Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2017-01-17

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:58 cr0wrx joined #fuel
01:11 fatdragon joined #fuel
01:36 astnbomb joined #fuel
01:44 mdnadeem joined #fuel
02:38 fandi joined #fuel
02:48 ilbot3 joined #fuel
02:48 Topic for #fuel is now Fuel 9.0 (Mitaka) https://www.fuel-infra.org/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
03:13 jose-phillips joined #fuel
03:15 jose-phillips joined #fuel
03:18 jose-phillips joined #fuel
03:20 jose-phillips joined #fuel
03:22 jose-phillips joined #fuel
03:25 astnbomb joined #fuel
03:56 mdnadeem joined #fuel
04:11 fatdragon joined #fuel
05:12 fatdragon joined #fuel
06:12 fatdragon joined #fuel
06:18 Sketch joined #fuel
06:28 jianghuaw omolchanov__, could you suggest how to trim the diagnostic snapshot down? It's huge - 3GB.
06:28 jianghuaw That's for this issue: https://bugs.launchpad.net/fuel/+bug/1656773
07:13 fatdragon joined #fuel
08:04 dmellado joined #fuel
08:12 DavidRama joined #fuel
09:15 fatdragon joined #fuel
10:15 jose-phillips joined #fuel
10:16 fatdragon joined #fuel
11:16 fatdragon joined #fuel
11:41 jose-phillips joined #fuel
12:17 fatdragon joined #fuel
12:34 AlexAvadanii joined #fuel
12:57 wessel joined #fuel
12:58 wessel Hi! I have a issue starting my Openstack msyql cluster
12:59 wessel Is someone here able to help me out?
13:18 fatdragon joined #fuel
14:19 fatdragon joined #fuel
15:01 acormier joined #fuel
15:03 Julien-zte joined #fuel
15:08 kur1j joined #fuel
15:17 acormier joined #fuel
15:18 acormier joined #fuel
15:36 acormier joined #fuel
16:05 fatdragon joined #fuel
16:16 johnavp19891 joined #fuel
16:20 thiagolib Hi I am also having the same problem I can not start the mysql cluster in my Mirantis Opentack 9 environment.
16:20 thiagolib I have 3 comtroller nodes, 2 compute nodes and 2 cinder nodes.
16:23 kur1j joined #fuel
16:25 kur1j how cna I adjust the /usr/bin/kvm command that starts up a VM? I need to add the kvm=off flag
16:37 Julien-zte joined #fuel
16:38 Julien-zte joined #fuel
16:39 Julien-zte joined #fuel
16:40 Julien-zte joined #fuel
16:40 mikese joined #fuel
16:41 Julien-zte joined #fuel
16:41 mikese Hey anyone here to help me out with something I'm confused with real quick?
16:42 Julien-zte joined #fuel
17:09 cr0wrx joined #fuel
17:15 raunak joined #fuel
17:20 cr0wrx anyone here familiar with partitioning in fuel and/or how ceph OSDs are handled
17:39 chopmann joined #fuel
17:44 evgenyl cr0wrx: Do you have a more specific question about partitioning?
17:47 cr0wrx evgenyl: A couple. I noticed each OSD disk has bios_grub and UEFI partitions on them - not sure if that's important, not important, just side affect of fuel and how it defaults to setting up partitions, etc....If I manually add more OSD disks later after initial fuel deploy I just want to be sure I won't hit issues if I don't add those
17:48 cr0wrx evgenyl: Also, I noticed that while I set partitions to be one size in fuel UI (50GB for journal), I noticed the actual partitions that were created for journal were 10-11GB in size
17:51 evgenyl evgenyl: Absence of bios_grub partition on new disks should not cause any problems.
17:51 evgenyl cr0wrx: The second issue is more interesting...
17:53 evgenyl cr0wrx: So you allocated a partition of size 50GB on a single disk and it's 10GB? Are you sure that you haven't done any changes with HW (like adding/removing disks)? It could cause reset of partitioning to default schema.
17:53 cr0wrx evgenyl: Yup this doesn't have any additional disks involved.
17:55 cr0wrx evgenyl: I can also see in /etc/ceph-osd.yaml some deployment related stuff and I see volumes that line up with partitions (the first two boot related, cephjournal, and ceph), except cephjournal has size: 53112 in the yaml but on disk via parted -l it's only 10.7GB
17:57 evgenyl cr0wrx: Could you please run `fuel provisioning --default --env YOUR_ENV_ID_HERE --node-id YOUR_CEPH_NODE_ID_HERE` which should generate a partitioning schema in a file, paste it to http://paste.openstack.org/ and send me a link?
18:12 cr0wrx evgenyl: paste here http://paste.openstack.org/show/595252/
18:16 evgenyl cr0wrx: That is interesting, it's indeed 53112MB on each disk (also, just for your information, it's advised to allocate cephjournal on separate disk, which is usually ssd). Ok, could you please send output of `lsblk` from the node?
18:21 cr0wrx evgenyl: here it is http://paste.openstack.org/show/595255/
18:21 cr0wrx ignore sde / sdf - those are additional disks (one I set up as a new OSD, the other still unallocated)
18:29 cr0wrx evgenyl: Thanks for note about cephjournal on separate disk. I've definitely read about running journals on SSDs but our cluster is small enough that I've been hesitant because losing an SSD holding journals for several OSDs means we lose the OSDs correct? And our cluster is pretty small currently so I'd prefer to spread them out over each disk and reduce failure points a bit (although I know it effectively cuts performance down a bit)
18:39 evgenyl скcr0wrx Usually it's done due to performance reasons. If it's not an issue for you, then it's fine.
18:40 evgenyl cr0wrx: I assume the reason you have the journal not fully allocated, is there is somewhere hardcode in puppet manifests. Trying to find it.
18:41 cr0wrx evgenyl: it would be nice to be faster, but since I'm new to ceph and we have limited OSDs I'm more concerned on stability. I figure if our cluster grows larger and OSD count grows enough so losing several at once wouldn't be a problem then we can switch everything to SSD journals
18:42 kur1j when you click "start instance" what script is called to start the instance /usr/bin/kvm ?
18:51 miroslav_ joined #fuel
18:53 miroslav_ cr0wrx: If you do not use separate disks for Ceph OSD journals - you do not need to create journal partitions at all.
18:54 miroslav_ cr0wrx: In such case Ceph uses journal file located directly on the OSD partition.
18:55 cr0wrx miroslav_: My understanding is doing that is even worse performance than separate partition on same disk due to overhead of filesystem related stuff that applies if journal is a file on OSD disk (vs raw partition for journal)
18:59 miroslav_ cr0wrx: About the journal sizes. If journal is located on spindel HDD - 2GB would be sufficient. If journal is located on SSD - for SSD with write speed about 0.5 GB/s journal size about 15 GB is sufficient. The simplified rule of thumb is journal size should provide 30 seconds of continuous write at full device speed.
19:03 Reepicheep joined #fuel
19:04 miroslav_ cr0wrx: significant write speed degradation may happen only when you re-create jounrl file and journal location on the disk is quite heavily fragmented. For the new OSD journal location is not fragmented and provides very similar write speed as for journal on the separate partition. However - separate partition for journal is OK, you only do not need 50 GB journal size,  it is too big.
19:06 cr0wrx miroslav_: I figured journal of 50gb may have had extra space but wasn't sure and didn't go through the math. If a 2GB journal is sufficient for HDD (I see that's the default in ceph.conf) I can run with that
19:07 cr0wrx HDD is like 110MB/s so sounds like 3GB journal would probably be better though? idk, I don't mind giving it a couple gigs extra
19:09 miroslav_ cr0wrx: journal partition size about 6 GB is optimal for the modern 4+ GB in size HDDs,  which provide write speed up to 200 MB/s. For 110MB/s - yes, 3GB is optimal.
19:14 jose-phillips joined #fuel
19:15 cr0wrx miroslav_: 6gb it is then. Thanks.
19:17 cr0wrx miroslav_: So my understanding is you would typically also only put something like 4 journals per SSD if using SSD - does that mean environments just use something like 64gb SSD disks or 128gb or so for journal? Something on the smaller size - any more than that seems wasted. I'm not sure if 64gb is readily available but 128gb is still relatively easy to find
19:19 evgenyl miroslav_: Have you heard about hardcodes of partitions sizes for ceph journal?
19:19 evgenyl miroslav_: I'm trying to find why cr0wrx env has cephjournal of size 50, but created partition is 10..
19:20 evgenyl Oh...
19:20 evgenyl That is bad.
19:20 evgenyl https://github.com/openstack/fuel-agent/blob/stable/mitaka/fuel_agent/drivers/nailgun.py#L373-L382
19:21 evgenyl cr0wrx: Basically you cannot set cephjournal of size more than 10G...
19:22 cr0wrx evgenyl: that code seems to explain it :P
19:22 cr0wrx and seems like it would be fine for HDD, but maybe not for SSD journal based on miroslav_ advice
19:22 cr0wrx I'm on liberty for what it's worth, but probably same code
19:25 miroslav_ It was fine for SAS/SATA SSDs. But may be not enough for modern NVMe SSDs with 3GB/s write speed.
19:25 cr0wrx evgenyl: this also seems to answer another question I had - how partitioning across journal disks works. It seems like you only need to create a journal disk and set entire disk to ceph journal in fuel, and it'll partition automatically for you and use for the OSD disks? But then how does that apply if you have 20 OSD disks and only set 1 journal disk - will it try to cram 20 journal partitions onto single journal disk?
19:27 kur1j Is there an easy way to add kvm=off?
19:27 thiagolib evgenyl: I'm having trouble with fuel 9.0 to mount the cluster with 3 controller 2 cinder nodes and 2 compute nodes. When it will mount the mysql cluster it breaks down by putting the following error
19:27 thiagolib (/ Stage [main] / Cluster :: Mysql / Exec [wait-initial-sync]) /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute '
19:27 thiagolib Have you seen this error?
19:28 miroslav_ cr0wrx: evgenyl: If I remember correct - you may set only single journal partition (or journal space) per disk drive and Fuel cuts the necessary number 10 GB each for each journal partition.
19:30 miroslav_ thiagolib: Do you mean MySQL on all controllers stuck in this state? And what is DB size?
19:30 cr0wrx miroslav_: I guess the way the UI works you can only add a single journal partition per disk. But does it do anything to limit how many journals are applied to a single journal disk? If I have 10 OSDs and 1 journal disk at 128gb, it seems like it would put 10 journals on one disk. If that's the case, assuming my 10 OSDs are all my OSDs that would be a single point of failure as journal disk failure means everything is lost?
19:31 evgenyl cr0wrx: I'm not a Ceph expert, but reading the code it looks like for each journal is used for all OSDs https://github.com/openstack/fuel-library/blob/stable/mitaka/deployment/puppet/ceph/lib/facter/ceph_osd.rb#L47
19:32 evgenyl But again, it's better to read the docs on how journaling exactly works.
19:34 miroslav_ cr0wrx: Not sure about the built-in upper limit for the number of journals per device. And yes, if you have only 10 OSDs the best option is not use the separate disk for journals.
19:34 cr0wrx evgenyl: Sure. I'm not a ceph expert either, but my understanding is you need 1 journal per OSD and if journal is lost then OSD is lost. Thus if you have all your journals on single disk as in above example, a disk failure would break all OSDs which would not be a good thing (single point of failure)
19:36 cr0wrx miroslav_: thanks. I know some of this is hypothetical and probably not important for most large deployments. But for smaller poc / lab environments or people just getting started (like us), a couple storage nodes with a handful of drives is where we started, so some of this does affect our decision process
19:36 cr0wrx I wouldn't complain if I had another 100 drives though :P
19:39 miroslav_ cr0wrx: If all journals were lost starting from Ceph 0.80.10 you still have the long in time and painful possibility to restore the most recent data replica,  with complete loss of every data written into journals - say the data written in last minute before journal disk was lost.
19:39 kur1j where is the libvirt template for nova?
19:42 miroslav_ kur1j: May be you need this http://docs.openstack.org/kilo/config-reference/content/hypervisor-configuration-basics.html
19:43 kur1j miroslav_: it doesn't have the kvm=off flag unfortunately
19:50 miroslav_ kur1j: At least one hypervisor type should be configured per compute node. If you only need to switch off a compute node and prevent scheduler from assigning new VMs on that node  you may use command like following: http://docs.openstack.org/admin-guide/cli-nova-manage-services.html
19:51 miroslav_ kur1j: Or you may use host aggregates http://docs.openstack.org/admin-guide/dashboard-manage-host-aggregates.html
19:52 kur1j miroslav_: huh? this has nothing to do with what I'm trying to do
19:54 miroslav_ kur1j: And what is your final goal in disabling KVM?
19:54 kur1j miroslav_: the kvm=off flag has nothing to do with turning a node on or off, it has to do with disabling the ability for the guest VM to tell it is running in a guest VM.
19:56 kur1j miroslav_: long story short, I'm trying to get PCI Passthrough working with a consumer grade NVidia GPU. The issue is the driver checks to see if it is running in a VM, if it is, it fails. If you pass kvm=off it is supposed to keep the GPU driver from being to tell it is in a VM
19:57 kur1j sorry for the confusion
20:02 miroslav_ kur1j: May be this guide help you: https://gist.github.com/claudiok/890ab6dfe76fa45b30081e58038a9215
20:02 kur1j miroslav_: I did that and it doesn't work unfortunately.
20:03 miroslav_ You need to switch virt_type from default KVM to qemu_kvm in nova.conf
20:05 miroslav_ kur1j: like here http://docs.openstack.org/kilo/config-reference/content/qemu.html
20:06 acormier joined #fuel
20:07 kur1j miroslav_: let me try that
20:09 kur1j so looking at my instances logs I don't see -cpu host,kvm=off :-/
20:09 kur1j should I?
20:11 cr0wrx any of you know what I should look into if I want a different volume group in openstack for SSDs and separate for HDD? From my understanding fuel deployment will just set up a single group that has all OSDs in it. If I manually add additional OSDs on SSDs, I imagine I'll need to manually add ceph pools and change some configurations in cinder.conf?
20:14 Sketch cr0wrx: my understanding is you need ceph 1.0 for that, which is not yet in the release version of fuel (it might be in 10.0?)
20:14 cr0wrx maybe volume group wrong word. volume type I guess (to create an SSD volume type backed by same ceph cluster, but specific disks)
20:15 miroslav_ cr0wrx: Ceph RBD operates by RBD pools - you may need to create 2 pools and 2 Cinder backends - one per pool.
20:18 cr0wrx yea I see in cinder.conf it has rbd_pool=volumes, so I was thinking I could just manually create a new pool in ceph (and then figure out how to specify which OSDs handle which pools, but it's probably doable), and then add second cinder backend using the second pool
20:18 cr0wrx it's quite possible I'm missing a step (and probably an important one), but hopefully I can figure it out
20:20 kur1j miroslav_: should there be a different mode of operation in this now?
20:20 kur1j miroslav_: im not getting anything different (I restarted nova-compute)
20:21 cr0wrx https://www.sebastien-han.fr/blog/2013/04/25/ceph-and-cinder-multi-backend/ seems to have the openstack specific stuff in it. Just gotta set up pools for SSD with ceph
20:25 miroslav_ kur1j: both kvm and qemu are managed by libvirt. And kvm is part of qemu-kvm. Setting virt_type to kvm actually switches off qemu hardware emulation layer and some other features.
20:26 kur1j miroslav_: well in either case, I don't see the flag being passed
20:29 cr0wrx https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/ and there is the ceph piece :P This blog has it all for me it seems.
20:30 cr0wrx so the only thing I wonder is this - if I manually update and modify ceph related stuff like OSDs and crushmap and etc. - will fuel have an issue with it? I know I've manually adjusted some other things post-deployment and fuel doesn't like it and breaks when adding additional controllers or compute nodes, for instance
20:33 miroslav_ kur1j: Well, there is another way. Please stop your VM and edit its XML definition, usually located in /var/lib/nova/instances/<instance ID>/libvirt.xml
20:34 miroslav_ Add <kvm>
20:35 miroslav_ <hidden state='on'/> </kvm>
20:35 miroslav_ Like here https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Complete_example_for_QEMU_with_libvirtd
20:35 miroslav_ And restart your VM
20:36 kur1j miroslav_: I tried that and the file gets overwritten
21:01 miroslav_ kur1j: Well, libvirt.xml is generated by driver.py and config.py located here /usr/lib/python2.7/dist-packages/nova/virt/libvirt/
21:02 miroslav_ kur1j: However, I cannot find kvm hidden option
21:03 kur1j i love how NONE of this shit works
21:03 kur1j (not your fault)
21:05 kur1j miroslav_: Yeah, I think I'll have to add it
21:12 miroslav_ kur1j: Yes, looks like this feature should go here, https://github.com/openstack/nova/blob/stable/mitaka/nova/virt/libvirt/config.py#L1938
21:13 miroslav_ kur1j: But I am far from being a great expert in Nova code.(
21:21 acormier joined #fuel
21:21 chopmann joined #fuel
21:25 kur1j miroslav_: so why would "-S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilties.pidfile -daemonize" be passed to qemu-system-x86_64 instead of -name instance instance-00001b8 -S -machine pc-i44fx-vivid,accel=kvm,usb=off -cpu Haswell-noTSX,...
22:17 ipsecguy_ joined #fuel
22:23 cr0wrx anyone here know feasibility of changing storage backend for ephemeral volumes (nova) to ceph rbd post deployment? When I deployed cluster I used ceph for everything except nova, but I'd like to back nova with ceph as well now
23:10 gardlt joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary