Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-01-19

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:51 emagana joined #fuel
00:54 emagana joined #fuel
02:47 ilbot3 joined #fuel
02:47 Topic for #fuel is now Fuel 5.1.1 (Icehouse) and Fuel 6.0 (Juno) https://software.mirantis.com | Fuel for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
03:35 benrodrigue joined #fuel
04:15 Longgeek joined #fuel
04:46 akas joined #fuel
05:34 emagana joined #fuel
06:31 rongze joined #fuel
06:32 dklepikov joined #fuel
06:39 benrodrigue joined #fuel
06:47 sambork joined #fuel
07:05 Miouge joined #fuel
07:06 monester_laptop joined #fuel
07:28 saibarspeis joined #fuel
07:47 Krast joined #fuel
07:50 sambork joined #fuel
08:02 e0ne joined #fuel
08:33 sambork joined #fuel
08:34 hyperbaba_ joined #fuel
08:40 avgoor joined #fuel
08:40 corepb_ joined #fuel
08:53 robklg joined #fuel
08:59 fandi joined #fuel
09:05 stamak joined #fuel
09:12 devstok joined #fuel
09:25 monester_laptop joined #fuel
09:43 saibarspeis joined #fuel
09:45 azemlyanov joined #fuel
09:48 kivio joined #fuel
09:52 mescanef joined #fuel
09:54 ddmitriev joined #fuel
10:16 benrodrigue joined #fuel
10:18 dkusidlo joined #fuel
10:23 e0ne joined #fuel
10:27 ChrisNBlum joined #fuel
10:53 e0ne joined #fuel
11:00 pal_bth joined #fuel
11:28 sambork joined #fuel
11:56 corepb_ joined #fuel
12:05 saibarspeis joined #fuel
12:13 dkusidlo joined #fuel
12:29 corepb_ joined #fuel
12:40 ChrisNBlum joined #fuel
12:43 ChrisNBlum joined #fuel
12:44 ChrisNBlum joined #fuel
12:47 ChrisNBlum joined #fuel
12:51 ChrisNBlum joined #fuel
12:53 ChrisNBlum joined #fuel
12:56 ChrisNBlum joined #fuel
12:57 sambork joined #fuel
13:00 ChrisNBlum joined #fuel
13:03 ChrisNBlum joined #fuel
13:06 e0ne joined #fuel
13:06 ChrisNBlum joined #fuel
13:09 ChrisNBlum joined #fuel
13:13 ChrisNBlum joined #fuel
13:14 sambork joined #fuel
13:16 ChrisNBlum joined #fuel
13:19 ChrisNBlum joined #fuel
13:22 ChrisNBlum joined #fuel
13:25 ChrisNBlum joined #fuel
13:28 ChrisNBlum joined #fuel
13:31 ChrisNBlum joined #fuel
13:34 ChrisNBlum joined #fuel
13:37 fandi joined #fuel
13:37 ChrisNBlum joined #fuel
13:40 ChrisNBlum joined #fuel
13:46 ChrisNBlum joined #fuel
13:49 ChrisNBlum joined #fuel
13:52 ChrisNBlum joined #fuel
13:55 ChrisNBlum joined #fuel
13:58 ChrisNBlum joined #fuel
13:59 dkusidlo joined #fuel
14:01 ChrisNBlum joined #fuel
14:04 ChrisNBlum joined #fuel
14:07 ChrisNBlum joined #fuel
14:10 ChrisNBlum joined #fuel
14:13 ChrisNBlum joined #fuel
14:16 ChrisNBlum joined #fuel
14:19 ChrisNBlum joined #fuel
14:31 e0ne joined #fuel
14:34 benrodrigue joined #fuel
14:35 pal_bth joined #fuel
14:39 mstrukov joined #fuel
14:39 dkusidlo joined #fuel
14:44 mstrukov joined #fuel
14:59 miroslav_ joined #fuel
15:08 mpetason joined #fuel
15:11 rmoe joined #fuel
15:12 corepb_ joined #fuel
15:23 jobewan joined #fuel
15:33 blahRus joined #fuel
15:54 e0ne joined #fuel
16:08 jaypipes joined #fuel
16:19 agordeev joined #fuel
16:28 emagana joined #fuel
16:33 e0ne joined #fuel
16:38 jaypipes_ joined #fuel
16:40 emagana joined #fuel
16:42 angdraug joined #fuel
16:46 mattgriffin joined #fuel
16:54 ChrisNBlum joined #fuel
16:55 emagana joined #fuel
16:57 ChrisNBlum joined #fuel
17:00 ChrisNBlum joined #fuel
17:01 emagana joined #fuel
17:03 ChrisNBlum joined #fuel
17:06 ChrisNBlum joined #fuel
17:09 rmoe joined #fuel
17:09 ChrisNBlum joined #fuel
17:12 ChrisNBlum joined #fuel
17:15 ChrisNBlum joined #fuel
17:16 championofcyrod1 left #fuel
17:18 ChrisNBlum joined #fuel
17:21 ChrisNBlum joined #fuel
17:22 championofcyrod1 joined #fuel
17:22 championofcyrod1 rabbitmq handshake_timeout started happening on fuel 5.1 (icehouse) controllers.  looks like the built in time out is 3 seconds and is compiled w/ rabbitmq.
17:23 championofcyrod1 but could be a config var in rabbitmq 3.3.5... thoughts anyone?
17:24 championofcyrod1 actually.. it looks like it's just 1 node.
17:24 championofcyrod1 (controller)
17:24 championofcyrod1 i'm just going to bounce the controller with a reboot.
17:25 ChrisNBlum joined #fuel
17:28 ChrisNBlum joined #fuel
17:31 ChrisNBlum joined #fuel
17:34 ChrisNBlum joined #fuel
17:35 jaypipes joined #fuel
17:36 ChrisNBlum joined #fuel
17:39 ChrisNBlum joined #fuel
17:42 ChrisNBlum joined #fuel
17:45 ChrisNBlum joined #fuel
17:48 ChrisNBlum joined #fuel
17:51 ChrisNBlum joined #fuel
17:54 ChrisNBlum joined #fuel
17:56 xarses joined #fuel
18:00 ChrisNBlum joined #fuel
18:03 ChrisNBlum joined #fuel
18:06 miroslav_ joined #fuel
18:06 ChrisNBlum joined #fuel
18:09 ChrisNBlum joined #fuel
18:09 championofcyrod1 well.. looks like the OSFTs got a little further, but ultimately failed... some of the rabbitmq stuff is getting through, and then sometimes it times out.
18:10 championofcyrod1 setting system swappiness to 0
18:10 championofcyrod1 and turning swapoff
18:12 * championofcyrod1 remembers with zeromq+zookeeper+kafka that swappiness breaks j2ee message queue timeouts.
18:12 championofcyrod1 maybe related?
18:12 ChrisNBlum joined #fuel
18:13 championofcyrod1 WOOHOO!!!!
18:13 championofcyrod1 health tests are working again.
18:14 championofcyrod1 so system swappiness of 60 (centos default) was causing rabbit to swap, even though there was still allocated RAM in buffer
18:15 ChrisNBlum joined #fuel
18:16 MiroslavAnashkin Hmm. So far we observed only Ceph OSD related-issues if swap is turned on and no swap partition defined...
18:18 ChrisNBlum joined #fuel
18:21 ChrisNBlum joined #fuel
18:24 ChrisNBlum joined #fuel
18:25 championofcyrod1 it sort of depends on how much ram, and how many services are running.
18:26 championofcyrod1 but it's definitely the issue.  i've been fighting it for about 4 days now...
18:26 championofcyrod1 and i just did this: http://championofcyrodiil.blogspot.com/2015/01/rabbitmq-handshaketimeout.html
18:26 championofcyrod1 and now everything is working again.  another indicator was that the health checks were failing at different spots
18:26 championofcyrod1 e.g. not always the same step
18:27 championofcyrod1 imo, centos 6's swappiness is a bit high anyway
18:27 ChrisNBlum joined #fuel
18:27 championofcyrod1 the JVMs/containers allocate so much heap/ram, that the OS thinks it should start swapping... even though the buffers are only half used.
18:30 ChrisNBlum joined #fuel
18:32 MiroslavAnashkin championofcyrod1: Thank you very much for figuring it out!
18:34 ChrisNBlum joined #fuel
18:34 championofcyrod1 MiroslavAnashkin: and thank you for your help in the past.  Apache Zookeeper has similar issues, so once i understood what openstack was trying to do, i was able to make further considerations about the environment.
18:35 championofcyrod1 rather, zookeeper doesnt have issues, but the services that use them can often be misconfigured in regard to swap.
18:35 championofcyrod1 lunch time!
18:37 ChrisNBlum joined #fuel
18:37 MiroslavAnashkin It may be due to NUMA. There is enough RAM, but not enough in that NUMA address space, which includes RabbitMQ or the process, going to swap.
18:39 * championofcyrod1 will have to do some reading on NUMA
18:40 ChrisNBlum joined #fuel
18:43 ChrisNBlum joined #fuel
18:45 ChrisNBlum joined #fuel
18:48 ChrisNBlum joined #fuel
18:51 ChrisNBlum joined #fuel
18:54 ChrisNBlum joined #fuel
18:58 ChrisNBlum joined #fuel
19:00 ChrisNBlum joined #fuel
19:07 ChrisNBlum joined #fuel
19:10 ChrisNBlum joined #fuel
19:13 ChrisNBlum joined #fuel
19:15 MarkDude joined #fuel
19:15 MarkDude joined #fuel
19:16 ChrisNBlum joined #fuel
19:17 corepb_ joined #fuel
19:19 ChrisNBlum joined #fuel
19:22 ChrisNBlum joined #fuel
19:25 ChrisNBlum joined #fuel
19:28 ChrisNBlum joined #fuel
19:28 monester_laptop joined #fuel
19:30 xarses joined #fuel
19:31 ChrisNBlum joined #fuel
19:33 ChrisNBlum joined #fuel
19:34 obcecado joined #fuel
19:36 ChrisNBlum joined #fuel
19:39 ChrisNBlum joined #fuel
19:42 ChrisNBlum joined #fuel
19:45 e0ne joined #fuel
19:45 ChrisNBlum joined #fuel
19:48 ChrisNBlum joined #fuel
19:51 ChrisNBlum joined #fuel
19:54 ChrisNBlum joined #fuel
19:57 ChrisNBlum joined #fuel
20:05 monester_laptop joined #fuel
20:50 jaypipes joined #fuel
20:57 benrodrigue joined #fuel
21:07 iunruh joined #fuel
21:10 Akshik joined #fuel
21:11 Akshik hi, does the Test-VM has cloud-init installed
21:11 Akshik how do i check the same
21:31 Akshik joined #fuel
21:40 benrodrigue joined #fuel
21:43 jaypipes joined #fuel
21:46 mpetason Cloud-init based instances should have /etc/cloud.cfg or a configration file in etc for cloud init. Most OpenStack built linux images, by vendors, include cloud-init.
21:48 xarses Akshik: the TestVM image is a cirros image and has cloud-init
21:49 Akshik xarses, thanks
21:50 xarses the default user:pass is cirros:cubswin:)
21:50 Akshik but i tried executing a user data script and it did not reflect
21:51 championofcyrod1 okay.. i've see this before... but it looks like the $HOME/.ssh/known_hosts isnt being read when ssh -vvv, only the /etc/ssh/known_hosts:
21:51 championofcyrod1 Unexpected error while running command. Command: ssh 10.20.0.9 mkdir -p /var/lib/nova/instances/2006d7db-d18e-4390-ae1c-40dd77644853 Exit code: 1 Stdout: '' Stderr: "Warning: Permanently added '10.20.0.9' (RSA) to the list of known hosts.\r\n"
21:51 championofcyrod1 trying to do a resize
21:51 championofcyrod1 i'm wondering if this has to do with some kind of inconsistent state from rabbitmq being flaky the last time it was attempted.
21:52 championofcyrod1 should have stricthostchecking set to no for host *
21:54 xarses championofcyrod1: the known hosts warning is normal when stricthostkeychecking is no and userknownhostsfile is /dev/null
21:54 xarses this is from fuel node, but the deployed node will be similar http://paste.openstack.org/show/158981/
21:54 Akshik mpetason, i could not find /etc/cloud.cfg in the vm
21:57 championofcyrod1 xarses: thanks.  I thought 'no' meant that the known hosts file was ignored.
21:57 xarses no, it means it doesn't have to be in the file
21:57 championofcyrod1 wait... yes that is what i see... what user account is used?
21:58 championofcyrod1 from the horizon dashboard when the call is made? root?
21:58 Akshik xarses, is there a way to test the same, also can you point me to some cloud-init installed image repo
21:58 championofcyrod1 (what OS user account)
21:58 xarses if you get a duplicate, it will cause validation errors, hence why userknownhostsfile is /dev/null so it's allways empty
21:59 xarses Akshik: sorry, no I don't have any off-hand
22:00 xarses Akshik: the cirros image that we bundle works in the CI tests and should work properly if the metadata service is functioning
22:01 Akshik xarses, so should i check my metadata services?
22:03 xarses Akshik: if it's not getting it's IP data automaticly, check DHCP and metadata
22:04 xarses championofcyrod1: if its in the nova service, it would be the nova user
22:05 Akshik xarses, im able to get ip
22:05 xarses then what part of cloud-init isn't working?
22:06 xarses championofcyrod1: are you using ceph?
22:06 Akshik im just trying to work with cloud-init like changing the host name installing few packages and things like that, the first basic runcmd: with echo did not work
22:07 championofcyrod1 xarses: well horizon ui seems to mostly do rest api calls... so i was surprised when i saw the SSH error as a response. seems like anything 'instance' related is nova api call related, whereas anything 'volume' related is cinder api call.
22:07 championofcyrod1 since the command i issued was a resize, wasnt sure which ssh call was failing
22:07 championofcyrod1 heh... nvm.. i guess the mkdir /var/lib/nova should have gave it away
22:08 championofcyrod1 alright, gotta run.  thanks a bunch xarses, i'll see what i can debug in the morning.
22:08 xarses championofcyrod1: the nova-compute service that is the source hosts will ssh into the destination node of the resize
22:08 xarses if you are using ceph, the code is broken
22:08 championofcyrod1 i'm using ceph
22:08 championofcyrod1 but a resize was successful for me about 2 months ago
22:09 Akshik xarses, this is what i passed as the post creation custom script http://paste.openstack.org/show/158985/
22:09 championofcyrod1 maybe i didnt resize the volume though... i am curious about 'flavor' resize in the context of a preserve after shutdown volume that is created.
22:09 championofcyrod1 in that i can set a custom volume size of say, 40GB that persists after shutdown, even if using a flavor that specifies a 100GB volume
22:10 championofcyrod1 when launch instance, with boot source: "Boot from image (creates a new volume)"
22:11 xarses championofcyrod1: I'm not sure about that, your moving from a flavor with a 40Gb volume, and then resize to a 100Gb?
22:11 championofcyrod1 I have an instance w/ 8GB RAM and a volume i specified to be like 20GB
22:11 championofcyrod1 the user requested 16GB
22:11 championofcyrod1 when resizing to the xlarge flavor, i'm curious about what happens to my volume.
22:12 championofcyrod1 when it was originally created with a custom size
22:12 championofcyrod1 for instance if i made it 300GB, which is larger than the xlarge flavor's disk
22:12 xarses if the image is larger than the flavor allows, then it will fail to spawn
22:13 championofcyrod1 so just always ensure i specify a flavor that has the same disk or larger, if i only want to increase ram>
22:13 xarses I'm not sure how that relates to custom volume size, but I would guess it's treated the same
22:13 championofcyrod1 i tried virsh from a controller... but the domain XML change didnt seem to matter to openstack when i rebooted the VM
22:13 xarses usually, you can also set params on the image so that it can only show flavors that will be big enough for it
22:14 championofcyrod1 (tried virsh to increase just the RAM)
22:14 championofcyrod1 but since mysql seems to hold the instance data, wasnt sure about what else is interconnected.
22:14 xarses that would probably mess with scheduling down the road
22:14 championofcyrod1 (the instance data reported on horizon)
22:14 championofcyrod1 yea
22:15 championofcyrod1 ultimately, i would just like to be able to scale ram, without touching volume information
22:15 xarses as long as you don't do something that would rebuild the libvirt data, that might work, but seems heavy handed
22:15 championofcyrod1 which virsh allows in libvirt
22:16 championofcyrod1 also would like to increase the vram to support some of the larger cirrus hd display modes, for the instances that are just running as NX forwarded desktops.
22:16 championofcyrod1 the only other option is something called SPICE and another plugin that i'm not sure how to integrate w/ openstack.
22:17 championofcyrod1 alright, really gotta run.  thanks for your help.  I'll be back in the morning.
22:17 xarses afaik the SPICE dispaly works properly in openstack
22:18 championofcyrod1 hmm... i'll have to give that a shot.  My openstack was quickly embraced by ~20 developers, so i have to be careful when making modifications.  Perhaps I'll try out devstack as a testing ground.
22:36 Akshik joined #fuel
22:58 e0ne joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary