Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-06-05

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 Kupo24z1 xarses: would i get much better performance out of ceph in ubuntu vs centos then?
00:06 xarses Kupo24z1: I would suspect better performance just because of the newer kernel
00:07 crandquist joined #fuel
00:13 jaypipes joined #fuel
00:47 rmoe joined #fuel
00:51 Kupo24z1 fresh install on ubuntu, dashboard 404's :(
01:03 crandquist joined #fuel
01:20 rmoe joined #fuel
01:31 xarses joined #fuel
01:32 vogelc joined #fuel
01:32 vogelc is there a way to increase the amount of memory available to each docker instance?
01:35 vogelc Also is there a know issue with deploying more than 4 CEPH nodes at a time?
03:16 crandquist joined #fuel
06:29 al_ex joined #fuel
06:56 artem_panchenko joined #fuel
07:07 dpyzhov joined #fuel
07:07 taj joined #fuel
07:40 odyssey4me joined #fuel
07:56 izinovik joined #fuel
07:57 pbrooko joined #fuel
08:06 e0ne joined #fuel
08:45 nikalgiv joined #fuel
09:08 dnikishov joined #fuel
09:08 dnikishov left #fuel
09:21 e0ne joined #fuel
09:39 meow-nofer__ joined #fuel
09:55 mrasskazov joined #fuel
09:56 evg vogelc: memory is not restricted, as i see. Run "docker inspect _instance_".
10:16 mattymo evg, vogelc is not online
10:16 mattymo but we don't set any docker container limits currently. if a particular component is breaking, we can work on it
10:31 meow-nofer__ joined #fuel
10:54 e0ne_ joined #fuel
12:50 crandquist joined #fuel
13:22 thehybridtech joined #fuel
13:23 izinovik joined #fuel
13:24 book` joined #fuel
13:41 jobewan joined #fuel
14:14 e0ne joined #fuel
14:22 gajsin joined #fuel
14:45 jaypipes joined #fuel
14:45 rmoe joined #fuel
14:55 odyssey4me_ joined #fuel
14:56 jues joined #fuel
15:09 albionandrew joined #fuel
15:09 albionandrew Good morning, is there a new ISO that has the dhcprelay patch included or should I just do a patch? I would prefer to use an iso if possible.
15:23 mattymo albionandrew, which patch are you talking about?
15:23 xarses joined #fuel
15:25 albionandrew mattymo https://review.openstack.org/gitweb?p=stackforge/fuel-library.git;a=patch;h=b9ede803a2cab9a79c51f37bebf0755719394940
15:27 mattymo that's mine :) I don't have any externally available ISO for you, unfortunately. You may patch your system manually or build a new iso from the patchset
15:27 jerrygb joined #fuel
15:27 mattymo I did some basic testing with it and it seems to solve conditions where dhcrelay quits or doesn't start correctly
15:29 albionandrew mattymo OK I’ll try it now. Thanks
15:45 rmoe joined #fuel
15:45 vkozhukalov joined #fuel
15:46 christopheraedo joined #fuel
15:53 odyssey4me_ joined #fuel
15:57 xarses joined #fuel
16:04 odyssey4me joined #fuel
16:05 angdraug joined #fuel
16:16 albionandrew mattymo can you talk me through using this patch? I downloaded to /tmp/relay.patch I changed into /etc/puppet I then ran patch -p 3 < /tmp/relay.patch but get File to patch: when I enter docker/manifests/dockerctl.pp .....docker/manifests/dockerctl.pp: No such file or directory
16:16 albionandrew what am I doing wrong?
16:23 mattymo your base directory isn't /etc/puppet/modules
16:26 albionandrew mattymo thanks. I don’t do a lot of patching so that is newish to me thanks
16:36 vogelc joined #fuel
16:38 vogelc I've run into a new issue with 5.0.  I can deploy an environment but if I try to add more than 4 storage nodes the deployment stalls and never responds.  The only fix is to delete the environment.
16:39 vogelc has ayone else seen this?
16:43 vogelc I see this in the nailgun log: Cannot find admin interface for node return first interface
16:43 vogelc but the interface does exist.  I booted into bootstrap
16:45 albionandrew mattymo - Sorry I’m clearly being a moron but I ran the patch http://pastebin.com/xZCgkjXC but nothing has changed.. What else do I need to do?
16:56 albionandrew anyone ^
17:03 christopheraedo albionandrew: someone should weigh in momentarily - weekly fuel meeting on IRC just ended
17:03 albionandrew thanks
17:05 vkozhukalov left #fuel
17:18 xdeller_ joined #fuel
17:39 e0ne joined #fuel
18:08 MiroslavAnashkin albionandrew: Looks like you have patched already installed master node. But the patch affect scripts which install docker to master node. So, you have to apply this patch to source code, build new ISO from updated source and reinstall master node. OR - you have to reinstall docker with patched scripts
18:09 MiroslavAnashkin albionandrew: And I am not sure about the right way to reinstall docker.
18:09 albionandrew MiroslavAnashkin OK you just prempted my next question.
18:10 albionandrew How time consuming to build iso? Do you have a script?
18:11 MiroslavAnashkin No, building ISO gets harder and harder.
18:11 albionandrew If it is difficult when will the next patched ISO be available?
18:12 MiroslavAnashkin 5.0.1 is expected closer to the end of June
18:13 albionandrew Does Mirantis have instructions on building iso from source ? We cant wait until the end of june.
18:14 christopheraedo albionandrew: Start here: http://docs.mirantis.com/fuel-dev/develop/env.html
18:14 christopheraedo I'm working on updating the docs as well, one tiny patch is here: https://review.openstack.org/#/c/98003/
18:15 christopheraedo getting the fuel dev environment ready takes probably 20-40 minutes (depending on speed of internet connection), and then building the ISO takes anywhere from 30 minutes to .. much longer if you have a slow connection
18:15 MiroslavAnashkin Well, I see reinstalling Docker is not complex
18:16 christopheraedo that's probably faster than building a fresh iso - I would take a shot at that first (then fall back to building an ISO if you run into problems there).
18:17 albionandrew MiroslavAnashkin Do you have instructions?
18:17 Kupo24z1 xarses: you around? quick question
18:18 MiroslavAnashkin albionandrew: First, you have to reinstall dockerctl
18:18 albionandrew just from yum?
18:18 MiroslavAnashkin No, with Puppet
18:19 MiroslavAnashkin Let me check where to get the inputs
18:31 albionandrew MiroslavAnashkin We don’t use puppet here so other than knowing what it is I don’t know how to use - would something like this http://pastebin.com/V84rrpTy achive what you need me to do?
18:34 MiroslavAnashkin No, it require couple of input parameters
18:35 xarses Kupo24z1: pong
18:38 MiroslavAnashkin albionandrew: You may take /etc/puppet/modules/nailgun/examples/host-only.pp, temporary comment out everything, except the first 10 lines (variable declarations) and except 2 classes - docker::dockerctl and docker:
18:39 MiroslavAnashkin albionandrew: Then, please run `puppet apply -d -v /etc/puppet/modules/nailgun/examples/host-only.pp`
18:39 albionandrew Thanks. I put what you suggest on pastebin first. Just so we’re clear.
18:39 Kupo24z1 xarses: I've heard that writeback cache is now usable due to ext4/xfs 'barrier' being enabled. is this enabled by default in Fuel deployed nodes?
18:40 xarses Kupo24z1: no, we don't modify the cache method
18:40 MiroslavAnashkin albionandrew: And also do not comment out lines 13 and 14
18:40 albionandrew received
18:41 MiroslavAnashkin albionandrew: comment only "->" part of line 14
18:45 albionandrew MiroslavAnashkin should it look like http://pastebin.com/kjMGVVFV
18:46 albionandrew Not sure on the last bit on line 14
18:46 MiroslavAnashkin albionandrew: No, like this http://paste.openstack.org/show/82992/
18:47 MiroslavAnashkin You may create new file in the same directory with  /etc/puppet/modules/nailgun/examples/host-only.pp and apply it instead
18:53 e0ne joined #fuel
18:55 albionandrew MiroslavAnashkin: I took your file and ran http://pastebin.com/cH1YDLu0 it errored.
19:03 MiroslavAnashkin albionandrew: Please comment out temporary /etc/puppet/modules/docker/manifests/init.pp, line 55 (before  => Service['supervisord'],)
19:03 albionandrew will do
19:03 MiroslavAnashkin And try again
19:05 albionandrew MiroslavAnashkin: That looks better. dockerctl build all is running now
19:06 albionandrew puppet is the next thing on my list to learn
19:09 jaypipes joined #fuel
19:29 e0ne joined #fuel
19:49 albionandrew MiroslavAnashkin: The ‘dockerctl build all'  process is still running does that seem right?
19:51 albionandrew http://pastebin.com/He42i2mQ
19:51 MiroslavAnashkin It's default timeout is 2 hours
19:51 albionandrew so …? Kill it? wait ?
19:51 MiroslavAnashkin It takea ~30 minutes on SSD and at least 1 hour on HDD
19:51 albionandrew OK. Thanks
20:18 blahRus1 joined #fuel
20:46 tsduncan__ joined #fuel
21:17 xarses joined #fuel
21:26 vogelc_ joined #fuel
21:51 vogelc_ Has anyone seen nailgun lock up after adding nodes?
21:53 xarses vogelc_: Not my self, but I heard of someone that was having an issue when they added nodes to a cluster and the nailgun api.log was full of "CanNotFindInterface" tracebacks
21:53 xarses for 4.1 its in /var/log/nailgun/api.log
21:54 xarses for 5.0 its 'dockerctl shell nailgun'
21:54 xarses and then you can inspect the same file
21:55 vogelc_ xarses:  I am using 5.0 and see that it can't find the mgmt interface,  I reboot the client and it says it found it but then immediately starts saying it can't find it again.
21:56 vogelc_ '2014-06-05 17:07:20,615 DEBG 'nailgun' stdout output: 2014-06-05 17:07:20.614 INFO [7f3426bfd700] (node) Node '720-23 (6f:6e)' is back online  2014-06-05 17:07:20,617 DEBG 'nailgun' stdout output: 2014-06-05 17:07:20.617 INFO [7f3426bfd700] (notification) Notification: topic: discover message: Node '720-23 (6f:6e)' is back online  2014-06-05 17:07:28,797 DEBG 'nailgun' stdout output: 2014-06-05 17:07:28.797 WARNING [7f343682e700] (
22:14 albionandrew joined #fuel
22:16 albionandrew MiroslavAnashkin: The puppet command we talked about ran. I rebooted put I still can’t pxe boot.
22:43 Kupo24z1 Hmm
22:44 Kupo24z1 trying to live-migrate a boot-from-volume instance and it just returns nothing
22:45 xarses rmoe ^
22:45 xarses did you enable ephmeral ceph support?
22:46 rmoe that shouldn't matter for boot-from-volume
22:46 Kupo24z1 xarses: No did not
22:46 Kupo24z1 I*
22:46 Kupo24z1 nova-all error reads nova-nova.virt.libvirt.driver ERROR: Live Migration failure: internal error Attempt to migrate guest to the same host
22:46 rmoe what is the exact command you used?
22:47 Kupo24z1 thats from horizon, with CLI....
22:48 Kupo24z1 Much bigger, pastebin inc
22:48 Kupo24z1 http://pastebin.mozilla.org/5356992
22:48 Kupo24z1 command: nova --debug live-migration b65b4ce6-73cc-4268-8a25-9da6e6599c7b node-44
22:49 rmoe it looks like the previous failed migration didn't get cleaned up on the destination
22:49 Kupo24z1 this instance was spawned from horizon using the 'Boot from volume snapshot"
22:49 rmoe the instance path already exists so it fails
22:49 rmoe can you delete that path on the dest (it should be empty) and try it again?
22:51 Kupo24z1 Got this error after deleting the path and retrying; <179>Jun  5 22:50:42 node-47 nova-nova.virt.libvirt.driver ERROR: Live Migration failure: internal error Attempt to migrate guest to the same host 00000000-0000-0000-0000-00000000efbe
22:51 Kupo24z1 deleted 'rm -rf /var/lib/nova/instances/b65b4ce6-73cc-4268-8a25-9da6e6599c7b'
22:51 rmoe so you're trying to move a vm from node-47 to node-44 correct?
22:51 Kupo24z1 full debug from command; http://pastebin.mozilla.org/5356993
22:51 Kupo24z1 this command was node-47 to node-43 but yes
22:51 rmoe ok
22:54 blahRus1 Maybe ubiquity can get some free support ;)
22:57 angdraug rmoe: 5.0 nova is built based on rbd-live-migration-stable-havana-fedorapeople in my github
22:58 angdraug oh wait, that's 4.1
22:58 angdraug rbd-ephemeral-clone-stable-icehouse
22:58 angdraug ^ is for 5.0
23:13 Kupo24z1 rmoe: any clues?
23:16 rmoe I think it's a config problem, we only only configure libvirt migration settings when ceph ephemeral storage is enabled
23:22 rmoe try setting live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST on your compute nodes
23:39 Kupo24z1 should i jsut redeploy with ceph ephemeral? this is a test env so its no probelm
23:54 rmoe live migration will definitely work in that case

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary