Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-08-18

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:33 teran joined #fuel-dev
00:47 nzrf joined #fuel-dev
00:47 e0ne joined #fuel-dev
01:19 AlexF joined #fuel-dev
01:34 teran joined #fuel-dev
01:36 teran_ joined #fuel-dev
01:36 AlexF joined #fuel-dev
01:38 teran__ joined #fuel-dev
01:40 teran joined #fuel-dev
01:48 e0ne joined #fuel-dev
02:49 e0ne joined #fuel-dev
02:51 e0ne joined #fuel-dev
03:25 teran joined #fuel-dev
03:52 e0ne joined #fuel-dev
03:55 Krast joined #fuel-dev
04:52 e0ne joined #fuel-dev
04:56 teran joined #fuel-dev
04:59 HeOS joined #fuel-dev
05:32 enikanorov joined #fuel-dev
05:53 e0ne joined #fuel-dev
06:12 teran joined #fuel-dev
06:12 salmon_ joined #fuel-dev
06:13 dshulyak joined #fuel-dev
06:29 asledzinskiy joined #fuel-dev
06:37 pasquier-s_ joined #fuel-dev
06:47 prmtl joined #fuel-dev
07:00 HeOS joined #fuel-dev
07:10 teran joined #fuel-dev
07:13 tatyana joined #fuel-dev
07:22 shauno_ joined #fuel-dev
07:22 Longgeek joined #fuel-dev
07:28 skolekonov joined #fuel-dev
07:32 nurla joined #fuel-dev
07:37 EeeBotu New bug #1358144 in fuel: "[Upgrade] Upgrade failed with Cannot find running container with name "fuel-core-5.0-postgres"" [Critical, New] https://bugs.launchpad.net/bugs/1358144
07:37 EeeBotu New bug #1358140 in fuel: "[astute] Cirros image: stderr: sh: 1: source: not found" [Critical, In Progress] https://bugs.launchpad.net/bugs/1358140
07:42 Longgeek joined #fuel-dev
07:43 prmtl_ joined #fuel-dev
07:44 Longgeek joined #fuel-dev
07:45 Longgeek joined #fuel-dev
07:46 zynzel joined #fuel-dev
07:50 e0ne joined #fuel-dev
07:50 tuvenen joined #fuel-dev
07:51 AlexF joined #fuel-dev
07:58 Longgeek_ joined #fuel-dev
07:59 tuvenen Hi guys, review is needed here: https://review.openstack.org/#/c/107639/
08:03 IvanBerezovskiy joined #fuel-dev
08:06 dmitryme fuel library team: it would be nice if you review and merge https://review.openstack.org/#/c/112677
08:07 dmitryme it is the final part of HA fix
08:09 ddmitriev joined #fuel-dev
08:31 e0ne joined #fuel-dev
08:39 EeeBotu New bug #1358168 in fuel: "Fuel master backup doesn't save ssh key" [High, In Progress] https://bugs.launchpad.net/bugs/1358168
08:39 EeeBotu New bug #1358160 in fuel: "Broken URL http://docs.mirantis.com/openstack/fuel/fuel-5.0/pdf/Mirantis-OpenStack-5.0-RelNotes.pdf" [Medium, New] https://bugs.launchpad.net/bugs/1358160
08:46 ikalnitsky Hi All! We have some problems in our host-system puppets. https://bugs.launchpad.net/fuel/+bug/1358144 That's a blocker for fuel upgrade / fuel patching. Could someone take a look?
08:46 ikalnitsky mattymo: aglarendil: ^^
08:47 mattymo so, the new docker-io rpm built friday causes docker service to restart
08:47 mattymo and also we're still restarting supervisord during puppet run
08:47 mattymo these containers should be autostarting but it looks like they aren't
08:47 sbartel joined #fuel-dev
08:47 sbartel hello
08:48 sbartel about the new version 5.0.1
08:48 sbartel does it come from the stable/5.0 branch
08:48 mattymo ikalnitsky, or asledzinskiy do you guys have an env still available?
08:48 sbartel or the master one?
08:48 mattymo sbartel, yes
08:49 ikalnitsky mattymo: i can give you env when all you need is just run upgrade.sh and wait ~2 mins to fail.
08:49 sbartel mattymo : the stable/5.0?
08:49 mattymo ikalnitsky, just an env after failure is all I want
08:51 asledzinskiy mattymo: ikalnitsky: I've given mattymo the env that I gave to you this morning
08:51 ikalnitsky asledzinskiy: ok. thanks.
08:58 mattymo so we have to ensure docker survives a restart
08:58 mattymo and we have to do it without supervisor because we're updating dockerctl config already at this point
08:59 mattymo repeating existing functionality is awesome
09:00 mattymo also stopping docker service seems to cause all sorts of mount issues
09:07 ddmitriev joined #fuel-dev
09:08 aglarendil alex_didenko: mattymo
09:08 ikalnitsky mattymo: is there something I can help you from python side?
09:08 aglarendil we have code that restarts supervisor but leaves containers untouched
09:09 tnurlygayanov Hi there
09:09 aglarendil okay, alex is on vacation
09:09 tnurlygayanov I have a question
09:09 mattymo I can fix it from the puppet side
09:09 aglarendil mattymo: Matt, could you look into this bug?
09:09 mattymo I think I have a solution already aglarendil :)
09:09 tnurlygayanov we released 5.0.1, but we have no tag 5.0.1 in Fuel repositories
09:10 mattymo 1 - add to dockerctl start of existing container the following:       umount -l $(grep "$id" /proc/mounts | awk '{print $2}' | sort -r)
09:10 mattymo 2 - add dockerctl start all as a trigger after docker service is restarted
09:10 aglarendil mattymo: will it be idempotent?
09:10 mattymo yes
09:11 aglarendil mattymo: excellent
09:11 mattymo if a container is already running it will say it's already started
09:15 teran joined #fuel-dev
09:23 xarses joined #fuel-dev
09:24 teran joined #fuel-dev
09:27 bookwar joined #fuel-dev
09:32 mattymo ikalnitsky, dshulyak1 I need help with 1 piece actually
09:33 mattymo we are updating the yum repo before backing up container data. we should not change repo config until after
09:34 mattymo evgeniyl, ^
09:36 mattymo asledzinskiy, ^
09:42 sbartel question about the 5.0.1 release
09:43 sbartel I thought that ml2 support should be included
09:43 sbartel and in the stable/5.0 branch the ml2 stuffs added in master branch are not avalaible
09:43 evgeniyl mattymo: did we accept patch from alex_didenko not to restart supervisor and upgarde only dockerctl and fuel-client?
09:43 sbartel did I miss something?
09:44 EeeBotu New bug #1358201 in fuel: "[devops] Pls create 5.1.1 milestone in LP" [High, Fix Released] https://bugs.launchpad.net/bugs/1358201
09:44 mattymo evgeniyl, yes I believe we did
09:44 mattymo so I think we can skip that concern for supervisord
09:45 mattymo the actual bug is docker isn't restarting containers properly because it creates broken mount points when restarting docker service
09:45 mattymo it's fine on reboot because broken mount points vanish
09:46 mattymo https://review.openstack.org/#/c/114921/.
09:56 mattymo asledzinskiy, your test env broke hard
09:56 mattymo "top: /proc is not mounted, required for output data" <- really cool error
09:57 dshulyak1 mattymo: evgeniyl: as for bug with ip address allocation, i think we will need to restart docker after patch is applied
09:57 teran joined #fuel-dev
09:58 dshulyak1 docker daemon
09:58 mattymo restarting daemon does not help
09:58 mattymo there are broken mounts
09:58 mattymo it's an issue with docker 0.10
09:59 dshulyak1 mattymo: but i am talking about ipaddress allocation issue
09:59 dshulyak1 duplicate ips for different docker containers
09:59 mattymo docker restarts when puppet runs and installs new rpm
10:03 Longgeek joined #fuel-dev
10:23 teran joined #fuel-dev
10:35 HeOS joined #fuel-dev
10:37 dshulyak joined #fuel-dev
10:44 adanin joined #fuel-dev
10:56 e0ne joined #fuel-dev
11:01 albs joined #fuel-dev
11:19 Longgeek joined #fuel-dev
11:28 Longgeek joined #fuel-dev
11:32 dpyzhov aglarendil: https://review.openstack.org/#/c/111010/ merge please
11:32 dpyzhov nurla: https://review.openstack.org/#/c/113290 remove -2 please
11:34 nurla dpyzhov: done
11:35 dpyzhov aglarendil: https://review.openstack.org/#/c/113290 merge please
11:44 EeeBotu New bug #1358255 in fuel: "NSX manifests conflist with ML2 introduced changes" [High, In Progress] https://bugs.launchpad.net/bugs/1358255
11:45 Longgeek joined #fuel-dev
11:50 dnikishov joined #fuel-dev
11:56 ikalnitsky mattymo: is there a workaround to avoid the issue with docker restarting?
11:57 mattymo I'm working on it and testing it
11:57 mattymo it doesn't seem to work well yet
11:57 mattymo https://review.openstack.org/#/c/114921/ ikalnitsky
11:57 ikalnitsky mattymo: yep. i apply your fix (patch set 2) but the puppet hangs :(
11:58 mattymo look at patchset 3
11:58 mattymo it makes sure dockerctl start all runs after dockerctl build all
11:58 mattymo start all runs in arbitrary order and build all runs in the needed order
11:59 mattymo and you can't start the containers w/o the storage containers (from build)
12:02 Longgeek_ joined #fuel-dev
12:09 mattymo new patch seems better ikalnitsky
12:09 mattymo hmm where is the jenkins task to build an upgrade patch based on a gerrit commit?
12:10 mattymo nevermind got it
12:11 ikalnitsky mattymo: patchset #3 - puppet still hangs :(
12:12 mattymo tail -1000 /tmp/puppet2014*
12:12 mattymo ikalnitsky, ^
12:13 ikalnitsky mattymo: http://paste.openstack.org/raw/96810/
12:13 mattymo nice! that's awful
12:14 * mattymo fixes checks
12:29 e0ne joined #fuel-dev
12:38 agordeev hi, i still can't build the bootstrap image against the master branch with little additional changes. https://review.openstack.org/#/c/113934/ can anyone who is more lucky then me build it for me? or just guide me to the right direction
12:45 EeeBotu New bug #1358265 in fuel: "Remove fabric monkey patch from Shotgun" [Medium, Triaged] https://bugs.launchpad.net/bugs/1358265
12:53 mattymo ikalnitsky, asledzinskiy evgeniyl it looks like we have an impasse here for solving the docker-io upgrade from 5.0 to 5.1
12:54 artem_panchenko mattymo ikalnitsky: I think puppet hangs because upgrade script update version.yaml before applying manifests, so after upgrading of docker-io package it tries to build/start new 5.1 containers, but they aren't unpacked yet. Also, I believe puppet shouldn't start new containers, because further upgrade script will try to attach to the old postgres container and dump database
12:54 mattymo the centos repo gets updated to 5.1, but puppet host-only.pp gets applied. I have a fix in a new commit for 5.1, but those manifests are not in place
12:54 dnikishov joined #fuel-dev
12:54 mattymo you need to have fuel version 5.0 and 5.0 yum repo when getting old container data
12:55 mattymo without it, it's not going to work
12:55 mattymo err sorry.. 5.0 manifests against 5.1 repo
12:56 mihgen dpyzhov: aglarendil: who is the best to help agordeev with bootstrap build? see message above
12:56 ikalnitsky mattymo: it's impossible to get 5.0 manifests against 5.1 repo becaus of centos-versions.yaml
12:56 evgeniyl mattymo: is mandatory to upgrade docker? I think it could be risky, and I had problems with upgrade script when they had changed their api.
12:57 mattymo we're upgrading docker to fix a bug in ip address allocation
12:57 mattymo it's a backported patch
12:57 evgeniyl mattymo: what bug? With iptable rules duplication when you stop a container?
12:58 evgeniyl mattymo: I thought we made workaround when we delete the rules.
12:58 mattymo this is not an iptables bug
12:58 mattymo this is a bug with docker allocating the same internal address, like 172.41.0.8 to two containers
12:59 ikalnitsky evgeniyl: we should support docker upgrades anyway, i believe. so let's solve this issue before 5.1.
13:00 mattymo ikalnitsky, evgeniyl please please I want you guys to accept my proposal to get 5.0 container data BEFORE running puppet
13:01 mattymo and then we can stop those containers safely and upgrade
13:03 msemenov joined #fuel-dev
13:03 xarses joined #fuel-dev
13:06 ikalnitsky mattymo: ok. We will talk about it right after meeting.
13:08 dnikishov hey guys, I'm working on MOS customizations for Wells Fargo project and having an issue when certain API services (glance, cinder, neutron) don't process http requests. Basically what I have is a 5.1 iso, built from master (+ customizations: SSL endpoints and PKI tokens) and Ubuntu/HA/Neutron OS env. Once deployment succeedes, API calls to these services most of the times just hang up, their log files don't contain any signs that such requests even have arr
13:08 enikanorov__ joined #fuel-dev
13:10 mattymo dnikishov, the end was cut off
13:10 mattymo dnikishov, looks like you need some packet captures
13:12 dnikishov mattymo: I'm not sure the problem is with packet loss as a curl to the endpoint succeedes
13:37 ylobankov joined #fuel-dev
13:40 adanin_ joined #fuel-dev
13:58 nzrf joined #fuel-dev
14:01 AlexF joined #fuel-dev
14:03 enikanorov_ joined #fuel-dev
14:04 Krast_ joined #fuel-dev
14:07 kaliya_ joined #fuel-dev
14:07 agordeev mihgen: dnikishov: aglarendil: please nevermind. I had built image successfully against stable/5.0
14:08 agordeev err, i meant dpyzhov ^
14:12 jaypipes joined #fuel-dev
14:17 zynzel joined #fuel-dev
14:19 jobewan joined #fuel-dev
14:34 angdraug joined #fuel-dev
14:38 zynzel joined #fuel-dev
14:48 EeeBotu New bug #1358313 in fuel: "Can not configure fixed network with mask more than 24" [High, New] https://bugs.launchpad.net/bugs/1358313
14:48 EeeBotu New bug #1358304 in fuel: "Community ISO build fails with fuel-astute installation" [Low, Confirmed] https://bugs.launchpad.net/bugs/1358304
14:52 zynzel_ joined #fuel-dev
14:57 dpyzhov aglarendil: https://bugs.launchpad.net/fuel/+bug/1355180 should it be closed?
14:57 akupko joined #fuel-dev
15:01 mattymo ikalnitsky, asledzinskiy dpyzhov dshulyak1 guys I am requesting OSCI to remove the new docker-io package for now
15:01 mattymo my patches aren't working and I don't want to leave this critical bug up for now
15:02 mattymo I will try my best to make a better patch tomorrow
15:03 ikalnitsky mattymo: what about next upgrades? we definitely must be able to upgrade docker on user systems
15:03 mattymo ikalnitsky, see my message) I'm trying to fix it!
15:03 mattymo it is possible, sure. the _most_ ideal scenario for upgrading docker is to docker stop $(docker ps), then upgrade, then dockerctl start all
15:15 dancn joined #fuel-dev
15:19 kaliya joined #fuel-dev
15:21 dancn hello, are really required more than 4GB of RAM for running succesfully 'make iso' in fuel-main?
15:22 dancn lrzip stop at around 80% with Failed to malloc ckbuf in hash_search
15:22 dancn Cannot allocate memory
15:22 salmon_ dancn: no, 4GB should be enough
15:22 xenolog joined #fuel-dev
15:24 dancn salmon_: I stated from 1GB, increasing the VM RAM size to 2, 3, 3.5 and finally 4GB but still no luck, any suggestion?
15:37 dancn with 5GB the process finished succesfully, but really this huge amount of needed RAM surprises me a lot, it should be at least documented. I used the prepare-build-env.sh from master and then switched to stable/5.0 before 'make iso'.  Any thoughts on this?
15:37 tzn joined #fuel-dev
15:51 EeeBotu New bug #1358369 in fuel: "[System tests] Need to add check that Fixed network CIDR on controller is default value" [High, New] https://bugs.launchpad.net/bugs/1358369
15:51 EeeBotu New bug #1358345 in fuel: "Need to increase vm_slave_memory_mb in config.sh for VBox" [Undecided, New] https://bugs.launchpad.net/bugs/1358345
15:51 EeeBotu New bug #1358342 in fuel: "revert docker-io to docker-io-0.10.0-2.mira2.x86_64" [Critical, Confirmed] https://bugs.launchpad.net/bugs/1358342
15:57 rmoe joined #fuel-dev
16:01 EeeBotu New bug #1358374 in fuel: "[osci] applying patches to sqlalchemy" [High, Confirmed] https://bugs.launchpad.net/bugs/1358374
16:23 tatyana joined #fuel-dev
16:28 teran joined #fuel-dev
16:28 teran_ joined #fuel-dev
16:31 dshulyak joined #fuel-dev
16:38 AlexF joined #fuel-dev
16:42 tatyana joined #fuel-dev
16:49 e0ne joined #fuel-dev
16:59 dilyin joined #fuel-dev
17:02 aviramb joined #fuel-dev
17:03 AlexF joined #fuel-dev
17:12 prmtl joined #fuel-dev
17:29 jaypipes joined #fuel-dev
17:46 jaypipes joined #fuel-dev
17:48 xarses joined #fuel-dev
17:58 agrebennikov_ joined #fuel-dev
17:59 agrebennikov_ hey guys, I'm on the important customer right now. I need to know whether we have any person who thought about regions implementations into Fuel
18:06 HeOS joined #fuel-dev
18:16 angdraug joined #fuel-dev
18:19 prmtl joined #fuel-dev
18:53 EeeBotu New bug #1358428 in fuel: "Delete compute service when deleting compute node" [Undecided, New] https://bugs.launchpad.net/bugs/1358428
19:22 MiroslavAnashkin We only have a lot of people who thought we need to postpone regions implementation in Fuel. Since it appears regions may be implemented easy enough.
19:40 teran joined #fuel-dev
19:41 e0ne joined #fuel-dev
19:58 salmon_ joined #fuel-dev
20:03 jkirnosova joined #fuel-dev
20:04 kat_pimenova joined #fuel-dev
20:04 teran joined #fuel-dev
20:11 e0ne joined #fuel-dev
20:15 enikanorov joined #fuel-dev
20:23 agrebennikov_ joined #fuel-dev
20:23 agrebennikov_ MiroslavAnashkin, that's why noone aclually implemented it yet
20:36 e0ne joined #fuel-dev
20:55 tzn joined #fuel-dev
20:59 teran_ joined #fuel-dev
21:00 teran__ joined #fuel-dev
21:14 jaypipes-afk joined #fuel-dev
21:49 e0ne joined #fuel-dev
22:02 msemenov_ joined #fuel-dev
22:09 ikalnitsky_ joined #fuel-dev
22:12 asledzinskiy joined #fuel-dev
22:13 dshulyak joined #fuel-dev
22:16 tzn joined #fuel-dev
22:50 teran joined #fuel-dev
22:56 AlexF joined #fuel-dev
23:16 teran joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary