Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-11-12

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:18 pboldin joined #fuel-dev
00:59 Iesvs joined #fuel-dev
01:11 Iesvs I had an issue lastnight where I discovered that my small fuel server had filled up /var and was no longer booting up.
01:11 Iesvs Today, I found that there were several log files that were massive!
01:12 Iesvs I cleared out about 22G of log files.
01:13 Iesvs A daily tar of the log files would have saved me some grief.
01:13 Iesvs Just a suggestion :)
01:19 Iesvs These logs were especially large:
01:23 Iesvs Hmm... on a reboot, I am now getting this error
01:24 Iesvs kernel: device-mapper: space map common: index_check failed: blocknr 0 != wanted 2179
01:24 Iesvs kernel: device-mapper: block manager: index validator check failed for block 2179
01:24 Iesvs kernel: device-mapper: transaction manager: couldn't open metadata space map
01:25 Iesvs When the drive filled up, it seems to have trashed the metadata.
01:45 Iesvs Also, sorry about the logs - I didn't notice it was reading /var as a command.
01:45 Iesvs var/log/docker-logs/nailgun/api.log
01:45 Iesvs var/log/docker-logs/remote/node-*/nailgun-agent.log
01:45 Iesvs var/log/docker-logs/remote/node-55.domain.tld/mongod.27017.log
01:46 Iesvs And atually, I was mistaken. That mongod log was by itself 22G. By removing those log files I actually cleared 50G of space on my fuel server.
06:02 dklepikov joined #fuel-dev
06:09 dklepikov joined #fuel-dev
06:12 xarses joined #fuel-dev
06:13 rmoe joined #fuel-dev
06:22 fandi joined #fuel-dev
06:22 prmtl_ joined #fuel-dev
06:26 prmtl_ joined #fuel-dev
06:27 AlexF joined #fuel-dev
06:27 dshulyak joined #fuel-dev
06:31 shauno_ joined #fuel-dev
07:19 AlexF joined #fuel-dev
07:28 stamak joined #fuel-dev
07:37 taj_ joined #fuel-dev
07:55 e0ne joined #fuel-dev
07:58 gilmeir joined #fuel-dev
08:03 e0ne joined #fuel-dev
08:04 Longgeek joined #fuel-dev
08:05 pasquier-s joined #fuel-dev
08:07 Longgeek joined #fuel-dev
08:10 sambork joined #fuel-dev
08:11 e0ne joined #fuel-dev
08:12 pasquier-s joined #fuel-dev
08:14 salmon_ joined #fuel-dev
08:24 azemlyanov joined #fuel-dev
08:26 adanin joined #fuel-dev
08:36 fandi joined #fuel-dev
08:38 ddmitriev joined #fuel-dev
08:40 EeeBotu New bug #1391776 in fuel: "Mellanox deployment missing intel_iommu=on kernel parameter" [Undecided, New] https://bugs.launchpad.net/bugs/1391776
08:40 EeeBotu New bug #1391773 in fuel: "Move ping_vip__public to astute post deploy hooks" [High, Triaged] https://bugs.launchpad.net/bugs/1391773
08:44 HeOS joined #fuel-dev
08:52 romcheg joined #fuel-dev
08:57 xarses joined #fuel-dev
09:02 romcheg Guys,  could you please review https://review.openstack.org/#/c/129900/ and https://review.openstack.org/#/c/129901/ ?
09:07 AlexF joined #fuel-dev
09:08 shauno_ joined #fuel-dev
09:09 shauno__ joined #fuel-dev
09:09 romcheg evgeniyl__: ^
09:16 rmoe joined #fuel-dev
09:20 romcheg akislitsky: ^
09:27 akislitsky_ joined #fuel-dev
09:40 xarses joined #fuel-dev
09:45 EeeBotu New bug #1391804 in fuel: "    Isolate sudo syslog logs locally" [Undecided, New] https://bugs.launchpad.net/bugs/1391804
09:45 EeeBotu New bug #1391797 in fuel: "update mellanox_openstack manifests for Fuel 6.0" [Undecided, In Progress] https://bugs.launchpad.net/bugs/1391797
09:45 EeeBotu New bug #1391796 in fuel: "[doc] Update information in READ.me file" [Undecided, New] https://bugs.launchpad.net/bugs/1391796
09:48 shauno_ joined #fuel-dev
09:51 IvanBerezovskiy joined #fuel-dev
09:52 AlexF joined #fuel-dev
09:53 e0ne joined #fuel-dev
09:53 d__k joined #fuel-dev
09:59 gilmeir joined #fuel-dev
10:00 AlexF joined #fuel-dev
10:10 Longgeek joined #fuel-dev
10:12 Longgeek_ joined #fuel-dev
10:18 akupko joined #fuel-dev
10:19 monester_laptop joined #fuel-dev
10:25 ddmitriev joined #fuel-dev
10:32 adanin joined #fuel-dev
10:44 AlexF joined #fuel-dev
10:46 EeeBotu New bug #1391819 in fuel: "Please create packages/precise/python-defaults repository" [Undecided, New] https://bugs.launchpad.net/bugs/1391819
10:46 EeeBotu New bug #1391812 in fuel: "[systest] Prescale test in system_test" [Undecided, New] https://bugs.launchpad.net/bugs/1391812
11:03 romcheg joined #fuel-dev
11:06 dkusidlo joined #fuel-dev
11:21 AlexF joined #fuel-dev
11:37 aviramb_ joined #fuel-dev
11:37 aviramb_ left #fuel-dev
11:39 aviramb__ joined #fuel-dev
11:47 EeeBotu New bug #1391833 in fuel: "Glance Lost connection to MySQL server at 'reading aut horization packet (System test deploy_bonding_ha_balance_slb)" [High, New] https://bugs.launchpad.net/bugs/1391833
11:51 shauno_ joined #fuel-dev
11:51 pboldin joined #fuel-dev
11:53 sambork joined #fuel-dev
11:59 gilmeir can nova guys please CR: https://review.openstack.org/133920
11:59 shauno_ joined #fuel-dev
12:01 gilmeir had a mistake, I meant to post this one - https://review.fuel-infra.org/#/c/534/
12:09 AlexF joined #fuel-dev
12:12 ddmitriev joined #fuel-dev
12:12 EeeBotu New bug #1391846 in fuel: "[ui] Environment with this name already exists -- support for error message from API" [Medium, New] https://bugs.launchpad.net/bugs/1391846
12:12 EeeBotu New bug #1391845 in fuel: "Update Mellanox OFED to version 2.3" [Undecided, In Progress] https://bugs.launchpad.net/bugs/1391845
12:27 dshulyak joined #fuel-dev
12:30 dmitryme gilmeir: that CR was created into a wrong branch. The correct one is here https://review.fuel-infra.org/#/c/750/ and it will take some time for CI to test it (1-2 hours, I think)
12:30 kat_pimenova Hi! I can't relogin on our launchpad.net . When I'm trying to relogin I've got: "Forbidden! You don't have permission to access /+openid-callback on this server." page.
12:31 gilmeir dmitryme: thanx, dburmistrov just explained that to me :)
12:33 romcheg joined #fuel-dev
12:41 dshulyak joined #fuel-dev
12:42 mihgen folks I'm pretty sure bvt centos hangs on staging http://jenkins-product.srt.mirantis.net:8080/job/6.0.staging.centos.bvt_1/64/
12:44 mihgen alex_didenko: nurla: can you guys start looking into this build? ^^
12:45 bookwar mihgen: it passed the deployment and started ostf
12:47 mihgen bookwar: that's good. THough it ran for 3 h instead of 2 ..
12:48 mihgen bookwar: please analyse whether it's infra issue or code
12:49 mihgen alex_didenko: please be on it too, to see if it's pacemaker/corosync thing you are working on
12:50 AlexF joined #fuel-dev
12:53 pasquier-s_ joined #fuel-dev
12:53 alex_didenko mihgen: ok
12:54 bookwar alex_didenko, mihgen: according to zabbix, there weren't any memory or CPU issues on the host
12:55 bookwar but there was torrent failure
12:55 bookwar iso has not been downloaded via torrent, test used the fallback to wget option
12:58 alex_didenko yeah, 50 minutes were spent on ISO downloading
13:00 pboldin joined #fuel-dev
13:05 mihgen bookwar: alex_didenko thanks
13:05 mihgen rvyalov: as 6.0 staging passed, but overall staging build failed, can we push update to public mirrors manually?
13:06 rvyalov mihgen: yes
13:07 mihgen dmitryme: igormarnat We expect to update public mirror now, by rvyalov's manual push - as both BVTs are green! ^^^
13:07 igormarnat mihgen, go
13:11 AlexF joined #fuel-dev
13:13 EeeBotu New bug #1391873 in fuel: "    Parallel test execution with py.test" [Undecided, New] https://bugs.launchpad.net/bugs/1391873
13:25 rvyalov mihgen: done
13:26 dkusidlo joined #fuel-dev
13:32 aviramb_ joined #fuel-dev
13:33 Longgeek joined #fuel-dev
13:45 bpiotrowsk joined #fuel-dev
13:46 akasatkin bookwar: is it supported to propose fixes to 5.0 branch ? https://fuel-jenkins.mirantis.com/job/stackforge-verify-fuel-web/4245/consoleFull
13:47 AlexF joined #fuel-dev
13:49 samuelbartel joined #fuel-dev
13:51 bookwar akasatkin: let's see if empty test passes https://fuel-jenkins.mirantis.com/job/stackforge-verify-fuel-web/4246/console
13:51 akasatkin bookwar: ok
13:53 EeeBotu New bug #1391881 in fuel: "refactor neutron  l3-agent" [Medium, In Progress] https://bugs.launchpad.net/bugs/1391881
13:56 sovsianikov joined #fuel-dev
14:03 adanin joined #fuel-dev
14:13 IvanBerezovskiy left #fuel-dev
14:13 mattymo https://review.openstack.org/#/c/133956/ reviewers needed
14:13 mattymo alex_didenko, dmitryme nurla xarses ? ^
14:15 EeeBotu New bug #1391899 in fuel: "UI related metadata serialized in deployment info" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1391899
14:15 EeeBotu New bug #1391896 in fuel: "hanging mount points when building iso" [High, New] https://bugs.launchpad.net/bugs/1391896
14:20 sunxin joined #fuel-dev
14:22 seeg hello, could we merge https://review.openstack.org/#/c/132543/ ?
14:30 agordeev bookwar: hi. bug/1391896 is extremely harmful for our build servers, right?
14:31 bookwar agordeev: i am not sure if it is the root cause. I mean i unmounted all the hanging mounts on the slave and rerun the build
14:31 bookwar but it fails with the same losetup error
14:32 bookwar http://jenkins-product.srt.mirantis.net:8080/job/6.0.iso/96/console
14:32 bookwar so it looks like the original problem is in the make process itself and not caused by some other builds on the server
14:33 AlexF joined #fuel-dev
14:35 agordeev bookwar: gah. are other builds sharing the same loop devices on that particular server?
14:35 bookwar we run one build at a time
14:36 bookwar so right now there aren't any other builds, but inside one build do we use this loop-device twice?
14:36 bookwar for centos first and then for ubuntu?
14:36 dkusidlo joined #fuel-dev
14:37 agordeev nope, that can't happen. pnly building of ubuntu target image needs loop devices
14:39 bookwar agordeev: we have local_mirror/centos already mounted when we start working on ubuntu one
14:43 mattymo you can use an arbitrary device. you don't need loop dev 10 necessarily
14:43 mattymo losetup doesn't require a specific one
14:45 bookwar mattymo: https://github.com/stackforge/fuel-main/blob/f09f3624e55da3b58bf9ef6e2614c4a96cb5b072/image/ubuntu/create_separate_images.sh#L69
14:45 bookwar build fails on this one https://github.com/stackforge/fuel-main/blob/f09f3624e55da3b58bf9ef6e2614c4a96cb5b072/image/ubuntu/create_separate_images.sh#L78
14:46 bookwar it looks like '-e' condition is wrong
14:47 mattymo shouldn't you umount it first
14:47 mattymo then losetup -d
14:47 mihgen folks, who can take https://bugs.launchpad.net/fuel/+bug/1391833? this is really needed to be triaged/ fixed asap, as it goes to 5.1.1. We are about to declare code freeze for 5.1.1 tomorrow, and better we would not see such things
14:47 mihgen mattymo: bogdando alex_didenko dilyin xarses ^^^
14:50 agordeev mattymo: the only problem is that it doesn't expect appearing of those mount points.
14:50 agordeev bookwar: '-e' condition is fine, since we're dealing with special block device file as loop device.
14:52 bookwar agordeev: http://paste.openstack.org/show/132402/ - i don't understand this
14:52 AlexF joined #fuel-dev
14:53 bookwar it says "loop: can't delete device /dev/loop_devices_fuel_build0: No such device or address" but it actually has deleted it
14:55 agordeev bookwar: it's better to take a look at what 'losetup -a' says. Because /dev/loop_devices_fuel_build0 is just the association to /dev/loop<X>, where X is unknown
14:58 lordd joined #fuel-dev
15:01 rpodolyaka2 joined #fuel-dev
15:02 rpodolyaka2 can you guys take a look at Egor's comments at https://bugs.launchpad.net/fuel/+bug/1384785 ?
15:02 rpodolyaka2 looks like a RabbitMQ cluster is not doing fine after failure of 1 node
15:03 rpodolyaka2 nova-compute tries to reconnect to any of 3 rabbitmq instances in a loop, but fails
15:12 romcheg joined #fuel-dev
15:16 EeeBotu New bug #1384785 in fuel: "Volumes/instances couldn't be created after destroy controller due to oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': [Errno 110] Connection timed out" [Undecided, New] https://bugs.launchpad.net/bugs/1384785
15:18 romcheg joined #fuel-dev
15:18 romcheg left #fuel-dev
15:19 romcheg joined #fuel-dev
15:21 dshulyak joined #fuel-dev
15:23 bookwar agordeev: have you tested the "removing stale loop devices" part? because without stale devices it build seems to work properly
15:25 jobewan joined #fuel-dev
15:35 dshulyak joined #fuel-dev
15:47 mihgen bookwar: any pregress with http://jenkins-product.srt.mirantis.net:8080/job/6.0.iso/ ?
15:47 mihgen how it appears in the build?
15:47 monester_laptop joined #fuel-dev
15:48 mihgen looks like regression
15:49 kozhukalov joined #fuel-dev
15:49 mihgen agordeev: kozhukalov: do you know anythign about it? We need urgent help here, as we can't build 6.0 ISO. If we find it to be a regression, let's jsut revert before we can find a right solution to it
15:49 mihgen kozhukalov: I'm about failures at 6.0 iso build, see internal  http://jenkins-product.srt.mirantis.net:8080/job/6.0.iso/
15:50 kozhukalov mihgen: looking at it
15:50 mihgen thx
15:50 mihgen nurla: 1391438 this seems to be committed, do you think we can restart promo bvt?
15:51 mihgen we need an ISO for scale lab, can we recommend any without this promo bvt passing?
15:51 bookwar mihgen, kozhukalov: the issue with iso happens when there exists /dev/loop_devices_fuel_build0 on the server
15:52 bookwar then the part which is responsible for removing stale loop devices fails
15:52 bookwar if we clean stale devices manually - wvwrything works, there is just a minor issue with changelog artifact handling
15:53 kozhukalov bookwar: as far as i can see the error is about changelog file
15:53 kozhukalov cp: cannot stat '*.changelog': No such file or directory
15:53 kozhukalov Build step 'Execute shell' marked build as failure
15:53 agordeev kozhukalov: mihgen: bookwar: i'm trying to fix the part with unmounted mount points after the build
15:53 bookwar kozhukalov: yes, the last one is about changelog and we are fixing it, it is just the matter of wrong path in bash script
15:54 kozhukalov bookwar: ok
15:55 nurla mihgen: for promo bvt check with Sergey's fix, we should have green iso build
15:55 EeeBotu New bug #1391935 in fuel: "504 gateway timeout occasionally blocks Web UI" [High, In Progress] https://bugs.launchpad.net/bugs/1391935
15:56 mihgen kozhukalov: please do it very quick then - we need new ISO and unblock other teams
15:56 mihgen nurla: once they are done with the fix ^^ - we can start a new build and see ..
15:56 nurla mihgen: yep
16:04 lordd joined #fuel-dev
16:08 msemenov joined #fuel-dev
16:10 asheplyakov joined #fuel-dev
16:14 pboldin joined #fuel-dev
16:18 EeeBotu New bug #1391947 in fuel: "OpenStack Dashboard gets 10.20.0.2 IP when assigned as default router" [Undecided, New] https://bugs.launchpad.net/bugs/1391947
16:18 EeeBotu New bug #1391943 in fuel: "[kilo] Add missed packages for Nova" [Medium, New] https://bugs.launchpad.net/bugs/1391943
16:30 angdraug joined #fuel-dev
16:30 sunxin joined #fuel-dev
16:30 nurla guys, please use for some custom testing jobs from "custom_iso" view in jenkins
16:30 nurla izinovik: ^
16:35 Longgeek joined #fuel-dev
16:40 lordd joined #fuel-dev
16:50 izinovik nurla, thank you for explanation
16:53 vtzan hello all. astute logs have stucked on "Attempt to assign lesser progress detected: Progress was: deploying, attempted to assign: 38. Skipping this node (id=4)" for very long time. bar on the right of fuel is on 100% but never ends
16:53 evgeniyl__ bogdando: looks like Fuel CI set +1 for https://review.openstack.org/#/c/132535/ and https://review.openstack.org/#/c/132534/ could you please merge this backports if they are ok?
16:56 EeeBotu New bug #1391956 in fuel: "Fuel+NSX can not complete deploy because of broken dependencies packages" [Undecided, New] https://bugs.launchpad.net/bugs/1391956
16:59 mihgen bogdando: angdraug: guys, do you think we can merge this? https://review.openstack.org/#/c/133526/
16:59 mihgen issue block scaling lab
17:04 angdraug mihgen: makes sense, approved
17:07 salmon_ joined #fuel-dev
17:08 vtzan the puppet log while deploying is showing many many times this: Puppet::Type::Neutron_net::ProviderNeutron) Neutron API not avalaible. Wait up to 78 sec.
17:08 vtzan any idea from anyone?
17:36 pboldin joined #fuel-dev
17:37 ikalnitsky joined #fuel-dev
17:38 dshulyak joined #fuel-dev
17:41 lordd joined #fuel-dev
18:02 dshulyak joined #fuel-dev
18:14 dshulyak joined #fuel-dev
18:42 lordd joined #fuel-dev
18:53 dshulyak joined #fuel-dev
18:54 e0ne joined #fuel-dev
19:02 EeeBotu New bug #1392004 in fuel: "Package for python Ceilometer client change log not equal to git commits history" [High, New] https://bugs.launchpad.net/bugs/1392004
19:02 EeeBotu New bug #1391999 in fuel: "The information about python-barbicanclient package isn't presented in change log files" [High, New] https://bugs.launchpad.net/bugs/1391999
19:02 EeeBotu New bug #1391997 in fuel: "Information about python-muranoclient not presented in ubuntu-packages.changelog" [High, New] https://bugs.launchpad.net/bugs/1391997
19:02 EeeBotu New bug #1391993 in fuel: "Murano apps package in not up to date in MOS 5.1.1" [High, New] https://bugs.launchpad.net/bugs/1391993
19:24 EeeBotu New bug #1392015 in fuel: "Package for python Heat client change log not equal to git commits history" [Undecided, New] https://bugs.launchpad.net/bugs/1392015
19:24 EeeBotu New bug #1392009 in fuel: "Information about python-designateclient not presented in changelog  files" [High, New] https://bugs.launchpad.net/bugs/1392009
19:24 EeeBotu New bug #1392005 in fuel: "Information about python-cinderclient not presented in ubuntu-packages.changelog" [High, New] https://bugs.launchpad.net/bugs/1392005
19:30 angdraug joined #fuel-dev
19:35 dshulyak joined #fuel-dev
19:41 dshulyak joined #fuel-dev
20:02 EeeBotu New bug #1392039 in fuel: "Information about python-troveclient not presented in  ubuntu-packages.changelog file" [High, New] https://bugs.launchpad.net/bugs/1392039
20:02 EeeBotu New bug #1392037 in fuel: "Package for python Swift client change log not equal to git commits history" [High, New] https://bugs.launchpad.net/bugs/1392037
20:02 EeeBotu New bug #1392032 in fuel: "Empty git history in centos-packages.changelog for python-saharaclient package" [High, New] https://bugs.launchpad.net/bugs/1392032
20:02 EeeBotu New bug #1392017 in fuel: "Information about python-ironicclient not presented in changelog files" [High, New] https://bugs.launchpad.net/bugs/1392017
20:02 jpf joined #fuel-dev
20:02 tatyana joined #fuel-dev
20:19 AlexF joined #fuel-dev
20:28 jobewan joined #fuel-dev
20:28 EeeBotu New bug #1392043 in fuel: "Information about Trove package not presented in ubuntu-packages.changelog & centos-packages.changelog files" [High, New] https://bugs.launchpad.net/bugs/1392043
20:28 EeeBotu New bug #1392041 in fuel: "Information about sahara-dashboard not presented in centos-packages.changelog file" [High, New] https://bugs.launchpad.net/bugs/1392041
20:28 EeeBotu New bug #1392040 in fuel: "Sahara package change log not equal to git commits history" [High, New] https://bugs.launchpad.net/bugs/1392040
20:50 e0ne joined #fuel-dev
20:51 tatyana joined #fuel-dev
21:21 AlexF joined #fuel-dev
21:25 e0ne joined #fuel-dev
21:29 EeeBotu New bug #1392066 in fuel: "Non user-friendly error message in ceilometer OSTF platform test" [Low, New] https://bugs.launchpad.net/bugs/1392066
21:39 dshulyak joined #fuel-dev
22:18 lordd joined #fuel-dev
22:42 adanin joined #fuel-dev
23:14 salmon_ joined #fuel-dev
23:15 IgorYozhikov joined #fuel-dev
23:36 salmon_ joined #fuel-dev
23:45 salmon_ joined #fuel-dev
23:53 teran joined #fuel-dev
23:58 salmon_ joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary