Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-05-20

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:21 teran joined #fuel-dev
00:26 IlyaE joined #fuel-dev
00:26 teran_ joined #fuel-dev
00:44 rmoe joined #fuel-dev
01:00 teran joined #fuel-dev
01:36 xarses joined #fuel-dev
03:07 teran joined #fuel-dev
03:42 IlyaE joined #fuel-dev
04:08 teran joined #fuel-dev
04:37 fandi joined #fuel-dev
04:43 IlyaE joined #fuel-dev
04:51 artem_panchenko joined #fuel-dev
05:01 dshulyak_ joined #fuel-dev
05:09 teran joined #fuel-dev
05:10 teran_ joined #fuel-dev
05:13 _daks_ joined #fuel-dev
05:19 e0ne joined #fuel-dev
05:33 e0ne joined #fuel-dev
06:03 e0ne joined #fuel-dev
06:06 _daks_ joined #fuel-dev
06:11 teran joined #fuel-dev
06:26 fandi joined #fuel-dev
06:27 sambork joined #fuel-dev
06:31 al_ex joined #fuel-dev
06:34 Ch00k joined #fuel-dev
06:39 EeeBotu New bug #1321095 in fuel: "Slaves don't boot via PXE after deployment stop on provisioning" [Undecided, New] https://bugs.launchpad.net/bugs/1321095
06:39 EeeBotu New bug #1321094 in fuel: "[devops] Fuel CI jobs for fuel-library have to be documented" [High, New] https://bugs.launchpad.net/bugs/1321094
06:48 mihgen warpc: hi
06:49 mihgen warpc: take a look at https://bugs.launchpad.net/fuel/+bug/1321095 please
06:49 warpc mihgen: hi, i will
06:53 warpc mihgen: i suggest to merge it: https://review.openstack.org/#/c/94163/
06:54 mihgen bogdando: I don't think https://review.openstack.org/#/c/93883/ is Fuel CI issue with -1. I think we need to try it manually.. something wrong there, when you deploy just one controller. Does Fuel CI gives you logs?
06:55 mihgen warpc: done
06:56 salmon_ joined #fuel-dev
06:57 warpc mihgen: thanks! About https://bugs.launchpad.net/fuel/+bug/1321095 - mark it as duplicate of https://bugs.launchpad.net/fuel/+bug/1316583
06:58 mihgen warpc: ok. there are good troubleshooting steps performed and patch provided by Artem - can we use it for proper fix?
06:59 mihgen warpc: at least we would add comment in original bug, that there are some useful info in that duplicate
07:01 warpc mihgen: fix i will look now in env. If it will solve problem, it will be great. In this case we also should change erase_node mcollective agent, because ssh is bash copy of ruby script
07:02 fandi joined #fuel-dev
07:03 AndreyDanin joined #fuel-dev
07:05 AndreyDanin joined #fuel-dev
07:05 artem_panchenko joined #fuel-dev
07:12 teran joined #fuel-dev
07:18 Ch00k_ joined #fuel-dev
07:22 salmon_1 joined #fuel-dev
07:26 warpc akasatkin: https://bugs.launchpad.net/fuel/+bug/1316761/comments/12 please answer)
07:28 akasatkin warpc: manually - what do you mean?
07:29 akasatkin warpc: I used UI only while deploying this.
07:30 warpc akasatkin: ok, it is interesting, thanks. Can you add comment in ticket, please?
07:30 akasatkin warpc: I'll add
07:31 e0ne joined #fuel-dev
07:31 rmoe_ joined #fuel-dev
07:36 warpc akasatkin: thanks. If it reproduce again, please attach a logs. It will be useful because before i only saw logs from Jenkins CI
07:36 teran joined #fuel-dev
07:36 akasatkin warpc: sure
07:48 warpc mihgen: https://bugs.launchpad.net/fuel/+bug/1316583/comments/4 i check possible patch
07:52 mihgen nurla: jkirnosova bogdando holser Folks I don't get what's our default cpu_allocation_ratio?
07:52 mihgen 16 or 1?
07:52 holser 8
07:52 mihgen wow
07:53 mihgen https://review.openstack.org/#/c/90106/1
07:53 mihgen where does 8 comes from?
07:54 holser There should another Gerrit from aglarendil
07:54 holser Let me find it
07:58 mihgen anyway if it is 8, then jkirnosova's env should work by default, should be other reason for "No valid host"
07:58 mihgen jkirnosova: did you enable debug for  openstack logs on settings tab?
07:59 mihgen without debug, it is very hard to understand what's the reason for "no valid hosts". With debug, scheduler log contains pretty good reason
08:00 jkirnosova mihgen: yes, debug is always enabled in my envs. I reproduce this issue not the first time.
08:01 mihgen jkirnosova: so what's the scheduler log says? is there a diag snapshot, I'd like to take a look myself..
08:02 mihgen bogdando: alex_didenko, other  - what do you think about https://review.openstack.org/#/c/93956/2, which is intended to fix "Rabbit service does not start after reboot the controller"
08:02 mihgen still WIP
08:05 mihgen nurla: do you know where we insert a check? https://bugs.launchpad.net/fuel/+bug/1319869
08:07 alex_didenko mihgen: need to try/test it but the idea is good, currently we're not running rabbitmq under corosync/pacemaker
08:07 mihgen Related to No valid host, but different one - as it is for volumes now, https://bugs.launchpad.net/fuel/+bug/1320248
08:09 mihgen wow jkirnosova's logs in scheduler: libvirtError: internal error no supported architecture for os type 'hvm'
08:09 mihgen from https://bugs.launchpad.net/fuel/+bug/1320261
08:14 mihgen jkirnosova: what env do you run it on? Did you try to choose KVM hypervisor?
08:16 mihgen jkirnosova: if nested virtualization is not enabled, you need to choose qemu, which is by default in wizard..
08:17 holser mihgen: https://github.com/stackforge/fuel-library/commit/d0d2f9e3009cdc99c3fb3a38b8a2249173ea11f4
08:17 mihgen holser: ok, thanks
08:18 jkirnosova mihgen, yes, all envs were created with KVM hypervisor
08:19 mihgen jkirnosova: can you retry using qemu?
08:19 mihgen jkirnosova: wait
08:20 mihgen actually can you login into compute node, find /etc/nova/nova.comf, find option … what is that option something like virt_type, and change it to qemu
08:20 mihgen then /etc/init.d/openstack-nova-compute restart
08:21 mihgen or there is similar service name in /etc/init.d/, and then try to create instance again
08:22 jkirnosova mihgen: ok, I'll do it on the next running env. Launched the 213 iso
08:25 mihgen I closed that ticket, please reopen if issue still persists with qemu. Folks, https://bugs.launchpad.net/fuel/+bug/1320248 still remains open and needs investigation why we don't have enough disk space for Cinder volumes
08:39 teran joined #fuel-dev
09:10 EeeBotu New bug #1321194 in fuel: "Need increase timeout for sahara test in system test run" [Medium, In Progress] https://bugs.launchpad.net/bugs/1321194
09:13 alex_didenko joined #fuel-dev
09:13 mirrorbox hi, I have a question again
09:14 mirrorbox I have a warning that is hard to trace
09:14 mirrorbox /var/log/nailgun/app.log:/usr/lib64/python2.6/site-packages/sqlalchemy/orm/dependency.py:458: SAWarning: Object of type <NodeRaidConfiguration> not in session, add operation along 'Node.raids' will not proceed
09:15 mirrorbox http://paste.openstack.org/show/80976/ <-- relevant code pieces are here
09:21 mirrorbox it's not obvious why this message is not shown for NodeAttributes, while the code is generally the same
09:21 anotchenko joined #fuel-dev
09:37 zinovik joined #fuel-dev
09:44 EeeBotu New bug #1321200 in fuel: "Docs need information about configuring syslog in Fuel" [Undecided, New] https://bugs.launchpad.net/bugs/1321200
09:50 mihgen alex_didenko: I think https://bugs.launchpad.net/fuel/+bug/1320248 is going to be hard to reproduce manually
09:51 mihgen alex_didenko: I would encourage you to analyze logs and ask QA team for specifics of test, or just look into test code yourself
09:51 alex_didenko mihgen: I'm going to run exactly the same system test groups that failed in the bug
09:51 mihgen alex_didenko: it is likely to be related to deployment setup used, so logs analysis should help
09:51 meow-nofer mirrorbox, welcome to the mysterious world of SQLAlchemy
09:52 mihgen alex_didenko: ok.
09:53 meow-nofer mirrorbox, if you're creating new model instance, then you definitely should use session.add(new_stuff) to write it into database
09:53 mirrorbox meow-nofer: why the existing code doesn't do that for attributes then?
09:55 meow-nofer mirrorbox, I'm not sure, maybe actual changes are submitted not just by db().flush(), but then some objects are expired within current session
09:55 meow-nofer mirrorbox, anyway, db().add() should be used in both cases :)
09:57 meow-nofer mirrorbox, also I would propose using objects for implementing business logic, not models
09:58 mirrorbox meow-nofer: https://github.com/stackforge/fuel-web/blob/bd28fd5a9da09f3fa8f02bbb43725eef457cd2ea/nailgun/nailgun/api/handlers/node.py#L196
09:58 mirrorbox so, basically, this should not work or I misunderstand something?
10:07 bogdando mihgen, >>> you suggest to increase vcpu, but we should have overcommitment enabled in scheduler, do we? >>>  Perhaps we should disable core filters in scheduler for system tests (https://bugs.launchpad.net/fuel/+bug/1300027)
10:09 bogdando mihgen, oh I see there is a nested kvm issue instead
10:11 mirrorbox haha
10:11 bogdando mihgen, about https://review.openstack.org/#/c/93956/2 it is still WIP, we're discussing this issue, I will put my ideas to the BP https://docs.google.com/a/mirantis.com/document/d/1ftMnLZsQAh9EgoYR4yO9ruyLqgtcoC7OJYgKC8ExtNo/edit#heading=h.106gosm62dg9
10:11 mirrorbox new_raid_conf = NodeRaidConfiguration()
10:11 mirrorbox db().add(new_raid_conf)
10:11 mirrorbox node.raids = new_raid_conf
10:11 mirrorbox db().add(node)
10:11 mirrorbox db().commit()
10:11 mirrorbox doesn't complain
10:11 mirrorbox does't create stuff
10:11 mirrorbox very well...
10:13 Ch00k joined #fuel-dev
10:16 tnurlygayanov Hi!
10:16 tnurlygayanov I have a question about the OpenStack installation
10:17 tnurlygayanov I have installed OpenStack with MOE 4.1 and when I run '/etc/init.d/openstack-nova-metadata-api restart' I can see that openstack-nova-metadata-api does not work
10:18 tnurlygayanov is it ok or it is bug?
10:26 tnurlygayanov https://bugs.launchpad.net/fuel/+bug/1321226
10:27 jkirnosova mihgen: https://bugs.launchpad.net/fuel/+bug/1320261 is not reproduced on Qemu
10:31 mirrorbox ah, finally fixed the db session problem
10:34 meow-nofer mirrorbox, sorry, was at lunch, what's the solution?
10:35 mirrorbox the final working piece looks this way:
10:35 mirrorbox raid_config = NodeRaidConfiguration()
10:35 mirrorbox node.raids = raid_config
10:35 mirrorbox db().add(raid_config)
10:35 mirrorbox + flush()
10:35 mirrorbox I'm still wondering how the code from the link above works... but nevermind
10:36 meow-nofer mirrorbox, I guess flush just does nothing in these cases
10:37 mihgen tnurlygayanov: https://bugs.launchpad.net/fuel/+bug/1321226 address already in use. You would need to looks who is keeping that address
10:37 meow-nofer mirrorbox, that's because you're just assigning some instance to some object field, and it doesn't mean some object to be created in DB unless you explicitly specify so (or specify foreign keys)
10:42 anotchenko joined #fuel-dev
10:47 EeeBotu New bug #1321226 in fuel: "Service openstack-nova-metadata-api does not work on controller nodes" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1321226
11:13 EeeBotu New bug #1321241 in fuel: "[system_test] Ceph multinode compact sys test fails on ubuntu with proggram mysql error in api" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1321241
11:23 sambork joined #fuel-dev
11:25 Ch00k joined #fuel-dev
11:28 al_ex Hi, is it possible to get the development guide of Fuel 3.2.1? thanks
11:29 al_ex the current version (http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso) has a different environment setup.
11:35 meow-nofer al_ex, hi, unfortunately, I think there is no online development doc for Fuel 3.2.1
11:35 meow-nofer al_ex, the good thing is, you can just get fuel-web 3.2.1 source code and build it yourself
11:36 meow-nofer al_ex, if I remember correctly, current article on building docs by hand works for 3.2.1 as well
11:37 al_ex mhhh...it is a problem for me. but is there an offline version? or is there a old PDF version? it is very important for me
11:40 al_ex <meow-nofer>: thanks for the suggestion
11:41 meow-nofer al_ex, I tried to build html version myself right now, here is the link: https://dl.dropboxusercontent.com/u/1786096/fuel-docs_3.2.1.tar.gz
11:41 meow-nofer al_ex, check this out, maybe it will work for you
11:42 al_ex <meow-nofer> thanks a lot :)
11:51 EeeBotu New bug #1321248 in fuel: "Sometimes instace failed to get Active state according to compute filter return 0 host" [Medium, New] https://bugs.launchpad.net/bugs/1321248
12:06 sambork joined #fuel-dev
12:11 Ch00k joined #fuel-dev
12:13 EeeBotu New bug #1321258 in fuel: "Deployment failed with puppet errors" [High, New] https://bugs.launchpad.net/bugs/1321258
12:26 mihgen warpc: any idea why Fuel CI doesn't update https://review.openstack.org/#/c/93960/ ?
12:28 bogdando ntp issue in current system tests https://bugs.launchpad.net/fuel/+bug/1320815 see comments #2 and #3
12:28 mihgen alex_didenko: bogdando folks you are in same location, please help ikalnitsky for initial troubleshooting, bug https://bugs.launchpad.net/fuel/+bug/1321258 has no hints on possible area of failure
12:28 warpc mihgen i don't see Build Started for patch-set #2
12:29 mihgen bookwar: any idea on Fuel CI ? ^^
12:30 alex_didenko mihgen: ok
12:31 mihgen holser: did you create bug regarding sahara?
12:33 holser mihgen, I’ve just come back from lunch, doing now …
12:36 salmon_1 mihgen: finalli I was able to reproduce https://bugs.launchpad.net/fuel/+bug/1317548 with system tests. Now I'm trying to reproduce it manually
12:39 salmon_ oh, it was fixed in the meantime :D
12:39 salmon_ I will check on new iso
12:51 EeeBotu New bug #1321276 in fuel: "RabbitMQ in master node lose connection from local clients" [High, Confirmed] https://bugs.launchpad.net/bugs/1321276
12:57 tatyana joined #fuel-dev
13:08 mihgen holser: let me know please update on this, and let this know to tnurlygayanov
13:08 holser k
13:08 holser https://bugs.launchpad.net/fuel/+bug/1321284
13:14 EeeBotu New bug #1321284 in fuel: "Sahara deployment fails" [High, New] https://bugs.launchpad.net/bugs/1321284
13:19 Ch00k joined #fuel-dev
13:19 e0ne_ joined #fuel-dev
13:21 bookwar Fuel CI voting is broken, several patchsets went without triggering tests, working on this right now
13:46 holser mihgen patch for https://bugs.launchpad.net/fuel/+bug/1321284 is under CI tests
13:49 bookwar Issue with Fuel CI has been resolved. If you find review without comments from Fuel CI - let me know.
13:53 EeeBotu New bug #1321295 in fuel: "Mistake in the inform message fuel cli." [Low, New] https://bugs.launchpad.net/bugs/1321295
13:53 EeeBotu New bug #1321289 in fuel: "fakeui build failed" [Critical, Confirmed] https://bugs.launchpad.net/bugs/1321289
13:55 mihgen alex_didenko: https://bugs.launchpad.net/fuel/+bug/1320248 was reproduced by apalkina manually
13:57 e0ne joined #fuel-dev
13:57 akupko joined #fuel-dev
13:59 alex_didenko mihgen: ok, will check the snapshot
14:07 salmon_ mihgen: I tested https://bugs.launchpad.net/fuel/+bug/1317548 on newest ISO with fix and I'm still getting the same error
14:09 mihgen salmon_: thanks, can you add your steps to reproduce to the ticket and diagnostic snapshot ?
14:09 mihgen may be some other observations in logs?
14:09 salmon_ mihgen: wait, I will try again
14:10 salmon_ I didn't update fuel-main... Just realized that ISO version shouldn't matter
14:20 mihgen vkramskikh: are you working on https://bugs.launchpad.net/fuel/+bug/1321289 ?
14:21 vkramskikh mihgen i'm talking with teran about it, still investigating
14:27 EeeBotu New bug #1321302 in fuel: "Deadlocks in tests in stop deployment task" [High, In Progress] https://bugs.launchpad.net/bugs/1321302
14:57 mattymo|home joined #fuel-dev
14:58 kat_pimenova joined #fuel-dev
14:59 IlyaE joined #fuel-dev
15:15 e0ne joined #fuel-dev
15:39 ikalnitsky hi guys. could you please take a look at my comment for this bug? https://bugs.launchpad.net/fuel/+bug/1320910 It would be great if someone suggest a good solution.
15:42 bogdando please review https://review.openstack.org/#/c/93896/ fix for ntp sync issue
15:56 jobewan joined #fuel-dev
16:13 rmoe joined #fuel-dev
16:43 dshulyak_ joined #fuel-dev
16:54 Ch00k joined #fuel-dev
16:55 EeeBotu New bug #1321358 in fuel: "Changes in the command line does not correspond to changes in the Web Ui" [Low, New] https://bugs.launchpad.net/bugs/1321358
16:56 mihgen vkramskikh: any update on https://bugs.launchpad.net/fuel/+bug/1321289 ?
16:57 vkramskikh nope :( it is very strange
16:58 xarses joined #fuel-dev
16:59 mihgen does it reproduce locally?
17:00 mihgen does it affect ISO build?
17:02 vkramskikh no it doesn't, only on fuelweb.vm
17:02 vkramskikh it works fine on my pcs
17:02 vkramskikh it doesn't affect ison build if it builds not on fuelweb.vm.mirantsi.net :)
17:03 vkramskikh it just stopped working there without any visible reason
17:03 mihgen ok, I move it out of 5.0 then, but it is still critical to fix. involve devops engineers into troubleshooting..
17:05 bookwar vkramskikh: let's see if i can regenerate the environment here on Fuel CI https://fuel-jenkins.mirantis.com/job/stackforge-verify-fuel-web/3518/console and then maybe rerun fakeui job with clean env
17:33 artem_panchenko joined #fuel-dev
18:00 teran joined #fuel-dev
18:01 teran_ joined #fuel-dev
18:07 testmaniac joined #fuel-dev
18:07 teran joined #fuel-dev
18:07 testmaniac wasn't the release of fuel 5.0 supposed to be out last Friday?  Where should I download it from to test it out?  All i see at the Mirantis site is fuel 4.1.
18:18 bookwar mihgen, vkramskikh do you know any reason why we have npm-1.2.15 installed locally on fuelweb fakeui vm?
18:20 bookwar there is npm-1.3.6-5.el6 available in EPEL and it works with no problems
18:37 teran joined #fuel-dev
18:43 mihgen bookwar: I think vkramskikh has no idea about it… someone hacked it before
18:43 mihgen bookwar: let's fix it, you are the boss
18:46 mihgen testmaniac: we have a few critical fixes still. tentatively early next week it should go out as Mirantis OpenStack: https://wiki.openstack.org/wiki/Fuel/5.0_Release_Schedule#Important_Dates_for_Fuel_5.0
18:47 teran joined #fuel-dev
18:47 mihgen testmaniac: if you would like to, you can create a community version of Fuel: http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso
18:47 mihgen you can check out our launchpad to get list of remaining defects we are still working on
18:48 testmaniac @mihgen: thanks a lot.
18:49 teran_ joined #fuel-dev
18:52 e0ne joined #fuel-dev
19:27 evgeniyl joined #fuel-dev
19:30 e0ne joined #fuel-dev
19:39 e0ne joined #fuel-dev
19:45 Bomfunk joined #fuel-dev
20:01 mihgen xarses: hi Andrew
20:02 mihgen xarses: do you know any status on ceph bug https://bugs.launchpad.net/fuel/+bug/1317548 ?
20:17 IlyaE joined #fuel-dev
20:19 xarses joined #fuel-dev
20:20 jaypipes joined #fuel-dev
20:27 xarses joined #fuel-dev
20:27 xarses mihgen: hi
20:48 mihgen xarses: hey
20:49 xarses how's it going
20:49 mihgen wondering if you guys verified that nailgun also fine with kombu & pyamqp upgrade (if it uses the latter)
20:49 xarses rmoe: ^^
20:50 mihgen and what's our plan for ceph bug https://bugs.launchpad.net/fuel/+bug/1317548
20:50 rmoe I didn't
20:50 rmoe I'll test it
20:50 rmoe but even if it doesn't work can't we just have two versions of each in our repo and force nailgun to use the old version?
20:51 xarses mihgen: 1317548, can't reproduce on 212 by hand
20:51 mihgen rmoe: we can I think :) but it will require changes somewhere else, not only just package upgrade
20:53 mihgen xarses: ok.. can you please comment in the ticket at least that.. I think we still need to analyze logs - wondering if provided exception http://paste.openstack.org/show/80887/ is any related to a bug
20:53 mihgen is angdraug hiding btw?)
20:54 EeeBotu New bug #1321451 in fuel: "Upgrade kombu and pyamqp" [High, Confirmed] https://bugs.launchpad.net/bugs/1321451
20:54 xarses mihgen: he's joining
20:54 angdraug joined #fuel-dev
20:55 mihgen we have a number of weird issues still…. http://fuel-launchpad.mirantis.com/project/fuel/bug_table_for_status/Open/5.0
20:59 xarses mihgen: can we get the failed ci tests marked with the iso/git SHA in future? Makes it easier to know if they may have been fixed already
21:00 mihgen xarses: are you talking about ..?
21:00 xarses like https://bugs.launchpad.net/fuel/+bug/1317548
21:01 xarses having output from /api/version would be helpful
21:01 mihgen that's aglarendil's bad that he didn't specify output of /api/version.. feel free to blame him for that :)
21:01 xarses without having to dive through the log bundle
21:01 mihgen yep
21:04 mihgen xarses: failed by timeout on latest run (ISO #213): http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_1/61/
21:05 mihgen same for ubuntu. further analysis requires going into snapshot
21:11 IlyaE joined #fuel-dev
21:22 IlyaE joined #fuel-dev
21:38 mihgen -1 bug: https://bugs.launchpad.net/fuel/+bug/1320248 NTP issues, and still 11 in a row…actually "migrate_vm_backed_with_ceph system test fails" could also be affected by NTP
21:50 mihgen angdraug: https://review.openstack.org/#/c/93883/ constantly fails on Fuel CI. I believe an issue is in patch
21:50 xarses mihgen: found another serious issue, but should be easy to fix, writing bug
21:52 mihgen xarses: ok...
21:54 angdraug mihgen, it failed on ubuntu and passed on centos, the patch is distro-agnostic
21:55 mihgen not really sure, ubuntu is 1 controller HA there as far as I know
21:55 mihgen centos is multimode simple mode
21:56 mihgen ubuntu test usually passes in 35 min, but in this case it doesn't pass in 1h10min, so that's why build fails
21:56 angdraug yet another reason why we shouldn't be nesting multimode anymore...
21:57 mihgen well thing is that multimode actually passes in 15-17 min
21:57 mihgen and it is 2x faster ability to validate many openstack services
21:58 mihgen so I don't really know what to do, even though I want to drop multinode no less than you
21:58 xarses mihgen: https://bugs.launchpad.net/fuel/+bug/1321466
21:59 angdraug 1) fix ha, it shouldn't take extra 20 minues
21:59 angdraug 2) move move tests into fast unit tests
21:59 mihgen yep, though not sure about 1st - we install a lot of stuff for HA
21:59 angdraug move move -> move more
21:59 xarses mihgen: deploy time is much longer again for ha in 212 as compared to 2 weeks ago
22:00 xarses its borderline not acceptable
22:00 angdraug installing it shouldn't take that much time, bringing it all up takes more
22:00 xdeller joined #fuel-dev
22:00 mihgen xarses: interesting. we had somewhere script which shows who eats the time...
22:00 xarses puppet took over 1800s for primary controller
22:02 xarses mihgen: yes, I know the evaltrace timers can be poked at, and there is a ruby parser that will pull it up. I made some other code around parsing more of it since evaltrace can be timed wrong
22:02 xarses It could be adapted to read the tarball's and produce a report
22:03 mihgen I think we need a CI job which would draw a graph on daily basis..
22:03 xarses mihgen: probably
22:04 mihgen xarses: that would be interesting, we can produce checks based on it, so Fuel CI can -1 if some limit is exceeded
22:05 mihgen xarses: do you have that script?
22:05 xarses the puppet evaltrace parser or mine? well i have both
22:05 angdraug when will we be able to detect puppet crashes?
22:05 mihgen we can create a bug on fuel-devops, and once they sort out other high priority staff they could make it happen
22:06 mihgen angdraug: what do you mean?
22:06 angdraug that's what I see in that failed CI job
22:06 mihgen I didn't hear any bugs regarding puppet crash and astute's inability to detect for months
22:07 mihgen so I don't know what you see… can you tell more?
22:07 angdraug https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/894/
22:07 angdraug first entry in puppet-apply.log from primary controller:
22:07 angdraug 2014-05-20T20:07:42
22:07 angdraug it all stops at 2014-05-20T20:08:34
22:08 angdraug 2014-05-20T20:08:34.404295+00:00 info: (/Stage[first]/Rsyslog::Client/File[/etc/rsyslog.d/]) Starting to evaluate the resource
22:08 angdraug which indicates that it crashed while parsing the catalog
22:09 angdraug there's one more entry at 2014-05-20T20:32:39
22:09 angdraug and ps output has: root     24351 11.5 30.4 560732 466616 ?       Ss   20:45   1:25 /usr/bin/ruby /usr/bin/puppet apply
22:09 angdraug which indicates that it was restarted at 20:45
22:10 angdraug interestingly enough there's a haproxy process started at 20:50
22:10 angdraug which implies that puppet did run (or haproxy wouldn't be running), just didn't log anything
22:11 mihgen angdraug: I'm pretty sure it is NTP issue again
22:11 angdraug not sure, there's a clock jump in astute.log, but it was at 2014-05-20T20:05:44, before puppet was started
22:12 angdraug also uptime has 20:57:53 up 52 min
22:13 mihgen astute is master node, remember
22:15 mihgen xarses: can you please create bug to fuel-devops about what we've discussed? and add your script there...
22:16 angdraug times in the controller logs add up with timestamps in astute.log on master
22:17 angdraug well of course they would since they're reported by rsyslog
22:18 EeeBotu New bug #1321466 in fuel: "vip__public_old cant reach other nodes mgmt addr" [Critical, Confirmed] https://bugs.launchpad.net/bugs/1321466
22:19 mihgen folks I'm off. have a good day.
22:22 xarses mihgen: have a good night https://bugs.launchpad.net/fuel/+bug/1321473
22:29 IlyaE joined #fuel-dev
22:55 EeeBotu New bug #1321473 in fuel: "[devops] create CI test around deployment-runtime and report puppet runtime breakdown" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1321473
22:58 IlyaE joined #fuel-dev
23:14 IlyaE joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary