Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2016-05-23

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:17 gardlt joined #fuel-dev
04:30 cartik joined #fuel-dev
05:44 nadya joined #fuel-dev
07:18 kszukielojc joined #fuel-dev
07:24 salmon_ joined #fuel-dev
07:31 shardy joined #fuel-dev
07:51 HeOS joined #fuel-dev
07:58 skolekonov_ joined #fuel-dev
08:14 14WAAC81Y joined #fuel-dev
08:14 noshankus joined #fuel-dev
08:26 Samos123 joined #fuel-dev
08:35 Samos123 joined #fuel-dev
08:35 yassine joined #fuel-dev
08:39 mmalchuk_ joined #fuel-dev
08:50 javeriak joined #fuel-dev
09:15 iberezovskiy joined #fuel-dev
09:16 iberezovskiy joined #fuel-dev
09:17 iberezovskiy joined #fuel-dev
09:19 degorenko joined #fuel-dev
09:21 ekosareva_ joined #fuel-dev
09:21 ekosareva joined #fuel-dev
09:47 javeriak joined #fuel-dev
10:03 BobBall joined #fuel-dev
10:12 BobBall Hey - we've got a f8 env where the bootstrap node keeps rebooting.  Don't see anything interesting in /var/log/remote/<ip>/bootstrap yet.  Any hints for debugging?  e.g. a way to stop a reboot in the bootstrap node?
10:14 mattymo BobBall, how far does the node get before rebooting?
10:15 BobBall Where would I monitor progress for them?
10:15 mattymo astute might not think the node was properly deleted. I'm not sure how, but there might be a task in nailgun remaining for delete/reset a node
10:15 BobBall The console shows the node starting up and I can login for a few seconds before it deletes
10:15 mattymo `fuel task` will show any outstanding tasks
10:15 mattymo on master node
10:15 BobBall Will check
10:16 mattymo then fuel task --delete -f --tid $tasknum
10:17 mattymo possibly another issue is you don't have local disk in the boot order on the node, it got provisioned okay, but it won't boot into the deployed OS
10:18 astupnikov joined #fuel-dev
10:18 astupnikov Dear core members of fuel-web team, please review https://review.openstack.org/#/c/305714/ . I need your +2 :)
10:19 BobBall We do have local disk, and previous deploy was successful.  We hit some controller issues (seemed that one of the controller nodes was broken and wouldn't get the floating IP) and we decided to redeploy that env rather than continuing to investigate that particular issue.
10:21 mattymo BobBall, if deleting the task doesn't help, can you send me /var/log/astute* (some useful info might be rotated already) and I'll try to figure out why it is rebooting
10:22 BobBall You're correct - there are 3x reset env and 1x node_deletion tasks
10:22 BobBall status ready, progress 100%
10:22 BobBall maybe they are finished?
10:22 mattymo 100 means it's done
10:23 mattymo are there any not at 100?
10:23 BobBall nope, all 100
10:24 mattymo ok we need logs to see why
10:27 BobBall Also, is there a known bug with the tftp server used?  We occasionally get lots of nodes hanging when downloading the initrd until we reboot the FM...
10:28 BobBall We were just getting lots of hangs, so rebooted the FM to clear those up and it seems that the bootstrap loop has been fixed too...
10:30 mattymo it happens.. moving to gpxe or ipxe is http based and better, but not all hardware supports it
10:32 BobBall Is that a configurable thing, or does Fuel only use tftp because of that hardware issue?
10:37 mattymo it's in dnsmasq config.. we don't make it configurable. it's incredibly hard to debug
10:37 mattymo so we leave standard tftp based pxe
10:38 BobBall Shame :)
10:39 BobBall We'll raise a bug on the bootstrap hangs when we repro it next; but it seems to be quite reliable after redeploying nodes lots of times
10:42 jianghuaw__ joined #fuel-dev
11:09 bookwar_ left #fuel-dev
11:18 HeOS_ joined #fuel-dev
11:32 nokes joined #fuel-dev
11:41 nadya joined #fuel-dev
12:13 nokes joined #fuel-dev
12:29 Liuqing joined #fuel-dev
12:33 nadya joined #fuel-dev
12:50 swann joined #fuel-dev
13:18 bogdando joined #fuel-dev
13:40 purplerbot joined #fuel-dev
13:48 krobzaur joined #fuel-dev
13:58 jobewan joined #fuel-dev
14:01 zigo mattymo: After provisionning of Debian with the 9.0 tag of Fuel, I get astute that reports:
14:02 zigo 2016-05-23 13:59:30 DEBUG [12135] Not target nodes will be rejected:
14:02 zigo 2016-05-23 13:59:30 DEBUG [12135] Not provisioned: 1,2, got target OSes:
14:02 zigo 2016-05-23 13:59:30 DEBUG [12135] Still provisioning following nodes: ["1", "2"]
14:02 zigo And that's it...
14:02 zigo It turns in loop.
14:02 nadya joined #fuel-dev
14:02 zigo In the nodes, I can see in nailgun-agent.log:
14:02 zigo I, [2016-05-23T14:01:10.579955 #6421]  INFO -- : Discover prevented by /etc/nailgun-agent/nodiscover presence.
14:02 zigo D, [2016-05-23T14:02:01.748355 #6433] DEBUG -- : Sleep for 5 seconds before sending request
14:03 zigo Is this normal?
14:03 zigo What's going wrong?
14:32 mattymo zigo, sorry I missed your message
14:33 mattymo zigo, I'm starting a local deploy. let me explain about node statuses in just a few mins
14:42 xarses_ joined #fuel-dev
14:46 omolchanov joined #fuel-dev
14:46 omolchanov left #fuel-dev
14:58 nadya joined #fuel-dev
15:07 zigo mattymo: Ok.
15:08 ekosareva joined #fuel-dev
15:12 BobBall Anyone know what might cause a repeatable failure on only one of two my secondary controller nodes during a full deploy?  Failure to deploy dns-server and on the failed controller I get http://paste.openstack.org/show/498206/? This first error is listed in a few LP bugs but several of those bugs have been marked as invalid - because it was fixed already or the user wasn't using the UI to do a full deploy
15:12 BobBall ment
15:13 mattymo zigo, I forget how nailgun determines target type... bootstrap or "target"
15:14 mattymo but that has something to do with how it figures out if we've installed a node right
15:14 mattymo it's some astute mco agent code
15:14 zigo mattymo: After the nodes have rebooted in the provisionned OS, it should be replying to "mco ping" no?
15:19 mattymo warpc, ping
15:19 mattymo zigo, it's /etc/nailgun_systemtype
15:20 mattymo it should say target or bootstrap
15:20 zigo It says target for the nodes.
15:22 zigo Fuel hasn't written /etc/mcollect/client.cfg correctly. It has plugin.activemq.pool.1.host = stomp1
15:22 zigo That's obviously wrong.
15:22 zigo I'm not sure what component would write it.
15:22 zigo Maybe it didn't go up to there...
15:51 nadya joined #fuel-dev
15:53 nadya joined #fuel-dev
16:34 xarses joined #fuel-dev
16:37 charz joined #fuel-dev
17:04 mattymo zigo, it should get it from cloud-init
17:04 mattymo maybe the cloud-init template isn't compatible with debian.. or the package isn't installed
17:04 mattymo (sorry I had a meeting)
17:25 jobewan joined #fuel-dev
17:32 jobewan joined #fuel-dev
18:13 degorenko joined #fuel-dev
19:41 nadya joined #fuel-dev
20:09 zigo mattymo: Cloud-init is installed, and it did run.
20:09 zigo mattymo: Where is stored the template?
20:27 nadya joined #fuel-dev
21:09 mwhahaha zigo: the cloud init templates come from fuel-agent
21:09 mwhahaha https://github.com/openstack/fuel-agent/tree/master/cloud-init-templates
21:09 zigo mwhahaha: Cheers.
21:10 zigo mwhahaha: I don't have them in my fuel-agent package indeed.
21:10 zigo mwhahaha: Where am I supposed to push them to (ie: what location in the filesystem of the master node)?
21:10 zigo Oh, I do have them.
21:11 zigo In /usr/share/fuel-agent/cloud-init-templates
21:11 mwhahaha might be we don't have a debian specific one so it's failing
21:11 mwhahaha not sure how they get picked
21:12 zigo I got a /usr/share/fuel-agent/cloud-init-templates/boothook_debian_jessie_x86_64.jinja2 :)
21:12 zigo But maybe it's outdated.
21:12 zigo mwhahaha: Do you remember if it changed between earlier this year and now?
21:13 mwhahaha the templates don't look changed
21:14 mwhahaha i think the config gets written out to a config disk so you could check in it
21:14 mwhahaha oh looks like they have versioned ones
21:14 mwhahaha https://github.com/openstack/fuel-agent/commit/11e535e6e396afed03627f48ac5e1229a7cf6cf9
21:14 zigo Hum... looks like at least names changed and stuff...
21:14 zigo Yup.
21:15 zigo I'll investigate this tomorrow then.
21:16 zigo Thanks a lot for this very useful hint.
21:16 mwhahaha sure
21:30 salmon_ joined #fuel-dev
22:49 izinovik- joined #fuel-dev
22:50 roman_vyalov joined #fuel-dev
22:50 Samos123_ joined #fuel-dev
22:51 tlbr_ joined #fuel-dev
22:51 dougshelley66_ joined #fuel-dev
22:52 intr1nsic_ joined #fuel-dev
22:52 zigo_ joined #fuel-dev
22:54 dburmistrov joined #fuel-dev
22:55 vkramskikh joined #fuel-dev
22:58 MiroslavAnashkin joined #fuel-dev
23:39 IvanKliuk_ joined #fuel-dev
23:39 apopovych joined #fuel-dev
23:39 omolchanov_ joined #fuel-dev
23:43 mescanef joined #fuel-dev
23:44 yantarou joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary