Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-07-24

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:23 sergmelikyan joined #fuel
00:55 skylerberg joined #fuel
00:56 skylerberg Hi all. I am getting an error when I try to pxe boot a node: permission denied.
00:56 skylerberg Anyone know why I might get this error?
00:58 mwhahaha permission denined where?
00:58 skylerberg On the node that is trying to boot from the network. It says tftp://10.0.2.4/fuel-node.pxe... Permission denied
00:58 mwhahaha you might try kicking the cobbler docker container
00:59 mwhahaha that seems odd
01:01 skylerberg That is probably it. I am not connected to the internet on the master, so there could be a problem with docker.
01:01 skylerberg Should I get a connect set up, destroy all the docker stuff and then run the bootstrap script?
01:04 mwhahaha might be a good thing to try, or just check file perms/selinux
01:08 mwhahaha could just try restarting it too
01:10 xarses joined #fuel
02:09 evgenyl joined #fuel
02:19 mattymo_ joined #fuel
02:31 hezhiqiang joined #fuel
02:42 tzn joined #fuel
02:43 _tzn joined #fuel
02:52 hakimo_ joined #fuel
02:59 bildz joined #fuel
03:31 jobewan joined #fuel
03:53 tzn joined #fuel
04:12 rmoe joined #fuel
05:40 julien_ZTE joined #fuel
05:44 Akshik joined #fuel
06:01 sergmelikyan joined #fuel
06:08 Miouge joined #fuel
06:36 mkwiek joined #fuel
07:17 monester joined #fuel
07:26 ub joined #fuel
07:34 tzn joined #fuel
07:36 evgenyl joined #fuel
07:39 hyperbaba joined #fuel
08:02 sergmelikyan joined #fuel
08:27 hezhiqiang joined #fuel
08:29 hezhiqia_ joined #fuel
08:44 hezhiqiang joined #fuel
08:57 hezhiqiang joined #fuel
09:03 sergmelikyan joined #fuel
09:10 rmallah joined #fuel
09:10 rmallah hi where to set global timezone in fuel environment for all nodes during provisioning
09:14 tatyana joined #fuel
09:35 hezhiqiang joined #fuel
09:42 e0ne joined #fuel
09:43 hezhiqiang joined #fuel
09:50 devvesa joined #fuel
09:59 HeOS joined #fuel
10:02 e0ne_ joined #fuel
11:02 xarses_ joined #fuel
11:03 andreww joined #fuel
11:03 sergmelikyan joined #fuel
11:04 andreww joined #fuel
11:10 teran joined #fuel
11:14 devvesa joined #fuel
11:35 asteer joined #fuel
11:56 teran_ joined #fuel
12:34 sergmelikyan joined #fuel
12:41 sergmelikyan joined #fuel
12:45 mquin joined #fuel
12:46 dkusidlo joined #fuel
12:59 jdandrea joined #fuel
13:05 jhova joined #fuel
13:30 xarses_ joined #fuel
14:01 mkwiek joined #fuel
14:06 claflico joined #fuel
14:17 dkusidlo joined #fuel
14:48 jaypipes joined #fuel
14:53 thansen joined #fuel
14:57 blahRus joined #fuel
14:58 dkusidlo joined #fuel
15:16 angdraug joined #fuel
15:17 richoid joined #fuel
15:20 _tzn joined #fuel
15:47 sergmelikyan joined #fuel
15:56 bitblt joined #fuel
16:08 rward joined #fuel
16:16 xdeller joined #fuel
16:45 mquin joined #fuel
17:13 rmoe joined #fuel
17:38 sergmelikyan joined #fuel
17:57 e0ne joined #fuel
18:06 skylerberg joined #fuel
18:12 xarses joined #fuel
18:42 jaycee joined #fuel
19:00 alwaysatthenoc joined #fuel
19:45 sergmelikyan joined #fuel
19:49 ub joined #fuel
19:54 ub2 joined #fuel
20:45 HeOS joined #fuel
20:53 Verilium Hmm, with 6.1 stable, are there any issues with a deployment with GRE?  I just got a "Deployment has failed. Timeout of deployment is exceeded." and not quite sure why.
20:54 mwhahaha nope, check the astute log
20:54 mwhahaha do you know what part of the process it was at?
20:54 mwhahaha did it deploy your OS already or was it early on in the building process
20:55 mwhahaha if it was early on in the image building process you may be having issues with IO on the fuel master as it builds the OS image
20:55 Verilium No, no idea specifically.  I kind of just left it running (it's been around 1h30, trying to deploy 5 nodes).
20:55 Verilium 2015-07-24T20:45:48 err: [543] Timeout of deployment is exceeded.
20:55 Verilium Before that though...
20:55 Verilium 2015-07-24T20:45:46 debug: [543] Node 6(controller) status: running
20:56 Verilium mwhahaha:  OS deployment was done, iirc.  Lemme try and connect to the nodes...
20:56 mwhahaha look in /var/log/docker-log/astute/astute.log and look for errors
20:57 mwhahaha deployment exceeded on a particular task so trying to narrow down what was hanging
20:57 Verilium I was trying to deploy 3 controllers and 2 computes.  The 2 computes show ubuntu is installed.  The 3 controllers are those with an error.
21:05 Verilium Hmm, not sure, looks like there's a lot of tasks that have been repeating for the last hour.
21:05 mwhahaha which one
21:06 mwhahaha you could also look in the puppet log for node 1 and search for 'err:' as that'll probably have what went wrong
21:09 Verilium mwhahaha:  http://paste.openstack.org/show/405464/
21:10 mwhahaha so it's waiting for the db to come up
21:10 mwhahaha if you run 'clustercheck' on one of the controller nodes, what does it say?
21:11 Verilium Hmm, the puppet logs on 2 of the 3 nodes are showing finished catalog run.
21:11 mwhahaha check the puppet.log for the nodes and look for an error there
21:13 Verilium 2015-07-24 21:10:44ERR
21:13 Verilium (/Stage[main]/Main/Haproxy_backend_status[mysql]) Could not evaluate: Could not get CSV from url http://192.168.0.2:10000/;csv
21:13 Verilium Hmm...
21:13 Verilium 2 out of the 3 nodes have this as ERR level.
21:14 mwhahaha right the node that it worked on is the one where that is running
21:14 mwhahaha so it might be a connectivity thing between the nodes over that network
21:15 mwhahaha you should be able to run 'ip netns list' on the good node and you should see a haproxy namespace
21:15 mwhahaha make sure that the other nodes can ping that ip and that haproxy is running in that namespace
21:18 Verilium Hmm, yeah, the 2 other nodes can't ping 192.168.0.2.
21:20 Verilium How is 192.168.0.2 configured on that node where it's pingable?  Not seeing the IP in question as being configured on the machine.
21:20 Verilium mwhahaha:  clustercheck comes back OK on all 3 nodes.
21:20 Verilium as in, "is synced".
21:20 mwhahaha it'd be in the network namespace
21:20 Verilium Oh, netns.
21:20 mwhahaha do an 'ip netns list'
21:21 mwhahaha it'll probably be in the haproxy one
21:21 mwhahaha where ever that is living
21:21 Verilium Right, was expecting a vip along the lines of keepalived or something, ok.
21:21 e0ne joined #fuel
21:22 Verilium Hmm, well, the 2 other nodes don't have a 192.168.0 ip on their haproxy ns...
21:23 mwhahaha so they shouldn't
21:23 mwhahaha they should be able to get to that over a different interface
21:23 mwhahaha i forget which interface let me see
21:24 Verilium Ahh, ok, I see the routes.  Through br-mgmt.  Hmm...
21:24 mwhahaha so it points to your management network not being setup correctly which is odd if you passed network validation
21:24 Verilium Yep, that part worked fine.
21:24 Verilium I wonder if I'm stumbling on something vmware related, again...
21:25 Verilium All 5 nodes are on one ESXI host.
21:25 mwhahaha you'd think that' d be less of a problem
21:26 mwhahaha but the issue points to network connectivity problems, so if you fix that you should be able to redeploy and it'll go through
21:26 Verilium Hmm, the 192.168.0 ips can ping each other.
21:26 mwhahaha but you can't ping the vip?
21:26 Verilium They just can't ping the .2 that's in the haproxy ns.
21:27 mwhahaha you can go into the network namespace and try and ping the other stuff
21:27 mwhahaha 'ip netns exec haproxy bash'
21:29 Verilium Running a tcpdump -i any icmp on one of the nodes that can't reach the vip at .2.
21:29 Verilium And doing a ping from the .2 namespace, and the node isn't seeing anything coming in.
21:30 Verilium Well, from the haproxy ns, I can ping the public IP's gateway...
21:32 Verilium From that haproxy ns, I can also ping 240.0.0.1, which is one of the other nodes.
21:32 mwhahaha there is a br-mgmt-hapr interface on the controller node right?
21:32 mwhahaha the one with haproxy running
21:32 mwhahaha you should at least be able to ping the br-mgmt interface of the same host from the haproxy namespace
21:33 Verilium There isn't.
21:33 mwhahaha i wonder if it's not there for gre
21:33 mwhahaha i've just got a vlan deploy i'm looking at
21:34 mwhahaha anyway you could try and submit a bug, https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Test_and_report_bugs as this seems weird
21:34 Verilium The few results I found googling for this error seemed to mention a GRE deployment, which made me wonder if that might be something.
21:34 mwhahaha i'm not sure what else to go look for
21:34 mwhahaha alternatively reset the deploy and try again, the good ol' reboot and try again plan :D
21:35 Verilium Haha, yeah, I will, nothing to lose at this point.
21:35 Verilium Maybe I'll try with just 1 controller at first, at least see if things complete.
21:36 mwhahaha that will probably work just fine
21:36 mwhahaha but you might run into compute issues if they too can't talk to the haproxy stuff
21:36 Verilium And here I was hoping it was all nice and dandy magic and I just had to click a button. ;)
21:36 Verilium Hmm, yeah.
21:39 Verilium I'll probly true out a vlan deployment too just to compare.
21:39 Verilium Or maybe try with the latest nightly build...
21:39 mwhahaha gre is supposed to just work
21:39 mwhahaha the nightly build may have more craziness, but you're welcome to try it :D
21:39 Verilium Hmm, 1 month difference with stable.
21:40 Verilium I have enough physical hosts.  Might try it out that way too, just in case this is all vmware-related, or something.
21:40 mwhahaha yea let's just say there's a lot of changes in 6.1 vs 7.0
21:40 mwhahaha ymmv
21:41 Verilium Yeah.  We actually want to wait for 7.0 for installation for our production environment.
21:41 Verilium I mostly want to get familiar with fuel with 6.1, but ultimately, I'm waiting for DVR with 7.0.
21:42 Verilium Still liking what I see with fuel up to now.  Especially considering my previous installs were straight from vanilla openstack.
21:47 Verilium Ah, this was from before.
21:47 Verilium 24-07-2015 20:45:48Deployment has failed. Timeout of deployment is exceeded.
21:47 Verilium 24-07-2015 20:45:48Failed to deploy node 'Untitled (aa:57)': Unknown error
21:47 Verilium 24-07-2015 20:45:48Failed to deploy node 'Untitled (af:b4)': Unknown error
21:47 Verilium Gotta love unknown errors, heheh.
21:47 mwhahaha yea our error messaging needs work :/
21:48 Verilium Hehe, all in time.
21:51 Verilium Anyhow, I'll test out a few other scenarios first and probably come back to this.  If it happens again, I'll submit a report.
21:51 mwhahaha ok
21:55 Verilium Thanks for the help btw.
21:56 mwhahaha no problem
21:59 * mwhahaha wanders off ... (╯°□°)╯︵ ┻━┻
22:01 Verilium Same here, friday 6pm, time to continue with this later. :P
23:50 osryan_ joined #fuel
23:50 skath_ joined #fuel
23:54 nurla joined #fuel
23:57 rmoe joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary