Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2016-07-20

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 Egyptian[Home] joined #fuel
00:33 code-R joined #fuel
00:35 code-R_ joined #fuel
00:54 xarses joined #fuel
01:34 Liuqing joined #fuel
01:37 Zer0Byte__ joined #fuel
01:40 Zer0Byte__ hey
01:48 ilbot3 joined #fuel
01:48 Topic for #fuel is now Fuel 8.0 (Liberty) https://www.fuel-infra.org/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
02:25 rmoe joined #fuel
02:31 rmoe joined #fuel
02:37 jianghuaw joined #fuel
02:52 jianghuaw joined #fuel
02:59 jianghuaw joined #fuel
03:00 Liuqing joined #fuel
03:23 xarses joined #fuel
03:51 charz joined #fuel
04:17 rmoe joined #fuel
04:45 code-R joined #fuel
05:20 javeriak joined #fuel
05:39 aneliubin joined #fuel
06:24 Zer0Byte__ joined #fuel
07:07 javeriak joined #fuel
07:13 Sesso joined #fuel
07:25 Egyptian[Home] joined #fuel
07:28 AlexAvadanii joined #fuel
07:40 DavidRama joined #fuel
07:53 Sesso joined #fuel
08:04 javeriak joined #fuel
08:06 azvyagintsev joined #fuel
08:09 code-R joined #fuel
08:25 Sesso joined #fuel
08:26 noshankus joined #fuel
08:51 ekosareva_ joined #fuel
08:54 rtikunov please, please review https://review.openstack.org/#/c/342855/
09:29 ekosareva_ joined #fuel
09:39 javeriak joined #fuel
10:49 vkulanov joined #fuel
10:56 sbog1 joined #fuel
10:56 sbog1 left #fuel
11:00 ekosareva_ joined #fuel
11:11 yassine joined #fuel
11:18 agordeev fuelers, please be informed that fuel-agent CI is broken due to https://bugs.launchpad.net/fuel/+bug/1580199
12:09 Liuqing joined #fuel
12:16 yassine joined #fuel
12:26 Stern_Winter joined #fuel
12:26 Stern_Winter left #fuel
12:44 javeriak joined #fuel
12:50 Sketch mwhahaha: well, fixed the typo and tried to rebuild the image, it still errors out.  http://pastebin.com/fLLS6XXE
12:52 Sketch aha, the real log points me to an actual build log
12:52 Sketch ok, the problem is dkms
12:53 Sketch it's trying to build against the centos kernel
12:54 Sketch which probably wouldn't work even if it was successful
12:54 Sketch i guess i'll just copy the dkms built binary from the bootstrap and make a hacky binary package
13:17 mwhahaha it shouldn't be building against the centos kernel, that's weird
13:17 goldenfri joined #fuel
13:25 javeriak_ joined #fuel
13:26 javeriak joined #fuel
13:32 Sketch i guess it has to build on the fuel server if it's building a pre-installed image
13:32 Sketch maybe i need to pass it options for what kernel to build against
13:33 Sketch since the default is probably the running kernel
13:33 Sketch but i'll just use my hacky binary package for now ;)
13:37 code-R joined #fuel
13:41 code-R_ joined #fuel
13:44 javeriak_ joined #fuel
13:45 javeria__ joined #fuel
13:53 Sketch yay, i have a new target image.
13:58 mwhahaha holser_, aglarendil, alex_didenko: can you guys please review https://review.openstack.org/#/c/339721/ it's holding up upstream work
14:02 azemlyanov joined #fuel
14:03 holser_ mwhahaha: let me have a look
14:05 fritchie joined #fuel
14:06 mwhahaha thanks
14:15 holser_ mmalchuk has just merged it
14:18 mmalchuk yep
14:27 GMAzrael @mwhahaha: https://bugs.launchpad.net/mos/+bug/1604846
14:29 mwhahaha GMAzrael: are there any errors in the puppet log for the horizon node(s)?
14:29 GMAzrael not that I have seen
14:30 mwhahaha you may need ot manually run collectstatic
14:31 mwhahaha see #4 http://docs.openstack.org/developer/horizon/topics/install.html
14:33 agordeev Sketch: afaik, DKMS is clever enough to detect that it needs to build the modules against the installed version of kernel, not against the one on which the system is running on.
14:34 kutija joined #fuel
14:37 jhova joined #fuel
14:38 GMAzrael mwhahaha: Added log. Seems to be failing to compress after collectstatic
14:43 xarses joined #fuel
14:46 romcheg vkulanov: hi
14:46 romcheg around?
14:46 GMAzrael mwhahaha: Bug is a duplicate of the other bug I filed: https://bugs.launchpad.net/mos/+bug/1603528
15:02 jhova joined #fuel
15:12 fritchie joined #fuel
15:20 DavidRama left #fuel
15:28 code-R joined #fuel
15:31 Sketch mwhahaha: so, now that my deploy image is fixed, i'm back to the database failure.  Failed tasks: Task[primary-database/9]
15:33 mwhahaha so anything in the mysql logs?
15:34 Sketch i only have a mysql/error.log, same as before.  ends with a signal11
15:37 Sketch https://rednsx.org/tmp/error.log
15:39 Sesso joined #fuel
15:43 mwhahaha do you not have a msyqld.log in /var/log/remote/node-9*/
15:44 Sketch i was looking on the node itself
15:44 Sketch hmm, but when i look at the haproxy log, i see the same database down error that i see in the log, so the error is from haproxy
15:44 Sketch but then 20 minutes later i see it say that it's up
15:45 Sketch so maybe it's just taking too long to come up, and fuel is erroring out before that?
15:46 Sketch i do have a mysqld.log in /var/log/remote, and the entries in it match up with when haproxy sees mysql as being up
15:47 Sketch as does the start time on the mysql process
15:48 * Sketch hits deploy again, to see if it continues now that mysql is up
15:48 Sketch not sure if it will try to reinstall mysql again or not
15:49 Sketch [ 6743.126299] WARNING: CPU: 16 PID: 0 at /build/linux-ZqOzIB/linux-3.13.0/net/core/dev.c:2223 skb_warn_bad_offload+0xcd/0xda()
15:49 Sketch maybe i should change my offload settings while i'm at it...
15:50 Sketch but i don't think i can do that without reinstalling (well, not via fuel anyway)
15:51 v1k0d3n joined #fuel
15:51 code-R_ joined #fuel
15:54 mwhahaha it seems very strange that it would take so long to start up especially if you only have a single controller
15:55 Sketch i wonder if these repeated stack traces about offload might be slowing it down
16:14 Vijayendra joined #fuel
16:15 javeriak joined #fuel
16:23 Sketch hmm, weird.  so my deployment errored out with the same error.  check the logs, and there are no new errors in the haproxy or mysql logs, just the same old ones.
16:24 Sketch it doesn't do some sort of grep on the logs afterwards does it? heh
16:31 javeriak joined #fuel
16:41 GMAzrael anyone able to assign my bug to MOS Horizon?
16:45 mwhahaha Sketch: no its failing because the haproxy backend isn't started. the task actually polls the haproxy status. pacemaker say?
16:46 Sketch ah.  well, i wiped and am trying again with offload disabled to see if that helps (at least get rid of the errors in dmesg)
16:47 Zer0Byte__ joined #fuel
16:53 DavidRama joined #fuel
17:04 code-R joined #fuel
17:10 javeriak_ joined #fuel
17:35 fritchie joined #fuel
17:40 flerfb0rt joined #fuel
17:40 code-R joined #fuel
17:59 Sketch hmm, deploying still in progress.  watching the logs while it's in progress this time, it still took about 20 minutes to go from mysql down to up, but the build hasn't failed yet...
17:59 Sketch 13, actually.  not quite 20.  last time it was a little over 20.
18:14 raunak joined #fuel
18:15 johnavp19891 joined #fuel
18:17 rtikunov joined #fuel
18:26 Sketch i'm starting to think the install hung.  it's at 65% (35% for one node and 60% for the other 2) for like an hour
18:27 mwhahaha you can check the astute log to see what it's currently working on
18:27 Sketch 2016-07-20 18:27:46 DEBUG [27095] Node 12(primary-database) status: running
18:27 Sketch 2016-07-20 18:27:46 DEBUG [27095] {"nodes"=>[{"status"=>"deploying", "uid"=>"12", "role"=>"primary-database"}]}
18:28 Sketch is repeating over and over...
18:28 mwhahaha yup probably hung waiting for it
18:28 mwhahaha can you provide the /var/log/remote/node-12/mysql*.log files?
18:29 Sketch https://rednsx.org/tmp/log/
18:30 mwhahaha so it's up
18:30 Sketch yeah, the process is running
18:30 mwhahaha what does haproxy-status.sh report in terms of the backend?
18:31 Sketch UP
18:31 mwhahaha ok so you can check the puppet log to see what it's working on
18:33 Sketch aha, errors
18:34 Sketch https://rednsx.org/tmp/log/puppet-apply.log
18:35 mwhahaha well that's weird
18:36 mwhahaha so it tries to query port 10000 on the management vip, is that up?
18:36 Sketch "Could not get CSV after 600 seconds!" 3 times, explains why it hangs for so long anyway ;)
18:37 mwhahaha so from the node you can get the management_vip via 'hiera management_vip'
18:37 mwhahaha then you can try and curl http://management_vip:10000/;csv
18:37 Sketch on the controller or the fuel server?
18:37 mwhahaha controller
18:37 Sketch nothing listening on that port
18:38 mwhahaha it'd be in the haproxy network namespace
18:38 mwhahaha whats weird is that this should be running on that host :D since you only have the one controller
18:39 mwhahaha are you doing any network templates or anything for this environment?
18:39 Sketch yeah
18:39 Sketch no network templates
18:39 Sketch network configured via fuel webui
18:39 javeriak joined #fuel
18:39 Sketch multiple vlans on bonded 10G interfaces
18:41 mwhahaha maybe it's running on the database vip
18:41 mwhahaha hiera database_vip
18:41 mwhahaha and try a curl http://<database_vip>:10000/;csv
18:41 Sketch connection refused
18:42 Sketch that's the same as the management_vip, also
18:42 Sketch probably makes sense with only 1 controller
18:42 mwhahaha does /etc/haproxy/conf.d/010-stats.cfg exist?
18:42 Sketch yep
18:43 Sketch and haproxy is running
18:43 mwhahaha ip netns exec haproxy netstat -nl | grep 10000
18:44 Sketch looks like the only port it's listening on is 3306
18:44 mwhahaha wat
18:45 Sketch hmm, just tried to restart haproxy, fatal errors in configuration:
18:45 Sketch Proxy 'horizon-ssl': no SSL certificate specified for bind '10.10.24.11:443' at [/etc/haproxy/conf.d/017-horizon-ssl.cfg:3] (use 'crt').
18:45 mwhahaha well there's your problem
18:46 Sketch "inconsistencies between private key and certificate loaded from PEM file"
18:46 Sketch so it doesn't like my certs for some reason
18:46 mwhahaha wrong cert/key perhaps?
18:48 Sketch i wonder if there's any chance it doesn't like the order of the certs in the file
18:48 Sketch or the intermediate CA bundle
18:49 mwhahaha i bet it tries the first cert or something
18:49 mwhahaha i'm not sure we ever tested with an intermedia ca bundle in there
18:50 Sketch heh, removed the intermediate CA bundle and it started
18:50 Sketch now i guess i try to redeploy and see what happens...
18:51 * mwhahaha throws intermediate CA bundles at sbog
18:54 johnavp19891 Hi all. I'm having an issue during node deployment. On reboot the node is not bootstrapping properly. It's looping on Could not find kernel image: /images/ubuntu_bootstrap/vmlinuz
18:54 mwhahaha johnavp19891: what version? have you tried deleteing and rebuilding the bootstrap image via the 'fuel-bootstrap' command?
18:55 johnavp19891 Fuel version 8.0
18:55 johnavp19891 I tried fuel-bootstrap build
18:55 mwhahaha you say on the reboot it's not bootstrapping properly, reboot of what?
18:56 mwhahaha did you successfully deploy to the node then remove it?
18:57 johnavp19891 Reboot of the slave. I attempted to upgrade a controller using octane upgrade-node ${SEED_ID} isolated <NODE-ID>
18:57 johnavp19891 which failed with that error on reboot.
18:58 johnavp19891 Then I delete the node and tried to recreate it and as normal and I'm still getting the error
18:58 mwhahaha does /var/lib/tftpboot/images/ubuntu_bootstrap/vmlinuz exist on the fuel master?
18:58 mwhahaha it might be in the cobbler docker container
18:59 johnavp19891 It's not on the host
18:59 mwhahaha try the cobbler container
18:59 johnavp19891 ok
19:00 johnavp19891 ok yea it is in the cobbler container
19:02 Sketch johnavp19891: i have found that sometimes leftover pxe configs in tftpboot cause clients to fail to pxeboot
19:02 Sketch so you might rm /var/lib/tftpboot/pxelinux.cfg/your-mac-address
19:02 Sketch (in the cobbler container)
19:05 johnavp19891 Sketch: That was the ticket
19:05 Sketch mwhahaha: now neutron fails becuase it doesn't like the incomplete-ca-chain cert.
19:05 johnavp19891 Thank you both
19:06 Sketch i guess it's time for a self-signed cert.
19:11 mwhahaha if you report the bug we can work on addressing that, as it seems to be a kinda important thing to support
19:11 Sketch yeah
19:12 javeriak joined #fuel
19:16 Zer0Byte__ joined #fuel
19:24 Sketch https://bugs.launchpad.net/mos/+bug/1604930
19:38 raunak joined #fuel
19:42 goldenfri is there a command a run after fuelmenu to make it finish setting up the fuel environment?
19:43 goldenfri usually it does it thing automatically but I was monkeying with it this time
19:43 AlexAvadanii the script that runs the whole thing is called bootstrap_admin_node.sh,, and it's in path
19:43 goldenfri ah, great thanks
19:43 AlexAvadanii but it is NOT re-entrant afaik
19:44 AlexAvadanii so you should take a look it at it and see what id does before/after fuel-menu, to get an idea
19:44 goldenfri right, will do
19:45 tsufiev joined #fuel
19:46 asilenkov joined #fuel
19:47 meow-nofer_ joined #fuel
19:58 Liuqing joined #fuel
20:04 Sketch success!  well, a successful deployment anyway.  the http link for horizon doesn't work, and the https link points to my fuel server.
20:06 Sketch so maybe not a working deployment ;)
20:37 Sketch hmm.  it seems like a routing and/or firewall issue.  i can get to horizon from the fuel server but not from a different subnet
20:38 goldenfri So what are the advantages of using Ubuntu+UCA?
20:40 GMAzrael goldenfri: when the next Stack releases in October, you will have access to those packages
20:40 goldenfri ah ok
20:40 goldenfri so easier to upgrade?
20:41 mwhahaha not necessarily
20:41 GMAzrael where as non-UCA will be stuck on Mitaka
20:41 mwhahaha Newton will require ubuntu 16.04
20:42 goldenfri oh hrm
20:42 mwhahaha it's more of a package source preference
20:42 Sketch well, that's interesting.  i left a ping running that i started 10 minutes ago.  it just started responding.
20:42 goldenfri is there a high chance of a package breaking somthing if you are using UCA?
20:44 GMAzrael only if you try to upgrade Mitaka to Newton
20:44 mwhahaha the UCA is only lightly tested so I can't guarantee it works in the same scenarios as the non-UCA version
20:45 goldenfri ok, makes sense. I think I'll save UCA for a test server at this point then.
20:53 DavidRama joined #fuel
21:26 Sesso joined #fuel
21:46 code-R joined #fuel
21:57 code-R_ joined #fuel
22:25 Sesso joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary