Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-06-04

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 xarses designated: Can you 'dockerctl shell nailgun grep /var/log/nailgun/api.log | grep CanNotFindInterface | sort | uniq -c'
00:04 xarses oops..
00:05 xarses designated: Can you 'dockerctl shell nailgun grep CanNotFindInterface /var/log/nailgun/api.log | sort | uniq -c'
00:13 designated xarses, ubuntu.  I no longer have access to fuel.  after failing multiple times and not getting a response from anyone for a couple of days I tore it down.
00:18 xarses so i haven't figured out what, or possibly why, but the node agent appearars to have sent 393 versions of the node data across 722 reports
00:18 xarses and the same two nodes had the issue, with 2 other nodes only experiencing the issue twice
00:19 designated xarses, in that snapshot yes...i continued to experience the same issue with other nodes randomly after i submitted that diagnostic snapshot.
00:19 xarses my guess is that the root cause here is that the agent failed to include some necessary piece of information about the one of the interfaces which caused the DB to puke resulting in that trace
00:20 xarses designated: it would happen possibly any time you talk to the cluster, with one of the problem nodes in it. Hence why I wanted to see if there where more nodes showing up the in log.
00:21 xarses 93 times mac=00:a0:d1:ed:73:68, 38 times mac=00:a0:d1:ed:73:68, 1 time mac=00:a0:d1:ed:73:68, 1 time mac=00:a0:d1:ed:73:68
00:21 designated xarses, the nodes are all identical with the exception of differences in hard drives
00:22 xarses id maybe look at 73:68 there and see if it has a flakey nic
00:22 xarses probably eth0, or eth1
00:22 xarses I'll plow through the actual agent data in the log here and try to identify what it's send that changed
00:22 xarses tomorrow
00:23 designated xarses, after more testing 73:68 stopped being the problem and other nodes randomly started doing the same thing.  I have since loaded all of the servers in maas and have not once had an issue with any of the onboard NICs.
00:30 xarses designated: like i said, I need to look at the data more in the log to see what's changing in the agent data over time. I'll follow up with you tomorrow.
00:31 IlyaE joined #fuel
00:37 designated xarses, thank you
00:52 rmoe joined #fuel
01:17 xarses joined #fuel
01:28 xarses joined #fuel
02:10 IlyaE joined #fuel
02:18 vogelc__ joined #fuel
02:19 vogelc__ I have to update my bootstrap and centos images for the broadcom firmware,  where do I put the updated images and make sure each docker instance has access to it?
02:21 vogelc__ I was dropping them in /var/www/nailgun/....  and then I rebooted the master node.  But it seemed the instances were still bootstrapping from the original image
03:46 angdraug joined #fuel
05:09 Alexande1 joined #fuel
05:34 tim|mint hey all, I'm getting a lot of dependency errors this time around when doing a make iso, anyone any idea? http://paste.openstack.org/show/82724/
05:35 tim|mint trying to build the stable/5.0 branch
06:35 al_ex joined #fuel
06:57 artem_panchenko joined #fuel
07:26 sotpuschennikov joined #fuel
07:52 akupko joined #fuel
08:12 pbrooko joined #fuel
08:38 e0ne joined #fuel
08:39 izinovik joined #fuel
08:48 rmoe joined #fuel
08:57 e0ne joined #fuel
09:03 xenolog joined #fuel
09:03 dpyzhov joined #fuel
09:04 brain461_ joined #fuel
09:43 e0ne joined #fuel
09:44 e0ne_ joined #fuel
09:52 nurla joined #fuel
09:52 brain461_ joined #fuel
09:52 holser joined #fuel
09:53 dpyzhov joined #fuel
09:53 mihgen joined #fuel
09:54 taj joined #fuel
09:54 aglarendil joined #fuel
09:54 teran joined #fuel
09:55 evg joined #fuel
09:56 kat_pimenova joined #fuel
09:56 jkirnosova joined #fuel
09:57 mrasskazov joined #fuel
09:57 meow-nofer joined #fuel
09:58 monester joined #fuel
09:58 vkramskikh joined #fuel
09:59 apalkina joined #fuel
10:07 tim|mint hey all, I'm getting a lot of dependency errors this time around when doing a make iso, anyone any idea? http://paste.openstack.org/show/82724/
10:07 tim|mint trying to build the stable/5.0 branch
10:07 dnikishov joined #fuel
10:07 dnikishov left #fuel
10:08 dburmistrov tim|mint: Could you pastebin your command and full log?
10:09 tim|mint sure
10:11 tim|mint dburmistrov: http://paste.openstack.org/show/82751/
10:14 dburmistrov tim|mint: Temporary failure resolving 'fuel-repository.mirantis.com'. It seems to be a dns-resolver issue
10:16 tim|mint thx! that was it indeed... silly ubuntu always clears /etc/resolv.conf :S
10:24 akislitsky joined #fuel
11:46 dpyzhov joined #fuel
11:56 e0ne joined #fuel
12:03 e0ne_ joined #fuel
12:37 foliveira joined #fuel
12:45 foliveira I have deployed Fuel 5.0 on some Dell servers that have a Mellenox ConnectX-3 dual port NIC. When I boot the discovery image the mlx4_en module was not getting loaded.  I modified initramfs.img with "modprobe mlx4_en" in /etc/rc.modules so that the discovery image can now detect the NIC and install openstack.  How would I setup /etc/rc.modules in the installtion logic so that I don't have to...
12:45 foliveira ...log into each server and configure this manually??
12:53 evg foliveira: hi, so you'v made your own bootstrap image?
12:54 vogelc evg: I have the same question as filiveira.  I have created the new images (I have done this before).  How do I update them in the new docker configuration?
12:54 foliveira Yes, I made my own bootstrap with the etc/rc.modules file
12:57 evg vogelc: you had to put it into /var/www/nailgun/bootstrap/ and run cobbler sync (on master node)
12:58 vogelc running that now
12:58 evg vogelc: foliveira: but wait, things's changed in 5.0. I should check it
12:58 vogelc evg:  Ill wait
12:59 foliveira yes. that all works for the bootstrap.  However, I want to modify the deploy process to install the rc.modules file.
13:00 foliveira I would also like to install the latest Mellanox driver.  Can I add that to the manifest?
13:09 vogelc joined #fuel
13:09 evg yes, it works. And described here http://docs.mirantis.com/fuel/fuel-5.0/release-notes.html (cannot point the anchor, sorry)
13:12 foliveira evg:thanks.  I will look.
13:13 evg foliveira: do you like to install drivers with puppet?
13:13 foliveira Yes.
13:17 vogelc evg: cobbler sync did work this time.  Thank you.  I am assuming I can do the same thing for the centos initrd.img?
13:20 evg vogelc: yes, related to cobbler, i think
13:21 vogelc evg: I will try it - thanks
13:26 evg foliveira: you can modify manifest, yes. Or may be it would be the right way to modify instalation packages (centos/ubuntu).
13:26 foliveira evg: thanks
13:45 jobewan joined #fuel
13:48 evg foliveira: rebuild linux-image-XXX-.deb with your drivers (in /var/www/nailgun/ubuntu/fuelweb/x86_64/pool/main on master) and renew Packages* files (/var/www/nailgun/ubuntu/fuelweb/x86_64/dists/precise/main/binary-amd64) for the new checksumms
13:57 e0ne joined #fuel
14:00 b-zone joined #fuel
14:49 blahRus joined #fuel
14:59 IlyaE joined #fuel
15:39 BillTheKat joined #fuel
15:43 BillTheKat MiroslavAnashkin: Just a note to update you - your observation about setting boot strings worked - successfully created an icehouse instance.  THANKS - Are you going to enter in a bug report to get the documentation updated and include a discussion about this boot string thing if changing IP subnets etc.
15:45 MiroslavAnashkin There is bug report and I am correcting the docs right now
15:50 BillTheKat MiroslavAnashkin: thanks !!!!!!!
16:15 rmoe joined #fuel
16:28 xarses joined #fuel
16:38 jues joined #fuel
17:15 angdraug joined #fuel
17:18 albionandrew joined #fuel
17:20 albionandrew Hi, We are testing Fuel 5. I am building the master node, I am now seeing a message “(null): Dropping Ts06 features since no CSUM feature. How do I fix this?
17:26 albionandrew xarses ^ any ideas? The redhat site says to update the kernel
17:27 xarses albionandrew: 5.1 do you mean 5.0?
17:27 xarses 5.1 is the next development version
17:27 albionandrew 5.0 but I think the “I am…” could look like a 1
17:29 xarses you can check https://<fuel-node>:8080/api/version. There should also be a number in the footer, but if your on development, then the api version is more helpful
17:30 albionandrew THe master node has not finsihed building seems hung.. but to confirm it is 5.0
17:31 albionandrew If I go to that link I just get a 404
17:33 xarses Where is that log message from?
17:33 albionandrew Just output to the console
17:34 albionandrew I see a lot of docker info and then 10? (null) Dropping …. and nothing has changed in the last 30 mins?
17:34 xarses Can you give a screenshot, or some of the surrounding messages? I don't know what the context is
17:35 albionandrew Sure I”ll get you a screen shot
17:35 xarses hmm, around docker. Thats odd. You should be able to switch to another pty and get onto the console
17:35 xarses let me know if you cant
17:36 albionandrew http://tinyurl.com/lygm8kq
17:36 MiroslavAnashkin You may ignore this message.
17:38 albionandrew xarses / MiroslavAnashkin: I changed the console. I now get the “fuel ui is available” … when I visit that url it eventually times out nginx 504 Gateway time-out
17:39 xarses albionandrew: yes, thats a misnomer because it's the login banner and was already set
17:39 xarses can you paste the results of 'top' and 'docker ps'?
17:39 albionandrew no nothing about docker just type “docker ps”
17:40 e0ne joined #fuel
17:40 xarses yes, its a command
17:42 albionandrew http://pastebin.com/x9E5ztjc and http://pastebin.com/JMvM2aFj
17:43 e0ne joined #fuel
17:47 xarses hmm, nothing obvious for me. MiroslavAnashkin do you have any insight?
17:51 albionandrew xarses / MiroslavAnashkin - I see this on the Redhat site - https://access.redhat.com/site/solutions/631123 otherwise I don’t see a lot about it
17:51 albionandrew not sure if it’s exactly the same but similar
17:52 thehybridtech joined #fuel
18:03 albionandrew xarses MiroslavAnashkin I’m going to try again - These are the settings I used - http://picpaste.com/Screen_Shot_2014-06-04_at_10.42.31_AM-pyCjaDbs.png but I never saw that setup screen http://docs.mirantis.com/openstack/fuel/fuel-5.0/pdf/Mirantis-OpenStack-5.0-UserGuide.pdf as show on page 6. Do these kernel args http://picpaste.com/Screen_Shot_2014-06-04_at_10.42.31_AM-pyCjaDbs.png seem correct .. I think they are thats what I used on the last 4.0
18:03 albionandrew deployment.
18:04 e0ne joined #fuel
18:07 xarses the look ok, honestly I don't look at them much so I don't know for certain.
18:10 albionandrew xarses I think they’re good, as you know we ran this an insane number of times when testing 4 so it was pretty well documneted from this side as to what we did. Would you suggest not setting them? Is there a place later on to set them ?
18:18 xarses albionandrew: as I understand, you should set your ip info in the kernel options and then further change stuff in fuelmenu, there is a bug about it already.
18:18 xarses normally, you should be able to skip setting the kernel, but there was some issue introduced with the containers.
18:19 albionandrew so I need to set it? Not an obtion to skip anymore?
18:21 albionandrew xarses and when the show menu comes up, which it did this time I should treat that docker interface exactly as I treated eth0 before?
18:21 xarses MiroslavAnashkin: ^
18:22 xarses albionandrew: I don't know myself
18:22 xarses The docker stuff is about as new to me awell
18:22 xarses s/awell/as well /
18:22 albionandrew xarses - it looks really cool.
18:24 MiroslavAnashkin albionandrew: https://bugs.launchpad.net/fuel/+bug/1326125 Page 6 in the docs is obsolete.
18:26 MiroslavAnashkin docker0 is bridge between ethX and internal Docker network. With 5.0 you may only select the PXE interface in Fuelmenu and only during the first boot
18:49 e0ne joined #fuel
18:59 mpetason joined #fuel
19:12 albionandrew MiroslavAnashkin / xarses  I seem to have a working Master node now. Thanks. The console says http://picpaste.com/Screen_Shot_2014-06-04_at_1.09.02_PM-4qhx4FQM.png but I can get to the gui via the IP I set here - http://picpaste.com/Screen_Shot_2014-06-04_at_12.36.33_PM-A0p8XmaR.png Is the console output a bug or have I done something wrong?
19:13 xarses albionandrew: that is likely a bug. That looks like the default address for the docker0 interface
19:13 albionandrew Yes it is
19:15 MiroslavAnashkin No. IP address and NIC where you get Fuel UI in 5.0 is not related to Docker or Admin network at all. For instance, you may start up second NIC, assign any IP address to this second NIC and get Fuel UI there as well by the assigned IP
19:16 xarses MiroslavAnashkin: I think the issue here is that the MOTD was written with the ip addr of docker0 not whichever interface has the default route
19:16 xarses erm, i guess its the banner not motd
19:20 MiroslavAnashkin Yes, almost everything from all NICs is redirected to docker network. I agree, displaying the docker IP instead of physical NIC is incorrect.
19:20 albionandrew xarses MiroslavAnashkin I also see this http://tinyurl.com/qya88p4 . I will continue building but if there’s any info you would like let me know.
19:21 MiroslavAnashkin I myself found a couple of non-clear places in Fuelmenu and cannot say how it works until took a look into the code.
19:21 xarses albionandrew: you need to clear your browser cache
19:22 xarses and possible wipe temp files
19:22 MiroslavAnashkin Or don't use Chrome
19:22 MiroslavAnashkin Or deep clean Chrome, if there is deep clean...
19:23 albionandrew DId this - https://support.google.com/chrome/answer/95582?hl=en
19:24 albionandrew Will try in safari which I never use
19:24 xarses after you clear your cache, you will still need to create a new cluster
19:25 MiroslavAnashkin We already have report that Chrome may work incorrect with Fuel UI 5.0 even after cache/cookie cleanup.
19:25 albionandrew Perfect in safari, thanks
19:26 MiroslavAnashkin And both use the same WebKit engine...
19:27 albionandrew I didn’t restart chrome I just used the instructions on that google link. Just for info.
19:28 MiroslavAnashkin Since Chrome uses separate processes for every tab - may be restart really required
19:34 crandquist joined #fuel
19:35 TVR_ do you have a link to fuel 4.1.img anywhere? My laptop died, and I want to keep the sites consistant
19:36 Arminder guys for fuel 5.0, ha, do we need a separate node for telemetry mongodb role?
19:36 Arminder or it can co-exisit with controller
19:36 Arminder in which case do i select mongo role for all the 3 controllers
19:36 Arminder or just 1
19:37 Arminder i know it can co-exist, but shall it be done
19:37 MiroslavAnashkin Separate node is extremely recommended.
19:37 Arminder and what kind of specs shall we need for it
19:38 Arminder i understand bigger drive would be one
19:38 Arminder anything else required?
19:39 xarses Arminder: It is required for any real workloads, it is quite noisy.
19:39 xarses The default disk allocator will attempt to consume at least one whole disk for the mongo role
19:40 MiroslavAnashkin Hardware specs are the same as for controller nodes.
19:42 Arminder ah great, thanks for the responses guys n you guys are doing a great job!
19:42 MiroslavAnashkin MongoDB+Telemetry data simply takes all disk I/O throughoutput, so other OpenStack services may simply stop working after I/O timeout
19:43 TVR_ anyone have a link to dload 4.1?
19:44 TVR_ please?
19:46 xarses http://software.mirantis.com/prior-releases/
19:46 TVR_ thank you, thank you
19:48 Arminder BTW, any timelines for 5.0.1 yet?
19:48 Arminder or is it too early?
19:49 Arminder or its going to be 5.1 directly?
19:51 MiroslavAnashkin We hope to release it by upcoming Tuesday, next week. But so far didn't made a decision, should it be an upgrade to existing installation or ISO. Depends on how many issues we will find in upgrade functionality.
19:54 TVR_ it hangs and the download never starts.....
19:57 MiroslavAnashkin Please try log in with your account first, then you may select Software>Prior Releases in the upper menu. Yeah, it took 5 minutes to me to find this link...
20:00 xarses Arminder: https://launchpad.net/fuel
20:00 xarses Look at series and milestones
20:00 xarses near the end of june for 5.0.1
20:02 MiroslavAnashkin Aah, yes, it is 4.1.1 targeted to Tuesday
20:25 albionandrew MiroslavAnashkin xarses  Does dhcp run from inside a docker container or somewhere else now? Can’t pxe any clients. I see the request on the master from the client but the master does not respond - http://pastebin.com/eWMqDkmJ
20:26 xarses albionandrew: see https://bugs.launchpad.net/fuel/+bug/1324152? and "also affect you" if it solves your problem
20:27 albionandrew xarses Thanks I’ll give that a go.
20:33 pbrooko joined #fuel
20:34 albionandrew xarses - I get Jun  4 20:19:09 fuelpxe03 dhcrelay: Error getting hardware address for "docker0": No such device
20:36 xarses albionandrew: sounds like that error
20:36 xarses bug even
20:37 albionandrew Is there a fix ? I tried restarting the relay service etc
20:37 albionandrew I will also put something on launchpad
20:38 xarses it identifies restarting relay, and then cobbler
20:38 xarses did you do both?
20:38 xarses also if you can post any of the logs that Matt was asking for, that would be helpful
20:38 albionandrew Yes did all.
20:38 jerrygb joined #fuel
20:43 albionandrew xarses - http://pastebin.com/d2Kgq1Ja
20:44 albionandrew I will put on launchpad too
20:47 xarses albionandrew: and 'dockerctl logs cobbler'?
20:47 albionandrew sure. just putting something on launchpad now.
20:53 albionandrew xarses - I added that stuff to lanuchpad. Anything else worth trying?
20:55 xarses albionandrew: check the IRC logs from yesterday for "relay" there was talk about it
20:57 wayneeseguin joined #fuel
21:02 e0ne joined #fuel
21:05 Kupo24z1 anyone know of a way to change cache='none' to a different cache method?
21:05 Kupo24z1 (for VM disks)
21:08 xarses for machines created by Neutron?
21:09 Kupo24z1 Think i found it http://www.sebastien-han.fr/blog/2013/08/22/configure-rbd-caching-on-nova/
21:09 Kupo24z1 Need to edit the caching method that VM's are created with
21:09 xarses yep, its a little old now, but that should be the gist of it
21:09 Kupo24z1 in the <disk> section
21:11 Kupo24z1 xarses: you know what would work best in an all-ssd ceph config?
21:43 xarses No, but I think there are some ceph docs about it
22:01 albionandrew xarses - http://pastebin.com/ibUJG7j7 is this correct? What does the dhrelay do? If I do an ip add | grep 10.0.0.2 nothing comes back? What is that IP for?
22:03 xarses dhcrelay forwards DHCP packets from one interface to a) another or b) a server IP, its used to span network domains, since cobbler and dnsmasq runs in a container its technically in on another subnet
22:03 albionandrew OK thats what I thought so where is that 10.0.0.2 supposed to be?
22:04 xarses dhcrelay should listen on the interface (-i <iface>) that you want to PXE from, if your admin is eth0, then -i eth0
22:04 xarses then you want to specify a destination interface (-d <iface> i think) or a server (-s <ipaddr> i think)
22:04 albionandrew so should I try starting it and have it only listen on docker0 ? since that 10.0.0.2 doesn’t seem to exist?
22:05 xarses dockerctl shell cobbler ip a
22:05 xarses it should be the ip for the cobbler host
22:06 albionandrew xarses that came back with that IP
22:06 xarses then thats the addr you want to forward to
22:07 xarses you should be able to ping it, or at least telnet to port 80
22:28 albionandrew xarses - After a restart I could get to the 10.0.0.2. dhblaz seemed to think that the arg should have eth0 in. We changed it in vi /etc/sysconfig/dhcrelay but something changes it back.
22:29 xarses supervisor maybe?
22:29 xarses we don't run puppet that way, so it should't do that
22:30 albionandrew so would it be worth killing supervisord and trying again?
22:30 xarses see if it has some random config for dhcrelay
22:32 albionandrew If I’m understanding you correctly  grep -i dhc -r /etc/supervisord.d/ comes back with nothing
22:54 Kupo24z1 Im seeing crazy higher throughput when using btrfs as the OSD filesystem, is there any reason why fuel doesnt use it?
22:55 xarses its not a production filesystem
22:56 xarses and I've seen it eat data in the ceph threads as recently as emperor
22:56 xarses it shouldn't be horrible if your replica factor is 3 or 4
22:56 xarses I use it on my laptop and it randomly locks up
22:57 xarses I haven't eaten the volume (yet)
22:57 * xarses knocks on wood
22:57 xarses the performance is due to the fact that ceph can use btrs's async journal and doesn't need to maintain a separate one
22:58 xarses if you wanted it to be btrfs, you should be able to change the puppet manifests to make it format btrfs instead of XFS with out too much trouble
23:00 xarses hmm, we might not have the btrfs packages, you'd need those
23:01 xarses and to change /etc/puppet/modules/ceph/manifests/ (grep for xfs and replace with btrfs)
23:10 Kupo24z1 ah okay, thanks
23:11 Kupo24z1 Where does fuel have the OSD mounts? dont see anything for them in /etc/fstab
23:11 Kupo24z1 per http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11530 its recommended to use inode64
23:11 xarses We use the magic udev script to mount and initalize them
23:11 Kupo24z1 for xfs
23:13 xarses Kupo24z1: afaiu that option dosn't even exist in CentOS 6.5 and is default enabled in Ubuntu so we don't pass it
23:13 Kupo24z1 ah okay
23:16 Kupo24z1 trying to get my 4k numbers faster, sitting at 20MB/s writes and 70MB/s with 7x nodes at 3 OSD's each so looking at all possibilities
23:35 thehybridtech joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary