Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-04-15

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 Arminder joined #fuel
00:02 Arminder joined #fuel
00:03 Arminder joined #fuel
00:04 Arminder joined #fuel
00:06 Arminder joined #fuel
00:08 Arminder joined #fuel
00:08 Arminder joined #fuel
00:09 Arminder joined #fuel
00:10 Arminder joined #fuel
00:11 Arminder joined #fuel
00:12 Arminder joined #fuel
00:13 Arminder joined #fuel
00:14 Arminder joined #fuel
00:39 xarses joined #fuel
00:39 thumpba_ joined #fuel
00:41 emagana joined #fuel
00:46 thumpba_ joined #fuel
01:53 LiJiansheng joined #fuel
02:23 fedexo joined #fuel
03:08 daniel3_ joined #fuel
03:18 Longgeek joined #fuel
04:37 Longgeek joined #fuel
05:48 Longgeek joined #fuel
06:13 meow-nofer_ joined #fuel
06:15 sc-rm joined #fuel
07:43 ddmitriev joined #fuel
08:02 e0ne joined #fuel
08:02 hyperbaba joined #fuel
08:08 e0ne joined #fuel
08:08 hyperbaba joined #fuel
08:14 hyperbaba joined #fuel
08:17 sambork joined #fuel
08:23 e0ne joined #fuel
08:25 stamak joined #fuel
08:33 e0ne joined #fuel
08:35 alecv joined #fuel
08:40 zerda joined #fuel
09:03 alecv joined #fuel
09:11 tzn joined #fuel
09:21 sambork joined #fuel
09:22 vasilygorin joined #fuel
09:23 [HeOS] joined #fuel
09:23 vasilygorin left #fuel
09:25 vasilygorin joined #fuel
09:34 e0ne joined #fuel
10:02 Longgeek_ joined #fuel
10:03 e0ne joined #fuel
10:09 teran joined #fuel
10:21 dklepikov joined #fuel
10:23 sambork joined #fuel
10:37 saibarspeis joined #fuel
10:41 teran joined #fuel
10:54 e0ne joined #fuel
10:56 sc-rm evg: After installing a new juno from fuel 6.0 I had a controller going down, then it rebooted, this controller was running the dhcp agent, but none of the controllers revocered the dhcp agent after the controller came back.
10:57 sc-rm evg: I had to do service neutron-dhcp-agent restart on the controllers to bring back the service. Was this not supposed to be fixed in fuel 6.0?
11:01 jskarpet joined #fuel
11:26 [HeOS] joined #fuel
11:29 Longgeek joined #fuel
11:38 e0ne joined #fuel
11:49 vt102 joined #fuel
11:58 vt102 joined #fuel
12:03 tzn joined #fuel
12:30 e0ne joined #fuel
12:35 vt102 joined #fuel
12:35 evg sc-rm: Hi, not this case? https://bugs.launchpad.net/fuel/+bug/1436414
12:36 vt102 joined #fuel
12:36 evg sc-rm: not fixed yet, but a fix was proposed.
12:46 sc-rm evg: Okay, cool that someone is working on it. Because we are moving into a production state
12:56 jaypipes joined #fuel
14:05 rongze joined #fuel
14:36 emagana joined #fuel
14:46 blahRus joined #fuel
14:57 newbism joined #fuel
14:57 newbism Hey
14:58 newbism So, still want to get the CX3 going, it seems like I can with eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1 but thats only from bootstrap, how will it function once its in the actual image... idk, whats going on?
15:07 evg newbism: Hi
15:08 evg newbism: I suppose the same result you can get with module's params
15:09 newbism oh? Is there a way I can rebuild or add in puppet manifest to set eth mode on boostrap?
15:09 newbism isnt it suppose to be like that anyway? I see the mirantis docs and they just show the 40GB IB show up
15:09 evg newbism: does modinfo say something about its params?
15:10 newbism logging in ...
15:11 evg newbism: and the first can you confirm the module is load before you run "echo eth....."?
15:11 newbism modinfo mlx4_core ?
15:11 newbism lsmod | grep mlx shows mlx4_ib               123345  0 ib_sa                  33416  1 mlx4_ib ib_mad                 47630  2 mlx4_ib,ib_sa ib_core                82699  3 mlx4_ib,ib_sa,ib_mad mlx4_en                81952  0 mlx4_core             206735  2 mlx4_ib,mlx4_en
15:12 evg newbism: I don't know. haven't got mellanox NIC
15:13 newbism are there two versions of 6.0? One from openstack dev and one from Mirantis ... it says it works? https://community.mellanox.com/docs/DOC-2077
15:13 evg newbism: Am I right the same card is used for ib and ethernet?
15:13 newbism yeah you can change modes
15:15 evg newbism: there is the only one 6.0
15:16 evg newbism: please run mnodinfo for the both modules
15:17 newbism evg: http://dpaste.com/2AH8S7B
15:17 newbism port_type_array
15:19 newbism do i have to create a custom bootstrap node? im confused on how this is suppose to work according to the docs
15:20 newbism like say I want to use neutron mlx ... but then when i configure interfaces its not there?
15:20 evg newbism: in the bad case. Let me see what else you can do....
15:21 newbism does puppet control the boostrap nodes? ill hack in the echo statement x'D
15:22 newbism OT: you know whats pretty cool too? ipmiutil .. it has an idiscover for ipmi
15:22 newbism OT again: lldpd can show switch ports
15:23 evg newbism: puppet doesn't control bootstrap
15:25 newbism damn so i gotta rebuild the image
15:25 newbism ugh
15:26 evg newbism: it's not a problem
15:27 newbism waat'
15:27 newbism how am i suppose to assign the storage network?
15:28 newbism its all on the eths
15:29 evg newbism: as I got it you will get two devices - eth and ib
15:30 newbism the only interfaces that show up after provisioning and ability to assign interfaces are the ethernet interfaces
15:30 evg newbism: provisioning?
15:30 newbism If I echo eth into the /sys channel... it will show up as a new ethX ... I can then discard action
15:31 newbism add it to the cluster again
15:31 newbism and the eth will show up
15:31 newbism the ib eth
15:31 evg newbism: got it
15:33 evg newbism: I saw such cards a year ago. (don't remember with ones). I got eth nics + iscsi devices.
15:34 evg newbism: I configured iscsi/NAS in bios
15:35 evg newbism: ok. let's rebuild bootstrap
15:36 evg newbism: mkdir BOOTSTRAP; cd BOOTSTRAP; cat /var/www/nailgun/bootstrap/initramfs.img | gunzip -c | cpio -imudv
15:37 newbism I believe its a CX3 from Intel
15:37 evg newbism: edit what you want. I think - etc/modprobe.d/modules.conf. Insert "option" command
15:37 newbism its a quad node
15:37 evg newbism: then find . -xdev | cpio --create --format='newc' | gzip -9 > /var/www/nailgun/bootstrap/initramfs.img
15:38 evg newbism: backup original image before
15:38 newbism yas :D
15:38 newbism ty for reminder
15:38 evg newbism: yes, it seemed to be some intel
15:39 newbism so this is bootstrap, what will happen with the actual centos img?
15:39 newbism we will find out... ?
15:40 evg newbism: you can configure ethtool options for NICs in .yalm file
15:40 newbism maybe need to put the mellanox tools on boostrap to determine wtf kind of card it is, and/or if its in eth/ib mode
15:41 newbism thats good, but wil the actual centos built node also have the same problem?
15:41 rongze joined #fuel
15:41 newbism and ill need to rebuild that in the same way
15:42 evg newbism: it seems simple but not elegant.
15:42 newbism nah it seems ok, programming hardware is rough I know
15:42 newbism just wanna know where were at :D
15:43 evg newbism: you can put a bunch of commands in local.rc
15:44 newbism So... normally this should work with Mellanox cards, its just mine is of course the odd ball... didnt get detected and could file a feature enhancement?
15:46 evg newbism: strange, I thought etherned mode should be enable by default
15:46 newbism yah thas what im sayinnnn xD
15:46 evg newbism: stop. from your past "HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet"
15:48 evg newbism: I think default is reasonable to get ethernet and ib
15:49 newbism default 1 for IB..?
15:49 evg newbism: 0
15:49 newbism ok
15:50 newbism well, i echo eth into the sys and she shows up...  I've always thought we are interacting with the CX3 through eth
15:50 newbism rather than something like lustre
15:51 sambork joined #fuel
15:52 evg newbism: I really haven't work with such nics. You have to experiment with your card to get both eth+ib. Then repack bootstrap.
15:53 newbism you can have both mode?
15:54 evg newbism: I had no problem, but it  were not mellanox cards
15:54 newbism what did you have?
15:56 evg newbism: ethernet device (ip link) + iscsi disks (in /dev/disk/)
16:00 evg newbism: if you're about NICs I don't remember exactly as I said
16:03 newbism so there is this
16:03 newbism but first boot to bootstrap is cat /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1     auto (ib)
16:03 newbism lemme rmmod try that again..
16:04 evg newbism: do it please. and auto is reasonable I think
16:04 evg newbism: aaaaa, got it
16:04 xdeller joined #fuel
16:05 newbism were trying to modprobe  mlx4_core port_type_array=2,
16:05 newbism 2,2
16:05 evg newbism: your card auto discovered the media you've pluged into it
16:06 newbism head hurts
16:09 emagana joined #fuel
16:17 sambork joined #fuel
16:18 daniel3_ joined #fuel
16:19 newbism would there be a config on the IB switch thats auto config it for IB?
16:22 championofcyrodi "HGST will ship a 10TB helium drive in the second half of 2015" -http://www.zdnet.com/article/hgst-goes-all-in-on-helium-drives/
16:32 emagana joined #fuel
16:37 evg newbism: hm, as I see from url you've posted fuel should configure all the things be itself. So let's try just to rebuild bootstrap image.
16:39 evg newbism: but check if module's option works on the current bootstrap.
16:40 evg newbism: rmmod <module(s)>; set port_type_array=2 option in /etc/modprobe.d/mellanox.conf; modprobe...
16:41 evg newbism: if it works you can rebuild bstrap
16:46 jobewan joined #fuel
16:46 newbism evg:  on the live bootstrap image I rmmod everything and modprobe mlx4_core port_type_array=2  ... still auto ib
16:47 newbism weird
16:47 newbism it just does not want to listen to the module optoin
16:51 xarses joined #fuel
17:02 evg newbism: strange, what if you blacklist mlx4_ib
17:11 sambork joined #fuel
17:17 tzn joined #fuel
17:19 vt102 joined #fuel
17:28 vt102 joined #fuel
17:37 e0ne joined #fuel
18:06 sambork joined #fuel
18:07 darkhuy joined #fuel
18:07 darkhuy anyone awake? :-)
18:09 darkhuy How do I find out what the nailgun user password is to gain psql access to the postgres db for nailgun on a fuel deployment?
18:17 evg darkhuy: please look inside /etc/fuel/astute.yaml
18:18 darkhuy evg:you are a livesaver...thank you :-)
18:19 teran joined #fuel
18:31 daniel3_ joined #fuel
18:37 sambork joined #fuel
18:49 thumpba joined #fuel
18:50 thumpba_ joined #fuel
18:55 newbism evg: it seems if i boot the nodes without the IB connected... the module will load mlx4_core as ib properly
19:00 [HeOS] joined #fuel
19:21 jaypipes joined #fuel
19:30 e0ne joined #fuel
19:31 newbism so now im really damn confused
19:31 newbism raaage
19:33 muadgib joined #fuel
19:33 newbism this is annoying as balls, what damn hardware do you need to make this work right
19:35 championofcyrodi newbism: Broadcom 1Gbps Ethernet NICs seem to be supported ;p
19:36 championofcyrodi a guy was in here last week struggling w/ the new Intel X710 40Gbps ethernet cards... turns out his newest firmware just dropped 7 days ago and drivers were not part of the centos base image... so it would pxe boot, but then no drivers for the 2.6.36 kernel.
19:38 championofcyrodi ubuntu openstack MAAS and their whole openstack deployment stack likely uses newer kernels and newer drivers.  i would be choosy with what you lock in to.  although it's all open source, you'll get locked in with the amount of time and money you put in to maintaining Openstack X on Linux Kernel Y running linux distro Z
19:40 newbism I could have swored I booted a vanilla cenos 7 on these same nodes, same ib switch ... and it worked fine
19:40 championofcyrodi right, centos 7 uses a 3.x kernel
19:40 newbism ugh
19:40 championofcyrodi centos 6 is 2.6 kernel...
19:40 championofcyrodi centos 6 is supported until 2019 i think.
19:41 championofcyrodi centos 7 offical release just rolled out sometime in the last 6 months
19:41 championofcyrodi likely you can get fuel w/ centos 6 to run on your hardware, but is not going to be trivial since your hardware is much newer than the software.
19:41 newbism I guess 3.x has newer OFED
19:41 championofcyrodi OFED?
19:42 newbism its still crazy it totally disobeys port_type_array
19:42 newbism the IB stuff
19:42 newbism shit
19:43 championofcyrodi i work in a low budget facility where nobody gets money for infrastructure, it all goes to developer salary, so i hobble together what i can find
19:43 championofcyrodi openstack is great for heterogeneous commodity hardware.
19:44 championofcyrodi have you tried just installing centos 6.5 on a node by itself?
19:44 championofcyrodi see if you can get a minimal install going and utilize your NICs, get static IPs and ping and dns resolve servers?
19:45 championofcyrodi thats a good way to troubleshoot your hardware and determine if it has compatability issues w/ centos 6 drivers.
19:45 newbism I can get it to work... the crazy problem is ... "IF THE IB IS CONNECTED" .... it will default to IB mode.... regardless of mlx4_core port_type_array=2,2
19:46 championofcyrodi what is 'it' ?
19:46 newbism I can echo into the /sys path and get it to work
19:46 newbism OR i can leave the IB unplugged... reboot the node
19:46 newbism and its in eth mode
19:46 newbism the ib interface = it
19:46 championofcyrodi is this for the fuel master?
19:46 championofcyrodi or are you trying to deploy an environment?
19:48 newbism trying to deploy env
19:48 newbism nodes are discovered np
19:59 newbism AHIGhasdiogh;asfg
20:05 newbism "the installer that just works"
20:08 newbism https://bugs.launchpad.net/fuel/+bug/1393414 seems like my prob?
20:12 teran joined #fuel
20:14 championofcyrodi newbism: try breathing slowly through the nose and out the mouth, while i look at this bug and try to make sense of it.
20:14 newbism imma go get some tea
20:15 championofcyrodi "Just upgrade the isolinux directory files (vmlinuz kernel, initrd.img, etc) to CentOS 6.6 ones"
20:15 championofcyrodi that sounds like what you want to do
20:17 newbism yisss...
20:17 championofcyrodi hmmm looks like this fix should already be in the lastest fuel iso...
20:17 championofcyrodi (i think)
20:18 newbism i'm on fuel-community-6.0.iso
20:19 tzn joined #fuel
20:19 tzn joined #fuel
20:19 championofcyrodi you could try a nightly build...
20:19 championofcyrodi https://fuel-jenkins.mirantis.com/view/ISO/
20:20 tzn joined #fuel
20:20 championofcyrodi (there is a 6.0 tab at the top)
20:20 newbism I will, if trying to update the images does not work
20:20 championofcyrodi http://seed.fuel-infra.org/fuelweb-iso/fuel-community-6.0.1-91-2015-04-15_03-39-09.iso.torrent?from=status
20:20 tzn joined #fuel
20:21 newbism ty
20:21 championofcyrodi hope it works out for you... this is a really good deployer and works pretty well once you fail at it a few dozen times.
20:24 newbism the bug says to install the new image to /var/www/nailgun/2014.2-6.0/centos/x86_64/ ...
20:24 newbism but I'm still at bootstrap?
20:26 mwhahaha newbism: i believe they mean put that on your admin node in that folder
20:31 newbism but there is /var/www/nailgun/2014.2-6.0/centos/x86_64/isolinux/ and /var/www/nailgun/bootstrap/
20:33 vt102 joined #fuel
20:34 newbism sooOo were updating the installed image but not the bootstrap? the bootstrap image is where im having the prob... i cant see the interface to define it to even begin to deploy
20:37 tzn joined #fuel
20:37 xdeller joined #fuel
20:40 tzn joined #fuel
20:40 championofcyrodi newbism: it may be part of the cobbler/nailgun docker container on the fuel maste
20:40 championofcyrodi r
20:41 championofcyrodi anyone get link bonding to work? my radosgw has 10Gbps between public and the gateway, but 1Gbps between gateway and switch w/ OSDs.
20:41 championofcyrodi so i'm seeing HUGE speeds uploading to object store...
20:41 championofcyrodi but then when its 'done' uploading... i see the internal 1Gbps just working at its own pace.
20:42 championofcyrodi long after the initial upload is done.
20:42 championofcyrodi i have eight 1Gbps NICs, so was hoping to bond the 8 of them for storage/management VLANs
20:42 championofcyrodi using VLANs by the way.
20:43 championofcyrodi by theory is that since i have 4 OSDs w/ SSDs and 1Gbps, i should be able to get > 1Gbps on internal read/write between radosgw and the nodes... right?
20:44 championofcyrodi the eth device pushing data in to ceph seems to be chunking it
20:44 championofcyrodi w/ avg of 216 MBit/s
20:45 championofcyrodi avg 7Mbit Outgoing, 220Mbit incomming, with a MAX incoming of 931Mbit ~ 1Gbps NIC...
20:46 championofcyrodi so it looks like at SOME point it was pushing at 1Gbps to ceph...
20:46 championofcyrodi but not consistently
20:47 tzn joined #fuel
20:47 tzn joined #fuel
20:48 championofcyrodi hmm looking at one of my osds...i'm seeing the outgoing is eating up IO on each ceph node...
20:48 championofcyrodi a lot...
20:48 championofcyrodi which is likely due to rebalancing for replication...
20:51 championofcyrodi maybe if i lower replication from 3 to 2 on the object store pool...
20:51 championofcyrodi things will balance a little faster
20:52 newbism So as expected... I replaced the isolinux files to where the bug said to place them.... rebooted cobber docker container
20:52 newbism rebooted node
20:52 newbism still centos 6.5
20:53 newbism you can even see when booting the node it says loading /images/bootstrap/linux ..... /images/bootstrapinitramfs
20:53 newbism which is like... the whole point
20:54 newbism i guess try with a nightly
20:55 newbism i was assuming using an actual release would have given me what I wanted considering its stated to be supported on the roadmap since 5.1
20:56 tzn joined #fuel
21:09 newbism ticket updated
21:18 tzn joined #fuel
21:24 tzn joined #fuel
21:31 tzn joined #fuel
21:38 Red5CHS joined #fuel
21:39 Red5CHS Hey guys and gals. I've got a fuel 6.0 install that didn't finish because of missing broadcom 58710 firmware so no ethernet.
21:39 Red5CHS found this link: https://bugs.launchpad.net/fuel/+bug/1416610
21:39 Red5CHS Put the firmware in /lib/firmware. rebooted node and it came back up with configured ethernet ports. eth0, eth2, and eth3.
21:40 Red5CHS however... now I'm stuck. it would seem that the docker containers can't start because I'm assuming they need to be rebuilt with the driver... or something.
21:40 Red5CHS with everything installed, is there a way to sorta... reboot the set-up process?
21:44 tzn joined #fuel
22:00 tzn joined #fuel
22:05 tzn joined #fuel
22:10 LiJiansheng joined #fuel
22:26 tzn joined #fuel
22:27 tzn joined #fuel
22:27 tzn joined #fuel
22:36 tzn joined #fuel
22:49 daniel3_ joined #fuel
23:21 jaypipes joined #fuel
23:30 teran_ joined #fuel
23:33 kaliya_ joined #fuel
23:35 francois joined #fuel
23:38 xdeller_ joined #fuel
23:38 lee__ joined #fuel
23:38 aarefiev_ joined #fuel
23:38 youellet joined #fuel
23:39 dpyzhov joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary