Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-02-28

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:10 designated joined #fuel
00:20 designated any word on when fuel 4.1 will be released?
00:28 xarses designated: hopefully within the next 24 hours
00:39 miroslav_ joined #fuel
00:42 marcinkuzminski joined #fuel
00:51 ilbot3 joined #fuel
00:51 Topic for #fuel is now Fuel for Openstack: http://fuel.mirantis.com/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
00:52 xarses dhblaz, haven't found solution, but here is bug https://bugs.launchpad.net/fuel/+bug/1285929
00:57 angdraug joined #fuel
01:00 GeertJohan joined #fuel
01:00 GeertJohan joined #fuel
01:28 dmit2k xarses: may I ask you one more thing on Fuel project logics?..
01:28 xarses sure, about to sign off for a bit though
01:28 warpig_ joined #fuel
01:28 dmit2k xarses: say I have deployed a cluster with Fuel, say it is a 4.1
01:29 dmit2k xarses: then I want to install a plugin for Nova
01:29 xarses ok
01:29 dmit2k which is the 'proper' way?
01:29 dmit2k from the Fuel logic?
01:29 xarses currently, install it by hand
01:30 xarses puppet doesn't normally re-run
01:31 jeremydei joined #fuel
01:31 dmit2k so I can consider that Fuel-deployed cluster as a stand-alone thing, with limited monitoring from Fuel master node and ability to deploy additional nodes, such as OSD or compute, right?
01:31 dmit2k everything beyond that should be done manually
01:32 xarses or adding another controller
01:33 xarses but yes, if you want it to do something it needs to be set up before deployment
01:33 dmit2k so, I should not afraid of things like "losing compatibility with future versions" as you don't have an upgrade path either way?..
01:33 xarses we don't have any good lifecycle management yet, thats a 5.0, 5.1 item
01:33 xarses which is the same point for third-party plugin support
01:34 dmit2k clear, so my proposal is more or less correct, right?
01:34 xarses correct, if you want to do something besides adding or deleting whole nodes after deployment, you need to do it your self
01:34 dmit2k I'm in a situation, when I have to take decision which way to go
01:35 dmit2k at the moment we have a CEPH cluster (standalone) plus CloudStack cluster
01:35 dmit2k I don't like the way CS goes so looking for another alternative
01:36 dmit2k OS seems to be the only one, and after of month of experiments Fuel seems to be the easiest way of deploying OS
01:37 dmit2k I just want to get an understanding, which way the Fuel with Mirantis will go
01:38 xarses dmit2k: general roadmap https://wiki.openstack.org/wiki/Fuel
01:38 xarses hmm, someone just re-wrote the page
01:39 dmit2k it is not informative
01:39 xarses ya, not anymore
01:39 xarses it had some more useful roadmap info on it a week or two ago
01:40 dmit2k maybe I'm a bit paranoic, but while digging through Fuel docs I've got a feeling that something important is hidden out of the public view
01:42 dmit2k manuals are not complete, no solutions and guidelines for solving problems, no real community around... Looks like the company is reserving some space for commercial support :)
01:42 dmit2k but opposite:
01:43 dmit2k for the last few days I've got A LOT of support from your tech guys either on the Launchpad or on the IRC
01:43 dmit2k so I'm frustrated :)
01:43 xarses ya, are manuals are incomplete because of lack of time
01:44 dmit2k you do operate as a big company, but still provide support as an opensource garage
01:44 xarses well, if you register your fuel node for your 30 day trial, you can have a go at our paid support group
01:45 dmit2k do I need it?
01:45 dmit2k 15 years in this business
01:45 xarses depends on you, and your organization if you want to receive SLA based support services
01:45 dmit2k every time when I go through Managers, things are not being solved
01:46 xarses I'm a fuel developer, so are alot of the other mirantis folks in here, we simply help when able like most community channels
01:47 dmit2k BTW, we were thinking that this 30 days support is a kind of a gift, but actually - after 3 days online with your tech guys I don;t have a feeling that I will get more
01:48 xarses dmit2k: I've got to go now, I'll be back in a bit
01:48 dmit2k OK
02:11 dmit2k * /t FUEL is COOL, it just lacks documentation
03:38 dburmistrov joined #fuel
03:42 rmoe joined #fuel
03:44 xarses joined #fuel
04:06 dhblaz joined #fuel
04:06 fandi joined #fuel
04:12 topshare joined #fuel
04:13 topshare joined #fuel
04:13 topshare joined #fuel
04:29 topshare joined #fuel
04:30 fandi hi
04:32 fandi we're from cyberindo mega persada.. we have problem when try to move dhcp-agent and dnsmasq from 1 server to another server..
04:48 jeremydei joined #fuel
04:55 topshare joined #fuel
05:01 jeremydei joined #fuel
05:13 vkozhukalov joined #fuel
05:15 jeremydei joined #fuel
05:28 meow-nofer_ joined #fuel
05:28 ogelbukh joined #fuel
05:28 bookwar joined #fuel
06:05 bookwar joined #fuel
06:34 mihgen joined #fuel
06:39 designated is it normal for fuel to only load 4 nodes at a time?
06:40 designated tried to deploy changes to 8 nodes at once, 4 loaded but the other 4 are just sitting there
06:51 Ch00k joined #fuel
07:31 anotchenko joined #fuel
07:32 saju_m joined #fuel
07:35 rvyalov joined #fuel
07:39 e0ne joined #fuel
07:55 anotchenko joined #fuel
07:57 Ch00k joined #fuel
08:01 topshare joined #fuel
08:07 mrasskazov1 joined #fuel
08:13 mihgen joined #fuel
08:19 miguitas joined #fuel
08:31 anotchenko joined #fuel
08:32 vkozhukalov joined #fuel
08:43 anotchenko joined #fuel
08:46 evgeniyl designated: good morning, could you clarify please, what does mean 'load 4 nodes'?
08:51 Ch00k joined #fuel
08:53 rvyalov joined #fuel
08:54 alex_didenko joined #fuel
09:18 topshare joined #fuel
09:23 vk joined #fuel
09:29 Ch00k joined #fuel
09:46 topshare joined #fuel
09:50 saju_m joined #fuel
09:51 tatyana joined #fuel
10:13 dburmistrov joined #fuel
10:13 dburmistrov joined #fuel
10:32 anotchenko joined #fuel
10:33 e0ne_ joined #fuel
10:36 vk joined #fuel
10:37 rverchikov joined #fuel
11:27 anotchenko joined #fuel
11:49 vk joined #fuel
12:10 rverchikov left #fuel
12:11 mihgen joined #fuel
12:45 Ch00k joined #fuel
12:46 Dr_Drache joined #fuel
13:08 dburmistrov joined #fuel
13:10 baboune joined #fuel
13:13 baboune we are having a problem with Fuel and Ubuntu.  After ubuntu is installed, the machine reboots, and in the PXE setup communicates back with cobbler to get the pxelinux.cfg, right after receiving the info, it says "Booting ..." and then hangs.  The machines have multiple disks, we tried to set them up as many volumes with RAID-0, and as one volume with RAID5.  It is the same result.  The Fuel UI seems to detect the disks fine though.
13:28 TVR___ joined #fuel
13:31 Dr_Drache baboune, dell?
13:36 Dr_Drache baboune, I have the same issue.
13:36 anotchenko joined #fuel
13:37 TVR___ I think between us select few, we are fully vetting the fuel system and doing a very good job of QA on it...
13:38 Dr_Drache TVR___, you missed it : baboune> we are having a problem with Fuel and Ubuntu.  After ubuntu is installed, the machine reboots, and in the PXE setup communicates back with cobbler to get the pxelinux.cfg, right after receiving the info, it says "Booting ..." and then hangs.  The machines have multiple disks, we tried to set them up as many volumes with RAID-0, and as one volume with RAID5.  It is the same result.  The Fuel UI seems to det
13:38 Dr_Drache ect the disks fine though.
13:38 Dr_Drache same exact symtoms as I've been saying for 3 weeks.
13:39 TVR___ it is only with dell, correct?
13:41 TVR___ a bit of a knee jerk comment here... but personally, I am not a big fan of the way dell presents either it's NICs to the OS or it's disks... I have seen anomaly with BSD and dell as well, although with the newest kernels in ubuntu being used, this should not be presenting this issue
13:41 baboune no it s HP HW
13:42 baboune HP DL 380S to be precise
13:42 TVR___ interesting.. OK..
13:42 baboune we also have one node with HP320G5s
13:42 baboune same problem for both
13:42 TVR___ P400 seties controller on H series?
13:42 baboune the same thing worked fine on HP dekstop machines though
13:42 baboune yes
13:42 baboune one has a P400
13:43 baboune the other is an embedded RAID
13:43 baboune (I know this P400 is crap but .. that is what there is)
13:43 TVR___ wait.. embedded raid AND a P4XX controller?
13:43 baboune two machines: HP 380 -> P400, HP 320G5s -> embedded
13:44 TVR___ What slot is the P400 in?
13:44 TVR___ let me explain...
13:44 TVR___ I dealt with a hadoop cluster.... twin P400's
13:44 Dr_Drache baboune, the p400 is embedded
13:45 Dr_Drache 380 g5?
13:45 baboune yes DL380 G5
13:45 Dr_Drache baboune, there is a patch for you.
13:45 TVR___ the issue is, if the controllers are not in the slot closest to the power side.. as in C0 slot.. they will not boot the controllers in the same order twice
13:46 Dr_Drache TVR___, this is an issue with how bootstrap identifys the drives.
13:46 Dr_Drache I had that issue with my dells
13:46 Dr_Drache known issue with the smart arrays
13:46 TVR___ the main boards split their controllers for board slots
13:46 baboune ok that is good info.  I can take a look at the slots
13:47 TVR___ make sure it is in the first slot....
13:47 baboune the thing is that the issue is consistent across two machines with quite different layout 320 vs 380
13:47 baboune ok, I will go check
13:47 Dr_Drache baboune, there is a patch, I have one, but I don't think it's safe to give out.
13:47 Dr_Drache damn
13:47 Dr_Drache wish it was with a dell
13:47 baboune Dells are coming in 3-4 weeks
13:48 TVR___ I would also try, for testing, disabling the onboard raid and just see if it boots / takes
13:48 Dr_Drache baboune, what dells?
13:51 Dr_Drache I need to find someone who can test with a R415
13:51 Dr_Drache see if it's just me or what
13:55 baboune on the HP DL380 the RAID controller is on Slot1 which is the closest to the power supply
13:55 baboune there is no other slot closer that we can see
13:57 topshare joined #fuel
14:00 baboune I made a mistake the DL320 is also using a P400 controller
14:01 baboune on the 320 it is on a RIZER card
14:03 likelion joined #fuel
14:03 vkozhukalov baboune, can you give us access to your fuel master node via ssh?
14:03 zer0tweets joined #fuel
14:03 vkozhukalov the problem is that we use centos based discovery ram image
14:04 vkozhukalov it finds hard drives and we then try to identify them via their /dev/disk/by-path links
14:05 vkozhukalov then when we boot ubuntu installer kernel it maps hard drives in other by-path links
14:05 vkozhukalov couple of weeks ago we made a patch for solving this issue
14:06 Dr_Drache vkozhukalov, AFAIK it's been fixed by a patch.
14:06 Dr_Drache patch worked great here,
14:06 vkozhukalov we added identification using /dev/disk/by-id links
14:06 vkozhukalov ok, have you applied this patch on your master node?
14:06 Dr_Drache acually, first patch, I did it manually
14:07 Dr_Drache vkozhukalov, I have, he hasn't
14:08 vkozhukalov Dr_Drache, do you have any installing problems with ubuntu right now?
14:08 vkozhukalov I've heard that now ubuntu is installed correctly
14:09 vkozhukalov but then after reboot you have some problems with booting system
14:09 Dr_Drache yes sir
14:09 Dr_Drache it doesn't find the drive /dev/disk/by-uuid
14:10 vkozhukalov to find out what is going on, we need you to boot this problem node with some emergency ram image like RIP linux
14:10 Dr_Drache drops to a (initramfs) with a hard lock
14:10 Dr_Drache well, I'd need to redeploy with ubuntu.
14:10 Dr_Drache would you like that to happen?
14:12 vkozhukalov ok, sounds like you have wrong grub configuration
14:12 Dr_Drache well, redeploying here in 5 min.
14:13 Dr_Drache install is rather fast
14:13 vkozhukalov Dr_Drache, yes, if you can give us remote access to your master node, we can put everything we need (RIP linux) and then we need to boot problem node into RIP linux via pxe
14:15 vkozhukalov Dr_Drache, ideally we need remote access (ssh) and webex session with you or IPMI access to slave node
14:15 baboune is the patch available somewhere?
14:15 Dr_Drache i attempted ssh last time, masternode was rejecting all connections, and was just given a patch.
14:16 Dr_Drache and webex doesn't work in linux very well :P
14:17 vkozhukalov Dr_Drache, ok, i believe skype is also ok.
14:19 vkozhukalov baboune, yes, patch is available, MiroslavAnashkin can provide you this  patch and shell script to apply this patch
14:21 vkozhukalov baboune, which version do you use?
14:21 vkozhukalov baboune, 4.0?
14:25 Dr_Drache vkozhukalov, working on ssh access
14:27 vkozhukalov Dr_Drache, great
14:40 Dr_Drache vkozhukalov, doesn't look like the firewall is liking it
14:47 Dr_Drache vkozhukalov, no ssh, we can't sort out why the firewall is blocking
14:51 vkozhukalov Dr_Drache, we've discussed this issue a bit and it looks like we have a solution
14:52 Dr_Drache vkozhukalov, you software guys rock
14:52 Dr_Drache lol
14:53 vkozhukalov Dr_Drache, it is likely that the problem is connected with the fact that dell raids usually delay exposing there volumes
14:54 vkozhukalov Dr_Drache,  grub has default timeout about several seconds and then it reports that it is unable to find a disk
14:54 Dr_Drache right, also when grub loads, it's a 2-3 sec refresh rate (can see the screen load)
14:55 vkozhukalov Dr_Drache, so we can try to fix this problem with adding parameter into grub config
14:55 Dr_Drache it seems I still have an issue with the old version of the patch too.
14:55 vkozhukalov Dr_Drache, we need to add GRUB_CMDLINE_LINUX="rootdelay=90 nomodeset" into /etc/default/grub
14:56 Dr_Drache once I can get it installed, I shall
14:57 vkozhukalov Dr_Drache, just a second, I'll tell you where you need to add this line
14:58 vkozhukalov Dr_Drache, i mean you need to add it into cobbler snippet
14:58 Dr_Drache ok
14:59 baboune ok. some update.  we checked in cobbler the generated script has a wrong by-path id
14:59 baboune we are making a test to verify
14:59 Dr_Drache baboune, that's the patch
15:00 Dr_Drache you need to edit the initramfs to fix it properly
15:04 baboune thx for the help so far. have a good week end
15:05 vkozhukalov Dr_Drache, vim /usr/lib/python2.6/site-packages/cobbler/pmanager.py, then find where GRUB_CMDLINE_LINUX is present and append "rootdelay=90 " right after "console=tty0"
15:06 vkozhukalov Dr_Drache, I believe this line is 902, if we have the same version of this file
15:06 vkozhukalov Dr_Drache, then you need to restart cobbler (/etc/init.d/cobblerd restart)
15:08 Dr_Drache i'm looking
15:09 vkozhukalov Dr_Drache, if it helps then ok, the problem is crearly connected to dell RAID delay
15:09 Dr_Drache can't find the line yet
15:11 vkozhukalov Dr_Drache, try to download it here https://drive.google.com/file/d/0BybDDjx4oqkYSEpFbHpYVnRURUE/edit?usp=sharing
15:13 Dr_Drache yea
15:13 Dr_Drache my file is COMPLETLY differnt
15:16 Ch00k joined #fuel
15:17 Dr_Drache vk
15:17 Dr_Drache vkozhukalov, http://paste.openstack.org/show/70664/
15:19 TVR___ anyone ever benchmark or do benchmark tests against their object store storage in here? Mirantis side?
15:19 TVR___ cosbench is being a bear to get setup....
15:19 TVR___ rados bench limits what I can get for data
15:21 TVR___ just wondering if the Mirantis side in here has a favorite tool for testing max connections, IO, and IOPs to the object store
15:21 vkozhukalov Dr_Drache, looks like your version of pmanager.py is ancient.
15:22 Dr_Drache vkozhukalov, yea, perhaps.
15:23 mihgen joined #fuel
15:24 miroslav__ Dr_Drache: Have you re-deployed your master node recently?
15:24 miroslav__ And have you forgotten to re-apply the patch after re-deploy?
15:25 MiroslavAnashkin http://download.mirantis.com/fuelweb/fuel_partition_manager_patch_40_to_41_ver4.run
15:25 Dr_Drache MiroslavAnashkin, I have not.
15:25 MiroslavAnashkin This is the link for the latest patch version
15:25 Dr_Drache it's the same node as before with the old patch you gave me
15:26 Dr_Drache i assume I need to revert first
15:27 MiroslavAnashkin BTW, there is one more, a backport of network settings from 4.1 to 4.0. It adds stability and fixes different cases, found last month
15:27 MiroslavAnashkin http://download.mirantis.com/fuelweb/fuel_l23_network_4.0_to_4.1_backport_ver1.run
15:28 MiroslavAnashkin Dr_Drache: The output you shared is possible only if master node is not patched at all. You might occasionally rolled back the patch, it has such functionality
15:28 Dr_Drache MiroslavAnashkin, but I assure you, I did nothing with the patch but what I was instructed to by you
15:29 MiroslavAnashkin Please re-apply. Select Apply and Overwrite then asked by the patch script
15:29 MiroslavAnashkin Latest version is anyway better.
15:30 jobewan joined #fuel
15:38 Dr_Drache MiroslavAnashkin, vkozhukalov
15:38 Dr_Drache patches applyed
15:38 Dr_Drache file edited
15:40 Dr_Drache cobber restarted, rebootstrapping the nodes
15:41 Dr_Drache MiroslavAnashkin,
15:42 Dr_Drache in fuel UI, setup tab.
15:42 Dr_Drache "use Qcow2 for images" should be unchecked, if i am doing a full CEPH backend right?
15:46 Dr_Drache brb
15:47 Dr_Drache left #fuel
15:47 Dr_Drache joined #fuel
15:50 Dr_Drache vkozhukalov, MiroslavAnashkin, non booting nodes
15:50 Dr_Drache MiroslavAnashkin, same error as the first patch.
15:50 Dr_Drache I ran cobbler sync
15:55 Dr_Drache seems only a reboot really fixed that
16:06 dburmistrov joined #fuel
16:07 asledzinskiy left #fuel
16:07 Dr_Drache vkozhukalov, MiroslavAnashkin, seems to have booted, but completly differnt error now. lol
16:09 MiroslavAnashkin Interesting, what else may block Cobbler sfter the patch... Sorry, I've been both eyes in the patch code
16:10 Dr_Drache https://www.dropbox.com/s/b0r2rep68f2xmc4/2014228110850.jpg
16:10 Dr_Drache MiroslavAnashkin, no idea, it works fine after a reboot
16:10 Dr_Drache but now that error
16:10 MiroslavAnashkin Dr_Drache: Was it master node?
16:11 MiroslavAnashkin Ah, I see
16:12 Dr_Drache MiroslavAnashkin, yes, masternode needed a reboot after the patch.
16:12 Dr_Drache that screen is the dell after ubuntu install
16:12 MiroslavAnashkin I am about the screenshot
16:13 Dr_Drache thew screen is a controller node on the dell r415
16:15 MiroslavAnashkin Looks like Ubuntu failed to load Matrox graphics driver. Anyway, there is no X installed to OpenStack nodes at all. Most probably we should ignore this message
16:15 Dr_Drache i'm talking about the system V
16:15 Dr_Drache it stalls there
16:15 Dr_Drache all nodes do
16:16 Dr_Drache all 3 stalled there
16:17 MiroslavAnashkin Dr_Drache: Please go to Fuel UI, anf click Log sign on the controller node to see what actually happened
16:17 Dr_Drache MiroslavAnashkin, nothing shows after the grub install
16:18 Dr_Drache unless I go to debug
16:18 MiroslavAnashkin Second known issue with Ubuntu - it does not show console in case error happened on boot
16:19 Dr_Drache compute nodes are stuck on the system V
16:19 MiroslavAnashkin Well, try the following way: 1 Go to the logs tab. 2.  Select Other nodes instead of Fuel master. 3  Select controller node. 4. Select puppet as source. 5. Click Show
16:21 Dr_Drache ...
16:21 Dr_Drache there is no puppet as source
16:21 MiroslavAnashkin BTW, what is the node status Fuel UI show - Installing Ubuntu or Ubuntu Installed?
16:22 Dr_Drache either way
16:22 Dr_Drache ubuntu failed to deploy.
16:22 Dr_Drache installing
16:23 MiroslavAnashkin Well, looks like Ubuntu really failed to start...
16:24 Dr_Drache I forced rebooted the controller
16:24 Dr_Drache and it came up
16:24 Dr_Drache but no communication to the master
16:47 MiroslavAnashkin Dr_Drache: Looks like all COM/Serial ports are disabled in BIOS on your Dell
16:49 MiroslavAnashkin Dr_Drache: Some time ago we fixed issue for machines without video card at all - but looks like this fix crashes OS on boot when no one serial port available.
16:49 MiroslavAnashkin Dr_Drache: Please check BIOS settings on the failed node
16:52 Dr_Drache joined #fuel
16:52 MiroslavAnashkin Probably, we should not enable serial console in GRUB by default, but have such setting in Fuel UI
16:52 Dr_Drache well
16:53 Dr_Drache all of them are enabled by default
16:53 Dr_Drache virtual and phyiscal
16:54 MiroslavAnashkin Hmm..
16:54 Dr_Drache ok
16:54 Dr_Drache on the master node.
16:54 Dr_Drache how do i clear logs?
16:54 Dr_Drache so when i make a new diag snapshot, it's not huge
16:58 MiroslavAnashkin Dr_Drache: Serial console option is also incompatible with selection of video drivers with default GRUB settings.
17:00 Dr_Drache MiroslavAnashkin, so, remove the "no mode set"
17:00 Dr_Drache or?
17:01 MiroslavAnashkin Wait a  bit, I'll check the remained options in GRUB defaults
17:02 Dr_Drache k
17:06 angdraug joined #fuel
17:08 TVR___ so future question... for the ceph backend... how would we do journaling on SSDs with fuel?
17:08 TVR___ I am sure this is on the roadmap somewhere...
17:09 e0ne joined #fuel
17:09 MiroslavAnashkin Dr_Drache: Please leave "nomodset" in place.
17:10 Dr_Drache k
17:11 likelio joined #fuel
17:11 angdraug TVR___: we already have that )
17:12 angdraug all you need to do is edit your disk configuration before deployment
17:12 angdraug and allocate your SSDs to Ceph Journal
17:12 MiroslavAnashkin Dr_Drache: We have to remove all instances of "console=tty0 console=ttyS0,9600" from /boot/grub/grub.cfg
17:14 Dr_Drache MiroslavAnashkin, I have that deployment deleted.
17:14 likelion joined #fuel
17:14 Dr_Drache is that something I can remove from the cobbler script?
17:14 TVR___ ok.. so set all the SSD to Ceph Journal and all the SATA or whatever to just Ceph in the disk settings... cool
17:15 MiroslavAnashkin Dr_Drache: Yes, in Cobbler script. And remove "GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600" from /etc/default/grub
17:15 TVR___ did not notice that..heh
17:17 Dr_Drache MiroslavAnashkin, I happen to not be good at python, here is a paste with how I think the edits should be : http://paste.openstack.org/show/70697/
17:22 MiroslavAnashkin Dr_Drache: Yes, please edit /usr/lib/python2.6/site-packages/cobbler/pmanager.py
17:23 Dr_Drache MiroslavAnashkin, done
17:23 dmit2k joined #fuel
17:24 dmit2k hello everyone! any news for 4.1 release?..
17:24 Dr_Drache dmit2k, they are trying to get a few more days
17:25 dmit2k OK, I had this feeling...
17:25 TVR___ dmit2k that is Dr_Drache's job to ask for 4.1 at least 3 times a day !!
17:25 TVR___ heh
17:25 dmit2k :-D
17:26 Dr_Drache LOL
17:26 Dr_Drache TVR___, not anymore
17:26 Dr_Drache time for 5.0
17:26 dmit2k mihgen yesterday discussed possible release today, so I asked :)
17:27 TVR___ good article on what you can expect from ceph as the back end in OpenStack  http://software.intel.com/en-us/blogs/2013/10/25/measure-ceph-rbd-performance-in-a-quantitative-way-part-i
17:27 MiroslavAnashkin Dr_Drache:  Like following http://paste.openstack.org/show/70698/
17:31 Dr_Drache MiroslavAnashkin, edited, cobbler restarted, and deploy attempting
17:33 MiroslavAnashkin Dr_Drache: It is good you don't hear what everybody around says about the usage of quotation marks in this python code piece :-)
17:41 designated has anyone encountered problems with mac to nic mappings being incorrect after you start the deployment process in fuel?  I can boot a node and it pulls the correct ip address on eth0 but when a role has been defined and the node is deployed, after bootstrap and during OS load, it seems like the udev mapping is incorrect because eth0 winds up on a different switch port.
17:41 designated it doesn't always happen, I tried deploying 8 nodes last night and four of them succeeded, the other four had problems with the udev mapping
17:49 jobewan joined #fuel
17:50 Dr_Drache joined #fuel
17:50 rmoe joined #fuel
17:50 MiroslavAnashkin designated: Fuel 4.0?
17:52 meow-nofer__ joined #fuel
17:53 angdraug joined #fuel
17:55 designated MiroslavAnashkin: yes 4.0
18:03 MiroslavAnashkin designated: Have you disabled selected network cards in machine BIOS?
18:04 Dr_Drache joined #fuel
18:04 rmoe joined #fuel
18:05 designated MiroslavAnashkin: I have not
18:06 miguitas left #fuel
18:13 MiroslavAnashkin designated: Are you deploy CentOS or Ubuntu?
18:13 rmoe joined #fuel
18:13 designated MiroslavAnashkin: ubuntu
18:28 angdraug_ joined #fuel
18:31 xarses joined #fuel
18:31 alex_didenko joined #fuel
18:35 Dr_Drache hmm
18:46 xarses designated: it's been fixed for 4.1
18:46 designated xarses: thank you
18:47 designated I heard it's going to be a couple more days for 4.1 now?
18:47 MiroslavAnashkin xarses: Cannot find link to appropriate pull request
18:47 xarses unfortunately, yes. We found a massive issue that we cant release with
18:48 xarses MiroslavAnashkin: for the ubuntu nic order?
18:48 MiroslavAnashkin xarses: yes
18:48 xarses MiroslavAnashkin: one min
18:49 designated does 4.1 address backing up an environment in case fuel has to be rebuilt?  exporting/importing environments?
18:49 designated if someone has a link to a changelog for 4.1 i would be more than happy to read through it
18:50 xarses we have a release notes around, but I don't think we have a nice looking change log other than the git logs on the repos
18:52 xarses designated: https://github.com/stackforge/fuel-docs/tree/master/pages/release-notes/v4-1
18:52 xarses I don't think there are any "built" copies around yet
18:53 designated xarses: thank you
18:53 xarses designated: MiroslavAnashkin: https://bugs.launchpad.net/fuel/+bug/1269920
18:54 Dr_Drache MiroslavAnashkin, trying a deploy again. but so far no dice
18:56 xarses http://paste.openstack.org/show/70698/ is missing text, not sure what thats supposed to do
18:57 xarses MiroslavAnashkin: ^
18:58 likelion joined #fuel
18:58 MiroslavAnashkin ?
18:58 Dr_Drache MiroslavAnashkin, that edit broke fuel
18:58 Dr_Drache lol
18:58 xarses that paste has some 's' missing, not sure the intent
18:59 MiroslavAnashkin Dr_Drache: OK, then all these quotations were not necessary
18:59 vkozhukalov joined #fuel
19:00 Dr_Drache at this point
19:01 Dr_Drache centOS and ubuntu are undeployable
19:02 Dr_Drache either pxe cannot locate boot server.
19:02 Dr_Drache or, when it does, it fails and drop to non-automated mode
19:05 Dr_Drache MiroslavAnashkin, I think we are missing a " somewhere or something
19:08 MiroslavAnashkin Dr_Drache:  Please try to change pmanager with the following file https://drive.google.com/file/d/0B4szgxpvz2aCRFV5eFI5YjFtLUE/edit?usp=sharing
19:11 MiroslavAnashkin Please rename the file under the link to pmanager.py
19:11 MiroslavAnashkin and place into  /usr/lib/python2.6/site-packages/cobbler/pmanager.py
19:11 Dr_Drache done
19:17 designated if you guys need someone to test 4.1 I do have a lab with 8 dell c6100s I am testing on.
19:26 Dr_Drache designated, we can test it when it's out :P
19:27 designated Dr_Drache: word
19:31 Dr_Drache designated, I mean, think about how many bugs just have been fixed by the 4-5 people here in the last 2 weeks
19:33 designated Dr_Drache: true
19:34 designated is it still worth submitting diagnostics for 4.0?
19:35 Dr_Drache if you have a reason to, then heck yea
19:35 Dr_Drache untill everyone can get 4.1, 4.0 is still the working version
19:36 MiroslavAnashkin designated:  It is still very helpful. And 4.0 remains in use after 4.1 out, since it is the only way to get RHOS deployed.
19:37 Dr_Drache people use RHOS?
19:37 Dr_Drache :P
19:39 designated going through a deployment right now and getting quite a few errors on controller install.
19:39 designated invoke-rc.d: initscript qemu-kvm, action "start" failed. dpkg: error processing qemu-kvm (--configure):
19:39 designated after it finishes or outright fails i can generate diagnostics.  who should i send them to or does it automatically upload somewhere?
19:41 designated Base/Exec[cinder-manage db_sync]) Skipping because of failed dependencies
19:42 designated fuel has internet access so im not sure why it's not resolving these dependancy issues
19:45 Dr_Drache designated, MiroslavAnashkin can handle it, or open a ticket
19:45 designated i don't want to step through each error, i'm just wondering where i should send the diagnostic report
19:49 dhblaz joined #fuel
19:49 dhblaz We are having problems with assigning floating IPs
19:49 dhblaz I see this in our logs:
19:49 dhblaz 2014-02-28T18:14:51.706557+00:00 debug:  2014-02-28 18:14:46 ERROR nova.api.open
19:49 dhblaz stack [req-c90f3004-a499-4583-8e9d-2d2f29634cc7 e649d971f45f448f96d0a15b11e9f690
19:50 dhblaz 63adb3d6a61243e78ad91b3831bd1f0b]  Caught error: 409-{u'NeutronError': {u'messa
19:50 dhblaz ge': u'No more IP addresses available on network 5c70fbde-ccfe-4ed6-89d5-9b2247c
19:50 dhblaz dfa8c.', u'type': u'IpAddressGenerationFailure', u'detail': u''}}
19:50 dhblaz But when I use neutron floatingip-list it shows we have several IPs available
19:51 dhblaz When I look in horizon at the list of floating IPs they all show down, even the ones that are assigned (previously) and working.
19:53 mutex dhblaz: what version of fuel ?
19:53 dhblaz 4.0
19:57 Dr_Drache MiroslavAnashkin,
19:57 likelion joined #fuel
19:57 Dr_Drache I think we are suscessful
20:00 MiroslavAnashkin Not me. I still have to assemble small patch to enable/disable serial console
20:04 mutex MiroslavAnashkin: enable serial console for which ?
20:06 MiroslavAnashkin For whose who needs it. It is necessary only for servers without any graphics card and conflicts with certain graphics drivers if card is on-board.
20:07 mutex ah
20:07 xarses odd, I've not seen that conflict with onboard before
20:07 mutex I wrote a hook for TCP serial console for instances
20:07 mutex my team needed it in case their windows VMs blue screened
20:07 MiroslavAnashkin Dr_Drache: And looks like we  (OK, me) shouild include timeout to the common partition manager patch as well.
20:12 Dr_Drache MiroslavAnashkin, controller fully deployed
20:12 Dr_Drache waiting on the computes
20:17 Dr_Drache shit
20:17 Dr_Drache error
20:20 Dr_Drache MiroslavAnashkin, LOL, I have one node that won't deploy openstack, but ubuntu is installed
20:20 designated with 4.0 how do i manually stop the deployment process?
20:20 xarses designated: wait for it to time out
20:21 designated ouch
20:21 xarses or cause it to error out
20:21 MiroslavAnashkin designated: Delete the env if OS is already deployed
20:21 xarses by say, turning off all the remaining nodes
20:21 designated I've turned them all off
20:22 Dr_Drache xarses, MiroslavAnashkin... I have a node that won't redeploy/deploy OS..... steps?
20:22 MiroslavAnashkin Dr_Drache: Probably it is related to network connection.
20:22 xarses Dr_Drache: goto cobbler https://<fuel_node>/cobbler_web
20:22 Dr_Drache shit....
20:22 Dr_Drache I screwed up
20:23 Dr_Drache I hit delete
20:23 Dr_Drache lol
20:23 Dr_Drache redeploy from scratch
20:23 Dr_Drache I am an idiot
20:23 xarses Dr_Drache: login and change netboot to enable on the system page
20:28 Dr_Drache ok
20:30 designated is there a way to update fuel to get the latest patches?
20:31 xarses designated: in a complicated and non-developer unfriendly way, "yes"
20:32 designated so the best way is to download the latest iso?
20:32 xarses There is one caviot, we can't rebuild the deb package repo in the CentOS fuel node so you have to use the iso tools to build it and scp it in
20:33 designated eww
20:33 xarses most of the everything else can be replaced in flight but its also hacky
20:33 Dr_Drache yea
20:33 xarses the best way at this point is to pull in the latest iso. 5.0 "should" be able to get some updates more easily
20:34 Dr_Drache even the patches arn't very nice.
20:34 xarses Dr_Drache: ya, especially for cobbler. manifests and nailgun themselves are easy to change
20:34 xarses naily and astute are more coplicated
20:35 Dr_Drache it takes a sync, and a reboot
20:35 xarses and packages are painful
20:35 Dr_Drache you'd think they'd be simple by now.
20:35 Dr_Drache this reminds me of linux in 1999
20:35 designated with all of the trouble i've had it seems easier to just install os and script out a manual openstack deployment, do it once and deploy it many times.
20:36 Dr_Drache designated, you must not have done many scripts.
20:36 Dr_Drache :P
20:37 designated I've built openstack by hand quite a few times and never had this many problems :/
20:37 MiroslavAnashkin Continuous automatic update for Fuel is planned feature. Not sure, for 5.0 or 5.1
20:37 Dr_Drache well, the real question, if it's that much easyer, why are you here?
20:38 designated trying to get it to work :)  I've also tried juju+maas
20:38 designated there is no way to determine if it will make your life easier until you try it right?
20:38 Dr_Drache true.
20:38 Dr_Drache I see promise
20:39 Dr_Drache that's why I am still around.
20:39 Dr_Drache too bad there are too many people trying to make their own.
20:39 designated i have definitely seen a lot of progress with fuel since back in the day, modifying config files and deploying via cli
20:39 designated i never did get that to work.
20:40 xarses designated: heh, I'm a dev for fuel and i barely got that to work
20:40 xarses =)
20:40 Dr_Drache OS does NOT have a future with 10 companies with their own deployment software,
20:40 designated xarses: i don't feel so bad then lol
20:41 xarses the new cli tool is much easier since its first rendered by the api so you at least dont start with a broken example
20:41 Dr_Drache if all these people with their own software came togeather, I say 6 months we could have one near flawless
20:41 xarses but it works if you dont go gun-ho custom
20:41 xarses and still allows you to hang yourself if you do
20:41 MiroslavAnashkin Dr_Drache: Did you meant Linux is dead?
20:41 designated the troubleshooting is problematic because it takes so long to get from one error to the next, resolve, try again etc...  it reminds me of trying to resolve dependancy problems back in the day
20:42 Dr_Drache MiroslavAnashkin, no.
20:42 designated MiroslavAnashkin: i think he's just trying to say it would be better if everyone focused on a single project
20:42 Dr_Drache but the same idea can be applyed to distros.
20:42 Dr_Drache super fragmented.
20:42 likelion joined #fuel
20:43 Dr_Drache xarses, cobbler_web
20:44 Dr_Drache login?
20:44 xarses Dr_Drache: ya, we are working towards integrating with ironic and Tripple0
20:44 Dr_Drache I forgot it again
20:44 xarses cobbler cobbler
20:44 xarses really secure supper secret password
20:44 xarses =P
20:44 designated juju worked great if you only wanted one interface with everything bridged to it.  other than that you have to develop custom charms and their irc channel is dead.  Fuel is a lot more active and helpful.
20:45 xarses designated: you might like the new bonding ui in 4.1
20:45 designated xarses: i read that in the new features, that is the feature i'm most excited about.
20:45 xarses its fancy looking, there is still more settings you can do in the cli way, but it helps alot for most deployments
20:46 designated single 1Gbe connection for administration, bond two 10Gbe NICS and trunk them.
20:46 Dr_Drache xarses, i'm getting unable to boot from pxe on a node I just tryed to deploy
20:47 xarses Dr_Drache: the node doesn't find the dhcp address? or does it load the menu and the kernel doesn't start?
20:47 Dr_Drache gets a dhcp, then nothing.
20:47 xarses so then no menu?
20:47 Dr_Drache "cannot locate boot server"
20:48 xarses do a cobbler sync from the command line and restart the node that won't boot
20:48 xarses make sure cobbler sync doesn't have any odd looking messages
20:49 Dr_Drache no change
20:50 pixelfog joined #fuel
20:51 xarses check that dnsmasq is running, check /etc/dnsmasq.conf for bad addresses
20:52 xarses check that xinetd is running
20:52 xarses (it starts tftp on demand)
20:53 xarses make sure there isn't another dhcp server on the subnet, and you can grab a tftp client and make sure tftp spawns
20:55 Dr_Drache the other 2 nodes deployed fine
20:57 xarses its also possible, however unlike that the node that doesn't work could benefit from some firmware/bios updates
20:57 xarses s/unlike/unlikely/
20:58 Dr_Drache yea
20:58 Dr_Drache that was done yesterday
20:58 xarses damn
20:59 Dr_Drache deleted the cluster it was part of
20:59 Dr_Drache and bootstrapped properly
21:03 xarses odd
21:04 Dr_Drache very
21:04 Dr_Drache but
21:04 Dr_Drache whatever
21:04 Dr_Drache hey
21:04 Dr_Drache using CEPH backend for everything
21:04 Dr_Drache i should UNCHECK the use qcow2 option, right?
21:05 designated by default bootstrap uses centos right?
21:05 Dr_Drache yes
21:05 designated if that is the case i assume that is the cause of incorrect mac address to nic mappings when loading ubuntu because of the differences in udev
21:06 designated is there a way to change the default bootstrap to ubuntu instead of centos?
21:07 designated so that my bootstrap mac address to nic mappings will be the same as when the OS starts loading
21:07 Dr_Drache would have to rebuild fuel
21:07 Dr_Drache but they are now.
21:08 designated surely that is stored in a config file somewhere
21:15 Dr_Drache what is in a config file?
21:17 designated whether or not to bootstrap a node using centos or ubuntu
21:18 Dr_Drache xarses, 2 nodes doing it this time
21:19 xarses Dr_Drache: thats, oddness.
21:19 xarses deleting them returns them to bootstrap too
21:19 xarses completely baffles me
21:19 Dr_Drache yes, but once out of bootstrap
21:19 Dr_Drache they can't boot.
21:21 xarses Dr_Drache: can you paste /var/lib/tftpboot/pxelinux.cfg/<MAC> for one of the nodes
21:22 xarses designated: no, we currently cant replace the bootstrap node, but the new patches fix that
21:23 xarses designated: *cant replace with ubuntu* I'd guess that you can build one your self if you wanted to hack on the iso build scripts
21:23 xarses it was something i was toying with trying as it was
21:23 designated i guess for testing then I'll deploy centos so my nic mappings stay consistent.
21:24 Dr_Drache xarses, http://paste.openstack.org/show/70758/
21:29 AndreyDanin joined #fuel
21:31 strictlyb joined #fuel
21:36 Dr_Drache xarses,
21:36 Dr_Drache is there a way to "clean out" the database?
21:36 xarses you want to reset to empty? yes
21:37 Dr_Drache I want to go back to fresh installed master
21:37 Dr_Drache but with my patches and fixes in tact
21:37 xarses yep
21:38 xarses one sec
21:38 xarses http://paste.openstack.org/show/70760/
21:41 Dr_Drache ./reset-db.sh: line 4: =/opt/nailgun/bin/manage.py: No such file or directory
21:41 Dr_Drache dropdb: missing required argument database name
21:41 Dr_Drache Try "dropdb --help" for more information.
21:41 Dr_Drache ./reset-db.sh: line 7: syncdb: command not found
21:41 Dr_Drache ./reset-db.sh: line 8: loaddefault: command not found
21:43 Dr_Drache but it's physically there
21:45 Dr_Drache blah
21:45 Dr_Drache [root@fuel ~]# ./reset-db.sh
21:45 Dr_Drache dropdb: missing required argument database name
21:45 Dr_Drache Try "dropdb --help" for more information.
21:45 Dr_Drache ./reset-db.sh: line 7: syncdb: command not found
21:45 Dr_Drache ./reset-db.sh: line 8: loaddefault: command not found
21:50 mutex is crm resource reprobe supposed to work ? I can force daemons to start but reprobe just times out after 60 seconds
21:52 mutex my neutron-l3-agent and dhcp-agent are in a 'stopped' state
21:52 mutex no amount of start, restart, reprobe, or force start seems to get it back to a started state
21:52 mutex I can manually start it with crm_resource --resource XYZ --force-start --node node-13
21:53 mutex but that still doesn't get the status into crm status
21:53 mutex into started state
21:53 Dr_Drache worked if i did them manually
21:53 Dr_Drache scripts suck sometimes
21:53 mutex ah
21:53 xarses Dr_Drache: odd, that works for me 100%
21:54 xarses mutex: crm resource <resource> cleanup
21:54 mutex yeah...
21:54 xarses force starts make it not track the current state correctly
21:55 mutex yeah
21:55 mutex it's interesting... it picked up the state momentarily
21:57 xarses mutex: is the pid of dhcp-agent changing?
21:59 mutex great question
22:00 mutex looks like no
22:02 xarses mutex: I ask, because there was a bug in 4.0 with the pacemaker scripts loosing track of the l3-agent and dhcp-agent pid and killing them off randomly
22:02 mutex yeah
22:02 mutex still investigating
22:03 xarses regardless if its in a funky state now, you have to wack on crm to stop the services, and not attempt to start them, once they are really stopped and not respaning, do a cleanup, and then ask crm to start them and it should return normal
22:03 mutex that may be what is happening here
22:03 mutex because it finds the service, the is running the mirantis/l3-agent stop ocf script
22:03 mutex k
22:04 xarses mutex: https://bugs.launchpad.net/fuel/+bug/1269334
22:04 xarses s/respaning/respawning /
22:10 rmoe joined #fuel
22:11 23LAAEV6P joined #fuel
22:11 meow-nofer joined #fuel
22:12 angdraug joined #fuel
22:12 AndreyDanin joined #fuel
22:12 mutex hrm, well it definitely is off
22:13 mutex can't seem to get crm to fix things up
22:13 vkozhukalov joined #fuel
22:14 strictlyb joined #fuel
22:20 mutex ah ha
22:20 mutex openvswitch agent is down
22:20 mutex that is why nothing else is starting
22:20 xarses ok, so thats totally the bug above
22:24 designated if i use trunks with centos do i have to enable ovs vlan splinters?
22:26 mutex ok, it is all better now
22:31 mutex sad that it doesn't tell you 'oh I cannot start these services because my dependancies are not met'
22:37 jobewan joined #fuel
22:39 meow-nofer__ joined #fuel
22:41 angdraug_ joined #fuel
22:46 strictlyb joined #fuel
22:48 AndreyDanin joined #fuel
23:03 xarses designated: it will be more reliable but some nics do work with out vlan splinters
23:03 xarses mutex: ya, crm isn't verry talkative about that kind of stuff
23:12 mutex sad
23:13 designated xarses: thanks I'll keep that in mind.
23:14 designated wth is this? Advanced_node_netconfig/L23network::L3::Ifconfig[eth5]/Notify[ifconfig_eth5]/message) defined 'message' as 'Interface:eth5 IP:none/'
23:15 designated i never designated eth5 for use
23:15 designated I'm guessing it rolls through every interface?
23:15 xarses puppet acts on it to make sure its not got any configuration it
23:20 xarses designated: its slightly annoying if you wanted to have config on it. but it's great if you left an interface in dhcp
23:27 designated xarses: fair enough
23:29 designated time is set correctly on fuel host os but log timestamps are incorrect.  any way to resolv that?
23:32 vk joined #fuel
23:34 ogelbukh joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary