Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-08-04

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 Nuc_ joined #fuel
02:23 thehybridtech joined #fuel
03:11 jaypipes joined #fuel
03:15 xdeller joined #fuel
03:16 aedocw joined #fuel
04:21 ArminderS joined #fuel
05:25 e0ne joined #fuel
06:17 dpyzhov joined #fuel
06:21 Nuc_ joined #fuel
06:22 ykotko_ joined #fuel
06:24 ddmitriev joined #fuel
06:30 adanin joined #fuel
06:49 Longgeek joined #fuel
06:55 TomekG joined #fuel
07:04 pasquier-s joined #fuel
07:06 e0ne joined #fuel
07:07 TomekG Hi, I've notice a quite serious SPOF in HA deployment. All nodes use dns resolver on fuel-master, so failure of master cause serious problems on controllers. All nodes have other nodes in their /etc/hosts, so what is the reason to set resolver to master instead of nameservers configured for particular environment or globally in fuel-menu?
07:08 TomekG What is a recommendiation for production environments?
07:21 holser joined #fuel
07:26 Longgeek_ joined #fuel
07:48 artem_panchenko joined #fuel
07:51 artem_panchenko joined #fuel
08:04 mattymo TomekG, are you still here?
08:15 artem_panchenko joined #fuel
08:35 TomekG mattymo, Yes I am
08:35 egor joined #fuel
08:37 mattymo TomekG, great. this poses a real serious issue for deployment if we use upstream DNS on controllers/compute nodes
08:37 mattymo if we point to upstream DNS servers, it will not know how to resolve other controllers unless you set up DNS delegation appropriately on your company's DNS server
08:37 mattymo we can't automate that or expect people to set it up for even very simple deployments
08:38 TomekG I suppose the problem occurs when /etc/hosts hasn't been updated yet?
08:38 mattymo /etc/hosts I believe now gets updated any time the cluster gets redeployed
08:39 TomekG so is it safe to change it after deployment?
08:39 mattymo you can change, sure. Either leave the /etc/hosts entries or leave Fuel Master as resolver. But if you change both, you'll definitely break rabbitmq
08:40 mattymo TomekG, is there a bug report already on launchpad about this?
08:41 TomekG in the previous release (4.1) it wasn't a big problem, because fuel master boots in seconds, now in 5.0 it takes much longer due to docker containters
08:41 TomekG no
08:42 mattymo sorry about the slower 5.0 booting. In 5.1 the containers are a little faster
08:42 mattymo and I cleaned up the manifests for Cobbler so it will restart a bit faster in 5.1
08:44 mihgen mattymo: I think it should not be SPoF as we have all required entries in /etc/hosts on controllers
08:44 mihgen so resolution inside of the env should work just fine
08:44 mattymo so which piece exactly is failing? resolution of external domains?
08:44 mihgen VMs are might be in trouble though, connectivity from VM
08:45 mihgen yep, if you ping google.com from VM, by default it gets DNS from external Neutron router I believe, which gets it from the host, and host gets it from Fuel master node
08:45 TomekG we had a problem with hypervisor with fuel-master, and I was not able to login in Horizon for example
08:45 mihgen something like this
08:45 mihgen TomekG: that's strange
08:45 TomekG after replacing resolvers in resolv conf everythings started to work fine
08:46 mihgen TomekG: did you use hostname of node with Horizon?
08:46 mihgen or just IP address?
08:46 TomekG I'll try to repeat this problem
08:46 mihgen TomekG: thanks, feel free to file bugs!
08:47 TomekG hostname, we've changed API endpoints to FQDN
08:47 TomekG this is the direct cause I think
08:48 mattymo you should add the new fqdn for your controllers in /etc/hosts as a second name
08:49 mattymo like 10.20.0.5 node-1 openstack-api-01.mycompany.com
08:49 TomekG good idea, I'll do it
08:50 mattymo but I'd like to see what kind of logs are showing up in apache. It might have some useful data
08:51 ebartz joined #fuel
08:54 TomekG @mihgen But does is still a bug if it was caused by my custom change of endpoints to FQDN?
09:01 sikor_sxe joined #fuel
09:01 mihgen TomekG: well I would need more details
09:02 mihgen did you put IP to your DNS configuration? Resolution worked, but Horizon didn't?
09:04 TomekG mattymo: I've run tail -f on apache and keystone logs on all controllers during failed login, and there is only POST /horizon/auth/login/ HTTP/1.0" 200 in apache, and <140>Aug  4 11:02:24 node-22 dashboard-openstack_auth.forms: WARNING Login failed for user "admin" in dashboard log, it seems it couldn't reach keystone api
09:06 TomekG mighen: no, when problem occurs, fuel-master was down, and I have it as a resolver in /etc/resolv.conf (default config). My custom change was replacing IP for FQDN in endpoint catalog, so django process couldn't connect keystone api
09:07 TomekG I think solution to this problem is to add endpoints to /etc/hosts on controller and compute nodes, as mattymo said
09:08 mattymo yeah that makes sense totally
09:08 mattymo just updating your /etc/hosts still sounds appropriate
09:10 TomekG The only process which needs to resolve external hostnames is Glance to download images from internet, so only glance is susceptible to dns problems when master is down
09:11 TomekG But it is quite rare.
09:13 TomekG So I'll keep master as a resolver, and add additional domains to /etc/hosts. I'll let you know, if I'll notice something new.
09:13 TomekG Thank for help
09:15 mihgen this is not gonna be a full solution. The issue is that in general, you run "nova list" from you local machine
09:15 mihgen request goes to keystone, which returns list of endpoints
09:15 mihgen as you say, some of them (or all) will contain FQDN, not IP
09:16 mihgen so your local machine would have to know what those FQDNs are
09:18 TomekG When I run "nova list"
09:18 TomekG I need to have fully functioning DNS on my local machine
09:19 TomekG And the case you described is not the same issue
09:20 TomekG I'm not sure, but there is no DNS resolution on contorllers when I use API from remote host, maybe only for logging purposes, or when command like "nova image-list" wants to connect to glance API.
09:27 TomekG I'll be back in about 1h
09:30 geekinut1h joined #fuel
09:40 sbartel hello guys
09:40 sbartel I have a problem when trying to bootsrap  HP Proliant DL360e
09:41 sbartel the servers are seen with no disk
09:42 sbartel and in the centos installed during bootstrap  there is no disk mounted (fdisk -l return is empty)
09:42 sbartel any one have an idea of what can be the problem
09:47 adanin joined #fuel
09:49 mihgen sbartel: hi
09:50 mihgen vkozhukalov and MiroslavAnashkin are experts in this
09:50 mihgen vkozhukalov seems to be not in this channel at the moment, MiroslavAnashkin is idle
09:50 sikor_sxe sbartel: sounds pretty much like a missing raid-driver issue
09:51 mihgen sikor_sxe: yep, that's my guess too
09:51 sikor_sxe can u boot those with vanilla ubuntu 14.04 or centos 6.5?
09:51 vkozhukalov joined #fuel
09:51 mihgen sbartel: what version of fuel are you using?
09:51 sikor_sxe when i had to mess with this kind of stuff, i remember i had to inject drivers into the images
09:52 sikor_sxe or use a custom driver-boot cd before the actual os installation
09:53 sbartel fuel 5.0
09:53 vkozhukalov mihgen: that is a good idea to try to boot with vanilla image, but my suggestion is to try vanilla centos first
09:53 sbartel seems quiet similar to this http://h30499.www3.hp.com/t5/General/CentOS-6-3-unable-to-install-in-DL360e-Gen8-Server/td-p/5887823#.U99V7_l_uDs
09:54 sbartel for which HP provide corrected driver
09:54 vkozhukalov it is certainly either driver compatibility lack or wrong raid configuration
09:54 sbartel so I guess I will have to inject drivers into the image to correct the problem
09:55 sikor_sxe i was actually quite amazed that fuel deployment did work on dell rack servers w/o any raid hazzle
09:55 sikor_sxe it used to be always a mess a few years ago
09:55 sikor_sxe any corporate raid hw would not work with RHEL/SLES out of the box
09:56 sikor_sxe not even considering ubuntu
09:56 mihgen sbartel: when you get a bit more info, please don't forget to file a bug about that to launchpad.net/fuel
09:56 mihgen sbartel: I'd like to get this fixed..
09:57 sbartel ok no problem
09:57 sbartel in order to inject driver into the centos image, is there an how to somewhere?
09:59 vkozhukalov that is a pretty standard procedure, lot of how to on the Internet, briefly, you need to unpack bootstrap image, put drivers into /lib/modules, reindex them and pack image again
10:02 sbartel ok thanks
10:04 sikor_sxe when i set up a local dns for internal nslookups i noticed that openstack instances do renew their dhcp-leases from the subnet every 45s. is this supposed to be that way or is there some misconfiguration involved?
10:06 decede sikor_sxe: I think thats the standard setup in openstack
10:14 ebartz left #fuel
10:19 Longgeek joined #fuel
10:21 sikor_sxe ah great
10:36 lordd joined #fuel
10:37 lordd hi guys
10:37 lordd why does Fuel has an option for start_guests_on_host_boot
10:37 lordd instead of resume_guests_state_on_host_boot
10:38 lordd I would rather have my users have their machine state restored than just started
10:39 lordd after all, if a machine is paused or stopped, there might be a good reason behind it
10:39 mihgen lordd: sounds like a bug.. could you please file one to launchpad?
10:40 lordd mihgen: Thanks, although I wouldn't really consider it fully a bug, rather an improvement over the actual functionality
10:41 lordd mihgen: Will try to open a ticket, hopefully won't get an "invalid" :)
10:41 mattymo it's a bug if it contains behavior a user wouldn't expect
10:41 mihgen lordd: yep.. but launchpad doesn't have tasks, and it is too small for blueprint, so the only option is to file bug )
10:41 bookwar joined #fuel
10:41 mihgen lordd: ping me if it ever gets invalid)
10:41 lordd mattymo: Actually it does what the user expects, but there is a better option
10:42 lordd ok, thanks both!
10:42 lordd every time a bug gets a "confirmed", I listen to the portal song
10:45 Longgeek joined #fuel
11:04 ebartz joined #fuel
11:05 ebartz Is there any information, when the 5.0.1 version will be available as iso-download?
11:24 Longgeek joined #fuel
11:48 TVR_ what release did fuel have the option for VCENTER added to it?
11:48 TVR_ that would be 4.1.1... right?
11:49 TVR_ NOT 4.1... as in, that option did NOT exist in 4.1 when building a new openstack environment... right?
11:50 TVR_ (under the third option under compute.. choices are KVM, QEMU and VCENTER)
11:51 pasquier-s joined #fuel
11:53 TVR_ The reason I ask is I am getting "Your OpenStack environment has been created, but configuration has failed. You can configure it manually" as my error, but manual configuration is not possible as half the networking options are gone...
11:53 Longgeek joined #fuel
12:09 teknomega joined #fuel
12:12 Longgeek_ joined #fuel
12:13 bookwar1 joined #fuel
12:13 christop1eraedo joined #fuel
12:14 gleam_ joined #fuel
12:23 mattymo joined #fuel
12:45 brain461 joined #fuel
12:48 lordd joined #fuel
12:49 rmoe joined #fuel
12:54 Longgeek joined #fuel
12:59 ebartz1 joined #fuel
13:00 MiroslavAnashkin sbartel: Hi, Sam! Please do the following check: Boot one node to the bootstrap and run `ohai > /tmp/ohai.txt`. Then, please share its output. Basically, I need to know your RAID controller model, the value of "Removable" bit and if it is fake RAID or true hardware RAID.
13:05 andriikolesnikov joined #fuel
13:15 atze joined #fuel
13:16 Guest862 Hi there, is there any documentation available for setting software raid on nodes? Or is hardware raid the only way to go
13:20 ebartz joined #fuel
13:26 lordd Hi Guest24601: Check out the architecture reference here:
13:27 lordd http://docs.mirantis.com/openstack/fuel/fuel-5.0/reference-architecture.html#nics-aggregation
13:27 lordd ah, sorry, wrong question
13:27 lordd wrong channel
13:27 KodiakF lordd:  actually that helped me haha
13:27 lordd :P
13:27 KodiakF I was preparing a list of questions just now for a Mirantis rep
13:28 KodiakF that probably answers one  of them
13:28 lordd ok, glad to help, even by mistake ;)
13:28 lordd Bonding is pretty neat as you can config it through the UI
13:29 KodiakF Now the devs just need to allow iscsi and nfs mount setups through the UI
13:29 lordd haha, that would be nice indeed
13:29 KodiakF (so I can get some block space onto my tiny blades)
13:29 KodiakF they have lots of cores and ram but 60GB of block isn't going to do anything
13:29 lordd well, you can still modify the puppet scripts on Fuel to do that chores
13:30 lordd *those
13:30 KodiakF yea but I'm not a whiz
13:30 KodiakF which is why Fuel seems nice
13:30 KodiakF I can manually do those things on a host no problem
13:30 KodiakF but making Puppet do those things is another story for me haha
13:30 lordd then board the jolly puppet boat, they have a very nice tutorial
13:30 lordd on vms
13:30 KodiakF my Puppet modules in the enterprise are middle-school level
13:31 lordd http://puppetlabs.com/blog/quest-learn-puppet-new-learning-vm
13:31 KodiakF Put a file, turn on a service, install a package etc etc
13:31 KodiakF that's what I can do in puppet
13:31 KodiakF Making it do iscsiadm stuff is a whole other story
13:31 lordd oh, ok, that might be then a bit too basic + is enterprise puppet
13:31 e0ne joined #fuel
13:31 lordd I simply liked their RPG approach to learn puppet
13:32 lordd Then get somebody to hack that for you in exchange of X
13:32 dpyzhov joined #fuel
13:33 lordd Guest862: Do you mean RAID for the storage nodes or the controllers/computes/etc
13:35 lordd offtopic: what does low-hanging-fruit means?
13:36 KodiakF the easy tasks in a larger subset of stuff that needs to be done
13:36 lordd ah, sounds like I found out how to get my summit tickets :)
13:36 KodiakF it's easier to grab the "low hanging fruit" on a tree than to get a ladder out of the shed and get the higher stuff
13:36 Guest862 raid on controller and compute
13:39 lordd Guest862: AFAIK you can do it manually by using mdadm or Disk Druid
13:40 lordd if you go with mdadm, then use LVM
13:40 lordd does that makes sense?
13:40 Guest862 yes, althous this would be a alternative solution
13:41 Guest862 although..
13:41 Guest862 would this be supported?
13:42 Guest862 (by mirantis support)
13:43 lordd if you have a raid controller, then there is a blueprint for it:
13:43 lordd https://blueprints.launchpad.net/fuel/+spec/lsi-raid-controllers
13:44 lordd there is also an another blueprint to get an improvement on how disk partitions work:
13:44 lordd https://blueprints.launchpad.net/fuel/+spec/improve-fuel-default-disk-partition-scheme
13:45 lordd but that last one is still not approved
13:45 Guest862 we don't hav a raid controller (yet)
13:45 lordd finally, you can edit the puppet scripts or get somebody (i.e. Mirantis or yourself) to customize it
13:47 lordd I think that for the moment, the next big thing would be the improvement on networking, ml2 and SDN stuff
13:47 Guest862 there is a raid reverence in the relnotes of 5.0
13:47 Guest862 http://docs.mirantis.com/openstack/fuel/fuel-5.0/release-notes.html
13:48 Guest862 it suggests raid is available.. but can't find in anyware
13:48 ebartz joined #fuel
13:48 Guest862 (search for raid-1)
13:49 lordd RAID-1 spans all configured disks on a node?
13:49 Guest862 yes
13:52 lordd hmm, I'm checking this: https://github.com/stackforge/fuel-library/blob/f46ccc151185fba1ddadd317b46d8c3741b2d78c/deployment/puppet/cobbler/templates/scripts/pmanager.py#L356
13:56 ebartz left #fuel
13:57 lordd so far, there is no mdadm config in my environments
13:57 Guest862 ok
13:57 Guest862 it seems there is some raid support, but notyet  fully implemented
13:58 lordd I think the mdadm support comes from cobbler
13:59 lordd check with Ryan Moe
14:01 lordd I mean, rmoe
14:01 Guest862 found him ;)
14:07 TVR_ what's the place we upload pics or files to here?
14:08 mattgriffin joined #fuel
14:08 lordd http://paste.openstack.org/
14:08 lordd pictures, I don't know
14:19 lordd Guest862: About Raid, I confirm you that, if you use it without configuring the provisioning data, it will automatically use Raid-1 on all drives
14:19 lordd so you will see a md0
14:20 lordd then again, this will work as a boot disk, so it has a specific space
14:20 lordd around 200mb
14:21 lordd a good question would be if this can be used for other disks as well
14:21 Guest862 using multiple disks for os fails in combination with ubuntu
14:22 Guest862 but thnx for the info!
14:22 Guest862 need to go!
14:28 angdraug joined #fuel
15:19 blahRus joined #fuel
15:49 xarses joined #fuel
15:57 rmoe joined #fuel
16:03 Nuc__ joined #fuel
16:21 jpf_ joined #fuel
16:22 vkozhukalov left #fuel
16:32 adanin joined #fuel
16:49 e0ne joined #fuel
17:07 angdraug joined #fuel
17:18 mutex joined #fuel
17:30 andriikolesnikov joined #fuel
17:34 e0ne joined #fuel
17:35 adanin joined #fuel
17:52 e0ne joined #fuel
18:21 tatyana joined #fuel
19:28 e0ne joined #fuel
19:51 teknomega is 5.0.1 out or 5.1 out.. or comming out soon ?
19:58 kupo24z teknomega: https://bugs.launchpad.net/mos/+bug/1341656 i think is whats stopping it
20:00 kupo24z & https://bugs.launchpad.net/mos/+bug/1340711
20:05 teknomega thanks
20:27 angdraug yep, ^ and fallout from fixes for them is what's holding 5.0.1 back
20:28 angdraug 5.1 has more stuff holding it back, e.g. patching OpenStack is still more flaky than we'd like
20:42 adanin joined #fuel
20:51 teknoprep any else here have the error in 5.0 that when installing multiple HA Controllers from the beggeinging
20:51 teknoprep it bombs out almost right away
21:16 angdraug that's a pretty generic bug description :)
21:17 angdraug is it 5.0 GA or a 5.0.1 pre-release build?
21:17 angdraug what's the error it's bombing with?
21:21 e0ne joined #fuel
21:31 e0ne joined #fuel
21:45 xarses_ joined #fuel
23:19 shane__ joined #fuel
23:24 KodiakF1 joined #fuel
23:24 shane__ where does one download the upgrade tarball for 5.0.1? not seeing a link anywhere on software.mirantis.com as stated in the docs
23:29 kupo24z shane__: afaik the general release of 5.0.1 isnt out yet
23:31 shane__ ok, its mentioned in the documenation with instructions
23:32 shane__ is there a workaround for getting lbaas installed without waiting on 5.1?
23:32 shane__ or is it possible to upgrade nodes by pointing them at th 5.1 stable repositories?
23:36 adanin joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary