Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2016-05-09

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 fatdragon joined #fuel
00:16 natarej joined #fuel
00:17 rongze joined #fuel
00:45 rongze joined #fuel
01:38 rongze joined #fuel
01:52 fatdragon joined #fuel
02:01 rongze joined #fuel
02:20 rongze joined #fuel
02:34 rongze joined #fuel
02:58 rongze joined #fuel
03:14 rongze joined #fuel
03:19 krobzaur joined #fuel
03:25 krobzaur joined #fuel
03:31 rongze joined #fuel
03:32 rongze_ joined #fuel
03:36 rongze joined #fuel
04:37 fedexo joined #fuel
05:32 rongze joined #fuel
05:34 kszukielojc joined #fuel
06:11 rongze joined #fuel
06:18 javeriak joined #fuel
07:00 rongze joined #fuel
07:05 rongze joined #fuel
07:15 javeriak joined #fuel
07:16 rongze joined #fuel
07:17 javeriak joined #fuel
07:46 rongze joined #fuel
07:49 fatdragon joined #fuel
08:02 rongze_ joined #fuel
08:06 Miouge joined #fuel
08:33 natarej joined #fuel
08:38 yassine joined #fuel
08:53 noshankus joined #fuel
08:59 rongze joined #fuel
09:06 sc-rm joined #fuel
09:08 noshankus Hi guys, running into problems during installation of fuel-community-9.0.iso - I always seem to get the error: "ERR anaconda: Error code 127 running the kickstart script at line 1" - this is during the "kickstart %%post script(s)"
09:09 noshankus Anyone seen anything similar or know where I can start to troubleshoot the actual script? I have access via the shell whihc is there from the installation
09:09 sc-rm When uploading images to glance, through the cli or browser, the uploading is making the traffic to the nodes become slow or unresponsive, what might be the stuff I need to look at? The controllers are dual Xeon quad cores and 32Gb of ram with dual 1Gbit networking, looking at iptraff reveals that the traffic is not going above 100Mbit and CPU load looks low compared to normal, so I’m kind of lost, what to do?
09:26 elo joined #fuel
09:52 artem_panchenko joined #fuel
10:03 neilus joined #fuel
10:33 javeriak joined #fuel
10:34 javeriak joined #fuel
10:44 rongze joined #fuel
10:58 noshankus Hi guys - looks like installation is failing due to missing "/sbin/grub" command - according to "/mnt/sysimage/root/anaconda-post-partition.log"
10:58 noshankus This is specified in line 269 in ks.cfg
11:00 noshankus On image, there are "/sbin/grub2-[...]" but no "/sbin/grub"
11:01 noshankus Any ideas? I guess this is a bug...
11:08 noshankus Has no-one else come across this during installation?
11:10 yassine joined #fuel
11:14 rongze joined #fuel
11:30 noshankus joined #fuel
11:38 LGee noshankus: I guess I'd open a bug report then
11:38 kishorebjv joined #fuel
11:40 noshankus LGee - I guess... just think it's strange that it hasn't been noticed before...especially as this is still in the 10.0 builds... and I thought it would affect ALL installs from the 9.0 iso
12:13 LGee I'm still on 7.0 here due to internal reasons :P
12:14 kodokuu joined #fuel
12:40 neilus joined #fuel
12:43 book` joined #fuel
12:49 srwilkers joined #fuel
13:06 v1k0d3n joined #fuel
13:26 pbelamge joined #fuel
13:39 rongze joined #fuel
13:48 srwilkers joined #fuel
13:51 fatdragon joined #fuel
14:04 krypto joined #fuel
14:06 gardlt joined #fuel
14:10 thiagolib joined #fuel
14:17 noshankus Hi guys, how can I increase the number of releases in fuel? I.e. I'd like to try Mitaka on CentOS7. I currently only have two Ubuntu and one CentOS6.5 which is offline
14:22 gomarivera joined #fuel
14:23 mwhahaha noshankus: there's a bunch of work that'll have to go into enabling CentOS7 as we don't have a way to build a centos7 image for deployment at the moment. You can check with #fuel-dev if you have questions on how to start that work
14:25 noshankus Ah cool, I see, so it's not there yet - I might just do that once I get this up and running - thanks mwhahaha
14:25 mwhahaha i know we want to get centos7/RDO support in, but we need help with it :)
14:45 srwilkers joined #fuel
14:46 natarej joined #fuel
14:47 MikeG451 joined #fuel
14:51 gomarivera joined #fuel
14:52 dancn joined #fuel
15:01 fatdragon joined #fuel
15:05 srwilkers joined #fuel
15:15 noshankus joined #fuel
15:28 elo joined #fuel
15:32 ericjwolf joined #fuel
15:34 ericjwolf Good morning.  I have a Fuel 8.0 install (Ubuntu/Liberty).  I am in the middle of a transistion from Fuel 6.1 to 8.0 so only 1 controller is configured.  I have seen an issue 2 times now where crm shows all the processes as stopped.  I looked in the log files and not finding anything useful.  If I reboot the server everything comes back up.  Can I get a pointer on where to look first to try and figure out what is going on?
15:34 ericjwolf http://paste.openstack.org/show/496482/
15:36 mwhahaha ericjwolf: if the public gateway goes unpingable it'll shut everything dow
15:36 mwhahaha down
15:36 mwhahaha Clone Set: clone_ping_vip__public [ping_vip__public]
15:36 mwhahaha Stopped: [ srvrhp601.hitachi-cta.com ]
15:36 mwhahaha also check disk space, that'll also do it
15:37 ericjwolf Disk space is all good:
15:37 ericjwolf root@srvrhp601:/var/log# df -k Filesystem                     1K-blocks     Used Available Use% Mounted on udev                            65987628       12  65987616   1% /dev tmpfs                           13199804     1324  13198480   1% /run /dev/dm-5                      116690928  2870568 107869752   3% / none                                   4        0         4   0% /sys/fs/cgroup none                                5120
15:37 javeriak joined #fuel
15:37 mwhahaha yea so it's likely the public gateway going unavailable
15:37 ericjwolf I can ping the gateway now.
15:37 mwhahaha check it from the namespace
15:38 ericjwolf short of rebooting the sever, what is the best way to recover?
15:38 mwhahaha i think in 8 we might have exposed a way to disable that check
15:38 mwhahaha at install time
15:38 mwhahaha let me go l ook
15:40 javeriak_ joined #fuel
15:40 mwhahaha https://github.com/openstack/fuel-library/blob/stable/8.0/deployment/puppet/osnailyfacter/modular/virtual_ips/public_vip_ping.pp
15:41 mwhahaha prior to deployment if you set run_ping_checker to false for the node it won't setup the public  gateway ping
15:42 ericjwolf If I look at the Namespaces what should be there?
15:42 mwhahaha you could remove the ping_vip__public resource
15:42 mwhahaha it needs to be able to ping the gateway
15:42 mwhahaha oh what namespaces?
15:42 ericjwolf http://paste.openstack.org/show/496484/
15:43 mwhahaha there should be a vrouter, haproxy, qhdp-<uuid> qrouter-<uuid>
15:44 mwhahaha if it's working for a while then stops, seems like an arp issue or something
15:45 ericjwolf The vrouter and haproxy Ns are there but no qrouter or qdhcp.  I only have the default public network.  We are not using the neutron L3 Agent for our app, we use the networking directly from the computes.
15:45 mwhahaha ok so would make sense you don't have the q* stuff
15:46 ericjwolf I will have to research why the public Ip is failing.  Is there a threashold that can be set (like 4 or 5 or 10 pings?)
15:46 mwhahaha yea i'm trying to see
15:46 ericjwolf Is there an easy way to recover (aside from rebooting the server)?
15:48 mwhahaha it should auto come up when the gateway goes pingable again, was there anything in the pacemaker logs?
15:50 ericjwolf Trying to find when if went down.
15:50 ericjwolf some time between Friday and Now.
15:51 krypto_ joined #fuel
15:54 srwilkers joined #fuel
15:57 ericjwolf About the time the pacemaker logs staart shoing the resources goign down I see a few logs:
15:57 ericjwolf May 07 02:55:11 [10162] srvrhp601.hitachi-cta.com      attrd:   notice: attrd_trigger_update:   Sending flush op to all hosts for: #health_disk (red)
15:58 ericjwolf May 07 02:55:11 [10165] srvrhp601.hitachi-cta.com    pengine:     info: native_color:   Resource p_haproxy:0 cannot run anywhere
15:58 mwhahaha so it was the disk
15:58 ericjwolf A lot more logs.
15:59 mwhahaha so once the disk fills it shuts everything down to prevent corruption of rabbit or mysql
15:59 mwhahaha the threshold is 95%
15:59 ericjwolf The disk is no where full..
15:59 mwhahaha it may not be now, but do you have trends of all the paritions?
16:00 mwhahaha there was an issue where large glance uploads via horizon could cause it to get triggered
16:00 mwhahaha but i thought we fixed it, but it might not have been fixed in 8
16:00 ericjwolf Typically we import images via glance cli, using a rmote http server.
16:01 krypto joined #fuel
16:01 ericjwolf seldom is horizon used.
16:01 mwhahaha crm node status-attr <hostname> delete "#health_disk"
16:01 mwhahaha that's the command to tell pacemake it's been dealt with
16:01 mwhahaha which should cause services to get restored
16:03 ericjwolf that did fix it.
16:04 ericjwolf I will investigate more.
16:04 mwhahaha your partition layout looked weird
16:04 mwhahaha do you customize them to just be a single parittion?
16:07 neilus joined #fuel
16:17 srwilkers joined #fuel
16:26 krypto joined #fuel
16:30 krypto_ joined #fuel
17:02 javeriak joined #fuel
17:15 srwilkers joined #fuel
17:34 MikeG451 joined #fuel
17:50 elopez joined #fuel
17:53 HeOS joined #fuel
18:11 ericjwolf joined #fuel
18:12 ericjwolf @mwhahaha - I do modify the default sizing but I use the default layouts.
18:12 mwhahaha hmm ok what you pasted seemed to only have /
18:13 mwhahaha maybe formatting
18:13 mwhahaha i assume that perhaps the logs are getting filled up then logrotate kicks in
18:13 ericjwolf yeah, the cut and paste in the webchat is terrible,
18:13 ericjwolf I did turn on some verbose logging.
18:13 ericjwolf Look at http://paste.openstack.org/show/496484/
18:14 ericjwolf The output is better formated.
18:14 mwhahaha yea, /dev/mapper/logs-log            10190136  6771924   2877540  71% /var/log
18:14 mwhahaha bet that's filling
18:14 mwhahaha which would explain why 2 days later you don't see anything because the log rotate triggered a few more times clearing out space
18:15 ericjwolf hummm....  Should that be a bit more robust?  The entire system will shutdown if a log file is fill?
18:15 ericjwolf full
18:15 mwhahaha well if you had 3 controllers and only one fills its disks you don't get a complete shutdown
18:15 mwhahaha it only stops the one node
18:16 ericjwolf understood....
18:16 ericjwolf what is the alarm clear threshold?
18:16 mwhahaha there is none for that
18:16 ericjwolf so the disk is back to 71% and the alarm did nto clear
18:17 mwhahaha you have to do something
18:17 mwhahaha basically the node goes red and the operator needs to respond
18:18 ericjwolf Is this a pacemaker thing or a Mirantis thing?
18:18 mwhahaha pacemaker
18:18 mwhahaha we just configure it
18:18 ericjwolf note I am a Telco guy...  Nodes are suppose to self heal :)
18:18 mwhahaha previously we had people filling their mysql disk or rabbitmq and crashing an entire cluster beause it corrupted
18:18 mwhahaha as a telco guy, you should have proper monitoring in place :D
18:18 ericjwolf I did have that issue back at the first install of 6.1
18:19 ericjwolf fair enough..  You got me
18:19 mwhahaha we added it in 7 i think
18:19 mwhahaha https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/manifests/cluster/health.pp#L10
18:19 mwhahaha that's what we exclude from monitoring
18:20 mwhahaha so if horizon gets filled (the previously mentioned glance thing) or /boot it won't stop it
18:20 mwhahaha but anything else will cause the node to shutdown services to prevent possible corruptions
18:20 mwhahaha the logs i could see being excluded but you should have more than a 10g log partition
18:20 mwhahaha imho
18:21 ericjwolf Does fuel use a percentage of the available disk to create the partions?  By default the Os only got 50g total.  I increased it to 120G.
18:22 mwhahaha for some partitions yea
18:22 mwhahaha let me see if i can go find the rules
18:23 ericjwolf I completely missed that in fuel
18:23 ericjwolf there is a separate log option.
18:23 ericjwolf I used the default.
18:23 mwhahaha https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/fixtures/openstack.yaml#L561-L721
18:24 mwhahaha by default we use 10g
18:24 mwhahaha as a minimum
18:25 krypto_ joined #fuel
18:26 ericjwolf 50G big enough?
18:27 ericjwolf I will re-install the controller when I migrate my other machines over to this cluster.
18:27 mwhahaha it all depends on how many nodes you plan on deploying
18:28 ericjwolf I am going to turn off verbose loggin.  So that should help.  Was debugging a nova issue.
18:28 mwhahaha 50 would be an improvement
18:28 gardlt joined #fuel
18:28 ericjwolf which I have a second question.  So last week we were talking about the CPU Pinning feature and it is not in your release.  Do you know if you guys picked up the multi-queue support?
18:29 ericjwolf These 2 features we did development on our app to utilize them.
18:29 mwhahaha when was it added upstream?
18:30 ericjwolf I will have to check..
18:31 mwhahaha doesn't look like we currently expose a way to configure it in configs, doesn't mean it's not there in the code however
18:32 ericjwolf I had planned to take the fuel admin course you guys developed but they cancelled it on me.  Do you know if this is a good class?
18:35 mwhahaha seems to be pretty informative
18:35 mwhahaha i have no idea what's actually in it, but the syllabus has some interesting/useful stuff in it
18:37 gardlt joined #fuel
18:37 ericjwolf Thanks again for your help.  I will make note of all these changes that are required and resolve the issues.
18:38 ericjwolf I will see if I can find when the multi-queue was dropped in.
18:38 ericjwolf Will there be a 8.1 to update any of the Liberty features/patches?
18:39 mwhahaha there are maintenance updates but no 8.1
18:49 neophy joined #fuel
18:56 natarej joined #fuel
19:45 jobewan joined #fuel
20:17 javeriak_ joined #fuel
20:54 eric_lopez joined #fuel
21:13 ericlopez joined #fuel
22:02 njhtran joined #fuel
22:03 njhtran Hey all.  i'm trying to install fuel9 on a kvm vm, after the grub menu 'install fuel' it loads initram and then hangs
22:03 njhtran anyone run into this before and know if there's a workaround?
22:13 MikeG451 joined #fuel
22:13 Reepicheep joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary