Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2016-03-17

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 tzn joined #fuel
00:07 MorAle joined #fuel
00:08 izinovik joined #fuel
00:09 MiroslavAnashkin joined #fuel
00:10 akislitsky joined #fuel
00:11 meow-nofer joined #fuel
00:12 sbog joined #fuel
00:13 vkramskikh joined #fuel
00:14 mihgen joined #fuel
00:15 apalkina joined #fuel
00:41 skamithi13 joined #fuel
00:44 skamithi13 joined #fuel
00:47 skamithi14 joined #fuel
01:16 kur1j joined #fuel
01:18 xarses joined #fuel
01:18 kur1j Hello. I'm struggling to get an Openstack implementation up and running for some reason and it revolves completely around the networking setup
01:21 kur1j I have a router -->192.168.1.0/24 --> Physical host (eth0, eth1, eth2).
01:23 jmhale joined #fuel
01:23 kur1j On the physical host I have CentOS 7.0 that has a VM running Fuel with eth0 as 10.20.0.0/24 and eth1 as 192.168.1.10 (this is so the machine can get out to the internet on VM)
01:25 kur1j I then created another VM and attached eth0 and eth1 to it and let fuel bootload it.
01:26 kur1j I make a new environment
01:26 kur1j add the VM1 that Fuel bootstrapped as controller 1
01:29 kur1j on controller 1 I have eth0 -->Admin (PXE), eth1 --> Public, eth2 --> Management, Storage, Private
01:30 kur1j in my networks setting I switched public to 192.168.1.0/24 and made the gateway 192.168.1.1 and pointed it towards my gateway that has external network connectivity.
01:31 kur1j I then deploy...it deployes...once the controller is deployed the controller VM is 192.168.1.5....I can ping it...but it started up 192.168.1.4 for Horizon...I can't ping 192.168.1.4 from any other machine
01:32 cr0wrx what do you mean any other machine
01:32 kur1j cr0wrx: as in if I plug up a laptop to the same switch that holds 192.168.1.0/24
01:33 cr0wrx hrm, if they are on same switched network I would assume you can hit unless your vlans are wrong
01:33 kur1j well there are no vlans
01:33 kur1j im just using different switches
01:34 cr0wrx sure there are (I'm no expert by the way, just happen to be here)
01:34 kur1j cr0wrx: any help (even talking things out) haha
01:34 cr0wrx fuel defaults to tagging management / public etc with vlans 101, 102, etc
01:35 kur1j well public never has a vlan for me
01:35 kur1j only storage, management and private
01:35 cr0wrx ok
01:36 cr0wrx if you ssh to controller can it ping out to other machines
01:36 kur1j oddly enough yes
01:36 kur1j i can ping the gateway
01:37 cr0wrx if you run tcpdump do you see the incoming ping requests if you ping 192.168.1.4
01:37 kur1j I assume the gateway for the public interface is the router for my network
01:37 kur1j cr0wrx: I didn't think to do that.
01:38 kur1j do you know if not setting the vlans up on the switch (even if its in a VM) cause it to not to work?
01:38 cr0wrx assuming both boxes are on 192.168.1.0/24, same switch without vlan restrictions, and no firewall blocking access, there shouldn't be an issue as far as I'm aware
01:39 kur1j cr0wrx: yeah it is confusing the hell out of me
01:39 cr0wrx uh, it's possible. again, I'm not an expert but I recall when I first deployed with fuel 7 I didn't pass network connectivity check until I created the vlans on switch and tagged the ports
01:39 cr0wrx did you do the network connectivity check before deploy
01:39 kur1j it won't let me do the connectivity check with only 1 node
01:40 cr0wrx ah
01:41 kur1j just to be sure im not crazy...the Public interface gateway isn't the cluster itself correct? it is the gateway to the system providing network connectivity right?
01:42 cr0wrx horizon will be on the public interface
01:42 cr0wrx and floating public IPs you assign to VMs come out of it
01:42 kur1j so for example...I have a pfsense box (internet)---(public)>pfsense--(192.168.1.0/24) (gateway, 192.168.1.1) --> LAN
01:42 cr0wrx I have pfsense too
01:43 cr0wrx But I did it differently
01:43 cr0wrx in pfsense I created a vlan called 'openstack' with vlan id of 100, when I configured networking in openstack I set public network to vlan 100 as well
01:43 cr0wrx and the gateway is 192.168.5.1 (which is the openstack vlan on pfsense), whereas my standard 'lan' on pfsense for normal machines is 192.168.7.1
01:44 cr0wrx 192.168.7.0/24 I mean
01:44 kur1j cr0wrx: don't you have to make these changes on a switch as well?
01:45 cr0wrx I just set up vlans on switch for 100, 101, 102, etc (all the openstack ones) and tagged the ports I wanted it to pass on
01:45 kur1j ahhh yeah, so you do have to make the changes on the switch
01:45 kur1j these idiots I work with are like "its just a setting in pfsense"
01:45 kur1j "
01:45 kur1j "..but you don't need vlans"
01:45 cr0wrx well, I did do the connectivity check, and I can tell you it fails until you tag the ports on the switch
01:46 cr0wrx but again, I'm not an expert :) I'm doing this all at home getting familiar with it still, but I've deployed a few times. Working on fuel 8 now
01:46 kur1j well ive been trying to get this crap working for like 2 days now
01:46 kur1j fuel is hellva time saver from deploying manually
01:46 cr0wrx I think networking has been the hardest part for me
01:46 kur1j OS is a total bitch to configure and install
01:47 kur1j you know more than me :)
01:47 cr0wrx once I got it working the first time, it worked great and I redeployed several times no issues, everything else worked fine for the most part (couple minor things here and there, but not bad considering all the components)
01:47 kur1j lol after you VLAN tagged your switch ;)
01:47 cr0wrx then I swapped out my switches for one with more ports, had to set it up again, am switching to fuel 8, and am having issues again
01:48 kur1j how bad is it to upgrade?
01:48 cr0wrx well I don't think you can upgrade easily unless you do a HA deployment (3 controllers, etc)
01:48 cr0wrx and even then it seems like a longer process, I haven't tried though, just read through some docs
01:49 cr0wrx by upgrading I meant, I am blowing away my old cluster and just deploying fresh
01:49 kur1j ah
01:49 kur1j yeah, that makes sense haha
01:49 kur1j im curious how it will be to redeploy new version of OS if even possible
01:50 cr0wrx from what I read upgrades historically meant just making a second cluster and then migrating stuff over to it...but the past few releases they have been working on upgrading in place so you don't have to double hardware
01:50 kur1j interesting
01:50 cr0wrx but it does require an HA deploy, you essentially take one of the controllers and upgrade it and then start switching stuff over to it or something along those lines
01:50 kur1j ah
01:50 kur1j well on a side not within OS hows the support for GPUs?
01:51 kur1j with PCI Passthrough
01:51 rmoe joined #fuel
01:51 cr0wrx ask that again tomorrow when someone else is around :)
01:51 kur1j lol
01:53 cr0wrx I gotta boogie for now anyways. Best of luck. Try checking vlans on switch and see if that fixes your problem. Sounds like you're almost there if you have node deployed and running
01:53 kur1j yup! thanks for the help!
02:01 [HeOS] joined #fuel
02:20 treaki__ joined #fuel
02:25 sbog joined #fuel
02:25 meow-nofer joined #fuel
02:25 akislitsky joined #fuel
02:25 Captain_Murdoch joined #fuel
02:25 aglarendil_ joined #fuel
02:25 dklenov joined #fuel
02:25 palexster joined #fuel
02:25 ashtokolov joined #fuel
02:25 evgenyl joined #fuel
02:25 mwhahaha joined #fuel
02:25 mescanef joined #fuel
02:48 srwilkers joined #fuel
02:49 srwilkers left #fuel
03:07 gariveradlt joined #fuel
03:29 xarses joined #fuel
04:09 krypto joined #fuel
04:13 Jeffrey4l joined #fuel
04:26 l4ng1t joined #fuel
04:29 treaki_ joined #fuel
04:47 fedexo joined #fuel
05:18 askb joined #fuel
05:45 HeOS joined #fuel
06:08 ArminderSingh joined #fuel
06:08 ArminderSingh can we uupgrade from fuel 6 to 8
06:08 ArminderSingh i can see in docs upgrade from 6 to 7
06:08 ArminderSingh but no upgrade to 8
06:09 ArminderSingh also can't find update package for 8.0 in downloads
06:10 ArminderSingh so if I had to setup a Liberty cloud, I do need to have fuel node from scratch using 8.0 ISO?
06:13 ArminderSingh or is the update in plans or from scratch is the only option?
06:15 skamithi13 joined #fuel
06:26 Obi-Wan joined #fuel
06:32 Jeffrey4l joined #fuel
06:57 akaszuba joined #fuel
07:01 HeOS joined #fuel
07:03 HeOS joined #fuel
07:57 magicboiz joined #fuel
08:05 HeOS joined #fuel
08:21 vsedelnik joined #fuel
08:24 pbelamge joined #fuel
08:31 neouf joined #fuel
08:43 pumphouse_p joined #fuel
08:43 xarses_ joined #fuel
08:45 sc-rm joined #fuel
08:46 sc-rm I have an installment with lma running on one node, after restarting this node, the Grafana dashboard gives : “Template variables could not be initialized: undefined” even after running for 30 min, any ideas?
08:52 pasquier-s sc-rm, which version of lma are you using?
08:54 sc-rm pasquier-s: I’m using 2015.1.0-7.0, but doing “service grafana-server restart” fixed the server problem, now I need to figure why it’s reporting everything as warning, even thought pcs status is reporting everything is running fine and all services in the stack is fine
08:56 pasquier-s sc-rm, ok, Fuel 7.0 with lma 0.7.3 or 0.8.0? I assume the latter probably
08:56 sc-rm pasquier-s: fuel 7.0 with lma 8.0
08:59 sc-rm pasquier-s: Since the dashboard is back, then I have the next problem, the dashboard is reporting no data for everything, which leads me to think that it’s not getting any data from the controllers, compute nodes and ceph nodes
09:08 pasquier-s sc-rm, could you paste the logs from the LMA collector running on the controller(s) to paste.openstack.org?
09:09 sc-rm pasquier-s: Yep, If I know which log you want to see, just digged around and saw that influx is the problem, giving “failed to store statistics: timeout” in the log, so probably related to https://github.com/influxdata/influxdb/issues/4121
09:10 pasquier-s sc-rm, that would be /var/log/lma-collector.log
09:12 sc-rm pasquier-s: http://paste.openstack.org/show/490848/
09:14 pasquier-s sc-rm, just to be sure, you have only 1 controller
09:14 pasquier-s sc-rm, please share /var/log/collectd.log also
09:15 sc-rm pasquier-s: I have 3 controllers and one lma node, then a lot of compute/ceph nodes
09:17 pasquier-s sc-rm, ok so please share /var/log/lma_collector.log and /var/log/collectd.log for all 3 controllers (at least the last lines)
09:17 sc-rm pasquier-s:  http://paste.openstack.org/show/490849/ the same on all 3 controllers
09:18 pasquier-s sc-rm, I would recommend to hard-restart the collectd service on the 3 controllers: kill -9 $(pidof collectd)
09:19 pasquier-s the collectd logs show that collectd fails to send metrics to the lma collector service
09:19 akaszuba joined #fuel
09:19 sc-rm pasquier-s: http://paste.openstack.org/show/490850/ the same for all 3 controllers, this is /var/log/lma_collector.log and the other was /var/log/collectd.log
09:19 sc-rm pasquier-s: http://paste.openstack.org/show/490851/ which one of these process is the relevant to kill?
09:20 sc-rm pasquier-s:  got it killed
09:23 pasquier-s sc-rm, maybe you would need to restart the lma_collector service too
09:23 pasquier-s sc-rm, I mean if restarting the collectd service didn't help
09:25 sc-rm pasquier-s: for now it did not help, so trying that out
09:26 pasquier-s sc-rm, so the steps are:
09:26 pasquier-s 1. stop collectd: /etc/init.d/collectd stop
09:26 pasquier-s 2. check that collectd is effectively stopped
09:27 pasquier-s 3. restart lma_collector: crm resource restart lma_collector
09:27 pasquier-s 4. wait a little bit of time and check the lma_collector logs
09:27 pasquier-s 5. start collectd again
09:29 magicboiz joined #fuel
09:32 samuelBartel joined #fuel
09:34 krobzaur joined #fuel
09:41 bhaskarduvvuri joined #fuel
10:01 sc-rm pasquier-s: now I see, why it might not work, the web interface for influx says : 0 error - Could not connect to https://172.16.4.11:8086
10:02 pasquier-s sc-rm, oh is there any influxd process running on your lma node?
10:02 sc-rm pasquier-s: http://paste.openstack.org/show/490859/
10:03 pasquier-s sc-rm, anything in the logs except the statistics timeout message?
10:05 dklepikov joined #fuel
10:20 sc-rm pasquier-s: nope, I don’t see any errors, just a lot of : “[http] 2016/03/17 10:20:27 192.168.0.4 - lma [17/Mar/2016:10:20:27 +0000] POST /write?db=lma&precision=ms HTTP/1.1 204 0 - Go 1.1 package http de39a2e1-ec29-11e5-8127-000000000000 4.311382ms” on all controllers
10:21 sc-rm pasquier-s: sorry that was the /var/log/influxdb/influxd.log
10:22 tzn joined #fuel
10:23 sc-rm pasquier-s: lma logs are showing no activity on all controllers, but I see I can use kibana, but it’s not using the influxdb I guess
10:24 skamithi13 joined #fuel
10:26 magicboiz joined #fuel
10:28 pasquier-s sc-rm, so influxdb seems to be running fine
10:28 sc-rm pasquier-s: Yep it seems so
10:28 pasquier-s sc-rm, having no logs in /var/log/lma_collector.log is good too btw
10:29 sc-rm is there any cli I could use to check i
10:29 pasquier-s sc-rm, checking influxdb?
10:29 sc-rm pasquier-s: Yep
10:29 sc-rm pasquier-s: show databases or something like that
10:30 pasquier-s sc-rm, the influx command should be available
10:30 pasquier-s sc-rm, check the options to pass the username and password
10:31 sc-rm pasquier-s: also my first thought, but it’s not there, command not found ;-)
10:31 pasquier-s sc-rm, look into /opt/influxdb/bin or /opt/influx/bin
10:34 sc-rm pasquier-s: found it, so now we are here ERR: error authorizing query: lma not authorized to execute statement 'SHOW DATABASES', requires admin privilege
10:35 pasquier-s sc-rm, try 'use lma' and then 'show measurements'
10:36 sc-rm pasquier-s: ERR: unable to parse Basic Auth credentials
10:38 pasquier-s sc-rm, can you double-checked the credentials you've passed to the CLI?
10:38 pasquier-s sc-rm, maybe the 'auth' command is available in this version, not sure.
10:38 sc-rm pasquier-s: did that a couple of times, and they are the same as use in fuel web
10:39 sc-rm pasquier-s: auth is there
10:39 sc-rm pasquier-s: ERR: error authorizing query: lma not authorized to execute statement 'SHOW MEASUREMENTS', requires READ on 'lma'
10:39 pasquier-s sc-rm, can you try with user 'root' and the same password as the 'lma' user?
10:40 sc-rm pasquier-s: then it gives some result
10:42 sc-rm pasquier-s: so I get dbs: _internal and lma
10:43 pasquier-s and if you run 'use lma' and 'show measurements'
10:43 pasquier-s ?
10:48 sc-rm pasquier-s: ERR: database not found: 'lma'
10:50 e0ne joined #fuel
10:51 pasquier-s sc-rm, hmm that is weird
10:51 pasquier-s sc-rm, can you check the disk space on the server (df -h)?
10:53 Jeffrey4l joined #fuel
10:53 sc-rm pasquier-s: df -h shows there is like 160G+ for /opt/influxdb, /opt/es-data / /var/log and seperatly, so like 1Tb in all free space
10:55 bhaskarduvvuri joined #fuel
11:36 skamithi13 joined #fuel
11:38 skamithi13 joined #fuel
11:50 sc-rm pasquier-s: in lma_collector.log : 2016/03/17 11:49:51 Plugin 'elasticsearch_output' error: HTTP response error status: 400 Bad Request
11:52 skamithi14 joined #fuel
11:59 bhaskarduvvuri joined #fuel
12:05 HeOS joined #fuel
12:18 bhaskarduvvuri joined #fuel
12:51 kur1j_ joined #fuel
13:00 kur1j_ hello. So I think I'm misunderstanding something. Im using fuel 8 and am selecting neutron with tunneling segmentation when I create an openstack deployment. In the network configuration it wants me to set a private network vlan along with the public, storage and management networks. However in the documentation I found on https://docs.mirantis.com/openstack/fuel/fuel-8.0/mos-planning-guide.html#plan-the-network doesn't even show the "
13:05 venkat_ joined #fuel
13:06 Kdecherf joined #fuel
13:06 Kdecherf hello world
13:06 Kdecherf is there a way to apply a puppet manifest (basically installs a new package) on an already deployed cluster?
13:36 pbelamge joined #fuel
13:53 tzn joined #fuel
14:00 e0ne joined #fuel
14:01 venkat_ joined #fuel
14:20 sc-rm pasquier-s: Now after running for a while the user lma can do show measurements, so the big question is why does grafana not show any data
14:22 claflico joined #fuel
14:24 pasquier-s sc-rm, I would check the datasources in Grafana and verify that Grafana can connect to InfluxDB
14:24 pasquier-s http://docs.grafana.org/datasources/influxdb/
14:26 sc-rm pasquier-s: That is apparently correct, becase stopping influx makes grafana complain, starting influx makse it only go “no data”
14:27 sc-rm pasquier-s: The biggest problem for me, I’m a mysql guy, I don’t know how to navigate the influxdb and se if there is any kind of data stored anywhere for lets say nova/ceph.
14:28 pasquier-s sc-rm, if you run the influx CLI, I can provide some queries to execute
14:28 sc-rm pasquier-s: Cool :-)
14:29 pasquier-s sc-rm, 'select value from cpu_util where time > now() - 5m' should return something
14:30 sc-rm pasquier-s: no result
14:31 pasquier-s sc-rm, have you restarted the collectd service on the controllers?
14:32 sc-rm pasquier-s: Not after show mesurements started working
14:32 pasquier-s ok so please restart the collectd services
14:35 sc-rm pasquier-s: Now we are getting somewhere :-D Thanks for patience and help
14:38 xarses_ joined #fuel
14:39 sc-rm pasquier-s: Now grafana is reporting again and I just have to figure out, which information is triggering the warn on rabbitmq. Is there a way to see the query grafana is doing for a given web visible field? like the warn part on this image http://snag.gy/ePbEN.jpg
14:42 krobzaur Kdecherf: If you log into the node to which you want to apply the manifest, I think you can just run "puppet_apply $your_manifest" or something of that nature
14:51 pasquier-s sc-rm, if you pass the mouse over the little down arrows (v), you should get the detailed message
14:52 pasquier-s sc-rm, afaict you have too many messages in RabbitMQ that are not consumed
14:52 pasquier-s sc-rm, running "rabbitmqctl list_queues | grep -v ' 0$'" should give you some info
14:57 sc-rm pasquier-s: The number of outstanding messages is too high, then how to deal with that...
14:58 pasquier-s sc-rm, if you run the rabbitmqctl command above, it should at least tell which queues aren't consumed
14:58 sc-rm pasquier-s: It did, and it’s a hell lot of queues ;-)
14:59 pasquier-s sc-rm, can you paste the list?
15:01 sc-rm pasquier-s: http://paste.openstack.org/show/490919/
15:01 sc-rm pasquier-s: So I guess it’s the ones with 1000+ ?
15:02 pasquier-s sc-rm, my rabbitmqctl command wasn't so great, "rabbitmqctl list_queues | grep -v '0$'" should be better
15:03 sc-rm pasquier-s: http://paste.openstack.org/show/490920/
15:03 Kdecherf krobzaur: well, is there anyway to use mcollective to push the change to all nodes?
15:04 pasquier-s sc-rm, I've seen this kind of issue many times but never completely understood it
15:04 sc-rm pasquier-s: The evn is working fine, but still annoying to have it in the logs. Will look more into it tomorrow, have to leave for today
15:05 sc-rm pasquier-s: evn = env
15:05 pasquier-s sc-rm, good
15:08 sc-rm pasquier-s: So thanks for the help :-)
15:21 askb_ joined #fuel
15:27 tatyana joined #fuel
15:33 angdraug joined #fuel
15:34 geekinutah joined #fuel
15:49 blahRus joined #fuel
16:02 gariveradlt joined #fuel
16:03 srwilkers joined #fuel
16:22 pbrooko joined #fuel
16:34 v1k0d3n joined #fuel
17:06 pbelamge joined #fuel
17:08 gariveradlt joined #fuel
17:23 skamithi13 joined #fuel
17:23 askb joined #fuel
17:44 pbelamge joined #fuel
17:47 tzn joined #fuel
17:56 tzn joined #fuel
17:59 v1k0d3n joined #fuel
17:59 tzn joined #fuel
18:04 HeOS joined #fuel
18:06 neilus joined #fuel
18:12 neilus joined #fuel
18:18 e0ne joined #fuel
18:21 krypto joined #fuel
18:59 angdraug joined #fuel
19:20 vsedelnik joined #fuel
19:33 Jeffrey4l_ joined #fuel
19:37 YuryK joined #fuel
19:38 shashurup joined #fuel
19:41 skamithi13 joined #fuel
19:42 skamithi14 joined #fuel
19:47 skamithi13 joined #fuel
20:10 skamithi13 joined #fuel
20:10 skamithi13 joined #fuel
20:11 skamithi13 joined #fuel
20:21 e0ne joined #fuel
21:22 vsedelnik joined #fuel
22:13 Alcest joined #fuel
22:14 pumphouse_p Hey all, what's the preferred approach to enable feature groups on an already deployed Fuel 8 master node? I can do it on firstboot in fuelmenu, but if I miss it at that point, do I go and relaunch fuelmenu? Or can I just edit the /etc/nailgun/settings.yaml in the nailgun container and then restart it?
22:23 vsedelnik joined #fuel
22:25 mihgen pumphouse_p: I didn't try recently, but I think you can just edit settings.yaml and restart container
22:43 BillC joined #fuel
23:03 HeOS joined #fuel
23:20 pumphouse_p mihgen: thanks! I tried and it and it took effect, I was just curious if it was the "right" way to do it.
23:25 vsedelnik joined #fuel
23:51 mihgen pumphouse_p: it was even documented in docs.fuel-infra.org somewhere..
23:51 mihgen not sure if it's still the case
23:53 fedexo joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary