Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2016-03-16

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:59 xarses joined #fuel
01:29 vsedelnik joined #fuel
01:42 n3m8tz joined #fuel
01:44 skamithi13 joined #fuel
01:50 skamithi14 joined #fuel
02:12 skamithi13 joined #fuel
02:22 rongze joined #fuel
02:30 vsedelnik joined #fuel
02:35 rmoe joined #fuel
02:46 l4ng1t joined #fuel
03:04 natarej__ joined #fuel
03:31 vsedelnik joined #fuel
03:56 natarej__ joined #fuel
03:57 vsedelnik joined #fuel
04:00 claflico joined #fuel
04:07 fedexo joined #fuel
04:13 [HeOS] joined #fuel
04:28 natarej joined #fuel
04:30 treaki__ joined #fuel
05:19 rmoe joined #fuel
05:35 rmoe joined #fuel
05:41 HeOS_ joined #fuel
05:42 rmoe joined #fuel
05:52 rmoe joined #fuel
06:05 rmoe joined #fuel
06:20 rmoe joined #fuel
06:31 rmoe joined #fuel
06:41 rmoe joined #fuel
06:53 rmoe joined #fuel
06:56 l4ng1t joined #fuel
07:06 rmoe joined #fuel
07:09 rongze joined #fuel
07:13 rmoe joined #fuel
07:15 HeOS_ joined #fuel
07:17 rmoe joined #fuel
07:21 akaszuba joined #fuel
07:27 rmoe joined #fuel
07:27 pbelamge joined #fuel
07:30 elo joined #fuel
07:34 rmoe joined #fuel
07:39 vsedelnik joined #fuel
07:40 vsedelnik joined #fuel
07:40 rmoe joined #fuel
07:48 rmoe joined #fuel
07:56 rmoe_ joined #fuel
08:06 rmoe joined #fuel
08:14 rmoe joined #fuel
08:20 rmoe joined #fuel
08:20 permalac joined #fuel
08:30 rmoe joined #fuel
08:41 rmoe joined #fuel
08:45 rmoe joined #fuel
08:46 rongze joined #fuel
08:47 cartik joined #fuel
08:49 cartik hi would like to know how are config files pushed to nodes,
08:49 cartik meaning in my env i have two compute node
08:50 cartik and one node is failing with puppet error, on troubleshooting i identified the global.yaml file is missing in one node
08:50 cartik would like to know how are these files being pushed, can someone guide me
08:52 rmoe joined #fuel
08:52 rongze joined #fuel
08:53 Miouge joined #fuel
08:56 neilus joined #fuel
08:59 Miouge joined #fuel
09:02 HeOS joined #fuel
09:02 rmoe joined #fuel
09:03 HeOS joined #fuel
09:06 Miouge joined #fuel
09:08 Chris___ joined #fuel
09:12 kaliya joined #fuel
09:13 rmoe joined #fuel
09:17 Miouge joined #fuel
09:25 rmoe joined #fuel
09:31 tzn joined #fuel
09:36 v1k0d3n joined #fuel
09:41 rmoe joined #fuel
09:46 rmoe joined #fuel
09:47 Miouge joined #fuel
09:54 Miouge joined #fuel
10:05 Miouge joined #fuel
10:07 magicboiz joined #fuel
10:10 rmoe joined #fuel
10:15 rmoe joined #fuel
10:15 Miouge joined #fuel
10:20 rmoe joined #fuel
10:28 rmoe joined #fuel
10:33 Miouge joined #fuel
10:33 rmoe joined #fuel
10:40 rmoe joined #fuel
10:48 kelepirci joined #fuel
10:48 rmoe joined #fuel
10:48 kelepirci hello all
10:48 kelepirci is there anybody that can help me?
10:53 rmoe joined #fuel
10:58 rmoe joined #fuel
11:04 neilus joined #fuel
11:06 rmoe joined #fuel
11:10 Damjanek kelepirci: Please describe your problem
11:15 rmoe joined #fuel
11:15 Miouge joined #fuel
11:19 kaliya joined #fuel
11:21 rmoe joined #fuel
11:27 vsedelnik joined #fuel
11:31 javeriak joined #fuel
11:39 rmoe joined #fuel
11:39 skamithi13 joined #fuel
11:43 rongze joined #fuel
11:46 skamithi14 joined #fuel
11:51 DaveJ__ joined #fuel
11:52 rmoe joined #fuel
11:52 Miouge joined #fuel
11:57 rmoe joined #fuel
12:02 vsedelnik joined #fuel
12:07 rmoe joined #fuel
12:11 rmoe joined #fuel
12:19 rmoe joined #fuel
12:25 vsedelnik joined #fuel
12:30 rmoe joined #fuel
12:31 vsedelnik joined #fuel
12:39 rmoe joined #fuel
12:44 rmoe joined #fuel
12:46 kaliya joined #fuel
12:46 omolchanov joined #fuel
12:53 rmoe joined #fuel
13:00 rmoe joined #fuel
13:02 neilus1 joined #fuel
13:04 neilus joined #fuel
13:05 rmoe joined #fuel
13:10 e0ne joined #fuel
13:11 dancn joined #fuel
13:12 rmoe joined #fuel
13:19 rmoe joined #fuel
13:23 Miouge joined #fuel
13:35 rmoe joined #fuel
13:44 rmoe joined #fuel
13:45 cjj joined #fuel
13:47 cjj after removing old nodes from an environment installed with fule 8.0 i can no longer pxe boot them to assign to a new environment, the pce image loads and then i have the error cannot get disk parameters
13:47 cjj if i reinstall the fuel-master node the slaves can be booted from PXE again
13:47 cjj any takers?
13:48 neilus1 joined #fuel
13:51 srwilkers_ joined #fuel
13:56 rmoe joined #fuel
13:59 l4ng1t joined #fuel
14:03 gariveradlt joined #fuel
14:05 rmoe joined #fuel
14:06 HeOS joined #fuel
14:10 rmoe joined #fuel
14:13 rongze_ joined #fuel
14:14 neilus joined #fuel
14:19 xarses joined #fuel
14:24 rmoe joined #fuel
14:31 rmoe joined #fuel
14:38 rongze joined #fuel
14:45 rmoe joined #fuel
14:46 dslevin joined #fuel
14:47 jcook_ joined #fuel
14:51 rmoe joined #fuel
14:56 vsedelnik joined #fuel
14:58 rmoe joined #fuel
15:00 magicboiz joined #fuel
15:14 krypto joined #fuel
15:21 rmoe_ joined #fuel
15:22 Miouge joined #fuel
15:30 magicboiz joined #fuel
15:32 vsedelnik joined #fuel
15:47 blahRus joined #fuel
15:57 krypto using 6.1 i have 3 controller+network nodes,can i add 2 more controller+network nodes is it supported to have even controller nodes.i see "(/Stage[main]/Openstack::Network/Exec[waiting-for-neutron-api]/returns) change from notrun to 0 failed: bash -c "neutron net-list --http-timeout=4 " 2>&1 > /dev/null returned 1 instead of one of [0]"
15:59 ohyeahhuh five controllers? how large is your env? im just curious.
16:01 e0ne joined #fuel
16:01 krypto 260 compute nodes and 3 controllers
16:01 * ohyeahhuh gulps
16:01 ohyeahhuh neat
16:01 krypto ahh not 260, 180+
16:02 krypto planning to add 80 more
16:02 * ohyeahhuh has six
16:03 krypto i guess this error has nothing to do with number of nodes "neutron net-list --http-timeout=4 " 2>&1 > /dev/null returned 1 instead of one of [0]"
16:03 ohyeahhuh yeah i dont think so either
16:03 skamithi13 joined #fuel
16:03 ohyeahhuh it looks like the list is not returned in time
16:05 xarses krypto: doe neutron net-list take longer than 4 sec to finish?
16:05 xarses you should always have an odd number of controllers
16:05 xarses even is bad for quorum
16:06 krypto xarses right now its 3.28 s
16:06 xarses so when it's doing that, it also just kicked the server
16:06 krypto any idea how to increase this timeout to a larger value
16:06 xarses so either double the timeout, or add more retries, possibly both
16:07 xarses grep for 'waiting-for-neutron-api' in /etc/puppet/manifests/ on the fuel node
16:07 xarses I don't recall where that was in 6.1
16:09 l4ng1t1 joined #fuel
16:10 n3m8tz joined #fuel
16:10 geekinutah joined #fuel
16:10 l4ng1t joined #fuel
16:11 tzn joined #fuel
16:12 xarses krogon: https://github.com/openstack/fuel-library/blob/stable/6.1/deployment/puppet/openstack/manifests/network.pp#L226-l238
16:12 xarses sorry, meant krypto
16:20 elopez joined #fuel
16:23 n3m8tz joined #fuel
16:36 vsedelnik joined #fuel
16:41 severion joined #fuel
16:42 vsedelnik joined #fuel
16:47 e0ne_ joined #fuel
16:50 e0ne joined #fuel
17:00 e0ne joined #fuel
17:02 venkat_ joined #fuel
17:02 HeOS joined #fuel
17:26 cr0wrx joined #fuel
17:26 krobzaur joined #fuel
17:27 cr0wrx anyone know where I can find more docs around fuel cli task commands and where additional logs may be? network connectivity check hanging and fuel task --list shows
17:27 cr0wrx id | status  | name                               | cluster | progress | uuid
17:27 cr0wrx ---|---------|------------------------------------|---------|----------|-------------------------------------
17:27 cr0wrx 7  | running | verify_networks                    | 1       | 0        | afd75ed0-b64d-4922-8c1b-f42528d0964d
17:27 cr0wrx 8  | error   | check_dhcp                         | 1       | 100      | c6e24da3-eb0c-4fa0-ab6a-f4cf555f43be
17:27 cr0wrx 9  | running | check_repo_availability            | 1       | 0        | e0bced42-e1ec-4887-84c7-ca708f022036
17:27 cr0wrx 10 | running | check_repo_availability_with_setup | 1       | 0        | 6b8f32af-f966-4788-a6c8-61ba053a2449
17:28 cr0wrx seems like check_dhcp hit 100%, error somehow, and now things are hung
17:28 mwhahaha i would start by checking mcollective logs on nodes (or in /var/log/docker-logs/remote/*/bootstrap on the master)
17:32 cr0wrx Main errors I see in those are
17:32 cr0wrx 2016-03-15T18:45:56.307967+00:00 err: 18:45:55.936710 #1774] ERROR -- : agent.rb:111:in `rescue in handlemsg' net_probe#check_url_retrieval failed: #<Class:0x007f5c54026fe0>: execution expired
17:32 cr0wrx 2016-03-15T18:45:56.307967+00:00 err: 18:45:55.936799 #1774] ERROR -- : agent.rb:112:in `rescue in handlemsg' /usr/lib/ruby/vendor_ruby/systemu.rb:93:in `read'
17:32 cr0wrx I'm not sure if they are current or not though
17:34 cr0wrx The only other error I see on one of the interfaces is
17:34 cr0wrx 2016-03-16T17:27:14.543351+00:00 err: 17:27:14.131707 #1819] ERROR -- : rabbitmq.rb:30:in `on_miscerr' Unexpected error on connection stomp://mcollective@192.168.7.47:61613: es_recv: connection.receive returning EOF as nil - resetting connection.
17:46 cr0wrx so I killed the error task, but if I try to delete a running task it says I can't. I'm guessing maybe I need to force it, is that the case? Will there be any impact? The running tasks (3 of them) are all at progress 0 and not changing
17:57 bhaskarduvvuri joined #fuel
17:59 magicboiz joined #fuel
18:02 krypto thanks xarses i will try that
18:07 geekinutah joined #fuel
18:11 krypto also any body know why master router namespace becomes backup  when a new network node comes back online in neutron with VRRP
18:13 skamithi14 joined #fuel
18:26 elopez joined #fuel
18:38 Miouge joined #fuel
19:14 Captain_Murdoch joined #fuel
19:14 cr0wrx digging a bit more into the check_dhcp, the errors I'm getting back in /var/log/docker-logs/nailgun all state 'node <name> discovered DHCP servir via eno2 with IP <fuel master ip>. This will conflict with the installation'
19:15 cr0wrx which brings up a couple questions - why would it conflict? The IP it discovers is the fuel master IP, and fuel is running DHCP...That's how it's supposed to be is my understanding.
19:15 cr0wrx and the second would be why it hangs the network connectivity check, but less important at this point
19:22 Marcel__ joined #fuel
19:22 mwhahaha i've never seen that message before, i wonder if your network configuration is wrong
19:24 Marcel__ Hey All, I proposed a patch about the https://bugs.launchpad.net/fuel/+bug/1557628 and the review page is https://review.openstack.org/#/c/293054/
19:24 Marcel__ Please can you take a look at it?
19:29 venkat_ joined #fuel
19:46 e0ne joined #fuel
20:23 skamithi13 joined #fuel
20:24 skamithi13 joined #fuel
20:44 ohyeahhuh i got another report from a coworker about network config not being retained by fuel master
20:46 mwhahaha should report a bug and attach a snapshot
20:47 mwhahaha wiki.openstack.org/wiki/Fuel/How_to_contribute#Test_and_report_bugs
20:59 blabla_ joined #fuel
21:02 blabla_ left #fuel
21:03 working joined #fuel
21:04 working left #fuel
21:09 working joined #fuel
21:09 working quit
21:09 working quit
21:13 gariveradlt joined #fuel
21:37 e0ne joined #fuel
21:39 gariveradlt joined #fuel
22:10 DevStok joined #fuel
22:10 DevStok p_rabbitmq-server_start_0 on node-2.domain.tld 'unknown error'
22:10 DevStok then i got all neutron services down
22:10 DevStok crm - resourcces - restart p_rabbit_server
22:10 DevStok nothing
22:18 ilbot3 joined #fuel
22:18 Topic for #fuel is now Fuel 8.0 (Liberty) https://www.fuel-infra.org/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
22:23 DevStok crm resource status master_p_rabbitmq-server resource master_p_rabbitmq-server is running on: node-1.domain.tld resource master_p_rabbitmq-server is running on: node-3.domain.tld resource master_p_rabbitmq-server is NOT running
22:39 v1k0d3n joined #fuel
22:40 severion joined #fuel
22:43 v1k0d3n_ joined #fuel
22:44 severion joined #fuel
22:57 DevStok Error description:    {error,{cannot_create_mnesia_dir,"/var/lib/rabbitmq/mnesia/rabbit@node-2/",                                     enospc}}
22:57 DevStok rabbit log
22:59 mwhahaha disk space or did the disk go read-only?
23:00 DevStok drwxrwxr-x  2 rabbitmq rabbitmq  4096 Mar 16 22:49 mnesia/
23:01 DevStok I found a page that say the mnesia files are not killed all
23:01 DevStok so this create a problem on the status
23:01 DevStok in my case
23:01 DevStok rabbit fu*k pcs
23:02 DevStok and all neutron agents are dead
23:02 DevStok i found now
23:03 DevStok /dev/dm-4                   50G   47G     0 100% /
23:03 mwhahaha yup there's your problem :(
23:04 DevStok ok but what i have to delete
23:07 mwhahaha not sure, in the newer versions of fuel we've created more partitions and added in a minor health check to prevent the disk from getting to 100%
23:10 DevStok this can be the key of the problem
23:10 DevStok because
23:10 DevStok the node running well
23:10 DevStok stressed with many health check
23:10 DevStok but a day start to crash agents etc..
23:11 DevStok is it the right explanation
23:11 DevStok ?
23:15 DevStok http://paste.openstack.org/show/490817/
23:15 DevStok can i delete ...
23:18 v1k0d3n joined #fuel
23:24 DevStok what kind of file is ./srv/node/2/objects/1675/d2b/d1743e60e304ffe623ef6f59cde47d2b/1457950948.55912.data
23:27 DevStok ok i removed the big files
23:27 DevStok and rabbit started weel again
23:27 DevStok all services are up
23:28 DevStok how can I avoid that partition fill up?
23:28 xarses this is also why we discourage putting mongo with the controller
23:28 xarses mongo tends to eat disks
23:29 DevStok ok
23:29 xarses do maintenance on mongo
23:29 xarses or shorten the alerts history
23:29 DevStok but with fuel I gave enough space
23:29 DevStok at the moment i dont use ceilometer
23:29 DevStok is better to stop from crm
23:30 DevStok ?
23:30 DevStok i have to kill the mongo process too?
23:30 xarses I don't think its in crm
23:30 xarses you will want to remove it from all of the service configs, since they will still send messages
23:30 xarses which will move the problem to rabbit
23:30 DevStok crm(live)resource# stop p_ceilometer-agent-central
23:31 xarses hrm
23:31 xarses I guess you want them off
23:31 skamithi13 joined #fuel
23:31 DevStok to free space i deleted this files crm(live)resource# stop p_ceilometer-agent-central
23:32 DevStok sorry ./srv/node/2/objects/1675/d2b/d1743e60e304ffe623ef6f59cde47d2b/1457950948.55912.data
23:32 skamithi13 joined #fuel
23:36 DevStok this path is the swift cache?
23:36 DevStok /srv/node/2/objects/
23:40 DevStok yes cache
23:40 DevStok deleted all
23:40 DevStok xarses give me a feedback please
23:41 xarses sorry, /srv/node is probably swift
23:42 xarses if you aren't using ceph anyway
23:42 xarses so, thats related to your glance images at the worst
23:43 Jeffrey4l joined #fuel
23:59 skamithi14 joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary