Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-03-19

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:10 loth Any reason why corosync-notifyd is using 100% cpu on one of my controllers?
00:22 rmoe joined #fuel
00:49 xarses joined #fuel
01:09 Longgeek joined #fuel
01:19 LiJiansheng joined #fuel
01:21 angdraug xarses: <loth> Any reason why corosync-notifyd is using 100% cpu on one of my controllers?
02:48 ilbot3 joined #fuel
02:48 Topic for #fuel is now Fuel 5.1.1 (Icehouse) and Fuel 6.0 (Juno) https://software.mirantis.com | Fuel for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
03:11 mattgriffin joined #fuel
04:09 gongysh_ joined #fuel
04:54 sanek joined #fuel
05:11 mattgriffin joined #fuel
06:11 adanin joined #fuel
06:34 xarses joined #fuel
07:00 dklepikov joined #fuel
07:04 stamak joined #fuel
07:06 sambork joined #fuel
07:42 saibarspeis joined #fuel
07:56 teran_ joined #fuel
07:57 corepb joined #fuel
08:01 tzn joined #fuel
08:07 kaliya joined #fuel
08:14 e0ne joined #fuel
08:34 stamak joined #fuel
08:35 jskarpet Installing Zabbix after initial deployment, how can I roll out Zabbix on already existing nodes?
08:48 alecv joined #fuel
08:54 kaliya jskarpet: this is not supported (yet but going to). You need to install and configure agents manually
08:54 jskarpet I can't just re-run with updated Puppet config?
08:55 kaliya You can. I never tried it personally
08:56 jskarpet How are those manifests generated? How to get an updated copy?
08:57 jskarpet or does the site.pp poll Fuel?\
08:59 jskarpet hmm, should probably be /etc/controller.yaml etc.
09:01 kaliya Manifests are rsynced to the nodes during the deployment. Then, a local puppet apply is run. After the provisioning, no connection nodes-master exists anymore
09:01 kaliya So, you have to hack the manifests locally, before applying, or rsyncing manually
09:01 jskarpet But the manifests are the same across no?
09:01 jskarpet only roles are different from the yaml files
09:02 jskarpet So how do I regenerate the controller.yaml etc. ?
09:02 jskarpet in order to account for the newly added Zabbix server
09:04 kaliya what is controller.yaml? MiroslavAnashkin will this work smoothly?
09:05 jskarpet It's variables for hiera
09:10 ChrisNBlum joined #fuel
09:19 sambork1 joined #fuel
09:19 monester_laptop joined #fuel
09:23 kaliya jskarpet: mmh this is sort of hacking of puppet, since I only know scenarios where the Zabbix server is installed at the first deployment, maybe better asking in #fuel-dev
09:23 HeOS joined #fuel
09:26 sambork joined #fuel
09:29 Longgeek joined #fuel
09:32 saibarspei joined #fuel
09:37 neophy joined #fuel
09:37 Longgeek joined #fuel
09:40 teran joined #fuel
09:43 aliemieshko_ joined #fuel
09:43 teran_ joined #fuel
09:57 neophy Hi all, I have openstack env deployed using fuel 5.1. Now fuel web UI shows all the three controllers are offline. I am able to ping all the controller ip and telnet port 22. when I try to ssh from fuel to  controller it hangs with message: debug1: Entering interactive session. after some time I am getting : Write failed: Broken pipe. Any one in the list faced similar issue? any help on this?
09:59 neophy Here is ssh debug log: http://paste.openstack.org/show/193499
10:00 neophy it  hangs at this level and Write failed: Broken pipe error after some time
10:00 neophy any help?
10:07 sambork joined #fuel
10:13 Longgeek joined #fuel
10:14 sambork joined #fuel
10:23 e0ne joined #fuel
10:26 kaliya neophy: virtual cluster?
10:27 neophy kaliya: all are physical machine
10:28 kaliya what does `fuel nodes` show for these controllers? Status
10:29 neophy kaliya: shows false on online column
10:30 neophy kaliya: but I am able to ping controller ip from fuel...
10:30 jskarpet Is adding more controller nodes after first deployment supported? (It failed miserably effectively taking down the other controllers)
10:30 samuelBartel joined #fuel
10:33 neophy kaliya: How do I trace the exact problem? It was working fine around one month...
10:33 kaliya jskarpet: it is since 5.1, where you can deploy only 1 controller in HA with the idea to add new nodes in the future
10:34 jskarpet I have 3 controller with HA
10:34 jskarpet adding 2 more
10:34 jskarpet Fuel 6.0
10:34 kaliya jskarpet: can you share a snapshot?
10:35 jskarpet Yeah, but it's going to be big :P
10:35 jskarpet 1-2GB
10:35 kaliya NP, if you can share on Drive, Dropbox or some other source and then share the URL, I will file a bug with your problem
10:36 jskarpet I see a lot of these in the Puppet run on existing controller: "Neutron API not avalaible. Wait up to 60 sec"
10:36 kaliya neophy: which network settings?
10:36 kaliya jskarpet: looks like you have HA issues? `pcs status`
10:36 neophy kaliya: Neutron with VLAN
10:37 jskarpet kaliya; Looks normal (like before attempted adding of more nodes)
10:37 kaliya jskarpet: do you have some failures in keystone-all.log?
10:37 jskarpet no
10:38 kaliya neophy: do you have other nodes in the cluster? Are they reachable?
10:38 neophy kaliya: yes. I have 2 compute node and 3 mongodb node they are reachable
10:38 kaliya ok jskarpet please generate a snapshot and upload to some public place. I will look into your logs
10:38 MartinHansen joined #fuel
10:38 kaliya neophy: only controllers like frozen so
10:39 MartinHansen Hey, i have a few questions about networking and fuel 6.0
10:39 neophy kaliya: yes. only controllers have issue
10:39 kaliya neophy: in `fuel nodes` other nodes are online True right?
10:39 kaliya neophy: Fuel version?
10:39 neophy kaliya:yes
10:39 neophy kaliya:furl 5.1
10:39 neophy kaliya: sorry fuel 5.1
10:40 kaliya MartinHansen: try to ask. But maybe better ask in #fuel-dev in case of implementation technicalities or in-depth questions ;)
10:40 kaliya evg: how can that controllers get frozen like in the neophy case (still pingable but offline)?
10:41 MartinHansen thanks
10:43 ikalnitsky joined #fuel
10:46 evg neophy: kaliya: there were talks about this "brocken pipe" a month ago. But I can't remember what was the root cause.
10:47 kaliya evg: in this chat?
10:47 evg neophy: I would check network consistency the first
10:48 evg kaliya: it seems to me yes
10:49 jskarpet If you have another device on the same ip, you can see that behavior also
10:49 kaliya neophy: no way to get a console from those servers? Rack software or so, and check their status?
10:49 evg neophy: could you try ping -s <big number>
10:50 neophy kaliya: when I reboot controllers they will come up. fuel nodes shows status true and able to login to controllers. after 5 to 10 mins it again fuel nodes shows False and unable to login
10:51 kaliya neophy: I would check the load. How many controllers? 3?
10:51 neophy kaliya: yes
10:52 kaliya neophy: SSHing same result on the primary and the other 2?
10:52 neophy kaliya: Now I rebooted one controller and logged into that controller
10:52 neophy kaliya: yes. all the controllers behaving the same way
10:56 neophy kaliya:do you want to see any log on the controller?
10:57 kaliya neophy: what's the load?
10:57 kaliya neophy: do you have Zabbix around running?
10:58 neophy kaliya: no Zabbix.
10:59 evg neophy: do you see domething fishy in mcolective/rabbit logs?
10:59 kaliya neophy: I would check at first /var/log/messages
10:59 evg neophy: daemon.log
11:02 kaliya jskarpet: today and tomorrow I'm practically like ooo, please file a bug yourself or share asap the snapshot URL
11:04 jaypipes joined #fuel
11:10 neophy evg: I got this error some time back in rabbit log: 253879:=ERROR REPORT==== 19-Mar-2015::10:52:47 ===
11:10 neophy 253880:Mnesia('rabbit@node-32'): ** ERROR ** (core dumped to file: "/var/lib/rabbitmq/MnesiaCore.rabbit@node-32_1426_762367_477655")
11:14 kaliya evg: neophy we have some ocf improvements, maybe they apply
11:15 kaliya neophy: we backported some improvements in rabbitmq clustering to 5.1
11:17 neophy kaliya: do you mean I have to apply the  rabbitmq  improvements release by fuel?
11:17 kaliya neophy: I have not the complete picture, but if Rabbit is the root cause, it will give relief
11:18 kaliya neophy: strange things in messages? in daemon.log?
11:18 kaliya (away for some hours more or less, I will spordically read)
11:18 evg neophy: please grep "error report" in rabbit*.log on other controllers
11:20 neophy kaliya: here is message.log last few lines:<kaliya> neophy: I have not the complete picture, but if Rabbit is the root cause, it will give relief
11:20 neophy <kaliy
11:20 neophy kaliya: sorry, http://paste.openstack.org/show/193522/
11:21 neophy kaliya: this few lines from /var/log/messages
11:30 neophy evg: here is last error report in rabbit log: http://paste.openstack.org/show/193528/
11:33 sambork joined #fuel
11:48 DaveJ__ joined #fuel
11:48 e0ne joined #fuel
12:07 evg neophy: I still think you should carefully check your network. ping -s 1472 ...
12:08 neophy evg: ping from fuel to controller mgmt ip?
12:09 evg neophy: yes
12:10 neophy evg:ok
12:20 Longgeek joined #fuel
12:25 neophy evg:I am getting ping replay while doing ping using: ping -s 1472 <controller_ip>
12:28 evg neophy: try run it for a time
12:28 evg neophy: does ssh still work?
12:28 neophy evg:sure. it is going on...
12:29 sambork1 joined #fuel
12:37 Longgeek_ joined #fuel
12:39 jskarpet 400 Bad Request on Ceph container creation through radosgw swift - where to look for this in logs on the server side?
12:47 Akshik joined #fuel
12:52 monester_laptop joined #fuel
12:53 corepb joined #fuel
12:56 sambork joined #fuel
12:57 neophy evg/kaliya: I am away for some time. will join again
13:15 corepb joined #fuel
13:27 e0ne joined #fuel
13:46 julien_ZTE joined #fuel
13:59 claflico joined #fuel
14:12 mattgriffin joined #fuel
14:15 julien_ZTE joined #fuel
14:17 tzn joined #fuel
14:31 adanin joined #fuel
14:41 CheKoLyN joined #fuel
15:11 wayneeseguin joined #fuel
15:18 neophy joined #fuel
15:29 adanin joined #fuel
15:55 kozhukalov joined #fuel
16:01 angdraug joined #fuel
16:03 blahRus joined #fuel
16:04 xarses joined #fuel
16:09 julien_ZTE joined #fuel
16:10 rmoe joined #fuel
16:17 pal_bth joined #fuel
16:26 emagana joined #fuel
16:28 championofcyrodi Hello again.  I was hoping to PXE boot a system to use as ceph osd only... however it is 32-bit and the pxe boot initrd/kernel does not work.
16:28 championofcyrodi all of my beefier systems are 64bit, but was hoping to try out lower powered systems to just run as ceph osds (w/o monitor or client)
16:34 championofcyrodi looks like this is cobbler related
16:46 thumpba joined #fuel
16:47 bjoernfan Does anybody have any good reading on how the OVS flows are set up when a compute reboots? My google fu is failing me.
16:47 bjoernfan I think we are getting loops in our network before the flow rules are properly set up, and sometimes they are not set up at all.
16:47 bjoernfan For a few random nodes.
16:48 angdraug championofcyrodi: we don't support 32bit systems
16:48 thumpba_ joined #fuel
16:49 angdraug effort to add such support would be comparable with adding support for another distro version, to wit ubuntu trusty support effort in 6.1
16:49 angdraug bjoernfan: l23network module in fuel-library is where it's done
16:50 angdraug http://docs.mirantis.com/openstack/fuel/fuel-6.0/reference-architecture.html#network-architecture
16:50 angdraug ^^ is a high level description of what it's trying to do
16:53 bjoernfan Thanks, I'll check it out and might have more questions tomorrow. :)
17:09 julien_ZTE joined #fuel
17:27 stamak joined #fuel
17:28 championofcyrodi angdraug: no biggie, i'll likely set this up stand alone w/ ceph only.
17:29 championofcyrodi to test
17:49 ikalnitsky left #fuel
17:59 mattgriffin joined #fuel
18:25 stamak joined #fuel
18:32 emagana joined #fuel
18:36 stamak joined #fuel
19:00 mattgriffin joined #fuel
19:01 monester_laptop joined #fuel
19:02 mattgriffin joined #fuel
19:07 jobewan joined #fuel
19:21 HeOS joined #fuel
19:27 julien_ZTE joined #fuel
19:39 angdraug joined #fuel
19:53 [HeOS] joined #fuel
20:01 teran joined #fuel
20:03 mattgrif_ joined #fuel
20:05 stamak joined #fuel
20:09 e0ne joined #fuel
20:17 admin0 joined #fuel
20:25 tzn_ joined #fuel
20:25 emagana joined #fuel
20:48 mattgriffin joined #fuel
20:54 emagana joined #fuel
21:37 julien_ZTE joined #fuel
22:10 e0ne joined #fuel
22:18 Killsudo joined #fuel
23:09 julien_ZTE joined #fuel
23:17 julien_ZTE joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary