Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-08-27

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 kupo24z Ah spoke too soon, critical error on neutron-openvswitch-agent
00:09 kupo24z http://pastebin.mozilla.org/6168619
00:10 kupo24z odd enough it marked the deployment as successful
00:11 xarses icks
00:11 kupo24z xarses: its non-critical for 5.1 apparently? https://bugs.launchpad.net/mos/+bug/1347612
00:12 xarses is your ovs agent still running?
00:12 kupo24z nova says it is..
00:12 kupo24z se
00:12 xarses neutron agent-list ?
00:13 kupo24z Yeah
00:13 kupo24z all are up
00:13 xarses ok, can you paste agent-list in the bug, and upload a snapshot
00:13 xarses I'll kick it back up
00:13 kupo24z In that mos bug?
00:13 xarses yes please
00:16 xarses OK, I changed the tags around, just waiting for your notes.
00:16 kupo24z waiting on snapshot
00:16 kupo24z this server isnt slow either...
00:16 xarses btw, you're awesome on the bug reporting
00:16 xarses It's helpful
00:17 kupo24z yeah its nice to have things fixed so fast
00:19 kupo24z Next week should be able to test 5.1 on a larger cluster, around 45-50 nodes
00:20 rmoe joined #fuel
00:20 kupo24z Added comment
00:22 xarses thanks
00:23 xarses that should light a fire on that one
01:29 xarses joined #fuel
01:31 kupo24z xarses: updated with my findings on https://bugs.launchpad.net/fuel/+bug/1361391
01:33 kupo24z also My docker-dnsmasq is getting flooded with Aug 27 01:44:41 dnsmasq[2809]: query[ANY] webpanel.sk from 172.17.42.1 Aug 27 01:44:41 dnsmasq[2809]: forwarded webpanel.sk to 8.8.8.8. 172.17.42.1 is set to docker0 which is causing (i think) intermittent ssh: Could not resolve hostname node-2: Temporary failure in name resolution
01:34 kupo24z not sure if thats coming from fuel or externally and i should just block
02:11 Kupo24z1 joined #fuel
02:14 klonhjcertusnet joined #fuel
02:22 nimissa joined #fuel
04:29 ArminderS joined #fuel
05:26 aepifanov joined #fuel
05:29 adanin joined #fuel
05:39 nimion joined #fuel
05:59 ddmitriev1 joined #fuel
06:07 aepifanov joined #fuel
06:20 Longgeek joined #fuel
06:50 tuvenen joined #fuel
06:56 artem_panchenko joined #fuel
06:58 artem_panchenko joined #fuel
07:07 HeOS joined #fuel
07:20 e0ne joined #fuel
07:36 e0ne joined #fuel
07:37 HeOS joined #fuel
07:42 e0ne joined #fuel
07:58 e0ne joined #fuel
08:00 aepifano1 joined #fuel
08:00 adanin_ joined #fuel
08:02 HeOS joined #fuel
08:04 omelchek1 joined #fuel
08:05 e0ne_ joined #fuel
08:08 sb- joined #fuel
08:10 monester_ joined #fuel
08:11 mattymo joined #fuel
08:11 gleam joined #fuel
08:11 fhond joined #fuel
08:11 eshumkher joined #fuel
08:12 dilyin joined #fuel
08:16 agordeev joined #fuel
08:16 eshumkher joined #fuel
08:17 nimissa joined #fuel
08:18 dshulyak joined #fuel
08:21 Longgeek joined #fuel
08:22 f13o joined #fuel
08:41 dancn joined #fuel
08:43 Longgeek joined #fuel
08:49 aepifanov joined #fuel
08:52 igajsin joined #fuel
09:04 e0ne joined #fuel
09:08 baboune joined #fuel
09:08 baboune hi, is the following error an issue or can it be ignored?  [7f5173479700] (receiver) RPC method deploy_resp received: {"status": "error", "task_uuid": "39568f07-aac3-49b5-847d-c66452e10338", "error": "Error occurred while running method 'deploy'. Inspect Orchestrator logs for the details."} /usr/lib64/python2.6/site-packages/sqlalchemy/sql/expression.py:1927: SAWarning: The IN-predicate on "nodes.id" was invoked with an empty sequence.
09:11 baboune I am using fuel 5.0.1
09:15 tdubyk joined #fuel
09:20 kaliya hi baboune, does that error occur when you deploy a new environment?
09:31 geekinutah joined #fuel
09:44 e0ne joined #fuel
09:50 nimissa joined #fuel
10:08 HeOS joined #fuel
10:11 dpyzhov joined #fuel
10:16 Longgeek joined #fuel
10:51 HeOS joined #fuel
11:00 Longgeek joined #fuel
11:37 nimissa joined #fuel
11:50 eshumkher joined #fuel
11:51 sc-rm_ joined #fuel
11:56 sc-rm_ When upgrading from 5.0 to 5.0.1 I get this error http://paste.openstack.org/show/100992/
11:56 sc-rm_ and it fails in upgrading
12:03 kaliya sc-rm_, that page is giving me error 500
12:04 sc-rm_ trye again
12:04 sc-rm_ it works now here, but saw the 500 error :-)
12:07 kaliya sc-rm_, seems that error 500 is becoming frequent on paste.openstack
12:08 akupko joined #fuel
12:08 sc-rm_ lets try with http://pastebin.com/wFYWhEYZ
12:08 kaliya sc-rm_, might be related to https://bugs.launchpad.net/fuel/+bug/1341492
12:10 sc-rm_ might be, but have tried to run upgrade 3-4 times with no luck, so just have to wait for a fix
12:14 homegrown joined #fuel
12:17 sc-rm_ I also cannot delete volumns inside openstack deployed with fuel, any suggestions to where to go?
12:21 eshumkher joined #fuel
12:23 adanin joined #fuel
12:48 sc-rm_ kaliya: http://pastebin.com/AP6SUWkX is what happens before it crashes with the first error
12:49 sc-rm_ kaliya: but I tried to edit the patch in manually and are now trying a now upgrade to se if it fixes the problem with more retries.
13:07 kaliya sc-rm_, which change are you trying to work on? what did you edit on patch
13:08 sc-rm_ the changes for fixing this bug https://bugs.launchpad.net/fuel/+bug/1341492 I tried to apply inside my upgrade and it did not fail and now says “(upgrade) *** UPGRADE DONE SUCCESSFULLY"
13:09 kaliya thanks, sc-rm_
13:09 sc-rm_ but afterwards I don’t see the buttons to upgrade openstack as described in the release not for fuel
13:10 sc-rm_ and I have a problem with when restarting nodes, fule insist on deploying changes to those nodes disk configuration
13:10 sc-rm_ but when trying to deploy fuel fails
13:19 evg sc-rm_: after upgrade please try rebooting already bootstraped nodes
13:19 sc-rm_ applying windows trick now - aka rebooting nodes
13:21 evg sc-rm_: :) only bootstrapped ones
13:25 sc-rm_ stated out by restarting fuel server and then moving on from there
13:32 Chocobo joined #fuel
13:32 Chocobo joined #fuel
13:33 Chocobo Has anyone been able to deploy OpenStack using Fuel 5.0.1?  It works fine when using the virtual box scripts but we can not seem to get HA,CEPH w/ Ceilometer to deploy on hardware.  Ringsync errors and eventual deployment timeouts.
13:34 Chocobo We even tried applying this patch to the master node: https://review.openstack.org/#/c/116247/
13:35 sc-rm_ evg: after rebooting fuel. Fuel still insists on deploying changes to the disk configuration, but it fails to do so.
13:38 xdeller joined #fuel
13:47 evg sc-rm_: sorry, it's not clean for me. Have you made some configuration withoutu deploying before upgrade?
13:47 sc-rm_ evg: let my explain it like this.
13:48 mattgriffin joined #fuel
13:48 sc-rm_ evg: I deployed Fueld successfully and deployed OpenStack successfully and was running
13:48 eshumkher joined #fuel
13:49 sc-rm_ evg: Then I had to reboot one of the storage nodes because we had to move the power for it. When it then came back online Fuel says it needs to deploy changes to disk configuration for that node
13:49 sc-rm_ evg: Even though there have been no changes to the node it self other than being moved from one power source to another
13:50 e0ne joined #fuel
13:51 Chocobo It looks like my first error occurs with Ceph with "Unable to pull /etc/ceph/ceph.conf from node-6"  What is likely to cause this?  Again... our network verification passed.
13:58 MiroslavAnashkin Either incorrect ceph monitor node address in initial /etc/ceph/ceph.conf or no connection from the deployed node to main ceph monitor.
13:59 sc-rm_ evg: and after rebooting the fuel computer it still says it needs to deploy changes
14:00 MiroslavAnashkin Please go to the failed node and check both ceph.conf files - one in /etc/ceph/ceph.conf and second in /root/ceph.conf. The config in the/root/ should be a link to the /etc/ceph/ceph.conf
14:02 eshumkher joined #fuel
14:02 Chocobo MiroslavAnashkin: We use the default network addresses in an attempt to get this up and running.  We changed nothing.   We will take a look though.
14:03 evg sc-rm_: please check "fuel node" and "fuel task". Don't you see something suspicious?
14:04 sc-rm_ evg: http://pastebin.com/7ewP88EL
14:07 sc-rm_ evg: I have to go, but will be back tomorrow looking at this issue again.
14:10 evg sc-rm_: ok. we'll think what we can do here.
14:10 sc-rm_ evg: Thanks and sounds cool :-)
14:10 Chocobo MiroslavAnashkin: We can ping all the host IPs in the mon_host in the ceph.conf file.
14:13 MiroslavAnashkin Please try on the failed node the following `ceph-deploy --overwrite-conf config pull <primary mon IP address>`
14:13 Chocobo MiroslavAnashkin: How do we know what the "Primary mon IP address" is?
14:14 MiroslavAnashkin node-6 i believe - it was listed in the error message
14:19 Chocobo MiroslavAnashkin: It looks like it is hanging on the .sudo_pushy step (will use a remote connection without sudo) then it fails with a "GenericError: Failed to fetch config from 1 hosts"
14:19 Chocobo Firewall issue?
14:24 Chocobo https://bugs.launchpad.net/fuel/+bug/1316524   Maybe this?
14:27 MiroslavAnashkin Do you use separate MongoDB node?
14:27 MiroslavAnashkin If yes - then looks like this is it.
14:30 Chocobo MiroslavAnashkin: Maybe?  MongoDB is running on two of the controller nodes, and CephOSD is running on the three remaining nodes.
14:34 Chocobo Huh, it looks that patch was already applied.   :/
14:35 MiroslavAnashkin Yes, it was fixed before 5.1 branch was created
14:36 MiroslavAnashkin It might be a firewall issue as well, all these firewall are always bring some issues
14:57 angdraug joined #fuel
14:58 mattgriffin joined #fuel
15:06 pasquier-s_ joined #fuel
15:20 ilbot3 joined #fuel
15:20 Topic for #fuel is now Fuel 5.0.1 for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
15:21 MiroslavAnashkin Please check promiscuous mode is allowed on the firewall or on the network ports, serving storage and management networks
15:21 Longgeek joined #fuel
15:22 homegrown joined #fuel
15:22 gleam joined #fuel
15:22 mattymo joined #fuel
15:22 christopheraedo joined #fuel
15:22 sanek joined #fuel
15:31 xarses joined #fuel
15:34 Kupo24z1 xarses: you around?
15:34 xarses nope
15:34 Kupo24z1 ah oh well
15:34 xarses ;)
15:35 tdubyk joined #fuel
15:35 Kupo24z1 So i found a newish bug with resource limits on rbd ephemermeral disks; http://www.gossamer-threads.com/lists/openstack/dev/40717 they already made a backports bug on it however I was wondering if its possible to apply that hack post deployment?
15:43 angdraug joined #fuel
15:47 ianunruh joined #fuel
15:47 ianunruh we just installed fuel in our organization and it can't reach DNS because it is on the 172.17.0.0/16 subnet
15:48 ianunruh which is what docker already uses on the fuel master
15:48 eshumkher joined #fuel
15:48 ianunruh could we possibly change the docker subnet?
15:48 xarses mattymo: ^
15:49 mattymo ianunruh, it selects this network on the daemon start based on networks currently configured
15:51 HeOS joined #fuel
15:51 HeOS left #fuel
15:51 mattymo you can work around this by editing the /etc/sysconfig/docker file to add -b docker0=172.99.99.0/24
15:51 mattymo add it to other_args=
15:51 HeOS joined #fuel
15:53 Kupo24z1 xarses: would that be possible or would it need to be done at iso creation?
15:53 homegrown left #fuel
15:53 adanin joined #fuel
15:54 xarses kupo24z: i haven't looked it over yet, in a meeting
15:56 Kupo24z1 kk np
15:57 HeOS_ joined #fuel
15:57 f13o joined #fuel
15:59 HeOS_ joined #fuel
16:03 [HeOS] joined #fuel
16:15 xarses kupo24z: you can patch nova after deployment yourself, or even have puppet apply the patch for you if you'd like
16:16 scroiset left #fuel
16:18 Kupo24z1 xarses: i found /usr/share/pyshared/nova/virt/libvirt/imagebackend.py with that code in it however unsure if i can just change that file on all nodes and see changes
16:22 xarses change it, and restart nova-compue
16:22 xarses compute even
16:24 Kupo24z1 then hard reboot instance and check for iotune in xml?
16:24 xarses just provision a new instance
16:25 Chocobo joined #fuel
16:25 Chocobo joined #fuel
16:25 Kupo24z1 xarses: does spacing work in that .py code? I have http://pastebin.mozilla.org/6181000
16:25 Kupo24z1 s/work/matter/
16:26 xarses yes, the indent needs to line up correctly
16:26 Chocobo Well we are going to try again without Ceph.  Ceph doesn't seem stable in 5.0.1
16:27 xarses Chocobo: ceph should be stable, can you share the topology you are attempting to deploy?
16:28 Kupo24z1 does that indent look correct?
16:28 xarses kupo24z: close enough, 17 and 14 could be indented again, but it's not required to function
16:29 Kupo24z1 Does that need to be changed on compute nodes for scheduler/api?
16:30 xarses no, scheduler/api doesn't run on the computes
16:31 xarses just the controllers, and they don't influence the libvirt xml
16:31 Chocobo xarses: we have 5 nodes we are doing HA with.  Two of the controller nodes have ceilometer running on them.  One controller has Ceph OSD.  The two compute nodes have Ceph OSD.   We are using the default networking configuration.  We have a managed switch with two vlans (101 and 102) on the Admin/Management/Storage ports.
16:31 Chocobo Public network and admin network is untagged.
16:32 Kupo24z1 You shouldnt have ceph OSD running on a controller
16:33 Kupo24z1 since the controllers are ceph mon's its very bad to have OSD's on those
16:34 xarses Well, its worse to run mongo on the controllers
16:34 Chocobo Kupo24z1: Ok, but the installer never even made it that far.  It deployed the Controller/MongoDB nodes first.  The first controller node finished fine then it moved to the second Controller/MongoDB node where it failed with a Ringsync timeout.  We patched the puppet modules and re-ran it the deployment.  This time we got an error with the ceph-deploy config.
16:34 xarses also, I don't think it will work with 2 nodes correctly
16:35 xarses Chocobo: Swift Ringsync?
16:35 xarses Chocobo: ceph-deploy config pull?
16:36 Chocobo xarses: yes.  We found a patch that seems to have fixed the ringsync issue:  https://review.openstack.org/#/c/116247/
16:37 Chocobo xarses: yes the ceph-deploy config pull failed.  We tried it manually and it failed.  Not sure if there is a firewall issue.    We have done a similar deployment with virtualbox without issue.
16:37 xarses Chocobo: ceph-deploy config pull only fails when the node can't reach the first controller on the management network or the ssh-keys didn't get installed
16:38 xarses most likely its the prior
16:38 Chocobo xarses: all the nodes can ping one another, but ssh between nodes (as root) does not work.
16:39 rmoe joined #fuel
16:39 Chocobo When I say ping... we only tested on the management network.
16:39 xarses Chocobo: make sure its the management network, not the admin network. The management is only configured if puppet has run at least once
16:39 xarses ok
16:40 Chocobo xarses: 192.168.0.0/24 is the default management network?   I believe so...  but yes all three of the ceph nodes could ping one another on that network.
16:41 xarses not the ceph-osd nodes, the first controller by node-id
16:41 xarses the lowest numbered controller is considered the first or primary controller
16:42 Chocobo What are the "rules" for HA, including what services can co-exist on a node and how many nodes you need of each thing?  I know you need 3 controller nodes, 2 Ceph nodes...  how many MongoDB/Ceilometer nodes?   Can we do HA with 5 nodes + 1 fuel master.
16:42 eshumkher joined #fuel
16:43 Chocobo xarses: It was the lowest numbered node (node-6).  It started at 6 because we had one failed attempt that we stopped/reset.
16:43 Kupo24z1 xarses: no luck on that, i replied to the list. Oh well
16:59 jobewan joined #fuel
17:09 Chocobo We are trying it again in regular multi-node.
17:32 Chocobo Ugh!  Even in non-HA mode we get the error where it can not fetch the ceph.conf!
17:35 MiroslavAnashkin 90% of errors comes from network, it is our current statistics...
17:37 Chocobo We can ping all nodes on the managment network (192.168.0.0/24) from the lowest numbered node.  Should we check anything on the Admin (10.20.0.0/24) network or the Storage (192.168.1.0/24) network?
17:37 Chocobo It is frustrating because the network verification works.   The public network does not need to have a gateway does it?   We do not currently have a router NATing the public network.
17:45 MiroslavAnashkin Network verification checks basic connectivity only, but is not able to check all the ports/modes/protocols are allowed
17:46 MiroslavAnashkin Please check Storage network as well
17:47 Chocobo MiroslavAnashkin: Is there any reason to manually configure any of the networking?  We are using the defaults, but we have tried modifying the network using "fuelmenu" on the first boot...  the defaults seem fine to me and I am not sure what else I could configure if the switch is set up correctly.
17:48 Chocobo We are trying another deployment.  If this fails we will check the storage network as well.
17:49 MiroslavAnashkin If the defaults seem fine - there is no reason to change it. Configuration required when you are limited of available IP addresses, VLANs, etc
17:49 mattgriffin joined #fuel
17:50 Chocobo MiroslavAnashkin: ok so if the defaults are ok and the switch is configured correctly (I am assuming the network verification tests this) can you think of anything else we might have misconfigured that could be causing our errors?
17:59 Chocobo_ joined #fuel
17:59 Chocobo_ MiroslavAnashkin: sorry... network dropped.  I don't know if I missed anything you said.
18:02 Chocobo_ Maybe our use of Neutron/GRE is causing problems?  Do I need to increate the MTU on the NICs?
18:02 xarses Chocobo_: It's not a requirement, but it would help guest performance
18:02 xarses it would not be able to cause these issues though
18:04 xarses kupo24z: http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg33282.html
18:05 xarses In case you didn't see it, they changed the ISO url slightly. It now shows build status, and we will keep building 5.1 once it's cut to stable
18:12 eshumkher joined #fuel
18:14 e0ne joined #fuel
18:15 Chocobo_ Ok, I also forgot to mention how we are accessing the Fuel Master node:  We have a machine connected to the Admin/PXE switch and we are getting an address via the Fuel DHCP server.  I don't see how this could cause a problem...  but could it?
18:16 eshumkher joined #fuel
18:21 angdraug joined #fuel
18:24 Chocobo_ Oh... we were not using VLAN splinters with CentOS.  I think we will try another deployment with Ubuntu.
18:30 MiroslavAnashkin Chocobo_: For GRE you may change default physical NICs settings in order to gain full performance. It is possible with Fuel CLI only, as following: http://paste.openstack.org/show/101163/
18:35 Chocobo_ MiroslavAnashkin: Doing this will only increase performance correct?  In past (manual) OpenStack installs I have run into issues with GRE and MTU.  The additional space GRE takes in a packet forced me to reduce the MTU on the guest VMs (or increase the MTU on the physical NICs and enable Jumbo Frames).
18:38 MiroslavAnashkin Yes, but default NIC settings may decrease GRE performance so terrible, that it may appear like where is no connectivity at all.
18:39 Chocobo_ Ahhh, so it sounds like we have two things to try... Ubuntu and the changes you just pasted.
18:39 Chocobo_ We could also use Neutron/VLAN for testing.
18:40 Chocobo_ GRE has a lot of benefits in a production environment though.
18:43 MiroslavAnashkin And it is recommended to avoid using VLAN splinters with Neutron (OpenVSwitch) unless absolutely required.
18:43 Chocobo_ MiroslavAnashkin: In the paste you sent me what is ethY?   It mentions that ethX is each interface, but I see ethY with "ethtool"
18:43 Chocobo_ MiroslavAnashkin: Ok, thank you.  We will avoid them by using Ubuntu.
18:44 MiroslavAnashkin Just the next NIC
18:44 Chocobo_ Ahh ok, thanks.
18:44 Chocobo_ We will give those changes a try.
19:01 aepifanov joined #fuel
19:03 aepifanov joined #fuel
19:09 aepifanov joined #fuel
19:14 aepifanov joined #fuel
19:16 aepifanov joined #fuel
19:17 aepifanov joined #fuel
19:18 aepifanov joined #fuel
19:18 aepifanov joined #fuel
19:20 aepifanov joined #fuel
19:22 Kupo24z1 joined #fuel
19:26 aepifanov joined #fuel
19:27 HeOS joined #fuel
19:32 adanin joined #fuel
19:55 eshumkher joined #fuel
20:13 aepifanov joined #fuel
20:14 aepifanov joined #fuel
20:16 aepifanov joined #fuel
20:16 aepifano1 joined #fuel
20:21 aepifano1 joined #fuel
20:24 aepifano1 joined #fuel
20:35 aepifano1 joined #fuel
20:36 aepifanov1 joined #fuel
20:40 aepifano2 joined #fuel
20:48 eshumkher joined #fuel
20:53 aepifano1 joined #fuel
20:55 aepifano1 joined #fuel
20:55 aepifanov joined #fuel
21:02 aepifanov joined #fuel
21:02 aepifanov joined #fuel
21:06 angdraug_ joined #fuel
21:15 e0ne joined #fuel
21:22 e0ne joined #fuel
21:52 akupko joined #fuel
21:53 ykotko joined #fuel
21:53 t_dmitry joined #fuel
21:53 tdubyk joined #fuel
22:49 angdraug joined #fuel
23:04 xarses joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary