Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-03-24

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:30 skore joined #fuel
02:31 skore left #fuel
03:51 aleksandr_null joined #fuel
04:20 fandi joined #fuel
04:41 vkozhukalov joined #fuel
04:45 vkozhukalov left #fuel
04:50 vkozhukalov joined #fuel
04:50 vkozhukalov left #fuel
05:22 Ch00k joined #fuel
05:43 IlyaE joined #fuel
06:00 mihgen joined #fuel
06:19 xarses joined #fuel
06:35 saju_m joined #fuel
07:12 saju_m joined #fuel
07:19 saju_m joined #fuel
07:19 aglarendil|pc joined #fuel
07:19 mihgen_ joined #fuel
07:19 MiroslavAnashkin joined #fuel
07:24 saju_m joined #fuel
07:31 Ch00k joined #fuel
07:31 saju_m joined #fuel
07:44 e0ne joined #fuel
07:55 evgeniyl joined #fuel
08:02 bookwar joined #fuel
08:24 likelion joined #fuel
08:24 likelion left #fuel
08:24 baboune joined #fuel
08:26 bogdando joined #fuel
08:30 e0ne joined #fuel
08:35 dburmistrov joined #fuel
08:35 Ch00k_ joined #fuel
08:49 dburmistrov joined #fuel
09:09 vk joined #fuel
09:11 topochan joined #fuel
09:11 topochan joined #fuel
09:21 rvyalov joined #fuel
09:24 dpyzhov joined #fuel
09:26 brain461 joined #fuel
09:27 e0ne_ joined #fuel
09:29 mihgen joined #fuel
09:50 vkozhukalov joined #fuel
10:07 DaveJ__ joined #fuel
10:11 vkozhukalov left #fuel
10:54 vkozhukalov joined #fuel
11:31 topochan joined #fuel
12:14 meow-nofer joined #fuel
12:17 Ch00k joined #fuel
12:32 MiroslavAnashkin joined #fuel
12:50 vkozhukalov1 joined #fuel
12:55 e0ne joined #fuel
12:55 dubmusic joined #fuel
12:58 dubmusic I have a question about dhcp in an HA-neutron-VLAN environment.  Is there anyone that is here to chat?
13:05 MiroslavAnashkin Simply ask your question. This chat is logged and we read these logs.
13:08 dubmusic Awesome
13:11 e0ne joined #fuel
13:11 dubmusic OK, I am running Fuel 4.1 and my instances are simply not getting their IP via DHCP.  The qdhcp, qrouter services are running as expected and I can easily set an IP manually, along with a gateway and reach the instance via the floating IP, so I have isolated it to the dhcp service
13:12 dubmusic My controller nodes are 8,9,10 and the single instance runs on node-11
13:12 justif joined #fuel
13:12 Ch00k joined #fuel
13:17 dubmusic root@node-8:~# neutron agent-list
13:17 dubmusic +--------------------------------------+--------------------+---------+-------+----------------+
13:17 dubmusic | id                                   | agent_type         | host    | alive | admin_state_up |
13:17 dubmusic +--------------------------------------+--------------------+---------+-------+----------------+
13:17 dubmusic | 1ebd4a6e-7a60-442b-b89a-c9f1d45ca514 | Open vSwitch agent | node-11 | :-)   | True           |
13:17 dubmusic | 46dde73d-e0ef-42a1-8e49-0a22ab707cf9 | L3 agent           | node-9  | :-)   | True           |
13:17 dubmusic | 4a7e8c6a-e3e3-4a25-bfae-c0d75ae3762d | DHCP agent         | node-10 | :-)   | True           |
13:18 dubmusic | 5d0558b7-9c1f-4108-b320-df08fb763e7d | Open vSwitch agent | node-3  | :-)   | True           |
13:18 dubmusic | 5d8981bd-9ca3-4b78-a164-feec3669025d | Open vSwitch agent | node-9  | :-)   | True           |
13:18 dubmusic | 6139e947-5193-49bc-8b32-6766bd3e79cb | Open vSwitch agent | node-8  | :-)   | True           |
13:18 dubmusic | 6a439de4-7bb0-4839-9623-3ee258f5832e | Open vSwitch agent | node-1  | :-)   | True           |
13:18 dubmusic | 7b229a6f-5632-4a00-9b72-739c97c5a041 | Open vSwitch agent | node-4  | :-)   | True           |
13:18 dubmusic | ea34ff2f-5895-4ee1-9732-4ec75722d239 | Open vSwitch agent | node-10 | :-)   | True           |
13:18 dubmusic +--------------------------------------+--------------------+---------+-------+----------------+
13:18 TVR_ heh.. there is always pastebin or http://paste.openstack.org/
13:18 dubmusic Sorry. I will use pastebin nexxt time
13:18 dubmusic Total cut and haste error
13:19 TVR_ no worries.. just makes it to where people can look back over your post rather than dealing with scrollback
13:19 dubmusic TVR, you posted a similar problem on Jan 31
13:20 TVR_ yes... so I am looking up what the issue was..
13:21 dubmusic I cannot get a dhcp address on my cluster, with a manual setting of IP, I can even reach the instance through the floating IP
13:21 TVR_ ok.. got it...
13:22 TVR_ so.. what is the dhcp server IP?
13:22 TVR_ 192.168.111.2?
13:23 dubmusic Also, which network should the BOOTP messages be sent over?
13:23 dubmusic Let me check
13:23 TVR_ ip netns
13:23 TVR_ ip netns exec qdhcp-<whatever it is> ip addr
13:24 TVR_ that will give you what it ~should ~be
13:24 dubmusic http://pastebin.com/MPZHKgZK
13:25 TVR_ look to see if you have 2 tap devices...
13:25 dubmusic I see only one
13:25 TVR_ good..good
13:26 TVR_ neutron port-list --all-tenants | grep ef892423-82
13:27 Dr_Drache joined #fuel
13:27 TVR_ you get results?
13:27 dubmusic One sec
13:29 dubmusic vpn glitch
13:30 TVR_ also.. what was your qrouter name from ip netns? Check that with "ip netns exec qrouter-<whatever it is> ip a" to be sure it's there
13:31 TVR_ if it all looks good, you may need to log into node-10 and run "service neutron-dhcp-agent stop ; killall dnsmasq ; service neutron-dhcp-agent start"
13:32 dubmusic_ joined #fuel
13:32 dubmusic_ http://pastebin.com/pVfpxhVS
13:32 tatyana joined #fuel
13:32 Dr_Drache TVR_, happy monday!
13:32 TVR_ yes, yes it is...
13:32 Dr_Drache lol
13:34 Dr_Drache wait for it..
13:34 Dr_Drache wait for it...
13:34 Dr_Drache BOOM
13:34 Dr_Drache ..soory
13:34 Dr_Drache s/soory/sorry :P
13:35 dubmusic http://paste.openstack.org/show/74132/
13:35 dubmusic Router looks happy with both internal and floating addresses present
13:36 tatyana joined #fuel
13:37 dubmusic I am seeing BOOTP traffic show up on different interfaces.  Where should the BOOTP traffic live?
13:41 TVR_ one sec.. looking at mine now
13:44 xdeller joined #fuel
13:46 TVR_ do you have a dhcp server on your public nic?
13:48 dubmusic no
13:48 jobewan joined #fuel
13:48 TVR_ I am seeing two networks on it
13:48 TVR_ inet 172.19.73.130/24 brd 172.19.73.255 scope global qg-730fc5c2-6c     inet 172.19.73.131/32 brd 172.19.73.131 scope global qg-730fc5c2-6c
13:49 Ch00k joined #fuel
13:50 TVR_ they seem to overlap... not sure if that is the issue.. but it isn't right
13:50 TVR_ check your qg-730fc5c2-6c int
13:57 vkozhukalov1 left #fuel
13:59 vkozhukalov1 joined #fuel
13:59 vkozhukalov1 left #fuel
13:59 TVR_ what does "crm status" show? anything out of the norm?
14:03 dubmusic http://paste.openstack.org/show/74136/
14:06 Ch00k joined #fuel
14:09 anotchenko joined #fuel
14:10 topochan_ joined #fuel
14:16 TVR_ dubmusic  Did you try the restarting the services? The only issue from what I have seen so far is the additional 172.19.73.131/32  on the public network... not sure why that would be there. Make sure you are not trying to issue dhcp from the public network, as this would be my first place to start.
14:24 MiroslavAnashkin dubmusic: You may restart networking with these commands `crm resource cleanup clone_p_neutron-plugin-openvswitch-agent` and then
14:25 MiroslavAnashkin `crm resource restart clone_p_neutron-plugin-openvswitch-agent`
14:26 MiroslavAnashkin Run these 2 commands on single controller only. These should clean and restart all networking services, including p_neutron-dhcp-agent and p_neutron-l3-agent
14:28 TVR_ leave it to MiroslavAnashkin to have the solid answers at his fingertips ..
14:28 vkozhukalov1 joined #fuel
14:30 mjeanson joined #fuel
14:50 anotchenko joined #fuel
14:56 IlyaE joined #fuel
15:03 vkozhukalov1 left #fuel
15:08 vkozhukalov joined #fuel
15:37 dubmusic joined #fuel
15:38 dubmusic OK.  I will try these. commands.  Thank you.  BTW, Did you see an issue with the CRM status, Miroslav?
15:39 anotchenko joined #fuel
15:54 dubmusic OK.  I restarted the CRM the neutron-dhcp is on a new server, but the ip netns indicates that the qdhcp is not on the same server.
15:55 dubmusic root@node-9:~# ip netns
15:55 dubmusic qrouter-c0ca0b20-8ccb-412c-b55a-df7ac0a59645
15:56 mutex dubmusic: you also might want to try restarting the openvswitch daemon on the compute node your instance is on
15:57 MiroslavAnashkin I see no issues in your CRM status output. And it is OK if L3 and DHCP agents exist in different nodes. There should be single agent instance for namespace
15:59 dubmusic I will do that, but first I have two qrouter instances now and the qdhcp is on a different host than the neutron dhcp service, is that OK?
16:00 dubmusic Do want to see it after I ran the commands?
16:01 dubmusic After I ran the two commands, the services changed.  I now have the 2 qrouters and a qdhcp that seems to be on the wrong server.  I will paste in the current crm status
16:02 anotchenko joined #fuel
16:02 dubmusic http://paste.openstack.org/show/74156/
16:05 MiroslavAnashkin Yes, DHCP agent may be running on every host
16:09 MiroslavAnashkin s/every/any/
16:11 dubmusic OK.  Still no luck with dhcp
16:15 MiroslavAnashkin Did you made any manual changes to /etc/neutron/dhcp_agent.ini?
16:17 dubmusic no.
16:17 dubmusic would you like me to look at it?
16:19 dubmusic I restart some services on the compute node and now I do see dhcp requests even leave the box.
16:19 MiroslavAnashkin Please share the output of `ps -ef | grep dnsmasq` from the machine where neutron-dhcp-agent is currently running.
16:19 dubmusic should it be the one with the qdhcp device?
16:20 dubmusic The one that CRM status says is running dhcp has no qdhcp device
16:21 MiroslavAnashkin No, dnsmasq should be started on the same node as DHCP agent (p_neutron-dhcp-agent(ocf::mirantis:neutron-agent-dhcp):Started node-9 )
16:21 dubmusic There is no dnsmasq service running either
16:21 MiroslavAnashkin So, it is node-9 if the last crm status info still actual
16:22 dubmusic correct.  It is not running on that node
16:23 dubmusic dnsmasq, that is
16:24 dubmusic It did not seem to start on that node.  any ideas?
16:24 MiroslavAnashkin DHCP agent should start dnsmasq with a bunch of input parameters.
16:25 dubmusic should I try to restart it manually?
16:26 dubmusic What is the best method
16:27 MiroslavAnashkin No, it should not be started manually. Normally it gets all the parameters from command line and it is neutron supplies these parameters
16:27 tatyana joined #fuel
16:29 dubmusic Right.  What would be the correct approach to get the service in the correct state?
16:29 justif2 joined #fuel
16:30 mutex you need to restart it through crm, like above
16:30 mutex crm resource restart p_neutron-dhcp-agent
16:31 dubmusic So I can do this from any node and it will apply the correct changes to all the HA controllers
16:31 mutex you can do it from any of the controller nodes
16:31 anotchenko joined #fuel
16:32 MiroslavAnashkin dubmusic: For which network your instances do not getting IP? Is it external or internal network?
16:32 dubmusic internal
16:33 dubmusic I an testing with a single cirrros instance
16:33 dubmusic I can manuallly set IP/GW etc and access it from the public side.  DHCP, however, fails
16:34 dubmusic I see the BOOTP packets go out from the Private interface on the compute node
16:35 mutex but the packets don't get to the dnsmasq server ?
16:36 MiroslavAnashkin Are there free IPs in the internai IP addresses pool?
16:36 dubmusic Yes on the free
16:36 mutex you should use tcpdump on the interface, and make sure you are monitoring the dnsmasq logs on the fuel-master node
16:36 dubmusic Why on the fuel master?
16:37 dubmusic That is not giving out addresses for the cluster…
16:37 dubmusic Or do you think it is interfering
16:37 mutex no no
16:37 dubmusic Here is another odd on
16:37 dubmusic odd one
16:38 mutex but during my debugging, I noticed the fuel-master node has the dnsmasq logs, the node where the process is running doesn't
16:39 dubmusic the dhcp request is going out on the private interface on the compute node.  I think that is correct.  That is where we have configured  our vlans for provate use.  There is also an untagged vlan on that port
16:39 dubmusic Native VLAN
16:39 dubmusic but the dhcp requests arrive on another port on the controllers
16:39 dubmusic The port that is not part of the private
16:41 mutex you want to run tcpdump on the private network interface
16:41 dubmusic from the neutron cleanup script there is this process sleeping also
16:41 dubmusic bash -c sleep 33 ; q-agent-cleanup.py --agent=dhcp --reschedule --remove-dead 2>&1 >> /var/log/neutron/rescheduling.log
16:41 mutex that's ok, are you using fuel 4.0
16:42 dubmusic 4.1
16:42 mutex k, then the ocf bug should be fixed ;-)
16:42 dubmusic hmmm
16:42 mutex the process should disappear when the sleep is complete
16:43 dubmusic now, the compute is sending the dhcp request on the private, but it is arriving on the wrong interface on the controller
16:44 mutex that sounds like your problem then
16:46 dubmusic After running the neutron restart, there are no nodes running the qdhcp device when doing an ip netns
16:47 mutex what do you mean neutron restart
16:47 mutex the neutron-server daemon ?
16:47 dubmusic yes
16:48 MiroslavAnashkin Private network should not have IP addresses. Internal and Private networks are different in Neutron.
16:48 dubmusic Correct.  I have no ips on that network
16:48 dubmusic crm resource restart p_neutron-dhcp-agent
16:49 mutex ah, ok
16:49 dubmusic That was the command I ran
16:49 mutex hmmm
16:49 mutex and all of the dnsmasq processes are gone now ?
16:50 dubmusic dnsmasq is there on two hosts, but not the one that crm marked as being the dhcp host
16:51 dubmusic There are no devices as qdhcp using ip netns on any host
16:51 dubmusic just qrouter
16:53 MiroslavAnashkin Ok, then please go to master node and check /var/log/remote/<node name with DHCP agent>/neutron-agent-dhcp.log
16:55 MiroslavAnashkin It should report why it does not start dnsmasq.
16:56 dubmusic master fuel node?
16:56 dubmusic no
16:56 dubmusic I am on the current master controller and there is no /var/log/remote directory
16:58 MiroslavAnashkin On fuel master node - the one which runs Fuel UI. All the logs by default are configured to be stored on master node
16:59 dubmusic OK.  Got it.  CRM says dhcp should be no node-9
17:01 vkozhukalov joined #fuel
17:02 dubmusic After restarting, this is what I see in the logs you specified.  http://paste.openstack.org/show/74165/
17:03 dubmusic last item is syncronizing state.
17:03 dubmusic still no dnsmasq on that node.
17:06 dubmusic Is it worth rebooting?
17:08 IlyaE joined #fuel
17:11 MiroslavAnashkin Please enable debug mode in /etc/neutron/dhcp_agent.ini on each controller.
17:12 MiroslavAnashkin And run `crm resource cleanup clone_p_neutron-plugin-openvswitch-agent` `crm resource restart clone_p_neutron-plugin-openvswitch-agent`
17:12 dubmusic Will do
17:13 xarses joined #fuel
17:24 mihgen joined #fuel
17:25 angdraug joined #fuel
17:26 xarses dubmusic: where you able to resolve your issue?
17:31 dubmusic Sorr…in a meeting.
17:38 Ch00k joined #fuel
17:39 fandi joined #fuel
17:48 xarses Dr_Drache: you got ubuntu going?
17:48 IlyaE joined #fuel
17:50 Dr_Drache xarses, yes
17:50 xarses huzzah!
17:51 Dr_Drache still trying to sort out dual network adaptors in instances.
17:58 xarses iirc you where having issues with only the first interface receiving an address?
18:06 Dr_Drache yes
18:06 Dr_Drache sorry, stepped away.
18:16 Dr_Drache xarses, yes.
18:16 Dr_Drache same thing again.
18:16 Dr_Drache 4.1 patch v2, ubuntu.
18:20 angdraug joined #fuel
18:28 Dr_Drache xarses, with net04_ext as first adaptor : http://paste.openstack.org/show/74171/
18:29 Dr_Drache xarses, with net04 as first adaptor : http://paste.openstack.org/show/74172/
18:30 e0ne joined #fuel
18:30 Ch00k joined #fuel
18:31 rvyalov joined #fuel
18:33 Ch00k_ joined #fuel
18:38 Ch00k joined #fuel
18:41 IlyaE joined #fuel
18:56 tatyana left #fuel
20:02 MiroslavAnashkin joined #fuel
20:20 meow-nofer joined #fuel
20:47 e0ne joined #fuel
21:08 bogdando joined #fuel
21:10 ruhe left #fuel
21:30 e0ne joined #fuel
21:44 BillTheKat joined #fuel
21:46 BillTheKat joined #fuel
21:53 justif anyone home?
22:05 dubmusic joined #fuel
22:09 dubmusic joined #fuel
22:19 Ch00k joined #fuel
22:22 xarses nope
22:37 justif aww
22:37 justif just as I was leaving for home
22:37 justif well be back soon
22:38 vk joined #fuel
23:23 justif joined #fuel
23:45 justif I have a deployment that failed with Centos on 4.1 with the v2 patch applied and all but one ceph+controller node failed to deploy with an error of could not find partion 3
23:47 justif any ideas? All are Hp dl380 g5's
23:47 justif with the p400 smart array controllers
23:47 justif previous deployments on this node have worked fine in the past
23:52 anotchenko joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary