Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-03-26

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 justif how do i access ceilometer?
00:43 dubmusic joined #fuel
01:14 rmoe joined #fuel
01:37 dubmusic left #fuel
01:37 dubmusic joined #fuel
01:38 dubmusic I am stuck at my cinder volumes hanging at the phase where they are trying to attach to an instance.  They were created successfully, but cannot attach.  I am scouring the logs, but I am not sure that I am looking in the correct place.
01:40 dubmusic Any help would be great
01:40 dubmusic or even a way to kill the ever spinning attempt to attach
01:47 dubmusic I do see the following in the logs …..    nova-nova.compute.manager.log:2014-03-25T20:17:13.873761+00:00 err:  ERROR: Failed to connect to volume c8f20cca-eaad-4289-8393-d72ce5e4fbe7 while attaching at /dev/vdb
01:59 xarses joined #fuel
02:16 justif2 joined #fuel
02:31 vkozhukalov joined #fuel
03:39 dubmusic is there any reason that using this directive in nova.conf on the compute node would not work?    cinder_catalog_info='volume:cinder:internalURL'
04:22 mihgen joined #fuel
05:12 Ch00k joined #fuel
05:18 Ch00k joined #fuel
05:24 dburmistrov joined #fuel
05:44 fandi joined #fuel
05:55 e0ne joined #fuel
06:03 IlyaE joined #fuel
06:19 saju_m joined #fuel
06:43 IlyaE joined #fuel
06:45 dburmistrov joined #fuel
06:48 e0ne joined #fuel
07:02 vkozhukalov joined #fuel
07:03 Ch00k joined #fuel
07:11 ToTsiroll joined #fuel
07:12 ToTsiroll hey guys, im having trouble with mysql..there was a power outage..and now i cant restart the mysql service (Permission denied even in root). My compute services cant connect to the mysql server
07:19 vkozhukalov ToTsiroll: which mode did you use for your cluster during deployment? HA or simple?
07:19 vkozhukalov ToTsiroll: if HA, then all services connect to mysql via HA proxy
07:21 aepifanov joined #fuel
07:23 aepifanov hello guys
07:23 aepifanov I have a problem with FUEL 4.1
07:23 vkozhukalov ToTsiroll: when you use HA mode, mysql instances are replicated via Galera HA service
07:23 vkozhukalov aepifanov: hello
07:24 vkozhukalov aepifanov: what is your problem?
07:24 aepifanov I have 3 nodes and after PXE booting 2 have status "pending addition" and 1 "discovered"
07:25 ToTsiroll oh ok..i get it...but how can i fix the problem regarding nova-compute nodes cant connect to mysql?
07:25 aepifanov I've looked at this node and didn't see at least nailgun-agent in processes
07:26 aepifanov I guess it might be a root cause that node doesn't change status
07:27 vkozhukalov aepifanov: agent is launched periodically by crontab
07:27 vkozhukalov aepifanov: did only one of your three nodes booted into "bootstrap"?
07:28 aepifanov all 3
07:28 mihgen aepifanov: pending addition is when you added node to the env, but have not provisioned it yet
07:28 mihgen if node is out of any environment, it should keep "discover" status
07:28 aepifanov all 3 are added into env
07:29 mihgen actually there is no such status as "pending_addition", it's another field in the DB
07:30 mihgen so there is status and pending_addition fields for Node
07:30 mihgen take a look at node.py in fuel-web repo
07:31 mihgen sorry I have to go, won't help more than it for now..
07:31 vkozhukalov ToTsiroll: try to find out that ha proxy and galera processes are running and try to find out if they listen ports
07:32 mihgen aepifanov: I'm still here for a moment. The file I'm talking about is here: https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/db/sqlalchemy/models/node.py#L85
07:32 mihgen please tell me more about an issue
07:33 vkozhukalov ToTsiroll: if not, try to restart them
07:33 ToTsiroll i did restart haproxy..will try with galera
07:38 aepifanov hm... after the env reseting, all 3 in the pending status ...
07:39 vkozhukalov aepifanov: where you can see their pending status? there are several possible statuses: 'ready',
07:39 vkozhukalov 'discover',
07:39 vkozhukalov 'provisioning',
07:39 vkozhukalov 'provisioned',
07:39 vkozhukalov 'deploying',
07:39 vkozhukalov 'error'
07:40 aepifanov for now, thank you guys and I'll let you know if I'll face it again
07:41 dburmistrov_ joined #fuel
07:41 aepifanov sorry for the messing, I mean node status  in GUI
07:42 ToTsiroll it seems the problem is in the rabbitmq-server
07:42 ToTsiroll i tried restarting it but its doesnt seems to be restarting
07:43 ToTsiroll i opened the log and saw Error Reports (error on AMQP connection : enotconn)
07:43 ToTsiroll it seems to drop all rpc requests
07:54 vkozhukalov ToTsiroll: In HA mode we use RabbitMQ cluster with HA proxy, so you need to ensure that RabbitMQ cluster is up, try to use "rabbitmqctl cluster_status"
07:55 vkozhukalov ToTsiroll: here are some commands which can be helpful https://www.rabbitmq.com/clustering.html
07:55 vkozhukalov ToTsiroll: Fuel HA architecture in general is described here http://docs.mirantis.com/fuel/fuel-4.1/pdf/Mirantis-OpenStack-4.1-ReferenceArchitecture.pdf
07:57 mihgen joined #fuel
08:10 saibarspeis joined #fuel
08:28 ToTsiroll yup ive been using the commands...but i try to stop one node's rabbit..it just hangs there on the stopping notification..it doesnt stop
08:29 ToTsiroll right now 2 of my rabbit are down since i cant seem to stop them or restart them
08:39 ToTsiroll ok i just fixed it...killed all processes of rabbit in the hosts that it hangs...removed it to cluster and added it again..seems to be working now
08:39 ToTsiroll all my compute nodes are up :) thanks for the help
09:06 mihgen joined #fuel
09:09 saibarspeis I am trying since a week ago to install a Fuel cluster for testing and I can't manage to do it
09:09 saibarspeis It never validates the network settings
09:10 saibarspeis I am trying to do it using Neutron with GRE segmentation
09:10 saibarspeis Are there any "special" hints that I don't get from the documentation (which by the way is crap and clearly was written  by at least 10 different people)
09:11 topochan joined #fuel
09:19 ToTsiroll describe your nodes
09:19 vkozhukalov saibarspeis: perhaps the problem is somehow connected with the network equipment you use. when we send probing packets, we add fake IP headers, and 1.0.0.0 as source and destination addresses. And it turn out that some equipment filters out such invalid IP packets. We've just fixed that issue https://review.openstack.org/#/c/80364/
09:20 saibarspeis Ahh
09:20 saibarspeis Ok
09:20 saibarspeis My attempts were both done in virtualized environment
09:20 saibarspeis One in vmware esxi
09:20 saibarspeis And the second on my laptop using Parallels for Mac
09:20 saibarspeis I don't know if parallels filters that in their virtual switch implementation, but they might just do that
09:21 saibarspeis And in ESXi I think I have to enable promiscuous mode
09:21 saibarspeis In order for that to work
09:21 vkozhukalov saibarspeis: ah, maybe I didn't  understand your issue correctly
09:21 saibarspeis You did
09:21 saibarspeis It's just I did not specified
09:21 saibarspeis I am trying to build a <testing platform> of OpenStack with fuel
09:22 saibarspeis So I've built a virtualized environment (because these are the resources I have right now)
09:23 saibarspeis Created the FUEL machine, four network portgroups (so I have four nics on each machine including the FUEL server)
09:23 saibarspeis I've created two new machines (VMs in our case) with eth0 connected to the portgroup on which FUEL serves PXE boot
09:23 saibarspeis and eth1, 2 and 3 to the rest of the port groups
09:23 saibarspeis Let's say the second is public and float
09:23 saibarspeis Third is management
09:23 vkozhukalov saibarspeis: just try to install cluster even it does not mark your network settings as valid
09:23 vk joined #fuel
09:23 saibarspeis And forth is storage
09:24 saibarspeis I did just that :)
09:24 saibarspeis Fuel does not see the nodes after
09:24 saibarspeis The nodes are booted and they are fine (with open stack installed)
09:24 saibarspeis But fuel does not see them
09:25 saibarspeis I can even see ovs-vswitchd running on one of the nodes
09:26 saibarspeis bridges are up on the nodes
09:26 saibarspeis I think some interfaces are not configured in FUEL
09:26 saibarspeis I will go there now
09:31 vkozhukalov saibarspeis: if fuel is not able to communicate with slave nodes at all, it is not connected with probing packets. do you mean it is even not possible to ping slave nodes from fuel master or ssh to them from master?
09:32 saibarspeis Let me try
09:33 saibarspeis No
09:34 rvyalov joined #fuel
09:39 e0ne joined #fuel
09:45 vkozhukalov saibarspeis: yes, please try to find out which interfaces and bridges are up on master as well as on slaves: "ip a" and "ovs-vsctl show"
09:46 saibarspeis I am copying config data now
09:49 saibarspeis I can't copy the data from any nodes
09:49 saibarspeis And I can't find out on what interfaces some bridges are up on
09:49 saibarspeis Because the brctl command is missing from the nodes
09:49 saibarspeis I can see, in my case, the management bridge is called br-mgmt and has IP 192.168.200.2/24
09:50 saibarspeis But I can't find out on what interface is that bridge up
09:51 vkozhukalov saibarspeis: if i remember correctly (i might be wrong) we use ovs bridges, not linux bridges
09:52 saibarspeis OK, is there a way to find out on what interface does a bridge binds to ?
09:52 vkozhukalov saibarspeis: so you need to use "ovs-vsctl list-ifaces br-mgmt" to find out which interfaces this bridge is connected to
09:53 vkozhukalov saibarspeis: or just use ovs-vsctl --help, there are plenty of command which can be helpful
10:15 DaveJ__ joined #fuel
10:37 Ch00k joined #fuel
10:38 anotchenko joined #fuel
11:14 anotchenko joined #fuel
11:21 saju_m joined #fuel
11:23 saibarspeis Thank you very much vkozhukalov
11:23 e0ne joined #fuel
12:00 e0ne joined #fuel
12:05 e0ne joined #fuel
12:09 fandi joined #fuel
12:10 anotchenko joined #fuel
12:11 bogdando joined #fuel
12:14 e0ne joined #fuel
12:23 e0ne_ joined #fuel
12:25 e0ne joined #fuel
12:30 justif joined #fuel
12:33 Ch00k joined #fuel
12:51 MiroslavAnashkin saibarspeis: To get OpenStack network working on virtualized environment it is mandatory set up the following:
12:51 MiroslavAnashkin 1. Enable promiscows mode. Starting from Havana release Neutron does not work without promiscous mode enabled.
12:52 MiroslavAnashkin promiscous
12:54 MiroslavAnashkin 2. Avoid using Intel e1000 emulated network cards. These drivers are optimized to cut off all broadcast/multicast traffic, which it considering unnecessary. It also breaks Neutron. Use default AMD cards or Virtio instead.
12:55 saibarspeis Pfff
12:55 saibarspeis That is exactly what I was using
12:56 jaypipes joined #fuel
12:57 MiroslavAnashkin http://docs.mirantis.com/fuel/fuel-4.1/install-guide.html#virtualbox - These settings are actual for other VM hypervisors.
12:57 saibarspeis MiroslavAnashkin: Thank you, you are awesome! Thank you. I will redeploy everything using this doc
12:57 saibarspeis And this new information
12:58 saibarspeis I have vmware envirnoment and I don't know if I can get AMD cards or virtio
12:58 MiroslavAnashkin Defauld Lance network cards in VMWare are actually AMD
12:58 saibarspeis I have E1000 and VMXnet (2 and 3)
12:59 MiroslavAnashkin Wait a minute, let me check
12:59 saibarspeis This is a vmware platform, not vmware desktop
12:59 saibarspeis We are actually trying to prove Fuel over vCenter and vCloud director
13:02 anotchenko joined #fuel
13:04 MiroslavAnashkin Aah, VMware sets first adapter as e1000 and second and further - as VMXnet
13:06 saibarspeis I can change them
13:06 saibarspeis But these are the only options
13:07 saibarspeis If you say e1000 is not ok ? I will try with vmxnet first
13:07 e0ne_ joined #fuel
13:07 saibarspeis I am now enabling promisc mode on all port groups of this instalation
13:07 MiroslavAnashkin VMXnet require vmware tools installed to guest OS. It may not work
13:09 MiroslavAnashkin VLance should work everywhere. You may change your .vmx file of your machine and set something like ethernet0.virtualDev = "vlance"
13:12 MiroslavAnashkin And same thing for other NICs.
13:19 jobewan joined #fuel
13:55 evg joined #fuel
14:03 IlyaE joined #fuel
14:04 saibarspeis Just so you know
14:04 saibarspeis I used E1000 cards in vmware for the nodes and the fuel server
14:06 saibarspeis But I went trhough all the portgroups and enabled all security access to it (MAC Address change, Promiscuous mode and Forged transmits)
14:06 saibarspeis I redeployed using the same settings
14:06 saibarspeis And the nodes installed open stack and worked
14:06 saibarspeis Fuel delviered me two links to connect to the open stack dashboard
14:06 saibarspeis Those links don't work now because of network filtering but they will soon
14:06 saibarspeis So E1000 is just FINE in vmware for open stack
14:17 Ch00k joined #fuel
14:18 MiroslavAnashkin Yes, it depends on how did they implemented NIC - VMWare platform may have different implementation.
14:19 mattymo saibarspeis, our e1000 emulation has issues in VirtualBox. Emulated hardware can vary from implementation to implementation :(
14:36 e0ne joined #fuel
15:06 angdraug joined #fuel
15:07 mihgen joined #fuel
15:47 justif How would I go about troubleshooting performance issues with deploying from large images? For example I have the cloudbase Server2012 QCOW2 image that is 16GB in size that fails to upload through the horizon interface but works fine from the cli and it took about 17 minutes. Now I am trying to deploy an instance from that image and seems to be taking it's sweet time
15:54 Ch00k joined #fuel
15:58 aglarendil you can try to tune haproxy/apache timeouts and restart corresponding services
15:59 MiroslavAnashkin it is Nova timeout.
16:00 justif what about speeding up the processing time? Any way to reduce the amount of time it takes to process the image? More memory? Disk speed? Network?
16:00 dubmusic joined #fuel
16:00 aglarendil all of them :-)
16:00 justif heh
16:00 justif so the controller/cinder nodes have 8 GB of ram
16:00 aglarendil no, MiroslavAnashkin, it is not nova timeout, because justif says it works just fine from cli
16:01 aglarendil it should be haproxy or apache timeout
16:01 justif with 4 nics, one for the PXE and 3 bonded with the storage/public/mgmt on it
16:07 MiroslavAnashkin Related bugs to your issue https://bugs.launchpad.net/nova/+bug/1280357 and https://bugs.launchpad.net/nova/+bug/1253612
16:08 justif cool, will look into that. Are there recommended specs for controller nodes for support X number of total nodes?
16:09 MiroslavAnashkin Please try this: http://docs.mirantis.com/fuel/fuel-4.1/pre-install-guide.html#node-server-hardware-recommendations
16:14 justif ok will look into that also
16:21 e0ne_ joined #fuel
16:24 dburmistrov joined #fuel
16:51 vkozhukalov joined #fuel
16:52 dubmusic How do I get my ssh keypair into and Ubuntu 12.04 install.  It appears to generate its own on the initial boot
16:53 dubmusic Also, i resolved that issue of the volumes hanging in the “Attaching” status
16:54 angdraug joined #fuel
16:56 rmoe joined #fuel
16:56 dubmusic It appears that the Fuel deployment is configured such that all API access from the cinder nodes hits the publicURL, even with a ….cinder_catalog_info=volume:cinder:internalURL…configured in the nova.conf of the compute node.
16:57 dubmusic I had to add the public VLAN and all is working again.
17:13 angdraug joined #fuel
17:25 dubmusic joined #fuel
17:38 Tatyanka_Leontov left #fuel
17:50 xarses joined #fuel
17:53 IlyaE joined #fuel
17:57 saju_m joined #fuel
18:08 Ch00k joined #fuel
18:10 ruhe joined #fuel
18:27 BigA joined #fuel
18:30 e0ne joined #fuel
19:41 xarses joined #fuel
19:53 alexdepalex joined #fuel
20:21 tatyana joined #fuel
20:29 saju_m joined #fuel
20:56 e0ne joined #fuel
21:16 dubmusic joined #fuel
21:17 larion89 joined #fuel
21:17 dubmusic left #fuel
21:17 dubmusic joined #fuel
21:18 larion89 Hi, I've got a question. is it possible to deploy openstack and get it working properly in a virtual environment based on esxi?
21:22 e0ne joined #fuel
21:22 larion89 i read the irclogs, and i might have found the problem. didnt have promiscious mode on 'accepted'
22:10 dubmusic joined #fuel
22:11 dubmusic left #fuel
22:12 dubmusic joined #fuel
23:14 IlyaE joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary