Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-03-25

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:19 IlyaE joined #fuel
00:45 justif must have been a fluke
00:45 justif deploy worked fine this time
00:51 fandi joined #fuel
02:00 ogelbukh joined #fuel
03:35 vkozhukalov joined #fuel
04:21 xarses joined #fuel
05:10 dburmistrov joined #fuel
05:15 vkozhukalov joined #fuel
05:24 Ch00k joined #fuel
05:29 mihgen joined #fuel
05:31 IlyaE joined #fuel
05:49 e0ne joined #fuel
07:03 vkozhukalov joined #fuel
07:11 anotchenko joined #fuel
07:24 StackKid joined #fuel
07:27 StackKid hi folks, need some help with Fuel 4.0 on virtualbox on Mac OSX 10.9.2. After rebooting the fuel master-node VM, I can't ping 10.20.0.2 from my host machine (10.20.0.1), can't get an ARP entry from the host machine. Seems to be related to Vbox network setting but appreciate any pointers.
07:28 StackKid Seems that other people ran into this issues from reading previous msges and the suggestion was to install net-tool but can't seem to find such info for Mac
07:29 StackKid hi fuel devs, need some help with Fuel 4.0 on virtualbox on Mac OSX 10.9.2. After rebooting the fuel master-node VM, I can't ping 10.20.0.2 from my host machine (10.20.0.1), can't get an ARP entry from the host machine. Seems to be related to Vbox network setting but appreciate any pointers.
07:30 saju_m joined #fuel
07:30 mihgen StackKid: hi
07:30 mihgen can you try to login from console to master node?
07:31 mihgen and verify that your eth0 still has 10.20.0.2
07:31 StackKid yes indeed
07:31 mihgen also, on you host machine, do you have 10.20.0.1 on vboxnet0 ?
07:31 StackKid ifconfig -a shows eth0 has status UP and ip 10.20.0.2
07:31 StackKid yes on host machien vboxnet0 is there
07:32 mihgen what about IP?
07:32 StackKid i'm rerunning the script to re-create my fuel environment from scratch but this issue comes up EVERYTIME
07:32 StackKid host machine : vboxnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ether 0a:00:27:00:00:00  inet 10.20.0.1 netmask 0xffffff00 broadcast 10.20.0.255
07:33 mihgen then, if you go to settings of your master node VM, network configuration should have bridged network and use vboxnet0 as first adapter
07:33 mihgen weird
07:33 mihgen why do you reboot master node actually?
07:33 StackKid very weird
07:33 StackKid this issue seems to occur even without rebooting
07:34 StackKid all of the sudden i can't connect/ping or login to FUEL GUI anylonger
07:34 StackKid I checked to make sure I don't have any conflicting networks/IP addresses on my setup ;-)
07:35 StackKid yes the VBOX manager shows VNIC/adapter setting correctly
07:35 mihgen the only thing I could imagine is that you have some conflicting network… but since you don't, I'm lost :)
07:36 StackKid also another question: i should be able to shutdown all the VMs (fuel master node, slave node 1, 2,3) and bring them back online and all of my environments should be saved and brought up as the last state right?
07:36 StackKid can't see to be able to do that with my setup. That is strange
07:36 StackKid im puzzled
07:36 mihgen yes, it supposed to work as such
07:37 mihgen unless you sort out this simplest IP thing nothing else will work
07:37 StackKid i need to try this on a Linux box. i suspect this has to do with Virtualbox on Mac
07:37 StackKid anyone successfully run this on VBOX on Mac OS X?
07:37 mihgen might be version of vbox ?
07:37 mihgen I do
07:37 mihgen and many more folks
07:37 StackKid ok that's good to know
07:37 mihgen I'm on mac 10.8.4
07:38 mihgen vbox 4.2.12 ..
07:38 StackKid i have VMWare fusion which shouldn't conflict with vboxnets and I double/tripple checked to make sure I have no conflicting IPs
07:38 StackKid there gotta be something really simple ..and I'm missing it
07:39 StackKid i'm on 4.3.8 R92456
07:39 StackKid bug on VBOX?
07:39 mihgen could be..
07:40 StackKid ok i'm going to try the 4.2.12 build
07:40 StackKid thanks @mihgen
07:40 mihgen np :)
07:40 StackKid much appreciated!
07:41 mihgen I have a god feeling that it's rather anything else, like vmware, and not vbox version..
07:52 StackKid i'm trying it on VBOX 4.2.24 build
07:52 StackKid will let you know. I think you're right. Just doing the usual elimination process
07:53 StackKid I might end up doing it on another fresh machine with VMWare fusion
07:53 Ch00k joined #fuel
07:54 mihgen ok. btw, why you are running 4.0, not 4.1 ?
07:56 StackKid haven't grabbed the 4.1 yet ..will do it
07:57 StackKid ran 4.1 on another mac w/ Fusion 6.0 and saw the same behavior
08:03 bogdando joined #fuel
08:08 evg joined #fuel
08:10 e0ne joined #fuel
08:12 stef____ joined #fuel
08:15 fandi joined #fuel
08:16 jkirnosova joined #fuel
08:52 saju_m joined #fuel
08:57 topochan joined #fuel
09:01 DaveJ__ joined #fuel
09:08 rvyalov joined #fuel
09:27 vk joined #fuel
09:34 dburmistrov joined #fuel
09:55 tatyana joined #fuel
09:56 saju_m joined #fuel
10:08 dburmistrov_ joined #fuel
10:13 Ch00k joined #fuel
10:16 e0ne_ joined #fuel
10:21 aglarendil joined #fuel
11:01 mihgen joined #fuel
11:08 saju_m joined #fuel
11:15 saju_m joined #fuel
11:33 e0ne joined #fuel
11:38 saju_m joined #fuel
12:12 e0ne joined #fuel
12:13 dburmistrov_ joined #fuel
12:13 dubmusic joined #fuel
12:14 e0ne joined #fuel
12:18 aglarendil|works joined #fuel
12:20 aglarendil joined #fuel
12:22 justif joined #fuel
12:23 Dr_Drache joined #fuel
12:51 topochan joined #fuel
12:55 saju_m joined #fuel
12:56 AndreyDanin joined #fuel
12:57 Dr_Drache joined #fuel
13:00 Gil_McGrath joined #fuel
13:00 saju_m joined #fuel
13:05 saju_m joined #fuel
13:10 saju_m joined #fuel
13:42 Ch00k joined #fuel
13:44 fandi joined #fuel
13:44 topochan Hi, I have install openstack with fuel 4.1 and there is not swift option in horizon, I use the default cinder and glance options but I cannot see the swift menu and non api access in horizon, is it normal?
13:47 Dr_Drache topochan, it sounds like you didn't enable swift.
13:47 Dr_Drache but other than that, i cannot help because I use CEPH.
13:48 MiroslavAnashkin topochan: What option did you selected for Glance backend when you created new OpenStack environment in Fuel UI - Swift or Ceph?
13:50 MiroslavAnashkin You may also check your Settings tab of the already deployed environment - there is appropriate check box for Glance backend
13:57 baboune joined #fuel
13:57 baboune Hi.  When using Fuel to setup OpenStack, is i tpossible to enable IP v6?  And how?
13:59 topochan MiroslavAnashkin: I check swift to glance, I am not using ceph
14:01 MiroslavAnashkin baboune: OpenStack does not support IPv6 so far
14:05 baboune u mean this is true: http://www.nephos6.com/blog/?p=160?
14:15 Dr_Drache joined #fuel
14:22 MiroslavAnashkin Yes, correct IPv6 work currently is possible with Nova-network only. But there is progressin Neutron.
14:24 baboune documentation of havana says it is doable with OVS, vlan and FlatDhcpManager
14:25 baboune http://docs.openstack.org/admin-guide-cloud/content/section_configuring-compute-to-use-ipv6-addresses.html
14:25 baboune oooh, I see
14:26 baboune so not with Neutron then...
14:37 baboune when using Fuel and setting up OpenStack it should be possible to select nova-network and then setup IP v6 or?
14:42 MiroslavAnashkin No. currently there is no settings for IPv6 in Fuel
14:52 BillTheKat joined #fuel
15:00 IlyaE joined #fuel
15:05 jobewan joined #fuel
15:13 topochan then if I don’t using CEPH, why I don’t have swift ? in the storage settings option only I have checked the cinden LVM over iSCSI
15:13 justif why would ceph nodes in a deployment show no OSD's after a fresh deployment?
15:13 justif http://paste.openstack.org/show/74259/
15:14 justif http://imgur.com/a/uRHra
15:19 e0ne joined #fuel
15:21 MiroslavAnashkin topochan: Please go to Horison Admin, select System info and check if there is Swift enabled for object store. After that, switch to Project and check Opject Store part is available
15:21 MiroslavAnashkin Horizon
15:22 MiroslavAnashkin justif: Am I right, you have deployed to HP hardware with P200/P400 RAID controller?
15:22 justif correct
15:22 justif 3 ceph nodes with p400
15:23 asledzinskiy joined #fuel
15:23 MiroslavAnashkin It is critical bug. We expect patch today or tomorrow.
15:23 topochan there isn’t a swift service in the service list
15:24 justif ok, willing to test if needed.
15:25 topochan I think it’s not installed
15:25 topochan :S
15:27 Ch00k_ joined #fuel
15:37 MiroslavAnashkin topochan: What is your env - Ubuntu or CentOS, and HA or non-HA?
15:38 Dr_Drache joined #fuel
15:42 BillTheKat joined #fuel
15:54 topochan non-HA and CentOS
15:54 topochan epicadmin
15:54 topochan sorry MiroslavAnashkin
15:57 vk joined #fuel
15:59 topochan I set vlan-manager, kvm, cinder lvm and glance default that it’s swift
16:00 topochan ok, I had read this : By default, Glance image service uses Swift object storage in HA deployment mode, and local storage on the Controller node in simple multi-node mode
16:00 topochan that means taht swift it’s not installed in multinode?
16:09 fandi joined #fuel
16:11 angdraug joined #fuel
16:28 dburmistrov joined #fuel
16:31 saibarspeis joined #fuel
16:53 rmoe joined #fuel
17:08 xarses joined #fuel
17:10 MiroslavAnashkin Actually Swift is always installed, but not used and not configured in non-HA configurations. Swift require several swift nodes to work properly.
17:10 topochan ok, thanks for the help and the information
17:35 Tatyanka_Leontov joined #fuel
17:37 Dr_Drache joined #fuel
17:41 justif http://paste.openstack.org/show/74281/ are lines 287 - 308 normal? Deploying the TestVM images in a fresh deployment
17:46 MiroslavAnashkin Yes, it is normal. Cirros image supports ec2 api. Since it is started inside the environment without Amazon EC2 API enabled - it fails to get its inctance ID.
17:47 justif the same happens to a Centos images and it seems the process to change the name of the host to the instance name fails also
17:47 justif the hostname is host-10-0-0-7
17:51 MiroslavAnashkin Please check your metadata service is configured correctly. Here is guide which also contains how to set inctance name part http://docs.openstack.org/user-guide/content/user-data.html
17:53 Dr_Drache joined #fuel
17:58 dubmusic joined #fuel
17:58 dubmusic Does anyone know why a cinder volume creation would be stuck in creating?
17:59 dubmusic I cannot delete either.
18:00 angdraug dubmusic: check your rabbitmq and mysql clusters
18:00 dubmusic OK.  What should I be checking?
18:01 angdraug rabbitmqctl list_queues |grep cinder
18:01 angdraug could be https://bugs.launchpad.net/fuel/+bug/1285449
18:01 MiroslavAnashkin cinder quota?
18:01 angdraug see if there's a buildup of messages in the scheduler queues
18:02 dubmusic Yes sir, there is
18:02 angdraug try to restart all your cinder services
18:02 angdraug ha?
18:02 angdraug MiroslavAnashkin: how do you check if its due to quota?
18:02 dubmusic http://paste.openstack.org/show/74283/
18:03 justif MiroslavAnashkin, Seems there was an issue with the net_04 network created. Deleted it the default router, created them again and launched a new instance and seems good to go now
18:03 angdraug dubmusic: lost/rebooted any controller nodes recently?
18:03 dubmusic Yes
18:04 dubmusic Tracking down a messed up vlan config on a switch
18:04 angdraug ah, in that case restarting openstack services one by one should help
18:04 dubmusic restart cinder on the controllers or  cinder nodes
18:04 dubmusic ?
18:05 angdraug on all of them
18:05 owentl joined #fuel
18:05 angdraug if that's the problem I linked above, it would be python interpreter being locked up in stuck tcp connections to rabbitmq (or mysql or both)
18:06 angdraug if you're reluctant to blanket restart them all, check which ones are stuck with strace
18:06 angdraug they would be sitting on a read() call
18:07 angdraug you can then check the fd against /proc/<pid>/fd and lsof to see which connection is stuck
18:07 Ch00k joined #fuel
18:08 Dr_Drache could I bother someone, I just want to be pointed in the direction that I need to look about my network devices, or maybe i'm just an idiot.
18:09 dubmusic restart api, scheduler and volume?
18:09 angdraug dubmusic: yes
18:09 dubmusic Is there an order they should be started in?
18:09 angdraug you might also check up on other openstack services, nova or neutron is vulnerable to the same problem
18:09 angdraug no, any order
18:10 dubmusic OK restarting.  Will let you know.
18:10 vkozhukalov joined #fuel
18:10 mswynnex joined #fuel
18:11 angdraug the queues should clear up almost immediately after restart
18:14 dubmusic OK.  All good
18:14 dubmusic Thank you
18:15 dubmusic neutron failed to start the services properly, so I restarted them… Could this affect swift?
18:18 dubmusic angdraug:  it now is hanging on attaching… same issue?
18:19 angdraug probably nova and neutron are stuck in the same manner
18:19 angdraug not sure about swift, don't think it has anything to do with rabbitmq or mysql
18:21 xarses joined #fuel
18:21 dubmusic What is the best to bounce nova and neutron?
18:25 Ch00k joined #fuel
18:31 mswynnex Is there any way to deploy a cluster with Fuel 4.1 using Ubuntu plus the older 3.2 kernel that seems to be included?
18:32 Dr_Drache do you have a special case in where a old kernel is required?
18:37 mswynnex Yes, older hardware. The 3.8 kernel does not support my storage. When I boot a node using 3.2 everything works great, except that deployment hangs.
18:41 e0ne joined #fuel
18:43 mswynnex Bascially I start deployment, Ubuntu installs, the node has storage errors when it boots following installation so I reboot it and at the GRUB prompt I boot from kernel 3.2 which works great, no storage errors. FuelWeb then reports that Ubuntu is installed, but once it gets to the Openstack installation it hangs if the node is running on kernel 3.2.
18:44 Dr_Drache whats the storage?
18:47 BillTheKat joined #fuel
18:48 mswynnex mptsas - LSI 1064e SAS controller.
18:54 dubmusic angdraug: I have restarted all of the services that I can, an it is still hanging on the attachment of the volume to the instance
18:54 dubmusic any thoughts?
18:59 vk joined #fuel
19:07 justif I am having some issues with windows instances, I have tried to upload the cloudbase kvm qcow2 image and it seems to fail and I cant see why
19:07 justif any ideas of wehre to look?
19:10 dubmusic Does the controller node need any access to the storage (cinder) nodes on anything other than the management?
19:11 dubmusic The docs state that it does not
19:15 dubmusic Does anyone know if the controllers need access to the storage network.  I see tons of arp requests on this interface
19:15 Dr_Drache yes
19:15 Dr_Drache they do
19:16 Dr_Drache all nodes need access to storage, managment, admin
19:20 dubmusic what are  the controllers doing on the sotrage network, out of curiousity?
19:20 dubmusic don’t they just instruct the compute nodes to attach to storage vi  admin?
19:22 Dr_Drache no
19:22 Dr_Drache they also control storage.
19:25 * angdraug is back
19:25 angdraug dubmusic: what exactly is it hanging on?
19:26 angdraug what about nova service-list and cinder service-list, any services reported down?
19:26 angdraug any stack traces in nova or cinder logs?
19:33 dubmusic Well, I just added the VLAN for storage to the compute nodes, and there is lots of traffic
19:34 dubmusic I do see this in the logs on the controller node-8 cinder-cinder.volume.manager INFO: Updating volume status
19:35 dubmusic …and the this… WARNING: Unable to update stats, driver is uninitialize
19:35 dubmusic services list looks clean
19:39 dubmusic http://paste.openstack.org/show/74287/
19:45 dubmusic I cannot delete the volume either
19:50 dubmusic angdraug:  This does not look right either…http://paste.openstack.org/show/74289/
19:51 angdraug no, that doesn't look right
19:51 angdraug does that vg exist on node-8?
19:53 dubmusic no.  I see a VG called ‘image’ when I run vgs
19:54 angdraug is that a cinder node?
19:55 angdraug did you configure cinder lvm or ceph for cinder?
19:55 dubmusic cinder lvm
19:55 angdraug fuel provisioning script should have created a cinder vg
19:56 dubmusic wait…there is one, on the cinder server   :   volume-0d5eef62-c416-4b26-9e5d-d13c0513269a cinder
19:56 angdraug ah wait a second
19:56 angdraug node-8 is a controller?
19:56 dubmusic it seems to be looking locally for this
19:57 dubmusic yes
19:57 angdraug you shouldn't have cinder-volume running on controller if you're using cinder-lvm
19:57 angdraug cinder tried to schedule your volume to the wrong node
19:58 dubmusic should it be shut off?
19:58 angdraug make sure you have cinder-volume running on your cinder nodes, stop it everywhere else
19:59 dubmusic OK
19:59 angdraug any node that has cinder-volume service running will be scheduled to create volumes
19:59 angdraug but nodes that don't have cinder vg can't host volumes
20:00 dubmusic I am surprised that it let me start it
20:01 dubmusic It starts automatically
20:02 dubmusic man, how can I cancel this attachment?
20:03 angdraug can you just delete that volume? or you want to keep it?
20:04 dubmusic I could delete, but it says I do not have permission
20:05 dubmusic also, when I killed cinder volume, cinder-manage reports those services as down
20:06 Matt_V joined #fuel
20:07 IlyaE joined #fuel
20:11 vk joined #fuel
20:15 e0ne joined #fuel
20:19 Matt_V Would anyone in here have some idea why qrouter namespaces wouldn't be getting created for new routers?  Existing routers work correctly however new routers aren't getting namespaces
20:25 xarses joined #fuel
20:28 dubmusic angdraug: I still see this in the rabbit queues actually…….cinder-scheduler_fanout_0940d514a9274bdbb2ad9c7042c016ac 638
20:29 dubmusic Looks like there are 638 messages in the queue
20:36 dubmusic Does anyone know how I can process the 638 messages sitting in Rabbit for cinder?
20:37 dubmusic It’s drivnig me crazy
20:45 angdraug it's fine for cinder-manage to report them down for nodes where they're not supposed to be running
20:47 angdraug tried cinder force-delete <volume>?
21:43 IlyaE joined #fuel
22:52 vk joined #fuel
23:01 justif joined #fuel
23:41 dubmusic joined #fuel
23:55 dubmusic_ joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary