Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-01-30

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
23:38 dhblaz joined #fuel
23:54 IlyaE joined #fuel
00:13 IlyaE joined #fuel
00:38 TVR_ joined #fuel
01:13 jkirnosova joined #fuel
01:51 xarses joined #fuel
02:18 rmoe joined #fuel
02:20 vkozhukalov joined #fuel
03:49 ArminderS joined #fuel
04:26 jouston__ joined #fuel
04:58 marcinkuzminski joined #fuel
04:59 dhblaz joined #fuel
05:17 ArminderS joined #fuel
05:36 dhblaz joined #fuel
06:25 bookwar joined #fuel
07:29 mihgen joined #fuel
07:49 e0ne joined #fuel
07:57 vkozhukalov joined #fuel
08:09 meow-nofer joined #fuel
08:19 vkozhukalov joined #fuel
08:23 e0ne_ joined #fuel
08:28 meow-nofer as I see, we don't have enough rights to set blueprint priority on Launchpad
08:29 meow-nofer or it's just me?
08:29 meow-nofer I mean, Dima asked to update it, but I can't do it, anyway
08:39 evgeniyl joined #fuel
08:40 mattymo meow-nofer, you can ask me
08:40 mattymo I can set blueprint priorities
08:42 vk joined #fuel
08:46 miguitas joined #fuel
08:46 meow-nofer mattymo, thanks, I'll ask you
08:58 meow-nofer joined #fuel
08:58 vkozhukalov joined #fuel
09:34 evgeniyl joined #fuel
09:52 xarses joined #fuel
09:59 Alexander joined #fuel
10:05 e0ne joined #fuel
10:05 Alexande1 joined #fuel
10:12 xarses joined #fuel
10:28 mihgen joined #fuel
10:39 e0ne_ joined #fuel
10:46 tatyana joined #fuel
10:54 e0ne joined #fuel
10:58 evgeniyl joined #fuel
11:00 vkozhukalov joined #fuel
11:28 e0ne_ joined #fuel
11:30 e0ne joined #fuel
11:46 e0ne joined #fuel
11:58 e0ne joined #fuel
12:00 mrasskazov1 joined #fuel
12:23 TVR_ joined #fuel
12:26 MiroslavAnashkin joined #fuel
12:29 MiroslavAnashkin joined #fuel
12:45 bookwar joined #fuel
12:49 e0ne_ joined #fuel
13:02 evgeniyl joined #fuel
13:15 e0ne joined #fuel
13:16 e0ne_ joined #fuel
13:18 vkozhukalov joined #fuel
13:19 TVR_ morning folks
13:20 e0ne joined #fuel
13:24 MiroslavAnashkin designated: Yes, there is issue with VMware player. Newer Neutron versions require multicast and promiscous mode. It was discovered, certain VMware and VirtualBox virtual NIC drivers has some optimizations and blocking multicast packets. So, we recommend to use quite old AMD PC-Net FAST III virtual network adapters - these do not block traffic.
13:26 MiroslavAnashkin It is almost evening here. We all are scattered around the world.
13:27 TVR_ Where are you in the world?
13:28 MiroslavAnashkin Moscow. Russia.
13:29 TVR_ wow, nice.. the engineer who sits next to me (Stas Alekseev) was in Moscow ~ 5 weeks ago
13:30 MiroslavAnashkin :)
13:30 TVR_ the globe is getting smaller every day
13:37 e0ne_ joined #fuel
13:39 Dr_Drache joined #fuel
13:41 e0ne joined #fuel
13:42 e0ne joined #fuel
13:44 e0ne_ joined #fuel
13:46 e0ne joined #fuel
13:48 e0ne_ joined #fuel
13:50 Dr_Drache hmmm, connection problem it seems
14:13 evgeniyl joined #fuel
14:19 vkozhukalov joined #fuel
14:26 evgeniyl joined #fuel
14:43 Bamberrouse joined #fuel
14:43 Bamberrouse left #fuel
14:54 dhblaz joined #fuel
15:01 Dr_Drache akupko, multi-region (federation) ceph replication is supported in emeperor. setup is manual, and it seems to break fuel
15:08 miguitas joined #fuel
15:15 TVR_ so after repeated tries, including the complete rebuild of the fuel server, a 6 node HA (3 controller + ceph with 3 compute + ceph) fails...
15:16 Dr_Drache TVR_, why are we failing?
15:16 TVR_ seems the ceph OSD's are dropping (ran ceph osd tree to see ~ half are down)
15:16 Dr_Drache yea, noticed ceph and fuel are not good friends.
15:16 TVR_ have not touched any configs... straight up fuel install and it just failed... looking at logs on node-1 now
15:18 TVR_ I am generating a log snapshot and will file a ticket...
15:20 Dr_Drache deeper I get into this the more bug reports I've filed
15:21 Dr_Drache I love the product.
15:23 Dr_Drache just get frustrated when what the marketing/sales tell me it can do, and then "maybe in version 5" LOL
15:24 Dr_Drache frustrated because managers listen to that.
15:24 Dr_Drache LOL
15:25 MiroslavAnashkin TVR_: What Ceph replication factor did you set?
15:26 TVR_ I set 2 (or left default) and with the 6 nodes, I am using 2 x 146G each for the ceph disks
15:26 TVR_ 12 disks total.. os on sepparate 72G
15:27 MiroslavAnashkin TVR_: Oh, this default is specially set for thode, who try Fuel on virtual machines. Please use at least 3
15:27 MiroslavAnashkin thode=>those
15:30 TVR_ I have tried 3 in the past with the same result
15:31 TVR_ are you on support?
15:32 TVR_ or are you part of support?
15:36 Dr_Drache him? I think he's a dev.
15:38 TVR_ Ah, ok... so I have been going and chatting with the source... good to know..
15:38 TVR_ support just picked up my ticket...
15:39 Dr_Drache I'd like to use my 30 day free support, but when I have at least that long to wait for my known errors to be fixed/commited. I don't see the point yet.
15:40 Dr_Drache sorry, I should shutup. had a bad meeting about openstack/ect this morning. :P
15:40 TVR_ The reason I filed the ticket as well was so another set of eyes could look over the plethora of logs as I generated a diagnostic snapshot for them as well...
15:40 Dr_Drache not trying to be a party pooper.
15:44 TVR_ hey.. no worries...
15:44 Dr_Drache i'm browsing the bugs seeing what I can help with. lol
15:45 designated MiroslavAnashkin, thank you for your response.  I am in the process of building everything out on dedicated hardware for lab testing.
15:45 TVR_ so what happened with your meeting? you take flack for not having the cluster up yet?
15:46 Dr_Drache take flack because the product doesn't do what the sales call says it can do.
15:46 Dr_Drache well "easily"
15:47 TVR_ ah, ok... I tried to roll my own for the last two months.... neutron screwed me... and my year end review took a hit for it... big hit
15:47 TVR_ I feel the pain..
15:47 Dr_Drache we have a 10 node KVM cluster.
15:48 Dr_Drache not true HA, but has live migration/etc
15:48 Dr_Drache that can be rebuilt in 2 days from scratch.
15:48 TVR_ the sad part is, in 6 months, I will be an expert at openstack and ceph..... but we will see how well that helps me then
15:49 Dr_Drache same, but in 4 months I need to have a DR site setup with HA. and we were convinced that OS was the direction to go with that.
15:50 TVR_ I hear ya... the centos / kvm part is easy... hell, I can get ceph talking as the backend and talking to a single all in one node for images and storage..... I just can't tie it all together yet.... seems fuel has solved my neutron issues, as that seems to work... while a little weak with ceph... and I am strong with ceph, but do it differently tha
15:50 TVR_ n the fuel team does....
15:51 TVR_ guess I need to be patient to see what they learn from my snapshot logs
15:51 Dr_Drache ohh, and we don't touch centOS usally here.
15:52 TVR_ we use oracle el6 so centos is the closest thing to our standard....
15:52 xdeller joined #fuel
15:52 Dr_Drache no one here has seen the point of "enterprise back-port mess" vs, say ubuntu LTS.
15:52 TVR_ I wouldn't care if I used the ubuntu install.... but that didn't load completely either with the same issues
15:53 rmoe joined #fuel
15:54 Dr_Drache I just wish marketing was truthful about timelines or what is "easy"
15:54 TVR_ I suspect the issue I am having will be with the fact that these servers were previously in another fuel cluster (created a non-HA setup, that worked) and I suspect that because they use ceph-deploy, they need to zap the disks first and instead are having ceph see old data and dropping the OSD because of it
15:54 IlyaE joined #fuel
15:56 TVR_ ok.. looking over the puppet manifest.. they never zap a disk.... I suspect this will be the issue
15:56 TVR_ ceph-deploy disk zap
15:57 TVR_ if the disk had info from a ceph cluster prior.... that needs to be run.... from my experience
15:58 Dr_Drache ahhh
15:58 Dr_Drache brb
15:58 xdeller TVR_: did you checked network connectivity at the moment when half of cluster was dead?
15:58 TVR_ yes.. they are talking over the management network.. yes?
15:58 designated if you don't delete the partition table, then you will most certainly have problems when trying to redeploy a drive for ceph
15:59 TVR_ yes.. exaclt
15:59 xdeller no, over storage segment
15:59 TVR_ exactly
15:59 TVR_ ok.. let me check
15:59 designated if that is not being done, perhaps you could add it to the puppet manifest, and/or suggest that as an included feature
16:01 jouston_ joined #fuel
16:01 TVR_ at no place in the existing recipies, does it do a zap
16:01 designated in my limited experience with fuel so far, there doesn't seem to be much accountability for variance.  it seems to do a handful of specific things well but lacks certain sanity checks and i fully expect those issues to be sorted out as the product is developed.
16:02 TVR_ sure.... and in no way do I critisize it... I am actually quite impressed...
16:02 xdeller TVR_: generally there _can_ be a drop of connectivity in most intensively used network (storage one) but it requires verification at least
16:02 TVR_ yes.. looking.. testing now
16:03 xdeller had you see oom killer events there?
16:03 designated 4.0 is leaps and bounds beyond the version of fuel i first tried.  I remember editing all of these configuration files, not having a full understanding of all of the options and never could get it to work quite right.
16:04 rmoe a zap shouldn't be necesssary as prior to provisioing the partition table is cleared as well as any existing lvm or md metadata on all disks
16:04 designated i think it's awesome that they've opened it up to the community, i just wish more people took an active interest.
16:04 TVR_ yes.. killer event.. it would seem my storage network is not responding...
16:04 TVR_ good catch... let me adress this
16:06 designated I'm not contributing to the solution, just expressing my opinion so I'll be quiet :)
16:07 xdeller TVR_ can you please check if it is switch connectivity issue or just local node` one? please do /etc/init.d/openvswitch restart
16:07 rvyalov joined #fuel
16:08 xarses joined #fuel
16:13 TVR_ it's my switch... will need to rectify this
16:20 xdeller how did you checked this? by switch restart?
16:20 TVR_ by switch is dead after ping failed
16:21 TVR_ went to switch.. no lights
16:21 TVR_ heh
16:21 e0ne joined #fuel
16:22 TVR_ storage ~was~ on a sepparate network... but looks like I need to move it..
16:23 MiroslavAnashkin TVR_: OMG, if you ran Verify Network settings on network tab in Fuel UI and it has passed - then it is definitely Fuel bug.
16:24 TVR_ well.....it wouldn't let me verify until there were nodes in it... and so after I didn't do that...
16:24 TVR_ so.. yea.. my bad...
16:25 MiroslavAnashkin TVR_: Yes, it require roles assigned to nodes.
16:25 TVR_ a switch failing isn't exactly something that happens often
16:27 MiroslavAnashkin TVR_: Yes, but we had 3 switch and one router failures in last 3 month. All were caused by improperly working WiFi access point.
16:36 TVR_ ok.. so I am re-deploying... without vlan tags (I have enough nics and makes it easy) and each designation has it's own nic.... I have a good feeling about this one
16:37 TVR_ You being not from support, I and I am sure others appreciate you kicking up support in here...
16:47 MiroslavAnashkin I am between dev and support. Like second line.
16:48 TVR_ ah.. ok.. cool... well, we apretiate your help here.
16:48 dhblaz joined #fuel
17:02 Dr_Drache hmm
17:13 MiroslavAnashkin Dr_Drache: Yes, I remember. Your patch is expected (at least vkozhukalov said it is) tomorrow.
17:14 Dr_Drache MiroslavAnashkin, that's fine, trying to sort out networking with another DHCP server on my external, and ceph replication.
17:18 rmoe joined #fuel
17:18 Dr_Drache seems there is no easy way to use fuel on a live network, unless I segment out my live DHCP.
17:20 designated put the fuel interface in a separate VLAN, create an SVI to route traffic between VLANs and don't configure a help address.  completely separate that layer two segment.
17:20 designated helper* address
17:21 mihgen joined #fuel
17:21 MiroslavAnashkin Dr_Drache: Yes, we are going to get rid of DHCP in external network, but currently no DHCP should exist in all Mirantis Openstack related network segments.
17:22 Dr_Drache MiroslavAnashkin, what I am saying, I could be saying it wrong.
17:22 Dr_Drache but I cannot deploy on a live network. i'd have to deploy, then make the network live.
17:22 designated i intend to separate the segment used for PXE booting and add an additional interface for interacting with fuel.  does the web server by default listen on all configured interfaces?
17:23 Dr_Drache fuel and PXE both run from the same interface.
17:23 Dr_Drache it can listen on the other for interaction.
17:23 Dr_Drache BUT... your cluster will be dead to the world
17:24 designated why will the cluster be dead to the world, public/flotaing ip networks will be on a different interface
17:24 designated floating*
17:25 MiroslavAnashkin designated: Fuel master node web UI is on 8000 port by default. Cobbler UI is on default port (80 I think)
17:25 Dr_Drache designated, I don't understand.
17:26 Dr_Drache PXE to deploy, needs dhcp from fuel.
17:26 designated traffic from the nodes does not have to traverse fuel
17:26 Dr_Drache your saying you'll have a PXE only interface for deploy, then deploy without usign that interface?
17:26 MiroslavAnashkin designated: So, yes, it is possible to separate UI from PXE
17:27 dhblaz joined #fuel
17:27 designated two interfaces on fuel, one for pxe and deployment to nodes, the other interface will be used for management of fuel and fuel to communicate with the internet
17:27 Dr_Drache but what about the nodes.
17:28 designated the controllers and compute nodes themselves will have separate interfaces trunked to switch with VLAN tagging
17:28 Dr_Drache ahhh, the part that doesn't work.
17:28 MiroslavAnashkin designated: But you have to configure Fuel API, which is by default is on the same 8000 port to the same interface with PXE
17:28 Dr_Drache at least with ceph.
17:28 designated 3 interfaces on nodes, a single 1Gb interface for pxe/fuel interaction, two 10Gb NICs, one for storage network, other for all other traffic.
17:29 designated fuel api and whatever web server is being used are separate processes, no?
17:29 Dr_Drache no
17:30 Dr_Drache they are deeply intergrated.
17:30 Dr_Drache as far as I can tell so far.
17:30 designated so you're saying adding an additional interface to fuel will cause all fuel services to run that interface as well?
17:31 mihgen joined #fuel
17:31 designated I don't see why you can't just listen for tcp/80 on the additional interface
17:32 Dr_Drache you can, but you'd have to edit the api
17:32 designated MiroslavAnashkin, already said it is possible to separate UI from pxe
17:32 Dr_Drache if you edit the api.
17:32 Dr_Drache so possible yes. likly? that's the question
17:32 designated i'm not familar enough with the api to argue one way or the other
17:33 designated if what i intended to do was based on misinformation or a misunderstanding then I'll have to figure out another way.
17:33 MiroslavAnashkin designated: Correction - API is not necessary for PXE and should be moved alongside with Fuel UI
17:33 angdraug joined #fuel
17:34 designated so it's possible to run the ui and api on both interfaces while limiting PXE to a single interface?
17:41 Dr_Drache MiroslavAnashkin, radosgw-agent, is that looking to be supported soon?
17:51 e0ne joined #fuel
17:53 miguitas joined #fuel
18:11 vk joined #fuel
18:20 angdraug Dr_Drache: not until we upgrade to a version of Ceph that has it, probably around the time we upgrade to IceHouse
18:25 Dr_Drache angdraug, so fuel 4.0 isn't on emperor?
18:27 MiroslavAnashkin designated: You may change /etc/nginx/conf.d/nailgun.conf to set Fuel UI listen on any port or interface.
18:28 xdeller no, it is based on dumpling
18:29 angdraug emperor is not an LTS release, we'll stay with dumpling until firefly, the next LTS release, is out and stable
18:31 designated MiroslavAnashkin, thank you
18:33 Dr_Drache well, that's on me. I thought since it's based on havanna, all parts of the puzzle would be updated.
18:35 TVR_ amazing how with the right networks it installed correctly... excellent...
18:35 Dr_Drache sweet TVR_
18:37 e0ne joined #fuel
18:45 angdraug well we did update it from cuttlefish :)
18:45 Dr_Drache heh.
18:56 xarses joined #fuel
19:19 vkozhukalov joined #fuel
19:31 e0ne joined #fuel
19:47 TVR_ So, when a cluster is built correctly, and I take the default for the internal network of 192.168.111.0/24 ... my new instances should be getting assigned an IP from this subnet, yes?
19:47 TVR_ assuming I choose network net04 and not the ext network that is...
19:48 TVR_ from the node a service network restart doesn't get an IP
20:09 IlyaE joined #fuel
21:01 jouston__ joined #fuel
21:07 tsduncan joined #fuel
21:34 TVR_ I can launch instances fine.. and everything looks good.. just the last thing it seems is the dhcp isn't getting to my instance... I checked security groups and it looks good as they are getting the default
21:34 Dr_Drache is that a vlan issue?
21:35 Dr_Drache the DHCP isn't on the same vlan as the instances "external"
21:35 TVR_ the default gave me two networks...net04 and net04_ext
21:35 TVR_ the dhcp should be from net04 ..... correct?
21:36 TVR_ net04 is connected to the subnet 192.168.111.0/24 as I took the defaults there...
21:36 Dr_Drache net04_ext is your external network (non-cluster) net04 should be floating from your clusterpool
22:23 designated joined #fuel
22:31 richardkiene_ joined #fuel
22:52 designate joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary