Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-01-28

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
23:05 e0ne joined #fuel
23:22 kpimenova joined #fuel
23:47 meow-nofer_ joined #fuel
23:47 xarses_ joined #fuel
23:49 vt102 joined #fuel
23:52 Bomfunk joined #fuel
02:15 vkozhukalov joined #fuel
02:33 rmoe joined #fuel
04:12 designated joined #fuel
04:14 designated I have installed Fuel and when trying to create an environment if I select anything other than nova-network it gives me a message "Cannot find suitable CIDR".  Anyone know why?
04:31 bioma_000__ joined #fuel
05:02 Arminder joined #fuel
05:26 IlyaE joined #fuel
05:50 MiroslavAnashkin joined #fuel
05:51 meow-nofer_ joined #fuel
06:32 AndreyDanin_ joined #fuel
06:50 mihgen joined #fuel
07:20 mrasskazov joined #fuel
07:23 vk joined #fuel
07:33 vkozhukalov joined #fuel
08:18 steale joined #fuel
08:22 meow-nofer_ joined #fuel
08:25 mihgen joined #fuel
08:27 e0ne joined #fuel
08:45 e0ne joined #fuel
09:17 steale joined #fuel
09:28 e0ne joined #fuel
09:42 ArminderS joined #fuel
09:43 Arminder guys, why do we have small wait time for instance creation to wait for volume to be created from image?
09:44 Arminder VolumeNotCreated: Volume d919db31-b1f5-440f-919f-2355c0ac45cf did not finish being created even after we waited 70 seconds or 60 attempts.
09:44 Arminder just 70 seconds?
10:06 e0ne_ joined #fuel
10:37 vkozhukalov joined #fuel
10:50 jouston joined #fuel
10:55 mihgen joined #fuel
12:24 TVR_ joined #fuel
12:46 miguitas joined #fuel
12:51 Dr_Drache joined #fuel
13:27 MiroslavAnashkin designated: Have you customized network configuration during the Fuel installation?
13:28 Alexander joined #fuel
13:52 Dr_Drache MiroslavAnashkin, I have a cluster finally deployed.
13:53 Dr_Drache with GRE.
13:53 MiroslavAnashkin Arminder: It is Openstack default timeout. You may change timeouts in nova.conf. Please see example here: https://github.com/openstack/nova/blob/master/etc/nova/nova.conf.sample
13:53 TVR_ sweet! I figured out my issue from yesterday as well
13:53 Dr_Drache no combination of VLANs would deploy with Ceph
13:53 Dr_Drache TVR_,
13:55 Dr_Drache hate to ask this
13:55 MiroslavAnashkin Arminder: Or, here - for Havana https://github.com/openstack/nova/blob/stable/havana/etc/nova/nova.conf.sample
13:55 Dr_Drache how do i "know" my ceph is working?
13:56 TVR_ want me to field this question?
13:56 Dr_Drache lol, it's a stupid question. but i've been up all night with this.
13:56 TVR_ from a compute node...
13:57 Dr_Drache prob sleep would do me some good
13:57 TVR_ rados df
13:57 TVR_ rados lspools
13:57 TVR_ rados -p rbd ls
13:57 TVR_ that tells if you can connect
13:57 TVR_ from a compute node:
13:57 TVR_ cinder create --display_name vm80G 80
13:58 TVR_ cinder list
13:58 MiroslavAnashkin Dr_Drache: Are you use Ceph as both Glance and Cinder backends? Then, simply launch instance from test image, which was uploaded in scope of Openstack installation.
13:58 TVR_ see if it's there
13:58 Dr_Drache MiroslavAnashkin, yes.
13:58 Dr_Drache all the fuel tests work
14:01 Dr_Drache thanks guys
14:01 Dr_Drache but yea, even opened a new dell 48 port switch
14:02 Dr_Drache even with proper vlan in the switch, still couldn't get that error to go away
14:02 TVR_ cool.. you a master of vlans on that bad boy now?
14:03 Dr_Drache lol
14:03 Dr_Drache never.
14:03 Dr_Drache at least that my bosses know :P
14:06 Dr_Drache TVR_, crazy error i get from your cinder command.
14:06 Dr_Drache but i'm set for now.
14:06 TVR_ yes?
14:07 TVR_ that should create a volume
14:07 TVR_ you do need to set your exports though
14:07 TVR_ for auth and such
14:08 Dr_Drache gives a python error.
14:09 TVR_ I will check it once my env is back up
14:11 TVR_ my issue is with there being too many networks here.. so I had to move my cluster and fuel node
14:12 Dr_Drache yea, mine is segrated.
14:12 Dr_Drache hmm..
14:12 Dr_Drache won't let me upload an image
14:16 Dr_Drache yea for testing,
14:16 Dr_Drache and for coffee
14:18 Dr_Drache "The webpage at http://10.20.0.12/dashboard/project/images_and_snapshots/create/ might be temporarily down or it may have moved permanently to a new web address.
14:18 Dr_Drache Error code: ERR_ACCESS_DENIED"
14:29 MiroslavAnashkin Dr_Drache: please run `source openrc` before running any nova/cinder related commands. openrc is located in /root
14:30 Dr_Drache ahhh, that's good to note.
14:30 Dr_Drache silly little things.
14:31 Dr_Drache just have to bend my will to horizon.
14:31 Dr_Drache like... vnc.
14:31 Dr_Drache lol
14:32 Dr_Drache question, it says my volume limit is 10...
14:33 Dr_Drache is that something I messed up?
14:38 RelayChatInfo joined #fuel
14:38 RelayChatInfo left #fuel
14:39 TVR_ when I deploy a cluster using HA of 6 nodes... I get an Error "Not enough free IP addresses in pool"
14:40 TVR_ what pool is that refferencing?
14:41 MiroslavAnashkin Yes, it is default Openstack limit. You may change limits with `nova quota*` and `cinder quota*` commands
14:41 TVR_ I have 30 floating... 50 public, 35 discovery and 35 for install....
14:41 MiroslavAnashkin TVR_: 35 discovery and install
14:42 MiroslavAnashkin TVR_: Fuel require IP from Admin segment on each NIC
14:42 TVR_ ok.. these are the first and ONLY 6 nodes I am or have tried to install on this fresh new fuel server
14:42 TVR_ for each nic.. OK.. makes sense then
14:43 Dr_Drache MiroslavAnashkin, cool, hanks
14:43 TVR_ 4 nics.. / server.. 6x4=24 should be enough.. yes?
14:43 MiroslavAnashkin TVR_: Yes, 6 nodes*4NICs require 24 IP addresses
14:43 TVR_ I have 35 in discovery and another 35 in install pool
14:44 MiroslavAnashkin TVR_: Bad news - you cannot change this pool size after master node is deployed.
14:44 Dr_Drache I don't like this "cannot change if you deploy" on all these options :P
14:44 TVR_ 10.7.212.45 <--> 10.7.212.79 for discovery and 10.7.212.80 <--> 10.7.212.115 for install
14:45 MiroslavAnashkin Dr_Drache: http://docs.openstack.org/user-guide/content/cinderclient_commands.html and http://docs.openstack.org/user-guide/content/novaclient_commands.html
14:45 Dr_Drache MiroslavAnashkin, thanks for the link.
14:46 MiroslavAnashkin We don't like inability to change admin pool size as well. But there are too many dependencies, so it is easier to reinstall master node instead of reconfigure it.
14:46 TVR_ with these ranges, I should be good.. yes?
14:48 MiroslavAnashkin TVR_: It is just 35 addresses. How many NICs do you have per node?
14:48 Dr_Drache MiroslavAnashkin, so, if my cluster gets above XX size, say my cluster doubles in size... I have to delete it to redo it. how is that easier?
14:49 TVR_ 4 nic's per node
14:49 TVR_ 6 nodes
14:50 MiroslavAnashkin Dr_Drache: It is simple. We strongly recommend to have separate physical network for Admin network. Inside this network you may set up any address space - even class A
14:50 MiroslavAnashkin TVR_: Yes, should be enough for 6 nodes
14:51 TVR_ ok.. it is erroring.. I will rebuild and steal a few more.. thanks
14:51 Dr_Drache MiroslavAnashkin, that part I understand, I mean, all these options that can dynamically change with cluster size. the only way to change them is to delete and restart. not what I'd look for for production.
14:52 Dr_Drache and, you cannot just have a seprate physical network, not if their are WANs involved.
14:54 MiroslavAnashkin Dr_Drache: our planned features are master node upgrade, master node injection to the controllers and clustered master node. All these features require admin network reconfiguration ability.
14:54 Dr_Drache what about convert from non HA to HA?
14:56 MiroslavAnashkin Dr_Drache: Instead, we are working on backup/restore feature. It should allow to restore controllers from non-HA to HA environment. Simple convertation is not easy, since HA mode uses additional clustering components.
14:57 Dr_Drache that part is fine. as long as there is a path
14:57 Dr_Drache that's my concern. is there a path..
14:58 IlyaE joined #fuel
14:59 Dr_Drache there is quite a bit for me to learn, but I've been doing KVM over 9 sites with live migration, so I THOUGHT I had a good foundation.
14:59 ArminderS MiroslavAnashkin: thanks for the updates
14:59 ArminderS once i do changes to nova.conf, what services shall need to be restarted
15:02 MiroslavAnashkin ArminderS: It depends on which timeout is changed. BTW, it was discovered, not all timeout settings from nova.conf work at all. So, I'd restart all *nova* services in a batch
15:03 ArminderS right, thanks
15:03 ArminderS a for loop shall do the trick then unless there is a specific order
15:06 MiroslavAnashkin ArminderS: There was specific order, described somewhere in docs.openstack.org
15:14 ArminderS hmmm
15:32 Dr_Drache now comes my hard part.
15:32 Dr_Drache networking
15:51 xdeller joined #fuel
15:53 vk joined #fuel
15:59 designated_ joined #fuel
16:11 JCxMLnblFl joined #fuel
16:11 JCxMLnblFl left #fuel
16:26 amartellone joined #fuel
17:04 e0ne joined #fuel
17:32 mihgen joined #fuel
17:37 angdraug joined #fuel
17:59 e0ne joined #fuel
18:00 dnovakovskiy joined #fuel
18:22 vkozhukalov joined #fuel
18:43 TVR_ so.. quick question.,..
18:43 TVR_ http://www.cdw.com/shop/products/HGST-Travelstar-7K1000-HTS721010A9E630-hard-drive-1-TB-SATA-600/2949655.aspx#TS
18:43 TVR_ on one of my nodes
18:43 TVR_ wrong paste..
18:43 TVR_ one sec
18:43 TVR_ (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) change from notrun to 0 failed: Command exceeded timeout at /etc/puppet/modules/ceph/manifests/osd.pp:33
18:44 Dr_Drache hmm
18:44 TVR_ there we go.. on one of my nodes...
18:44 TVR_ can I just re-deploy that one node, or will the whole process start again
18:45 Dr_Drache beats me. LOL
18:45 Dr_Drache was it deployed alone?
18:45 TVR_ ok.. thanks... (hate it when something else is in my paste buffer)
18:45 TVR_ no.. it was part of a HA setup...
18:47 Dr_Drache hmm, is not having that bring your ceph nodes down past the relication #?
18:47 Dr_Drache *replication
18:48 TVR_ I was asking because I didn't want to spend 2 hours waiting for the whole thing again... I will try adding a timeout to the recipe
18:49 TVR_ ah, looks like clicking deploy only redeploys the bad node...
18:49 TVR_ sweet !!
18:50 Dr_Drache nice
18:50 TVR_ it's little things like that that make a product nice to use
18:51 TVR_ I added a timeout value to the recipe...
18:53 korn__ joined #fuel
18:55 e0ne joined #fuel
18:56 ArminderS joined #fuel
19:13 mihgen joined #fuel
19:38 albionandrew joined #fuel
19:51 TVR_ it built now and I can connect to the dashboard...
19:52 TVR_ seems I am having your image issue as I see the TestVM image is saving indeffinetely and when I upload an image, it too saves indeffinetely..
19:54 Dr_Drache TVR_, i can't upload at all.
19:54 Dr_Drache but, i'm not very great at what network settings I need to connect.
19:54 TVR_ you uploading by hostname?
19:54 Dr_Drache via horizon.
19:58 TVR_ yea.. you uploaded it via http yes? so.. create image ==> name, image location => http://
19:59 TVR_ did you use the IP of the location or the hostname of where it's located?
20:03 Dr_Drache neither.
20:03 Dr_Drache I used a local
20:03 Dr_Drache from the workstation.
20:04 Dr_Drache ...openstack over a MPLS.
20:04 TVR_ ah, ok
20:04 Dr_Drache how the hell am I going to route that.
20:04 TVR_ it's why I start httpd on a box I can route to and put images there...
20:05 Dr_Drache I'm going to have to find someone to talk to about the networking
20:05 TVR_ right now I am dealing with auth so I can run cli commands and see what's going on
20:06 Dr_Drache yea, auth seems backwards from what i'd expect to see.
20:16 TVR_ source openrc seemed to fix my issues
20:16 TVR_ from node-1
20:16 TVR_ this allowed me to run all my nova or cinder CLI commands...
20:17 TVR_ but it also told me the storage wasn't completely happy yet as when I tried to create a volume.. it also hung in the creating state forever...
20:17 TVR_ so I rebooted the whole cluster to see what happens
20:20 MiroslavAnashkin TVR_: It is bad idea to rebot whole cluster, if you use HA mode
20:20 MiroslavAnashkin reboot
20:20 TVR_ heh.. yea.... I am sure this will cause other isssues
20:20 TVR_ heh
20:21 Dr_Drache ok, I am sure I can read this somewhere... but where would I find info, so that my instances can contact my current network, and vice versa?
20:23 MiroslavAnashkin joined #fuel
20:23 Dr_Drache also, if I assign a floating IP to an instance, can I also set that IP as static in the VM as well?
20:23 Dr_Drache MiroslavAnashkin, you missed my question :P
20:24 Dr_Drache <Dr_Drache> ok, I am sure I can read this somewhere... but where would I find info, so that my instances can contact my current network, and vice versa? <-- lol
20:24 MiroslavAnashkin Dr_Drache: No, static and floating IPs belong to different networks
20:25 Dr_Drache ok, I must have misunderstood that.
20:27 Dr_Drache MiroslavAnashkin, is there a openstack network documentation for non-network dummies? Because I think I need it
20:28 MiroslavAnashkin Dr_Drache: Please start from this first: http://docs.mirantis.com/fuel/fuel-4.0/reference-architecture.html
20:29 MiroslavAnashkin http://docs.mirantis.com/fuel/fuel-4.0/reference-architecture.html#network-architecture
20:31 TVR_ hey MiroslavAnashkin, what would the usual causes be for volumes going into the creating state forever? My images are 'saving' forever as well..... I checked the cinder.conf and nova.conf and the virsh secret-list and uuid match... so have you seen this?
20:32 MiroslavAnashkin TVR_: Do you use Ceph? If yes - for which backend - cinder or glance?
20:33 Dr_Drache MiroslavAnashkin, that sadly, I have read.
20:33 TVR_ I chose ceph for images and volumes..
20:33 TVR_ looks like  rbd -p rbd ls hangs
20:34 TVR_ rados commands work
20:34 Dr_Drache MiroslavAnashkin, TVR_, ok... so my public IP's, those can be my "internal" network IPs for my other networks.
20:34 Dr_Drache say like a 10.10.10.x type local network.
20:41 TVR_ OK.. so I guess I will actually use the 30 day support now... oh well...
20:41 MiroslavAnashkin TVR_: I am not strong in Ceph:-(
20:41 TVR_ no worries...
20:41 TVR_ I had cinder on all my nodes when I rolled my own... but the compute host here dont... so I will check with support
20:42 MiroslavAnashkin TVR_: Xarces and Angdraug are
20:43 TVR_ thanks man!
20:44 MiroslavAnashkin Dr_Drache: Internal network is for interconnect between Openstack hosts and for service traffic between VMs. External (Floating) is for outbound connection to VMs - it is usually segment of your local network, reserved for Openstack
20:51 Dr_Drache external floating, that's labled public in some spots right?
20:55 e0ne joined #fuel
21:05 jouston joined #fuel
21:06 angdraug TVR_: fuel creates separate images and volumes pools instead of using rbd pool for everything
21:06 TVR_ yes.. I see the volumes and images pool, yes
21:06 angdraug is ceph -s reporting any problems?
21:06 TVR_ I am rebuilding the cluster as the first time not all nodes deployed completely..
21:06 angdraug what about osds, do they all show up in ceph osd tree?
21:07 TVR_ are you on still in say, 1.5 hours?
21:07 angdraug sure
21:07 TVR_ ok.. let me reach out to you then, as I will have a clean env
21:08 Dr_Drache angdraug, so, can you explain that a tiny bit more? if i have ceph backend for everything, how does it acually use the space? (say a 2TB drive on each OSD)
21:08 TVR_ also.. I added a timeout of 600 to /etc/puppet/modules/ceph/manifests/osd.pp for the osd activate as it timed out on one node from the last cluster
21:11 Dr_Drache wow, TVR_ just a random #?
21:11 Dr_Drache or just a guess.
21:12 Dr_Drache i mean, if it works, it works, just thought 600 be a bit high. good job though
21:14 angdraug if you use ceph backend for everything, including nova, all your drives will be shared between glance, cinder, and nova
21:14 angdraug have you seen this presentation? http://www.slideshare.net/mirantis/fuel-ceph
21:15 angdraug slide 5 should answer your question
21:15 Dr_Drache sure have.
21:15 Dr_Drache I assumed such, but only asked because you mentioned it
21:15 angdraug ah I see
21:16 Dr_Drache I hate to assume and then be shown as the idiot.
21:16 angdraug different pools are for logical segregation, it doesn't impact physical layout
21:16 angdraug no, it's a legit question
21:16 angdraug in theory you can alter physical layout between different pools, we just don't do that by default
21:17 angdraug customizing crushmaps is left as an excercise to the user :)
21:18 Dr_Drache NOOO! :P but yea, so when my ceph storage is "90%" of my drive, that is shared between all needed storages.
21:18 angdraug yup
21:18 Dr_Drache beautiful.
21:18 angdraug better yet, it's CoW in all cases except one tiny corner
21:19 Dr_Drache right. so, in fuel, when i have a automatic portion called "virtual storage" what is that then?
21:20 angdraug that's left for the benefit of LP #1262914
21:20 angdraug we plan to get rid of it as soon as we're confident we got rid of all the places in Nova where it insists on downloading a local copy of an image to the compute node
21:21 Dr_Drache awesome.
21:21 angdraug if you enable all ceph options, it's save to give it minimum amount of storage, enough to keep your largest image or two
21:21 angdraug mind that you'd want to pull ceph packages from fuel 4.1 repo if you use ceph for nova ephemeral drives
21:22 angdraug there's a nasty race condition that was fixed days after 4.0 was released
21:22 Dr_Drache ahh, yea, I can't deploy in ubuntu because of ceph.pp issues.
21:22 Dr_Drache happen to know the information I need to pull/build from the 4.1?
21:23 angdraug I was referring to LP #1260911, is that what you have?
21:23 angdraug http://ss0078.svwh.net/fwm/4.1/ubuntu/pool/main/
21:24 angdraug just grab the ceph packages from there, 4.1 is in active development right now so you don't want the rest of it yet
21:24 Dr_Drache k, i'll get those.
21:24 Dr_Drache I have one more question, then I'll let you be :P
21:25 angdraug there's another ubuntu specific problem you might see if you use SATA drives: LP #1263648
21:25 angdraug there's a bug in Linux SATA subsystem that can break the way Fuel tracks SATA drive allocations
21:26 Dr_Drache I have that issue with my HPs and smart arrays
21:26 Dr_Drache I will have a remote site, over a MPLS link (subnet of my external) I will want part of my HA cluster... how would i look into that?
21:26 angdraug I wouldn't recommend having a single ceph cluster on two sides of an MPLS link
21:26 Dr_Drache I gave logs and dumps to MiroslavAnashkin and someone else.
21:27 Dr_Drache ahhh, any suggestion of how to tackle that? it's for a DR site.
21:27 angdraug ceph is very sensitive to time, 50ms of clock skew reportedly can be enough to bring down your cluster
21:28 angdraug have a look at radosgw-agent
21:28 angdraug although that's only applicable to rgw objects (s3/swift)
21:28 angdraug I think Inktank is working on a similar solution for rbd, but I haven't seen anything tangible yet
21:29 Dr_Drache right, I mean, I have decent CoW with my NFS/KVM right now.
21:29 Dr_Drache but, we are looking to go to openstack.
21:31 angdraug what's the latency between the sites, and how far behind can you afford the DR site to be?
21:31 Dr_Drache somewhere I thought with emperor and soon "new version" the time frame thing was "less of an issue"
21:32 Dr_Drache right now, 25-32ms
21:32 angdraug yes, emperor introduced radosgw-agent, that must be what they're referring to
21:32 angdraug not applicable to rbd just yet, unfortunately
21:32 angdraug 32ms is definitely too much for ceph
21:32 Dr_Drache and, a few min is fine. i'm not looking for no downtime.
21:32 Dr_Drache sadly our SLA gives up to 50ms
21:33 angdraug well that means that you're definitely looking for async replication
21:33 angdraug and I mean for OpenStack MySQL stuff, too, not just Ceph
21:34 Dr_Drache right, I have critical data on multipath SANs.
21:34 angdraug depending on your use cases, cooking up your own replication solution a layer above openstack that would backup volumes from one site to another might be reasonable
21:34 Dr_Drache this is just for the instances/etc
21:35 Dr_Drache I don't belive we are big enough to be able to take on that sort of project. but, i'll look into it.
21:36 angdraug try to bring it up on ceph-users@ ml, or #ceph on OFTC
21:36 angdraug that's not a unique problem so maybe someone already has cooked something up )
21:37 Dr_Drache yea, I do appreciate the information. now if i could deploy ubuntu (we've been a ubuntu shop for 5+ years)
21:38 angdraug ubuntu is a priority for us, so please submit bugs for anything you come across and we'll try our best to fix them
21:39 angdraug http://www.slideshare.net/Inktank_Ceph/sweil-tomorrowsceph-27299726
21:39 angdraug slides 43-46 talk about multi-site plans for Ceph
21:39 angdraug so if you're patient you can just wait for them to implement it :)
21:39 Dr_Drache that's why i'm here, not just to get info.
21:40 angdraug thanks!
21:40 Dr_Drache I like submitting bugs to people who can ask for more info than i think to give.
21:40 Dr_Drache i've done 2 so far, I am sure I'll get more once I figure out networking a bit more tomarrow.
22:22 IlyaE joined #fuel
22:31 e0ne joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary