Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-02-10

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 xek joined #fuel
00:16 kdavyd joined #fuel
00:56 Longgeek joined #fuel
02:01 rmoe joined #fuel
02:06 ahg joined #fuel
02:12 Longgeek joined #fuel
02:27 xarses joined #fuel
02:49 mattgriffin joined #fuel
03:18 mattgri__ joined #fuel
03:37 Longgeek joined #fuel
03:42 ahg joined #fuel
04:57 Akshik joined #fuel
07:01 Akshik joined #fuel
07:04 dklepikov joined #fuel
07:15 Longgeek joined #fuel
07:23 Miouge joined #fuel
07:28 monester_laptop joined #fuel
07:30 sambork joined #fuel
07:40 tzn joined #fuel
07:43 avgoor joined #fuel
07:46 hyperbaba joined #fuel
07:48 sambork joined #fuel
08:06 ahg joined #fuel
08:14 Longgeek joined #fuel
08:24 ahg joined #fuel
08:41 saibarspeis joined #fuel
08:43 e0ne joined #fuel
08:45 sambork joined #fuel
08:55 alecv joined #fuel
08:59 HeOS joined #fuel
09:18 ahg joined #fuel
09:33 dkaigarodsev joined #fuel
09:42 stamak joined #fuel
09:45 tzn joined #fuel
09:52 e0ne joined #fuel
10:07 andriikolesnikov joined #fuel
10:37 ddmitriev joined #fuel
10:40 dilyin joined #fuel
10:45 Longgeek joined #fuel
10:55 monester_laptop joined #fuel
11:46 warpc__ joined #fuel
11:52 andriikolesnikov joined #fuel
11:59 ahg joined #fuel
12:08 ahg joined #fuel
12:33 Miouge joined #fuel
12:41 andriikolesnikov joined #fuel
13:25 e0ne joined #fuel
13:42 devstok joined #fuel
13:42 devstok hi
13:42 devstok yesterday night glance image-create seems to be solved
13:42 devstok but
13:43 devstok now I got this WARNING glance.store.base [-] Failed to configure store correctly: Store sheepdog could not be configured correctly. Reason: Error in store configuration: [Errno 2] No such file or directory Disabling add method.
13:55 devstok glance answer with a good response but the images still in queued status
13:55 sambork joined #fuel
13:58 emagana joined #fuel
14:15 aliemieshko hi, please run 'glance image-list' and provide us
14:17 devstok image-list works well
14:17 devstok the list of the images
14:20 Miouge joined #fuel
14:28 aliemieshko These are harmless warnings https://ask.openstack.org/en/question/29511/glance-cache-prefetcher-displays-errors-related-to-storage-backends-not-configured/
14:30 devstok If i run that command
14:30 devstok 2015-02-10 15:29:19.882 18676 ERROR glance.store.sheepdog [-] Error in store configuration: [Errno 2] No such file or directory 2015-02-10 15:29:19.893 18676 WARNING glance.store.base [-] Failed to configure store correctly: Store sheepdog could not be configured correctly. Reason: Error in store configuration: [Errno 2] No such file or directory Disabling add method. 2015-02-10 15:29:19.897 18676 WARNING glance.store [-] Deprecated: g
14:38 aliemieshko what command ?
14:40 devstok glance-cache-prefetcher
14:41 devstok however is the same warnings errors that i get when i launch image-create
14:57 aliemieshko does glance create images ?
14:57 julien_ZTE joined #fuel
15:05 claflico joined #fuel
15:09 emagana joined #fuel
15:14 Longgeek joined #fuel
15:20 championofcyrodi joined #fuel
15:21 mpetason joined #fuel
15:30 sambork joined #fuel
15:32 devstok no
15:32 devstok the image creation remain in queued status
15:33 devstok but now I've got a forbidden message in the end
15:33 devstok and the log show this error  ERROR glance.store.sheepdog [-] Error in store configuration: [Errno 2] No such file or directory
15:33 mattymo we don't use sheepdog store
15:33 mattymo that's not the actual error. you can ignore that
15:34 devstok HTTP/1.1 403 Forbidden date: Tue, 10 Feb 2015 15:30:45 GMT content-length: 50 content-type: text/plain; charset=UTF-8 x-openstack-request-id: req-edd42ad4-862f-4bee-8978-f400963361a6  403 Forbidden  Attribute 'file' is read-only.    Request returned failure status. 403 Forbidden Attribute 'file' is read-only.     (HTTP 403)
15:34 devstok this is the output of glance --debug image-create
15:35 devstok I've updated the python files with last version
15:35 devstok images.py
15:35 devstok V2
15:36 mattymo The real error is somewhere in glance logs
15:38 devstok no I only got some warnings but no others
15:40 blahRus joined #fuel
15:47 devstok ok now i sync all python files in the controllers
15:47 devstok the command works
15:47 devstok but the status of the creation still remain at 25%
15:47 devstok SAVING
15:50 MiroslavAnashkin What is the image format and what is the filesystem size? It is recommended to use RAW images with Ceph - Ceph provides copy-on-write itself and have to convert qcow images on the fly to the full size, then convert it back.
15:55 saibarspeis joined #fuel
15:55 devstok in the last env test with HA the qcow2 was used normally
15:55 devstok btw I'll test raw image
15:56 devstok I have to use qemu-convert?
15:56 devstok i'm using precise ubuntu .img
15:56 sambork joined #fuel
16:00 devstok https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
16:03 MiroslavAnashkin qemu-img convert -f <input image file format> -o raw <input file> <output file>
16:04 MiroslavAnashkin http://docs.openstack.org/image-guide/content/ch_converting.html
16:05 devstok I created the raw
16:05 devstok qemu-img convert -O raw precise-server-cloudimg-amd64-disk1.img ubuntu.raw
16:05 devstok launched the glance image-create
16:08 devstok still saving status 25%
16:13 MiroslavAnashkin Please log in to second console and check available free space on the controllers
16:15 devstok total            33G  7.6G   24G  25%
16:17 Miouge joined #fuel
16:18 devstok but I'm using CEPH
16:18 devstok but not for nova
16:18 devstok so the images go to the CEPH storage
16:25 Miouge joined #fuel
16:34 MiroslavAnashkin Yes, but Glance may download/upload the image to temporary storagewith curl first, then, if necessary,  transfers image to the temporary storage on the proper node, then uploads it to target backend.
16:36 devstok so simple :)
16:36 devstok what is the problem for you?
16:38 devstok I saw that in DB there is not the image_loacations for the images that I tryed to create
17:09 daniel3_ joined #fuel
17:09 devstok ?
17:14 rmoe joined #fuel
17:18 sambork joined #fuel
17:24 MiroslavAnashkin Got out of ideas :-(
17:40 emagana joined #fuel
17:41 emagana_ joined #fuel
17:51 e0ne joined #fuel
17:53 andriikolesnikov joined #fuel
18:05 xarses joined #fuel
18:15 jaypipes joined #fuel
18:17 xarses mihgen: https://review.openstack.org/#/c/150987/
18:30 stamak joined #fuel
19:04 ahg joined #fuel
19:06 tobiash_ joined #fuel
19:52 ahg joined #fuel
20:01 sambork joined #fuel
20:49 tatyana joined #fuel
21:06 julien_ZTE joined #fuel
21:30 sambork joined #fuel
21:32 emagana joined #fuel
22:03 championofcyrodi so i got rbd to export an image w/ my old 5.1 icehouse instance..
22:03 championofcyrodi as raw... 60GB
22:04 championofcyrodi i then did glance --image-create --name yum --disk-format raw --file ./yum.raw --is-public True --is-protected False --human-readable --progress
22:04 championofcyrodi on the new 6.0 Juno environment
22:05 championofcyrodi the image upload was success, and i can see the full RAW image in rbd
22:05 championofcyrodi however the state is still 'saving' in glance.
22:06 championofcyrodi glance api log has: glance.api.v1.upload_utils [-] Unable to kill image 016abb38-508f-4041-bdc6-f38d156c95cd:
22:06 championofcyrodi where 016... is the uuid generated for the image i tried to create
22:07 championofcyrodi hmm
22:07 championofcyrodi with
22:07 championofcyrodi glance.api.v1.upload_utils NotAuthenticated: Authentication
22:07 championofcyrodi at the bottom of the trace.
22:10 championofcyrodi since it was 60GB... maybe the auth token expired???
22:16 championofcyrodi also the log says it's using glance.api.v1.upload_utils
22:16 championofcyrodi this is ceph for cinder+glance
22:28 championofcyrodi so since rbd seemed to have to data... tried this: glance image-create --size 64424509440 --name yum-rbd --store rbd --disk-format raw --container-format bare --location rbd://016abb38-508f-4041-bdc6-f38d156c95cd
22:28 championofcyrodi and it seemed to work and the image is 'active' in glance...
22:29 championofcyrodi trying to spawn the VM now... (crossing fingers)
22:30 championofcyrodi Every 2.0s: ceph osd pool stats compute : pool compute id 5 [\r\n] client io 1795 B/s wr, 0 op/s
22:30 championofcyrodi so it looks like ceph is doing something
22:31 championofcyrodi the instance should be using ephem storage, so i'm watching the compute rbd pool
22:32 championofcyrodi B/s and op/s is changing though...  saw ~8000B/s at one point.
22:58 championofcyrodi okay... it looks like this is working... but VERY slow since its 60GB of RAW disk in bare format.  if it works, i'll try converting the rbd exports to qcow and see if that speeds things up copying from rbd images pool, to compute node disk, convert to raw, and back to rbd.
22:58 dilyin joined #fuel
23:00 omolchanov joined #fuel
23:02 jonmills_nasa joined #fuel
23:03 jonmills_nasa I'm having a really difficult time deploying multiple controllers in HA config.  Moreover, when it fails to deploy, I'm having a near impossible time cleaning it up.  Fuel 5.1.1.  Any advice would be much appreciated.
23:05 jonmills_nasa I'm trying to explore what I can get done using the Fuel cli command.  Is there a magic cli command where I can force a wedged deployment to go away?  My Fuel UI is hung up in a "Stopping...." dialogue while it tries to clean up.
23:06 championofcyrodi you should be able to use the fuel ui, click the environment, then 'actions' tab
23:06 championofcyrodi then just delete it
23:06 championofcyrodi or reset
23:06 jonmills_nasa Thanks, man.  I'm familiar with that one.  I was hoping to not trash all the nodes though.
23:06 championofcyrodi also we are in the middle of upgrading to 6.0
23:07 jonmills_nasa Understood.
23:07 championofcyrodi which has already been released.  there are a lot of fixes in 6.0 you'll likely want
23:07 jonmills_nasa I'm running 5.1.1 because it's what Mellanox supports.  I'm running this entirely on top of Infiniband.
23:07 jonmills_nasa Long story there.
23:07 championofcyrodi sounds like it.
23:07 championofcyrodi but that is awesome (infiniband)
23:08 championofcyrodi Mirantis Fuel 6.0 does have Mellanox as an option in it's configuration.
23:08 jonmills_nasa Interesting.
23:08 championofcyrodi so it will probably support that as well
23:08 championofcyrodi i never saw mellanox as a config option in 5.0
23:09 jonmills_nasa I think Mellanox took Mirantis 5.1.1, hacked on it for awhile, and gave it to us.
23:09 championofcyrodi nice
23:09 jonmills_nasa So the version I see offers me the option to deploy worker nodes that come up with a working OFED stack with support for pkeys, etc.  that all works.
23:10 jonmills_nasa but I can't seem to make the HA controllers behave.
23:10 championofcyrodi http://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#mellanox-neutron-components
23:10 jonmills_nasa Each deployment fails
23:10 championofcyrodi maudgib: you did you troubleshoot the 6.0 deployment issues when it failed?
23:10 championofcyrodi 's/you did/how did/g'
23:11 jonmills_nasa Basically, I had it all working, but with a single controller.  Again, this is 5.1.1 on IB.  I've not used 6.0.
23:11 jonmills_nasa The problems came when I tried to then deploy a second controller, for HA.
23:12 jonmills_nasa Not only did the deploy fail
23:12 jonmills_nasa but it also broke the existing, working, controller
23:12 jonmills_nasa and now it's wedged
23:13 championofcyrodi so we started w/ 5.0, upgraded to 5.1, then 5.1.1 then 6.0...
23:13 championofcyrodi one of the first things we found out is that EVERYTHING runs through rabbitmq
23:13 jonmills_nasa well, sure
23:13 championofcyrodi coordination wise...
23:13 championofcyrodi first thing we saw was a lot of exceptions with 'broken pipe'
23:14 championofcyrodi do you're rabbitmq logs have anything like that?
23:14 championofcyrodi your*
23:14 jonmills_nasa So I can look at the output of 'rabbitmqctl list_connections'
23:14 jonmills_nasa and I can tell you that there looks to be far fewer connections than there should be
23:16 jonmills_nasa 'dockerctl check' passes all tests though
23:16 jonmills_nasa I think rabbitmq isn't busy because the controller is sideways.  none of the openstack components are trying to talk anymore
23:18 jonmills_nasa In fact, this thing is so wedged, that the 'Reset Environment' button, under the Actions tab, is grayed out.
23:18 championofcyrodi check /var/log/rabbitmq on the 1 controller you had running
23:19 championofcyrodi i think the docker containers are volume mounting the host /var/log/<service> folders
23:19 championofcyrodi let me check...
23:20 championofcyrodi okay wait a min... docker is just on the fuel controller..
23:20 jonmills_nasa What is shows is an orderly shutdown, in fact.  It's as if Fuel used its Puppet config against the controller to make it not be a controller anymore
23:21 championofcyrodi you want to look at the rabbitmq on your openstack icehouse controller
23:21 championofcyrodi hmm... that sounds odd
23:21 jonmills_nasa Correct.  That's what I did.
23:21 championofcyrodi okay
23:21 jonmills_nasa *shrugs*
23:22 championofcyrodi unfortunately i'm stumped... we are using a very diff. config, so unfortunately you're likely hitting issues we didnt.
23:22 jonmills_nasa I think I'm just going to have to delete env and start over.  Which is very frustrating.
23:22 championofcyrodi however, the good news is... after being frustrated and banging our heads against the wall for about 2 months...
23:22 championofcyrodi we are finally getting comfortable enough w/ openstack+fuel to get good use out of it
23:22 championofcyrodi our config at least.
23:23 jonmills_nasa My expertise in Openstack was of the Essex generation.  So I'm behind the curve now and really relying on Fuel to make it work.
23:23 jonmills_nasa But I'll keep trying
23:23 championofcyrodi #fuel: exporting the image from ceph+rbd in 5.1 and importing it into ceph+rbd in 6.0, followed by glance image-create with the --location switch worked like a charm.  just took a while to spin up the huge RAW image.
23:25 jonmills_nasa Thanks for chatting
23:25 championofcyrodi jonmills_nasa: good luck to you.  I have to run, but i'm here often these days so feel free to bounce ideas about anything that isnt mellanox ;) because we are running on supermicros and hoping to get 10Gbps network soon.
23:26 jonmills_nasa gotcha
23:26 jonmills_nasa we are hoping to get Gluster+RDMA running over the IB instead of using Ceph
23:26 jonmills_nasa we also want virtualized IB into the VMs using SRIOV
23:27 jonmills_nasa not holding my breathe though
23:43 julien_ZTE joined #fuel
23:47 emagana joined #fuel
23:53 codybum joined #fuel
23:54 codybum Hi All
23:54 codybum Has anyone had to set sysctl vm.swappiness=10 on their controllers for RabbitMQ to behave?
23:55 codybum On Fuel 6.0.1 changing the swappiness seems to really help rabbitmq problems, but there are still issues.  I still see lots of "failed to publish" messages.

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary