Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-02-18

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 jobewan joined #fuel
00:13 jobewan anyone answer a quick neutron question real quick?
00:14 jobewan the docs say that for vlan segmentation, you need 3 physical nics... We have 2 that we are lag'ing together and trunking all vlans to.  Can we get fuel to support that?
02:05 vkozhukalov joined #fuel
04:09 besem9krispy joined #fuel
04:50 padlock joined #fuel
04:51 padlock hi
04:53 padlock left #fuel
04:56 jobewan joined #fuel
04:58 jobewan can anyone answer a quick neutron question?
04:58 jobewan the docs say that for vlan segmentation, you need 3 physical nics... We have 2 that we are lag'ing together and trunking all vlans to.  Can we get fuel to support that?
05:08 mihgen joined #fuel
06:48 saju_m joined #fuel
07:25 dburmistrov joined #fuel
07:27 e0ne joined #fuel
07:40 meow-nofer joined #fuel
07:41 meow-nofer__ joined #fuel
08:24 mrasskazov joined #fuel
08:30 vkozhukalov joined #fuel
08:32 rvyalov joined #fuel
08:33 mihgen joined #fuel
08:49 richardkiene_ joined #fuel
09:07 e0ne joined #fuel
09:25 vk joined #fuel
09:25 tatyana joined #fuel
09:29 miguitas joined #fuel
09:38 saju_m joined #fuel
09:38 e0ne_ joined #fuel
09:40 tatyana left #fuel
09:40 e0ne__ joined #fuel
09:40 e0ne_ joined #fuel
09:44 anotchenko joined #fuel
10:05 saju_m joined #fuel
10:09 vk joined #fuel
10:14 vk joined #fuel
10:17 vk joined #fuel
10:21 vk joined #fuel
10:22 anotchenko joined #fuel
10:23 sanek joined #fuel
10:30 besem9krispy joined #fuel
10:34 anotchenko joined #fuel
11:05 anotchenko joined #fuel
11:31 anotchenko joined #fuel
12:21 MiroslavAnashkin jobewan: http://docs.mirantis.com/fuel/fuel-4.0/reference-architecture.html#assign-admin-network-to-the-ovs-bridge
12:36 e0ne joined #fuel
12:53 anotchenko joined #fuel
13:04 xdeller joined #fuel
13:14 phamby joined #fuel
13:45 Dr_Drache joined #fuel
13:45 Dr_Drache xarses, MiroslavAnashkin, I forgot who I was talking to about qcow/raw storage.
13:46 MiroslavAnashkin Dr_Drache: xarses
13:48 Dr_Drache MiroslavAnashkin, question though... raw = better, but what about the qcow setting in fuel, for snapshotting?
13:49 MiroslavAnashkin Hmm, my opinion - qcow is better :)
13:49 Dr_Drache I agree, but we were talking space usages.
13:50 Dr_Drache guess I need to revisit that conversation. see what I need to do.
13:55 Dr_Drache MiroslavAnashkin, what about the slowness of ceph/cinder when it needs to convert qcow2 to raw, and take up 2x diskspace?
13:55 Dr_Drache (this is only based on what I am told)
14:05 MiroslavAnashkin qcow require only space, which is actually used. RAW a getting expanded in Glance cache to the full formatted size, including empty space.
14:05 besem9krispy joined #fuel
14:08 MiroslavAnashkin So, if you image has 2 GB used and 100Gb of free filesystem space then convertation to raw may create 100Gb image, while original qcow remains just 2Gb
14:10 Dr_Drache not how it was explained to me
14:10 Dr_Drache that it has to convert to raw, then make a volume in raw, every time.
14:13 MiroslavAnashkin Yes, true. Raw volume means libvirt/openstack know nothing of it and fully rely to the OS in the image
14:40 Dr_Drache joined #fuel
14:44 Dr_Drache MiroslavAnashkin, sorry bout that
14:44 Dr_Drache killed the wrong VM
14:44 Dr_Drache MiroslavAnashkin, so, should I upload in qcow, or raw. so confused today
14:46 anotchenko joined #fuel
14:48 phamby left #fuel
15:01 MiroslavAnashkin It depends on your configuration and intentions. As Xarses said :xarses so, with ceph for glance and cinder, its better to store images as raw, as it will just do a copy on write clone.
15:01 jaypipes joined #fuel
15:02 Dr_Drache yea
15:02 Dr_Drache guess I need to redeploy AGAIN
15:15 xarses Dr_Drache: in a ceph env, raw would be preferred
15:16 Dr_Drache xarses, that's what I gathered, but fuel pretty much tells you otherwise
15:16 xarses Dr_Drache: ya, ceph is a newer feature.
15:17 Dr_Drache guess, I'm going to redeploy.
15:17 Dr_Drache lol
15:23 Dr_Drache xarses, hate to ask : "Ceph RadosGW for objects" what does that mean?
15:25 xarses configures object storage (swift-api) to use RadosGW instead
15:27 Dr_Drache if ceph is my primary storage, that's pretty much a no-brainer then.
15:28 xarses radosgw doesn't support a small handfull of features that swift does, so they are seperated
15:29 xarses but in most cases, yes you want it
15:30 Dr_Drache awesome, thank you for the assistant with my stupid questions today
15:33 anotchenko joined #fuel
15:39 tatyana_ joined #fuel
16:12 Dr_Drache xarses, is uploading through the gui ok regardless of source img type? or there a better way?
16:13 xarses Dr_Drache: its fine, you can also upload from the glance cli client
16:14 Dr_Drache so, I upload a qcow, choose qcow, but once it's in glance, it's raw. got it. thanks!
16:14 xarses erm, if you upload qccow, glance won't convert it
16:14 xarses it just stores whatever you send it
16:15 Dr_Drache oh
16:15 Dr_Drache so i have to pre-convert it.
16:15 xarses https://github.com/stackforge/fuel-library/tree/master/deployment/puppet/ceph#testing-openstack
16:15 xarses example of using the glance cli
16:16 Dr_Drache yea, i'll use that, I guess that was my real question, does the gui convert.
16:17 xarses nope, conversion is only done by cinder/nova if the backend says its needed
16:19 Dr_Drache ...but I am using raw, doesn't that make it needed?
16:38 jobewan joined #fuel
17:02 vkozhukalov joined #fuel
17:06 jobewan joined #fuel
17:16 xarses joined #fuel
17:18 besem9krispy joined #fuel
17:41 rmoe joined #fuel
17:43 crandquist joined #fuel
17:45 e0ne joined #fuel
17:53 richardkiene joined #fuel
18:14 mihgen joined #fuel
18:26 meow-nofer joined #fuel
18:27 meow-nofer__ joined #fuel
18:30 vkozhukalov joined #fuel
18:43 e0ne joined #fuel
18:55 dhblaz joined #fuel
18:58 dhblaz When I try to run this command on Fuel 4.0 configured with HA and ceph the task is starts never seems to complete:
18:58 dhblaz cinder upload-to-image b8eb63c7-b67a-4043-8b11-e7414ace452a hb_fullstack_2014021702 --force True
18:59 dhblaz The image stays stuck in queuing
19:00 dhblaz Can anyone confirm that this is specific to me or offer helpful advice about where I might try troubleshooting?
19:15 MiroslavAnashkin dhblaz: What is the filesystem size of the image? What is image format (raw, qcow, other)? How long it is in status queued?
19:15 dhblaz The volume format is RAW
19:15 dhblaz and it is 20gb
19:16 dhblaz I tried not specifying the image format (defaults to RAW) and specifying qcow2
19:16 MiroslavAnashkin For instance, I have 40 Gb qcow image and it takes about 40 minutes to upload on the fast machine with SSD only.
19:18 dhblaz Been in status queued for 14 hours
19:18 MiroslavAnashkin Since you use Ceph - it may last twice longer time to upload than the same image converted to raw.
19:19 MiroslavAnashkin Hmm, 14 hours for 20 gigs is abnormal.
19:21 MiroslavAnashkin dhblaz: Please start investigation with `ceph -s` command. Your case looks like Ceph backend is offline or misconfigured.
19:23 dhblaz http://paste.openstack.org/show/66990/
19:26 MiroslavAnashkin Well, then what is the total disk (disks) size of the image? Does these 20 gigs include free space?
19:26 dhblaz I just ran the health checks and see this one failing that is probably relevant:
19:26 dhblaz Create volume and attach it to instance
19:26 dhblaz Timed out waiting to become in-use Please refer to OpenStack logs for more details.
19:27 dhblaz MiroslavAnashkin: 20 gigs includes the free space
19:27 dhblaz my OSDs have 149G free
19:28 dhblaz This health check passed though:
19:28 dhblaz Launch instance, create snapshot, launch instance from snapshot
19:28 dhblaz So it looks like an issue with cinder instead of glance
19:29 dhblaz MiroslavAnashkin: I have to change locations, but I will be back in 15 minutes and will read the logs.  I'll see you on the other side.
19:38 Dr_Drache hmmm
19:38 Dr_Drache MiroslavAnashkin, I have a suggestion.
19:38 Dr_Drache wait, this is OS, not fuel
19:40 dhblaz joined #fuel
19:41 dhblaz MiroslavAnashkin: back
19:56 angdraug joined #fuel
20:10 MiroslavAnashkin Hmm, what does `rbd ls images` says?
20:15 dhblaz It lists a bunch of uuids
20:15 dhblaz Do you want me to put it up on paste.openstack.org?
20:19 MiroslavAnashkin Please save first list, try to upload problematic image one more time and check if new uuid appears in ceph.
20:20 MiroslavAnashkin You may check after minute or 2 and then interrupt upload
20:21 dhblaz I just booted the instance that I used to build that volume to make sure that the volume was still good.  I'm going to shut it down and do as you suggest.
20:23 MiroslavAnashkin Or try `cinder --debug upload-to-image`
20:25 MiroslavAnashkin xarses: Hmm, interesting, does `cinder upload-to-image` use glance cache or performs everything inside Ceph?
20:25 besem9krispy joined #fuel
20:25 e0ne_ joined #fuel
20:30 MiroslavAnashkin dhblaz: And you also may check if there are read-write activity in ceph while cinder downloads the volume
20:32 dhblaz http://paste.openstack.org/show/67007/
20:33 dhblaz I don't know how to check for read-write activity separate from whatever else would be going on
20:43 dhblaz I tried this on the ceph osds
20:43 MiroslavAnashkin So, cinder gets image, but glance does not store it. Despite there is request for glance in debug output.
20:44 dhblaz vmstat -w -S m -p sda4 10
20:44 dhblaz I didn't really see any large uptick in writes
20:44 dhblaz Reads don't move at all.  I guess it is getting all the reads from cache.
20:45 xarses MiroslavAnashkin: no clue
20:45 MiroslavAnashkin I may assume there is not enough free space on the volume where cinder (curl) stores temporary data
20:46 MiroslavAnashkin But we usually have some related errors in the logs in such cases
20:48 dhblaz In which logs should I find such errors
20:48 dhblaz I made the mistake of ticking the Debug logs
20:48 dhblaz and the signal to noise ratio is terrible
20:48 MiroslavAnashkin Please start from glance logs
20:49 Dr_Drache signal to noise ratio.... how does that terminolgy fit?
20:49 xarses dhblaz: this looks like an iteration of glance upload --location bug
20:50 Dr_Drache MiroslavAnashkin, xarses, it seems when I upload raw, I can only make one instance from a image. (offical ubuntu cloud img)
20:50 dhblaz Dr_Drache: The signal (something useful in the logs) is overwhelmed by the number of lines written a second
20:51 Dr_Drache after that, the instance can never finish a cloudinit.
20:52 Dr_Drache dhblaz, I find it to be the exact opposite. but I still stand confused with the radio term applied to logs.
20:52 xarses Dr_Drache: cloudinit... sounds like networking issues, would need to see logs
20:53 justif joined #fuel
20:53 Dr_Drache xarses, http://paste.openstack.org/show/67017/
20:54 Dr_Drache I still think i'm having security group growing pains
20:54 dhblaz Dr_Drache: | are you looking in /var/log/remote on the master?
20:55 dhblaz xarses: my cloud-init problems were related to redhat's zeroconf making a route that circumvented the connection to the metadata service
20:55 xarses Dr_Drache: yep, line 309, un-able to contact the metadata service
20:56 Dr_Drache xarses, yea. I agree, looking at what changed other than a new instance
20:59 dhblaz MiroslavAnashkin: I see two things in the log that might be relevant
20:59 dhblaz Stderr: '/bin/sh: collie: command not found\n'
21:00 dhblaz And 2014-02-18 09:56:42 WARNING glance.store.base [-]  Failed to configure store correctly: Store cinder could not be configured correctly. Reason: Cinder storage requires a context. Disabling add method.
21:04 besem9krispy joined #fuel
21:04 dhblaz Dr_Drache: You may what to confirm that your metadata proxy is running (Listening on port 8775) on your controllers
21:04 dhblaz and that you can connect to it from your compute nodes
21:12 xarses Dr_Drache: what network manager are you using?
21:19 dhblaz MiroslavAnashkin: As an update.  I installed sheepdog and ran find /etc/rc3.d/ -type l -name "S*openstack*" -exec {} restart \;
21:19 dhblaz on node-16 (my primary controller)
21:19 dhblaz It didn't seem to make a difference
21:23 justif How do I go about trouble shooting the openstack install on a Compute node? It always seems to fail at the same point for both nodes and can't figure out why
21:25 dhblaz MiroslavAnashkin: I installed sheepdog on nodes 17 and 18
21:25 dhblaz (other controllers
21:26 dhblaz )
21:26 dhblaz tried again and node-17's disk filled up
21:26 dhblaz I believe it was writing in /tmp
21:26 dhblaz The file disappered before I could find it.
21:26 dhblaz I'll increase the filesystem size and try again
21:27 dhblaz real bummer that /tmp needs to be so big.
21:27 dhblaz I have to go for today
21:27 dhblaz thanks for your help MiroslavAnashkin and xarses
21:29 dhblaz joined #fuel
21:49 justif ping
22:01 Dr_Drache xarses, nuetron
22:01 Dr_Drache (spelling)
22:02 Dr_Drache sorry, was in a impromptu meeting
22:04 xarses neutron gre or vlan?
22:04 xarses Dr_Drache: the worst // best kind
22:04 Dr_Drache GRE
22:04 Dr_Drache vlans don't want to work
22:04 xarses ha deployment or multi node?
22:05 xarses oh, are you using centos?
22:05 Dr_Drache multinode, and sadly centOS
22:05 Dr_Drache ubuntu won't deploy
22:06 xarses hmm, why no ubuntu?
22:07 Dr_Drache doesn't boot on the controller node.
22:07 Dr_Drache can't find boot device, and none of my logs seem to have helped.
22:08 xarses what kind of hardware was that node?
22:09 Dr_Drache dell R415
22:09 Dr_Drache acually have 3 of them
22:09 Dr_Drache my compute nodes are dell R715s
22:09 Dr_Drache 7 of those
22:13 xarses but the R715's would run ubuntu fine?
22:13 Dr_Drache I can install ubuntu manually and they all run perfectly
22:13 Dr_Drache fuel can't deploy them
22:15 Dr_Drache or, deploys, but won't boot.
22:15 xarses ok, I think it might be a bug we fixed recently, would you be able to test booting one of the machines against the discovery (PXE) image and another off the Ubuntu loader (you can probably start this from the PXE menu too)
22:16 Dr_Drache I would be able to test in the morning EST.
22:16 xarses Otherwise, I'd love to see if we could otherwise fix the issue
22:16 Dr_Drache we have building maintence going on soon, I'm going to have to leave the building.
22:17 Dr_Drache but, if we could get togeather AM EST, I'd love to work with you.
22:18 xarses I'm awake around your 10AM
22:18 Dr_Drache I will be here.
22:18 xarses sometimes slightly earlier
22:21 xarses Dr_Away: might be this bug https://bugs.launchpad.net/fuel/+bug/1277151
22:21 e0ne joined #fuel
23:18 jeremydei joined #fuel
23:18 asledzinskiy joined #fuel
23:21 designated joined #fuel
23:32 mswynnex joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary