Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-05-12

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
03:28 fedexo joined #fuel
04:13 vtzan joined #fuel
04:56 jaypipes joined #fuel
05:08 teran joined #fuel
06:30 dklepikov joined #fuel
06:47 vtzan joined #fuel
06:50 vtzan Dear Team, I am having huge performance issues with ceph. I am using fuel 5.1.1 and i am using ceph for instances/images etc. The performance inside the instances is extremely low
07:10 hyperbaba joined #fuel
07:32 homegrow_ joined #fuel
07:54 akostrikov joined #fuel
07:56 HeOS joined #fuel
07:57 HeOS joined #fuel
08:01 holser_ joined #fuel
08:09 Longgeek joined #fuel
08:17 e0ne joined #fuel
08:27 Longgeek joined #fuel
08:30 Longgeek joined #fuel
08:38 hyperbaba_ joined #fuel
08:49 akupko joined #fuel
08:56 stamak joined #fuel
09:04 akupko joined #fuel
09:33 holser_ joined #fuel
09:58 DaveJ__ joined #fuel
10:10 dklepikov vtzan: Hello. How do you check it?  what kind of network do you use? What version of ceph do you use 'ceph -v'? Do you use RBD cache?
10:29 homegrow_ joined #fuel
11:03 vtzan dklepikov, checked with server instance which had low write performance with dd, also checked with rados bench. ceph is 0.80.4 and the deployement doesn't have rbd cache enabled.
11:03 vtzan dklepikov, the network is 2x10Gbit lacp
11:04 vtzan dklepikov, for each node on cluster
11:07 vtzan dklepikov, enviroment is fuel 5.1.1 HA with ubuntu + GRE
11:23 holser_ joined #fuel
11:53 dklepikov vtzan: please show 'ethtool -k <IFACE>'  for all ifaces (eth*, br-*). post results on http://paste.openstack.org/
11:54 dklepikov vtzan: What was speed in tests?
12:16 vtzan dklepikov, http://paste.openstack.org/show/220758/
12:17 vtzan dklepikov, I have included the speedtest also
12:19 dklepikov vtzan: What replication factor into ceph?
12:19 vtzan dklepikov, replication 2
12:23 dklepikov vtzan: please run on instance 'rm -rf /tmp/test && spew -i 50 -v -d --write -r -b 4096 10M /tmp/test' and post the result. when you will run it please run at the same time  on node-6 and nose-7 'iostat -x 1' and post the results to
12:27 vtzan dklepikov, its running right now. I am waitting to finish
12:27 Longgeek joined #fuel
12:30 vtzan dklepikov, its still very slow, here are the results until now node6 : http://paste.openstack.org/show/220763/ and node7: http://paste.openstack.org/show/220762/
12:30 vtzan dklepikov, I had to stop the spew
12:33 dklepikov what spew showed?
12:33 vtzan dklepikov, still running
12:34 vtzan dklepikov, Can i run a smaller test because performance is struggling
12:36 vtzan dklepikov, check this one http://paste.openstack.org/show/220766/ with smaller test
12:37 dklepikov vtzan: rm -rf /tmp/test && spew -i 4 -v -d --write -r -b 4096 10M /tmp/test
12:37 vtzan dklepikov, its big one and needs way more time to finish. is that ok?
12:38 dklepikov yes, it will write files with size 10M by 4M blocks
12:39 vtzan dklepikov, ok I am waitting to finish. the performance is very very bad
12:41 vtzan dklepikov, customer sites are down because of bad IO can we do something else because the spew strungles the performance way too much
12:43 dklepikov ok
12:43 dklepikov vtzan: please see https://bugs.launchpad.net/fuel/+bug/1275650
12:43 vtzan dklepikov, sorry but we are in a very very bad situation atm regarding this problem.
12:44 dklepikov vtzan:  You have "generic-segmentation-offload: on" on your ifaces
12:45 vtzan dklepikov, should i do it for all interfaces? I have done this on controllers eth2 & eth3
12:46 dklepikov vtzan: What networks are on controller's eth2 & eth3?
12:47 vtzan dklepikov, bonded and it includes all public,storage,management with vlans 100,102,101 accordingly
12:55 vtzan dklepikov, When I first deployed I had a bandwidth problem and i had to put on controlers ethtool --offload eth2 gro off &  ethtool --offload eth3 gro off
12:55 vtzan dklepikov, these are the physicals of the bonded interface
12:55 vtzan dklepikov, do i need the offload flag too?
13:01 tzn joined #fuel
13:01 ppal joined #fuel
13:10 vtzan dklepikov, do you have any other idea that might be helpful?
13:11 dklepikov vtzan:  I ask our experts about gso.
13:11 vtzan dklepikov, ah you did so?
13:12 vtzan dklepikov, just fyi iperf between the nodes is tested and its ~20gbit
13:13 dklepikov vtzan:  What is VM to VM, placed on different computes, speed?
13:14 dklepikov vtzan: and can you show 'ceph -w' for ~1 minute
13:16 vtzan dklepikov, yes here it is: http://paste.openstack.org/show/220827/
13:16 vtzan dklepikov, vm to vm you mean iperf right?
13:23 vtzan dklepikov, the iperf speed between 2 guest servers from different hypervisors is ~ 576Mbits/sec
13:25 vtzan dklepikov, is this limited somewhere?
13:50 dklepikov vtzan: no, not limited
13:56 vtzan dklepikov, maybe in neutron or somewhere that's why i asked
13:56 dklepikov vtzan: no
13:58 vtzan dklepikov, so ~530mbits/sec vm to vm is not so good. Where are the rest?
13:59 vtzan dklepikov, all the network is 2x10gbit lacp for each node/compute/controller/ceph....
14:00 vtzan dklepikov, I am very very curious regarding the ceph part, we have done all according the docs and the performance is very bad :-(
14:01 dklepikov vtzan: RBD cache can increase speed
14:01 vtzan dklepikov, Can i use memory for this?
14:02 vtzan dklepikov, is there any way to enable it on the fly?
14:03 vtzan dklepikov, but still 10 HD x 6 TB serial ata has so bad performance? how is it possible?
14:03 vtzan dklepikov, the disks has 128MB cache each one
14:03 dklepikov vtzan: what OS? what cep version 'ceph -v'?
14:04 vtzan ceph -v
14:04 vtzan ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f) - ubuntu 12.0.4
14:04 vtzan 12.04
14:08 julien_ZTE joined #fuel
14:08 dklepikov vtzan: 80.4 has an issue with it. You need to update it to 80.9
14:09 vtzan dklepikov, aha
14:10 dklepikov vtzan: 80.4 has an issue with librbd
14:10 holser_ joined #fuel
14:11 vtzan dklepikov, do you have any idea what is the recommended way to upgrade?
14:12 vtzan dklepikov, is it fixed on 0.80.7 do you know it
14:12 dklepikov vtzan: yes I do
14:13 dklepikov vtzan: but 80.9 have some fix of a performance regression in librbd
14:13 vtzan dklepikov, I see the 0.80.7 packages on http://mirror.fuel-infra.org/fwm/5.1.2/ubuntu/pool/main/  that's why i asked
14:16 dklepikov vtzan: you can use it
14:17 dklepikov vtzan: please pay attention, that once you upgrade a daemon, you cannot downgrade it.
14:19 vtzan dklepikov, I need to find a way to upgrade to 0.80.9
14:19 vtzan dklepikov, I understand
14:20 vtzan dklepikov, can you provide any guide with the upgrade procedure plz? thank you
14:21 dklepikov vtzan: About gso: turn it off on  physical, bonds, bridges.
14:21 dklepikov vtzan: http://ceph.com/docs/master/install/upgrading-ceph/
14:21 vtzan dklepikov, all of them ok
14:22 vtzan dklepikov, i am just skeptical about fuel that has the old version if i need to deploy an extra node
14:23 vtzan dklepikov, which version do you suggest to go?
14:23 vtzan dklepikov, 0.80.9 is ok?
14:25 dklepikov vtzan:  0,80.9 is ok, but you need to test an update procedure on test environment to prevent data lost. Also create a backup of all configs first
14:26 dklepikov vtzan: and customers buckups to
14:26 vtzan dklepikov, I will test on virtualbox first, the configs are overwriten?
14:26 vtzan dklepikov, I will test yes
14:34 dklepikov vtzan: > if i need to deploy an extra node (update your nailgun repo with new packet, and make some changes into puppet for ceph nodes)
14:37 vtzan dklepikov, ok that makes sense. thank you
14:39 vtzan dklepikov, thank you for your help. I really appreciate it.
14:41 emagana joined #fuel
14:52 rmoe joined #fuel
15:14 blahRus joined #fuel
15:34 vtzan exit
15:45 jaypipes joined #fuel
15:55 daniel3_ joined #fuel
16:08 holser_ joined #fuel
16:17 xarses joined #fuel
16:50 emagana joined #fuel
17:02 asilenkov_ joined #fuel
17:02 jobewan_ joined #fuel
17:02 kaliya joined #fuel
17:02 emagana joined #fuel
17:04 championofcyrodi joined #fuel
17:08 championofcyrodi left #fuel
17:26 holser_ joined #fuel
18:11 holser_ joined #fuel
18:11 angdraug joined #fuel
18:37 MaverickHunter joined #fuel
18:38 julien_ZTE joined #fuel
18:51 e0ne joined #fuel
18:52 stamak joined #fuel
18:59 holser_ joined #fuel
19:04 HeOS joined #fuel
20:38 julien_ZTE joined #fuel
20:44 stamak joined #fuel
20:53 e0ne joined #fuel
21:30 emagana joined #fuel
21:40 angdraug joined #fuel
21:46 jobewan joined #fuel
21:59 DaveJ__ joined #fuel
22:08 holser_ joined #fuel
22:12 championofcyrodi joined #fuel
22:19 championofcyrodi left #fuel
23:47 rmoe joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary