Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 jermudge_ OK. So you have one brick that's online and good, right? And one brick that's offline? Is the offline brick the failed brick or its replacement?
00:01 redsolar joined #gluster
00:07 jisk both bricks were fine before this command
00:07 jisk the offline brick is the one I am trying to replace, which was OK until the replace-brick command
00:08 jermudge_ so the replace-brick command failed to change the entry in the volume info? it still points to the old brick? or did you try to replace a brick with the same path?
00:10 jisk I issued this command:  volume replace-brick storage1 ss2:/exports/exp1 ss2:/exports/exp3 start
00:10 jisk After which it took /exports/exp1 offline and spewed into the log
00:11 jisk I've just commit forced it as I needed to get the volume back oniine
00:12 jisk at least the data is still there, I'm just going to have to merge it
00:23 jisk Thanks jermudge_
00:23 jisk It would be nice if gluster didn't randomly break on me like it has so many times :-(
00:34 joeto joined #gluster
00:36 kwevers joined #gluster
00:39 hchiramm_ joined #gluster
00:39 red_solar joined #gluster
00:40 redsolar_office joined #gluster
00:43 redsolar joined #gluster
00:46 red_solar joined #gluster
00:57 FyreFoX anyone still around?
01:01 FyreFoX interestingly its only when I use  -p on the cp command
01:06 JoeJulian I wish a2 could look at that.
01:07 JoeJulian 2 things, one, I was really tired yesterday and though it was the timing thing where the mount was happening before the brick server started.
01:07 JoeJulian Sorry for identifying your problem incorrectly.
01:08 FyreFoX thats ok
01:08 JoeJulian Since you're actually hitting a crash bug, please file a bug report.
01:08 glusterbot http://goo.gl/UUuCq
01:09 FyreFoX ok
01:12 JoeJulian That'll be the best way to get the devs themselves to take a look at it.
01:16 FyreFoX yep
01:21 redsolar joined #gluster
01:28 yinyin joined #gluster
01:30 hchiramm_ joined #gluster
01:31 bulde joined #gluster
01:45 red_solar joined #gluster
01:47 redsolar_office joined #gluster
01:59 jermudgeon joined #gluster
02:04 redsolar joined #gluster
02:07 __Bryan__ joined #gluster
02:16 FyreFoX JoeJulian: which component should I use?
02:23 red_solar joined #gluster
02:31 hagarth joined #gluster
02:33 yinyin joined #gluster
02:47 jermudge_ joined #gluster
02:57 semiosis FyreFoX: sorry i'm not up to speed on your issue, but it's probably safe to pick glusterfs core and whoever triages your bug will change it as necessary
03:04 johnmark greetings yall
03:05 johnmark ive been in san diego at the lisa conference
03:06 m0zes anyone here using ganeti to manage their vms on glusterfs?
03:06 m0zes I'm looking into it today...
03:06 johnmark after talking to some postgres peeps im convinced that using glusterfs with postgres + libgfapi should be possible
03:07 johnmark m0zes: oh?
03:07 y4m4 joined #gluster
03:07 johnmark sounds imminenrtly possible
03:07 JoeJulian Oh, my
03:07 m0zes johnmark: libgfapi and postgres sounds awesome :D
03:08 johnmark m0zes: i think the open spource lab at oregon state has done that
03:08 johnmark m0zes: heh :-)
03:08 glusterbot New news from newglusterbugs: [Bug 887098] gluster mount crashes <http://goo.gl/QjeMP>
03:08 johnmark joeju
03:08 johnmark oops... i hate connectbot
03:08 JoeJulian heh
03:09 johnmark yeah, they do all their writes async
03:09 johnmark so not particularly sensitive to latency
03:10 JoeJulian Wait, so you're saying multiple servers could share the data stored on gluster? Can it be sharded to take advantage of distribute?
03:10 johnmark and now that libgfapi has much of the posix stuff it *should* be possible
03:10 m0zes johnmark: Ramereth gave a talk about ganeti at k-state today. I don't think the ganeti folks added any functionality to handle qemu's libgfapi stuff yet, though.
03:10 johnmark thats what im thinkig
03:11 johnmark m0zes: interesting
03:11 swkide_ joined #gluster
03:12 johnmark joejulian: theres probably some customization that needs to happen but it should be possible
03:12 JoeJulian That's the first sign I've yet seen that would allow me to continue leaving sql in my future design plans.
03:13 johnmark and i thought about you and mysql thinking, ok, maybe we just missed this opportunity
03:13 johnmark heh ok
03:14 johnmark m0zes: the qemu integration is very bleeding edge. give it time :-)
03:14 JoeJulian I wonder if Monty would be interested in doing (or able to do) that with the mysql backends.
03:14 johnmark joeju
03:14 johnmark grrr
03:14 JoeJulian ... via mariadb
03:15 m0zes johnmark: I know. I jsut saw the qemu docs updated with glusterfs stuff, and it makes me want to play with it that much sooner ;)
03:15 johnmark joejulian: you read my mind
03:17 hagarth joined #gluster
03:18 johnmark m0zes: me too
03:18 johnmark so yeah,
03:19 johnmark lots of interesring things are possible with libgfapi
03:19 johnmark i think...
03:21 johnmark joejulian: theyre just starting the mariadb foundation
03:21 JoeJulian Yeah, I saw that just now too.
03:22 johnmark so now is a good time to talk to them
03:25 johnmark m0zes: speaking of kstate thats still on my list of unis to talk to
03:27 m0zes johnmark: hooray :)
03:28 m0zes I got an e-mail from Jason today. I'm going to respond in the morning :)
03:28 johnmark m0zes: ah good. was wondering about that
03:29 johnmark fudcon is in january, so if i can combine 2 things in one trip then its all good
03:31 johnmark m0zes: and thats in lawrence
03:31 m0zes we'll probably still be trying to finish up our power and cooling upgrades then, but I certainly wouldn't mind giving you the tour.
03:31 johnmark oh oj
03:31 johnmark er ok
03:32 johnmark on another note, if someone can show me how to do tab complete with connectbot ill be yours forever
03:32 m0zes I know our LUG here is planning on going FUDCon... I'm probably going to go with them :)
03:32 johnmark ah cool ok
03:34 johnmark anyway gotta get ready tp board. see yall 'round
03:39 daMaestro joined #gluster
03:50 m0zes johnmark: ctrl+i is tab sequence. or install hacker keyboard.
03:50 JoeJulian +1 hacker keyboard
03:51 m0zes hacker keyboard is the only reason I can actually work on my phone ;) (in the unfortunate event where I have to.)
03:51 mdarade1 joined #gluster
03:52 m0zes the two android phones I had before this one had hardware keyboards (G1 and G2). it just became too much work to find a *good* /rooted/ phone with a hardware keyboard, so now I have an SGS3.
03:54 m0zes I have to work from my phone less, now that I actually have hired a "clone" of my self. still bringing him up to speed, but he can be on call without me sometimes...
04:14 mdarade1 left #gluster
04:15 yinyin joined #gluster
04:17 mohankumar joined #gluster
04:34 FyreFoX thanks semiosis
04:38 hagarth joined #gluster
04:55 shylesh joined #gluster
04:55 shylesh_ joined #gluster
04:59 vpshastry joined #gluster
05:06 y4m4 joined #gluster
05:08 sgowda joined #gluster
05:16 harshpb joined #gluster
05:20 yinyin joined #gluster
05:25 yinyin_ joined #gluster
05:32 pranithk joined #gluster
05:33 pranithk JoeJulian: ping
05:36 bala joined #gluster
05:49 hagarth joined #gluster
05:50 ngoswami joined #gluster
06:02 ramkrsna joined #gluster
06:09 rastar joined #gluster
06:23 vimal joined #gluster
06:27 yinyin joined #gluster
06:27 bala joined #gluster
06:28 swkide left #gluster
06:36 harshpb joined #gluster
06:43 harshpb joined #gluster
06:47 raghu joined #gluster
06:55 harshpb joined #gluster
06:57 harshpb joined #gluster
06:57 mohankumar joined #gluster
06:58 daMaestro left #gluster
07:01 rgustafs joined #gluster
07:02 harshpb joined #gluster
07:08 dobber joined #gluster
07:11 harshpb joined #gluster
07:15 harshpb joined #gluster
07:16 bala joined #gluster
07:19 harshpb joined #gluster
07:19 guigui1 joined #gluster
07:22 harshpb joined #gluster
07:23 harshpb joined #gluster
07:40 mrquesty joined #gluster
07:45 mrquesty Hi Guys, I'm running the latest Gluster (3.3.1) on the latest CentOS (6.3 with kernel 2.6.32-279.11.1.el6.centos.plus.x86_64) with ext4. There is only a single node (replica factor 1) and the mount goes thru fuse.glusterfs on a localhost. glusterfs process eats up all RAM and goes to SWAP rather frequently. There is nothing suspicious in logs and the configuration is just basic. Any ideas, what could be done to eliminate this (this really stops machi
07:45 mrquesty ne from being productive) :(
07:45 JoeJulian First of all, ,,(ext4)...
07:45 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
07:45 JoeJulian How much ram?
07:46 mrquesty 8GB
07:46 mrquesty Yep, i read about ext4 bug, but it's solved in 3.3.1, ain't it?
07:46 JoeJulian no
07:47 mrquesty Any work around except to migrate to XFS?
07:47 JoeJulian None that aren't in that article.
07:49 mrquesty OK, got it. Thanks, Joe
07:50 mohankumar joined #gluster
07:54 Nevan joined #gluster
07:54 puebele joined #gluster
07:54 ekuric joined #gluster
07:58 mohankumar joined #gluster
08:02 Nr18 joined #gluster
08:03 harshpb joined #gluster
08:06 michaelb_ joined #gluster
08:08 harshpb joined #gluster
08:09 bharata joined #gluster
08:09 nightwalk joined #gluster
08:13 puebele joined #gluster
08:18 michaelb_ Hello, I was wondering if there's someone who could give me some advice. I've got 3 servers in as many separate data centers and need to keep a few directories in sync. Only one of the servers is being used at any one time but if the 'live' server goes down, failover and fallback/restore should not involve any administrative effort as far as the file replication is concerned.  Regular gluster replication isn't performing well (a
08:19 michaelb_ Is gluster just not the right tool for this job or is there a configuration that would meet my needs? Geo replication appears to require intervention on restoration of service from a failure and the docs don't say anything about a scenario/configuration that might fix that: ie A=>B=>C=>A.
08:20 mdarade1 joined #gluster
08:20 mgebbe_ joined #gluster
08:21 JoeJulian I haven't heard of anybody finding the magic to handle that. No matter what, if there's writes being done to this storage it's a recipe for split-brain.
08:21 JoeJulian See my blog @ http://joejulian.name and part 1 of why replication is hard.
08:21 glusterbot Title: JoeJulian.name (at joejulian.name)
08:23 michaelb_ I've got MySQL running with Galera, which holds a quorum and recovers from such a failure scenario quite nicely as long as 2 servers are alive and talking. They are then considered the authorative source of data.
08:23 andreask joined #gluster
08:23 michaelb_ Was hoping the same might be possible for regular files somehow.
08:24 JoeJulian Not yet. jdarcy's working on all that and a pony though. ;)
08:27 mdarade1 left #gluster
08:32 michaelb_ Thanks for the link to your blog, will have a read. Having done some research I know of some of the issues surrounding this kind of synchronisation of data.
08:32 JoeJulian Cool, you're ahead of most of the people out there then. :)
08:33 michaelb_ It just seems so odd that a problem like this is fairly common yet no good solution has been found to it yet.
08:42 michaelb_ Will think of something suitable I'm sure, thanks again.
08:42 JoeJulian You're welcome
08:42 JoeJulian One thing I've heard of...
08:43 JoeJulian Doing two separate volumes, one at each end, that geosynch's to the other. Apparently they're doing some sort of heartbeat thing to automate failover.
08:44 mooperd joined #gluster
08:48 michaelb_ Hmm, will investigate that. Or maybe keep the normal replicate gluster cluster but let clients write and read from the regular file system (nothing to do with the cluster) and use intotify events to move / update files into (and out of - on remote nodes) the cluster. speed improve for the clients while keeping all the distribute magic happening at its own pace.
08:50 michaelb_ inotify*
08:51 JoeJulian Let me know if you come up with anything. I'm sure it would be popular.
08:51 hagarth joined #gluster
08:56 yinyin_ joined #gluster
09:00 harshpb joined #gluster
09:00 guest2012 joined #gluster
09:04 harshpb joined #gluster
09:05 Uguu joined #gluster
09:10 harshpb joined #gluster
09:14 gbrand_ joined #gluster
09:15 ramkrsna joined #gluster
09:16 harshpb joined #gluster
09:17 michaelb_ Will do, cheers. Have a nice weekend.
09:24 x4rlos JoeJulian: Nice blog. We have a similar setup to your environemnt too.
09:26 JoeJulian x4rlos: thanks
09:28 x4rlos freeswitch vs. asterisk?
09:28 36DACLOFQ joined #gluster
09:28 JoeJulian Yep, we're using freeswitch.
09:30 manik joined #gluster
09:32 duerF joined #gluster
09:39 vpshastry joined #gluster
09:43 sripathi joined #gluster
09:45 x4rlos Nice to see you mentioned drbd too. This was an option.
09:48 yinyin joined #gluster
09:50 rastar joined #gluster
09:52 JoeJulian I can't remember if I said anything bad about that... I know I was thinking horribly dark things.
09:52 x4rlos hehe. Biggest problem is that its master->slave scenario.
09:53 JoeJulian Even bigger is that everything's in one big binary blob.
09:54 x4rlos oh, yeah, thats bad :-)
09:55 x4rlos We wanted it originally just for database write-logs, but now looking to implment it for user shares and possibly backups.
09:55 passie joined #gluster
10:00 x4rlos Gluster seems to be the way.
10:13 shireesh joined #gluster
10:15 dblack|away joined #gluster
10:16 Jippi joined #gluster
10:17 kkeithley joined #gluster
10:18 21WAAE2GZ joined #gluster
10:18 21WAAE2GZ left #gluster
10:25 bala joined #gluster
10:26 nissim just read the following blog: http://blog.jebpages.com/archives/f​edora-17-openstack-and-gluster-3-3/ , used its rpm repository for glusterfs 3.3.1-4 on fedora17, created a regular distrbuted volume over 5 nodes ontop ipoib, created a file but ls vol/ shows nothing...
10:26 glusterbot <http://goo.gl/iPBfL> (at blog.jebpages.com)
10:27 nissim I breaking my head here, installed different versions of gluster over and over again but nothing is working as claimed in their documentation.
10:27 nissim at first I used rdma which worked for 2|3 nodes, added another node/brick and ls volume stuck again
10:28 JoeJulian I'm opening Jason's page so I can see where you're having trouble.
10:28 nissim tested tcp instead of rdma and again ls volume over more than 2/3 nodes gives empty folder although I created a 10GB file few seconds before
10:28 nissim please do
10:29 nissim I will be happy to get help here, since I am not sure I can build on such a solution if it gives me so many problems
10:29 nissim using EXT4
10:32 JoeJulian nissim: Ah... that explains it. ,,(ext4)
10:32 glusterbot nissim: Read about the ext4 problem at http://goo.gl/PEBQU
10:33 vpshastry joined #gluster
10:35 the-me interesting
10:35 x4rlos "as NFSv2 wasn't designed to accomidate anything larger" <-- this why we only use nfs v3?
10:35 JoeJulian Yeah, that's one word for it. ;)
10:36 JoeJulian Gluster implemented v3 initially because it was a project that was a manageable size.
10:37 x4rlos I was using ext4 for my test bed, i have v3.3.1. Im gonna have to read up on that :-)
10:37 JoeJulian Easiest thing to do is just use xfs for your bricks if you're starting from scratch.
10:40 dbruhn joined #gluster
10:40 mdarade1 joined #gluster
10:40 passie joined #gluster
10:43 x4rlos Any problems hosting these on an lvm2 partition? :-)
10:43 harshpb joined #gluster
10:44 * x4rlos /note to self: use xfs.
10:44 JoeJulian lvm2 is what I use.
10:44 x4rlos good to know :-)
10:48 vpshastry joined #gluster
10:49 harshpb joined #gluster
10:52 harshpb joined #gluster
10:53 ekuric joined #gluster
10:55 jmara joined #gluster
10:55 sgowda joined #gluster
11:01 manik joined #gluster
11:04 ekuric joined #gluster
11:09 nissim joined #gluster
11:10 nissim Thanks Julian, will reformat my storage as XFS and give it another try
11:10 harshpb joined #gluster
11:15 harshpb joined #gluster
11:15 rgustafs joined #gluster
11:16 kevein joined #gluster
11:17 x4rlos Just gonna firewall off my instances of gluster, and try and write to both nodes with conflicting content in a file.
11:18 x4rlos Anyone know what i should expect to see?
11:24 nissim have a look at this, creating a 10GB file over RDMA takes 33sec vs less than 10sec locally on the same disk used by gluster : [root@nova-node1 instances]# time dd if=/dev/zero of=swapfile bs=1M count=10240
11:24 nissim 10240+0 records in
11:24 nissim 10240+0 records out
11:24 nissim 10737418240 bytes (11 GB) copied, 33.7866 s, 318 MB/s
11:24 nissim real    0m33.797s
11:24 nissim user    0m0.011s
11:24 nissim sys     0m9.148s
11:25 nissim this time I created a distributed volume on top of XFS based on Julian's recommendation & link
11:26 nissim Julian, do you have ant idea how performance can be increased here? gluster optimization ???
11:27 nissim also, my glusterfs nfs.log is full of these: [2012-12-14 13:26:52.016412] E [rdma.c:4604:tcp_connect_finish] 0-nova-vol1-client-0: tcp connect to  failed (Connection refused)
11:27 nissim [2012-12-14 13:26:52.016530] W [rdma.c:4187:gf_rdma_disconnect] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x57f9) [0x7f2b3d0aa7f9]))) 0-nova-vol1-client-0: disconnect called (peer:)
11:27 nissim [2012-12-14 13:26:52.016617] W [rdma.c:4521:gf_rdma_handshake_pollerr] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x5690) [0x7f2b3d0aa690]))) 0-rpc-transport/rdma: nova-vol1-client-0: peer () disconnected, cleaning up
11:27 nissim [2012-12-14 13:26:52.020067] E [rdma.c:4604:tcp_connect_finish] 0-nova-vol1-client-1: tcp connect to  failed (Connection refused)
11:27 nissim [2012-12-14 13:26:52.020185] W [rdma.c:4187:gf_rdma_disconnect] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x57f9) [0x7f2b3d0aa7f9]))) 0-nova-vol1-client-1: disconnect called (peer:)
11:27 nissim [2012-12-14 13:26:52.020259] W [rdma.c:4521:gf_rdma_handshake_pollerr] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x5690) [0x7f2b3d0aa690]))) 0-rpc-transport/rdma: nova-vol1-client-1: peer () disconnected, cleaning up
11:27 nissim [2012-12-14 13:26:52.023480] E [rdma.c:4604:tcp_connect_finish] 0-nova-vol1-client-2: tcp connect to  failed (Connection refused)
11:27 nissim [2012-12-14 13:26:52.023597] W [rdma.c:4187:gf_rdma_disconnect] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x57f9) [0x7f2b3d0aa7f9]))) 0-nova-vol1-client-2: disconnect called (peer:)
11:27 nissim [2012-12-14 13:26:52.023671] W [rdma.c:4521:gf_rdma_handshake_pollerr] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x5690) [0x7f2b3d0aa690]))) 0-rpc-transport/rdma: nova-vol1-client-2: peer () disconnected, cleaning up
11:27 nissim [2012-12-14 13:26:52.026993] E [rdma.c:4604:tcp_connect_finish] 0-nova-vol1-client-3: tcp connect to  failed (Connection refused)
11:27 nissim [2012-12-14 13:26:52.027109] W [rdma.c:4187:gf_rdma_disconnect] (-->/usr/sbin/glusterfs(main+0x344) [0x4047e4] (-->/lib64/libglusterfs.so.0(+0x3e658) [0x7f2b43b35658] (-->/usr/lib64/glusterfs/3.3.1/​rpc-transport/rdma.so(+0x57f9) [0x7f2b3d0aa7f9]))) 0-nova-vol1-client-3: disconnect called (peer:)
11:27 nissim was kicked by glusterbot: message flood detected
11:29 nissim joined #gluster
11:29 nissim sorry, disconnected from some reason
11:30 z00dax nissim: no, you were kicked out for pasting stuff in here use a pastebin instead
11:30 nissim I am sorry about that
11:30 nissim didnt know I am not alowed
11:30 nissim won't do it again
11:31 shireesh joined #gluster
11:32 nissim z00dax?
11:32 z00dax I'm easy, i just saw the kick happen,
11:33 x4rlos When a client writes a file to gluster over nfs - does it pass both files from the client as the glusterfs mount does, or does server1 pass to server2 as it would over a samba share?
11:33 nissim ok, I didnt know that I am not allowed to paste ...
11:33 bala joined #gluster
11:33 x4rlos or maybe i misunderstood both implementations.
11:34 nissim anyway, assuming you saw what I pasted, is there a solution for these messages?
11:35 sgowda left #gluster
11:36 nissim How can I optimize gluster performance?
11:36 nissim anyone?
11:44 sripathi joined #gluster
11:49 mooperd joined #gluster
11:52 mdarade joined #gluster
11:52 mdarade left #gluster
12:03 edward1 joined #gluster
12:09 x4rlos In the event of a split brain scenario, and 'testfile.txt' has contents on server1: "this is server1" and server2: "this is server2" - I then reenable the connection between the two - selfheal seems to leave the both in their separate states.
12:13 x4rlos this something that you can control? Not that i dont believe that this is not desired behaviour :-)
12:13 x4rlos all them double negatives, haha.
12:22 rudimeyer joined #gluster
12:24 rudimeyer_ joined #gluster
12:24 nissim I move to XFS based on Julian recommedation and now cluster is working, only problem is the locally I can write at 1.5GB/s while on gluster write speed is reduced to 300MB/s. Can anyone help undertstand why such a huge difference?
12:24 ekuric joined #gluster
12:28 hagarth joined #gluster
12:33 bala joined #gluster
12:35 ekuric1 joined #gluster
12:48 bauruine joined #gluster
12:49 jtux joined #gluster
12:50 harshpb joined #gluster
12:55 shylesh joined #gluster
12:55 shylesh_ joined #gluster
12:56 samcooke joined #gluster
12:58 harshpb joined #gluster
13:03 x4rlos Can a trigger be set in gluster so if it has a file conflict error it does something? So i could say "on error 1234 (content mismatch in files on bricks) do 'mv %f %f.$(+%d%m%y-%H%M-%S)' on: OLDEST_FILE" ?
13:04 harshpb joined #gluster
13:11 x4rlos Is there a consistency-check that can be run on replica bricks to check they are in sync?
13:12 harshpb joined #gluster
13:16 harshpb joined #gluster
13:16 x4rlos hmmm. using manual volume heal says successful on file contents when there are differences.
13:24 samcooke We're currently rebalancing our volume and gluster seems to be holding open files for every file it moves for the duration of the rebalance (in our case 100,000s) - the only mention of this I've found online is neofob mentioning it on IRC before - has anyone else seen this before? Is it a bug or intended?
13:27 rgustafs joined #gluster
13:27 tryggvil joined #gluster
13:29 hagarth joined #gluster
13:45 manik joined #gluster
13:45 rudimeyer_ joined #gluster
13:49 plarsen joined #gluster
13:49 ekuric joined #gluster
13:50 Humble joined #gluster
13:53 aliguori joined #gluster
13:54 efries joined #gluster
13:56 alan_ joined #gluster
13:57 passie left #gluster
13:57 Guest835 hi, can someone help me to create raid 10 on 3.3
14:01 kkeithley Gluster doesn't do RAID-10
14:01 Guest835 http://community.gluster.org/q/can-i-create-distr​ibuted-striped-and-replicated-volume-like-raid10/
14:01 glusterbot <http://goo.gl/eUErl> (at community.gluster.org)
14:03 Guest835 I want have mirror then stripe mirrors
14:03 m0zes Guest835: @stripe is not something you probably want.
14:04 m0zes @stripe
14:04 glusterbot m0zes: Please see http://goo.gl/5ohqd about stripe volumes.
14:05 Guest835 stripe it's only for win performance?
14:06 kkeithley stripe is mainly for when you want to write files that are bigger than the individual bricks.
14:08 Guest835 hmm, you want say impossible with bluster have stripe over lot mirrors? Like raid10?
14:09 Guest835 we try find solutions for backup storage
14:10 glusterbot New news from newglusterbugs: [Bug 887268] glusterfsd process crashed <http://goo.gl/TSYCO>
14:11 kkeithley There's no parity when you use stripe. RAID 10 is RAID 0 (mirror) + RAID 1 (stripe with parity)
14:12 chirino joined #gluster
14:12 inodb joined #gluster
14:17 Guest835 ok, can I have big filesystem on gluster over 5-10 servers? And have availability?
14:18 shylesh joined #gluster
14:20 kkeithley yes, certainly. That's the point of gluster. (Even though there's no RAID 1 and thus no RAID 10.) You can make a replica+distribute volume using your 5-10 servers.
14:22 kkeithley Distribute (dht) lets you aggregate the bricks into a large volume, replication (afr, mirroring) gives you availability.
14:23 Guest835 and then i can without any impact miss one server?
14:24 Guest835 or one brick
14:26 x4rlos JoeJulian: http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/ <--- I have an instance where it cannot detect split brain on file contents.
14:26 glusterbot <http://goo.gl/FPFUX> (at joejulian.name)
14:26 harshpb joined #gluster
14:28 Guest835 kkeithley: Can you post link to manual, how configure distribute fs. I'm new on bluster and don't understand diffidents between dht and far
14:28 Guest835 @gluster
14:28 glusterbot Guest835: I do not know about 'gluster', but I do know about these similar topics: 'Can I use glusterfs on an i386 platform', 'What is the difference between Gluster Storage Platform (GSP) and GlusterFS?', 'What ports does glusterfs use for nfs?', 'python-gluster'
14:30 kkeithley @manual
14:31 Guest835 @manual
14:31 Guest835 ok
14:31 kkeithley c'mon glusterbot, you can do it
14:33 kkeithley There's docs on www.gluster.org. AFR is the gluster term for mirroring, DHT is distribute. If you have four bricks, you create a afr+dht volume with `gluster volume create $volname replica 2 node1:$path_to_brick1 node2:$path_to_brick2 node3:$path_to_brick3 node4:$path_to_brick4.
14:34 kkeithley If each brick is 2TB, that'll give you a 4TB mirrored volume.
14:34 ekuric joined #gluster
14:35 kkeithley gah, there is docs. There are docs on www.gluster.org. AFR is the gluster term for mirroring, DHT is distribute. If you have four bricks, you create a afr+dht volume with `gluster volume create $volname replica 2 node1:$path_to_brick1 node2:$path_to_brick2 node3:$path_to_brick3 node4:$path_to_brick4`
14:37 m0zes @rtfm
14:37 glusterbot m0zes: Read the fairly-adequate manual at http://goo.gl/E3Jis
14:38 harshpb joined #gluster
14:38 Guest835 yes, I try it, work fine. Now i want do s1:hdd1 mirro s2:hdd1, s1:hdd2 mirro s2:hdd2, s1:hdd3 mirro s2:hdd3 …. ant marge all in one
14:40 Guest835 and when i lost one disk or one server i have not problems
14:40 kkeithley Just do what I showed you above, that's what you'll get what you want
14:40 Guest835 ok, thank you
14:40 glusterbot New news from newglusterbugs: [Bug 887272] glusterfs process crashed <http://goo.gl/HCF2B>
14:40 kkeithley gah, my writing this morning is the pits
14:40 harshpb joined #gluster
14:41 pepe123 hello there again : ) are there suppose to be any issues when using a server in 3.2.5 and clients in 3.0.2 (using gluster native client and ubuntu)
14:42 harshpb joined #gluster
14:42 kkeithley I would seriously surprised  if 3.0.2 native clients would work with 3.2.x servers.
14:43 x4rlos me too. It actually mounts even?
14:43 harshpb joined #gluster
14:45 harshpb joined #gluster
14:46 harshpb joined #gluster
14:46 shylesh joined #gluster
14:46 shylesh_ joined #gluster
14:49 Guest835 kkeithley: yes, work fine. And if i want add new server i can do it "volume add-brick <VOLNAME> replica 2 s1:/hdd4 s2:hdd5" ?
14:50 kkeithley yes
14:55 Guest835 kkeithley: nice. and last questions, have you some technology like radosgw in cephfs? One mountserver proxy who know loads and can forward to less loaded server.
14:57 noob2 joined #gluster
14:57 kkeithley Gluster's REST interface (object store) is Unified File and Object (UFO).
14:58 stopbit joined #gluster
14:58 khushildep joined #gluster
14:58 kkeithley I'd argue that gluster doesn't need a mountserver proxy. When you use AFR (replication, mirror) the writes go to both mirrors regardless of the load. Read requests are sent to both mirrors, and the first one to respond will almost certainly be the less loaded one.
15:00 Guest835 very good! Thank you!!!
15:02 wN joined #gluster
15:07 tryggvil joined #gluster
15:07 neofob left #gluster
15:08 tryggvil_ joined #gluster
15:09 bala joined #gluster
15:14 pepe123 <kkeithley>: actually if i download the .vol file to the client the mount works but then it shows  the "Transport endpoint is not connected" when doing cd
15:15 pepe123 but the mount -f glusterfs method doesnt work at all
15:16 kkeithley `mount -f ...` is a "fake" mount. from the mount man page: ...everything to be done except for the ac;tual system call....
15:17 kkeithley did you mean `mount -t glusterfs ...`?
15:18 kkeithley And maybe the gluster rpc calls for mount didn't change between 3.0.x and 3.2.x, so mount works, but for the rest of the f_ops?
15:21 wushudoin joined #gluster
15:25 shylesh joined #gluster
15:26 x4rlos what does afr stand for btw?
15:28 bennyturns joined #gluster
15:30 andreask Advanced File Replication
15:30 x4rlos thanksyou :-)
15:36 glusterbot New news from resolvedglusterbugs: [Bug 887274] Fix format string usage in gf_log/cli_out <http://goo.gl/RDo3M>
15:46 y4m4 joined #gluster
15:48 samcooke joined #gluster
15:49 daMaestro joined #gluster
15:51 ekuric joined #gluster
15:55 dstywho joined #gluster
15:57 sripathi joined #gluster
16:03 shylesh joined #gluster
16:08 cicero joined #gluster
16:09 semiosis cicero: ?
16:09 cicero semiosis: thx for the ppas -- how would i get it installed on natty?
16:09 semiosis do-release-upgrade :)
16:09 cicero sorry, for 3.3
16:10 semiosis seriously though a few people have been asking for packages for older ubuntus, mostly lucid (meh)
16:10 semiosis maybe i ought to do some builds
16:11 cicero that would be swell
16:11 glusterbot New news from newglusterbugs: [Bug 887301] Container listing of objects is sorted resulting is higher latency in response time <http://goo.gl/789il>
16:11 cicero i'm happy to compile things myself but i haven't figured out ppas just yet
16:13 cicero in the meantime, i'm connecting to my brand new 3.3 volume via nfs... SPoF but i can deal
16:13 cicero i mean, my natty & non-precise clients
16:13 cicero anyhow, thanks again for the PPAs - they're awesome!
16:14 semiosis yw
16:17 chandank joined #gluster
16:20 tryggvil joined #gluster
16:32 shylesh joined #gluster
16:34 neofob joined #gluster
16:35 samcooke Hi, has anyone had the experience of a rebalance keeping lots of files open? It looks like it keeps every file it moves open for the duration of the rebalance
16:36 wushudoin joined #gluster
16:40 manik joined #gluster
16:40 mooperd joined #gluster
16:53 samcooke joined #gluster
17:05 rastar joined #gluster
17:09 dbruhn Is the performance read-ahead option still a tunable option, I am not seeing it in the manual.
17:11 glusterbot New news from newglusterbugs: [Bug 884381] Implement observer feature to make quorum useful for replica 2 volumes <http://goo.gl/rsyR6>
17:11 elyograg hmm.  didn't know glusterbot sent notices when you update an existing bug.
17:16 bulde joined #gluster
17:17 dbruhn Are there any disadvantages to increasing the read cache, if ram is available?
17:20 Humble joined #gluster
17:23 nightwalk joined #gluster
17:35 wushudoin joined #gluster
17:39 chirino joined #gluster
17:44 Gualicho joined #gluster
17:44 Gualicho hey guys
17:44 Gualicho qq: How do I set the performance options on the client side?
17:45 Gualicho for example: performance/io-cache
17:45 Gualicho I know how to do it on the server, using the "gluster" command
17:46 Gualicho but docs are not clear about the options on the glusterfs native client
17:48 nightwalk joined #gluster
17:51 chirino joined #gluster
17:59 semiosis Gualicho: my advice: dont
17:59 semiosis and generally speaking all options should be set via gluster command... some will be set server site, others will be set client side
18:00 semiosis s/server site/server side/
18:00 glusterbot What semiosis meant to say was: and generally speaking all options should be set via gluster command... some will be set server side, others will be set client side
18:00 semiosis after changing volume options you can review the generated vol files (but don't edit them) in /var/lib/glusterd/vols/<VOLNAME>/*.vol
18:00 semiosis client is -fuse.vol
18:01 semiosis check out that -fuse.vol and see, maybe that option *did* change client-side settings
18:03 Gualicho semiosis, thanks, I changed everything using gluster command
18:03 Gualicho but that works on the server side, where glusterd is running
18:03 Gualicho how do I do that on the client side?
18:03 Gualicho where I only use mount -t glusterfs
18:03 Gualicho there's no daemon running on the client, and I didn't install the glusterd rpm, so there's no gluster command
18:04 kkeithley the client-side mount is going to retrieve the client-side -fuse.vol file where the options have been set by the gluster command.
18:04 kkeithley clients that are already mounted will have to unmount and remount to pick up the new option(s)
18:05 Gualicho oh, ok, so everything I set using gluster set volume option, is also picked up by the client
18:06 kkeithley should be
18:06 Gualicho thanks, that explains it
18:06 kkeithley that's what semiosis meant when he wrote: check out that -fuse.vol and see, maybe that option *did* change client-side settings
18:07 Gualicho ok, I will
18:09 chirino joined #gluster
18:17 RobertLaptop joined #gluster
18:17 tryggvil joined #gluster
18:23 Mo__ joined #gluster
18:27 zaitcev joined #gluster
18:29 y4m4 joined #gluster
18:42 quillo joined #gluster
18:52 wushudoin joined #gluster
18:54 gbrand_ joined #gluster
18:57 nightwalk joined #gluster
19:12 y4m4 joined #gluster
19:13 rudimeyer joined #gluster
19:21 wN joined #gluster
19:23 wN joined #gluster
19:23 jiffe98 you guys do those office hours anymore?
19:24 rbennacer joined #gluster
19:25 rbennacer is it necessary to rebalance the bricks when you add a new node?
19:28 JoeJulian That depends on whether your node is a client or a server. Assuming server you should at least the fix-layout portion to make new files use the new brick.
19:29 rbennacer so all my previous bricks are 99% full, and this new node is a server
19:29 noob2 JoeJulian: i'm noticing on a lot of the client machines that fuse seems to disconnect from one brick once in awhile and never reconnect.  is that a problem with the 3.3.0-x fuse client?
19:29 JoeJulian yep
19:29 noob2 ok
19:29 rbennacer upgrade to 3.3.1
19:29 noob2 so i need to get moving on that upgrade
19:30 noob2 can i upgrade just the clients without upgrading the servers?
19:30 wushudoin joined #gluster
19:30 rbennacer mmm i think you have to upgrade the servers too
19:30 noob2 darn
19:30 rbennacer it is very easy to do it though
19:30 noob2 i'm worried about doing that
19:31 noob2 how did you go about it?
19:31 rbennacer i did it a week ago and everything is still working :)
19:31 noob2 lol
19:31 JoeJulian rbennacer: I've been reading reports from people (or at least one person over and over again) saying that rebalance has issues with bricks that full. I'd be interested in how it goes for you and diagnosing any problems you might have with it.
19:32 rbennacer JoeJulian, i just want to prevent future issue, for now i am not experiencing any problem
19:32 rbennacer well, not yet
19:33 noob2 rbennacer: did you just shutdown a node, upgrade and then restart?
19:33 noob2 or did you shutdown the entire cluster and upgarde
19:33 rbennacer nope
19:33 JoeJulian With bricks that full, you'll probably want to do a full rebalance, otherwise it'll get pretty inefficient with linkfiles for everything that ends up on the new brick.
19:33 rbennacer just add the package source to the repository and do an apt-get install glusterfs
19:34 puebele joined #gluster
19:34 rbennacer am i going to have some downtime when i do the rebalance?
19:35 JoeJulian noob2: What I do is upgrade 1 server. Wait until the heal info shows everything clean for all the volumes that had bricks on that server, check for heal-failed and split-brain, finding none, do the next server.
19:35 JoeJulian rbennacer: nope
19:35 noob2 JoeJulian: ok i was hoping it would go like that
19:35 JoeJulian noob2: Once the servers are upgraded, I upgrade the clients. This does require and umount/mount.
19:35 JoeJulian s/and/an/
19:35 glusterbot What JoeJulian meant to say was: noob2: Once the servers are upgraded, I upgrade the clients. This does require an umount/mount.
19:35 noob2 right
19:36 JoeJulian ... for now.
19:37 dbruhn rbennacer: I personally have been having issues getting through a rebalance my servers/bricks aren't as full as 99% but they are fairly full
19:37 noob2 that's ok.  i can remount the clients without too much notice
19:37 dbruhn with being slow.
19:38 rbennacer so rebalancing the bricks will improve performance without any risk during the rebalance operation?
19:38 JoeJulian Yep
19:38 rbennacer hahah
19:38 rbennacer ok sounds good
19:39 rbennacer well let's work on it then
19:40 xinkeT joined #gluster
19:41 red_solar joined #gluster
19:45 elyograg JoeJulian: am I the "one person over and over again" you just mentioned? :)  bug 863427.  remove-brick appears to use rebalance under the covers.
19:45 glusterbot Bug http://goo.gl/DUy9i is not accessible.
19:45 elyograg bug 862347
19:45 JoeJulian Not sure... The mailing list is what came to mind.
19:45 glusterbot Bug http://goo.gl/QjhdI medium, medium, ---, sgowda, ASSIGNED , Migration with "remove-brick start" fails if bricks are more than half full
19:46 wushudoin joined #gluster
19:47 JoeJulian No, that's remove-brick... That one doesn't surprise me quite as much (though I agree that it needs to work).
19:48 redsolar_office joined #gluster
19:49 elyograg From what I saw during remove-brick, its rebalance appears to require significantly more space on the target volume(s) than is ultimately being moved there.  If the expansion factor is large enough, it could explain problems for "regular" rebalances with very full bricks.
19:49 y4m4 joined #gluster
19:49 elyograg it might also be completely unrelated. (:
19:50 JoeJulian Perhaps. I found something similar with self-heal a while back (when I upgraded from 3.1 to 3.3) and that might be related too.
20:04 nissim joined #gluster
20:07 nissim Hi, I am running fedora 17 with gluster 3.3.1-4 over 5 nodes. tested both tcp & rdma and cant write more than 450MB/s. local write is 1.5GB/s. I don't really understand whats need to be done to get closer numbers to local write results. anyone??
20:10 nissim anyone getting more than 800MB/s over RDMA or tcp ???
20:10 rbennacer left #gluster
20:14 red_solar joined #gluster
20:16 JoeJulian ~pasteinfo | nissim
20:16 glusterbot nissim: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
20:17 JoeJulian And what speed is your infiniband?
20:17 nissim 40Gbps
20:18 nissim I have a mellanox connect X3 interface + mellanox switch
20:18 nissim will paste the vol info in a few minutes to http://fpaste.org
20:18 glusterbot Title: Fedora Pastebin (at fpaste.org)
20:22 nissim do you want me to paste "gluster vol info" ?
20:22 nissim or do you need other input?
20:25 nissim paste the output to http://fpaste.org
20:25 glusterbot Title: Fedora Pastebin (at fpaste.org)
20:25 nissim can you have a look?
20:25 nissim many thanks ...
20:28 JoeJulian "... then paste the link that's generated here."
20:28 nissim what link, I see no link
20:29 nissim JoeJulian??
20:30 dbruhn When you use the paste bin, it should provide you with a link after you past the stuff and choose continue or whatever
20:31 nissim http://fpaste.org/ZeuO/
20:31 glusterbot Title: Viewing Cant pass 400MB/s over RDMA/TCP by nissim (at fpaste.org)
20:32 JoeJulian Ok, thanks. How are you testing throughput?
20:32 nissim ran a simple command: time dd if=/dev/zero of=testfile bs=1M count=10240
20:33 nissim create a 10GB file
20:33 rosco__ joined #gluster
20:34 JoeJulian @wonka
20:34 JoeJulian ... damn... semiosis!!! ^
20:35 Nuxr0 joined #gluster
20:35 nissim do you need more info??
20:35 JoeJulian What if you run that on 100 clients (with different outfiles of course) and aggregate?
20:36 stopbit joined #gluster
20:36 JoeJulian When you run this in production, will you be creating a lot of 10Gig files using dd?
20:37 nissim In production, it will be used as openstack instances shared datastore
20:37 nissim so I guess I will have lots of virtual machines files which are not small
20:39 nissim I dont understand how come creating a local file is 3 times faster than using gluster while using the same datastore
20:39 JoeJulian So that'll be a lot of random writes from a myriad of vm hosts to a lot of files. dd doesn't really seem to model that well, imho.
20:39 nissim it has to be something that I am missing
20:40 nissim can't be that RDMA provides 400MB/s over 40Gbps HCA
20:40 JoeJulian Also, ib support is considered to be in "tech preview" which basically means it still needs some qa love.
20:40 dbruhn is tcp,rmda actually showing it using the infiniband?
20:41 nissim I am willing to pass RDMA if I get 1GB/s from IPOIB
20:41 nissim I read in multiple gluster blogs and site, users are getting such rates, so how come I am stuck in the 400MB/s sealling
20:42 JoeJulian From what I've heard (I don't have the luxury of the equipment) when you spread those requests among multiple clients, you're aggregate bandwidth seems to get maxed out.
20:42 JoeJulian That's the other thing. Some are reporting perfect results, some are reporting the ceiling you're noticing.
20:43 nissim I know that
20:43 JoeJulian It makes me wonder if there's some hardware difference between the two.
20:43 nissim what do you mean?
20:44 JoeJulian Well, something /has/ to be different between the successful installations and the ones that experience this ceiling. If it's not glusterfs, then what is it?
20:45 nissim I also read a redhat document: nfs over rdma that reports performance close the local machine
20:45 bfoster_ joined #gluster
20:45 nissim I am not sure
20:46 nissim is there any performance tunning document by gluster out there?
20:46 jdarcy joined #gluster
20:46 dbruhn the manual has a who tuning section
20:46 nissim Do I need to do some fine tunning to get better performance
20:47 dbruhn http://www.gluster.org/wp-content/up​loads/2012/05/Gluster_File_System-3.​3.0-Administration_Guide-en-US.pdf
20:47 glusterbot <http://goo.gl/bzF5B> (at www.gluster.org)
20:47 kkeithley1 joined #gluster
20:47 JoeJulian The people that don't have that ceiling have submitted configurations that have no changes to the options.
20:49 nissim what results would you expect to have from 5 nodes running over IPOIB (node: 64GB + 12cores + 2TB (SSD))?
20:49 nissim 400MB/s ...
20:50 nissim btw, read this PDF several time, saw the different gluster options but expected to get some recommendations or example setups from gluster guys
20:51 JoeJulian Since you're writing to one file, your client will be writing to one server (with your distribute volume configuration) you /should/ be able to get 40mbit.
20:51 JoeJulian er, gbit
20:51 nissim thats correct, or at least 1.5gbps like when writing locally
20:51 JoeJulian er, right.
20:52 * JoeJulian waits for the caffeine to kick in...
20:52 nissim so i ask my self , what else need to be done to get to the 1.5gbps which i know is possible
20:52 JoeJulian Are there any rdma performance tests like iperf?
20:53 nissim i am not an rdma expert, that why I am popping the question here
20:53 nissim I am new to infiniband
20:53 JoeJulian I wish I was... ;)
20:54 nissim but it looks like I will be in the next couple of days ;)
20:54 JoeJulian Wanna pick up a few cards and a switch and send them my way and I'll figure it out. :D
20:54 nissim wish I could
20:55 nissim its kind of odd, i am the only user want to use RDMA here
20:55 nissim I would expect it to be more common
20:57 nissim did you get your coffee already ;)
20:57 dbruhn There are people running RDMA out there, just hard for people to get their hands on it for testing purposes, kind of a high buy in.
20:57 nissim it wasnt that expensive ...
20:58 nissim same price as 10Gbe NIC and SWICH
20:59 dbruhn Yeah, and a lot of guys who do a lot of work for the community are running 1gb systems.
21:01 nissim JoeJulian, there is a tool name qperf which is used to test RDMA between two nodes
21:01 nissim this is what I am going to do now
21:02 nissim it also available in fedora yum repository
21:05 JoeJulian I just have 1Gbe stuff, but yeah, if I were deciding between implementing 10Gbe and IB, it would be IB.
21:07 JoeJulian I think they guys that run IB are too busy to hang out here and help (of course I am too, but I do anyway).
21:08 * m0zes runs IB. my backing store can't handle more than 400MB/s per server, though.
21:12 nissim I have a feeling IB link is no more than 4gbps
21:13 nissim I am running qperf (RDMA perf tool) to testing bidirectional stream and i got almost 8gbps for both connections
21:13 nissim i dont understand why
21:13 sjoeboo_ nissim: there are different ib standards/rates
21:13 nissim I need an IB guy, some that can help
21:13 sjoeboo_ we're hoping to do a FDR install this spring which will be 56GBs
21:13 nissim what do you mean?
21:14 JoeJulian Is there an #infiniband channel?
21:14 sjoeboo_ there are different data-rate standars
21:14 sjoeboo_ standards*
21:14 semiosis JoeJulian: dd-damn
21:14 sjoeboo_ single, double, quad, full, etc
21:14 dbruhn nissim are all of your cards QDR cards and your switch?
21:14 nissim how do I know what I am using?
21:14 m0zes SDR DDR QDR FDR FDR10
21:15 m0zes ibv_devinfo
21:15 sjoeboo_ lspic should say
21:15 sjoeboo_ of that
21:15 sjoeboo_ or*
21:15 dbruhn and are they gen1 or gen2 pci
21:15 m0zes or ibstat
21:16 m0zes I'd also check something like 'ibqueryerrors' for links that are dropping.
21:16 nissim ran ibstat, but I see no such info
21:16 nissim nothing about channel or any of the words you just mentioned
21:17 nissim I dont have ibv_devinfo on fedora infiniband-diag rpm
21:17 nissim I know its available under ubuntu since I tested ubuntu before
21:19 nissim ok, ran ibqueryerrors and I got some errors , let me check ...
21:21 nissim here is an example: PortXmitWait == 2120772
21:21 nissim PortXmitDiscards == 6321
21:22 nissim LinkErrorRecoveryCounter == 7
21:22 nissim when I run ibstat, I see all card are active
21:22 nissim how can I get more info??
21:30 m0zes to get actual link stats 'iblinkinfo'
21:30 nissim it looks like all my card are FDR-10
21:31 nissim switch is 40gbs
21:32 nissim 10min qperf read/write between 2 nodes provide no more than 3.89gbps
21:33 nissim which is around 490MB/s RDMA
21:33 nissim I don't understand why do I get such low rate
21:33 dbruhn Do you have support with mellenox?
21:33 dbruhn I am assuming you have a mellenox switch
21:34 JoeJulian As someone who volunteers to hang out here supporting GlusterFS, I can't help but be happy to hear that. I've always suspected that was the case when people were pointing at Gluster, and hoped I was right.
21:35 nissim yep a mellanox switch
21:36 nissim I will check the support on sunday
21:36 nissim sound like a good support question
21:37 nissim and I will be more than happy to share this info here contributing to this nice forum
21:38 dbruhn nissim, here is a decent link talking about infiniband throughput http://www.mellanox.com/content/pa​ges.php?pg=performance_infiniband
21:38 glusterbot <http://goo.gl/2I1Q8> (at www.mellanox.com)
21:39 dbruhn the gen1 pci slots will only be able to provide half of the rate of the qdr
21:42 nissim one sec
21:45 nissim still 400MB/s is very low even for gen 1
21:45 nissim I guess I have gen 2
21:45 nissim but I will check on sunday
21:50 quillo joined #gluster
21:50 ron-slc joined #gluster
21:50 GLHMarmot joined #gluster
21:50 m0zes I'd also check that the IB card itself is in a port that has enough pci-e lanes. pci-e does autonegotiation and will slow itself down if it has to.
21:51 noob2 left #gluster
21:54 m0zes I had a 10GbE nic that thought it was in a 1 lane pci-e slot (getting roughly 200MB/s through it). once I pulled it an reseated it I could get 800MB/s through it
21:58 nissim will check it too
22:06 nissim thank you: JoeJulian, dbrun,m0zes and all the others for helping ... will update on sunday
22:07 nissim since I am currently on the dark side of the globe, I say good night ;)
22:15 tryggvil joined #gluster
22:15 hattenator joined #gluster
22:23 gbrand_ joined #gluster
22:26 badone joined #gluster
22:40 rudimeyer joined #gluster
22:47 tryggvil joined #gluster
22:53 mathmonkey joined #gluster
23:31 badone joined #gluster
23:33 hchiramm_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary