Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-06-17

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 koguma joined #gluster
00:02 plarsen joined #gluster
00:04 vpshastry joined #gluster
00:11 gildub joined #gluster
00:12 elico JoeJulian: see you..
00:12 JoeJulian later
00:35 itisravi joined #gluster
00:40 calum_ joined #gluster
00:45 RicardoSSP joined #gluster
00:45 RicardoSSP joined #gluster
00:48 primechuck joined #gluster
00:54 haomaiwang joined #gluster
00:58 jmarley joined #gluster
00:58 jmarley joined #gluster
01:00 purpleidea JoeJulian: you've got 48tb bricks now? what happened?
01:28 JoeJulian purpleidea: I changed jobs. I now work for IO on this stuff: http://www.amax.com/enterprise/pdfs/AMAX%20IO%20OCP%20Reference%20Architecture.pdf
01:34 itisravi joined #gluster
01:45 edong23 joined #gluster
01:49 bala joined #gluster
01:51 harish joined #gluster
01:56 vpshastry joined #gluster
02:02 lpabon joined #gluster
02:21 vimal joined #gluster
02:25 jag3773 joined #gluster
02:29 sjm joined #gluster
02:31 vimal joined #gluster
02:32 vpshastry joined #gluster
02:36 davinder13 joined #gluster
02:38 coredump joined #gluster
02:41 harish joined #gluster
02:53 hagarth joined #gluster
02:53 itisravi joined #gluster
03:06 Matthaeus joined #gluster
03:12 saurabh joined #gluster
03:37 itisravi joined #gluster
03:43 eryc joined #gluster
03:43 eryc joined #gluster
03:46 RameshN joined #gluster
03:47 kanagaraj joined #gluster
03:47 shubhendu_ joined #gluster
03:56 spandit joined #gluster
04:01 kdhananjay joined #gluster
04:01 vpshastry joined #gluster
04:05 bala joined #gluster
04:06 rastar joined #gluster
04:09 kumar joined #gluster
04:09 RameshN joined #gluster
04:10 ndarshan joined #gluster
04:21 ppai joined #gluster
04:23 kshlm joined #gluster
04:37 prasanthp joined #gluster
04:38 nishanth joined #gluster
04:39 haomaiwang joined #gluster
04:45 deepakcs joined #gluster
04:48 lalatenduM joined #gluster
04:52 qdk joined #gluster
04:54 haomai___ joined #gluster
04:55 rjoseph joined #gluster
04:56 bharata-rao joined #gluster
05:01 psharma joined #gluster
05:03 davinder13 joined #gluster
05:05 vimal joined #gluster
05:08 nshaikh joined #gluster
05:09 dusmant joined #gluster
05:12 vpshastry joined #gluster
05:12 rjoseph joined #gluster
05:19 sjm joined #gluster
05:22 ramteid joined #gluster
05:24 aravindavk joined #gluster
05:24 hchiramm__ joined #gluster
05:34 hagarth joined #gluster
05:37 bala1 joined #gluster
05:43 aravindavk joined #gluster
05:45 wgao_ joined #gluster
05:49 systemonkey joined #gluster
05:49 dusmant joined #gluster
05:51 systemonkey joined #gluster
05:56 raghu joined #gluster
06:05 meghanam joined #gluster
06:06 mbukatov joined #gluster
06:11 meghanam_ joined #gluster
06:23 ktosiek joined #gluster
06:24 kdhananjay joined #gluster
06:29 gildub joined #gluster
06:39 hagarth joined #gluster
06:39 ekuric joined #gluster
06:41 aravindavk joined #gluster
06:51 ctria joined #gluster
06:52 eseyman joined #gluster
06:56 spiekey joined #gluster
07:02 kanagaraj joined #gluster
07:03 dusmant joined #gluster
07:05 RameshN joined #gluster
07:08 karnan joined #gluster
07:16 fsimonce` joined #gluster
07:18 andreask joined #gluster
07:21 keytab joined #gluster
07:22 fsimonce joined #gluster
07:29 stickyboy Oh wow... healing a failed brick is going very slowly.
07:29 stickyboy 1.5TB in 24 hours.
07:32 Magne joined #gluster
07:35 capri stickyboy, 1GB or 10GB network?
07:35 ctria joined #gluster
07:36 Magne Q: (new to glusterfs)  I need to have a replicated emailstorage.  Using postfix and dovecot today, on single server. Want to have at least two servers. I note that dovecot says there is a problem using glusterfs for this.  Anyone with experience/hints?
07:36 deepakcs joined #gluster
07:38 stickyboy capri: 10GbE
07:39 capri stickyboy, ok thats slow. 2 node gluster replicated setup?
07:40 liquidat joined #gluster
07:41 stickyboy capri: 2 nodes, yeah
07:42 rjoseph1 joined #gluster
07:50 SpComb joined #gluster
07:50 SpComb is there a way to just disable the NFS server funtionality if I'm not using it?
07:51 SpComb like, for all volumes by default :)
07:51 SpComb I know there's `volume set ... nfs.disable on'
07:52 hagarth SpComb: "volume set all nfs.disable on" should work
07:52 harish joined #gluster
07:54 SpComb volume set: failed: Not a valid option for all volumes
07:59 stickyboy capri: Yeah, I even turned off the iptables on the system.
08:00 stickyboy capri: Is there a way to manually rsync the data over or something?
08:00 prasanthp joined #gluster
08:00 stickyboy rsync with preserving the extended attributes... hmm.  I think I've seen that before.
08:20 liquidat joined #gluster
08:20 eseyman joined #gluster
08:20 hagarth joined #gluster
08:20 ktosiek joined #gluster
08:20 meghanam_ joined #gluster
08:20 wgao_ joined #gluster
08:20 ramteid joined #gluster
08:20 vimal joined #gluster
08:20 ndarshan joined #gluster
08:20 kumar joined #gluster
08:20 eryc joined #gluster
08:20 Pupeno joined #gluster
08:20 StarBeast joined #gluster
08:20 dblack joined #gluster
08:20 nage joined #gluster
08:20 jrcresawn joined #gluster
08:20 sjoeboo joined #gluster
08:20 jbrooks joined #gluster
08:20 morse joined #gluster
08:20 kkeithley joined #gluster
08:20 elico joined #gluster
08:20 hybrid512 joined #gluster
08:20 XpineX joined #gluster
08:20 ninkotech joined #gluster
08:20 TheDingy joined #gluster
08:20 johnmark joined #gluster
08:20 ry joined #gluster
08:20 marmalodak joined #gluster
08:20 GabrieleV joined #gluster
08:20 sadbox joined #gluster
08:20 silky joined #gluster
08:20 weykent joined #gluster
08:20 portante joined #gluster
08:20 partner joined #gluster
08:20 Licenser joined #gluster
08:20 d-fence joined #gluster
08:20 VeggieMeat joined #gluster
08:20 glusterbot joined #gluster
08:20 rturk|afk joined #gluster
08:20 cyber_si joined #gluster
08:20 coreping joined #gluster
08:20 m0zes joined #gluster
08:20 jiqiren joined #gluster
08:20 sauce joined #gluster
08:20 nixpanic_ joined #gluster
08:20 RobertLaptop joined #gluster
08:20 JoeJulian joined #gluster
08:20 ndevos joined #gluster
08:20 necrogami joined #gluster
08:20 anotheral joined #gluster
08:20 Dave2 joined #gluster
08:20 semiosis joined #gluster
08:20 brosner joined #gluster
08:20 SteveCooling joined #gluster
08:20 stigchristian joined #gluster
08:21 SpComb joined #gluster
08:21 JordanHackworth joined #gluster
08:21 yosafbridge joined #gluster
08:21 Andreas-IPO joined #gluster
08:21 ultrabizweb joined #gluster
08:21 osiekhan1 joined #gluster
08:21 lezo joined #gluster
08:21 samkottler joined #gluster
08:21 decimoe joined #gluster
08:21 Georgyo joined #gluster
08:21 masterzen joined #gluster
08:21 _jmp_ joined #gluster
08:21 mjrosenb joined #gluster
08:21 pasqd joined #gluster
08:21 foster joined #gluster
08:21 basso joined #gluster
08:21 johnmwilliams__ joined #gluster
08:21 eshy joined #gluster
08:22 yosafbridge joined #gluster
08:27 haomaiwang joined #gluster
08:28 eseyman joined #gluster
08:28 ninkotech joined #gluster
08:28 dblack joined #gluster
08:30 koguma joined #gluster
08:32 primusinterpares joined #gluster
08:36 eightyeight joined #gluster
08:36 SpComb if I'm running gluster on top of LVM, should I just use one big LV with all the different volume-bricks inside it, or should I use a separate LV per volume-brick?
08:37 ninkotech joined #gluster
08:38 ProT-0-TypE joined #gluster
08:44 koguma joined #gluster
08:44 dblack joined #gluster
08:44 eseyman joined #gluster
08:44 liquidat joined #gluster
08:44 hagarth joined #gluster
08:44 ktosiek joined #gluster
08:44 meghanam_ joined #gluster
08:44 wgao_ joined #gluster
08:44 ramteid joined #gluster
08:44 vimal joined #gluster
08:44 ndarshan joined #gluster
08:44 kumar joined #gluster
08:44 eryc joined #gluster
08:44 Pupeno joined #gluster
08:44 StarBeast joined #gluster
08:44 nage joined #gluster
08:44 jrcresawn joined #gluster
08:44 morse joined #gluster
08:44 kkeithley joined #gluster
08:44 elico joined #gluster
08:44 hybrid512 joined #gluster
08:44 XpineX joined #gluster
08:44 TheDingy joined #gluster
08:44 johnmark joined #gluster
08:44 ry joined #gluster
08:44 marmalodak joined #gluster
08:44 GabrieleV joined #gluster
08:44 sadbox joined #gluster
08:44 glusterbot joined #gluster
08:44 rturk|afk joined #gluster
08:44 cyber_si joined #gluster
08:44 coreping joined #gluster
08:44 m0zes joined #gluster
08:44 jiqiren joined #gluster
08:44 sauce joined #gluster
08:44 nixpanic_ joined #gluster
08:44 RobertLaptop joined #gluster
08:44 JoeJulian joined #gluster
08:44 ndevos joined #gluster
08:44 necrogami joined #gluster
08:44 anotheral joined #gluster
08:44 Dave2 joined #gluster
08:44 semiosis joined #gluster
08:44 brosner joined #gluster
08:44 SteveCooling joined #gluster
08:44 stigchristian joined #gluster
08:44 VeggieMeat joined #gluster
08:44 d-fence joined #gluster
08:44 Licenser joined #gluster
08:44 partner joined #gluster
08:44 portante joined #gluster
08:44 weykent joined #gluster
08:44 silky joined #gluster
08:49 [o__o] joined #gluster
08:50 bharata-rao joined #gluster
08:58 primusinterpares joined #gluster
09:01 kaushal_ joined #gluster
09:02 Intensity joined #gluster
09:03 calum_ joined #gluster
09:05 rjoseph joined #gluster
09:07 lalatenduM hagarth, is gluster 3.5 is backward compatible with gluster 3.4?
09:08 vpshastry joined #gluster
09:13 SpComb heh, `git clone .../linux.git` into a glusterfs is.... slow :)
09:14 SpComb 3x replica, should try a 3x distributed
09:14 stickyboy So does anyone have experience with rsyncing bricks in a recovery situation, then using self-heal later?
09:14 stickyboy Self heal on 22TB is going at about 1TB per day. :)
09:19 eseyman joined #gluster
09:30 ctria joined #gluster
09:31 SpComb yeouch, git clone into a 3x replica is 12m27.295s, into a 3x distributed 7m37.710s... local fs is 0m9.262s
09:31 SpComb must be a pretty pathological case for gluster... I wonder if git is able to do IO in parallel, or just one file at a time
09:32 SpComb ok, given, that local time is with hardlinkin..
09:36 jmarley joined #gluster
09:37 mrspastic joined #gluster
09:37 mrspastic hello
09:37 glusterbot mrspastic: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:38 T0aD joined #gluster
09:39 mrspastic i am using NFS on top of GFS (not using native NFS support in GFS). Are there known issues by doing so?
09:39 mrspastic [mounting glusterfs on server locall and exporting mount via NFS]
09:41 nshaikh joined #gluster
09:43 prasanthp joined #gluster
09:44 stickyboy mrspastic: Some people talk about "protocol stacking" and thought it might be a nice tactic to leverage Gluster's replicated backend with NFS's client-side caching.
09:44 stickyboy I have no experience, but there was talk on the mailing list last year.
09:44 stickyboy Maybe look for it.
09:47 mrspastic will do, thanks for the ideas
09:52 hagarth lalatenduM: yes, 3.5 is protocol compatible with 3.4 and 3.3
10:01 lalatenduM hagarth, cool thanks
10:04 nbalachandran joined #gluster
10:05 rjoseph joined #gluster
10:09 kaushal_ joined #gluster
10:13 vpshastry joined #gluster
10:14 [ilin] joined #gluster
10:14 SpComb oh wow, turns out one of my nodes has a faulty network port and was stuck on 100Mbps :)
10:15 SpComb probably explains some of the poor perf
10:15 mrspastic oh weird, i have been having bad performance, it makes me think i should double check
10:17 SpComb I noticed that some transfers were just capping at 11MB/s while others were well over
10:17 [ilin] hi, we have two node glusterfs 3.4.1 and we are thinking about upgrading to 3.5. can we just upgrade the packages on the nodes one by one or we need to stop glusterfs before?
10:20 Paul-C joined #gluster
10:21 koguma joined #gluster
10:22 Paul-Cuz left #gluster
10:27 kkeithley1 joined #gluster
10:28 nthomas joined #gluster
10:32 Pupeno Is anybody mounting volumes from localhost at boot time?
10:34 davinder14 joined #gluster
10:37 ctria joined #gluster
10:37 calum_ joined #gluster
10:40 stickyboy Ok, I'm going to forgoe the self-heal daemon to restore this brick, because it's too much data.
10:41 stickyboy Trying on a subset of my data first...
10:45 japuzzo joined #gluster
10:47 glusterbot New news from newglusterbugs: [Bug 1110262] suid,sgid,sticky bit on directories not preserved when doing add-brick <https://bugzilla.redhat.com/show_bug.cgi?id=1110262>
10:47 koguma I'm running Gluster 3.4.4.  Is it a problem that the nfs log has: Using Program GlusterFS 3.3, Num (1298437), Version (330)?
10:47 stickyboy koguma: That's more or less a protocol version.
10:47 stickyboy It's ok.
10:47 stickyboy There's another place where it prints the program version, but I can't remember.
10:53 koguma Ok cool. Thanks. :)
11:03 kanagaraj joined #gluster
11:09 koguma Weird problem.  2 replicated servers.  I can mount nfs only on one of them.  The other gives a 'failed, reason given by server: Permission denied'
11:16 shubhendu_ joined #gluster
11:16 ndarshan joined #gluster
11:16 dusmant joined #gluster
11:17 glusterbot New news from newglusterbugs: [Bug 1101647] gluster volume heal volname statistics heal-count not giving desired output. <https://bugzilla.redhat.com/show_bug.cgi?id=1101647>
11:20 edward1 joined #gluster
11:33 bene2 joined #gluster
11:37 RicardoSSP joined #gluster
11:38 hybrid512 joined #gluster
11:38 diegows joined #gluster
11:38 hybrid512 joined #gluster
11:42 tryggvil joined #gluster
11:57 primechuck joined #gluster
12:01 tryggvil joined #gluster
12:06 jvandewege joined #gluster
12:07 jag3773 joined #gluster
12:09 ekuric joined #gluster
12:10 partner any rumours around about 3.4.4-2 ? i'm in the middle of maintenance window for upgrading but we decided not to as the two pretty much only ever used commands are add-brick and rebalance :)
12:11 partner could have done with 3.4.3 and apply patch myself as i will anyways need to build packages for the few squeeze boxes that we cannot upgrade
12:11 SpComb is there any particular reason why nfs performs better for many small files vs glusterfs?
12:11 Nightshader joined #gluster
12:11 SpComb does the kernel nfs client just have more agressive metadata and write caches, and the glusterfs implementation is more conservative in the default settings?
12:12 SpComb or is there some kind of roundtrip-serialization bottlenecking going on the in the glusterfs client?
12:13 Nightshader Hi all, i'am new to Gluster and just setup a two-node replica 2 test. As test I changed one file directly on the brick filesystem but Gluster didnt notice - any advice?
12:13 Nightshader The changed file became empty..
12:15 SpComb Nightshader: why would you modify the brick fs directly? Oo
12:15 T0aD joined #gluster
12:16 jtux joined #gluster
12:16 partner SpComb: have you read this or would that shed some light to your question: http://joejulian.name/blog/nfs-mount-for-glusterfs-gives-better-read-performance-for-small-files/
12:16 glusterbot Title: NFS mount for GlusterFS gives better read performance for small files? (at joejulian.name)
12:16 Nightshader SpComb: in order to test the filesystem's resilience
12:17 glusterbot New news from newglusterbugs: [Bug 1107649] glusterd fails to spawn brick , nfs and self-heald processes <https://bugzilla.redhat.com/show_bug.cgi?id=1107649>
12:17 SpComb Nightshader: against brick corruption?
12:18 SpComb partner: yeah, metadata caches are probably a big factor... but I also read something about fuse not write-combining small writes etc
12:18 Nightshader (my background has been ZFS for the last few years, so we are testing Gluster with that in mind)
12:19 partner i hope to get rid off the fuse soon
12:20 SpComb but avoiding multiple serial roundtrips for metadata operations is a really big deal for random write perf
12:21 SpComb I'd assume NFS has really been optimized for that... but then I dunno what the glusterfsd actually does with the NFS operations :)
12:22 SpComb ..at the cost of some coherency nastyness that applications then have to deal with via locking etc
12:22 [o__o] joined #gluster
12:23 SpComb but on the other hand who would use a mail server that wasn't tested to work on NFS :P
12:26 SpComb is the glusterfs server serving nfs able to gain some kind of perf advantage over a glusterfs client doing the same operation on the mountpoint?
12:27 jiku joined #gluster
12:39 partner i don't use nfs but don't you save one extra step there? like if the nfs would be non-glusterfs you would need to mount the volume to that host and then serve that over the nfs for the clients?
12:39 marcoceppi joined #gluster
12:39 [o__o] joined #gluster
12:39 partner now the internal one is aware of the translators involved with the volumes/bricks
12:41 SpComb partner: I mean client running glusterfs and talking to glusterfsd directly vs client with nfs kernel client talking to one glusterfsd that then talks to other glusterfsd's
12:43 haomaiwa_ joined #gluster
12:44 partner ah ok. i have no answer to that so i'll just shut up :)
12:44 SpComb yeah, you'd need someone who's actually worked on the glusterfsd nfs-server implementation to know :P
12:48 SpComb glusterfs seems to perform really well for bulk writes (dd if=/dev/zero of=... bs=1M) now, but yeah, a `git clone` is still pretty catastrophical :/
12:49 haomaiwa_ joined #gluster
12:52 haomai___ joined #gluster
12:53 sroy_ joined #gluster
12:55 coredump joined #gluster
12:58 chirino joined #gluster
13:05 primechuck joined #gluster
13:06 [o__o] joined #gluster
13:07 julim joined #gluster
13:07 sjm joined #gluster
13:11 jmarley joined #gluster
13:11 jmarley joined #gluster
13:13 simulx joined #gluster
13:13 JoeJulian partner: yes, I tested the patch for rebalance. 3.4.4-2 should be coming soon.
13:14 deeville joined #gluster
13:18 JoeJulian Nightshader: erasure coding is a feature in development. I don't see it in the list for 3.6. Maybe it will make 3.7.
13:19 hagarth joined #gluster
13:20 ndk joined #gluster
13:23 dusmant joined #gluster
13:27 koguma joined #gluster
13:28 koguma Weird problem.  2 replicated servers.  I can mount nfs only on one of them.  The other gives a 'failed, reason given by server: Permission denied'
13:28 deeville koguma, firewall?
13:29 koguma deeville:  Hmmm.. the firewall is only setup on the external interfaces..  but I can easily test to rule that one out..
13:30 deeville koguma, yah turn off iptables on that server to check
13:30 koguma deeville: nope, not the firewall
13:30 bene2 joined #gluster
13:30 koguma I've tested mounting to localhost, the localserver, and the remote server.  mounting the remote server works, but same permission error on the localhost and server.
13:30 partner JoeJulian: yeah, noticed the comment in bz, thanks
13:31 deeville so this is just NFS
13:31 koguma btw, someone was mentioning something about nfs in the kernel before.  I think gluster 3.5 uses user-mode nfs, not kernel mode.
13:32 koguma the glsuter.org site mentions nfs-ganesh for 3.4.. which is userspace nfs..kinda like fuse.
13:33 koguma deeville: volume status shows nfs is up on localhost...
13:34 mtanner_ joined #gluster
13:40 shyam joined #gluster
13:40 JoeJulian koguma: make sure kernel nfs isn't running on that machine.
13:41 koguma JoeJulian: oh man, I think that's the prob...
13:41 sroy_ joined #gluster
13:42 koguma JoeJulian: I'm pretty sure that was the problem.. I found it running.. shutting it down hosed the instance...
13:42 koguma oh it's back...
13:44 koguma JoeJulian: That was it.
13:45 koguma Thanks man.  It's usually the simple things that kill you.  I forgot I had it running on the second server...
13:45 plarsen joined #gluster
13:45 koguma Weird that I can't mount it via localhost, but I can live with that.
13:46 koguma Btw, we need glusterfs to get killed when glusterd is brought down.. that's a bug...
13:46 JoeJulian koguma: not a bug
13:47 JoeJulian koguma: why do you want to lose access to your data when you restart the management daemon?
13:47 deeville koguma, sorry got pulled into a meeting, glad you sorted it out
13:47 JoeJulian that would be like all your VMs going down when you restart libvirt.
13:48 koguma JoeJulian: no prob.  NFS doesn't failover, so wouldn't you lose access anyway?  I mean if both glusterd and glusterfsd are down, glusterfs stays up.
13:49 bala1 joined #gluster
13:50 koguma JoeJulian: Or does it connect to another server?  I haven't tested it, but from the docs I've seen, only mounting glusterfs does the connection get maintained (by failing over).
13:50 JoeJulian not failing over... but close.
13:50 koguma JoeJulian: I can test right now... :)
13:50 koguma JoeJulian: the nfs part
13:51 JoeJulian even fuse never fails over, simply maintains connection to remaining replica.
13:51 JoeJulian You can use a floating ip with nfs. Some do.
13:51 koguma JoeJulian: oooh, you're right.  :P
13:52 koguma I killed both daemons and it did connect to the other replica.
13:52 JoeJulian Point being, we've had bugs that only affect the management daemon before. If I tell someone to restart glusterd on all their servers, I don't want to spend the rest of the day helping them debug split-brain.
13:52 koguma JoeJulian:  This is exciting. :)  I feel vindicated spending the last few days fighting in getting gluster properly working.  Looks like it's good to go...
13:53 deeville Is Gluster still known to be slow on writes for many small files? I read on older versions, all the bricks in a replicated volume are checked if the file exists before writing.
13:54 mrspastic joined #gluster
13:54 JoeJulian I've never known gluster to be slow for small files. I've known network latency and tcp overhead to add latency to consistency queries. If you're not worried about consistency and/or have no need for fault tolerance, you can overcome that.
13:54 simulx gluster unpriv mount question:  i get no "errors" in the log (http://pastebin.com/ipN5b0Kb) all the auth seems to work..., connections are ok, etc.... but it still won't mount as non-root.   if i run the same sommand as root, it works fine
13:54 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
13:55 simulx sorry: http://ur1.ca/hjs7p
13:55 JoeJulian simulx: oh where did that go... I wrote about how to do that on my blog some time ago...
13:55 glusterbot Title: #110456 Fedora Project Pastebin (at ur1.ca)
13:56 mrspastic if nfs.enable-ino32 is set to on, will all inodes have 32bit values? or will this only occur when you mount to the native NFS daemon
13:56 simulx i read a zillion blogs
13:56 JoeJulian http://joejulian.name/blog/mounting-a-glusterfs-volume-as-an-unprivileged-user/
13:56 glusterbot Title: Mounting a GlusterFS volume as an unprivileged user (at joejulian.name)
13:56 mrspastic Ideally, i would like to have 32bit values indepentent of the methodoly for mounting
13:56 simulx key things were: putting the user in the "fuse" group, making sure the volume authentication was correct, etc.
13:56 simulx Yep... that's the one i read
13:59 mrspasti1 joined #gluster
13:59 simulx the user_allow_other is et, etc.
13:59 mrspasti1 exit
13:59 mrspastic lol oops
13:59 simulx you should add the "put the user in the fuse group" thing too... i found that was essential
13:59 simulx usermod -aG fuse <NAME>
14:00 mrspastic anyone? 32 inodes? i would like to force usage of 32 bit inodes regardless of the mounting methodology
14:00 simulx that got rid of a bunch of errors
14:00 JoeJulian simulx: failed to exec fusemount, no such file or directory?
14:00 simulx no...i used to get that
14:00 simulx fixed though
14:00 simulx you can click on the fpaste.org link to see the log
14:00 simulx http://ur1.ca/hjs7p
14:00 glusterbot Title: #110456 Fedora Project Pastebin (at ur1.ca)
14:00 simulx notice... there's no errors
14:01 JoeJulian I did. That's where I saw the error. :P
14:01 lpabon joined #gluster
14:01 simulx oh
14:01 simulx wait
14:01 simulx argh
14:01 julim joined #gluster
14:01 simulx yeah it's still there
14:02 JoeJulian mrspastic: not sure. I'd be looking through "gluster volume set help" to try to answer that if I weren't in the middle of other things at the moment.
14:03 simulx well i have no idea why that's showing up... i know "fusermount" is there... in /bin/fusermount
14:04 JoeJulian maybe it's spurious... Wouldn't be the first non-error to be logged...
14:05 simulx when i run as root... that doesn't show up in the log
14:05 daMaestro joined #gluster
14:05 simulx and the mount works
14:05 simulx so probably it's some issue
14:05 JoeJulian Can you fpaste an strace?
14:08 sjm joined #gluster
14:10 bennyturns joined #gluster
14:11 DV__ joined #gluster
14:11 simulx strace log mounting: http://fpaste.org/110464/14030142/
14:11 glusterbot Title: #110464 Fedora Project Pastebin (at fpaste.org)
14:14 simulx if i run the same strace as root.. i get the same set of ENOENT returns
14:15 bala1 joined #gluster
14:16 JoeJulian Gah, that's useless. That just traces the bash script. I should have guessed.d
14:18 simulx indeed
14:18 simulx the only difference is the exit status of the child
14:18 simulx i guess i can use gdb
14:19 simulx or strace -f
14:19 simulx i'll try that
14:20 simulx ah!  that's it
14:20 simulx ls /usr/bin/fusermount-glusterfs
14:20 simulx missing
14:21 JoeJulian provided by glusterfs-fuse
14:21 JoeJulian which distro?
14:23 simulx ubuntu
14:23 JoeJulian semiosis: ^
14:24 mrspastic will look more into ir
14:24 simulx yes
14:24 simulx semiosis:
14:24 JoeJulian @semiosis--
14:24 glusterbot JoeJulian: semiosis's karma is now 1
14:24 JoeJulian hehe
14:25 simulx so that's just a file not in the repo
14:26 JoeJulian Looking that way.
14:28 haomaiwang joined #gluster
14:28 JoeJulian I'm rearranging my home-office so my desktop machine is currently unplugged or I'd boot up my ubuntu vm and look at the packaging process to see what I could figure out.
14:28 sjoeboo joined #gluster
14:30 wushudoin joined #gluster
14:37 bene2 joined #gluster
14:38 davinder14 joined #gluster
14:38 simulx ok i recompile gluster from scratch... should i just copy over the missing executables?  or really run make-install ?
14:39 simulx i'll try the simpler route first
14:40 tdasilva joined #gluster
14:41 lmickh joined #gluster
14:43 shubhendu joined #gluster
14:43 haomaiw__ joined #gluster
14:48 simulx that works
14:48 simulx so for ubuntu:  1. recompile:  2. copy in the fusermount helper 3. set sticky bit ... 4.  everything works
14:48 purpleidea JoeJulian: sweet :) that means you can host all of our movie collections now ; ;)
14:48 simulx sigh
14:49 simulx essentially... don't use the ppa if you want to use fusermount
14:50 vpshastry joined #gluster
14:51 thiago joined #gluster
14:52 JoeJulian purpleidea: I was already doing that... ;)
14:53 purpleidea JoeJulian: hehe, well 48TB bricks sounds pretty big!!
14:54 JoeJulian Aren't you in my Google+ circles? I posted pictures...
14:54 purpleidea JoeJulian: I don't have the google+ :(
15:00 kanagaraj joined #gluster
15:01 sjm joined #gluster
15:04 tdasilva left #gluster
15:04 tdasilva_ joined #gluster
15:06 spwk joined #gluster
15:09 purpleidea JoeJulian: I think i found a bug in the article... it talks about "
15:10 purpleidea 18 racks" but the diagram shows 1 - 17, not 0-17
15:15 kshlm joined #gluster
15:20 XpineX joined #gluster
15:26 jobewan joined #gluster
15:27 XpineX_ joined #gluster
15:32 davinder14 joined #gluster
15:37 partner the missing one is the movie collection..
15:37 sroy__ joined #gluster
15:39 purpleidea damn!
15:41 Slashman joined #gluster
15:48 _dist joined #gluster
15:48 sroy_ joined #gluster
15:50 zerick joined #gluster
15:55 jag3773 joined #gluster
16:02 Slashman joined #gluster
16:02 davinder14 joined #gluster
16:13 partner but interesting indeed, i'd love to hear more about the setup(s), its something that is pretty difficult to do on your own. i mean to setup test environments of that size to know how it works on such scale
16:14 bala joined #gluster
16:15 partner but now its more of a testing in production.. i'm not at all sure should i rather put 40 TB brick or as i've done now put LVM in and slice it into initially 3x 10TB logical volumes which make the bricks for me
16:17 partner i guess there is no right or wrong on that area, i just felt its somewhat easier to manage with "reasonable-sized" bricks rather than one that fills the whole box. of course i'm left with 3x more bricks and that might show up as additional latency and what not when searching for files from all the bricks..
16:18 partner and i can always remove-brick and then expand the remaining ones if this turns out to be Wrong(tm) way of doing things
16:19 semiosis a parallel find tool would be neat.  afaict the limitation is that find just goes through directories sequentially, so it doesnt matter how many bricks you have, you'll only be searching them one at a time with find anyway
16:20 semiosis if i'm in a hurry to find a file i just go run find on the bricks directly instead of through a client mount (but that's very rare)
16:21 semiosis see ,,(targeted self heal) for an example of parallel find
16:21 glusterbot https://web.archive.org/web/20130314122636/http://community.gluster.org/a/howto-targeted-self-heal-repairing-less-than-the-whole-volume/
16:21 semiosis kinda hackey though
16:22 semiosis and of course since v3.3 the self heal daemon removes any need for targeted self heal
16:23 partner i would be actually interested in seeing the big picture of fragmentation, i'm sure there's plenty but how much is plenty.. no idea
16:24 partner nor have i any idea how much it might slow things down
16:27 partner 423M inodes in use according to client on my volume
16:28 _polto_ joined #gluster
16:28 _polto_ hello
16:28 glusterbot _polto_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:30 Matthaeus joined #gluster
16:34 _polto_ Can GlusterFS with Geo-Replication serve as two SAMBA servers in two different locations providing the same datas ?
16:35 _dist _polto_: I beleive so but much of the data _could_ come from geo-replication servers "master"
16:36 hagarth joined #gluster
16:37 _polto_ _dist, What do you mean ? I want people in the two cities to use their local SAMBA file server. But I want the content to be the same
16:37 _polto_ Initially I was thinking about DRDB, but the low bandwidth (2Mbps) do not allow this.
16:37 Matthaeus _polto_: Have you heard of the CAP triangle?
16:38 _dist _palto_: the local server that is a geo-replication receiver will only have so much "recent" stuff. If someone at site A writes a new file or modifies and existing one, site B may be forced to read from A until B's files are "synced"
16:38 _polto_ Matthaeus, no
16:39 Matthaeus Consistency, Availability, Performance.  Pick any two.
16:39 _polto_ Matthaeus, :)
16:40 Matthaeus If you have a 2 meg link, I strongly suspect that using gluster across it will incur a much higher performance penalty than just using one samba server.
16:40 _polto_ _dist, it's OK if this process is transparent.
16:41 _dist _polto_: it would be, but Matthaeus is right, if by 2 meg you mean 2Mbit/s I suspect your latency will be pretty high. I haven't done atest like that, or crunched the numbers
16:42 _polto_ ok, I'll test it. :)
16:43 _polto_ few months ago, before the geo-replication I tryed normal replication in sync mode - it was teribly slow. I'll try now with geo-replication.
16:43 Matthaeus _polto_: You may find, after testing, that consistency is the part of CAP that you're willing to give up.  If that's the case, check out something like owncloud.
16:44 semiosis _polto_: with geo-rep you'll have to designate one location as read-only, the other as the writer
16:44 semiosis they'll have the same data, but writes can only go in to one location
16:44 Matthaeus That's going to -murder- performance.
16:44 _polto_ buerk... ownlcoud... it work fine for my own, but with 15 people you need a dualxeon server... ;)
16:44 Matthaeus Writes from the read-only office will have to traverse the 2meg link twice.
16:45 _polto_ semiosis, aiii !! I need RW on every locations.
16:45 semiosis gluster is not a good fit for that use case
16:45 semiosis maybe one day we'll have multi-master geo replication, but not today
16:46 semiosis you could set up two volumes, geo-replicating in opposite directions
16:46 semiosis so each location has it's own RW volume and a RO copy of the other location's volume
16:47 _polto_ too complicated. and not transparent for users.
16:47 semiosis however you slice this, having both locations collaborate on the same files will be complicated
16:47 zaitcev joined #gluster
16:47 semiosis that's teh nature of the prbolem
16:47 Matthaeus _polto_: How often will two people need near-simultaneous access to the same file?
16:48 partner alright, another 24TB online, go away disk alert, must love (a bit) gluster ;)
16:49 _polto_ Matthaeus, same file - probably not so often. Twice per day..
16:49 Mo__ joined #gluster
16:50 partner i wish i could rebalance but given the fact the servers run out of memory there's not much i can do about it, new stuff will end up to newest bricks
16:52 Matthaeus _polto_:  You may wish to very seriously consider some kind of eventual consistency model, then.  Dropbox, box.com, owncloud, even crashplan would be a better fit for you than a filesystem that will sacrifice performance to keep availability and consistency up.
16:52 JoeJulian partner: Which version do they run out of memory on?
16:53 partner JoeJulian: 3.3.2, my servers only have 8 gigs of memory as i recall hearing extra memory won't do much good but turns out even RH suggests at least 32 GB for archival purposes
16:53 _polto_ Matthaeus, thanks. we are considering seafile.
16:54 JoeJulian partner: yeah, memory leak in rebalance. 3.4.4-2 is the first I might consider recommending it.
16:54 JoeJulian Speaking of which...
16:55 partner JoeJulian: good to hear its gone. my plan was to upgrade to 3.4.4 today 2PM but given the nasty bug i will wait
16:55 JoeJulian hagarth: I verified that patch. Can we get that merged into release-3.4 please?
16:55 partner i can run it for 3-5 days but if i forget it bad things will start to happen, brick procs dying and what not
16:56 partner and it seems to start from beginning so cannot even stop in the middle to free up mem and continue
16:57 hagarth JoeJulian: sure, kkeithley_ - can you please pick up http://review.gluster.org/8029 for release-3.4 ?
16:57 glusterbot Title: Gerrit Code Review (at review.gluster.org)
16:58 JoeJulian semiosis: Did you see that bit about the ubuntu package not including fusermount-glusterfs
16:59 semiosis what is fusermount-glusterfs
16:59 semiosis ?
16:59 JoeJulian semiosis: userspace fuse mount. It's in the rpm but not in the deb
16:59 JoeJulian for unprivileged users to mount the volume.
17:00 semiosis thx for pointing it out, i missed it
17:00 semiosis got a link?
17:02 JoeJulian No link, scroll back to 14:24utc
17:02 JoeJulian Also, do you want the build stuff I did that includes the rebalance patch, or do you want to just build your own?
17:02 semiosis thx for offering but i'll do it, it's no trouble at all
17:03 semiosis i've had to roll a patch for every 3.5 release due to gf-error-codes.h, i'm quite used to it by now :)
17:11 plarsen joined #gluster
17:20 semiosis Me: We're only getting 1 Mbit downstream. Comcast: Well you should be getting 50. Me: THATS WHY I'M CALLING YOU
17:21 Matthaeus semiosis: tell me you're on business class.
17:21 semiosis yes
17:22 Matthaeus Then you have half a prayer of getting it resolved.
17:22 semiosis haha
17:26 rotbeard joined #gluster
17:35 kkeithley_ hagarth, JoeJulian: re: http://review.gluster.org/8029 for release-3.4. Sure
17:35 glusterbot Title: Gerrit Code Review (at review.gluster.org)
17:42 spwk Hello. I have this replicated gluster volume which has been running fine for a while, but the brick log is growing large with mostly this line "0-server: inode for the gfid (...) is not found. anonymous fd creation failed". Should I worry and is there anything that I can do?
17:45 kkeithley_ JoeJulian, hagarth: http://review.gluster.org/8029 needs a +2 code review
17:45 glusterbot Title: Gerrit Code Review (at review.gluster.org)
17:46 hagarth kkeithley_: you can probably treat pranith's +1 as +2, I suppose ;)
17:47 kkeithley_ mkay. I'll review it.
17:52 primechu_ joined #gluster
17:54 glusterbot New news from resolvedglusterbugs: [Bug 961615] Hypervisor mount crashed after rebalance <https://bugzilla.redhat.com/show_bug.cgi?id=961615>
18:02 jruggiero joined #gluster
18:06 ramteid joined #gluster
18:06 qdk joined #gluster
18:09 jruggiero left #gluster
18:15 cfeller joined #gluster
18:30 [o__o] joined #gluster
18:40 jcsp_ joined #gluster
18:44 barnim joined #gluster
18:46 _polto_ left #gluster
18:47 mjsmith2 joined #gluster
18:52 primechuck joined #gluster
18:54 dtrainor joined #gluster
18:54 dtrainor Hi.
18:54 glusterbot dtrainor: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
18:54 cfeller On my main four node Gluster setup (version 3.4.4 on RHEL 6), I just now realized that I somehow failed to adjust the inode size to 512 bytes when I formatted the bricks. (It is currently the default of 256 bytes.)  How much of a performance hit am I taking because of this?  I know it _was_ an issue, and if it still is, I can take a weekend and backup my entire setup, reformat the bricks and then copy everything back.  However, some reading last n
18:56 dtrainor Hah.  Okay.  So with the help of JoeJulian the other day, I was able to replace a failed brick on a single-server setup.  It's starting to resync - very, very slowly.  Over two days, approximately 11G of 1.2T have been restored to that brick.  It's a 2x2=4 distributed-replicate volume.  If it's doing its thing, it's doing its thing.... but it seems like this is really, really slow.
19:02 partner cfeller: your message was cut at ..."last ni"
19:03 cfeller oops... ok.  Here is the rest:
19:03 cfeller However, some reading last night seemed to indicate that recent improvements to XFS make this less of an issue (but possibly still is somewhat?).
19:03 cfeller That said, how much of a performance hit am I taking and would it be worthwhile to reformat (as I don't know of any way to fix the inode size without reformatting).
19:04 tdasilva_ joined #gluster
19:04 kkeithley_ cfeller: me personally, I wouldn't go to the effort of backup-reformat-restore. There may be a small performance hit. I don't know what it really amounts to.
19:05 kkeithley_ If you're really finding that you need every last iota of performance, then do it.
19:06 XpineX_ joined #gluster
19:07 jag3773 Hello, it's been a while since I've chimed in here, but I've got an issue on one of our glusters that I need a hand solving
19:07 jag3773 I've put some relevant details here: https://gist.github.com/jag3773/59da457e629c647a1f26
19:07 glusterbot Title: Gluster Volume Checksum Mismatch (at gist.github.com)
19:07 jag3773 basically, i have a 4 node cluster in a distributed replicate... one of the nodes has dropped out due to a cksum mismatch
19:08 jag3773 i just need to know the best course of action to fix that node that dropped out
19:08 jag3773 I think i'll need to remove it from the volume and re-add it, but I'm not quite sure of the specifics there
19:08 jag3773 any thoughts?
19:13 cfeller kkeithley_: I was kind of poking around, as my main (production) Gluster setup doesn't seem (without any formal benchmarking other than "time" on certain transfers) that much faster than my "test" Gluster setup.
19:13 cfeller My main Gluster setup has faster disks and and hardware RAID, so in theory it should bet a bit faster. That was when I realized that my "test" Gluster cluster has the correct inode size but my main cluster doesn't.
19:13 cfeller But I also have a boatload of more data on my main Gluster setup too, so that may be a contributing factor?
19:15 dtrainor re: brick sync, I just *happened* to be doing a find . on that volume, and that appears to be enough to make Gluster start sync'ing
19:18 JoeJulian cfeller: I've read somewhere, and I forget where, that the inode size difference is negligible to the point of being irrelevant.
19:20 cfeller JoeJulian: some of the more recent threads I was reading last night seemed to allude to that, but I figured I'd ask in here to be sure.
19:21 JoeJulian dtrainor: did you "gluster volume heal $vol" or maybe even "gluster volume heal $vol full"
19:21 dtrainor i did not, no.  i'll look in to how those work, thank you for the suggestion
19:21 cfeller Thanks for your feedback JoeJulian and kkeithley_ !
19:21 JoeJulian jag3773: I would just rsync the /var/lib/glusterd/vols from a good server
19:22 jag3773 ok, thanks JoeJulian, i thought it was something like that, but i wasn't quite sure what files to sync
19:22 jag3773 i'll give that a try here in a bit
19:22 partner jag3773: filled up your /var ?
19:23 jag3773 no
19:25 jag3773 i'm not sure why the node dropped out in the first place, there are logs noting that it had a communication problem for a few days, that may have been it
19:27 jrcresawn_ joined #gluster
19:36 Sunghost joined #gluster
19:42 Sunghost hello i have a 2 node glusterfs in version 3.5 on debian, i mount via nativ client vol1 on node2 but cant write any new files - input output error in mc - any idea?
19:45 Sunghost glusterlog for vol1 says no space left, but df -h says 5tb ?! i added the new brick yesterday and run rebalance after that
19:46 Matthaeus joined #gluster
19:46 jag3773 for posterity, I've added the solution to my gist at: https://gist.github.com/jag3773/59da457e629c647a1f26  -- thanks again JoeJulian
19:46 glusterbot Title: Gluster Volume Checksum Mismatch (at gist.github.com)
19:47 jag3773 partner, see that gist link for the fix too if you are interested
19:52 [o__o] joined #gluster
19:54 Sunghost Idea?
19:59 JoeJulian df -i  maybe?
19:59 Sunghost i now start a second rebalance and see in etc-gluster.. log -> i now start a second rebalance and see in etc-gluster.. log -> [glusterd-utils.c:8053:glusterd_volume_rebalance_use_rsp_dict] 0-: failed to get index
19:59 partner jag3773: thanks, good you resolved it
19:59 Sunghost Hello JoeJulian ;)
19:59 Sunghost clusternode02:/vol1                                    1178248352   390130 1177858222    1% /mnt/clustervol1
20:00 Sunghost both systems are new, perhaps you remember before i reinstalled both i had a problem with ext4 on raid6 on boot / grub device
20:00 Sunghost this are the new ones
20:00 Sunghost so brick1 is full and i added yester the new and fresh installed 2 brick
20:01 Sunghost actual rebalance is in progress
20:03 Sunghost now i see some xfs kernel messages from new md2->  xfs_dir2_block_lookup+0x48/0xdc [xfs] <- problem?
20:05 gildub joined #gluster
20:08 Sunghost the problem is that i must be sure that both bricks run without problems, if so than another 20tb whould be added
20:08 JoeJulian It will probably crash the clients before it's complete. Which distro are you using?
20:09 bene2 joined #gluster
20:09 JoeJulian oh, debian... I just re-read that.
20:09 JoeJulian @ppa
20:09 glusterbot JoeJulian: The official glusterfs packages for Ubuntu are available here: 3.4 stable: http://goo.gl/u33hy -- 3.5 stable: http://goo.gl/cVPqEH -- introducing QEMU with GlusterFS 3.4 support: http://goo.gl/7I8WN4
20:09 JoeJulian @beta
20:09 glusterbot JoeJulian: I do not know about 'beta', but I do know about these similar topics: 'beta-yum', 'yum-beta'
20:09 JoeJulian hmm
20:10 JoeJulian @qa
20:10 Matthaeus joined #gluster
20:10 JoeJulian semiosis: Are you building the current 3.5.1 betas?
20:11 JoeJulian Sunghost: If he is, you want the latest beta. There's a bug fix for crashing clients during rebalance.
20:12 Sunghost beta? mh. you think this could be the problem and not the xfs on one of the bricks?
20:14 Sunghost still in progress but it seems to be very low only 6.mb in size and 316 files on both with 12 skipped on brick2
20:17 partner so umm is it a 2 brick distributed volume? then this all would make sense, at least i've got it broken that way in my testings, thought with older version
20:19 Sunghost yes 2 brick distributed - in version 3.4x all was fine
20:20 Sunghost joined #gluster
20:20 primechuck joined #gluster
20:20 zaitcev joined #gluster
20:20 _dist joined #gluster
20:20 jobewan joined #gluster
20:20 bennyturns joined #gluster
20:20 daMaestro joined #gluster
20:20 japuzzo joined #gluster
20:20 eryc joined #gluster
20:20 nage joined #gluster
20:20 jrcresawn joined #gluster
20:20 elico joined #gluster
20:20 TheDingy joined #gluster
20:20 johnmark joined #gluster
20:20 marmalodak joined #gluster
20:20 silky joined #gluster
20:20 weykent joined #gluster
20:20 portante joined #gluster
20:20 partner joined #gluster
20:20 Licenser joined #gluster
20:20 d-fence joined #gluster
20:20 VeggieMeat joined #gluster
20:20 stigchristian joined #gluster
20:20 SteveCooling joined #gluster
20:20 semiosis joined #gluster
20:20 Dave2 joined #gluster
20:20 anotheral joined #gluster
20:20 necrogami joined #gluster
20:20 ndevos joined #gluster
20:20 JoeJulian joined #gluster
20:20 RobertLaptop joined #gluster
20:20 nixpanic_ joined #gluster
20:20 sauce joined #gluster
20:20 jiqiren joined #gluster
20:20 m0zes joined #gluster
20:20 coreping joined #gluster
20:20 cyber_si joined #gluster
20:20 rturk|afk joined #gluster
20:20 glusterbot joined #gluster
20:23 MacWinner joined #gluster
20:23 AaronGr joined #gluster
20:24 Sunghost how is the current state? column status say in progress but below stay volume rebalance: vol1: cuccess ?! is it finish or not?
20:31 Sunghost now failed for localhost = brick2
20:31 Sunghost disconnecting connectionfrom clusternode02
20:32 Sunghost Shutting down connection clusternode02
20:33 bene2 joined #gluster
20:36 Sunghost partner JoeJulian?
20:36 plarsen joined #gluster
20:36 julim joined #gluster
20:36 wgao_ joined #gluster
20:36 Pupeno joined #gluster
20:36 morse joined #gluster
20:36 kkeithley joined #gluster
20:36 ry joined #gluster
20:36 GabrieleV joined #gluster
20:36 sadbox joined #gluster
20:36 sadbox joined #gluster
20:36 primusinterpares joined #gluster
20:37 dblack joined #gluster
20:40 Sunghost i added 3.5 in my sources.list http://download.gluster.org/pub/gluster/glusterfs/3.5.... how to change to beta 3.5.x ?
20:40 Intensity joined #gluster
20:40 partner sorry, playing at the background with 3.5.0
20:40 fyxim_ joined #gluster
20:41 Sunghost oh ok, nop
20:41 Sunghost strange rebalance status says for brick2 failed and for brick1 in progress - whats hapend now
20:42 partner i lost the track of how your setup is so i have hard time guessing where the issue could be, i'll leave it for experts
20:42 Sunghost faild or in progress?
20:42 Sunghost ok any questions for my setup?
20:42 Sunghost its simply installed via own source list -> deb http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/Debian/apt wheezy main
20:42 glusterbot Title: Index of /pub/gluster/glusterfs/3.5/3.5.0/Debian/apt (at download.gluster.org)
20:44 Sunghost debian wheezy was installed from installer nothing self compiled or anything like this
20:44 semiosis Sunghost: i haven't made any deb packages for 3.5.0 beta yet
20:44 Sunghost hello semiosis - you are the one who build the dep packages ;) good to know
20:44 Sunghost good work so fare
20:44 Sunghost far
20:45 semiosis thank you
20:46 semiosis i'll make debs for 3.5 beta tonight, i'll publish them here: http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.1beta2/Debian
20:48 Sunghost ok for me its not urgent so use the night and sleep ;) i can wait 2 days or so
20:48 partner yeah, big thanks to semiosis for the packages, added with all the sources and dsc stuff, makes my life so much easier compiling them further (yeah, i still have the squeeze boxes around :)
20:49 partner once the 3.4.4-2 is out i cast a spell on you giving ^10 karma :)
20:50 Sunghost short question version 3.5.0 is stable or did i install a beta?
20:50 semiosis i guess i should make the 3.4.4-2 packages tonight too
20:50 partner and was doing some packaging for jessie too but was left with some unsorted issues there which might be related more to my build environment
20:50 semiosis 3.5.0 is "stable" as much as any .0 release can be stable :)
20:51 partner yeah, i refused to go to 3.5.0 on production for that very reason :)
20:51 Sunghost ok, i read the list in the last days and saw lots of problems i think
20:53 partner i wonder if 3.5.0 refuses to rebalance my bricks because a) they are on the same host and b) they are on the same mount under different dirs.. :)
20:53 Sunghost ah ok so you have the same problem like i
20:54 partner i wouldn't count on that as i have this rather weird setup which i used for testing removing replication
20:55 Sunghost ok - back to my initial problem - this rebalance problem leads to the message disk full, or may it another failure?
20:56 partner as its distributed - would it be so that the hash points your file to be written to that full brick instead of the one having free space?
20:57 semiosis rebalance is always tricky, and filling a disk is always bad.  combine the two...
20:58 semiosis now, there's bugs being fixed in rebalance (that we know of) with patches causing new releases any day now
20:58 partner yeah, i tested that at around 3.3.1 and the results we for example disk full while df reported still free, the writes hit the full brick..
20:58 sjm left #gluster
20:58 semiosis so, even if you get past the no free space error, the rebalance might *still* fail for some other reason, like this rebalance memory leak bug
20:58 Sunghost detail sais 2,5GB for brick1 and 5,4TB for new brick2
20:59 Sunghost so i whould say full is not full ,
20:59 Sunghost ;)
20:59 semiosis insufficient info
21:00 semiosis if you want to convince me, give me a pastie with: df -m & df -i from your servers and the output of 'gluster volume info'
21:00 semiosis ...while i go get a cup of coffee.  brb
21:01 Sunghost df -m /dev/md2                                                 5697803     579   5697225    1% /media/node02
21:01 Sunghost df -i /dev/md2                                               1167327424    66499 1167260925    1% /media/node02
21:02 Sunghost log -> 0-glusterfs: Files migrated: 0, size: 0, lookups: 305, failures: 15, skipped: 12
21:02 Sunghost hope your cup of coffee is ready ,)
21:02 Sunghost more from log -> Fix layout failed for
21:02 Sunghost and 0-glusterfs: Rebalance is failed. Time taken is 1871.00 secs
21:02 partner better paste them all into one, also from both servers added with the volume info
21:03 Sunghost all from new brick2
21:03 partner not here, gets flooded
21:03 Sunghost sure
21:03 partner its difficult to say from just pieces of log entries, hard to draw the big picture
21:04 partner also full log lines and possibly the ones surrounding if they seem related
21:05 semiosis Sunghost: my coffee is ready, but your pastie is not.  i only see one brick, from one server, and no gluster volume info
21:06 semiosis use pastie.org to send multi line pastes
21:07 semiosis ping me when you get all that info together, i have to get back to work
21:07 Sunghost i understand but there are lot of lines and some have private state and not for the eyes of third persons and at all not for the world of pastie... you know
21:08 Sunghost some more info -> [dht-layout.c:640:dht_layout_normalize] 0-vol1-dht: found anomalies in
21:08 Sunghost [dht-selfheal.c:742:dht_fix_layout_of_directory] 0-vol1-dht: 1 subvolume(s) are down. Skipping fix layout.
21:09 Sunghost [dht-rebalance.c:1515:gf_defrag_fix_layout] 0-vol1-dht: Fix layout failed for
21:09 Sunghost thats what it is in general and continue sometimes
21:11 partner hmm one subvolume is down?
21:12 Sunghost mh i only have one volume
21:12 partner did the gluster volume status all detail tell its online or not? we can't help as we don't see the output
21:12 partner one volume yes but subvolume is different thing
21:13 partner you can quickly view that with "gluster volume status" command if everything is online or not
21:13 Sunghost gluster volume status all
21:13 Sunghost Status of volume: vol1
21:13 Sunghost Gluster process                                         Port    Online  Pid
21:13 Sunghost ------------------------------------------------------------------------------
21:13 Sunghost Brick clusternode01:/media/node01/vol1                  49152   Y       3149
21:14 Sunghost Brick clusternode02:/media/node02/vol1                  49152   Y       3087
21:14 Sunghost NFS Server on localhost                                 2049    Y       3100
21:14 Sunghost NFS Server on 192.168.0.41                              2049    Y       3154
21:14 Sunghost Task Status of Volume vol1
21:14 Sunghost ------------------------------------------------------------------------------
21:14 Sunghost There are no active volume tasks
21:14 Sunghost ok understand subvol is not the same as volume
21:18 Sunghost i remember one thing from yesterday - i first run volume add-brick - it looks good, after checking the status i dindt saw the new brick so i found the failure and run peer probe and than add-brick
21:19 semiosis Sunghost: please use pastie.org for multi-line paste
21:19 Sunghost ok thought this lines are not to much for irc, sorry
21:21 wushudoin joined #gluster
21:23 partner i recall seeing similar issue on mailinglist and it turned out to be some connectivity issue. but hard to say, we don't see logs to spot such
21:24 partner ohwell, past midnight, i go background not to hassle more :o
21:25 Sunghost ok i will check my network, while its very simple with one nic for each brick and a simple switch ;) which works for all other clients
21:25 partner on that case everything was online and was looking perfect, the issues started when rebalance was issued
21:26 Sunghost i dont want to hassle too, its just a discussion ;)
21:27 Sunghost ok so i have to wait for deb of beta
21:28 partner its just so hard to debug when we can't see enough, that's all. if you are afraid of putting something out then replace some domain names and such with example.com or something such
21:29 partner but its often really valuable to see everything in the log instead of random snippets
21:30 Sunghost ok i understand that.
21:31 Sunghost but i wonder if the system is such unstable, while its setup and installed out of the box with no custom things
21:36 andreask joined #gluster
21:36 andreask joined #gluster
21:40 tryggvil joined #gluster
21:46 partner there are so many parameters involved. if we all would install a "vanilla" debian to work with they would be still all be different. sometimes its just a tiny thing that makes all the difference
21:47 partner or might be a bug aswell if it was all alright with the previous version, maybe better wait (if you can) for new packages to test with
21:50 Sunghost you are right, but normaly i am carefull before posting information and logs belongs to that, but your right - yes previous version seems to be right and i can wait for the package - thank for help
21:57 Matthaeus joined #gluster
21:59 jag3773 joined #gluster
22:01 tryggvil joined #gluster
22:05 dwilsonci joined #gluster
22:07 dwilsonci hullo. 1 question RE geo-rep. how to tell master to use the unprivileged account I've setup on slave?
22:09 dwilsonci adding "option mountbroker-geo-replication.<unprivileged_account> <slave_volume>" to /etc/glusterfs/glusterd.vol appears to be insufficient
22:13 dwilsonci ssh -i /var/lib/glusterd/geo-replication/secret.pem <unprivileged_account>@<slave_node_fqdn> give me following output:
22:13 dwilsonci [2014-06-17 22:13:05.519536] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
22:13 dwilsonci [2014-06-17 22:13:05.519598] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
22:13 dwilsonci [2014-06-17 22:13:05.519954] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
22:13 dwilsonci [2014-06-17 22:13:05.519993] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
22:13 dwilsonci [2014-06-17 22:13:05.688020] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
22:13 dwilsonci [2014-06-17 22:13:06.688225] I [cli-cmd.c:135:cli_cmd_process] 0-: Exiting with: 110
22:13 dwilsonci gsyncd initializaion failed
22:15 dwilsonci I think that means passwordless auth is setup … but 'gluster volume geo-replication <master_volume> <slave_node_fqdn>::<slave_volume> create push-pem fails with
22:15 dwilsonci "Passwordless ssh login has not been setup with gfs01.vpc242.customink.com."
22:15 dwilsonci hints anyone?
22:27 semiosis dwilsonci: use pastie.org for multi-line pastes please
22:29 dwilsonci apologies
22:32 plarsen joined #gluster
22:58 fidevo joined #gluster
23:00 _dist is anyone here experienced with tuning 10gbe on linux? I can't seem to push past 600MB/s
23:08 jbrooks joined #gluster
23:08 JoeJulian _dist write or read?
23:09 RicardoSSP joined #gluster
23:11 _dist JoeJulian: the specific test I'm doing is a dd over netcat from /dev/zero to /dev/null
23:12 JoeJulian So you're getting exactly half your possible bandwidth...
23:12 _dist Yeap, that's fair to say
23:12 JoeJulian duplex?
23:12 JoeJulian The number seems awfully suspect.
23:12 _dist it's actually an RR setup both sides have dual ports
23:13 _dist "bond_mode active-backup"
23:13 _dist The cards are Intel X540s
23:14 Matthaeus Actually using all 8 lanes?
23:15 _dist that would make perfect sense actually
23:15 * _dist checks
23:15 gildub joined #gluster
23:16 _dist Looks like "LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-"
23:18 _dist iperf has always given full speed, but I wanted to have something I test data to disk with as well so NC made sense.
23:19 Matthaeus _dist: also, I think that link bonding scheme will only give you the bandwidth of one of the links.
23:20 _dist Matthaeus: yeap, that's expected. The plan is that each cable will eventually go to a different switch, right now that's not the case :)
23:20 Matthaeus So, you're seeing 600 MB/s which is 4800 mbit
23:20 _dist right...
23:20 Matthaeus Which is in the right ballpark, at least.
23:20 Matthaeus What's your system load do when you're running iperf?
23:21 _dist iperf gives 10Gbps, it's just NC that gives 600Mbytes
23:21 _dist but I'll run it and let you know right now
23:21 Matthaeus I'd guess you're bottlenecking on something else.
23:22 _dist with iperf the load is extremely small, I'll retry the NC now
23:22 Matthaeus Also, what's your source/dest for nc?
23:22 Matthaeus dev/zero and /dev/null?
23:22 _dist I send /dev/zero to /dev/null
23:24 gildub joined #gluster
23:24 Matthaeus What happens if you nc to localhost?  What speeds do you get?
23:24 _dist same
23:24 Matthaeus It's not the network.
23:25 _dist I felt that way too, but I wasn't 100% certain of it.
23:26 Matthaeus It's likely that it's some part of the network subsystem that's bottlenecking.  I can get 21 GB/s on a dd if=/dev/zero of=/dev/null on my workstation.
23:26 Matthaeus What's your MTU?
23:26 _dist default, 1500
23:27 Matthaeus Jumbo frames may help here.
23:27 _dist I'll try it, but I doubt it'll take me from 600 - 21,000 :) <-- if you meant bytes
23:28 Matthaeus With the bonding mode you're using, your max will be around 10,000.
23:29 _dist ah, you meant bits.
23:29 * _dist trying same test using 9000
23:31 _dist unfortunately, mtu 9000 still at 600Mbytes
23:31 Matthaeus localhost too?
23:32 _dist yeap
23:32 Matthaeus paste your ifconfig?
23:32 _dist sure, btw dd locally to dev/null gets 9Bytes/s
23:34 _dist in that vmbr0 is the bridge that uses bond0
23:35 Matthaeus Your MTU on lo is pretty high, so we're not seeing a frame processing problem.
23:35 Matthaeus This is about where I get off the train and start aimlessly googling for a few hours.
23:36 _dist I figured NC to /dev/null would have the least steps in between, perhaps there's a simpler way? Less steps, less bottlenecks
23:37 Matthaeus dd to dev null gives tens of gigabytes, right?
23:37 _dist yeap if I don't go through NC
23:37 Matthaeus Adding nc into that mix cuts it down to 600 megabytes.
23:37 _dist right, well not actually 10s of it gives 9 gigabytes /sec
23:37 _dist (but I'm using 1M blocks)
23:37 Matthaeus srsly?  What hardware is this?
23:38 Matthaeus On a new-ish i7, I got ~21 GB/s
23:38 Matthaeus With 1M blocks.
23:38 _dist hmm, I should test my home stuff
23:38 Matthaeus But that might be your issue right there.
23:39 Matthaeus But what does iperf say?
23:39 _dist iperf says 10gbps, or 9.8 or there abouts depending
23:40 _dist processor on one server is 2xe5-2620 xeon and the other is 2xe5645 xeon (second is a bit older)
23:40 _dist newer one actually gets 11Gbytes/sec not 9
23:40 Matthaeus Given how slow the dd was, I'd say nc is probably running about as fast as it's gonna.
23:41 Matthaeus And changing the bonding mode isn't likely to fix that.
23:41 Matthaeus In fact, I'd bet that even if you changed the bonding mode to one that uses both links, you wouldn't see too much faster even with iperf.
23:42 Matthaeus But I'd be curious to see what happened.
23:42 _dist I can't imagine how an 11Gbyte bottleneck would limit me to 600Mbyte, but I'm missing something?
23:42 Matthaeus If the data stream has to pass through that bottleneck twice, it would.
23:43 _dist ?? Bytes! :) we're talking 88 gbps wouldn't twice still be 44 gbps ?
23:44 Matthaeus …nevermind.  I was doing bad math.
23:44 Matthaeus I shall don the cone of shame.
23:45 _dist np :) but you are getting 20 giga bytes on yours ? Honestly so I am on my home server which is _way_ slower
23:45 _dist but, my home server is running a fairly new kernel. I don't know how to properly compare that to the rhel6 that proxmox uses
23:46 Matthaeus https://dpaste.de/2aLD
23:46 glusterbot Title: dpaste.de: Snippet #271925 (at dpaste.de)
23:46 _dist thx
23:46 JoeJulian semiosis: I have passed along your expectation to build the patched 3.4.4-ubuntu1~precise2 tonight and everyone made plans around it happening, so if you can, I would sure appreciate it.
23:47 _dist any idea where else my bottleneck could be ?
23:49 Matthaeus netcat uses a 1k buffer.
23:50 Matthaeus When I run the dd test with a 1K buffer, I get 1.1 GB/s
23:50 Matthaeus Curious what you get?
23:50 mjsmith2 joined #gluster
23:50 _dist dd over nc?
23:51 Matthaeus dd if=/dev/zero of=/dev/null bs=1K count=10M
23:52 _dist 740Mbytes locally
23:52 _dist you get that with localhost test on 10gbe ?
23:53 Matthaeus There's your bottleneck.
23:53 Matthaeus It's the buffer size in nc.
23:53 Matthaeus What command line are you using to test nc?
23:53 _dist nc -lp 5000 > anywhere
23:54 _dist dd if=/dev/zero bs=1M count=4096 | nc -vvv IP port
23:56 Matthaeus Just a sec
23:56 _dist but what I'm using specifically is " nc -lp 5000 > /dev/null & dd if=/dev/zero bs=1M count=4096 | nc -vvv localhost 5000"
23:58 Matthaeus https://dpaste.de/r75h
23:58 glusterbot Title: dpaste.de: Snippet #271926 (at dpaste.de)
23:58 _dist so looks like netcat is to blame
23:59 _dist how were you getting higher speed before using nc ?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary