Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-07-03

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 mjsmith2 joined #gluster
00:06 pureflex joined #gluster
00:13 bennyturns joined #gluster
00:41 coredump joined #gluster
01:11 gildub joined #gluster
01:12 StarBeast joined #gluster
01:15 dtrainor how common is it to do a rebalance?
01:15 dtrainor is that something that people do regularly as part of maintenance?
01:16 JoeJulian dtrainor: I know a lot of people have actively avoided it because I said it was broken.
01:16 JoeJulian Which was kind of a shock to me at summit. :D
01:16 dtrainor hah
01:16 dtrainor I didn't know that
01:17 dtrainor Which summit?
01:17 JoeJulian The most recent, in SFO.
01:18 dtrainor Ok, so Red Hat Summit
01:18 dtrainor thought you'd be at OpenStack Summit
01:18 JoeJulian Ah, right. I wanted to be but the timing just didn't work.
01:19 dtrainor I co-preso'd for an OpenShift talk
01:19 dtrainor First time, I died on stage.
01:19 JoeJulian Hehe
01:20 JoeJulian I've been on stage since I was 5, so the stage doesn't bother me.
01:20 dtrainor Ah well.  Next year.
01:20 JoeJulian What bothers me is when I F up the preparation.
01:20 dtrainor I was voluntold for that track so I was kinda wtf....
01:20 JoeJulian Hehe, JMW?
01:20 dtrainor Yeah I didn't prepare much.  Like I said, next time.
01:21 dtrainor ?
01:21 JoeJulian Guess not.
01:21 dtrainor No.
01:21 StarBeast joined #gluster
01:21 dtrainor I don't get it?
01:22 JoeJulian johnmark
01:22 dtrainor ah, no.
01:23 dtrainor oh, hi.
01:25 dtrainor well i started a rebalance before I asked about it.  is it going to break me?
01:26 JoeJulian what version?
01:26 dtrainor 3.5
01:26 JoeJulian minor version?
01:27 dtrainor 3.5.1
01:27 JoeJulian You *should* be ok.
01:27 dtrainor What *normally* happens during a broken rebalance?
01:27 JoeJulian There was a bug prior to 3.5.1 that would crash the clients if you had open files.
01:27 dtrainor ah
01:28 JoeJulian Also prior to 3.4.5 in the 3.4 releases.
01:31 dtrainor cool, thanks
01:31 dtrainor i was going to reboot this fileserver buuuut that's prob not a good idea.
01:31 JoeJulian yeah.. I wouldn't.
01:36 dtrainor Alright... sun is down.  Time to hit two wheels and get out of the house.  Thanks again for the help.
01:37 JoeJulian You're welcome.
01:38 Alex 09:27:31 < JoeJulian> There was a bug prior to 3.5.1 that would crash the clients if you had open files.
01:38 * Alex blinks a few times
01:38 * Alex chmod -xs a few binaries just in case some sysadmin feels... plucky
01:39 Alex JoeJulian: Is there a page covering affected versions? (He asks, googling...)
01:40 JoeJulian Just bugzilla entries.
01:40 Alex We have built a reasonable amount of our infrastructure around Gluster, and yet still have very little internal -fu. :) Thanks, will dig.
01:40 JoeJulian I've tested the fix in my staging environment successfully.
01:41 JoeJulian Hoping to do production a week from Thursday.
01:47 Peter1 when would 3.5.2 be out?
01:48 JoeJulian Not sure. Is there a bug fix that you're looking for?
02:06 Peter1 several :)
02:06 Peter1 https://bugzilla.redhat.co​m/show_bug.cgi?id=1113778
02:06 glusterbot Bug 1113778: medium, unspecified, ---, pkarampu, ASSIGNED , gluster volume heal info keep reports "Volume heal failed"
02:07 Peter1 https://bugzilla.redhat.co​m/show_bug.cgi?id=1113403
02:07 glusterbot Bug 1113403: unspecified, unspecified, ---, pkarampu, POST , Excessive logging in quotad.log of the kind 'null client'
02:07 Peter1 https://bugzilla.redhat.co​m/show_bug.cgi?id=1115197
02:07 glusterbot Bug 1115197: medium, unspecified, ---, rgowdapp, NEW , Directory quota does not apply on it's sub-directories
02:07 Peter1 https://bugzilla.redhat.co​m/show_bug.cgi?id=1115199
02:07 glusterbot Bug 1115199: unspecified, unspecified, ---, pkarampu, NEW , Unable to get lock for uuid,  Cluster lock not held
02:07 pureflex joined #gluster
02:08 Peter1 The 1st two already known to be fix in 3.5.2
02:08 Peter1 1st one is more like ubuntu packaging
02:22 StarBeast joined #gluster
02:24 [o__o] joined #gluster
02:25 vpshastry joined #gluster
02:35 harish joined #gluster
02:52 Alex if I start to quite regularly see split brain errors (https://gist.github.com/3b0936e5aa01c11cea6a) - what kind of things should I start to look into? I"m a bit worried that I see these every day on new and old files alike, and the boxes themselves don't ever seem to have availability issues so it doesn't feel like 'true' split brain
02:52 glusterbot Title: gist:3b0936e5aa01c11cea6a (at gist.github.com)
02:56 Alex (I definitely see some of files on the underlying bricks have differing Modify times, for instance)
02:57 srjb joined #gluster
03:07 sputnik13 joined #gluster
03:09 sputnik1_ joined #gluster
03:10 sputnik13 joined #gluster
03:10 sonicrose can anyone tell me if i have 6 gluster VMs with 3 HDD each, would I get more performance using hardware RAID to make a 3 disk RAID 0 and having each VM have one brick, or to use stripe volume in gluster and export 3 bricks?
03:11 sputnik13 joined #gluster
03:12 bennyturns sonicrose, I wouldn't use stripe
03:13 nbalachandran joined #gluster
03:13 sonicrose well i'll be using the stripe xlator no matter, its just wether i'm doing gluster as stripe 9 replica 2 or stripe 3 replica 2
03:13 sonicrose im guessing having 6 bricks instead of 18 bricks would be alot less overhead in userland?
03:13 bennyturns stripe has too many perf problems for mwe to use for anything
03:14 sonicrose bennyturns, i know what you mean.  it took me 6 months of tweaking to finally get a stable and fast config but i finally figured it out the other day
03:14 bennyturns sweet!
03:14 sonicrose im now getting like 700MB/s out of just 3 servers with dual GbE
03:15 bennyturns I wouldn't even know what to recommend for that config
03:15 bennyturns wowo!
03:16 sonicrose i was only getting like 110MB/sec before i switched to stripe from distributed
03:16 bennyturns you should do a write up on it or something
03:17 sonicrose http://fpaste.org/115207/57402140/
03:17 glusterbot Title: #115207 Fedora Project Pastebin (at fpaste.org)
03:18 Alex sonicrose: So what does that translate to in terms of redundancy? (he says, googling, again :))
03:19 sonicrose 3 physical servers with 4 x 2TB drives and Dual GbE running Citrix 6.2, 2 gluster VMs per physical server, each one has its own NIC and 2 hard drives direct attached to the VM
03:19 sonicrose it means i can lose any 1 of the 3 physical servers and stay up
03:19 Alex Got it, makes sense
03:20 kshlm joined #gluster
03:21 Alex atm I have 2 servers, each with 10 bricks, in distributed-replicate. I'm seeing kinda poor performance but my load is lots of small (fseek/fstat) requests to large files. Just by swapping out gluster mounts for NFS I've managed to increase from ~1k (HTTP equiv) requests to 5k (presumably because it's now properly being fscached) - but I'll have a look at striped, too. Thanks :)
03:22 sonicrose it does make it so that you can't just go to any of the backend bricks and copy the files out of them though
03:22 sonicrose be warned
03:22 sonicrose all files are on all servers and only pieces
03:23 StarBeast joined #gluster
03:23 sonicrose dont wanna send anyone down that road without understanding the risks
03:23 sonicrose my new project though replaces these 3 servers with 3 new ones that have 6 x 2TB drives and dual 10GbE instead
03:25 sonicrose its weird though how the stripe xlator works in terms of reads
03:25 sonicrose no wait nvm
03:26 Alex Yeah, that's a good point - although trying to mush together the files is something I think we always think is a good to have but we never actually... do.
03:26 sonicrose its the replicate xlator that works weird in terms of reads
03:27 sonicrose when doing a striped replicated when I read from it I would expect all servers to be sending me data, but they dont, only the ones that respond first
03:27 sonicrose but that's kinda not so bad because it means that it is like a form of load balancing
03:27 sonicrose i just get 1/2 the read speed i was expecting
03:29 rastar joined #gluster
03:30 sonicrose http://gyazo.com/c0b309fcbe0d15309bdc2da4555d5005  my read speeds aren't as good as when i first tested it, but i have alot of VMs up and using it now too
03:30 glusterbot Title: Gyazo - c0b309fcbe0d15309bdc2da4555d5005.png (at gyazo.com)
03:36 glusterbot New news from newglusterbugs: [Bug 1115748] Bricks are unsync after recevery even if heal says everything is fine <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115748>
03:43 itisravi joined #gluster
03:50 shubhendu joined #gluster
04:00 JoeJulian ~extended attributes | Alex
04:00 glusterbot Alex: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
04:01 JoeJulian ~stripe | sonicrose
04:01 glusterbot sonicrose: Please see http://joejulian.name/blog/sho​uld-i-use-stripe-on-glusterfs/ about stripe volumes.
04:09 bennyturns joined #gluster
04:12 ndarshan joined #gluster
04:19 kanagaraj joined #gluster
04:20 Alex JoeJulian: if they differ, does that mean it is splitbrain? I mean, I agree there's a difference, but I'm struggling to work out the cause of the split brain, sorry :)
04:23 rjoseph joined #gluster
04:23 StarBeast joined #gluster
04:26 JoeJulian Alex: Study up on the #2 link.
04:26 JoeJulian Off to read a bedtime story...
04:26 Alex Night, cheers as always!
04:26 srjb joined #gluster
04:27 JoeJulian I'm going to make an expert out of you yet... ;)
04:33 ramteid joined #gluster
04:41 sahina joined #gluster
04:42 ppai joined #gluster
04:45 hagarth joined #gluster
04:46 kdhananjay joined #gluster
04:51 saurabh joined #gluster
04:54 psharma joined #gluster
04:56 spandit joined #gluster
05:02 dusmantkp__ joined #gluster
05:02 vimal joined #gluster
05:03 srjb joined #gluster
05:08 prasanthp joined #gluster
05:09 lalatenduM joined #gluster
05:09 vpshastry joined #gluster
05:13 RameshN joined #gluster
05:15 nishanth joined #gluster
05:21 rastar joined #gluster
05:24 StarBeast joined #gluster
05:32 vkoppad joined #gluster
05:36 srjb joined #gluster
05:36 davinder16 joined #gluster
05:44 kumar joined #gluster
05:47 dusmantkp__ joined #gluster
05:47 srjb joined #gluster
05:47 hagarth joined #gluster
05:49 Peter1 joined #gluster
05:49 Peter1 what is the native NFS block size of gluster?
05:50 rjoseph joined #gluster
05:54 srjb joined #gluster
06:01 mjrosenb woah, qemu has gluster support?
06:16 raghu joined #gluster
06:20 fsimonce joined #gluster
06:25 StarBeast joined #gluster
06:32 lalatenduM mjrosenb, yeah
06:33 RameshN_ joined #gluster
06:40 Thilam joined #gluster
06:48 nishanth joined #gluster
06:53 ekuric joined #gluster
06:58 ctria joined #gluster
06:59 stickyboy Anyone seen this Petabyte thing from last year?  http://jread.us/2013/06/one-petabyte-red-h​at-storage-and-glusterfs-project-overview/
07:01 stickyboy He's daisy chaining DAS boxes via SAS...
07:02 sijis joined #gluster
07:02 sijis joined #gluster
07:11 SpComb is using raw disks as bricks really annoying enough to not be worth it?
07:12 SpComb I mean, the nicely thing about hardware raid arrays generally is that replacing disks is pretty trivial - the controller blinks a LED, you yank the drive out of the slot and put a (blank?) new one in and it starts rebuilding onto the new drive without even asking you (?) about it
07:12 keytab joined #gluster
07:12 SpComb how much would it take to get a similar level of maintenance ease with glusterfs, using individual disks directly a bricks
07:14 SpComb you'd need something to a) figure out a brick is broken b) pull it out of service c) determine the physical slot and set its failure LED d) detect a new drive being inserted e) pull it into service f) replicate back onto it
07:14 SpComb all automatically
07:14 SpComb does anybody do that?
07:25 Philambdo joined #gluster
07:26 StarBeast joined #gluster
07:27 ktosiek joined #gluster
07:29 vpshastry joined #gluster
07:29 [o__o] joined #gluster
07:34 ramteid SpComb: What I can answer: your assumptions about HW RAID is correct (LED, blank disk, auto-rebuild) at least that's my experience with HW ctrl in DL380. other questions I can not answer though
07:48 ninkotech joined #gluster
07:55 StarBeast joined #gluster
08:17 stickyboy SpComb: That's my approach too, about hardware RAID.  Red Hat's recommendation in Red Hat Storage Server is RAID6.
08:18 stickyboy I lost 2 disks in my RAID5 a few weeks ago and syncing my 30TB of data was a pain in the ass.
08:18 stickyboy For me, all new storage servers will be RAID6.
08:19 SpComb you could get a lot more out of network storage clustering stuff if you could just write to disks directly without an additional layer of RAID on the hardware
08:20 stickyboy SpComb: It's always a tradeoff. :P
08:20 SpComb if you had suitable automation for the glusterfs bricks
08:20 SpComb I wonder if ZFS would help
08:25 tty00 SpComb: do you want to run glusterfs on a linux machine with zfs support in userland?
08:26 tty00 ah, glusterfs servers can be run on fbsd :)
08:30 SpComb ZOL has native linux ZFS
08:31 SpComb *zfsonlinux
08:31 tty00 i found it now, but would you really want to use ZOL in prod and as disklayer for glusterfs?
08:32 SpComb why not - but the main point I was wondering about was non-raided disks and automated replacement
08:32 dusmant joined #gluster
08:34 aravindavk joined #gluster
08:36 Slashman joined #gluster
08:37 liquidat joined #gluster
08:38 meghanam joined #gluster
08:38 nishanth joined #gluster
08:39 sijis joined #gluster
08:39 ppai joined #gluster
08:47 stickyboy SpComb: There is discussion on the mailing list about ZFS from time to time.  I don't think it's ready.
08:50 shubhendu joined #gluster
08:54 ndarshan joined #gluster
09:00 fraggeln imho, I think a hw-raid is faster then zraid/zraid2, depending on the raidcontroller ofc.
09:00 fraggeln and, the battery powered cache in raidcontrollers is very nice as well.
09:09 glusterbot New news from newglusterbugs: [Bug 1115850] libgfapi-python client integration issue <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115850>
09:11 hagarth_ joined #gluster
09:20 jag3773 joined #gluster
09:21 aravindavk joined #gluster
09:21 SpComb but if you can reliably handle failover of individual disks then you can just replicate across three raw disks using glusterfs, and not have to pay the raid disk space penalty
09:22 SpComb *failure
09:23 fraggeln disks are cheap :)
09:29 deepakcs joined #gluster
09:32 ndarshan joined #gluster
09:38 dusmant joined #gluster
09:39 glusterbot New news from newglusterbugs: [Bug 1115850] libgfapi-python fails on discard() and fallocate() due to undefined symbol <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115850> || [Bug 1115852] libgfapi-python client fails on creat and open functional tests <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115852> || [Bug 1115864] Better naming/structuring required for libgfapi-python modules <https://bugzilla.redhat.com/show_bug.
09:39 bala1 joined #gluster
09:41 [o__o] joined #gluster
09:45 qdk joined #gluster
09:50 rwheeler joined #gluster
09:50 hagarth joined #gluster
09:59 Pupeno joined #gluster
10:06 davinder16 joined #gluster
10:14 kkeithley1 joined #gluster
10:15 RameshN joined #gluster
10:23 d-fence joined #gluster
10:33 ndarshan joined #gluster
10:39 davinder16 joined #gluster
10:42 foster joined #gluster
10:53 gildub joined #gluster
10:57 DV joined #gluster
10:58 eclectic joined #gluster
10:58 dusmant joined #gluster
11:01 shubhendu joined #gluster
11:03 ctria joined #gluster
11:24 monotek joined #gluster
11:26 ninkotech joined #gluster
11:26 77CAAO8ZF joined #gluster
11:27 monotek joined #gluster
11:28 RameshN joined #gluster
11:30 RameshN_ joined #gluster
11:39 rjoseph joined #gluster
11:40 glusterbot New news from newglusterbugs: [Bug 1115748] Sparse file healing of VM image files is not healing holes properly <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115748>
11:41 RicardoSSP joined #gluster
11:43 somebub joined #gluster
11:43 somebub left #gluster
11:47 RameshN joined #gluster
11:49 lalatenduM joined #gluster
11:49 vpshastry joined #gluster
11:53 nhayashi joined #gluster
11:55 LebedevRI joined #gluster
12:05 kshlm joined #gluster
12:08 theron joined #gluster
12:10 gildub joined #gluster
12:10 glusterbot New news from newglusterbugs: [Bug 1115949] [USS] : "ls -l" on .snaps directory from nfs mount gives " Remote I/O error " <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115949>
12:14 kanagaraj joined #gluster
12:16 ctria joined #gluster
12:21 hagarth joined #gluster
12:35 edward1 joined #gluster
12:44 nshaikh joined #gluster
12:44 hchiramm__ joined #gluster
12:47 B21956 joined #gluster
12:49 chirino joined #gluster
12:58 RioS2 joined #gluster
13:00 edong23 joined #gluster
13:08 plarsen joined #gluster
13:11 ctria joined #gluster
13:12 Pupeno_ joined #gluster
13:13 cyberbootje joined #gluster
13:13 cyberbootje joined #gluster
13:14 cyberbootje joined #gluster
13:17 davinder16 joined #gluster
13:20 dusmant joined #gluster
13:21 julim joined #gluster
13:28 bennyturns joined #gluster
13:33 RioS2 joined #gluster
13:38 japuzzo joined #gluster
13:45 hchiramm__ joined #gluster
13:49 diegows joined #gluster
13:54 qdk joined #gluster
14:04 ctria joined #gluster
14:09 Pupeno joined #gluster
14:10 dusmant joined #gluster
14:11 coredump joined #gluster
14:11 sahina joined #gluster
14:14 wushudoin joined #gluster
14:15 Slashman_ joined #gluster
14:20 Pupeno joined #gluster
14:22 bala1 joined #gluster
14:25 srjb joined #gluster
14:47 srjb left #gluster
14:48 ctria joined #gluster
14:56 dino82 joined #gluster
14:58 Pupeno_ joined #gluster
14:59 deepakcs joined #gluster
15:00 bennyturns joined #gluster
15:03 tru_tru joined #gluster
15:05 coredump joined #gluster
15:07 mortuar joined #gluster
15:08 jobewan joined #gluster
15:11 _Bryan_ joined #gluster
15:18 Slashman_ joined #gluster
15:30 harish_ joined #gluster
15:31 coredump joined #gluster
15:42 bala joined #gluster
15:45 Peter1 joined #gluster
15:56 obelix_ joined #gluster
15:57 Peter1 i m still having the du and df space usage different issue over NFS on gluster
15:57 vpshastry joined #gluster
15:58 Peter1 i umount and mount the NFS and still holding the space
15:58 Peter1 is there a way to find unlinked inode on gluster??
15:59 ctria joined #gluster
16:03 dino82 Is there a way to have gluster automatically sync/replicate upon rejoining the cluster without need of I/O to trigger it?
16:03 semiosis dino82: that has been standard since version 3.3.0
16:03 semiosis there's a self-heal daemon running on each server
16:03 semiosis ,,(processes)
16:03 glusterbot The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal).
16:07 dino82 Hmm let me try it again to see if it does it
16:07 dino82 I assume killing all gluster-related processes is enough to simulate a failure without bringing down a node completely?
16:07 RameshN joined #gluster
16:07 RameshN_ joined #gluster
16:08 semiosis correct, though you would only need to kill a brick export daemon process (glusterfsd)
16:08 torbjorn1_ Peter1: maybe I'm misunderstanding your question, but how about "lsof -n | grep deleted" ?
16:08 semiosis or firewall its port
16:08 dino82 Ah I don't see glustershd running, should it be, on each server instance?
16:08 semiosis what version of glusterfs are you using?  what distro?
16:09 dino82 3.3.1 CentOS 6.4
16:09 semiosis thats pretty old, but it should have the self heal daemon
16:11 dino82 It's the latest version in EPEL.  I may be missing a package or two.  Here's what is installed:  glusterfs-fuse-3.3.1-15.el6.x86_64 glusterfs-3.3.1-15.el6.x86_64  glusterfs-server-3.3.1-15.el6.x86_64
16:11 JoeJulian @yum
16:11 glusterbot JoeJulian: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
16:11 semiosis JoeJulian: just in time i was about to have dino82 upgrade to ubuntu!
16:12 dino82 ha
16:12 semiosis ;)
16:12 JoeJulian lol
16:12 Peter1 torbjorn1_:i did lsof and doesn't exist
16:12 Peter1 i know process are tie to a folder
16:12 Peter1 i tried to mv it and didn't free up space
16:12 dino82 Ah good thanks for the link
16:12 semiosis see also ,,(3.4 upgrade notes)
16:12 glusterbot http://vbellur.wordpress.com/2013/​07/15/upgrading-to-glusterfs-3-4/
16:12 semiosis dino82: ^
16:12 Peter1 i just did a rm on the folder and that cleared up all the empty space
16:13 Peter1 i wonder if we can identify the inode of these kind of empty space issue
16:13 dino82 Is 3.4 preferred over 3.5, any gotchas with the latest version?
16:13 Peter1 more like an unlinked inode
16:14 JoeJulian Peter1: If I'm understanding your question correctly, you have a file in a quota restricted directory. You mv a file out of that directory and the quota remains unchanged. If you rm the file, the space is released from the quota.
16:15 lpabon_test joined #gluster
16:21 Peter1 JoeJulian: i mv as as rename
16:21 Peter1 in the same directory
16:21 Peter1 if i delete the folder or move it out the directory, it free up the space and align the df and du
16:22 JoeJulian Peter1: So you mv a file within a quota directory and now twice the size of the file is used up out of the quota.
16:23 Peter1 yes
16:23 Mo_ joined #gluster
16:25 vpshastry joined #gluster
16:25 dino82 I upgraded to 3.4.4 but it doesnt' seem to be self-healing, I took one server down, touched a file in the volume, and brought the process back online, but it didn't sync the new data until I performed i/o on it from a client
16:26 dino82 There's no glustershd process, if there needs to be
16:27 torbjorn1_ dino82: the "glustershd" process is actually a special invocation of the glusterfs process
16:27 torbjorn1_ dino82: what about "pgrep -f glustershd -l" ?
16:28 dino82 Ok, yes that does exist. glusterfs being the process that called it
16:30 kkeithley_ @repo
16:30 glusterbot kkeithley_: I do not know about 'repo', but I do know about these similar topics: 'git repo', 'ppa repo', 'repos', 'repository', 'yum repo'
16:30 kkeithley_ @git repo
16:30 glusterbot kkeithley_: https://github.com/gluster/glusterfs
16:30 kkeithley_ @yum repo
16:30 glusterbot kkeithley_: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
16:31 dino82 I suppose it isn't a huge deal that the files don't immediately appear until there's i/o
16:31 dino82 the files still exist in the volume regardless
16:31 kkeithley_ there shouldn't be any glusterfs left in EPEL repos. Use the YUM repos shown above for RPMs of community GlusterFS
16:32 torbjorn1_ dino82: AFAIK the glustershd process works by walking the filesystem and triggering the self-heal, pretty much the same way "find . -type f | xargs stat" would
16:32 torbjorn1_ dino82: so you might have to wait until glustershd kicks in, or the walk hits the file your looking at
16:32 dino82 Does it do it on an interval or is there a trigger?
16:33 torbjorn1_ dino82: I think there is an interval, but I'm not 100% sure .. I seem to remember it being tunable as well
16:33 torbjorn1_ *vague stuff*
16:33 dino82 Hmm ok, thanks, I'll dig deeper
16:34 dino82 key  = "cluster.heal-timeout" ?
16:34 dino82 "Heal timeout is time interval for checking the need to self-heal in self-heal-daemon default is 10minutes can be changed to any efficient value."
16:35 ghenry joined #gluster
16:37 dino82 That seemed to work better
16:41 dino82 thanks ++
17:00 zerick joined #gluster
17:01 theron joined #gluster
17:11 jbrooks joined #gluster
17:12 cfeller joined #gluster
17:15 Bardack joined #gluster
17:20 johnmark JoeJulian: you rang?
17:20 dtrainor joined #gluster
17:21 LebedevRI joined #gluster
17:22 vpshastry joined #gluster
17:24 ctria joined #gluster
17:25 sonicrose whats it mean when status says Brick n1:/bra/nass1r1                                   N/A     Y       1131
17:25 sonicrose N/A on Port Number
17:45 pureflex joined #gluster
17:47 sonicrose 14 random seeks per second is kinda bad for a 3 disk raid0 huh :/
17:48 andreask joined #gluster
17:54 redbeard joined #gluster
18:00 vpshastry joined #gluster
18:11 michael__ joined #gluster
18:12 corwin7 I am unable to get gluster to run on a new Debian testing install. apg-get install glusterfs-server results in: "
18:12 corwin7 Setting up glusterfs-server (3.5.0-1) ...
18:12 corwin7 [FAIL] Starting glusterd service: glusterd failed!
18:13 sputnik13 joined #gluster
18:14 RioS2 corwin7, what do the logs say?
18:15 corwin7 # tail -n1 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
18:15 corwin7 [2014-07-03 18:13:33.424978] W [glusterfsd.c:1095:cleanup_and_exit] (-->glusterd(main+0x503) [0x7f379f1e8b93] (-->glusterd(glusterfs_volumes_init+0xb0) [0x7f379f1eba90] (-->glusterd(glusterfs_process_volfp+0x106) [0x7f379f1eb996]))) 0-: received signum (0), shutting down
18:16 corwin7 [2014-07-03 18:13:33.345436] E [glusterd-store.c:1979:glus​terd_store_retrieve_volume] 0-: Unknown key: brick-1
18:17 corwin7 I was attempting to rejoin a volume that the old machine was connected to.
18:19 corwin7 If I run "dpkg --purge glusterfs-client glusterfs-common glusterfs-server"  and reinstall it, it fails. Is there some place it's storing files that are not being cleaned out?
18:19 corwin7 I see /var/gluster is not being removed
18:23 corwin7 also /var/lib/glusterd/
18:24 corwin7 deleting /var/lib/gluster, var/gluster and purging and reinstalling cleared the error/
18:25 dtrainor that's not awesome
18:26 corwin7 This seems really flakey. Has anyone had good luck with gluster in a production environment or is this fuse stuff just hobby level? The zfs fuse implementation was cool as hell but only a novelity.
18:27 dtrainor I've not encountered that.  Is the hostname 'brick-1' by chance?
18:28 corwin7 nope, the old machine was bleys, new machine caine.
18:28 dtrainor Just did a quick search, saw some things to indicate that DNS might be involved
18:30 corwin7 The names of the machines are all in the local /etc/hosts file...
18:30 dtrainor Just throwing it out there
18:30 dtrainor Do you still have the contents of the old /var/lib/glusterd ?  Can you compare them against what you have now?
18:32 corwin7 you could easily be right dtrainor. No, I deleted it.  I never had this machine configured, I've been trying to replace the old bleys machine but haven't gotten anything working.
18:32 dtrainor ah ok understood
18:33 dtrainor http://www.gluster.org/pipermail/gl​uster-users/2013-August/036984.html
18:33 glusterbot Title: [Gluster-users] Need help understanding the glusterd issue (at www.gluster.org)
18:35 ricky-ti1 joined #gluster
18:40 corwin7 # gluster volume info
18:40 corwin7
18:40 corwin7 Volume Name: obsidian
18:40 corwin7 Type: Replicate
18:40 corwin7 Volume ID: b4ae50fd-6c10-4a91-a3de-0494923232f3
18:40 corwin7 Status: Started
18:40 corwin7 Number of Bricks: 1 x 2 = 2
18:40 corwin7 Transport-type: tcp
18:40 corwin7 Bricks:
18:40 corwin7 Brick1: bleys:/var/gluster/data
18:40 corwin7 Brick2: dimitar:/var/gluster/data
18:41 corwin7 I need to change bleys to caine. Anyone have a clue how to do this? The documentation I've found at gluster.org is unusable.
18:43 theron joined #gluster
18:44 theron_ joined #gluster
18:48 theron joined #gluster
18:54 corwin7 # gluster volume replace-brick obsidian bleys:/var/gluster/data caine:/var/gluster/obsidian start
18:54 corwin7 volume replace-brick: failed: brick: bleys:/var/gluster/data does not exist in volume: obsidian
18:56 corwin7 so "gluster volume info" says that's brick1, and gluster volume replace-brick says it doesn't exist.
19:00 corwin7 I'm starting to think I've been wasting my time with gluster as it appears to be a toy
19:11 theron joined #gluster
19:30 MacWinner joined #gluster
19:30 jbrooks left #gluster
19:37 theron_ joined #gluster
19:37 semiosis corwin7: please use pastie.org or similar for mutliline pastes
20:01 semiosis corwin7: I haven't looked at the packages in debian in a while (i probably should).
20:01 semiosis corwin7: you'll probably have better luck with the debs on download.gluster.org, see ,,(latest)
20:01 glusterbot corwin7: The latest version is available at http://download.gluster.org/p​ub/gluster/glusterfs/LATEST/ . There is a .repo file for yum or see @ppa for ubuntu.
20:01 semiosis http://download.gluster.org/pub/​gluster/glusterfs/LATEST/Debian/
20:02 glusterbot Title: Index of /pub/gluster/glusterfs/LATEST/Debian (at download.gluster.org)
20:02 semiosis [14:26] <corwin7> This seems really flakey. Has anyone had good luck with gluster in a production environment or is this fuse stuff just hobby level? The zfs fuse implementation was cool as hell but only a novelity.
20:02 semiosis yes lots of people use this in production, with deployments varying from a few GB all the way to PBs
20:09 corwin7 I set my web servers up to share files with gluster and also set up gluster between a couple of desktops. I now believe I made a huge mistake and need to get the production servers off gluster before there is a failure and I'm screwed as the tools contradict each other and the volumes seem unrepairable.
20:11 MacWinner joined #gluster
20:13 corwin7 fuse should have been enough of a clue that this is a toy. I'm off to set up Luster. I use zfs anyway.
20:13 corwin7 quit
20:13 corwin7 exit
20:16 jonathanpoon joined #gluster
20:18 azenk joined #gluster
20:23 theron joined #gluster
20:24 wushudoin joined #gluster
20:24 theron joined #gluster
20:30 azenk I'm seeing a lot of LOOKUP (null) (<uuid>) ==> "Stale file handle" messages in the brick logs, I'm also seeing a "0-graph: init failed" error whenever I try to set the brick log level.  Does this make sense to anyone?
20:36 ninkotech_ joined #gluster
20:44 necrogami joined #gluster
20:51 jonathanpoon hi everyone, I have 2 glusterfs data nodes that I'm running a replicated volume on.  I'm looking to increase the performance.cache-size parameter.  Each data node contains 16GB of RAM.  If I set the cache-size parameter to 8GB, would that mean each data node would use 8GB of RAM?  or split between the data nodes so each one uses 4GB of RAM?
20:53 sputnik13 joined #gluster
21:00 obelix_ joined #gluster
21:06 wushudoin joined #gluster
21:06 stickyboy joined #gluster
21:11 plarsen joined #gluster
21:33 semiosis wow corwin7 was trolling with skill.  knew all the right things to say.  bravo!
21:38 semiosis azenk: maybe if you put the whole log (or a big chunk of it) on pastie.org i might see something interesting, but that message alone doesnt mean much to me
21:38 cicero joined #gluster
21:38 TheSov joined #gluster
21:38 TheSov does anyone here use gluster-nfs in production in a vmware environment?
21:39 semiosis jonathanpoon: i believe that parameter applies to each server individually, but i'm not certain
21:40 Peter1 semiosis: did u experience du super slow on gluster with many small files?
21:40 coredump joined #gluster
21:41 semiosis Peter1: i dont use du on gluster, but i would guess that, like ls, du would be slow if you have a lot of files
21:41 Peter1 what do u use to check system usagE
21:42 Peter1 i m having issues that the du is diff then df
21:42 Peter1 i m running 3.5.1 on ubuntu
21:42 glusterbot New news from newglusterbugs: [Bug 1116150] [DHT:REBALANCE]: Rebalance failures are seen with error message " remote operation failed: File exists" <https://bugzilla.redhat.co​m/show_bug.cgi?id=1116150>
21:44 semiosis Peter1: i have nagios watching the disk usage on my bricks.  I alert by percentage.  since the bricks fill up evenly, if one brick is 80% full, then the volume is 80% full.  that works great for me.
21:45 semiosis Peter1: how much of a difference?
21:45 semiosis is it possible du is counting twice as much because it is adding up the space reported in .glusterfs (which are just hard links that dont really take up space)
21:45 Peter1 for a 4TB filesystem it's around 600GB diff
21:46 semiosis hrm
21:48 Peter1 looks like the ls and lookup are slower on gluster then other NFS
21:49 semiosis that is not surprising
21:51 Peter1 is there a way to optimize that?
21:54 semiosis usually it comes down to network latency
21:54 Peter1 latency between gluster nodes?
21:54 sadbox joined #gluster
21:55 semiosis Peter1: for fuse clients, latency between client & all the servers.  for nfs clients, latency between the gluster-nfs server and the rest of the servers.
21:55 semiosis metadata ops (like directory listing) need to go to all bricks, because glusterfs is fully distributed (with no metadata server)
21:56 Peter1 thanks
21:56 semiosis file lookups trigger a self-heal check which checks the xattrs of the file on all replicas to make sure the replicas of that file are all in sync
21:56 Peter1 i m running NFS so it must be latency between glusteter nodes and clients
21:56 semiosis all of these things require multiple RTT across the network
21:57 semiosis although there is parallelism, it still takes some time, and with lots & lots of files, it adds up
21:57 Peter1 right
21:58 Peter1 so a flat directory with tons of small files are no good for gluster....
21:59 semiosis well, depends on what you do
21:59 semiosis my application never does a directory list or a du, so it wouldnt bother me
21:59 semiosis but if you depend on speedy directory list then yeah that would be bad
21:59 Peter1 log processing, job spooling
22:18 mjsmith2 joined #gluster
22:24 theron joined #gluster
22:43 sputnik13 joined #gluster
23:01 Ch3LL_ joined #gluster
23:02 HACKING-FACEBOOK joined #gluster
23:02 HACKING-FACEBOOK joined #gluster
23:02 HACKING-FACEBOOK joined #gluster
23:07 ThatGraemeGuy joined #gluster
23:11 PLATOSCAVE joined #gluster
23:14 SNow joined #gluster
23:14 SNow joined #gluster
23:14 social joined #gluster
23:15 MacWinner joined #gluster
23:15 irated joined #gluster
23:15 purpleidea joined #gluster
23:16 Norky joined #gluster
23:19 coredump joined #gluster
23:22 wushudoin joined #gluster
23:30 mortuar joined #gluster
23:35 Norky joined #gluster
23:43 glusterbot New news from newglusterbugs: [Bug 1116168] RFE: Allow geo-replication to slave Volume in same trusted storage pool <https://bugzilla.redhat.co​m/show_bug.cgi?id=1116168>
23:44 mortuar_ joined #gluster
23:56 mortuar joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary