Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-03-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 Guest19520 joined #gluster
00:04 elyograg rather than delete from r2, I am moving them to a save location.  don't want to take any risks.
00:10 JoeJulian +1
00:18 georgeh|workstat joined #gluster
00:22 elyograg what's the full getfattr syntax?  I have some very specific ones, but not the one that gets all attributes.
00:25 xymox joined #gluster
00:26 elyograg foudn it.
00:26 JoeJulian "-m" does a partial regex match against the name of the extended attribute.
00:27 khushildep_ joined #gluster
00:27 elyograg says -z is an invalid option for setfattr
00:27 JoeJulian sorry, -x
00:28 mattapp__ joined #gluster
00:28 JoeJulian ... these developers and their nonsensical letter shortcuts.
00:37 mattappe_ joined #gluster
00:38 xymox joined #gluster
00:39 mattapperson joined #gluster
00:45 cjanbanan joined #gluster
00:48 xymox joined #gluster
00:48 [o__o] joined #gluster
00:52 gmcwhistler joined #gluster
00:52 vpshastry joined #gluster
00:56 xymox joined #gluster
01:11 tokik joined #gluster
01:12 xymox joined #gluster
01:20 cjanbanan joined #gluster
01:20 xymox joined #gluster
01:29 xymox joined #gluster
01:31 ec2-user joined #gluster
01:36 cjanbanan joined #gluster
01:40 ec2-user left #gluster
01:41 ec2-user joined #gluster
01:41 ec2-user left #gluster
01:44 griz1 joined #gluster
01:45 cjanbanan joined #gluster
01:45 griz1 Looking for more information about using root squash with management host exceptions but not finding much in the manual. Anyone doing this?
01:47 vpshastry joined #gluster
01:51 sputnik13 joined #gluster
01:54 gdubreui joined #gluster
02:00 cjanbanan joined #gluster
02:08 cjanbanan joined #gluster
02:13 Guest19520 joined #gluster
02:17 cjanbanan joined #gluster
02:44 velladecin is there any way how to resolve trusted.glusterfs.dht.linkto="XX". I have number of these files and I don't like them :)
02:48 JoeJulian velladecin: Why don't you like them?
02:49 velladecin I'd rather if the files were where in the location instead of the pointer. According to my reading this happens when eg: one of the bricks runs out of space but I still have plenty of space on all the bricks.
02:49 JoeJulian Or if a file is renamed.
02:50 velladecin I didn't do any renaming either. I guess the question is if this is by design and can be left without intervention?
02:51 JoeJulian It's not really a big deal. You can rebalance. It may require a "force" at the end if a file would need to be moved from a less-full to a more-full brick to clear the link.
02:51 JoeJulian It is by design.
02:51 JoeJulian For more information:
02:51 JoeJulian @lucky dht misses are expensive
02:51 glusterbot JoeJulian: http://joejulian.name/blog/dht-misses-are-expensive/
02:51 velladecin Ok, I did try rebalance force and it didn't help, but I'll give it another go. But you say that I don't have to worry too much?
02:52 JoeJulian Nope
02:52 velladecin yep, I read that one
02:52 velladecin Ok, thanks.
02:52 JoeJulian If you delete them all, "find -size 0 -perm 1000 -exec /bin/rm {}\;" they'll be recreated if they're needed.
02:53 velladecin I did delete some and then they came back :) the self-heal daemon brought them back I guess?
02:53 velladecin I left them in peace after that
02:53 JoeJulian Might as well.
02:56 velladecin One more question. In 'heal info' I had files that I couldn't fix by 'healing'. Then I checked attrs and all the attrs for replication were set to '1'. I had no split-brain anywhere in the logs and the files were 'un-healable'. After I changed the value for one of bricks to '0' and then the self-healing worked. Is this the right course of action?
03:07 sputnik13 joined #gluster
03:07 bharata-rao joined #gluster
03:14 cjanbanan joined #gluster
03:25 jporterfield joined #gluster
03:27 cjanbanan joined #gluster
03:33 Oneiroi joined #gluster
03:34 cjanbanan joined #gluster
03:35 shylesh joined #gluster
03:35 badone joined #gluster
03:37 itisravi joined #gluster
03:37 RameshN joined #gluster
03:37 harish_ joined #gluster
03:37 chirino joined #gluster
03:52 cjanbanan joined #gluster
03:52 shubhendu joined #gluster
03:53 kanagaraj joined #gluster
03:53 cp0k_ joined #gluster
03:54 yhben joined #gluster
04:00 rjoseph joined #gluster
04:04 elyograg JoeJulian: Not sure if you're still hanging around.  If files that are split brain are the ---------T links, they can be simply deleted, right?  oh, it says there are two files linked to that entry, so I need to find the other one.  searching by inum, that will be fun.  this is one of my gfid entries.
04:05 elyograg hoping xfs will be speedy at this: find . -inum 1074818825
04:10 velladecin if you're talking about the files 'partner' :) in the .glusterfs directory you can do this:
04:10 velladecin find <BRICKROOT> -type f | xargs ls -l | grep '\-T'
04:10 velladecin this will give you the 'T' file and it's 'partner'. This is only good when you have a few of them
04:10 elyograg i think there are a lot.  which is why I need to rebalance, which kicked off this whole problem.
04:10 velladecin otherwise: find <BRICKROOT> -samefile <FILE>
04:10 velladecin will give you both the files for the <FILE>
04:11 elyograg seems like there should be a faster way.  It knows there's another file linking to it, can't it know what that file is?
04:11 ndarshan joined #gluster
04:12 velladecin you can script it I guess. When you add '-delete' at the end of the second find command both the files will get deleted
04:13 elyograg i suspect that for all these, all I will have is a gfid.  which doesn't do me a lot of good in figuring out what file I need to actually test via the mount point.
04:13 elyograg I have to do a find across the entire filesystem.  millions of files and directories.
04:14 velladecin the 'T' files have 'trusted.glusterfs.dht.linkto' to where the file is actually located
04:14 elyograg how do I determine what brick that is on?
04:15 elyograg it has trusted.glusterfs.dht.linkto=0x6d6466732d7265706c69636174652d3900
04:15 velladecin the 'trusted.glusterfs.dht.linkto' will have smth like '=volume-replica-0'
04:15 velladecin get rid of the '-e hex' from getfattr
04:16 elyograg it's volume-replicate-9 ... is that the tenth server listed in a volume info?
04:16 elyograg s/server/brick
04:18 velladecin grep -r 'volume-replicate-9' /var/lib/glusterfs and follow the chain of client/replica in the config file
04:18 velladecin that will tell you exactly which servers are the correct replica.
04:19 CheRi joined #gluster
04:19 velladecin */var/lib/glusterd
04:19 sputnik1_ joined #gluster
04:19 elyograg that's a directory, not a file.
04:20 cjanbanan joined #gluster
04:20 velladecin I meant do the 'grep' from this dir not the one I initially said
04:20 CheRi joined #gluster
04:21 gmcwhistler joined #gluster
04:21 mattappe_ joined #gluster
04:22 elyograg i still don't have a "real" pathname to look for.
04:24 elyograg i think there's no way to avoid locating the other hardlink.  since I have zero idea what the file is other than .glusterfs/96/4e/964e1c50-a511-4699-a4e1-d6f0a9bf89e0 ... looks like I just get to crawl the brick for that inode number.
04:30 ppai joined #gluster
04:31 saurabh joined #gluster
04:31 johnbot11 joined #gluster
04:33 johnbot11 joined #gluster
04:34 lpabon joined #gluster
04:39 satheesh joined #gluster
04:47 pvh_sa joined #gluster
04:47 jporterfield joined #gluster
04:52 deepakcs joined #gluster
04:53 kdhananjay joined #gluster
04:54 hagarth joined #gluster
04:58 johnbot11 joined #gluster
04:59 elyograg i was getting REALLY worried about how things are going when I looked at split-brain and heal-failed, but after finding bug 864963 I think I might only need to be concerned about heal info.
04:59 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=864963 low, medium, ---, vsomyaju, ASSIGNED , Heal-failed and Split-brain messages are not cleared after resolution of issue
05:00 bala joined #gluster
05:00 lalatenduM joined #gluster
05:04 kshlm cd
05:04 meghanam joined #gluster
05:05 meghanam_ joined #gluster
05:09 cp0k_ joined #gluster
05:14 pvh_sa joined #gluster
05:16 mohankumar__ joined #gluster
05:21 spandit joined #gluster
05:25 davinder joined #gluster
05:32 prasanth joined #gluster
05:32 chirino_m joined #gluster
05:34 vpshastry joined #gluster
05:36 ndarshan joined #gluster
05:36 Guest52795 joined #gluster
05:36 ajha joined #gluster
05:36 prasanth joined #gluster
05:37 itisravi_ joined #gluster
05:40 shylesh joined #gluster
05:40 itisravi_ joined #gluster
05:40 nshaikh joined #gluster
05:41 edward1 joined #gluster
05:51 raghu` joined #gluster
05:52 RameshN joined #gluster
05:55 prasanth joined #gluster
06:03 Philambdo joined #gluster
06:07 kevein joined #gluster
06:15 vimal joined #gluster
06:17 jporterfield joined #gluster
06:23 TvL2386 joined #gluster
06:26 snehalphule_ joined #gluster
06:26 snehalphule_ deepakcs, ping
06:28 rahulcs joined #gluster
06:29 deepakcs snehalphule_, hi
06:30 snehalphule_ deepakcs, nice to see you here again :)
06:33 deepakcs snehalphule_, same here.. sent a pvt msg to u
06:33 deepakcs snehalphule_, lets talk over there
06:33 itisravi_ joined #gluster
06:34 itisravi joined #gluster
06:35 itisravi joined #gluster
06:43 shylesh joined #gluster
06:45 cjanbanan joined #gluster
06:49 elyograg ok, so JoeJulian has either found better things to do with his evening or he's gone to sleep.  anyone around that knows how to fix heal problems that are not covered by his blog posts?
06:52 pk1 joined #gluster
06:53 CheRi joined #gluster
06:59 jporterfield joined #gluster
07:14 yhben joined #gluster
07:18 harish_ joined #gluster
07:27 rossi_ joined #gluster
07:33 jtux joined #gluster
07:35 itisravi joined #gluster
07:35 ajha joined #gluster
07:35 prasanth joined #gluster
07:35 RameshN joined #gluster
07:35 yhben_ joined #gluster
07:37 bala joined #gluster
07:37 pk1 joined #gluster
07:37 shubhendu joined #gluster
07:39 pvh_sa joined #gluster
07:40 ndarshan joined #gluster
07:40 prasanth joined #gluster
07:40 itisravi joined #gluster
07:41 hagarth joined #gluster
07:41 ppai joined #gluster
07:41 elyograg If I go through everything in the volume via a fuse mount and stat it, is that roughly equivalent to a heal full?
07:42 deepakcs joined #gluster
07:42 spandit joined #gluster
07:43 shylesh joined #gluster
08:01 ctria joined #gluster
08:02 ngoswami joined #gluster
08:04 aravindavk joined #gluster
08:06 liquidat joined #gluster
08:16 cjanbanan joined #gluster
08:21 rahulcs joined #gluster
08:22 keytab joined #gluster
08:26 rgustafs joined #gluster
08:28 andreask joined #gluster
08:29 al joined #gluster
08:33 T0aD[FR] joined #gluster
08:37 junaid joined #gluster
08:37 cjanbanan joined #gluster
08:45 rfortier joined #gluster
08:49 lalatenduM joined #gluster
08:53 satheesh1 joined #gluster
09:06 haomaiwa_ joined #gluster
09:08 rahulcs joined #gluster
09:11 jiqiren joined #gluster
09:13 satheesh joined #gluster
09:16 ninkotech joined #gluster
09:16 ninkotech_ joined #gluster
09:22 ricky-ti1 joined #gluster
09:28 rahulcs joined #gluster
09:38 yhben joined #gluster
09:45 cjanbanan joined #gluster
09:49 rwheeler joined #gluster
10:10 yhben joined #gluster
10:19 hybrid512 joined #gluster
10:27 jporterfield joined #gluster
10:35 ctria joined #gluster
10:43 doekia joined #gluster
10:54 jporterfield joined #gluster
10:54 gdubreui joined #gluster
11:00 ProT-0-TypE joined #gluster
11:00 khushildep joined #gluster
11:09 haomaiwa_ joined #gluster
11:11 satheesh3 joined #gluster
11:15 RameshN joined #gluster
11:18 ctria joined #gluster
11:26 cjanbanan joined #gluster
11:30 jporterfield joined #gluster
11:37 yhben joined #gluster
11:39 latha joined #gluster
11:42 __123_cyber joined #gluster
11:47 ndarshan joined #gluster
11:50 kanagaraj joined #gluster
11:52 social kkeithley: Hi, do you know how is it with tests for gluster? Can I turn off the posix mode of shell or should I write thest purely in it? It helps to be able to use syntax like this to me:
11:52 social TEST  diff <( getfacl $M0 2> /dev/null | tail -n +2 ) <( getfacl ${V0}-${i} 2> /dev/null | tail -n +2 )
11:59 shubhendu joined #gluster
12:01 ppai joined #gluster
12:06 rahulcs joined #gluster
12:13 kkeithley social: are you talking about the tests in .../tests/{basic,bugs,performance/*.t ?
12:14 social kkeithley: yes
12:14 social kkeithley: it works fine in bug test when I just set +o posix so it's probably OK
12:22 kkeithley Yeah, I don't see why not. If it's not okay I would expect that to come out in the gerrit review of the whole patch.
12:32 social kkeithley: what about tests should I make one big set of tests for bug for all issues I'm chasing there or split it into smaller ones
12:32 social kkeithley: probably for speed considerations of tests it should stick together shouldn't it?
12:37 B21956 joined #gluster
12:37 kkeithley are all the issues you're chasing filed under a single BZ?
12:38 ccha joined #gluster
12:40 social kkeithley: yep
12:40 kkeithley Although if there are different bugs I'd personally prefer to see a separate BZ for each bug.
12:40 social kkeithley: I don't know whether it is different bug probably some same issue with different results
12:41 hagarth joined #gluster
12:41 kkeithley But bug-$BZ_NUMBER.t is the the convention for tests, so if they're all rolled up into an omnibus BZ then one bug-$BZNUM.t is okay
12:41 social kkeithley: right now I need to find out how to kill one brick so I can simulate outage :/
12:41 kkeithley okay, seems reasonable. I don't want to make more work, unnecessarily
12:42 aravindavk joined #gluster
12:42 RameshN joined #gluster
12:44 vpshastry joined #gluster
12:46 kkeithley don't you hang on to the pid of the glusterfsds when you start them? kill -9 it ought to do the trick I'd think
12:46 ndarshan joined #gluster
12:47 tokik joined #gluster
12:57 kanagaraj joined #gluster
12:58 shubhendu joined #gluster
12:58 ppai joined #gluster
13:06 ndarshan joined #gluster
13:07 Peanut Hey, is anyone at the Gluster Community meeting in Amsterdam? We're just starting.
13:12 social kkeithley: ok I have added tests now to finish the commit so it fixes the issues
13:16 lpabon joined #gluster
13:17 B21956 joined #gluster
13:17 hagarth joined #gluster
13:18 kanagaraj joined #gluster
13:18 shubhendu joined #gluster
13:18 ppai joined #gluster
13:18 ndarshan joined #gluster
13:25 andreask joined #gluster
13:25 mattappe_ joined #gluster
13:26 JoeJulian kill -9 will kill without shutting down the tcp connection, leading to the 42 second ping-timeout if that's what you want. kill -15 if you do want to close it.
13:38 sprachgenerator joined #gluster
13:39 B21956 joined #gluster
13:44 diegows joined #gluster
13:51 bennyturns joined #gluster
13:54 rahulcs joined #gluster
13:56 Guest19520 joined #gluster
13:56 owenmurr joined #gluster
13:56 eryc joined #gluster
14:00 vpshastry joined #gluster
14:06 khushildep joined #gluster
14:12 RayS joined #gluster
14:16 theron joined #gluster
14:25 theron joined #gluster
14:27 aravindavk joined #gluster
14:31 hybrid512 joined #gluster
14:34 RayS joined #gluster
14:35 mkzero joined #gluster
14:35 davinder joined #gluster
14:35 vpshastry joined #gluster
14:35 japuzzo joined #gluster
14:37 aixsyd joined #gluster
14:37 pk1 left #gluster
14:38 aixsyd JoeJulian, semiosis - after that bout of non-healing yesterday, I still am left with a few files that simply wont heal. they dont fail, split-brain or anything, they just hang out forever in the heal info output as needing to be healed. any ideas what to do?
14:39 aixsyd the one is a freshly put file, too. and its been long-done transfering. theres also no IO on either node.
14:41 lalatenduM aixsyd, have u tried a full heal?
14:42 aixsyd lalatenduM: i have not yet. shall i?
14:42 aixsyd lalatenduM: that usually involved a full crawl of both nodes, doesnt it?
14:42 aixsyd *involves
14:42 RayS joined #gluster
14:42 lalatenduM aixsyd, yeah, it goes through file by file and heal them
14:43 aixsyd if i'm (trying) to host VM disks from that cluster, performance should surely diminish greatly during said crawl, no?
14:44 lalatenduM aixsyd, yes, I also think so
14:44 aixsyd :(
14:44 JoeJulian Shouldn't. Heal ops are done in a lower priority queue.
14:46 aixsyd really?
14:46 aixsyd Hm.
14:47 lalatenduM JoeJulian, aixsyd I think it is not direct, self-heal uses good amount of memory, so if you have lots of RAM it should not be much of an issue
14:47 aixsyd 32gb :P
14:48 lalatenduM aixsyd, should be fine I guess, JoeJulian ^^
14:48 * JoeJulian shrugs
14:48 JoeJulian I don't know why it would use that much memory.
14:49 haomaiwa_ joined #gluster
14:50 aixsyd really strange, i'm still getting tons of "no active sinks" in my SHD logs
14:50 rpowell joined #gluster
14:50 aixsyd I just wrote a large (50gb) file to the volume, it wrote on both nodes, but it says the file needs to be healed. its long done transfering
14:50 JoeJulian What's a "getfattr -m . -d -e hex" say for one of those files?
14:51 aixsyd all i have is a gfid: cb771ee3-e456-4825-a135-b2323ae56cd6
14:52 aixsyd i assume its in <brick>/.glusterfs/indices/?
14:52 aixsyd says that command is no good
14:56 aixsyd Ahh
14:56 aixsyd JoeJulian: http://fpaste.org/82200/13939449/
14:56 glusterbot Title: #82200 Fedora Project Pastebin (at fpaste.org)
14:57 jobewan joined #gluster
14:58 JoeJulian That shows pending data operations for both bricks. That's normal during a write.
14:58 aixsyd *pending* - but nothing ever transfers
14:59 JoeJulian stop that vm and see if those go to 0x0.
14:59 aixsyd Whoa, wait a tic. my IB connection is at 1gbps
15:01 aixsyd what theheck.
15:04 nikk anyone familiar with the quorum feature?
15:05 nikk what i've read so far says that gluster will take a brick offline entirely if x number of peers aren't reachable.. i'm wondering if it's possible to leave it online in r/o mode instead of just killing it all together
15:06 nikk killing = making no read or write
15:06 nikk (note i haven't tested this at all yet)
15:07 bugs_ joined #gluster
15:12 haomai___ joined #gluster
15:15 davinder joined #gluster
15:15 aixsyd man, wheres jclift when I need him? XD
15:16 primechuck joined #gluster
15:17 primechuck Has anyone ever run gluster fuse + ~ 12 bricks + KVM/Qemu on the same box?
15:18 primechuck We had a setup that, under load, would just load up with kernel sys CPU load, and slow to a crawl with everything in user space.
15:18 cjanbanan joined #gluster
15:19 primechuck brick processes using 1800% load just doing futex
15:20 JoeJulian I have 60 bricks per server, and one of those servers runs 11 VMs.
15:26 davinder2 joined #gluster
15:26 primechuck That sounds promising
15:27 jobewan joined #gluster
15:27 vpshastry joined #gluster
15:27 primechuck Was half hoping for the easy answer of you can't do that.
15:27 primechuck :)
15:27 rahulcs joined #gluster
15:27 rotbeard joined #gluster
15:29 JoeJulian with that many bricks per server, I did set performance.cache-size = 8M
15:30 davinder joined #gluster
15:38 ndk joined #gluster
15:39 davinder joined #gluster
15:40 nikk JoeJulian: did you happen to see what i wrote earlier about quorum?
15:42 tdasilva joined #gluster
15:42 primechuck Would settings like this disable that?
15:42 primechuck performance.io-thread-count: 64
15:42 primechuck network.remote-dio: on
15:42 primechuck cluster.eager-lock: enable
15:42 primechuck performance.stat-prefetch: off
15:43 primechuck performance.io-cache: off
15:43 primechuck performance.read-ahead: off
15:43 primechuck performance.quick-read: off
15:43 primechuck cluster.data-self-heal-algorithm: diff
15:43 JoeJulian nikk: There are two types. Server quorum kills the server if it loses quorum. cluster quorum makes it read-only if the client loses a quorum connection.
15:43 irctc720 joined #gluster
15:44 lmickh joined #gluster
15:45 kanagaraj joined #gluster
15:46 davinder joined #gluster
15:47 JoeJulian primechuck: Why 64 io threads? Do you have that many io channels in your hardware?
15:47 chirino joined #gluster
15:48 JoeJulian As an aside, more than 3 lines is considered flooding. Please use a pastebin like fpaste.org for that kind of stuff. :)
15:49 RicardoSSP joined #gluster
15:49 RicardoSSP joined #gluster
15:49 JoeJulian back in a bit...
15:50 nikk JoeJulian: cluster quorum is awesome sounding.. i'll give that a try
15:50 primechuck yeah, my bad.  I clicked the wrong button when I pasted.
15:51 nikk i want all of my nodes to be able to live on their own in case of failures but i feel uncomfortable commiting new changes unless a certain % of nodes are online
15:51 davinder joined #gluster
15:53 primechuck Really only 20 channels on the disk.
15:53 primechuck *for disks.
15:56 davinder2 joined #gluster
15:56 primechuck Hmmm, wonder how the threading is done, that could cause the issues we've been seeing
15:58 vpshastry joined #gluster
16:02 rossi_ joined #gluster
16:03 davinder joined #gluster
16:07 irctc720 joined #gluster
16:08 plarsen joined #gluster
16:10 davinder joined #gluster
16:10 bennyturns joined #gluster
16:15 hybrid512 joined #gluster
16:16 haomaiwa_ joined #gluster
16:20 dewey joined #gluster
16:21 Philambdo joined #gluster
16:22 Philambdo joined #gluster
16:26 rossi_ joined #gluster
16:30 davinder joined #gluster
16:36 davinder joined #gluster
16:36 Slash_ joined #gluster
16:40 TvL2386 joined #gluster
16:44 davinder joined #gluster
16:45 gmcwhistler joined #gluster
16:49 davinder joined #gluster
16:54 gmcwhistler joined #gluster
16:54 davinder joined #gluster
17:01 hagarth joined #gluster
17:01 davinder joined #gluster
17:05 davinder joined #gluster
17:10 baoboa joined #gluster
17:11 davinder joined #gluster
17:15 davinder joined #gluster
17:23 mattappe_ joined #gluster
17:29 Mo__ joined #gluster
17:29 irctc720 joined #gluster
17:34 johnbot11 joined #gluster
17:35 jmarley joined #gluster
17:35 jmarley joined #gluster
17:35 mattappe_ joined #gluster
17:39 Matthaeus joined #gluster
17:45 rahulcs joined #gluster
17:48 rossi_ joined #gluster
17:58 Matthaeus1 joined #gluster
17:59 jiffe98 joined #gluster
18:00 tjikkun_work joined #gluster
18:06 rahulcs joined #gluster
18:10 zerick joined #gluster
18:11 irctc720 joined #gluster
18:13 kaptk2 joined #gluster
18:17 SFLimey joined #gluster
18:34 Yaz HI, has anyone here used BTrfs on brick partitions?
18:35 Yaz btrfs has data integrity functions I would like to use and wondered if there are any gotchyas
18:51 YazzY Yaz: i know people use ZFS for that
18:51 YazzY nice nick btw, bit short though ;)
18:51 rossi_ joined #gluster
18:52 cp0k Hey everyone, I am about to add new bricks to an existing volume and will need to fix the layout + migrate existing data. It seems this can be done via two separate steps or all in one (http://gluster.org/community/documentation/index.php/Gluster_3.2:_Rebalancing_Volume_to_Fix_Layout_and_Migrate_Existing_Data)
18:52 Yaz Great minds think alike YazzY
18:52 glusterbot Title: Gluster 3.2: Rebalancing Volume to Fix Layout and Migrate Existing Data - GlusterDocumentation (at gluster.org)
18:52 cp0k Any recommendations on which path to take?
18:53 stickyboy Yaz: ... I wouldn't recommend btrfs for production.
18:53 YazzY Yaz: :)
18:54 YazzY stickyboy: can you please elaborate why you wouldn't ?
18:54 stickyboy YazzY: Elaborate?  It's not stable yet.
18:54 YazzY but it is :)
18:55 Yaz thats not true stickyboy
18:55 stickyboy Ok.
18:55 kkeithley Is there a btrfs.fsck yet?
18:55 stickyboy If you think it's stable, then go put your data on it. :)
18:55 Yaz they are just polishing up things, but the main functions are solid
18:55 stickyboy Yaz: Have you used it?
18:55 mwoodson joined #gluster
18:55 stickyboy I've been trying it in various forms since 2009.
18:55 Yaz ive tested it
18:55 Yaz and in the process of testing it
18:55 Yaz im just worried more of the snapshot features could cause conflicts with gluster
18:56 YazzY Yaz: i was at the same point a few weeks ago and I chose to use the inkernel ZFS instead
18:56 Yaz but i dont know the inner workings of gluster enough to know
18:56 mwoodson i'm looking for some help with gluster.  I have 2 nodes that are not joining the cluster after an ip change
18:56 stickyboy Don't say I didn't tell you btrfs wasn't ready.
18:57 YazzY at least the storage format is stable now
18:57 stickyboy Yeah.
18:57 stickyboy That much is true.
18:57 YazzY the makers of btrfs say it is stable but to be honest i have no experience with it myself other than what I read about it
18:58 stickyboy I'm running it on my local NAS and decided recently to run Linux 3.14-rc so I can get the latest and greatest...
18:58 YazzY ZFS works great for me
18:58 stickyboy To quote Theodore Tso, ext4 is a stopgap until BTRFS is ready.
18:58 stickyboy But it's not ready yet. ;)
18:59 YazzY i have a 10TB volume with ZFS, speedy and stable as they come
18:59 YazzY speaking of stability,  is gluster stable yet?
19:00 kris joined #gluster
19:00 nikk YazzY: in theory
19:00 Yaz YazzY, I think gluster is still in development stages
19:00 Yaz last I heard its 2.0
19:00 Yaz amiright?
19:00 YazzY it sucked when i tested it with KVM, the VM broke badly when I removed the main node
19:00 stickyboy Well hundreds of us are using it in production. ;)
19:00 stickyboy I have a few minor niggles... but it's pretty magical.
19:01 nikk there were recently some improvements for hosting vm images.. saw it in a changelog someplace
19:01 nikk i think it was for 3.4
19:01 YazzY stickyboy: do you use it with live migration of KVM guests?
19:01 YazzY or any virtualization for that matter
19:01 nikk http://www.gluster.org/2013/07/glusterfs-3-4-is-here/
19:02 YazzY nikk: i'm on 3.4
19:02 nikk is that what you were having problems with before?
19:02 nikk or an older ver
19:02 samppah i'm using RHEV (kvm) with GlusterFS
19:02 YazzY nikk: on 3.4
19:02 nikk aah
19:02 stickyboy YazzY: Nope, only /homes on our compute cluster.  Haven't been brave enough to do KVM backing stores on gluster.
19:02 YazzY samppah: have you been testing node outtages?
19:03 YazzY stickyboy: good call ;)
19:03 JoeJulian developers are always sure their stable software is stable... then the real world gets their hands on it.
19:03 nikk userspace file system code is always a little scary to me but some very large companies use it for lots of different things
19:03 nikk sounds about right haha
19:03 samppah YazzY: yes, also I have had unexpected failure of storage node.. so far so good :)
19:04 YazzY samppah: so your running VMs survived broken main node?
19:04 YazzY samppah: can you show me how you mount the bricks on your KVM host?
19:05 YazzY this is my fstab entry: eddie.xyz.net:/KVM /gluster/kvm glusterfs  defaults,_netdev,backupvolfile-server=ziggy.xyz.net 0 0
19:05 samppah YazzY: mounting shouldn't matter as long as you are using native glusterfs client.. client only fetches volume information from one server and then connects to all servers
19:06 YazzY yeah, I'm using FUSE mount
19:06 samppah YazzY: that sounds good
19:06 YazzY still when I took down the eddie server, the VM's filesystem got badly damaged
19:07 YazzY it was still running until i took it down and could never take it up again (the VM)
19:07 YazzY seems like there could be issues with locking or something
19:07 samppah what's the exact things you did to bring eddie down etc?
19:08 Yaz Anyone here have experience with gluster and infiniband?
19:08 Yaz That low latency seems to nice to have
19:09 YazzY samppah: just rebooted it
19:09 samppah YazzY: hmm nothing else?
19:09 YazzY samppah: nothing else
19:09 samppah YazzY: eddie is only running glusterfs or also running KVM on it?
19:09 YazzY samppah: just glusterfs
19:10 YazzY samppah: the KVM server has the brick mounted
19:10 YazzY the brick is exported from eddie and ziggy
19:10 YazzY they are only running glusterfs
19:10 stickyboy Yah, I think I'm having locking issues...
19:11 samppah hmm.. sounds like there is something wrong really bad
19:11 stickyboy Need to mail the mailing list..
19:11 YazzY stickyboy: this is my gluster config http://pastebin.com/5Y0E2fiu
19:11 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
19:12 samppah brb.. i need to take the dogs for a walk
19:13 stickyboy YazzY: Not much to it
19:14 stickyboy YazzY: 3.4.2?
19:14 mwoodson hey guys, I'm using gluster in aws.  We recently changed IP's on the machines.  When  I did, I can not get the peers to get connected again
19:14 mwoodson i just have a simple 2 node replicated cluster
19:15 mwoodson any suggestions how to debug or help?
19:16 Matthaeus1 mwoodson, when you did your initial peer probe, what did you use for hostnames?
19:16 mwoodson i believe it was hostnames.
19:16 Matthaeus1 Fantastic.  Which hostnames?
19:16 mwoodson the hostnames for each of the server.
19:16 mwoodson i didnt' actually set this up, i inherited it
19:16 JoeJulian ~hostnames | mwoodson
19:16 glusterbot mwoodson: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
19:17 Matthaeus1 What does gluster peer status show?
19:17 JoeJulian "YazzY: it sucked when i tested it with KVM, the VM broke badly when I removed the main node" - there is no "main node"
19:17 Matthaeus1 EC2 instances sometimes have hostnames that reflect their ephemeral IP address, and thus these hostnames change after an IP address change.
19:18 mwoodson Matthaeus1: http://pastebin.com/Q9AKCuqH
19:18 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
19:19 mwoodson ahh..wrong pastebin.
19:19 mwoodson Matthaeus1: the names resolv to the right ips.
19:20 JoeJulian "Yaz: Anyone here have experience with gluster and infiniband?" Not I, directly, but yes. The lower latency is very nice to have, as well as rdma to save a context change.
19:20 Matthaeus1 mwoodson: Check security groups and ensure that you can connect from each host to the others on the gluster ports.
19:20 Matthaeus1 ~ports | mwoodson
19:20 glusterbot mwoodson: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
19:20 Matthaeus1 If not, it's probably your security groups.
19:21 YazzY stickyboy: yes, 3.4
19:21 mwoodson i can connect 24007 and 24009 (24008 is dead).
19:21 YazzY 3.4.2
19:21 mwoodson i'm on 3.2.
19:21 doekia joined #gluster
19:22 YazzY JoeJulian: the node defined as the first one in the fstab config
19:22 YazzY when i rebooted that node
19:22 doekia Q: how to specify more than one volume backupserver?
19:23 JoeJulian Point being, don't think of there being a "primary" and a whatever. They're all equals. Keeping that in mind usually makes troubleshooting more efficient.
19:23 YazzY doekia: backupvolfile-server=server1,server2,server3
19:23 JoeJulian ~mount server | doekia
19:23 glusterbot doekia: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
19:23 JoeJulian @rrdns
19:23 glusterbot JoeJulian: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
19:24 doekia thx I wasn't sure ... documentation is so sparse
19:24 YazzY doekia: i have in my fstab server1:/VOLUME /mount/point glusterfs  defaults,_netdev,backupvolfile-server=server2,server3 0 0
19:25 doekia huh own to put such a syntax in fstab?
19:25 YazzY what?
19:25 doekia weird doesn't comma collide with mltiple options?
19:26 YazzY no, you're weird! :D
19:27 doekia :-D I mean in fstab coma is used to separate de different options such as ro default ... I was wondering that server1,server2 could not collide
19:27 YazzY JoeJulian: that rrdns does not seem like a safe choice for running KVM
19:28 JoeJulian Been doing it for years.
19:28 YazzY doekia: it's been like that for ages
19:28 YazzY doekia: check mount options for NFS
19:29 doekia ok no prob ;-)
19:31 mwoodson Matthaeus: tell me if this means anything to you.  When I tcpdump, i see a lot of connections trying to/from 1022,1023 to 24007.
19:31 mwoodson those are probably blocked
19:32 m0zes_ joined #gluster
19:34 Matthaeus To -and- from 1022, 1023?
19:36 mwoodson Matthaeus:  i see communication going back and forth on those ports
19:36 mwoodson to 24007
19:36 JoeJulian Sure... from <1024 to the other 24007.
19:37 mwoodson JoeJulian:  do i need to open firewall for those?
19:38 JoeJulian drop your firewall and see if that cures your problem. If so, see what you did wrong.
19:39 JoeJulian Unless you're blocking outbound ports, I don't /think/ it should be a problem, but that's easy to test.
19:39 mwoodson should be...
19:47 mwoodson JoeJulian: i enabled all ports in my sg/firewall for testing.  It connected right up.
19:48 mwoodson so it is a firewall issue
19:49 Matthaeus mwoodson: I have a poster above my desk that says, and I quote, "It's the firewall, dumbass"
19:49 Matthaeus It's a handy reminder for me.
19:49 JoeJulian :D
19:49 mwoodson Matthaeus: i liked it.. I need add to it "or the security groups"
19:49 mwoodson double layer in aws
19:50 Matthaeus With AWS security groups, you can specify a security group instead of a CIDR address and it'll automatically include all instances contained in the specified security group.
19:50 mwoodson yeah, i did that.
19:50 mwoodson i dont' know if it's kosher to accept everything from within the sg
19:51 Matthaeus It's not perfect, but it's safer than accepting everything from 0.0.0.0/0
19:51 mwoodson sure, in testing.
19:51 mwoodson that's what i have setup now
19:52 stickyboy I have problems with files like ~/.Xauthority when I log into several systems which have ~/ on Gluster within a short time period.
19:52 stickyboy Not sure if it's locking or caching or what...
19:53 samppah YazzY: btw, are you using cache with KVM?
19:58 doekia Yazzy...?
19:59 doekia Got this with your fstab entry: [mount.c:298:gf_fuse_mount] 0-glusterfs-fuse: mount of web2:/www to /var/www.remote (default_permissions,web3,web1,allow_other,max_read=131072) failed
20:01 stickyboy What is cluster.eager-lock for?
20:02 gdubreui joined #gluster
20:04 mwoodson JoeJulian: do you know anyone who uses gluster in aws? i'd like to ask them how they configured their ports
20:04 JoeJulian stickyboy: Lock phase of a transaction has two sub-phases. First is an attempt to acquire locks in parallel by broadcasting non-blocking lock requests. If lock aquistion fails on any server, then the held locks are unlocked and revert to a blocking locked mode sequentially on one server after another.  If this option is enabled the initial broadcasting lock request attempt to acquire lock on the entire file. If this fails, we revert back to the
20:04 JoeJulian sequential "regional" blocking lock as before. In the case where such an "eager" lock is granted in the non-blocking phase, it gives rise to an opportunity for optimization. i.e, if the next write transaction on the same FD arrives before the unlock phase of the first transaction, it "takes over" the full file lock. Similarly if yet another data transaction arrives before the unlock phase of the "optimized" transaction, that in turn "takes ove
20:04 JoeJulian r" the lock as well. The actual unlock now happens at the end of the last "optimized" transaction.
20:04 JoeJulian mwoodson: semiosis
20:05 nikk JoeJulian: you were right, cluster quorum is perfect for what i'm doing, server quorum is bad juju.
20:05 mwoodson thanks! semiosis: you around to ask a few questions about aws configuration?
20:05 rotbeard joined #gluster
20:05 * semiosis is
20:05 semiosis dont ask to ask, just ask
20:05 mwoodson semiosis: i realize i have an issue with security groups in aws with my gluster config.
20:06 mwoodson i have ports 24007, 24009 opened
20:06 mwoodson but i get peers disconnected status
20:06 mwoodson i have port 111 tcp/udp as well
20:06 mwoodson when i opened all in my sg, everything started working.
20:06 cjanbanan joined #gluster
20:06 mwoodson what ports do you have opened to get gluster to work in aws?
20:07 semiosis mwoodson: what version of glusterfs are you using?  the latest is 3.4.2
20:07 mwoodson no, 3.2
20:07 semiosis you ought to use the latest
20:07 semiosis since you're just starting out (right?)
20:07 semiosis what distro?
20:07 mwoodson i'm inheriting a project. we are running rhel
20:07 semiosis ah hm
20:07 semiosis well ok
20:08 mwoodson 3.4.0.59rhs-1.el6rhs  I see that available..
20:08 mwoodson but i haven't upgraded to it
20:08 semiosis ,,(yum)
20:08 glusterbot The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
20:08 semiosis but upgrading is another thing altogether
20:08 mwoodson yeah
20:09 stickyboy JoeJulian: Guess that's from the "documenting the undocumented" ;)
20:09 stickyboy JoeJulian: If I don't understand it, I guess I don't need it.
20:09 semiosis mwoodson: the log /var/log/glusterfs/etc-glusterfs-glusterd.log should have more info about the connection issue between servers
20:09 JoeJulian stickyboy: Actually, that's from "gluster volume set help"
20:10 stickyboy JoeJulian: Ah.  Well it's the same then. :D
20:10 semiosis mwoodson: usual causes are name resolution and iptables/security groups
20:10 semiosis mwoodson: peer in cluster (disconnected) is talking about the connection between glusterd ,,(processes) on port 24007
20:10 glusterbot mwoodson: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/F6jqx for more
20:10 glusterbot information.
20:10 mwoodson semiosis: i see alot of things like: Error (Transport endpoint is not connected), peer (10.146.181.124:1021)
20:11 mwoodson where they are referring to port 1021, 1023, 1022.
20:11 mwoodson these aren't listed anywhere to open up
20:11 andreask joined #gluster
20:11 semiosis mwoodson: those are source ports, from which a connection is made to 24007
20:12 semiosis is glusterd running on the servers?
20:12 mwoodson semiosis: yes
20:12 stickyboy JoeJulian: Thought the eager locks thing might be related to the issues I'm having with files like ~/.Xauthority which get accessed nearly simultaneously by different hosts throwing read errors (also ~/.cpan/ and others).
20:13 mwoodson semiosis: when i disabled my security group and things were working, i did an lsof -i | grep gluster.
20:13 mwoodson I saw things like this:
20:13 mwoodson glusterd  18685   root   13u  IPv4 628836      0t0  TCP drupal1.xxxx.com:24007->ip-10-146-175-95.ec2.internal:1020
20:13 JoeJulian stickyboy: perhaps, or perhaps mount with entry-timeout=0 and/or negative-timeout=0
20:13 mwoodson where it looks like those connections are being made to those lower ports
20:14 stickyboy JoeJulian: Yah, seems the cluster.eager-lock is more a performance option from what I read (and the related HekaFS post).
20:14 semiosis mwoodson: a connection is established with the lower port, but the connection was initiated (first SYN) from the lower port to port 24007
20:14 mwoodson i see.
20:14 stickyboy JoeJulian: re: timeouts... lemme read up.  Thanks for pointer.
20:14 elyograg problems fixing heal entries continue today, but I'm dealing with another fire.  remote datacenter unplugs a cable from a router aux port ... and the T1 line protocol goes down.  It's making no sense.
20:15 semiosis mwoodson: are you in VPC?
20:15 semiosis or ec2 classic?
20:15 mwoodson no...traditional,
20:15 mwoodson err..classic, that's what it called
20:15 semiosis hmm, then security groups only block inbound connection requests
20:15 semiosis er, inbound traffic
20:15 semiosis not stateful
20:15 semiosis afaik
20:16 mwoodson i just allowed ports 1000-1023 and things started working again
20:16 JoeJulian If it's not stateful then that makes sense.
20:16 JoeJulian You'll need to open 1-1023.
20:16 semiosis so you need port 24007/tcp allowed in from any security group (including the same security group) where other gluster servers are
20:16 andreask joined #gluster
20:16 mwoodson i have that.
20:17 semiosis JoeJulian: well maybe it is stateful, right.  you dont need the priv ports allowed
20:17 mwoodson that's not what i'm seeing.
20:17 chirino_m joined #gluster
20:18 semiosis mwoodson: well if you can figure it out then by all means go ahead
20:18 mwoodson semiosis: this seems to be working, but i am the odd man out, so i don't feel like its valid
20:18 mwoodson semiosis: i appreciate your help
20:18 JoeJulian semiosis: Do you have a reference ec2 security group config?
20:18 semiosis mwoodson: but for me, all I have is 24007-24500 or something like that allowed
20:19 semiosis JoeJulian: no, it's not complicated
20:19 semiosis no one's ever had this kind of trouble that I know of
20:19 mwoodson yeah, that's why i feel like something else is going on
20:19 semiosis you just add a rule to allow 24007 & your briick ports and get on with life
20:20 semiosis i suspect there's some routing issue going on here... most likely name resolution
20:20 semiosis is it possible you're resolving to public-ipv4 instead of local-ipv4?
20:21 semiosis actually get back to the log
20:21 semiosis the message you found just says that a connection died, doesnt say why
20:21 semiosis maybe if you restart glusterd you'll get log messages showing the reason why connection cant be made
20:22 mwoodson let me try...
20:22 JoeJulian Network ACLs are stateless. You're not using those, are you?
20:22 mwoodson JoeJulian: im using iptables + security groups
20:23 semiosis brb, doing birthday pie for a coworker
20:25 JoeJulian I only like two kinds of pie...
20:25 JoeJulian ... hot and cold.
20:29 YazzY samppah: yes, I do, <driver name='qemu' type='qed' cache='writeback' io='native'/>
20:30 YazzY samppah: cache='writeback' gives me the most optimal write/read speeds
20:31 elyograg lovely.  got a bug in IOS that results in routers freaking out when something's connected to a serial port.  CSCsk25405.
20:40 aixsyd joined #gluster
20:41 aixsyd JoeJulian: so I now have two large files in a state of flux. http://fpaste.org/82316/65679139/
20:41 glusterbot Title: #82316 Fedora Project Pastebin (at fpaste.org)
20:41 mattappe_ joined #gluster
20:43 aixsyd JoeJulian: and I keep on getting these in the shd logs: [afr-self-heal-data.c:655:afr_sh_data_fix] 0-gv0-replicate-0: no active sinks for performing self-heal on file /images/303/vm-303-disk-1.qcow2
20:44 B21956 joined #gluster
20:44 aixsyd nothing has accessed this file in hours. theres no IO activity on either node. asking it to heal does nothing, changes nothing.
20:46 JoeJulian assuming the same state as earlier, where it has two trusted.afr entries, both non-zero, the no active sinks is probably because the non-zero status would define it as only a source with no sinks. After I get back this afternoon I'll verify that the code agrees with me.
20:46 borreman_dk joined #gluster
20:47 aixsyd what if i setfattr to all zeros?
20:47 elyograg wtf is happening here?
20:47 elyograg http://fpaste.org/82321/39660351/
20:47 glusterbot Title: #82321 Fedora Project Pastebin (at fpaste.org)
20:48 elyograg gluster is mounted via NFS.
20:50 aixsyd well, not sure if I should have done that, but it looks like its healing now o.o'
20:59 theron joined #gluster
20:59 aixsyd looks like one disk was fixed...
21:02 aixsyd question for anyone - if i delete a file from a brick of a node, will a triggered heal replicate that file back from the other node?
21:05 elyograg with 3.3 and later, that file is a hardlink to another file in the .glusterfs structure.  Don't know what happens if you delete one link but not the other.
21:06 JoeJulian usually the hardlink is recreated and the file will be in exactly the same state as it was before you deleted it.
21:07 aixsyd JoeJulian: gotcha.
21:08 aixsyd JoeJulian: also, setfattr to all 0's seemed to fix what was going on
21:08 aixsyd VMs boot up as normal, no data loss
21:09 JoeJulian I need to get my son to start doing animations of how this stuff works.
21:10 * semiosis getting trolled by someone spreading ubuntu package fud :(
21:11 semiosis i offer to give support in IRC, guy says something like people dont like beginners on IRC.  then goes & tells others my packages are broken.
21:11 semiosis argh
21:11 semiosis the thanks I get
21:13 elyograg I don't feel like beginners are badly treated here.  or the other tech channels I idle in.
21:14 Matthaeus And yet somehow, that one misanthrope has more impact on your day than the thousands of people who are grateful for your packages.
21:14 Matthaeus Myself included, btw.
21:14 semiosis Matthaeus: !!!!
21:14 JoeJulian Some... Don't go in to #centos as a beginner. "If you don't know THAT then your company should hire a REAL sysadmin."
21:14 semiosis Matthaeus: bugs me because there's no way I can help this person
21:15 semiosis Matthaeus: and if he discourages someone else from seeking help, that's bad for everyone
21:17 semiosis elyograg: glad to hear that.
21:18 chirino joined #gluster
21:18 elyograg of course, i know how to communicate what I need.  If someone who can't communicate effectively comes in, they may be unable to get their question across, and may go away frustrated.
21:19 semiosis elyograg: thats why we ask people for logs & command output :)
21:19 JoeJulian Most of the time I think we're pretty good at picking out what someone's really having a problem with, even when they don't know.
21:21 mattapperson joined #gluster
21:33 cfeller elyograg: I agree completely - I don't think beginners are treated badly here at all.  It was the help I got here, while trying to make heads and tails of the somewhat scattered documentation that really helped me out.  JoeJulian, semiosis and other have been a great help for me over the past year.
21:34 cfeller I've been running gluster in production since late last year and couldn't have done it w/out the help I got from everyone along the way.
21:37 badone joined #gluster
21:38 semiosis woo!
21:40 JoeJulian I will admit, though, when someone comes in here with a chip on their shoulder an an attitude expecting failure, I'm probably not as positive and helpful as I normal am.
21:40 JoeJulian s/normal/normally/
21:40 glusterbot What JoeJulian meant to say was: I will admit, though, when someone comes in here with a chip on their shoulder an an attitude expecting failure, I'm probably not as positive and helpful as I normally am.
21:43 social who here knows a bit more about dht self heal and xattrs?
21:44 JoeJulian No such thing.
21:44 semiosis @learn ask as dont ask who you can ask.  don't ask if you can ask.  just ask.
21:44 glusterbot semiosis: The operation succeeded.
21:44 JoeJulian dht is just the distribution process. There is not healing of that.
21:45 social JoeJulian: what does dht_selfheal_directory in xlators/cluster/dht/src/dht-selfheal.c then?
21:57 social anyway I slowly start to understand it but I'm bit puzzled about the acls there, it can be seen in http://review.gluster.org/#/c/7178/5/tests/bugs/bug-1063832.t < the acl on volume root and a_duck will fail atm
21:57 glusterbot Title: Gerrit Code Review (at review.gluster.org)
21:59 zaitcev joined #gluster
22:06 Joe630 joined #gluster
22:06 Joe630 greeting all, I am going thru the gluster quickstart and found an error in the docs.  What is the best way to report it?
22:07 Joe630 http://www.gluster.org/community/documentation/index.php/Getting_started_install
22:07 glusterbot Title: Getting started install - GlusterDocumentation (at www.gluster.org)
22:07 Joe630 the url for the packages for redhat is not quite right, so the wget command below it does not work.
22:08 semiosis Joe630: it's a public wiki, if you know the fix, you can make it yourself
22:09 semiosis or else this is a fine place to report it
22:09 Joe630 i thought so, but i can't find the edit button
22:09 semiosis create account first
22:09 semiosis logged in users get the edit button
22:09 Joe630 roger that.
22:09 Joe630 thanks
22:10 primechuck Is there an eqivelent of -o remount for glusterfs?
22:10 FarbrorLeon joined #gluster
22:10 semiosis no
22:10 semiosis or it's new and i dont know about it
22:12 primechuck That is kind of what I thought
22:12 nuxD ndevos: hi https://issues.apache.org/jira/browse/CLOUDSTACK-6181 do you see a problem with this from a GlusterFS point of view?
22:12 glusterbot Title: [CLOUDSTACK-6181] Root resize - ASF JIRA (at issues.apache.org)
22:15 elyograg now all of the directories in my 'heal $vol info' output are no longer directories, they are gfids. with readlink I can figure out what directory it is, but how do I fix this?
22:17 elyograg is there a way to get it to show them again as actual pths rather than gfids?
22:17 Joe630 semiosis: fixed, thanks for the tip.
22:17 semiosis thank *you* for improving the wiki!
22:20 khushildep joined #gluster
22:21 FarbrorLeon Anyone done any interesting performance testing native client vs nfs on gluster?
22:22 divbell yes
22:22 * divbell grins
22:22 Joe630 i think meant /me runs
22:22 Joe630 *think you meant
22:23 * Joe630 hides
22:23 divbell i don't want to hurt any feelings here, but the FUSE "native client" is expensive and slow
22:23 divbell and neither the FUSE client nor the NFS client allow POSIX acls, so integration with Samba+AD is not possible using either of those methods
22:24 divbell there is a vfs_gluster for samba now though
22:24 Joe630 hmm glusterd and glusterfsd aren't in chkconfig right.
22:24 Joe630 centos 6.5
22:25 FarbrorLeon divbell: How do you mean expensive?
22:25 divbell don't try to run something like a PHP-enabled document root on gluster fuse, the stats will kill your performance
22:25 semiosis divbell: well thats just like your opinion
22:25 divbell semiosis, haha <3
22:25 semiosis divbell: see ,,(php) for optimizations
22:25 glusterbot divbell: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
22:25 glusterbot --negative-timeout=HIGH --fopen-keep-cache
22:26 semiosis theres a lot you can do to get decent performance from php over gluster, but also lots you can do (or not do) to get terrible perf
22:26 elyograg i see them in chkconfig on a centos 6.5 server.  glusterd is on, glusterfsd is off.  glusterfsd is only called during shutdown, or manually.
22:26 divbell ooh, i'll have to take a fresh look.. i lost that customer though
22:26 divbell aye
22:27 Joe630 elyograg: this is from a 10 minute old installation:
22:27 Joe630 [root@gluster1 init.d]# chkconfig glusterd --list
22:27 Joe630 glusterd       0:off1:off2:off3:off4:off5:off6:off
22:27 cfeller joe630: just run "chkconfig glusterd on" that is all you need for 3.4.x.
22:27 Joe630 right, but it isn't default
22:27 cfeller but yes, it isn't enabled by default.
22:27 Joe630 hoaha jinx!
22:28 Joe630 back to the wiki i go
22:28 elyograg i like the fact that I have to enable it, myself.
22:28 social divbell: huh, we use gluster acls for some time :/
22:28 divbell with samba?
22:29 social divbell: no, with fuse mount
22:29 chirino joined #gluster
22:30 divbell yeah, i was just saying that gluster fuse or gluster nfs are incompatible with samba when also using active directory and posix acl-based permissions to bring over windows groups and whatnot
22:30 divbell i should get back to work now ;)
22:30 FarbrorLeon divbell: thanks for the tip. My use case is a HPC with 20 nodes. Currently it is performing worse than our 7 year old cluster..
22:33 larsks joined #gluster
22:41 johnbot11 joined #gluster
22:44 theron joined #gluster
22:47 Joe630 i feel like i am missing a huge amount of "config"
22:48 Joe630 we go from install to gluster peer probe
22:49 Joe630 [root@gluster1 init.d]# gluster peer probe gluster2
22:49 Joe630 peer probe: failed: Probe returned with unknown errno 107
22:49 cfeller iptables
22:49 cfeller trust your peer.
22:50 cfeller or you can open a port range.
22:50 Joe630 wow.
22:50 Joe630 i am a big dummy
22:50 cfeller but in my config, I just trust the IP of my gluster peer nodes.
22:50 Joe630 using a fresh install and not one of my standards ones, where iptables is off.
22:51 Joe630 cfeller: thank you.
22:51 cfeller Joe630: np.
23:03 calum_ joined #gluster
23:05 elyograg JoeJulian: (or anyone else who knows) I have worked out what to do for *files* on my 'heal info' report and for gfid values that refer to files or gluster links.  but I don't know what to do for directories.  most of these are directories: http://fpaste.org/82388/97427213/
23:05 glusterbot Title: #82388 Fedora Project Pastebin (at fpaste.org)
23:06 kshlm joined #gluster
23:17 Yaz Does anyone know if the gluster rdma code is stable enough to use? From the sounds of it as of 3.4 it isnt http://www.gluster.org/pipermail/gluster-users/2013-July/036440.html
23:17 glusterbot Title: [Gluster-users] tips/nest practices for gluster rdma? (at www.gluster.org)
23:18 Yaz in my testing it seems to work, but i havnt been able to mimic my production environment enough to get a good benchmark
23:33 mattappe_ joined #gluster
23:35 mattapperson joined #gluster
23:43 JoeJulian social: Don't confuse me with facts... ;)
23:44 elyograg JoeJulian: I'm writing up an email asking for help with healing directories.  I tried a few things but it didn't help.  Look for it on gluster-users momentarily.
23:46 JoeJulian Ok, my answer, though, is to delete the trusted.afr xattrs.
23:46 elyograg tried it.  must be something I'm not seeing about the exact steps. :)
23:51 chirino_m joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary