Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2013-12-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:28 theron joined #gluster
00:36 gdubreui joined #gluster
00:51 ProT-0-TypE joined #gluster
01:01 johnbot11 joined #gluster
01:02 Remco joined #gluster
01:04 sticky_afk joined #gluster
01:05 stickyboy joined #gluster
01:08 sarkis joined #gluster
01:12 yinyin joined #gluster
01:17 vpshastry joined #gluster
01:22 hflai joined #gluster
01:30 JoeJulian "Absolutely. Backup-to-disk is a great use case for gluster, lots of groups use us for that. We work well with RMAN, Netbackup, Commvault, etc. “
01:30 JoeJulian Anybody in here do that?
01:38 harish joined #gluster
01:41 shyam joined #gluster
01:47 jag3773 joined #gluster
01:59 nage joined #gluster
01:59 Remco joined #gluster
02:01 daMaestro joined #gluster
02:22 _Bryan_ joined #gluster
02:31 glusterbot New news from newglusterbugs: [Bug 1037267] network disconnect/reconnect does not resume data access to server <https://bugzilla.redhat.com/show_bug.cgi?id=1037267>
02:31 sonicrose semiosis, OMG i love you
02:39 sonicrose i am setting up an 8 Node GigE cluster tonight to see how it goes
02:45 bharata-rao joined #gluster
03:05 kshlm joined #gluster
03:07 johnbot11 joined #gluster
03:13 bsaggy joined #gluster
03:13 satheesh joined #gluster
03:13 bsaggy joined #gluster
03:21 sarkis joined #gluster
03:24 smellis sonicrose: i'm building smaller hypervisor clusters, 2 or 3 nodes with KVM and gluster, using infiniband cards from ebay, really improves the performance
03:25 johnbot11 joined #gluster
03:26 mattapp__ joined #gluster
03:30 sticky_afk joined #gluster
03:30 stickyboy joined #gluster
03:32 sonicrose how does infiniband cost stack up against 10GbE cards and switches?
03:36 bsaggy_ joined #gluster
03:36 bsaggy_ joined #gluster
03:40 smellis sonicrose: new it's about 99% of the cost of 10Gig, but it's 40Gig
03:40 smellis on ebay, I bought dual port mellanox DDR cards (20Gig) for $25 each
03:41 smellis the cables were about $15 each
03:41 smellis CX4-CX4 4x cables to be exact
03:42 smellis needs a little fiddling to get it working right, but it's pretty awesome for the cost (esp second hand ebay)
03:43 smellis I'm working on a 3 node ring of hosts right now on this DDR kit, getting near native performance to the disk using libgfapi
03:43 smellis My biggest problem before was gige
03:44 smellis it's definitely awesome if you don't have money, like me lol
03:49 itisravi joined #gluster
03:49 DV joined #gluster
03:50 rjoseph joined #gluster
03:55 saurabh joined #gluster
03:57 dhyan joined #gluster
04:02 bala joined #gluster
04:02 johnbot11 joined #gluster
04:04 johnbot11 joined #gluster
04:17 theron joined #gluster
04:23 shyam joined #gluster
04:24 sarkis_ joined #gluster
04:24 Nuxr0 joined #gluster
04:24 xymox joined #gluster
04:24 xymox joined #gluster
04:25 ulimit joined #gluster
04:26 johnbot11 joined #gluster
04:28 RameshN joined #gluster
04:30 bala joined #gluster
04:37 kanagaraj joined #gluster
04:38 anands joined #gluster
04:47 MiteshShah joined #gluster
04:50 dusmant joined #gluster
04:52 nshaikh joined #gluster
04:56 aravindavk joined #gluster
04:56 prasanth joined #gluster
04:57 shylesh joined #gluster
05:00 ppai joined #gluster
05:05 meghanam joined #gluster
05:10 raghu joined #gluster
05:13 sahina joined #gluster
05:14 spandit joined #gluster
05:16 CheRi joined #gluster
05:19 anands joined #gluster
05:20 mattapp__ joined #gluster
05:26 MiteshShah joined #gluster
05:31 vpshastry1 joined #gluster
05:37 shruti joined #gluster
05:46 anands joined #gluster
05:52 prasanth joined #gluster
05:58 Paul-C joined #gluster
06:00 Paul-C joined #gluster
06:02 glusterbot New news from newglusterbugs: [Bug 1040275] Stopping/Starting a Gluster volume resets ownership <https://bugzilla.redhat.com/show_bug.cgi?id=1040275>
06:02 Paul-C left #gluster
06:02 sarkis_ joined #gluster
06:12 ndarshan joined #gluster
06:19 Atin joined #gluster
06:26 bulde joined #gluster
06:28 NeatBasis joined #gluster
06:33 johnmwilliams joined #gluster
06:42 yinyin joined #gluster
06:45 ngoswami joined #gluster
06:52 krypto joined #gluster
06:55 ProT-0-TypE joined #gluster
06:56 dhyan joined #gluster
07:10 ctria joined #gluster
07:21 jtux joined #gluster
07:25 psyl0n joined #gluster
07:30 stickyboy joined #gluster
07:33 ekuric joined #gluster
07:37 thogue joined #gluster
07:46 stickyboy Need a GlusterFS logo to put in a slide...
07:47 stickyboy Why are all the logos on Google Images so small? :\
08:02 glusterbot New news from newglusterbugs: [Bug 1037501] All the existing bricks are not marked source when new brick is added to volume to increase the replica count from 2 to 3 <https://bugzilla.redhat.com/show_bug.cgi?id=1037501>
08:03 abyss^ stickyboy: check in google images: large at least one image with logo is very large;)
08:10 stickyboy abyss^: Cool, forgot about the search parameters hehe
08:10 eseyman joined #gluster
08:16 bharata-rao joined #gluster
08:16 keytab joined #gluster
08:18 ctria joined #gluster
08:29 ababu joined #gluster
08:42 zeittunnel joined #gluster
08:43 stickyboy joined #gluster
08:55 mohankumar joined #gluster
09:00 ricky-ti1 joined #gluster
09:02 glusterbot New news from newglusterbugs: [Bug 969461] RFE: Quota fixes <https://bugzilla.redhat.com/show_bug.cgi?id=969461>
09:06 pithagorians joined #gluster
09:07 pithagorians hey guys, what can cause cpu wait time double more than user on cluster members ?
09:08 pithagorians and if i should worry about it
09:19 harish joined #gluster
09:22 ctria joined #gluster
09:23 calum_ joined #gluster
09:25 Norky joined #gluster
09:25 hagarth joined #gluster
09:28 vpshastry1 joined #gluster
09:29 bharata-rao joined #gluster
09:29 geewiz joined #gluster
09:32 glusterbot New news from newglusterbugs: [Bug 1040348] mount.glusterfs needs cleanup and requires option validation using getopt <https://bugzilla.redhat.com/show_bug.cgi?id=1040348>
09:35 vimal joined #gluster
09:49 gdubreui joined #gluster
09:56 andreask joined #gluster
10:03 glusterbot New news from newglusterbugs: [Bug 1040355] NT ACL : User is able to change the ownership of folder <https://bugzilla.redhat.com/show_bug.cgi?id=1040355>
10:09 ricky-ticky joined #gluster
10:15 mbukatov joined #gluster
10:21 badone joined #gluster
10:22 harish joined #gluster
10:40 vpshastry1 joined #gluster
10:46 psharma joined #gluster
10:53 rjoseph joined #gluster
11:06 mohankumar joined #gluster
11:06 psyl0n joined #gluster
11:20 hybrid5121 joined #gluster
11:23 ninkotech_ joined #gluster
11:23 ninkotech__ joined #gluster
11:31 nshaikh joined #gluster
12:07 clag_ joined #gluster
12:17 CheRi joined #gluster
12:19 badone joined #gluster
12:20 hagarth joined #gluster
12:21 ppai joined #gluster
12:21 sgowda joined #gluster
12:30 B21956 joined #gluster
12:33 ctria joined #gluster
12:45 sgowda joined #gluster
12:52 vpshastry1 joined #gluster
13:02 ppai joined #gluster
13:04 marbu joined #gluster
13:08 mkzero has anybody tried out the cluster.min-free-disk on non-uniform brick setups w/ gluster 3.4.x? i'm currently migrating data to a replicated gluster setup and it didn't seem to work very well :/
13:13 chirino joined #gluster
13:13 bennyturns joined #gluster
13:14 dhyan joined #gluster
13:14 zeittunnel joined #gluster
13:18 kanagaraj_ joined #gluster
13:19 mohankumar joined #gluster
13:23 edward1 joined #gluster
13:27 cfeller joined #gluster
13:30 ricky-ticky joined #gluster
13:33 ccha4 does there any cache for glusterfs client ?
13:33 ccha4 or any fuse optimizations ?
13:35 mkzero ccha4: if i recall correctly, performance.cache-size is what you are looking for
13:35 ccha4 but is for server side right ?
13:36 ccha4 I already have performance.cache-size: 512MB
13:36 ccha4 for the volume
13:37 vpshastry joined #gluster
13:41 bala joined #gluster
13:41 dhyan joined #gluster
13:42 andreask is there somewhere a description how the self-heal daemon works? i mean how does it find out what to heal without accessing all files?
13:42 davidbierce joined #gluster
13:42 vpshastry2 joined #gluster
13:43 FooBar any pointers to what a good 'cache-size' should be (my machines have 64G dedicated to gluster ;P )
13:44 ccha4 andreask: I check with gluster volume status
13:45 tqrst has anyone running 3.4.1 noticed anything weird with rebalance status? Mine has entries that are all localhost except for the last one: http://pastie.org/private/oxkqt3awlqtcu061innq
13:45 glusterbot Title: Private Paste - Pastie (at pastie.org)
13:45 tqrst the only way I can get some approximation of the status is to run status on all servers and grep for non-localhost entries
13:46 tqrst and even then, that only gives me a handful of them
13:46 kmai007 joined #gluster
13:46 tqrst I've reported a bug for this already but I'm curious if anyone else has encountered this (and, more importantly, if there is a workaround)
13:46 kmai007 when you run a rebalance in gluster3.4.1-3, do you only need to run it on 1 gluster node, or must you run it on all servers?
13:46 mkzero tqrst: had that problem too. if you run it multiple times on the same node the last one changes and the others stay localhost.
13:47 tqrst kmai007: you launch it from one
13:47 kmai007 thank you tqrst
13:47 tqrst and then the rest joins in the memory-hogging fun
13:47 tqrst and then some will inexplicably finish days before others, leaving you scratching your head
13:47 tqrst mkzero: glad to see I'm not the only one :)
13:47 kmai007 so from the doc. it appears that the function of migrate-data is no longer an option in the rebalance cmd ?
13:47 sroy joined #gluster
13:48 kmai007 must be for 3.3.1 release
13:48 kmai007 http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Rebalancing_Volume_to_Migrate_Existing_Data
13:48 glusterbot Title: Gluster 3.2: Rebalancing Volume to Migrate Existing Data - GlusterDocumentation (at www.gluster.org)
13:49 tqrst kmai007: rebalance used to be split in fix-layout and migrate-data - now it just does both
13:49 kmai007 thanks for the clarification
13:49 kmai007 it completed successfully from lastnight
13:50 kmai007 but it's reporting is leaves more to be desired
13:50 tqrst I might actually get to finish rebalancing this time around - killed ~80k folders and 20+ million files to bring things down to something a bit more manageable
13:50 tqrst er, 10+
13:51 kmai007 i do not envy you
13:51 kmai007 so daily, it has become more of a headache for my web content developers
13:51 kmai007 they get a 403 permission denied error, when they make a modification to a file served by gluster
13:52 kmai007 [2013-11-25 17:20:24.338351] W [fuse-bridge.c:705:fuse_attr_cbk] 0-glusterfs-fuse: 42913615: STAT() /valvora/htdocs/index.html => -1 (Structure needs cleaning)
13:52 kmai007 i've checked over and over, and its not the underlying xfs bricks
13:52 bala joined #gluster
13:52 kmai007 to fix i'd do an 'ls -l' of the directory where the file lives, and then the cleaning error goes away
13:53 kmai007 and the page is served by apache
13:53 kmai007 i've seen several emails about this behavior but none, with a long term solution
13:53 kmai007 i've found that if you mount up the filesystem to the clients as NFS, the issue goes away
13:54 mkzero kmai007: how do they modify the files? do they change the content or do the recreate the files? so does the inode number change on the client?
13:54 kmai007 they would edit the file in a text editor
13:54 kmai007 save,
13:54 kmai007 then hit F5 to see their changes in the browser
13:55 kmai007 some times they get a 403 error, and i would find in the client fuse logs that the structure needs cleaning
13:55 mkzero kmai007: not really helpful, editors sometimes have different behaviours when saving a file ;) with what options are the clients mounted?
13:55 kmai007 i've straced a dirty file
13:55 kmai007 how to I share a paste ?
13:55 kmai007 instead of pasting it on here glusterbot
13:57 mkzero tqrst: how long does a rebalance with that many files take you?
13:57 mkzero kmai007: pastie.org or something like that?
13:57 gmcwhistler joined #gluster
13:59 kkeithley ,,(paste)
13:59 glusterbot For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
13:59 tqrst mkzero: I can't remember managing to rebalance this volume without either segfaults or excessive memory usage, so I don't know
13:59 tqrst mkzero: more than a week
14:00 tqrst should be more reasonable now that I've cleaned things up
14:00 _Bryan_ joined #gluster
14:02 mkzero tqrst: wow.. i hope when i hit the limits of my current small cluster that works better.. no idea how many files are on that cluster atm but it should be near your numbers.. just with more folders
14:03 kmai007 http://ur1.ca/g69qt
14:03 glusterbot Title: #60786 Fedora Project Pastebin (at ur1.ca)
14:04 FooBar hmm... I just set my gluster performance.cache-size to 10GB .... and then I couldn't mount the volume again on the client
14:04 FooBar [2013-12-11 14:02:10.596481] E [quick-read.c:827:check_cache_size_ok] 0-gv0-quick-read: Cache size 10737418240 is greater than the max size of 4017340416
14:04 FooBar so I set it to 4017340416 now...
14:04 FooBar seems to work again
14:04 FooBar but weird that I can configure it to illegal values
14:05 kkeithley file a ,,(bug)
14:05 glusterbot I do not know about 'bug', but I do know about these similar topics: 'fileabug'
14:05 kkeithley ,,(fileabug)
14:05 glusterbot Please file a bug at http://goo.gl/UUuCq
14:05 FooBar will do
14:05 tqrst mkzero: I get the impression that a lot of the problems we've encountered are leftovers from when we were running 3.2 a long time ago
14:07 kmai007 here is an strace i did on a file that needed structure needs cleaning, and how it became clean again.
14:07 kmai007 http://ur1.ca/g69rb
14:07 glusterbot Title: #60787 Fedora Project Pastebin (at ur1.ca)
14:08 mbukatov joined #gluster
14:10 hagarth joined #gluster
14:10 kmai007 http://ur1.ca/g69rv cleaned structure
14:10 glusterbot Title: #60792 Fedora Project Pastebin (at ur1.ca)
14:15 vpshastry1 joined #gluster
14:15 FooBar https://bugzilla.redhat.com/show_bug.cgi?id=1040494 <-- cache-size limit
14:15 glusterbot Bug 1040494: medium, unspecified, ---, csaba, NEW , Volume unmountable when performance.cache-size is set over 4GB
14:17 kmai007 @FooBar, can you help me understand what performance.cache-size is for?  I was going through the features and I haven't enabled anything yet in my environment
14:31 sroy joined #gluster
14:33 bala joined #gluster
14:36 glusterbot New news from newglusterbugs: [Bug 1040494] Volume unmountable when performance.cache-size is set over 4GB <https://bugzilla.redhat.com/show_bug.cgi?id=1040494>
14:40 aravindavk joined #gluster
14:56 ira joined #gluster
14:59 zaitcev joined #gluster
15:00 bugs_ joined #gluster
15:00 hagarth gluster community meeting starting in #gluster-meeting
15:01 krishnan_p joined #gluster
15:01 vpshastry1 left #gluster
15:02 Technicool joined #gluster
15:03 japuzzo joined #gluster
15:04 krishnan_p left #gluster
15:04 krishnan_p joined #gluster
15:04 shyam joined #gluster
15:05 badone joined #gluster
15:12 andreask how does the self-heal daemon avoid to crawl the complete fs-tree?
15:16 jzb joined #gluster
15:16 dbruhn joined #gluster
15:17 pk joined #gluster
15:18 shylesh joined #gluster
15:18 andreask joined #gluster
15:20 dhyan joined #gluster
15:23 sarkis_ joined #gluster
15:25 hagarth joined #gluster
15:25 wushudoin joined #gluster
15:28 kdhananjay joined #gluster
15:29 jag3773 joined #gluster
15:31 hagarth joined #gluster
15:32 TvL2386 joined #gluster
15:35 dewey joined #gluster
15:40 psyl0n joined #gluster
15:47 hagarth1 joined #gluster
15:58 jskinner_ joined #gluster
16:00 mkzero joined #gluster
16:07 DV joined #gluster
16:07 lalatenduM joined #gluster
16:08 zerick joined #gluster
16:14 zerick joined #gluster
16:14 kaptk2 joined #gluster
16:21 kmai007 i just completed my first successful removal of a bad file/its .glusterfs counter part
16:22 kmai007 i just wish i knew how it got in that state to begin with
16:22 dbruhn split-brain?
16:22 kmai007 how reliable is the split-brain status cmd ?
16:23 kmai007 when i ran it, it said, 0 files split-brain
16:23 kmai007 only reason i knew about it was there were these links to some while path , that shouldn't exist from the the client
16:23 dbruhn Most of the time, I end up finding split-brain issues out when my application stack tries to access the file in question. I get an Input/Output error
16:24 kmai007 correct
16:24 kmai007 a user discovered it b/c they could not delete the links nor the directory that stores them
16:24 kmai007 it was all out of wack on the gluster bricks too
16:25 kmai007 is there a how-to-doc to read on how to catch them all?
16:25 kmai007 besides relying on the volume heal cmd ?
16:26 dbruhn Sounds like you didn't have a split-brain issue, but something else
16:26 geewiz joined #gluster
16:27 kmai007 the client log showed [2013-12-11 13:15:47.829804] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-devstatic-replicate-1: /employees/htdocs/emp/ec/devper/craft/track/courses/esrue/firefox3: Skipping entry self-heal because of gfid absence [2013-12-11 13:15:47.848345] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-devstatic-replicate-1: background  entry self-heal failed on /employees/htdocs/emp/ec/dev
16:30 dbruhn agh ok, looks like the .glsuterfs entry was screwed up
16:30 dbruhn yeah, that wouldn't show up in a split brain
16:30 kmai007 i'm not sure how it got that way either
16:30 dbruhn what you could do is grep your logs for part of that error and see if anything else is showing up
16:31 psyl0n joined #gluster
16:31 dbruhn the file could have been corrupted on disk
16:31 kmai007 yeh I don't see any raid errors, but i'll check again
16:31 semiosis marcoceppi: ping
16:32 marcoceppi semiosis: pong
16:32 semiosis hi!
16:32 glusterbot semiosis: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:32 semiosis d'oh!
16:32 semiosis marcoceppi: johnmark just brought this to my attention: https://launchpad.net/ubuntu/trusty/+source/glusterfs/3.4.1-1ubuntu1
16:32 glusterbot Title: “glusterfs” source package : Trusty (14.04) : Ubuntu (at launchpad.net)
16:33 semiosis marcoceppi: i'm concerned that the upstart jobs included in the merge are not the latest & greatest -- specifically the one to block mounting
16:33 marcoceppi semiosis: did you verify this, or are you speculating?
16:34 semiosis marcoceppi: verified
16:34 marcoceppi semiosis: okay, so you want to patch the debian file to include better upstart scripts?
16:35 semiosis http://paste.ubuntu.com/6556797/
16:35 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
16:36 semiosis i suppose the merge used the upstart jobs that are in the glusterfs source tree... i will update those
16:37 marcoceppi semiosis: that'd be the best way to get them in
16:37 semiosis will do, thanks
16:37 marcoceppi semiosis: I'll ping some package experts to see if there will be another sync, if not (at least for trusty) get instructions on how you can submit a patch to universe
16:39 semiosis cool
16:41 dhyan joined #gluster
16:43 marcoceppi semiosis: so if you get the updated source tree in to debian before package freeze (with the updated upstart) we can just merge them in
16:43 semiosis i was hoping you'd say that :)
16:44 marcoceppi semiosis: if not, you'll want to open a merge request against the gluster packaging branch in lp
16:44 marcoceppi semiosis: also, ping jamespage in #ubuntu-server if  you need any help regarding either of those actions
16:44 vpshastry joined #gluster
16:44 semiosis excellent, thanks!
16:44 marcoceppi semiosis: np! Thanks for the heads up!
16:45 semiosis yw
16:45 LoudNoises joined #gluster
16:49 thogue joined #gluster
16:50 badone joined #gluster
16:52 semiosis marcoceppi: before i do that i'd like to get a review of the new upstart job, should I ask jamespage for a review?  or someone else?
16:53 marcoceppi semiosis: jamespage is also a kung-fu upstart master, you can also check in #upstart
16:54 semiosis great
16:54 semiosis yeah last time i went to #upstart for help with this they sent me to ubuntu people
16:54 semiosis and i ended up getting help from spamaps
16:54 semiosis who was super helpful
16:55 semiosis that was ~2 years ago though
16:55 marcoceppi semiosis: spamaps is still around, he worked on upstart when he was with Canonical, though he's not Canonical anymore, I'm sure he'd be happy to help
17:00 vpshastry left #gluster
17:05 jbd1 joined #gluster
17:16 psyl0n joined #gluster
17:16 samppah @ports
17:16 glusterbot samppah: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
17:16 keytab joined #gluster
17:29 JordanHackworth joined #gluster
17:30 Alpinist joined #gluster
17:35 ndk joined #gluster
17:36 johnbot11 joined #gluster
17:37 johnbot11 joined #gluster
17:41 bennyturns joined #gluster
17:41 Mo__ joined #gluster
17:43 badone joined #gluster
17:51 kmai007 is there a better time to run a volume rebalance fix-layout in 3.4.1 ?  I dont want my clients to cry while i'm at work.....
17:51 kmai007 and how often should it be ran?
17:55 kmai007 also when the gluster vol file is updated, the clients appear to be notified, but where on the client is it stored?  Is it just running in memory?
17:56 bulde joined #gluster
17:58 johnbot1_ joined #gluster
18:02 JordanHackworth joined #gluster
18:04 dusmant joined #gluster
18:08 Remco kmai007: Clients fetch the volfile from one of the nodes, then use that. So I'd say it's just in memory
18:08 Remco Don't know about the other questions though
18:11 semiosis kmai007: you only need to do a rebalance after adding bricks
18:17 badone joined #gluster
18:17 thogue joined #gluster
18:36 aliguori joined #gluster
18:37 SFLimey joined #gluster
18:39 neofob left #gluster
18:39 SFLimey I'm looking for some help configuring AWS ELB, namely how to implement health checks to see if the gluster node is responding and what to set the sticky sessions to. I've goggled and chased my tail for a couple of days but not finding anything. Anyone have any experience?
18:40 chirino_m joined #gluster
18:43 semiosis SFLimey: what problem are you trying to solve?  i doubt ELB is the right solution.
18:45 SFLimey I'd like to have multi zone gluster nodes but not have to have my clients point to a specific node but point to a pool A record/load balancer.
18:45 semiosis using fuse or nfs clients?
18:48 SFLimey I'm open to suggestions, currently we use nfs on a older rev of gluster but we're building out a fresh test environment with CentOS and Ubuntu boxes.
18:49 semiosis what's your use case/workload?
18:49 semiosis i'd usually recommend fuse clients.  see ,,(mount server)
18:49 glusterbot The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
18:49 semiosis and ,,(rrdns)
18:49 glusterbot You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
18:50 semiosis so you dont need any load balancer with fuse clients, and if you want the mount server address to be more available, use rrdns
18:52 SFLimey At some point this will scale to be multi region, the load isn't huge but uptime has to be guaranteed. 99.999% isn't quite good enough apparently. ;)
18:53 semiosis currently glusterfs geo-replication is one way, async, so the slave in another region is effectively read only
18:54 semiosis and regular replication is latency sensitive, so you probably wouldn't be happy with the performance running that between regions
18:57 SFLimey Yeah, I did see that although that's down the line six months or so. Maybe Gluster will be in a different place at that point. So silly question, so to use fuse I just install the glusterfs-fuse binary on each client that needs access to the Gluster cluster?
18:58 semiosis @latest
18:58 glusterbot semiosis: The latest version is available at http://download.gluster.org/pub/gluster/glusterfs/LATEST/ . There is a .repo file for yum or see @ppa for ubuntu.
18:59 SFLimey I've been living on Semiosis ppa for the last day. Thanks btw.
18:59 semiosis hmm, i suppose so, i'm not up to speed on exactly what is in all those rpms
18:59 semiosis yw
18:59 semiosis for the ubuntu packages just install glusterfs-client
19:00 SFLimey Last question, I really appreciate the help btw, why would you use fuse over nfs?
19:00 andreask joined #gluster
19:04 semiosis fuse client does HA automatically (when connecting to a replicated volume).  also, since it connects directly to all bricks, it is better for a highly parallel workload
19:04 semiosis since requests do not need to be proxied through a single host like with nfs
19:06 SFLimey Oh gosh, okay that pretty huge for use. Thanks again Semiosis, you've been a huge help!
19:06 semiosis yw, any time
19:11 japuzzo joined #gluster
19:11 semiosis SFLimey: i've been running gluster in ec2 for a couple years feel free to ping me if you have trouble
19:27 Paul-C joined #gluster
19:37 andreask joined #gluster
19:46 DV joined #gluster
19:48 chirino joined #gluster
20:12 kmai007 would this be caused by concurrent use to a file by 2 different clients?
20:12 kmai007 W [fuse-bridge.c:705:fuse_attr_cbk] 0-glusterfs-fuse: 7235433: STAT() /headsup/htdocs/ddate/tla_20131206.cfm => -1 (Structure needs cleaning)
20:12 kmai007 1 server is listing it fine, while the other is listing it as ls: cannot access tla_20131206.cfm: Structure needs cleaning
20:15 kmai007 4 node distr./repl setup
20:15 kmai007 the xfs filesystem on the bricks are fine and not displaying errors
20:17 facemangus joined #gluster
20:17 facemangus Quick question if anybody is awake and alive, concerning v3.2.7: How do you handle split brains before 3.3? Everything I see is for post 3.3
20:18 facemangus Without heal, how do you even determine the brick in conflict...
20:20 semiosis facemangus: client log files & ,,(extended attributes)
20:20 glusterbot facemangus: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
20:26 facemangus I inherited a production gluster replication network about a month ago, apparently the rest of the staff wasn't taught to not cause splitbrains
20:26 facemangus Great thing to have to learn in a production environment >:(
20:26 semiosis how are the split brains being caused?  that shouldn't normally happen
20:27 semiosis usually causes are network partitions or alternating server failures
20:28 facemangus It seems he set them up so that both servers have a /shared directory, and then it populates /var/www/html. If they modify /var/www/html it is causing them.
20:28 facemangus its just 2 bricks :\
20:28 semiosis hmmm
20:29 facemangus also I have been unable to get any information from getfattr, 1is this going to throw stuff into the log or should I expect output directly? Still reading the document linked.
20:30 semiosis getfattr -m .  -d -e hex {filename}
20:30 semiosis that will output the xattrs for a file
20:30 facemangus I am receiving nothing :\
20:30 semiosis have to run that on the brick, not through a client mount
20:31 mattappe_ joined #gluster
20:32 daMaestro joined #gluster
20:32 facemangus Der, now I am getting somewhere
20:33 facemangus and they are identical...
20:34 sroy joined #gluster
20:34 semiosis all zeros means file is in sync on all replicas
20:35 semiosis non-zero means that brick has changes that are newer than the other brick
20:35 semiosis it is ahead
20:35 facemangus odd
20:35 facemangus it seems it healed itself
20:35 semiosis if two bricks have non zero, thats split brain
20:35 facemangus :|
20:35 facemangus it was solid zeros for me, and I can cat the file now
20:35 facemangus but the logs do not indicate a heal
20:35 semiosis heh
20:35 facemangus *sigh* this work day has not had enough coffee in it
20:36 facemangus spoke too soon
20:36 facemangus wow
20:36 facemangus that was amazing
20:37 dbruhn ?
20:37 dbruhn what was amazing?
20:37 facemangus It healed itself, but logs are reporting healing as failed
20:37 facemangus haha
20:37 JoeJulian Probably because another process completed the heal so the client couldn't get a lock and failed the heal.
20:38 JoeJulian Probably means that glustershd healed it on one of the servers.
20:38 semiosis howdy stranger
20:38 JoeJulian Hey there...
20:38 JoeJulian Been overloaded lately... :/
20:38 facemangus Well thank you very much for the insight so far, I haven't had a chance to read up on our gluster clusters yet as the amount of stuff I am supporting is too much for one person.
20:38 semiosis clearly
20:38 JoeJulian Still am, somewhat, but it's getting there.
20:39 JoeJulian facemangus: +1
20:39 facemangus Waiting on an assistant to do tickets while I research our more advanced implementations (if heartbeat breaks I am screwed)
20:40 andreask joined #gluster
20:41 morse joined #gluster
20:56 facemangus semiosis: need some brief advice so I can make a ruling in the office.
20:56 semiosis you've come to the right place lol
20:57 facemangus if the brick /var/www/html (shows up in df as servername:/www mounted to /shared) and then public_html under the user symlinks to /shared/html
20:57 facemangus where should changes be made
20:57 facemangus that will not cause brain spluts
20:57 psyl0n joined #gluster
20:57 facemangus Networking issues aside
20:57 dbruhn You should only be making changes in from the mount point
20:57 dbruhn for normal operations
20:57 dbruhn if you are accessing the bricks it's due to repairing issues
20:58 facemangus so for here, if a developer needs to make a surgical change to a file, that mount point would be /shared?
20:58 dbruhn what is the output of df on your servers?
20:59 facemangus It has some privileged information, but otherwise the relevant portion is that servername:/www 916G 30G 840G 4% /shared
20:59 dbruhn So the gluster share is mounted to /shared
20:59 facemangus servername is in /etc/hosts with the private IP etc (its a functioning setup)
20:59 facemangus right
21:00 semiosis can you sanitize & pastie the output of 'gluster volume info www'
21:00 dbruhn they shouldn't be changing any files except for through /shared
21:00 facemangus what is the ban send-q here? its 8 lines and my irssi doesn't ratelimit
21:00 facemangus can pastebin if needed
21:00 semiosis dbruhn: they might not be, split brain can happen in this situation even if all acceess goes through client
21:01 semiosis facemangus: please use a ,,(paste) site
21:01 glusterbot facemangus: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
21:01 facemangus http://pastebin.ca/2496349
21:01 glusterbot Title: pastebin - Untitled - post number 2496349 (at pastebin.ca)
21:02 facemangus thanks semiosis, gonna note that for later
21:02 semiosis facemangus: i suppose you have a web server reading directly from /var/www/html?  hope it doesnt write
21:02 semiosis idk how the writes happen to this volume, but the two usual causes of split brain are....
21:02 facemangus webserver doesn't write anything (Except in one case that i am having the developers correct, stupid selfupdate didn't account for gluster)
21:04 semiosis 1. network partitions... consider replica 2 where servers are also clients (your case) if the server/client machines are disconnected from each other, and both modify the same file on their client mount, then that file will be split brained
21:04 facemangus I assumed the /var/www/html is off limits, just trying to make sure I get clarification before I send out a company wide mandate/warning
21:04 dbruhn semiosis, I realize split-brain can happen even through the client, but it is guaranteed to happen if you are writing to the bricks directly
21:04 semiosis dbruhn: not quite
21:04 facemangus So if they disconnect for just a second and someone happens to :wq in vim just then...
21:04 dbruhn really?
21:04 semiosis dbruhn: try it yourself
21:05 semiosis facemangus: has to be writes to the same file on both servers, while the servers are not connected to each other
21:05 facemangus well thats complicated
21:05 facemangus Is it possible to set up a master that would populate the others with authority?
21:06 semiosis facemangus: you might be able to set one of the systems' client mount points to be read only
21:06 facemangus only receive changes from gluster then?
21:06 semiosis i don't always mount my gluster volumes locally, but when i do, i prefer a read-only mount
21:06 facemangus heh
21:06 dbruhn I'll have to a bit later, i've always been under the impression editing a file from the brick level would leave it in a split-brain state. Interesting bit of info.
21:07 semiosis dbruhn: split-brain is a state of the xattrs, when you write to a brick directly, you (usually) are not updating xattrs
21:08 facemangus The set up seems to work wonderfully otherwise, just that this particular cluster of servers is receiving a lot of changes right now. Both webservers sync static content over gluster
21:08 dbruhn That makes sense, didn't know that bit
21:08 badone joined #gluster
21:10 facemangus I also read that reading from socket failed. errors were fixed in 3.3?
21:10 facemangus the log spam
21:10 semiosis facemangus: i suspect you have networking issues between these machines
21:10 semiosis that would explain the split brains
21:10 facemangus yes, indeed it would
21:11 facemangus seems like it is getting minor disconnects every 3 seconds on the dot...
21:11 semiosis although doesnt explain how split-brains could have been healed... by definition split brain means it can't be automatically healed
21:12 semiosis facemangus: is some monitor connecting to the gluster port every 3 sec to check if the service is up?
21:13 facemangus Not that I am aware of, buuuuuut who knows what the previous admin configured. (He actually was very good, left this company for microsoft)
21:13 facemangus the socket reading fails are coming from localhost on varied ports in the 1000 range, could be related to the other disconnected errors short of a second here and there variance
21:14 semiosis i'm guessing that's in the etc-glusterfs-glusterd.log?
21:15 semiosis what about client logs?  any sign of connectivity issues in them?  and brick logs?  on either machine?
21:15 facemangus etc-*-glusterfs shows the 0-client: disconnected, etc-*-glusterd shows the more verbose socket.management error
21:15 facemangus I haven't seen anything
21:16 facemangus we have a postgres cluster on the same private network operating on the two servers that don't error either
21:16 facemangus *replicating cluster
21:16 facemangus brick log is empty entirely haha
21:34 markr_ joined #gluster
21:38 kmai007 guys i'm still unsure how files get to this state: http://ur1.ca/g6bnv
21:38 glusterbot Title: #60974 Fedora Project Pastebin (at ur1.ca)
21:39 kmai007 structure needs cleaning, when i can stat the directory of the file and it fixes it
21:39 kmai007 nothing on the gluster nodes
21:39 CLDSupportSystem joined #gluster
21:39 kmai007 indicates xfs errors
21:39 kmai007 err...bricks
21:40 CLDSupportSystem Hello, I am trying now for a week, on and off to setup a standalone glusterfs server and client. Standalone for now. I can't get the nfs to work...
21:41 CLDSupportSystem sudo mount -t nfs -o vers=3 localhost:/sharevol ~/gluster_mount/
21:41 CLDSupportSystem mount.nfs: access denied by server while mounting localhost:/sharevol
21:41 CLDSupportSystem I think I am close, I get this as above
21:41 facemangus ENONET, my cursed mistress
21:41 mattapp__ joined #gluster
21:41 CLDSupportSystem so I know something is working here but its a permissions issue
21:41 CLDSupportSystem Volume Name: sharevol
21:41 CLDSupportSystem Type: Distribute
21:41 CLDSupportSystem Status: Started
21:41 CLDSupportSystem Number of Bricks: 1
21:41 CLDSupportSystem Transport-type: tcp
21:41 CLDSupportSystem Bricks:
21:42 CLDSupportSystem Brick1: ec2-23-21-245-172.compute-1.amazonaws.com:/gfsd_data
21:42 CLDSupportSystem Options Reconfigured:
21:42 CLDSupportSystem nfs.rpc-auth-allow: localhost
21:42 CLDSupportSystem nfs.addr-namelookup: off
21:42 CLDSupportSystem nfs.register-with-portmap: on
21:42 CLDSupportSystem nfs.disable: off
21:42 CLDSupportSystem that is my volume info
21:42 facemangus whoa slow down my friend
21:42 CLDSupportSystem I setup all firewall rules , sure thing
21:43 kmai007 did you try mount -v  to see verbosely?
21:43 mattap___ joined #gluster
21:43 CLDSupportSystem the error is not much more detailed if i do
21:43 CLDSupportSystem mount.nfs: mount(2): Permission denied, mount.nfs: access denied by server while mounting localhost:/sharevol
21:44 dbruhn What distro?
21:44 CLDSupportSystem ubuntu
21:44 kmai007 nfs.disable: off  <---- could that be your problem?
21:44 kmai007 what
21:44 kmai007 i read it wrong
21:44 CLDSupportSystem umm well thats what it said to try in the gluster docs
21:44 kmai007 LOL
21:44 CLDSupportSystem yeh, double negative lol
21:45 dbruhn Is the OS provided NFS server disabled and off?
21:45 CLDSupportSystem yes it is, its not even installed
21:45 CLDSupportSystem i only installed nfs-common
21:45 CLDSupportSystem and portmapper is running
21:46 kmai007 i just did it
21:46 kmai007 and it worked
21:46 CLDSupportSystem what did you do?
21:46 kmai007 maybe 'localhost' = actual hostname ?
21:46 CLDSupportSystem i tried
21:46 kmai007 mount -t nfs <gluster_hostname>:/vol   /mnt/point
21:47 dbruhn whats "showmount -e localhost"
21:47 dbruhn show
21:47 kmai007 omg, i should have thought of that
21:47 CLDSupportSystem i tried ip which times out, using localhost or domain name gets the access denied error
21:47 kmai007 if the vols were started
21:47 CLDSupportSystem showmount -e localhost : Export list for localhost:
21:47 CLDSupportSystem /sharevol localhost
21:48 CLDSupportSystem yes started
21:48 kmai007 what do you have for RPC
21:48 kmai007 chkconfig --list|grep rpc rpcbind         0:off   1:off   2:on    3:on    4:on    5:on    6:off rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off
21:49 CLDSupportSystem sudo gluster volume start sharevol
21:49 CLDSupportSystem Volume sharevol already started
21:49 CLDSupportSystem portmapper
21:49 CLDSupportSystem i didn't use rpcbind, isn't portmapper same thing?
21:49 kmai007 i guess you're not getting this message
21:50 kmai007 mount.nfs: rpc.statd is not running but is required for remote locking.
21:50 CLDSupportSystem no im not getting that
21:50 kmai007 what does /var/log/messages say when you try to mount
21:52 kmai007 localhost:/sharevol ~/gluster_mount/                   <----- what if you tried another mount point?  like /mnt
21:52 CLDSupportSystem nothing reported in syslog
21:52 CLDSupportSystem doubt that would make a diff, but i'll try
21:52 dbruhn what does the /var/log/glusterfs/nfs.log show?
21:53 CLDSupportSystem gluster_mount belongs to me and is in my home dir
21:54 CLDSupportSystem i think i may have found the problem
21:54 CLDSupportSystem portmapper config, which i didn't see in the docs at all when setting up
21:54 CLDSupportSystem I didn't spot this in the mount -v output at first glance, but here it is, take al look
21:54 CLDSupportSystem mount.nfs: timeout set for Wed Dec 11 16:55:58 2013
21:54 CLDSupportSystem mount.nfs: trying text-based options 'vers=3,addr=127.0.0.1'
21:54 CLDSupportSystem mount.nfs: prog 100003, trying vers=3, prot=6
21:54 CLDSupportSystem mount.nfs: trying 127.0.0.1 prog 100003 vers 3 prot TCP port 38467
21:54 CLDSupportSystem mount.nfs: prog 100005, trying vers=3, prot=17
21:54 CLDSupportSystem mount.nfs: portmap query retrying: RPC: Program not registered
21:54 CLDSupportSystem mount.nfs: prog 100005, trying vers=3, prot=6
21:55 CLDSupportSystem mount.nfs: trying 127.0.0.1 prog 100005 vers 3 prot TCP port 38465
21:55 CLDSupportSystem mount.nfs: mount(2): Permission denied
21:55 CLDSupportSystem mount.nfs: access denied by server while mounting localhost:/sharevol
21:55 dbruhn you should use fpaste instead of flooding the room so you don't get kicked out by glusterbot
21:57 CLDSupportSystem sorr
21:57 CLDSupportSystem sorry im a noob lol
21:57 dbruhn No worries, just wanted to make sure you didn't get booted.
21:57 CLDSupportSystem haven't used IRC for like years, plus using some weird client on here
21:57 nueces joined #gluster
21:58 CLDSupportSystem in mountain lion.. heck i though IRC was dead :S
21:59 CLDSupportSystem mount.nfs: portmap query retrying: RPC: Program not registered <- so this tells me portmap is not setup to map nfs through to gluster-server correct?
21:59 dbruhn Welcome to the open source world, I am a mac user too. Using xchat here. IRC is still alive and well.
21:59 CLDSupportSystem ;)
22:00 JoeJulian kmai007: Yeah, when I last had the "structure needs cleaning" error, one of my bricks needed xfs_repair
22:00 CLDSupportSystem was looking for the gluster google group lol when i found this
22:00 CLDSupportSystem dbruhn, can you point me to a good resource to getting portmapper configured with gluster for nfs ?
22:02 CLDSupportSystem or maybe I should uninstall portmapper and use rpcbind, resource for that ?
22:03 kmai007 @JoeJulian while no errors were logged on the brick servers, would you try to run xfs_repairs on it?
22:03 JoeJulian kmai007:yep
22:03 dbruhn I honestly have no idea, I am using the FUSE client.
22:03 CLDSupportSystem hmm ok, i'll look into FUSE
22:04 kmai007 is the how-to-safely run xfs_repairs on your site?  I hate to lose data
22:04 JoeJulian CLDSupportSystem: Can you telnet to port 38465?
22:04 CLDSupportSystem i'll use native if that's the way to go, was hoping it would work nicely with nfs and no need for extras
22:04 dbruhn There are use cases for both
22:04 CLDSupportSystem @JoeJulian, yep one of the 1st things i tried
22:04 CLDSupportSystem checked all the listed ports in the doc
22:05 CLDSupportSystem just checked agian, it worked
22:06 JoeJulian CLDSupportSystem: I'd probably "gluster volume reset" all your options and try again.
22:06 CLDSupportSystem Thanks Joe, I will go from there, and do some more reading on port mappers, maybe use rpcbind instead
22:07 CLDSupportSystem Thanks guys, I'll be back tomorrow if resetting doesn't work
22:07 JoeJulian As an aside, nfs mounting from localhost isn't a good idea. Memory deadlocks have been observed in the kernel doing that.
22:07 semiosis CLDSupportSystem: you need to use tcp... see ,,(nfs)
22:07 glusterbot CLDSupportSystem: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
22:07 JoeJulian [13:55] <CLDSupportSystem> mount.nfs: trying 127.0.0.1 prog 100005 vers 3 prot TCP port 38465
22:08 semiosis eh
22:08 CLDSupportSystem i am on vers=3 and specify vers=3 as an option, if i dont' it times out
22:08 mattapp__ joined #gluster
22:08 CLDSupportSystem i can telnet into 34865
22:08 CLDSupportSystem * 38465
22:09 cogsu joined #gluster
22:09 JoeJulian Which suggests to me that EPERM must be coming from glusterfs. If resetting those options doesn't find the issue, perhaps increase the client logging to debug to find a clue.
22:10 semiosis CLDSupportSystem: you're just starting to use gluster?  i'd suggest cleaning it out & starting over.  it really should not be this complicated.  all the defaults should "just work" with ubuntu nfs clients, as long as you use mount -t nfs -o tcp,vers=3
22:10 semiosis oh well
22:18 fyxim joined #gluster
22:20 gdubreui joined #gluster
22:20 kmai007 JoeJulian
22:21 kmai007 do you recall how long the xfs_repair took on the brick, and what was the size of your brick?
22:21 JoeJulian Sorry, it's been a year.
22:21 kmai007 glad to see its been smooth for a year
22:22 jskinner_ joined #gluster
22:27 facemangus joined #gluster
22:27 facemangus Ugh, semiosis it returned
22:28 facemangus Gluster doesn't see it as split-brained (0=0x000000000000000000000000), but error reporting does
22:28 facemangus fml
22:29 leblaaanc joined #gluster
22:31 leblaaanc Hey guys. I'm curious. I'd like to setup a replicated volume with two nodes. How does glusterfs know which is the "master" ?
22:31 leblaaanc Or more specifically can I preemptively copy the brick data before creating the volume?
22:33 kmai007 oh here is a strange incident
22:33 kmai007 so i've updated a feature option on the gluster CLI
22:34 kmai007 as I tail a fuse-client vol.log
22:36 kmai007 i see [2013-12-11 02:48:35.537992] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-11 02:48:35.569749] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-11 02:48:35.598306] I [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
22:37 kmai007 does that mean, it received the feature change, knows about it, and say there is no MORE changes, continuing?
22:39 davidbierce joined #gluster
22:39 JoeJulian leblaaanc: There is no master. There is an undocumented reality that the left-hand brick in a replica volume, if pre-loaded with data, will replicate.
22:40 JoeJulian kmai007: Sounds like a fair interpretation.
22:40 rotbeard joined #gluster
22:41 leblaaanc left hand?
22:41 facemangus oh wow
22:41 facemangus so
22:41 facemangus it is split brained
22:41 facemangus but if I move the split file on one
22:41 facemangus or remove it
22:41 facemangus it copies that change to the other brick
22:41 facemangus instead of healing it
22:42 JoeJulian leblaaanc: When you're creating the volume, ie. "gluster volume create myvol replica 2 server1:/brick server2:/brick" server1 is on the left of the replica pair.
22:43 JoeJulian ~split-brain | facemangus
22:43 glusterbot facemangus: To heal split-brain in 3.3+, see http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/ .
22:43 facemangus 3.2.7
22:43 facemangus believe me, I wish it was 3.3
22:43 JoeJulian @meh
22:43 glusterbot JoeJulian: I'm not happy about it either
22:43 leblaaanc JoeJulian: should I delete the .glusterfs info prior to recreating the volume?
22:44 JoeJulian leblaaanc: Ah, you didn't say this was already a brick from a volume. Now there's xattrs to deal with too... :/
22:45 facemangus oh JoeJulian I actually had already read yours, heh. I wish...
22:45 JoeJulian Yes, I'd probably do the steps from the following link, path or prefix
22:45 JoeJulian hmm
22:45 semiosis path or prefix is already
22:45 semiosis path or prefix is already part of a volume
22:45 JoeJulian a path or prefix is already part
22:45 semiosis path or prefix of it is already part of a volume
22:45 * JoeJulian beats glusterbot
22:45 semiosis worst game ever
22:46 semiosis let's play "guess that regex!"
22:46 leblaaanc o.O
22:46 semiosis path or a prefix
22:47 leblaaanc JoeJulian: well, I have a brick I can empty and copy everything to omitting the xattrs? then use that one as the "left-hand" ?
22:48 JoeJulian path or prefix of it is already part
22:48 semiosis path or a prefix of it
22:48 glusterbot semiosis: To clear that error, follow the instructions at http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/ or see this bug https://bugzilla.redhat.com/show_bug.cgi?id=877522
22:48 * semiosis wins
22:48 facemangus ugh, how can a file be in sync and split-brained at the same time
22:49 JoeJulian They can be different sizes, times, owners...
22:49 JoeJulian have different gfid
22:49 facemangus I mean, getfattr shows no issues
22:49 facemangus yet I/O error
22:50 facemangus and if i remove one it will remove the other...
22:50 facemangus shouldn't that not happen
22:50 semiosis remove it through the client or on the brick?
22:50 JoeJulian You might be encountering that bug where even though you've healed a split-brain, the client needs to be remounted to accept it...
22:50 facemangus client
22:50 facemangus oh?
22:50 JoeJulian yeah, fixed a long time ago
22:51 MrNaviPacho joined #gluster
22:51 facemangus I have been meaning to update to 3.3.+ but production environment, etc
22:51 JoeJulian Not sure if it's in 3.2.7 though
22:51 JoeJulian I'd personally jump straight to 3.4
22:51 facemangus are there any conflicts?
22:52 JoeJulian Besides 3.2 being rpc incompatible with 3.4, no.
22:52 calum_ joined #gluster
22:52 facemangus tried remounting, still failed to open as split brain seen, returning EIO
22:52 facemangus *sigh* why is it always at the end of the workday
22:53 JoeJulian Pfft... it's not Friday. You're not that bad off... ;)
22:53 facemangus I mean I am working until it is fixed
22:53 facemangus and it killed my .svn/entires too so I can't even re-co the file :D
22:53 facemangus wunderbar
22:54 JoeJulian facemangus: fpaste me the "getfattr -m . -d -e hex" and a stat of the files on the bricks and the error message from the log.
22:54 andreask joined #gluster
22:56 facemangus mind if I PM it to you so I don't have to redact everthing?
22:58 facemangus JoeJulian: ^
22:59 JoeJulian facemangus: sure
23:02 facemangus done
23:08 leblaaanc joined #gluster
23:12 leblaaanc woops sorry back
23:28 mattappe_ joined #gluster
23:28 facemangus thank you so much #gluster, in general, and thank you 4 fold to JoeJulian
23:30 JoeJulian @yum repo
23:30 glusterbot JoeJulian: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
23:30 facemangus noted for tomorrow
23:48 gdubreui joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary