Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-26

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 social_ y4m4: thanks, seems to have worked fine
00:12 y4m4 partner: cool at this point you would see the files being distributed properly.
00:12 y4m4 partner: that is the idea :-)
00:13 y4m4 social_: perfect
00:15 partner y4m4: that is not really the problem but the open files left by the rebalance which keeps growing.. plus the disk utilization still being 100%
00:17 partner y4m4: nevertheless, thank you for pointing out the bug on percentage, once i get this one fixed That should apply and get files written to new brick and i can continue my holiday :)
00:19 partner ok, i'm off now, i'm just way too tired to do anything more, thank you JoeJulian and y4m4 for your help, i will try to figure this out somehow tomorrow
00:19 mooperd joined #gluster
00:23 glusterbot New news from newglusterbugs: [Bug 978082] auxiliary group permissions fail via kerberized nfs export <http://goo.gl/pXiZz>
00:28 social_ y4m4: seems I hit some bug in 3.3.1 http://paste.fedoraproject.org/20903/22064621/
00:28 glusterbot Title: #20903 Fedora Project Pastebin (at paste.fedoraproject.org)
00:28 social_ y4m4: to me this looks like overflow of split-brain messages :/
00:29 sprachgenerator joined #gluster
00:29 rcoup hey folks, I can deal with normal split-brain files easily, but I have a LOT (500) that are just gfids
00:30 rcoup I can grep the entire filesystem to find the file with the right xattr, but as you can imagine that takes forever^2
00:30 rcoup any tips?
00:30 portante joined #gluster
00:30 social_ y4m4: from graph I guess someone let gluster to have split brain for a week and it crashed \o/ it hit 4096+ messages
00:32 y4m4 social_: is this a client side message?
00:32 y4m4 social_: like a crash?
00:33 social_ y4m4: no server side self-heal daemon
00:34 y4m4 social_: ah! ok
00:35 social_ yep I can see it it hit 4k+ split brain and crashed
00:36 rcoup I can find the gfid in .glusterfs on the bricks, but there's no link back to the path from there (afaict)
00:36 JoeJulian rcoup: Find the gfid file (.glusterfs/01/23/0123456...etc) and stat that file. If it has only 1 link ("Links: 1") then it should be safe to delete it.
00:37 rcoup k, one example says links:2
00:37 rcoup (random one of many)
00:37 rcoup hopefully that'll prune my list a bit though
00:38 JoeJulian If "Links: 2" you can get the inode and "find -inum $inode_number" on the brick to find it.
00:38 JoeJulian (or more than 2)
00:40 y4m4 rcoup: better is to find out the "ls -id <path_to_gfid_in_glusterfs> | awk {'print $1'}"
00:40 y4m4 rcoup: then what JoeJulian said!
00:40 rcoup JoeJulian: find basically traverses the entire filesystem though?
00:40 bala joined #gluster
00:41 JoeJulian y4m4: He's already got the inode number from the stat. :P
00:41 y4m4 social_: it could be a fixed bug - you should file a bug report
00:41 glusterbot http://goo.gl/UUuCq
00:41 JoeJulian ... and why weren't you at summit? :P
00:41 kevein joined #gluster
00:42 y4m4 JoeJulian: no conferences for me :-)
00:42 rcoup hmm, wonder if I can do *one* walk for all inode #s rather than 500x
00:43 rcoup (seems walk is inevitable)
00:46 JoeJulian find -type f -printf "%i %P\n"
00:47 JoeJulian (find does fewer system calls than ls -idR)
00:49 jag3773 joined #gluster
01:14 mooperd joined #gluster
01:21 y4m4 joined #gluster
01:21 semiosis chirino: i don't suppose you're around at this late hour, but if you are, i'm stumped by this:
01:21 semiosis [ERROR] Failed to execute goal org.fusesource.hawtjni:mave​n-hawtjni-plugin:1.8:build (default) on project glfsjni-linux32: build failed: org.apache.maven.plugin.MojoExecutionException: Add a '-Dnative-src-url=file:...' to have maven download the native package -> [Help 1]
01:22 semiosis i dont understand where that zip file comes from.  i think i should be providing it, but no idea how
01:22 JoeJulian @kick semiosis off topic ;)
01:22 glusterbot JoeJulian: Error: You don't have the #gluster,op capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
01:22 semiosis lmao
01:23 semiosis JoeJulian: chirino and I started a project today to make a libgfapi/java binding
01:23 semiosis so this is on topic
01:23 semiosis :D
01:23 JoeJulian I met several people that would probably get all hot and bothered about that...
01:23 semiosis +1
01:24 semiosis see pm
01:24 JoeJulian Careful, though, you might end up working at Red Hat.
01:25 lalatenduM joined #gluster
01:27 rcoup JoeJulian: sure, find vs ls for printing inodes, but if I want to take a list of 500x inodes and find the files/dirs they map to...
01:31 JoeJulian rcoup: You were inquiring about walking it just one time. I was simply suggesting that would be the most efficient. Pipe it somewhere and do your magic. Hell, maybe even pipe it through sort.
01:32 rcoup JoeJulian: hmm, I guess I could make a master index :) I think find -inum <i1> -o -inum <i2> -o -inum <i3> ...
01:32 rcoup might do what I want too
01:32 rcoup if I don't exceed the maximum command length
01:33 JoeJulian Right, but I think you would fairly quickly.
01:35 JoeJulian If you use that find command and use the printf to make json you could even pump those into elasticsearch to make your queries very fast...
01:43 sprachgenerator joined #gluster
01:44 y4m4 joined #gluster
01:45 badone joined #gluster
01:48 rcoup started. lets see how long this takes :)
01:51 rcoup related question. I'm assuming split-brain files would be a problem for geo-replication (did 75% of the volume, now just goes to 'faulty' pretty quickly). Then I discovered the SB files
02:00 semiosis sounds like a fair assumption.  once a file is determined to be split-brained glusterfs usually wont touch it
02:10 robo joined #gluster
02:12 harish joined #gluster
02:30 portante_ joined #gluster
02:56 bharata joined #gluster
03:02 bulde joined #gluster
03:22 rjoseph joined #gluster
03:23 robo joined #gluster
03:33 frakt joined #gluster
03:36 sprachgenerator joined #gluster
03:52 krishnan_p joined #gluster
03:53 bala joined #gluster
03:55 bala joined #gluster
03:56 bala joined #gluster
04:03 itisravi joined #gluster
04:08 sgowda joined #gluster
04:14 hagarth joined #gluster
04:17 CheRi joined #gluster
04:23 vpshastry joined #gluster
04:29 bulde joined #gluster
04:42 anands joined #gluster
04:56 aravindavk joined #gluster
05:13 JoeJulian Oy! Using 3.3 with fedora 19 is a pain.
05:14 jclift_ ?
05:14 jclift_ Doesn't F19 have 3.4 beta something in the yum repos?
05:15 JoeJulian glusterfs-api is depended on.
05:15 jclift_ Heh.. I guess that's why using 3.3 is a pain on F19
05:15 JoeJulian Yeah. Since I'm running 3.3 in production though....
05:16 jclift_ Makes sense.
05:16 JoeJulian I wonder if they got the rpc compatibility this time...
05:16 jclift_ Self compiled rpms?
05:16 rastar joined #gluster
05:16 * jclift_ hasn't tried out the compile instructions with 3.3
05:16 jclift_ Only tested with 3.4 and git master :/
05:16 jclift_ (yeah, lazy)
05:17 * JoeJulian is lazy too...
05:17 jclift_ :)
05:17 JoeJulian I just rpm -e --nodeps glusterfs-api
05:17 JoeJulian 'course now yum complains about missing dependencies for qemu.*
05:18 JoeJulian Maybe I'll just make a dummy rpm that claims to provide the library...
05:20 jclift_ It might just be easier to try compiling your own.  With 3.4/git master it's pretty quick
05:20 jclift_ (though, it does pollute the system by installing a bunch of dev tools)
05:22 JoeJulian Point is, I need to downgrade. I can downgrade to the http://download.gluster.org/pub/gl​uster/glusterfs/3.3/3.3.1/Fedora/ rpms, but qemu's going to complain about missing dependencies (3.3 doesn't have libgfapi)
05:22 glusterbot <http://goo.gl/NWfQP> (at download.gluster.org)
05:24 glusterbot New news from newglusterbugs: [Bug 978148] Attempting to mount distributed-replicate volume on RHEL 6.4 hangs in upstream 3.4.0 Beta 3 <http://goo.gl/oVBwo>
05:24 jclift_ Hmmm, found another new interesting bug too.  "showmount -e [gluster node]" is only showing 1 out of 5 of the NFS volumes
05:25 jclift_ Can mount the others (except the broken distribute-replica volume) fine using native client.  NFS server is showing up for all volumes in the gluster volume info too.
05:25 jclift_ New BZ time. :D
05:26 raghu joined #gluster
05:27 shireesh joined #gluster
05:32 sgowda joined #gluster
05:33 JoeJulian That's wierd. It doesn't even look like it's trying to connect to gluster2-2:/export/brick1/test4 in that bug.
05:34 anands joined #gluster
05:34 JoeJulian Regardless, though, being a replicated volume it shouldn't hang.
05:37 andreask joined #gluster
05:38 JoeJulian jclift_: Are you up early or late?
05:38 jclift_ JoeJulian: When it hangs there, it effectively kills any use of the other volumes too.  So, even if I mount (say) the "test5" volume perfectly fine after ^C the test4 volume... if I do a "df" then the command line hangs.
05:38 jclift_ 6:38am.  Up late.  Will probably hit the sack in a few hours.
05:38 JoeJulian yikes
05:38 jclift_ I kind of started late anyway.
05:39 jclift_ JoeJulian: Hmmm, when there are multiple volumes, it's expected they'd ALL be available through NFS at the same time yeah?
05:39 JoeJulian yeah
05:40 JoeJulian iirc, I've seen that before. They all can be mounted though, just don't show up in showmount
05:40 jclift_ Heh, except now they can't be mounted either. :(
05:40 jclift_ Me only tried showmount after the NFS was complaining the volumes don't exist
05:40 jclift_ Writing up BZ now
05:41 JoeJulian nifty
05:41 jclift_ I'm trying to be thorough with my testing. :)
05:41 jclift_ (and BZ's)
05:42 JoeJulian in 3.4, is there only 1 glusterfsd even if you have multiple bricks?
05:43 jclift_ Nope
05:44 jclift_ Just checked on the gluster1-2 box.  ps -ef |glusterfsd is showing 6 processes
05:44 jclift_ Hmmm, 5 volumes, 6 processes
05:44 jclift_ Two are for the test4 volume
05:45 JoeJulian That would probably be a bad thing.
05:45 jclift_ I wonder if that's due to the volume type (distributed-replica), or if that's some kind of bug that's causing the problem
05:45 jclift_ Interesting
05:45 jclift_ k, I'll copy the ps -ef output to the BZ
05:47 JoeJulian Another wierd thing. There's not two copies of everything in the log file. That makes me wonder if that's left-over from some previous test.
05:48 jclift_ The logs should be very clean
05:49 jclift_ After finding the problem, and seeing that it survives a reboot (ie not transient), I disabled gluster at boot, rebooted the boxes, wiped /var/log/glusterfs/* before starting gluster, then triggered the mount problem again then captured the logs
05:50 jclift_ JoeJulian: Just to point out, that distributed-replica volume is only on two servers.  I don't have enough boxes to spread it across 4 storage nodes
05:51 jclift_ So, the bricks are doubled up into separate dirs
05:51 jclift_ AFAIK though, that shouldn't cause issues
05:51 jclift_ (except maybe for non-optimal performance, etc)
05:52 JoeJulian jclift_: Right.
05:53 hagarth joined #gluster
05:53 JoeJulian That's interesting. Could you add /var/lib/glusterfs/vols for that server?
05:56 jclift_ Good idea.  I'll have to grab it right now though.  It's not as clean.
05:56 jclift_ But, I haven't changed the volume layout at all since.
05:56 JoeJulian Dangit, kkeithley, I forgot that the "official" rpms are 3.3.1-1. I wanted your bugfix releases.
05:59 jclift_ JoeJulian: k, attached the /var/lib/glusterd/ dirs from both storage nodes.  tarballs
06:01 JoeJulian btw... I always replace the word "node" with "smurf" when I read it. We prefer to use the correct ,,(glossary) terms around here. :P
06:01 glusterbot A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
06:03 jclift_ Meh
06:07 JoeJulian jclift_: Ok, that's just weird. And it starts up the extra brick server every time, even after a reboot?
06:08 jclift_ JoeJulian: I haven't checked if that extra process is there every time.  Give me a few minutes to finish off this new BZ and I'll do a reboot and look. :)
06:09 JoeJulian jclift_: That's it for me tonight. The wife apparently has need of my services in bed...
06:10 jclift_ JoeJulian: Heh, no worries.  Have a good sleep dude. :)
06:10 JoeJulian ... and now she's whacking me in the shoulder. I guess I'm not supposed to share that information.
06:10 jclift_ Just make sure there's no camera on the laptop/tablet, you're all set. :D
06:11 Debolaz I'm reading about split brain now (Background: 2nd day of using GlusterFS, so I'm quite new). While it doesn't really bother me from a perspective of using GlusterFS or not, I'm fine with it potentially happening, are there any specific steps I should do to prevent it that isn't done for me with a standard replica setup?
06:12 JoeJulian You can enable quorum. And we'll have to re-post the article about how to cause split-brain. The site it was on has gone away.
06:12 JoeJulian @split-brain
06:12 glusterbot JoeJulian: (#1) To heal split-brain in 3.3, see http://goo.gl/FPFUX ., or (#2) learn how to cause split-brain here: http://goo.gl/Oi3AA
06:14 puebele joined #gluster
06:15 Debolaz So I should simply type: gluster volume set web01 cluster.quorum-type Auto
06:16 JoeJulian http://webcache.googleusercontent.com​/search?q=cache:Jnyrx6G9D-kJ:goo.gl/O​i3AA&amp;hl=en&amp;gl=us&amp;strip=1
06:16 glusterbot <http://goo.gl/8RPZU> (at webcache.googleusercontent.com)
06:16 JoeJulian Debolaz: Yes
06:16 JoeJulian That link is the google cached version of that "how to cause split-brain" page.
06:18 JoeJulian ... and I should have actually answered, "Yes. Provided your use case will allow that."
06:18 JoeJulian ok, really gone (again). Goodnight.
06:20 jtux joined #gluster
06:20 krokarion joined #gluster
06:22 rotbeard joined #gluster
06:24 glusterbot New news from newglusterbugs: [Bug 978205] NFS mount failing for several volumes with 3.4.0 beta3. Only last one created can be mounted with NFS. <http://goo.gl/6epdm>
06:24 psharma joined #gluster
06:26 ngoswami joined #gluster
06:33 puebele joined #gluster
06:34 vimal joined #gluster
06:42 kshlm joined #gluster
06:44 ricky-ticky joined #gluster
06:52 ekuric joined #gluster
06:54 dobber joined #gluster
07:00 mmalesa joined #gluster
07:06 krokar joined #gluster
07:10 hybrid512 joined #gluster
07:19 sgowda joined #gluster
07:28 y4m4 joined #gluster
07:37 ProT-0-TypE joined #gluster
07:37 anands joined #gluster
07:44 icemax joined #gluster
07:54 glusterbot New news from newglusterbugs: [Bug 969461] RFE: Quota fixes <http://goo.gl/XFSM4>
08:03 theron joined #gluster
08:08 mmalesa joined #gluster
08:08 mmalesa joined #gluster
08:29 hagarth joined #gluster
08:33 rjoseph joined #gluster
08:37 deepakcs joined #gluster
08:39 anands joined #gluster
08:42 ramkrsna joined #gluster
09:11 bulde joined #gluster
09:19 mmalesa joined #gluster
09:22 bulde joined #gluster
09:28 spider_fingers joined #gluster
09:29 vshankar joined #gluster
09:30 mmalesa_ joined #gluster
09:36 mmalesa joined #gluster
09:45 benashmead joined #gluster
09:46 pkoro joined #gluster
09:47 benashmead anyone around who could maybe answer a question or two for me?
09:47 manik joined #gluster
09:50 benashmead I installed GlusterFS from the CentOS 5 repos, which unfortunately stuck me with v2.0.9 - I'm trying to work out the upgrade path from there to 3.2, and from there to 3.3. Can't seem to find any info though - it almost seems as though 2.0.9 and 3.0 are equivalent? Would that be fair to say?
09:55 glusterbot New news from newglusterbugs: [Bug 978297] Glusterfs self-heal daemon crash on split-brain replicate log too big <http://goo.gl/A1CwU>
09:55 ToMilesS joined #gluster
10:06 slava_dp joined #gluster
10:08 slava_dp hi. got an issue, hope someone will help. I have two servers running drupal on top of glusterfs and a balancer in front. page access with bare filesystem is 50ms, and access on glusterfs mounted volume is 7500ms. any hints please?
10:09 slava_dp one node shows peer status as Connected, and the other shows Disconnected, if that's relevant at all.
10:13 duerF joined #gluster
10:20 mmalesa_ joined #gluster
10:22 rastar joined #gluster
10:36 bulde joined #gluster
10:36 edward1 joined #gluster
10:46 ngoswami joined #gluster
10:48 rjoseph joined #gluster
10:55 pkoro joined #gluster
10:58 hagarth joined #gluster
11:01 msvbhat joined #gluster
11:02 ctria joined #gluster
11:10 benashmead just repeating this, in case someone can help:
11:10 benashmead I installed GlusterFS from the CentOS 5 repos, which unfortunately stuck me with v2.0.9 - I'm trying to work out the upgrade path from  there to 3.2, and from there to 3.3. Can't seem to find any info though - it almost seems as though 2.0.9 and 3.0 are equivalent?  Would that be fair to say?
11:13 ToMilesS ndevos: you around? looked in wireshark and the READDIRP Reply coming from one of the bricks does include the unlisted directory
11:14 kkeithley @repos
11:14 glusterbot kkeithley: See @yum, @ppa or @git repo
11:14 kkeithley @yum
11:14 glusterbot kkeithley: The official community glusterfs packages for RHEL (including CentOS, SL, etc), Fedora 17 and earlier, and Fedora 18 arm/armhfp are available at http://goo.gl/s077x. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates repository.
11:19 CheRi joined #gluster
11:20 harish joined #gluster
11:38 jclift_ joined #gluster
11:43 piotrektt joined #gluster
11:43 piotrektt joined #gluster
11:48 ngoswami joined #gluster
11:48 mmalesa joined #gluster
11:51 mmalesa_ joined #gluster
11:53 tziOm joined #gluster
11:55 rcheleguini joined #gluster
12:02 bulde joined #gluster
12:02 y4m4 joined #gluster
12:03 mmalesa joined #gluster
12:05 mmalesa__ joined #gluster
12:05 mmalesa joined #gluster
12:10 CheRi joined #gluster
12:14 ujjain joined #gluster
12:16 rjoseph joined #gluster
12:17 ddp23 joined #gluster
12:18 ddp23 hi, mounting a volume via nat and seeing port 24007 established but the usual other ports 24011 that would be established don't seem to be, though I can telnet to them individually through NAT. Anyone else seen this? Seems to work fine from the same subnet... ideas?
12:19 ddp23 version is 3.2.5-1ubuntu1 in case that matters...
12:20 ngoswami joined #gluster
12:25 aliguori joined #gluster
12:37 jthorne joined #gluster
12:37 jthorne steeeeelllllaaaaaa
12:40 plarsen joined #gluster
12:46 bet_ joined #gluster
12:47 mmalesa joined #gluster
12:48 anands joined #gluster
12:49 mmalesa_ joined #gluster
12:50 hagarth joined #gluster
12:55 mmalesa joined #gluster
12:55 T0aD joined #gluster
12:58 mmalesa_ joined #gluster
12:59 mmalesa joined #gluster
13:01 aliguori joined #gluster
13:02 robo joined #gluster
13:02 deepakcs joined #gluster
13:04 mmalesa joined #gluster
13:06 mmalesa_ joined #gluster
13:06 robo joined #gluster
13:16 joelwallis joined #gluster
13:28 dewey joined #gluster
13:28 ollivera joined #gluster
13:33 bala joined #gluster
13:34 kshlm joined #gluster
13:37 kedmison joined #gluster
13:42 dberry joined #gluster
13:42 dberry joined #gluster
13:43 kedmison Hi, Gluster (and IRC) newbie here... I'm having some trouble with my gluster setup and am hoping to find some help.  'too many open files' is the problem on the gluster nodes, and with something close to 4M files open, I'd have to agree.
13:45 kedmison I definitely don't have enough users or traffic to the gluster servers to need 4M files open, so I am wondering about the possibility of a FD leak somewhere.
13:50 portante joined #gluster
13:54 semiosis kedmison: usually the fd leak is in the application... have you checked open files on your client machines using lsof?
13:54 semiosis also, what version of glusterfs?  what distro version?
13:54 kaptk2 joined #gluster
13:56 rwheeler joined #gluster
13:56 kedmison good question: I just check the clients and the only 2 clients active at this point have around 3K FDs open.
13:57 kedmison glusterfs 3.3.1
13:58 semiosis are you sure those are really the only clients?  you could verify that by looking at tcp connections to the bricks on your glusterfs servers
13:59 ddp23 left #gluster
14:01 spider_fingers left #gluster
14:02 kedmison Yes, quite sure, but I double checked via netstat just in case.  (those are the only clients I've configured at this point).
14:03 dberry left #gluster
14:03 manik joined #gluster
14:03 _br_ joined #gluster
14:05 kedmison I'm looking quite closely at rebalance as a possible cause right now.  I ran a rebalance months ago, and thought it was over and done with long ago.  However, the rebalance log file on the problem node has seen a lot of activity today, and in intervening days too.
14:06 kedmison rebalance status indicates 'failed' for the current node and 'not started' for the other node.
14:06 semiosis a ha
14:07 semiosis sorry but i have to run, bbl
14:07 kedmison no problem; grateful for your help so far semiosis.
14:08 JoeJulian joined #gluster
14:11 vpshastry joined #gluster
14:13 bugs_ joined #gluster
14:16 kedmison I'm not sure what happened, but the open FD counts are back down to reasonable levels.  I did take a couple of actions in between discovering the problem and seeing the counts back down, but I don't know what, if anything, was responsible.
14:29 gmcwhistler joined #gluster
14:40 bsaggy joined #gluster
14:44 failshell joined #gluster
15:00 vdrmrt joined #gluster
15:00 vdrmrt_ joined #gluster
15:00 vdrmrt Maybe somebody can help me
15:00 vdrmrt I deleted a volume with gluster volume delete <volume>
15:01 vdrmrt now I want to create a new volume with the bricks from the old volume
15:01 vdrmrt and now I get
15:01 vdrmrt <brick path> or a prefix of it is already part of a volume
15:01 glusterbot vdrmrt: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
15:04 wushudoin joined #gluster
15:05 social_ I continually keep seeing this on two clients >  [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-Staging-client-0: remote operation failed: Stale NFS file handle. Path: /aA/kac/oink (0f0abac7-dbf4-4955-a346-d0b064402233)  any idea how to debug it?
15:07 failshell is there a way to configure gluster to return only the space used by a volume with df?
15:07 failshell quota maybe?
15:10 theron joined #gluster
15:13 dberry joined #gluster
15:17 lbalbalba joined #gluster
15:17 semiosis failshell: df works for me
15:18 failshell im getting values for the entire cluster
15:18 awheeler_ joined #gluster
15:18 failshell how do you disable NFS globally and not per volume?
15:18 semiosis chirino: tried getting started with the glfsjni project last night but ran into trouble.  basically i dont know where to begin.  i tried a simple 'mvn -Pfull clean test' but that said i needed to provide -Dnative-src-url or something like that
15:19 semiosis failshell: disable it on all volumes, thats the only way i know of
15:20 awheeler_ I have a 4 node gluster setup, with replica set at 2.  Sometimes when I post a file, I see on the XFS level that 2 nodes have the file, and 2 nodes have the file with length 0 and a flag of T.  Other times, it's just on two nodes.  I'm using 3.3.  This seems odd and inconsistent, but otherwise all seems fine.  Can someone tell me if this is a bug or?
15:21 chirino semiosis.. which linux you on?
15:21 chirino 32 or 64  bit?
15:22 chirino semiosis: if your on 64 bit, try: mvn install -P linux64
15:22 chirino then paste me the build output.
15:23 failshell Request received from non-privileged port. Failing request
15:24 failshell i keep getting that error
15:24 failshell since i upgraded to 3.3. and when that happens, the cluster becomes unresponsive
15:24 atrius_ joined #gluster
15:25 daMaestro joined #gluster
15:28 Gugge awheeler_: if i remember correct a file is not moved when it is renamed, and the new name should have been on other servers.
15:28 failshell http://www.gluster.org/pipermail/glu​ster-users/2012-December/035047.html
15:28 glusterbot <http://goo.gl/TbBkN> (at www.gluster.org)
15:28 failshell ah ah
15:29 wushudoin left #gluster
15:30 ndevos ToMilesS: so, if the READDIRP Reply contains the missing directory, the issue definitely is client side
15:30 awheeler_ gugge: Interesting, but all versions of the file (0-length or no) have the same name.
15:31 ndevos ToMilesS: I dont want to say anything stupid, but does the directory does not start with a dot '.', does it?
15:31 aliguori joined #gluster
15:32 semiosis chirino: yeah tried that too but it tried&failed downloading a jar from fusesource's repo
15:32 ndevos ToMilesS: btw, good digging with wireshark, thats not something everyone can find out :)
15:32 semiosis chirino: will get the build output
15:32 chirino semiosis: I think it's cause you don't have autotools installed
15:32 semiosis have it
15:32 chirino but I'd need to see the build output
15:33 chirino then again I've not really tried the native build yet :)
15:33 krishnan_p joined #gluster
15:34 awheeler_ Gugge: nm, I did search on glusterfs and sticky, and it looks like the T flag is indicating that the file has been moved to another node.
15:34 semiosis chirino: one thing i'm confused about... does this need the actual glusterfs source?
15:34 ToMilesS ndevos: no, its a bunch of folders counting up like 'instance-00000266'
15:34 semiosis i tried following the leveldbjni instructions, setting up GLFS_HOME, but i dont know how to make it go
15:34 statix_ left #gluster
15:34 chirino will need a dev env
15:34 semiosis chirino: seems like maven should kick off configure with it's generated prefix
15:34 chirino like the headers
15:35 Gugge awheeler_: correct, and that can happen when you rename a file (it isnt moved to where it should be, but a link is created there)
15:35 chirino semiosis: yeah
15:35 ndevos ToMilesS: hmm, so, it could be some issue in the caching on the client-side, but I think that is unlikely
15:35 awheeler_ Gugge: Perhaps swift is moving the file?  So it arrives on one node, but the dynamic hash puts on a different node?
15:35 chirino semiosis: so that's something that you might be able to avoid.
15:36 chirino i.e. just have an m4 script detect where gluster dev env is installed at
15:36 ToMilesS ndevos: its the folder were openstack/nova stores its KVM info and disk images
15:36 chirino based on standard install conventions.
15:36 lbalbalba left #gluster
15:36 ToMilesS ndevos: de volume is mounted on diff machines and they all have the same 'ghost' directory not showing up in listing
15:37 ndevos ToMilesS: that is very strange, and it rules out client-side caching too :)
15:37 Gugge awheeler_: or switch copies the data to a tmpfile and renames it to the right name
15:38 Gugge swift
15:38 awheeler_ Gugge: Ah, so the 0-lenght file is the correct location for the temporary file name, but the wrong loction for the new name?  I think I like that answer.
15:38 semiosis chirino: https://gist.github.com/semiosis/5868477
15:38 glusterbot Title: glfsjni mvn -Plinux64 install (at gist.github.com)
15:39 ndevos ToMilesS: one last idea: you can check if the gfid of that directory (getfattr -m trusted.gfid -d -e hex instance-00000266) is unique, maybe you have a gfid collision that causes the issue
15:39 failshell any idea what could lead to this message with georepl: https://gist.github.com/fai​lshell/d656c0dc2ffec26ab5d7
15:39 glusterbot <http://goo.gl/EGbxy> (at gist.github.com)
15:40 ToMilesS ndevos: when I look in wireshark i can see 'LOOKUP' call for every folder in the listing except the 'ghost' dir, even tho it did got the dir from the bricks readdir
15:40 ndevos ToMilesS: I'll be back tomorrow, and may have more time to look at it, I'd be interested in the tcpdump with the full readdirp call/replies
15:41 chirino semiosis: problem is at line 33
15:41 ToMilesS ndevos: I'll check for gfid collision and we can talk again tomorrow then
15:41 awheeler_ Gugge: On a related note, when the hasing algorythm places more files on one set of nodes than another, and the first set fills up, will a rebalance, possibly with fix-layout correct this?  If not, is there any way to change the hashing seed to get a different distrubution?
15:41 awheeler_ s/hasing/hashing/
15:41 bulde joined #gluster
15:41 glusterbot awheeler_: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:41 ndevos ToMilesS: hmm, that does not ring a bell with me...
15:41 bala joined #gluster
15:42 chirino configure generation failed.
15:42 ndevos ToMilesS: good luck, and we'll speak tomorrow again
15:42 ToMilesS ndevos: ok good, thanks for the help
15:43 awheeler_ Gugge: Incidentally, a rebalance removes the 0 length files.
15:45 aliguori joined #gluster
15:45 chirino semiosis: ok, I commented out most of the custom.m4 bits for now
15:46 chirino pull and give it another try
15:46 chirino it should actually get to the C compiler stage.
15:47 Gugge awheeler_: what happens is this, you copies data to a tempfile, and it is placed where the tempfile name hash dictates. then you rename it, and the data is not moved, but a link is created where the new name hash dictates it should be. when you rebalance the data is moved
15:48 semiosis chirino: yes!
15:48 semiosis chirino: now i see failing output from configure, i think i can work with this
15:48 semiosis unfortunately my time for this is limited during the day while i'm at work
15:48 chirino k
15:48 semiosis thanks
16:02 awheeler_ Gugge: Ah, excellent, thanks for that.
16:06 hagarth joined #gluster
16:06 vpshastry joined #gluster
16:11 Mo__ joined #gluster
16:20 bulde joined #gluster
16:32 failshell JoeJulian: you around?
16:36 JoeJulian failshell: Just got in to the office. What's up?
16:36 failshell talk about good timing
16:36 failshell JoeJulian: ever seen that with georepl https://gist.github.com/fai​lshell/0acbc3311f3023077da7
16:37 glusterbot <http://goo.gl/Gmzvi> (at gist.github.com)
16:37 failshell seems to be only one volume with that problem
16:37 JoeJulian I never use georep, so probably not. :/
16:37 failshell ah damn :)
16:38 JoeJulian EBADF but not sure when end is throwing that.
16:39 manik joined #gluster
16:40 JoeJulian what version is that?
16:40 failshell 3.3.1
16:40 failshell destroyed the destination, hopefully that's the one borking up
16:40 JoeJulian Hmm... I wonder why it doesn't match mine...
16:41 JoeJulian Because I'm looking at the wrong file....
16:41 failshell well, its the source
16:42 JoeJulian Looks like the child process must have died.
16:43 failshell weird thing is, even if gluster reports the georepl as faulty, its still syncing data
16:43 mmalesa joined #gluster
16:44 JoeJulian Ok, now Gnome has gone too far. They took transparency out of gnome-terminal so now I can't see my background through the terminal window. :(
16:44 failshell i gave up on using linux on the desktop
16:44 JoeJulian lol!!!!
16:45 failshell at work, we're pretty much an OSX shop on the desktops
16:45 tg2 joined #gluster
16:45 failshell its not bad, but these days i prefer Win7 on the desktop
16:45 failshell seems to be the simpler get-out-of-my-way
16:45 JoeJulian right...
16:46 JoeJulian ... anyway ... how about that weather, eh?
16:46 failshell lol
16:46 jclift_ failshell: Yeah, I went to OSX a few years ago coz I got sick of updating every x months with the desktop linux releases (eg fedora), and having these releases commonly spontaneously break while I'm trying to get work done (ie yum update -> new kernel version -> doesn't like my nVidia card)
16:47 jclift_ ... and now OSX has moved to a yearly model.  Dammit.
16:47 failshell i gave Linux a spin again on my home box
16:47 JoeJulian jclift_: True.... solved that buy replacing the video card.
16:47 failshell recently, latest fedora
16:47 semiosis JoeJulian: make your background the same shade of absolute black as your terminal background -- pretend it's transparent
16:47 failshell and latest ubuntu
16:47 JoeJulian semiosis: lol
16:47 failshell none could use my wifi card
16:47 failshell so
16:47 jclift_ JoeJulian: GTX285 video card 2GB ram. ($$$)  So, that wasn't an option. :D
16:47 failshell but that said, the nouveau driver now works fine with my nvidia gtx 560
16:48 failshell i gotta say though, im a gamer, so on the home PC, i need win7
16:48 JoeJulian Actually, it's a background I took while hiking at Mt. Saint Helens. It's very zen and I like having that feeling of being able to see outdoors.
16:48 lpabon joined #gluster
16:48 jclift_ failshell: Agreed
16:48 jclift_ Interestingly, Borderlands 2 has an OSX port that's actually playable on HD2500 graphics
16:48 jclift_ (without looking completely crap)
16:48 semiosis failshell: have you tried kde?
16:49 deepakcs joined #gluster
16:49 semiosis failshell: kubuntu is the best desktop imo
16:49 failshell semiosis: never liked kde, was always a gnome
16:49 JoeJulian I do have to boot into windows occasionally to play Firefall.
16:49 failshell my favorite was always xfce
16:49 * jclift_ installed Fedora 19 (xfce desktop) on laptop recently.  I don't use the laptop much, but it's actually fairly decent for once
16:49 failshell ah yeah, a former coworker of mine works on Firefall
16:49 jclift_ When I'm next sick of OSX, I might try that (then prob run back to OSX) :D
16:50 semiosis jclift_: would you try kde?
16:50 failshell i kinda like fedora's gnome 3 look. but there's no way im gonna have a network cable running from my office to the living room for connectivity : )
16:50 JoeJulian OSX doesn't run on that GTX285 either, though, does it?
16:50 jclift_ JoeJulian: Sure does
16:51 jclift_ JoeJulian: OSX can run a GTX285 "out of the box".  Not modding required.
16:51 failshell i miss the configuration flexibility of linux though
16:51 failshell this damn command+F1 on OSX, can't diable that hotkey and it sucks
16:52 jclift_ Purely by luck, the GTX285 happened to be the exact chipset that Apple used in it's high end graphics card offering for Mac Pro users some years back.  It "just worked".
16:52 jclift_ Since then I've moved to real Apple hw tho
16:53 failshell OSX is a good compromise between linux and windows
16:54 jclift_ Yeah
16:54 jclift_ Has GUI, has usable CLI interface, + modern (commonly available) programs run on it
16:54 * jclift_ gets back on topic :D
16:54 failshell W [rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
16:54 failshell how do i get rid of that message?
16:55 jclift_ No idea.  I'm told it can be ignored tho
16:56 JoeJulian Nice... took a nice simple dialog and made it this: https://wiki.gnome.org/Terminal/FAQ#Ho​w_can_I_make_the_terminal_transparent.3F
16:56 glusterbot <http://goo.gl/1qNPL> (at wiki.gnome.org)
16:56 failshell JoeJulian: yeah that's Gnome for you. Removing usefull stuff.
16:56 NuxRo JoeJulian: you could install MATE :)
16:57 JoeJulian Then telling you you can still make it work, but you have to use other tools and obscure settings. Might as well be a windows registry...
17:07 balunasj joined #gluster
17:16 Eco_ joined #gluster
17:38 joelwallis joined #gluster
17:39 zaitcev joined #gluster
17:47 mooperd joined #gluster
17:53 JoeJulian failshell: You would have to add "option transport-type {[tcp],[socket],[rdma]}" to the .vol file. Not worth the effort, imo.
17:53 failshell did that
17:53 failshell cant wait to have migrated my data to the new cluster
17:53 failshell the old one is in a very weird state right now
17:57 * JoeJulian hates pdf files...
17:59 robos joined #gluster
18:00 nightwalk joined #gluster
18:07 krishnan_p joined #gluster
18:10 failshel_ joined #gluster
18:14 aliguori joined #gluster
18:14 JoeJulian hmm, what's in glusterfs-resource-agents?
18:15 kkeithley This package provides the resource agents which plug glusterd into
18:15 kkeithley Open Cluster Framework (OCF) compliant cluster resource managers,
18:15 kkeithley like Pacemaker.
18:16 JoeJulian Pfft.. you wanted me to actually look at the info? :P
18:16 kkeithley no
18:16 kkeithley ;-)
18:16 JoeJulian I'm writing a blog article about how broken fedora 19 is thanks to libgfapi integration.
18:17 kkeithley oh?
18:17 JoeJulian I have to use my existing 3.3.1 volumes.
18:17 JoeJulian ... though, I admit I haven't tried to see if I can mount the volumes with 3.4... maybe I should do that first.
18:18 JoeJulian 3.4's supposed to do rpc translation to 3.3, isn't it?
18:18 kkeithley ??, not that I know of
18:18 JoeJulian dammit
18:18 kkeithley bug don't quote me
18:19 kkeithley s/bug/but/
18:19 glusterbot What kkeithley meant to say was: but don't quote me
18:19 JoeJulian I've been pushing for that since 3.1.
18:19 kkeithley yes, that would be nice
18:20 hagarth 3.4 and 3.3 are rpc compatible
18:21 JoeJulian Woo-hoo!
18:21 kkeithley And I use one volume to test 3.3.x, 3.4, and master. I don't push on them very hard though.
18:22 JoeJulian hagarth: Now for the harder question... My fedora box is a peer, though it doesn't provide any bricks (just for management). If it's 3.4, can I still be a peer and manage volumes?
18:23 hagarth JoeJulian: just the 3.3 commands from fedora 3.4?
18:23 JoeJulian Right.
18:23 hagarth it should work.
18:23 JoeJulian time to start testing... and I was going to get work done today....
18:24 hagarth sorry for being a distraction :).
18:24 anands joined #gluster
18:25 JoeJulian Hehe. At least it's a distraction that changes the tone of my day in a more positive direction. My blog will turn from rant to rave. :D
18:25 * hagarth hopes it works ;)
18:25 JoeJulian kkeithley: Did I see that beta3 is about to hit the Fedora repo?
18:26 kkeithley 3.4.0-0.6beta3 was pushed to from Updates-Testing to Updates. IOW yes
18:26 kkeithley for f19
18:27 kkeithley for everything else, beta3 has been in the official (download.g.o) repo for a couple of weeks now
18:28 hagarth kkeithley: are 3.3 ufo rpms available for epel somewhere?
18:29 vdrmrt semiosis: I have question for you, do you have some time?
18:29 kkeithley The swift part of UFO is available for el6 from http://repos.fedorapeople.org/re​pos/openstack/openstack-grizzly/
18:29 glusterbot <http://goo.gl/24dOQ> (at repos.fedorapeople.org)
18:30 hagarth kkeithley: ok
18:30 kkeithley the glusterfs-ufo rpm is included with the rest of glusterfs
18:30 kkeithley and for f19 just get the swift bits from yum
18:31 hagarth ok
18:33 kkeithley It's still a bit messy. You can also get the swift bits for el6 from YUM.
18:34 andreask joined #gluster
18:34 kkeithley "we" only build our own swift rpms for f18 and earlier now. Not for el6, f19, and f20/rawhide.
18:36 hagarth think we should evolve a howto on this.
18:36 kkeithley That's for Fedora and EPEL builds. Builds from our git tree with glusterfs.spec.in don't build swift or ufo
18:36 kkeithley confused yet? ;-)
18:37 * hagarth is totally dazed ;)
18:41 kkeithley I'm waiting patiently for lpabon (and jvyas) to get their Fedora Secret Decoder rings and start building their own packages so I can remove ufo from the Fedora/EPEL glusterfs packages.
18:42 JoeJulian oooh, 3.4 client does, indeed, mount my 3.3 volumes. :D <happy dance>
18:43 failshel_ ye for backward compatibility!
18:47 m0zes joined #gluster
18:52 portante joined #gluster
19:02 JoeJulian http://joejulian.name/blog/fedo​ra-19-with-legacy-glusterfs-33/
19:02 glusterbot <http://goo.gl/AFBrL> (at joejulian.name)
19:09 anands joined #gluster
19:18 bulde joined #gluster
19:18 tziOm joined #gluster
19:20 semiosis vdrmrt: ask away
19:20 vdrmrt thx, I'm struggling to mount my glustervolumes in fstab
19:21 vdrmrt sometimes they mount sometimes they don't
19:21 semiosis distro?  glusterfs version?
19:21 vdrmrt ubuntu 12.04
19:21 vdrmrt gluster 3.3.1
19:21 semiosis mounting from localhost or remote?
19:21 vdrmrt localhost
19:21 vdrmrt I'm just experimenting with 3 virtual machines
19:21 nueces joined #gluster
19:22 vdrmrt I don't know if that has anything to do with it
19:22 vdrmrt I tried mount with the hostname and 127.0.0.1
19:23 semiosis how many mounts?
19:23 semiosis in fstab?
19:23 vdrmrt 1 for gluster
19:23 vdrmrt 10 for hdd
19:24 semiosis i meant gluster, thx
19:24 semiosis and do you have client logs showing the mount failures?
19:24 vdrmrt yes
19:24 semiosis pastie.org them?
19:24 vdrmrt hmm
19:24 vdrmrt ok one moment
19:25 vdrmrt I have a work around with rc.local now :-)
19:25 semiosis i'm sure
19:35 vdrmrt http://pastie.org/8083571
19:35 glusterbot Title: #8083571 - Pastie (at pastie.org)
19:35 vdrmrt I'm not sure If you can make anything out of it.
19:35 vdrmrt I can't
19:36 semiosis that's the glusterd log file, i need to see a client log file, should be named for the mount point, like /var/log/glusterfs/mount-point.log
19:37 vdrmrt ah
19:37 vdrmrt I think I see it now
19:37 vdrmrt [2013-06-26 19:33:33.722489] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 127.0.0.1
19:38 vdrmrt strange
19:38 semiosis indeed
19:38 semiosis i have seen that too, intermittently, i dont know what the cause is, or why it's intermittent
19:38 vdrmrt sometimes it does work
19:38 vdrmrt sometimes not
19:38 krishnan_p joined #gluster
19:38 vdrmrt let me try with the hostname again
19:39 semiosis shouldn't matter
19:42 vdrmrt I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
19:42 vdrmrt [2013-06-26 19:39:37.493058] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
19:42 vdrmrt [2013-06-26 19:39:37.493095] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host mvs1
19:42 vdrmrt [2013-06-26 19:39:37.493125] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
19:42 vdrmrt the same
19:42 vdrmrt when I reboot I get an error that the it can't mount
19:43 vdrmrt with the question to skip or manually recover
19:43 vdrmrt I skip
19:43 vdrmrt And now I see that it's mounted
19:43 plarsen joined #gluster
19:44 vdrmrt mvs1:gv0                                        /mntpoint     glusterfs       defaults        0       0
19:44 vdrmrt something wrong with that?
19:45 semiosis i'd suggest replacing defaults with nobootwait
19:45 semiosis defaults is a placeholder when you dont set any options
19:45 vdrmrt ah ok
19:46 semiosis but since this is, ultimately, a network mount, you should use nobootwait
19:46 semiosis so problems with it won't delay system booting
19:46 semiosis just a safety valve
19:47 vdrmrt I looked at nobootwait but documentation was not always clear for me
19:48 vdrmrt ok the system boots to login now
19:48 vdrmrt so that's ok
19:48 vdrmrt the volume is also mounted
19:48 vdrmrt that's great
19:48 vdrmrt but
19:49 vdrmrt I get following error in console
19:49 vdrmrt mountall: Plymounth command failed
19:49 semiosis ?!
19:49 vdrmrt mountall: disconnected from plymounth
19:49 vdrmrt skipping mounting since plymouth is not available
19:49 vdrmrt event failed
19:50 vdrmrt so I get a boot error but the volume is mounted
19:50 semiosis idk what plymou(n)th is
19:50 * semiosis googles
19:50 vdrmrt me neither saw it in /etc/init
19:50 semiosis "Plymouth is the application which provides the graphical "splash" screen when booting and shutting down an Ubuntu system." - https://wiki.ubuntu.com/Plymouth
19:51 glusterbot Title: Plymouth - Ubuntu Wiki (at wiki.ubuntu.com)
19:51 vdrmrt omg
19:51 vdrmrt strange running ubuntu-server in vm
19:52 semiosis anyway, probably unrelated to any gluster issues
19:52 vdrmrt idd
19:52 semiosis so, this DNS resolution failure thing, it's a pain
19:53 semiosis i ran into it myself, and although i did come up with a solution, i'm not happy with it
19:53 vdrmrt do you mount volumes with fstab?
19:53 semiosis yes, i refuse to engage in any rc.local foolishness
19:53 semiosis :)
19:53 vdrmrt :-)
19:53 semiosis so what i did was to modify the mounting-glusterfs.conf upstart job to wait for static-network-up
19:54 semiosis i'm not sure even if that's the right event, but it seemed to work every time and fail none
19:54 vdrmrt let me try that
19:54 semiosis one problem though is with multiple mounts
19:54 semiosis hang on i'll pastie an example
19:54 vdrmrt thx
19:55 semiosis http://pastie.org/8083658
19:55 glusterbot Title: #8083658 - Pastie (at pastie.org)
19:55 semiosis so i mentioned multiple mounts are a problem
19:56 semiosis if you have more than one glusterfs mount in fstab then you need an blocker/waiter job for each one, which is why you see /the/mount/point appearing twice in that file
19:57 semiosis the reason for this is if you don't make the waiter job specific to the mount point, with a MOUNTPOINT= parameter, then the waiter will only catch one of your mounts, the others will not wait, will start immediately, and possibly fail with the DNS resolution error
19:58 semiosis so i have puppet generate one of these waiters for each client mount, since of course i model all my mounts in puppet
19:58 semiosis this works, but i'm not happy with it
19:58 vdrmrt I can see that
19:58 vdrmrt puppet is on my learning list
19:58 semiosis would rather have a real, native solution, worthy of submitting to ubuntu.
19:59 vdrmrt I created a script that checks if the volume is started and run it from rc.local
19:59 vdrmrt very simple :-)
20:00 semiosis i see
20:02 vdrmrt In your solution does this mean that you have change your init script needs to be changed for every volume that you want to mount
20:02 vdrmrt * sry let me try that again
20:02 lpabon_ joined #gluster
20:02 vdrmrt does this mean that you have to change the init script for every volume you want to mount?
20:03 vdrmrt * that's better :-)
20:03 johnmorr joined #gluster
20:08 rcoup joined #gluster
20:09 lpabon joined #gluster
20:10 vdrmrt semiosis: got it "so i have puppet generate one of these waiters for each client mount"
20:11 vdrmrt have to read more carefully
20:11 vdrmrt semiosis: thx for the support
20:11 semiosis was in eclipse, didnt see your chats until you called my name
20:11 semiosis yes you got it
20:11 vdrmrt np
20:11 semiosis yw
20:20 bsaggy hey guys, got a directory that shows the error when running ls on it from a client: ls: reading directory .: File descriptor in bad state
20:20 bsaggy Any ideas?
20:30 vdrmrt bye
20:36 bsaggy if it helps, this is the error shown in the client's log: http://pastebin.com/UCKbGK7R
20:36 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
20:37 bsaggy here it is in fpaste: http://fpaste.org/21173/
20:37 glusterbot Title: #21173 Fedora Project Pastebin (at fpaste.org)
20:38 JoeJulian bsaggy: What version is that?
20:38 bsaggy glusterfs 3.1.3
20:39 * JoeJulian makes warding signs...
20:40 JoeJulian It's probably a bug. Perhaps mismatched gfids. Check the ,,(extended attributes) on the directory that it's complaining about.
20:40 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
20:40 JoeJulian On each brick.
20:41 JoeJulian Also, you should upgrade. There's some critical bug fixes out since then.
20:42 nightwalk joined #gluster
20:43 bsaggy Yes, I'd like to upgrade; I'll be looking into that soon.
20:43 bsaggy I'll check out the extended attributes, thanks.
20:53 mooperd joined #gluster
20:56 JoeJulian oldbug 3705
20:56 glusterbot Bug http://goo.gl/YORlt low, medium, ---, kparthas, ASSIGNED , [FEAT] Use uuid in volume info file for servers instead of hostname or ip address
20:58 joelwallis joined #gluster
21:00 bsaggy JoeJulian: so the getfattr returned different trusted.gfid's on the directory in question. Now what?
21:01 JoeJulian Make them match. Pick one and setfattr -n trusted.gfid -v {replacement value} $path
21:01 JoeJulian That version will do that. It's fixed in 3.1.5.
21:02 bsaggy Alright, cool. I'll give it a try.
21:06 bsaggy That seems to have worked so far, I can ls the directory in question successfully.  However, I still see the log entries about self healing everytime I list that directory.
21:07 JoeJulian Check the trusted.afr values on those directories. Zero them out if they're not.
21:08 JoeJulian There may be other split-brain beneath that directory, too. There's a script way back on my blog that you can use to check the clean status of your files.
21:08 JoeJulian http://joejulian.name
21:08 glusterbot Title: JoeJulian.name (at joejulian.name)
21:12 bsaggy Thanks, yes, so far I see one sub directory with the issue.
21:14 bsaggy afr's for both clients are 0x000000000000000000000000
21:14 bsaggy i'm sorry, both bricks*
21:21 JoeJulian Perhaps remount?
21:29 bsaggy JoeJulian: It was showing some split brain errors, but seemed to be working, then showed a (fuse_loc_fill() failed) error. I backed out of the directory in question, then listed it and it lists without errors.
21:29 bsaggy seems to be fine now.
21:34 jskinner joined #gluster
21:37 fidevo joined #gluster
21:41 bsaggy JoeJulian: Thanks for your help, I appreciate it!
21:41 nueces joined #gluster
21:42 JoeJulian bsaggy: excellent. Seriously, though, you are going to lose data if you don't upgrade to at least 3.1.5. I can't stress that enough.
21:42 bsaggy Thanks for the heads up - I'll get on that.
22:26 nueces joined #gluster
23:04 Koma joined #gluster
23:15 ninkotech_ joined #gluster
23:16 ninkotech joined #gluster
23:29 robo joined #gluster
23:35 Oneiroi joined #gluster
23:42 nixpanic joined #gluster
23:42 nixpanic joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary