Camelia, the Perl 6 bug

IRC log for #gluster, 2013-01-07

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:35 daMaestro joined #gluster
01:34 chirino joined #gluster
02:30 bharata joined #gluster
02:51 raven-np joined #gluster
03:05 smellis dudes, I am having performance issues on 3.3
03:05 smellis what should I look for?
03:11 smellis left #gluster
03:11 smellis joined #gluster
03:12 smellis hello all, i'm getting poor performance on good hardware with a replicated volume
03:12 smellis any ideas on where to start
03:12 edong23_ joined #gluster
03:12 semiosis smellis: what are you doing?  what are you seeing?  what do you expect to see instead?
03:15 smellis i've got two nodes, a raid 10 vol on both with xfs as the backing store, and a replicated gluster vol between them
03:15 smellis 153MB/s on the backing store, 12 MB/s on the gluster mount
03:16 smellis those are write speeds
03:17 semiosis what are you doing to produce those numbers?
03:17 smellis dd if=/dev/zero of=/mnt/gluster/test.img count=1024000 bs=1024
03:18 semiosis first of all, dd is not representative of any actual workload you're likely to use glusterfs for
03:18 smellis haven't tested read performance yet
03:18 smellis i plan to put vm images on this vol
03:18 semiosis however, if you insist, you should use bs=1M or larger
03:18 smellis but I should get similar numbers at least right?
03:18 smellis roger, i'll run that
03:18 semiosis so put some vm images on it and see if you get useable performance
03:19 smellis i'll do that too
03:19 smellis lol
03:19 semiosis glusterfs allows you to scale out, achieving more aggregate performance than a single server can provide, when using many servers and many clients
03:20 semiosis although all things being equal single thread performance will likely be less through glusterfs than on bare metal local filesystem
03:20 smellis well, crap
03:20 smellis bs=1M and we got 96 MB/s
03:20 smellis kewl
03:20 semiosis :)
03:20 semiosis still not representative of an actual workload, but at least the numbers look good
03:20 semiosis sorry to burst your bubble, we've all been there... but that's the reality of distributed cluster filesystems
03:22 semiosis bbiab
03:23 smellis well, we'll put some vms on and test there as well
03:28 vex Hi. So, I've resolved some split-brain files, how do I now clear them from 'gluster volume heal <volume> info split-brain' ?
03:28 bulde joined #gluster
03:28 sunus joined #gluster
03:35 vex oh good.
03:35 vex https://bugzilla.redhat.com/show_bug.cgi?id=864963
03:35 glusterbot <http://goo.gl/8tcCO> (at bugzilla.redhat.com)
03:35 glusterbot Bug 864963: low, medium, ---, vsomyaju, ASSIGNED , Heal-failed and Split-brain messages are not cleared after resolution of issue
03:40 sripathi joined #gluster
03:40 JoeJulian semiosis: Does it sound sane that installing your glusterfs-server package doesn't create /etc/init/mounting-glusterfs.conf or is something wrong here?
03:41 sripathi joined #gluster
03:54 sgowda joined #gluster
04:07 overclk joined #gluster
04:10 vpshastry joined #gluster
04:35 badone_ joined #gluster
04:39 cicero joined #gluster
05:02 vpshastry left #gluster
05:04 vpshastry joined #gluster
05:07 Humble joined #gluster
05:18 lala joined #gluster
05:18 vijaykumar joined #gluster
05:21 mohankumar joined #gluster
05:27 lala_ joined #gluster
05:28 cicero joined #gluster
05:30 lala_ joined #gluster
05:34 bulde joined #gluster
05:34 hagarth joined #gluster
05:49 shylesh joined #gluster
05:52 cicero joined #gluster
05:53 flin joined #gluster
05:55 lala_ joined #gluster
06:01 rastar joined #gluster
06:03 JoeJulian semiosis: https://gist.github.com/4472816
06:03 glusterbot Title: The glusterfs-server upstart conf would start before the brick was mounted. I added this ugly hack to get it working. I ripped the pre-start script from mysql and decided to leave the sanity checks in place, just because. (at gist.github.com)
06:05 cicero joined #gluster
06:13 shylesh joined #gluster
06:16 mdarade1 joined #gluster
06:21 mdarade1 joined #gluster
06:22 shylesh joined #gluster
06:36 cicero joined #gluster
06:38 raghu joined #gluster
06:43 lala joined #gluster
06:45 vimal joined #gluster
07:06 guigui1 joined #gluster
07:08 ngoswami joined #gluster
07:08 shylesh joined #gluster
07:09 test joined #gluster
07:09 test Hi, I have a use case. Could anyone please help me in understanding how glusterFS can help me to achieve that
07:10 test I want to separate my storage such that two EC2 instances can access it
07:18 dobber joined #gluster
07:18 jtux joined #gluster
07:19 joeto joined #gluster
07:30 khushildep joined #gluster
07:48 tjikkun joined #gluster
07:48 tjikkun joined #gluster
07:52 ekuric joined #gluster
07:58 hateya joined #gluster
08:00 deepakcs joined #gluster
08:00 Azrael808 joined #gluster
08:03 ctria joined #gluster
08:05 andreask joined #gluster
08:05 badone_ joined #gluster
08:05 Nevan joined #gluster
08:36 hagarth joined #gluster
08:37 shylesh joined #gluster
08:40 sripathi joined #gluster
08:41 TheHaven joined #gluster
08:45 tjikkun_work joined #gluster
08:46 ramkrsna joined #gluster
08:46 ramkrsna joined #gluster
09:03 cicero joined #gluster
09:06 zhashuyu joined #gluster
09:23 shireesh joined #gluster
09:33 sripathi1 joined #gluster
09:45 khushildep joined #gluster
09:46 sripathi joined #gluster
09:48 LoadE joined #gluster
09:49 LoadE hi. im trying to set up a gluster replicated volume. I have 2 hosts, one with some data. I have created the volume and only have data on one. i've tried self heal but it does not seem to trigger a sync. what should i do to get the data from host1 synced to host2? (glusterfs 3.3.1)
09:52 mdarade1 left #gluster
09:54 hagarth joined #gluster
09:59 guigui4 joined #gluster
10:05 overclk joined #gluster
10:05 raven-np joined #gluster
10:16 bulde joined #gluster
10:26 andrei__ joined #gluster
10:33 shireesh joined #gluster
10:33 rastar1 joined #gluster
10:45 duerF joined #gluster
10:55 glusterbot New news from newglusterbugs: [Bug 880241] Basic security for glusterd <http://goo.gl/rOZ3P>
11:00 morse joined #gluster
11:15 andrei__ joined #gluster
11:16 vpshastry joined #gluster
11:20 cicero joined #gluster
11:21 raven-np joined #gluster
11:23 glusterbot New news from resolvedglusterbugs: [Bug 890509] kernel compile goes into infinite loop <http://goo.gl/ax3HE>
11:37 raven-np1 joined #gluster
11:41 mohankumar joined #gluster
11:44 guigui3 joined #gluster
11:56 edward1 joined #gluster
12:06 guigui3 joined #gluster
12:10 vpshastry joined #gluster
12:12 khushildep joined #gluster
12:13 rwheeler joined #gluster
12:19 raven-np joined #gluster
12:22 social_ joined #gluster
12:23 dobber joined #gluster
12:23 social_ Hi, I have issue with gluster, I can see on 3.3.0 that mkdir takes ages (1min) what can be cause of this?
12:29 edward1 joined #gluster
12:38 wintix joined #gluster
12:38 wintix joined #gluster
12:39 shireesh joined #gluster
12:45 H__ social_: I see thsi too from time to time. What's your drives' IOPS when this happens for you ?
12:46 social_ H__: underlying raind seems to be slacking :/
12:47 social_ oh iops on gluster volumes, let me look
12:54 social_ H__: seen higher, it's not in use at all :/
13:01 bulde1 joined #gluster
13:07 raven-np joined #gluster
13:13 social_ it seems to be something with locks
13:27 social_ H__: I turned on debug output on brick and I see this: D [mem-pool.c:457:mem_get] (-->/usr/lib64/libglusterf​s.so.0(call_resume+0x8fb) [0x347822ecfb] (-->/usr/lib64/glusterfs/3.3.0/xlator/perform​ance/io-threads.so(iot_fxattrop_wrapper+0x46) [0x7feda76f00c6] (-->/usr/lib64/libglusterfs.so.0(mem_get0+0x1b) [0x347824093b]))) 0-mem-pool: Mem pool is full. Callocing mem
13:32 manik joined #gluster
13:34 gbrand_ joined #gluster
13:44 H__ social_: cannot really help you there (just a gluster user myself), I'm sure others here can.    How's your memory pressure on the gluster boxen ?
13:45 JoeJulian LoadE: Step 1: Create GlusterFS Volume. Step2: Mount that volume someplace. Step3: Copy the files onto it. Step4: ... Step 5: Profit!
13:46 social_ JoeJulian: any idea how to debug Mem pool is full. Collocing mem errors?
13:46 JoeJulian LoadE: Though practically, if you create a replicated volume and you start with bricks where the left-hand one in the volume definition is the one with pre-existing data, it generally works.
13:47 JoeJulian social_: getting through the scrollback, but the first problem I see is 3.3.0
13:50 JoeJulian social_: The only possible response to your question is that you cannot diagnose that error simply because it isn't one. It's a debug message.
13:50 bennyturns joined #gluster
13:51 JoeJulian So you're doing a mkdir of one directory and it takes a full minute?!?
13:51 social_ JoeJulian: yep
13:51 social_ at best only 30sec
13:51 JoeJulian How many bricks?
13:51 social_ 10
13:51 JoeJulian latency?
13:52 rwheeler joined #gluster
13:52 social_ JoeJulian: how can I look at latency? btw write of 100MB file takes under second
13:52 social_ so I wouldn't say it's latency :(
13:54 JoeJulian One thing that's different with a directory vs a file is that a directory has to be created on all the bricks whereas a file is only on one dht subvolume.
13:54 JoeJulian ping all 10 bricks from the client for a simple latency check.
13:55 JoeJulian (and a minute's still WAY to long unless you're replicating to the moon).
13:56 social_ Well it's on amazon and you know they have those floppy drives underneeth
13:56 JoeJulian hehe
13:57 JoeJulian If you set your client log to TRACE level, you should be able to see which brick server is slow to respond. That might help you diagnose.
13:57 social_ 23ms at max
13:58 JoeJulian Check the root partition size (assuming logs are written to the root partition) to make sure they're not full on your bricks.
13:59 JoeJulian That really slows things down when the brick can't write it's logs. (bug already open for that)
13:59 JoeJulian ( has been since 3.0... :/ )
14:06 social_ http://paste.stg.fedoraproject.org/2939/ hm?
14:06 glusterbot Title: #2939 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
14:07 JoeJulian jdarcy: damn. I hit that 3-way self-heal bug you've just fixed in gerrit
14:09 JoeJulian social_: wow. that's amazingly unhelpful
14:10 social_ JoeJulian: well it's grep on dir name
14:11 social_ I'll try original log
14:11 JoeJulian The stuff between lines 4 and 5 is what I was hoping to find.
14:14 social_ JoeJulian: http://paste.stg.fedoraproject.org/2940
14:14 glusterbot Title: #2940 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
14:19 JoeJulian lines 54 & 55 appear to have data loss
14:20 JoeJulian The first thing I would check would be a df on that client
14:21 hagarth joined #gluster
14:21 hateya joined #gluster
14:23 JoeJulian Actually, yeah, there's lots of incomplete log lines in that.
14:23 * social_ looks if it wasn't caused by copy paste
14:24 social_ huh it's  in the log sorry
14:24 stigchri_ joined #gluster
14:26 rwheeler joined #gluster
14:26 social_ JoeJulian: http://paste.stg.fedoraproject.org/2941/ seems like it was cathing on '
14:27 glusterbot Title: #2941 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
14:34 andrei__ joined #gluster
14:37 JoeJulian ~pasteinfo | social_
14:37 glusterbot social_: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
14:38 JoeJulian or stg or whatever
14:39 JoeJulian brick 0 took 2 seconds to respond to the lock request. brick1 took 15.
14:39 social_ http://paste.stg.fedoraproject.org/2942/
14:39 glusterbot Title: #2942 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
14:39 stopbit joined #gluster
14:39 JoeJulian So that's dfs01 and dfs02
14:43 vijaykumar left #gluster
14:47 puebele1 joined #gluster
14:47 H___ joined #gluster
14:49 m0zes ~/cball
14:54 QuentinF joined #gluster
14:54 QuentinF Hi
14:54 glusterbot QuentinF: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:55 QuentinF Self-healing is just for modification files or also for creation files ?
14:57 QuentinF We need to have rebalance in glusterfs 3.3.1 or self healing automatically rebalance ?
14:58 lala joined #gluster
14:58 JoeJulian modification, creation, deletion, renaming...
14:59 JoeJulian You should only need to rebalance if you add or remove bricks.
14:59 andrei__ joined #gluster
15:00 QuentinF but when a brick down and up after ?
15:00 JoeJulian The self-heal daemon will handle that.
15:00 JoeJulian rebalancing is a distribute thing. Self-heal is a replicate one.
15:03 erik49 joined #gluster
15:04 social_ JoeJulian: I've build similar gluster next to it and I'm having 0,1s on mkdir ;.;
15:04 QuentinF Is there any config to set self-healing auto (gluster 3.3.1) ?
15:05 bala joined #gluster
15:05 JoeJulian QuentinF: nope, that's automatic.
15:07 JoeJulian social_: Check the brick logs on 0,1 and see if anything pops out.
15:07 shireesh joined #gluster
15:07 JoeJulian That's creepy. LinkedIn just recommended a connection to a name that I do, indeed, know. They've been dead for 20 years.
15:08 JoeJulian ... was just someone else with the same name but still... creepy...
15:09 lala_ joined #gluster
15:09 hagarth joined #gluster
15:10 lala_ joined #gluster
15:11 plarsen joined #gluster
15:11 lala_ joined #gluster
15:13 jdarcy JoeJulian: Are you saying you just hit bug 802417?
15:13 glusterbot Bug http://goo.gl/V2r2t high, high, ---, jdarcy, ASSIGNED , [glusterfs-3.3.0qa27]: gfid change in the backend leading to EIO
15:14 QuentinF JoeJulian:  This is my config http://blog.quentinfonteneau.com/wp-conte​nt/uploads/2012/11/UntitledDocument11.png
15:14 glusterbot <http://goo.gl/Y8Vgm> (at blog.quentinfonteneau.com)
15:15 QuentinF While server 2 is down, I create files In volume gluster. Files are distributed on bricks in server 1 & 3
15:16 QuentinF But doesn't replicate when server up
15:16 QuentinF It's work with files update
15:18 JoeJulian jdarcy: you mean you work on more than one bug at the same time? ;) The replica 3 self-heal clearing the pending attributes prematurely bug.
15:18 johnmark greetings, earthlings
15:19 JoeJulian Salutations
15:19 johnmark JoeJulian: I can't use the word salutations without thinking of Charlotte's Web
15:19 JoeJulian I always think of Heathers.
15:20 JoeJulian http://www.youtube.com/watch?v=2LkFNZauk90
15:20 glusterbot Title: Heathers (JD & Veronica meet) - YouTube (at www.youtube.com)
15:23 JoeJulian Oh, jdarcy, maybe it is that bug...
15:26 JoeJulian Yes, indeed. That's what happened.
15:26 jbautista joined #gluster
15:27 JoeJulian We had two switches simultaneously fail. To get things up and running quickly since I didn't know, nor have time to determine, the state of the servers and didn't have time to resolve any split-brain that might have occurred, I chose to punt and just use one of the replica.
15:28 JoeJulian Later, I added back in the second.
15:28 JoeJulian When I added the third, lots of split-brain due to that bug.
15:29 JoeJulian I think I'll have to just wipe the third replicas and let them heal.
15:29 gmcwhistler joined #gluster
15:32 JoeJulian QuentinF: The self-heal daemon works on a timer. Perhaps you didn't wait long enough? Also see "gluster volume heal help" for more tools related to self-heal.
15:32 QuentinF Ok thx
15:35 jbrooks joined #gluster
15:39 tc00per joined #gluster
15:44 bugs_ joined #gluster
15:45 JoeJulian jdarcy: Are you already explaining to Pranith or should I give it a go?
15:48 lh joined #gluster
15:48 lh joined #gluster
15:55 social_ JoeJulian: I cascade restarted glusterd on and it somehow got down to 15 sec, still too much
15:57 chouchins joined #gluster
15:58 JoeJulian social_: glusterd's just the management daemon. glusterfsd is probably the culprit.
16:02 chouchins Morning Joe.  One of our mirrored glusterfs volumes lost its entire phyiscal mirrored node with all bricks a month ago.  Glusterfs kept working fine off of the working mirror.  We've just finished replacing the original node with all of its bricks but the self heal doesn't seem to be running completely.  Any suggestions on how to troubleshoot it?  There should be around 1.5T per brick and the mirror only has around 300G per brick wi
16:02 chouchins th no signs that any more is transferring.
16:03 dbruhn joined #gluster
16:05 JoeJulian The first choices would be the cli: gluster volume heal $vol info; gluster volume heal $vol info split-brain; gluster volume heal $vol info heal-failed
16:06 JoeJulian If those all look clean and I hadn't had my coffee yet, "gluster volume heal $vol full" and wait and see if anything changes.
16:06 raven-np joined #gluster
16:06 chouchins cool, I think I've tried most of these already but I'm going to go through it again with a fine tooth comb
16:07 JoeJulian If you know python at all, you could go with a variation of http://joejulian.name/blog/quick-and-d​irty-python-script-to-check-the-dirty-​status-of-files-in-a-glusterfs-brick/
16:07 glusterbot <http://goo.gl/grHFn> (at joejulian.name)
16:08 JoeJulian hmm, I thought jdarcy had a newer script to do that, but it's not on his github so maybe I'm remembering wrong.
16:09 chouchins hmm well that's not good.  Showing the bricks with the full info as Online N on a vlume status
16:09 chouchins heal $vol info showed the bricks as not connected
16:10 chouchins I'll check out that python script
16:11 chouchins ah the underlying xfs fs for the bricks on my good node are giving i/o errors heh
16:11 JoeJulian argh
16:12 chouchins reboot and pray on this one.  It took my purchasing department a MONTH to order my replacement drives so if the others failed...
16:14 johnmark ouch :(
16:15 daMaestro joined #gluster
16:16 chouchins we are setting up our first locally mirrored and georeplicated glusterfs cluster for clincal patient data this week actually.  It will be our fourth glusterfs cluster here at WashU.
16:16 johnmark chouchins: wow.
16:16 johnmark that sounds pretty cool
16:16 johnmark chouchins: you're in St. Louis?
16:17 chouchins Yeah
16:17 chouchins I talked to you briefly at the last Redhat summit in Boston
16:17 chouchins but we're pretty much full steam ahead with glusterfs here
16:17 johnmark chouchins: awesome. I'm horrible at remembering names and faces
16:17 johnmark chouchins: but I'll mkae it a point of remembering you next time
16:17 chouchins me too, no sweat.
16:18 johnmark speaking of, sounds like we should pay you a visit
16:18 JoeJulian I'm horrible with names, but I remember people. :D
16:18 johnmark heh :)
16:18 chouchins I worked with JoeJulian on an actual outage during the summit heh
16:18 johnmark chouchins: haha awesome
16:18 chouchins it was fun trying to bang that out on my ipad
16:18 johnmark chouchins: we've been making a tour of great midwestern universities
16:18 chouchins oh cool
16:18 JoeJulian Better than on an Evo 4G in the mall (been there, done that).
16:18 johnmark u Wisc, U Iowa, and soon K State
16:19 johnmark JoeJulian: ha!
16:19 johnmark chouchins: so might as well add Wash U to the list
16:19 chouchins great :)  We'll have our georeplicated cluster in place by then.  Its only replicating from one side of the campus to the other but its enough to ensure against building destruction of the primary site
16:20 johnmark nice!
16:29 ninkotech_ joined #gluster
16:29 zaitcev joined #gluster
16:31 JoeJulian jdarcy: Would it be more sane to mark everything with trusted.afr.${volume}-client-${brick_number} to 0x0000..0001 for the 3rd brick or should I just wipe it? Would marking it dirty be safe? It would sure save on bandwidth...
16:31 wN joined #gluster
16:34 jdarcy JoeJulian: If you're sure the third copy is in sync, then you can either delete or zero the xattrs.  If not, then you can delete the file.
16:35 erik49 chouchins, how are you dealing with hipaa?
16:35 chouchins this is georeplicating over our internal secure network between campus buildings so I haven't looked into whether its encrypted or not
16:36 chouchins I was under the impression it ran over ssh like rsync
16:36 jdarcy It does use rsync, not sure about the ssh part.
16:37 JoeJulian jdarcy: I was thinking the other way around. Force the 3rd copy to be stale. Let the delta self-heal do it's magic.
16:37 sjoeboo so, whats the "wake-up" like on geo-rep? how often should i expect it to be moving changes over...?
16:37 jdarcy JoeJulian: How is that the other way around?
16:37 JoeJulian 3rd rep is currently marked clean on the first two.
16:37 sjoeboo i seem to see a big jump when i start it up, then a whole lot of nothing/no changes being sync'd. I changed a config, and things got sync'd suddenly...
16:40 JoeJulian sjoeboo: iirc, it's checks every 10 minutes by default. "gluster volume geo-replication <valid-master> <valid-slave> config" will dump out the default config. "gluster volume geo-replication <master> <slave-url> config <key> <value>" to make changes.
16:40 sjoeboo JoeJulian: Thanks, any docs on all those keys? The ones I'm finding don't have much listed, seems like there are more...
16:41 sjoeboo would that be the timeout?
16:41 JoeJulian No clue. I just noticed that command in a gluster-users thread last night.
16:41 sjoeboo hm
16:42 JoeJulian I haven't even run it myself yet. :D
16:43 shireesh joined #gluster
16:45 JoeJulian chouchins: It does use ssh if you geo-replicate to a directory. If the target is a gluster volume, it mounts that volume on the slave and rsyncs from that.
16:45 JoeJulian er, no..
16:46 JoeJulian it mounts the target volume on the master if the target's a gluster volume.
16:46 JoeJulian sjoeboo: yeah, that thread. ;)
16:46 sjoeboo yah
16:48 social_ hmm nice bug
16:49 social_ anyone using ext4 on gluster bricks? I have 3.3.1 and when I do sftp to node where it's mounted and do ls I'll end up in getdent loop
16:49 semiosis ,,(ext4)
16:49 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
16:49 jdarcy 18 seconds.  You're quick, semiosis.
16:50 semiosis sometimes
16:50 cicero robots are usually quick
16:50 cicero beep boop
16:51 kkeithley 29 to 37 looks like 8 second to me, not 18 ;-)
16:52 semiosis gf told me this song reminded her of me http://www.youtube.com/watch?v=BNC61-OOPdA
16:52 glusterbot Title: Robots - Flight of the Conchords - YouTube (at www.youtube.com)
16:52 cicero BINARY SOLO
16:52 cicero 011011010101
16:52 semiosis affirmative
16:52 cicero :)
16:52 social_ semiosis: ugh I thought it was resolved, ook still broken and still need to downgrade kernel
16:53 JoeJulian social_: You should abuse your kernel contacts over that one.
16:54 jdarcy I think their reaction boiled down to "ha ha suckers, shouldn't have been relying on that field anyway"
16:54 JoeJulian If I'm deducing who social_ is, he's a kernel contributor.
16:55 social_ JoeJulian: I'm not kernel contributor :(
16:55 JoeJulian Dang
16:55 JoeJulian Guess I need to watch more Sherlock.
16:56 cicero and play more violin
16:56 jdarcy Too bad, because I think they're the ones who need to see how such an attitude plays out among users.
16:56 jdarcy Didn't Linus recently go on a rant about "you don't break userspace"?  And yet, they did.
16:56 semiosis maybe fuserspace gets an excemption
16:56 social_ jdarcy: send that link to linus :P
16:56 kkeithley wait, wasn't there just a thread about something a certain someone about not breaking userland?
16:56 jdarcy Heh.
16:56 kkeithley s/someone/someone said/
16:56 jdarcy kkeithley: :-P
16:56 glusterbot What kkeithley meant to say was: wait, wasn't there just a thread about something a certain someone said about not breaking userland?
16:57 semiosis s/xce/xe/
16:57 glusterbot What semiosis meant to say was: maybe fuserspace gets an exemption
16:57 jdarcy "Don't break userland, unless it's something in userland for which we already have irrational hatred."
16:57 kkeithley exception exemption
16:58 JoeJulian Which, wrt Linus, does seem to be FUSE.
16:58 semiosis kkeithley: lol
16:58 kkeithley yes, might point exactly
16:59 jdarcy JoeJulian: It competes (some might say) with NFS, 9p, and Ceph - all dear to some relevant kernel developer or other.
16:59 kkeithley oh the irony
17:00 johnmark jdarcy: sounds like we need a blog post decrying the kernel devs' tyranny
17:00 semiosis jdarcy: saw something about you & sage weil at linux.conf.au... that should be most interesting!
17:00 jdarcy kkeithley: You mean the fact that 9p diverged from the One True Way in other respects?
17:00 johnmark semiosis: it's actually going to be me now :)
17:00 jdarcy semiosis: Unfortunately, it is not to be.
17:00 semiosis johnmark: ooh
17:00 johnmark heh
17:00 kkeithley I don't even know what 9p is, let me go look it up
17:01 johnmark kkeithley: +1
17:01 jdarcy I think it would be awesome to get on stage with Sage, been trying to do just that for years, but I can't do it in Australia.
17:01 jdarcy Maybe we can arrange something at FAST.
17:02 johnmark jdarcy: would be cool
17:02 johnmark so what message should we relay to kernel dev types?
17:02 johnmark seriously. this stuff makes me angry
17:02 johnmark and we should say something - because *not* saying anything sends the wrong message
17:03 johnmark ie. acquiessence
17:03 jdarcy johnmark: That userspace storage - not just us but databases, object stores, etc. - is what users seem to want, and that digging in their heels in opposition is just going to make their opinions irrelevant.
17:03 johnmark jdarcy: ok
17:04 kkeithley Er, what Torvalds said to Mauro is appropriate, don't you?
17:04 johnmark s/acquiessence/acquiescence/
17:04 glusterbot What johnmark meant to say was: ie. acquiescence
17:04 johnmark kkeithley: link?
17:04 semiosis johnmark: you havent seen it?  you're in for a treat!
17:04 jdarcy https://lkml.org/lkml/2012/12/23/75
17:04 glusterbot Title: LKML: Linus Torvalds: Re: [Regression w/ patch] Media commit causes user space to misbahave (was: Re: Linux 3.8-rc1) (at lkml.org)
17:05 JoeJulian From: gluster-users, "I'm running into performance trouble ..."
17:05 jdarcy If we don't make distributed filesystems that work for users, then they'll just use Swift or Cassandra instead.  Not exactly a win for the kernel crowd.
17:06 JoeJulian I always want to recommend viagra when comments start like that.
17:06 jdarcy ViagraFS
17:06 JoeJulian ViagraFS - The cure for performance problems.
17:07 JoeJulian I suppose Pfizer might oppose that name.
17:08 jdarcy SildenafilFS just doesn't have the same punch.
17:08 ekuric joined #gluster
17:09 JoeJulian BostonMedicalGroupFS doesn't roll off the tongue... wait, no! ... Is a bit of a mouthful... Aack! ...is hard to .... oh, nevermind.
17:13 johnmark JoeJulian: lulz
17:13 johnmark no no no, the prescription for better uptime!
17:14 johnmark Splunk used to have a t-shirt that said that.... until new corporate hacks prevented them from making more :)
17:16 johnmark semiosis: ha! that *is* entertaining
17:17 johnmark I do appreciate Linus' candor there. Now we just need to educate them a bit more about what's going down in userspace
17:17 JoeJulian That's an awesome email.
17:17 lala joined #gluster
17:24 lala_ joined #gluster
17:26 H___ joined #gluster
17:28 chirino joined #gluster
17:29 lala_ joined #gluster
17:30 lala_ joined #gluster
17:32 Mo___ joined #gluster
17:33 lala_ joined #gluster
17:33 chouchins joined #gluster
17:34 chouchins its healing the glusterfs volume now that both nodes are up with disks.  We did lose a drive and it locked up the thecus8900 it was mounting.  Should rebuild now though.
17:34 chouchins glad we're building everything new on dell R520s
17:34 social_ last stupid question from me, isn't there some sneaky super quick delete on files? I mean if I run find --delete in cron it is quite slow
17:40 JoeJulian social_: find is probably the slow part.
17:41 hagarth joined #gluster
17:43 JoeJulian Here's an idea... not sure if it will be faster though. The tree walking part may be the expensive part as it's doing a stat on every dirent to find out if it's a file or a directory... I wonder if it would be faster to rm *, then whatever's left you know is a directory. You could then open those dirents and repeat as necessary.
17:55 edong23 joined #gluster
17:58 jdarcy setfattr -n trusted.glusterfs.remove-recursive -v mydirectory /mountpoint
17:58 jdarcy Doesn't work, but we could do it.
17:58 social_ even more interesting would be default ttl for file >.>
17:58 social_ something like s3 stuff
17:59 cicero i cheat
17:59 cicero on the brick itself
18:00 cicero i list it out from the backend
18:00 cicero and then replace it with the mount name
18:00 jdarcy cicero: That's basically what the magic xattr would do, except that it does the fan-out automatically.
18:00 jdarcy cicero: Also, bear in mind that in some installations the bricks won't be directly accessible, and might some day be protected by SELinux so that only glusterfsd can modify them.
18:01 cicero ah yeah
18:01 cicero i know this cheating is full of peril and dangers
18:01 cicero certainly not a best practice
18:01 cicero how do you mean by the magic xattr bit?
18:01 jdarcy cicero: The setfattr above.
18:01 JoeJulian Ugh, I'm following too many bugs. Pranith keeps asking me for feedback and I miss the requests half the time.
18:02 cicero jdarcy: ah
18:02 cicero the one that doesn't work?
18:02 cicero but would be cool if it did?
18:02 JoeJulian yep
18:02 cicero rgr
18:03 cicero yeah that'd be nice :)
18:03 cicero also, death to selinux
18:03 cicero i've never met a selinux i liked
18:03 cicero :x
18:04 johnmark cicero: ha. you and pretty much everyone else I talk to
18:04 JoeJulian cicero: Set up a hackable box sometime. It's fun to let the script kiddies get frustrated.
18:04 edong23 ^
18:04 edong23 i like selinux
18:04 edong23 sometimes its weird
18:04 elyograg one of the first things i do on a new centos box is disable selinux and the firewall. both cause me tons of grief.
18:04 edong23 but as you learn it, you get through it
18:04 edong23 again, education is key
18:05 johnmark edong23: agreed. I can see why it's needed
18:05 johnmark more security or easier to use. Pick one.
18:06 edong23 right, thats how i felt at first
18:06 edong23 but as you keep usingit.. it gets easier
18:06 JoeJulian Too bad rackerhacker doesn't hang out in here anymore. Security is his new gig at Rackspace.
18:06 akenn-1 joined #gluster
18:07 JoeJulian I inadvertently congratulated him on it before he announced it. :o
18:08 johnmark haha
18:08 edong23 im addicted to harvest cheddar sunchips with guacamole
18:08 cicero yeah i've never been properly educated in selinux, only come across it tearing my hair out over apache selinux interactions
18:08 cicero it's never worth it for me
18:09 cicero ah well
18:09 cicero yes edong23 i will take two please
18:10 JoeJulian jdarcy: If it didn't succeed it's reset? I'm confused.
18:10 edong23 cicero: eeven with good security on the network side.. you have risks of sql injections... or http post scripts.. or   or   or   or
18:10 edong23 selinux would help with such
18:11 edong23 if you have a very minimal system, you are probably ok with just network firwalling..
18:11 cicero yeah
18:11 edong23 soo... gluster
18:11 edong23 i like it
18:11 edong23 i really do
18:12 edong23 its super simple to set up
18:12 edong23 testing some performance on it over the next few days
18:12 edong23 need to get vmware to support it
18:12 edong23 ;)
18:12 JoeJulian I'd be a little surprised if they're not considering that themselves.
18:12 edong23 yeah
18:13 edong23 im sure
18:13 * JoeJulian is not a closed-source proponent.
18:13 edong23 me either
18:13 edong23 but...
18:13 edong23 still
18:13 edong23 it would make my migration from vmware to kvm cheaper and simpler
18:13 edong23 just use gluster storage
18:14 JoeJulian How many vm?
18:14 edong23 and migrate a physical machine to kvm
18:14 edong23 600
18:14 edong23 i dont know
18:14 edong23 lol
18:14 edong23 seems like more and more every day
18:14 edong23 vm sprawl
18:14 JoeJulian hehe
18:14 edong23 i think it was 120 last count
18:14 edong23 but i recently deleted several that i think we didnt need
18:15 edong23 and since i havent had calls i think i am right
18:15 JoeJulian Hey! Sounds like you subscribe to ,,(Joe's performance metric).
18:15 glusterbot nobody complains.
18:15 cicero haha
18:15 cicero what just happened
18:15 edong23 JoeJulian: you are hammering gluster pretty hard?
18:15 cicero i don't know how to play this MUD
18:16 johnmark edong23: curious - are you familiar with ovirt?
18:16 JoeJulian edong23: Not really, no.
18:16 edong23 johnmark: its on my list
18:16 edong23 i am not "familiar" with using it, but i am familiar with it
18:16 johnmark edong23: cool. and you have no doubt heard of our QEMU integration?
18:16 edong23 as in i have read about it several times
18:16 JoeJulian cicero: You haven't encountered glusterbot yet?
18:16 JoeJulian @meh
18:16 glusterbot JoeJulian: I'm not happy about it either
18:17 edong23 johnmark: for better formance?
18:17 edong23 wow, 3 letter typo
18:17 edong23 for better performance?
18:17 JoeJulian ViagraFS
18:17 * JoeJulian so wants to code that trigger into glusterbot...
18:17 johnmark edong23: haha... :) yes, better performance
18:18 edong23 interesting
18:18 johnmark edong23: with the combination of the new libgfapi and the BD xlator and the gluster device driver in qemu
18:18 edong23 im happy on vmware for now, its not currently costing anything... im just taking my time preparing for the switch
18:18 johnmark there's talk of 3x read increase and 2x write increase
18:18 johnmark edong23: ah, ok
18:18 edong23 yeah
18:18 edong23 im excited
18:18 edong23 seriously
18:18 edong23 libgfap sounds great
18:18 johnmark edong23: so, I'd be very curious to see what you think
18:18 johnmark yup
18:19 edong23 johnmark: well, im working with smellis on it
18:19 haidz what other places other than /var/lib/gluster and <brick>/.glusterfs is volume information stored?
18:19 edong23 we live near eachother
18:19 edong23 and over the last few days have been workign on the differences in kvm and vmware   but those differences are fading away quick
18:19 JoeJulian The volume information is only stored in /var/lib/glusterd/vols
18:20 edong23 johnmark: my only problem with vmware (other than my iscsi daemon dying all the time) is that when something does go wrong, which isnt often, it is hard as hell to troubleshoot and fix
18:20 edong23 lol
18:20 edong23 i like the idea of being in an environment that i built, and understand full
18:20 edong23 with a full set of tools to do whatever i want
18:21 edong23 and gluster is the nice sprinkles on top
18:21 edong23 dont get me started on all the other dual-primary systems
18:21 edong23 lol
18:21 edong23 i have horror stories
18:21 joshcarter joined #gluster
18:22 edong23 johnmark: yesterday, smellis and i tested performance one a 2 host setup with gluster on raid10 near with 2 drives
18:22 edong23 when we finish, we are going to put it up on an 8 drive setup with raid 10
18:22 edong23 and see what performance increase there is, and then hopefully judge what the difference would be on adding more drives
18:23 JoeJulian When I first got the idea of virtualizing (not even having any truly good reason at the time) I tried qemu (kqemu was a kernel patch at the time), then vmware. Both were so horribly slow that I almost was forced to give up. The settled on Xen.
18:23 edong23 but im satisfied withthe performance of it on 2 disks honestly
18:23 haidz JoeJulian, yeah.. its weird.. i keep getting this.. "/appdata/brick or a prefix of it is already part of a volume
18:23 haidz "
18:23 glusterbot haidz: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
18:23 edong23 JoeJulian: i have no performance issues with vmware
18:23 edong23 but i have very good hardware
18:23 JoeJulian When Red Hat bought kvm, I figured I'd try that and it performed better than xen, so I switched.
18:23 haidz thank you glusterbot
18:24 edong23 god
18:24 edong23 that bot is pretty bad ass
18:24 edong23 lol
18:24 JoeJulian hehe
18:24 edong23 matrix shit
18:25 haidz JoeJulian, basically had some weirdness where a raid volume in amazon didnt mount and started using the underlying filesystem.. its a new deployment so i just recreated it.. but very annoying
18:25 haidz went from md0 to md127
18:25 * glusterbot whispers, Take the red pill...
18:25 edong23 haidz: that means your superblock isnt reporting properly
18:26 edong23 haidz: you can fix it with mdadm --detail --scan >> /etc/mdadm.conf
18:26 haidz edong23, will generating an mdadm.conf fix it?
18:26 haidz cool
18:26 edong23 after you assemble it as /dev/md0
18:26 edong23 yeah
18:26 JoeJulian haidz: Precisely why we recommend putting your brick into a subdirectory of the mounted filesystem.
18:26 edong23 haidz: the idea is to get that stuff into the superblock, but if you didnt, you can rebuild the sb  or just use mdadm.conf
18:26 JoeJulian Then if the filesystem doesn't mount, the directory won't exist and glusterfsd won't start.
18:27 edong23 JoeJulian: i was about to ask if it would create the dir or just die
18:27 edong23 but you answered
18:27 edong23 interesting
18:27 haidz ah
18:27 edong23 smellis: you need to change that
18:27 haidz perfect
18:27 edong23 lol
18:27 nueces joined #gluster
18:27 haidz thanks JoeJulian
18:27 haidz do you guys work at redhat?
18:28 haidz (formerly gluster)?
18:28 JoeJulian I work at Ed Wyse Beauty Supply in Seattle.
18:28 haidz ah ok
18:28 JoeJulian hehe
18:28 JoeJulian I'm a ,,(volunteer).
18:28 glusterbot A person who voluntarily undertakes or expresses a willingness to undertake a service: as one who renders a service or takes part in a transaction while having no legal concern or interest or receiving valuable consideration.
18:28 haidz im trying to get a call with redhat for a largish deploy
18:29 kkeithley JoeJulian is too modest too.
18:29 JoeJulian Yes, I'm very good at modesty.
18:30 JoeJulian johnmark: ^ haidz
18:32 erik49 joined #gluster
18:35 Staples84 joined #gluster
18:37 bauruine joined #gluster
18:44 johnmark haidz: what? and no one is getting back to you?
18:53 smellis edong23: aha, I understand, I just assumed that the gluster volume wouldn't start because the metadata wouldn't be there
18:55 andreask joined #gluster
18:55 y4m4 joined #gluster
18:55 chouchins if we update the glusterfs server nodes to 3.3.1 from 3.3.0 will the clients running 3.3.0 be ok ?
18:58 JoeJulian chouchins: You would if the 3.3.0 clients would reconnect to a server after it's gone down. It doesn't do that reliably though.
18:58 hateya joined #gluster
18:58 chouchins Hmm ok, we'll have to hit them all at once then
18:59 JoeJulian chouchins: In theory, if you made a volume setting change that affected the client, it should cause it to reconnect.
19:08 haidz johnmark, i asked dave thomas to set something up.. i think i specified the wrong folks to talk to though
19:15 johnmark haidz: oh? ok
19:16 elyograg fedora 18 has a new firewall.  took me a while to figure out how to turn it off.
19:16 DaveS joined #gluster
19:17 elyograg "chkconfig firewalld off" -- which forwarded to systemctl.
19:17 _Bryan_ Just to check:  If I am replacing a server in a replicated setup...I can build the new server..get the UUID for the dead server from the other glsuter sustems and put in the /etc/glusterd/glsuterd.info file...start up the server and trigger a self heal and it will copy all the files over...correct?
19:18 danci1973 joined #gluster
19:18 _Bryan_ Do I need to copy over the peers files as well?
19:19 _Bryan_ Oh..and this is 3.2.5
19:19 twx_ elyograg: what? new firewall?
19:19 twx_ :o
19:20 elyograg twx_: firewalld instead of iptables.  i installed system-config-firewall-tui and it didn't work.
19:21 andreask1 joined #gluster
19:21 andreask joined #gluster
19:21 twx_ gee, haven't even had time to become good friends with systemctl yet
19:22 JoeJulian elyograg: That was there in 17 as well, btw...
19:22 sjoeboo_ joined #gluster
19:23 elyograg hmm.  i had a F17 setup, thought the tui command worked.
19:23 JoeJulian @replace
19:23 glusterbot JoeJulian: (replace [<channel>] <number> <topic>) -- Replaces topic <number> with <topic>.
19:23 JoeJulian @replace server
19:23 glusterbot JoeJulian: Error: 'server' is not a valid topic number.
19:23 JoeJulian @whatis replace
19:23 glusterbot JoeJulian: Useful links for replacing a failed server... if replacement server has different hostname: http://goo.gl/4hWXJ ... or if replacement server has same hostname:
19:23 glusterbot JoeJulian: http://goo.gl/rem8L
19:23 _Bryan_ Thanks glsuterbot/JJ
19:24 JoeJulian _Bryan_: Just refreshing my memory. The second one is the way I think I remember doing it.
19:26 elyograg latest distro-sync on F18 downgraded over 100 packages.
19:26 _Bryan_ That worked...thanks..
19:27 JoeJulian cool
19:27 JoeJulian downgraded? Now you've got me wondering...
19:28 _Bryan_ nope....not gone to 3.3
19:28 _Bryan_ a little scared to be honest...I have to drop my entire DR setup..in order to upgrade
19:28 JoeJulian Sorry, _Bryan_, that was to elyograg.
19:28 _Bryan_ over 300TB
19:28 _Bryan_ 8-)
19:29 elyograg JoeJulian: including the kernel - 3.6.11 to 3.6.10.
19:31 kkeithley elyograg: glusterfs in f18 should be 3.3.1-4
19:31 kkeithley er, sorry, 3.3.1-3
19:32 elyograg kkeithley: yes.  I first tried your repo, which didn't work. :)
19:33 elyograg my brick servers are centos, these will be providing network access to the volume(s).
19:33 kkeithley I don't have 3.3.1 f18 rpms in my repo because they're in the f18 proper
19:34 H__ joined #gluster
19:34 gbrand_ joined #gluster
19:34 kkeithley It's a HekaFS thing ;-)
19:35 kkeithley Otherwise I'd have 3.3.1 RPMs in all versions of Fedora (and EPEL)
19:36 JoeJulian kkeithley: I don't know if you've been following the "Rethinking EPEL at FUDcon Lawrence 2013" thread, but it sounds like they don't care if you break enterprise anymore.
19:37 kkeithley JoeJulian: I have not, but that's good to know.
19:41 kkeithley Although I never broke enterprise. I was shipping glusterfs for EPEL before RHES came out, and RHES is the one that released RPMs with conflicting names.
19:43 JoeJulian No, but back when daMaestro created the package, and I started trying to become the maintainer, the reason newer (actually usable) versions couldn't go in was because they broke both existing installs.
19:44 JoeJulian (both, as in the 2 people that were still using the package)
19:44 JoeJulian (ok, I'm being facetious...)
19:48 bitsweat joined #gluster
19:54 kkeithley I never heard that that was a criteria. The only grumbling I ever heard was that RHES RPMs, which are available in the RHEL Storage channel (for those who know about RHEL channels and such) had the same names as the Fedora/EPEL RPMs; as such someone who had both the Storage channel and EPEL would have two conflicting sources of Gluster RPMs.
19:56 _Bryan_ When a native mounted client drops a brick due to loss of connectivity....how/when does it add that brick back into its pool?
19:57 _Bryan_ I have a client that still does not see the brick I replaced..but if I go to a new client and mount then everything is working as it should
20:01 bdperkin joined #gluster
20:04 JoeJulian _Bryan_: When working correctly, within 3 seconds.
20:04 _Bryan_ not what I wanted to hear....but at least it confirms what I thought was supposed to happen
20:05 JoeJulian You could make a change to the volume that affects the client graph. That would probably cause it to reconnect.
20:05 JoeJulian At least it will if it's the same bug that was affecting 3.3.0.
20:10 _Bryan_ for isntance...what attribute?
20:13 JoeJulian client.io-threads
20:14 * JoeJulian is switch shopping...
20:14 JoeJulian These switch vendors don't even try to compete against Arista in that price range. It's out of my league, but still...
20:17 H___ joined #gluster
20:18 elyograg I love Cisco switches.  There is most definitely a price premium to them, though.
20:21 JoeJulian Cisco Nexus 3000 switches tout "Sub-microsecond" latency. Arista:380ns
20:23 JoeJulian And the price for the Arista is roughly a quarter of the Cisco.
20:23 * JoeJulian should probably stop drooling and start looking in his actual price-range.
20:24 semiosis JoeJulian: buy used equipment
20:24 semiosis then you can get last years new hawtness in your budget
20:24 semiosis i got some nice extreme network switches for the office that do everything i need for probably 1/10 the price they cost new
20:25 semiosis and they have a 10 year warranty that still has like 6 years left
20:25 chouchins we're a dell shop so its easy for us
20:26 jdarcy People used to pay huge premiums to get the latest greatest car that was only marginally better than last year's.  Apparently those dynamics now apply to switches.
20:26 semiosis jdarcy: yep
20:27 semiosis when i was buying these extreme's new stuff was like 10Gbps full bisection blah blah blah when all i needed was a decent 100Mbit with gig trunk/uplink and good L3 capability, so i got a great deal
20:30 semiosis now if i could just hire a developer who knows wtf L3 means i'd be all set
20:33 dbruhn so? I know this has probably been hashed to death. I have some extremely high CPU usage from glusterfsd? Is that the fuse client side driver?
20:34 jdarcy Glusterfsd is the server side.
20:35 JoeJulian @processes
20:35 dbruhn weird I am seeing peaks of like 409% in top right now
20:35 glusterbot JoeJulian: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
20:35 dbruhn Awesome, that helps.
20:39 JoeJulian jdarcy: wrt 4034 and Pranith's requested chante: If it didn't succeed it's reset? I'm confused.
20:39 JoeJulian s/chante/change/
20:40 glusterbot JoeJulian: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
20:41 JoeJulian Wouldn't that mean if it fails then it's just reset?
20:44 m0zes semiosis: ew extreme...
20:45 semiosis m0zes: whys?
20:45 semiosis whys that?
20:47 m0zes semiosis: just bad experience with them in the past. I couldn't keep a telnet session open to them. multicast would drop 90% of the packets. weird port/ip filtering (I couldn't I couldn't pass packets on the prod side addressed to anything in the management sides subnet)
20:48 semiosis heh that does sound bad
20:48 semiosis i've not had any issues like that
20:48 m0zes not quite worded right. two separate networks, same subnet range, the switch would filter packets out of the prod side of the network if it thought they /might/ belong in the management side.
20:49 semiosis i understood that :)
20:49 semiosis and it does sound bad!
20:49 bitsweat left #gluster
20:49 andreask joined #gluster
20:50 m0zes they were annoying enough with those issues that I won't touch them anymore. My uni switched to extreme briefly, and quickly got bit by the same multicast issue.
20:50 semiosis only issue i ran into was trying to do wifi ap management, works ok with one ap connected, but when I connected a second ap the two aps started taking turns resetting every 5 min making wifi unusable
20:50 m0zes multicast is really nice if you are wanting to image entire labs from one 'master'
20:51 semiosis i guess the lesson to take away is that extreme is ok as long as the switch is more extreme than your needs
20:51 semiosis my needs being not extreme, but rather mundane, are a good fit for extreme network switches :)
20:52 m0zes yeah. it may be that they've fixed those issues, but I couldn't wait for them to do it :)
21:23 dbruhn why would my system export my volume on all but one of the servers
21:29 semiosis dbruhn: glusterd stopped?
21:29 haidz when you mount volumes, is there a way to specify a directory underneath?
21:30 haidz rather than the root of the volume
21:30 semiosis haidz: not sure about nfs, but for fuse clients no
21:30 haidz for native
21:30 haidz hrm ok
21:31 semiosis feel free to file a bug requesting that feature though... there may even already be one you can add your support for
21:31 glusterbot http://goo.gl/UUuCq
21:31 haidz ok cool
21:31 haidz maybe ill do a combination of lvm to create multiple expandable volumes
21:31 haidz are there issues with expanding a gluster volume online with lvm?
21:32 semiosis afaik that works great
21:32 haidz ok cool.. ill give that a goi
21:32 semiosis the problems happen if you wait too long and your bricks fill up
21:32 semiosis no fs likes a full disk
21:33 haidz yeah
21:33 semiosis of course, try things out in a staging/test environment before you go to production
21:34 _Bryan_ ok so today is not my day,....
21:34 haidz semiosis, for sure.. i just ran in to this issue in staging.. so now i need to recreate with lvm :/
21:34 _Bryan_ now I have one users whose home directory is all ????? and input/output error when i try to go into it...
21:34 _Bryan_ any ideas what would cause this?
21:34 haidz _Bryan_, missing brick?
21:34 _Bryan_ nope...
21:35 semiosis _Bryan_: logs logs logs... client & brick logs
21:35 _Bryan_ checked all of them...
21:35 _Bryan_ logs tell only give alot of "pure path INFO statements
21:35 _Bryan_ at the prompt I get the input/output error
21:35 dbruhn semiosis: nope running
21:36 _Bryan_ It has to be in gluster since I get the error on any client that mounts the main volume...this is jsut one dir down in the path
21:36 dbruhn and I have the system up and running via the fuse client, I am just having some performance issues, so I was going to test NFS and see if it is any better
21:37 _Bryan_ the dir is just completely skipped in a self heal as well
21:41 haidz semiosis, im going to create a feature request for this
21:42 badone_ joined #gluster
21:45 JoeJulian _Bryan_: Mount the volume to a different directory and try accessing those files.
21:46 JoeJulian If nothing else, it'll give you a clean client log to post.
21:46 cicero joined #gluster
21:46 badone joined #gluster
21:47 lh joined #gluster
21:47 lh joined #gluster
21:47 _Bryan_ I did...on acompletely diff host...
21:47 _Bryan_ and same issue...
21:47 _Bryan_ You will love this..
21:47 _Bryan_ I compared a replica pair and they were different..
21:47 _Bryan_ so I shut down a galuster daemon and moved the dir with the issue out of the way...to re replicate it over...
21:48 _Bryan_ the restarted glsuter...
21:48 _Bryan_ then I was able to self heal and it was fine....but then the issue moved to two otehr dirs...
21:48 _Bryan_ so I am slowly steping through them...
21:48 _Bryan_ New Error:  [2013-01-07 13:48:32.506871] I [dht-layout.c:192:dht_layout_search] 0-gfs-vol0-dht: no subvolume for hash (value) = 3877369762
21:49 JoeJulian Did you do a remove-brick?
21:49 _Bryan_ nope...
21:49 _Bryan_ nothng changed...jsut the one dir went off line...
21:49 _Bryan_ I only noticed because a users could not log into thier NX session
21:49 _Bryan_ none of my monitoring checks picked anything up
21:50 _Bryan_ ok third round....this time a single dir is messed up...but the two before it are good again
21:50 _Bryan_ I am thinking about fully replacing this brick I am moving the dirs on...
21:51 _Bryan_ as it seems isolated to it...just very odd that no real error was reported
21:51 JoeJulian Check the xattrs for a directory that's being a pain. Figure out which one should have 0xe71bf7a2. That's the problem brick(s).
21:52 JoeJulian getfattr -n trusted.glusterfs.dht -e hex for the directory will show the range.
21:53 JoeJulian http://joejulian.name/blog​/dht-misses-are-expensive/ has a lot of the details
21:53 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
21:54 JoeJulian You didn't set lookup-unhashed off I presume.
21:54 _Bryan_ nope
21:54 _Bryan_ what is .landfill it just showed up
21:55 JoeJulian Has something to do with deleted files.
22:04 twx_ hmm.. i see subvolume nfs export-options.. can one selectively set export options for glusterfs native client on subvolume (directory?) level
22:05 sjoeboo_ joined #gluster
22:07 twx_ what I wonder is, can i set diff ip-auth perms for glusterfs:/gvol01/dir{1,2} ?
22:08 twx_ or should I just create separate gluster volumes?
22:09 JoeJulian separate gluster volumes
22:09 JoeJulian Be aware that each brick reserves it's own cache so play accordingly.
22:09 twx_ oh
22:09 twx_ ty
22:10 twx_ set size cache or computed size cache ?
22:11 JoeJulian For a 16gig server with 62 bricks, I set performance.cache-size 8MB.
22:12 twx_ oh
22:12 twx_ default is 32 right?
22:13 JoeJulian I was going to be all snarky and direct you gluster volume set help, but it's not in there. :(
22:14 JoeJulian I'm pretty sure I remember 32 being the default, yes.
22:14 _Bryan_ I think I am running 64MB
22:14 twx_ :-)
22:15 _Bryan_ OH.....I think I understand whatyou are doing now...8MB x 62 bricks = 496MB of cache...
22:16 JoeJulian Yep
22:16 _Bryan_ hmm....woudl that improve performance....if I split my brick smaller so I could go over the max cache allowed for a single brick because it would have muliple caches?
22:16 cicero wow python's ctypes looks badass.. didn't even know you could do that
22:17 JoeJulian hehe, I didn't really either before I wrote that.
22:17 JoeJulian I mean I did, but I didn't realize how easy they made it.
22:17 cicero very cool.
22:17 JoeJulian _Bryan_: Not really sure. I backed into doing it this way.
22:20 JoeJulian I knew going into using gluster that I wanted bricks to be per-disk. I didn't want anything striped, don't need massive speed, and since fault tolerance was my priority, it made no sense to use mirroring.
22:21 JoeJulian So from there, I set up 4 bricks per server, one on each disk.
22:21 jayunit100 joined #gluster
22:21 jayunit100 anyone know how to use systemtcl to figure out why glusterd won't start?
22:22 jayunit100 http://pastebin.com/raw.php?i=gQAEpFWf
22:22 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
22:22 JoeJulian systemctl status glusterd
22:23 jayunit100 systemctl status glusterd.service did it for me.  but I'm not sure what the information (re: above paste bin) is telling me.
22:23 JoeJulian Oh, right... I always forget that the first time I type that damned command.
22:25 twx_ may I ask why the per disk brick policy?
22:26 jayunit100 twx_: is that directed at me?
22:26 JoeJulian That's the rest of that paragraph. ;)
22:26 twx_ anyone who has that setup basically, JoeJulian mentioned it
22:27 twx_ yes but why not traditional raid on disk on each server
22:27 JoeJulian Worst case scenario, I want a drive I can send to a clean room and get data back.
22:27 JoeJulian raid's striped or mirrored. I don't like stripes.
22:28 * JoeJulian waits for semiosis to add some smart-ass comment to that and is disappointed.
22:28 semiosis o_O
22:28 twx_ was thinking more like raid5ish disk backend and gluster on top of that in a distributed manner
22:28 JoeJulian twx_: A lot of people do that.
22:29 semiosis me?  smart-ass?
22:29 JoeJulian In a good way.
22:29 twx_ so what, you have replica=2? that would basically use the same amount of disk as a raid10 if i got it right?
22:29 semiosis :)
22:29 twx_ or just not that bothered about losing data maybe
22:29 twx_ :>
22:29 JoeJulian twx_: Right, it would (except I do replica 3).
22:29 twx_ oh.
22:29 twx_ :)
22:30 JoeJulian So with my replica 3 I have the same overhead as raid5 with replica 2.
22:32 JoeJulian And I don't have stripes. I don't run the risk of losing the raid if 2 drives fail (replica mitigates that worry though). I have "superior fault isolation" as jdarcy so eloquently put it.
22:35 JoeJulian Then I had a drive fail and put my design to the test. Everything went as planned until the vm images and innodb data started to self-heal. Back in 3.0 these all happened simultaneously. Two hours later, I could access the files again.
22:38 JoeJulian That's when I decided to split up my volumes. Now if a disaster happens (your entire network goes down due to some unidentified environmental issue), you bring one server back online and let your users use it. As priority demands and usage allows, return individual volumes to operation on the other servers, allowing them to complete their self-heal before reactivating the next one.
22:39 JoeJulian And if jdarcy gets bug 802417 fixed, that'll go even smoother the next time.
22:39 glusterbot Bug http://goo.gl/V2r2t high, high, ---, jdarcy, ASSIGNED , [glusterfs-3.3.0qa27]: gfid change in the backend leading to EIO
22:42 andreask joined #gluster
22:44 andreask1 joined #gluster
22:49 jayunit100 left #gluster
22:55 cicero joined #gluster
22:59 johnmark JoeJulian: sounds like a blog post :)
22:59 JoeJulian I've thought about it...
23:00 JoeJulian Every time I do I end up doing some other topic.
23:00 JoeJulian I've actually got that post started.
23:00 andreask joined #gluster
23:01 andrewbogott joined #gluster
23:02 andrewbogott I have some runaway log files that are full of nonstop errors like these two:
23:03 andrewbogott 0-deployment-prep-project-server: 814427505: LINK /apache/common-local/p (1821982b-90f2-43c5-b7dd-9a261bb1c4c2) ==> -1 (File exists)
23:03 andrewbogott (-->/usr/lib/glusterfs/3.3.0/xlator/de​bug/io-stats.so(io_stats_lookup+0x13e) [0x7f84ffaf9c5e] (-->/usr/lib/glusterfs/3.3.0/xlator/fe​atures/marker.so(marker_lookup+0x300) [0x7f84ffd0fd90] (-->/usr/lib/glusterfs/3.3.0/xlator/f​eatures/marker.so(mq_req_xattr+0x3c) [0x7f84ffd1956c]))) 0-marker: invalid argument: loc->parent
23:03 andrewbogott Is that problem familiar to anyone?
23:04 JoeJulian Not familar, but are you using geo-replicate?
23:04 andrewbogott I don't know what that is, so I don't think so.  Three replications on boxes in the same data center.
23:08 raven-np joined #gluster
23:09 JoeJulian I find it funny when I search for a solution and fine that it's mine.
23:10 JoeJulian http://irclog.perlgeek.de/g​luster/2012-10-19#i_6079428
23:10 glusterbot <http://goo.gl/MYEeP> (at irclog.perlgeek.de)
23:10 JoeJulian "gluster volume set $vol geo-replication.indexing off"
23:13 andrewbogott JoeJulian, should it appear in 'volume info' if it's on?
23:13 JoeJulian I don't think so.
23:14 JoeJulian And it is in the graph or it wouldn't be throwing errors. :D
23:15 duffrecords joined #gluster
23:16 andrewbogott I take it 'geo-replication' is different from 'replication' somehow?  What will I lose from turning it off?
23:16 andrewbogott (And, I have 100+ volumes running with the same settings, only this one throwing all those errors.)
23:16 JoeJulian Yes, geo-replication is a one-way replication to a remote location.
23:17 andrewbogott Is this something that another team member would've turned on on purpose, or is it on by default?
23:18 hattenator joined #gluster
23:19 cicero joined #gluster
23:20 JoeJulian Hmm, it does seem to be in the graph by default...
23:21 andrewbogott s'pose I need to restart the volume now?  The log file is just as noisy as before.
23:21 JoeJulian You could probably get away with just restarting that brick.
23:22 JoeJulian @processes
23:22 glusterbot JoeJulian: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
23:22 JoeJulian kill the glusterfsd for that brick, then "gluster volume start $vol force" to start it again.
23:26 andrewbogott Hmph.  Restarted the whole volume, still getting log spam.
23:26 andrewbogott Is it possible for me to query the setting to see if geo-replication is on still?
23:26 JoeJulian info should show it
23:27 andrewbogott gluster volume info <volume>?  It doesn't, or at least not by that name.
23:28 andrewbogott Oh, my mistake, it does:  geo-replication.indexing: off
23:28 andrewbogott back in 20...
23:29 cyr_ joined #gluster
23:48 akenney joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary