Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-01-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 mattappe_ joined #gluster
00:05 jbrooks joined #gluster
00:07 psyl0n joined #gluster
00:12 mattappe_ joined #gluster
00:21 dbruhn joined #gluster
00:29 purpleidea johnmark: i can't clear them for you, you have to use JMWbot: @forget <id>
00:30 purpleidea johnmark: the JMWbot works for you!
00:36 psyl0n joined #gluster
00:37 TrDS left #gluster
00:44 harish joined #gluster
00:45 mattappe_ joined #gluster
00:48 aixsyd joined #gluster
00:49 aixsyd Sup bros - stupid question, i think - theres no FUSE for windows server is there?
01:04 mattappe_ joined #gluster
01:12 _pol joined #gluster
01:12 lyang0 joined #gluster
01:13 theron joined #gluster
01:15 gmcwhistler joined #gluster
01:19 psyl0n joined #gluster
01:20 psyl0n joined #gluster
01:24 harish joined #gluster
01:27 _pol joined #gluster
01:29 mattappe_ joined #gluster
01:37 mattappe_ joined #gluster
01:38 theron joined #gluster
01:51 mattapperson joined #gluster
01:52 mattappe_ joined #gluster
01:56 mattapperson joined #gluster
02:00 micu joined #gluster
02:01 glusterbot New news from newglusterbugs: [Bug 1052759] The default value from help for option 'self-heal-daemon' is incorrect. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1052759>
02:03 bala joined #gluster
02:09 mattapperson joined #gluster
02:09 dbruhn__ joined #gluster
02:14 mattapperson joined #gluster
02:18 gmcwhistler joined #gluster
02:22 mattapperson joined #gluster
02:27 r0b joined #gluster
02:30 bala joined #gluster
02:34 mattappe_ joined #gluster
02:37 mattappe_ joined #gluster
02:42 theron joined #gluster
13:41 ilbot3 joined #gluster
13:41 topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
13:49 vpshastry joined #gluster
13:55 mattappe_ joined #gluster
14:02 mattappe_ joined #gluster
14:02 aixsyd joined #gluster
14:03 aixsyd semiosis: i hate life :(
14:04 aixsyd semiosis: my cluster is having terrible read and writes - sub MB/s D:
14:07 japuzzo joined #gluster
14:09 ira joined #gluster
14:14 sroy_ joined #gluster
14:15 jag3773 joined #gluster
14:16 r0b joined #gluster
14:16 robo joined #gluster
14:17 calum_ joined #gluster
14:22 mattappe_ joined #gluster
14:28 vpshastry joined #gluster
14:29 nshaikh joined #gluster
14:33 psyl0n joined #gluster
14:35 mattapperson joined #gluster
14:37 aixsyd yeah, these operating speeds are abysmal. a debian installatio took 4.5 hours last night
14:38 T0aD lol
14:39 aixsyd this is seriously bad.
14:40 aixsyd somethings messed up badly. and i dont know where to even begin v.v'
14:44 cfeller joined #gluster
14:44 skered aixsyd: Is this inside a VM?
14:45 aixsyd skered: yessir
14:51 sroy_ joined #gluster
14:52 bennyturns joined #gluster
14:54 aixsyd figured it out - its most definitely the server that i'm using as a hypervisor - i tried the same install on another hypervisor server and its speedy as fsck.
15:00 theron joined #gluster
15:06 jobewan joined #gluster
15:12 klaas joined #gluster
15:18 TrDS left #gluster
15:20 KORG|2 joined #gluster
15:20 ndk joined #gluster
15:21 yinyin joined #gluster
15:21 vpshastry left #gluster
15:23 zerick joined #gluster
15:23 jbrooks joined #gluster
15:32 jag3773 joined #gluster
15:34 bugs_ joined #gluster
15:36 khushildep joined #gluster
15:37 doc|holliday joined #gluster
15:38 jclift joined #gluster
15:39 Technicool joined #gluster
15:42 kkeithley_ http://www.gluster.org/community/docu​mentation/index.php/Backport_Wishlist  is open for backport requests for 3.3.3 and 3.4.3. No dates set yet, but I'm guessing mid-February for 3.3.3 and 3.4.3 will follow. Don't delay, get your requests on the lists ASAP.
15:42 glusterbot Title: Backport Wishlist - GlusterDocumentation (at www.gluster.org)
15:43 jbrooks joined #gluster
15:43 Philambdo joined #gluster
15:46 _Bryan_ joined #gluster
15:51 japuzzo_ joined #gluster
15:51 shubjero joined #gluster
15:51 shubjero anyone here at the red hat data protection tour in toronto?
15:54 doc|holliday I've got a small gluster (v3.3.1) setup with 20 servers with 5TB each. distributed only. need to search through few thousand files. started off with 'find' -- it was painfully slow. then wrote my own app, which using hints is able to show most relevant files first. googled some and found out that stat() is expensive, removed it from my app, but that didn't seem to help. looking for other suggestions
15:57 doc|holliday s/5TB/4TB/
15:57 glusterbot What doc|holliday meant to say was: I've got a small gluster (v3.3.1) setup with 20 servers with 4TB each. distributed only. need to search through few thousand files. started off with 'find' -- it was painfully slow. then wrote my own app, which using hints is able to show most relevant files first. googled some and found out that stat() is expensive, removed it from my app, but
15:57 glusterbot that didn't seem to help. looking for other suggestions
15:57 doc|holliday heh, thanks glusterbot
15:58 XATRIX left #gluster
15:58 shubjero did you try locate?
16:00 purpleidea shubjero: tell me about this data protection in toronto thing
16:01 ndevos doc|holliday: pmux might be of your interest: https://forge.gluster.org/pmux
16:01 glusterbot Title: pmux - Gluster Community Forge (at forge.gluster.org)
16:01 shubjero purpleidea: its a one day talk/conference... seems to be an intro to RH storage server... which is backed by glusterfs
16:01 shubjero http://redhatstorageworkshops.com/
16:01 glusterbot Title: Red Hat Storage Modern Data Protection Tour | Red Hat (at redhatstorageworkshops.com)
16:02 shubjero im here, just trying to get through the marketing stuff :)
16:02 ndevos I love the simple example from the docs - https://forge.gluster.org/pmux/pages/Home - a distributed grep :)
16:02 glusterbot Title: pmux - Home - Open wiki - Gluster Community Forge (at forge.gluster.org)
16:03 purpleidea shubjero: weird! i'm in toronto right now. anyways don't need to attend. you there?
16:03 shubjero yeah
16:04 daMaestro joined #gluster
16:06 purpleidea shubjero: any good?
16:06 sroy_ joined #gluster
16:06 doc|holliday shubjero: locate won't work cuz new files get constantly created
16:07 shubjero purpleidea: still early, cant say just yet.. havent got to the meat and potatoes yet
16:08 purpleidea shubjero: yeah it says lunch is included.
16:08 shubjero lol
16:08 purpleidea shubjero: are you new to gluster?
16:08 shubjero yeah
16:08 purpleidea shubjero: are you running a fedora machine or do you have access to one?
16:08 purpleidea (like fedora20)
16:08 shubjero i just rolled out two centos 6.5 vm's and installed gluster on it
16:09 purpleidea shubjero: if you're interested, i've recently released a cool (easy) new way to try out glusterfs:
16:09 purpleidea ~vagrant | shubjero
16:09 glusterbot shubjero: (#1) Part 1 @ https://ttboj.wordpress.com/2013/12​/09/vagrant-on-fedora-with-libvirt/, or (#2) Part 2 @ https://ttboj.wordpress.com/2013/12​/21/vagrant-vsftp-and-other-tricks/, or (#3) Part 3 @ https://ttboj.wordpress.com/2014/01/​02/vagrant-clustered-ssh-and-screen/, or (#4) Part 4 @
16:09 glusterbot https://ttboj.wordpress.com/2014/0​1/08/automatically-deploying-glust​erfs-with-puppet-gluster-vagrant/
16:09 shubjero i was following the guides on gluster.org
16:09 shubjero cool thanks
16:10 purpleidea shubjero: well if you have a look at what i've published, i'd love to hear your feedback. i'm the puppet-gluster guy, and now the glusterfs vagrant guy, but both are just tools to make it easy to try out/deploy glusterfs
16:10 shubjero awesome, thanks
16:11 purpleidea shubjero: np. i'd appreciate it if you pass out those links (#4 is sufficient) and ask those people to try and send me feedback. makes it pretty easy to deploy a glusterfs test cluster, but feedback will let me know how well it went for you
16:14 davidjpe1cock purpleidea++
16:14 purpleidea davidjpeacock: you tried it?
16:15 purpleidea (sadly the bot doesn't support karma++ commands)
16:15 davidjpeacock purpleidea: nope but I will; very glad to see you did this - thank you
16:15 davidjpeacock It's ok - I like karma for karma's sake ;-)
16:16 purpleidea davidjpeacock: my pleasure, please feel free to pass out the links and let me know you feedback on how it went
16:16 purpleidea s/you/your/
16:16 glusterbot What purpleidea meant to say was: davidjpeacock: my pleasure, please feel free to pass out the links and let me know your feedback on how it went
16:19 yinyin joined #gluster
16:19 davinder joined #gluster
16:22 doc|holliday ndevos: that looks interesting... I am really only just listing the files, don't need to grep through them, but I might be able to make something out of it
16:24 plarsen joined #gluster
16:29 doc|holliday it looks like gflocator & msgpack-rpc might be all I need
16:31 doc|holliday but in terms of gluster itself are there any other optimizations I can do to speed it up?
16:31 semiosis run it on infiniband
16:33 psyl0n joined #gluster
16:33 purpleidea or bonded 10gE
16:37 doc|holliday too late for that :)
16:37 Philambdo joined #gluster
16:39 doc|holliday I guess I will have to do something similar to gflocate
16:40 doc|holliday which brings me to another question: if I delete files directly from the underlying brick, am I going to break something?
16:40 purpleidea doc|holliday: yes (simple answer)
16:42 _pol joined #gluster
16:42 theron joined #gluster
16:43 doc|holliday purpleidea: :) is there a "careful" way of doing it? I am basically trying to run a daemon on each brick to auto purge oldest files to keep the brick filled at 70-80%
16:47 purpleidea doc|holliday: i think you'll need to give me more information on what you want to do...
16:47 semiosis doc|holliday: the standard advice is that you dont modify the bricks
16:47 semiosis except through a client mount
16:47 purpleidea you can get away with deleting files in the bricks sometimes, but that will change and it's hard to know all the edge cases.
16:48 purpleidea doc|holliday: listen to semiosis ^^^
16:50 doc|holliday ok I am tracking
16:50 shubjero           wont the brick self-heal if you delete from the underlying filesystem?
16:51 rotbeard joined #gluster
16:52 semiosis shubjero: probably
16:52 semiosis ...if it's a replicated volume
16:52 doc|holliday here is my scenario: I have an autonomous system (20 bricks x 4TB each, distributed only). new files get recorded periodically and as the bricks fill up, I need to purge the oldest files
16:53 semiosis ah, no replication, that might make things easier
16:53 purpleidea doc|holliday: you need to write a script that runs on a _client mount_
16:53 purpleidea doc|holliday: like 'tidy' (package manager search it)
16:53 doc|holliday is there a guarantee that the bricks will be filled evenly? if that's the case, I can work on the mount
16:53 semiosis but i forsee problems due to caching (in glusterfs process memory) of brick content
16:53 boomertsfx joined #gluster
16:54 semiosis you might be able to get away with this if you disabled all the performance xlators :/
16:54 semiosis but this is uncharted territory afaik
16:54 semiosis doc|holliday: the files are distributed evenly, yes.  files, not bytes.
16:55 doc|holliday schidt
16:55 semiosis doc|holliday: here's a solution...
16:56 purpleidea doc|holliday: you're not thinking about this right... let gluster ensure that there's room to store files... set a max capacity per brick, and just delete when the total volume is close to full...
16:56 semiosis it's a two-part system... 1. a search tool that crawls the bricks looking for "old" files, which exports its results to... 2. a deletion tool which deletes the files in the list via a client mount
16:56 semiosis that should perform better than doing the search through the client mount
16:56 purpleidea semiosis: you have to convince him to write this tool in java with jnio, etc :P
16:57 semiosis esp since the searches are parallel & distributed :D
16:57 semiosis purpleidea: lol
16:57 purpleidea ...and fund your work...
16:59 semiosis this could be done in a couple lines of bash using parallel, xargs, ssh, and find
16:59 doc|holliday sorry guys no java for me, not even ruby. bash and python is as far as I will go :)
16:59 semiosis doc|holliday: he was joking about java.  see my last line re bash
17:00 doc|holliday semiosis: I know :) I am tracking with your approach
17:02 doc|holliday here is my idea: run a daemon on each server, which monitors the brick and once it gets up to max capacity, figure out which files belong to this brick, temporarily mount the volume and nuke 'em
17:02 semiosis doc|holliday: sounds good to me
17:02 KORG joined #gluster
17:03 semiosis doc|holliday: "figure out which files belong to this brick" -- you can just run find in the brick with -mtime to get old files on that brick
17:04 doc|holliday semiosis: doesn't it also creates files / dirs that are on other bricks? I remember at least seeing dirs with funny size
17:04 semiosis here's a trick...   find /path/to/brick -mtime -10 -printf '/client/mount/path/%P\0' | xargs -0 rm
17:05 semiosis that produces paths in the client mount to files that are on the local brick which are older than 10 days
17:05 portante joined #gluster
17:05 semiosis and deletes the paths, through the client mount
17:06 jclift joined #gluster
17:07 LoudNoises joined #gluster
17:07 purpleidea semiosis: don't you want find -0 and no \0 ?
17:07 semiosis purpleidea: no
17:07 semiosis thats the trick, let me explain
17:08 semiosis consider... find /path -printf '%P'
17:08 semiosis and you have a file /path/foo
17:08 semiosis that find command will print /foo
17:09 semiosis the %P prints the subpath under the find target path
17:09 semiosis so you can prepend that with the client mount path
17:09 semiosis then the \0 gives you null terminated paths to pass into xargs -0
17:10 doc|holliday semiosis: doesn't gluster create the dirs that are not on this brick but are on other bricks? can I delete those?
17:10 purpleidea i agree with all that, but why doesn't the -0 do that with find in this scenario?
17:10 semiosis purpleidea: well for all i know it might, i just found a way it works using -printf and went with that
17:11 doc|holliday or -print0 '%P'
17:11 semiosis doc|holliday: don't delete dirs, only files
17:11 semiosis keeps it simpler
17:11 semiosis and never delete dirs from bricks directly, that can cause big problems with distributed volumes
17:12 doc|holliday semiosis: there is a new dir for each day, inside it about 200 dirs (categories), inside which are the files
17:12 doc|holliday after a year I will have 6000 empty dirs :)
17:13 purpleidea doc|holliday: delete those from the client mount when empty
17:13 semiosis doc|holliday: -print0 does not take an argument.  see man find(1)
17:13 semiosis doc|holliday: well then prune dirs too, but i'd recommend using rmdir to be safe
17:15 doc|holliday purpleidea: semiosis: oh yeah, find doesn't have -0 (xargs does). find's equivalent is -print0, so it would be: -prints '%P' -print0, but semiosis' example ('%P\0') is shorter
17:16 doc|holliday s/-prints/-printf/
17:16 glusterbot What doc|holliday meant to say was: purpleidea: semiosis: oh yeah, find doesn't have -0 (xargs does). find's equivalent is -print0, so it would be: -printf '%P' -print0, but semiosis' example ('%P\0') is shorter
17:16 semiosis neat
17:17 purpleidea doc|holliday: semiosis: yeah, this! sorry i retract my find -0 comment. my brain is not on. goodnight
17:17 semiosis lol.  g'nite purpleidea
17:17 doc|holliday :) later
17:18 doc|holliday semiosis: if you don't mind I will bug you one more time
17:18 Liquid-- joined #gluster
17:19 TvL2386 joined #gluster
17:20 doc|holliday so I am looking at the brick, it will have dirs from this brick as well as "fake" dirs from other bricks. is there a way to distinguish between the two?
17:20 semiosis every brick has the whole directory tree
17:20 semiosis nothing fake about them
17:21 semiosis the directories (not files) are on all bricks, period.
17:22 doc|holliday oh ok
17:24 jclift joined #gluster
17:26 Liquid-- joined #gluster
17:27 jclift joined #gluster
17:28 diegows joined #gluster
17:30 doc|holliday huh, I can just run rmdir -p (via client mount) on the dirs that I removed files from
17:32 doc|holliday thanks semiosis, I think I have enough data to make it work now
17:32 semiosis there you go
17:32 semiosis yw
17:32 Mo_ joined #gluster
17:34 doc|holliday left #gluster
17:40 SFLimey joined #gluster
17:41 zaitcev joined #gluster
17:46 aixsyd heya semiosis
17:47 semiosis heyo
17:48 aixsyd if i'm running a VM hypervisor with gluster as vm storage and one Hypervisor works fine and the other is slowwwwwwwwwwwww as fsck - one would think its a hardware issue with the hypervisor - not the cluster, ya?
17:49 aixsyd i had a linux vm install take 4.5 hours last night, iotop showing no greater than 1MB/s speeds on either cluster node
17:49 aixsyd tried it today with a different hypervisor server hardware, same cluster, and got 15-30MB/s - which is acceptable
17:53 yinyin joined #gluster
17:54 sprachgenerator joined #gluster
17:59 daMaestro|isBack joined #gluster
18:06 semiosis aixsyd: hard to say
18:06 semiosis but sounds reasonable
18:13 SFLimey I rebuilding a Gluster cluster for our production environment at AWS, we currently host them on 2 x m2.xlarge instances. They run without issues. I'm planning to build the new cluster on 4 instances. Wondering if I should stick with m2.xlarge memory optimized instances or switch to the compute optimized instances that used ssd back storage? Anyone have experience with this?
18:15 Rocky__ joined #gluster
18:18 theron joined #gluster
18:20 tryggvil joined #gluster
18:25 qdk joined #gluster
18:26 Rocky__ joined #gluster
18:30 flrichar joined #gluster
18:31 daMaestro joined #gluster
18:41 boomertsfx joined #gluster
18:44 theron joined #gluster
18:54 yinyin joined #gluster
18:56 sroy_ joined #gluster
18:57 lpabon joined #gluster
18:57 boomertsfx joined #gluster
18:58 tryggvil joined #gluster
18:59 andreask joined #gluster
19:03 Liquid-- joined #gluster
19:03 dbruhn__ joined #gluster
19:05 sjoeboo joined #gluster
19:06 khushildep joined #gluster
19:16 _pol joined #gluster
19:24 boomertsfx I have a couple dozen files and directories that are split-brained... some of the files on the backend fs don't exist anymore but show up in the client directory listings (as ? ? ? w/ io error)... I'm assuming that there are still orphaned files in the .glusterfs backend node -- what's the best way to clean it up, preferably without taking the cluster down?
19:25 _pol joined #gluster
19:32 _pol joined #gluster
19:36 dbruhn__ boomertsfx, http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
19:36 glusterbot Title: Fixing split-brain with GlusterFS 3.3 (at joejulian.name)
19:38 boomertsfx saw that earlier, will give it a whirl
19:38 boomertsfx thanks
19:38 dbruhn__ It's not hard, and you don't have to take the cluster down to complete it.
19:39 boomertsfx we're doing heavy rsyncs to get the data to the cluster, saw that mentioned as a possible culprit... we're on 3.3.1
19:40 dbruhn__ You're rsyncing into the mount point right?
19:40 boomertsfx yeah via the native client
19:40 dbruhn__ kk
19:41 tryggvil joined #gluster
19:41 boomertsfx 99.999% works fine, but a few times a month we end up with horqued files
19:41 dbruhn__ How did you end up with split brain files already? And any reason you aren't on a newer version?
19:42 diegows joined #gluster
19:43 boomertsfx not sure why it happens initially... we're on centos 5 still... is it pretty easy to upgrade to 3.4.2?  i haven't looked yet
19:44 dbruhn__ Indeed it is, from what I've seen as feedback. I am honestly still on 3.3.2 on my systems. But I am stuck in RDMA land on a couple of them, and need to test 3.4.2 in my lab, haven't have time.
19:47 boomertsfx humm, I see a 3.3.1 change log, but not 3.3.2
19:47 dbruhn__ http://download.gluster.org/pu​b/gluster/glusterfs/3.3/3.3.2/
19:47 glusterbot Title: Index of /pub/gluster/glusterfs/3.3/3.3.2 (at download.gluster.org)
19:48 dbruhn__ There are a lot of people running 3.4.x in production, redhat is still on 3.3.x last I heard for redhat storage
19:48 jag3773 joined #gluster
19:52 sroy_ joined #gluster
19:53 lpabon_ joined #gluster
19:54 yinyin joined #gluster
19:58 aixsyd i'm running 3.5 in production >)
19:59 dbruhn__ aixsyd, do you still have that test cluster in place?
19:59 dbruhn__ with your IB?
20:00 aixsyd yep
20:00 aixsyd and i was kidding about 3.5 in production
20:00 dbruhn__ is that running 3.4.x?
20:00 aixsyd 3.4.1
20:00 dbruhn__ would you want to do me a huge favor, and try creating an RDMA volume on it, and try and mount it as a client to one of the servers?
20:01 aixsyd i heard RDMA sucked with gluster currently
20:01 dbruhn__ I've heard the same thing, mixed reviews on if it's working in 3.4.x
20:02 aixsyd and mount the volume on a 3rd server? or on itself
20:02 dbruhn__ on itself
20:02 dbruhn__ so it mounts on the RDMA
20:02 aixsyd i gotcha
20:02 aixsyd for you buddy, anything ;)
20:03 dbruhn__ I need to go rebuild my stuff, my lab equipment is a 45/60 min drive from my office, and needs fresh OS's loaded on it, it was built out for cloud stack previously
20:05 aixsyd gluster volume create gv0 replica 2 rdma
20:05 aixsyd right?
20:05 aixsyd oh crap
20:05 aixsyd my IB cards are set up for ipoib
20:06 aixsyd not sure how to get them in rdma
20:08 dbruhn__ they should work as both
20:08 dbruhn__ the ipoib runs over the top
20:08 dbruhn__ and yep that looks right
20:09 dbruhn__ and when you mount it's mount -t glusterfs server1:/gv0.rdma /mnt/gv0
20:09 aixsyd i got this
20:10 aixsyd http://fpaste.org/68398/13897302/
20:10 japuzzo joined #gluster
20:10 glusterbot Title: #68398 Fedora Project Pastebin (at fpaste.org)
20:11 dbruhn__ you probably need to peer probe back from the second server, it's seeing itself as localhost I bet
20:19 aixsyd already in peer list
20:19 aixsyd both ways
20:20 semiosis aixsyd: ,,(hostnames)
20:20 glusterbot aixsyd: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
20:20 semiosis you can check with 'gluster peer status' if you run that command on EVERY SERVER
20:20 aixsyd connected on both
20:23 dbruhn__ well then, looks like I have a bunch of work to do :)
20:23 dbruhn__ thanks for checking that for me
20:23 aixsyd apologies bro :(
20:23 dbruhn__ Nothng to be sorry about, I really appreciate you checking it out for me.
20:24 aixsyd ^_^
20:24 _pol joined #gluster
20:32 tryggvil joined #gluster
20:32 SFLimey_ joined #gluster
20:35 aixsyd dbruhn__:  whoa
20:35 dbruhn__ ?
20:35 aixsyd http://fpaste.org/68406/13897317/
20:35 glusterbot Title: #68406 Fedora Project Pastebin (at fpaste.org)
20:35 aixsyd this is on a cliient
20:35 dbruhn__ interesting
20:35 aixsyd dufuq?
20:36 dbruhn__ looks like split-brain
20:36 dbruhn__ on the root of the file system
20:36 aixsyd O_O
20:36 aixsyd heal info shows no split brain
20:36 dbruhn__ make sure the permissions are identical on all bricks, make sure the directories directly under the root are the same,
20:36 dbruhn__ and check the extended attributes
20:37 ekool joined #gluster
20:37 ekool Just setup a 2 master server of gluster and mounted it, running df the process is sitting there... been a few minutes.
20:37 psyl0n joined #gluster
20:38 dbruhn__ also make sure your /brick/.glusterfs/00/00/000000​00-0000-0000-0000-000000000001 is a soft link to ../../..
20:38 dbruhn__ on all bricks
20:40 aixsyd sec
20:40 dbruhn__ i've had /brick/.glusterfs/00/00/000000​00-0000-0000-0000-000000000001 turn into a file, and a directory before and cause this issue
20:43 aixsyd wtf
20:43 aixsyd /data/gv0/brick1 used to be a directory
20:43 aixsyd and now its a file
20:43 aixsyd cant access it, IO error
20:43 aixsyd same on both bricks
20:44 aixsyd how the christ does this just happen?
20:44 aixsyd what if this happens in production? both bricks are screwed? all data screwed?
20:45 dneary joined #gluster
20:45 boomertsfx dbruhn, I got the split brained files working, so now I'm left with the directories...  client says unable to heal perms/ownership of the dir .. backend ownerships and perms are the same on both backend fs
20:46 aixsyd volume start: gv0: failed: Failed to find brick directory /data/gv0/brick1 for volume gv0. Reason : Input/output error <--*RAGE*
20:50 dbruhn__ boomertsfx, you will need to make sure the directory is on all of the bricks, has the same permissions, and has the same directories under it. Once you have done that you will want to check and make sure the gfid is the same on all of the directories.
20:50 dbruhn__ aixsyd, that's weird, is this on the test system you just tried the RDMA stuff on?
20:50 aixsyd nope
20:51 aixsyd i have two clusters running, this is on the first. rdma was on the second
20:51 dbruhn__ kk, just wasn't sure if it would be related.
20:51 boomertsfx will check the gfid, I think everything else is the same
20:52 dbruhn__ boomertsfx, if that all checks out you will need to find the gfid in the .glusterfs directory and make sure it's a link, it's a bit of a process
20:53 dbruhn__ aixsyd, I've never seen gluster turn a directory into a file, the only times I've had issues is with corruption on a drive, that being said, very weird.
20:53 yosafbridge joined #gluster
20:53 dbruhn__ I am assuming you have server1:/data/gv0/brick1 as your brick?
20:53 dbruhn__ or is it gv0?
20:54 aixsyd brick1 is the brick
20:54 aixsyd volume start: gv0: failed: Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /data/gv0/brick1. Reason : No data available
20:55 dbruhn__ So the directories are now files?
20:55 yinyin joined #gluster
20:55 aixsyd wait a tic
20:56 dbruhn__ boomertsfx, here is a gfid resolver script https://gist.github.com/semiosis/4392640
20:56 glusterbot 'Title: Glusterfs GFID Resolver\r \r Turns a GFID into a real path in the brick (at gist.github.com)'
20:56 aixsyd sweeeeeet. /dev/sdb doesnt exist
20:56 aixsyd er
20:56 aixsyd /dev/sdb1
20:56 dbruhn__ so your file systems are not mounted
20:56 aixsyd and wont
20:56 aixsyd cant
20:56 aixsyd shows up as unallocated disk space now
20:57 aixsyd no partition table
20:57 aixsyd WTF HAPPENED.
20:57 dbruhn__ :/
20:57 aixsyd am i crazy, or is this more than a little bit crappy?
20:57 semiosis why not both?
20:57 aixsyd i like the cut of your jib
20:58 boomertsfx yeah I'm look at the pair of backend nodes and the dir listing in the gfid directory is totally different between machines
20:58 dbruhn__ no partition table on both servers?
20:58 semiosis boomertsfx: what version of glusterfs?
20:58 aixsyd correct
20:58 boomertsfx 3.3.1
20:58 badone joined #gluster
21:02 aixsyd i quit for today
21:03 aixsyd ive spent a month getting this working and i'm really no further than a month ago :(
21:03 dbruhn__ aixsyd, you should check those logs, it's weird that something would nuke the file system on both machines, being the data is replicant, it could be a raid issue or something.
21:03 kaptk2 joined #gluster
21:04 aixsyd i recreated the partition table and partitions and its working just fine now
21:04 dbruhn__ Sorry about the frustration, I know it can be a pain in the ass when this stuff isn't working.
21:04 T0aD or so you think
21:05 aixsyd T0aD: yep
21:06 aixsyd http://fpaste.org/68422/73356813/
21:06 glusterbot Title: #68422 Fedora Project Pastebin (at fpaste.org)
21:17 jskinner_ joined #gluster
21:17 bchilds joined #gluster
21:21 boomertsfx joined #gluster
21:23 TrDS joined #gluster
21:25 dbruhn joined #gluster
21:36 plarsen joined #gluster
21:46 sroy_ joined #gluster
21:49 31NAAC520 joined #gluster
21:55 sroy_ joined #gluster
21:56 yinyin joined #gluster
22:01 cfeller joined #gluster
22:02 Liquid-- joined #gluster
22:04 SpeeR joined #gluster
22:10 dalekurt joined #gluster
22:11 dalekurt joined #gluster
22:11 dalekurt joined #gluster
22:15 parad1se joined #gluster
22:19 tryggvil joined #gluster
22:34 dalekurt joined #gluster
22:40 boomertsfx how do you get the gfid with dashes if you know the file/dir name
22:40 semiosis boomertsfx: it's one of the ,,(extended attributes)
22:40 glusterbot boomertsfx: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
22:40 boomertsfx i can dump it with getfattr -e hex
22:41 boomertsfx but your scripts seems to use gfid with dashes?
22:41 boomertsfx the resolver one
22:41 semiosis hmm
22:41 boomertsfx which is what gluster puts in log files I guess
22:43 semiosis well
22:44 semiosis the resolver works like this.... you give it a "GFID" it searches for something with that name in /brick/path/.glusterfs
22:44 semiosis if it finds the thing, then...
22:44 semiosis if the thing is a dir, read the symlink targer
22:44 semiosis if the thing is a file, find hard links to that file in the brick
22:45 semiosis i suppose there's a transformation you can do to the GFID to go from UUID style formatting to hex
22:46 semiosis maybe it's as simple as removing the leading 0x and adding dashes?
22:46 semiosis that would be easy
22:57 yinyin joined #gluster
22:59 jporterfield joined #gluster
22:59 boomertsfx yeah that's what it is...  I just wanted to confirm I wasn't crazy... basically the volume heal info has a bunch of directories that claim to have ownership/permission issues but they all match
22:59 Liquid-- joined #gluster
22:59 boomertsfx on the replicated pairs
23:03 semiosis boomertsfx:  see ,,(split brain)
23:03 glusterbot boomertsfx: I do not know about 'split brain', but I do know about these similar topics: 'split-brain', 'splitbrain'
23:04 semiosis boomertsfx:  see ,,(split-brain) for info how to resolve
23:04 glusterbot boomertsfx: To heal split-brain in 3.3+, see http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/ .
23:04 semiosis hth, idk what else to recommend
23:04 boomertsfx yeah that worked for the split brain files, not the dirs
23:04 boomertsfx time to head home
23:04 boomertsfx heh
23:04 semiosis hm ok
23:04 boomertsfx thanks for all your help
23:04 semiosis yw
23:49 tryggvil joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary