Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-03-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 philv76 joined #gluster
00:15 velladecin thanks Joe and the Bot..
00:15 velladecin @ports
00:15 glusterbot velladecin: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
00:17 velladecin BTW, how does one find these 'at ports' and other commands?
00:25 tokik joined #gluster
00:27 Leolo ... has geo-replication been removed?  I have 3.4.0 installed and can't find it in the docs
00:31 Leolo @geo-replication
00:31 glusterbot Leolo: See the documentation at http://download.gluster.org/pub/gluste​r/glusterfs/3.2/Documentation/AG/html/​chap-Administration_Guide-Geo_Rep.html
00:34 Leolo that link is 404
00:35 Leolo forget geo-replication
00:35 JoeJulian Leolo: Did you install it? It's a separate package.
00:36 Leolo i see references to glusterfs-geo-replication
00:36 JoeJulian velladecin: /msg glusterbot help factoids search
00:36 Leolo but yum doesn't see those
00:36 Leolo (CentOS 6, btw)
00:37 JoeJulian @yum repo
00:37 glusterbot JoeJulian: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
00:37 JoeJulian That one?
00:39 jag3773 joined #gluster
00:41 tokik joined #gluster
00:42 tokik joined #gluster
00:46 Leolo nope
00:46 Leolo the CentOS one
00:48 Leolo thank you
00:57 yosafbridge joined #gluster
00:59 purpleidea semiosis: should we do a startup? we rack the servers in snowbanks, so cooling is free. unfortunately, so is frostbite!
00:59 cp0k joined #gluster
01:00 cp0k_ joined #gluster
01:02 Leolo if you wanted IaaS in Canada, I'd rent a rack from iWeb
01:02 JoeJulian "And coming up after the break, is an Alberta startup company responsible for global warming?"
01:03 Leolo ...seems iWeb claims to do IaaS
01:15 cp0k_ Hey guys, it would not surprise me if this question has been asked here dozens of time - is there any way to speed up an existing re-balance? I am open to doing anything possible to help Gluster out
01:21 harish joined #gluster
01:25 Alex Hey cp0k_. I believe the balance speed is affected by the priority of the balancer.
01:26 cp0k_ Alex: how do you mean that?
01:26 Alex Rebalance traffic is throttled into the least priority scheduling by default.
01:26 Alex oops, mispaste there.
01:26 Alex Have a look at http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented - specifically the bit I mispasted :)
01:26 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
01:27 cp0k_ huh?
01:27 cp0k_ ahh, I see now :)
01:28 JoeJulian cp0k_: Quick way, from a client mountpoint just do a find.
01:30 Alex That actually reminds me, I need to re-ask my question from a couple of days ago...
01:30 cp0k_ JoeJulian: doing a find will automatically trigger a heal, which will distribute the existing file onto the new bricks that have been already added in...correct?
01:30 JoeJulian Heal != rebalance.
01:31 Alex How does Gluster keep attrs like mtime in sync? I have an odd (repeated) issues where on different servers I see different mtimes for the same file. (Setup is two bricks, one on each server, each server also mounts localhost:/volumename). I'm wondering if it's just split brain, or if I'm missing something.
01:31 JoeJulian Heal is for healing split replicas.
01:32 JoeJulian Alex: My first answer is ,,(mount server)
01:32 glusterbot Alex: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
01:33 JoeJulian I'm not sure of the entire process behind mtime. It's always advised to make sure your servers are in sync (ntpd)
01:33 Alex ah okay, so localhost: bit shouldn't matter, makes sense.
01:33 Leolo @rrdns
01:33 glusterbot Leolo: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
01:33 Alex Indeed - they were within 0.000mumble seconds
01:33 _dist left #gluster
01:33 cp0k_ JoeJulian: ah, right...so since my new bricks are part of the volume, doing a find on the file will make sure it is distributed properly among the bricks based on the replica type?
01:34 Alex I shall go look at the source a little for mtime then, as I couldn't find anything authoratitive on the web. Thanks as always JoeJulian :)
01:34 JoeJulian Alex: if heal info, heal info split-brain and heal info heal-failed are empty, then I'm not sure how to explain that.
01:35 JoeJulian Alex: Might also run a "heal full" and see if it corrects it.
01:35 Alex JoeJulian: Yeah, it stumped me too, checked the basics and saw no fires, but it really badly affects my user traffic (because it breaks IMS requests from my caches, and gluster acts as the underlying origin storage for a CDN) - so after some cursory WTF'ing I tend to just fix it manually.
01:35 Alex Will definitely look at that next time, thanks
01:35 JoeJulian "fix it manually"? touch through the client, or what?
01:36 Alex The difference is visible on the underlying brick fs - so a touch to make them both match.
01:36 Alex Which is grim - don't judge me. Things were on fire. :)
01:37 JoeJulian No judgement, was just curious if a touch -m -d through the client was how you cured it.
01:38 JoeJulian My guess is that it would work, but I was just fishing for confirmation.
01:39 Alex *nod*
01:39 Alex In this case: https://gist.github.com/becd38df0866adcebe6a - I'm not sure whether the modify jitter is just ntp-a-like
01:39 glusterbot Title: gist:becd38df0866adcebe6a (at gist.github.com)
01:40 cp0k_ JoeJulian: If I remember correctly you once pasted the optimal 'find' command to crawl files, or had it up on your site?
01:40 cp0k_ JoeJulian: At this point I am looking at about 3 months of rebalancing with the speeds I am seeing
01:40 Alex sorry, ntp-a-like - I actually meant 'about the same as the offset reported by ntp'. Sigh. :)
01:41 Alex Anyway, if it reoccurs, will dig. Thanks!
01:41 JoeJulian cp0k_: probably, came from gluster themselves. Now I think it was overkill since find calls stat to determine the dirent type in case it's a directory it needs to crawl. That stat triggers lookup() which triggers a self-heal check.
01:44 jag3773 joined #gluster
01:45 cp0k_ JoeJulian: got it, so when gluster does not see the file on the newly added bricks it sees the case as a split brain and heals it?
01:46 JoeJulian Only if the newly added bricks are part of a replica
01:46 JoeJulian If you have a replica 2 volume, add two bricks and make it a replica 4 volume - then yes.
01:47 cp0k_ Type: Distributed-Replicate
01:47 cp0k_ Number of Bricks: 15 x 2 = 30
01:47 cp0k_ it was previously 10 x 2 = 20
01:48 Alex Hm, one other thing I've been digging a little bit on - I see a big performance hit with a client that does lots of random seek requests (imagine bittorrent a like small byte range requests, but over HTTP - resulting in lots of fseek) against larger files hosted on Gluster. The problem only seems to show up if I use the gluster client rather than the server - that is, if I address the underlying storage it works fine - otherwise, I get glusterfsd processes ...
01:48 JoeJulian Right, so what you're trying to do is rebalance, not self-heal.
01:48 Alex ... running at *very high load*. Is there anything commonly applied in this kind of scenario to deal with lots of small seeks?
01:48 cp0k_ JoeJulian: Correct, I already have the rebalance operation running
01:48 JoeJulian cp0k_: Then that's all there is. No way to speed that up.
01:49 neurodrone__ joined #gluster
01:49 cp0k_ JoeJulian: :/ so a find on a file will not tell gluster to rebalance that then
01:50 JoeJulian cp0k_: Shouldn't really be any need, either. If you're concerned with filling bricks, do the fix-layout first - should finish pretty quickly - then do the full rebalance.
01:50 JoeJulian The fix-layout will allow new files to be created on the new bricks.
01:51 cp0k_ I started with a fix-layout in the beginning, that broke the production env due to i/o blocking
01:51 JoeJulian Alex: I haven't seen any thing written up about that.
01:51 JoeJulian Alex: But if you find something, please do.
01:51 Alex Noted, ta
01:51 JoeJulian cp0k_: what blocking?
01:52 JoeJulian cp0k_: That shouldn't block at all. It's just updating the masks on the directories.
01:52 cp0k_ JoeJulian: I have about 189TB of data, does not feel like the fix-layout would be fast
01:52 JoeJulian cp0k_: how many directories?
01:53 cp0k_ in the volume root dir?
01:53 JoeJulian In the volume. That just gets an xattr change.
01:53 JoeJulian Each directory gets an xattr change. That's all.
01:54 cp0k_ I am not sure how many dirs in total in the volume, it is probably safe to say several thousands
02:01 nage joined #gluster
02:01 nage joined #gluster
02:12 dusmant joined #gluster
02:20 Elico joined #gluster
02:21 Elico are there any videos or tuts about using gluster as object storage?
02:23 Elico can I find jeff darcy?
02:30 harish joined #gluster
02:32 mattapp__ joined #gluster
02:38 Elico afors
02:38 Elico gfors
02:41 jmarley joined #gluster
02:52 harish joined #gluster
03:06 Janus1 joined #gluster
03:08 bharata-rao joined #gluster
03:14 neurodrone__ joined #gluster
03:22 tokik joined #gluster
03:24 hagarth joined #gluster
03:28 gmcwhistler joined #gluster
03:29 cp0k joined #gluster
03:30 cp0k_ joined #gluster
03:34 shubhendu joined #gluster
03:42 itisravi joined #gluster
03:44 snehal joined #gluster
03:47 aravindavk joined #gluster
03:54 ajha joined #gluster
03:56 JoeJulian Elico: Jeff stopped hanging out on IRC a while back. Best bet for contacting him is through direct email.
03:58 elyograg JoeJulian: is there any helpful info about my symlink problems, or anything else I can do to evaluate whether my volume is healthy enough to finally do a rebalance?
04:00 elyograg perhaps I just need to ignore them problem and proceed, but the results for the last two attempts were so bad.
04:03 cp0k joined #gluster
04:05 elyograg I just replied to my thread on the mailing list, complaining a little bit about the lack of response.  I want to say that my irritation is not directed at you.  You've been beyond helpful, and I completely understand that you can't single-handledly shoulder every "help me, I'm desperate" plea that comes your way.
04:07 JoeJulian elyograg: Was just sitting down at my computer to look at that.
04:07 JoeJulian The filename, line 5 for instance, 484 is a directory, isn't it?
04:08 saurabh joined #gluster
04:08 elyograg which thing are you looking at?
04:09 JoeJulian http://fpaste.org/83806/43825451/
04:09 glusterbot Title: #83806 Fedora Project Pastebin (at fpaste.org)
04:09 elyograg line 5 -- 484 is a file.
04:09 JoeJulian Oh, ok.
04:10 JoeJulian Does it have a gfid?
04:10 elyograg i will check.
04:10 tokik joined #gluster
04:11 JoeJulian check all the bricks, too. It looks like maybe there's a dht-linkfile with no gfid, I think?
04:12 JoeJulian 0-mdfs-dht, in fact, which should be the first replica set.
04:14 JoeJulian ... unless that's just an effect to the errors that lead to that one... which I'm starting to think is an increasingly strong possibility.
04:14 JoeJulian a/to/of/
04:14 elyograg http://fpaste.org/84171/94511210/
04:14 glusterbot Title: #84171 Fedora Project Pastebin (at fpaste.org)
04:14 elyograg I had to build myself a little script to spit out hex and ascii attrs. :)
04:15 elyograg d00v01 is the second replica set.  d00v00 is the first.
04:16 elyograg http://fpaste.org/84172/11372139/
04:16 glusterbot Title: #84172 Fedora Project Pastebin (at fpaste.org)
04:17 JoeJulian got it
04:20 elyograg i find myself somewhat flabbergasted that the glibc max symlinks level is only 8. i think that might be fairly crippling to the .glusterfs tree.
04:21 ngoswami joined #gluster
04:21 elyograg historical data suggests that it used to be 20, though I don't know when it changed.
04:21 JoeJulian I doubt it, but it's worth checking. Line 6 is the original culprit so I'll follow that traceback and see if any of them are attempting to dereference the symlink.
04:22 gtobon joined #gluster
04:24 JoeJulian So client3_3_inodelk throws up because the gfid in the structure is null that it's supposed to be altering returns null.
04:29 elyograg it looks like gfid 3f629e64-ccf0-4d9b-b7ba-1782f8abeb2b is in *every* one of these repeating log sequences.  This looks like it's the gfid for the newscom/mdfs directory directly under each brick.
04:29 JoeJulian Does the symlink for that directory exist?
04:29 JoeJulian the gfid symlink
04:30 elyograg so that's /bricks/d00v00/mdfs/newscom/mdfs ... and so on for each of the other d??v?? combinations.  It'll take me a minute to check all 32 bricks.
04:30 ndarshan joined #gluster
04:30 elyograg looks like it does.
04:30 JoeJulian $brick_root/.glusterfs/3f/62/3f62​9e64-ccf0-4d9b-b7ba-1782f8abeb2b should be a symlink
04:31 elyograg it is./
04:31 JoeJulian It should point to the real directory name's parent directory
04:31 elyograg colorization shows it's not a broken symlink.
04:32 JoeJulian Check the gfid of the parent directory
04:32 deepakcs joined #gluster
04:33 elyograg looks good.
04:33 elyograg http://fpaste.org/84176/45124201/
04:33 glusterbot Title: #84176 Fedora Project Pastebin (at fpaste.org)
04:34 JoeJulian I've got to find something in the fridge to snack on, at least. It's 9:30 and I'm starving... back in a sec...
04:35 mohankumar joined #gluster
04:36 elyograg checking gfids of the parent directories, they do indeed match the symlink info.
04:40 bala joined #gluster
04:43 prasanth joined #gluster
04:44 ravindran joined #gluster
04:48 mohankumar joined #gluster
04:51 elyograg microwave corndogs.  kinda limp and rubbery, but takes 3 minutes for three of them instead of 18 for the regular oven. :)
04:51 JoeJulian Canned chili for me...
04:51 JoeJulian If we shared, that'd go pretty good together... :D
04:51 elyograg that would.
04:52 JoeJulian http://fpaste.org/84178/13945135/
04:52 glusterbot Title: #84178 Fedora Project Pastebin (at fpaste.org)
04:55 hagarth joined #gluster
04:55 elyograg http://fpaste.org/84179/45137071/
04:55 glusterbot Title: #84179 Fedora Project Pastebin (at fpaste.org)
04:55 satheesh1 joined #gluster
04:57 JoeJulian Lets try the same thing for 02
04:58 JoeJulian I'm trying to build something I can duplicate... :/
04:59 elyograg gfid numbers look the same.  slightl differences in other things.  http://fpaste.org/84180/13945139/
04:59 glusterbot Title: #84180 Fedora Project Pastebin (at fpaste.org)
04:59 ira joined #gluster
05:00 JoeJulian Gah!
05:00 JoeJulian Ok, let's see if this is easy to fix...
05:00 JoeJulian Just a sec while I find the xattr I'm trying to remember...
05:01 elyograg this isn't making any sense to me.  I've been around everything I can figure out in that error log and don't know what's up.  I do know that I have broken symlinks, though.
05:01 ppai joined #gluster
05:03 kdhananjay joined #gluster
05:05 elyograg I'm more worried about those broken symlinks than these error messages, although it does concern me that these are continuing to show up in my nfs log even though I'm not actually doing anything.  I do not know what other people might have kicked off, though.
05:06 dusmant joined #gluster
05:06 JoeJulian I get the same error when looking at symlinks, but I have no errors.
05:07 vpshastry joined #gluster
05:08 JoeJulian On the client... actually, hang on while I build a paste
05:08 haomaiwa_ joined #gluster
05:08 elyograg I'm thinking that I would like to shut down all gluster processes across ell eight servers, but that takes a lot of coordination to set up.
05:10 haomai___ joined #gluster
05:10 elyograg Just went poking in /var/log/glusterfs/bricks ... no logs for about a week.  atleast on this server.
05:10 prasanth joined #gluster
05:10 JoeJulian http://fpaste.org/84183/94514642/
05:10 glusterbot Title: #84183 Fedora Project Pastebin (at fpaste.org)
05:11 JoeJulian substitute your own mount point of course.
05:11 JoeJulian That's through a fuse mount
05:11 rjoseph joined #gluster
05:12 JoeJulian Once that's done, stat 484 through your nfs mount and see if it generates an error.
05:12 haomaiwang joined #gluster
05:12 JoeJulian I'm hoping that it doesn't.
05:12 elyograg I'll make a copy of it first. ;)
05:13 JoeJulian If I'm right, I'll want you to do the rebalance fix-layout. You have layout mismatches.
05:14 elyograg shouldn't that complete quickly?  it's just sorta hanging there.
05:14 JoeJulian Yep, it should. :(
05:14 elyograg got a bunch of errors.
05:15 spandit joined #gluster
05:15 elyograg this is supposed to be via the fuse mount, right?
05:15 JoeJulian Right
05:15 elyograg got localhost:mdfs mounted at /mnt/mdfs
05:15 JoeJulian I wonder if it's because I walked from the bottom up...
05:16 elyograg mount seems very broken now.  crap.  I hope this hasn't broken production.
05:17 lalatenduM joined #gluster
05:18 elyograg ok, production mount is good.
05:18 JoeJulian whew
05:18 JoeJulian Can we do the getfattr loops again on the bricks and see if it fixed the mappings?
05:20 elyograg http://fpaste.org/84184/94515228/
05:20 glusterbot Title: #84184 Fedora Project Pastebin (at fpaste.org)
05:21 elyograg going to unmount and remount.
05:21 JoeJulian Oh, that's interesting
05:21 elyograg oh, it's not mounted at all now.
05:21 JoeJulian Oh, crap. I think I caused the client to crash by rebalancing a file.
05:21 JoeJulian heh, that's a bug.
05:21 JoeJulian I have to see if I can do that again!
05:22 elyograg it did let me unmount it once, though.
05:22 elyograg a second umount said not mounted.
05:23 elyograg the getfattr on the brick works still.  the layout attr is not present.
05:24 JoeJulian There's not that many... just mount it again and setfattr -n distribute.fix.layout -v 1 from /mnt/mdfs down to 568
05:26 elyograg ok, that's done.
05:26 elyograg didn't do the end file, 484.
05:27 calum_ joined #gluster
05:28 JoeJulian cool
05:28 JoeJulian Though I just did a file on my test volume with no problem... <shrug>
05:29 JoeJulian The client log even says it fixed the layout of my file.
05:29 JoeJulian Ok, stat the the file, 484
05:30 VerboEse joined #gluster
05:31 snehal joined #gluster
05:33 vkoppad joined #gluster
05:34 elyograg did that.  retunred quickly.  I see a layout fixed for each directory in the path in the client log.
05:34 JoeJulian Sounds good. No errors for the file then?
05:35 JoeJulian Try accessing the file through nfs. Since fscache is a pain for this step, head -c 1k 484 >/dev/null
05:35 elyograg none that I could see.  I see a lot of the same errors in this client log as I am getting in nfs.log -- my monitoring doesn't watch the local client log.
05:36 JoeJulian Just check for a lack of error.
05:37 JoeJulian Since I'm 99% sure that's the problem, feel free to try the same process with some other file, but I think a fix-layout will be the order of the day.
05:37 elyograg i guess solaris doesn't have a -c option.  need to get on a linux machine with the nfs mount. :)
05:37 elyograg no errors on the head command.
05:38 haomaiwa_ joined #gluster
05:38 JoeJulian If it were my volume, I would fix-layout.
05:39 JoeJulian I'm not sure how you ended up with different layouts on the replicas, but that's got to be what caused it.
05:39 elyograg I'm getting constant messages in nfs.log like this: http://fpaste.org/84186/45163571/
05:39 glusterbot Title: #84186 Fedora Project Pastebin (at fpaste.org)
05:39 elyograg no change in the message.  just repeating several times per second.
05:40 JoeJulian Oh, good. That explains the first error a little better.
05:40 elyograg MAX is a new provider.  this wouldn't be something from the 3.3.1 rebalance.
05:40 JoeJulian Now that root has a fixed layout, it's able to get to the correct error message.
05:41 elyograg the gfid there is the same one that's been in the error messages.
05:42 glusterbot New news from newglusterbugs: [Bug 1021686] refactor AFR module <https://bugzilla.redhat.co​m/show_bug.cgi?id=1021686>
05:42 raghu joined #gluster
05:43 elyograg I just did the same fix-layout down the tree with that path, and those messages stopped.
05:43 JoeJulian +1
05:43 elyograg so I should do a rebalance fix-layout?  will probably take forever.
05:43 JoeJulian I love figuring stuff out. :D
05:44 JoeJulian You should, yes.
05:44 xymox joined #gluster
05:46 elyograg should 'readv failed' warnings worry me?
05:46 JoeJulian No
05:47 elyograg rebalance log is error free aside from those warnings.  it'll have to go through the fix-layout again when I do it for real.  I think it could take a couple of days to do the fix-layout.
05:48 JoeJulian I'm feeling pretty confident in your volume health now.
05:48 JoeJulian for what that's worth.
05:49 elyograg making the constant logs in nfs.log stop with a few setfattr commands ... that's pretty awesome.
05:49 JoeJulian Did you see https://forge.gluster.org/gstatus
05:49 glusterbot Title: gstatus - Gluster Community Forge (at forge.gluster.org)
05:49 elyograg no, i'
05:49 JoeJulian I haven't played with it yet, but it looks pretty cool.
05:49 elyograg ll look into it though.
05:51 harish joined #gluster
05:52 ndarshan joined #gluster
05:52 elyograg that was easy to install. :)
05:53 bala joined #gluster
05:53 elyograg apparently my self-heal backlog is extensive.  sitting at that for a while now.
05:54 elyograg gstatus says status unhealthy.  but doesn't say why. :)
05:54 JoeJulian Well that's par for the course.
05:55 dusmant joined #gluster
05:56 shubhendu joined #gluster
05:59 elyograg the specific volume info says backlog data is unavailable.
05:59 elyograg probably why it sat so long at the self heal backlog.
06:02 elyograg bedtime.  i'll check on the status tomorrow.
06:04 JoeJulian Me too. goodnight.
06:04 elyograg interesting.  this was in the log just a minute ago: 0-mdfs-dht: mismatching layouts for /newscom/mdfs/RTR/rtrphotosfour/docs/290
06:04 JoeJulian Probably hasn't made it there yet.
06:04 chirino joined #gluster
06:05 elyograg If I'm ever in your area, I'm buying you a beer.  I might buy you some anyway, remotely. :)
06:05 JoeJulian :D
06:05 JoeJulian I may have to head down to Idaho Falls this summer.
06:05 JoeJulian That's our closest store to you.
06:10 rahulcs joined #gluster
06:12 shylesh joined #gluster
06:14 psharma joined #gluster
06:19 Elico left #gluster
06:20 ajha joined #gluster
06:22 micu joined #gluster
06:25 nshaikh joined #gluster
06:25 ricky-ti1 joined #gluster
06:25 chirino_m joined #gluster
06:27 vkoppad joined #gluster
06:27 gdubreui joined #gluster
06:28 zerick joined #gluster
06:30 benjamin_ joined #gluster
06:32 ravindran joined #gluster
06:33 92AAAHVBD joined #gluster
06:35 meghanam joined #gluster
06:35 meghanam_ joined #gluster
06:44 vimal joined #gluster
06:51 jporterfield joined #gluster
06:51 rastar joined #gluster
06:53 yinyin joined #gluster
07:06 haomai___ joined #gluster
07:07 satheesh joined #gluster
07:11 DV joined #gluster
07:11 shubhendu joined #gluster
07:12 ndarshan joined #gluster
07:12 edward2 joined #gluster
07:13 bala joined #gluster
07:19 jtux joined #gluster
07:19 dusmant joined #gluster
07:30 pk1 joined #gluster
07:32 ngoswami joined #gluster
07:35 mohankumar joined #gluster
07:42 ngoswami joined #gluster
07:45 rahulcs joined #gluster
07:50 ekuric joined #gluster
07:50 Pavid7 joined #gluster
07:55 ajha joined #gluster
07:57 vkoppad joined #gluster
08:00 eseyman joined #gluster
08:00 elyograg joined #gluster
08:02 foster joined #gluster
08:03 rahulcs joined #gluster
08:09 iamben_tw joined #gluster
08:23 keytab joined #gluster
08:24 cjanbanan joined #gluster
08:28 hagarth joined #gluster
08:31 gdubreui joined #gluster
08:32 andreask joined #gluster
08:36 RameshN joined #gluster
08:40 kdhananjay1 joined #gluster
08:40 kam270__ joined #gluster
08:43 KORG joined #gluster
08:48 meghanam joined #gluster
08:48 meghanam_ joined #gluster
08:51 satheesh2 joined #gluster
09:00 liquidat joined #gluster
09:07 satheesh joined #gluster
09:15 X3NQ joined #gluster
09:16 jporterfield joined #gluster
09:28 chirino joined #gluster
09:34 ngoswami joined #gluster
09:38 Pavid7 joined #gluster
09:39 rgustafs joined #gluster
09:43 glusterbot New news from newglusterbugs: [Bug 1074947] add option to bulld rpm without server <https://bugzilla.redhat.co​m/show_bug.cgi?id=1074947>
09:49 tokik joined #gluster
09:50 lalatenduM joined #gluster
09:50 meghanam_ joined #gluster
09:50 meghanam joined #gluster
09:56 jmarley joined #gluster
09:56 jmarley joined #gluster
10:01 jporterfield joined #gluster
10:02 doekia_ joined #gluster
10:02 andreask joined #gluster
10:13 glusterbot New news from newglusterbugs: [Bug 1041109] structure needs cleaning <https://bugzilla.redhat.co​m/show_bug.cgi?id=1041109>
10:24 vkoppad joined #gluster
10:33 yinyin joined #gluster
10:54 ProT-0-TypE joined #gluster
10:54 doekia_ joined #gluster
10:55 rfortier joined #gluster
10:56 doekia__ joined #gluster
10:59 chirino_m joined #gluster
11:00 lpabon joined #gluster
11:02 neurodrone__ joined #gluster
11:05 ngoswami joined #gluster
11:06 B21956 joined #gluster
11:09 aravindavk joined #gluster
11:12 neurodrone__ joined #gluster
11:15 dusmant joined #gluster
11:16 ProT-O-TypE joined #gluster
11:17 ajha joined #gluster
11:20 rahulcs_ joined #gluster
11:22 hagarth joined #gluster
11:26 doekia joined #gluster
11:30 rwheeler joined #gluster
11:32 cp0k_ joined #gluster
11:35 vkoppad joined #gluster
11:35 Pavid7 joined #gluster
11:44 diegows joined #gluster
11:51 dusmant joined #gluster
11:54 yinyin joined #gluster
11:59 gmcwhistler joined #gluster
12:00 rossi_ joined #gluster
12:01 cp0k joined #gluster
12:02 itisravi joined #gluster
12:05 tokik joined #gluster
12:20 gmcwhistler joined #gluster
12:20 dusmant joined #gluster
12:23 neurodrone joined #gluster
12:25 owenmurr joined #gluster
12:33 jag3773 joined #gluster
12:40 andreask joined #gluster
12:43 lalatenduM @learn mailinglists as http://www.gluster.org/interact/mailinglists
12:43 glusterbot lalatenduM: The operation succeeded.
12:43 lalatenduM @mailinglists
12:43 glusterbot lalatenduM: http://www.gluster.org/interact/mailinglists
12:43 glusterbot New news from newglusterbugs: [Bug 1030580] Feature request (CLI): Add an option to the CLI to fetch just incremental or cumulative I/O statistics <https://bugzilla.redhat.co​m/show_bug.cgi?id=1030580>
12:44 Janus1 left #gluster
12:53 jag3773 joined #gluster
13:07 kdhananjay joined #gluster
13:07 RayS joined #gluster
13:15 tokik joined #gluster
13:20 japuzzo joined #gluster
13:22 sroy joined #gluster
13:31 qdk_ joined #gluster
13:31 dkorzhevin joined #gluster
13:33 philv76 joined #gluster
13:35 sjoeboo joined #gluster
13:37 dkorzhevin joined #gluster
13:40 bennyturns joined #gluster
13:42 tdasilva joined #gluster
13:43 glusterbot New news from newglusterbugs: [Bug 1075087] [Rebalance]:on restarting glusterd, the completed rebalance is starting again on that node <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075087>
13:47 theron joined #gluster
13:48 shyam joined #gluster
13:50 rwheeler joined #gluster
13:54 B21956 joined #gluster
13:58 jobewan joined #gluster
14:03 neurodrone__ joined #gluster
14:10 rahulcs joined #gluster
14:12 benjamin_ joined #gluster
14:13 chirino joined #gluster
14:20 tryggvil joined #gluster
14:20 ninkotech_ joined #gluster
14:22 yinyin joined #gluster
14:22 quebre joined #gluster
14:28 rpowell joined #gluster
14:28 quebre i'm missing 'gluster' command, is that normal ?
14:29 quebre i have glusterfs-client and glusterfs-common packages installed
14:29 quebre and still no 'gluster' command
14:30 partner hmm sounds about right to me
14:30 quebre http://www.jamescoyle.net/how-t​o/439-mount-a-glusterfs-volume - im trying to follow this howto
14:31 quebre but can't do without the 'gluster' command, please advice
14:32 partner gluster command is not used a single time on that example?
14:32 partner and you don't need that command to exist for mounting the volume on the client side
14:33 ndk joined #gluster
14:34 partner what exactly is the issue, where are you stuck at?
14:34 yinyin joined #gluster
14:35 quebre partner: i have two proxmox servers, made cluster from it, also i have two supermicro ipmi servers, now i have two nodes in the cluster and i want to apply glusterfs
14:36 quebre how exactly i am supposed to do it ?..
14:38 partner ah, so there is no glusterfs running yet, the howto you are following does not say anything about that part
14:38 quebre no glusterfs running..
14:39 partner well there's plenty of instructions for that, not sure what exactly is the best one but maybe starting with the official site stuff: http://www.gluster.org/community/d​ocumentation/index.php/QuickStart
14:39 glusterbot Title: QuickStart - GlusterDocumentation (at www.gluster.org)
14:39 quebre ok..
14:39 partner or this: http://www.gluster.org/community/document​ation/index.php/Getting_started_overview
14:39 glusterbot Title: Getting started overview - GlusterDocumentation (at www.gluster.org)
14:42 cp0k joined #gluster
14:43 doekia joined #gluster
14:46 kkeithley1 joined #gluster
14:48 seapasulli joined #gluster
14:50 bazzles how do I configure geo-replication 'rsync command to use --inplace rather --sparse ?
14:51 bazzles s/rather than/rather
14:54 benjamin_____ joined #gluster
14:57 lmickh joined #gluster
15:00 bugs_ joined #gluster
15:01 yinyin joined #gluster
15:02 daMaestro joined #gluster
15:04 elyograg quebre: on rhel/centos with the latest packages, i think you need glusterfs-cli.  it'll take me a minute before I can find out for sure.
15:04 calum_ joined #gluster
15:06 elyograg JoeJulian: rebalance fix-layout completed without error in the rebalance log.  I did just get errors from my nfs server monitoring again, though.  from after the rebalance finished.
15:07 yinyin joined #gluster
15:07 jag3773 joined #gluster
15:15 doekia joined #gluster
15:15 doekia_ joined #gluster
15:16 swat30 joined #gluster
15:17 ProT-0-TypE joined #gluster
15:18 nueces joined #gluster
15:23 daMaestro|isBack joined #gluster
15:23 qdk joined #gluster
15:25 jmarley joined #gluster
15:25 jmarley joined #gluster
15:29 seapasulli joined #gluster
15:31 theron joined #gluster
15:31 rpowell On gluster 3.4.2 what is the best way to replace a brick without using the  replace-brick command? 'gluster volume add-brick' followed by a 'gluster volume remove-brick …….. start' ?
15:33 theron joined #gluster
15:45 pk1 left #gluster
15:49 Norky if your volume is replicated then you will have to add bricks equal to your replica level, likely 2 or 3
15:49 Norky why not replace-brick?
15:51 zaitcev joined #gluster
15:52 Matthaeus joined #gluster
15:53 kkeithley1 joined #gluster
15:53 rpowell Norky: thought replace-brick wasnt recommended and was being removed
15:55 Norky hmm, if that's the case I apologise, I' must admit I've not seen the lastest release notes
15:56 Matthaeus joined #gluster
15:58 Norky ahh, yes, I've just found the mailing list post
15:58 Norky wow, that passed me by
15:58 Norky please disregard what I said, I'm clearly out of touch
16:01 Matthaeus joined #gluster
16:05 rpowell Norky: no worries I just saw it all recently myself.
16:05 yinyin joined #gluster
16:08 elyograg quebre: yes, the package installed on my system is glusterfs-cli-3.4.2-1.el6.x86_64
16:11 kkeithley1 joined #gluster
16:19 Leolo norky - care to share a link to that post?
16:20 semiosis :O
16:20 yinyin joined #gluster
16:33 yinyin joined #gluster
16:36 Mo_ joined #gluster
16:43 JoeJulian elyograg: fpaste?
16:46 glusterbot New news from newglusterbugs: [Bug 1075182] Despite glusterd init script now starting before netfs, netfs fails to mount localhost glusterfs shares in RHS 2.1 <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075182>
16:47 Pavid7 joined #gluster
16:48 bazzles whats the supported ( or best ) way of changing geo-replication so that rsync uses --inplace and not --sparse ?
16:49 Norky Leolo, http://www.gluster.org/pipermail/glus​ter-users/2013-September/037487.html
16:49 glusterbot Title: [Gluster-users] Phasing out replace-brick for data migration in favor of remove-brick. (at www.gluster.org)
16:55 hagarth joined #gluster
17:00 yinyin joined #gluster
17:06 chirino_m joined #gluster
17:14 _dist joined #gluster
17:15 micu joined #gluster
17:18 JoeJulian Damn. I wish I'd had time to chime in on that thread when it happened. That's just ugly.
17:19 JoeJulian Amar and Avati both talked about how it's "recommended" to add servers in quantities of replica count, but it's not *required* and shouldn't be. Amar's answer for adding a 3rd server requires twice the amount of storage for a brick replacement.
17:20 sputnik1_ joined #gluster
17:23 elyograg JoeJulian: http://fpaste.org/84402/58560139/
17:23 glusterbot Title: #84402 Fedora Project Pastebin (at fpaste.org)
17:23 elyograg looks like the same gfid.
17:24 elyograg they all say 0-mdfs-dht, but I wonder if it's not the first server.  I've got four, each with eight bricks.
17:25 elyograg that's on the nfs server, in nfs.log, which has no bricks.  I haven't seen any errors in brick logs on the four servers, could it be in another log on the brick servers?
17:25 elyograg actually the brick logs are size zero.
17:26 Leolo norky - ty
17:27 yinyin joined #gluster
17:30 JonnyNomad joined #gluster
17:34 elyograg fix-layout took 6.5 hours.  I thought it would be a lot more. :)
17:34 elyograg lunch.
17:44 madphoenix joined #gluster
17:50 tryggvil joined #gluster
17:51 Joe630 joined #gluster
17:51 Joe630 hello superfriends
17:52 Joe630 is there a way to adjust the failover timeout for gluserfs mounts
17:52 semiosis ,,(ping timeout)
17:52 Joe630 mount -t glusterfs -o backupvolfile-server=gluster1 gluster2:/gv0 /mnt
17:52 glusterbot I do not know about 'ping timeout', but I do know about these similar topics: 'ping-timeout'
17:52 semiosis Joe630: ,,(ping-timeout)
17:52 glusterbot Joe630: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
17:52 Joe630 looks like its 45 seconds
17:52 failshell joined #gluster
17:52 Joe630 okay.
17:52 Joe630 is there a way to adjust it?
17:52 semiosis 'gluster volume set' command
17:52 Joe630 "very expesive" is a lot cheaper than 45 seconds of wall clock for my app
17:53 semiosis see 'gluster volume set help' for a list of options
17:53 Joe630 thanks
17:53 semiosis yw
17:54 yinyin joined #gluster
17:56 yinyin_ joined #gluster
18:02 tryggvil joined #gluster
18:02 Joe630 can I find a definition of "very expensive"?
18:03 Joe630 i need to weigh the costs of moving it vs wall clock
18:03 jmarley joined #gluster
18:05 plarsen joined #gluster
18:07 _dist Joe630: I'm new to this conversation, but I believe that setting should only affect the fuse mounts failover time when the brick you were talking to goes down ungracefully. Like pulling the power cord, raid array dying completely, etc
18:08 Joe630 that is exactly what i am concerned about
18:08 Joe630 42 seconds is an eternity.
18:08 Joe630 i am looking for <1 second failover.
18:09 Joe630 but I don't know what "Very expensive" means in terms of remapping the indoes and locks.
18:09 Joe630 like is it 10 seconds per file?
18:09 Joe630 or is it a mmap
18:09 Joe630 or is it open and read from disk
18:10 _dist Joe630: Yeah, it is a long time so I agree with you that as long as lowering it doesn't tank your performance. I honestly don't know the latency increase you'd have to test. However, the other side of the coin is why would a brick just drop out of your volume? Personally I don't use gluster for unplanned downtime, I use it for shraed storage for vms, planned outages.
18:11 _dist If a machine just drops off, I expect things will go wrong, even if it did switch over immediately, things could still be wrong if a brick just "dissapears"
18:16 elyograg you can.  it's a volume option.  as glusterbot just said, it's that high so that transient problems do not result in full recovery operations.
18:17 elyograg lot said about it.  was afk for a bit. :)
18:17 Joe630 So you guys have never seen a kernel panic?
18:17 Joe630 or a box go down unexpectadly?
18:17 Joe630 Things fail, and I need to make sure my whole infrastructure that is build on this storage doesn't go down with it.
18:17 elyograg yes.  doesn't happen often, but I have seen it.  sudden catastrophic failures in power supplies, motherboards ...
18:18 Joe630 I think it is a reasonable question to ask and prepare for.
18:18 Joe630 I am using gluster instead of netapps.  netapps faill *all the time* but have redundancy built in that is *not* 42 seconds.
18:19 elyograg I don't think re-establishing those file descriptors and locks is expensive so much in terms of time, but in system load.  if you get into a flapping situation, the servers will be thrashing trying to keep up.
18:20 Joe630 thrashing on the file server and client is preferred to the NFS hangs for those 42 seconds
18:20 Joe630 isn't that exactly what would happen anyway?
18:21 Joe630 flapping would case the same behavior, it would just be seperated by 42 seconds?
18:21 Joe630 *cause
18:21 andreask joined #gluster
18:22 _dist Joe630: I'm honestly not experienced enough to say what the tradeoff is, and I agree with you. Without knowing the trade off there's not way to make that call well. I expect the trade off is different for each type of load, and the only way to know what it will be is to test yourself, on your load.
18:22 elyograg maybe.  but let's say you have a series of network events that cause a bunch of 2 second outages for a few minutes.  a 42 second timeout wouldn't cause a failover, it would just weather the storm.  a subsecond failover would fail over a LOT.
18:23 Joe630 why would it fail a lot?
18:23 doekia_ joined #gluster
18:23 Joe630 why can't i set 3 subsecond fails in a row cause the flap
18:23 elyograg network problems are sometimes WEIRD.
18:23 Joe630 WEIRD is prefered to totally STOPPED
18:23 yinyin joined #gluster
18:24 Joe630 this is why i argued against NFS
18:24 Joe630 it is fundamentally broken for failover
18:24 Joe630 gluster helps.
18:24 Joe630 lets make it better.
18:24 jag3773 joined #gluster
18:25 _dist Joe630: if I were you, I'd set the failover to something more reasonable, say 10 seconds. And then try to make the events happen that the gluster documentation considers "expensive", I'm not sure if/where that is documented
18:26 _dist or maybe even lower, and work you way up if need be
18:26 Joe630 so if you were running a VM pool, could your whole farm deal with a 10 second stoppage?
18:26 Joe630 I feel like this is Dark City
18:26 Joe630 SLEEEEEEEEP
18:27 elyograg I would never put virtual machines on a network filesystem at all.  assuming I ran virtual machines much ... i prefer bare metal.
18:27 _dist Joe630: definitely, none of my guests would freak out in that amount of time, dark city :)
18:27 Joe630 elyograg: that is not how the world works.
18:27 Joe630 not aws, not vmware, not openstack
18:28 _dist elyograg: we're running 37 kvm guests on a gluster replicate, works fine. Latency is a little high (30ms) throughput is decent. Live migration is beautiful. If I hit the power button a vm hosts, it migrates everything over to the other hosts BEFORE it shuts down
18:29 _dist but yeah, for certain applications, a network file system could be too slow, we just don't have any
18:29 rossi_ joined #gluster
18:31 Joe630 live migration is a beautiful thing
18:31 Joe630 and exactly why we are using NFS
18:31 Joe630 _dist: how is the latency compared to using mount.nfs
18:35 _dist Joe630: I'm using libgfapi in qemu, not mount.glusterfs so my comparison to nfs wouldn't be fair. Also it's over 2x10gbe bonded 802.3ad
18:36 Joe630 ooh
18:36 Joe630 is that fast?
18:36 Joe630 does it failover more betterer?
18:36 Joe630 that may be an option
18:36 _dist Honestly I've never had an event where failover occurred yet, only been live for 3 weeks, never considered it until just now you asked :)
18:38 _dist the latency is lower, I get about 300-400 MBytes/sec of real throughput if it's random io, and around 2k iops (but that FS bottleneck, not gluster)
18:38 sputnik1_ joined #gluster
18:39 _dist Joe630: is VM storage your use case?
18:41 elyograg JoeJulian: Here's a section of log that gets repeated in nfs.log with a different path after that same gfid: http://fpaste.org/84438/39456323/
18:41 glusterbot Title: #84438 Fedora Project Pastebin (at fpaste.org)
18:41 elyograg this includes two "I" logs that aren't in my monitoring alarm emails.
18:43 elyograg the replicate number in the I logs is different in various repeats.
18:43 Joe630 _dist: yes
18:44 JoeJulian "Joe630> So you guys have never seen a kernel panic? or a box go down unexpectadly?" I had a kernel panic about 6 years ago and the magic smoke came out of a motherboard 5 years ago.
18:45 _dist JoeJulian: you should bottle that stuff, sell it on ebay
18:45 JoeJulian JoeJulian: I moved a network cable to another switch probably a dozen times in the last year.
18:45 Joe630 I have 9000 servers under my control.
18:45 elyograg looks like it's not always the same gfid, just that's the most common one.  setattr of uid/gid on <gfid:14c41119-97a0-4924-9​40c-58ec10b2a9a1>/374/761 :<gfid:00000000-0000-0000-0000-000000000000> failed (Invalid argument)
18:45 Joe630 i see it every day.
18:45 _dist Joe630: physical ?
18:45 Joe630 at least that many
18:45 Joe630 yes, physical.
18:45 Joe630 ooh, this is my chance to say "over 9000"
18:47 _dist that's a good poewr level
18:47 _dist power*
18:47 JoeJulian Are all 9k going to be storage servers?
18:47 Joe630 oooh that would be rad.
18:48 _dist Joe630: I think what JoeJulian is saying, is how many will be running gluster? That is a massive deployment
18:49 failshell joined #gluster
18:49 JoeJulian Joe630: Basically it's a math problem. Figure out your commitment requirements, your failure possibilities and the amount of time each failure would exist. Use the longest ping-timeout that fits within that calculation.
18:50 JoeJulian What you don't want is an ACKs that took 1ms too long triggering split-brain.
18:50 yinyin joined #gluster
18:51 JoeJulian That's the worst-case scenario, imho.
18:51 Joe630 That is my original question, beacuse "very expensive" is an unknown quantity
18:51 JoeJulian It's true.
18:51 kmai007 joined #gluster
18:52 Joe630 one man's ceiling is another man's floor.
18:52 _dist JoeJulian: Do you know if libgfapi behaves different? I assume not, but haven't tested it
18:53 JoeJulian From my own experience, "very expensive" was all my storage servers maxing out their cpus and all my clients working at about (and this is not a scientific observation) 30% speed for half an hour.
18:53 JoeJulian _dist: I'm sure it would be different, just because it avoids some of the context switching, but probably not all that much.
18:54 _dist Joe630: I think the reason I was never as concerned as you are, is because my hypervisors ARE my storage nodes.
18:54 JoeJulian Joe630: Are you here in Seattle?
18:55 elyograg oh fun.  one of the CPUs in my gluster nfs server had an ecc error in the L3 cache.
18:55 JoeJulian Speaking of oops...
18:55 Joe630 JoeJulian: I YAM
18:56 Joe630 There can be onle one Joe
18:56 Joe630 I am actually on the East side
18:56 Joe630 but I live over there.
18:56 kmai007 is everyone using the default logrotate by glusterfs?
18:56 JoeJulian 9k vms... east side... hmm...
18:57 Joe630 nor 9k vms.
18:57 Joe630 not
18:57 Joe630 actual servers
18:57 Joe630 my data center is far away
18:57 Joe630 and it isn't MS
18:57 JoeJulian I wouldn't have thought MS.
18:58 JoeJulian They're more likely to throw good money after bad and pay EMC or someone.
18:59 JoeJulian Ok, I give up... who? :D
18:59 Philambdo joined #gluster
18:59 JoeJulian ... if you can say...
19:00 Joe630 are you on the east side?!
19:00 JoeJulian Right now I'm in Edmonds, working from my home office. My corp office is in SODO.
19:02 JoeJulian Wife works out of City Center Bellevue.
19:03 JoeJulian ... for anyone who's stalking me...
19:03 * elyograg scribbles some notes.
19:03 Joe630 In a big black skyscraper?
19:04 JoeJulian No, I work for Ed Wyse Beauty Supply, a regional distributor.
19:04 Joe630 No, your lady
19:04 JoeJulian Inome/Intellius.
19:04 Joe630 I work at Taco Time
19:04 Joe630 we have a side thing as a data warehouse
19:04 JoeJulian 9k servers for TT???
19:04 Joe630 yes.
19:05 Joe630 amazon started as a bookseller.
19:05 JoeJulian And all they can buy for kids meal toys are sunglasses?
19:05 Joe630 TT started selling tacos
19:05 Joe630 someone has to compete with TT
19:05 JoeJulian Interesting.
19:05 rotbeard joined #gluster
19:07 elyograg JoeJulian: after it finishes zipping, I'm going to pm you a dropbox URL.  It hasn't been redacted in anyway.  It's my full nfs.log.  750MB before compression, going back to april of last year.
19:09 _dist taco town SNL skit :)
19:11 Joe630 taco time has the epiphany stick.
19:13 JoeJulian elyograg: nah
19:14 JoeJulian elyograg: I'm interested in the getfattr walk we did from an error reporting file on any servers that file is on now.
19:14 Joe630 http://www.tacotime.com/menu/crisp_burritos.html
19:14 glusterbot Title: TacoTime Menu - Crisp Burritos (at www.tacotime.com)
19:14 JoeJulian Now I want lunch...
19:15 Joe630 sorry that's the other company.  I work for TTNW
19:15 Joe630 http://www.tacotimenw.com/CrispBurrito.aspx
19:15 glusterbot Title:  (at www.tacotimenw.com)
19:17 skochok joined #gluster
19:18 yinyin joined #gluster
19:18 rpowell joined #gluster
19:18 Joe630 off to the company store for some delicious lunch
19:20 elyograg JoeJulian: not sure what you mean by that. I'm ready to poke around.
19:20 junaid joined #gluster
19:22 JoeJulian elyograg: Basically, last night when I had you do that while loop getfattr script, I noticed that the trusted.glusterfs.dht values were different for the same directory on different servers. They shouldn't be.
19:25 elyograg I didn't look at it super closely, but I -thought- they were the same.
19:27 elyograg If I need to, I can copy the tree out to other storage, wipe out out on gluster, then recopy it back.  i'll need to make sure that's OK, but I think it might be.  If I copied it into a different directory name, then did renames at the directory level for an instanteous swap, that would work, right?
19:28 elyograg the gfid that almost all of those logs have in common is the parent of everything, so if I have to rename that, there'd be some problems. :)
19:31 sputnik1_ joined #gluster
19:34 elyograg ok, i found the while thing you sent me in the scrollback.  I'll get some new info.
19:35 JoeJulian Well it worked when we used the distribute.fix.layout xattr...
19:39 elyograg oh, it's the dht values that are different.
19:39 elyograg those were from different bricks on the same server, though.
19:40 JoeJulian Ah, ok.
19:40 elyograg so i'd think they'd be different.  I didn't check two bricks from the same replica set (which are on different servers)
19:40 JoeJulian I completely missed that last night. I must have been more tired than I thought.
19:41 JoeJulian Odd that it still worked though.
19:42 elyograg checking one of these new log entries, I do see differences on the two replicas, though.
19:43 elyograg oh, wait.  sorry, did the wrong server.
19:43 elyograg when I do the right server, everything matches up.
19:44 plyushkin joined #gluster
19:45 yinyin joined #gluster
19:46 elyograg 15 minutes until we have a 'gluster status' meeting.
19:46 elyograg I wonder if I should run another rebalance fix-layout.
19:47 elyograg that fixed several thousand directory layouts according to the log.
19:47 JoeJulian I think it's going to say it fixed them even if there was no change.
19:49 elyograg there were LOTS of entries where it said it was fixing the layout, but only a few thousand were like this: 0-mdfs-dht: subvol: mdfs-replicate-2; inode layout - 2684354550 - 2952790004; disk layout - 3758096370 - 4026531824
19:49 JoeJulian hmm
19:49 JoeJulian brb... daughter wants out of her bath.
19:55 elyograg these logs make no sense at all.
19:56 kmai007 elyograg: how do you handle timestamps in your logs?
20:05 criticalhammer joined #gluster
20:07 criticalhammer Does anyone know whats up with the 3.4 gluster builder in the RHEL repo and why it doesnt come with init script to start up the daemons nor does it place the gluster command into the user path?
20:07 criticalhammer the only commands that show up are glusterfs and glusterfsd
20:08 semiosis criticalhammer: gluster command should be in root's path (/usr/sbin ?)
20:08 criticalhammer i am root
20:08 semiosis criticalhammer: initscript should only start glusterd
20:08 semiosis glusterd starts the rest
20:09 criticalhammer there is no glusterd
20:09 criticalhammer just glusterfs and glusterfsd
20:09 kris left #gluster
20:09 semiosis possible you have only the client package, but not server package, installed?
20:10 semiosis i'm not familiar with the rpms but if we were talking about my debs thats what it sounds like you've got
20:10 criticalhammer there is no "server" package
20:11 semiosis well, that's the problem then
20:11 semiosis are you using the ,,(yum repo)?
20:11 glusterbot The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
20:11 criticalhammer yes
20:11 criticalhammer i havent screwed around with the channels yet
20:11 criticalhammer i thought i wouldnt have to
20:11 semiosis idk channels
20:11 criticalhammer looks like i do
20:12 yinyin joined #gluster
20:14 JMWbot joined #gluster
20:14 JMWbot I am JMWbot, I try to help remind johnmark about his todo list.
20:14 JMWbot Use: JMWbot: @remind <msg> and I will remind johnmark when I see him.
20:14 JMWbot /msg JMWbot @remind <msg> and I will remind johnmark _privately_ when I see him.
20:14 JMWbot The @list command will list all queued reminders for johnmark.
20:14 JMWbot The @about command will tell you about JMWbot.
20:15 Amanda I feel like that is both the best and worst idea in the world. How many people leave ascii art?
20:16 semiosis wrong channel?
20:16 Matthaeus joined #gluster
20:16 Amanda I think I'm in the right one?
20:17 kkeithley1 joined #gluster
20:21 JoeJulian criticalhammer: What you had installed is the client for Red Hat Storage. Since that's a Red Hat package, EPEL isn't supposed to step on it so you're stuck with the older packages that are part of RHS. We're upstream and conveniently provide our own repos for most distros.
20:21 criticalhammer seems like the gluster server package is not in the RHEL repo
20:21 JoeJulian Right. The server is part of the RHS channel.
20:21 criticalhammer ive checked every supplementary channel and the server package is not there.
20:22 criticalhammer kk well that makes sense
20:22 JoeJulian I think it's a premium package, or whatever they call it.
20:22 criticalhammer lol premium
20:22 semiosis unleaded?
20:22 JoeJulian I've never paid for it, so I'm not sure.
20:22 JoeJulian Leaded? :D
20:22 criticalhammer well epel here I come
20:23 criticalhammer fully leaded!
20:23 criticalhammer im going to test out btrfs on my test bed
20:23 JoeJulian Like I say, its not in epel either due to packaging restrictions. Only our own upstream repo.
20:23 JoeJulian Though it is in Fedora proper.
20:24 criticalhammer hmm
20:24 JoeJulian ... I wonder how that's going to work for RHEL 9... hehe.
20:25 criticalhammer sorry I didnt catch what you said earlier JoeJulian about it being in the RHS channel
20:25 criticalhammer thanks for pointing that out to me
20:26 criticalhammer all you have to do is repeat yourself a few times and it will eventually sink into my brain
20:26 JoeJulian hehe
20:28 kkeithley_ you don't get the RHS server unless you're paying for RHS support; you won't find it in any channel. If you're running on RHEL, you'll want to uninstall the RHS client bits and use the bits from download.gluster.org.
20:29 criticalhammer yeah, i just realized that kkeithley
20:29 criticalhammer just a pain when you want to update the server and have to be mindful where things are coming from
20:30 criticalhammer what channel and making sure not to install the same thing twice
20:30 kkeithley_ probably add an exclude= clause in one of the /etc/yum.repos.d/*.repo files
20:30 JoeJulian Doesn't priorities work with updated?
20:30 criticalhammer JoeJulian: idk, i havent tested it out
20:31 criticalhammer it would be better if RHEL just didnt put gluster client and RDMA packages in their channel
20:31 criticalhammer just let gluster manage it themselves
20:31 JoeJulian +1
20:31 JoeJulian I think the idea is that you don't have to pay for clients, only servers.
20:32 criticalhammer makes sense
20:32 criticalhammer but still just one extra step a sys admin has to be mindful of
20:32 kkeithley_ things like libvirt/oVirt and more can be clients, and they need the client bits. And for RHEL from Red Hat, that means RHS client bits, not community client bits
20:32 JoeJulian ... btw... that's true with CentOS as well as the RHS client packages are included there, too.
20:33 JoeJulian I just manage my yum.repos.d with puppet so I never have to worry about forgetting.
20:33 criticalhammer yeah im in the process of learning salt
20:34 criticalhammer im a python junky
20:34 kkeithley_ yeah, CentOS is a RHEL clone, not a Fedora clone
20:34 JoeJulian me too. salt wasn't around when I started with puppet.
20:34 * JoeJulian hates ruby.
20:34 criticalhammer ive never used ruby
20:34 criticalhammer but ive heard mixed reviews
20:35 criticalhammer and also hasnt there been massive ruby exploits recently?
20:35 JoeJulian I wouldn't be surprised.
20:35 criticalhammer ill just stick to python
20:35 criticalhammer it does what I want, and with salt, gives me a reason to program in it more
20:36 semiosis wow thats really soggy thinking
20:36 criticalhammer soggy thinking?
20:36 semiosis you dont need to know, or even see, ruby to use puppet
20:36 criticalhammer what do you mean
20:37 semiosis i mean, so what if puppet is written in ruby
20:37 Matthaeus Puppet uses ruby on the backend to provide a domain-specific language to the admin, to the point where puppet config files can look just like regular config files.
20:37 criticalhammer oh, i didnt know that
20:38 criticalhammer i chose salt because its written in python
20:38 semiosis and yes there were some bugs in ruby, but that doesnt mean anything written with ruby is tainted
20:38 criticalhammer and ive been programming in python for 10+ years
20:38 semiosis people say the same things about java, and php, and it annoys me
20:38 criticalhammer everything is exploitable
20:38 semiosis anyone can write a bad/insecure app in any language, doesnt reflect on any other app written in that language
20:38 criticalhammer give someone enough time and they will eventually break it
20:39 criticalhammer i agree
20:39 JoeJulian (puppet/ruby is tainted... according to people who know, that's one of the reasons that environments don't do what you would expect in puppet)
20:39 semiosis JoeJulian: you trolling me bro?
20:39 semiosis :D
20:39 JoeJulian Hehe
20:40 criticalhammer well ne way, thanks for clearing this up with me folks
20:40 JoeJulian I go to a meetup with a couple of the devs every month.
20:40 semiosis JoeJulian: are they wishing they could start over in golang?
20:40 JoeJulian lol!
20:40 criticalhammer meetings with devs.... im sorry :P
20:41 JoeJulian No, it's great... I think it's the only way that pull requests actually get looked at.
20:41 jag3773 joined #gluster
20:43 ricky-ticky1 joined #gluster
20:43 [o__o] left #gluster
20:43 criticalhammer so whats peoples thoughts of gluster and btrfs?
20:43 kkeithley_ if you value your data, I wouldn't not suggest using btrfs until it has an fsck
20:43 metroreklama joined #gluster
20:44 criticalhammer btrfsck?
20:44 theron joined #gluster
20:45 criticalhammer https://btrfs.wiki.kernel.org/index.php/FAQ#What.27s_the_difference_between_btrfsck_and_fsck.btrfs
20:45 glusterbot Title: FAQ - btrfs Wiki (at btrfs.wiki.kernel.org)
20:45 criticalhammer well thats somthing ill test out then
20:46 criticalhammer besides the snapshot feature, I think btrfs should work under gluster
20:46 [o__o] joined #gluster
20:46 criticalhammer ill do a blog sometime next week with my findings
20:46 JoeJulian kkeithley_: would you say that for xfs as well? :)
20:46 criticalhammer ^^
20:47 kkeithley_ there's xfs_repair
20:48 JoeJulian I've missed the logic that makes that different from btrfsck.
20:49 [o__o] left #gluster
20:49 kkeithley_ oh, I hadn't noticed. Is that a real fsck?
20:49 kkeithley_ they finally got one.
20:49 criticalhammer yeah
20:51 kkeithley_ btrfs works fine under gluster. It's just a question of whether you think it's stable enough for _your_ data.
20:51 criticalhammer well under RHEL its still sketchy
20:51 criticalhammer at least with RHEL 6.4
20:51 [o__o] joined #gluster
20:52 criticalhammer i ended up getting a kernel panic when trying to format a disk with btrfs
20:52 kkeithley_ I don't believe there's a btrfsck in RHEL6.4, is there? Maybe it's there in 6.5
20:52 criticalhammer if you manually install everything it will work on any rhel 6 version
20:52 criticalhammer i didnt pull it from repo
20:54 JoeJulian I wouldn't use btrfs with anything less than kernel 3.5, imho.
20:54 criticalhammer yeah im starting to think that as well
20:55 criticalhammer it just has all the features I want in a file system
20:55 criticalhammer i want it to be good in RHEL so badly!
20:55 JoeJulian I actually do have one use case that uses it, and I run Fedora for that.
20:55 JoeJulian But that's throw-away so I don't really care if it's a problem.
20:57 Pavid7 joined #gluster
20:57 _dist criticalhammer: I think we spoke before, but I'm just going to mention that I'm using gluster on zfs without issue in production right now. As far as I know, it has all the same features as brtfs
20:58 semiosis _dist: how's zfs work on rhel these days?
20:59 criticalhammer we have but my spidey sense is telling me zfs is going the way of the dinosaur
20:59 criticalhammer in linux
21:00 _dist semiosis: I'm not running it on rhel right now but I have. The newest ZoL can be installed with yum (if you're asking about their 2.6 kernel) that's actually what its coded for since LLNL is basically the beta tester for all releases. It's probably more stable on rhel than debian (my use)
21:01 semiosis heh, neat! last time i looked they had ubuntu packages & iirc no others
21:01 semiosis but i was only looking for ubuntu packages, so
21:01 _dist semiosis: you can just yum install it now, or apt-get as well yeah
21:01 criticalhammer yeah its there in RHEL
21:02 _dist criticalhammer: maybe, but I think that'd be pretty far off. I feel like that's similar to saying xen will go away because of kvm. Personally I think ZoL will become more popular than on BSD or Solaris. As far as I know Oracle is backing BRTFS and it bough (and closed) open solaris. Linux kernels don't have ZoL because of licensing issues
21:03 _dist bought*
21:05 criticalhammer we will see
21:05 criticalhammer i hope ZFS sticks around
21:05 * _dist not fighting any wars :)
21:05 criticalhammer me either
21:06 MacWinne_ joined #gluster
21:08 andreask joined #gluster
21:09 ninkotech joined #gluster
21:11 mkzero_ joined #gluster
21:20 elyograg ZFS is beautiful.  I love it.  Unfortunately Oracle is now in charge so you can't get Solaris for free any more.  Linux would be a great place for it, if not for the license incompatibility.  I'm sure that BSD, where the license is no problem for ZFS, is a good system ... but the leading/bleeding edge in almost all open source software is on linux.
21:21 criticalhammer I havent checked the numbers, but linux most likely has a larger user base
21:21 glusterbot` joined #gluster
21:22 criticalhammer and that larger user base gives to potentially more use cases which helps future development
21:24 twx_ joined #gluster
21:24 social_ joined #gluster
21:24 juhaj_ joined #gluster
21:24 jezier_ joined #gluster
21:27 cogged joined #gluster
21:27 Amanda__ joined #gluster
21:27 delhage_ joined #gluster
21:28 _dist on a side note I think oracle's similar direction with MySQL (now mariaDB) was really smart.
21:28 crashmag joined #gluster
21:28 uebera|| joined #gluster
21:28 eightyeight joined #gluster
21:28 iwsdedo joined #gluster
21:28 kkeithley_ What's the URL of LLNL's yum repo?
21:28 * kkeithley_ is too lazy to google for it
21:28 semiosis @lucky llnl yum repo
21:28 _dist http://zfsonlinux.org/epel.html
21:28 _dist more of a zfsonlinux question :)
21:28 glusterbot Title: ZFS on Linux (at zfsonlinux.org)
21:28 glusterbot semiosis: http://www.opensfs.org/wp-content/uploa​ds/2013/04/Morrone_Sequoia_LUG2013.pdf
21:29 cjh973 joined #gluster
21:29 HeisSpiter joined #gluster
21:29 tjikkun joined #gluster
21:29 tjikkun joined #gluster
21:30 JoeJulian joined #gluster
21:30 verdurin joined #gluster
21:31 zaitcev joined #gluster
21:33 kmai007 Guys, i am in a pickle....
21:33 kmai007 on all of my 8 gluster nodes
21:33 yinyin joined #gluster
21:33 kmai007 i have some delete processes go glusterd
21:33 kmai007 http://fpaste.org/84513/94573570/
21:33 glusterbot Title: #84513 Fedora Project Pastebin (at fpaste.org)
21:34 kmai007 i don' t know how to get rid of them; other than a reboot?
21:34 elyograg looks like you (or someone) upgraded glusterd.
21:34 elyograg you've got processes running the previous version.,
21:34 kmai007 that would be me
21:35 kmai007 darnit
21:35 kmai007 so a reboot is the action to take?
21:35 elyograg you should have done this before upgrading, but you should be able to do it now.
21:35 kmai007 i tried service glusterd restart, but no changes
21:35 elyograg service glusterd stop
21:35 elyograg service glusterfsd stop
21:35 elyograg killall glusterfs
21:35 elyograg then wait a few seconds and restart glusterd.
21:36 elyograg note that this will unmount any fuse mounts.
21:36 kmai007 this is prod, i'll have to wait off hours
21:36 elyograg most likely this is a mount process.
21:37 kmai007 when u say mount, do you mean on the clients?
21:37 elyograg a mount on whichever server you ran the lsof.  might also be self-heal daemon processes.
21:38 al joined #gluster
21:38 bfoster joined #gluster
21:38 kmai007 oh ok, the lsof output is on the storage node
21:38 VeggieMeat joined #gluster
21:38 kampnerj joined #gluster
21:38 lanning joined #gluster
21:38 elyograg ps axwww | grep "glusterfs "
21:38 m0zes joined #gluster
21:38 jiffe98 joined #gluster
21:38 haomaiwa_ joined #gluster
21:38 tjikkun_work_ joined #gluster
21:38 al joined #gluster
21:38 pdrakeweb joined #gluster
21:38 kmai007 elyograg: [root@omhq1140 ~]# ps axwww | grep "glusterfs "
21:38 kmai007 6842 pts/0    S+     0:00 grep glusterfs
21:38 kmai007 43108 ?        Ssl    0:04 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/37d0cad5a8d92e4da1b1938076b75cdc.socket
21:38 kmai007 43115 ?        Ssl    0:04 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/d605f5d0b9ccc47d9fa9ac509a9aadfa.socket --xlator-option *replicate*.node-uuid=281114a​d-a227-4cc7-a4e6-63f9db4e7d91
21:38 kmai007 [root@omhq1140 ~]#
21:38 kmai007 sorry
21:39 masterzen joined #gluster
21:39 NeatBasis joined #gluster
21:39 al joined #gluster
21:39 eryc joined #gluster
21:39 eryc joined #gluster
21:39 elyograg the selfheal daemon runs as a glusterfs process.  so do the nfs server process and fuse mounts.
21:39 kmai007 meant to paste the link instead
21:39 kmai007 http://fpaste.org/84516/57392213/
21:39 glusterbot Title: #84516 Fedora Project Pastebin (at fpaste.org)
21:39 elyograg looks like those are the nfs server and self-heal daemon.
21:40 kkeithley1 joined #gluster
21:40 qdk joined #gluster
21:41 stickyboy joined #gluster
21:41 wrale joined #gluster
21:41 bennyturns joined #gluster
21:41 ndk joined #gluster
21:41 FrodeS joined #gluster
21:41 tryggvil joined #gluster
21:41 delhage joined #gluster
21:41 Amanda__ joined #gluster
21:41 jmarley joined #gluster
21:41 Joe630 joined #gluster
21:41 JonnyNomad joined #gluster
21:41 yosafbridge joined #gluster
21:41 sputnik13 joined #gluster
21:41 jurrien_ joined #gluster
21:41 ColPanik joined #gluster
21:41 cyberbootje joined #gluster
21:41 sac`away` joined #gluster
21:41 sulky joined #gluster
21:41 a2 joined #gluster
21:41 larsks joined #gluster
21:41 abyss^_ joined #gluster
21:41 neoice joined #gluster
21:41 primusinterpares joined #gluster
21:41 johnmwilliams__ joined #gluster
21:41 dcmbrown_ joined #gluster
21:41 edong23 joined #gluster
21:41 fyxim joined #gluster
21:41 Peanut joined #gluster
21:41 Kins joined #gluster
21:41 fsimonce joined #gluster
21:41 delhage joined #gluster
21:41 elyograg oh, you have brick processes too -- glusterfsd according to your first fpaste.
21:41 kmai007 yeh, looks like it will be a reboot
21:41 elyograg actually, that might be what all of them are now that I look at it.
21:41 kmai007 is teh only solution
21:41 kmai007 it is
21:41 kmai007 i just mapped the PIDs
21:41 elyograg running those three commands followed by a glusterd restart should take care of it.
21:41 stickyboy joined #gluster
21:42 sulky joined #gluster
21:42 elyograg as long as you don't have any fuse mounts locally on that machine.
21:42 kmai007 yeh these are strictly storage
21:42 kmai007 no local fuse mounts
21:43 elyograg i can't be 100% sure that this won't cause an explosion, but it is highly unlikely as long as it's a replicated volume and all replicas are on different servers.
21:43 kmai007 yep they are
21:44 kmai007 elyograg: thanks for bouncing the ideas around
21:44 semiosis joined #gluster
21:44 semiosis joined #gluster
21:44 micu joined #gluster
21:45 kmai007 damn, sure enough every gluster node has a "delete" in lsof for glusterd
21:45 jag3773 joined #gluster
21:45 kmai007 looks like i didn't upgrade properly
21:45 kmai007 from 3.4.1-3 -> 3.4.2-1
21:45 X3NQ joined #gluster
21:45 kmai007 ugh
21:46 elyograg IMHO, it's best to run those three commands, do the upgrade, and then reboot.
21:46 kmai007 gotcha, next time i'll do that
21:46 elyograg that way gluster is entirely shut down during the upgrade (service glusterd stop is not enough), and everything is sure to be cleared out when it starts back up.
21:47 kmai007 agreed, i recalled seeing a newsletter about this, but didn't click
21:48 kmai007 i guess my rolling upgrade was only half way done
21:49 kmai007 so who is going to summit?
21:50 morse joined #gluster
21:51 chirino joined #gluster
21:53 kshlm joined #gluster
21:59 kmai007 is the gluster feature that needs to be set network.ping-timeout <in sec> what I need to tune for the clients to "failover" to another storage brick?
22:00 elyograg i believe that's all you need to change.  at the volume level.
22:00 kmai007 what have you modified yours to, if i may ask?
22:01 elyograg I haven't.
22:01 kmai007 default 42 sec.
22:13 Joe630 kmai007: you need to specify a backup server.  I've only found it in the mount options:
22:13 Joe630 mount -t glusterfs -o backupvolfile-server=gluster1 gluster2:/gv0 /mnt
22:14 JoeJulian Another option is ,,(rrdns)
22:14 glusterbot You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
22:14 Joe630 rr DNS is a really bad way to do that.
22:14 Joe630 it doesn't allow fo failover.
22:14 Joe630 it's just a poor man's load balancer
22:14 JoeJulian yes it does
22:15 JoeJulian And no. There's no load balancing.
22:15 JoeJulian @mount server
22:15 Joe630 no it doesn't, cached dns entries don't go away when the server goes away
22:15 glusterbot JoeJulian: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
22:15 Joe630 ok
22:15 Joe630 so what do you do on broken dns servers that only return one name
22:15 Joe630 but randomize it
22:16 JoeJulian I use not-broken dns servers. :D
22:16 Joe630 or when you can't reach the server specified
22:16 Joe630 DNS doesn't work for this
22:16 JoeJulian I didn't realize that there were DNS servers that were broken in that way. Do you know which dns software does that?
22:16 Joe630 this does NOT do failover:
22:16 Joe630 http://edwyseguru.wordpress.com/2012/01/09/usin​g-rrdns-to-allow-mount-failover-with-glusterfs/
22:17 JoeJulian It does for me.
22:17 JoeJulian Always has.
22:17 elyograg Joe630: It's only for *mount*.
22:17 Joe630 http://www.gluster.org/community/doc​umentation/index.php/Glossary:_RRDNS
22:17 glusterbot Title: Glossary: RRDNS - GlusterDocumentation (at www.gluster.org)
22:17 Joe630 where does it say that
22:17 JoeJulian # host glusterfs
22:18 JoeJulian glusterfs.ewcs.com has address 10.0.0.120
22:18 Joe630 JoeJulian: so you have rrdns, shut a machine down, and your mounts stay active?
22:18 JoeJulian glusterfs.ewcs.com has address 10.0.0.145
22:18 JoeJulian glusterfs.ewcs.com has address 10.0.0.2
22:18 JoeJulian and yes, I know that host isn't a valid tool.
22:18 Joe630 mount it and shut down 10.0.0.2
22:18 elyograg the fuse client doesn't maintain a connection to the server listed in the mount.
22:18 JoeJulian regardless of rrdns, if you shut down a machine your mounts stay active.
22:18 Joe630 but the data isn't accessible
22:18 Joe630 I did all of this work today
22:19 JoeJulian As long as it's replicated to a different machine it is.
22:19 Joe630 what you are saying just does not work.
22:19 elyograg it connects to it, downloads the volume definition which has a list of servers and brick paths.  probably includes port numbers.  It connects directly to every single **BRICK**.
22:19 JoeJulian doesn't include port numbers. That's a separate rpc call to glusterd.
22:20 elyograg got it.  but it does connect to every brick, the name in the mount is discarded.
22:20 JoeJulian hmm, there's an interesting question I need to check... which glusterd does it ask for port numbers...
22:21 JoeJulian I assume each server's own glusterd, but I'm just guessing there.
22:21 tdasilva left #gluster
22:22 JoeJulian So the bigger question, Joe630, with rrdns if I shut down a server and /mount/ a volume using the glusterfs hostname, does it mount every time. Can I strace and see it failing to connect to the down server and trying the next one in the list?
22:22 JoeJulian The answer is, yes.
22:22 Joe630 sweet
22:22 Joe630 i am running test now
22:22 Joe630 but its hare because I am using hosts files
22:23 Joe630 in a test vm
22:23 Joe630 glustercluster:/gv0 on /mnt type fuse.glusterfs (rw,default_permissions,al​low_other,max_read=131072)
22:23 elyograg you don't want to use rrdns for the server names, just the name you use for mount.
22:23 JoeJulian rrdns isn't everyone's tool, but I like it.
22:23 Joe630 elyograg: i know how to internet
22:23 JoeJulian lol
22:23 elyograg ain't no noun that can't be verbed. :)
22:23 Joe630 over 9000 servers
22:23 Joe630 they are all named "server"
22:24 kmai007 JoeJulian: your splitmount sir, how does it handle GFID ?
22:24 Joe630 and they are in a /8 nerwork
22:24 JoeJulian kmai007: Yeah, that's another article...
22:24 kmai007 JoeJulian: thank you sir
22:25 kmai007 you can call me sir too if you like
22:25 JoeJulian kmai007: I think that the gfid can be (by some bit fiddling) interpreted as the inode.
22:26 Joe630 other than tcpdump is it possible to shee what i'm connected to?
22:26 Joe630 *see
22:26 kmai007 yeh i was trying to repeat my steps when finding the path; then ls -li, then finding the inum
22:26 JoeJulian netstat
22:26 JoeJulian They're tcp connections.
22:26 elyograg lsof -Pni
22:26 kmai007 netstat -anp|grep <:port#>
22:26 Joe630 but i just heard it doesn't keep an open connection
22:27 JoeJulian It does for the client<->brick communications.
22:27 kmai007 hand shake and bake
22:27 Joe630 also elyograg ain't noun that can't verb.
22:27 Joe630 also elyograg ain't no noun that can't verb.
22:31 kmai007 JoeJulian: can you help me understand how a volume can get into a GFID split-brain situation?
22:32 kmai007 or anyone?
22:32 JoeJulian I'll blog that later. I've got to get this sphinx server up and running today and it seems like every vendor we do business with wants to interrupt that today.
22:32 kmai007 gotcha
22:33 kmai007 GL
22:43 YazzY joined #gluster
22:43 YazzY joined #gluster
22:46 rwheeler joined #gluster
22:57 chirino joined #gluster
23:02 ColPanik_ joined #gluster
23:09 robo joined #gluster
23:21 elyograg JoeJulian: we think we've figured out why the nfs log is showing accesseson these files and killed the processes, but the reason for the errors still eludes us.  A remaining question is whether or not this should be an indication that we should avoid doing the rebalance.
23:25 JoeJulian Is heal info empty?
23:25 JoeJulian Or empty-ish, except for current activity?
23:26 neurodrone joined #gluster
23:26 elyograg no entries there.
23:28 JoeJulian I can't think of any reason why not, then.
23:34 kam270 joined #gluster
23:37 robo joined #gluster
23:43 theron joined #gluster
23:44 kam270 joined #gluster
23:50 RicardoSSP joined #gluster
23:50 RicardoSSP joined #gluster
23:54 sroy_ joined #gluster
23:54 kam270 joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary