Camelia, the Perl 6 bug

IRC log for #gluster, 2013-05-31

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 Kripto joined #gluster
00:07 Kripto left #gluster
00:07 Kripto joined #gluster
00:09 Kripto Hello, I'm seeing some odd behavior in GlusterFS (3.3.1) where stating the dir path takes so long that files are failing to be written completely to. Is there any way to speed Gluster up so that it's as fast as a local filesystem?
00:10 Kripto There are approximately 32k directories that are being statd in the path
00:23 _pol joined #gluster
00:26 vpshastry joined #gluster
00:33 _pol joined #gluster
00:37 wNz joined #gluster
00:37 wNz whats the command to see what attributes are currently set on a brick?
00:38 Kripto wouldn't that be gluster volume info ?
00:38 wNz i thought it was getfattr
00:39 wNz getfattr -md /path/to/brick
00:39 wNz is what i was looking for
01:02 joelwallis joined #gluster
01:13 hjmangalam1 joined #gluster
01:18 majeff joined #gluster
01:20 hjmangalam3 joined #gluster
01:21 bala joined #gluster
01:23 robo joined #gluster
01:49 Technicool joined #gluster
01:57 inevity joined #gluster
01:57 baul joined #gluster
02:02 majeff joined #gluster
02:04 portante joined #gluster
02:09 majeff joined #gluster
02:23 theron joined #gluster
02:38 dmojorydger joined #gluster
02:45 vshankar joined #gluster
02:49 hagarth joined #gluster
02:51 theron joined #gluster
03:03 bharata joined #gluster
03:12 baul joined #gluster
03:15 inevity joined #gluster
03:38 Airbear joined #gluster
03:44 baggar11 joined #gluster
03:44 baggar11 Did Redhat kill the gluster ISO's when they bought?
03:56 badone joined #gluster
04:06 majeff joined #gluster
04:11 JoeJulian baggar11: No, they were out-of-date, buggy, and totally targeted at the wrong type of user. They killed themselves.
04:13 baggar11 eh, dang
04:13 JoeJulian Kripto: No, clustered filesystems (any of them) serve a different function than local filesystems do. As such they're subject to an entirely different set of criteria.
04:14 hjmangalam1 joined #gluster
04:15 JoeJulian That's like comparing Apples and Orchards.
04:21 baggar11 left #gluster
04:22 y4m4 joined #gluster
04:23 y4m4 joined #gluster
04:25 saurabh joined #gluster
04:26 mohankumar__ joined #gluster
04:39 badone joined #gluster
04:43 sgowda joined #gluster
04:45 majeff1 joined #gluster
04:45 waldner joined #gluster
04:45 waldner joined #gluster
04:46 kke joined #gluster
04:47 tjikkun_work joined #gluster
04:48 anands joined #gluster
04:51 hagarth joined #gluster
04:53 vpshastry joined #gluster
04:57 shylesh joined #gluster
04:59 shireesh joined #gluster
05:02 bulde joined #gluster
05:05 mohankumar__ joined #gluster
05:11 sgowda joined #gluster
05:17 ndevos joined #gluster
05:18 pkoro joined #gluster
05:19 kshlm joined #gluster
05:26 vpshastry1 joined #gluster
05:31 aravindavk joined #gluster
05:38 an joined #gluster
05:47 sgowda joined #gluster
05:50 saurabh joined #gluster
06:02 lalatenduM joined #gluster
06:08 lalatenduM joined #gluster
06:08 ngoswami joined #gluster
06:14 StarBeast joined #gluster
06:16 jtux joined #gluster
06:21 vimal joined #gluster
06:32 satheesh joined #gluster
06:35 ollivera joined #gluster
06:38 dobber_ joined #gluster
06:39 an joined #gluster
06:45 jtux joined #gluster
06:48 16WAAER4U joined #gluster
06:49 jtux joined #gluster
06:57 ollivera left #gluster
07:02 yinyin joined #gluster
07:06 majeff joined #gluster
07:07 majeff1 joined #gluster
07:09 andreask joined #gluster
07:09 ricky-ticky joined #gluster
07:10 hybrid512 joined #gluster
07:12 vpshastry1 joined #gluster
07:14 ndevos joined #gluster
07:16 jiku joined #gluster
07:17 ekuric joined #gluster
07:18 puebele1 joined #gluster
07:46 _ilbot joined #gluster
07:46 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
07:49 ninkotech_ joined #gluster
07:54 ujjain joined #gluster
08:02 rb2k joined #gluster
08:05 ninkotech_ joined #gluster
08:05 ninkotech joined #gluster
08:20 ccha joined #gluster
08:20 VSpike joined #gluster
08:20 primusinterpares joined #gluster
08:20 stopbit joined #gluster
08:20 Supermathie joined #gluster
08:21 harish joined #gluster
08:21 ninkotech joined #gluster
08:21 VSpike If I have a pair of replica glusterfs servers, with three clients, should it be safe to reboot one of the servers?
08:22 VSpike How can I check if a client is correctly balancing across both servers?
08:23 nightwalk joined #gluster
08:25 Airbear joined #gluster
08:42 VSpike It seems to have worked OK this time :)
08:43 VSpike I see a *lot* of stuff like this in the server's brick log [2013-05-31 09:38:59.188092] I [server3_1-fops.c:1085:server_unlink_cbk] 0-gv0-server: 1517: UNLINK <gfid:63953b43-8b64-4f49-bf65-c9b9ba69b2​ee>/17ae031b-82b5-4791-bae7-e80be8a37cf7 (58b06d08-81d2-49a3-9498-90f005bc85a7) ==> -1 (No such file or directory)
08:43 VSpike And similar in the client logs too
08:43 VSpike Is this fairly normal chatter?
08:43 manik joined #gluster
08:49 36DAAP83P joined #gluster
08:52 rastar joined #gluster
08:56 spider_fingers joined #gluster
09:02 ccha hum how can I delete an geo-replication ?
09:06 vpshastry2 joined #gluster
09:06 duerF joined #gluster
09:14 satheesh joined #gluster
09:19 JordanHackworth joined #gluster
09:25 sgowda joined #gluster
09:27 mohankumar joined #gluster
09:28 yinyin joined #gluster
09:31 ccha joined #gluster
09:31 stopbit joined #gluster
09:33 nightwalk joined #gluster
09:46 Norky joined #gluster
09:51 lh joined #gluster
09:52 majeff joined #gluster
09:52 Guest86059 left #gluster
10:03 Supermat1ie joined #gluster
10:03 mohankumar__ joined #gluster
10:05 VSpike joined #gluster
10:07 edward1 joined #gluster
10:07 primusinterpares joined #gluster
10:18 nightwalk joined #gluster
10:20 shylesh joined #gluster
10:22 ccha joined #gluster
10:38 atrius_ joined #gluster
10:39 glusterbot New news from newglusterbugs: [Bug 969384] [FEAT] Reduce the number of crawls the self-heal daemon needs to make <http://goo.gl/V8GzT>
10:44 jiku joined #gluster
10:48 inevity joined #gluster
10:48 baul joined #gluster
10:50 an joined #gluster
10:59 lpabon joined #gluster
11:14 juhaj joined #gluster
11:17 yinyin_ joined #gluster
11:17 baul joined #gluster
11:17 inevity joined #gluster
11:23 inevity joined #gluster
11:23 baul joined #gluster
11:29 baul joined #gluster
11:29 inevity joined #gluster
11:35 andreask joined #gluster
11:48 baul joined #gluster
11:48 inevity joined #gluster
12:01 inevity joined #gluster
12:01 baul joined #gluster
12:08 root__ joined #gluster
12:11 yinyin_ joined #gluster
12:27 dewey joined #gluster
12:29 baul joined #gluster
12:29 inevity joined #gluster
12:30 dewey_ joined #gluster
12:46 tziOm joined #gluster
12:49 nickw joined #gluster
12:50 balunasj joined #gluster
12:54 robo joined #gluster
13:01 harish joined #gluster
13:04 plarsen joined #gluster
13:05 yinyin_ joined #gluster
13:20 jdarcy joined #gluster
13:20 manik joined #gluster
13:20 aliguori joined #gluster
13:26 ccha how can I delete an geo-replication ?
13:26 pkoro joined #gluster
13:33 andrewjs1edge joined #gluster
13:36 portante joined #gluster
13:40 glusterbot New news from newglusterbugs: [Bug 969461] RFE: Quota fixes <http://goo.gl/XFSM4>
13:41 vpshastry1 joined #gluster
13:44 bennyturns joined #gluster
13:47 sgowda joined #gluster
13:51 aliguori joined #gluster
13:52 guigui3 joined #gluster
13:58 manik joined #gluster
13:59 yinyin_ joined #gluster
14:02 wushudoin joined #gluster
14:06 sw_ joined #gluster
14:07 robo joined #gluster
14:08 18VAA7UPV joined #gluster
14:15 aliguori joined #gluster
14:18 guigui1 joined #gluster
14:18 arolfe joined #gluster
14:20 nueces joined #gluster
14:33 dbruhn joined #gluster
14:34 dbruhn is there a way to force a volume stop short of killing the services on all the servers that contain bricks?
14:34 jdarcy Did you try "volume stop xyz force"?
14:35 dbruhn yep
14:35 dbruhn [root@ENTSNV04001EP 1]# gluster volume stop ENTV04EP force
14:35 dbruhn Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
14:35 dbruhn operation failed
14:35 hchiramm_ joined #gluster
14:36 jdarcy Anything in the client or glusterd logs about the failure?
14:36 dbruhn actually yeah, looks like it over flowed one of the bricks
14:36 dbruhn ugh
14:37 dberry joined #gluster
14:37 dberry joined #gluster
14:37 jdarcy Overflowed?
14:37 jdarcy Doesn't seem like a volume stop should fail because a brick is out of space.
14:38 dbruhn sorry, no, The log made it look like the volumes full
14:38 dbruhn but it's not and none of the bricks are out of space either
14:38 dbruhn http://pastie.org/7989056
14:38 glusterbot Title: #7989056 - Pastie (at pastie.org)
14:38 dbruhn Here is the log output
14:40 dbruhn I was going to reboot the system due to some wonky behavior with the application on top of it, and then I couldn't stop the volume
14:41 jdarcy ENOSPC on a lookup?  Weird.
14:41 rb2k If gluster loses a connection between two bricks (e.g. network gets separated), how long would you guys expect gluster to reestablish the connection once the network connection works fine again?
14:42 dbruhn lol, you see where my confusion is
14:42 Technicool joined #gluster
14:42 jdarcy dbruhn: If it were me, I'd kill all of the gluster processes on the machines with client-6 and client-7 (from the volfile/log), check whether they are actually out of space or inodes, and then restart glusterd
14:43 jdarcy rb2k: Should be on the order of 30 seconds at most.
14:43 rb2k hm
14:43 rb2k thanks
14:43 jdarcy rb2k: IIRC it will retry every *three* seconds, but there are other timeouts in the 30-second range that might be involved.
14:44 rb2k [2013-05-31 12:00:27.169265] C [client-handshake.c:126:rpc​_client_ping_timer_expired] 0-test-fs-cluster-1-client-0: server 10.84.119.65:24009 has not responded in the last 42 seconds, disconnecting.
14:44 rb2k is that fuse -> local gluster brick
14:44 rb2k or in between gluster servers?
14:44 rb2k maybe fuse has a different timeout
14:44 rb2k (although that should be influenced)
14:45 jthorne joined #gluster
14:46 jdarcy That's between the client and a brick.  Generally servers don't talk to each other except by pretending to be clients.
14:47 jdarcy If you're repeatedly getting that kind of log message, it means you're connecting but then the server daemon on the other end isn't responding.
14:48 pkoro joined #gluster
14:51 sjoeboo_ so, i've got a situation where my gluster volume is UP and working...but ALL cli commands basically sit there and timeout/fail...only logs entires are
14:51 sjoeboo_ https://gist.github.com/sjoeboo/5685522
14:51 glusterbot Title: gist:5685522 (at gist.github.com)
14:52 duerF joined #gluster
14:53 yinyin_ joined #gluster
14:54 jdarcy sjoeboo_: Looking at the cluster-lock code now.
14:56 sjoeboo_ awesome, thank you.
14:59 daMaestro joined #gluster
14:59 jdarcy Hm, this doesn't make any sense in the version of the code I have.  What version are you on, sjoeboo_?
15:00 sjoeboo_ 3.3.1, but w/ the gid caching fix backported.
15:02 jdarcy That shouldn't affect the glusterd stuff, fortunately.  Looking again...
15:03 jdarcy OK, those messages don't actually indicate a problem.  The problem is what's *not* there.
15:04 sgowda joined #gluster
15:05 jdarcy Hi Shishir.
15:05 jdarcy sjoeboo_: So even "gluster peer status" times out?
15:06 * jdarcy wonders if "gluster system fsm log" will show anything at all.
15:11 dbruhn jdarcy looks like /var was full on all of the servers? ugh
15:13 majeff joined #gluster
15:14 jdarcy dbruhn: That's an odd way for that to manifest, but glad you figured it out.
15:14 dbruhn well I am working on it at the moment
15:14 dbruhn 10 severs 20 bricks on this system
15:14 jdarcy sjoeboo_: Also, do the messages appear when the CLI command is issued, or when it terminates?
15:15 rwheeler joined #gluster
15:15 _Bryan_ joined #gluster
15:21 jag3773 joined #gluster
15:23 arolfe Is it safe to use replace-brick to move data onto a brick that already has data? that is, to consolidate a set of smaller bricks into one larger one?
15:25 jdarcy arolfe: I think that would have a strong potential for data loss.
15:26 jdarcy arolfe: If your brick is not at a local mountpoint, it should be trivial to create a second empty brick on the same local filesystem.
15:26 arolfe that's what I was worried about.
15:27 arolfe hmm, all my bricks are mountpoints.
15:27 arolfe What's the right way to vacate and decommission a brick without having a new/empty brick available?
15:29 bennyturns joined #gluster
15:31 jdarcy arolfe: Remove-brick implicitly rebalances data away from the brick that's being removed.
15:31 jdarcy arolfe: That's why it can take a long time.
15:32 kaptk2 joined #gluster
15:33 bugs_ joined #gluster
15:33 arolfe ah.  the remove-brick example in the docs shows a warning "removing bricks can result in data loss" so I assumed it didn't do any rebalancing.
15:35 dberry joined #gluster
15:35 hchiramm_ joined #gluster
15:35 wushudoin joined #gluster
15:36 jdarcy arolfe: Just tried it a couple of times to be sure.  Seems to work as advertised.
15:36 arolfe great.  thanks for the help!
15:38 jdarcy arolfe: Any time.  As always, don't take a developer's word for anything.  ;)  It's not too hard to test on a small config yourself, to make sure you understand what's going on (and maybe how it could fail).
15:41 theron joined #gluster
15:43 dberry joined #gluster
15:48 yinyin_ joined #gluster
15:53 bennyturns joined #gluster
15:56 dbruhn ok now I have a completely new issue my glistered service won't start, I wonder if config files could be messed up between servers?
15:57 sjoeboo_ jdarcy: sorry had a meeting to get to. Those logs show up when the cli command is issued.
16:02 dbruhn I have a 10 Server system with 20 bricks total. Server 5,7, and 10 will start the service, the rest will not start. I think because the drive was full when I issues the stop command it couldn't populate the changes between the servers in the system. If I were to copy my /var/lib/glusterd/(volume) directory to the other servers would it fix those servers?
16:03 semiosis dbruhn: ,,(peer rejected)
16:03 glusterbot dbruhn: I do not know about 'peer rejected', but I do know about these similar topics: 'peer-rejected'
16:03 semiosis dbruhn: ,,(peer-rejected)
16:03 glusterbot dbruhn: http://goo.gl/nWQ5b
16:03 dbruhn getting a page not found from the link
16:03 semiosis @forget peer-rejected
16:03 glusterbot semiosis: The operation succeeded.
16:03 semiosis @learn peer-rejected as http://gluster.helpshiftcrm.com​/q/how-do-i-fix-peer-rejected/
16:03 glusterbot semiosis: The operation succeeded.
16:04 semiosis dbruhn: check on ***ALL*** of your servers the output of 'gluster peer status' to see if any peers report any other peers in state Peer Rejected
16:04 semiosis i suspect that may be the case
16:04 dbruhn I can't even get the service to run on all but three of them right now
16:04 semiosis oh right that's weird
16:05 semiosis so you ran out of space on root?
16:05 semiosis then glusterd stopped working?
16:05 dbruhn on /var
16:05 semiosis oh right
16:05 dbruhn then I issued a gluster volume stop command
16:05 semiosis i think the solution for peer rejected described in that link will help you in this case too
16:05 semiosis try it on one server
16:06 ndevos dbruhn: you should also check /var/lib/glusterd/peers/* sometimes empty files prevent glusterd from starting at all
16:11 devoid joined #gluster
16:12 sjoeboo joined #gluster
16:14 dbruhn silly question before I do this, back up my volume files on the servers, and there isn't a chance this will hose the data on the bricks or screw the volume up right?
16:18 semiosis money back guarantee ;)
16:18 dbruhn haha, well I am less worried about the work and more worried about the 10TB of customer data on the system that is down right now *sigh* nothing like a friday fire drill
16:19 semiosis i dont think your data is at risk from doing this... unless something goes nuts with self heal once the bricks are back online
16:19 semiosis but that's unlikely
16:20 semiosis highly unlikely, based on what you've said so far
16:20 dbruhn thanks or the insight, I appreciate it
16:20 dbruhn I'll buy you a beer someday if I get the opportunity
16:20 semiosis woo
16:20 dbruhn I take those debts seriously!
16:20 dbruhn where are you located anyway?
16:20 semiosis FL
16:21 ninkotech joined #gluster
16:21 ninkotech_ joined #gluster
16:22 ccha what after beta2 ? beta3 or rc ?
16:23 semiosis ccha: betas then GA, usually
16:23 ninkotech joined #gluster
16:23 ninkotech_ joined #gluster
16:23 semiosis beta = rc
16:24 ccha will it have a beta3 ? before GA ?
16:25 semiosis i dont know.  that will probably be discussed on the gluster-devel mailing list, but i havent been following the conversation
16:27 ccha is there already documentation for 3.4 as 3.3 ?
16:35 lpabon joined #gluster
16:35 vpshastry1 joined #gluster
16:39 ninkotech_ joined #gluster
16:46 sjoeboo joined #gluster
16:48 vpshastry1 left #gluster
16:52 dbruhn when ever I try and run a gluster peer status I get a connection failed message
16:52 dbruhn weird
16:53 _pol joined #gluster
16:54 thomaslee joined #gluster
16:54 _pol joined #gluster
16:59 zaitcev joined #gluster
17:06 dbruhn has anyone ever seen behavior like this http://pastie.org/7989759
17:06 glusterbot Title: #7989759 - Pastie (at pastie.org)
17:06 dbruhn nothing has changed between it running happily and not
17:13 bulde joined #gluster
17:18 hagarth joined #gluster
17:32 Uzix joined #gluster
17:43 _ilbot joined #gluster
17:43 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
17:46 GabrieleV joined #gluster
17:47 sjoeboo jdarcy: sorry, in and out today. gluster peer status works just fine. its volume operations that fail.
17:48 sjoeboo jdarcy:  and gluster system fsm log does indeed return.
17:50 rb2k joined #gluster
18:00 vpshastry joined #gluster
18:06 kbsingh joined #gluster
18:11 glusterbot New news from newglusterbugs: [Bug 959069] A single brick down of a dist-rep volume results in geo-rep session "faulty" <http://goo.gl/eaoet>
18:12 plarsen joined #gluster
18:26 joelwallis joined #gluster
18:27 andreask joined #gluster
18:29 lbalbalba joined #gluster
18:30 andreask joined #gluster
18:36 bulde joined #gluster
18:38 Mo__ joined #gluster
18:43 dbruhn more info: etc-glusterfs-glusterd.vol.log contains this error "[rpcsvc.c:491:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request" when gluster peer status fails
18:47 vpshastry left #gluster
18:58 lbalbalba @later tell semiosis thanks! that seems to work. someone should add that bit to the admin guide(s) ;)... although i still think its funny when you cant just edit config files on disk when the demon's not running...
18:58 glusterbot lbalbalba: The operation succeeded.
19:15 hjmangalam1 joined #gluster
19:19 hagarth joined #gluster
19:27 semiosis :O
19:28 semiosis repeating what i said in PM...
19:28 semiosis [15:25] <semiosis> thank jdarcy he's the one who solved your problem, i just relayed :)
19:28 semiosis [15:26] <semiosis> make a new page from a link here: http://www.gluster.org/communit​y/documentation/index.php/HowTo
19:28 glusterbot <http://goo.gl/0Y2v2> (at www.gluster.org)
19:28 semiosis [15:27] <semiosis> your contribution of a HOWTO would be very appreciated!
19:28 semiosis lbalbalba: ^^^
19:30 lbalbalba no problem. not sure what i should name that howot, though. 'modifying .vol files with a filter' ?
19:31 JoeJulian sounds good to me
19:32 semiosis +1
19:36 rastar joined #gluster
19:41 Uzix joined #gluster
19:42 lbalbalba done. http://www.gluster.org/community/docume​ntation/index.php/HowTo#Tips_and_Tricks
19:42 glusterbot <http://goo.gl/UTvj3> (at www.gluster.org)
19:43 lbalbalba http://www.gluster.org/community/docu​mentation/index.php/Glusterfs-filter
19:43 glusterbot <http://goo.gl/dMhlL> (at www.gluster.org)
19:44 JoeJulian Thanks lbalbalba. That's new to me, too.
19:45 * JoeJulian ponders the possibilities...
19:45 JoeJulian semiosis: ^^ do a logrotate on the brick files when the filter is triggered?
19:46 lbalbalba still think its funny when you cant just edit config files on disk when the demon's not running... doesnt the demon read the config on startup, and then write back what it read when changes are made through the cli admin interface?. i didnt expext that at all, its unlike any demons ive come across
19:46 JoeJulian It builds vol files from the info file.
19:46 lbalbalba hu ? info file ?
19:47 JoeJulian /var/lib/glusterd/vols/$vol/info
19:47 semiosis glusterfs isn't like other software.  details at 11 ;)
19:48 JoeJulian I'm more and more liking jdarcy's plan of using an external configuration management tool, like zookeeper or (my preference) ConCoord.
19:49 semiosis JoeJulian: concoord... ZK in python?
19:49 semiosis or is there more to it than that?
19:49 JoeJulian yep
19:50 dbruhn I will say I have now learned a hard lesson around those config files. If the storage space they are located on is full and can't be written to and you try making any changes even as simple as stopping a volume you will hose up the configuration data on all of the servers with bricks most likely.
19:50 lbalbalba so whats the pint of having .vol files at all then, when they dont accurately reflect the current config anyway
19:50 JoeJulian It had more of an object focus, which I think suits volumes and peers better, too.
19:50 lbalbalba pint/point
19:50 JoeJulian which is why I /always/ put /var/log on it's own partition.
19:51 dbruhn yep
19:51 semiosis JoeJulian: your logrotate idea... to have a history of changes?
19:52 JoeJulian And a fresh log file after a change happens.
19:52 dbruhn how much i/o do the config files end up going through?
19:52 semiosis oh you mean logrotate on the actual logs, i thought you meant use logrotate to preserve /var/lib/glusterd
19:52 JoeJulian Ah
19:53 JoeJulian No, if I was going to do that, I'd probably use git.
19:53 sjoeboo so, i had asked this before and was tlaking a tiny bit to jdarcy, but, figured i might as well ping again...
19:53 sjoeboo https://gist.github.com/sjoeboo/5685522
19:53 glusterbot Title: gist:5685522 (at gist.github.com)
19:53 sjoeboo thats what i'm seeing when i try any  gluster volume cli commands
19:54 sjoeboo and the command just sits there and then returns, no output/times out.
19:54 sjoeboo i can check peer status just fine.
19:54 sjoeboo and jdarcy had asked me to run
19:54 sjoeboo gluster system fsm log|less
19:54 JoeJulian lbalbalba: It's a legacy thing. The .vol files came first. The cli was not designed to read the vol files, but rather to keep state and rebuild them. Probably a lazy technique to just be able to get it done.
19:54 sjoeboo which runs and returns:
19:54 sjoeboo https://gist.github.com/sjoeboo/5687561
19:54 semiosis sjoeboo: looks like you got some polish spam on your gist?
19:54 glusterbot Title: gist:5687561 (at gist.github.com)
19:55 sjoeboo ha, apparently? oh well.
19:55 JoeJulian sjoeboo: That looks completely normal. Lock, respond (no errors = 0), unlock.
19:55 lbalbalba JoeJulian; thx
19:55 sjoeboo yeah, thats what i thought.
19:59 tylerflint joined #gluster
20:00 tylerflint does gluster not support nfs v4?
20:02 NeatBasis_ joined #gluster
20:04 lbalbalba tylerflint: no, i dont think so
20:05 hagarth joined #gluster
20:09 JoeJulian not yet
20:09 tylerflint well shoot :) nfsv3 doesn't support ipv6
20:10 tylerflint and gluster doesn't support nfsv4 :)
20:10 JoeJulian You could ipv4 tunnel
20:11 JoeJulian wait... it doesn't?
20:12 tylerflint well, the solaris nfs client doesn't
20:13 Chr1z joined #gluster
20:15 Chr1z If I setup a 2-server glusterfs where both are replicating the other… say 100gb each… If I wanted to add a few more server for redundancy but don't necessarily need extra space, how's the best way to go about that?  I just want it where if a few of them drop I'd still be fine… (basically adding more for throughput and redundancy improvements)
20:15 JoeJulian Chr1z: add-brick replica N <brick>
20:16 JoeJulian by increasing the replica count by the number of bricks you're adding, you'll add those bricks to the replica.
20:16 semiosis Chr1z: please do yourself a favor and enable quorum
20:16 semiosis and read up on ,,(split-brain)
20:16 glusterbot (#1) To heal split-brain in 3.3, see http://goo.gl/FPFUX ., or (#2) learn how to cause split-brain here: http://goo.gl/Oi3AA
20:16 semiosis the latter
20:17 JoeJulian btw, unless you're scaling for single file popularity (like netflix does for a new movie release) you might be better off adding distribution.
20:18 Chr1z split brain is similar to mysql replication being out of sync… I'm wanting to prevent that completely if possible.. what's the best way to ensure that doesn't occur?
20:18 semiosis JoeJulian: does netflix use glusterfs?
20:19 JoeJulian That's what I've read somewhere... though I don't /think/ they use it for content delivery.
20:19 Chr1z JoeJulian: I haven't even set it up yet.. I'm mainly wanting this to allow multiple apache servers to share a /home to keep from having to rsync between several servers constantly so everything is always up to date.  Just trying to plan the best way to go about it at this point.
20:19 JoeJulian But it's the best analogy I could think of.
20:20 Chr1z using drbd right now so limited to 2 apache servers… hoping this will allow me to scale that up a bit
20:20 JoeJulian Chr1z: You mount the volume on each apache server (GlusterFS client). By adding more replicas, you add more latency and decrease write bandwidth as each client connects to every server in the cluster. Writing to replca sets writes to each brick simultaneously.
20:21 JoeJulian So really you just need replication for fault tolerance. Distribution will provide load distribution.
20:22 JoeJulian So your question is, how many storage servers can I tolerate failing simultaneously? Add 1 and that's your replica count.
20:22 JoeJulian And yes, your apache servers /can/ also be storage servers.
20:24 Chr1z so… I can have say 4 servers and replicate between 2 and the 3rd and 4th are for load distribution only?
20:26 JoeJulian you would have 2 replica sets. The files would be distributed across both of them (whole files, not stripes). So "foo" might be on servers 1 & 2 and "bar" might be on 3 & 4
20:26 JoeJulian The client mount gives you a unified namespace.
20:27 Chr1z JoeJulian: ok.. and in your scenario… servers 1 and 3 (or 2 and 4) could fail at the same time and all would still be ok ? or basically any one server from each replica ?
20:28 JoeJulian Correct.
20:31 Chr1z ok.  cool… only other thing… if say all servers rebooted at the same time (power outage etc) will it just come back up and require some intervention?
20:31 JoeJulian If you have quorum enabled, you should be safe.
20:32 JoeJulian otherwise, there's the possibilty that files that were in the process of being written will be split-brain.
20:32 Kripto_ joined #gluster
20:32 Chr1z is quorum something that can be simply enabled or something that must be installed and setup separately?  Sorry for so many questions.. I'm sure this will make much more sense once I'm actually setting it up later.
20:33 JoeJulian just enabled
20:33 semiosis volume set $vol cluster.quorum-type auto
20:33 semiosis or something like that
20:33 semiosis see gluster volume set help for a list of opts
20:33 semiosis it's in there
20:34 jdarcy joined #gluster
20:34 Chr1z so a "brick".. is it a certain size or a size I set?
20:34 semiosis ,,(glossary)
20:34 glusterbot A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
20:35 jdarcy A brick is just a directory.  It's as big as the server lets it be (usually up to the size of the local filesystem where it lives).
20:35 Chr1z gotcha...
20:35 hchiramm__ joined #gluster
20:35 Chr1z greatly appreciate all the help… BBL gonna go see what I can break lol
20:36 semiosis have fun
20:37 jdarcy You keep on using that word.  ;)
20:37 JoeJulian lol
20:37 rastar joined #gluster
20:38 JoeJulian I totally read that with Mandy Patinkin's accent even.
20:39 jdarcy I have a friend who is like the god of Princess Bride.  He can recite all the lines at will, with perfect inflection.
20:40 jdarcy Same guy who famously replied, when I complained about how having four Jeffs in one small company was annoying, that "*One* was annoying."  Worked together three times, trying for four.
20:43 JoeJulian jdarcy: did your zookeeper/concoord proposal get any traction? The more I consider it, the more I like it.
20:44 joelwallis joined #gluster
20:47 jdarcy JoeJulian: It got negative traction.  Everybody hated it, and me for pointing out the need.  But there's good news.
20:48 jdarcy JoeJulian: There's a new consensus protocol, called Raft.  All of us who are dorky enough to care about such things are excited, and there are many implementations in progress.
20:48 jdarcy JoeJulian: Some of them are in languages like C++ or Go that don't have the nasty dependency problems that ZK does.
20:49 JoeJulian That was one of the thinks I liked about concoord. Which of the implementations of Raft do you like so far?
20:49 JoeJulian s/thinks/things/
20:50 glusterbot JoeJulian: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
20:50 lbalbalba glusterbot got no love for sed :P
20:51 jdarcy JoeJulian: Probably logcabin so far.  Here's the list. https://ramcloud.stanford.edu/​wiki/display/logcabin/LogCabin
20:51 glusterbot <http://goo.gl/XebGL> (at ramcloud.stanford.edu)
20:55 lbalbalba hrm. some 'prove' tests seem to fail and/or freeze when i add 'option transport.socket.own-thread on' :(
20:57 jdarcy That's interesting, in a bad way.
21:00 jdarcy Using separate threads seriously affects the whole execution model, so it often exposes race conditions and deadlocks that have crept in since the last time I tested it.
21:00 lbalbalba running './tests/basic/volume-status.t ' results in 'mount.nfs: mounting host2.localdomain:/patchy failed, reason given by server: No such file or directory'
21:00 lbalbalba but CTRL_Z ing it to the background and then running it from the cmdline runs fine
21:01 lbalbalba looks like the race you mention ;)
21:01 Mo__ joined #gluster
21:03 jdarcy Welp, looks like we have some bugs to fix.  Thanks!
21:04 lbalbalba youre welcome. i like to break things ;)
21:05 war|chil1 joined #gluster
21:08 ehg joined #gluster
21:19 Staples84 joined #gluster
21:20 duerF joined #gluster
21:23 satheesh joined #gluster
21:23 satheesh1 joined #gluster
21:29 jag3773 left #gluster
21:34 robinr joined #gluster
21:34 ricky-ticky joined #gluster
22:04 _pol joined #gluster
22:17 majeff joined #gluster
22:20 Skunnyk joined #gluster
22:20 wN joined #gluster
22:20 soukihei joined #gluster
22:20 rosco joined #gluster
22:20 gmcwhistler joined #gluster
22:34 nightwalk joined #gluster
22:41 joelwallis joined #gluster
22:43 tylerflint left #gluster
22:45 ccha joined #gluster
22:49 aliguori joined #gluster
22:54 theron joined #gluster
23:14 badone_ joined #gluster
23:39 jthorne joined #gluster
23:58 StarBeast joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary