Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-21

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 mattapperson joined #gluster
00:15 elyograg_ joined #gluster
00:18 elyograg_ lots of frustration.  Seeing extremely high load averages on brick servers for one of my volumes.  access is NFS, via another gluster server that has no bricks.  seems to be triggerred by processes doing lots of stats - with 'find' or 'chown -R'. gluster 3.3.1.  Most of them are 3.3.1-11, got 3.3.1-15 on the ones that are showing this problem.
00:18 tokik joined #gluster
00:19 JoeJulian The only changes between those were for stuff you're not using.
00:21 elyograg_ this is only happening on one of my volumes.
00:21 glusterbot New news from newglusterbugs: [Bug 1067733] Rename temp file created in /var/lib/glusterd/peers/ during peer probe <https://bugzilla.redhat.co​m/show_bug.cgi?id=1067733> || [Bug 1062287] On breaking the connection between replicated volumes certain files return -ENOTCONN <https://bugzilla.redhat.co​m/show_bug.cgi?id=1062287>
00:22 elyograg_ if I run 'df /mdfs' and 'df /mdfs2' ... which are on the same cluster and both NFS mounted through the same host, the first is instant and the other takes a couple of seconds.
00:23 elyograg_ mdfs is comprised of bricks on four servers, mdfs2 is comprised of bricks on two servers.  those two servers are having the high load - 32 on one, 16 on the other.
00:24 JoeJulian same number of files?
00:26 elyograg_ no.  fpaste.org/79102/13929422/
00:26 elyograg_ mdfs2 is the problematic volume.
00:27 sarkis joined #gluster
00:27 elyograg_ peer status: http://fpaste.org/79103/94243413/
00:27 glusterbot Title: #79103 Fedora Project Pastebin (at fpaste.org)
00:27 elyograg_ stupid firefox.  i don't like how it gets rid of http:// ... but only sometimes.
00:28 elyograg_ first paste again, this time clickable. http://fpaste.org/79102/13929422/
00:28 glusterbot Title: #79102 Fedora Project Pastebin (at fpaste.org)
00:28 JoeJulian At least glusterbot was smart enough to figure it out. :D
00:28 elyograg_ (if your irc client does clickable, irssi doesn't. :)
00:29 cjanbanan joined #gluster
00:31 elyograg_ one detail not apparent there: the six brick servers are accessed between themselves on a different LAN than they are accessed from the NAS servers.
00:32 m0zes joined #gluster
00:32 JoeJulian I wonder if readdirplus helps with nfs mounted volumes.
00:33 JoeJulian I'm guessing not since there's no fuse layer.
00:33 elyograg_ oh, there's another difference.  Centos 6.5 on the new servers.  6.4 on the old ones.
00:33 elyograg_ the new ones with that high load problem are on 6.5.
00:34 elyograg_ when do we get the readdir support in centos?
00:34 elyograg_ mdfs is in full production, so I can't reboot anything but the two new servers.
00:34 shyam joined #gluster
00:34 JoeJulian More files, fewer servers - if I were to guess, I'd say about 240% as many files per directory on mdfs2 as mdfs..
00:35 JoeJulian readdirplus is in the current centos kernels. Has been for a while now.
00:35 elyograg_ mdfs has 49 terabytes on it, mdfs2 has 3.8.
00:35 elyograg_ so the new volume has about one tenth the space, and probably about that in number of files too.
00:35 JoeJulian Oh, right.. I forgot which column was which.
00:36 elyograg_ that said, there are over 5 million files on mdfs2.
00:36 JoeJulian caching? If mdfs is in production, perhaps the stats are already in memory? I'm guessing here...
00:37 m0zes_ joined #gluster
00:38 JoeJulian I'm also assuming you've already checked the logs, network settings, and schedulers.
00:38 elyograg_ don't know much about thaqt.  I did just notice that the nfs server process on slc01nas1 is using 7.3GB of RAM.  If that continues to go up, NFS is going to shut down entirely.
00:39 sputnik13 joined #gluster
00:39 elyograg_ last time this happened, logs were silent on the new new machines.  until I completely shut down gluster on those two machines and started it back up, at which point everything was fine.  I'll be doing that soon, to get us out of this problem.
00:40 elyograg_ what I'm hoping is that this is a known 3.3.1 problem and when we can eventually upgrade to 3.4.2, everything will magically get better.
00:42 elyograg_ we were running for quite a while with no problems, until we added the mdfs2 volume.  we ran into the NFS server process going out of control on memory after the new volume was added.
00:43 JoeJulian I don't see any memory leak patches relating to nfs
00:43 elyograg_ that problem we eventually found, and adding the nfs.write-behind option seemed to make that get better, but now I'm not so sure.
00:43 elyograg_ it's supposed to be fixed in 3.4.
00:43 elyograg_ with the write-behind cache as a workaround.
00:43 JoeJulian Well then they didn't include that in the patch note...
00:44 elyograg_ can't remember the bug ID.  It was a bug that I don't ahve rights to view, but i found it in the release note for RHS.
00:46 mtanner_ joined #gluster
00:46 elyograg_ I need to kill things now.
00:47 overclk joined #gluster
00:48 eastz0r_ joined #gluster
00:48 haomaiw__ joined #gluster
00:49 d-fence joined #gluster
00:49 elyograg_ is there any chance that updatedb running on the brick servers would cause problems?  I need to figure out how to make it skip /bricks and its subdirectories.
00:49 johnmark_ joined #gluster
00:50 JoeJulian It does eat up a lot of cycles and i/o.
00:50 cjanbanan joined #gluster
00:50 JoeJulian I just disable it on my servers.
00:50 Peanut__ joined #gluster
00:51 jclift joined #gluster
00:51 cyberbootje joined #gluster
00:56 sprachgenerator joined #gluster
00:57 elyograg_ and another difference just noticed.  still using cfq for the i/o scheduler.  that's changed to noop on the ones that are working without a problem.
00:57 16WAAR7KB joined #gluster
00:57 lyang0 joined #gluster
00:57 nikk joined #gluster
00:57 khushildep joined #gluster
00:57 crazifyngers joined #gluster
00:57 askb joined #gluster
01:02 DV joined #gluster
01:02 16WAAR7KB joined #gluster
01:02 lyang0 joined #gluster
01:02 nikk joined #gluster
01:02 khushildep joined #gluster
01:02 crazifyngers joined #gluster
01:02 askb joined #gluster
01:03 wrale joined #gluster
01:04 refrainblue aha!  so it looks like a vm that runs OL6.5 with the latest uek doesnt work, but when i try rhck 2.6.32 it does work
01:05 JoeJulian refrainblue: good to know. Thanks for testing that.
01:05 refrainblue a guy at oracle asked me to email him the results so they can fix it in OL so ill be doing that
01:06 JoeJulian Probably should file a bug report too. It may not be gluster's problem, but maybe they'll see a workaround.
01:06 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
01:10 sroy_ joined #gluster
01:23 refrainblue uh what component would i file this under...
01:23 cjanbanan joined #gluster
01:24 refrainblue i think ill just go with unclassified
01:24 JoeJulian sounds like a plan.
01:32 sprachgenerator joined #gluster
01:39 eastz0r joined #gluster
01:52 glusterbot New news from newglusterbugs: [Bug 1067756] GlusterFS client mount fails when using Oracle Linux Unbreakable Enterprise Kernel <https://bugzilla.redhat.co​m/show_bug.cgi?id=1067756>
01:57 KaZeR_ joined #gluster
01:59 cjanbanan joined #gluster
02:07 satheesh2 joined #gluster
02:08 haomaiwa_ joined #gluster
02:11 REdOG hehe
02:19 zoldar joined #gluster
02:20 zoldar Hi. I've just done an update from 3.3.x to 3.4.2 and now when I try to mount the volume with any options besides defaults and _netdev, mount fails with bad arguments error. Is that expected? What's alternative way to enforce noexec and/or nosuid?
02:28 aliguori joined #gluster
02:39 cjanbanan joined #gluster
02:45 sputnik13 joined #gluster
02:53 sputnik13 joined #gluster
02:58 bharata-rao joined #gluster
03:01 nightwalk joined #gluster
03:05 sputnik13 joined #gluster
03:16 harish_ joined #gluster
03:22 shubhendu joined #gluster
03:28 RameshN joined #gluster
03:28 rfortier joined #gluster
03:36 itisravi joined #gluster
03:39 aquagreen joined #gluster
03:42 saurabh joined #gluster
03:49 kanagaraj joined #gluster
03:59 shyam joined #gluster
03:59 satheesh1 joined #gluster
04:04 ndarshan joined #gluster
04:09 jobewan joined #gluster
04:10 sprachgenerator joined #gluster
04:14 dusmantkp_ joined #gluster
04:16 jag3773 joined #gluster
04:23 kdhananjay joined #gluster
04:24 shylesh joined #gluster
04:24 KaZeR joined #gluster
04:26 jobewan joined #gluster
04:30 junaid joined #gluster
04:39 ppai joined #gluster
04:39 mohankumar__ joined #gluster
04:39 quique_ left #gluster
04:45 spandit joined #gluster
04:45 rjoseph joined #gluster
04:54 mohankumar__ joined #gluster
04:54 hagarth joined #gluster
04:56 tokik joined #gluster
05:00 bala joined #gluster
05:01 jobewan joined #gluster
05:05 davinder joined #gluster
05:09 m0zes joined #gluster
05:10 sarkis joined #gluster
05:10 sputnik13 joined #gluster
05:13 mohankumar__ joined #gluster
05:15 jbrooks left #gluster
05:16 tokik joined #gluster
05:16 davinder joined #gluster
05:17 ajha joined #gluster
05:19 mohankumar__ joined #gluster
05:22 sputnik13 joined #gluster
05:35 badone joined #gluster
05:40 prasanth joined #gluster
05:41 elyograg left #gluster
05:44 lalatenduM joined #gluster
05:45 bharata-rao joined #gluster
05:46 jporterfield joined #gluster
05:48 sahina joined #gluster
05:53 sputnik13 does it make a difference whether I'm running xfs on top of lvm vs just xfs?
05:53 sputnik13 with respect to block alignment
05:54 ajha joined #gluster
05:58 jporterfield joined #gluster
06:00 aravindavk joined #gluster
06:03 rastar joined #gluster
06:03 meghanam joined #gluster
06:03 meghanam_ joined #gluster
06:03 REdOG I don't think so... I had to do my xfs at 512 though
06:04 REdOG 4k was causing all kind of errors
06:09 vpshastry joined #gluster
06:12 vpshastry1 joined #gluster
06:13 mohankumar__ joined #gluster
06:14 badone joined #gluster
06:25 mohankumar__ joined #gluster
06:27 Philambdo joined #gluster
06:31 mohankumar__ joined #gluster
06:32 benjamin_____ joined #gluster
06:35 vpshastry2 joined #gluster
06:45 bharata-rao joined #gluster
06:51 vimal joined #gluster
06:55 dusmantkp_ joined #gluster
06:56 wica joined #gluster
07:00 cjanbanan joined #gluster
07:04 mohankumar__ joined #gluster
07:11 rfortier joined #gluster
07:13 jag3773 joined #gluster
07:14 mohankumar__ joined #gluster
07:18 vpshastry1 joined #gluster
07:20 aravindavk joined #gluster
07:21 aquagreen joined #gluster
07:21 jtux joined #gluster
07:21 shubhendu joined #gluster
07:21 ndarshan joined #gluster
07:23 rossi_ joined #gluster
07:23 kanagaraj joined #gluster
07:23 harish_ joined #gluster
07:24 bharata-rao joined #gluster
07:24 mohankumar__ joined #gluster
07:30 bala joined #gluster
07:33 mohankumar__ joined #gluster
07:46 bala joined #gluster
07:47 rgustafs joined #gluster
07:47 RameshN joined #gluster
07:51 ekuric joined #gluster
07:51 mgebbe joined #gluster
07:52 mgebbe joined #gluster
07:57 RedShift joined #gluster
07:59 cjanbanan joined #gluster
08:02 kanagaraj joined #gluster
08:03 ndarshan joined #gluster
08:05 _VerboEse joined #gluster
08:05 eseyman joined #gluster
08:07 ctria joined #gluster
08:11 ngoswami joined #gluster
08:12 mohankumar joined #gluster
08:12 badone joined #gluster
08:14 sahina joined #gluster
08:18 cjanbanan joined #gluster
08:18 eseyman joined #gluster
08:19 aravindavk joined #gluster
08:23 glusterbot New news from newglusterbugs: [Bug 1067852] Usage of Libgfapi and License Agreement <https://bugzilla.redhat.co​m/show_bug.cgi?id=1067852>
08:24 mohankumar joined #gluster
08:27 andreask joined #gluster
08:31 rastar joined #gluster
08:33 keytab joined #gluster
08:37 al joined #gluster
08:45 dusmantkp_ joined #gluster
08:45 shubhendu joined #gluster
08:57 REdOG joined #gluster
08:57 fsimonce joined #gluster
09:00 mohankumar joined #gluster
09:00 liquidat joined #gluster
09:06 X3NQ joined #gluster
09:09 franc joined #gluster
09:12 ndarshan joined #gluster
09:16 ajha joined #gluster
09:17 bala joined #gluster
09:18 mohankumar joined #gluster
09:29 kanagaraj joined #gluster
09:33 mohankumar__ joined #gluster
09:38 wgao joined #gluster
09:41 nikk joined #gluster
09:41 bharata_ joined #gluster
09:42 REdOG joined #gluster
09:42 satheesh4 joined #gluster
09:43 ndarshan joined #gluster
09:44 davinder joined #gluster
09:48 bala joined #gluster
09:49 lyang0 joined #gluster
09:52 an_ joined #gluster
10:24 mohankumar__ joined #gluster
10:24 X3NQ joined #gluster
10:36 DV__ joined #gluster
10:47 rossi_ joined #gluster
10:51 ThatGraemeGuy_ joined #gluster
10:52 ngoswami joined #gluster
10:53 REdOG joined #gluster
11:05 kanagaraj joined #gluster
11:06 RameshN joined #gluster
11:09 DV joined #gluster
11:13 raghu joined #gluster
11:13 Slash joined #gluster
11:16 badone joined #gluster
11:25 REdOG joined #gluster
11:26 rfortier joined #gluster
11:40 REdOG well I waited quite sometime and 3 of my bricks still don't match one another
11:40 * REdOG has lost faith in heal
11:47 an_ joined #gluster
11:55 itisravi joined #gluster
12:07 edward2 joined #gluster
12:25 mohankumar joined #gluster
12:34 mohankumar joined #gluster
12:43 rgustafs joined #gluster
12:44 X3NQ joined #gluster
12:45 ira joined #gluster
12:50 neurodrone__ joined #gluster
13:04 X3NQ joined #gluster
13:05 mohankumar joined #gluster
13:14 sprachgenerator joined #gluster
13:17 plarsen joined #gluster
13:19 hagarth joined #gluster
13:20 benjamin_____ joined #gluster
13:38 mohankumar joined #gluster
13:38 sahina joined #gluster
14:06 bennyturns joined #gluster
14:09 Slash joined #gluster
14:12 ctria joined #gluster
14:13 psyl0n_ joined #gluster
14:14 sroy_ joined #gluster
14:16 sprachgenerator joined #gluster
14:20 lman4821 joined #gluster
14:25 japuzzo joined #gluster
14:28 burn420 joined #gluster
14:30 primechuck joined #gluster
14:32 theron joined #gluster
14:36 P0w3r3d joined #gluster
14:43 mgebbe_ joined #gluster
14:44 benjamin_ joined #gluster
14:45 liquidat joined #gluster
14:45 psyl0n_ joined #gluster
14:47 mgebbe joined #gluster
14:50 mgebbe_ joined #gluster
14:51 sarkis joined #gluster
14:52 mgebbe joined #gluster
14:54 mgebbe joined #gluster
14:55 TvL2386 joined #gluster
14:55 mgebbe_ joined #gluster
14:55 cjanbanan joined #gluster
14:56 johnmilton joined #gluster
14:56 20WABBH8G joined #gluster
14:57 77CABKD7A joined #gluster
14:58 77CABKD7O joined #gluster
14:58 6JTABAXE3 joined #gluster
15:01 mgebbe_ joined #gluster
15:01 KyleG joined #gluster
15:01 KyleG joined #gluster
15:02 mgebbe joined #gluster
15:03 21WACEBB2 joined #gluster
15:05 ThatGraemeGuy joined #gluster
15:07 mgebbe_ joined #gluster
15:08 21WACEBEX joined #gluster
15:09 17WAA5W1C joined #gluster
15:11 B21956 joined #gluster
15:16 Slash joined #gluster
15:16 ctria joined #gluster
15:23 bugs_ joined #gluster
15:29 chirino joined #gluster
15:30 sputnik13 joined #gluster
15:32 calum_ joined #gluster
15:34 wrale joined #gluster
15:36 burnalot joined #gluster
15:37 burn420alot joined #gluster
15:39 nage joined #gluster
15:39 nage joined #gluster
15:41 rpowell joined #gluster
15:44 an_ joined #gluster
15:46 kmai007 joined #gluster
15:47 kmai007 morning guys, can somebody give me a break down of what each report in heal means: healed, heal-failed, split-brain
15:48 kmai007 like which of the reports require action from a human.  I believe split-brain is needing action, but im not sure about the other 2 reports
15:54 redbeard joined #gluster
15:56 mgebbe_ joined #gluster
15:57 an_ joined #gluster
15:58 natgeorg joined #gluster
15:59 77CABKFDU joined #gluster
15:59 jobewan joined #gluster
16:00 lmickh joined #gluster
16:01 mgebbe_ joined #gluster
16:02 77CABKFFW joined #gluster
16:03 japuzzo joined #gluster
16:05 mgebbe joined #gluster
16:06 vpshastry joined #gluster
16:08 japuzzo joined #gluster
16:09 jag3773 joined #gluster
16:10 japuzzo joined #gluster
16:11 cjanbanan joined #gluster
16:12 failshell joined #gluster
16:16 nikk anyone know if there's a good web ui or other monitoring/management tool?
16:17 nikk or any sort of way to monitor "stuff" via snmp (or anything)
16:17 benjamin_ joined #gluster
16:23 vpshastry joined #gluster
16:24 kaptk2 joined #gluster
16:25 vpshastry left #gluster
16:28 cjanbanan joined #gluster
16:30 XpineX joined #gluster
16:43 wrale anyone have experience with SR-IOV here? having an issue which is impeding my cluster build
16:45 al joined #gluster
16:50 bala joined #gluster
16:52 Mo__ joined #gluster
16:57 rpowell1 joined #gluster
16:57 cjanbanan joined #gluster
17:00 sprachgenerator joined #gluster
17:10 KyleG joined #gluster
17:10 KyleG joined #gluster
17:11 GabrieleV joined #gluster
17:12 theron joined #gluster
17:13 tdasilva joined #gluster
17:24 neofob left #gluster
17:36 cjanbanan joined #gluster
17:36 sputnik13 joined #gluster
17:36 vpshastry joined #gluster
17:47 redbeard joined #gluster
17:51 vpshastry left #gluster
17:57 cjanbanan joined #gluster
18:05 natgeorg joined #gluster
18:05 natgeorg joined #gluster
18:13 rwheeler joined #gluster
18:19 Slash joined #gluster
18:20 Matthaeus joined #gluster
18:25 purpleidea joined #gluster
18:26 cjanbanan joined #gluster
18:27 rossi_ joined #gluster
18:28 neurodrone joined #gluster
18:30 rossi_ nikk:  icinga, nagios, ganglia, munin ... waht do you want to monitor?
18:30 rossi_ s/waht/what/
18:30 glusterbot What rossi_ meant to say was: nikk:  icinga, nagios, ganglia, munin ... what do you want to monitor?
18:44 nikk rossi_: mostly disk latency
18:45 theron joined #gluster
18:45 nikk i'm seeing some curious behavior.. maybe someone is familiar with this.  i have one volume, four servers, one brick per server.  on two of the servers if i look at the actual files on local disk i see all of the files on the folume.. on two of the servers i see nothing.
18:45 nikk all four are mounting from localhost
18:45 nikk sometimes the files show up but promptly dissapear
18:45 nikk note that everything is always available from the actual gluster mountpoint
18:46 nikk unless i shut down the first two servers.. in which case the second two have nothing cached at all
18:46 nikk very confusing
18:47 theron joined #gluster
18:48 rossi_ nikk: monitoring: seekwatcher for one time check, maybe you can do something with systemtap (http://dtrace.org/blogs/brenda​n/2011/10/15/using-systemtap/)
18:48 rossi_ nikk: sorry, i am not very familiar with gluster yet
18:48 rossi_ nikk: did you check the log files frome each server?
18:52 samppah nikk: what kind of configuration you have and how did you set it up?
18:57 KyleG left #gluster
18:57 cjanbanan joined #gluster
18:58 qubit left #gluster
19:02 nikk samppah: four physical servers, four bricks (one brick per server), i created the volume starting with two bricks then added the other two after (was testing being able to add servers)
19:02 nikk the weird thing is on the second two servers the files are not always stored on local disk where as on the first two servers they are always there
19:03 nikk rossi_: yeah i can use standard disk i/o checks for the volume, was wondering if there was anything more gluster-specific
19:03 nikk i haven't really seen anything
19:04 nikk i can create a nagios check by having a file like /check that is incremented with the date every minute, nagios checks all servers to make sure that date isn't less than x minutes less than the current date
19:05 chucky_z joined #gluster
19:08 tg2 any idea why one of my bricks would be accepting more files than another?
19:08 aixsyd joined #gluster
19:09 samppah nikk: can you send output of gluster volume info to pastie.org?
19:10 aixsyd I know this isnt gluster-specific - but i know how knowledgable ya'll are - and we can take it to PMs, but does anyone here know anything about AP UPS monitoring with apcupsd and monitoring multiple (like, 2) UPSes at once?
19:10 aixsyd *APC UPSes
19:11 nikk http://fpaste.org/79319/30098041/
19:11 glusterbot Title: #79319 Fedora Project Pastebin (at fpaste.org)
19:17 zaitcev joined #gluster
19:17 samppah nikk: thanks.. do you still have commands which you ran to add two more bricks?
19:18 nikk the window has been closed since.. it should be something like.. volume add-brick gfs1 replica 2 centos3:/mnt/sdb1/gluster1 centos4:/mnt/sdb1/gluster1
19:19 nikk the bricks show up in the volume and i can view the files perfectly fine from the mount
19:19 nikk each server is running glusterd
19:19 nikk and each server is mounting localhost:/gfs1
19:19 nikk the idea is to have a horizontally scalable stack instead of having everything acting as a client
19:19 ndk joined #gluster
19:19 nikk in the case of a network outage clients end up breaking
19:25 samppah nikk: you should also run rebalance fix-layout at least after add brick to able to spread new data to new servers
19:26 samppah nikk: your current configuration is distributed replicated and data is replicated between two bricks
19:26 cjanbanan joined #gluster
19:27 nikk yeah tried fix-layout, heal, and sync
19:27 samppah ach
19:28 nikk erm, it should have been replica only, not distributed replica
19:28 rwheeler joined #gluster
19:28 nikk that would certainly explain it
19:28 nikk derp
19:28 samppah yes indeed.. although you should definetly see some data in new bricks
19:28 samppah btw.. i'm not sure if it's a good idea to use replica 4 :)
19:33 nikk hmm
19:33 nikk easy to convert that from distributed-replica to regular replica?
19:33 nikk i specified replica 2
19:33 nikk but i'll try removing the second set and re-adding
19:34 samppah yeah, you need to remove 2 bricks and then readd them and specify replica 4
19:35 nikk you said that i wouldn't want replica 4 - what's wrong with that?
19:39 samppah nikk: replication is handled on client side.. basicly your writespeed is your bandwidth / 4.. it will also affect reads as well becaus it has to check all 4 bricks for file consistency
19:40 nikk yeah that's what i thought
19:40 semiosis tg2: are those bricks replicas?  if they are then that's a problem, as they should have exactly the same files.  if not, then it might be normal.  glusterfs distributes files by hashing the filenames
19:40 nikk so i'm looking for something that scales horizontally (start with four servers, will add two more per site, then may double that later)
19:40 semiosis tg2: in large numbers files will be distributed evenly.  but for small number of files sometimes it's a little off balance
19:40 nikk each server needs to be able to exist on it's own if a wan link breaks and if one of the servers at each location is down
19:41 nikk so yeah..
19:41 samppah hmm
19:42 MacWinner joined #gluster
19:44 nikk i've looked at a lot of things.. ceph, leo, hadoop, nfs+cachefs, lsync
19:44 nikk my two best options are gluster for everything or local files with nfs for dynamic files
19:44 nikk both have their own pros & cons
19:45 nikk so to confirm.. if a file is written to one server, that server will tell everyone else that it's written
19:45 nikk when it reads a file, does it check with all other servers to see if it's proper?
19:45 nikk or just use whatever it has already
19:45 nikk (assuming it's not cached in memory already)
19:46 nightwalk joined #gluster
19:48 nikk any ideas what a good solution for that would be?  these are servers in multiple datacenters across a wan that all need sync'd filesystems fairly quickly
19:48 nikk gluster or otherwise
19:49 samppah nikk: do you have multiple machines in one datacenter and do you need rw access to all files?
19:50 nikk four (possibly more) hosts in each location.. starting with three locations, might go to four or five in the next five years
19:50 nikk all would need rw
19:50 nikk 99% of traffic is read
19:50 nikk very few writes
19:51 nikk and of those, most writes would be deploying new code
19:52 samppah hmm.. read locally and use gluster geo-replication to write/distribute to other bricks?
19:52 nikk i'm not familiar with the geo-replication at all but i've been meaning to look
19:53 nikk i still have to worry about scaling out per-datacenter though
19:53 nikk going from two -> four servers in one dc
19:53 nikk and it'll be many more over the next few years
19:55 nikk i know limelight usesdistributed-replicated volumes with 50+ bricks across wan without geo so i figured i'd give that a shot first :)
19:55 * REdOG can't seem to self heal even after deleting bricks
19:57 nikk gluster geo-replication is master/slave it looks like (from my three seconds of reading) instead of master-master like i'd need
20:01 theron joined #gluster
20:03 semiosis nikk: replica 4 over wan is probably not going to meet your performance expectations
20:05 semiosis [14:45] <nikk> when it reads a file, does it check with all other servers to see if it's proper?
20:05 semiosis yes every time a file is opened the client checks all replicas to make sure the file is in sync
20:06 semiosis nikk: how do you know about limelight?  got a link?
20:12 pdrakewe_ joined #gluster
20:14 puddles joined #gluster
20:14 B21956 joined #gluster
20:16 puddles anyone know why a replica 2 with 4 bricks of (1tb, 250gb, 300gb, 1tb) would be showing only 550gb free?  There's enough space across the 4 to have 1.25tb available.
20:17 puddles glusterfs 3.4.2
20:19 cjanbanan joined #gluster
20:19 REdOG is there a clean way to manually delete a volume?
20:20 * REdOG has one fubar and is ready to just restore from backup
20:20 nikk semiosis: yeah that'll end up getting slower and slower the more nodes added then :\
20:20 nikk especially over a wan
20:21 nikk i might have to look at another solution, not sure if gluster is going to satisfy master/master over a wan
20:21 semiosis nikk: it's really not meant to do that
20:21 nikk yeah
20:21 nikk square peg :)
20:21 nikk one thing i saw people doing is doing full replica then reading from local disk instead of from the actual gluster mount
20:21 nikk then writing to the gluster mount
20:22 nikk slow writes fast reads
20:26 nikk seems ghetto but my buddy said it's worked well for them
20:32 semiosis yep just make sure you mount that brick filesystem with noatime,nodiratime -- so that your reads don't modify the filesystem at all
20:37 nikk *sigh*
20:40 ktosiek_ joined #gluster
20:45 semiosis nikk: why do you need multi-master for code deployments?  code should be stored in, and deployed from, your source of truth -- i.e. VCS
20:46 dbruhn joined #gluster
20:46 nikk this particular use case is for things like vbulletin, wordpress, static content headed by apache, lighttpd, etc.
20:47 dbruhn jbrooks talk at SCALE 12 was pretty sweet
20:48 semiosis nikk: hmm, then why not just have a single instance and use a CDN?
20:48 nikk we do use a cdn
20:48 nikk three actually :)
20:49 semiosis hmm
20:49 semiosis ok
20:49 ctria joined #gluster
20:49 nikk it's to satisfy the requirements of DR
20:49 nikk multiple countries, different rules
20:50 semiosis nikk: seems like master/slave would be enough for DR
20:50 nikk it gets better though
20:50 semiosis fail over to the DR site & make it master
20:50 semiosis hehe i'm sure it does
20:50 nikk they need to be active/active
20:50 semiosis whyyy?
20:50 nikk haha
20:50 nikk that's what i keep saying
20:51 nikk it's just our industry
20:51 nikk but i'm going to find a way around it
20:51 nikk because i've had no luck with any product satisfying the requirements
20:51 semiosis well, wordpress needs a sql db, right?  what are you doing about the db req's?
20:51 semiosis can you do multi-master mysql across wan?
20:52 semiosis how'd you solve that?
20:52 nikk both percona and mariadb play nice with high-latency replication
20:52 nikk and are *ok* at resolving split brain without intervention
20:52 semiosis gotcha
20:53 nikk i'm wondering if varnish is able to frontend an object using multiple backends
20:53 semiosis i have varnish set up to round-robin between identical backends
20:53 nikk for example if i request /crap.html and it doesn't find the file in /var/www/crap.com... if i can go to /mnt/otherserver1/crap.com and see if it's there
20:53 nikk then make an nfs mesh
20:53 nikk haha
20:53 nikk yeah we do that now
20:55 kmai007 Should I be concerned with this message?   [fuse-bridge.c:3526:fuse_setlk_cbk] 0-glusterfs-fuse: 378921773: ERR => -1 (No such file or directory)
20:55 kmai007 it doesn't tell me what file or directory
20:55 kmai007 where it logged in the FUSE logs
20:55 kmai007 i suppose its 2 messages with the same meaning
20:56 kmai007 [2014-02-21 20:53:59.715332] W [client-rpc-fops.c:2203:client3_3_lk_cbk] 0-devstatic-client-0: remote operation failed: No such file or directory
20:56 kmai007 [2014-02-21 20:53:59.715394] W [fuse-bridge.c:3526:fuse_setlk_cbk] 0-glusterfs-fuse: 378921773: ERR => -1 (No such file or directory)
20:56 glusterbot New news from newglusterbugs: [Bug 1068781] glfs_read fails for large read <https://bugzilla.redhat.co​m/show_bug.cgi?id=1068781> || [Bug 1068776] Sharing RHS volume subdirectory via Samba causes error messages in log.ctdb <https://bugzilla.redhat.co​m/show_bug.cgi?id=1068776>
21:05 cjanbanan joined #gluster
21:20 psyl0n joined #gluster
21:23 jobewan joined #gluster
21:26 cp0k joined #gluster
21:30 REdOG how do I manually delete a volume? gluster vol delete isn't working and I want to obliterate it
21:31 REdOG Ive tried what I can but when i try to recreate it I get told volume already exists
21:45 cjanbanan joined #gluster
21:50 badone joined #gluster
21:56 rpowell1 REdOG:  did you delete or atleast clear the xattrs on the brick directory ?
21:56 cjanbanan joined #gluster
21:57 orion7644 joined #gluster
21:57 rpowell1 REdOG:  gluster vol delete will just remove the it from config files and the like.  The underlying filesystem (thankfully) remains untouched
22:09 bennyturns joined #gluster
22:18 failshel_ joined #gluster
22:24 xavih_ joined #gluster
22:27 glusterbot New news from newglusterbugs: [Bug 1066128] glusterfsd crashes with SEGV during catalyst run <https://bugzilla.redhat.co​m/show_bug.cgi?id=1066128>
22:34 xavih_ joined #gluster
22:44 primechuck joined #gluster
22:52 cjanbanan joined #gluster
22:59 qdk joined #gluster
23:03 cjanbanan joined #gluster
23:27 cjanbanan joined #gluster
23:29 failshell joined #gluster
23:32 sprachgenerator joined #gluster
23:32 jobewan joined #gluster
23:39 tokik joined #gluster
23:56 cjanbanan joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary