Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-26

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 daMaestro so you'd have to probe 1 -> 2 and 2 -> 1
00:00 nullsign_ damaestro: ah, not active.
00:00 nullsign_ i probed both ways, yes.
00:00 nullsign_ and yes, i stopped, remove the peer files, restarted, showed no peers, then probed each other.
00:01 nullsign_ then got 1 peer each, each one shows connected.
00:02 daMaestro k, let's put glusterd into debug mode so we can see what it's seeing
00:03 daMaestro iirc there is a configuration setting you can configure and then restart glusterd for it to take affect
00:03 daMaestro let me start up my test cluster
00:06 daMaestro ah, that was only for volume settings
00:07 daMaestro nullsign_, in /etc/sysconfig/glusterf* set the log levels to DEBUG
00:08 daMaestro restart the service and try the volume create again
00:08 daMaestro look for the failure in the logs and pastebin it please
00:08 nullsign_ k
00:09 nullsign_ hrm.. im using ubuntu, gotta find that file on ubuntu...
00:09 nullsign_ sec
00:13 nullsign_ ug.
00:19 nullsign_ hrm..
00:19 nullsign_ progress..
00:20 nullsign_ damaestro: http://pastie.org/pastes/5433764/reply
00:20 glusterbot Title: New - Pastie (at pastie.org)
00:27 daMaestro nullsign_, so do the volume create and then see what the glusterd says in debug mode
00:28 nullsign_ i did. i posted it in the pastie.
00:28 daMaestro i don't see it
00:28 daMaestro i'm over here trying to build my lxc container to stand up some new peers to see if i can reproduce this issue
00:29 daMaestro and everything is all f*kkered because we changed the repo layouts
00:32 mnaser can you run glusterfs all in a single process?
00:32 nullsign_ sorry, pasted wrong log; corrected here - http://pastie.org/5433887
00:32 glusterbot Title: #5433887 - Pastie (at pastie.org)
00:38 daMaestro nullsign_, for my quick check .. blow stuff away again and use the IPs of the servers
00:38 daMaestro if that works, it's a dns issue that is easily fixed
00:56 UnixDev joined #gluster
00:58 nullsign_ it tells me i can't use ips...
00:58 nullsign_ when trying to do the add :/
00:58 nullsign_ hrm.. dns, im using hostnames, not dns.. maybe that is the issue
00:58 nullsign_ /etc/hosts, i mean
00:59 daMaestro joined #gluster
01:13 yinyin joined #gluster
01:18 kevein joined #gluster
01:21 jluls joined #gluster
01:21 jluls JoeJulian - are you around?
01:22 jluls Anyone around in #gluster?
01:49 mohankumar joined #gluster
01:49 jluls hey
01:57 daMaestro joined #gluster
01:57 jluls anyone alive?
02:02 daMaestro just us chickens
02:02 jluls Need some tech support.  Wanna help?
02:07 yosafbridge joined #gluster
02:17 daMaestro just ask your question
02:17 yinyin joined #gluster
02:23 bala1 joined #gluster
02:28 sunus joined #gluster
02:32 pdurbin heh
02:47 sunus joined #gluster
02:52 daMaestro joined #gluster
03:21 bulde joined #gluster
03:27 yeming joined #gluster
03:40 johnmark oh dear
04:02 hagarth joined #gluster
04:04 UnixDev_ joined #gluster
04:07 saz joined #gluster
04:17 raghu joined #gluster
04:25 hagarth joined #gluster
04:50 yinyin joined #gluster
04:52 glusterbot New news from newglusterbugs: [Bug 832632] [Feature Request] Copy On Write (COW) <http://goo.gl/kKjq6> || [Bug 878652] Enchancement: Replication Information in gluster volume info <http://goo.gl/dWQnM> || [Bug 878004] glusterd segfaults in remove brick <http://goo.gl/KCswd> || [Bug 874554] cluster.min-free-disk not having an effect on new files <http://goo.gl/xbQQC>
04:58 yinyin joined #gluster
05:07 UnixDev joined #gluster
05:25 vijaykumar joined #gluster
05:30 sgowda joined #gluster
05:34 mdarade1 joined #gluster
05:42 vpshastry joined #gluster
05:46 pranithk joined #gluster
05:49 hagarth joined #gluster
05:52 glusterbot New news from newglusterbugs: [Bug 880050] RFC: replace passwordless ssh with gluster CLI <http://goo.gl/YQzks>
05:53 mdarade1 left #gluster
05:57 mohankumar joined #gluster
05:59 bala2 joined #gluster
06:22 ramkrsna joined #gluster
06:22 ramkrsna joined #gluster
06:35 ackjewt joined #gluster
06:36 overclk joined #gluster
06:46 khushildep joined #gluster
07:01 ngoswami joined #gluster
07:01 sunus hi, i am reading /api/src.glfs-fops.c  i notice there is a FIXME in line 571, but no explaination at all, what is that?
07:02 rgustafs joined #gluster
07:03 rudimeyer joined #gluster
07:04 Humble joined #gluster
07:19 puebele joined #gluster
07:23 shireesh joined #gluster
07:23 ngoswami joined #gluster
07:26 ally joined #gluster
07:29 ankit9 joined #gluster
07:30 Humble joined #gluster
07:33 guigui3 joined #gluster
07:51 rudimeyer There must be somebody here who runs Gluster on Amazon EC2 on EBS disks?
07:51 toruonu joined #gluster
07:52 toruonu Quick Q, I have volume of 12 bricks with 3x replication. I had one node down for ca 40 minutes and now I see in gluster volume heal home0 info that it lists a bunch of files. I hit the heal already a few times and it's been 3 days, but it's still in the same state
07:54 lkoranda joined #gluster
07:57 Azrael808 joined #gluster
07:57 ekuric joined #gluster
07:59 hagarth joined #gluster
08:04 toruonu anyone alive here?
08:04 deepakcs joined #gluster
08:05 ndevos joined #gluster
08:05 ctria joined #gluster
08:08 andreask joined #gluster
08:11 toruonu how can I see volume options? I only see gluster volume set <vol> <key> <val> in help, but no get function or similar
08:14 bulde toruonu: what do you mean by volume options? 'gluster volume info' ? or 'gluster volume set help' ? or xlator options which are written in gluster's volspec file?
08:14 toruonu bulde: I wanted to know what quorum setting was currently enabled for my volume
08:15 toruonu it seems that gluster volume info only shows settings that have been changed with respect to default
08:15 toruonu not showing what the default is sadly
08:17 Humble joined #gluster
08:18 bulde toruonu: quorum? are you using the git head?
08:19 bulde because i don't think we have any release with quorum support out yet... the patches are just committed to upstream
08:19 toruonu joined #gluster
08:20 toruonu grr… network hiccup… any chance to re-say what I lost since I mentioned that only changed settings are mentioned in gluster volume info
08:21 bulde toruonu: which version are you using?
08:23 toruonu 3.3.1
08:23 bulde i doubt the quorum patches are in that release
08:23 toruonu mm… that's the latest :)
08:23 toruonu and it did change the quorum when I set the setting
08:23 bulde i was just saying, quorum patches just made it to the upstream (ie, master branch, without any release)
08:24 toruonu Options Reconfigured:
08:24 toruonu cluster.quorum-type: auto
08:24 bulde toruonu: hmm... let me check with more informed folks :-)
08:24 toruonu anyway, right now my more immediate problem is the heal info showing files that it won't heal… according to split-brain there is no split-brain
08:25 toruonu what's the recommended pastebin like pasting location here?
08:26 toruonu anyway, here's the info
08:26 toruonu http://pastebin.com/f8A3cnyp
08:26 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
08:26 toruonu btw I don't see any ads on pastebin :P
08:26 andreask left #gluster
08:29 mario_ joined #gluster
08:34 toruonu I need to kick my ISP's ass …
08:36 bulde toruonu: you didnt miss much while you were out.. :-)  let me check the pastebin
08:45 kevein joined #gluster
08:47 vijaykumar joined #gluster
08:49 rgustafs joined #gluster
08:52 hchiramm_ joined #gluster
08:53 sshaaf joined #gluster
08:55 mario_ joined #gluster
08:55 hagarth joined #gluster
08:57 deepakcs @paste
08:57 glusterbot deepakcs: For RPM based distros you can yum install fpaste, for debian and ubuntu it's dpaste. Then you can easily pipe command output to [fd] paste and it'll give you an url.
09:00 hagarth @channelstats
09:00 glusterbot hagarth: On #gluster there have been 48312 messages, containing 2124515 characters, 353718 words, 1472 smileys, and 190 frowns; 352 of those messages were ACTIONs. There have been 17076 joins, 594 parts, 16561 quits, 3 kicks, 32 mode changes, and 5 topic changes. There are currently 162 users and the channel has peaked at 188 users.
09:01 mgebbe_ joined #gluster
09:04 dobber joined #gluster
09:05 tjikkun_work joined #gluster
09:08 gbrand_ joined #gluster
09:13 hchiramm_ joined #gluster
09:14 toruonu joined #gluster
09:14 toruonu ok, let's hope my network issues are solved for now … was there anything I missed about the pastebin
09:21 sgowda joined #gluster
09:21 vpshastry joined #gluster
09:27 ekuric joined #gluster
09:28 frakt Hi. Is features.limit-usage working properly in glusterfs 3.3? Not sure if I should use it or not :)
09:29 mooperd joined #gluster
09:29 mooperd Hello
09:30 glusterbot mooperd: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:30 mooperd go away glusterbot
09:30 ekuric1 joined #gluster
09:30 mooperd How much memory is recomended with gluster?
09:31 ekuric1 joined #gluster
09:31 yinyin joined #gluster
09:34 shireesh joined #gluster
09:36 frakt should I be using _netdev when mounting?
09:36 frakt unknown option _netdev (ignored)
09:39 toruonu as it's fairly possible that someone answered, but I got disconnected (already pissed at my ISP and had heated words with their tech support), then I'm reposting here the paste: http://pastebin.com/f8A3cnyp those files don't seem to heal away, ideas how to fix? and glusterbot: there are no ads so stop yammering about pastebin
09:39 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
09:43 frakt :D
09:45 vpshastry joined #gluster
09:45 sgowda joined #gluster
09:52 mario_ joined #gluster
09:59 toruonu ok, what's the support e-mail as it seems I'm not getting an answer here (probably too early for EU, too late for US) so better to send an e-mail and wait ...
10:02 bala1 joined #gluster
10:03 20WABMED0 joined #gluster
10:07 Alpinist joined #gluster
10:17 ekuric joined #gluster
10:17 ekuric joined #gluster
10:20 bala1 joined #gluster
10:27 mooperd www.gluster.org
10:27 guigui1 joined #gluster
10:32 hagarth joined #gluster
10:37 kspr joined #gluster
10:41 torbjorn__ joined #gluster
10:42 torbjorn__ There is a small error in http://download.gluster.org/pub/gluster​/glusterfs/3.3/LATEST/Debian/readme.txt  n line 3, piping wget output into apt-key. apt-key needs "-" as an argument to read stdin, so I guess the last part of the pipe should read "apt-key add -"
10:42 glusterbot <http://goo.gl/qG0YB> (at download.gluster.org)
10:49 khushildep joined #gluster
10:53 rudimeyer_ joined #gluster
10:54 manik joined #gluster
11:04 Hymie http://pastebin.org
11:04 Hymie http://pastebin.com/
11:04 glusterbot Title: Pastebin.com - #1 paste tool since 2002! (at pastebin.org)
11:04 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
11:16 mgebbe_ joined #gluster
11:41 andreask joined #gluster
11:56 glusterbot New news from newglusterbugs: [Bug 880157] libgfapi - samba integration <http://goo.gl/GkTQw>
11:57 guigui1 joined #gluster
12:04 bala1 joined #gluster
12:28 toruonu joined #gluster
12:29 toruonu are there any tuning recommendations for various configurations or use cases of glusterfs?
12:39 toruonu ok, let's ask it differently… anyone using glusterfs as shared /home ?
12:42 Humble joined #gluster
12:44 ndevos toruonu: yes, people do (not me)
12:46 toruonu yes, would love ot hear about how they've configured it and what kind of performance they see
12:46 toruonu I'm mostly interested in getting simple commands like ls etc to be faster
12:46 toruonu right now moving between directories and doing ls is slow
12:46 toruonu actual transfers are fine
12:47 toruonu but that kind of affects the generic workflow
12:47 ndevos well, you dont want to use an alias for ls, try /bin/ls instead
12:47 toruonu hmm.. why is the use of alias bad?
12:48 ndevos all the fancy --color etc options cause more data to be transferred, 'ls' may even do a stat() per file, and not only read all the directory entries
12:49 toruonu well … tab completion isn't fast either, definitely lagging behind local use
12:49 toruonu and I have to say the color stuff is actually quite useful if you have a large mixture of stuff in the directory :)
12:50 ndevos it definitely is useful, but will cause many more rpc traffic between the client <-> server
12:50 toruonu I'd assume filesystem index isn't terribly large usually … can't that be somehow pre-loaded and cached...
12:51 toruonu I'm guessing it partially is cached as second ls is faster usually
12:52 toruonu just did a time ls in a new directory that has 190 entries in it (half are symlinks). It took 9.3 seconds
12:52 toruonu that's not normal
12:54 balunasj joined #gluster
12:55 toruonu though /bin/ls is indeed way faster … 2.2s for 3235 files in a directory
12:56 toruonu same directory just plain ls (the alias) is still running
12:56 edward1 joined #gluster
12:59 deepakcs joined #gluster
13:00 toruonu took 101 seconds!!!
13:10 toruonu ok, I can understand that disabling the ls alias for color is a nice thing, but a lot of applications are horrendously slow as well…
13:10 vincent_1dk joined #gluster
13:10 zoldar_ joined #gluster
13:10 MinhP_ joined #gluster
13:10 Zenginee1 joined #gluster
13:11 samppah_ joined #gluster
13:13 duerF joined #gluster
13:14 toruonu hmm… a software that's supposed to create a tarball and a bunch of subdirectories is failing with disk I/O error
13:14 toruonu I can list files I can create files so I'm a bit puzzled
13:16 tc00per joined #gluster
13:17 zwu joined #gluster
13:18 vincent_1dk joined #gluster
13:20 toruonu is there a problem running sql databases on glusterfs? some feature issue? because I think the reason the software crashes with disk i/o error is because it's trying to put a small database where it keeps job status information
13:20 toruonu crab: DB Installation error : 'disk I/O error'
13:20 toruonu there's a 0 byte crabDB file that's created, but not filled...
13:21 toruonu I'm gonna run an strace to see what it does and where it fails
13:24 ankit9 joined #gluster
13:25 toruonu oook … now I have a feeling that some of my files have been "restored" over … I rewrote a bunch of them to a new config and suddenly between commands they reverted to their old selves
13:25 ally left #gluster
13:26 toruonu ok, that may have been my issue actually
13:26 toruonu odd
13:27 vijaykumar joined #gluster
13:28 toruonu oook… and now it works in creating directorie
13:28 toruonu dafuck...
13:31 toruonu ok just out of curiosity … am I the only one active on the channel? :P
13:31 toruonu besides glusterbot...
13:31 kkeithley define active. ;-)
13:32 toruonu well the whole history screen seems to be me yammering and people logging in/out :)
13:32 bala1 joined #gluster
13:33 toruonu @paste
13:33 glusterbot toruonu: For RPM based distros you can yum install fpaste, for debian and ubuntu it's dpaste. Then you can easily pipe command output to [fd] paste and it'll give you an url.
13:33 ndevos toruonu: where are you creating/modifying the files? through a mounted volume?
13:33 toruonu ndevos: yes
13:33 toruonu volume is running on storage nodes and mounted on client nodes
13:34 ndevos toruonu: okay, and you dont touch the files on the storage nodes at all?
13:34 toruonu nop
13:35 toruonu so I had about 3-4x attempted the command and got this Disk I/O error and now all of the sudden it's working
13:35 toruonu and I know I'm not the only one, I got reports from a user who also attempted to use the same tool and got the same error, while she also can work ordinarily on the filesystem
13:36 toruonu but the task creation itself is also extremely slow … hence my question if there's a generic way to tune the mount or the filesystem so that it's more responsive to small iops
13:37 ndevos hmm, strange... maybe thats part of the reason why databases are not a suitable use-case for glusterfs
13:37 ndevos toruonu: you could try mounting over nfs and see if that performs better
13:37 toruonu it has 12 bricks behind it and a fast 10G fabric (each of the 6 storage nodes has 2 bricks for this and 2x10G network) so it should be faster or only marginally slower than local disk (single)
13:39 toruonu I googled and there are reports since 3.1.0 that MySQL at least can run on glusterfs and is only about 35% slower than local disk…
13:40 ndevos well, disks is one thing, cluster-wide locking of files is something that is needed too
13:41 toruonu yes but that would only be a problem that would affect me if I did stuff in parallel on multiple machines
13:42 toruonu with locks preventing stuff… right now I know I'm working in this directory only on one node and on one shell
13:42 toruonu aaand again the disk io error
13:43 plarsen joined #gluster
13:43 toruonu that "database" is probably a postgres local db used only by the process which I run to keep state info of the hundreds of jobs I'm about to submit to Grid … so it's not a parallel access by various processes or users, but only by me so it should not have those locking issues … and then it's even weirder that creating the task sometimes works and sometimes gives i/o error
13:47 raghu joined #gluster
13:49 toruonu it looks like a read lock setting fails:
13:49 toruonu http://fpaste.org/VBOw/
13:49 glusterbot Title: Viewing 20075 stat("/home/mario/Summer12/CMS ... e/crabDB", O_RDWR|O_CREAT, 0644) = 5 ... 1 ENOENT (No such file or directory) (at fpaste.org)
14:00 toruonu no thoughts?
14:00 toruonu ok, I have to run now, be back in 3-4h to discuss again
14:02 duerF joined #gluster
14:03 aliguori joined #gluster
14:03 hchiramm_ joined #gluster
14:05 shireesh joined #gluster
14:08 mario_ joined #gluster
14:10 robo joined #gluster
14:12 vijaykumar left #gluster
14:15 tqrst joined #gluster
14:16 tqrst is a lot of "ls: cannot access /mnt/.../somefolder: Invalid argument" to be expected if only one out of several gluster servers are running?
14:16 tqrst I just updated to 3.3.1 and started only one server to make sure everything was running ok
14:16 tqrst I've only seen this happen on files before - not folders
14:17 plarsen joined #gluster
14:18 tqrst (I thought the whole folder structure was replicated on all bricks, so why would I see an error message like this on *folders*?)
14:21 tqrst Starting the other servers seems to have gotten rid of the error messages. I'm still curious why that would have happened at all, though.
14:21 mario__ joined #gluster
14:26 hagarth joined #gluster
14:26 Azrael808 joined #gluster
14:31 plarsen joined #gluster
14:39 rudimeyer joined #gluster
14:41 lh joined #gluster
14:41 lh joined #gluster
14:42 ankit9 joined #gluster
14:46 duerF joined #gluster
14:46 neofob joined #gluster
14:51 sjoeboo joined #gluster
14:59 stopbit joined #gluster
14:59 rwheeler joined #gluster
15:02 Humble joined #gluster
15:05 saz joined #gluster
15:06 noob2 joined #gluster
15:16 rudimeyer joined #gluster
15:19 chirino joined #gluster
15:19 chirino Are you guaranteed to be protected against split-brain if cluster.quorum-type is set to auto?
15:20 bit4man joined #gluster
15:24 wushudoin joined #gluster
15:26 glusterbot New news from newglusterbugs: [Bug 880241] Basic security for glusterd <http://goo.gl/rOZ3P>
15:30 ekuric joined #gluster
15:31 tqrst ever since I updated to 3.3.1, 'gluster volume myvol rebalance start' errors out with 'Starting rebalance on volume myvol has been unsuccessful', along with the following errors in the logs: http://pastie.org/private/mfw5r8e7zepo1urhspwgvg  . Any ideas?
15:31 glusterbot <http://goo.gl/dfAs6> (at pastie.org)
15:33 noob2 chirino: no you're not
15:34 chirino noob2: could you comment on http://community.gluster.org/q/how-can​-i-cause-split-brain-in-glusterfs-when​-cluster-quorum-type-is-set-to-auto/ then?
15:34 glusterbot <http://goo.gl/5RvKA> (at community.gluster.org)
15:34 noob2 what's your replica level?
15:34 chirino lets assume 3.
15:35 noob2 ok
15:35 noob2 at 3 i think you're safe
15:35 noob2 at 2 i don't think it does much
15:35 johnmark noob2: right. we're coming up with a solution for replicat 2 setups
15:36 johnmark which should be available for 3.4
15:36 noob2 johnmark: do you know approx when 3.4 is landing?
15:37 chirino So what's the problem /w 2?
15:38 chirino if you have network spit, only the first brick is allowed to write I thought?
15:38 noob2 it's still possible with a network split you could write to both nodes
15:39 noob2 when you say network split are you saying one side of your network goes down or that your network becomes segmented?
15:39 chirino segmented.
15:39 noob2 ok in that case i think it's still possible you could write to both nodes and have a conflict
15:41 jdarcy The most common cause of split brain is actually "split brain in time" rather than network partition.
15:41 noob2 exactly
15:41 noob2 i have those occur most often
15:41 jdarcy Server X goes down, changes are made, server X comes up and Y goes down, more changes are made *to the same files/directories*, voila!  Split brain.
15:41 chirino how do you get that to occur?
15:42 noob2 that's what i experienced when i was rsyncing and rebooted a node
15:42 johnmark noob2: early to mid january
15:43 jdarcy Even quorum doesn't completely solve that.  Imagine rotating failures among three replicas while updates continue.  Easy to end up with no two copies being alike.
15:43 chirino so it's basically because a brick allows you to write to it before it gets healed?
15:43 noob2 johnmark: thanks :)  i look forward to it
15:44 noob2 chirino: you can set it so that bricks go read only in a conflict until you resolve it but you'll have to intervien to fix it
15:44 jdarcy chirino: Basically yeah.  There's a feature called "outcast" that's supposed to prevent that, but that only kicks in between the start/end of self-heal.  A separate mechanism is necessary for the time before self-heal even kicks in.
15:44 chirino would be nice if a brick started as outcast if it's part of a quorum.
15:45 jdarcy IMO a brick shouldn't come up (from clients' perspective) until self-heal has had a chance to look at the change list and either heal or mark as outcast for everything on it.
15:45 chirino jdarcy: +1
15:45 jdarcy chirino: Working on it.  ;)
15:45 ron-slc joined #gluster
15:45 ngoswami_ joined #gluster
15:46 pdurbin any thoughts on benchmarking VM performance if we move the disk image from NFS (or local disk) to gluster? http://irclog.perlgeek.de/cr​imsonfu/2012-11-26#i_6185600
15:46 glusterbot <http://goo.gl/bPHQ2> (at irclog.perlgeek.de)
15:46 jdarcy chirino: The tricky bit is the bootstrap case.  You don't want all replicas coming up outcast.
15:46 ngoswami__ joined #gluster
15:47 mdarade1 joined #gluster
15:47 Staples84 joined #gluster
15:47 bdperkin joined #gluster
15:47 chirino jdarcy: on startup get a quorum to decide who was the eldest online brick. and have him heal the rest
15:47 jdarcy pdurbin: Without replication, should be pretty comparable.  With replication, you'd be taking a pretty significant performance hit (but getting data protection in return).
15:48 pdurbin jdarcy: ok. we are replicating. we are finding our kickstarts take over 10 minutes longer on gluster than NFS
15:49 pdurbin the data protection and high availability is great. we're able to reboot a gluster server and the VM keep running
15:49 pdurbin keeps
15:49 jdarcy chirino: Actually youngest (most up to date) would be better, but more importantly they shouldn't be talking to clients until they've talked to each other, and the "talking to each other" needs to terminate in the bootstrap case.
15:49 Humble joined #gluster
15:49 chirino jdarcy: yeah.
15:50 chirino Guess the healing process is where the magic is at.
15:50 jdarcy It definitely is.
15:51 jdarcy Distribution can add some interesting wrinkles with rebalancing and so on, but replication is where all the real deep dark voodoo lives.
15:51 chirino jdarcy: hardest bit might be making sure your only running 1 healing process at a time.
15:52 chirino guess it just needs to lock all the bricks /w out getting into a deadlock.
15:53 jbrooks joined #gluster
15:54 chirino btw.. does glusterfs compile on OSX?
15:56 bennyturns joined #gluster
15:56 gbr joined #gluster
15:56 glusterbot New news from newglusterbugs: [Bug 847619] [FEAT] NFSv3 pre/post attribute cache (performance, caching attributes pre- and post fop) <http://goo.gl/qbDjE> || [Bug 847626] [FEAT] nfsv3 cluster aware rpc.statd for NLM failover <http://goo.gl/QBwN9>
15:57 gbr I just tried to fix a split brain, using Joe Julian's script at http://www.joejulian.name/blog/fix​ing-split-brain-with-glusterfs-33/  The rm of the file worked, but the rm of the .gluster stuff failed with 'bad substitution'.  How do I get rid of my split brain now?
15:57 glusterbot <http://goo.gl/FzjC6> (at www.joejulian.name)
15:58 elyograg joined #gluster
15:59 tqrst I have a gluster server "node54" on which 'gluster volume list' gives 'no volumes present in cluster', yet 'gluster peer status' on any other node shows node54 as being connected. What gives? Its logs are filled with "Unable to find hostname: [some ip that is valid and can be reached from that server just fine]"
16:00 tqrst along with "unable to find peer by uuid"
16:01 robos joined #gluster
16:03 tqrst oh, the rpm scripts in the update destroyed my config on that server
16:03 tqrst no wonder
16:05 daMaestro joined #gluster
16:06 rwheeler joined #gluster
16:11 pdurbin i'll probably try iozone to test the performance of VMs on gluster vs. NFS
16:17 sjoeboo pdurbin: iozone is running currently
16:17 jbrooks joined #gluster
16:18 pdurbin sjoeboo: heh. nice. thanks! :)
16:28 lkoranda joined #gluster
16:29 tqrst and clearing the config directory except for the uuid and doing 'volume swync a-good-server myvol' makes glusterd crash
16:36 tqrst '[root@ml54 glusterd]# gluster volume sync ml43 bigdata' -> 'Volume ▒t) does not exist'
16:44 tc00per left #gluster
16:44 Kathor joined #gluster
17:04 tc00per joined #gluster
17:05 Kathor Can someone help to understand, why one of the gluster_3.3.1 servers does not creates a new, simple volume?
17:05 Kathor With just one local brick.
17:05 Kathor The command: gluster volume create VolName3 ServerName:/data/HDD
17:05 Kathor In the cli.log file there is only two lines:
17:05 Kathor [2012-11-26 18:34:04.793043] W [rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
17:05 Kathor [2012-11-26 18:36:04.890520] I [input.c:46:cli_batch] 0-: Exiting with: 110
17:05 Kathor AFter I execute the command, it stays for 2 minutes and finishes without any message.
17:05 Kathor I'm  trying to resolve this already for three evenings. There is other similar posts in the web, but no solution.
17:05 Kathor i'm  running Centos 6.3 / Gluster 3.3.1 compiled from source.
17:05 Kathor Previously, couple weeks ago - I have created two volumes on this server - in the same way, and they are used in production at the moment.
17:05 Kathor There is 3 other servers in the cluster - with lcoa, separate volumes in them.
17:05 Kathor When I make copy of the /var/lib/glusterd/vols/VolName1 to  /var/lib/glusterd/vols/VolName3; make changes in the files, and restart the gluster server - the "new" volume appears in 'gluster volume info', and is usable.
17:05 Kathor Yesterday, I was able to create the new volume in one other server in the cluster. But i'm  not able to start it now - operation failed.
17:10 Kathor Question #1) How to understand, why the server is not creating the volume?
17:10 Kathor Question #2) Does the other peer servers influence the particular server, in the job of creating a new volume?
17:11 Bullardo joined #gluster
17:15 Kathor Question #3) I have 4-6 peers, with a "local brick only" volumes in them, mounted in several other web production servers.
17:15 Kathor Is it safe to delete the peer information in all of the glusterd servers and restart the glusterd to start a "fresh" cluster?
17:19 Shdwdrgn joined #gluster
17:31 Kathor ok, I was sick of it. I deleted all peer info from the server, restarted it. Now I'm able to create new volumes.
17:32 rudimeyer_ joined #gluster
17:32 Kathor The mount points is broken in clients.
17:34 andreask joined #gluster
17:36 Mo___ joined #gluster
17:47 toruonu joined #gluster
17:47 toruonu alright, it's evening now … I'm starting to think that our gluster cluster is somehow in an inconsistent state. How best to understand if everything is 100% healthy and working perfectly or not?
17:48 toruonu all the 12 bricks in the volume seem present
17:49 toruonu btw the healing information from before that the files were in odd state and didn't heal seems to have gone away to most extent, but I do have one brick that still gives heal information about gid's, not files
17:51 toruonu those gids seem to be impossible to replicate because they're not there in the remote system:
17:51 toruonu [2012-11-26 19:31:18.506942] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-home0-client-11: remote operation failed: No such file or directory. Path: <gfid:4122172a-5e33-4b56-97cb-62252b659f2b> (00000000-0000-0000-0000-000000000000)
17:52 toruonu http://fpaste.org/GX5T/
17:52 glusterbot Title: Viewing Gathering Heal info on volume home0 ... e9d-bd44-4a31-8cb1-4cd36ce1d287> (at fpaste.org)
17:55 toruonu also, the last entries on the affected node (where the brick with the last gfid's is) after a re-initiated heal isn't that good:
17:55 toruonu [2012-11-26 19:53:06.554057] E [afr-self-heald.c:685:_link_inode_update_loc] 0-home0-replicate-3: inode link failed on the inode (00000000-0000-0000-0000-000000000000)
17:56 DaveS joined #gluster
17:56 toruonu this is the relevant Error level logs:
17:56 toruonu http://fpaste.org/IPP8/
17:56 glusterbot Title: Viewing [2012-11-26 19:51:19.640086] E [afr- ... file or directory) [2012-11-26 19:53 ... 0000000-0000-0000-0000-000000000000) (at fpaste.org)
17:58 JoeJulian ello
17:58 JoeJulian Anybody have any questions that need addressed while I have a few minutes?
18:00 toruonu yes I have :) if you can look up a few lines :)
18:00 Daxxial_ joined #gluster
18:00 toruonu basically summarized:
18:00 toruonu <toruonu> http://fpaste.org/IPP8/
18:00 toruonu <toruonu> http://fpaste.org/GX5T/
18:00 glusterbot Title: Viewing [2012-11-26 19:51:19.640086] E [afr- ... file or directory) [2012-11-26 19:53 ... 0000000-0000-0000-0000-000000000000) (at fpaste.org)
18:00 glusterbot Title: Viewing Gathering Heal info on volume home0 ... e9d-bd44-4a31-8cb1-4cd36ce1d287> (at fpaste.org)
18:01 toruonu the odd thing being that as far as I know this node has been for sure one that's been up since the glusterfs was installed on the nodes and the volume started without interruptions
18:02 JoeJulian looking...
18:04 toruonu the gid's only appeared a while after I enabled quorum on the 3-replica volume. Prior to that I had files listed on other bricks, but surprisingly they've fixed themselves in the last few hours after days of nothing happening
18:04 JoeJulian I'm guessing the "link failed" errors are related to the "No such file" on the gfid opendirs. Can I get the volume info?
18:04 toruonu http://fpaste.org/BXWO/
18:04 glusterbot Title: Viewing Volume Name: home0 Type: Distributed ... k4: 192.168.1.244:/d35 Brick5: 192.1 ... onfigured: cluster.quorum-type: auto (at fpaste.org)
18:05 elyograg here's a question.  Is quroum decided by a majority of peers, or by a majority of nodes containing bricks?
18:05 JoeJulian Hey, a fellow 4x3er. :D
18:05 toruonu :)
18:05 rwheeler joined #gluster
18:05 elyograg s/quroum/quorum/
18:05 glusterbot What elyograg meant to say was: here's a question.  Is quorum decided by a majority of peers, or by a majority of nodes containing bricks?
18:05 JoeJulian iirc, auto = more than half.
18:07 gbr I just tried to fix a split brain, using Joe Julian's script at http://www.joejulian.name/blog/fix​ing-split-brain-with-glusterfs-33/  The rm of the file worked, but the rm of the .gluster stuff failed with 'bad substitution'.  How do I get rid of my split brain now?
18:07 glusterbot <http://goo.gl/FzjC6> (at www.joejulian.name)
18:09 JoeJulian gbr: Just try reading the file through the client now. It /probably/ worked.
18:09 elyograg I'm asking because initially I will only have two peers that contain bricks, but I will have at least 4 peers total.  The machines providing NFS, UFO, and Samba will be peers without bricks.  I'll be replica 2.  If quorum is on a per-replica-set basis and only the nodes with the actual bricks will participate, then turning quorum on will be a bad thing for me.
18:10 gbr Will do.  System is down now for RAM exchange (memtest86 showed bad RAM).  Will try when it's back up.
18:10 JoeJulian toruonu: ls -l /d36/.glusterfs/d0/6a/d06a383​d-49cf-46d9-ad4f-ac92a60139df
18:10 JFK I have an xfs partion mounted as /node, i made on it gluster and mounted gluster as /replicated, than i've wrote something to /node instead of /replicated. It looks like it did replicated to other machines. Am i in big trouble?
18:11 toruonu JoeJulian: http://fpaste.org/SymU/
18:11 glusterbot Title: Viewing lrwxrwxrwx 1 root root 69 Nov 26 15: ... 3d-49cf-46d9-ad4f-ac92a60139df -> .. ... 0d-847f7ea8ccfa/crab_0_121116_154626 (at fpaste.org)
18:11 JoeJulian elyograg: Yes, quorum is only used on replicate
18:11 XmagusX joined #gluster
18:11 JFK for now everythnig looks fine but i don't know if it is goig to blow in a minute
18:12 JoeJulian JFK: You're probably ok, but (of course) you shouldn't do that. :D
18:12 JoeJulian No, nothing's going to blow up. If it was going to break, it would only break that one file.
18:13 JFK JoeJulian: but i can read only from /node directory safely?
18:13 JoeJulian toruonu: Does that show as a broken link? I'm guessing that the directory it's linking to doesn't exist.
18:14 JoeJulian JFK: You /can/. It's not advised but it won't break anything.
18:15 JoeJulian toruonu: Notice that in /d36/.glusterfs/d0/6a/d06a383d​-49cf-46d9-ad4f-ac92a60139df, the two directory parents are the first four digits of the gfid. Assuming they're all broken links, you can just delete them (on all three bricks).
18:16 toruonu it's not a broken link
18:16 toruonu http://fpaste.org/X1eI/
18:16 glusterbot Title: Viewing [root@se4 bricks]# cd /d36/.glusterf ... 847f7ea8ccfa/crab_0_121116_154626 to ... 0005 1000002 4096 Nov 19 20:57 share (at fpaste.org)
18:17 JFK i do understand that sometimes i can read from there not updated file, but im ok with that as long as i can read file localy/fast
18:17 JoeJulian toruonu: Ok, let's check getfattr -m . -d -e hex /d36/.glusterfs/d0/6a/d06a383​d-49cf-46d9-ad4f-ac92a60139df
18:17 toruonu getfattr: /d36/.glusterfs/d0/6a/d06a383d​-49cf-46d9-ad4f-ac92a60139df: Too many levels of symbolic links
18:18 JoeJulian JFK: My opinion is always that if you know what you're doing and why, then it can't possibly be wrong.
18:19 JoeJulian toruonu: Oops, add -h
18:19 toruonu mmm -h didn't help that :)
18:20 toruonu [root@se4 6a]# getfattr -h -m . -d -e hex /d36/.glusterfs/d0/6a/d06a383​d-49cf-46d9-ad4f-ac92a60139df
18:20 toruonu getfattr: /d36/.glusterfs/d0/6a/d06a383d​-49cf-46d9-ad4f-ac92a60139df: Too many levels of symbolic links
18:21 toruonu which is kind of odd as it should mean no rerefs
18:21 toruonu s/rerefs/derefs/
18:21 glusterbot What toruonu meant to say was: which is kind of odd as it should mean no derefs
18:21 bitsweat joined #gluster
18:23 JoeJulian Yeah, shouldn't be able to get an ELOOP from that. :/
18:25 toruonu can I get the actual path name for all the gids? maybe it's crap that I don't really use and the easier way out is to remove it :)
18:26 tqrst ever since I updated to 3.3.1, one of my servers has been rejecting its peers (even though the other servers think that one is up and running just fine), and rebalancing doesn't work: http://community.gluster.org/q/rebalance-uns​uccessful-after-update-from-3-2-6-to-3-3-1/  any ideas?
18:26 glusterbot <http://goo.gl/KE0qR> (at community.gluster.org)
18:26 JoeJulian If it's a gfid for a directory entry, which all of those are, then it can be safely deleted anyway.
18:26 tqrst I tried to wipe its configuration folder and sync from another server, but that failed (see the bottom of that post)
18:26 JoeJulian Files are a little different because it could break hardlinks, but you can't hardlink directories.
18:26 mario_ joined #gluster
18:27 toruonu grrr
18:27 toruonu network hiccup
18:27 toruonu JoeJulian: did I miss any replies?
18:28 Azrael808 joined #gluster
18:29 JoeJulian tqrst: "Volume ▒▒a does not exist" tells me that at least one of them is running 3.3.0
18:29 JoeJulian toruonu: If it's a gfid for a directory entry, which all of those are, then it can be safely deleted anyway.
18:29 JoeJulian toruonu: Files are a little different because it could break hardlinks, but you can't hardlink directories.
18:30 toruonu ok so if I remove those gfid's then nothing's gonna be messed up? :) and is there some easy one-liner that you use to convert the gfid to pathname?
18:30 tqrst JoeJulian: I just did gluster --version on all of them, and got 3.3.1 everywhere. Then again, this doesn't guarantee that the running glusterd processes really are 3.3.1 (although I did do 'service glusterd stop' before updating, followed by start, on all machines...)
18:30 tqrst is there a way to find out what version a glusterd process is on?
18:34 toruonu JoeJulian: it seems the references we're tracking are the .. directory!
18:34 toruonu http://fpaste.org/O9zA/
18:34 glusterbot Title: Viewing [root@se4 ~]# ls /d36/.glusterfs/d0/ ... 2012 db4e6034-8788-43eb-b7ec-7df5be ... 5 15:39 TriggerFilter [root@se4 ~]# (at fpaste.org)
18:35 toruonu I think the path is /mario/Summer12/CMSSW_5_3_4/src/AnalysisCo​de/Configuration/done/crab_0_121116_154626  and we're seeing it pointing first to the crab, then to done, then to Configuration and then to AnalysisCode
18:35 tqrst JoeJulian: actually, I just restarted glusterd on the server I am syncing from (the foo in 'gluster volume sync foo myvol'), and...now syncing just segfaults
18:35 toruonu or I think I just messed up that right now :D
18:36 toruonu nah ignore my last :D
18:39 toruonu ok, I've verified all of the gid's that give odd results are directories, but they're not empty. If I remove them from .glusterfs what is the consequence? Why are they there at all if I can just remove them :)
18:40 tqrst JoeJulian: let's back up for a sec - let's say one of my gluster servers' configuration file somehow gets corrupted by a bad script (coughcentosrpmscough). To recover it, is it sufficient to clear out /var/lib/glusterd/ and create a new /var/lib/glusterd/glusterd.info with the original uuid, followed by a sync?
18:40 JoeJulian toruonu: They'll get recreated when the directory is accessed. Hopefully they'll be recreated correctly.
18:42 JoeJulian tqrst: I think you might have to probe that server again since it won't know it's a peer of anything.
18:43 toruonu ok, in this case the directories in question are something I can scrap in theory so I'll try
18:43 toruonu whoops
18:43 toruonu some of those directories now show duplicate content :P
18:43 tqrst JoeJulian: (also, it looks like my shared root setup doesn't play well with the gluster rpms. I made /var/lib/glusterd private to each machine, but pretty much everything else is shared across all servers. The rpm installed things in /var/lib/glusterd, but that change only happened on the machine that I ran the yum update command on - blargh)
18:43 toruonu ah that was temporary I think
18:43 toruonu or not… interesting
18:44 toruonu http://fpaste.org/zWGt/
18:44 glusterbot Title: Viewing [mario@ied crab_0_121116_153306]$ ls ... 2 mario HEPUsers 16384 16. nov 15:3 ... o HEPUsers 16384 19. nov 20:56 share (at fpaste.org)
18:45 tqrst JoeJulian: probing got rid of the 'peer status is empty' problem, but syncing still fails as before
18:46 JoeJulian tqrst: Try just restarting glusterd on that one. I suspect it'll sync automatically now.
18:46 tqrst JoeJulian: I've done so several times already
18:47 tqrst JoeJulian: (what with all the segfaulting)
18:47 JoeJulian That segfaulting went away with 3.3.1 for me... :/
18:47 aliguori joined #gluster
18:49 toruonu btw is it normal for small files like /mario/.viminfo to show up on two bricks out of three for a while?
18:50 tqrst 'repoquery --list glusterfs' shows files both in /usr/lib64/glusterfs/3.3.1 and /usr/lib/glusterfs/3.2.7
18:50 JoeJulian toruonu: I've never seen that myself, but I hear it occasionally.
18:51 JoeJulian tqrst: I was afraid of that.
18:51 toruonu well the gfid's after removal did disappear from healing, but those .viminfo ones did appear now
18:52 tqrst JoeJulian: do you see a problem with "yum --enablerepo=glusterfs-epel update glusterfs glusterfs-debuginfo glusterfs-fuse glusterfs-rdma glusterfs-server"?
18:52 tqrst because that's what I did
18:52 JoeJulian tqrst: So you're using a glusterfs root? I thought about doing that for a while, but never got around to trying it. You bake in the client and libraries into initrd?
18:52 tqrst JoeJulian: no, I do not have a glusterfs root
18:52 mario_ joined #gluster
18:52 tqrst my cluster is a shared root setup, where a lot of the files are shared between all nodes
18:53 tqrst that's what I meant by shared root
18:53 toruonu grr… another hiccup.
18:53 JoeJulian Ah
18:53 JoeJulian brb...
18:53 tqrst certain paths, like /var/lib/glusterd, are set to be private to each node
18:56 tqrst I knew this shared root setup would come back to bite us, but I'm not the one who created this cluster so I don't have much freedom there (tm)
19:05 JoeJulian tqrst: No, I don't see any problem with that yum upgrade, but if it didn't complete, it could leave the old rpm installed as well.
19:05 tqrst JoeJulian: afaik, it did complete
19:06 tqrst however, any extra scripts in the rpm would have run only on one node
19:06 mario_ joined #gluster
19:06 JoeJulian Ah, I see where your upgrade problem came from then. :(
19:07 Daxxial_ joined #gluster
19:07 tqrst (one thing to keep in mind is that we're already with the /var/lib/glusterd path instead of /etc/glusterd, and that I ran 'glusterd --xlator-option *.upgrade=on -N' on all nodes after the update)
19:08 JoeJulian Ok, that should have solved that.
19:09 JoeJulian rpm -qa 'gluster*' should only show 3.3.1, right?
19:09 tqrst yes
19:09 JoeJulian If so, I would remove /usr/lib/glusterfs/3.2.7
19:10 tqrst interesting
19:10 toruonu JoeJulian: is it normal that those files show up all the time. After we ended the gfid thing I had 0 files in the heal list and now mere 20 minutes later I see this:
19:10 toruonu http://fpaste.org/b1DJ/
19:10 glusterbot Title: Viewing Gathering Heal info on volume home0 ... .240:/d35 Number of entries: 0 Brick ... .168.1.240:/d36 Number of entries: 0 (at fpaste.org)
19:10 tqrst the files aren't actually there
19:10 toruonu it's as if 2 nodes are out of sync in communications
19:11 toruonu but it's a 3-way replication so ...
19:11 JoeJulian toruonu: Are those frequently written files? Files can be dirty on-and-off if they're heavily written to.
19:12 toruonu I'm assuming those are probably in use right now
19:12 toruonu so it's normal to have them pop up there for a while
19:12 kkeithley what problem do you see that would allow an upgrade to complete and leave 3.2.7 bits behind?
19:12 JoeJulian Yes.
19:12 JoeJulian kkeithley: I was wondering that too.
19:12 tqrst kkeithley: I think it might be an issue with repoquery
19:13 * JoeJulian isn't familiar with repoquery.
19:13 tqrst I was using it to get the list of files owned by the 'glusterfs' package
19:13 kkeithley JoeJulian: oh, I thought you said ^^^ that you knew what the problem was.
19:14 theron joined #gluster
19:14 JoeJulian Ah, no. It's just acting like it's not the right version, even though --version says it is.
19:14 kkeithley mkay, I misunderstood.
19:16 * kkeithley reads repoquery man page and wonders how it's different then, e.g., `yum list ...`
19:16 JoeJulian tqrst: Maybe do: rpm -qaV 'glusterfs*'
19:16 tqrst kkeithley: yum list shows the installed version, whereas repoquery lists the installed files
19:17 tqrst but it looks like I misunderstood how it works
19:17 kkeithley yeah, looks like a mashup of yum+rpm
19:17 tqrst yum list glusterfs shows that glusterfs.x86_64 3.3.1-1.e16 is installed, and glusterfs.i686 3.2.7-1.el6 is *available*. repoquery probably lists the files from both even though only one is installed
19:17 kkeithley interesting
19:18 tqrst I think it's safe to assume that I do not have 3.2.7 files lying around
19:18 tqrst sorry about the confusion
19:19 kkeithley okay
19:19 * kkeithley breathes a small sigh of relief
19:19 tqrst any idea how to find out what commands yum will run after installing an rpm?
19:21 JoeJulian rpm -q --scripts
19:21 tqrst oh wow
19:21 kkeithley besides stracing it? I have in mind that yum just manages fetching the rpms from the repos, and then invokes rpm to do the install
19:21 tqrst that script is dangerous
19:21 tqrst http://pastie.org/private/hxtccu7imd7jqpgfwxlqa
19:21 glusterbot Title: Private Paste - Pastie (at pastie.org)
19:22 tqrst check out the first if
19:23 kkeithley you mean the `if [ -d /etc/glusterd ] ...`
19:23 tqrst yeah
19:23 kkeithley that's straight out of the glusterfs.spec file for the rpm
19:23 tqrst for some reason, ours is the other way around (/etc/glusterd is a real folder, and /var/lib/glusterd is a symlink to it)
19:24 toruonu joined #gluster
19:25 kkeithley yes, that's consistent with older installs.
19:25 kkeithley I suppose
19:25 toruonu JoeJulian: using your online time … any hints on tuning the mount or volume to speed up operations? using the gluster volume as /home and it's terribly slow for users
19:25 kkeithley wait...
19:25 toruonu the nodes are all high-end nodes with 2x 10Gb NIC's etc … but basic ls commands take a lot of time
19:26 kkeithley ick
19:26 Azrael808 joined #gluster
19:28 toruonu oh and … I'm still seeing the Disk I/O errors
19:31 tqrst JoeJulian: looks like I need to run the step that renames the /var/lib/glusterd/vol/**.vol files to .vol.rpmsave - not that it should have anything to do with the issues I've encountered so far
19:31 JoeJulian I use it for /home, too, and since 3.3.1 I keep checking to make sure it's actually mounted and I'm not using a local filesystem. It seems too fast.
19:32 toruonu I'm using 3.3.1, but for example I tested today and doing a simple ls (using the alias that has color on) in a folder with 3520 files took 101 seconds
19:32 JoeJulian Oh! Well don't have 3520 files in one directory. ;)
19:33 toruonu well even 120 files took a good 10 seconds
19:34 toruonu doing it without the color option does give about 5-6x faster result
19:34 toruonu anyway right now I get this:
19:34 toruonu [mario@ied Configuration]$ strace -f -o crab.strace crab -cfg qcd_bctoe_170.cfg -create all
19:34 toruonu crab: DB Installation error : 'disk I/O error'
19:34 toruonu it fails in creating a local SQLite database file
19:34 toruonu I've got the strace if needed
19:35 JoeJulian 229 files in my home directory took 2.573s.
19:35 toruonu have you done an ls already once? :) it seems to cache somewhat
19:35 JoeJulian Ah, of course... without --color it was nearly instant...
19:36 y4m4 joined #gluster
19:36 toruonu anyway that slowness is secondary to users not being able to work … and that crab disk I/O failed error is a showstopper right now
19:36 JoeJulian Can you remount?
19:37 toruonu tough … it's a shared system so I have users in all the time
19:37 JoeJulian If you haven't re-mounted since upgrading from 3.3.0 to 3.3.1 then I know what bug you're hitting.
19:37 toruonu and as it's /home
19:37 toruonu it's a clean install
19:37 toruonu did it only a few days ago
19:37 toruonu 3.3.1
19:37 toruonu all the way
19:38 JoeJulian Ok, two possibilities. 1, that bug is still around, 2, there's a split-brain error on that.
19:38 kkeithley tqrst: you like http://fpaste.org/HJy3/ better?
19:38 glusterbot Title: Viewing if [ -d /etc/glusterd -a ! -h /var/l ... ib/glusterd cp -a /etc/glusterd /var ... f /var/lib/glusterd /etc/glusterd fi (at fpaste.org)
19:38 JoeJulian For 1, a workaround that I found is to make a graph change that affects the client.
19:39 JoeJulian Change client.log-level or client.io-threads
19:39 tqrst JoeJulian: ok I'm at the point where every server is as if the update had run fully on each, and I've restarted every glusterd. node54 is still getting rejected by everyone, and syncing still outputs a garbage volume name 'does not exist'
19:39 kkeithley JoeJulian: ^^^
19:39 toruonu JoeJulian: in the strace link I sent you the file in question on which everything fails is crabDB
19:40 tqrst kkeithley: seems better
19:41 toruonu if I'm not mistaken the error is this: http://fpaste.org/BbnF/
19:41 glusterbot Title: Viewing fcntl(3, F_SETLK, {type=F_RDLCK, whe ... b_0_121126_212942/share/crabDB", O_R ... ad(5, "", 100) = 0 5030 close(3) = 0 (at fpaste.org)
19:44 toruonu is there some issue with locking? or how best to debug this
19:44 gbrand_ joined #gluster
19:44 toruonu I'm only seeing this with specific kind of command and always on the SQLite DB file, creating, writing, deleting of files/directories works just fine
19:44 JoeJulian toruonu: fuse client, right?
19:44 toruonu yes
19:44 toruonu fuse client on hardnode, simfs mounted (bind mount) onto the VM of users server
19:48 Technicool joined #gluster
19:48 semiosis toruonu: what version of sqlite?
19:48 JoeJulian jdarcy: Are you around and do you have any insight on this? F_SETLK returns ENOENT. The file does exist though.
19:49 semiosis i recall people having issues with sqlite but cant find the details & there's no obviously related bugs in BZ since 2011
19:49 toruonu ./crab_0_121022_173411/share/crabDB: SQLite database (Version 3)
19:49 semiosis there was this: http://community.gluster.org/q/how-can-i-make​-sure-glusterfs-is-working-with-posix-locks/
19:49 glusterbot <http://goo.gl/RKZwa> (at community.gluster.org)
19:49 toruonu this from a successful creation
19:49 semiosis but that cgo q/a is not too helpful imho
19:50 toruonu nope in this case the thing crashes and what is left is a 0 byte file
19:51 JoeJulian semiosis: The only issues I recall was where there were multiple clients all trying to lock the same sqlite database.
19:51 toruonu well doesn't crash, exits with Disk I/O error :)
19:51 toruonu in this case however I can guarantee noone else is accessing that file or for that matter the directory where the file resides at the same time
19:51 toruonu it fails at creation
19:51 semiosis ooohhhhh
19:51 semiosis thx for clarifying, i missed that
19:52 gbrand__ joined #gluster
19:52 semiosis check/pastie your brick log file for the brick that contains the zero-len file please
19:52 sjoeboo_ joined #gluster
19:53 semiosis client log may have some info as well, but my guess is this is posix-related so on the brick side
19:53 toruonu how do I determine which brick it went on?
19:54 semiosis @pathinfo
19:54 glusterbot semiosis: I do not know about 'pathinfo', but I do know about these similar topics: 'pasteinfo'
19:54 semiosis hrm
19:54 semiosis from the client, getfattr -m . -d trusted.gluster.pathinfo
19:54 semiosis or somethign like that, i dont remember exactly
19:55 semiosis you can also of course do an ls /path/to/file on all your bricks
19:55 layer7switch joined #gluster
19:55 snarkyboojum_ joined #gluster
19:55 morse_ joined #gluster
19:55 toruonu ah … found one brick at least where the file is present
19:55 semiosis s/trusted.gluster/trusted.glusterfs/
19:55 glusterbot What semiosis meant to say was: from the client, getfattr -m . -d trusted.glusterfs.pathinfo
19:55 toruonu [root@se4 ~]# ls /d35/mario/Summer12/CMSSW_5_3_4/src/AnalysisC​ode/Configuration/crab_0_121126_212942/share/
19:55 toruonu crabDB
19:56 m0zes_ joined #gluster
19:56 torbjorn1_ joined #gluster
19:56 semiosis great, could you pastie the log for that brick and its replicas from the time when the operation failed?
19:56 toruonu http://fpaste.org/NYHr/
19:56 glusterbot Title: Viewing [2012-11-26 17:50:13.712717] I [serv ... file or directory) [2012-11-26 17:50 ... ) ==> -1 (No such file or directory) (at fpaste.org)
19:56 ndevos_ joined #gluster
19:57 toruonu ah forget the whole log
19:57 toruonu only the last 2 lines
19:57 toruonu [2012-11-26 21:30:15.047271] I [server3_1-fops.c:252:server_inodelk_cbk] 0-home0-server: 1365533: INODELK (null) (--) ==> -1 (No such file or directory)
19:57 toruonu [2012-11-26 21:30:15.082376] I [server3_1-fops.c:346:server_entrylk_cbk] 0-home0-server: 1365534: ENTRYLK (null) (--) ==> -1 (No such file or directory)
19:57 toruonu that's around the time I think (the file shows 21:29 as time)
19:57 toruonu might be a few seconds mismatch or time difference
19:57 semiosis um
19:57 semiosis ok
19:58 kkeithley @stripe
19:58 glusterbot kkeithley: Please see http://goo.gl/5ohqd about stripe volumes.
19:58 semiosis are any of the replicas of that brick missing the directory containing that file
19:58 toruonu no striping
19:58 toruonu hang on
19:59 semiosis i'm just guessing wildly here, afaik this kind of problem "should not happen" when your volume is in good health
19:59 toruonu all three blocks seem to be there
19:59 toruonu sorry bricks have all that file
20:00 semiosis and none of the other brick log files have anything else
20:00 semiosis ?
20:00 semiosis they all show the same thing as the one you pastied?
20:01 toruonu well that's from another brick:
20:01 toruonu http://fpaste.org/JAfc/
20:01 glusterbot Title: Viewing [2012-11-26 21:27:45.263288] I [serv ... I [server3_1-fops.c:203:server_lk_cb ... f-f8d9556ef4b7) ==> -1 (File exists) (at fpaste.org)
20:02 helloadam joined #gluster
20:02 toruonu it's showing a lot of those lock issues
20:03 semiosis and what filesystem are your bricks formatted with?
20:03 toruonu ext4
20:03 toruonu we've been bitten by XFS issues too many times
20:04 semiosis and you're not affected by the ,,(ext4) issue, i presume
20:04 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
20:05 toruonu to be fair I've read it, but not understood what the implication is :)
20:05 toruonu and looking at comments JoeJulian is claiming that this is fixed in 3.3.1 :)
20:05 toruonu which we run
20:07 aliguori joined #gluster
20:07 kkeithley no, the ext4 fix did not make it into 3.3.1
20:08 semiosis toruonu: would it be possible for you to try stopping & starting the volume & umounting/remounting the client?
20:08 toruonu can someone tell me what the actual symptoms are for the ext4? :)
20:08 shireesh joined #gluster
20:08 semiosis i know that's drastic, but sometimes it magically shakes things loose that should work but dont
20:08 semiosis toruonu: ls hangs due to infinite loop iirc
20:09 semiosis maybe other things too but that's usually what people turn up here complaining about
20:09 toruonu ah, ls does take time, but it completes always as a function of elements in the folder
20:09 semiosis thats normal
20:09 toruonu a sec or so for < 50; 100s for 3500
20:09 toruonu so probably not bitten unless this locking issue is related
20:10 toruonu but I'm assuming it's not as the behavior should be infinite loop
20:10 toruonu not error file not found
20:10 kkeithley then you're good, unless you update your kernel to one that has the problem
20:10 puebele joined #gluster
20:10 toruonu well we run a kernel that is supposed to have it, but we don't run vanilla RHEL kernels… we run OpenVZ patched ones
20:10 toruonu so I'd not be surprised if the patch is removed in the OpenVZ one :)
20:11 toruonu ah hang on that's not the case … it should only affect the brick nodes
20:11 toruonu those are SL 5.7
20:11 toruonu based on EPEL 5.7
20:12 JoeJulian toruonu: It wasn't.... I guess I should add a comment about that.
20:12 bennyturns joined #gluster
20:13 mario_ joined #gluster
20:13 toruonu lost what ever was said for ca 2 minutes
20:13 toruonu anyway … remounting might be an issue
20:13 toruonu it's /home afterall
20:13 neofob joined #gluster
20:14 semiosis yeah and who knows if it will even help
20:14 semiosis but when i'm out of ideas, that's my last resort troubleshooting step
20:14 toruonu bunch of users have vncserver running can't say what they'd lose if I did kick them out
20:14 bennyturns joined #gluster
20:14 toruonu I'm guessing lazy umount on a busy fs is not a good option for getting health up :D
20:15 toruonu well the volume heal info gives 0 entries for all bricks
20:15 toruonu so that's good...
20:15 toruonu maybe I should just restart glusterd on all brick nodes one by one?
20:15 toruonu how does the system survive that?
20:15 toruonu and would that help or not
20:15 semiosis restarting glusterd does not necessarily restart the brick export daemon ,,(processes)
20:16 glusterbot the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
20:16 toruonu probably can't restart the one that's mounted
20:16 toruonu so if the export remains running and I restart glusterd it should leave the volume up and about?
20:16 semiosis you could kill (-15/TERM, default) the glusterfsd process for a brick, then restart glusterd, which will respawn the killed brick export daemon
20:17 semiosis doing that for each brick in the volume, one at a time
20:17 semiosis but check between each one to make sure your client(s) reconnect to the brick
20:17 toruonu ok, what happens when I restart the glusterd on the server that's used for the client mount
20:17 semiosis they should, but i've seen them fail to
20:18 semiosis ,,(mount server)
20:18 glusterbot (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
20:18 semiosis bbiab, lunch
20:18 tqrst to add to today's fun: my logs are getting spammed with 'I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now' every 3 seconds
20:20 JoeJulian tqrst: gluster volume status
20:20 tqrst JoeJulian: it's inconsistent across machines, which one do you want?
20:20 JoeJulian 3 seconds is a reconnect try interval.
20:21 JoeJulian It shouldn't be.... :(
20:21 tqrst http://pastie.org/private/agl5ve9gzwmcvxqaxsjg
20:21 glusterbot Title: Private Paste - Pastie (at pastie.org)
20:21 tqrst here's 3
20:22 tqrst 54 is the one that got completely messed up by the update
20:23 JoeJulian I would pick one server, stop all the glusterd, rsync /var/log/glusterd/vols from the one you're happy with to *all* the rest of the peers. Start glusterd on all of them again.
20:23 JoeJulian That won't interfere with operations but should at least make sure all your volume definitions are in sync.
20:23 tqrst isn't that what 'volume sync' is essentially supposed to do?
20:24 JoeJulian Yes, but it only does it if there are no volumes defined, and that funky character you listed earlier worries me.
20:26 tqrst (for the record, trying that on only ml54 didn't help - same status as before)
20:26 tqrst (tried that earlier)
20:30 toruonu hmm I just did [root@ganymede ~]# for i in `cat glusterlist`; do ssh $i "pkill glusterfsd; service glusterd restart"; sleep 5; done and seems that right now I can create crab tasks (i.e. the sqlite db creation succeeds)
20:31 toruonu will have to see if that works for all tasks, but that I'll leave to run overnight
20:35 tqrst JoeJulian: before I rsync everywhere, are you sure there's nothing machine-specific in glusterd/vols/... ?
20:37 tqrst (is there a description of the files in /var/lib/glusterd somewhere?)
20:39 JoeJulian There is nothing machine specific in that directory.
20:39 toruonu ok, off to sleep, but before I go … is there some kind of tuning possible to improve iops of the volume? as it's used for /home I'd really like the users to have as snappy as possible experience and if there's something I can tune for it I'd love to hear it...
20:40 JoeJulian There really isn't. jdarcy's got some patches that he's worked on to improve that though.
20:40 toruonu we even have a few enterprise grade SSD's available that are meant for replicated MySQL central databases, but we've not migrated there yet and doubt we need the full 100GB for it so could probably spare some of it to SSD speeds
20:41 JoeJulian That "--color" option, for ls, is the culprit. That pulls a stat() for each file in the directory, which triggers a self-heal check, in order to determine which colors each file should be.
20:43 ctria joined #gluster
20:43 toruonu but would that impact only the basic ls or also tab completions and running scripts that tarball items and do listings through other means than shell ls alias
20:45 JoeJulian That color option, instead of just opendir();readdir();closedir() it then adds a stat() for each entry.
20:46 toruonu I'm assuming that only affects shell scripts that use plain ls or interactive use
20:46 JoeJulian I wouldn't necessarily expect that for tab completions, but I haven't looked at how those work.
20:46 toruonu I guess the rest are down to how the code's written
20:47 JoeJulian tarring files shouldn't add all that much overhead since the files would have had to be read anyway.
20:47 toruonu but I'm just wondering if there might be some parallelization options that might improve things
20:47 toruonu as you mentioned client thread count etc
20:47 JoeJulian Client io-threads are off by default.
20:47 JoeJulian You can try that, but I haven't seen any difference.
20:47 toruonu so if I have 10+ users doing stuff in parallel it may be useful to tune things on the communication side / server side
20:49 kkeithley Hmmm. I should look at why that is. I wonder if our self heal checking is done synchronously in the readdir loop. (Perhaps we could queue the self heal check to a worker thread so the readdir can return quickly. Maybe conditionalize it on stats while in a readdir loop somehow.)
20:49 toruonu oh btw does glusterfs require the bricks to be only used by glusterfs? or can I share the bricks with hadoop datanode data directories?
20:50 toruonu if I could I could try out how glusterfs behaves in a very large installation and the available space would be a dynamic variable between hadoop and glusterfs
20:50 JoeJulian Using gluster's hadoop plugin?
20:50 toruonu no no independt
20:50 toruonu just sharing mountpoint
20:50 tqrst JoeJulian: same issue as before - volume status is different on ml54
20:50 JoeJulian nothing else should modify the bricks.
20:50 toruonu what do you mean modify :)
20:50 semiosis imho that means write or lock files
20:51 JoeJulian tqrst: grep working-directory /etc/gluterfs/glusterd.vol
20:51 JoeJulian write, lock, change attributes...
20:51 JoeJulian But using the hadoop plugin, you can then share use with hadoop.
20:52 tqrst JoeJulian: 'option working-directory /var/lib/glusterd' everywhere
20:52 toruonu joined #gluster
20:52 toruonu ok, gonna shoot my ISP tomorrow
20:52 toruonu anyway ...
20:53 toruonu if glusterfs sees /d2/glusterfs/ as the "brick" and hadoop sees "/d2/hdfs" as it's datanode dir, then they're orthogonal except for total available space and usage pattern
20:53 toruonu neither touches the contents of the others directories
20:53 toruonu but the available space does change depending on the filling of the differnet directories
20:54 JoeJulian tqrst: You mentioned that /var/lib/glusterd is a symlink to /etc/glusterd. Is that true (still) on all of them?
20:54 JoeJulian toruonu: OH! ok. That's fine. I misunderstood what you were wanting to do.
20:54 tqrst JoeJulian: yes
20:54 toruonu we run hadoop balancer so no hadoop brick should get full unless the whole of hadoop's getting full and we've got right now 1.9PB configured, and 1.35PB in use
20:55 JoeJulian tqrst: well, dammit... something's got to be different. :P
20:55 toruonu ok, that's then an option to give a full posix compatible FS as a data directory also to the shared /home :) also test how glusterfs fares in speed tests in comparison to hadoop and how it scales to 200 brick nodes with 200 mounts
20:56 toruonu and if we ever decide to migrate from hadoop to glusterfs it's as simple as starting to move data from one to another as deleteing stuff from hadoop would clean up space for glusterfs to put it on :)
20:56 tqrst JoeJulian: I would just reboot the whole thing if I didn't have 5-10 users using it right now
20:56 toruonu and vice versa if gluster misbehaves and we axe it :D
20:56 tqrst sigh
20:57 toruonu tqrst: I know … damn users :P got the same feel at times :D
20:58 tqrst JoeJulian: (thanks for the help so far btw)
20:59 JoeJulian That's where it was good to be a consultant. "Everybody log out. I'm going to reboot."
21:00 toruonu Ok I've got wife pinging me from bedroom for a while now so going to log off for today … will see if all oddities have disappeared by tomorrow morning when I'll get to see if all of the crab commands worked meaning the locking issue disappeared
21:00 toruonu thanks for the help
21:00 tqrst urge to file a ticket with IT titled "please admin this cluster ktnx"...rising
21:02 tqrst what's the difference between 'peer in cluster' and 'accepted peer request' in 'peer status'?
21:02 tqrst ml54 is showing up as 'State: Accepted peer request (Connected)', but all the others are 'Peer in Cluster (Connected)'
21:03 JoeJulian I was just about to ask about that.
21:04 JoeJulian peer probe 54 from another (other than 43) and see if that changes anything.
21:04 tqrst (asked http://community.gluster.org/q/what-is-the-diff​erence-between-peer-in-cluster-connected-and-ac​cepted-peer-request-connected-in-peer-status/ btw)
21:04 glusterbot <http://goo.gl/VUv4J> (at community.gluster.org)
21:04 tqrst JoeJulian: says it's already in the peer list
21:06 tqrst rebootign 54 now because why not
21:07 JoeJulian Ok, stop glusterd on 54. rsync peers from 2 other servers to it. On 54 delete the one entry for 54 from peers. Start glusterd.
21:08 tqrst ah and peer status shows almost all 'Peer rejected (connected)' on 54
21:09 tqrst wait what - vols/bigdata emptied itself on 54
21:09 tqrst rs,l.mkgdfajfgjljkh
21:09 semiosis peer rejected usually means that the server has volume configuration(s) that are out of sync with the other peers
21:11 Shdwdrgn joined #gluster
21:14 semiosis tqrst: to resolve the peer rejected, on the rejected server, stop glusterd, move /var/lib/glusterd/* (except glusterd.info) out to somewhere else, start glusterd, probe the server from one of the good peers, probe the a good peer from the bad server, restart glusterd on the bad server
21:14 tqrst semiosis: will try as soon as the server is back up
21:14 semiosis if gluster volume info shows no volumes, you can do a gluster volume sync <good-server> on the bad server to fetch the volume configs, though that often happens automatically
21:14 semiosis s/the a/the/
21:14 glusterbot What semiosis meant to say was: tqrst: to resolve the peer rejected, on the rejected server, stop glusterd, move /var/lib/glusterd/* (except glusterd.info) out to somewhere else, start glusterd, probe the server from one of the good peers, probe the good peer from the bad server, restart glusterd on the bad server
21:15 semiosis again, when you set /var/lib/glusterd/* aside, remember to keep the /var/lib/glusterd/glusterd.info in place, as it contains the UUID other servers know it as
21:19 tqrst [root@ml54 glusterd]# gluster volume status
21:19 tqrst No volumes present
21:19 tqrst [root@ml54 glusterd]# gluster volume sync ml51 bigdata
21:19 tqrst Volume ▒uH does not exist
21:19 tqrst (this is after stopping, clearing out everything but glusterd.info, restarting and probing)
21:21 tqrst and glusterd crashed afterward
21:21 semiosis wow
21:21 tqrst urgh
21:22 tqrst here's the bt http://pastie.org/5439360
21:22 glusterbot Title: #5439360 - Pastie (at pastie.org)
21:22 semiosis is it just my irc client, or did that vol doesn't exist error really have some unprintable character followed by uH?
21:22 tqrst hilariously/worryingly enough, that is not your client
21:24 JoeJulian Try just plain ol' "gluster volume sync all"
21:25 tqrst when it says "please delete all the volumes before full sync", can I do that while glusterd is running?
21:26 JoeJulian yes
21:26 tqrst it's still complaining about this even after rm -rf vols/*
21:27 JoeJulian restart glusterd (again... <sigh>)
21:27 tqrst same error
21:28 tqrst vols/bigdata got created when glusterd restarted
21:31 tqrst at this point I'm tempted to just downgrade back to 3.2.7, but I have the feeling that won't be as easy as I think
21:35 JoeJulian tqrst: Did you ever check that rpm command I suggested?
21:36 tqrst JoeJulian: rpm -qa 'gluster*'?
21:37 JoeJulian qaV
21:37 tqrst all this outputs is '....L....    /var/lib/glusterd'
21:37 * JoeJulian grumbles
21:37 JoeJulian That's true on 54 vs another one that works right?
21:38 tqrst yes
21:40 JoeJulian Well, ok... so since it got recreated then I would say the sync worked properly. If stat's still wrong, then it's got to be the peer status.
21:40 y4m4 joined #gluster
21:40 tqrst I'll try the peer thing from earlier
21:40 tqrst although that folder looks ok at first glance
21:43 tqrst same issue
21:43 tqrst "[2012-11-26 16:42:42.272291] E [glusterd-utils.c:1926:glus​terd_compare_friend_volume] 0-: Cksums of volume bigdata differ. local cksum = -1818218846, remote cksum = 858523319" during startup too
21:45 JoeJulian peer status still broken?
21:45 tqrst yes
21:45 tqrst that is probably because of the differing cksum
21:46 JoeJulian I guess
21:46 tqrst any idea how that's even computed?
21:47 JoeJulian Not without looking at the source. I suspect it uses the same hash as it uses for dht.
21:47 johnmark question for you'uns
21:47 JoeJulian You could confirm that using the script on http://joejulian.name/blog​/dht-misses-are-expensive/
21:47 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
21:47 johnmark let's say I wanted to create an EBS-style service with OpenStack
21:47 tqrst but what am I hashing?
21:48 JoeJulian johnmark: cinder
21:48 johnmark how would you go abou tit?
21:48 johnmark JoeJulian: ok
21:48 johnmark would you use GlusterFS for that?
21:48 JoeJulian But we need that block device xlator that's in gerrit.
21:48 johnmark right, ok
21:48 johnmark JoeJulian: that's what I thought
21:49 JoeJulian Though it could probably be done with images and loop...
21:51 nickhatz joined #gluster
21:51 nickhatz hi!
21:51 glusterbot nickhatz: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
21:51 nickhatz how safe is gluster with a replication on three disks?
21:54 rwheeler joined #gluster
21:55 semiosis at least three times as safe as no replication, why not
21:55 circut joined #gluster
21:56 nickhatz why then two is not recommended?
21:57 semiosis recommended by whom?
21:57 tqrst JoeJulian: glusterd_volume_compute_cksum in xlators/mgmt/glusterd/src/glusterd-utils.c:1568 hints at it being different from dht.
21:57 nickhatz a guy that has a presentation on gluster I don't remember where
21:58 semiosis nickhatz: well you'd have to ask that guy then
21:58 semiosis or give us a link to the presentation and maybe someone here will be able to explain better
22:00 tqrst what happens if I downgrade to 3.2.7 now that I've done "glusterd –xlator-option *.upgrade=on -N" everywhere during the upgrade to 3.3.1?
22:00 tqrst can I even downgrade any more?
22:03 copec joined #gluster
22:07 eightyeight joined #gluster
22:07 johnmark JoeJulian: re: images and loop - that's what I was wondering about
22:07 johnmark ie is there a reason *not* to do it that way?
22:17 Bullardo joined #gluster
22:30 tqrst welp, I'm out of ideas
22:30 tqrst going home for now. Hopefully my volume doesn't burst into flames overnight, and the other replicas for those bricks don't die.
22:33 TSM2 joined #gluster
22:42 nickhatz hmm.. I guess they fix it: http://www.slideshare.net/Gluster slide 23
22:42 glusterbot Title: Glusterfs, Glusterfs | SlideShare (at www.slideshare.net)
22:44 Daxxial_ joined #gluster
22:53 JoeJulian johnmark: Sorry, work and all that... There's no specific reason not to. Just breaks KISS.
22:54 hattenator joined #gluster
23:13 johnmark JoeJulian: ok, got it
23:16 layer3switch joined #gluster
23:18 aliguori joined #gluster
23:34 inodb joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary