Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 en0x left #gluster
00:06 _pol joined #gluster
00:10 jdarcy joined #gluster
00:12 glusterbot New news from newglusterbugs: [Bug 921302] Can't set quota for a mounted volume <http://goo.gl/gzPy6>
00:45 _pol joined #gluster
00:51 tjstansell anyone know if glusterbot is gluster-made or some other open-source bot?  it's pretty nice.
00:57 yinyin joined #gluster
00:58 puebele joined #gluster
01:02 cyberbootje1 joined #gluster
01:05 cyberbootje joined #gluster
01:06 JoeJulian @version
01:06 glusterbot JoeJulian: The current (running) version of this Supybot is 0.83.4.1+limnoria 2012-07-13T14:08:55+0000. The newest versions available online are 2013-03+12T18:3608Z (in testing), 2013-03+12T18:3608Z (in master).
01:06 JoeJulian tjstansell: ^^ I prefer the limnoria fork. It's actively maintained.
01:07 tjstansell cool. thanks.
01:08 jdarcy joined #gluster
01:12 robo joined #gluster
01:25 kevein joined #gluster
01:50 mjrosenb joined #gluster
01:50 mjrosenb is trusted.glusterfs.dht.linkto being used anymore?
01:52 JoeJulian yes
01:53 JoeJulian http://joejulian.name/blog​/dht-misses-are-expensive/
01:53 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
01:54 mjrosenb JoeJulian: so when would these be created?
01:54 mjrosenb it looks like they are occasionally getting created spuriously
01:54 mjrosenb or just the data is being ignored
01:55 JoeJulian f the file's not there, hopefully there's a file there with the same filename, zero bytes, mode 1000 with the extended attribute "trusted.glusterfs.dht.linkto". This is what we call the sticky-pointer, or more correctly the dht link pointer. This tells the distribute translator "yes, the file should be here based on it's hash but it's actually at...". This happens, for instance, when a file is renamed. Rather than use a bunch of network resourc
01:55 JoeJulian es moving the file, a pointer is created where we expect the new filename to hash out to that points to where the file actually is. Two network calls, no big deal.
01:55 mjrosenb because I have that on my brick 'memoryalpha' and the real file on 'memorybeta', but ls on a client shows both.
01:55 JoeJulian What's getfattr -n trusted.dht.linkto show?
01:56 mjrosenb what does -n do?
01:56 JoeJulian name
01:56 JoeJulian It'll just show that one attribute as opposed to the -m which is a partial match
01:57 mjrosenb oh wait, did its name change somewhat recently?
01:57 JoeJulian Er, sorry, trusted.glusterfs.dht.linkto
01:57 mjrosenb ahh, it contains "magluster-client-1"
01:58 mjrosenb there is also an attribute trusted.gfid, but I suspect that is supposed to be there.
01:58 JoeJulian So that says that the file actually exists on magluster's second brick.
01:59 mjrosenb which sholud be the other machine.
01:59 JoeJulian The gfid is a psudo inode number that's used to keep inode numbers consistent.
01:59 mjrosenb ok, this sounds like it is almost certainly a problem with my freebsd port
01:59 mjrosenb speaking of which
01:59 mjrosenb I should actually put that patch up for review
01:59 JoeJulian Yep
01:59 mjrosenb now that it is like a 6 months old :(
02:01 JoeJulian That page on my blog that I linked to explains the distributed hash translation algorithm, including mentioning the sticky pointers.
02:01 mjrosenb for now, I'm just going to rm the link, and hope nothing explondes :/
02:01 robo joined #gluster
02:03 JoeJulian If the link's needed, the next time that dirent is accessed, the link will be created again.
02:09 mjrosenb JoeJulian: do you also have an explanation why an ls of a directory with a large number of files seems to take forever (at least much longer than it does on either brick)
02:12 sahina joined #gluster
02:26 bulde joined #gluster
02:30 kkeithley1 joined #gluster
02:34 Humble joined #gluster
02:42 jdarcy joined #gluster
03:07 Humble joined #gluster
03:11 yinyin joined #gluster
03:16 sahina joined #gluster
03:21 Ryan_Lane joined #gluster
03:39 sgowda joined #gluster
03:47 aravindavk joined #gluster
03:52 kris_ joined #gluster
03:59 jules_ joined #gluster
04:10 sjoeboo_ joined #gluster
04:10 pai joined #gluster
04:11 shylesh joined #gluster
04:24 yinyin joined #gluster
04:25 mohankumar joined #gluster
04:29 Humble joined #gluster
04:38 vpshastry joined #gluster
04:49 rastar joined #gluster
04:51 bharata joined #gluster
04:58 bulde joined #gluster
04:58 satheesh joined #gluster
05:00 jules_ joined #gluster
05:01 pai joined #gluster
05:01 bharata If I have a volume with one brick and later add another brick in replica mode, is there a way to enforce the data copy to 2nd brick or will gluster do it over the period of time ?
05:01 sripathi joined #gluster
05:03 yinyin joined #gluster
05:06 bala joined #gluster
05:11 hagarth joined #gluster
05:12 isomorphic_ joined #gluster
05:17 lalatenduM joined #gluster
05:17 saurabh joined #gluster
05:18 lala_ joined #gluster
05:21 saurabh joined #gluster
05:48 vshankar joined #gluster
05:48 raghu` joined #gluster
05:51 bharata Would volume heal command help in the case I mentioned above ?
05:52 tjstansell bharata: yeah. should be able to do a heal full to force it to sync everything.
05:52 bharata tjstansell, nice, thanks
05:58 sjoeboo_ joined #gluster
06:04 sripathi joined #gluster
06:10 bharata In a distributed volume, is there a way I can move a file from one brick to another ?
06:10 sgowda joined #gluster
06:14 satheesh joined #gluster
06:23 mooperd joined #gluster
06:36 glusterbot New news from resolvedglusterbugs: [Bug 868801] gluster volume creation says 'success' but volume does not exist on any of the peers <http://goo.gl/opEQ6> || [Bug 796603] [FEAT] `gluster volume create' should create parent directories during volume creation <http://goo.gl/oXUHS>
06:51 yinyin joined #gluster
06:51 vimal joined #gluster
06:56 hagarth joined #gluster
07:00 cw joined #gluster
07:08 rotbeard joined #gluster
07:10 Troy joined #gluster
07:10 Troy Hi guys .. need little help
07:10 Troy how do I stop geo replication ?
07:10 jcaputo Hello folks.  Can somebody tell me if this is normal behavior?
07:11 Troy I removed the package but it still going on
07:11 jtux joined #gluster
07:11 jcaputo http://dpaste.com/1022433/
07:11 glusterbot Title: dpaste: #1022433: Crawling in progress, by joe (at dpaste.com)
07:12 Troy removed the gsyncd.conf
07:12 Troy but as I restart glusterd it is being created again
07:12 jcaputo @bharata, i think you'll want to rebalance
07:13 kevein_ joined #gluster
07:13 dobber joined #gluster
07:14 bharata jcaputo, but that will apply to the entire volume right ? I was looking for a fine grained control where I should be able to move a file of my choice to a given brick
07:14 jcaputo oh, sorry @bharata, I was referring to your previous post
07:15 jcaputo about adding another brick
07:15 bharata jcaputo, ah ok
07:17 jcaputo @bharata http://gluster.org/community/documentation/​index.php/Gluster_3.2:_Rebalancing_Volumes
07:17 glusterbot <http://goo.gl/tWGZe> (at gluster.org)
07:17 sripathi1 joined #gluster
07:19 bharata jcaputo, It will be nice to have file level movement b/n bricks for virtualization usecase to move the VM storage to the node where VM runs
07:20 bharata bulde, ^
07:25 Troy how to kill geo-replication sessions ??
07:25 Troy anyone please
07:28 jcaputo hmm, sorry Troy, I'm not really certain ...
07:28 jcaputo perhaps this page may help?
07:28 jcaputo https://access.redhat.com/knowledge/docs/en-US/R​ed_Hat_Storage_Software_Appliance/3.2/html/User_​Guide/chap-User_Guide-Geo_Rep-Starting-Stop.html
07:28 glusterbot <http://goo.gl/fiE55> (at access.redhat.com)
07:29 Troy thanks jcaputo ...I have tried this but nothing works
07:30 jcaputo anything in the logs?
07:31 Troy I have installed geo-replication today on production .. but its taking 100% of cpu usage .. and lots of errors    related to python in log file
07:32 Troy no I am just trying to get rid of it
07:32 Troy I removed geo-replicaiton
07:32 Troy package
07:32 Troy removed rsync
07:32 Troy killed rsync process .. and CPU came down to 2 % usage
07:33 Troy but gluster volume info shoes geo-replication.indix.. on
07:33 Nevan joined #gluster
07:33 Troy can not kill it while replication sessions exists
07:34 Troy when I stop replication it says not active
07:34 Troy command failed
07:34 Troy but as per logs it is still trying to do the replication
07:37 sripathi joined #gluster
07:37 Troy [2013-03-14 00:37:51.412887] I [monitor(monitor):81:monitor] Monitor: starting gsyncd worker
07:42 Troy geo-replication session between gv0 & backup-server-1:/backup/gv0 not active
07:42 Troy geo-replication command failed
07:42 Troy geo-replication.indexing cannot be disabled while geo-replication sessions exist
07:43 Troy Set volume unsuccessful
07:48 jcaputo hmmm
07:48 jcaputo need somebody else on here that is more experienced with geo-replication
07:48 jcaputo i'm unsure
07:52 guigui joined #gluster
07:55 ctria joined #gluster
07:58 ngoswami joined #gluster
08:01 isomorphic joined #gluster
08:02 ekuric joined #gluster
08:12 Staples84 joined #gluster
08:18 andreask joined #gluster
08:19 yinyin joined #gluster
08:25 harshpb joined #gluster
08:29 yinyin joined #gluster
08:33 johndesc1 joined #gluster
08:36 hybrid5123 joined #gluster
08:37 zetheroo joined #gluster
08:38 zetheroo so after I make my two zfs bricks on a single server ... how do I get the OS to see them as one disk?
08:39 sahina joined #gluster
08:41 sjoeboo_ joined #gluster
08:41 inodb joined #gluster
08:47 neofob joined #gluster
08:50 stickyboy joined #gluster
08:58 zetheroo ok, so I am going to try to use LVM for that ...
08:58 displaynone joined #gluster
08:59 hybrid5123 joined #gluster
09:00 puebele joined #gluster
09:02 syoyo__ joined #gluster
09:02 sripathi joined #gluster
09:03 aravindavk joined #gluster
09:09 hagarth @channelstats
09:09 glusterbot hagarth: On #gluster there have been 99515 messages, containing 4354842 characters, 731434 words, 2959 smileys, and 367 frowns; 677 of those messages were ACTIONs. There have been 35316 joins, 1160 parts, 34174 quits, 14 kicks, 109 mode changes, and 5 topic changes. There are currently 209 users and the channel has peaked at 210 users.
09:13 zetheroo is there a way to tell glusterfs to only use eth1 and eth2 ?
09:13 Rocky joined #gluster
09:13 aravindavk joined #gluster
09:16 bulde1 joined #gluster
09:17 ndevos zetheroo: no, not really, but maybe http://hekafs.org/index.php/2013/01/sp​lit-and-secure-networks-for-glusterfs/ helps you?
09:17 glusterbot <http://goo.gl/nufzN> (at hekafs.org)
09:25 tjikkun_work joined #gluster
09:26 JasonG joined #gluster
09:36 jcaputo joined #gluster
09:39 duerF joined #gluster
09:41 rastar joined #gluster
09:43 Alpinist joined #gluster
09:45 sgowda joined #gluster
09:47 jdarcy joined #gluster
09:51 jdarcy joined #gluster
09:56 jcaputo joined #gluster
09:57 gbrand_ joined #gluster
10:05 sgowda joined #gluster
10:10 sripathi joined #gluster
10:14 shireesh joined #gluster
10:23 rastar joined #gluster
10:40 yinyin joined #gluster
10:50 aravindavk joined #gluster
10:54 sgowda joined #gluster
10:56 bulde joined #gluster
11:05 hybrid512 joined #gluster
11:28 neofob joined #gluster
11:29 lpabon joined #gluster
11:32 puebele2 joined #gluster
11:37 jtux joined #gluster
11:41 yinyin joined #gluster
11:42 manik joined #gluster
11:55 balunasj joined #gluster
11:58 aliguori joined #gluster
12:03 Shrini joined #gluster
12:08 plarsen joined #gluster
12:09 jdarcy joined #gluster
12:09 shireesh joined #gluster
12:14 sjoeboo joined #gluster
12:22 Shrini left #gluster
12:33 Staples84 joined #gluster
12:38 yinyin joined #gluster
12:50 shireesh joined #gluster
12:51 puebele joined #gluster
12:55 robo joined #gluster
12:58 yinyin_ joined #gluster
13:02 robos joined #gluster
13:04 rwheeler joined #gluster
13:07 x4rlos Does anyone know if the truncation on the line length of a volume has been resolved on the "#gluster volume status" command? Seems to be 52 characters in length when issued from command line under "gluster process". My pathnames aren't exactly massive.
13:07 x4rlos (if that makes sense).
13:13 jtux joined #gluster
13:14 wN joined #gluster
13:14 cw joined #gluster
13:17 bennyturns joined #gluster
13:18 yinyin joined #gluster
13:29 tqrst has anyone here gone from 256 byte to 512 byte inodes on ext4? tune2fs is taking forever. Just wondering how long I should expect this to last.
13:30 tqrst there's ~5 million files on there, so I guess that's a bit of rearranging, but still
13:30 tqrst might as well take the data off, format and put it back on at this rate
13:34 dustint joined #gluster
13:38 lh joined #gluster
13:38 tqrst I found a mailing list post complaining that tune2fs took an hour and a half on a 30G partition... hopefully this doesn't scale >= linearly
13:38 vpshastry joined #gluster
13:40 x4rlos tqrst: don't be too hasty to leave if you want an answer - this room can stay quiet for a while :-)
13:40 tqrst x4rlos: I'm always here :p
13:40 tqrst complaining about something or other
13:42 vpshastry left #gluster
13:43 andreask joined #gluster
13:45 glusterbot New news from newglusterbugs: [Bug 918917] 3.4 Beta1 Tracker <http://goo.gl/xL9yF>
13:46 x4rlos hehe. Me too :-p
13:46 sjoeboo joined #gluster
13:50 harshpb joined #gluster
13:50 hagarth joined #gluster
13:56 harshpb joined #gluster
13:59 tqrst looks like this was addressed partly back in 2008: http://marc.info/?l=linux-ext4​&amp;m=122672822822037&amp;w=2
13:59 glusterbot <http://goo.gl/ijtHn> (at marc.info)
14:02 sripathi joined #gluster
14:09 manik joined #gluster
14:12 mooperd_ joined #gluster
14:15 lala_ joined #gluster
14:21 harshpb joined #gluster
14:22 mooperd joined #gluster
14:25 torbjorn1_ My setup running 3.4.0-alpha2 running on squeeze-backport kernel 3.2.0 (which seems to be 3.2.35) seems to hang the kernel pretty
14:25 torbjorn1_ hard
14:25 stickyboy torbjorn1_: During compile?
14:25 torbjorn1_ IPMI-SOL gave me a traceback that seemed to indicate something bad with locking
14:26 torbjorn1_ a reset was necessary to bring it back, using login over IPMI-SOL seemed to hang when requiring IO
14:26 torbjorn1_ mounting went fine, these problems came when putting some strain on the system
14:26 wushudoin joined #gluster
14:26 torbjorn1_ stickyboy: no, this was using the FUSE client, server running on different boxes
14:27 torbjorn1_ If I were to investigate this, should that be as a kernel bug ? .. Given that Gluster lives in user-space, should it be able to hang the kernel ?
14:27 torbjorn1_ This was my own compile, done on a box running 2.6 kernel
14:28 deepakcs joined #gluster
14:34 harshpb joined #gluster
14:37 lala_ joined #gluster
14:37 harshpb joined #gluster
14:39 rastar joined #gluster
14:43 sripathi joined #gluster
14:57 lpabon joined #gluster
14:57 BSTR joined #gluster
15:05 rwheeler_ joined #gluster
15:06 sripathi joined #gluster
15:08 jag3773 joined #gluster
15:11 plarsen joined #gluster
15:14 Cenbe joined #gluster
15:22 daMaestro joined #gluster
15:23 wN joined #gluster
15:47 hybrid5121 joined #gluster
15:50 manik joined #gluster
15:53 hybrid512 joined #gluster
15:59 rotbeard joined #gluster
16:02 wN joined #gluster
16:06 HaraldJensas joined #gluster
16:11 eryc joined #gluster
16:13 Staples84 joined #gluster
16:18 zetheroo left #gluster
16:20 harshpb joined #gluster
16:28 mooperd joined #gluster
16:32 aliguori joined #gluster
16:39 harshpb joined #gluster
16:39 robos joined #gluster
16:43 hybrid512 joined #gluster
16:45 mooperd joined #gluster
16:48 harshpb joined #gluster
16:48 semiosis torbjorn1_: i wonder if the crash happened because you compiled on 2.6 kernel & ran on 3.2 kernel
16:48 semiosis try compiling on the same 3.2 kernel as you run on and see if that crashes too
16:52 Mo___ joined #gluster
16:56 _pol joined #gluster
16:57 _pol joined #gluster
17:00 bulde joined #gluster
17:09 ngoswami joined #gluster
17:12 harshpb joined #gluster
17:15 shylesh joined #gluster
17:16 glusterbot New news from newglusterbugs: [Bug 903396] Tracker for gluster-swift refactoring work (PDQ.2) <http://goo.gl/wiUbE>
17:17 f0urtyfive joined #gluster
17:18 f0urtyfive Hello... Wondering if I can use gluster to make two redundant NFS servers that can seamlessly transition between them when one goes down
17:19 mooperd joined #gluster
17:20 f0urtyfive my actual plan is to have two VMware VMs, on seperate hosts, on local storage... running OpenSolaris W/ ZFS with gluster on top serving NFS redundantly to both hosts, so either host could be rebooted without causing an outage
17:30 elyograg f0urtyfive: gluster can serve as the backend for such a system, but the failover must be handled separately. if you use heartbeat, ucarp, or something else like that to move a virtual IP between two servers running glusterd, that would likely do it.
17:43 f0urtyfive elyograg: ah ok
17:49 y4m4 joined #gluster
17:51 f0urtyfive elyograg: any idea if it would be more favorable to use Gluster to handle multi-site replication, or zfs send?
17:53 Troy joined #gluster
17:53 robos joined #gluster
17:53 elyograg f0urtyfive: I couldn't say.  I haven't used georeplication.
17:59 aravindavk joined #gluster
18:01 aravindavk joined #gluster
18:06 hateya joined #gluster
18:09 disarone joined #gluster
18:13 dustint joined #gluster
18:14 ctria joined #gluster
18:30 manik joined #gluster
18:31 Ryan_Lane joined #gluster
18:39 plarsen joined #gluster
18:42 dustint joined #gluster
18:45 ninkotech_ joined #gluster
18:50 _pol joined #gluster
18:51 _pol joined #gluster
19:03 dustint joined #gluster
19:06 dustint joined #gluster
19:39 Nagilum hmm what do I do when gluster volume info isn't returning anything anymore?
19:41 Nagilum the fs seems to be still operational though
19:41 Nagilum I did a replace brick before, which reported completion way too early :>
19:41 elyograg do any other commands give info, like 'gluster peer status'
19:42 Nagilum hmm, depends on the node
19:44 Nagilum on the node where gluster volume info isn't returning anything it just hangs
19:44 Nagilum on another node it works
19:45 Nagilum there the gluster info also works
19:46 dblack joined #gluster
19:53 JoeJulian Nagilum: Start glusterd on the unit that's hanging.
19:55 JoeJulian Nagilum: You might want to check "the log" on "the node".
19:55 * JoeJulian hates ambiguity.
19:57 Nagilum ah, restarting clusterd helped!
20:02 H__ Nagilum: which gluster version ?
20:02 Nagilum 3.3.1
20:03 H__ damn, that's not what i wanted to hear ! I'm about to move lots of bricks when i've upgraded to 3.3.1
20:09 plarsen joined #gluster
20:12 Nagilum hmm, when I try to redo the replace its just says "replace-brick failed to start"
20:13 Nagilum I'm trying a heal but the heal info output is not very enlightening :-/
20:13 JoeJulian If a glusterd wasn't responding, it would fail to start.
20:14 Nagilum k, let me go through the nodes
20:14 JoeJulian No need.
20:14 JoeJulian The client's don't need to be checked, just the servers.
20:15 Nagilum its a set of 6 servers which mount the glusterfs from localhost, so the hosts are both at the same time
20:16 JoeJulian Ah, ok. Just wanted to try to save you time as you don't need to check every endpoint on a network, just servers.
20:16 JoeJulian You did say you were going to check all your "nodes".
20:17 Nagilum true
20:18 JoeJulian Didn't Dante have a circle of hell dedicated to ambiguity? I think it wasn't a numbered one though.
20:18 semiosis ha
20:21 Nagilum hmm, restarted all the gluterd's but I'm still getting "replace-brick failed to start" any way to find out what the problem is?
20:22 semiosis logs.  first one i'd check is /var/log/etc-glusterfs-glusterd.log on the server where you issued the command
20:24 JoeJulian Check the log on the node.
20:24 Nagilum yep, checking
20:24 * JoeJulian makes himself chuckle.
20:30 Nagilum hmm, Replace brick is already started for volume
20:30 Nagilum Staging failed
20:31 Nagilum status says migration complete
20:31 Nagilum but the mountpoint usage shows it hasn't finished
20:32 Nagilum ok, abort seems to have worked
20:34 Nagilum ok, replace-brick started again
20:45 tqrst the answer to how long tune2fs -I 512 takes on a 2T partition with 5 million files is 28 hours
20:45 tqrst followed by a few hours of fscking
20:45 Nagilum hmm glusterfs process uses 100% CPU and almost no network activity on the server which is supposed to receive the brick :-/
20:51 Nagilum hmm, again it says migration complete without actually being complete :-/
20:52 andreask joined #gluster
20:52 JoeJulian Is this a replicated volume?
20:53 Nagilum yes
20:53 Nagilum replicated-distributed
20:53 JoeJulian Then you can just force it and self-heal if you want.
20:53 JoeJulian What makes you sure it's not complete?
20:53 Nagilum the brick holds >1TB of data, the new brick is at 30GB
20:53 JoeJulian I believe you, just wondering what the problem could be.
20:54 Nagilum should I try to commit?
20:55 JoeJulian Seems like a reasonable step.
20:55 JoeJulian Then I'd do a heal...full.
20:56 Nagilum hmm, commit is taking its time
20:56 JoeJulian Probably migrating fd's and locks I would guess.
20:56 Nagilum ok
21:00 Nagilum it finished (without saying anything), gluster volume info hangs again though :-/
21:00 JoeJulian Hmm, sounds like a timeout.
21:01 Nagilum on another volume info worked, but it doesn't list the new brick :-/
21:01 Nagilum s/volume/node/
21:01 glusterbot What Nagilum meant to say was: on another node info worked, but it doesn't list the new brick :-/
21:02 JoeJulian restart all glusterd. check to see what's using 100% on that target server (or if anything still is)
21:02 Nagilum k
21:04 Nagilum hmm, kill -9 not working :-o
21:05 JoeJulian What're you trying to kill -9?
21:06 Nagilum glusterd
21:06 * JoeJulian raises an eyebrow...
21:08 JoeJulian ps -C glusterd -o ppid
21:08 JoeJulian I expect it to be 1, but if it's not, try killing whatever it is.
21:08 Nagilum it is 1
21:08 JoeJulian wow...
21:09 JoeJulian That's kernel debugging time...
21:10 JoeJulian I don't /think/ this should affect glusterd, but none of these bricks are ext4?
21:10 Nagilum correct
21:10 Nagilum alls xfs
21:11 JoeJulian Is glusterd Z now?
21:11 Nagilum top shows it as R
21:12 Nagilum oh, wait
21:12 Nagilum thats glusterfs not the glusterd
21:12 JoeJulian Ah, that's what I originally expected.
21:12 JoeJulian What's the whole command line for that one
21:13 Nagilum /usr/sbin/glusterd -p /run/glusterd.pid
21:14 JoeJulian No, for the blocked glusterfs
21:14 JoeJulian Or were you saying glusterfs is R and glusterd is Z?
21:14 Nagilum glusterfs is R
21:15 JoeJulian that was a logical and.
21:15 Nagilum ps aux shows "Ds" for glusterd
21:17 glusterbot New news from newglusterbugs: [Bug 895528] 3.4 Alpha Tracker <http://goo.gl/hZmy9>
21:18 mricon joined #gluster
21:19 Nagilum let me just reboot that box
21:19 mricon Could I have a setup where I have a replicated volume that is mounted read-write on "master" but read-only on the clients?
21:19 mricon s/clients/"slaves"?
21:20 mricon e.g. if I just wanted to mirror a large collection of files around the world, but I don't want those files modified.
21:20 mricon except on the "master"
21:21 JoeJulian Nagilum: Basically that meant that the process was in iowait. It shouldn't have been. It barely touches the hard drive except to read/edit the configuration files.
21:22 Nagilum I see
21:22 JoeJulian mricon: as of 3.3, you can mount ro.
21:23 mricon JoeJulian: looks like it's only 3.2 in EPEL, bummer
21:23 JoeJulian @yum repo
21:23 glusterbot JoeJulian: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
21:25 mricon ah, @redhat.com, me likes.
21:25 mricon how good is gluster at replicating over fairly slow links?
21:26 JoeJulian Not very. If that's what you're looking for (and you don't want to write to the slaves) then geo-replication sounds perfect for your needs.
21:27 mricon ah, nice
21:28 mricon sorry, still learning
21:28 JoeJulian That's why I hang out here. :)
21:28 mricon I may prod your brain a few times, then. :)
21:28 mricon I'm looking to replace a highly inefficient rsync setup
21:28 JoeJulian geo-replication is a much more efficient rsync setup. :)
21:30 mricon are there any benchmarks I can look at? E.g. if I have a volume with ~300GB that receives 100MB worth of writes -- how quickly will that replicate to 4 worldwide locations over 100Mbps and such?
21:31 mricon I don't need 3.3 for geo-replication, right?
21:34 JoeJulian Right, but I would recommend it.
21:35 JoeJulian kkeithley is the packager for epel. 3.3 isn't in for political reasons.
22:04 tqrst "tar: Cannot stat: Structure needs cleaning; tar: File shrank by 597 bytes; padding with zeros." oh joy
22:05 tjstansell jdarcy: any way i can bribe you to review the backport for release-3.3 for bug 918437? :)
22:05 glusterbot Bug http://goo.gl/1QRyw urgent, unspecified, ---, pkarampu, MODIFIED , timestamps updated after self-heal following primary brick rebuild
22:05 JoeJulian I'm pretty sure jdarcy's in bangalore so he might not be around.
22:07 tjstansell ah. well, maybe he'll see this at some point ... i just don't want it to get forgotten.
22:07 JoeJulian They're in India for the glusterfs developer conference.
22:08 tjstansell oh, i thought that was last week.
22:08 JoeJulian I could be wrong.
22:08 JoeJulian They're all running together.
22:13 tjstansell hm.. yeah, he left this chat room an hour and a half ago... oh well.
22:13 jdarcy joined #gluster
22:14 JoeJulian strange, too, since he uses quassel.
22:25 tqrst problem of the day: cat alternates between erroring out with Structure needs cleaning and just outputting nothing. Glusterd logs on that machine show http://pastie.org/private/iovgef8vhwcfabbvtd28oq
22:25 glusterbot <http://goo.gl/IhtGq> (at pastie.org)
22:25 tqrst this is in an area of our volume that hasn't been touched in forever (read: 3.2.5 days)
22:26 tqrst it happens on multiple files in there, too
22:26 tqrst ideas? cookies? chairs?
22:26 tqrst permissions are ---------T
22:27 JoeJulian First one... why is there a dht translator in /etc/glusterfs/glusterd.vol?
22:28 tqrst oddly enough, there isn't
22:28 JoeJulian But you're getting bricks logs in glusterd?
22:28 tqrst ah sorry about that - multitail interleaved in a weird way when I pasted
22:29 JoeJulian Ah, ok.
22:29 JoeJulian Whew.
22:29 tqrst that was actually mostly from the client logs
22:30 tqrst let me just repaste that, it's confusing
22:30 JoeJulian So does that error correspond to the right timestamp for each of those "stucture needs cleaning"?
22:31 JoeJulian And is it the same error each time?
22:33 JoeJulian Afaict, that might actually be an xfs error. Try running an xfs_repair on  bricks 25 and 26 (24 and 25 if you count from 0).
22:33 lh joined #gluster
22:33 lh joined #gluster
22:34 tqrst JoeJulian: same each time; http://pastie.org/private/mmlhagoll7ktrwkfm8xqq
22:34 glusterbot Title: Private Paste - Pastie (at pastie.org)
22:35 tqrst these are actually ext4
22:36 tqrst I see an increase in unreadable (pending) sectors in /var/log/messages
22:40 tqrst I can read it just fine if I go through the bricks
22:42 JoeJulian Can you read the extended attributes just find if you go through the bricks?
22:43 JoeJulian s/find/fine/
22:43 glusterbot What JoeJulian meant to say was: Can you read the extended attributes just fine if you go through the bricks?
22:45 JoeJulian If the extended attributes exceeded the inode size, they would have spilled over into an additional inode. If that inode's on a bad sector, this could cause that error. This wouldn't cause that error with things like "cat", because they never try to read that inode.
22:45 tqrst yes. trusted.afr.bigdata-client-24 and trusted.afr.bigdata-client-25 are both 0 on both bricks, and trusted.gfid is 0xa9e7f51545614b55b467d9b395d1575e on both too
22:45 tqrst but on both drives?
22:45 JoeJulian Are those two drives a replica pair?
22:45 tqrst yes
22:46 tqrst this is a 25x2 distributed-replicate
22:47 JoeJulian no, google, I did not mean euclidean! I actually meant euclean when I typed euclean.
22:48 tqrst heh, it doesn't try to correct it for me, although the wiki page for euclidean distance is in the top 10 results
22:48 JoeJulian hehe
22:48 JoeJulian what kernel are you running?
22:49 tqrst 2.6.32-131.17.1.el6.x86_64
22:49 tqrst this is scientific linux 6.1, which is basically just centos
22:49 tqrst I haven't updated the kernel recently because most of my volume is ext4
22:49 JoeJulian which is basically just rhel.
22:51 tqrst I've also seen "[2013-03-14 18:08:59.495801] E [posix.c:2770:posix_fgetxattr] 0-bigdata-posix: fgetxattr failed on key trusted.glusterfs.dht.linkto (No data available)" in the brick logs of another server, although I'm not sure what caused that one given the timestamp.
22:51 tqrst I've been poking through old files all afternoon
22:52 JoeJulian You can use the code in http://joejulian.name/blog​/dht-misses-are-expensive/ to see if that missing linkto is involved.
22:52 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
22:52 JoeJulian Compute the hash, check the bricks and see which one it belongs on.
22:53 JoeJulian Unless you've set lookup-unhashed off, though, it should create the linkto if it's missing.
22:55 tqrst argh, gotta run
22:55 tqrst I'll have another look tomorrow; thanks for the help so far
22:55 JoeJulian o/
22:56 JoeJulian Heh, looks like I partially figured this out already: http://irclog.perlgeek.de/g​luster/2012-10-29#i_6109523
22:56 glusterbot <http://goo.gl/FHsLx> (at irclog.perlgeek.de)
22:56 mricon does geo-replication require ssh?
22:56 rwheeler joined #gluster
22:57 mricon I already have a trusted openvpn uplink, so I'd rather avoid having to encrypt twice
22:57 andreask no, it only requires rsync
22:57 mricon ah, excellent
22:58 andreask but I must admit I have not tried without ssh
22:59 mricon Most documentation lists setting it up with ssh
22:59 mricon which is why I was confused
22:59 andreask sure, that is the default remote shell
23:00 mricon so, all I need on the slave is a listening rsync running with permissions to write to the glusterfs volume?
23:01 andreask it is all handled by special daemons
23:01 andreask I don't think it uses rsyncd
23:01 mricon ok, let me set up a test environment before I ask any more questions :)
23:02 JoeJulian glusterd needs to be able to remotely execute commands as root. I'm not sure if ssh is a hardcoded requirement or not.
23:02 andreask I'd assume so
23:03 JoeJulian btw... the way I usually answer these kinds of questions is by grepping the source. "git grep ssh" would probably give a pretty clear indication of how it's used.
23:24 jdarcy joined #gluster
23:39 jdarcy joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary