Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-20

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 lh joined #gluster
00:03 lh joined #gluster
00:27 duffrecords joined #gluster
00:28 khushildep joined #gluster
01:01 weplsjmas joined #gluster
01:05 nueces joined #gluster
01:06 weplsjmas joined #gluster
01:12 neofob_ joined #gluster
01:13 neofob_ when i try to replace a brick on by another new brick on the same server; it fails; the error in cli.log is "0-rpc-transport: missing 'option transport-type'. defaulting to "socket""
01:13 neofob_ how do i fix this?
01:15 JoeJulian That's not a failure (notice there's no " E " in the log for that line).
01:17 neofob_ JoeJulian: so what's wrong?
01:18 JoeJulian So far all I know is that "it fails".
01:18 JoeJulian Does it error?
01:19 neofob_ after the cmd "gluster volume replace-brick..." I get the returned msg: "replace-brick failed to start"
01:20 JoeJulian Sure, makes sense. This is for just remounting the brick in a different location, right?
01:21 neofob_ i have one brick on gluster0:/data/brick0 now i want to replace it with another brick on the same server gluster0:/data/brick3
01:22 neofob_ brick0 is 1TB; brick3 is 3TB
01:22 JoeJulian Ah, ok.. this wasn't the one I was thinking of.
01:23 JoeJulian Could you fpaste "gluster volume status" please?
01:23 neofob_ sure
01:24 neofob_ http://pastebin.com/1N5KSiph
01:24 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
01:26 JoeJulian Wanna try again? [17:23] <JoeJulian> Could you fpaste "gluster volume status" please?
01:31 JoeJulian No? Ok... See you later then. I'm off to have a 20th anniversary dinner with my wife. Bye!
01:32 neofob_ oh, ok
01:32 nhm JoeJulian: congrats!
01:33 nhm JoeJulian: Hope you got her something good. ;)
01:33 neofob_ JoeJulian: http://fpaste.org/iNWi/
01:33 glusterbot Title: Viewing gluster volume info by neofob (at fpaste.org)
01:34 neofob_ hah, i missed that; JoeJulian: congrat!
01:43 kevein joined #gluster
01:50 robinr joined #gluster
01:58 FyreFoX hi are there recommended formatting options for XFS for use with glusterfs?
02:15 robinr hi, i find that when i have a client writing a big file via gluster protocol, gluster volume heal VOL info will always shows that there are entires not synchronized. so, i take that a safe path to do minor upgrade, say from 3.3.0 to 3.3.1 is to (1) upgrade first gluster machine, (2) sync up, then (3) upgrade second machine. But if there's always a client writing a big file, then step (2) is impossible to accomplish ?
02:15 robinr am i missing something on what unsynchronized entries mean ?
02:16 sunus joined #gluster
02:34 semiosis FyreFoX: -i size=512 normally, or if you're going to use UFO, then size=1024.  this is the inode size, and making it larger than default allows gluster's extended attributes to fit in a single inode (rather than overflowing into another)
02:35 semiosis ~extended attributes | robin
02:35 glusterbot robin: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
02:35 semiosis robinr: ^^
02:35 semiosis see that article
02:36 semiosis to summarize, it's safe
02:36 robinr semiosis, let me read that. looks heavy
02:36 semiosis well, i assume that what you're reporting is normal behavior, idk that for sure
02:37 FyreFoX thanks semiosis
02:37 semiosis yw
02:48 robinr thanks, semiosis. i have probably mis-used gluster volume heal VOLNAME info as a means of monitoring (that is if there are unsynchronized entries, send a warming). what's a better way of doing monitoring ?
02:48 semiosis that's a very good question
02:49 semiosis first of all... are you triple sure that your client really is connected to both bricks?  and that the file is being written to on both of them continuously?
02:50 m0zes hrm, I've got a bit of an oddity here. I have a 5-node gluster cluster, upgraded from 3.2.7 to 3.3.1 today. When checking gluster peer status, I have 3 entries that are 'State: Peer Rejected (Connected)'
02:50 m0zes the thing is, all 5 share a volume.
02:50 semiosis robinr: i'm not familiar enough with volume heal info
02:50 semiosis 5-node?  that's an odd number
02:50 semiosis ~peer rejected | m0zes
02:50 glusterbot m0zes: I do not know about 'peer rejected', but I do know about these similar topics: 'peer-rejected'
02:51 semiosis ~peer-rejected | m0zes
02:51 glusterbot m0zes: http://goo.gl/nWQ5b
02:51 robinr semiosis: i simply get number of unsynchronized entries, if it's non-zero sound a warning
02:51 semiosis robinr: yeah but i mean have you validated your assumption that some entries are normal for in-flight writes?
02:51 robinr it seems like it's raising a lot of false alarms, especially if the clients are busy
02:51 semiosis because i never have
02:51 semiosis so i dont want to give you that impression
02:52 semiosis robinr: then when the clients calm down the number goes back to zero?  that is reassuring
02:52 robinr semisios, oh ok. yes.. when the clients stop writing, the number of unsynchornized entries go back to 0
02:54 semiosis robinr: you can find the nagios checks I use in my ,,(puppet) module
02:54 glusterbot robinr: (#1) https://github.com/semiosis/puppet-gluster, or (#2) https://github.com/purpleidea/puppet-gluster
02:54 robinr semisios, got you. at this point, you do not have a position (agree/disagree) with the statement that "some unsynchronized are normal during big writes".
02:54 semiosis basically watching processes, mounts, and scraping logs
02:54 semiosis it's not pretty, but it works
02:55 m0zes semiosis: how do I figure out which are the good and which are the bad? it switches depending on which of the 5 I do a peer status on.
02:55 robinr semiosis, thanks for additional ideas.
02:55 semiosis m0zes: eek
02:55 semiosis robinr: yw, monitoring is an ongoing issue
02:57 m0zes could this happen if two servers have rdma and share have several tcp,rdma volumes? maybe the three without rdma don't know how to cope...?
02:57 semiosis m0zes: idk
02:57 m0zes fair enough
02:57 semiosis i've seen it happen when I change a volume option while a server is offline, then when it comes back online it gets rejected
02:58 m0zes hrm, I don't know when that would have happened... it may be they've been in this state for a while...
02:58 semiosis m0zes: you could probably just choose one and sync all the others from it.  just make sure you keep backups of the data in /var/lib/glusterd you remove from each server... in case things go nuts
02:59 semiosis or you could go through and diff the .vol files on all of them, see what's different
02:59 m0zes I was thinking that would probably be the best bet.
03:00 semiosis i'm not up to speed on all the other stuff in /var/lib/glusterd these days so idk what impact that stuff has on peer status
03:00 semiosis i guess, if you find that the /var/lib/glusterd/vols/** is the same on all servers, then move on to other subdirs
03:01 semiosis but i'd def. start with vols
03:05 m0zes it seems like the rdma differences are what partially trigger it. the nodes without rdma rewrite the tcp,rdma volumes to just tcp.
03:07 m0zes a little annoying when they don't have any bricks in that volume.
03:07 semiosis wow that sounds like a bug for sure
03:07 semiosis m0zes: well now you know which ones are your "bad" nodes :)
03:07 m0zes indeed.
03:08 m0zes I'll resync the volumes and see if it happens again. it may have been something fixed in the 3.3 line, and I just didn't notice the peer rejected before now.
03:23 m0zes hrm. something is definitely buggy. followed instructions from peer-rejected link, one "good" hosts shows connected, the other "good" host is now rejected, from the "bad" server.
03:25 semiosis yikes
03:26 semiosis tbh, i've never actually deal with multiple rejected servers at once
03:27 m0zes fair enough. I am sure this relates to rdma at this point. can I sync the volumes "behind the scenes"?
03:27 semiosis i dont see why not, just stop a glusterd before you go messing around in its head
03:28 sazified joined #gluster
03:28 m0zes I'll give that a whirl
03:31 m0zes whee. all connected now.
03:32 m0zes probably would have only been an issue if I changed settings on the volume all 5 share, but it is good to make them think everything is in working order ;)
03:33 semiosis cool
03:33 semiosis ok outta here
03:33 m0zes thanks for your help :)
03:33 semiosis any time
03:33 semiosis yw
03:34 duffrecords left #gluster
03:42 theron joined #gluster
03:54 bulde joined #gluster
03:54 robinr semiosis, i've come to conclusions that unsynchronized entries are normal (especially on big files); http://gluster.org/pipermail/gluste​r-users/2012-September/011450.html
03:54 glusterbot <http://goo.gl/Eo6sn> (at gluster.org)
03:54 robinr thanks for the help
04:03 vpshastry joined #gluster
04:11 neofob_ left #gluster
04:11 Humble joined #gluster
04:32 shylesh joined #gluster
04:32 shylesh_ joined #gluster
04:40 isomorphic joined #gluster
04:51 hagarth joined #gluster
05:22 sgowda joined #gluster
05:42 Humble joined #gluster
05:47 bala joined #gluster
05:49 raghu joined #gluster
05:49 hagarth joined #gluster
05:51 rastar joined #gluster
05:56 an joined #gluster
05:58 Guest40477 Hey, I was doing some experimental benchmarking with glusterfs.... I seem to run into an issue, if I write files with similar names from 2 nodes to a 2 node distributed and non replicated setup
05:58 Guest40477 performance goes down?
05:58 Guest40477 any clue what may be happening?
06:00 bauruine joined #gluster
06:31 mohankumar joined #gluster
06:34 badshaa__ joined #gluster
06:35 vimal joined #gluster
06:37 shireesh joined #gluster
06:43 raven-np joined #gluster
06:46 robinr joined #gluster
06:47 raven-np weird issue with replicated setup; writes hitting the same node intermittently ..
06:48 badshaa__ @raven-np you seemed to be hit by the same issue
06:49 raven-np file sizes are same but names are different (generated)
06:52 ekuric joined #gluster
06:54 hagarth joined #gluster
06:55 66MAADNMH joined #gluster
06:58 santanu-sinha joined #gluster
07:00 ngoswami joined #gluster
07:00 badshaah joined #gluster
07:01 guigui3 joined #gluster
07:05 bulde joined #gluster
07:06 hagarth1 joined #gluster
07:16 jtux joined #gluster
07:23 bulde joined #gluster
07:25 hagarth joined #gluster
07:30 santanu-sinha hi there...
07:30 santanu-sinha need some info/help
07:33 santanu-sinha1 joined #gluster
07:38 robinr joined #gluster
07:39 robinr joined #gluster
07:40 santanu-sinha joined #gluster
07:42 santanu-sinha1 joined #gluster
07:42 santanu-sinha1 left #gluster
07:42 bulde joined #gluster
07:44 shireesh joined #gluster
07:47 dobber joined #gluster
07:49 mooperd joined #gluster
07:51 badshaah_ joined #gluster
07:55 khushildep joined #gluster
07:56 ctria joined #gluster
07:57 tjikkun_work joined #gluster
08:03 passie joined #gluster
08:04 glusterbot New news from resolvedglusterbugs: [Bug 765526] [glusterfs-3.2.5qa6]: glusterfs server crashed <http://goo.gl/ypBMS>
08:13 jtux joined #gluster
08:16 hateya joined #gluster
08:19 tryggvil joined #gluster
08:22 hateya joined #gluster
08:26 andreask joined #gluster
08:34 bauruine joined #gluster
08:39 ramkrsna joined #gluster
08:39 ramkrsna joined #gluster
08:42 m0zes joined #gluster
08:55 Norky joined #gluster
08:59 gbrand_ joined #gluster
09:02 Nevan joined #gluster
09:04 glusterbot New news from resolvedglusterbugs: [Bug 830701] glusterfs memory leak <http://goo.gl/YLSJI>
09:10 hateya joined #gluster
09:23 x4rlos JoeJulian: Sorry, i left for home yesterday and missed your reply.
09:23 x4rlos JoeJulian: I just wonder how your creating the mountpoint.
09:24 mooperd joined #gluster
09:28 DaveS joined #gluster
09:29 Norky joined #gluster
09:31 Norky joined #gluster
09:33 GLHMarmot joined #gluster
09:34 guest2012 joined #gluster
09:37 QuentinF joined #gluster
09:37 QuentinF Hi
09:37 glusterbot QuentinF: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:37 QuentinF I'va a pb with glusterfs 3.3.1
09:39 ekuric joined #gluster
09:41 QuentinF http://pastebin.com/PUYgBgPE
09:41 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
09:42 ekuric joined #gluster
09:42 QuentinF http://dpaste.org/PSe9z/
09:42 glusterbot Title: dpaste.de: Snippet #215401 (at dpaste.org)
09:42 bida joined #gluster
09:44 inodb_ joined #gluster
09:49 Humble joined #gluster
09:49 x4rlos I love how GB handles people.
09:54 haakond_ joined #gluster
09:59 mooperd joined #gluster
10:00 gbrand_ joined #gluster
10:03 greylurk_ joined #gluster
10:04 QuentinF [2012-12-20 11:03:55.610450] E [rpcsvc.c:491:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
10:04 QuentinF even if i've set nfs.ports-insecure: on
10:04 QuentinF Any ideas ?
10:09 vimal joined #gluster
10:11 weplsjmas joined #gluster
10:12 bala joined #gluster
10:20 Tekni joined #gluster
10:24 VeggieMeat_ joined #gluster
10:25 cbehm_ joined #gluster
10:26 jds2001_ joined #gluster
10:29 ackjewt joined #gluster
10:29 jiqiren joined #gluster
10:41 passie joined #gluster
10:42 ndevos QuentinF: try http://lists.nongnu.org/archive/html​/gluster-devel/2012-12/msg00031.html
10:42 glusterbot <http://goo.gl/59SD1> (at lists.nongnu.org)
10:47 QuentinF thx
10:48 bala joined #gluster
10:50 puebele1 joined #gluster
11:00 RobertLaptop joined #gluster
11:19 mooperd joined #gluster
11:27 khushildep joined #gluster
11:43 edward1 joined #gluster
11:44 nullck joined #gluster
11:53 badshaah joined #gluster
12:00 khushildep joined #gluster
12:15 kkeithley1 joined #gluster
12:22 badshaah joined #gluster
12:25 badshaa__ joined #gluster
12:26 badsh____ joined #gluster
12:36 mohankumar joined #gluster
12:42 khushildep joined #gluster
12:45 Norky joined #gluster
12:47 Norky joined #gluster
12:49 Norky joined #gluster
12:49 guigui1 joined #gluster
12:51 hagarth joined #gluster
12:52 sazified joined #gluster
12:53 balunasj joined #gluster
12:54 guest2012 joined #gluster
13:01 rwheeler joined #gluster
13:02 manik joined #gluster
13:04 vpshastry joined #gluster
13:08 puebele joined #gluster
13:24 bala joined #gluster
13:28 tryggvil_ joined #gluster
13:30 shireesh joined #gluster
13:40 thekev joined #gluster
13:41 JordanHackworth joined #gluster
13:41 sunus joined #gluster
13:59 Netsurge joined #gluster
14:04 Humble joined #gluster
14:05 raven-np joined #gluster
14:06 shireesh joined #gluster
14:12 pkoro joined #gluster
14:19 wN joined #gluster
14:20 Netsurge Hello #gluster, I have some questions regarding using gluster for an enterprise storage system
14:21 Netsurge WE currently have some older backup servers that use off the shelf dirves and its very unreliable
14:21 Netsurge WE also have an hp 3par for primary storage.  Looking to archive data and place backups on something that can be scaled
14:24 shireesh joined #gluster
14:24 H__ sounds like a fit to me
14:25 vpshastry left #gluster
14:27 vpshastry joined #gluster
14:31 Netsurge H__ : Would you put your job on it?
14:31 Netsurge We have the budget to go VNX / the SAN route
14:32 Netsurge but i would like to see something more Distributed
14:32 Netsurge and able to span datacenters
14:34 bennyturns joined #gluster
14:35 flowouffff joined #gluster
14:39 H__ Netsurge: I run it in production
14:41 m0zes spanning datacenters *can* be done, currently I wouldn't use it that way unless latency is really low and the connection is pretty fast. 3.4 looks like it will have multi-master georeplication that could be very interesting http://gluster.org/community/doc​umentation/index.php/Planning34
14:41 glusterbot <http://goo.gl/VoH8X> (at gluster.org)
14:41 H__ there's issues, sure, geo-replication does not work for me and I'm not happy about the lots-of-files speed, and the rebalance did the Wrong Thing after 8 add-bricks
14:42 H__ Netsurge: what's the bandwidth and ping between the datacenters ?
14:43 m0zes 3.4 looks like it may help some of the "lots-of-files" speed issues. if the 'readdirplus' attribute makes it in, and the kernel patch for FUSE is accepted.
14:45 50UAB2N2D joined #gluster
14:47 H__ yes that piece looks interesting. have been following it on the mailing list
14:47 H__ m0zes: for now I'm stuck at 3.2.5, I cannot upgrade to 3.3 as my storage system cannot go down
14:48 tryggvil joined #gluster
14:49 m0zes yeah, downtime is a hard thing to schedule. I was able to force some extra time for downtime due to whole building power upgrades yesterday...
14:50 guigui1 joined #gluster
14:55 hagarth joined #gluster
14:55 sunus joined #gluster
14:55 Netsurge H: Right now its 100mbs, we are upgrading to somewhere between 500mbs - 2Gbs ping right now is 1.51ms
14:55 Netsurge hmm
14:55 Netsurge Here is a question.
14:56 Netsurge Can i use glusterFS for the fileshare then use DFS-R in MSwindows for syncing the data?
14:56 H__ that bandwidth and ping times would work well for my load
14:56 Netsurge the second site is a DR site and not production
14:56 Netsurge H: thats what she said??
14:56 H__ don't know what DFS-R is
14:57 Netsurge sorry, I couldnt resist that one
14:57 Netsurge Its basically rsync for windows
14:57 H__ i think i'm missing a pun here
14:57 H__ (non-native english speaker here)
14:58 Netsurge Ah, ok.
15:00 stopbit joined #gluster
15:00 _Bryan_ I am rpetty sure we have addressed this before...but Why do the stale NFS file handle errors not go away over time....I understand that they are not really NFS errors but having a few thousand of them in a self heal out put is a bit of a pain
15:01 _Bryan_ and it is not always available to bounce the moutn to clean them up for quite some time
15:07 vpshastry joined #gluster
15:08 guigui1 joined #gluster
15:25 raven-np joined #gluster
15:28 stopbit joined #gluster
15:29 obryan joined #gluster
15:31 wushudoin joined #gluster
15:32 khushildep joined #gluster
15:41 vpshastry left #gluster
16:03 passie left #gluster
16:07 raven-np joined #gluster
16:09 daMaestro joined #gluster
16:10 nueces joined #gluster
16:12 duerF joined #gluster
16:15 wN joined #gluster
16:27 hackez joined #gluster
16:33 QuentinF I
16:33 QuentinF Hi
16:33 glusterbot QuentinF: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:33 QuentinF I've a problem with glusterfs 3.3
16:34 QuentinF my client  clients do not synchronize completely
16:34 khushildep joined #gluster
16:35 QuentinF and i have errors when i try to move in directory
16:35 QuentinF "Invalid argument"
16:36 QuentinF same when i try to create directory
16:36 QuentinF And, i haven't this pb in glusterfs 3.2.7
16:42 isomorphic joined #gluster
16:44 obryan joined #gluster
16:47 JoeJulian _Bryan_: 3.3.0 had that problem, but it seems to be fixed in 3.3.1. Which version are you seeing that with?
16:48 _Bryan_ 3.2.5  8-)
16:48 _Bryan_ I just thought KK told me there was another way it cleaned up those stale handles....and that remounting was not the only option
16:48 JoeJulian Not sure what to tell you then. I skipped the whole 3.2 series.
16:49 JoeJulian bbl...
16:49 _Bryan_ thanks for reply
17:03 Nevan1 joined #gluster
17:09 mdarade joined #gluster
17:09 mdarade left #gluster
17:28 mohankumar joined #gluster
17:31 isomorphic joined #gluster
17:35 wN joined #gluster
17:36 Mo___ joined #gluster
17:37 morse joined #gluster
17:56 andreask joined #gluster
17:56 zaitcev joined #gluster
17:59 Netsurge left #gluster
18:01 plarsen joined #gluster
18:04 QuentinF What is procÃedure when glusterfs 3.2.7 bricks down ? (I'm in replicate (x2) and distribute with 3 servers so 6 bricks
18:13 Humble joined #gluster
18:17 chirino joined #gluster
18:20 bauruine joined #gluster
18:25 partner joined #gluster
18:37 elyograg jdarcy: I really like this channel.  I can personally thank you for picking up my observer BZ report.
18:42 dbruhn joined #gluster
18:58 JoeJulian _Bryan_: You could try changing a volume setting, aka "gluster volume set $vol performance.io-cache off; gluster volume reset $vol performance.io-cache"
18:58 chirino joined #gluster
18:59 _Bryan_ on the client?
19:00 _Bryan_ did not know you could run any of the gluster commands on the client....
19:00 JoeJulian No, not on the client.
19:00 JoeJulian Not unless you installed -server and added the client to the peer group.
19:02 arusso_znc joined #gluster
19:03 JoeJulian jdarcy: memoryalligator = java?
19:04 danishman joined #gluster
19:04 JoeJulian QuentinF: When a brick is down, there's not really any procedure. When it comes back up, though, you should run a ,,(repair).
19:04 glusterbot QuentinF: http://goo.gl/uA812
19:07 weplsjmas joined #gluster
19:07 arusso_znc joined #gluster
19:19 greylurk joined #gluster
19:31 arusso- joined #gluster
19:31 arusso joined #gluster
20:02 _Scotty joined #gluster
20:04 _Scotty Howdy all, can someone help me with a cluster.min-free-disk question?
20:05 JoeJulian Possibly, but I won't guarantee you'll like the answers. :D
20:07 _Scotty LOL!  Fair enough :-)  I'm really trying to figure out how that setting is supposed to work versus what I am observing.  I read that setting it as a percentage doesn't always work (which I can verify) and to use a size instead, e.g. 2TB.  That doesn't seem to really work, either.
20:08 _Scotty My test configuration consists of two nodes, each with a single brick utilizing ZFS.  You shouldn't really use more than 80% of the ZFS partition because ZFS will start to slow down.  I'm using a distributed volume with volume-level quotas set.
20:09 _Scotty However, in my observation, Gluster will happily let that ZFS partition fill right up to 100% utilization.  It seems to ignore the min-free-disk setting.
20:09 darth joined #gluster
20:10 _Scotty Any thoughts on that one?
20:10 sazified joined #gluster
20:10 JoeJulian If that's for existing files, that's not surprising. I'm pretty certain that setting only affects creating new files.
20:10 _Scotty I guess I should also mention this is under CentOS 6.3, GFS 3.3.1
20:11 _Scotty So, I set the min-free-disk then kept copying files until the volume filled to 100%, which in turn filled up ZFS to 100%.
20:12 _Scotty I guess I could set the volume quota to be 80% of the total available space, but that doesn't help me on a per-brick basis with heterogeneous nodes.
20:14 JoeJulian That's an interesting observation. I'm not sure I've seen that discussed before.
20:14 JoeJulian It should prevent new files from exceeding the limit on a per-brick basis, but I've never encountered the whole volume exceeding those limits.
20:15 _Scotty Hmm.
20:16 JoeJulian I can't test it myself right now (critical work stuff) but that's probably worth having you file a bug report.
20:16 glusterbot http://goo.gl/UUuCq
20:16 _Scotty Thanks Joe.
20:16 _Scotty I'll fill one out now.
20:18 AndroUser joined #gluster
20:18 AndroUser Hi every one my name is cristiano corsani
20:19 AndroUser From florence national library
20:19 _Scotty @JoeJulian: Would that be considered a quota and core bug?
20:19 AndroUser We work with gluster and is great
20:19 _Scotty @AndroUser Hi Cristiano
20:20 AndroUser See www.slideshare.net/mobile/ccors​ani/ohacc-120510114446phpapp02
20:20 AndroUser And (sorry in italian) www.slideshare.net/mobile/ccorsani/bncf-​digital-long-term-preservation-glusterfs
20:21 JoeJulian _Scotty: Not sure... just pick one and if it's wrong it'll get moved during triage.
20:21 AndroUser We are project a huge storage for document long preservation
20:22 AndroUser I would like to know if comunity could be interested to help
20:22 AndroUser And develop some feature
20:22 AndroUser I don't want to abuse of this channel
20:23 bronaugh http://www.gluster.org/2011/11/trans​lator-101-class-1-setting-the-stage/
20:23 glusterbot <http://goo.gl/TL8tl> (at www.gluster.org)
20:23 bronaugh uh, isn't there supposed to be, you know, content here?
20:23 bronaugh ah; the actual content's at hekafs.org. not that there's any link in the glusterfs docs.
20:23 bronaugh who the hell curates those?
20:24 AndroUser I would like to leave my email: cristiano.corsani@beniculturali.it
20:25 AndroUser If someone is interested in more details
20:26 JoeJulian bronaugh: That's a legacy problem from before some upgrade, I believe, that resulted in data loss.
20:26 bronaugh JoeJulian: glusterfs seems to have a real problem with documentation :/
20:27 JoeJulian You should have seen it 2 1/2 years ago...
20:27 bronaugh I kind of did. I checked out glusterfs around that long ago.
20:27 JoeJulian So which documentation problem are you having now?
20:28 bronaugh well; I was looking for what translators are available.
20:28 bronaugh found the page eventually.
20:28 bronaugh but yeah.
20:28 JoeJulian Ah, well since volumes are built from the cli now, most of those docs have become irrelevant.
20:29 bronaugh http://www.gluster.org/community/docum​entation/index.php/Gluster_Translators
20:29 glusterbot <http://goo.gl/U2g8m> (at www.gluster.org)
20:29 bronaugh this looks pretty relevant.
20:30 JoeJulian So you're looking for default option values?
20:30 bronaugh even a list of all option values along with meanings, how to use them, etc.
20:31 bronaugh reference documentation.
20:31 JoeJulian gluster volume set help
20:37 bronaugh JoeJulian: obvious question: where does it say you can specify "help" after options?
20:37 bronaugh it being docs.
20:39 glusterbot New news from newglusterbugs: [Bug 889334] cluster.min-free-disk does not appear to work as advertised <http://goo.gl/eOt3c>
20:41 JoeJulian bronaugh: I see several mentions of help being a cli option throughout the Admin Guide.
20:42 bronaugh ok, that'd be my fault for not reading that. though I'm not sure where it's linked from.
20:43 bronaugh nonetheless, thanks for linking me to that. looks fairly cohesive / comprehensive.
20:46 JoeJulian From www.gluster.org click "Latest Docs" under "GlusterFS 3.3" there is the Admin Guide.
20:47 bronaugh blah.
20:47 bronaugh ok, PEBKAC.
20:47 JoeJulian hehe, no worries.
20:48 bronaugh it could be a wee bit more prominent.
20:48 bronaugh but I guess that's really the only complaint I can legitimately make.
20:56 andreask1 joined #gluster
20:58 tjikkun_work joined #gluster
20:59 bronaugh ok, so semi-random question.
21:00 bronaugh if you're running a replicated environment and a disk starts throwing bad data, is there a checksum to confirm which is the bad, and which is the good, copy?
21:05 bronaugh and another question.
21:05 bronaugh is there any way to snapshot glusterfs?
21:07 JoeJulian No to each. I handle snapshots with lvm.
21:08 ron-slc joined #gluster
21:10 bronaugh hmm
21:10 bronaugh so basically you handle it at an underlying layer.
21:12 bronaugh so can you create a read-only glusterfs volume on top of the snapshots trivially?
21:17 gbrand_ joined #gluster
21:21 lh joined #gluster
21:21 lh joined #gluster
21:38 bronaugh hm, ok, more qeustions.
21:38 bronaugh is it possible to force gluster to put files on specific bricks? ie: I want this tree to go on this brick, etc.
21:38 kkeithley1 left #gluster
21:39 bdperkin joined #gluster
21:40 bdperkin joined #gluster
21:48 johnmark semiosis: howdy
21:48 johnmark I recall somone saying they had an API for the perlgeek irclog foo
21:49 manik joined #gluster
21:51 mooperd joined #gluster
21:53 semiosis johnmark: heyo
21:53 semiosis bronaugh: that is a feature called "custom layouts" but it's not available yet
21:54 semiosis idk what the status is but it's come up several times
21:54 bronaugh ok.
21:54 semiosis last i heard the groundwork was set up in the core but it's not accessible through gluster CLI
21:54 johnmark semiosis: were you the one telling me about the API for getting info from perlgeek's irc logger?
21:54 semiosis johnmark: not me, maybe pdurbin?
21:55 johnmark semiosis: ah, good point. I'll ask him
21:57 lh joined #gluster
21:57 schmidmt_ left #gluster
22:03 Ryan_Lane joined #gluster
22:03 Ryan_Lane I have a volume that's supposed to be sharing nfs, but it isn't
22:03 Ryan_Lane it was previously
22:03 Ryan_Lane any steps to see why nfs isn't running?
22:03 Ryan_Lane am I supposed to be running any other services for this?
22:04 Ryan_Lane hm. one of the four is running it
22:11 andreask1 joined #gluster
22:12 Ryan_Lane hm. it seems that portmap and status are running on all the servers, but mountd nfs and nlockmgr is running on the one that's working
22:15 Ryan_Lane oh. I wonder if gluster is starting before portmapper service
22:16 Ryan_Lane seems to be the case. restarting glusterd solved the issue
22:19 JoeJulian Hey, Ryan_Lane, good catch. Is this rpm or deb based?
22:19 Ryan_Lane deb
22:19 Ryan_Lane I may be using an old upstart file
22:19 Ryan_Lane I have a custom one in puppet
22:20 Ryan_Lane I'll check out the newer ones
22:26 lh joined #gluster
22:26 lh joined #gluster
22:27 bronaugh ok yet more questions. I presume based on how glusterfs works that there are checksums stored for each file. am I correct in assuming this?
22:34 raven-np joined #gluster
22:35 Ryan_Lane if the upstart is broken in newer versions, I'll put a bug in :)
22:41 semiosis bronaugh: not correct.  glusterfs doesnt do any checksums for data at rest
22:42 hattenator joined #gluster
22:46 dbruhn Is there anyway to speed up a rebalance operation?
22:47 JoeJulian throw hardware at it?
22:48 JoeJulian But no.
22:48 dbruhn It's just been running way slower than I would expect
22:49 JoeJulian Omg... the Mayan jokes are flying like there's no tomorrow...
22:49 dbruhn like like 331GB over the course of 3 weeks from one node, and 272GB from the second and those are the first two of the 6 nodes
22:49 bronaugh semiosis: ok, how about at writeout?
22:49 bronaugh semiosis: ie: does an uptodate checksum follow files around?
22:49 semiosis bronaugh: idk what youre talking about
22:50 bronaugh ok. on zfs, each block is actually checksummed.
22:50 bronaugh now, I know glusterfs does not work at the block level (unless you're doing striping) so I know it doesn't do that.
22:50 bronaugh but, I also know that the DHT algorithm used to place objects -does- generate a hash of some variety.
22:50 bronaugh so the question becomes, is that hash stored somewhere?
22:52 semiosis bronaugh: it generates a "gfid" which is a unique id stored in the xattr of the file but that's not generated from the data in the file
22:52 JoeJulian No, the hash is only a hash of the filename (you can generate it yourself using the code on my blog).
22:52 semiosis bronaugh: what's your goal?
22:53 semiosis if you want to learn more about how glusterfs works, see ,,(extended attributes) article
22:53 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
22:53 bronaugh semiosis: goal is to know if a file on disk has become corrupted.
22:54 semiosis bronaugh: glusterfs does not provide that
22:54 bronaugh really, I think we can probably use the underlying filesystem (zfs) to solve this. but I'm curious anyways.
22:54 semiosis and imho should not
22:54 bronaugh you do replication. so how do you know which copy's valid if you have two copies?
22:54 semiosis read that article about xattrs
22:54 bronaugh there's two answers: quorem, using 3 or more copies, or checksums.
22:54 semiosis it explains it all very well
22:55 _br_ joined #gluster
22:55 semiosis actually there's several other ways in addition to those
22:55 bronaugh ok, reading.
22:56 tryggvil joined #gluster
22:57 _br_ joined #gluster
23:00 JoeJulian semiosis: Could you imagine the whining if self-heal checks included checksumming the file? :D
23:04 bronaugh if you're reading data out anyways, it's virtually free to compute a checksum.
23:04 bronaugh though blah.
23:04 bronaugh yeah you can only really do that real-time with blocks, not files.
23:04 bronaugh some pretty obvious problems with files.
23:05 bronaugh so yeah. pretty much impossible to do on gluster.
23:05 bronaugh guessing snapshotting is a maybe-3.5 feature.
23:06 semiosis you can snapshot with lvm today
23:06 bronaugh what guarantees does glusterfs make about point-in-time snapshots like that?
23:06 bronaugh basically, are they guaranteed to be consistent?
23:06 semiosis that's impossible
23:07 semiosis an application could always be in the middle of a "transaction"
23:07 bronaugh so then the snapshot you make from the underlying filesystem has no guarantees of snapshotting a consistent glusterfs state.
23:07 semiosis unless you stop your application from writing when its in a consistent state
23:07 bronaugh no, I'm talking about FS state, not file state.
23:08 bronaugh hmm, you know, maybe I'm just trying to solve a problem with the wrong tool here.
23:08 semiosis whats the difference?
23:09 bronaugh difference? a consistent filesystem state is a state in which the filesystem's metadata matches the data written to disk.
23:09 bronaugh consistency from an application perspective can mean all sorts of crazy things depending upon the application (ie: an app may write out first one file, then another, but the data in the two files is interdependent)
23:10 semiosis glusterfs has no state besides whats in the files
23:10 semiosis no centralized metadata to sync to disk
23:10 semiosis just xattrs on files
23:11 bronaugh right; and I guess the xattrs are infrequently changed anyways.
23:11 bronaugh based on what I've read so far.
23:11 JoeJulian As infrequently as the file is changed.
23:11 semiosis right
23:11 semiosis so to reiterate, there's app consistency, above glusterfs, and fs consistency, below glusterfs...
23:12 bronaugh semiosis: not quite, because there's still a window between when file data is written and when xattrs are written. that operation is not atomic.
23:12 semiosis did you read that article yet?
23:13 bronaugh yes; thought I don't see how it pertains to what I'm talking about here.
23:14 bronaugh metadata first, then data.
23:14 bronaugh (write out sequence)
23:14 bronaugh hmm.
23:15 bronaugh ok, so maybe I'm misunderstanding here.
23:15 semiosis we've all been there :)
23:16 JoeJulian The metadata is basically saying, "I have a change that didn't hit the other replica".
23:16 bronaugh JoeJulian: yeah in the case of AFR
23:16 JoeJulian That's the only case in which there's any metadata operations. *
23:17 bronaugh it would seem that with DHT the ranges would have to be updated.
23:17 JoeJulian That's only for a rebalance.
23:17 JoeJulian after adding or removing bricks
23:17 JoeJulian Doesn't affect the file state though.
23:17 bronaugh hmm
23:18 bronaugh so wait a minute, if you're doing striping, how does glusterfs know which brick to look for which block on?
23:18 semiosis don't do striping :)
23:18 semiosis ,,(stripe)
23:18 glusterbot Please see http://goo.gl/5ohqd about stripe volumes.
23:18 bronaugh well; yes :) but I'm asking.
23:18 bronaugh striping is asking for glusterfs to lose your shit.
23:18 JoeJulian For more info on Distributed Hash Table lookups, see http://www.joejulian.name/bl​og/dht-misses-are-expensive/
23:18 glusterbot <http://goo.gl/hO5LR> (at www.joejulian.name)
23:19 semiosis just guessing here... floor( byte offset / stripe size )
23:19 bronaugh yeah ok. fair enough. so again, no metadata update needed in that case.
23:19 semiosis just guessing here... floor( byte offset / stripe size ) % stripe count
23:19 JoeJulian Would make sense. All stripes start on the first brick of the stripe set.
23:20 JoeJulian ... and don't use stripe... ;)
23:20 bronaugh well yes. I already thought through just what a bad idea that is.
23:21 bronaugh I'm not entirely comfortable with glusterfs' way of distributing files either though :/ I'm not sure it's all that much better.
23:21 bronaugh you can have one brick go AWOL and have every dataset you own be partial.
23:21 bronaugh which is pretty much as bad.
23:21 semiosis distributed+replicated
23:21 JoeJulian But that would be true for any storage system with no replication.
23:21 semiosis that's the most popular type of volume
23:22 bronaugh JoeJulian: yes, which is why we use raid6/raidz2.
23:22 lh joined #gluster
23:22 lh joined #gluster
23:22 JoeJulian Not even in the same genre.
23:22 semiosis with glusterfs i have datacenter replication
23:23 semiosis granted they're connected by low latency metro lan, not wan, but still
23:23 semiosis much more resilient than raid*
23:24 bronaugh JoeJulian: the math is pretty similar, though. if you compose a volume of n bricks with distribution but not replication, your probability of data loss is the product of (1 - p(brick_fail[x]))
23:25 JoeJulian Comparing clustered filesystems to raid is like comparing nations to households.
23:25 bronaugh semiosis: yeah I've been thinking about doing replication on top of distribution as a replacement for the backup system that we currently use (because we're looking at $200/TB for backup, which is insane)
23:25 JoeJulian yikes
23:25 bronaugh yeah LTO-* is pretty retarded at this point :/
23:25 semiosis JoeJulian: good analogy
23:26 bronaugh drives at $12-20k/drive, tapes at $30/tape pretty much regardless of generation.
23:26 bronaugh plus frame space, plus plus plus.
23:26 bronaugh not our backup system; we're just using it. but no one wants to pay for it, so...
23:28 bronaugh so yeah. what I've been thinking about is either a) glusterfs with distribution but not replication on top of 12-18 disk raidz2 (read: raid6) volumes, with a separate batch of disks dedicated to backup (using rsync and snapshots), or b) using distribution + replication to avoid the backup problem entirely.
23:28 bronaugh the problem with b) of course is that you still have a single point of failure -- glusterfs.
23:28 bronaugh though hmm
23:28 bronaugh maybe it's not as bad as I am thinking it is.
23:30 bronaugh so basically the question becomes, with distribution and replication, what can go so terribly wrong as to eat data?
23:32 JoeJulian bitrot if the filesystem can't handle it, sounds like you're thinking zfs so that should be okay. Don't buy all one lot of hard drives (mix manufacturers) so you're not bit by some firmware bug that takes out 50 hard drives in the same week...
23:33 semiosis redundancy & snapshots serve two different purposes
23:33 bronaugh JoeJulian: done the whole "bit by a firmware bug" thing with ST31500341AS
23:33 semiosis i use both, redundancy for infrastrucutre issues, and snapshots for administrative issues
23:33 bronaugh that was a very unhappy month.
23:33 dbruhn left #gluster
23:33 JoeJulian That's the drive I was thinking of.
23:34 bronaugh we didn't lose any (significant) data.
23:34 bronaugh they just dropped out of the RAID array, which required some dark art to reassemble the array into somethnig sensibel.
23:35 bronaugh we did have I belive 1 500GB drive of the same vintage actually become inacesssible; but that's out of >30 drives.
23:35 bronaugh so no clustering of drives becoming inaccessible; just FLUSH CACHE taking >35s
23:37 bronaugh semiosis: regarding snapshots vs redundancy, I agree with your asssesment.
23:37 bronaugh semiosis: how do you handle pulling data out of snapshots?
23:38 semiosis bronaugh: well i'm in ec2 so my snapshots are actually provided by the infra.
23:38 semiosis bronaugh: to restore i just click a machine snapshot, click restore, and it boots up a new machine with the exact state of my target machine at the time the snapshot was taken
23:39 semiosis data & all
23:41 bronaugh right. but our use case here is, someone deleted a file. we want to be able to pull that file from a previous point-in-time snapshot.
23:41 bronaugh but obviously we don't want to pull -everything-
23:42 bronaugh so as far as I can see, one can specify that glusterfs use a set of snapshots as its filesystem in read-only mode, and then copy the files you want off that. but that does seem like a fair bit of admin work.
23:42 bronaugh just wondering if other people have done this and thus found a cleaner and/or easier way.
23:43 semiosis what do you mean by "specify that glusterfs use a set of snapshots as its filesystem in read-only mode"?
23:43 semiosis i've never heard of that... where do you get that idea?
23:43 bronaugh I should be using glusterfs terminology here; sorry.
23:44 bronaugh glusterfs uses a bunch of filesystems as its storage bricks. zfs allows you to mount point-in-time snapshots (read only)
23:44 semiosis you can simply mount the snapshot like a normal (ext/x/z)fs disk, fetch your file, and overwrite the bad version in the real glusterfs
23:44 semiosis don't restore into glusterfs' bricks
23:44 bronaugh ok, yeah. I get where you're going.
23:44 semiosis in fact, as a rule, never touch the bricks glusterfs has in use, only write through the client
23:44 bronaugh how can you figure out which brick a file resides on?
23:45 bronaugh (fwiw I wasn't suggesting overwriting files directly on a brick)
23:45 semiosis there's a virtual xattr for that
23:45 bronaugh ok. good to know.
23:45 semiosis i think you were suggesting replacing a brick beneath glusterfs?
23:45 semiosis which is usually not good
23:45 bronaugh never.
23:45 semiosis but tbh sometimes you do have to do it :)
23:45 semiosis @pathinfo
23:45 glusterbot semiosis: I do not know about 'pathinfo', but I do know about these similar topics: 'pasteinfo'
23:45 bronaugh no, I was suggesting mounting the snapshots of the bricks (read only) and then assembling a glusterfs filesystem out of the bricks.
23:45 semiosis hm
23:46 bronaugh a read-only one.
23:46 bronaugh so that one could then copy files transparently from that temporary, read-only filesystem to the present-time read-write glusterfs filesystem.
23:46 semiosis never tried that, but sounds like it would work
23:47 bronaugh oh, I'm sure it'll work. I'm just trying to figure out how to make it not be an inordinate pain in the ass :)
23:47 semiosis as long as your reads don't cause any modification, like atimes or whatever
23:48 _Scotty joined #gluster
23:51 bronaugh so can you mount glusterfs read-only?
23:51 bronaugh basic question, I know. but figured I should ask.
23:51 bronaugh and what does it do if its bricks are read-only?
23:53 Ryan_Lane bronaugh: can't mount gluster read-only
23:53 Ryan_Lane bronaugh: can export nfs, which can be shared read-only, though
23:58 bronaugh so that leads to the next question... which is, what does glusterfs do when the underlying filesystems are RO?
23:58 semiosis write errors to the brick logs
23:59 bronaugh ok, so long as it doesn't stop working or cause problems, I don't really care :
23:59 semiosis Ryan_Lane: are you sure about that?  i have glusterfs clients mounted read only
23:59 bronaugh :)
23:59 bronaugh just want to make sure I don't hit some nasty path in the code that's poorly tested and takes the server down.
23:59 semiosis but that's read-only clients... my bricks are r/w
23:59 Ryan_Lane semiosis: yes, but you can't share it read ony
23:59 Ryan_Lane *only
23:59 semiosis Ryan_Lane: true you can't force clients to be readonly
23:59 semiosis it's up to each client
23:59 Ryan_Lane right
23:59 Ryan_Lane which assumes you trust the clients ;)
23:59 bronaugh oh; so you -can- mount it RO

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary