Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-24

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 inodb joined #gluster
00:13 inodb joined #gluster
00:28 inodb_ joined #gluster
00:38 TSM2 joined #gluster
01:00 tryggvil joined #gluster
01:16 hchiramm_ joined #gluster
01:34 hchiramm_ joined #gluster
01:54 hchiramm_ joined #gluster
02:29 atrius joined #gluster
02:43 pull joined #gluster
03:01 quillo joined #gluster
04:51 bala joined #gluster
05:01 Daxxial_ joined #gluster
05:52 Humble joined #gluster
06:17 Humble joined #gluster
06:44 daMaestro joined #gluster
07:03 Humble joined #gluster
07:33 quillo joined #gluster
07:55 Nr18 joined #gluster
07:56 ekuric joined #gluster
08:10 sunus joined #gluster
08:16 sunus joined #gluster
08:19 mohankumar joined #gluster
08:39 yinyin joined #gluster
09:23 sunus joined #gluster
09:24 inodb joined #gluster
09:25 Humble joined #gluster
10:03 gbrand_ joined #gluster
11:12 mohankumar joined #gluster
11:27 Humble joined #gluster
11:47 Nr18 joined #gluster
12:03 sunus joined #gluster
12:06 Humble joined #gluster
12:28 Humble joined #gluster
12:35 yinyin joined #gluster
13:03 Jippi joined #gluster
13:06 lh joined #gluster
13:06 lh joined #gluster
13:16 sunus joined #gluster
13:16 Humble joined #gluster
13:32 gbrand_ joined #gluster
13:45 Humble joined #gluster
14:38 sunus joined #gluster
14:38 Nr18 joined #gluster
14:47 Humble joined #gluster
15:09 Azrael808 joined #gluster
15:16 Humble joined #gluster
15:29 bulde joined #gluster
15:44 khushildep joined #gluster
15:48 Humble joined #gluster
15:53 TSM2 joined #gluster
16:06 sunus joined #gluster
16:14 mohankumar joined #gluster
16:41 Humble joined #gluster
16:48 badone_ joined #gluster
16:56 gbrand_ joined #gluster
17:04 inodb joined #gluster
17:37 bala joined #gluster
17:38 rudimeyer_ joined #gluster
17:39 inodb_ joined #gluster
17:41 sac__ joined #gluster
18:07 sac_ joined #gluster
18:27 daMaestro joined #gluster
18:33 Bullardo joined #gluster
18:40 gbrand__ joined #gluster
18:59 edward1 joined #gluster
19:00 Hymie joined #gluster
19:00 Hymie I'm wondering if someone here recalls info about glusterfs 2.0.7?
19:00 daMaestro what's up?
19:00 Hymie specifically, I have a few replicated volumes
19:01 Hymie and I want to add a new server in
19:01 Hymie I can't upgrade things until I get the new server in
19:01 Hymie because it was neglected for so long that there's no failout capability (there's not enough power for me to swap out existing servers for upgrades)
19:01 daMaestro upgrading from 2.x -> 3.x is ..... ummm... yeah i don't personally recommend it
19:01 Hymie daMaestro: that's another important thing, thanks for that info
19:01 daMaestro even from 3.0 to 3.1 was ... yeah
19:02 Hymie daMaestro: once I get this extra server in, and, I can then swap servers out -- I can install a test server and move through the upgrae process
19:02 Hymie daMaestro: but, do you know if it is easy/hard to add a new server to a replicated volume?
19:02 Hymie I have 3 servers
19:02 Hymie I want to add a 4th
19:02 daMaestro you will have more success building a new pool with the latest code and copying over data
19:02 Hymie can I just edit all the fluster.vol files, and be done?
19:02 Hymie I can't do that :/
19:03 daMaestro i had a lot of issues going from 2.x -> 3.x
19:03 Hymie at worse, I may have to kill a glusterfs and manually rsync updates until I can swap all servers out
19:03 Hymie I don't want to go from 2.x to 3.x
19:03 Hymie I want to add a 2.0.7 replicated volume
19:03 Hymie I have three 2.0.7 servers
19:03 Hymie I have a forth I've cloned
19:03 daMaestro additionally, up until 3.1 rpc calls were not compatible
19:03 daMaestro ah, so you don't even want to upgrade?
19:03 Hymie I just need to know -- can I add another 2.0.7 server
19:03 Hymie not yet
19:03 Hymie as I said, I can't
19:04 daMaestro yeah, you can add another server
19:04 Hymie I need to get another server in there, so I can do swap outs and testing on upgrades
19:04 Hymie I've looked using wayback, mailing lists, there seems to be no docs on it
19:04 Hymie or, it's MIA now
19:04 Hymie so, I just edit glusterfs.vol, add the new server on all servers,and -- what?
19:04 Hymie it just slowly replicates to the new volume?
19:04 daMaestro can you pastebin your volume definitions?
19:04 Hymie sec
19:05 daMaestro also, what clients are accessing the cluster? fuse?
19:05 daMaestro (iirc in 2.x fuse was the only option)
19:05 bulde joined #gluster
19:05 Hymie http://pastebin.ca/2254944
19:05 glusterbot Title: pastebin - Miscellany - post number 2254944 (at pastebin.ca)
19:05 Hymie sorry, have 4 volumes, want to add a 5th
19:05 Hymie my mistake
19:06 Hymie sec
19:06 Hymie d coming
19:06 andreask joined #gluster
19:06 daMaestro okay, that is a very simple translator layout
19:06 daMaestro i assume that all bricks are the same size?
19:06 Hymie http://pastebin.ca/2254945
19:06 glusterbot Title: pastebin - Miscellany - post number 2254945 (at pastebin.ca)
19:06 Hymie yeah
19:06 daMaestro or at least have enough capacity to hold the total contents?
19:07 Hymie yes, for sure
19:08 daMaestro okay, so what you want to do is a few things
19:08 daMaestro i assume you want to prevent *any* impact on the clients?
19:08 Hymie thanks man -- you are the last remaining docs :D
19:08 Hymie yes, preferably... we don't really have any slack in use
19:08 daMaestro okay, so stand up the 5th server just like the others
19:09 daMaestro get a client configured with the extra server added and the brick added to the replicate and on that client do a ls -al on the mount
19:09 daMaestro that should force that client to do the healing to the additional server
19:10 daMaestro the other clients should not know about the new server until you've finished at least one ls -alR on the temp client
19:10 daMaestro basically, it's the clients responsibility in 2.x to make sure the bricks are correctly in sync
19:10 Hymie hmm, ok
19:10 Hymie ok, I see
19:10 daMaestro if you inform all of your clients about that, they will also take the time to replicate
19:10 daMaestro if your clients have the throughput... then tell them all
19:10 daMaestro but i've always used a side client that is non-customer facing in 2.x
19:11 daMaestro it is my understanding that now server side this can be done, but that is not until 3.2+
19:11 Hymie daMaestro: I have that
19:11 Hymie one server that isn't really customer facing
19:11 Hymie it gets updated, but not really very often
19:11 Hymie it's more for us for maintenance of the filesystem
19:12 daMaestro well so you mean to say you can use that as a "client" in the case of gluster right?
19:12 daMaestro how much data are we talking about? how many files?
19:13 Hymie # find /srv/clusterfs/uploads/ -type f | wc -l
19:13 Hymie 254975
19:13 Hymie small though
19:14 Hymie all very small
19:14 daMaestro df -h?
19:14 Hymie jpgs, html, php, etc
19:14 Hymie # du -sh /srv/clusterfs/uploads/
19:14 Hymie 80G     /srv/clusterfs/uploads/
19:14 daMaestro i assume all your servers are at least 1ge?
19:14 Hymie /etc/glusterfs/glusterfs.vol
19:14 Hymie 99G    93G   2.6G  98% /mnt/clusterfs
19:14 Hymie yeah
19:15 Hymie gigbit and proud since 2007 :P
19:15 Hymie gigabit even
19:15 daMaestro so are you adding another server to get more space? because that is not gonna happen with your current plan...
19:15 Hymie no, it's just because all the web servers are using glusterfs to server certain files
19:15 Hymie and I just need to add another
19:15 Hymie +1 web server
19:15 Hymie then, after, I can do upgrades and tests, prior to upgrading everything
19:16 Hymie right now I can't swap any servers out, the load is too high
19:16 Hymie due to neglect
19:16 Hymie if I had a failure, right now?  I'm not redundant
19:16 daMaestro well from a glusterfs perspective you are
19:16 daMaestro but i don't know what else you have going on
19:16 Hymie just raw requests / second
19:16 Hymie I don't have any more ram to add more apache workers
19:17 Hymie I can't even swap a server out to add ram
19:17 daMaestro yea you can add another gluster server into the load as i described
19:17 Hymie because I'm not redundant :/
19:17 daMaestro if you are worried about load, do not use the customer facing clients to do the initial replication
19:17 Hymie ok, thanks.. I'll give this a try.. although I wish there was a dev env here to try it on first
19:17 daMaestro you will notice a performance hit
19:17 Hymie hmm, ok
19:18 daMaestro well so you are not even telling the current clients about what you are doing
19:18 Hymie so I can use my "control" server, that isn't outward facing, to replicate
19:18 daMaestro use a different client
19:18 daMaestro yeah
19:18 Hymie btw, thanks for this.. one other thing I was wondering.... I initially cloned this entire directory tree when rsynfcing over the entire old server to the new
19:18 daMaestro once you get through one ls -alR (which should be relatively quick for the amount of files we are talking about here) you can inform the rest of the clients
19:19 Hymie so, I'm taking from what you've said, that I should wipe out the existing /srv/glusterfs dir, and just start with it empty?
19:19 daMaestro it depends on how you "cloned" it
19:19 Hymie rsync
19:19 daMaestro yeah, let me dig up the command i need you to run
19:20 daMaestro getfattr -dm - /path/to/a/file
19:20 Hymie man, management aren't idiots, but... they think you can just upgrade in 5 minutes ;P  I have 500 customers hosting on these servers.. imagine their pain if I don't get them a test server, to test their code on, when I do a massive OS upgrade of the cluster
19:20 Hymie damnit, I already eat too much pizza
19:20 daMaestro i would not recommend that, at all
19:21 Hymie I know I'm getting fatter ;P
19:21 daMaestro i would recommend a fresh cluster with the latest code
19:21 daMaestro or you are just asking for a headache
19:21 Hymie oh, that comes eventually
19:21 daMaestro additionally, due to the rpc incompatibilities, you are talking downtime anyways
19:21 Hymie but, first, before I do that, I have to get people up on the newest PHP version
19:22 Hymie I can freeze uploads of new hosting files, and rotate in new servers as I upgrade them
19:22 Hymie but for that to work, I need to slowly get to the point tat I can do that entire OS upgrade
19:22 Hymie then I can just rsync stuff over
19:22 Hymie on each fresh install
19:22 Hymie # getfattr -dm - /srv/clusterfs/uploads/ads/img/trans.gif
19:22 Hymie hmm
19:22 Hymie no output
19:22 Hymie # getfattr -dm - /srv/clusterfs/uploads/ads/img/trans.gif
19:22 Hymie getfattr: Removing leading '/' from absolute path names
19:22 Hymie # file: srv/clusterfs/uploads/ads/img/trans.gif
19:22 Hymie trusted.afr.nu=0sAAAAAAAAAAAAAAAA
19:22 Hymie trusted.afr.omicron=0sAAAAAAAAAAAAAAAA
19:22 Hymie trusted.afr.rho=0sAAAAAAAAAAAAAAAA
19:23 Hymie trusted.afr.xi=0sAAAAAAAAAAAAAAAA
19:23 Hymie that's on other servers
19:23 Hymie so, I guess rsync wasn't good enough
19:24 daMaestro yeah, so the rsync doesn't copy the xattr data that *is* how glusterfs works
19:24 Hymie interesting, that alone is quite helpful to me, thanks ;)
19:24 daMaestro however, it's possible that gluster will do the right thing and just look at what is on disk and then add the xattr data needed
19:25 daMaestro so then you don't have to wait for the fuse client to move the bulk of the data around.. just update xattrs
19:25 Hymie if I can just sync it over from my control server, there won't be any impact on my other clients, so I should be OK then I'd think
19:25 Hymie I'm not really IO bound here btw
19:25 Hymie it's all ram to the servers, for apache instances
19:25 Hymie very low CPU or IO load
19:25 daMaestro leave the data there, use the control server to do the ls -alR with the client configured with the 5th server and then let's check if the metadata is correct
19:25 daMaestro and actually... just do the ls -al /srv/clusterfs/uploads/ads/img/trans.gif
19:25 daMaestro that will trigger a heal
19:25 Hymie ok, so, sorry to go over this, but this is prod, and it's hard for me to even play
19:26 daMaestro then we can confirm that with getfattr
19:26 Hymie so, right now
19:26 Hymie I have a new server
19:26 Hymie it's a clone of an old server
19:26 Hymie I'm going to create a new config file
19:26 Bullardo joined #gluster
19:26 Hymie the config file will only have my "control" server in it
19:26 Hymie and, the "control" server will have this new server added
19:26 Hymie and all other servers remain the same, I don'ty cange their config?
19:27 Hymie (I'm assuming this is how I can isolate the first sync process)
19:27 daMaestro 1) don't change any of the current clients or servers
19:27 daMaestro 2) add your 5th server, configured just like your other servers
19:28 daMaestro 3) use a new client, whatever that is, with an updated config that knows about the new server
19:28 daMaestro 4) mount the glusterfs
19:28 daMaestro 5) ls -al /srv/clusterfs/uploads/ads/img/trans.gif
19:28 Hymie so, all my servers are identical right now.. all have the same configs, all have servers and clients configured
19:28 daMaestro 6) confirm on the 5th server it did what it's supposed to: getfattr -dm - /srv/clusterfs/uploads/ads/img/trans.gif
19:28 Hymie so in this case
19:29 daMaestro 7) assuming things are good: ls -alR /srv/clusterfs on the temp client
19:29 daMaestro 8) update all of your current client configurations to know about the new server
19:29 daMaestro and you are done
19:30 Hymie so, since I cloned an existing server
19:30 Hymie that means, according to the above
19:30 Hymie that I'd just edit /etc/gluster/glusterfs.vol, and add the new server there?
19:30 Hymie then mount and do my stuff?
19:31 daMaestro that sounds correct, yes
19:31 Hymie hmm....
19:31 daMaestro i'm now seeing that your gluster servers are also clients
19:31 Hymie I guess the glusterfsd.vol doesn't really do anything but server then (I'm absorbing how this older stuff works at the same time here)
19:31 daMaestro which is fine
19:31 Hymie er, but serve I mean, then
19:31 daMaestro you'll have to update glusterfsd.vol with the correct IP, but that should be it
19:32 daMaestro and yes, glusterfsd only serves the bricks
19:32 Hymie option auth.addr.brick.allow 192.168.1.*
19:32 Hymie that's the only IP I have, so I'm good to go I think
19:32 Hymie (in d.conf)
19:32 daMaestro in 2.x servers don't know about eachother, unless you configure them to)
19:32 daMaestro they are just dumb filers
19:32 Hymie going to try this now... will be back in 10... would you be able to hang around a few?
19:33 daMaestro yeah, i'll be here
19:33 Hymie thanks man
19:36 Hymie hmmmm
19:36 Hymie doesn't seem to be healing
19:37 Hymie maybe I should just go with an empty dir?
19:37 Hymie how can I force it not to replicate changes back to other clients/servers
19:38 Hymie hmm
19:38 * Hymie umounts
19:40 Hymie ok, moved it
19:40 Hymie hmm, that did heal
19:40 Hymie so, it had to be empty
19:41 Hymie who knows if the behaviour you're thinking of was 2.0.8 or something ;)
19:41 Hymie 20 years ago ;)
19:41 daMaestro i was not really confident it would heal correctly
19:41 daMaestro gluster *tries* to be safe in that regard
19:41 Hymie safe sounds like a good plan
19:41 * Hymie isn't doing a -R ls yet
19:42 Hymie just one dir at a time
19:42 Hymie just to see
19:42 daMaestro well do one file.
19:42 Hymie oh, I did
19:42 Hymie it copied
19:42 Hymie it was empty, now I have a directory tree, etc
19:42 daMaestro what does the getattr look like?
19:42 Hymie yeah, works
19:42 Hymie and, has the new server in the attr info too
19:43 Hymie so, what happens to the attr info on the other servers, as I add the new servers in?
19:43 Hymie er
19:43 Hymie new server
19:43 Hymie does it just "know" and add the attr info?
19:43 daMaestro show me both please
19:43 daMaestro only the new one should see the afr data for the new server
19:43 daMaestro that is what i want to confirm
19:43 Hymie interesting
19:44 Hymie on another server I checked, it has the attr data for the file for the new server
19:44 daMaestro ah yeah, that makes sense
19:44 Hymie I guess when it copies it, it says "ok, this has this now"
19:44 daMaestro correct
19:44 Hymie confirmed on two servers
19:44 Hymie old servers
19:45 * Hymie does some more work
19:45 Hymie thanks man, again -- I googled for days, no one directly spoke of this, the old docs didn't seem complete, wayback didn't help, etc ;)
19:45 daMaestro the temp client makes sure all servers it knows about has the correct xattr afr ... which is how glusterfs stores "this is on servers x, y and z"
19:46 daMaestro it is also how it knows where it needs to do a heal
19:46 Hymie find /srv/clusterfs/ -type f > /tmp/ooo.txt
19:46 Hymie on the new client, I'm going to cat the file into ls with a delay of 1 second per file
19:46 Hymie I don't care how many hours it takes
19:46 Hymie or maybe .1 seconds
19:46 Hymie whatever ;)
19:47 Hymie sleep .2 should be good enough
19:48 Hymie then tomorrow I can look again ;)
19:51 daMaestro Hymie, there is no need for that
19:52 daMaestro the new client is not in the load, so you can go full on ls -alR
19:52 daMaestro that should be relatively quick and then you can update the configs on all your other clients and drop the server into the load
19:52 daMaestro copying that many files and that much storage should be quick and non-impacting to existing clients
19:53 daMaestro as long as you are not i/o bound, you will be fine
19:54 daMaestro the only risk you have is the extra on-disk inode space required for the additional xattr data, but that should already be allocated and it does not look like extended inodes will need to be used for the amount of data being added
19:54 daMaestro so that is a very low risk
19:54 Hymie ok, thanks
19:55 copec joined #gluster
19:55 Hymie so, one other thing -- is 2.0.7 fairly good at auto healing and such.  I take a server out, reboot it, etc... halt it for a day, then bring it back up
19:56 Hymie I really only have to worry about load on this, but healing should be fine I guess?
19:56 Hymie ls -laR /mnt/clusterfs/
19:56 Hymie been sitting there for 3 minutes now
19:57 Hymie I guess it won't return until the whole tree exists?
19:59 Hymie yeah, I see df usage climbing
19:59 Hymie ok, thanks again man
19:59 Hymie I'll poke around at this over the next day or two ;)
20:00 Hymie (seriously man, I was stressing out, since I couldn't find anything doc wise, and there's a time limit on this too...)
20:00 Hymie heh
20:00 Hymie as if there isn't always, but still ;)
20:00 Hymie thnaks
20:04 Hymie ah, there some files show ;)
20:05 daMaestro 2.x is okay at self healing
20:05 daMaestro but it is the duty of a client to notice the issue and then update all bricks
20:06 daMaestro as you can see, i/o hangs before a self-heal is complete on a given request
20:06 daMaestro so you will have some read performance hits when a self-heal is needed
20:06 daMaestro but with small files, it's no big deal
20:06 daMaestro this has been improved considerably in newer versions
20:07 daMaestro as long as you don't end up in a split brain situation, glusterfs usually Does the Right Thing (tm)
20:07 daMaestro lol
20:08 daMaestro i ran into a lot of issues when doing a replicate on top of a distribute
20:08 Hymie well, my next goal is to read the current documentation, in prep for my OS upgrades
20:09 daMaestro 3.3 is faaar different then 2.x
20:09 Hymie because, "replicate" seems logical, but I'll have to read up a lot on the other roles
20:09 Hymie well, heh
20:09 Hymie I was looking for all these gluster commands that didn't exist on my installs :D
20:09 daMaestro start with a new cluster too... don't even try to upgrade
20:09 Hymie then I realised "damn, this must be old"
20:10 daMaestro even if it's a one node cluster to start, do it
20:10 Hymie oh I will, you've scared me enough ;)
20:10 Hymie I'll upgrade servers, and use rsync for a day or two until I can get all the old cluster / servers rotated out to the new OS / cluster
20:11 Hymie but that's a month away at least :/
20:11 daMaestro 3.3.x looks really sexy and the people working on it are smart and dedicated to the project
20:12 Hymie I imagine a lot of the additions / enhancements are to make sure it works well when spawning new VMs all over the place
20:13 daMaestro there is a lot of work in a lot of places.
20:14 Hymie ok, I'm going to go and try to watch a movie with lots of explosions to celebrate ;)
20:14 daMaestro i still have a bitter taste in my mouth.. but i also moved a cluster of over 300TB from 1.0.2 all the way to 3.0.x
20:14 daMaestro err 1.0.6+patches*
20:14 Hymie heh, that's a cluster ;)
20:15 daMaestro we were using xfs as the underlying FS and had some non-recoverable corruption that really screwed us because of the was distribute works
20:15 daMaestro so we got into a round hole square peg situation
20:16 daMaestro so there was a lot of coding on our side to get data back
20:16 Hymie bleh
20:16 Hymie sounds fun yet unpleasant
20:16 Hymie if you know what I mean
20:16 daMaestro it is also why i'm recommending a fresh cluster with the latest code base
20:16 daMaestro and not upgrades ;-)
20:17 Bullardo joined #gluster
20:18 daMaestro but to ease some concern, 3.2+ is going to be upgrade compatible going forward
20:18 daMaestro which is great.
20:18 Hymie yeah, I can't take down tonnes of machines every upgrade
20:18 Hymie so that is awesome
20:18 Hymie ok, bbl!
20:18 daMaestro and it directly intregrates into libvirt
20:23 daMaestro integrates*
20:30 khushildep joined #gluster
20:53 primusinterpares joined #gluster
20:54 gm__ joined #gluster
21:12 rudimeyer joined #gluster
21:17 Bullardo joined #gluster
21:33 hagarth joined #gluster
21:37 JoeJulian Hymie: Your load could be too high because you're doing replica 4. Replica 5 won't help that any. See my blog about http://www.joejulian.name/bl​og/dht-misses-are-expensive/
21:37 glusterbot <http://goo.gl/hO5LR> (at www.joejulian.name)
21:37 JoeJulian ~php | Hymie
21:37 glusterbot Hymie: php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
21:39 JoeJulian If this is for static content, with 2.0.7 if you turn self-heal checks off for your web clients, then that replica 4 or 5 might be okay, depending on the type of load.
21:49 JoeJulian Hymie: In 2.0.7 to disable self-heal, edit the replicate block like so: https://gist.github.com/4141516
21:49 glusterbot Title: Replicate block with self-heal disabled Gist (at gist.github.com)
21:50 TSM2 joined #gluster
21:50 JoeJulian Hymie: Of course, leave at least one client without those options and run a find on a cron job on that client to trigger a self-heal check with some regularity.
22:14 Hymie JoeJulian: thanks, but I specifically said my load wasn't high.  Good info to be aware of though.
22:15 * Hymie saves the disable info, who knows what the future holds
22:17 Hymie this is all temp too, I'll be at 3.something in a month or two
22:17 daMaestro joined #gluster
22:21 wN joined #gluster
22:30 Hymie JoeJulian: probably why I have no issues -- there's no PHP in here, only shared uploads and images and such
23:04 JoeJulian Eh, no biggie. I based that statement on "right now I can't swap any servers out, the load is too high"
23:04 JoeJulian ... but I see that you meant there was too much traffic to be able to take a server out of operation.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary