Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-07

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
05:25 _ilbot joined #gluster
05:25 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
05:32 satheesh joined #gluster
05:34 hjmangalam1 joined #gluster
05:34 mohankumar joined #gluster
05:38 Guest23596 joined #gluster
05:39 mohankumar__ joined #gluster
05:43 ricky-ticky joined #gluster
05:48 badone joined #gluster
05:51 satheesh joined #gluster
06:07 lalatenduM joined #gluster
06:07 rcoup joined #gluster
06:15 shylesh joined #gluster
06:24 jtux joined #gluster
06:26 vimal joined #gluster
06:27 mooperd joined #gluster
06:30 rastar joined #gluster
06:31 Guest23596 joined #gluster
06:31 mohankumar__ joined #gluster
06:37 Kins joined #gluster
06:39 dobber_ joined #gluster
06:49 guigui3 joined #gluster
06:55 majeff joined #gluster
06:56 ekuric joined #gluster
07:04 raghu joined #gluster
07:08 ctria joined #gluster
07:08 puebele1 joined #gluster
07:09 shylesh_ joined #gluster
07:26 andreask joined #gluster
07:37 kke what happens if there is no space for link files?
07:38 kke [2013-06-07 10:12:24.285687] E [client3_1-fops.c:232:client3_1_mknod_cbk] 0-m1_data-client-4: remote operation failed: No space left on device
07:38 kke [2013-06-07 10:12:24.285781] W [dht-linkfile.c:66:dht_linkfile_create_cbk] 0-m1_data-dht: /attachments/2012/10/3/6/0/604c2f16-d8d2-46b6​-be4d-faf7ba3aa12d/1227602MAVEN03N01499.pdf: failed to create link file (No space left on device)
07:41 puebele joined #gluster
07:41 puebele left #gluster
07:43 manik1 joined #gluster
07:44 madd joined #gluster
07:50 ricky-ticky joined #gluster
07:54 ujjain joined #gluster
08:00 Norky joined #gluster
08:01 ollivera joined #gluster
08:14 mooperd joined #gluster
08:15 ndevos joined #gluster
08:24 satheesh joined #gluster
08:24 satheesh1 joined #gluster
08:38 thomaslee joined #gluster
08:40 ricky-ticky joined #gluster
08:47 kevein joined #gluster
08:47 rb2k joined #gluster
08:54 kelkoobenoitr joined #gluster
09:08 shylesh joined #gluster
09:14 shylesh joined #gluster
09:17 rastar1 joined #gluster
09:17 rb2k semiosis: The solution to my troubles yesterday was adding these params to the configure step: ./configure --enable-fusermount --prefix=/usr --sysconfdir=/etc --localstatedir=/var
09:24 duerF joined #gluster
09:55 ultrabizweb joined #gluster
10:09 ultrabizweb joined #gluster
10:15 goerk joined #gluster
10:17 glusterbot New news from newglusterbugs: [Bug 971796] gluster peer status shows two entries with same UUID for same machine on one of the peer in cluster <http://goo.gl/793QD>
10:46 rb2k semiosis: I take that back. It's still missing "libglusterfs.a"
10:46 lalatenduM joined #gluster
10:46 ndevos rb2k: what is missing that?
10:46 rb2k I'm trying to compile gluster and package it as a deb file
10:47 rb2k Using the debian folder by semosis
10:47 ndevos oh, I've got no experience with that, sorry
10:47 rb2k but I think I might be missing a configure switch :(
10:47 rb2k I'm not the best when it comes to compiling c code
10:57 rcoup joined #gluster
11:04 lpabon joined #gluster
11:11 DEac- joined #gluster
11:20 harish joined #gluster
11:23 kke how come layout fix on identical node has gone to 2.5 million already but completed on another at 1.5 million or so
11:27 koubas joined #gluster
11:27 vpshastry joined #gluster
11:48 charlescooke_ joined #gluster
12:01 edward1 joined #gluster
12:09 Chiku|dc joined #gluster
12:09 kelkoobenoitr left #gluster
12:09 Chiku|dc hi, I have 1 volume with 2replica and another volume with 2 replica both volume are geo-replication
12:10 Chiku|dc but geo is very very long
12:10 Chiku|dc any way to speed up geo-replication ?
12:15 kke does it make any difference which node i point my clients to? will the servers somehow balance the connections?
12:15 stickyboy kke: For connections, no.  But subsequent reads/writes, yah.
12:16 stickyboy As I understand it.
12:17 stickyboy ie, the `mount -t glusterfs server1:/homes` tells mount to get the volume informations from server1.
12:22 andreask joined #gluster
12:23 kke i wonder if this rdma thing would be any good
12:23 kke or could i get it working
12:23 StarBeast joined #gluster
12:23 stickyboy rdma is the transport for Infiniband, no?
12:23 bennyturns joined #gluster
12:23 kke maybe
12:24 kke better not mess stuff up, i'll use the tcp as before.
12:24 stickyboy kke: Are you using Inifinband hardware?
12:24 kke no, so i guess that leaves it out
12:24 stickyboy Yah, you're stuck with TCP :P
12:27 nueces joined #gluster
12:34 DEac- hi
12:34 glusterbot DEac-: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
12:37 jbourke joined #gluster
12:38 lalatenduM joined #gluster
12:38 DEac- i run glusterfs-client on ubuntu, so i use upstart. i read some documentations about upstart+glusterfs and i wrote an own job for mounting glusterfs-mountpoints. but my first mountpoint will never be mounted. it is possible, that glusterfsd is not immidiatly ready, if i start it? if i do a sleep 3 before mount, it works fine.
12:58 ricky-ticky joined #gluster
13:00 jdarcy joined #gluster
13:01 jdarcy Hey all.
13:01 stickyboy jdarcy: Morning
13:18 36DAASL14 joined #gluster
13:23 vpshastry joined #gluster
13:26 rob__ joined #gluster
13:26 GabrieleV joined #gluster
13:39 ThatGraemeGuy joined #gluster
13:40 RicardoSSP joined #gluster
13:40 RicardoSSP joined #gluster
13:46 manik joined #gluster
13:49 theron joined #gluster
13:50 bennyturns joined #gluster
13:53 chirino joined #gluster
13:54 jdarcy GlusterFS presentation at the Akron LUG - by a high-school freshman.  http://www.meetup.com/Akron-Lin​ux-Users-Group/photos/12428652/
13:54 glusterbot <http://goo.gl/XSTOV> (at www.meetup.com)
13:54 jdarcy That's just too awesome for words.
13:55 stickyboy High-school freshman?!
13:55 stickyboy whoa
13:56 * stickyboy tries to think what he was doing as a Freshman.
13:56 tqrst- joined #gluster
13:56 stickyboy Running a warez server on hotline. :P
13:56 stickyboy hahahaha
13:57 charlescooke_ left #gluster
13:58 kaptk2 joined #gluster
13:59 jdarcy In my case it was wilderness survival and rifle practice.  Really.  We won't talk about the year after that, though.
14:00 stickyboy jdarcy: :P
14:01 jdarcy Wow, trying to use any of these so-called Raft implementations makes me realize just how unready for public consumption so many open-source projects are.  Now I don't feel so bad about HekaFS.
14:04 tqrst- jdarcy: is your healer.py script from https://github.com/jdarcy/glusterfs/​tree/heal-script/extras/heal_script still good for 3.3.1? I've got a few hundred gazillion split brains to fix.
14:04 glusterbot <http://goo.gl/ZUIru> (at github.com)
14:05 failshell joined #gluster
14:08 jdarcy tqrst-: I'm confident that the xattrs it uses haven't changed their meanings in any significant way.  What I'm less sure about is the volfile-fetching and mounting part, but at least if that fails it fails early and safely.
14:09 tqrst- jdarcy: worth a shot then
14:09 jdarcy I thought I added a "dry run" mode so you could see what it *would* do without actually doing it.
14:09 jdarcy But I don't see that in the code.
14:10 failshell JoeJulian: are you around?
14:10 tqrst- (for some reason, one of my servers has started filling up its glustershd.log with a ton of split-brain errors (20-30/second), while the other servers are going along just fine. That's without any down time for that server afaik.)
14:11 jdarcy You could always stub out remove_dups, touch_file, and clear_xattrs to get an effective dry run, I guess.
14:11 jdarcy Have you manually checked the xattrs to see if there's really a problem?
14:12 jdarcy I should write a simple script just to do that and convert GlusterFS gobbledygook into a plain English description of what state a file's in.
14:13 jbourke left #gluster
14:13 tqrst- that's actually what I was going to do, because going through log files is a pain
14:13 tqrst- and 'gluster volume heal info' tends to segfault/fail here for reasons I don't have the time to investigate
14:13 MrNaviPacho joined #gluster
14:14 jdarcy I'll be honest, AFAICT "heal info" alternates between silent and spammy, never stopping at *useful* in between.
14:15 tqrst- yeah... the split brain list doesn't seem to keep up to date, and contains dupes etc
14:16 jdarcy I think I'll add a dry-run mode and test against 3.3.1 right now.
14:17 tqrst- heh, first gfid I checked has "too many levels of symbolic links"
14:21 pkoro joined #gluster
14:22 portante joined #gluster
14:28 vpshastry joined #gluster
14:30 mohankumar__ joined #gluster
14:34 bala joined #gluster
14:39 joelwallis joined #gluster
14:39 jbrooks joined #gluster
14:41 jdarcy Huh.  Switching between 3.3 and master runs into some interesting build issues.  I guess that's what we get for mooshing all of that UFO stuff into the GlusterFS build.
14:42 semiosis rb2k: remove the .a files from the install file.  see my 3.3 QA ,,(ppa) package's .debian.tar.gz for example
14:42 glusterbot rb2k: The official glusterfs 3.3 packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.3 QA: http://goo.gl/5fnXN -- and 3.4 QA: http://goo.gl/u33hy
14:42 rb2k semiosis: thanks, I did that :)
14:42 rb2k as a matter of fact, I had already done that before
14:43 rb2k but forgotten
14:43 rb2k and when updating to your current lucid .debian.gz it disappeared again :)
14:43 rb2k (the 3.3 one, not the qa)
14:43 semiosis lucid yuck
14:43 semiosis upgrade
14:43 semiosis precise is nice
14:44 hjmangalam1 joined #gluster
14:44 jdarcy What's after Quetzal?  Retarded Raptor?
14:45 tqrst- ridiculous reindeer
14:46 * jdarcy can't wait for Vengeful Vole
14:48 semiosis jdarcy: raring ringtail, released in april
14:48 kkeithley What will they do for X
14:48 semiosis next is saucy salamander
14:48 semiosis https://wiki.ubuntu.com/DevelopmentCodeNames
14:48 glusterbot Title: DevelopmentCodeNames - Ubuntu Wiki (at wiki.ubuntu.com)
14:48 semiosis the community proposes names, mark shuttleworth chooses
14:49 kkeithley Xanthocroic Xanthis?
14:49 * jdarcy suggests Treacherous Tanager
14:49 kkeithley ooh, Xenodochial Xanthis
14:49 tqrst- xenophobic xeme?
14:50 jdarcy Unoriginal Ungulate
14:50 kkeithley just rolls off the tongue
14:51 jdarcy OK, 3.3 built OK, now to check out the self-heal script.
14:58 bugs_ joined #gluster
15:01 plarsen joined #gluster
15:02 rosco joined #gluster
15:06 failshell im running 3.2. on my master cluster, in the slaves logs, im getting a split brain error concerning the remote (slave) volume.
15:06 failshell how do i fix that?
15:06 failshell and where?
15:06 failshell JoeJulian gave me a getfattr command to run, but im not sure where
15:06 failshell and that command returned thousands of lines
15:07 failshell getfattr -R -m trusted.afr -d -e hex /data/glusterfs/
15:08 jdarcy Oh, FFS.  That "system getspec" bug was in 3.3.1?  Grr.
15:09 semiosis failshell: log should indicate which file is split brain.  that's the file you want to get attributes of, i guess
15:09 failshell semiosis: says path: /
15:09 failshell [afr-common.c:1039:afr_launch_self_heal] 1-microsite-prod-backup-replicate-6: background  meta-data self-heal triggered. path: /
15:09 semiosis good thats easy to fix
15:10 failshell enlighten me :)
15:10 semiosis verify on the bricks that the brick dir itself has the same perms, times & owners on all bricks
15:10 semiosis i.e. if your bricks are server1:/path/to/brick & server2:/path/to/brick, then you're looking at those properties of /path/to/brick on both servers
15:11 failshell ok, but im getting that message on the master cluster, concerning the slave cluster volume, so im checking that on the slavE?
15:11 failshell or the master?
15:11 semiosis slave
15:11 Chiku|dc about geo-replication is it safe to use option use-rsync-xattrs ?
15:13 Chiku|dc because my geo is on high latence
15:13 failshell semiosis: im looking at the brick root and not the volume root?
15:13 semiosis i.e. if your bricks are server1:/path/to/brick & server2:/path/to/brick, then you're looking at those properties of /path/to/brick on both servers
15:17 failshell semiosis: they all have the same owner/group/perms
15:17 failshell time is in sync
15:17 semiosis now run the getfattr command (without -R, we dont want recursion) on those paths
15:17 semiosis pastie.org the results if you want
15:18 semiosis we expect the afr attrs to be non-zero
15:19 semiosis also, in the log, near the background meta-data self-heal triggered line, is there a line saying why self-heal failed?
15:19 jthorne joined #gluster
15:23 failshell the command didnt return any results
15:24 failshell https://gist.github.com/fai​lshell/edff95f7ca6c735e310b
15:24 glusterbot <http://goo.gl/u7xt9> (at gist.github.com)
15:24 failshell those are the 3 recurring errors on the master cluster
15:25 semiosis no xattrs???
15:25 failshell none
15:25 failshell getfattr -m trusted.afr -d -e hex /data/glusterfs/
15:25 semiosis try with "-m ."
15:25 semiosis as root
15:26 semiosis this has to be on the backend brick path on the server, not through a client mount
15:26 failshell it is
15:26 semiosis that's bizarre
15:26 failshell not getting anything
15:26 semiosis maybe impossible :0
15:26 failshell oh wait
15:27 semiosis ha
15:27 failshell https://gist.github.com/fai​lshell/70589324683cae11caa3
15:27 glusterbot <http://goo.gl/C24vv> (at gist.github.com)
15:27 failshell with the -m "."
15:27 failshell on some bricks
15:28 semiosis ,,(pasteinfo)
15:28 glusterbot Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
15:28 semiosis or add to that gist
15:30 jag3773 joined #gluster
15:30 failshell updated the last gist
15:33 semiosis failshell: your getfattr output says "file: data/glusterfs/" however that is not a brick path in any of your volumes (it is a parent of brick paths)
15:34 johnmark joined #gluster
15:34 failshell ah ok, now i have something
15:44 hjmangalam1 joined #gluster
15:48 MrNaviPa_ joined #gluster
16:03 hjmangalam joined #gluster
16:03 bstromski joined #gluster
16:08 tg2 kkeithley, any idea if it's possible to remove 2 bricks with the same remove-brick command?  I don't see anything documented about it being possible or not.
16:21 kkeithley gluster help seems to say that remove brick takes one or more bricks in the same remove-brick command
16:22 majeff joined #gluster
16:25 kkeithley And if you're doing replica X, then you need to remove X bricks at once.
16:25 tg2 well its a distributed setup, 2 bricks ont eh same server, I want to remove them both at the same time so that brick 1 doesn't rebalance to brick 2 which will have to then be rebalaned again after it's removed
16:25 tg2 I'll give it a try.
16:27 edward1 joined #gluster
16:31 neofob is there an example of block device usage that i could look at?
16:31 neofob gluster doc only describes the feature http://www.gluster.org/community/do​cumentation/index.php/Planning34/BD
16:31 glusterbot <http://goo.gl/LglqL> (at www.gluster.org)
16:33 tg2 I tried to load my volumes like this: http://pastie.org/pastes/8019896​/text?key=pnlgvklrw41hhzx3hzxzqa   in order to prevent clietns writing to storage02 bricks, it seemed to work, when creating new files, it put them on the non-read-only bricks, while reads worked from any node
16:33 glusterbot <http://goo.gl/s5XC0> (at pastie.org)
16:33 tg2 but, when trying to remove a folder, it fails saying that its read only, even though that folder exists only on the non-read only node
16:34 kkeithley the bd "device" is not a real block device; it's used by oVirt (RHEV-M) as a "pseudo block storage" device for VM guests.
16:37 neofob kkeithley: hah, thanks for the info
16:44 vpshastry joined #gluster
17:10 Mo_ joined #gluster
17:21 hjmangalam1 joined #gluster
17:22 jdarcy tqrst-: I updated the heal script to work on 3.3.1+
17:23 jdarcy tqrst-: You'll probably need to use the -f flag to get the volfile from /var/lib/glusterd... because "system getspec" is broken in 3.3.1
17:23 jdarcy tqrst-: I also added a dry-run flag so you can make sure it's doing the right thing before you actually change anything
17:24 balunasj joined #gluster
17:26 thomaslee joined #gluster
17:29 thomaslee joined #gluster
17:35 failshell semiosis: still around?
17:38 hjmangalam joined #gluster
17:42 plarsen joined #gluster
17:43 failshell ye i fixed my first split-brain
17:50 joelwallis joined #gluster
17:51 lpabon joined #gluster
17:51 shanks joined #gluster
17:56 zaitcev joined #gluster
18:08 sjoeboo_ joined #gluster
18:09 hjmangalam joined #gluster
18:12 balunasj joined #gluster
18:20 kke what kind of speeds do you get on copying stuff to gluster? i just set up a new volume and i'm copying data from the old bricks, seems to go at steady 6gb/hour
18:21 kke which i guess roughly translates into 1.5mb/s
18:21 kke it's a replicated volume
18:33 vpshastry left #gluster
18:37 failshell i copy around at 15-20mb/sec
18:37 failshell but then again, i run everything in a vmware environment
18:37 failshell not dedicated hardware
18:37 failshell so there's a lot of network traffic in the way
18:39 kke well these are 24 core servers with 10gbit connections between them
18:39 kke a lot of small files
18:39 kke and directories
18:42 failshell how are you copying? rsync?
18:48 kke yeah
18:49 kke from brick to gluster mount
18:49 kke old brick
18:49 failshell are you using --inplace?
18:51 kke yes
18:51 failshell then i dont know
18:54 kke same server read + write and replicate to secondary node over network
18:54 kke i guess that takes it's toll
19:01 wN joined #gluster
19:08 brunoleon_ joined #gluster
19:14 JoeJulian kke: You /might/ be able to get better throughput if you could have multiple threads each handling different directory.
19:16 kke rsync already seems to be running 3 processes
19:16 kke but yeah maybe i could give that a try for a moment
19:22 brunoleon_ joined #gluster
19:24 kke got 400gb to transfer
19:24 kke hmm i could have copied to the new volume's brick directly and run a self heal or something after that
19:43 hjmangalam1 joined #gluster
19:44 JoeJulian If you needed no down time you could have added the new brick(s) to the existing volume while increasing the replica count.
19:44 JoeJulian After the self-heal completed, you'd then remove-brick the old one(s)
19:45 kke scp copies 40mb/s in the same cluster without gluster
19:45 JoeJulian For that matter, add-brick without changing the replica count and let remove-brick handle the dht migration.
19:45 hjmangalam joined #gluster
19:45 kke the old one is busted somehow
19:45 kke that's why i'm setting up a whole new one
19:45 JoeJulian Comparing apples and orchards.
19:45 kke screwed up with fs selection
19:45 JoeJulian Ah
19:46 kke ext4 with too little inodes, ran out of them on several bricks, tried removing one brick which which didn't help at all, now applications are hanging infinitely while trying to access some files
19:47 JoeJulian Then, yeah, it's probably not too late to do that. Stop glusterd and glusterfsd, rsync the old brick to the new brick - preserving the xattrs.
19:47 JoeJulian Hanging indefinitely is probably more ,,(ext4) than running out of inodes.
19:47 glusterbot (#1) Read about the ext4 problem at http://goo.gl/xPEYQ or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
19:52 kke yes but inodes are 100% in use
19:52 kke getting no space left on device when trying to create dirs or stuff
20:04 hjmangalam1 joined #gluster
20:07 smellis joined #gluster
20:17 brunoleon_ joined #gluster
20:18 hjmangalam1 joined #gluster
20:53 semiosis failshell: been in a meeting most of the day, just back now.  glad to see you got it resolved by yourself.
20:56 failshell yeah thanks for your help
20:59 chirino joined #gluster
21:01 semiosis yw
21:04 failshell 1-microsite-prod-backup-stat-prefetch: invalid argument: inode
21:04 failshell what does that mean?
21:09 wN joined #gluster
21:11 hjmangalam1 joined #gluster
21:30 joelwallis joined #gluster
21:55 tg2 joined #gluster
22:16 RicardoSSP joined #gluster
22:39 war|child left #gluster
22:55 chirino joined #gluster
23:05 chirino joined #gluster
23:07 hagarth joined #gluster
23:47 StarBeast joined #gluster
23:59 vrturbo joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary