Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 yinyin joined #gluster
00:09 RicardoSSP joined #gluster
00:09 RicardoSSP joined #gluster
00:16 premera joined #gluster
00:34 yinyin joined #gluster
00:40 disarone joined #gluster
01:01 pmuller_ joined #gluster
01:01 jules_ joined #gluster
01:26 yinyin joined #gluster
01:35 NeatBasis joined #gluster
01:48 kevein joined #gluster
02:11 Chiku|dc joined #gluster
02:11 Chiku|dc joined #gluster
02:14 glusterbot New news from newglusterbugs: [Bug 924132] reports a 503 error when download a container <http://goo.gl/pDZ8M>
02:24 kshlm|AF1 joined #gluster
02:24 awheeler joined #gluster
02:26 mnaser joined #gluster
02:34 bala1 joined #gluster
02:37 jules_ joined #gluster
02:38 ultrabizweb joined #gluster
02:38 pmuller_ joined #gluster
02:38 Han joined #gluster
02:38 Guest77353 joined #gluster
02:39 ndevos joined #gluster
02:40 torbjorn1_ joined #gluster
02:43 awheeler_ joined #gluster
02:49 helloadam joined #gluster
02:56 badone joined #gluster
02:56 bharata joined #gluster
03:06 sgowda joined #gluster
03:14 glusterbot New news from newglusterbugs: [Bug 928575] Error Entry in the log when gluster volume heal on newly created volumes <http://goo.gl/KXsmD>
03:34 ricky-ticky joined #gluster
03:47 badone joined #gluster
03:51 bala1 joined #gluster
03:55 rastar joined #gluster
03:57 jag3773 joined #gluster
04:02 hagarth joined #gluster
04:14 raghu joined #gluster
04:21 yinyin joined #gluster
04:28 sripathi joined #gluster
04:38 bala1 joined #gluster
04:44 glusterbot New news from newglusterbugs: [Bug 927616] root-squash: root-squashing does not get disabled dynamically <http://goo.gl/tZW0X>
04:48 bala1 joined #gluster
04:51 yinyin joined #gluster
04:52 hagarth joined #gluster
04:56 aravindavk joined #gluster
04:58 nocko left #gluster
05:10 vpshastry joined #gluster
05:21 deepakcs joined #gluster
05:27 yinyin joined #gluster
05:27 rastar joined #gluster
05:38 test joined #gluster
05:39 saurabh joined #gluster
05:50 anmol joined #gluster
05:50 shylesh joined #gluster
05:56 vshankar joined #gluster
06:00 sgowda joined #gluster
06:03 mohankumar joined #gluster
06:04 lalatenduM joined #gluster
06:06 lala_ joined #gluster
06:31 sripathi1 joined #gluster
06:32 venkatesh joined #gluster
06:35 yinyin joined #gluster
06:36 ekuric joined #gluster
06:37 vimal joined #gluster
06:46 sripathi joined #gluster
06:54 sripathi joined #gluster
06:54 sgowda joined #gluster
07:03 ctria joined #gluster
07:09 sripathi joined #gluster
07:21 raghug joined #gluster
07:22 piotrektt joined #gluster
07:49 sripathi joined #gluster
07:52 shireesh joined #gluster
08:08 camel1cz joined #gluster
08:08 camel1cz left #gluster
08:09 ladd joined #gluster
08:17 ngoswami joined #gluster
08:26 ricky-ticky joined #gluster
08:29 ladd joined #gluster
08:35 camel1cz1 joined #gluster
08:37 sripathi1 joined #gluster
08:39 sripathi joined #gluster
08:47 MrAbaddon joined #gluster
08:48 ladd joined #gluster
09:05 anmol joined #gluster
09:09 camel1cz1 left #gluster
09:11 venkatesh joined #gluster
09:12 satheesh joined #gluster
09:29 vpshastry joined #gluster
09:31 brunoleon___ joined #gluster
09:36 deepakcs joined #gluster
09:48 bharata joined #gluster
09:53 dobber joined #gluster
09:55 dobber_ joined #gluster
10:01 venkatesh joined #gluster
10:06 raghug joined #gluster
10:23 spai joined #gluster
10:24 saurabh joined #gluster
10:36 saurabh joined #gluster
10:39 venkatesh joined #gluster
10:52 sripathi joined #gluster
10:56 ekuric joined #gluster
10:56 ekuric left #gluster
10:57 ekuric joined #gluster
10:57 camel1cz joined #gluster
10:57 camel1cz left #gluster
11:05 shireesh joined #gluster
11:13 rcheleguini joined #gluster
11:15 vpshastry joined #gluster
11:15 hagarth joined #gluster
11:32 venkatesh joined #gluster
11:40 sohoo joined #gluster
11:48 sohoo hello chanel, we have some issues with self-heal and rebooted servers. i was wondering if anyone here can help deal with this situations. for example files created on the remaining nodes replicates to the come back node with zero length then the clients get input/output when requesting that file. if you delete the file and its gfid then the file replicates back ok. so i was thinking to monitor
11:48 sohoo the logs then if file gets this then block the failed node with iptables rule then delete the file + gfid from the failed node then ls -lR on that file is that sounds ok or is there a better way?
11:49 sohoo the clients io errors when nodes reboot is real pain
11:50 sohoo i mean reboots then come back online if they say off then there is no io errors
11:52 hagarth joined #gluster
11:53 sohoo hope someone here can help with that, we realy like gluster its doing nice on some things but the maintaince is not claer
11:56 xiu /wg 8
11:58 yinyin_ joined #gluster
12:16 glusterbot New news from newglusterbugs: [Bug 895528] 3.4 Alpha Tracker <http://goo.gl/hZmy9> || [Bug 918917] 3.4 Beta1 Tracker <http://goo.gl/xL9yF>
12:22 hagarth joined #gluster
12:24 flrichar joined #gluster
12:42 vshankar joined #gluster
12:42 sohoo anybody here can explain the self-heal proccess a little more?
12:49 dustint joined #gluster
12:56 aliguori joined #gluster
12:59 robos joined #gluster
13:15 awheeler_ joined #gluster
13:16 bennyturns joined #gluster
13:20 raghug joined #gluster
13:30 lh joined #gluster
13:30 lh joined #gluster
13:34 duerF joined #gluster
13:38 yinyin_ joined #gluster
13:41 plarsen joined #gluster
14:00 vpshastry joined #gluster
14:03 rob__ joined #gluster
14:08 badone joined #gluster
14:12 jbrooks joined #gluster
14:15 bugs_ joined #gluster
14:19 lpabon joined #gluster
14:28 vpshastry1 joined #gluster
14:30 mohankumar joined #gluster
14:35 jskinner joined #gluster
14:37 vpshastry joined #gluster
14:39 rwheeler joined #gluster
14:40 rastar joined #gluster
14:41 xymox joined #gluster
14:47 bstromski joined #gluster
14:53 joshcarter joined #gluster
14:55 daMaestro joined #gluster
14:55 rbennacer joined #gluster
14:56 rbennacer hello, i am mounting glsuterfs through NFS and it is not showing all the available space when i do a "df -h", after umount it and remounting it , it fixes the problem. any idea how is this possible?
14:58 rosmo maybe your bricks are not in sync?
14:58 rbennacer what do you mean? remounting the volume sync them?
15:00 mweichert joined #gluster
15:00 mweichert hello! does gluster have pre/post hook support yet?
15:00 rosmo i mean... do your bricks have equal free space?
15:00 rbennacer no, some of them are full
15:01 rosmo my thought was that gluster might read the free space from a random brick and return that as free space
15:01 rosmo i mean bricks within same volume
15:02 rastar joined #gluster
15:03 mweichert I'd like to add a post hook on "gluster volume head", which would create a zfs snapshot on the underlying brick of each server
15:04 Scotch joined #gluster
15:05 shylesh joined #gluster
15:07 vpshastry joined #gluster
15:11 rastar joined #gluster
15:13 neofob left #gluster
15:15 Goatbert joined #gluster
15:16 glusterbot New news from resolvedglusterbugs: [Bug 832693] glusterfs-3.2.6-2 segfaults <http://goo.gl/Bgd6P>
15:17 rbennacer left #gluster
15:17 rastar1 joined #gluster
15:18 glusterbot New news from newglusterbugs: [Bug 852224] Crash in fuse_thread_proc with kernel-3.5.2-3.fc17.x86_64 <http://goo.gl/jOste> || [Bug 859248] Mount fails <http://goo.gl/8P2QB>
15:19 bdperkin joined #gluster
15:19 bdperkin joined #gluster
15:32 zaitcev joined #gluster
15:42 \_pol joined #gluster
15:44 hagarth joined #gluster
15:44 H__ joined #gluster
15:44 H__ joined #gluster
16:00 disarone joined #gluster
16:04 hagarth joined #gluster
16:12 \_pol joined #gluster
16:13 hagarth joined #gluster
16:27 rotbeard joined #gluster
16:45 ctria joined #gluster
16:45 Mo____ joined #gluster
16:47 awheeler_ mweichert: http://gluster.org/community/docum​entation/index.php/Features/Hooks
16:47 glusterbot <http://goo.gl/QsCKv> (at gluster.org)
16:48 jskinner_ joined #gluster
16:49 glusterbot New news from newglusterbugs: [Bug 902953] Clients return ENOTCONN or EINVAL after restarting brick servers in quick succession <http://goo.gl/YhZf5>
16:54 theron joined #gluster
16:55 mweichert awheeler_: thanks
16:56 awheeler_ mwiechert: Just tested on using the set command, and that works.  So seems like instead of using head, you could set a value and create a snapshot if the value says to do that.
16:58 awheeler_ So, for example, you could use: "gluster volume set <volume> cluster.min-free-disk 1" to take a snapshot
16:58 awheeler_ And maybe: "gluster volume set <volume> cluster.min-free-disk 0" to delete it.
16:59 awheeler_ mweichert: The passed parameters look like: --volname=<volume> -o cluster.min-free-disk=0
17:01 lalatenduM joined #gluster
17:05 portante joined #gluster
17:06 sohoo im asking all day hoping someone will have short answare :) is it normal for clients to get io errors on files in situation when server is rebooting(in replicated/distrebuted) ?
17:06 sohoo rebooting and comeing back online that is
17:07 lpabon joined #gluster
17:10 soukihei joined #gluster
17:15 NuxRo sohoo: for how long do you get i/o errors? is it permanent or just temporary until the brick comes back online?
17:16 jskinner joined #gluster
17:22 bala joined #gluster
17:22 mooperd joined #gluster
17:29 jskinner_ joined #gluster
17:48 samppah awheeler, mweichert: that's cool
17:49 badone joined #gluster
17:50 sohoo when the bricks is offline thus server is offline clients dont get errors, the errors come only when it comes back online for files that was writen to node 1 when node2 was offline
17:52 sohoo its contsant at list for half hour when you cant allow this to continue and delete the entire brcik on that server to initial full sync then it replicates and there is no errors on clients
17:54 sohoo i understand that the heal includes probe the file for attributes then if need a heal take it from the good server and write it back to server 2 but what if the file doesnt exists on node 2 at all is it still get replicated?
17:57 sohoo its a 10 nodes cluster with replication pairs so it happen quit a lot
18:04 lpabon joined #gluster
18:05 andreask joined #gluster
18:08 _pol joined #gluster
18:09 _pol joined #gluster
18:35 jdarcy joined #gluster
18:48 jclift joined #gluster
18:57 andreask joined #gluster
19:14 rwheeler joined #gluster
19:30 sohoo realy strange that nobody can help on ThatGraemeGuy, makes you wonder if its all about moeny this days. The developers keep this to themself this is quit disapoiting
19:30 xymox joined #gluster
19:31 jclift sohoo: Or maybe that it's a public holiday in both the US and UK today, so many of them are doing Easter things with their families?
19:31 sohoo just to reiamnd, the situation happen on replicated nodes when 1 restart and come back clients see io errors :)
19:32 rosmo sohoo: my understaing is that this stuff is pretty hard to understand, you ought to pay for consulting if it's that's important to you
19:33 rosmo truth is, i have diven (dived? what's the past tense of dive) into glusterfs with just general idea of what's possible and not, and i've been bitten by some stuff
19:33 sohoo :) hard to understand? a storage need to be handeled by sysadmins not developers, if so please write it on your website no?
19:34 sohoo so i said, its strange open source project. Normaly the comunity helpes on all issues here there is no comunity just moeny :)
19:35 jclift rosmo: "I dived" is generally the right past tense there. :)
19:35 rosmo developers are mostly retarded when it comes to deployment, that's why we (hopefully) have devops guys
19:35 twx_ usually the ppls here are very helpful
19:35 rosmo joshcarter: i have dived? that sounds just not right...
19:36 rosmo jclift: you even :)
19:36 jclift Kill the "have"
19:36 rosmo there's like 20 guys with "jo" prefix ;)
19:36 jclift "I dived into ..."
19:36 sohoo thx rosmo i may check up later after the holidays, it looks like the hart of the product and strange that it has no answare
19:36 rosmo ohhh... no it makes sense
19:36 rosmo s/no/now/
19:36 glusterbot What rosmo meant to say was: joshcarter: i have dived? that sounds just nowt right...
19:36 jclift sohoo: Which part of "its a public holiday and people are with their families today" are you missing?
19:37 rosmo glusterbot saves the day, eh?
19:37 sohoo i got it :)
19:37 jclift sohoo: So, the response today is not like your average day, etc.
19:37 sohoo we have holiday here as well
19:37 sohoo :)
19:38 rosmo who here ACTUALLY has a day off for the holidays? there's bound to be some guys that have been drafted to monitor stuff
19:38 rosmo i have a "day off" and i'm been slashing my wrists trying to sync spare files from a brick to another
19:39 rosmo protip: whatever you read on the 'nets, it doesn't work
19:39 sohoo rosmo, i know thanks for the explnations.
19:39 rosmo (sparse files, that is, not spare files)
19:40 sohoo null files?
19:40 rosmo nope
19:40 sohoo ok got it images
19:40 sohoo KVM etc..
19:40 rosmo i've got some ovirt virtual machine images that are "thin provision" (eg. sparse files)
19:40 sohoo yes, got it
19:41 rosmo neither rsync --sparse nor rsync --inplace work (even when i have pre-created the files using truncate before)
19:41 sohoo in my situations it not, just normal small files(not that small) but not gb in size
19:41 rosmo so i'm trying tar cfvS and netcat right now
19:41 sohoo it will be mistake to use gluster to host cloud images now
19:41 rosmo if that doesn't work, i'm writing my own sync stuff....
19:42 rosmo this has to with gluster that i have a single brick replica and i want to shut down everything, sync and and add another replica to the set
19:42 jclift rosmo: For general copying of sparse images around (on CentOS/OSX/Fedora) systems, I just using rsync -aPS [source] [destination]
19:42 sohoo my big questune is if 1 node of replication pair reboots, meanwhile file writen to node 1 then 2 come back online does the self heal deal with that?
19:43 jclift rosmo: That definitely works with sparse files.  I haven't tried it in combination with gluster though.
19:43 sohoo on my tests it doesnt
19:43 rosmo jclift: that's the problem, i tried multiple times
19:43 jclift rosmo: Damn
19:43 rosmo jclift: it always looks like it's finishing the copy and they suddenly a sparse file that ought to be 8.4G is 16G
19:43 jclift sohoo: In theory it should.  Do the log files say anything useful about the self healing daemon not being able to connect to the right ports or something?
19:44 rosmo also i know they fixed some sparse file stuff in rsync 3.0.8 and i've upgraded both ends to 3.09
19:44 jclift rosmo: Just a sec, so the rsync isn't working, or gluster is screwing it up somehow?
19:44 rosmo jclift: just rsync
19:44 rosmo jclift: i'm rsyncing sparse files directly from brick to brick
19:44 rosmo jclift: (although the second one isn't a brick yet)
19:45 sohoo the servers log is not in debuging mode and the client logs has io errors only when node 2 is back onlime and doesnt have the new files yet
19:45 jclift rosmo: On a CentOS 6.4 box on the net, it's using rsync 3.0.6.  That definitely works to other CentOS boxes with sparse files.  (done it many times)
19:45 rosmo jclift: i've also tried cp -a --sparse=always through nfs but it just ain't working, so i'm trying tar now
19:45 jclift On my OSX 10.7 desktop, it has rsync 2.6.9, and that definitely works between itself and the CentOS versions. (also done many times)
19:46 rosmo jclift: it does seem to work for me a bit (eg. the file stays at the right size for quite a bit and they suddenly it's the actual size of the file)
19:46 jclift rosmo: Which OS are you using?
19:46 rosmo jclift: rhel 6.4 on one end with upgraded rsync and fc16 on one end
19:46 rosmo both are rsync 3.0.9
19:47 jclift sohoo: Is there any chance the self-healing daemon isn't able to connect to the other host?
19:47 jclift sohoo: ie firewall rules blocking it
19:47 ferrel joined #gluster
19:47 sohoo the both working the connection also fine
19:47 sohoo :) no firewall
19:47 jclift rosmo: Hmmm... that's weird.  I wonder if it's an xfs+rsync bug thing
19:47 _pol joined #gluster
19:47 sohoo we do extended monitoring
19:48 jclift sohoo: Yeah, that was my next suggestion.
19:48 sohoo any helpfull tips?
19:48 jclift sohoo: I have no idea how to help you more though.  The stuff I'm working on with gluster hasn't touched self healing in the slightest yet, so zero idea. :(
19:48 rosmo jclift: could be, since i've tried everything... i'm just waiting for the tar results... next time if it's doesn't work out, i'm just xfs_dump
19:49 jclift rosmo: As a way to find out, do you have an ext4 or something filesystem around (/ maybe?) that you could create a sparse file on, and try the rsync from that?
19:49 sohoo ok thnx, hope someone will later know. i realy dont want to dive into this code this days
19:49 jclift rosmo: That might at least help narrow down the problem.
19:49 sohoo for a simple thing as an answare :)
19:50 rosmo jclift: i'm using xfs all round
19:50 jclift sohoo: Yeah, I know the feeling.  Also, email the gluster-users mailing list, just in case someone does come back online and is able to check, etc, when you're not around.
19:50 jclift rosmo: Was a thought. :)
19:51 rosmo jclift: it wouldn't be a problem unless this specific filesystem was our ovirt master data...
19:51 sohoo tnx
19:51 jclift rosmo: Hopefully tar works. :)
19:52 rosmo jclift: hopefully yes, but if it doesn't, i can always write my own "just fseek over zeroes" sync software
20:14 noche joined #gluster
20:15 jbrooks joined #gluster
20:16 awheeler_ How do I know when a heal has completed?
20:17 awheeler_ When "gluster volume heal info" returns no files?  or...?
20:21 rosmo awheeler_: i guess yes
20:22 semiosis i need to file a bug
20:22 glusterbot http://goo.gl/UUuCq
20:23 awheeler_ rosmo: They seem to be growing.  Both for info, and info heal-failed
20:24 rosmo awheeler_: i guess gluster keeps seing files that aren't healed?
20:25 jclift awheeler_: Co-incidentally I'm looking through the Gluster 3.3 admin guide, and it shows a bunch of options that can be set on volumes (section 7.1, pg 32+).
20:26 jclift awheeler_: Things like "diagnostics.brick-log-level".
20:26 awheeler_ Dunno.  This is 3.3.1-11, and I'm using the heal command, rather than the find info command.
20:26 awheeler_ jclift: Have to check that out
20:26 jclift awheeler_: Some of these options sound useful.  There's even option to change the way self-heal works.  i.e. full file copy vs diff
20:26 jclift options
20:29 awheeler_ So what does it mean when a heal failed?
20:30 awheeler_ for some files that is
20:30 jclift That sounds less than good.
20:30 jclift awheeler_: Hmmm, pg 47 has info about healing failure.
20:30 awheeler_ always happens to me -- of course, I am hitting the volumes via swift the whole time.
20:31 awheeler_ Su, but not what it means.
20:31 jclift awheeler_: http://www.gluster.org/community/​documentation/index.php/Main_Page   "Admin Guide" entry under the "GlusterFS 3.3" heading.
20:31 glusterbot <http://goo.gl/eAVvs> (at www.gluster.org)
20:31 awheeler_ The most palatable explanation is that the file has been deleted, and so can't be healed.
20:31 awheeler_ Looking at that right now.
20:32 awheeler_ Just says self heal failed, not whether that's a temporary, or permanent issue, or really whether I shoudl be concerned.
20:32 jclift Yeah, not a bunch of useful stuff there for "when shit happens".
20:33 jclift Troubleshooting section, pg 113.
20:33 awheeler_ And, actually, the text is wrong, it's not info failed, but info heal-failed
20:34 jclift awheeler_: If you file a BZ for that, the docs guys will fix it. :)
20:34 awheeler_ So, in my case, I'm pretty sure that the heal failed files were deleted before they could be healed.
20:34 awheeler_ Be nice if that's the case if the text was obsolete, rather than failed.
20:35 jclift awheeler_: Interesting idea, and probably worthwhile too.
20:35 jclift awheeler_: Please file a BZ for that as well? :D
20:38 awheeler_ I was hoping this was already seen and going to be fixed in 3.4.  :)
20:39 awheeler_ Is there any way to get a list of the status of all of the options?
20:39 semiosis gluster volume set help
20:39 semiosis all but the undocumented ones :)
20:40 awheeler_ That's useful, but doesn't list their current status, just the default -- or is that the same?
20:50 glusterbot New news from newglusterbugs: [Bug 947226] CLI option --xml doesn't have any effect on debian/ubuntu <http://goo.gl/jEc9n>
20:52 jdarcy joined #gluster
20:53 awheeler_ Well, looks like heal might be finally catching up -- could be the loaded disk subsystem I'm using.
21:23 jskinner_ joined #gluster
21:33 ferrel left #gluster
21:44 plarsen joined #gluster
21:46 Han joined #gluster
21:55 masterzen joined #gluster
22:01 ninkotech_ joined #gluster
22:38 semiosis awheeler_: ah current status of options is given by 'gluster volume info' -- any option thats been changed, even if it's been changed back to default, will be listed with its current value
22:39 semiosis options that have never been modified are not listed
22:44 plarsen joined #gluster
22:48 robo joined #gluster
23:09 redsolar_office joined #gluster
23:15 Chiku|dc joined #gluster
23:15 Chiku|dc joined #gluster
23:17 duerF joined #gluster
23:18 Rydekull_ joined #gluster
23:21 sonne joined #gluster
23:23 nhm joined #gluster
23:24 Gugge joined #gluster
23:27 georgeh|workstat joined #gluster
23:32 GLHMarmot joined #gluster
23:38 Gugge_ joined #gluster
23:41 badone joined #gluster
23:42 andrewbogott joined #gluster
23:47 georgeh|workstat joined #gluster
23:49 jbrooks joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary