Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-06

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 hagarth joined #gluster
00:05 andrei joined #gluster
00:12 H__ joined #gluster
00:15 yinyin_ joined #gluster
00:16 edward2 joined #gluster
00:23 hagarth joined #gluster
00:27 nueces joined #gluster
00:28 yinyin joined #gluster
00:34 bala joined #gluster
01:01 yinyin joined #gluster
01:19 kevein joined #gluster
01:25 lpabon joined #gluster
01:34 yinyin joined #gluster
01:39 hagarth joined #gluster
01:42 lh joined #gluster
01:58 stopbit joined #gluster
02:00 yinyin joined #gluster
02:06 lng joined #gluster
02:13 lng Hi! I have Distributed Replicated Volume, but now I'd like to remove replica nodes and make the volume as Distributed Volume. How do I do it safe way?
02:14 lng Is it okay just to remove bricks and stop the nodes?
02:18 misuzu http://gluster.org/community/documentation​/index.php/Gluster_3.1:_Shrinking_Volumes
02:18 glusterbot <http://goo.gl/4LyxY> (at gluster.org)
02:20 jag3773 joined #gluster
02:20 _pol joined #gluster
02:20 lng misuzu: Naruhodo
02:37 misuzu '-')b
02:40 hagarth joined #gluster
02:59 bharata joined #gluster
03:04 Humble_away joined #gluster
03:08 pipopopo_ joined #gluster
03:13 rastar joined #gluster
03:16 sgowda joined #gluster
03:19 vshankar joined #gluster
03:29 shylesh joined #gluster
03:38 sgowda joined #gluster
03:42 anmol joined #gluster
03:51 en0x left #gluster
04:14 raghu joined #gluster
04:18 timothy joined #gluster
04:27 bulde joined #gluster
04:29 yinyin joined #gluster
04:36 bulde1 joined #gluster
04:42 vpshastry joined #gluster
04:43 lh joined #gluster
04:47 sripathi joined #gluster
04:51 lala joined #gluster
04:52 test joined #gluster
04:58 vpshastry joined #gluster
04:58 bharata joined #gluster
04:59 yinyin joined #gluster
05:06 satheesh joined #gluster
05:11 aravindavk joined #gluster
05:15 bala joined #gluster
05:17 phase5 joined #gluster
05:18 mohankumar joined #gluster
05:25 bala joined #gluster
05:28 yinyin joined #gluster
05:33 sripathi joined #gluster
05:36 shireesh joined #gluster
05:37 sahina joined #gluster
05:43 timothy joined #gluster
06:05 test_ joined #gluster
06:05 _pol joined #gluster
06:07 sripathi1 joined #gluster
06:08 sripathi joined #gluster
06:10 mooperd joined #gluster
06:13 phase5 joined #gluster
06:23 satheesh joined #gluster
06:25 ramkrsna joined #gluster
06:30 phase5 left #gluster
06:35 ngoswami joined #gluster
07:02 yinyin joined #gluster
07:03 sripathi1 joined #gluster
07:07 Nevan joined #gluster
07:09 sripathi joined #gluster
07:16 vimal joined #gluster
07:17 rgustafs joined #gluster
07:18 Humble joined #gluster
07:20 jtux joined #gluster
07:28 ThatGraemeGuy joined #gluster
07:44 guigui1 joined #gluster
07:49 Humble joined #gluster
07:59 ctria joined #gluster
08:01 jtux joined #gluster
08:16 abyss^_ it is possible to expand volumes on the same server? For example I have working gluster and I add new disk and I wanted to expand size...? On documentation is only about adding new servers...
08:22 sripathi1 joined #gluster
08:23 Nagilum abyss^_: I've only started yesterday, but a brick is a brick, whether it's local or remote
08:27 vshankar joined #gluster
08:29 glusterbot New news from newglusterbugs: [Bug 826958] geo-replication.indexing cannot be reset <http://goo.gl/hfn0u>
08:38 sgowda joined #gluster
08:40 sripathi joined #gluster
08:44 abyss^_ Nagilum: so you suggested if I do smth like this: gluster volume add-brick exist_volume server1:/new_location server2:/new_location that should work (server1 and server2 is a servers where the gluster exist now)
08:44 Nagilum yep
08:45 abyss^_ ok. Thank you, it's logical, but I'd like to be sure - yes I am going to check it one way or another:)
08:50 Staples84 joined #gluster
08:54 rotbeard joined #gluster
08:58 yinyin joined #gluster
08:58 inodb joined #gluster
08:59 glusterbot New news from newglusterbugs: [Bug 918437] timestamps updated after self-heal following primary brick rebuild <http://goo.gl/1QRyw>
09:00 Norky joined #gluster
09:03 puebele joined #gluster
09:06 tryggvil joined #gluster
09:12 mooperd joined #gluster
09:20 sgowda joined #gluster
09:22 puebele joined #gluster
09:24 dobber_ joined #gluster
09:32 tryggvil joined #gluster
09:41 deepakcs joined #gluster
09:45 tryggvil joined #gluster
09:47 mooperd joined #gluster
09:57 tryggvil joined #gluster
10:07 gbrand_ joined #gluster
10:11 mooperd joined #gluster
10:16 qwerty123 left #gluster
10:24 tryggvil joined #gluster
10:31 _benoit_ joined #gluster
10:40 tryggvil joined #gluster
11:00 tryggvil joined #gluster
11:01 sahina joined #gluster
11:01 ctria joined #gluster
11:03 shireesh joined #gluster
11:14 tryggvil joined #gluster
11:15 duerF joined #gluster
11:16 andrei joined #gluster
11:28 tryggvil joined #gluster
11:33 yinyin joined #gluster
11:40 tryggvil_ joined #gluster
11:47 vpshastry joined #gluster
11:50 vpshastry left #gluster
11:50 tryggvil joined #gluster
11:51 vpshastry joined #gluster
11:53 ctria joined #gluster
11:54 glusterbot New news from resolvedglusterbugs: [Bug 823304] [1d939fe7adef651b90bb5c4cd5843768417f0138]: geo-replication status goes to faulty state due to corrupted timestamp <http://goo.gl/Qk3am>
11:56 vpshastry1 joined #gluster
12:01 tryggvil_ joined #gluster
12:05 shireesh joined #gluster
12:07 vpshastry joined #gluster
12:13 tryggvil joined #gluster
12:22 vex joined #gluster
12:23 tryggvil joined #gluster
12:34 flrichar joined #gluster
12:37 tryggvil joined #gluster
12:40 vpshastry1 joined #gluster
12:49 bulde joined #gluster
12:49 andrei joined #gluster
12:53 dustint joined #gluster
12:56 rgustafs joined #gluster
13:01 yinyin joined #gluster
13:03 bala1 joined #gluster
13:17 nueces joined #gluster
13:18 timothy joined #gluster
13:19 shireesh joined #gluster
13:19 vpshastry joined #gluster
13:26 stopbit joined #gluster
13:28 bennyturns joined #gluster
13:31 puebele joined #gluster
13:48 vpshastry1 joined #gluster
13:51 puebele joined #gluster
13:53 shylesh joined #gluster
13:56 tryggvil joined #gluster
13:58 lala joined #gluster
13:58 balunasj joined #gluster
14:06 vpshastry joined #gluster
14:17 edward1 joined #gluster
14:23 rwheeler joined #gluster
14:26 lh joined #gluster
14:26 lh joined #gluster
14:31 shireesh joined #gluster
14:39 bennyturns joined #gluster
14:54 vpshastry joined #gluster
14:56 lpabon joined #gluster
14:58 mattr01 Hey when I try to mount a glustervolume remotely this is the error I get .. Does anyone know off hand what this could be ? E [dht-common.c:1372:dht_lookup] 0-virtdir-dht: Failed to get hashed subvol for
14:58 mattr01 the volume name is virtdir
15:00 mattr01 when I mount it locally it works .. I have a open network to the system
15:03 guigui1 joined #gluster
15:03 theron joined #gluster
15:04 fleducquede what is the command you run ?
15:04 mattr01 mount -t glusterfs <ipofhost>:/virtdir /mnt/glusterfs
15:05 fleducquede it seems fine
15:05 mattr01 But when I try to mount it locally it works
15:06 mattr01 just not when I try to have a remote client mount it
15:06 fleducquede your client fulfill the requirements ? fuse, glusterfs package installeD?
15:06 mattr01 grrr i dont have server installed
15:07 mattr01 fuse is installed
15:07 mattr01 installed server same problem
15:07 bugs_ joined #gluster
15:08 fleducquede what version of glusterfs do u use?
15:09 balunasj joined #gluster
15:09 mattr01 3.3.1-10.fc18
15:10 mattr01 the remote host is 3.3.1-1.el6
15:11 fleducquede output of gluster volume info ?
15:13 mattr01 on the remote or server?
15:13 fleducquede on the server
15:15 mattr01 where is that?
15:16 fleducquede type "gluster volume info"
15:16 fleducquede send me the output
15:16 mattr01 http://pastebin.com/3jsurhRf
15:16 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
15:18 mattr01 Same as last .. just doing what the bot says http://fpaste.org/tk1b/
15:18 glusterbot Title: Viewing Paste #282713 (at fpaste.org)
15:19 fleducquede what package have u installed on the client?
15:19 fleducquede (or on the remote machine).
15:19 mattr01 glusterfs 3.3.1-10.fc18 fuse and server
15:20 mattr01 just the heck of it I started glusterd and tried
15:20 mattr01 did work
15:20 mattr01 didntwork
15:20 mattr01 sorry
15:21 fleducquede grep " E " /var/log/glusterfs/* on both client and server
15:21 fleducquede in order to check all the logs on both sides
15:22 mattr01 okay trying now
15:24 mattr01 Server : http://fpaste.org/d6Pb/
15:24 glusterbot Title: Viewing Paste #282715 (at fpaste.org)
15:24 elyograg I posed a question on the mailing list but haven't gotten a fix yet.  Latest message in the thread: http://supercolony.gluster.org/pipermai​l/gluster-users/2013-March/035632.html
15:24 glusterbot <http://goo.gl/cvR9L> (at supercolony.gluster.org)
15:24 shylesh joined #gluster
15:24 Staples84 joined #gluster
15:24 elyograg it's about S3 support in the latest glusterfs-swift packages.
15:24 mattr01 Client(remote) http://fpaste.org/eUDu/
15:24 glusterbot Title: Viewing Paste #282716 (at fpaste.org)
15:25 fleducquede gimme 5 min
15:26 mattr01 fleducquede: thanks
15:31 lala joined #gluster
15:34 mattr01 I figured the problem out fleducquede
15:34 mattr01 I had to add the search domain for the hosts
15:34 mattr01 fleducquede: thanks for looking into for me
15:41 mattr01 the remote server was using a different DNS server .. so when I added the search domain for the servers it works .. seems gluster always returns the names or ips configured
15:41 Gilbs joined #gluster
15:47 timothy joined #gluster
15:49 tqrst anyone else experiencing memory leaks in 3.3.1's rebalance? (http://gluster.org/pipermail/glus​ter-users/2013-March/035636.html)
15:49 glusterbot <http://goo.gl/tEXYe> (at gluster.org)
15:50 tqrst (not necessarily a leak per se, I guess, but very high memory usage that climbs by ~50-80M/hour)
15:51 tqrst JoeJulian: I wonder how many of those 2991 error messages are seen in practice, and whether the seen subset overlaps between users :)
15:52 fleducquede good to know it's working now mattr01
15:52 fleducquede :)
15:56 guigui joined #gluster
15:57 bitsweat joined #gluster
16:02 daMaestro joined #gluster
16:05 Gilbs Is there a way to adjust the replica in a distributed replicated volume?  Can I do this prior to a brick removal or with the brick removal command?
16:09 satheesh joined #gluster
16:25 semiosis Gilbs: 'gluster volume remove-brick help' gives... Usage: volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... {start|stop|status|commit|force}
16:25 Gilbs semiosis:  thanks!
16:26 semiosis i just converted a 2-brick replica 2 volume to a 1-brick volume with that command
16:28 Gilbs I need to cut down a 4 replica to 2 and move over to geo-replication on the other servers.  Had a long night thinking I could not do this.
16:30 Gilbs I tried the help but on my side i'm only getting:  Usage: volume remove-brick <VOLNAME> <BRICK> ...
16:30 semiosis that feature was added in 3.3 :/
16:30 semiosis brb
16:31 Gilbs I see... I thought i was on 3.3...  ok, update time.
16:34 clag_ joined #gluster
16:34 Gilbs I see the problem:  W: Failed to fetch http://ppa.launchpad.net/semiosis/ubuntu-gluster​fs-3.3/ubuntu/dists/oneiric/main/source/Sources  404  Not Found
16:34 glusterbot <http://goo.gl/Y860n> (at ppa.launchpad.net)
16:43 phase5 joined #gluster
16:44 nemish joined #gluster
16:47 shylesh_ joined #gluster
16:48 semiosis Gilbs: yeah there's no package for oneiric.  i'll upload one but it will probably take a few hours for launchpad to build it
16:49 _br_ joined #gluster
16:52 _br_ joined #gluster
16:53 timothy joined #gluster
17:01 lala joined #gluster
17:01 Gilbs np, i'll just upgrade to 12.04
17:01 semiosis awesome
17:01 Gilbs Lab scenerio, i goofed and installed oneiric.
17:01 semiosis cancelling these builds
17:02 _br_ joined #gluster
17:02 Gilbs But i'll be back with geo-rep questions :)
17:42 Mo___ joined #gluster
17:43 cjohnston_work left #gluster
17:43 cjohnston_work joined #gluster
17:47 luckybambu joined #gluster
17:47 luckybambu Anyone have experience with georep across ~2mil file Gluster? I've got about 15gb worth of stuff that's not syncing
17:47 cjohnston_work luckybambu I have only tried a few files so far
17:48 cjohnston_work couple 100 large ones
17:48 luckybambu hmm
17:48 luckybambu I've got roughly 2 million files synced
17:48 luckybambu But it's just short a few
17:48 luckybambu Tried rebuilding the index etc but it won't sync the damn things
17:56 cjohnston_work I am not sure on the limitations there
17:57 luckybambu I don't know if there are limitations, I just suspect I'm making it suicide
17:57 disarone joined #gluster
18:01 vpshastry joined #gluster
18:16 y4m4 joined #gluster
18:17 balunasj joined #gluster
18:18 H__ mattr01: I do a peer probe of the other machines on all nodes to get to names
18:19 y4m4 joined #gluster
18:24 mattr01 H__: thanks, my problem was rooted with it returned the DNS names as "det-jboql01 and det-jboql02" but the client would only resolve them names as det-jboql01.hosts.domain.com
18:24 mattr01 do I added hosts.domain.com to my search domain in /etc/resolve
18:33 eightyeight i know that glusterfs uses "eventual consistency". can someone explain that in layman's terms? i'm having a hard time understanding it, versus "strong consistency", and the advantages/disadvantages to both
18:34 eightyeight it seems to be that "eventual consistency" is asynchronous replication, whereas "strong consistency" is synchronous
18:34 semiosis geo-replication is asynchronous
18:35 eightyeight but, from what i'm reading online, the "eventual" aspect is slower
18:35 semiosis regular replication is synchronous
18:35 eightyeight so what makes glusterfs eventually consistent?
18:35 semiosis who says it is?
18:36 eightyeight a number presentations i've been to- parallels summit, scale 11x. pdfs, papers, etc.
18:38 semiosis interesting
18:38 _pol joined #gluster
18:38 semiosis we should ask those authors/presenters why they call glusterfs eventually consistent
18:39 eightyeight http://www.parallels.com/summit/2013/agenda/. click on "Technical". the PDF of "Evaluating cloud storage strategies: Guidelines for hosters" mentions it, as an example
18:39 _pol joined #gluster
18:39 semiosis if you consider regular glusterfs replication, and rule out split-brain scenarios, then glusterfs is continuously/strongly consistent, afaict
18:40 eightyeight it does say, however that for eventual consistency, "Gluster does have a much slower strong consistency quorum enforcement mode"
18:40 eightyeight mentions Sheepdog, CEPH and pStorage as examples of strongly consistent
18:40 eightyeight anyway
18:42 semiosis imho, this pdf is wrong
18:42 semiosis http://www.parallels.com/fileadmin/parallels/doc​uments/Summit_presentations/JamesBottomley_Evalu​ating_Cloud_Storage_Strategies_20413-Final.pdf
18:42 glusterbot <http://goo.gl/OW7hc> (at www.parallels.com)
18:43 semiosis as far as its points about glusterfs anyway
18:43 semiosis would love to get some other opinions though
18:43 semiosis @later tell jdarcy any thoughts on this? http://www.parallels.com/fileadmin/parallels/doc​uments/Summit_presentations/JamesBottomley_Evalu​ating_Cloud_Storage_Strategies_20413-Final.pdf
18:43 glusterbot semiosis: The operation succeeded.
18:43 tjstansell parallels is also trying to position their own cloud storage platform...
18:43 semiosis oh really?!?!
18:44 tjstansell yeah. which is part of the point of that first document.  "we're going to help fix fuse for everyone" ...
18:44 eightyeight yes
18:44 eightyeight of course
18:44 eightyeight but it's not the only one i've seen where gluster was referred to as eventually consistent
18:44 eightyeight at the scale11x conferenece, the glusterfs presenter himself referred to it as such as well
18:44 eightyeight as did sage weil for his ceph presentaiton
18:45 tjstansell not sure what they're using or if they started from scratch.  i was at that conference and couldn't get any info on that...
18:45 eightyeight so, i was curious where that is coming from, and what i'm missing
18:45 semiosis eightyeight: in short, geo-rep is asynchronous aka eventually consistent, but regular glusterfs replication (AFR) is strongly/continuously consistent
18:45 semiosis as far as i understand it anyway
18:45 Alknelt joined #gluster
18:45 eightyeight semiosis: ok. that is what i was thinking, and thus why i wanted a clear definition between the two, because i was clearly missing it
18:46 semiosis who was the gluster presenter at stale11x?
18:46 eightyeight it was...
18:46 semiosis nm, found it
18:46 eightyeight http://www.socallinuxexpo.org/scale1​1x/presentations/glusterfs-sysadmins
18:46 glusterbot <http://goo.gl/cFdO7> (at www.socallinuxexpo.org)
18:46 eightyeight yeah
18:47 Gilbs Hooray!  Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y   Remove Brick commit force successful
18:47 semiosis Gilbs: nice
18:47 eightyeight someone asked a question about the replication, and the presenter mentioned it in his answer. i don't think it was in his slides
18:47 Alknelt Hi everyone. I'm new to gluster, and have recently (yesterday) and created a 70TB 2 disk striped filesystem with 48 bricks and am having some problems. I can't delete a directory. The client is running the glusertfs client…
18:48 Gilbs semiosis: any additional info on configuring geo-replication?
18:48 tjstansell the parallels cloud storage is tailored for their platform and ties into their virtuozzo containers well ... but you have to pay parallels a per GB cost for how much of your own storage you use ... ;)
18:48 semiosis Gilbs: nope, just ,,(rtfm)
18:48 glusterbot Gilbs: Read the fairly-adequate manual at http://goo.gl/E3Jis
18:48 semiosis havent really done much geo-rep myself
18:49 Gilbs will do, but lunch first...
18:49 semiosis Alknelt: pastie client log file?
18:50 eightyeight tjstansell: yeah. you can either do ther rev share, where you pay 10%, and the storage is free, or pay $6 per 1 GB, iirc
18:50 Alknelt semiosis: Is that located under  /var/log/glusterfs ? How much of it do you want?
18:50 eightyeight tjstansell: i had many disagreements with their "senior engineer" about glusterfs
18:51 eightyeight i know they're pushing their product, but at least get the facts straight
18:51 semiosis Alknelt: it is, i guess ~50 lines surrounding the time when your rm failed
18:51 eightyeight meh
18:51 tjstansell eightyeight: agreed.
18:51 tjstansell were you at summit?
18:51 eightyeight yeah
18:52 eightyeight my employer uses their products
18:52 ctria joined #gluster
18:53 semiosis Alknelt: had to /ignore you, sorry.  please do not send multiline pastes over irc, neither in channel nor private message.  please use a pastebin site like pastie.org instead :)
18:58 Alknelt semoisis: Sorry about that.
18:58 Alknelt http://pastie.org/6406006
18:58 glusterbot Title: #6406006 - Pastie (at pastie.org)
18:59 semiosis thanks, no problem :)
18:59 Alknelt I haven't used IRC for over 10 years...
19:03 semiosis lots of transport endpoint is not connected.  maybe an iptables or name resolution problem?
19:03 Alknelt iptables is off, Direct IP connect, no DNS
19:04 Alknelt I do see on the client the iptables on.
19:05 semiosis connection to 10.2.119.252:24015 failed (Connection refused) -- client unable to reach bricks.  possible causes, iptables, ip addr conflict (wrong server receives connection & refuses)
19:05 semiosis conn refused means a server sent RST because no process is bound to that port, or iptables is rejecting packets on that port
19:05 JoeJulian ... or glusterfsd not running...
19:05 semiosis ah, forgot to start the volume?
19:06 Alknelt It could be the client side iptables. I just turned the service off...
19:06 semiosis JoeJulian: good point
19:07 Alknelt volume is started, glusterd is running, no ipconflict…. possible client side iptables. But now disabled. Still can't delete directory.
19:08 JoeJulian check gluster volume status for "N"s
19:08 Alknelt All online
19:08 Alknelt I started a bonnie++ against it last night
19:09 JoeJulian pastie the output... you've got me intrigued
19:09 JoeJulian of gluster volume status
19:09 Alknelt bonnie++ died with the transport disconnect problem… haven't been able to delete a directory since then. Although I hadn't tried before either
19:09 Alknelt http://pastie.org/6406074
19:09 glusterbot Title: #6406074 - Pastie (at pastie.org)
19:10 JoeJulian semiosis: We should start a company that just does performance analysis based on the client's real-world needs.
19:11 semiosis i suspect ,,(ext4) problem
19:11 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
19:11 akshay joined #gluster
19:11 Alknelt It is running on ext4 bricks..
19:11 semiosis Alknelt: are your bricks formatted with ext4?  and are you running centos/redhat?
19:11 Alknelt Yes, yes.
19:11 JoeJulian Good call.
19:11 semiosis so thats most likely the problem then
19:11 semiosis everything else checked out
19:11 Alknelt Would xfs be a better choice?
19:11 semiosis +1
19:11 semiosis with inode size 512
19:12 semiosis yes
19:12 Alknelt default mkfs.ext4 options
19:12 Alknelt Any special considerations I should consider for xfs?
19:12 JoeJulian mkfs.xfs -i size=512
19:13 semiosis ^^^ inode size 512
19:13 Alknelt possibly filled the inode count up on the ext4 volumes then?
19:14 akshay hey
19:14 semiosis Alknelt: doubtful, but you can check inode usage on your bricks with df -i
19:14 flrichar if you use volumes to make your bricks a particular size, should you change your default PE size?
19:15 Alknelt not even close to full on inode utilization
19:15 flrichar someone somewhere mentioned 32m (not in gluster context though)
19:15 Alknelt Bricks are presented as a LUN over SAN… no volume usage
19:15 akshay i have having issues mouting the gluster volumes
19:15 semiosis Alknelt: it's an issue with the linux kernel's ext code, a change from mainline kernel 3.3 (which was backported to older redhat kernels) broke glusterfs
19:15 JoeJulian ~ext4 | Alknelt
19:15 glusterbot Alknelt: Read about the ext4 problem at http://goo.gl/PEBQU
19:15 akshay havinf endpoint transport errors
19:16 akshay any one knows how to fix them
19:16 Alknelt Thank you peeps. I'll get this going again except with xfs. :-)
19:16 semiosis Alknelt: the inode size thing is just an optimization because glusterfs uses ,,(extended attributes) so making inodes larger accomodates them
19:16 JoeJulian akshay: Paste the logs errors you're referring to at fpaste.org and paste the link generated here.
19:16 glusterbot Alknelt: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
19:17 akshay ok
19:17 akshay will do it
19:17 sergevn joined #gluster
19:17 semiosis akshay: usual causes: volume is stopped, iptables is blocking, or hostnames can't be resolved by the client
19:18 tryggvil joined #gluster
19:19 akshay volumes are not stopped
19:19 akshay i have rebooted the gluster nodes also
19:19 akshay or IP are resolving
19:26 en0x joined #gluster
19:26 en0x hi. i have issue. i have gluster cluster for testing when i do /etc/init.d/glusterd stop the mounted share is fine but when i reboot one of the servers i get unresponsive share until the server comes back
19:26 en0x any idea at what i should look to fix it?
19:27 akshay hey JoeJulian the link is http://fpaste.org/OsEq/
19:27 glusterbot Title: Viewing Paste #282771 (at fpaste.org)
19:27 semiosis en0x: client log file, /var/log/glusterfs/client-mount-point.log
19:28 JoeJulian akshay: Says right in there what the problem is and how to fix it.
19:28 dustint joined #gluster
19:29 en0x semiosis: http://pastie.org/private/oj7iqfuktkiih9sjznsa
19:29 glusterbot Title: Private Paste - Pastie (at pastie.org)
19:29 JoeJulian akshay: Here's more info: http://joejulian.name/blog/mounting-a-gl​usterfs-volume-as-an-unprivileged-user/
19:29 glusterbot <http://goo.gl/3kpJW> (at joejulian.name)
19:29 akshay ok
19:29 en0x seems like when i did decreate the ping timeout was better
19:30 JoeJulian ping-timeouts are inversely proportional to the probability of split-brain.
19:30 JoeJulian @ping-timeout
19:30 glusterbot JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
19:31 JoeJulian en0x: The problem is that your shutdown routine is not killing the glusterfsd processes before the network stops.
19:32 en0x or mayi see
19:32 en0x i see
19:32 en0x yes because when i shut down glusterfs by hand it happens instantly
19:32 en0x so i think there must be some issue with the default init script in centos6
19:32 bitsweat left #gluster
19:32 JoeJulian Not kkiethley's....
19:33 tjstansell anyone know where i can get glusterfs-3.3.0 repo packages for centos 6?  download.gluster.org only seems to have 3.3.1...
19:33 JoeJulian @yum repo
19:33 glusterbot JoeJulian: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
19:33 JoeJulian Oh, 3.3.0? no clue.
19:33 JoeJulian It's pretty broken. What do you need from 3.3.0?
19:33 tjstansell trying to see if the timestamp issues i'm seeing is only on 3.3.1
19:33 JoeJulian Ah
19:34 tjstansell i'm hesitant to go back to 3.2 ... but not sure what else to try to help narrow anything down.
19:34 JoeJulian I wonder if the 3.3.0 build can still be found on koji...
19:35 JoeJulian http://koji.fedoraproject.org/​koji/buildinfo?buildID=355244
19:35 glusterbot <http://goo.gl/WQU4z> (at koji.fedoraproject.org)
19:36 JoeJulian oops... State deleted...
19:36 tjstansell hm... interesting that the rpms on kkeithly's repo have -11 versions, where the ones I got from download.gluster.org have -1.
19:36 tjstansell glusterfs-3.3.1-11.el6.x86_64.rpm vs glusterfs-3.3.1-1.el6.x86_64.rpm
19:36 tjstansell are those really 10 revs different?
19:37 JoeJulian Yeah, the gluster.org packages will only ever have the -1 releases. iirc, most of the changes from -4 or -5 to -11 were swift related.
19:37 akshay thank you JoeJulian
19:37 JoeJulian akshay: You're welcome.
19:39 Mo___ joined #gluster
19:45 JoeJulian tjstansell: Once this finishes building, that will be the same as the official package was: http://koji.fedoraproject.org​/koji/taskinfo?taskID=5086581
19:45 glusterbot <http://goo.gl/Vhffd> (at koji.fedoraproject.org)
19:47 tjstansell i'm confused. what are you building? :)
19:47 JoeJulian 3.3.0-11
19:47 tjstansell oh. i see.
19:48 JoeJulian This is the system those actually get built on. I issued the same command kkiethley did when he built that package (with the exception that I'm building it as a scratch build instead of a release build).
19:49 tjstansell so do you think his 3.3.1-11 packages would have anything different (other than swift) than the centos 3.3.1-1 ones?
19:50 tjstansell it'll be good to test 3.3.0 too, but if his 3.3.1-11 packages include something that fixes things, that would be better, imo.
19:51 JoeJulian tjstansell: Here's the changelog http://paste.fedoraproject.org/4480/13625994
19:51 glusterbot Title: #4480 Fedora Project Pastebin (at paste.fedoraproject.org)
19:52 tjstansell that looks like 3.3.0-11 ... i'm more curious about 3.3.1-11 ...
19:54 JoeJulian http://paste.fedoraproject.org/4482/36259963
19:54 glusterbot Title: #4482 Fedora Project Pastebin (at paste.fedoraproject.org)
19:55 GabrieleV joined #gluster
19:56 tjstansell hm... that doesn't look like there's much chance anything would change... but i'll give it a shot and see how it goes.
19:56 semiosis tjstansell: what is the timestamp issue you're dealing with?
19:56 tjstansell bug 918437
19:56 glusterbot Bug http://goo.gl/1QRyw unspecified, unspecified, ---, pkarampu, NEW , timestamps updated after self-heal following primary brick rebuild
19:57 semiosis thx
19:57 JoeJulian tjstansell: If you really want to diagnose how that bug is happening, I would perform the operation with a TRACE log level for both the client and bricks. That will likely tell you (if you're reasonably good at diagnostics) where in the code to look for the problem. If you can read C then next what I do is look at the code pieces and try to conceptualize what's happening.
19:57 semiosis wow what a bug!
19:58 JoeJulian cool, huh.
19:58 tjstansell i did have DEBUG set ... the scary thing is that the logs show the *correct* thing happening, but the timestamp doesn't follow what the logs showed.
19:58 JoeJulian I could swear I've done this and didn't encounter that, but who knows...
19:59 tjstansell well, whatever we're doing ... it happens *every* time. :)
20:00 JoeJulian Ok, well if the logs show the correct thing happening, then the bug /should/ be pretty obvious.
20:00 JoeJulian Do you have that debug log anywhere?
20:01 tjstansell not currently, i've rebuilt several times, but i can turn on TRACE and do it again.
20:01 JoeJulian Probably would be good to add that to the bug report...
20:02 JoeJulian semiosis: http://www.openstack.org/blog/2013/02/brin​g-on-the-crazy-zero-to-book-in-five-days/
20:02 glusterbot <http://goo.gl/fpYot> (at www.openstack.org)
20:04 tc00per joined #gluster
20:04 JoeJulian semiosis: Maybe we should get RH to sponsor a gathering for us to do that with GlusterFS...
20:04 semiosis sure many fine books have been produced by collecting essays from many authors
20:04 semiosis see pm
20:05 JoeJulian They actually wrote it.
20:10 H__ joined #gluster
20:15 tc00per left #gluster
20:31 Gilbs Not sure if I missed this in the manual, how long between checks on geo-replication and sync?  Can the time be changed?
20:34 tjstansell joejulian: i have trace logs now of this happening.
20:34 tjstansell though i don't see anything obviously wrong.
20:35 tjstansell my file went from: -r-xr-xr-x 1 root root 2717 Mar  5 17:11 sync-data
20:35 mooperd joined #gluster
20:35 tjstansell to: -r-xr-xr-x 1 root root 2717 Mar  6 20:23 sync-data
20:36 JoeJulian attach those logs to the bug report. I'm off to a PT appointment for now...
20:39 mooperd_ joined #gluster
20:43 akshay hey JoeJulian
20:44 phase5 joined #gluster
20:50 mooperd_ joined #gluster
20:52 phase5 good even all,
20:52 tjstansell JoeJulian: i've attached the log to the bug
20:52 phase5 is it possible to run 2 (master) mysql server in top of glsuterfs, do i need special option for that in glusterfs and/or mysql?
20:53 phase5 same question would be interesting about postgress
20:54 phase5 if that would be possible life would be much easier in the sence of master /master dabases
20:54 phase5 can anybody advise me?
20:55 Gilbs good question phase5, we're going postgress in the near future.
21:06 phase5 i tested mysql on glusterfs 2.x a time ago - but the problem was that i need to restart the other mysqlserver to see the changes which i made on the mysql serverA
21:06 phase5 right now i am running 3.3 (latest)
21:07 phase5 what means "near future"? any version targets?
21:11 Gilbs That's all on the sql guys, it's my job to work out the storage portion. :)
21:17 _pol joined #gluster
21:18 _pol joined #gluster
21:19 Alknelt I've rebuilt my 70TB filesystem on top of XFS bricks with inodes size=512. The filesystem is behaving much better, a bonnie++ benchmark comes back with reasonable results.
21:19 semiosis \o/
21:20 Alknelt Is there anyway to optimize Gluster for smaller file read/writes?
21:25 Gilbs Alknelt:  performance.flush-behind   is what I remember turning to to help with smaller files.  Anyone second that?
21:25 Gilbs turning on*
21:29 tqrst is there a reason why gluster moved away from having documentation in indexable html instead of pure pdf? Searching for "gluster performance.flush-behind" on google yields 3.1 and 3.2 docs, but nothing for 3.3. You have to hunt down the admin guide pdf and search in there.
21:29 Alknelt Gilbs: Is there a way to list the OPTIONS as they are configured?
21:30 tqrst even then, the string "flush-behind" is wrapped in the pdf, so searching for "flush-behind" yields 0 hits whereas "flush-\nbehind" finds the correct page
21:31 tqrst Alknelt: gluster volume info, coupled with the documentation that shows the default values for various options
21:31 tqrst Alknelt: according to the 3.3.0 pdf, flush-behind is already on by default
21:35 semiosis 'gluster volume info' shows options that have been changed from default (even if they've been changed *back* to default since)
21:35 Gilbs reading from a different page it is off by default, is this right?  http://gluster.org/community/documenta​tion/index.php/Translators/performance
21:35 glusterbot <http://goo.gl/ikBtj> (at gluster.org)
21:36 semiosis 'gluster volume set help' shows all available options & some info
21:36 tqrst Gilbs: that page was last updated in 2011, so I would assume the docs in the admin guide pdf are more up to date
21:37 Alknelt Reading through the admin guide Tuning section now.
21:37 semiosis 'gluster volume set help' on my 3.3.1 test vm says flush-behind is on by default
21:37 Gilbs ah
21:37 semiosis and maybe thats why people say gluster is eventually consistent?!
21:38 tqrst the wiki is eventually consistent, yes ;)
21:38 semiosis i'm surprised an option that sacrifices consistency for performance is on by default.  i thought gluster went with consistency by default. hmm
21:41 H__ semiosis: i did not find time yet to dig deeper in the upstart scripts race
21:41 mattr01 is anyone firmiliar with the GlusterFS / Hadoop jar?
21:41 semiosis H__: neither have i
21:42 Alknelt performance.io-thread-count documentation is in error. Range is not 0-65, but 0-64
21:42 Alknelt erm 1-64
21:45 Alknelt performance.cache-size is not clear to me. Default is 32 MB, but available is size in bytes. Does that mean if I want to use 1MB Id input 1024?
21:46 fidevo joined #gluster
22:01 dendazen joined #gluster
22:01 dendazen Guys having issues with glusterFS
22:02 dendazen lsof -nPi | grep 16913
22:02 dendazen glusterfs 16913      root    6u  IPv4 1145649      0t0  TCP 10.147.194.200:1023->10.151.26.150:24007 (ESTABLISHED)
22:02 dendazen i have connection established to the client
22:02 dendazen to the server i am sorry
22:02 dendazen from the client
22:02 dendazen but when i do df -h
22:02 dendazen df: `/var/www/public/documentation': Transport endpoint is not connected
22:02 dendazen shows me this error.
22:03 dendazen Everything just worked yesterday.
22:03 dendazen nothing was changed
22:03 dendazen and mount shows.
22:04 dendazen 10.151.26.150:/data_pdf on /var/www/public/documentation type fuse.glusterfs (rw,allow_other,default_pe​rmissions,max_read=131072)
22:04 dendazen what can be the problem?
22:04 dendazen Thanks.
22:05 semiosis dendazen: client seems to have lost connection to brick(s) there should be connections on ,,(ports) above 24007 in addition to 24007 itself
22:05 glusterbot dendazen: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
22:05 semiosis dendazen: check client log file /var/log/glusterfs/client-mount-point.log for more info
22:07 dendazen ok
22:07 dendazen looking i see this
22:07 dendazen [2013-03-05 03:46:22.746137] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.147.194.200:1020)
22:07 glusterbot dendazen: That's just a spurious message which can be safely ignored.
22:07 dendazen Oh so what my coworker is saying
22:08 dendazen he stopped
22:08 dendazen daemon
22:08 dendazen than removed everything in /vols
22:08 dendazen folder and recreated the bricks to bind to the new Private IP
22:08 dendazen as he stopped AWS instance
22:08 dendazen and it gave a different private IP
22:08 semiosis should use ,,(hostnames) instead of ip addresses
22:09 glusterbot Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
22:09 semiosis to avoid this kind of problem
22:09 dendazen semiosis
22:09 dendazen the thing is
22:09 dendazen right now
22:10 dendazen we can only have one server in this glusterfs
22:10 dendazen no additional peers
22:10 dendazen but we mount stuff to the client
22:10 dendazen should not be matter
22:10 dendazen Also since the server was stopped on AWS
22:11 dendazen i assume hard drive UID most likely changed
22:11 dendazen can it be an issue?
22:11 semiosis should not be an issue, glusterfs has its own UUID in /var/lib/glusterd/glusterd.info -- not related to hard drive UUID
22:12 dendazen ok.
22:12 semiosis dendazen: please ,,(pasteinfo)
22:12 glusterbot dendazen: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:13 dendazen http://fpaste.org/gKrp/
22:14 dendazen and also on the client there is no  /var/log/glusterfs/client-mount-point.log
22:16 semiosis does that link work for you?
22:16 tqrst doesn't work for me
22:16 dendazen oh one sec
22:17 semiosis also while youre at it, fpaste 'ls /var/log/glusterfs'
22:17 dendazen http://fpaste.org/iV3N/
22:17 glusterbot Title: Viewing Paste #282807 (at fpaste.org)
22:17 tjstansell JoeJulian: I uninstalled 3.3.1 on my test boxes, installed the 3.3.0 build you created for me, corrected the UUID for each node so they were the same as before, started things up, and see the same timestamp issue
22:17 dendazen on the client?
22:17 semiosis yes
22:18 semiosis and 1-brick volumes?  why not use NFS if you only need 1-brick volumes?
22:18 dendazen http://fpaste.org/E6R2/
22:18 glusterbot Title: Viewing Paste #282808 (at fpaste.org)
22:18 dendazen well its going to grow
22:18 dendazen soon
22:19 dendazen in testing now.
22:19 dendazen + nfs has client issues.
22:19 semiosis ok fair enough
22:20 semiosis by the way, client-mount-point.log means the path where you mounted your client, so in your case probably  var-www-public-documentation-.log
22:20 semiosis seems to be the most recent
22:20 dendazen yeah
22:20 dendazen thats the one.
22:21 dendazen ALso i have weird stuff
22:21 dendazen when i restart flusterfsd
22:22 dendazen the server box tries to connect to those brick by itslef
22:22 dendazen there might be some issues with hostname
22:22 dendazen does it matter?
22:22 semiosis sorry i dont understand what youre talking about
22:23 dendazen ok i'll paste it.
22:25 dendazen http://fpaste.org/Gxxb/
22:25 glusterbot Title: Viewing Paste #282809 (at fpaste.org)
22:25 dendazen something is not right.
22:26 semiosis looks normal to me.  see ,,(processes)
22:26 glusterbot the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
22:26 dendazen oh ok.
22:26 dendazen so now i should be able to remount
22:27 dendazen the client?
22:27 semiosis sure
22:27 dendazen as i would kill processes with
22:27 dendazen fs stuff
22:27 dendazen as i thought it wasn't looking right
22:40 mitzip joined #gluster
22:40 ackjewt joined #gluster
22:52 tg2 joined #gluster
23:03 dendazen oh and he mounted /tmp
23:03 dendazen to the thing
23:04 dendazen but since it is now is not there
23:04 dendazen fuse_fs uses /tmp
23:04 dendazen in the mounting process
23:09 _pol joined #gluster
23:17 andrei joined #gluster
23:18 nueces joined #gluster
23:25 Gilbs left #gluster
23:39 _pol joined #gluster
23:46 _pol_ joined #gluster
23:53 tjstansell JoeJulian and semiosis: i've narrowed down my test case to simply nuking both references to a file on the primary brick (simulating brick corruption/failure).  I've also verified this is still the case in 3.4.0alpha too.
23:54 tjstansell as a reminder, this is bug 918437
23:54 glusterbot Bug http://goo.gl/1QRyw unspecified, unspecified, ---, pkarampu, NEW , timestamps updated after self-heal following primary brick rebuild

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary