Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 gbrand__ joined #gluster
00:08 kevein joined #gluster
00:08 lhawthor_ joined #gluster
00:14 gbrand__ joined #gluster
00:39 nightwalk joined #gluster
01:07 dedis11 joined #gluster
01:09 jiffe1 joined #gluster
01:34 nightwalk joined #gluster
02:08 nightwalk joined #gluster
02:32 davdunc joined #gluster
02:32 davdunc joined #gluster
02:34 yeming joined #gluster
02:37 sunus joined #gluster
02:46 bala1 joined #gluster
02:53 robo joined #gluster
02:59 nightwalk joined #gluster
03:20 bharata joined #gluster
03:30 lng joined #gluster
03:32 lng Hi! I have errors like this: "ls: cannot access /storage/3110000/saved.dat: Input/output error"
03:32 lng ??????????  ? ?        ?            ?            ? saved.dat
03:32 lng ls output ^
03:33 lng what can I do about that?
03:36 lng I even can't remove this file
03:39 lng is it split-brain?
03:46 nightwalk joined #gluster
03:49 shylesh joined #gluster
04:03 quillo joined #gluster
04:13 nightwalk joined #gluster
04:28 raghu joined #gluster
04:29 bharata joined #gluster
04:42 hagarth joined #gluster
04:45 sripathi joined #gluster
04:46 m0zes lng: @split-brain
04:46 m0zes @split-brain
04:46 glusterbot m0zes: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
04:46 m0zes lng: ^^
04:46 lng m0zes: probably, I know the reason
04:47 m0zes okay, the healing portion is what I thought you'd like to see
04:47 lng m0zes: I scaled up the nodes
04:47 lng yes
04:47 lng thansk
04:47 lng thanks
04:47 lng files have been updtaed on node A while node B was down
04:48 lng filenames are always the same
04:48 lng in result - two different files after node B started
04:48 lng m0zes: should it be the reason?
04:49 m0zes how did node B do down? power outage or a proper shutdown?
04:49 lng m0zes: I know very well this page
04:50 lng m0zes: just shutdown over EC2 console
04:51 lng m0zes: is there better way?
04:51 m0zes a split brain shouldn't normally be caused by a proper shutdown, as the nodes should have had a chance to make sure they were in relative sync before the daemons on B turned off.
04:51 lng I want to test it
04:51 lng m0zes: now I have a lot of errors
04:51 m0zes then when B came back up B should have recognized that its files were outdated.
04:52 lng Input/output error
04:52 lng m0zes: so it's okay to `$ halt` it, right?
04:53 m0zes I wonder if the shutdown scripts sent a kill -9 (SIGKILL) versus a kill -15 (SIGTERM). The SIGTERM will allow the daemons to communicate and properly shutdown any active writers to B.
04:54 m0zes thus making A the clear primary.
04:54 lng m0zes: maybe it's better to `pkill -f gluster` before shutdown
04:54 m0zes that might be a better way.
04:55 lng now I need to write shell script to heal the cluster
04:56 m0zes in any case, you know that A has the true copies of the files, and you should delete the split brained files from the brick(s) on B.
04:57 m0zes it sounds like 3.4 will have better handling of split-brains by way of an arbiter daemon that will act as a tie-breaker if both claim to be "primary"
04:57 lng I think I can get the list of these files by `gluster volume heal volume info split-brain`
05:01 deepakcs joined #gluster
05:03 mohankumar joined #gluster
05:14 ika2810 joined #gluster
05:17 vpshastry joined #gluster
05:19 jdarcy joined #gluster
05:28 bulde joined #gluster
05:45 nightwalk joined #gluster
05:56 mdarade1 joined #gluster
05:56 mdarade1 left #gluster
06:05 lng m0zes: I was not able to reproduce this problem
06:07 harshpb joined #gluster
06:08 abyss joined #gluster
06:11 vpshastry joined #gluster
06:12 bala2 joined #gluster
06:15 shylesh_ joined #gluster
06:16 lng m0zes: but what should I do with <gfid:438d0817-6b98-4dbf-8996-8b77555835ae>?
06:16 lng can I just delete them?
06:20 rgustafs joined #gluster
06:27 ngoswami joined #gluster
06:35 deepakcs joined #gluster
06:41 vpshastry joined #gluster
06:51 lng I have removed all gfid files returned by split-brain command, but 'Input/output' errors are still present
06:54 rudimeyer joined #gluster
06:58 ramkrsna joined #gluster
06:58 ramkrsna joined #gluster
06:58 lng why are not these files shown by `gluster volume heal storage info split-brain`?
07:04 guigui3 joined #gluster
07:22 y4m4 joined #gluster
07:30 shireesh joined #gluster
07:32 nightwalk joined #gluster
07:40 sripathi joined #gluster
07:40 glusterbot New news from newglusterbugs: [Bug 877885] [cb884f7eac134866ca47701d3208055acecbf9a0]: nfs server asserted due to null gfid <http://goo.gl/tRjHM>
07:50 ctria joined #gluster
07:50 ekuric joined #gluster
07:59 lkoranda joined #gluster
08:16 nightwalk joined #gluster
08:27 dobber joined #gluster
08:31 sshaaf joined #gluster
08:32 deepakcs joined #gluster
08:35 andreask joined #gluster
08:35 deepakcs joined #gluster
08:36 quillo joined #gluster
08:37 gbrand_ joined #gluster
08:40 Nr18 joined #gluster
08:41 glusterbot New news from newglusterbugs: [Bug 877903] Crash while running SSL unit test <http://goo.gl/1qdTg>
08:44 rosco 5
08:47 nightwalk joined #gluster
08:57 tjikkun_work joined #gluster
09:05 nightwalk joined #gluster
09:05 sunus joined #gluster
09:07 vpshastry joined #gluster
09:13 Azrael808 joined #gluster
09:17 inodb joined #gluster
09:25 tryggvil joined #gluster
09:34 DaveS joined #gluster
09:52 tryggvil joined #gluster
10:02 rudimeyer joined #gluster
10:04 guigui4 joined #gluster
10:13 duerF joined #gluster
10:16 vpshastry joined #gluster
10:18 nightwalk joined #gluster
10:36 ekuric joined #gluster
10:36 andreask joined #gluster
10:57 andreask left #gluster
10:59 * jdarcy o_O
11:04 nightwalk joined #gluster
11:21 glusterbot New news from resolvedglusterbugs: [Bug 866456] gluster volume heal $ full keeps increasing the No. of entries for gluster volume heal $ info healed even if healing is not done <http://goo.gl/5Opvq>
11:31 quillo joined #gluster
11:39 webwurst joined #gluster
11:39 bauruine joined #gluster
11:44 nightwalk joined #gluster
11:45 shireesh joined #gluster
11:56 hagarth1 joined #gluster
11:56 lkoranda joined #gluster
11:58 lkoranda joined #gluster
12:02 nightwalk joined #gluster
12:30 nueces joined #gluster
12:41 glusterbot New news from newglusterbugs: [Bug 878004] glusterd segfaults in remove brick <http://goo.gl/KCswd>
12:43 tryggvil joined #gluster
12:45 tryggvil_ joined #gluster
12:52 edward1 joined #gluster
13:01 rgustafs joined #gluster
13:04 manik joined #gluster
13:06 bala joined #gluster
13:12 quillo joined #gluster
13:13 nightwalk joined #gluster
13:16 ika2810 joined #gluster
13:22 manik joined #gluster
13:26 sripathi joined #gluster
13:26 balunasj joined #gluster
13:28 tmirks joined #gluster
13:33 kleind joined #gluster
13:35 kleind Hi there. I'm in the process of evaluating glusterfs. Installed 3.3.1-1 on debian squeeze. After creating and mounting a striped-replicated volume, I cannot write to it. dd says "invalid argument" about the of= option and cp says "invalid argument" to the destination file. Any advice?
13:35 kleind bricks are xfs with -i size=512, mounted rw. gluster volume status indicates everything to be ok.
13:37 andreask joined #gluster
13:44 dobber joined #gluster
13:44 nightwalk joined #gluster
13:46 puebele1 joined #gluster
13:48 hagarth joined #gluster
13:53 dobber joined #gluster
13:53 kleind i'm seeing this in the logs each time i try to write: [2012-11-19 14:52:52.733650] W [fuse-bridge.c:2025:fuse_writev_cbk] 0-glusterfs-fuse: 141: WRITE => -1 (Invalid argument
13:54 kleind and also "failed to get stripe size"
13:55 kleind this is the output of gluster volume status http://pastebin.ca/2253057
13:55 glusterbot Title: pastebin - Unnamed - post number 2253057 (at pastebin.ca)
13:55 kleind and here's an update with "gluster volume info" http://pastebin.ca/2253060
13:55 glusterbot Title: pastebin - Unnamed - post number 2253060 (at pastebin.ca)
13:57 ndevos hmm, Striped-Replicate with 1 x 2 x 2 = 4 bricks looks strange to me
13:58 ndevos kleind: whats the reason to use striped? Don't you want to use distribute-replicate instead?
13:59 kleind i am just evaluating. if this mode is causing, i am willing to try differently.
13:59 ndevos kleind: JoeJulian wrote a nice article about it -> http://joejulian.name/blog/sho​uld-i-use-stripe-on-glusterfs/
13:59 glusterbot <http://goo.gl/5ohqd> (at joejulian.name)
13:59 kleind i am rather sure i want to use distr-repl later, but i am not quite there yet
13:59 kleind i want to see myself
14:00 kleind i read that blog btw
14:00 bennyturns joined #gluster
14:00 ndevos sure, but striped-replicate is relatively uncommon and it was not available in earlier releases at all
14:00 kleind okay. but is it known to not let me write anything? i guess not. so i figured i'd report it?
14:01 ndevos striped is normally used for throw-away, recreatable data, like scratch space, no need to replicate that
14:02 ndevos it *should* work, but it definitely less tested, and I think I've seen a similar bug before to what you are experiencing
14:02 * kleind notes "don't use striped-replicated"
14:04 oneiroi joined #gluster
14:06 ndevos kleind: Bug 842752 seems to describe the same issue
14:06 glusterbot Bug http://goo.gl/mZQOL urgent, high, RHS 2.1, sgowda, CLOSED ERRATA, dd fails with "Invalid argument" error on the mount point while creating the file for first time on stripe with replicate volume
14:06 kleind thanks a lot ndevos
14:07 ndevos which is actually Bug 842364
14:07 glusterbot Bug http://goo.gl/we3JJ urgent, unspecified, ---, sgowda, MODIFIED , dd fails with "Invalid argument" error on the mount point while creating the file for first time on stripe with replicate volume
14:07 ndevos and as it is MODIFIED, the change is not yet in a released version
14:08 kleind briefly read it. seems like exactly what i am seeing. touch also succeeds. thanks a lot for posting that link
14:08 ndevos you're welcome
14:15 spn joined #gluster
14:17 kleind nvedos: since you mentiond 1-2-2 looking strange. would you explain how to read that line? which is replicate, which is stripe and which is the third number?
14:18 bauruine joined #gluster
14:27 robo joined #gluster
14:32 nightwalk joined #gluster
14:40 aliguori joined #gluster
14:44 rwheeler joined #gluster
14:54 kevein joined #gluster
14:56 dobber joined #gluster
14:57 stopbit joined #gluster
14:59 nightwalk joined #gluster
15:00 Azrael808 joined #gluster
15:05 robo joined #gluster
15:16 layer3switch joined #gluster
15:39 Designeru joined #gluster
15:39 Designeru hello
15:39 glusterbot Designeru: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
15:40 Designeru i'm testing gluster for a personal project and I was wondering couple of things:
15:40 Designeru - are there any limits in the filesize name/size ?
15:42 Designeru - if i set the "replica" to 2, but I want to add 3 servers i get an error message "Incorrect number of bricks supplied 1 with count 2"
15:42 Designeru does that mean that i need to always add 2 new nodes ?
15:43 Designeru thank you!
15:43 guigui1 joined #gluster
15:44 tqrst joined #gluster
15:45 kleind what about files with mode "---------T" in a gluster brick?
15:46 kkeithley the only file size limits are the same as the those imposed by the backing file systems.
15:46 kleind oh, i'm sorry, that was an faq
15:46 kkeithley If you want replica 2, they you need to add bricks in multiples of 2; replica 3, in multiples of 3, and so on
15:47 kkeithley s/they/then/
15:47 glusterbot kkeithley: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:47 Designeru thanks kkeithley
15:47 hurdman left #gluster
15:49 Designeru kkeithley - thanks for the answers :)
15:51 kkeithley yw
15:51 ndevos kleind: such a file is a link-file, similar to a symlink, but it can point to a file on a different brick/server, try 'getfattr -e hex -d -m . $file' on the brick
15:51 ndevos ah, thats in the FAQ?
15:53 kleind at least it's on the community website. when searching for "---------T" :)
15:56 jbrooks joined #gluster
16:00 zaitcev joined #gluster
16:03 mohankumar joined #gluster
16:07 tryggvil joined #gluster
16:11 rwheeler_ joined #gluster
16:13 aricg_ joined #gluster
16:14 nightwalk joined #gluster
16:21 ekuric joined #gluster
16:33 semiosis :O
16:58 ekuric left #gluster
17:05 Nr18 joined #gluster
17:07 ika2810 left #gluster
17:14 neofob where is the default log file of 3.3.1?
17:15 kkeithley /var/log/glusterfs/*
17:16 neofob hm...i don't see it
17:16 sjoeboo joined #gluster
17:16 kkeithley which linux dist?
17:16 neofob debian wheezy
17:20 neofob found it; it's in /usr/local/var/log/glusterfs
17:24 rudimeyer joined #gluster
17:31 bauruine joined #gluster
17:44 ceocoder joined #gluster
17:45 semiosis that's ood
17:46 semiosis odd
17:48 Nr18 joined #gluster
17:55 ceocoder1 joined #gluster
18:04 nightwalk joined #gluster
18:08 johnmark and ood
18:08 johnmark neofob: did you build this? or install our .deb?
18:09 neofob johnmark: i downloaded the 3.3.1 source and compiled it
18:09 rudimeyer joined #gluster
18:11 neofob so, fyi, by default, gluster installs everything to /usr/local
18:11 neofob i didn't specify PREFIX when i ran ./configure in the source
18:22 _Bryan_ Is there 3.2.5 docs still online?  it is not listed on the archived docs
18:26 johnmark _Bryan_: forgot to include the link: http://www.gluster.org/community/documentation/ind​ex.php/Gluster_3.2_Filesystem_Administration_Guide
18:26 glusterbot <http://goo.gl/J3Iz9> (at www.gluster.org)
18:26 johnmark will update that now
18:26 _Bryan_ Thanks..bringnign new guy up to speed and did not want him reading 3.3. as I am not running that
18:30 semiosis neofob: http://www.gnu.org/software/autoconf/manual/​autoconf-2.67/html_node/Default-Prefix.html
18:30 glusterbot <http://goo.gl/MSzCz> (at www.gnu.org)
18:30 semiosis neofob: that's not a glusterfs default
18:31 semiosis neofob: why are you building from source anyway?  are the packages not sufficient for you?
18:31 nightwalk joined #gluster
18:31 semiosis johnmark: good call on the source build!
18:31 semiosis nailed it :)
18:40 Mo__ joined #gluster
18:41 Technicool joined #gluster
19:02 neofob semiosis: the source there to be built ;)
19:08 bauruine joined #gluster
19:10 tryggvil joined #gluster
19:16 johnmark neofob: heh heh :)
19:17 johnmark semiosis: well, I couldn't figure out what else it woudl be
19:19 johnmark and I was just thinking of all those horrible situations I found myself after forgetting to add to configure "prefix=/usr"
19:32 _Bryan_ johnmark:  Thanks alot for fixing that link for me...
19:33 pdurbin johnmark: in case you missed this from sjoeboo: https://twitter.com/sjoeboo​/status/269850351602110464
19:34 glusterbot <http://goo.gl/0maLv> (at twitter.com)
19:37 tryggvil joined #gluster
19:38 johnmark pdurbin: woah! holy crap!
19:40 johnmark pdurbin: can he add his findings to this page: http://www.gluster.org/community/documentat​ion/index.php/Talk:Planning34/PuppetModule
19:40 glusterbot <http://goo.gl/UbWEC> (at www.gluster.org)
19:40 sjoeboo just a little pool to play around in for kvm storage, see how well it stands up
19:40 johnmark sjoeboo: very nice
19:40 johnmark would like to get the puppet module discussion kicked off again, too
19:41 johnmark _Bryan_: you're welcome
19:41 _Bryan_ Ooo....I want to hear about the puppet module...
19:41 sjoeboo johnmark: yeah, or this the gluster end, aside form install pkgs/service isn't really puppetized yet
19:42 sjoeboo that is to say, puppet doesn't make the bricks/join the pool/etc
19:43 glusterbot New news from newglusterbugs: [Bug 876214] Gluster "healed" but client gets i/o error on file. <http://goo.gl/eFkPQ>
19:43 johnmark sjoeboo: oh, ok
19:47 sshaaf joined #gluster
19:51 johnmark JoeJulian: ping
19:56 nhm joined #gluster
19:57 toruonu joined #gluster
19:57 toruonu out of curiosity, how large glusterfs installations are there and what kind of issues one might face?
19:58 toruonu contemplating here of about a 200 node 2PB (raw) config :)
19:59 rwheeler joined #gluster
20:02 johnmark toruonu: the 2PB part is easily done. The 200 node part may be a bit challenging, depending on your setup
20:02 johnmark ...with the usual caveats about use cases differing, along with expectations, yada yada
20:04 toruonu well the contemplation here is that we're right now using hadoop hdfs across all the nodes that have 3; 24 or 36 internal drives. The ones with 3-drives are the ones where the hdfs is also mounted for rw operations. Now if we were to stage that out to glusterfs (because hdfs isn't 100% posix and therefore not usable for user home directories etc), then we'd like to set up something similar. Probably replication factor 2 one single volume across all d
20:04 toruonu isks.
20:06 toruonu so right now the 3-disk nodes have about 2/3 of the space given raw to hdfs datanodes and 1/3 used in mirrored way locally for job scratch area… if we were however to move to a single uniform volume across the whole disk area, then that would have many benefits ...
20:06 toruonu the question here is wether glusterfs is ready for it :) and as I've only started reading about it I've decided to just ask :)
20:07 jdarcy The I/O path can handle 200 nodes OK.  It's the management paths I'd worry about.
20:08 toruonu got any information on what people have running live?
20:09 jdarcy Not sure how much I can disclose.  We do have customers with 200+ servers, but all that I know of are running unusually "well behaved" workloads like media streaming and they have very low server churn.
20:11 rudimeyer_ joined #gluster
20:12 toruonu well our use case would be CERN data analysis, it's not quite media streaming, but it's mostly huge amounts of reads with the occasional write to a local skimmed/analyzed file ...
20:12 toruonu however it would be about 5000 jobs doing it in parallel :)
20:12 toruonu on 170 servers :)
20:16 H__ that's quite a bit more than what i run in production, so i cannot add much useful ;-)
20:18 toruonu :)
20:19 toruonu ok, we'll probably start small… create a test volume of say 30-60TB with 2-3x replication and churn users stuff on this for a while… if it seems ok we can test expanding it ...
20:19 toruonu ok, got to run
20:33 johnmark doh
20:35 badone joined #gluster
20:36 johnmark samkottler|afk: ping :)
20:39 chandank joined #gluster
20:40 semiosis sjoeboo: are you going to monitorama?
20:40 sjoeboo when is that? (also, where)
20:40 semiosis march 28 at the msft nerd center in kendall sq
20:40 sjoeboo oh boston, march, right? yeah, i was planning on it
20:41 semiosis cool
20:41 * semiosis will be there
20:41 semiosis we could have a boston gluster pow wow /cc jdarcy kkeithley johnmark
20:41 semiosis s/pow wow/drinkup/
20:41 glusterbot What semiosis meant to say was: we could have a boston gluster drinkup /cc jdarcy kkeithley johnmark
20:42 semiosis or whatever they're called these days
20:42 sjoeboo yeah, that sounds good!
20:42 semiosis oh yeah pdurbin too
20:43 johnmark heh
20:43 johnmark semiosis: you bet!
20:43 johnmark sjoeboo: where is the msft nerd center?
20:43 * johnmark needs to find a place to host meetups
20:43 semiosis microsoft new england r&d center in kendall square, near mit
20:44 pdurbin mmm, drinkup
20:44 sjoeboo johnmark: i'd be amazed if places @ harvard were not available as well in one way or another...
20:46 pdurbin sjoeboo: there was an openstack meetup at seas a few months back
20:46 eightyeight joined #gluster
20:50 johnmark semiosis: ah, ok
20:50 kkeithley johnmark: mit, google, vmware, msft — they're all within a block of each other in Cambridge. Not sure what sjoeboo is thinking about wrt places at Harvard being available. For one thing if we're all in Kendall Sq, Harvard's a bit of a schlep to get to. Two stops on the T, but I'd say sticking around MIT would be better.
20:50 johnmark pdurbin: mmm... yes!
20:50 johnmark kkeithley: ok. I have no problem with MIT
20:50 sjoeboo yeah yeah, i jsut meant if looking for places, i bet there is stuff here as well
20:50 johnmark sjoeboo: but I <3 our friends at Hahvahd, too ;)
20:51 sjoeboo getting to kendel is not big deal for me!
20:51 johnmark heh
20:51 kkeithley Looking for places to go drinking? Yeah, there's plenty
20:51 kkeithley Miracle of Science
20:51 kkeithley to name one
20:51 johnmark kkeithley: I was thinking since we know of a few projects at harvard, it makes sense to host a meetup there
20:51 johnmark kkeithley: oh? sounds pretty cool
20:51 pdurbin kkeithley: jeez you're gonna make us walk all the way to mit? ;)
20:52 johnmark haha :)
20:52 kkeithley From Westford even ;-)
20:52 semiosis i hear the weather is nice in march :)
20:52 semiosis should be good for a walk
20:52 kkeithley Nicer than February usually.
20:52 JoeJulian johnmark: pong
20:52 johnmark kkeithley: the trick is finding a place that meets these criteria: 1. drinking + food allowed 2. can hold crowds of up to 50 people or sometimes larger
20:52 johnmark JoeJulian: see query
20:53 johnmark kkeithley: and 3. will be quiet enough such that we can hear each other talk
20:53 johnmark which rules out most bars / eating establishments
20:53 johnmark unless they have a private room
20:54 johnmark if they have AV equipment, ++ good
20:54 kkeithley Okay, wasn't sure what exactly you were looking for. I thought we were starting at msft in Kendall Sq.
20:54 semiosis i think johnmark is talking about glusterfs meetups generally, not necessarily a drinkup
20:54 kkeithley yup
20:54 semiosis i was just suggesting we get together for a beer while i'm in town :D
20:55 johnmark semiosis: heh. in my mind, those are synonymous :)
20:55 johnmark but yeah, I see what you mean
20:55 semiosis no av equip needed for that
20:55 johnmark semiosis: that sounds good to me!
20:55 semiosis as you can see on this slide, i enjoy lagers
20:55 semiosis :P
20:55 johnmark heh
20:56 pdurbin semiosis: absolutely a beer when you're in town
21:08 nightwalk joined #gluster
21:11 blendedbychris joined #gluster
21:11 blendedbychris joined #gluster
21:24 blendedbychris JoeJulian: you around… I have the scenario sort of as a while ago you helped me with.
21:24 blendedbychris http://pastie.org/private/4k9cwl0renu6jjkr9uaka
21:24 glusterbot Title: Private Paste - Pastie (at pastie.org)
21:27 blendedbychris http://pastie.org/private/2rh0hoese9q8tcs2aakv5a
21:27 glusterbot <http://goo.gl/Oql7r> (at pastie.org)
21:27 blendedbychris I have sld-wowza-2 which is currently working, replica but with a single failed node (sld-wowza-1)
21:29 blendedbychris I can't probe each of the nodes from one another though… the new node, sld-wowza-1 says sld-wowza-2 is already part of another cluster
21:30 purpleidea joined #gluster
21:30 purpleidea joined #gluster
21:34 bennyturns joined #gluster
21:37 TSM joined #gluster
21:43 lh joined #gluster
21:48 Azrael808 joined #gluster
21:48 blendedbychris http://pastie.org/private/lfxgijafpowgwdft9a2yfw
21:48 glusterbot <http://goo.gl/hpfa4> (at pastie.org)
21:54 andreask left #gluster
21:58 blendedbychris this is the error it returns http://pastie.org/private/4ctkwv4rjo4xvg98whrrw
21:58 glusterbot Title: Private Paste - Pastie (at pastie.org)
21:58 blendedbychris when I try to join sld-wowza-2 from the new sld-wowza-1 server
21:59 semiosis blendedbychris: please ,,(pastestatus) from *both* servers
21:59 glusterbot blendedbychris: Please paste the output of "gluster peer status" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:10 eightyeight joined #gluster
22:13 glusterbot New news from newglusterbugs: [Bug 877563] Metadata timestamps ignored potentially causing loss of new metadata changes <http://goo.gl/UH1ZB>
22:14 Andrew_ joined #gluster
22:24 SpeeR joined #gluster
22:30 eightyeight joined #gluster
22:41 blendedbychris semiosis: i did one sec i'll paste them together
22:41 blendedbychris semiosis: actually it's this one on sld-wowza-2 http://pastie.org/private/4k9cwl0renu6jjkr9uaka
22:41 glusterbot Title: Private Paste - Pastie (at pastie.org)
22:42 semiosis yeah, i saw that, then i asked for the status from *both* servers :)
22:43 blendedbychris my bad
22:43 blendedbychris http://pastie.org/private/rkgrl0tw8yoowj9iz7wkq
22:43 glusterbot Title: Private Paste - Pastie (at pastie.org)
22:43 semiosis it's just 'gluster peer status'
22:44 blendedbychris oh crap i knew that
22:44 semiosis i know :)
22:44 blendedbychris No peers present
22:44 blendedbychris (that's why i didn't pastie it). sorry getting caught back up too
22:44 semiosis ah ha
22:45 blendedbychris should that return something?
22:45 semiosis cat /var/lib/glusterd/glusterd.info please
22:45 blendedbychris UUID=59659cd7-6390-4b31-9af8-80dd6e53a499
22:45 semiosis odd
22:46 semiosis that doesnt match either of the uuids from your peer status on wowza-2
22:46 semiosis did this machine get wiped out?
22:46 blendedbychris ya it's a new machine
22:46 blendedbychris basically i'm readding, same hostnma
22:47 semiosis ~replace | blendedbychris
22:47 glusterbot blendedbychris: Useful links for replacing a failed server... if replacement server has different hostname: http://goo.gl/4hWXJ ... or if replacement server has same hostname:
22:47 glusterbot http://goo.gl/rem8L
22:47 semiosis ^^^
22:47 semiosis bbiab
22:47 blendedbychris this is 3.3
22:47 blendedbychris does that make a difference?
23:00 inodb_ joined #gluster
23:08 kminooie joined #gluster
23:08 kminooie hi everybody
23:09 rwheeler joined #gluster
23:11 kminooie so i have gluster volume which I am mounting through nfs. I keep getting the 'mount point has become stale' error. I have been searching for it online and although many people have been reporting it, i haven't yet been to find why it happens and how i can prevent it from happening. anyone has any idea?
23:14 JoeJulian kminooie: What version?
23:14 kminooie also i was wondering? how can add a brick to a volume that consist of only one brick with replica factor of 1, in a way that the replica farctor will be 2
23:14 kminooie 3.3
23:15 blendedbychris semiosis: that didn't work :(
23:15 semiosis blendedbychris: what are you trying to do, again?
23:15 * semiosis confused
23:17 kminooie in regard to the stale issue i have to say that i can overcome the issue by umounting and mounting again. but it keeps happening everyonce in a while and want to prevent it from happening at all
23:17 JoeJulian kminooie: I cannot find that error message. Is that a quote or are you paraphrasing?
23:17 semiosis JoeJulian: stale nfs filehandler
23:17 semiosis -r
23:17 semiosis guessing
23:17 kminooie it is not a quote and it is an nfs error message not gluster
23:18 blendedbychris semiosis: okay woo I got somewhere… I was able to probe the existing (running) server.
23:18 blendedbychris oh wait
23:18 * semiosis waits
23:19 blendedbychris ya volume info is missing and running "gluster volume sld-wowza-2 all"  returns volume sync: unsuccessful
23:19 blendedbychris i have the peers set
23:19 blendedbychris http://pastie.org/private/nrtklqyrt4zp0bltolqsiw
23:19 glusterbot <http://goo.gl/5vDhs> (at pastie.org)
23:20 semiosis restart glusterd on both of these servers
23:20 blendedbychris http://pastie.org/private/hfjjt8p72xbrp7bpvz72q
23:20 glusterbot Title: Private Paste - Pastie (at pastie.org)
23:21 blendedbychris same issue
23:21 blendedbychris volume info is missing
23:21 blendedbychris can i not just delete all the peers and start over?
23:22 semiosis fine by me
23:22 * semiosis &
23:23 JoeJulian kminooie: My memory is way into swap space. In order to successfully refresh it from cache, I need keywords that actually produce results. The exact problem you're encountering is the best way to make that happen.
23:24 hattenator joined #gluster
23:24 kminooie give me a second
23:26 blendedbychris semiosis: got it...
23:29 plantain_ joined #gluster
23:29 plantain_ joined #gluster
23:32 blendedbychris if this thing says Status: Brick is Not connected … what gives? the volume is there
23:32 blendedbychris (underlying that is)
23:33 kminooie so this a sample of the error i get on my client machine:
23:33 kminooie [alan@alan-pc storage]$ ll
23:33 kminooie ls: cannot access adsimages: Stale NFS file handle
23:33 kminooie total 0
23:33 kminooie d????????? ? ? ? ?            ? adsimages
23:33 kminooie [alan@alan-pc storage]$
23:34 kminooie i can't find anything related to this on my log file (both system logs and gluster logs (on the server s) )
23:35 JoeJulian And you actually are mounting via nfs... that's a little different.
23:36 kminooie most often it happens right after I write something in the root of the mount point , but it can happen in other situations as well such as after a mount point has been mounted for a long time ( a few hours)
23:36 JoeJulian If there's an error, I expect it to be in /var/log/messages or on the server you nfs mount from in /var/log/glusterfs/nfs.log
23:36 semiosis kminooie: is there NAT between your nfs client & server?
23:36 kminooie but unfortunately I can't pin point what exactly causes this , and that is actually one of my questions
23:37 kminooie no
23:37 kminooie they are all even on the same switch for that matter
23:39 kminooie as i said I can easily unmount and mount and everything would be fine. but in my production environment I don't have that luxury
23:39 JoeJulian Yeah, mount -o remount would work too, but I do hear what you're saying.
23:39 JoeJulian Shouldn't be necessary at all.
23:40 JoeJulian Oh, I wonder...
23:40 kminooie if it helps in any way, I sat up the volume using the command line and didn't touch any of the config file manually at all
23:41 JoeJulian kminooie: Try setting "gluster volume set $vol nfs.enable-ino32 on"
23:41 kminooie also most of the time it happens right after some sort of write operation on that mount point
23:42 kminooie ok . $vol would be the volume name right?
23:42 JoeJulian "Stale file handles occur when a file or directory was held open by an NFS client, and then was either removed, renamed, or replaced. For example a file gets removed and a new file is created using the same inode, or if the file was renamed and the inode changed." The same would be true if you received a 64 bit inode but the kernel cache truncated it to 32 bit.
23:42 JoeJulian yes
23:42 kminooie do  I have to remount after that command?
23:42 JoeJulian no
23:43 JoeJulian kminooie: Out of curiosity, what do you use for your brick filesystem?
23:44 kminooie well i wouldn't what inode is being used but in my experience it happens most often after i just create a new file or a new directory (not removing or replacing anything)
23:44 kminooie ext4
23:44 JoeJulian kminooie: That may be a problem as well. See ,,(ext4)
23:44 glusterbot kminooie: Read about the ext4 problem at http://goo.gl/PEBQU
23:45 blendedbychris Do you guys usually hit drive speed limitations before network speed?
23:45 JoeJulian I wish.
23:45 blendedbychris I have a 2GigE port and feel like I am getting the same performance with 1GigE
23:46 semiosis blendedbychris: there's other explanations for that
23:46 blendedbychris 565Mb/s sound slow?
23:47 TSM check yr iostat
23:47 blendedbychris am i looking at waits?
23:47 TSM whn you say 2G port are you bonding 2 1G
23:47 kminooie gluster> volume set adsimages nfs.enable-ino32 on
23:47 blendedbychris bonding
23:47 kminooie Set volume successful
23:47 kminooie ok will see how it works :)
23:48 blendedbychris TSM: http://pastie.org/private/i7iwdwvyfa6kfndxpzekew
23:48 glusterbot <http://goo.gl/nTkCZ> (at pastie.org)
23:48 TSM look at util and waits
23:48 JoeJulian Please let me know how it turns out kminooie
23:48 TSM iostat -x i meant
23:48 kminooie sure how about adding brick and increasing the replica to 2?
23:49 nightwalk joined #gluster
23:49 JoeJulian Yep, just see gluster volume help add-brick
23:49 blendedbychris http://pastie.org/private/o7rio4bc4dgim0lfqdmq0g
23:49 elyograg A hitachi 1TB enterprise drive has a sustained transfer rate in excess of what a single gigabit network link can do.
23:49 glusterbot <http://goo.gl/Gz9Cu> (at pastie.org)
23:49 blendedbychris one of these things i don't get
23:49 blendedbychris this is a RAID0 1TB volume
23:50 TSM not on random io elyograd
23:50 elyograg gluster's performance bottleneck is in metadata operations, not reading/writing the actual data.
23:50 semiosis blendedbychris: what kind of app/use case are you running?
23:51 blendedbychris semiosis: we share humongo mp4 files that we stream
23:51 blendedbychris basically it's at transcoder + media streamer
23:52 TSM so metadata is not the bottleneck here
23:52 blendedbychris I'm just curious if I need to spend the extra money on the 2GigE port
23:52 blendedbychris it's like 40 bucks
23:52 TSM what are you talking about 2G port
23:52 blendedbychris sorry
23:52 blendedbychris interface
23:53 TSM so is it a bond?
23:53 blendedbychris yes (I thought i said that)
23:53 blendedbychris TSM: did that iostat look normal?
23:53 blendedbychris that's one tool i don't nderstand
23:53 TSM what setup are you running on your bond
23:53 semiosis istr something about a kind of bond that worked by src/dst mac pairs, so traffic between two systems would always go over one link or the other
23:53 semiosis idk much about bonding though
23:55 TSM ive seen very poor performance when lag not setup correctly and/or using alb somtimes, finiky with switches
23:55 blendedbychris I dunno to be honest beyond maybe the nic
23:55 blendedbychris i didn't set up the bond
23:55 TSM why not go back to simples, one port, check speeds
23:55 TSM you can also use nc to test netowrk speeds
23:56 blendedbychris Craps already in production :P
23:56 blendedbychris that iostat didn't look off did it?
23:56 blendedbychris sdb btw
23:57 TSM is that under full load?
23:59 blendedbychris iperf gets me [  3]  0.0-10.0 sec  1.10 GBytes   944 Mbits/sec

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary