Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-28

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 nueces joined #gluster
00:30 designbybeck_ joined #gluster
00:37 chirino joined #gluster
00:46 Humble joined #gluster
01:32 saz joined #gluster
01:35 manik joined #gluster
01:35 glusterbot New news from newglusterbugs: [Bug 880157] libgfapi - samba integration <http://goo.gl/GkTQw>
01:41 zwu joined #gluster
01:41 kevein joined #gluster
01:41 * m0zes really wants to play with 40Gb connectivity for glusterfs. http://www.colfaxdirect.com/store/pc/vie​wPrd.asp?idproduct=1376&amp;idcategory=7 http://www.colfaxdirect.com/store/pc/vie​wPrd.asp?idproduct=1326&amp;idcategory=6
01:41 glusterbot <http://goo.gl/6xooP> (at www.colfaxdirect.com)
01:42 m0zes honestly not bad for either.
01:42 m0zes price that is. 36 port 40Gb switch for $16,000 and nics at $500 each.
01:53 zhashuyu joined #gluster
01:54 joaquim__ joined #gluster
01:58 yosafbridge joined #gluster
01:58 mohankumar joined #gluster
02:09 sunus joined #gluster
02:20 robo joined #gluster
02:21 lee_ joined #gluster
02:53 bharata joined #gluster
02:59 bulde1 joined #gluster
03:05 saz joined #gluster
03:10 sgowda joined #gluster
03:36 bulde joined #gluster
03:46 Daxxial_ joined #gluster
03:51 vicelow joined #gluster
03:52 vicelow someone knows why I are getting this error "have overlapping export directories from the same peer"
03:56 Tobarja joined #gluster
04:21 bulde joined #gluster
04:25 zhashuyu joined #gluster
04:34 atrius joined #gluster
04:36 sripathi joined #gluster
04:50 hagarth joined #gluster
04:55 ankit9 joined #gluster
05:01 lee_ joined #gluster
05:05 sripathi joined #gluster
05:36 raghu joined #gluster
05:45 ramkrsna joined #gluster
05:52 sripathi joined #gluster
06:05 lee_ joined #gluster
06:07 mohankumar joined #gluster
06:14 vijaykumar joined #gluster
06:24 Humble joined #gluster
06:28 JoeJulian elyograg: Yep, that was the expert advice you gave ChiTo.
06:29 ankit9 joined #gluster
06:30 sripathi joined #gluster
06:33 Bullardo joined #gluster
06:36 puebele joined #gluster
06:40 bala1 joined #gluster
06:50 Bullardo joined #gluster
06:51 sripathi joined #gluster
06:55 rgustafs joined #gluster
07:02 vijaykumar joined #gluster
07:04 ngoswami joined #gluster
07:12 vijaykumar joined #gluster
07:18 dobber joined #gluster
07:23 mooperd joined #gluster
07:24 layer3switch joined #gluster
07:28 rudimeyer_ joined #gluster
07:40 lee_ joined #gluster
07:44 lkoranda joined #gluster
07:51 y4m4 joined #gluster
08:12 deepakcs joined #gluster
08:18 overclk joined #gluster
08:19 rascasoft joined #gluster
08:22 zhashuyu joined #gluster
08:24 ctria joined #gluster
08:31 andreask joined #gluster
08:34 gbrand_ joined #gluster
08:34 andreask joined #gluster
08:35 ramkrsna joined #gluster
08:48 duerF joined #gluster
08:50 Humble joined #gluster
08:51 hagarth joined #gluster
08:58 Humble joined #gluster
09:04 toruonu joined #gluster
09:05 toruonu I'm trying to figure out why one particular process is running extremely slow on gluster. And under extremely slow I mean that a comparable process on local disk used to take about 1/10th of the time
09:06 toruonu when I run profiling I see that majority of bricks spend 90+% of the latency on LOOKUP
09:06 toruonu 90.79     369.75 us      31.00 us   83324.00 us          29340      LOOKUP
09:06 raghu left #gluster
09:06 toruonu with only a minor part going to READDIR, CREATE and READDIRP
09:07 toruonu ca 1-2% each
09:07 JoeJulian ~pasteinfo | toruonu
09:07 glusterbot toruonu: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
09:07 toruonu volume info:
09:07 toruonu http://fpaste.org/RO6k/
09:07 glusterbot Title: Viewing Volume Name: home0 Type: Distributed ... 2.168.1.245:/d35 Brick6: 192.168.1.2 ... rement: on cluster.quorum-type: auto (at fpaste.org)
09:07 toruonu your fellow 4x3-er :P
09:08 JoeJulian Ah, yes...
09:08 toruonu I mean I can take a 30-40% performance hit at the advantage of shared home, but this is ridiculously slow
09:08 JoeJulian Well, that's either self-heal checks and/or dht misses.
09:09 toruonu I used to be able to run on a 10-task system overnight 8-10 babysit scripts (that did status, get, resubmit) + plotting. Right now it's 80% done of the first babysit run that I started ca 12h ago
09:10 toruonu grr… sometimes I hate fpaste
09:10 toruonu I had a nice 10 minute profiling in the interval and ran the full profile info into fpaste only to get a 500 server error
09:10 toruonu and now I'll have to wait a long while because it's in the process of downloading output which is fully dominated by network speed from US to EU
09:11 toruonu though I could probably do an independent crab -status and profile that
09:13 toruonu I'm assuming that this task is so slow because of the SQLite DB file that is used heavily.. sad part is that I cannot make it move elsewhere either, it has to be in the task directory
09:13 JoeJulian What's this crab thing?
09:13 toruonu it's a toolkit to send jobs to grid to analyze physics data
09:14 toruonu ok, this is one crab -status request profile: http://fpaste.org/tfkZ/
09:14 glusterbot Title: Viewing Brick: 192.168.1.241:/d35 ---------- ... Duration: 98 seconds Data Read: 6964 ... 72 bytes Data Written: 6666719 bytes (at fpaste.org)
09:14 toruonu actually seems that profiling this particular task shows that lookup is reduced to 70% of latency and reading directory properties comes up a lot
09:15 mooperd joined #gluster
09:15 toruonu though can't be 100% sure which bricks it hit most … how can I find out easily which bricks a particular file is on without looking through all the bricks with ls
09:15 mooperd joined #gluster
09:16 JoeJulian Damn, that's a lot of lookups.
09:16 toruonu the overnight profile had lookups at 100% latency
09:16 toruonu for almost all bricks
09:17 JoeJulian Would probably be useful to analyze an strace (not that I want to at 1am...)
09:18 toruonu creating it just in case anyway :P
09:18 toruonu strace -o crab.strace -f cmd enough I guess?
09:18 JoeJulian yeah
09:19 toruonu overall it's python code that executes some system commands for submission/retrieval/status
09:20 toruonu but the basic -status should be that it hits the SQLite DB for job information, collects it in and then for those job states that are not in done runs the status check… howeer it takes a good 30s before it even gets to status checks
09:20 ankit9 joined #gluster
09:20 JoeJulian Ah, so it's all sqlite related then?
09:20 toruonu likely … I'm not an author of the code so can't be 100% it doesn't do something else as well in that time
09:21 toruonu but for a pure -status request I know that on local disk for 100% done task it runs real fast as it doesn't need to go outside and if it's got to check it'll take time depending on how long it takes to check all jobs
09:21 toruonu ok, got the trace
09:21 webwurst joined #gluster
09:21 toruonu does fpaste take 3.3MB pastes? :D or should I post it some other way
09:22 bauruine joined #gluster
09:22 JoeJulian probably some other way....
09:22 toruonu well managed to execute already so let's see if it eats it :D
09:22 toruonu haha WARNING: your paste size (4101.3KiB) is very large and may be rejected by the server. A pastebin is NOT a file hosting service!
09:23 toruonu JoeJulian: sent you the link to strace in private message
09:25 JoeJulian Wow, that's a lot of misses. See my blog about DHT misses
09:26 JoeJulian There's an option you can set that might actually make this fast.... I forget what it is though off the top of my head, and I'm too tired to go look.
09:27 toruonu this one: http://joejulian.name/blog​/dht-misses-are-expensive/
09:27 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
09:28 JoeJulian yeah
09:30 toruonu I guess you mean this part: Finally, you can set lookup-unhashed off. This will cause distribute to assume that if the file or linkfile cannot be found using the hash, the file doesn't exist.
09:30 JoeJulian Yeah, that's the ticket.
09:30 JoeJulian According to avati, that should work.
09:30 toruonu ok, gonna act the dumb user now, but … how do I set it :D gluster volume set home0 x=y?
09:31 JoeJulian gluster volume set $vol lookup-unhashed off
09:31 toruonu because the odd thing I found is that unlike most filesystems I wasn't able to find the current values for all parameters
09:31 toruonu gluster volume info only shows changes with regard to default
09:31 sripathi joined #gluster
09:31 JoeJulian gluster volume set help
09:32 toruonu ah, so that's how I see what defaults are
09:32 JoeJulian but it only shows documented settings. I'd be surprised if that setting wasn't undocumented.
09:32 toruonu [root@se1 ~]# gluster volume set help|grep -A 3 unhash
09:32 toruonu [root@se1 ~]#
09:32 toruonu :)
09:33 toruonu btw what's the overhead for profiling?
09:33 JoeJulian Never checked.
09:33 toruonu I turned it off now to retest how crab works now, but I guess it has some
09:34 toruonu ah indeed the code is tons faster
09:34 toruonu :)
09:34 JoeJulian excellent.
09:34 toruonu finished the status in ca 10-15 seconds :) previously it hadn't even outputted its version by that time :D
09:35 JoeJulian I'd be very careful with checking your bricks after renames and stuff like that though. I haven't heard of anybody testing this yet.
09:35 * toruonu is feeling like christmas has come early :)
09:35 JoeJulian If it works, though, I hope you're a blogger. :)
09:35 toruonu haha :) am not though I have thought about it many times as I run a lot of exotic configurations on a shitload of hardware
09:36 JoeJulian You should. I started after I realized that everything I ever search for is on rackerhacker's blog. I learn a lot writing articles.
09:36 toruonu so what's the possible risk with this unhashed lookup turned off? files that do not have a hash will get claimed not to exist
09:36 JoeJulian ... and put what I know into better perspective in my own mind as well.
09:37 JoeJulian Files that aren't on the brick they hash out to and don't have a dht.linkto won't exist.
09:37 toruonu well I know from experience that the best way to learn about something is teach a course of it to people who know nothing of it … you really have to work through the internals to be able to answer all questions :D
09:37 toruonu how could such files happen?
09:37 toruonu shouldn't they get a hash for any create/move operation?
09:37 JoeJulian They shouldn't. Renaming files is probably the most likely candidate though.
09:37 toruonu not at home with glusterfs yet so asking probably a load of idiot questions
09:38 JoeJulian Not at all.
09:38 Alpinist joined #gluster
09:38 JoeJulian You're not profiling with dd and expecting native performance, so you're already on my nice list.
09:39 glusterbot New news from newglusterbugs: [Bug 880948] Segfault in write-behind xlator when installing a distro on VM using QEMU-GlusterFS <http://goo.gl/wSNF8>
09:39 JoeJulian wow, that's a cool bug to have... :D
09:40 toruonu oh I did run dd, but traced that to various parallel streams to see how it distributes the load and was quite satisfied with the results
09:41 toruonu on a 4x3 I got dd writes of 210MB/s when doing 4+ streams
09:41 lee_ joined #gluster
09:42 toruonu which means about 50MB/s on average per theoretical physical drive while getting 3-way replication and a 100% posix compatible FS that's shared across the board…
09:42 toruonu that was good enough for me :)
09:42 toruonu I was more worried about the slowness of crab, but we figured that out today
09:42 JoeJulian Ah, I see you use ,,(Joe's performance metric)
09:42 glusterbot nobody complains.
09:44 toruonu :)
09:45 toruonu well previously the home was on a single drive and with 2+ people writing on it at a time you'd be hard pressed to get more than 50-60MB/s out of it anyway … so pretty close to that with loads of added benefits :)
09:50 toruonu any tips or tricks for improvements in speed also for larger code compilation / git operation on glusterfs
09:50 toruonu that's the only other side of grudging comments I get from users
09:51 toruonu that their code compilation takes longer now (guessing possibly some small improvement there from the dht miss fix) and that git commands were very slow
09:51 toruonu as they have loads of small operations
09:53 sripathi joined #gluster
09:54 toruonu a user is claiming that scram build (it's again a custom tool to include packages as needed, kind of make like) takes 2 minutes on local disk and when run at the same time on glusterfs it was only at first lines when the other had already completed… it's still running and he's waiting to see how much longer it takes, but esimate was a good many times longer
09:56 JoeJulian Just a wag, but if this lookup-unhashed setting makes that much difference, I'd be interested in seeing if that's the same problem.
09:57 toruonu well he runs it now after we did the lookup fix
09:57 JoeJulian Ah
09:57 toruonu 9.5 minutes vs 2.25 minutes
09:57 toruonu same command
09:58 JoeJulian Probably not that surprising then. Self-heal checks I would suspect.
09:58 Azrael808 joined #gluster
09:58 JoeJulian Maybe try setting client-io-threads on and see if that helps? Not sure if that build tool is multi-threaded...
09:59 toruonu hm let me ask how he ran it, usually we do run scram b -j 9 or smth
09:59 JoeJulian Or is it client.io-threads... guess I'll look that one up...
10:00 JoeJulian gluster volume set <volname> client-io-threads on/off will load/remove the io-threads xlator in the client volfile.
10:01 toruonu btw that's another undocumented option :p
10:01 toruonu he didn't use -j option so probably wasn't multithreaded
10:02 JoeJulian They're undocumented until they've received sufficient qa testing. Client-side io-threads had a lot of user testing though so afaict it's just a technicality.
10:03 toruonu indeed I can confirm that this is very slow… considering that I've not changed any code the scram command should run through in 10s or so after checking all directories, but it's not even started displaying output after 20 seconds
10:03 toruonu running an strace with it to see what the hell it's doing
10:04 JoeJulian readdir/stat probably
10:05 toruonu still no output … no a good minute has passed
10:05 toruonu it should have started at least claiming what it's doing
10:07 sripathi joined #gluster
10:08 toruonu ok, now I'm getting worried … it's still not started to ouput anything …
10:10 toruonu *sigh* … it's stating over the thousands of files in my CRAB directories … something it really ought not to do…
10:10 toruonu it's the ls all over again
10:11 toruonu is there any way to make the stat for all files happen faster? anything in the future plans etc?
10:13 JoeJulian I know there's progress being made. Patches submitted, etc. Not sure where those are for the 3.4 release though.
10:14 JoeJulian If it's stating the same files more than once, keeping the directory open will allow it to cache.
10:15 JoeJulian I understand both sides of this stat caching conundrum, but I have no idea where I stand on it.
10:15 toruonu it's basically doing a walk of it's software area so anything below a certain path no matter which submodule you try to compile
10:16 toruonu therefore that walk will include all job directories and their output files (stdout, stderr, output tarball and job report per job, multiply that with average 2000 jobs per task and ca 20 tasks). That alone will take a goodly 10-20 minutes for me
10:16 hagarth joined #gluster
10:16 JoeJulian Which it can't do without triggering a self-heal on every file because it has to find out if each one is a directory.
10:16 toruonu and that's actually got nothing to do with the compilation :( so I'll have to write and ask if I can somehow make scram ignore the crab state directories during the walk
10:17 toruonu any hints how to remedy this in the short term?
10:19 JoeJulian If this is still python we're talking about, and you're using os.path.walk, you could make your function spawn a thread which would stop the walk from blocking.
10:20 JoeJulian disable self-heal checking....
10:21 JoeJulian In theory, but I'm afraid to try it, you could disable self-heal checks. I would make sure quorum was on though. Then the self-heal daemon would be responsible for handling self-heals only. There would be a risk of the client getting stale data if there were an outage though.
10:21 JoeJulian Ok, my eyes are closing on their own. I'm going to go to bed.
10:22 * JoeJulian waves
10:22 toruonu it's not python
10:23 ankit9 joined #gluster
10:23 toruonu ok, how do I turn it off?
10:23 toruonu and then good night :)
10:23 toruonu and I do have quorum on
10:26 toruonu cluster.data-self-heal off ok did that … doesn't seem to help though
10:26 sripathi joined #gluster
10:27 tripoux joined #gluster
10:51 lee_ joined #gluster
11:08 pdurbin joined #gluster
11:09 glusterbot New news from newglusterbugs: [Bug 880993] XML tags are improperly nested or incorrectly closed for volume top commands in some cases <http://goo.gl/zBQfV>
11:13 pdurbin jdarcy: when you have a moment, can you please look at this bug my team member just filed against an app (ROOT) that's misbehaving on gluster: https://savannah.cern.ch/bugs/index.php?99118
11:13 glusterbot Title: ROOT - Bugs: bug #99118, system call return value / errno... [LCG Savannah] (at savannah.cern.ch)
11:15 pdurbin johnmark: you too, please. we can all chat about this when we get together next week. we just wanted to put this on your radar
11:16 syoyo joined #gluster
11:19 ndevos pdurbin: sounds very much like you're interested in bug 850352
11:19 glusterbot Bug http://goo.gl/9KbJb medium, medium, ---, ndevos, MODIFIED , 32bit support in Fuse, related to special option nfs.enable-ino32
11:23 pdurbin ndevos: thanks! i passed it along to my team
11:23 ndevos pdurbin: cool, let me know if you have any issues with it, recent glusterfs packages should support the enable-ino32 mount option already
11:26 pdurbin ndevos: great. i think we're running 3.3
11:28 ndevos pdurbin: you can check the /sbin/mount.glusterfs script and see if it contains that option, you may need 3.3.1
11:34 pdurbin ndevos: hmm. ok. can't right now but thanks
11:35 ndevos pdurbin: sure, I'm just giving some hints :)
11:38 hagarth joined #gluster
11:51 Humble joined #gluster
11:51 manik joined #gluster
11:51 glusterbot New news from resolvedglusterbugs: [Bug 845213] [FEAT] Need O_DIRECT support in translators <http://goo.gl/IFZ1z>
11:59 lee_ joined #gluster
12:11 abyss^_ joined #gluster
12:16 guigui3 joined #gluster
12:22 kkeithley joined #gluster
12:22 kkeithley left #gluster
12:22 kkeithley joined #gluster
12:26 toruonu so the directory walk speed depends on what?
12:27 toruonu is there any way I can speed that up?
12:27 toruonu I'm guessing adding bricks will actually NOT help because if the walk does a stat on most bricks, then that means more bricks means more latency
12:27 toruonu or...
12:29 kkeithley by "directory walk" do you mean /bin/ls? On linux ls is often aliased to do color listings, and that's what is doing the stat on each file
12:35 designbybeck_ joined #gluster
12:39 glusterbot New news from newglusterbugs: [Bug 881013] dht does not handle the errors properly and crashes. <http://goo.gl/D2lPP>
12:40 shireesh joined #gluster
12:45 toruonu kkeithley: no I mean software that traverses directories and checks the content
12:46 kkeithley It's the stat on every file that kills tree walks. Each stat triggers a self heal check, and that's what makes things slow.
12:46 toruonu we have scram, which is kind of a software management and compilation toolkit that checks the src/ directory of the project for changes and to do that it runs through all directories. I see loads and loads of stat() calls to files, basically every file in every folder is quickly checked
12:46 toruonu well I disabled the self heal
12:46 toruonu and it's not better
12:47 toruonu got 3 changes already for speeding stuff up, but except for the lookup-unhashed change the others don't seem to change anything
12:47 toruonu cluster.data-self-heal: off
12:47 toruonu performance.client-io-threads: on
12:47 toruonu cluster.lookup-unhashed: off
12:47 kkeithley I'd have to check; I'm not sure if disabling self heal stops the self heal check.
12:47 kkeithley Maybe someone else knows the answer?
12:48 toruonu that's the point … right now glusterfs is quite unusable for everyday work with our tools … we really really need the stat to improve in latency
12:48 toruonu any ideas on how to achieve that are welcome … even throwing hardware at it
12:50 edward1 joined #gluster
13:03 vicelow joined #gluster
13:04 vicelow hello all, someone knows why this error "have overlapping export directories from the same peer" is?
13:15 balunasj joined #gluster
13:16 toruonu googled around and came upon a performance tune called: performance/stat-prefetch
13:17 toruonu anyone got a quick intro to that and if that could speed up the stat issue I have?
13:19 deckid joined #gluster
13:22 bauruine joined #gluster
13:24 puebele1 joined #gluster
13:28 grzany joined #gluster
13:30 kkeithley looks to me like performance/stat-prefetch morphed into performance/md-cache, and a quick look at the source suggests that stat-prefetch is enabled by default.
13:30 kkeithley /suggests/suggests to me/
13:30 kkeithley s/suggests/suggests to me/
13:30 glusterbot What kkeithley meant to say was: looks to me like performance/stat-prefetch morphed into performance/md-cache, and a quick look at the source suggests to me that stat-prefetch is enabled by default.
13:31 guigui3 joined #gluster
13:31 aliguori joined #gluster
13:31 toruonu interesting … but then not helpful...
13:35 shireesh joined #gluster
13:41 kkeithley and then md-cache is on the client side usually. So if it's issuing batches of stats (just guessing here) then it's not going to help on the server side (guessing again)
13:42 mooperd kkeithley: hi
13:42 andreask joined #gluster
13:43 toruonu it really would help if the self heal was detached from the stat check
13:43 toruonu somehow
13:43 toruonu or detachable
13:43 rascasoft joined #gluster
13:43 mooperd curl -v -X PUT -T hello -H 'X-Auth-Token: AUTH_tk9727d74767824b359fc35168e3b86b4c' -H 'Content-Length: 5242880' https://cheese25:443/v1/AUT​H_gv0/free_palestine/hello -k
13:43 mooperd gives me a 401 Unauthorized
13:44 mooperd curl -v -X PUT -H 'X-Auth-Token: AUTH_tk9727d74767824b359fc35168e3b86b4c' https://cheese25:443/v1/AUTH_gv0/free_palestine_8 -k
13:45 mooperd 201 Created - Works perfectly however
13:45 mooperd am I missing something?
13:49 kkeithley you tried to put the file (in a container) before you created the container?
13:50 mooperd putting the file in a container that doesnt exist?
13:51 Alpinist joined #gluster
13:51 mooperd kkeithley: same..401
13:52 mooperd The containers are there on the filesystem
13:53 guigui3 joined #gluster
13:55 kkeithley and memcached is running?
13:56 mooperd kkeithley: yep
13:56 mooperd kkeithley: I can create as many containers as I like
13:56 mooperd that works fine
14:03 mooperd localhost:gv0 on /mnt/gluster-object/AUTH_gv0 type fuse.glusterfs (rw,default_permissions,al​low_other,max_read=131072)
14:04 mooperd actually. I cannot create files in /mnt/gluster-object/AUTH_gv0/$container
14:04 robo joined #gluster
14:04 mooperd kkeithley: Im not sure if I should be able to however
14:04 JoeJulian selinux?
14:05 kkeithley I think you should
14:05 mooperd touch does not return an errror
14:05 mooperd but does not create a file
14:07 mooperd linking is broken…?
14:07 mooperd dd does the same
14:10 toruonu ok, does anyone how to optimize the OS library lookup in general… I just checked why one command ran extremely long (half an hour instead of 2 minutes) and from what I can tell in the strace output the majority of time is spent in doing open and stat syscalls:
14:10 toruonu 77886 stat
14:10 toruonu 131959 open
14:10 toruonu of which most fail because the lookup is against a crap location:
14:10 toruonu $ grep ENOENT scram.strace |wc -l
14:10 toruonu 177385
14:11 tqrst anyone here use the hadoop connector? I'm wondering if it's stable, and if it works well.
14:14 mooperd kkeithley: I have just mounted gluster on another machine and it also cannot create files
14:14 tqrst http://gluster.org/community/d​ocumentation/index.php/Hadoop says it will be available in "3.3 beta"
14:14 glusterbot <http://goo.gl/m2Zix> (at gluster.org)
14:15 mooperd tqrst: you will have to test it I think
14:16 bitsweat joined #gluster
14:21 theron joined #gluster
14:23 JoeJulian toruonu: My understanding is that ld.so.cache is supposed to mitigate that. Try adding your library paths to /etc/ld.so.conf (or /etc/ld.so.conf.d/mylibraries.conf) and running ldconfig to rebuild the cache maybe?
14:23 * JoeJulian resolved his crisis and is heading back to bed.
14:26 mooperd Anyone know why I can create directories but not files....?
14:33 vijaykumar left #gluster
14:40 glusterbot New news from newglusterbugs: [Bug 881062] Brick processes crash when an attempt to mount a volume is made by an unauthorized client <http://goo.gl/JjgKX>
14:40 mooperd @ext4
14:40 glusterbot mooperd: Read about the ext4 problem at http://goo.gl/PEBQU
14:40 tqrst mooperd: darn
14:42 rudimeyer_ joined #gluster
14:44 mooperd oo
14:44 mooperd o_O
14:45 kkeithley mooperd: am I reading that correctly, you think you have the ext4 problem?
14:45 mooperd kkeithley: maybe. Im going to redo with xfs
14:46 * mooperd wants gluster to work with zfs
14:46 nueces joined #gluster
14:47 kkeithley Get Oracle to change their license and it could happen.
14:47 rwheeler joined #gluster
14:47 mooperd kkeithley: that is going to happen soon
14:47 Tobarja left #gluster
14:48 kkeithley Do they ship zfs in OEL?
14:48 mooperd kkeithley: dont think so
14:48 toruonu yes ZFS would be nice …
14:48 mooperd Lawrence Livermore National Laboratory is working on ZFS for linux
14:48 toruonu kernel level zfs
14:49 mooperd http://zfsonlinux.org/zfs-disclaimer.html
14:50 stopbit joined #gluster
14:50 mooperd The CCDL _IS_ an open source licence
14:50 NuxRo @ppa
14:50 glusterbot NuxRo: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
14:50 dberry joined #gluster
14:51 dberry joined #gluster
14:54 johnmark mooperd: barely :)
14:56 toruonu can anyone recommend a good profiling tool… I'd basically want to know what amount of time is spent doing what… are the stat's with ENOENT the culprit or something else… basically I'd like to know per syscall time consumed on average as well as in sum
14:56 johnmark mooperd: from http://en.wikipedia.org/wiki/Common_​Development_and_Distribution_License
14:56 glusterbot <http://goo.gl/rvx6o> (at en.wikipedia.org)
14:57 mooperd johnmark: Yep. I read it
14:57 jdarcy pdurbin: Just saw that, I'll look at the BZ in a second.
14:58 pdurbin jdarcy: thanks!
14:58 torbjorn1_ toruonu: Check out the "-c" argument to strace
14:58 pdurbin jdarcy: internally, we have a tiny bit of C to reproduce the behavior
14:58 jdarcy pdurbin: Wow, I tripped over exactly the same problem using Isilon systems at LASP.
14:59 pdurbin @lucky LASP
14:59 glusterbot pdurbin: http://lasp.colorado.edu/
15:00 chirino joined #gluster
15:00 jdarcy Yep, that one.  Same one where cosmic rays caused memory errors, and thin air caused inadequate cooling of the power supplies resulting in voltage sags *five minutes after I left the building*.
15:00 pdurbin heh
15:01 jdarcy I thought we had a 32-bit-ino option buried somewhere.  Let me look.
15:03 jdarcy Interesting.  We have an enable-ino32 option for NFS, but there are also references in non-NFS code.
15:03 jdarcy pdurbin: BTW, I assume you're one of the people I'll be meeting with next week . . . ?
15:04 ndevos jdarcy: that enable-ino32 mount option for fuse was introduced for bug 850352
15:04 glusterbot Bug http://goo.gl/9KbJb medium, medium, ---, ndevos, MODIFIED , 32bit support in Fuse, related to special option nfs.enable-ino32
15:04 mooperd \/dev/vdb on /vdb type xfs (rw)
15:04 mooperd is ok?
15:04 mooperd or you need some other special options?
15:05 jdarcy ndevos: There seem to be references in NFS, FUSE, and glusterfsd itself.  Still trying to figure out how they're related or different.
15:06 vicelow I couldn't mount a volume, could you please helpme?
15:06 pdurbin jdarcy: well... about that... i'm happy to attend but that's actually my last week in my current department... i'm moving to a new job within harvard but it's unlikely i'll do much with gluster in that new job: https://twitter.com/philipdur​bin/status/273229031082692608
15:06 glusterbot <http://goo.gl/JTSHf> (at twitter.com)
15:06 toruonu Hmm… according to strace -c the command that executed and took 2 minutes to complete didn't use that much time:
15:06 toruonu http://fpaste.org/O6wT/
15:06 glusterbot Title: Viewing % time seconds usecs/call calls erro ... n 0.00 0.000000 0 33 rt_sigprocmask ... ------- 100.00 1.640080 241 90 total (at fpaste.org)
15:06 ndevos jdarcy: the nfs option is per volume, but for fuse it's client-side only
15:06 jdarcy pdurbin: Wow, missed that one.  Congrats!
15:07 ndevos (the Linux kernel nfs-client has a module parameter to force 32-bit inodes)
15:07 toruonu according to that it used wait4 for a good 1.6 seconds and that's the whole time
15:07 pdurbin jdarcy: thanks. like i said in that tweet... super excited :)
15:07 vicelow this is the error that I'm getting http://fpaste.org/FBqL/
15:07 glusterbot Title: Viewing Gluster mount error (at fpaste.org)
15:09 toruonu ok, so according to this the command spends hardly any system time… as it amounts to only 1.6 seconds and most of it is waiting…
15:09 toruonu so I need to know the time spent in userspace
15:11 jdarcy So if you pass "-o enable-ino32" (which doesn't show up in --help) to mount.glusterfs, that will pass the option through to FUSE, which might or might not do the right thing.  ;)
15:11 nightwalk joined #gluster
15:12 * ndevos is pretty sure it should show up in the --help
15:12 pdurbin jdarcy: sounds like something to try. my co-worker was up late last night filing the bug against ROOT. i'll try to get him to pop in here soon-ish
15:12 jdarcy ndevos: Not in master.
15:13 ndevos hmm
15:13 jdarcy pdurbin: Let me check which versions have that.
15:14 jdarcy I only see it in master, not even in v3.3.1
15:15 jdarcy See the change, that is.  Doc still missing even in master.
15:15 NuxRo semiosis: I'm struggling to mount a gluster volume on ubuntu, the mount command returns no error, but nothing gets mounter. sounds familiar?
15:17 kkeithley vicelow: do yoiu have iptables (firewall) running?
15:17 ndevos http://review.gluster.org/3885 has been included on 7 sept, was that after v3.3.1?
15:17 glusterbot Title: Gerrit Code Review (at review.gluster.org)
15:19 ndevos http://review.gluster.org/3955 is needed too, and was merged a little later, 18 septemer
15:19 glusterbot Title: Gerrit Code Review (at review.gluster.org)
15:19 ndevos s,septemer,september,
15:19 jdarcy ndevos: A *ton* of stuff going back almost to the beginning of the year has never been pulled anywhere but master.  :(
15:19 vicelow kkeithley, I think so
15:20 ndevos jdarcy: oh, thats a shame :-/
15:20 kkeithley I expect that's your problem. @ports
15:20 kkeithley @ports
15:20 glusterbot kkeithley: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
15:21 kkeithley vicelow: ^^^
15:22 jdarcy The "winner" seems to be my own 5b176fdb, written on March 12 and merged to master on May 31 but still not in any release branch.
15:24 semiosis NuxRo: not familiar at all
15:25 ndevos oh, wow
15:26 bitsweat left #gluster
15:26 jbautista joined #gluster
15:26 NuxRo gluster volume set volume auth.reject *.*.*.* <- is this the correct way to reject from any IP ?
15:26 toruonu I smell 3.3.2 coming
15:26 toruonu from this discussion :p
15:27 semiosis NuxRo: iirc just *
15:28 NuxRo If I try only * I get:
15:28 NuxRo Usage: volume set <VOLNAME> <KEY> <VALUE>
15:28 NuxRo it replied with "success" after using *.*.*.*
15:28 semiosis ok then
15:29 ankit9 joined #gluster
15:29 nightwalk joined #gluster
15:46 mario_ joined #gluster
15:57 _Bryan_ joined #gluster
16:12 NuxRo hm, so setting a quota on a volume also updates the available size of the remote mountpoint, pretty awesome :D
16:18 chirino joined #gluster
16:30 kkeithley JohnMark: What's the date on www.gluster.org? I just published an article and the byline says November 30, 2012.
16:30 toruonu ok…. overall my performance loss comes still from the ENOENT errors
16:30 dastar joined #gluster
16:31 jbrooks joined #gluster
16:33 toruonu times are in seconds and I'm using grep to find specific lines of the strace with -r -t:
16:33 toruonu Total execution:227.527
16:33 toruonu Execution of ENOENT:181.89
16:33 toruonu Execution of ENOENT on glusterfs volume:156.106
16:37 toruonu so the unhashed-lookup setting to off I guess didn't really work or it's just the expensive stat calls...
16:42 mohankumar joined #gluster
16:44 puebele joined #gluster
16:45 DaveS joined #gluster
17:00 raghu joined #gluster
17:02 puebele joined #gluster
17:05 rascasoft left #gluster
17:21 hagarth joined #gluster
17:29 Humble joined #gluster
17:37 y4m4 joined #gluster
17:49 Bullardo joined #gluster
17:52 jdarcy Red Hat chooses Gluster, *of course* Suse and Ubuntu focus on Ceph.  https://www.suse.com/company/press/2012/11/inkt​ank-partners-with-suse-to-deliver-enterprise-cl​ass-support-for-ceph-storage-in-suse-cloud.html
17:52 glusterbot <http://goo.gl/uBxsG> (at www.suse.com)
17:53 jdarcy Maybe I'll barrage them with build-system patches submitted the wrong way, like they did to us.
17:56 kkeithley Not according to The Register: SUSE joins Canonical and Red Hat in using Ceph to puff OpenStack cloud (http://www.theregister.co.uk/2012/11​/28/suse_linux_cloud_ceph_inktank/)
17:56 glusterbot <http://goo.gl/ty1ur> (at www.theregister.co.uk)
17:57 kkeithley ...  like its peers Canonical and Red Hat, SUSE Linux seems to be favoring the Ceph distributed object store ...
17:58 jdarcy That's El Reg for you.
17:58 m0zes psh no one knows or cares about facts.
17:58 kkeithley indeed
17:59 jdarcy I have to admit, Inktank is beating us badly on the social-media and analyst-schmoozing fronts.
17:59 plarsen joined #gluster
18:00 mnaser joined #gluster
18:08 tqrst if a node goes down in a N x 2 distribute-replicate setup, will the files it held have a new copy created on another node to compensate?
18:08 toruonu ok, I've spent the last 3 hours hacking and slashing at my environment variables and trying to find a way that would work all the time and would be relogin safe (still not 100% done). The whole reason I do this is because of the open/stat syscalls again glusterfs that result in ENOENT take up 90% of the execution time
18:09 toruonu if I manage to rearrange the paths so that all files are found before it hits the gluster mount unless it really had to go there, then execution is 13 seconds. If I do no fiddling so that gluster mount is hit by some 4000 ENOENT calls, then it takes 3.5 minutes
18:09 jdarcy tqrst: No.
18:10 toruonu seriously, can't we get the time spent on those down
18:10 toruonu I'm kind of understanding for actual files and the stat and self-heal logic, but for ENOENT where the file doesn't even exist… why does it take so long?
18:10 jdarcy toruonu: Sounds like you might like negative-lookup.
18:10 vicelow anyone could help me with this log http://fpaste.org/UktO/
18:10 glusterbot Title: Viewing Paste #255691 (at fpaste.org)
18:10 toruonu what is negative lookup?
18:11 jdarcy https://github.com/jdarcy/negative-lookup
18:11 glusterbot Title: jdarcy/negative-lookup · GitHub (at github.com)
18:11 vicelow my iptables on server is http://fpaste.org/VQs3/
18:11 glusterbot Title: Viewing Paste #255692 (at fpaste.org)
18:11 vicelow but I can't mount the volume in client side
18:12 toruonu jdarcy: so this is different from what we debugged with JoeJulian and which culminated in:  <jdarcy> https://github.com/jdarcy/negative-lookup
18:12 glusterbot Title: jdarcy/negative-lookup · GitHub (at github.com)
18:12 toruonu baaah
18:12 toruonu wrong paste :)
18:12 toruonu cluster.lookup-unhashed: off
18:12 toruonu that's what I meant
18:13 toruonu ok … that aside… how does one implement your negative-lookup code? as I see that's a translator, but it goes where :)
18:14 kkeithley vicelow: and portmapper (rpcbind) is running on all the servers?
18:14 kkeithley what does rpcinfo on a server show?
18:15 kkeithley I presume you're using nfs mounts, right?
18:18 jdarcy There's a way to turn on negative-lookup caching in FUSE, but I'd be surprised if it's in a release branch.
18:21 vicelow I'm just testing on vmware virtual machines
18:22 jdarcy NFS will also do negative-lookup caching automatically.
18:23 toruonu so.. maybe I should mount over fuse and then over nfs (I remember there was some doc about something of the kind?) this way getting caching + higher performance?
18:26 jdarcy toruonu: Well, mounting NFS looped back to FUSE on the client *mostly* works, but under high load it can fall prey to a nasty virtual-memory deadlock.
18:26 jbrooks joined #gluster
18:26 jdarcy IIRC, the NFS client tries to push out a page, but to satisfy the request FUSE/GlusterFS needs to allocate one.  Boom.
18:27 toruonu ok … what would be the most reasonable solution to this? I'd rather have a system wide solution than having all users hack their environment …
18:29 kkeithley vicelow: the mount that's failing is an nfs mount? That's the only thing that would need to do a port lookup. What does /usr/sbin/rpcinfo show?
18:30 Technicool joined #gluster
18:31 Bullardo joined #gluster
18:31 kkeithley and you should open 38468 too in your iptables
18:32 jdarcy toruonu: What about running NFS (or CIFS) on the clients to a GlusterFS back end?  You still get the advantage of scalable shared storage, but avoid a lot of the protocol issues around directory listing etc.
18:33 jdarcy toruonu: I'm one of the biggest proponents of the native-client approach, but for the home-directory use case it can be rather painful.
18:33 toruonu so what's the upside/downside of nfs based mount?
18:34 vicelow is ext3
18:34 toruonu in comparison
18:34 vicelow sorry glusterfs mount
18:34 vicelow mount -t glusterfs -o log-level=WARNING,log-file=/var/log/gluster2.log nodo1:/test-volume /mnt/limpet/
18:34 vicelow this line
18:34 jdarcy toruonu: Write latency will be a bit higher, because requests have to go from client to proxy (NFS server / GlusterFS client) to GlusterFS servers.  That also means more load on the network between servers.
18:35 rwheeler joined #gluster
18:35 jdarcy toruonu: OTOH, all forms of caching are likely to be better - both data and metadata.
18:35 jdarcy toruonu: Also, NFS clients can make full use of readdirplus, which FUSE can't do (yet - patches in progress) so even uncached directory listings will be faster.
18:35 toruonu network load is not an issue … everything is on 10G fabric (quite low latency) and the storage nodes each runs a bonded 2-port network
18:36 jdarcy toruonu: The other big negative is that since NFS doesn't have built-in failover, you have to provide some yourself e.g. via UCARP or at least RRDNS.
18:36 vicelow kkeithley, http://fpaste.org/Rr0T/
18:36 glusterbot Title: Viewing rpcinfo (at fpaste.org)
18:37 jdarcy toruonu: The *storage* will fail over by itself just fine, but the protocol-proxy function needs some help.
18:37 toruonu well … I'm contemplating … I've got loads of nodes and I was planning on running each one also as part of glusterfs allocating a directory in some part to gluster as a brick
18:37 toruonu if I were to extend the /home volume with a brick listing from a count of nodes including that of the client nodes
18:37 toruonu then the gluster server would be running on the node
18:37 toruonu and I could do a local NFS mount?
18:37 toruonu getting no need for RRDNS etc
18:38 jdarcy toruonu: You could.  You'd need to watch out for the VM-pressure deadlock I mentioned, but for a lot of people that never actually seems to be a problem.
18:38 toruonu I was anyway planning to do the gluster mounts from local machine when I could run all nodes in gluster
18:38 Bullardo joined #gluster
18:39 jdarcy toruonu: Worst case, *that node* needs a reboot but the rest of the cluster keeps running.  Only you can say whether that's OK for your environment.
18:39 toruonu jdarcy: would that actually be a point where there is no fuse mount on the server, but just the nfs
18:39 toruonu it'd be an NFS local mount, but not over a local fuse mount
18:39 jdarcy toruonu: Each machine would have a GlusterFS native mount, then an NFS mount over that to get caching etc.  Note that in that particular environment the readdirplus point wouldn't apply.
18:40 toruonu nah I'm talking without a native mount … I'm assuming that I can NFS mount from one of my 6 storage servers that are running the bricks and the volume therefore. It could be any one of them. Yet none of them have the volume mounted
18:41 toruonu now what I'd do is make the client node as part of the node serving bricks therefore making it also a possible target in native and NFS mounts
18:41 toruonu except I'd mount it locally over NFS
18:41 toruonu just to be sure that's still affected by VM pressure
18:43 jdarcy toruonu: What do you mean by "client node as part of the node serving bricks..."?  Isn't that client also a server then?
18:44 toruonu yes
18:44 toruonu :)
18:44 toruonu but there would be no native mount
18:45 toruonu so my question was wether this VM pressure deadlock ONLY exists in case of native mount + nfs looped on it or if it's also in case the NFS mount is done locally within the server
18:45 tqrst I'm still getting "[socket.c:1798:socket_event_handler] 0-transport: disconnecting now" exactly every 3 seconds. Any ideas how to get rid of these? (20x2 distributed-replicate volume, nfs.disable=on, nfs.register-with-portmap=off, rest is default)
18:45 jdarcy toruonu: So you're saying that some machines would be GlusterFS servers and NFS clients (potentially talking to themselves) while others would perform the proxy function?
18:46 toruonu kind of
18:47 jdarcy toruonu: The deadlock can happen any time user-space code is involved in handling an NFS page-out requests, so it equally affects the GlusterFS client and server.
18:47 toruonu as an example of what we run. we run a 2PB hadoop installation. Every node is a datanode, yet all of the nodes also mount over fuse the hadoop storage locally. In the case of hadoop there's a central namenode so the mount comes from there, but in the case of glusterfs I understand that every server is also an NFS server
18:48 jdarcy toruonu: In the case where X is an NFS client talking to Y, which as a GlusterFS client goes back to X, then the glusterfsd process on X might need to allocate a page while processing a page-out also on X.
18:48 jdarcy toruonu: I'd say it's less likely than a loopback-NFS mount, but still possible.
18:49 toruonu well I'm guessing it's also only going to happen if there is some serious memory draining
18:49 kkeithley vicelow: are your bricks ext3?
18:49 vicelow joined #gluster
18:49 kkeithley And did you mount them with -o user_xattr?
18:49 jdarcy toruonu: I'd say try it.  If it works then great, if it doesn't then we all learn something.
18:49 toruonu our config is such that every node has 64GB ram and in the client machine case I only give 60GB to the OpenVZ container running the users OS. Therefore there's 4GB for hadoop datanode, gluster mount, nfs or what not
18:50 vicelow If I running a volume with replica 2, is possible to change the volume configuration and obtain a volume without replica? or I need to create a new one
18:51 tqrst (community link to go with my question http://community.gluster.org/q/why-are-m​y-glusterd-logs-getting-spammed-with-soc​ket-c-1798-socket-event-handler-0-transp​ort-disconnecting-now-every-3-seconds/)
18:51 glusterbot <http://goo.gl/nWK3n> (at community.gluster.org)
18:51 jdarcy vicelow: In 3.3.1 (I believe) you can change the replication level as part of adding or removing bricks.
18:51 tc00per left #gluster
18:52 vicelow thank's jdarcy
18:53 * jdarcy has to go do some non-IRC stuff for a while.  ;)
18:54 toruonu there's non-IRC out there?
18:54 tc00per joined #gluster
18:58 tc00per left #gluster
18:59 tc00per joined #gluster
19:01 tc00per left #gluster
19:03 aliguori joined #gluster
19:10 toruonu ok, I have one node where the gluster mount keeps getting stuck
19:10 toruonu it's a native fuse mount
19:11 toruonu right now I cannot do an ls and everything is stuck that has open fd's to the gluster mount
19:11 toruonu the other node that has it mounted seems to work fine
19:12 toruonu it's stuck right now so if someone has ideas to debug this would be nice
19:12 andreask joined #gluster
19:12 toruonu strace seems to only show nanosleep and occasional futex activity
19:15 tc00per joined #gluster
19:16 tc00per left #gluster
19:17 toruonu ok I killed gluterfs mount and am attempting nfs mount, but the mount command just hangs there
19:20 toruonu hmm and it timed out
19:21 toruonu do I need to start the nfs service or just some parts of it for gluster nfs to work?
19:21 * toruonu waves around frantically
19:21 Technicool it on automatically
19:21 toruonu hmm
19:21 toruonu ideas why there's a timeout then? both ends have iptables off
19:21 tqrst jdarcy: it's hard to see anything in wireshark given how much traffic those ports are seeing :\
19:21 Technicool so you should actually not have the kernel nfs running
19:22 toruonu [root@se1 glusterfs]# service nfs status
19:22 Technicool do you have portmap running?
19:22 toruonu rpc.mountd is stopped
19:22 toruonu nfsd is stopped
19:22 toruonu rpc.rquotad is stopped
19:22 toruonu yes
19:22 Technicool or rpcbind
19:22 Technicool were they running when you started glusterd?
19:22 toruonu no clue
19:22 toruonu but I'd assume yes
19:23 Technicool are you using -o vers=3?
19:23 toruonu no
19:23 Technicool try that
19:23 semiosis @nfs
19:23 glusterbot semiosis: To mount via nfs, most distros require the options, tcp,vers=3 -- Also portmapper should be running on the server, and the kernel nfs server (nfsd) should be disabled
19:23 toruonu just basic mount -t nfs 192.168.1.240:/home0 /home
19:23 Technicool what semiosis said better
19:23 toruonu ah indeed
19:24 toruonu now worked
19:24 Technicool semiosis is a genius like that
19:24 semiosis bah
19:24 semiosis just been around the block a few times
19:25 toruonu let's see if that fixes the lockup
19:25 toruonu somehow that node doesn't like glusterfs mount
19:25 mythzib joined #gluster
19:25 mythzib hi
19:25 glusterbot mythzib: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
19:27 mythzib using version 3.3 on Debian, the "cluster.min-free-disk" isn't working. I have 4 data nodes but they are fill ramdomly
19:27 mythzib I try to setup a hard limit (using cluster.min-free-disk 5GB by example)
19:27 mythzib but it's the same
19:28 semiosis what is your use case?
19:28 mythzib I'm testing gluster for the moment
19:28 mythzib for i in `seq -w 1 10`; do dd if=/dev/zero of=`date +%s` bs=1M count=1000; done
19:28 semiosis at best that option could only affect placement of new files, if you have existing files growing that option won't help -- use LVM to expand bricks for that imho
19:28 mythzib i use this command
19:29 semiosis ah
19:29 semiosis yeah, i've heard that option isn't very useful, never tried it myself though
19:29 semiosis do you get messages in the client or brick logs about running out of space?
19:29 mythzib nop
19:30 mythzib on one node I have : (No space left on device)
19:30 Technicool mythzib, what is your setup?  how many nodes, what volume type?
19:30 mythzib but nothing on the other
19:30 semiosis ~pasteinfo | mythzib
19:30 glusterbot mythzib: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:30 semiosis Technicool: pasteinfo :)
19:31 mythzib Technicool: 4 nodes, 1 client. Each have 19GB space
19:31 Technicool semiosis, i already called you a genius, not sure how much higher i can get the bar, sir  ;)
19:31 Technicool mythzib, that only answers half what i asked
19:31 mythzib I'm using vmware
19:31 mythzib hmm?
19:32 mythzib ho
19:32 mythzib distributed
19:33 Technicool use the pasteinfo and add df -h from the four nodes
19:33 Technicool its the first i've heard of that option not working, curious about it
19:34 mythzib http://pastebin.com/dnx5H5WY
19:34 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
19:34 semiosis mythzib: ,,(pasteinfo)
19:34 glusterbot mythzib: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:34 semiosis mythzib: that^^^
19:35 mythzib http://fpaste.org/8ppM/
19:35 glusterbot Title: Viewing Paste #255728 (at fpaste.org)
19:35 semiosis bbl, lunch
19:36 mythzib Technicool: http://fpaste.org/8ppM/ :)
19:36 glusterbot Title: Viewing Paste #255728 (at fpaste.org)
19:36 Technicool mythzib, you wrote 1000 files, correct?
19:36 mythzib about 75
19:36 mythzib each 1GB
19:37 mythzib I'm running the command again to fill up disk
19:37 Technicool what am i missing here then?  i only see < 1GB consumed per node
19:37 JoeJulian ~pasteinfo | mythzib
19:37 glusterbot mythzib: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:37 JoeJulian ~pasteinfo | mythzib
19:37 glusterbot mythzib: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:38 Technicool ^ thats JoeJulian's way of saying we would really also like to see the gluster volume info
19:38 Technicool and glusterbot
19:38 Technicool glusterbot loves gvi output
19:38 mythzib :)
19:38 mythzib I deleted the files
19:38 mythzib recreate it
19:39 Technicool before you run it again, lets see the volume info
19:39 JoeJulian Show me the volume info or I ban... ;) <jk>
19:39 Technicool lol
19:39 mythzib http://fpaste.org/MrI2/
19:39 glusterbot Title: Viewing Paste #255730 (at fpaste.org)
19:39 Technicool he's banned me for less, i wouldn't test him
19:40 JoeJulian You banned yourself.... :P
19:40 Technicool i have to say, thats fair
19:40 johnmark lulz
19:40 * johnmark capturing for posterity
19:41 Technicool so, with this config and the test you are running, it seems like it would be somewhat difficult to pinpoint the issue
19:41 mythzib :(
19:41 Technicool you need to first write a single file, say 15GB
19:41 Technicool that will land on one node
19:42 Technicool then run the test and create 20 1GB files
19:42 kkeithley johnmark: did you check the time on www.gluster.org?
19:42 Technicool or 100 250MB files is probably better
19:42 JoeJulian kkeithley: It's right, why?
19:42 JoeJulian Well, the system time is right.
19:43 vicelow Uuid: 00000000-0000-0000-0000-000000000000 means something wrong right?
19:43 Technicool mythzib, that should show you that the min-free is working
19:43 kkeithley I published a blog article earlier today and the byline says I posted it on November 30
19:44 Technicool kkeithley, impressive....most impressive
19:44 kkeithley vicelow: yes
19:44 kkeithley I do like to think that I'm ahead of my time.
19:44 mythzib Technicool: it's running, let's see
19:44 elyograg it's astounding ... time is fleeting ...
19:44 Technicool i wish i had a delorean to do my blog posts
19:44 Technicool kkeithley, lulz  nice
19:44 mythzib writting ~100MB/s
19:44 kkeithley November 12, 2012 even
19:44 Technicool system time on the main and dl nodes looks correct
19:45 Technicool not sure where the drift would come in
19:45 Technicool do you have a link?
19:45 kkeithley http://www.gluster.org/2012/11​/a-better-ufo-is-coming-soon/
19:45 glusterbot <http://goo.gl/T6o1N> (at www.gluster.org)
19:45 Technicool no wonder you can go forward in time...you have a ufo
19:46 Technicool 404
19:46 semiosis 404
19:46 JoeJulian Ah, right. It's not published yet.
19:46 Technicool 808
19:46 Technicool replicated...
19:46 * Technicool hates myself for that attempted joke
19:46 JoeJulian I make special door hangers I take with me that says "Room Not Found" and take them when I stay in hotels. I hang them on room 404.
19:46 kkeithley huh, I scheduled it to be published on the 30th originally, then I pushed it up to today
19:46 _br_ joined #gluster
19:47 Technicool lol
19:47 * Technicool buys kkeithley a beer
19:48 JoeJulian vicelow: I have no idea why that shows up in logs.
19:48 kkeithley so really, you get a 404?
19:48 JoeJulian vicelow: Is something not working correctly?
19:48 zaitcev joined #gluster
19:48 Technicool kkeithley, yes
19:48 JoeJulian kkeithley: yep. Log out and you'll see it too.
19:49 vicelow JoeJulian, it was my fault Im trying to add another brick right now but I can't
19:49 toruonu yay … I never thought I'd say this, but after moving to NFS mount it's like a breath of fresh air… everything works fast again :p
19:49 _br_ joined #gluster
19:50 mythzib Technicool: 15GB finished... I run an other command to create 100 files 250MB each
19:50 kkeithley hah, published now, but now the byline is November 21, 2012.
19:50 JoeJulian toruonu: Just remember you're dealing with caches now, so you could be looking at stale attributes. Probably won't matter for most things, but always good to know the possibilities.
19:50 mythzib on the data I have the 15GB file, it fill from 2,5GB to 2GB free (my limit)
19:51 mythzib and now fill other data
19:51 Technicool so, if you have just created 100 files, you would see each nodes got about 25 of them, about 2.5GB each if i am caffeinated sufficiently
19:51 JoeJulian Speaking of caffeine, I'll be back shortly.
19:51 Technicool so if you look at the backend of the machine that had the 15GB file, you will only see a handful of the 250mb files
19:51 toruonu JoeJulian: I'll move back to native when you guys get the stat thing under control :) it really is painful, especially the time spent on ENOENT failed calls
19:53 mythzib Technicool I have a lot of this file "---------T  2 root root    0 28 nov.  20:48 1354132123"
19:53 mythzib on the data node where the 15GB file is
19:53 Technicool mythzib, yes
19:53 Technicool those are the pointer files
19:53 mythzib so it's working in fact
19:53 Technicool that confirm that it is working
19:54 Technicool ^^
19:54 mythzib Technicool but this is because my command never stop when I create the file
19:54 mythzib (for i in `seq -w 1 100`; do dd if=/dev/zero of=`date +%s` bs=1M count=250; done)
19:55 mythzib it could be explain the behaviour? because of the "for"
19:55 Technicool mythzib, i don't follow
19:55 Technicool did 100 250MB files finish getting created?
19:56 mythzib yep
19:56 Technicool but the command didn't finish?
19:56 mythzib hmm no. I mean
19:57 _br_ joined #gluster
19:57 mythzib when I create 100GB using one command (for i in `seq -w 1 100`; do dd if=/dev/zero of=`date +%s` bs=1M count=1000; done) it doesn't stop create file on a full disk
19:58 Technicool mythzib, correct
19:58 Technicool because its a single file
19:58 mythzib it's because the bash never stop until the "for" loop is not over that the option is not working?
19:59 Technicool no...if you create a 100GB single file, there isn't even enough space in the cluster to hold it correct?
19:59 mythzib I don't create one single file
19:59 mythzib but 100 files, 1GB each
19:59 Technicool ah, sorry misread
19:59 mythzib but using one bash command using a for
20:00 Technicool no, i use almost the exact same for loop all the time
20:00 mythzib it looks like the "min free disk" isn't working for this purpose
20:00 Technicool except that i name the files as foo-$i
20:00 mythzib I did too in fact
20:01 elyograg you could add a check for the exit value with a break if it goes nonzero, that would exit the for loop.
20:01 mythzib did you already fill disk using a loop?
20:01 Technicool just mentioning that's the only difference, doesn't have anything to do with the issue
20:01 Technicool mythzib, yes
20:01 mythzib ok :(
20:01 Technicool but i typically use much smaller files
20:02 _br_ joined #gluster
20:02 mythzib ok
20:03 elyograg actually, there's even shorthand. this would probably work: for i in `seq -w 1 100`; do dd if=/dev/zero of=`date +%s` bs=1M count=1000 || break; done
20:03 Technicool you did prove that its working with the for loop though, correct?
20:03 mythzib Technicool : I did prove it's working when I run in two times the loop
20:03 Technicool when you have the 15GB file on one of the backends
20:04 mythzib with only one loop, it fill everything
20:04 Technicool so, if you have one of the nodes almost full and run the for loop, it fails?
20:04 Technicool was this with 1GB files?
20:05 slowe joined #gluster
20:05 Technicool try this
20:05 Technicool make a 17.5GB file (consume nearly all space on one node)
20:06 Technicool then try with 1GB
20:06 Technicool before we had 15GB
20:06 Technicool if you write 1GB to that, you are at 16
20:06 Technicool min free is 2GB
20:06 mythzib for glusterFS, it looks like "for i in `seq -w 1 100`; do dd if=/dev/zero of=`date +%s` bs=1M count=1000; done" it's the same as "dd if=/dev/zero of=`date +%s` bs=1M count=10000100" (because of the "bash command run only one in the for loop)
20:06 _br_ joined #gluster
20:07 mythzib ok, I try
20:07 Technicool otherwise, the much easier test is to create 1MB files
20:08 Technicool from what i was saying about, if you are at 16GB consumed, you probably have more than 2GB of space total, even if only by a few bytes
20:08 Technicool that would allow one more write of a 1GB file
20:09 Technicool which would not technically be 100%, but would be close
20:09 mythzib I don't care if it's a bit below as my limit.. but I don't want more file to be write on it :)
20:09 Technicool by writing 1MB files, you would still have plenty of overhead
20:09 mythzib (17,5GB is writting)
20:13 _br_ joined #gluster
20:16 mario_ joined #gluster
20:17 itamarjp joined #gluster
20:17 mythzib Technicool: ok it's working
20:17 _br_ joined #gluster
20:18 itamarjp hello guys
20:18 Technicool mythzib, ok good
20:18 mythzib after the 17GB create. I had 550MB disk left
20:18 itamarjp when I run a php script in a glusterfs mounted directory I receive PHP Fatal error:  Unknown: Failed opening required , how can I fix that ?
20:18 Technicool which was too small for the 1GB write
20:18 mythzib but when I run the second loop it's ok
20:19 mythzib Technicool : I really think it's because of the "bash loop"
20:19 mythzib 1GB should not be a mess
20:19 Technicool mythzib, wait, it still failed?
20:19 mythzib using one loop, it fill all disk
20:19 mythzib using too loop, it's ok
20:20 mythzib two loop
20:20 tc00per joined #gluster
20:20 Technicool mythzib, sorry if i don't follow
20:20 mythzib two different bash command
20:20 JoeJulian "toruonu> JoeJulian: I'll move back to native when you guys get the stat thing under control" ... Don't look at me! I just hang out here and BS will fellow sysadmins. :O
20:20 Technicool that i follow...what i don't follow is why having two makes a difference
20:20 toruonu :D
20:21 Technicool with the disk full, it should only write the pointer
20:21 toruonu well I'm happy right now with the performance so I'm not complaining :)
20:21 mythzib hmmm
20:21 _br_ joined #gluster
20:21 Technicool mythzib, there is one other possible explanation
20:21 toruonu JoeJulian: But I did take up blogging thanks to your advice
20:22 Technicool in quota, writing one file after quota is reached is allowed to compensate for multiple writes
20:22 JoeJulian Woot!
20:22 Technicool so, delete everything except the 17.5GB file
20:22 Technicool then try again with a single loop and 1MB files
20:23 Technicool if that doesn't work, you are probably correct about the bash command part
20:23 Technicool although i really want that not to be true
20:23 bauruine joined #gluster
20:23 JoeJulian No, it's got to be per fd...
20:24 Technicool oh JoeJulian, using words like must and got to with Gluster...who filled your heart with all that hope?
20:24 Technicool ;)
20:24 mythzib Technicool : already did it, it works this way
20:24 JoeJulian hehe
20:24 Technicool with 1MB files, works as expected, with a single bash loop?
20:25 JoeJulian Not so much hope as it is knowing how the structures are managed.
20:25 mythzib Technicool : when it doesn't work is when I have a single loop doing >80GB
20:25 pdurbin left #gluster
20:25 Technicool mythzib, in that case, i would put two bits on the fact that one file write is allowed after the threshold is reached
20:25 JoeJulian Since everything's attached to fds, that's why we don't have stat caches that last beyond the destruction of the fd.
20:25 mythzib I mean >80GB with 1GB each
20:25 Technicool ^fact^my wild speculation
20:26 mythzib I will need to take this case in concideration for my developpement
20:27 Technicool mythzib, bear in mind, in production you will (hopefully) have more than a few spare GB and users writing less than 1TB files
20:27 mythzib sure :)
20:27 _br_ joined #gluster
20:28 Technicool so it should be considered as an outside possibility, but not something to spend an extra six figures in hardware to accomodate...
20:28 Technicool unless you are selling the hardware  ;)
20:28 mythzib no, i'm not
20:28 mythzib :)
20:29 mythzib but I need to take it in concideration :)
20:30 mythzib consideration
20:30 mythzib I will have a lot of "small" nodes (2TB each)
20:30 * Technicool considers a script to constantly write and destroy millions of 1 byte files onto the gluster mount
20:31 Technicool sure, performance might get the *teensiest* ding...
20:31 Technicool but this won't be an issue for sure  ;)
20:31 semiosis so what was the verdict on min-free-disk?
20:31 semiosis will it blend?
20:31 semiosis i mean, does it work?
20:31 Technicool semiosis, yes, mostly
20:31 _br_ joined #gluster
20:32 Technicool the quirk being that it looks like it has the same failsafe as quota
20:32 semiosis ??
20:32 Technicool that allows one more file to be written after quota is reached, or in this case, threshhold
20:32 mythzib oops
20:32 Technicool so, if the next file that gets written is gigantoriffic
20:33 Technicool it will fill the disk
20:33 Technicool mythzib, don't say oops i was going to put this down as my good deed for the day
20:33 Bullardo joined #gluster
20:33 Technicool still working on that learning something new part.
20:33 vicelow my storage pool has a weird behavior, I just can connect 1 server, the second one fails...  first nodo3 is connected, after nodo2 is connected but no at the same time http://fpaste.org/wzD8/
20:33 glusterbot Title: Viewing peer probe (at fpaste.org)
20:34 semiosis vicelow: did you clone your servers?
20:34 vicelow yes
20:34 JoeJulian nodo = frodo's little brother...
20:34 semiosis each one needs a *unique* UUID in /var/lib/glusterd/glusterd.info
20:34 * Technicool thinks semiosis is a gluster shark
20:34 * Technicool refuses to bet against him
20:35 semiosis vicelow: if you clone, they all have the same uuid
20:35 vicelow aaa I see
20:35 Technicool unless you wait to install gluster after the clone
20:35 vicelow can I obtain a new uuid?
20:35 Technicool or do some sort of quick trick to fix it
20:35 JoeJulian You have to apply to the Elders of the Internet for a new uuid.
20:35 mythzib Technicool thanks for help and understanding
20:35 semiosis vicelow: delete the glusterd.info and when you restart glusterd it will regenerate a new one
20:35 JoeJulian or that
20:35 mythzib I was disapointed before come here.. because of the great glusterFS hope :)
20:35 semiosis JoeJulian: IANA
20:35 Technicool rm -rf /var/lib/glustserd && /etc/init.d/glusterd restart
20:36 vicelow nice thank you!
20:36 Technicool mythzib, yeah, we love it
20:36 _br_ joined #gluster
20:36 mythzib performance are really good !
20:36 mythzib really surprised
20:36 * Technicool <3's gluster so much, he didn't even get to take a lunch break
20:36 semiosis Technicool: picked up food, eating as we speek :)
20:37 Bullardo joined #gluster
20:37 Technicool semiosis, im actually going to grab some now, but i want people to start using that phrase
20:37 JoeJulian How did you get converted to my lunch schedule?
20:37 Technicool and giving me credit for it
20:37 Technicool JoeJulian, i had that lunch schedule for 8 years i think
20:37 semiosis i usually wait until ~3 pm to get lunch, to avoid the rush
20:38 Technicool its weird that the hairs on the back of my neck never broke from all the times they raised at people asking "aren't you taking a lunch"
20:38 semiosis we have good snacks in the office kitchen so it's easy :)
20:38 JoeJulian Yesterday I waited 'till 5:30pm... :/
20:38 JoeJulian no snacks.
20:38 semiosis JoeJulian: i thought you wfh
20:38 Technicool its raining here in cali land so i guess i should wait a bit
20:39 * semiosis does a shot of hot sauce
20:39 JoeJulian I only do that a couple days a week (sometimes 3). Yesterday, however, I had to go to Tacoma to work with the Integra tech to solve an intermittent dsl problem.
20:40 semiosis ah
20:43 _br_ joined #gluster
20:45 corenuy joined #gluster
20:49 _br_ joined #gluster
20:51 _br_ joined #gluster
20:52 vicelow semiosis, thank you! is working well now
20:52 semiosis yw
20:56 tqrst let's say I'm in socket.c:socket_event_handler - can I somehow figure out who initiated the connection that is being handled?
20:56 tqrst I'm not too familiar with the code
20:57 tqrst figured it would be easier to attach gdb to glusterd than wading through pages of tcpdump/wireshark to find out what is geenrating those periodic "[socket.c:1798:socket_event_handler] >> 0-transport: disconnecting now"
20:57 JoeJulian jdarcy: ^?
20:58 tqrst s/geen/gene
20:59 semiosis tqrst: how periodic are they?  consistent interval?  how long?
20:59 tqrst semiosis: 3 seconds, on every glusterd
20:59 JoeJulian Frankly, I think wireshark might be easier.
20:59 semiosis are you pinging your glusterds with some monitoring app like nagios?
20:59 JoeJulian 3 seconds is the reconnect interval from the client.
20:59 semiosis ooh
20:59 _br_ joined #gluster
21:00 tqrst semiosis: hm, we do have nagios tucked away somewhere, but I don't think it's set up to monitor any gluster-specific things
21:00 tqrst JoeJulian: what am I searching for in wireshark's output then?
21:00 tqrst just filtering by port shows a lot of noise
21:01 semiosis needs more signal
21:01 JoeJulian Should be something that corresponds to the timestamps in your logs.
21:01 tqrst right now I have 'tshark|egrep 24012\|24013\|24014', which is what ports glusterfsd listens on
21:01 JoeJulian It'll be on port 24007
21:01 semiosis to be sure.... tqrst: which log file is this?
21:02 tqrst semiosis: /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
21:02 JoeJulian yep, 24007
21:02 JoeJulian unless you're using rdma
21:02 tqrst ok, 24007 is slightly less noisy
21:02 semiosis @ports
21:02 glusterbot semiosis: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
21:06 _br_ joined #gluster
21:09 _br_ joined #gluster
21:11 tqrst still quite noisy though
21:12 JoeJulian Are you using the gluster filter?
21:12 tqrst there's a gluster filter?
21:12 JoeJulian @wireshark
21:12 glusterbot JoeJulian: Nixpanic has done some work on a wireshark decoder for glusterfs: http://goo.gl/gXXQ4 and http://goo.gl/8ZrVP
21:12 tqrst I was just doing tshark -t a|grep 24007
21:13 tqrst does this work with 3.3.1?
21:14 mythzib_ joined #gluster
21:14 _br_ joined #gluster
21:16 jdarcy_ joined #gluster
21:16 JoeJulian yep
21:20 tqrst Error unpacking rpm package wireshark-1.8.3-1.el6.0.glusterfs.x86_64 error: unpacking of archive failed on file /usr/sbin/dumpcap: cpio: cap_set_file
21:21 _br_ joined #gluster
21:23 mythzib joined #gluster
21:24 gbrand__ joined #gluster
21:25 _br_ joined #gluster
21:27 mythzib Technicool: when I have a lot of pointer file, because of the min free disk limit, how can I get back files? rebalance do nothing
21:29 tqrst filtering by 'glusterd' doesn't show anything, 'glusterfs' yields over 1000 packets per second
21:29 semiosis add more space, then rebalance?
21:29 semiosis mythzib: ^
21:31 bjoern joined #gluster
21:32 mythzib semiosis ok :)
21:32 semiosis just a guess :)
21:32 tqrst semiosis: (killed nagios, still getting the messages btw)
21:32 tqrst just in case
21:32 semiosis tqrst: yeah well we kinda new it wasn't that
21:33 semiosis tqrst: JoeJulian suggested that the gluster cli has a 3s reconnect timer for it's tcp connection to glusterd... do you have any gluster clis open anywhere?
21:36 tqrst semiosis: no, only glusterfs, glusterd and glusterfsd (checked across the whole cluster with pgrep -fl gluster)
21:37 _br_ joined #gluster
21:39 shireesh joined #gluster
21:40 stopbit joined #gluster
21:41 bjoern joined #gluster
21:42 45PABEK6R joined #gluster
21:46 _br_ joined #gluster
21:49 carrar joined #gluster
21:51 carrar w/connect -ssl irc.paraphysics.net 6697
21:51 carrar err
21:51 carrar woops
21:51 tqrst semiosis: found a related bug report https://bugzilla.redhat.com/show_bu​g.cgi?format=multiple&amp;id=847821
21:51 glusterbot <http://goo.gl/DUmHv> (at bugzilla.redhat.com)
21:52 _br_ joined #gluster
21:53 * semiosis wonders which IRC clients send the most "oops wrong window" messages
21:54 semiosis tqrst: cool!
21:55 semiosis JoeJulian: bug 847821
21:55 tqrst semiosis: I vaguely remember there being a reason why the previous guy disabled nfs, though
21:55 glusterbot Bug http://goo.gl/gJor4 low, medium, ---, rabhat, ASSIGNED , After disabling NFS the message "0-transport: disconnecting now" keeps appearing in the logs
21:55 tqrst something about a conflict with our own nfs server
21:55 semiosis yes it would conflict
21:55 semiosis @nfs
21:55 glusterbot semiosis: To mount via nfs, most distros require the options, tcp,vers=3 -- Also portmapper should be running on the server, and the kernel nfs server (nfsd) should be disabled
21:55 semiosis can only have one nfs server on a host
21:56 JoeJulian Ooh, kill the gluster nfs client.
21:57 tqrst what nfs client?
21:57 JoeJulian Ooh, kill the gluster nfs client on all your servers.
21:57 semiosis @processes
21:57 _br_ joined #gluster
21:57 glusterbot semiosis: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
21:57 vicelow good bye, thanks all for your help!
21:58 tqrst there are indeed two glusterfs processes per machine
21:58 JoeJulian pgrep -f 'glusterfs.*nfs'
21:58 tqrst ...none of which have 'nfs' in them, though :\
22:00 semiosis JoeJulian: why would glusterd and the nfs daemon communicate over a unix socket?  could it be glusterd is trying to reach the missing nfs daemon?
22:00 semiosis that would be a bug
22:00 JoeJulian Oh, I read the bug backwards...
22:00 semiosis one interpretation of the bug report, tho idk if it's the right interpretation
22:00 JoeJulian yeah, glusterd is trying to connect to the missing nfs daemon.
22:01 JoeJulian At least in that report it is.
22:01 semiosis well we all learned soemthing new
22:01 tqrst in case I missed something obvious in there: http://www.fpaste.org/AtcJ/
22:01 glusterbot Title: Viewing Paste #255786 (at www.fpaste.org)
22:02 rudimeyer_ joined #gluster
22:03 _br_ joined #gluster
22:04 tqrst so... I guess I'll ignore it for now
22:04 UnixDev i have a volume that i can mount on one brick but not the other, its replicate 2… how can this be fixed? running cluster 3.3.0
22:05 semiosis strictly speaking you should never mount a volume on a brick
22:05 semiosis but i suspect that's not really what you mean
22:05 JoeJulian tqrst: I assume you've restarted all your glusterd
22:05 semiosis could you please ,,(pasteinfo) and also both of your mount commands... the successful one & the failing one
22:05 glusterbot Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:05 UnixDev semiosis: what about accessing via nfs from a brick?
22:05 tqrst JoeJulian: several dozen times since my miserable attempt at upgrading to 3.3.1 :p
22:06 semiosis UnixDev: ,,(glossary) to keep our terms straight
22:06 glusterbot UnixDev: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
22:06 semiosis UnixDev: but terminology aside, if you could provide what i asked for that would be most helpful
22:06 tqrst which sort of turned out ok if you ignore the periodic segfaults
22:06 UnixDev clients access data on servers through nfs, can on 1 server, but not the other, volume is replicate 2… why?
22:07 JoeJulian UnixDev: "Strictly speaking" meaning if your brick in server1:/foo you shouldn't "mount -t glusterfs server1:myvol /foo"
22:07 UnixDev right, i have not done that
22:08 semiosis UnixDev: you can have the client mount the volume on a server using "localhost:/volname /some/client/path" but you should only do that with the FUSE client, not NFS client
22:08 JoeJulian tqrst: I wish I knew why yours is so damned unstable. The only thing that I know for sure you're doing differently is that you have /var/lib/glusterd symlinked to /etc/glusterd, right?
22:08 UnixDev mounted via  "mount -t glisters server1:/iso /mnt/iso" (brick path is /data/something)
22:08 UnixDev semiosis: I am using the fuse client
22:08 semiosis UnixDev: should be ok, lets find out whats going wrong
22:09 semiosis UnixDev: your client log would be /var/log/glusterfs/mnt-iso.log, please pastie that, along with the ,,(pasteinfo) i asked for earlier
22:09 glusterbot UnixDev: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:09 tqrst JoeJulian: yes (not that it should matter)
22:09 tqrst does gluster keep state information anywhere else?
22:09 JoeJulian tqrst: And you have some sort of shared store that holds the libs and application. None of /etc/glusterd is shared, right?
22:09 UnixDev semiosis: http://dpaste.org/n4n8y/
22:09 glusterbot Title: dpaste.de: Snippet #214033 (at dpaste.org)
22:10 tqrst JoeJulian: that's correct. /etc/glusterd is in statetab
22:11 JoeJulian What is the shared store? I'm going to see if I can duplicate this, since I've got more important things to do but don't want to do them right now.
22:11 _br_ joined #gluster
22:15 tqrst I'm not sure to be honest - all I know is that non-shared paths should be in /etc/statetab
22:17 _br_ joined #gluster
22:21 UnixDev semiosis: http://dpaste.org/4cjd3/
22:21 glusterbot Title: dpaste.de: Snippet #214034 (at dpaste.org)
22:23 UnixDev semiosis: any thoughts?
22:24 mario_ joined #gluster
22:25 semiosis UnixDev: idk but i'd suggest upgrading to the latest official release, 3.3.1
22:25 semiosis UnixDev: what version are you using?
22:25 UnixDev semiosis: build from branch 3.3 from git
22:26 UnixDev pretty recent, 2-3 days i believe
22:26 semiosis from how long ago?
22:26 semiosis oh
22:26 semiosis well
22:26 johnmark kkeithley: doh
22:27 UnixDev this is something relatively new, these types of "locks" where sometimes it doesn't mount, i also get times where all gluster locks up and "gluster vol status" just hangs… I didn't experience this behavior prior.. but ever since building new 3.3 branch from git a few days ago, it seems to be happening
22:27 JoeJulian release-3.3, right?
22:28 UnixDev correct, thats why I was asking the other day
22:28 UnixDev we're getting ready to use this in limited production, and I wanted to have the latest version.. but maybe it needs some more testing :P
22:29 semiosis what you have is not "the latest version" :/
22:29 UnixDev ahh, ok
22:29 UnixDev should I update and build again?
22:29 JoeJulian I wish there was a tag for v3.3.1 so I could tell if anything changed since then... :(
22:29 UnixDev oh.. I see
22:29 semiosis UnixDev: get 3.3.1 from download.gluster.org
22:30 UnixDev 3.3.1 is older than most recent commits in release-3.3
22:30 semiosis yeah i know
22:30 UnixDev oh, ok
22:31 UnixDev let me give it a try
22:31 * JoeJulian beats hagarth_ about the head and shoulders with a month old trout.
22:31 semiosis release-3.3 has newer code that, one day, will become the *next* release in the 3.3 series
22:31 UnixDev when they fix the locks perhaps
22:31 UnixDev lol
22:31 semiosis after it goes through some ,,(qa releases) and people think it's ready
22:31 glusterbot The QA releases are available at http://bits.gluster.com/pub/gluster/glusterfs/ -- RPMs in the version folders and source archives for all versions under src/
22:32 JoeJulian btw... what's wrong with the yum repo or the ppa?
22:33 johnmark JoeJulian: that's what I was going to ask
22:34 UnixDev well, last time I used that it would not work with vmware (newer fixes in git) … so building from git its easier in that respect…
22:35 UnixDev not having a release tag really sucks though
22:37 mario__ joined #gluster
22:38 tqrst semiosis: I tried to 'volume set myvol nfs.disable off' and then back on just to see if it would jiggle anything by magic, and got: "error" "set volume unsuccessful", with "GlusterFS[9319]: [2012-11-28 17:36:30.550250] C [glusterd-rpc-ops.c:887:glusterd3_1_stage_op_cbk] 0-: Stage response received from unknown peer: 00000000-0000-0000-0000-000000000000" in /var/log/messages
22:38 tqrst needless to say there is no peer with uuid 00000000-0000-0000-0000-000000000000
22:40 semiosis tqrst, UnixDev, did either of you clone servers, in such a way that /var/lib/glusterd/glusterd.info would have the same content on more than one server?
22:40 * semiosis batch processes
22:40 tqrst semiosis: no
22:40 UnixDev semiosis: nope, not in any way
22:40 semiosis excellent
22:40 UnixDev server 1 and server 2 are physical machines
22:40 semiosis then idk anything about the 0000000000000 uuids
22:41 tqrst I just tried again and couldn't reproduce the 000000 error
22:41 * tqrst sighs
22:41 semiosis fixed
22:48 tqrst at least my symptoms are consistent with that bug report - setting nfs.disable to off stops the log spam
22:55 Daxxial_ joined #gluster
22:56 UnixDev semiosis: I installed 3.3.1, restarted cluster on both servers. seems I still can't mount on server1 , mounting on server2 works fine. Now when i try "gluster vol info iso" i get "operation failed"
22:56 UnixDev cluster=gluster
22:57 blendedbychris joined #gluster
22:57 blendedbychris joined #gluster
22:57 UnixDev sorry, "gluster vol status iso" returns "operation failed".. vol info shows correctly
23:00 semiosis iptables/
23:00 semiosis ?
23:04 berend` joined #gluster
23:14 tc00per left #gluster
23:20 tc00per joined #gluster
23:26 lh joined #gluster
23:26 lh joined #gluster
23:34 duerF joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary