Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-02-29

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 haomaiwa_ joined #gluster
00:20 Iouns joined #gluster
01:06 EinstCrazy joined #gluster
01:17 nishanth joined #gluster
01:28 nangthang joined #gluster
01:41 haomaiwang joined #gluster
01:53 harish joined #gluster
01:57 mmckeen joined #gluster
01:57 hagarth joined #gluster
02:01 mmckeen joined #gluster
02:01 haomaiwa_ joined #gluster
02:19 shubhendu joined #gluster
02:41 ovaistariq joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:48 Norky joined #gluster
02:50 mdavidson joined #gluster
02:52 jbrooks joined #gluster
03:01 haomaiwa_ joined #gluster
03:07 baojg joined #gluster
03:22 arcolife joined #gluster
03:24 kanagaraj joined #gluster
03:26 gem joined #gluster
03:36 atinm joined #gluster
03:38 ashiq joined #gluster
03:42 nehar joined #gluster
03:46 nbalacha joined #gluster
03:48 sakshi joined #gluster
03:59 itisravi joined #gluster
03:59 ahino joined #gluster
04:01 haomaiwa_ joined #gluster
04:03 overclk joined #gluster
04:06 nathwill joined #gluster
04:11 ramteid joined #gluster
04:16 ovaistariq joined #gluster
04:26 itisravi joined #gluster
04:27 atalur joined #gluster
04:28 kshlm joined #gluster
04:30 baojg joined #gluster
04:30 kdhananjay joined #gluster
04:34 hchiramm joined #gluster
04:42 Manikandan joined #gluster
04:43 Trefex_ joined #gluster
04:49 ackjewt joined #gluster
04:50 dataio joined #gluster
04:51 ppai joined #gluster
04:52 hgowtham joined #gluster
04:55 gowtham joined #gluster
04:57 pppp joined #gluster
04:57 arcolife joined #gluster
04:59 archit_ joined #gluster
05:03 jdossey joined #gluster
05:15 kasturi joined #gluster
05:17 nehar joined #gluster
05:22 jiffin joined #gluster
05:24 karthikfff joined #gluster
05:24 ggarg joined #gluster
05:24 baojg joined #gluster
05:26 ndarshan joined #gluster
05:27 Apeksha joined #gluster
05:28 nbalacha joined #gluster
05:31 skoduri joined #gluster
05:31 Bhaskarakiran joined #gluster
05:33 merp_ joined #gluster
05:33 nehar joined #gluster
05:48 R0ok_ joined #gluster
05:50 shubhendu joined #gluster
05:51 ppai joined #gluster
05:54 poornimag joined #gluster
06:02 karnan joined #gluster
06:03 karnan_ joined #gluster
06:03 nishanth joined #gluster
06:06 dlambrig joined #gluster
06:07 ramky joined #gluster
06:08 karnan joined #gluster
06:08 gem joined #gluster
06:12 gem_ joined #gluster
06:12 kovshenin joined #gluster
06:13 gem_ joined #gluster
06:16 cpetersen joined #gluster
06:17 jdossey joined #gluster
06:25 pppp joined #gluster
06:32 Manikandan_ joined #gluster
06:35 atalur joined #gluster
06:36 kdhananjay joined #gluster
06:37 Manikandan joined #gluster
06:40 nbalacha joined #gluster
06:47 kovsheni_ joined #gluster
06:50 kovshenin joined #gluster
06:50 hchiramm joined #gluster
06:55 kovsheni_ joined #gluster
06:56 ggarg joined #gluster
06:59 cvstealth joined #gluster
07:00 kovshenin joined #gluster
07:07 ovaistariq joined #gluster
07:08 David_Varghese joined #gluster
07:09 mhulsman joined #gluster
07:10 jtux joined #gluster
07:12 kovshenin joined #gluster
07:18 baojg joined #gluster
07:19 jtux joined #gluster
07:22 post-factum joined #gluster
07:23 nangthang joined #gluster
07:37 kdhananjay1 joined #gluster
07:41 haomaiwa_ joined #gluster
07:50 [Enrico] joined #gluster
07:53 atalur joined #gluster
07:57 hackman joined #gluster
08:01 haomaiwa_ joined #gluster
08:02 hchiramm joined #gluster
08:02 ahino joined #gluster
08:04 Pupeno joined #gluster
08:04 Pupeno joined #gluster
08:05 mbukatov joined #gluster
08:06 jvandewege_ joined #gluster
08:07 ovaistariq joined #gluster
08:09 mhulsman1 joined #gluster
08:09 jri joined #gluster
08:10 siel joined #gluster
08:12 kovshenin joined #gluster
08:14 Wizek_ joined #gluster
08:15 David_Varghese joined #gluster
08:16 ivan_rossi joined #gluster
08:16 siel joined #gluster
08:17 Akee1 joined #gluster
08:20 Trefex joined #gluster
08:23 itisravi joined #gluster
08:26 baojg joined #gluster
08:30 Wizek joined #gluster
08:35 Akee joined #gluster
08:44 itisravi joined #gluster
08:46 Pupeno joined #gluster
08:56 baojg joined #gluster
08:58 ctria joined #gluster
09:01 haomaiwa_ joined #gluster
09:07 kovsheni_ joined #gluster
09:09 Pupeno joined #gluster
09:09 Pupeno joined #gluster
09:10 kovshenin joined #gluster
09:12 kovsheni_ joined #gluster
09:12 Slashman joined #gluster
09:16 kotreshhr joined #gluster
09:20 kovshenin joined #gluster
09:34 Lee1092 joined #gluster
09:35 kotreshhr left #gluster
09:38 kovsheni_ joined #gluster
09:50 hchiramm joined #gluster
09:51 ninkotech joined #gluster
09:51 ninkotech_ joined #gluster
09:53 n-st joined #gluster
09:56 jiffin joined #gluster
10:00 aravindavk joined #gluster
10:01 haomaiwa_ joined #gluster
10:04 baojg joined #gluster
10:08 ovaistariq joined #gluster
10:31 kdhananjay joined #gluster
10:49 atalur joined #gluster
11:01 haomaiwa_ joined #gluster
11:02 ashiq joined #gluster
11:02 morse joined #gluster
11:06 Trefex joined #gluster
11:07 atalur joined #gluster
11:11 mhulsman joined #gluster
11:18 kdhananjay1 joined #gluster
11:29 Simmo joined #gluster
11:31 Simmo Hi All! : )
11:31 morse joined #gluster
11:32 Simmo I have a strange problem: the setup is a replica volume (2+1 arbiter). When a client mounts the gluster volume then the files size is 0 bytes (checked with ls -l).
11:32 Simmo Is it normal ? If not, how can i fix it ? :-/
11:36 post-factum Simmo: so you mount volume to, saying, /mnt/volume, and ls -lha /mnt/volume shows that all files have zero size?
11:41 itisravi joined #gluster
11:42 ggarg joined #gluster
12:01 haomaiwang joined #gluster
12:02 baojg joined #gluster
12:09 ovaistariq joined #gluster
12:12 mhulsman1 joined #gluster
12:13 Simmo post-factum: yes, as you said!
12:14 kovshenin joined #gluster
12:14 Simmo post-factum: but not all the files: just the one created by my application. If i create a txt file manually then I can see its size
12:15 post-factum Simmo: what is that application? does it work ok on regular fs?
12:16 Simmo post-f: yes, it does. Locally it is generating that files in a consistent way
12:17 EinstCrazy joined #gluster
12:17 Simmo post-f: instead, the arbiter and the clients see the files with zero bytes.
12:18 post-factum does you app flushes/closes the file after it has been written to?
12:18 jdossey joined #gluster
12:19 kayn joined #gluster
12:19 johnmilton joined #gluster
12:19 Simmo yes, it does. But i can double-check. Anyway, the strange part is: if I open the "zero byte" file then the content looks OK.
12:20 post-factum sounds like this bug: https://bugzilla.redhat.co​m/show_bug.cgi?id=1310171
12:21 glusterbot Bug 1310171: unspecified, unspecified, ---, ravishankar, POST , Incorrect file size on mount if stat is served from the arbiter brick.
12:21 post-factum this patch should fix your issue: http://review.gluster.org/#/c/13539/
12:21 glusterbot Title: Gerrit Code Review (at review.gluster.org)
12:21 Simmo another hint: has anyone experienced issue with Boot framework (boost::interprocess::interprocess_exception) ?
12:22 Simmo post-f: oh thanks! I'll have a look immediately
12:23 mhulsman joined #gluster
12:23 post-factum (I really hope this patch will be merged into 3.7.9, and at last I could upgrade :))
12:25 Simmo post-f: it sounds great! : )
12:25 Simmo post-f: do I have a way to apply the patch on my glusterfs installation now ? : )
12:25 Simmo post-f: glusterfs 3.7.8 built on Feb 12 2016 13:08:20
12:25 post-factum sure, you must recompile glusterfs :)
12:26 Simmo I was afraid that the answer was that one :)
12:26 post-factum distro?
12:26 Simmo Ubuntu 14.04 LTS
12:26 post-factum sorry. if it was arch or centos, i could help
12:27 Simmo post-f: no worries. You already helped A LOT! : )
12:27 Simmo post-f: another way to fix it... would be to remove the arbitrer ?
12:27 Simmo *arbiter
12:27 post-factum dunno if that is possible now
12:27 Simmo a bit drastic, but it would be fine for me : )
12:27 post-factum you'd better stay with arbiter
12:28 post-factum as a silly temporary solution you could kill arbiter brick
12:29 post-factum i guess it will work but it is weird
12:29 Simmo luckily for me, i'm setting up a new production environment so that I could re-create a replica volume without the arbiter
12:30 post-factum you would be more happy compiling 3.7.8 with several extra patches
12:30 post-factum including this one
12:31 Simmo I'll check the steps online.. never did that : )
12:31 Simmo thanks a lot post-factum.. you let me see the light at the end of the tunnel : )
12:32 post-factum np
12:34 post-factum Simmo: https://github.com/semiosis/gluster​fs-debian/tree/trusty-glusterfs-3.7 probably this could help
12:35 glusterbot Title: GitHub - semiosis/glusterfs-debian at trusty-glusterfs-3.7: Debian source package for glusterfs (at github.com)
12:35 Simmo wow : )
12:35 Simmo thx!
12:36 post-factum or even this:
12:36 aravindavk joined #gluster
12:36 post-factum https://launchpad.net/~gluster/+archiv​e/ubuntu/glusterfs-3.7/+files/glusterf​s_3.7.8-ubuntu1~trusty2.debian.tar.gz
12:38 ira joined #gluster
12:40 B21956 joined #gluster
12:43 haomaiwang joined #gluster
12:43 Akee joined #gluster
12:46 overclk joined #gluster
12:53 nottc joined #gluster
12:56 julim joined #gluster
13:07 post-fac1um joined #gluster
13:09 post-factum joined #gluster
13:10 abrt joined #gluster
13:13 haomaiwa_ joined #gluster
13:15 sebamontini joined #gluster
13:22 abrt hi, i've installed gluster 3.6.3 with a replica 2, it seems to work fine, but...Every node has also kvm and all the vm every 3 second write an error to their own log file reporting "...0-gv0-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))", someone have any hint to resolve the issue?
13:37 kdhananjay joined #gluster
13:37 shyam joined #gluster
13:40 kshlm joined #gluster
13:45 post-factum abrt: could you try 3.6.8 first?
13:46 mhulsman joined #gluster
13:47 kanagaraj joined #gluster
13:49 abrt my fault sorry, i've noticed that one node print "0-glusterd: Request received from non-privileged port. Failing request" on etc-glusterfs-glusterd.vol.log I had executed "gluster volume set <volname> server.allow-insecure on" but not edited the glusterd.vol on the node. After the correction and a restart the message is disappeared
13:56 B21956 joined #gluster
13:58 plarsen joined #gluster
13:58 post-factum ah, you use gfapi to access vm images
14:00 theron joined #gluster
14:01 haomaiwa_ joined #gluster
14:05 chirino_m joined #gluster
14:08 harish joined #gluster
14:11 abrt yes, on 3.6.8 now :)
14:11 pdrakeweb joined #gluster
14:11 ovaistariq joined #gluster
14:15 wnlx joined #gluster
14:15 baojg joined #gluster
14:22 ira joined #gluster
14:25 theron joined #gluster
14:28 B219561 joined #gluster
14:29 jiffin1 joined #gluster
14:30 _Bryan_ joined #gluster
14:34 baojg joined #gluster
14:40 karnan joined #gluster
14:41 skylar joined #gluster
14:41 TealJax joined #gluster
14:45 DV joined #gluster
14:48 gem joined #gluster
14:50 hamiller joined #gluster
14:57 baojg joined #gluster
14:58 [diablo] joined #gluster
15:01 haomaiwa_ joined #gluster
15:01 EinstCra_ joined #gluster
15:03 EinstCrazy joined #gluster
15:14 taavida joined #gluster
15:16 farhorizon joined #gluster
15:18 vmallika joined #gluster
15:19 rwheeler joined #gluster
15:20 taavida hi. I've an issue when trying to restart glusterfsd on one member in a simple replica setup with 2 members. The service fails to rebind and I get some strange errors in glfsheal-[vol].log
15:20 taavida [2016-02-29 15:16:56.044254] W [dict.c:612:dict_ref] (-->/usr/lib64/glusterfs/3.7.5/xlator/clus​ter/replicate.so(afr_get_heal_info+0x16c) [0x7fc9e8a59fac] -->/usr/lib64/libglusterfs.so​.0(syncop_getxattr_cbk+0x43) [0x7fc9f77e9123] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x89) [0x7fc9f779a949] ) 0-dict: dict is NULL [Invalid argument]
15:20 glusterbot taavida: ('s karma is now -124
15:21 post-factum @paste
15:21 glusterbot post-factum: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
15:21 post-factum taavida: ^^
15:21 post-factum stop killing ('s karma
15:22 post-factum (++
15:22 glusterbot post-factum: ('s karma is now -123
15:30 jdossey joined #gluster
15:32 taavida post-factum: would love to try that, but I'm not allowed to access termbin.com from my location
15:34 DV joined #gluster
15:35 taavida http://pastebin.com/QCZ34b5E
15:35 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
15:36 taavida ok then, http://fpaste.org/331161/60165145/
15:36 glusterbot Title: #331161 Fedora Project Pastebin (at fpaste.org)
15:36 David_Varghese joined #gluster
15:37 arcolife joined #gluster
15:38 post-factum taavida: could you please update your setup to latest version available first?
15:42 post-factum this one could be your case: http://review.gluster.org/#/c/11409/
15:42 glusterbot Title: Gerrit Code Review (at review.gluster.org)
15:43 robb_nl joined #gluster
15:44 taavida post-factum: ok thanks. I'll consider upgrading but first need to check changelog between 3.7.5 (which we use in prod) and 3.7.8
15:44 post-factum changelog is pretty big :). do you compile packages by yourselves?
15:45 taavida nope, using official rpms from http://download.gluster.org
15:45 glusterbot Title: Gluster.org Download Server (at download.gluster.org)
15:45 post-factum ok, then probably you would want to wait for 3.7.9
15:46 post-factum because of fuse client performance regression
15:46 nathwill joined #gluster
15:46 nbalacha joined #gluster
15:48 taavida I guess you mean this https://bugzilla.redhat.co​m/show_bug.cgi?id=1309462 Thanks for the hint
15:48 glusterbot Bug 1309462: low, unspecified, ---, ravishankar, ASSIGNED , Upgrade from 3.7.6 to 3.7.8 causes massive drop in write performance.  Fresh install of 3.7.8 also has low write performance
15:48 poornimag joined #gluster
15:49 ovaistariq joined #gluster
15:50 kotreshhr joined #gluster
15:50 kotreshhr left #gluster
15:50 post-factum yup
15:53 taavida post-factum: do you know the roadmap for 3.7 stable release?
15:53 julim joined #gluster
15:53 wolsen joined #gluster
15:54 post-factum afaik, 3.7.9 release is scheduled to the end of february
15:54 post-factum given today is the last day of february...
15:55 post-factum i hope new release to happen this week or so, but we'd better ask 3.7.9 release manager
15:55 shubhendu joined #gluster
15:56 gem joined #gluster
15:57 ChrisHolcombe joined #gluster
15:58 post-factum but in case new regressions are introduced, i strongly advice you compiling your own packages with cherry-picked patches
15:58 post-factum that is what i did for 3.7.6
15:59 wushudoin joined #gluster
16:00 wushudoin joined #gluster
16:01 ayma joined #gluster
16:01 haomaiwa_ joined #gluster
16:02 atinm joined #gluster
16:06 ivan_rossi left #gluster
16:19 karnan joined #gluster
16:27 _Bryan_ joined #gluster
16:40 sebamontini joined #gluster
16:45 jugaad joined #gluster
16:48 dnoland1 joined #gluster
16:48 squizzi_ joined #gluster
16:52 honzik666 joined #gluster
16:54 Bhaskarakiran joined #gluster
16:54 jugaad Hi everyone, has anyone got any time to help me with a strange "ls" issue on a gluster mount?
16:56 dnoland1 jugaad: Does the issue happen to be really long time to list the dirs, but perfectly fine read speed after the dir is listed?
17:00 jugaad pretty much
17:00 jugaad takes about 1 minute to "ls" the directory
17:01 jugaad but "strace ls" takes just a fraction of a second
17:01 dnoland1 Mine is sitting at about 15 seconds.  Are you on version 3.7.8?
17:01 jugaad which makes it hard to try and determine what is going wrong
17:01 dnoland1 recently upgraded from 3.7.6 perhaps?
17:01 haomaiwang joined #gluster
17:01 dnoland1 jugaad: Hmmm, you are right.  $(strace ls) does run really fast
17:02 jugaad 3.7.8 using at the minute
17:02 jugaad never done an upgrade - this is a first time install
17:03 jugaad I was wondering whether it is because we have about 500,000 files in this mount, totalling about 100GB
17:03 jugaad lots of small files
17:04 jugaad I wondered whether there was a performance parameter that I could tune to make it cross check (whether it should heal) less
17:04 F2Knight joined #gluster
17:04 dnoland1 jugaad:  Thank you!  That is actually really useful information to me.  I have been trying to determine if this issue was related to my recent upgrade or not, and your experience suggests "no"
17:05 dnoland1 jugaad: but I do have a tip that could help you (and anyone else) debug this
17:05 ovaistariq joined #gluster
17:06 dnoland1 jugaad: go to the .glusterfs dir on one of the bricks you are using to host your slow fs
17:07 dnoland1 jugaad: then cd into one of the two digit hex name directories and run this command: $(file */* | grep -i broken)
17:07 dnoland1 jugaad: you may well find that you have a mess of broken symlinks in those directories.  I know I do.
17:08 jugaad which directories would this be?
17:08 dnoland1 jugaad: They are circular, and seem to cause the heal daemon to thrash around madly, eating all the IO and or starving all the other threads
17:09 dnoland1 jugaad: My bricks are mounted on /srv.  So I have /srv/hdd-1, /srv/hdd-2, ... /srv/hdd-9.
17:09 jugaad I can't see any hex directories :-(
17:10 dnoland1 jugaad: Inside those mounts are the volumes, and inside each volume dir is a hidden file called .glusterfs
17:10 dnoland1 jugaad: e.g. /srv/hdd-1/${volume name}/.glusterfs
17:10 dnoland1 jugaad: ls that dir and you should see a bunch of directories with two digit hex names.
17:11 jugaad I have /export/glusterhome as the mount
17:11 jugaad and /export/glusterhome/brick as the brick (just a single brick setup here...)
17:11 robb_nl joined #gluster
17:12 matclayton joined #gluster
17:13 dnoland1 jugaad: then your experience may be different.  I am running 9 bricks on each of three servers.  Setting it up that way was likely a mistake.  In any case, if your brick is mounted at /export/glusterhome/brick/ you should have a /export/glusterhome/brick/.glusterfs directory
17:13 calavera joined #gluster
17:13 jugaad aahh yes, got that!
17:13 dnoland1 jugaad: Good deal!
17:14 dnoland1 jugaad: now cd into 00 or whatever the first of the hex dirs is and run $(file */* | grep -i broken)
17:14 dnoland1 It may take a moment to run, but I will be **very** interested to know if you have a mess of broken symlinks in there.
17:14 jugaad that returns nothing
17:15 jugaad I take it you get lots of broken symlinks there?
17:15 dnoland1 jugaad: Yes.  Your result is still very interesting.  The slow ls and broken symlink issues may be unrelated.  What do you have in your glustershd.log file?
17:16 jugaad I ran it on the directory above - i.e. the .glusterfs directory itself, with $(file */*/* | grep -i broken) and it brings stuff back
17:17 jugaad 6 broken restuls
17:17 jugaad can I just delete these do you reckon?
17:17 jugaad I could give it a try if you aren't sure, as this is just a test setup at the minute
17:17 dnoland1 jugaad: Will you forgive me if I jump for joy!  I'm sorry you are having the same problem, but I am really glad I'm not going insane.
17:17 jugaad haha
17:18 jugaad you jump away mate
17:18 Philambdo1 joined #gluster
17:18 dnoland1 jugaad: :D If you are on a test setup, go ahead and toast those files.  I would do it like this to keep things as clean as possible.
17:19 dnoland1 jugaad: $(sudo gluster volume stop ${your test volume})
17:19 dnoland1 jugaad: then fry those directories on each brick (should be pretty easy since you only have the one)
17:19 atinm joined #gluster
17:20 dnoland1 jugaad: then $(sudo gluster volume start ${your test volume})
17:20 dnoland1 jugaad: then $(sudo gluster volume heal ${your test volume} full)
17:21 dnoland1 jugaad: watch your logs during the heal (may be a noop due to the one brick config)
17:22 jugaad Just going to do that now#
17:22 dnoland1 Thanks!
17:28 jugaad I'll have to do it on each of my 2 nodes, yup?
17:28 dnoland1 jugaad: Yes.
17:28 dnoland1 keep in mind, I am making this up as I go.  Just advising based on what I would do.
17:29 dnoland1 It is not clear to me why this issue is happening in the first place.  The broken links may be a symptom and not the core problem
17:29 jugaad I've been making it up as I go for weeks now
17:29 jugaad :-p
17:29 jugaad Thanks for all the help so far though :-)
17:30 dnoland1 Glad we are on the same page
17:32 jugaad deleted all the files, started it back up
17:32 jugaad healing now
17:33 jugaad let's see how this goes!
17:33 dnoland1 jugaad: very nice.  fingers crossed
17:36 jugaad How long does a heal usually take would you say?
17:37 jugaad I'm not sure whether this is stuck or not!
17:37 dnoland1 Depends on how much data we are talking and how fast your network / processor / memory is
17:37 jugaad I'm doing another "time ls"
17:37 jugaad see what happens
17:37 dnoland1 sure
17:37 dnoland1 You can also run iotop and see how much disc io you are getting
17:37 jugaad it still baffles me that "time ls" takes about a minute, and "time strace ls" takes milliseconds
17:38 jugaad maybe i should try "strace time ls" instead, mix it all around a bit!#
17:39 jugaad "time ls" took 2 mins this time
17:39 nishanth joined #gluster
17:39 dnoland1 In my case it seems like the broken links were causing the heal daemon to thrash about in circles and starve all the legitimate heals.  This caused my system to go into massive split-brain, and things have gone south from there
17:39 jugaad so that hasn't fixed it
17:39 dnoland1 It could be that the heal is eating all your disc / network io
17:39 jugaad what sort of performance hits did you see on the processor / RAM / IO?
17:40 dnoland1 They have been various.  The issue has become progressively worse over the last two weeks.  It started as some random timeouts on my webhosts, and progressed to the point that nobody can ssh into my servers because it times out before the home dir can fetch the .ssh/authorized_keys file.
17:41 dnoland1 I am currently eating 80G of memory and using 80% of 24 cores on each of my three nodes...
17:42 dnoland1 People are getting pretty angry at the new sys admin (i.e. me)
17:42 jugaad wow
17:42 jugaad those numbers are bigger than what I am working with
17:42 mhulsman joined #gluster
17:42 jugaad not sure what my boss would say if I went to him asking for that much!!
17:42 theron joined #gluster
17:43 jugaad just 2 cores here and 8GB
17:43 dnoland1 I have lost quite a bit of sleep to this problem.  Just started this job and want to come out strong, but I seem to have been struck by lightning.
17:43 jugaad but when I run these commands, I don't see more than about 5% utilisation with top
17:43 dnoland1 Well your system should be much easier to debug.  Far fewer moving parts
17:43 jugaad that's on the CPU and RAM-wise it's pretty much nothing
17:44 jugaad Also with the benefit of not having any real users yet!
17:44 jugaad but I'm running out of ideas
17:44 dnoland1 Yeah, I have about 3500 of em.
17:44 dnoland1 (users that is)
17:44 jugaad I was starting to think about going down the geo-replication route next
17:45 jugaad it's setup with replication at the minute, with the two datacentres being hundreds of miles apart
17:45 jugaad although we do have a nice quick dedicated 400Mbps link
17:45 jugaad 3500 users is far too many
17:45 jugaad get some of them fired!!
17:45 jugaad make your life easier!!
17:47 dnoland1 I will also be trying that as well.  I am at a large university, and I have access to solid hardware and resources.  My network is 40 gigabit and almost totally switched.  It makes things hard to debug, since I hardly ever get to shut anything down to isolate variables.
17:50 jugaad so... what I've found
17:50 jugaad "strace ls" comes back real quick
17:50 jugaad "ls" takes a while
17:50 jugaad strace must cache something somewhere
17:50 jugaad or do less
17:50 jugaad it is probably a red herring
17:51 dnoland1 Yeah, I don't know what to make of that
17:51 dnoland1 If I run ls twice in sequence it get the second result quickly
17:51 dnoland1 $(ls && ls)
17:52 jugaad let me try that
17:52 jugaad I'll  $(time ls && time ls) though...
17:53 jiffin joined #gluster
17:54 jugaad 1st one is back, 1 min 7 seconds
17:54 theron joined #gluster
17:55 jugaad second one is 1 min 10 seconds
17:55 jugaad :-s
17:55 dnoland1 strange.  My result is likely an artifact of the large read cache I set for my volume.  Defaults to 32M, but mine is 4096M.
17:57 jugaad so maybe I should change mine one way or another?
17:58 dnoland1 Could be.  I don't really understand why the default is so small, but I may have made a mistake in setting it so large.  You could try it since you don't have much to loose with respect to a test system.
18:00 F2Knight joined #gluster
18:01 theron joined #gluster
18:01 haomaiwa_ joined #gluster
18:02 dnoland1 Are you getting a huge number of errors like this (http://sprunge.us/YNgg) in you glustershd.log file?
18:02 cpetersen Better late than never, bug submitted.
18:02 cpetersen https://bugzilla.redhat.co​m/show_bug.cgi?id=1313017
18:02 glusterbot Bug 1313017: high, unspecified, ---, bugs, NEW , Quorum is not met and writes are restricted with quorum-type auto in a 3 node system
18:03 cpetersen JoeJulian:  I was too excited about going on holidays to submit that bug last week... ;)
18:04 ovaistariq joined #gluster
18:06 semiosis joined #gluster
18:06 raginbajin joined #gluster
18:07 jugaad dnoland1 - I will take a look at the log file now
18:07 jri joined #gluster
18:09 jugaad Got some entries that look similar in there
18:09 jugaad but "remote operation failed"
18:10 dnoland1 jugaad: drop the output of this in a pastebin for me: $(sudo gluster volume status all).
18:11 dnoland1 jugaad: totally possible that the only issue is a disconnected node
18:14 jri_ joined #gluster
18:14 jugaad pastebin going to be a bit of an issue - on a segregated network :-(
18:15 dnoland1 jugaad: oh....
18:15 jugaad aye and it's getting late too here going to pop off home
18:15 dnoland1 jugaad: well do you have a column of Y in the online field?
18:15 jugaad come back tomorrow and give geo-replication a go
18:15 jugaad aye I can check
18:15 jugaad give me a sec
18:16 dnoland1 jugaad: Thank you very much
18:16 jugaad everything is online
18:16 dnoland1 jugaad: then you have me stumped.  Thanks for the info tho.  And I will try and be online tomorrow.
18:16 dnoland1 jugaad: best of luck
18:16 jugaad my gut thoughts on my issue are either that I have not opened up all the correct firewall ports
18:17 arcolife joined #gluster
18:17 jugaad or that because I have so many small files, I need to performance tune it
18:18 jugaad and I may be back on tomorrow night so maybe see you then
18:18 dnoland1 jugaad: there are some good articles on that topic.  I will see if I can dig them up (I have a similar issue).  Will drop them in this room when I get a chance.  Have a good night :)
18:18 jugaad thanks very much for your help!
18:18 dnoland1 jugaad: my pleasure
18:19 hchiramm joined #gluster
18:44 sebamontini joined #gluster
18:53 ovaistariq joined #gluster
19:00 calavera_ joined #gluster
19:01 haomaiwa_ joined #gluster
19:09 theron joined #gluster
19:10 julim joined #gluster
19:10 ovaistariq joined #gluster
19:11 johnmilton i have a volume that is refusing to rebalance
19:11 johnmilton no errors, the other node is balanced
19:14 Trefex joined #gluster
19:14 B21956 joined #gluster
19:17 jri joined #gluster
19:19 dnoland1 Where are you looking for errors?
19:22 chirino joined #gluster
19:30 karnan joined #gluster
19:37 F2Knight joined #gluster
19:40 btpier joined #gluster
19:46 sebamontini johnmilton whats the output of volume heal info ?
19:48 johnmilton oh, i forgot to mention...this is an ancient gluster installation
19:48 johnmilton no volume heal
19:48 johnmilton i have, however performed a self heal (circa gluster 3.2 instructions), no errors
19:50 ovaistariq joined #gluster
19:56 theron joined #gluster
19:59 gbox joined #gluster
19:59 chirino joined #gluster
20:01 haomaiwa_ joined #gluster
20:25 chirino_m joined #gluster
20:26 ovaistariq joined #gluster
20:27 calavera joined #gluster
20:30 theron joined #gluster
20:32 chirino_m joined #gluster
20:37 robb_nl joined #gluster
20:47 theron joined #gluster
20:54 theron joined #gluster
20:56 ctria joined #gluster
21:01 haomaiwa_ joined #gluster
21:03 chirino_m joined #gluster
21:06 theron joined #gluster
21:11 dnoland1 Anyone have any insight as to why I would have 1. circular / broken symlinks in my .glusterfs dirs and 2. have **very** slow directory listing, but perfectly reasonable read speed on a 3 replicated glusterfs share?
21:12 dnoland1 I have all logs at my disposal and I am happy to provide whatever information I can
21:15 rwheeler joined #gluster
21:26 ovaistariq joined #gluster
21:28 theron joined #gluster
21:38 ira joined #gluster
21:43 jugaad joined #gluster
21:53 theron joined #gluster
22:01 haomaiwang joined #gluster
22:05 jri_ joined #gluster
22:16 ovaistariq joined #gluster
22:37 B21956 left #gluster
22:50 theron joined #gluster
23:01 haomaiwa_ joined #gluster
23:01 jbrooks joined #gluster
23:05 theron joined #gluster
23:12 calavera joined #gluster
23:21 siel joined #gluster
23:33 ovaistariq joined #gluster
23:34 EinstCrazy joined #gluster
23:39 ovaistariq joined #gluster
23:44 hackman joined #gluster
23:48 hackman joined #gluster
23:49 hackman joined #gluster
23:50 cliluw joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary