Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-11-20

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 EinstCrazy joined #gluster
00:02 bluenemo JoeJulian, are you still there? I'm still fighting to get clients to stop caching for such a long time.. it sometimes takes about 30 seconds until a echo fo > /var/www/test is propagated to another client. how can I speed this up? Preferably would be one second. I know this will kill some performance
00:14 hgichon joined #gluster
00:15 Romeor joined #gluster
00:18 Mr_Psmith joined #gluster
00:24 nishanth joined #gluster
00:30 mlhamburg joined #gluster
00:35 Champi joined #gluster
00:47 SpComb joined #gluster
00:47 plarsen joined #gluster
01:01 EinstCrazy joined #gluster
01:02 kminooie joined #gluster
01:05 zhangjn joined #gluster
01:05 kminooie hi again, does this mean anything to anyone? http://fpaste.org/292578/  the debug output is for that ls command. also found other posts talking about having to turn off acl to not get remote io error . is this a thing (turning off acl) ?  why is rpcdebug showing this line : "NFS reply getacl: -121"
01:05 glusterbot Title: #292578 Fedora Project Pastebin (at fpaste.org)
01:13 nicnicnic joined #gluster
01:16 nicnicnic left #gluster
01:17 TheSeven joined #gluster
01:22 Merlin__ joined #gluster
01:25 shyam joined #gluster
01:30 rideh joined #gluster
01:32 Lee1092 joined #gluster
01:41 dlambrig_ joined #gluster
01:53 kovshenin joined #gluster
01:54 kminooie left #gluster
01:58 DV joined #gluster
01:59 Romeor joined #gluster
02:01 nangthang joined #gluster
02:20 Romeor joined #gluster
02:22 gem_ joined #gluster
02:29 harish joined #gluster
02:32 haomaiwa_ joined #gluster
02:35 DV__ joined #gluster
02:38 haomaiwa_ joined #gluster
02:45 dlambrig_ joined #gluster
02:50 kdhananjay joined #gluster
02:59 dlambrig_ joined #gluster
03:07 jtux joined #gluster
03:07 overclk joined #gluster
03:23 hchiramm_home joined #gluster
03:38 atinm joined #gluster
03:38 sakshi joined #gluster
03:44 dgandhi joined #gluster
03:45 dgandhi joined #gluster
03:47 dgandhi joined #gluster
03:48 dgandhi joined #gluster
03:50 dgandhi joined #gluster
03:51 dgandhi joined #gluster
03:54 dgandhi joined #gluster
03:56 mlncn joined #gluster
04:01 nbalacha joined #gluster
04:04 spalai joined #gluster
04:04 kanagaraj joined #gluster
04:06 gem joined #gluster
04:09 rideh joined #gluster
04:10 jatb joined #gluster
04:14 trapier joined #gluster
04:16 shubhendu joined #gluster
04:18 swebb joined #gluster
04:22 [7] joined #gluster
04:24 spalai1 joined #gluster
04:29 zhangjn joined #gluster
04:51 overclk joined #gluster
04:53 pppp joined #gluster
04:56 jiffin joined #gluster
05:03 ppai joined #gluster
05:04 kotreshhr joined #gluster
05:07 Manikandan joined #gluster
05:07 spalai joined #gluster
05:08 ramky joined #gluster
05:08 hgowtham joined #gluster
05:09 skoduri joined #gluster
05:10 neha_ joined #gluster
05:11 zhangjn joined #gluster
05:12 hgowtham_ joined #gluster
05:18 zhangjn joined #gluster
05:20 ashiq joined #gluster
05:25 kotreshhr joined #gluster
05:27 vimal joined #gluster
05:27 mjrosenb joined #gluster
05:28 kotreshhr left #gluster
05:38 anil_ joined #gluster
05:40 rafi joined #gluster
05:42 vmallika joined #gluster
05:49 bharata-rao joined #gluster
05:52 R0ok_ joined #gluster
05:57 spalai left #gluster
05:59 haomaiwa_ joined #gluster
06:12 Saravana_ joined #gluster
06:14 kshlm joined #gluster
06:17 Humble joined #gluster
06:18 haomaiwang joined #gluster
06:24 overclk joined #gluster
06:29 raghu joined #gluster
06:34 spalai joined #gluster
06:37 hchiramm joined #gluster
06:39 atalur joined #gluster
06:46 davidself joined #gluster
06:51 Akee joined #gluster
06:52 mlhamburg1 joined #gluster
06:52 spalai joined #gluster
06:53 haomaiwa_ joined #gluster
06:55 nangthang joined #gluster
06:59 Kenneth__ joined #gluster
07:02 haomaiwa_ joined #gluster
07:03 overclk joined #gluster
07:05 mhulsman joined #gluster
07:20 primehaxor joined #gluster
07:20 [Enrico] joined #gluster
07:26 primehaxor joined #gluster
07:26 haomaiwa_ joined #gluster
07:44 [Enrico] joined #gluster
08:02 haomaiwa_ joined #gluster
08:07 Merlin__ joined #gluster
08:09 Merlin__ joined #gluster
08:17 fsimonce joined #gluster
08:18 zhangjn joined #gluster
08:20 armyriad joined #gluster
08:24 haomaiwa_ joined #gluster
08:25 F2Knight joined #gluster
08:30 kmkswamy joined #gluster
08:30 mhulsman1 joined #gluster
08:32 kattamm joined #gluster
08:33 kattamm how is data replication done in distributed-replicated mode. I have 3 servers with 2 disks and require 1 volume
08:34 kattamm making a total of 6 bricks
08:35 kattamm what is the best way to set this up structure
08:35 jiffin kattamm: say if you create 2x2 volume (total of four bricks)
08:35 kattamm with replication of 2
08:36 jiffin yes
08:36 kattamm jiffin: you mean only 2 node with 2 disks
08:36 zhangjn joined #gluster
08:36 kattamm at that time the replicated data will be placed in the same machine right
08:37 jiffin kattamm: say from x nodes and x disks , i can create  volume of 2x2
08:37 kattamm yes
08:37 jiffin there are two pairs
08:37 jiffin containing two bricks each
08:38 jiffin data will be distributed among those pairs
08:38 jiffin and inside each pair data will be replicated
08:39 kattamm yes, then how do i create the volume server1:disk1 server2:disk1 server1:disk2 server2:disk2 as the order in create volume command?
08:39 jiffin kattamm: it ways to keep replicated brick in another node
08:39 jiffin kattamm: that will fine
08:40 kattamm does gluster take care that the replicated copy is not present in the same server? and does not care about the order in which the bricks are specified in the create command
08:41 jiffin kattamm: here the order matters
08:41 jiffin here the pair one will be (server1:disk1 server2:disk1)
08:42 jiffin i.e same data will be replicated on server1:disk1 and server2:disk1
08:44 kattamm i am using https://github.com/shortdudey123/chef-gluster cookbook to spinup the setup and setup and he create the volume as server1:disk1 server1:disk2 server2:disk1 server2:disk2. which i feel stepping on my foot
08:44 glusterbot Title: shortdudey123/chef-gluster · GitHub (at github.com)
08:45 shortdudey123 kattamm: whats up?
08:46 kattamm hey thanks, i was looking for help on setting up a gluster cluster with your cookbook
08:46 kattamm a 3 node 2 disk , distributed-replicated set
08:46 shortdudey123 ah, yeah i think i put logic in there that breaks other people... need to test the PR thats there
08:47 shortdudey123 haven't tried more nodes w/ more than 1 disk actually
08:47 kattamm ok
08:48 shortdudey123 yeah
08:48 kattamm then may be i will create volume-group of the 2 disks to get more space per brick
08:49 shortdudey123 thats one option, i am just reading the history here a sec to get context
08:49 kattamm or is it better to move to distributed-stripped
08:49 kattamm k
08:49 shortdudey123 distributed-stripped has no fault tolerance :p
08:50 kattamm bummer.. i cant go to prod with that
08:51 shortdudey123 yup hahaha
08:53 kattamm what would be a good config for a 1TB availability, i am using aws.
08:54 shortdudey123 i need to solve that thought too... i eventually need triple digit TB
08:54 shortdudey123 keep in mind that the bigger the disks are the longer recovery takes if you have to replace one
08:54 msvbhat kattamm: You can create 3*2. 3 way distributed and 2 way replicated. Using all 6 bricks
08:55 kattamm yes.. learnt it the hardway
08:55 shortdudey123 personally i wouldn't go much more than 500gb per replca but its up to you
08:56 bhuddah for real?
08:56 ivan_rossi joined #gluster
08:56 kattamm ya, some guy messed up hosts files and i had to rebuild a 1 tb cluster after it failed to recover after a week
08:56 shortdudey123 i have a 3 node replicated thats 300gb
08:56 kattamm painful days
08:57 shortdudey123 wow that sucks :/
08:58 bhuddah are you talking of that small sizes for real?
08:58 bhuddah well. i'll report back in a few weeks.
08:58 kattamm for real
08:59 kattamm this is a temp storage in staging
08:59 bhuddah interesting.
08:59 bhuddah and why do you use gluster for that?
08:59 shortdudey123 i replaced a node on a 3x2 w/ 300gb per repla set and it took a day or 2
08:59 shortdudey123 (that was AWS thought between AZ's)
08:59 kattamm same aws setup
09:00 shortdudey123 Facebook runs their gluster clusters in the PB
09:00 kattamm i have 2 more setups with 3tb but running mirrored in prod
09:01 bhuddah mhm. i'm planning to set up a ~130TB installation soon.
09:01 mhulsman joined #gluster
09:01 haomaiwa_ joined #gluster
09:02 shortdudey123 bhuddah: i need to do double that in the coming months :/
09:02 shortdudey123 yeah
09:03 kattamm https://gist.github.com/kmkswamy/07f01cf3c6501932545b
09:03 glusterbot Title: volume status · GitHub (at gist.github.com)
09:03 shortdudey123 kattamm: anyway, what was your origional question? just how to do 3x2?
09:03 shortdudey123 ah i see
09:03 kattamm this is the config i have, i want to confirm that the replicas do not reside on the same machine
09:03 shortdudey123 you have 3 hosts and want a replica on each of them
09:03 Merlin__ joined #gluster
09:03 kattamm yes
09:04 bhuddah shortdudey123: mhm. yeah, well this is the initial phase. i've tried to plan it in a way that'll allow easy growing.
09:04 shortdudey123 (Brick1, Brick2, Brick3) are one replica and (Brick4, Brick5, Brick6) are the other replica
09:04 bhuddah but for me it's all real hardware and not aws this time.
09:04 kattamm brick1 and brick2 are on the same machine
09:04 shortdudey123 lucky haha
09:04 shortdudey123 yes
09:06 kattamm so if i lose server1, i loose data
09:08 bhuddah :(
09:09 kattamm seems like i will go for 3*3 replicated setup for the sanity of my manager :(
09:10 kattamm or change the create command to jumble the volume create command causing me to move away from the cookbook
09:10 shortdudey123 2nd one would be the option
09:10 shortdudey123 i am testing some stuff right now w/ the new PR w/ is a big rewrite
09:10 aravindavk joined #gluster
09:10 kattamm yes, i saw that
09:11 Norky joined #gluster
09:14 kattamm <shortdudey123> any timeline you are looking for the PR?
09:14 shortdudey123 testing it right now haha
09:14 kattamm lol, you said weekend
09:15 shortdudey123 by the weekend
09:15 shortdudey123 'before the weekend' :)
09:16 kattamm i will test this tomorrow (technically in 6 hrs)
09:17 kattamm thanks <shortdudey123> <jiffin> . Will catch some sleep now :)
09:17 shortdudey123 haha ok, mention me later if you need anything from me
09:18 jiffin kattamm: no problem
09:18 kattamm i will be lurking here
09:20 kovshenin joined #gluster
09:21 hchiramm joined #gluster
09:21 Humble joined #gluster
09:25 hchiramm_ joined #gluster
09:27 haomaiwa_ joined #gluster
09:27 Merlin__ joined #gluster
09:28 Humble joined #gluster
09:31 mhulsman joined #gluster
09:31 zhangjn joined #gluster
09:33 ppai joined #gluster
09:42 haomaiwa_ joined #gluster
09:47 Merlin__ joined #gluster
09:47 Slashman joined #gluster
09:47 arcolife joined #gluster
09:49 madnexus joined #gluster
09:50 Merlin__ joined #gluster
09:51 zoldar I have a 3 node setup with one arbiter (3.7.6 at the moment). The volume contains 1-2 big KVM image files. Now every now and then a self-heal operation is underway. There's no loss of connectivity during that time or anything like that. Is that normal? I'm asking because I have a script for nagios monitoring, which treats a presence of files in healed state as critical - I will turn it down to warning if that's nothing unusual.
09:56 madnexus joined #gluster
10:01 [Enrico] joined #gluster
10:06 kovshenin joined #gluster
10:17 haomaiwa_ joined #gluster
10:29 haomaiwa_ joined #gluster
10:35 nadley hi all
10:36 bluenemo joined #gluster
10:44 [Enrico] hi there. Is there any solution to the SELinux problem about glusterfsd not being able to write to (for example) /var/run/ea534bd0cf6dd830352f407206a58b94.socket ? Socket name seems not to be predictable so adding an fcontext to the profile is not possible
10:44 [Enrico] it is not in a dedicated subfolder either
10:44 [Enrico] (like /var/run/glusterfs/ that would help)
10:46 csim you could put glusterd in a permissive domain as a workaround
10:46 [Enrico] that's cheap come on :)
10:47 [Enrico] I was looking for a workaround letting me keep selinux
10:47 csim it let you keep selinux, but not for glusterd :/
10:47 [Enrico] it used to work in gluster 3.4 times
10:47 csim what AVC do you have ?
10:47 [Enrico] oh you mean only glusterd
10:47 [Enrico] that would be fine for me
10:48 [Enrico] csim, I use targeted mode, the default in red hat based systems
10:48 [Enrico] and gluster from gluster.org
10:49 csim csaba: yeah, but in audit.log ? the selinux error message ?
10:50 csim arfgfdfsf
10:50 csim [Enrico]: yeah, but in audit.log ? the selinux error message ?
10:50 * csim is on a train and with spotty wifi
10:50 [Enrico] type=AVC msg=audit(1447997549.221:336021): avc:  denied  { write } for  pid=5308 comm="glusterd" name="ea534bd0cf6dd830352f407206a58b94.socket" dev=dm-0 ino=526022 scontext=unconfined_u:system_r:glusterd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=sock_file
10:51 ppai joined #gluster
10:51 csim ok, indeed, nothing suspicious
10:51 [Enrico] csim, I added gluster to permissive for now, thank you for the idea, I didn't thought about it eheheh
10:51 [Enrico] csim, nono, that's just the wrong default context for the socket file
10:51 [Enrico] being named randomly no context could be preset so it inherits the parent folder one
10:53 csim yeah, but I wonder if it should have some kind of transition, like file created by glusterd end being some context, etc
10:53 [Enrico] nope
10:53 [Enrico] I think it's just easier to put the socket in a dedicated subfolder like and set the correct context for that
10:54 EinstCrazy joined #gluster
10:56 mhulsman joined #gluster
10:57 csim yeah, and it would also be safer, in case there is a collision ( which might never happen in practice )
10:58 [Enrico] agreed
10:58 csim and also cleaner when someone look and ask "what is this socket" :)
10:59 [Enrico] true it's not obvious at all who created it by looking at stats. Need to look at who has it open
10:59 [Enrico] actually also a fixed prefix could work, but I like the subfolder idea better
11:11 nishanth joined #gluster
11:28 overclk joined #gluster
11:30 atalur joined #gluster
11:35 harish joined #gluster
11:43 jwd joined #gluster
11:45 primehaxor joined #gluster
11:47 itisravi joined #gluster
11:48 mhulsman joined #gluster
11:48 mlncn joined #gluster
11:50 ppai joined #gluster
11:50 mhulsman1 joined #gluster
11:53 spalai joined #gluster
11:55 csaba csim: how can I be of your help with selinux? I do not have en expertise in selinux apart from recognizing by good chance those cases when something misteriosly doesn't work and in fact it's because of some selinux mess.
11:55 csim csaba: nope, it was a error of tabulation due to the slowness of my link, sorry
11:55 csaba csim: ok, np
11:58 aravindavk joined #gluster
12:02 zhangjn joined #gluster
12:03 zhangjn joined #gluster
12:04 haomaiwang joined #gluster
12:05 nbalacha joined #gluster
12:09 primehaxor joined #gluster
12:13 neha_ joined #gluster
12:26 mhulsman joined #gluster
12:28 primehaxor joined #gluster
12:38 ira joined #gluster
12:40 mhulsman1 joined #gluster
12:49 creshal joined #gluster
12:49 mhulsman joined #gluster
12:50 creshal Are there any downsides to using Gluster over NFS versus using Gluster over Fuse? I have to admit, I've never used NFS before, and the much better performance (more than order of magnitude) seems to make it preferable over gluster-fuse.
12:52 B21956 joined #gluster
12:53 Mr_Psmith joined #gluster
12:54 jiffin creshal: I don't have the exact no.s, for writes fuse mount should have upper hand
12:54 mhulsman1 joined #gluster
12:54 tomatto joined #gluster
12:55 jiffin but for nfs, since there is client side caching, the reads should have better performance
12:55 creshal jiffin: We have extremely read-heavy workloads with lots of tiny files, yes.
12:55 Manikandan joined #gluster
12:56 creshal Like "one write per week, one read per second"
12:57 jiffin creshal: as said before, I don't have any no.s to prove , but theoretically NFS will be a better choice for you
12:58 creshal Yeah, that order of magnitude improvement was from my tests on production data sets. :) 5 seconds for every run with gluster-fuse, 3 seconds on the first and 0.3 seconds on any further runs with NFS.
12:58 mhulsman joined #gluster
12:58 creshal So I just want to know whether there's any catch.
12:58 jiffin creshal: :)
12:59 creshal Obviously there's the usual locking thing on NFS, but I'm not using any databases on it. Just plain ol' files for web services and stuff.
13:06 Trefex joined #gluster
13:12 nadley question of the day :-), I have a gluster setup to provide a volume using two bricks and replica 2. I would like to replace one server by a new one how should I do that ?
13:13 creshal gluster volume replace-brick, I think?
13:13 nadley creshal: but it will resync all data no ?
13:14 creshal Good question.
13:15 nadley because in the new server I would like to replug the harddrive to avoid a full sync..
13:16 Manikandan joined #gluster
13:20 ctria joined #gluster
13:22 jiffin nadley: did u go through http://gluster.readthedocs.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick , the doc may be helpful
13:22 glusterbot Title: Managing Volumes - Gluster Docs (at gluster.readthedocs.org)
13:23 RedW joined #gluster
13:27 primehaxor joined #gluster
13:29 unclemarc joined #gluster
13:31 nadley jiffin: going to look thx
13:32 nadley jiffin: I wasn't knowing about gluster doc on RTD
13:34 nadley jiffin: regarding your link they start with a new brick which is empty. In my case I'll preferer to avoid starting from an empty brick
13:36 jiffin nadley: I don't try it my own,
13:46 nishanth joined #gluster
13:47 kenansulayman joined #gluster
13:49 mhulsman1 joined #gluster
13:51 DV joined #gluster
13:57 ashiq joined #gluster
14:00 karnan joined #gluster
14:01 mhulsman joined #gluster
14:03 jermudgeon joined #gluster
14:06 primehaxor joined #gluster
14:10 ctria joined #gluster
14:19 shaunm joined #gluster
14:19 arcolife joined #gluster
14:34 hamiller joined #gluster
14:36 ashiq joined #gluster
14:52 MrAbaddon joined #gluster
14:58 nishanth joined #gluster
14:58 spalai joined #gluster
15:00 skylar joined #gluster
15:02 rafi joined #gluster
15:04 spalai left #gluster
15:05 bennyturns joined #gluster
15:05 bennyturns joined #gluster
15:06 bennyturns joined #gluster
15:11 ayma joined #gluster
15:17 squizzi_ joined #gluster
15:20 mhulsman1 joined #gluster
15:21 lpabon joined #gluster
15:29 pppp joined #gluster
15:30 trapier joined #gluster
15:32 maserati joined #gluster
15:33 bennyturns joined #gluster
15:42 hagarth_ joined #gluster
15:46 plarsen joined #gluster
15:46 rafi joined #gluster
15:55 primehaxor joined #gluster
16:00 Manikandan joined #gluster
16:01 ayma1 joined #gluster
16:02 jiffin joined #gluster
16:14 vimal joined #gluster
16:37 rafi joined #gluster
16:46 rafi joined #gluster
17:02 Manikandan joined #gluster
17:07 coredump joined #gluster
17:13 Merlin__ joined #gluster
17:16 calavera joined #gluster
17:17 arcolife joined #gluster
17:20 mobaer joined #gluster
17:22 josh__ joined #gluster
17:22 josh__ \connect #pfsense
17:24 josh__ joined #gluster
17:25 BrettM I was wondering if someone could explain some behavior to me: when bringing a brick-server back online (with ~300 files need heailng) If the cluster.data-self-heal, cluster.entry-self-heal, cluster.metadata-self-heal) are enabled the fuse mount hangs for awhile (40-60s), however if i disable these parameters it does not appear to hang  -- I thought self healing was suppose to be a background task?
17:27 coredump joined #gluster
17:34 semiosis BrettM: pretty sure that, although there is a background self heal, the hang happens because your accessing the mount causes an immediate self heal check for the files/directories you're accessing.  just a guess.
17:36 BrettM semiosis, thank you for your input.  it seems that if i disable those parameters during node start, and then re-enable them ~1 minutes after start then the disruption in service appears much less.
17:36 skoduri joined #gluster
17:40 ivan_rossi left #gluster
17:42 adamaN joined #gluster
17:47 Merlin__ joined #gluster
17:56 social joined #gluster
18:02 mobaer1 joined #gluster
18:10 mobaer Hi, I have a strange problem using rdma mounts on a two node replicated setup (two nodes with replicated bricks, two nodes as clients). The volumes are set to transport-type tcp,rdma, all systems running Centos7.1. Running the glusterd and glusterfsd services via systemd allows me to mount the volumes in transport mode tcp on the client side. However rdma mounts are not possible ( rdma_cm_event_rejected Error 8 or something lik
18:12 mobaer Trying to debug the problem I stopped glusterd (systemctl stop glusterd) restarted glusterfsd (systemctl restart glusterfsd) and started glusterd --debug in the shell. That moment mounting also works with rdma transport-type.
18:13 Rapture joined #gluster
18:13 mobaer Sombody has a hint on how to get rdma working using systemd?
18:13 jiffin mobaer: you can check with rastar or rafi when they are online
18:14 jiffin mobaer: can u check with behavior of rdma only volume?
18:15 jiffin mobaer: the volume is not converted from tcp to tcp,rdma right?
18:16 jiffin mobaer: if yes , glusterd restart is required
18:17 mobaer jiffin: not converted, freshly created (multiple times), had quite some restarts with the systemd service. Will give an rdma only volume a try lateron, did not work in the past, but that was running another OFED...
18:18 mobaer jiffin: thanks for feedback!
18:19 jiffin mobaer: no problem
18:28 hagarth_ joined #gluster
18:29 bennyturns joined #gluster
18:30 Manikandan joined #gluster
18:36 BrettM so according to this post: http://www.gluster.org/pipermail/gluster-users.old/2015-January/020197.html, there is a difference between self healing occuring when mounting occurs, versus self healing based on the self healing daemon
18:37 bennyturns joined #gluster
18:41 BrettM empirical tests indicate that self-healing on mount is MUCH more disruptive than the self-healing daemon
18:42 JoeJulian I thought pranith said that (at least in 3.7) clients don't do self-heal anymore and it's all managed by shd.
18:43 cabillman joined #gluster
18:44 calavera joined #gluster
18:44 BrettM well my constant work load test hangs every time i bring a brick server back on line with cluster.*self-heal set to "on" and it doesn't when they are all set "off".
18:45 JoeJulian what does the client log show/
18:45 JoeJulian ?
18:45 BrettM when i profiled it with them "on"  i would get a huge storm of lookups
18:45 BrettM and the creating of a file would be hanging on getting a response to the lookup
18:46 JoeJulian But it doesn't say self-heal, so the block must be coming from somewhere else.
18:46 JoeJulian Are you on 3.7?
18:47 BrettM 3.7.6
18:47 JoeJulian ok
18:48 BrettM client log would need debug enabled for the volume?
18:49 BrettM because i dont see anything very useful with Info
18:51 JoeJulian Maybe, but I kind-of doubt it would help much. If I were tracking down the block, I would probably consider trace logs at the client and server. I might also look at the traffic with wireshark. I suspect that the client sends the lookup and then just sits there waiting for a response which would imply the delay is in the server code.
18:52 BrettM agreed, that is the feeling I have
18:53 BrettM in fact i turned on debug in the server and would see the lookup held as "least priority"
18:53 JoeJulian Oh, wow.. that's a good find.
18:53 JoeJulian Can you share that bit of log?
18:53 BrettM which didn't make sense to me because i thought the purpose of least priority was specifically to prioritize user traffic over self-heal traffic
18:54 JoeJulian Right
18:54 BrettM ok, i will have to dig through and find
18:54 JoeJulian That would explain why people have been complaining of slowdowns during heal events.
18:54 JoeJulian It's probably just a bug.
19:07 kminooie joined #gluster
19:16 CP|AFK joined #gluster
19:33 mlncn joined #gluster
19:35 BrettM JoeJulian, so here is an example from the client log of a 4 second lookup delay that occurs immediately after a brick server coming back online:
19:35 BrettM [2015-11-20 19:11:20.957246] D [MSGID: 0] [dht-common.c:2282:dht_lookup] 0-eschat_message-dht: calling revalidate lookup for /EPTT_clienttest/bb/30131-summary at eschat_message-replicate-0
19:35 BrettM [2015-11-20 19:11:20.979881] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-eschat_message-replicate-0: performing entry selfheal on a70d588b-1e8c-403d-a3a9-5ba1fc5b952c
19:35 BrettM [2015-11-20 19:11:24.601907] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-eschat_message-replicate-0: Completed entry selfheal on a70d588b-1e8c-403d-a3a9-5ba1fc5b952c. source=0 sinks=1
19:35 BrettM [2015-11-20 19:11:24.607488] D [MSGID: 0] [dict.c:166:key_value_cmp] 0-eschat_message-replicate-0: 'glusterfs.content' is different in two dicts (61, 0)
19:35 BrettM [2015-11-20 19:11:24.611662] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-eschat_message-replicate-0: performing metadata selfheal on 525fd7b6-3352-4e77-bfce-32e73bd6b14f
19:35 BrettM [2015-11-20 19:11:24.620132] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-eschat_message-replicate-0: Completed metadata selfheal on 525fd7b6-3352-4e77-bfce-32e73bd6b14f. source=0 sinks=1
19:35 BrettM [2015-11-20 19:11:24.621705] D [MSGID: 0] [dht-common.c:720:dht_revalidate_cbk] 0-eschat_message-dht: revalidate lookup of /EPTT_clienttest/bb/30131-summary returned with op_ret 0
19:35 semiosis BrettM: ,,(paste)
19:35 glusterbot BrettM: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
19:35 semiosis BrettM: ,,(pastebin)
19:35 glusterbot BrettM: I do not know about 'pastebin', but I do know about these similar topics: 'paste', 'pasteinfo'
19:35 ramky joined #gluster
19:35 semiosis ah i dont know anymore
19:35 semiosis but a pastebin-like site is more helpful than pasting in channel
19:37 BrettM ah, thank you semiosis
19:37 hgowtham_ joined #gluster
19:38 BrettM JoeJulian, http://termbin.com/zp7w
19:43 F2Knight joined #gluster
19:47 JoeJulian That's a client or shd?
19:48 BrettM JoeJulian, mnt-
19:48 BrettM which i believe is client?
19:48 JoeJulian That's odd.
19:48 BrettM and so i have a series of reads that all produce that behavior until the mount-based healing completes
19:49 BrettM err mount-triggered
19:50 BrettM so i think i was erroneous in saying before that it "hangs for 40-60 seconds"   it really should be "poor performance for 40-60 seconds"
19:53 BrettM JoeJulian, i looked for reference to the file being revalidated in the brick log file, however there was no debug that i could key on the file name
19:56 kminooie joined #gluster
19:59 kminooie hi I post this last night as well. don't know if anybody got a chance to look at it. I am trying to mount a 3.6 gluster over nfs. but I am getting error first It seemed to be an autofs issue but now I am getting the same error whether I use autofs or use mount on command line. it mounts but it generates error message when I try to access it : http://fpaste.org/292578/
19:59 glusterbot Title: #292578 Fedora Project Pastebin (at fpaste.org)
19:59 kminooie I am not sure what the correct result of getacl should be, but -121 doesn't seem right to me
20:00 JoeJulian Do you see any issues in /var/log/glusterfs/nfs.log on the server to which you're trying to connect?
20:00 srsc joined #gluster
20:04 mhulsman joined #gluster
20:05 kminooie JoeJulian: no, nothing at the time that the error is happening. that is why I tried rpcdebug . I also found Bug 1238318, but that one is reported on 3.7.2. mine is 3.6.6
20:06 kminooie https://bugzilla.redhat.com/show_bug.cgi?id=1238318
20:06 glusterbot Bug 1238318: medium, unspecified, ---, ndevos, ASSIGNED , NFS mount throws Remote I/O error
20:08 kminooie that bug report ^^^ says "NFS is showing Remote I/O errors. When disable and re-enable acl, it works." how do I do that? disable and re-enable acl? or just leave it disable :)
20:12 JoeJulian What filesystem is on your bricks?
20:13 JoeJulian gluster volume set nfs.acl off
20:13 kminooie ext3
20:13 JoeJulian gluster volume set $vol nfs.acl off
20:14 JoeJulian ext3, unless you mounted it with the acl option, doesn't support acl. That may be a factor.
20:14 kminooie JoeJulian: while I have you, so what is the deal with restarting gluste? why none of the processes die when i stop it? should i alway just pkill gluster?
20:15 JoeJulian Because there are occasions where you want to restart a specific service or the management daemon itself. Most people do not want their bricks to go offline for every event.
20:15 kminooie this one is not even a gluster. it is just a single server running gluster and it doesn't even have the health daemon. why doesn't it die when I stop it?
20:16 JoeJulian That doesn't grok.
20:16 JoeJulian @glossary
20:16 glusterbot JoeJulian: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
20:18 kminooie so when you need to actually restart it you have to actually kill it? as in that is the official way of doing that?  ( sorry that line was .. this one is not even a cluster ...  I meant that I have only one node on this volume )
20:21 JoeJulian So let's say you've found an issue with glustershd and want to try restarting it. Would you rather stop glusterd, kill that one process, then start glusterd? or would you rather have everyone stop what they're doing so you can bounce the storage server?
20:21 JoeJulian If you're going to do that, you might as well be running windows and have to reboot for everything. ;)
20:24 kminooie I get the argument but shouldn't it result in seperate init files for glusterfsd and glusterfshd so they can be operate independently? and also I am talking about restarting the glusterfsd itself. it just never goes down. regardless of the fact that it is working or not :)
20:25 kminooie anyway I am restarting everything so I can test that acl thingy. what was it ? nfs.acl off
20:28 kminooie what is the opposite of volume set command? how can I verify that I've turned of acl?
20:29 kminooie volume info did the trick
20:36 kminooie ok it seems to be working after turning off acl. I still need to test autofs. I'll report back :) as alway thanks JoeJulian. If I ever get a chance to visit north, I'll buy you a drink
21:12 JoeJulian I humbly request that you spend that $4.00 on open-licensed cancer research. See the link on my blog.
21:39 calavera joined #gluster
21:40 mobaer jiffin: fyi: I can't get a rdma only volume started using systemd. I need to start glusterd in bash (using --debug or not doesn't change anything)
21:40 mobaer however both run as root, I can't find a difference between starting with systemd and bash
21:41 jiffin mobaer: even i don't have any idea about that
21:42 shaunm joined #gluster
21:44 JoeJulian mobaer: selinux?
21:47 mobaer JoeJulian: Jepp... Gooood hint... looking at it
21:51 dblack joined #gluster
21:59 linagee joined #gluster
22:06 mobaer JoeJulian: thanks, you're right. setenforce 0 on the servers and using systemctl made the volumes available for the clients. I'll have a look on how to use gluster with rdma and selinux in enforcing mode.
22:11 coredump joined #gluster
22:25 JoeJulian +1
22:25 JoeJulian Be sure to blog about your solution. That's valuable stuff.
22:30 skylar joined #gluster
22:31 skylar joined #gluster
22:33 hagarth_ joined #gluster
22:36 mobaer sure. usual suspects don't provide any hints so far... Will do some try and error during the weekend.
22:44 calavera joined #gluster
22:47 dcmbrown joined #gluster
22:50 plarsen joined #gluster
22:50 atrius_ joined #gluster
22:53 dcmbrown So I hate the "one of the bricks contain the other" issues with a couple of centos servers i am reconfiguring. I already have a preexisting volume that I'd like to have as little downtime as possible.  if I stop and remove the UUID file on one of the servers and then start it back up, is this going to adversely affect the current volume?
22:53 glusterbot dcmbrown: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
22:54 dcmbrown thanks glusterbot but not _quite_ the answer. :P
22:58 JoeJulian dcmbrown: No idea. That seems like it would play havoc with the peering process.
22:58 JoeJulian I would spin up a couple VMs and try it.
23:01 dcmbrown yeah that makes sense.  I should add that I'm trying to add a second volume (to eventually replace the first) which is why I'm running into this error.  This will work as a way to figure out the issues involved with the solution however. I thought I'd check to see if anyone has tried it first.
23:06 JoeJulian If I were going to make a second volume to replace the first: my original volume would have bricks at ${brick_filesystem_mountpoint}/brick. My new volume would then have bricks at ${brick_filesystem_mountpoint}/brick1. As I mv files from the original volume to the new volume, my space requirements would remain unchanged.
23:17 mlncn_ joined #gluster
23:17 MrAbaddon joined #gluster
23:20 scubacuda joined #gluster
23:21 gbox joined #gluster
23:27 cliluw Is there anything I can hook into to be notified when a Gluster volume is created?
23:29 cliluw Suppose I have servers A and B in my Gluster pool. If A calls "gluster volume create..." using a brick on A and a brick on B, server B will be hosting a volume but it won't know.
23:32 gbox Hi I'm enjoying gluster but an update to 3.7.6 created an odd situation.  I have a distributed-replicated volume across 3 nodes: #1 & #2 nodes with #3 replicating.  Peer status on #1 & #2 show Disconnected status from each other but Connected to #3.  Peer status on #3 shows no Connections.  Weird, no?
23:39 gbox gluster --remote-host=replication_host peer status shows all nodes Disconnected!  Has anyone seen that?
23:40 JoeJulian check the logs.
23:43 gbox JoeJulian: Thanks always a good way to start.  Logs are extensive.  Should I start with etc-glusterfs-glusterd.vol.log?
23:45 JoeJulian yes
23:45 JoeJulian That's the management daemon, and that's what manages the peering.
23:47 gbox Rough in there!  Lots of "0-nfs: readv on /var/run/gluster/7cd5ce6b4c0ca0110607045704f943ef.socket failed (Invalid argument)"
23:50 gbox The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-11-20 23:46:22.2911
23:50 gbox 32] and [2015-11-20 23:48:19.489840]
23:51 gbox Is the nfs service enabled automatically?
23:54 JoeJulian Yes, it is.
23:54 JoeJulian Disabling is per-volume.
23:55 gbox Interesting I'll look into that.  Probably not the issue here.  Does anything stand out in these errors & warnings: http://fpaste.org/292999/48063664/raw/ ?
23:57 srsc hello. having an issue where one particular gluster client (v 3.7.4) errors when reading more than a few hundred kB of a file and is generally very slow to list directories, etc
23:58 DRoBeR joined #gluster
23:58 srsc it complains about a bad file descriptor and the transport endpoint not being connected when it errors
23:59 JoeJulian Sounds like network to me.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary