Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-06-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 Jandre joined #gluster
00:12 doubt_ joined #gluster
00:33 social joined #gluster
00:34 fink joined #gluster
00:36 sankarshan_ joined #gluster
01:21 markd_ joined #gluster
01:22 Innomina1 joined #gluster
01:32 baoboa joined #gluster
01:33 kkeithley joined #gluster
01:35 edwardm61 joined #gluster
02:05 nangthang joined #gluster
02:12 RajeshReddy joined #gluster
02:13 Alssi_ joined #gluster
02:24 harish_ joined #gluster
02:38 gildub joined #gluster
02:41 nishanth joined #gluster
02:42 victori joined #gluster
02:52 bharata-rao joined #gluster
03:06 [7] joined #gluster
03:15 overclk joined #gluster
03:28 hagarth joined #gluster
03:49 sripathi joined #gluster
03:54 ccha2 joined #gluster
03:57 DV joined #gluster
03:58 ppai joined #gluster
04:01 atinmu joined #gluster
04:09 fink joined #gluster
04:15 dusmant joined #gluster
04:19 kanagaraj joined #gluster
04:23 shubhendu_|afk joined #gluster
04:31 spandit joined #gluster
04:33 nbalacha joined #gluster
04:36 kshlm joined #gluster
04:37 shubhendu_ joined #gluster
04:37 sakshi joined #gluster
04:41 rafi1 joined #gluster
04:46 doubt_ joined #gluster
04:46 victori joined #gluster
04:57 RameshN joined #gluster
04:57 shubhendu__ joined #gluster
05:00 kanagaraj_ joined #gluster
05:07 kanagaraj__ joined #gluster
05:08 ccha3 joined #gluster
05:10 pppp joined #gluster
05:10 hagarth joined #gluster
05:11 karnan joined #gluster
05:12 shubhendu__ joined #gluster
05:16 gem joined #gluster
05:18 hchiramm_pto joined #gluster
05:18 soumya joined #gluster
05:22 deepakcs joined #gluster
05:26 kdhananjay joined #gluster
05:26 Manikandan joined #gluster
05:26 ashiq joined #gluster
05:27 RameshN joined #gluster
05:27 maveric_amitc_ joined #gluster
05:29 vimal joined #gluster
05:39 kdhananjay joined #gluster
05:40 Bhaskarakiran joined #gluster
05:43 jiffin joined #gluster
05:44 arcolife joined #gluster
05:49 atalur joined #gluster
06:01 jiffin joined #gluster
06:02 ppai joined #gluster
06:02 hgowtham joined #gluster
06:05 dcroonen joined #gluster
06:24 schandra joined #gluster
06:33 hagarth joined #gluster
06:42 kshlm joined #gluster
06:43 amukherj_ joined #gluster
06:44 spalai joined #gluster
06:44 RajeshReddy joined #gluster
06:44 kdhananjay joined #gluster
06:45 doubt_ joined #gluster
06:46 rgustafs joined #gluster
06:47 aravindavk joined #gluster
06:48 coredump joined #gluster
06:54 raghu joined #gluster
07:07 nbalacha joined #gluster
07:09 [o__o] joined #gluster
07:09 nsoffer joined #gluster
07:13 saurabh_ joined #gluster
07:14 cuqa hello, I have a file on a volume which produces input/output errors
07:14 cuqa seems like split-brain situation on different nodes. Whats the best way to recover
07:15 cuqa deleting all files would be okay, but since I receive io errors, it seams that I cannot do this
07:19 Manikandan joined #gluster
07:20 anrao joined #gluster
07:26 autoditac joined #gluster
07:27 cyberbootje1 joined #gluster
07:28 victori joined #gluster
07:35 ppai joined #gluster
07:36 fsimonce joined #gluster
07:37 victori_ joined #gluster
07:37 coredump joined #gluster
07:43 Leildin joined #gluster
07:46 amukherj_ joined #gluster
07:54 gem joined #gluster
07:54 doubt_ joined #gluster
07:59 LebedevRI joined #gluster
08:04 glusterbot News from newglusterbugs: [Bug 1228093] Glusterd crash <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228093>
08:06 c0m0 joined #gluster
08:09 ctria joined #gluster
08:18 autoditac joined #gluster
08:19 s19n joined #gluster
08:25 cyberbootje1 joined #gluster
08:27 rafi1 joined #gluster
08:32 anil joined #gluster
08:34 glusterbot News from newglusterbugs: [Bug 1228100] Disperse volume: brick logs are getting filled with "anonymous fd creation failed" messages <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228100>
08:36 ToMiles joined #gluster
08:47 Slashman joined #gluster
09:00 Trefex joined #gluster
09:04 glusterbot News from newglusterbugs: [Bug 1228111] [Backup]: Crash observed when glusterfind pre is run after deleting a directory containing files <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228111>
09:17 sysconfig joined #gluster
09:31 spandit_ joined #gluster
09:43 arcolife joined #gluster
09:50 gildub joined #gluster
09:50 Trefex joined #gluster
09:50 ppai joined #gluster
10:04 glusterbot News from newglusterbugs: [Bug 1228142] clang-analyzer: adding clang static analysis support <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228142>
10:06 bene2 joined #gluster
10:08 Trefex joined #gluster
10:10 spandit joined #gluster
10:12 vimal joined #gluster
10:24 Trefex joined #gluster
10:24 joshin joined #gluster
10:33 harish_ joined #gluster
10:34 glusterbot News from newglusterbugs: [Bug 1228160] linux untar hanged after the bricks are up in a 8+4 config <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228160>
10:34 glusterbot News from newglusterbugs: [Bug 1228157] Provide and use a common way to do reference counting of (internal) structures <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228157>
10:41 vimal joined #gluster
10:42 ppai joined #gluster
10:50 rafi1 joined #gluster
10:51 rgustafs joined #gluster
10:53 spandit joined #gluster
11:07 dcroonen joined #gluster
11:09 Ramereth joined #gluster
11:16 arcolife joined #gluster
11:21 B21956 joined #gluster
11:22 jcastill1 joined #gluster
11:25 kkeithley joined #gluster
11:27 jcastillo joined #gluster
11:34 TheOtter joined #gluster
11:34 TheOtter Hi, am very new to gluster.  Tried setting up version 3.7 on 4 servers as "replica 2".  When I reboot 1 server writes stop on the others.  I tried setting "cluster.quorum-type: auto", are there other settings I need to change here pls?
11:38 ndevos ~ping-timeout | TheOtter
11:38 glusterbot TheOtter: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
11:39 ndevos TheOtter: "writes stop" is that until the other server gets back online, or would that be only for 42 seconds?
11:39 TheOtter good question
11:39 TheOtter let me check
11:42 gildub joined #gluster
11:43 TheOtter you are right, it does come back
11:43 rgustafs joined #gluster
11:44 jvandewege_ joined #gluster
11:44 TheOtter I would have hoped that it would say "can I reach at least 50% of my hosts", ok, I'll carry on and ler you write?
11:45 kkeithley2 joined #gluster
11:45 TheOtter but I guess I'm missing a key concept here
11:47 anoopcs ndevos, I think there is a grace timeout option too for re-acquiring the locks after server reboot.
11:47 jiffin1 joined #gluster
11:49 kkeithley joined #gluster
11:49 ppai joined #gluster
11:50 arcolife joined #gluster
11:53 ndevos anoopcs: yes, thare are different options for that, re-acquire on the client, release on the server if a client becomes unresponsive, and maybe more
11:58 TheOtter Thanks for the help guys, starting to make a little more sense to me now
11:58 jvandewege joined #gluster
11:58 ToMiles joined #gluster
11:59 mator ndevos, can you please look at https://bugzilla.redhat.com/show_bug.cgi?id=847821 ? thanks
11:59 glusterbot Bug 847821: low, medium, ---, bugs, NEW , After disabling NFS the message "0-transport: disconnecting now" keeps appearing in the logs
12:00 jvandewege joined #gluster
12:03 meghanam joined #gluster
12:10 Manikandan_ joined #gluster
12:18 jiffin joined #gluster
12:18 Trefex joined #gluster
12:23 mkzero joined #gluster
12:24 Trefex joined #gluster
12:28 cuqa hello, should 'gluster volume heal <brick> info' return 0 entries?
12:35 glusterbot News from newglusterbugs: [Bug 1228216] Disperse volume: gluster volume status doesn't show shd status <https://bugzilla.redhat.co​m/show_bug.cgi?id=1228216>
12:35 DV_ joined #gluster
12:36 raghu joined #gluster
12:46 bene2 joined #gluster
12:48 ppai joined #gluster
12:54 wkf joined #gluster
12:56 Jandre joined #gluster
12:59 dgandhi joined #gluster
13:00 jayunit1000 joined #gluster
13:03 stickyboy joined #gluster
13:04 julim joined #gluster
13:04 deepakcs joined #gluster
13:05 ndevos mator: that sounds like something atin analyzed earlier, there should be an email in the archives about that
13:08 theron joined #gluster
13:08 ndevos mator: http://www.gluster.org/pipermail/g​luster-devel/2015-May/045164.html suggests bug 1222065
13:08 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1222065 high, high, ---, kparthas, MODIFIED , GlusterD fills the logs when the NFS-server is disabled
13:12 karnan joined #gluster
13:12 aaronott joined #gluster
13:17 meghanam joined #gluster
13:17 pppp joined #gluster
13:18 maveric_amitc_ joined #gluster
13:21 nangthang joined #gluster
13:24 georgeh-LT2 joined #gluster
13:27 Twistedgrim joined #gluster
13:27 prg3 joined #gluster
13:28 hamiller joined #gluster
13:31 meghanam_ joined #gluster
13:33 Bhaskarakiran joined #gluster
13:33 kkeithley2 joined #gluster
13:38 Bhaskarakiran_ joined #gluster
13:39 Trefex joined #gluster
13:41 chirino joined #gluster
13:42 arcolife joined #gluster
13:43 Bhaskarakiran_ joined #gluster
13:49 jmarley joined #gluster
13:52 plarsen joined #gluster
13:53 chirino joined #gluster
13:54 rafi1 joined #gluster
14:02 bene3 joined #gluster
14:03 marbu joined #gluster
14:10 wushudoin joined #gluster
14:12 Bhaskarakiran joined #gluster
14:12 bene2 joined #gluster
14:17 ToMiles any one know where to get the ububtu trusty package for 3.7 ? or have suggestions how I can build it myself?
14:26 csim_ joined #gluster
14:33 csim joined #gluster
14:33 ekuric joined #gluster
14:44 chirino joined #gluster
14:47 mbukatov joined #gluster
14:48 mator ndevos, thanks
14:54 maklemenz joined #gluster
14:57 arcolife joined #gluster
15:00 lpabon joined #gluster
15:01 prg3 joined #gluster
15:03 spalai joined #gluster
15:07 atinmu joined #gluster
15:08 julim joined #gluster
15:10 ghenry joined #gluster
15:10 ghenry joined #gluster
15:10 elico joined #gluster
15:22 hagarth joined #gluster
15:25 nbalacha joined #gluster
15:26 R0ok_ joined #gluster
15:32 Trefex joined #gluster
15:33 rwheeler joined #gluster
15:35 doubt_ joined #gluster
15:43 s19n left #gluster
15:44 ghenry joined #gluster
15:44 ghenry joined #gluster
16:02 meghanam joined #gluster
16:02 Gill joined #gluster
16:05 mkzero joined #gluster
16:05 julim joined #gluster
16:06 [7] JoeJulian: one more reason why removing replace-brick is bad:
16:06 plarsen joined #gluster
16:06 [7] during setup of my system I want to first create a replica1 distribute volume
16:07 [7] then I add a second host, switch to replica2
16:07 [7] then the third host, replica3
16:07 [7] at that point the system is operational, if I want to add more hosts later I add additional bricks without further increasing the replica count
16:07 [7] IIUC doing replace-brick while the volume is replica1 is fatal
16:08 [7] so I can't use that to move the correct brick numbers to the second host
16:08 [7] add-brick doesn't help either, it always adds the new bricks at the end
16:09 [7] so if I want to create the bricks with the correct size (3 bricks per host) from the beginning, I end up with the first 3 bricks being all on the same host
16:09 [7] is there any workaround for that?
16:10 victori joined #gluster
16:10 JoeJulian If you add the second host and add-brick replica2 <brick1> <brick2> <brick3> it will create the replicas on the second server.
16:20 jcastillo joined #gluster
16:21 [7] er, not sure if I understood that
16:21 [7] so assume I do on host A: gluster volume create vol replica 1 A:/b1 A:/b2 A:/b3
16:22 Trefex joined #gluster
16:22 [7] then I do on host B: gluster volume add-brick vol replica 2 B:/b4 B:/b5 B:/b6
16:23 [7] then I do on host C: gluster volume add-brick vol replica 3 C:/b7 C:/b8 C:/b9
16:23 [7] after that I'd assume that the volume contains the bricks in this order: A:/b1 A:/b2 A:/b3 B:/b4 B:/b5 B:/b6 C:/b7 C:/b8 C:/b9
16:24 [7] which, IIUC, would mean that A:/b1 A:/b2 and A:/b3 would be 3 replicas of the same data
16:24 [7] does add-brick inject the new bricks in different locations if it changes the replica count?
16:26 [7] JoeJulian: i.e. does it automatically interleave them, so that the volume contains A:/b1 B:/b4 C:/b7 A:/b2 B:/b5 C:/b8 A:/b3 B:/b6 C:/b9 in that order?
16:27 bennyturns joined #gluster
16:29 JoeJulian It does interleave them to satisfy the replication requirement.
16:29 nsoffer joined #gluster
16:30 ekuric joined #gluster
16:31 [7] so, generally speaking, when changing the replica count, I always have to supply (old_number_of_bricks * (new_number_of_bricks - old_number_of_bricks)) new bricks, and it will insert (new_number_of_bricks - old_number_of_bricks) consecutive ones of those after every (old-number_of_bricks) consecutive old bricks in the volume?
16:33 [7] btw, another set of old replace-brick CLI instructions is hiding on https://github.com/GlusterFS/Notes
16:34 [7] also if I want to clean everything up as if glusterd had never freshly installed on a system, in order to start over with a new attempt, what do I have to delete? just /var/lib/gluster contents?
16:36 JoeJulian Right, /var/lib/glusterd
16:47 B21956 left #gluster
16:49 coredump joined #gluster
16:56 spalai joined #gluster
16:59 cholcombe joined #gluster
16:59 Bhaskarakiran joined #gluster
17:03 CyrilPeponnet Hey guys I have the weirdest issue of the world
17:04 CyrilPeponnet using client 3.6.3 with server in 3.5.2
17:04 CyrilPeponnet from time to time, when un tar a tarball containing symlinks, they become files....
17:05 CyrilPeponnet ---------- 1 testlab testlab        0 Jun  4 09:48 cvs_tag
17:05 glusterbot CyrilPeponnet: --------'s karma is now -4
17:05 CyrilPeponnet hihi
17:05 CyrilPeponnet if I do it several time, some links are fine and sometime they are been converted to this strangeness
17:09 victori joined #gluster
17:11 victori joined #gluster
17:14 hagarth CyrilPeponnet: that is a known strange behavior of tar
17:14 CyrilPeponnet really?
17:15 hagarth yes, trying to look up my own mail in gluster-users/devel archives about this
17:15 CyrilPeponnet @hagarth it never occurred over nfs
17:15 CyrilPeponnet only since we start to use gfs
17:16 hagarth CyrilPeponnet: yes, the chances of occurrence is more with a distributed fs. .. IIRC the -P switch can be used to avoid this from happening
17:17 CyrilPeponnet we only have replicated vol
17:18 hagarth CyrilPeponnet: yes, this happens only with replicated volumes
17:19 CyrilPeponnet who, what is the underlying issue
17:20 hagarth CyrilPeponnet: http://www.gluster.org/pipermail/glu​ster-devel/2014-December/043356.html
17:20 haomai___ joined #gluster
17:21 CyrilPeponnet Doh
17:21 CyrilPeponnet indeed the -P fix the issue
17:21 CyrilPeponnet that's so weird
17:21 CyrilPeponnet but make sens with the explanation
17:21 CyrilPeponnet s/sens/sense/
17:21 glusterbot What CyrilPeponnet meant to say was: An error has occurred and has been logged. Please contact this bot's administrator for more information.
17:22 CyrilPeponnet @glusterbot you're buggy bro
17:22 [7] glusterbot-- :P
17:22 glusterbot [7]: glusterbot's karma is now 8
17:22 CyrilPeponnet Lol
17:23 Trefex joined #gluster
17:23 ToMiles joined #gluster
17:23 bennyturns joined #gluster
17:25 Rapture joined #gluster
17:25 hagarth lol
17:28 JoeJulian Yeah, maybe I'll get some time to look at that when I go on vacation later this month.
17:37 vimal joined #gluster
17:48 haomaiwang joined #gluster
17:50 B21956 joined #gluster
17:56 jiffin joined #gluster
17:57 CyrilPeponnet @glusterbot Can you shoot yourself in the foot?
17:58 CyrilPeponnet Damn! he's smarter than I thought...
18:08 haomaiw__ joined #gluster
18:12 [7] hm, add/remove-node is more clever than I thought
18:12 [7] which I kinda hate, because it makes it hard to tell what any command would do
18:12 [7] and there aren't really any docs on that either
18:21 [7] and, from what I can tell from my limited testing, a remove-brick operating can easily take weeks
18:21 [7] as it basically moves every single file in the whole FS
18:22 [7] so using that as a replacement for replace-brick doesn't work well
18:22 CyrilPeponnet I think it depends if you use replicated or distributed
18:22 [7] both
18:22 [7] replica3, at least 3*3 bricks
18:23 CyrilPeponnet with replicated it's instant for distributed I think I will rebalance files on the brick you want to remove (not sure but make sens)
18:23 [7] from what I observed it will start rebalancing everything everywhere for replicated as well
18:24 CyrilPeponnet *rebalance file on other brick
18:25 [7] the way it looks right now, adding a server to a replica 3 filesystem requires at least 3 consecutive rebalances if done via add-brick/remove-brick
18:25 [7] hm, or maybe I can get away with 2
18:25 [7] but there's really no point in doing more than 1
18:30 harish_ joined #gluster
18:33 [o__o] joined #gluster
18:56 rotbeard joined #gluster
20:26 bennyturns joined #gluster
20:37 spalai joined #gluster
20:48 victori joined #gluster
20:53 ira joined #gluster
20:58 victori joined #gluster
21:13 cholcombe gluster: does vol add-brick work for all translator types?  It only shows <stripe|replica> in the CLI docs
21:15 JoeJulian That simply states that those keywords may be used in that position.
21:15 JoeJulian Those would be used for changing the stripe or replica count.
21:15 Vortac seeing lots of [2015-06-04 21:14:57.857857] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/5f357526509​8894f71acfdfc3bb04b54.socket failed (Invalid argument)
21:15 Vortac in my etc-glusterfs-glusterd.vol.log - Any good pointers to troubleshooting this?
21:16 JoeJulian Vortac: Is nfs enabled?
21:16 JoeJulian Or, more correctly, is it disabled?
21:17 Vortac In volume status I seen NFS server on localhost online N - I do have nfs running
21:18 JoeJulian Does "gluster volume info" show nfs.disable as being set for your volumes?
21:18 Vortac JoeJulian: No
21:19 JoeJulian Did you upgrade recently?
21:19 Vortac JoeJulian: no - installed 3.7.0 - that's what currently running
21:20 Vortac I see there's an update for 3.7.1 avail
21:20 JoeJulian Oh, I think I saw something about that. Can you use 3.7.1?
21:20 Vortac JoeJulian: Sure.. was thinking about upgrading tonight.. What's best for the upgrade stop all (two) nodes? or can do rolling upgrade?
21:21 Vortac Rolling would be cool.. :)
21:23 al joined #gluster
21:25 [7] hm, so apparently the "clean" way to do things now is add-brick/remove-brick
21:26 [7] which will move around the data of at least 3 bricks, 2 of which will move inside the same partition and are completely unneccessary
21:26 [7] and I guess it doesn't use a move operation there, but rather copy&delete?
21:35 [7] http://pastie.org/pastes/10224102/text
21:35 [7] status stays like this indefinitely
21:40 kkeithley joined #gluster
21:41 wkf joined #gluster
21:42 [7] I really love to experiment with innovative technologies, but this seems way too unstable to ever trust it with real data
21:43 [7] right before that I had a case where I did the same "remove-brick" and it said completed, despite not having done anything. "remove-brick commit" ended up killing the only replica of all contents of that brick
21:47 [7] tried exactly the same again, this time I got "volume remove-brick start: failed: Volume test needs to be started before remove-brick (you can use 'force' or 'commit' to override this behavior)"
21:47 DV joined #gluster
21:47 [7] if course I stopped glusterd and deleted the contents of /var/lib/glusterd and the bricks between any attempts
22:05 corretico joined #gluster
22:06 [7] [2015-06-04 22:04:53.625523] E [glusterd-rebalance.c:917:glust​erd_defrag_event_notify_handle] 0-management: Failed to update status
22:06 [7] [2015-06-04 22:04:53.629487] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/gluster-rebalance-fa5​fe5d1-5e01-47c9-b004-ad426dcaaf0d.sock failed (No data available)
22:06 [7] [2015-06-04 22:04:53.693788] I [MSGID: 106007] [glusterd-rebalance.c:164:​__glusterd_defrag_notify] 0-management: Rebalance process for volume test has disconnected.
22:06 [7] ideas what might be causing this?
22:19 [7] reproduced the case again where it says completed despite not having replicated anything at all
22:20 Vortac JoeJulian: Just some feedback on the socket issue - upgraded to 3.7.1 and still getting the error on one node..
22:23 [7] I'm starting to get the impression that this whole mess is the result of replicate subvolumes not self-healing newly added bricks
22:25 aaronott joined #gluster
22:27 cholcombe JoeJulian, ok thanks :)
22:27 akay1 joined #gluster
22:27 afics joined #gluster
22:27 ndk joined #gluster
22:30 [7] so how do I trigger the healing of the new bricks after adding new bricks to increase the replica count?
22:30 cholcombe JoeJulian, vol add-brick should have a disperse param also no?
22:30 cholcombe [7]: i don't think you have to.  it occurs every 15 mins i think
22:31 cholcombe [7]: did you do a vol rebalance fix-layout ?
22:31 [7] fix-layout, IIUC, only affects the distribute on top of the replicate
22:31 [7] the problem here is the replicate below
22:32 [7] apparently adding additional replicas means that data will be inaccessible until these have healed
22:32 cholcombe you could force it with a find
22:32 [7] and operations such as removing other bricks will have catastrophic results (they migrate the empty brick and think they're done, discarding the good one)
22:33 DJClean joined #gluster
22:34 cholcombe just run a find on the mount. I think JoeJulian detailed the command on his blog
22:34 cholcombe that'll only work if you have inline heal on for the mount though i think
22:34 Drakonstein joined #gluster
22:36 kkeithley joined #gluster
22:37 [7] the fact that a not fully built replica will make files seem missing by itself gives me a very bad feeling
22:37 [7] it really shouldn't start using replicas before they've been fully built
22:38 cholcombe just keep an eye on the heal count
22:39 cholcombe what's your replica number?
22:39 [7] how does that help?
22:39 [7] I'm playing around with replica 1 to 3 here
22:39 [7] creating the volume as replica one on one server, then adding bricks from another server to bump it to replica2, then the same again to replica3
22:40 Drakonstein Is this an appropriate place to ask a question?
22:40 [7] sure
22:41 cholcombe yes
22:41 [7] just don't get annoyed by my rants :P
22:42 ProT-0-TypE joined #gluster
22:42 Drakonstein I used replace-brick to trade one a node in a triple replication system
22:42 Drakonstein the new one is building properly, but the other 2 are throwing a lot of errors
22:42 * [7] would be happy if he even got that far :P
22:42 Drakonstein [2015-06-04 22:41:56.723594] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-xvdb1-client-2: remote operation failed: No such file or directory. Path: <gfid:14c539a2-2fdf-43a5-bfc0-f13d1a1e014a> (14c539a2-2fdf-43a5-bfc0-f13d1a1e014a)
22:43 Drakonstein [2015-06-04 22:39:33.225084] E [afr-self-heal-entry.c:2353:afr​_sh_post_nonblocking_entry_cbk] 0-xvdb1-replicate-0: Non Blocking entrylks failed for <gfid:96027134-0265-4db6-aac5-e3680fb1b2fb>.
22:43 cholcombe no need to ask permission :)
22:43 Drakonstein those are the 2 types of errors being thrown on both servers
22:44 cholcombe i believe those errors are normal on the replica partners
22:44 [7] JoeJulian: do you consider it a bug if increasing the replica count through add-bricks makes the mounts of that volume suddenly appear empty?
22:44 cholcombe the self heal daemon could really use some better logging about what is really an error
22:45 Drakonstein All of the NFS mount points are freezing on a lot of requests after this was done.  Since those are the only errors being thrown, I assumed it was related ot a problem
22:46 cholcombe the nfs mounts are probably appearing to freeze because heals are happening in the background
22:46 cholcombe well in the foreground in your case
22:46 * [7] generally has the impression that incomplete replicas make glusterfs fail to handle requests appropriately
22:47 cholcombe i believe you can change that behavior but i don't remember the flags you need to set
22:47 [7] the whole redundancy seems to only work as lon as a brick is actually offline, but fails badly (at least in my tests - please show me what I'm doing wrong!) if a replica is incomplete or damaged, but reachable
22:49 Drakonstein I actually have the rebuilding node offline because the load was spiking much worse.
22:49 cholcombe oh you took it out of DNS round robin or whatever ip balancer you have?
22:49 Drakonstein I did remove it from the dns round robin, killed all gluster processes
22:50 cholcombe [7], I can't really comment because i've never tried to grow a replica set like that.  i've always bolted on 3x bricks or whatever my replica count was
22:50 Drakonstein And the mount points are still hanging accessing files
22:50 cholcombe is your replica 2?
22:50 cholcombe that might explain your hanging mounts
22:50 Drakonstein Our replication is set to 3.  so with 1 server out of the rotation, the other 2 are still active
22:51 Drakonstein we have 3 sets of those servers so 8 out of 9 are online
22:51 Drakonstein the other 2 clusters seem to be fine
22:51 cholcombe yeah that should be fine
22:53 cholcombe do you have an nfsstat you can run to see what's being requested?
22:54 cholcombe it's been awhile since i've dived into gluster nfs issues
22:54 Drakonstein It seems more like it's an issue with accessing a file that isn't available than something to do with NFS
22:55 Drakonstein NFS just happens to be how we're accessing it
22:56 Drakonstein I've had similar symptoms while nodes have rebuilt in the past, but they resolved themselves within a day through self heal
22:56 cholcombe right
22:56 cholcombe anything in the logs regarding your inaccessible file?
22:56 Drakonstein But these are going on 40+ hours now without healing themselves
22:57 cholcombe check the heal count.  it could be going down slowly depending on the file size
22:57 cholcombe what version of gluster are you running?
22:57 Drakonstein Just the No such file or directory and Non Blocking entrylks failure errors that I pasted
22:58 Drakonstein the heal count numbers are steadily increasing.  They just went over 1 million
22:58 cholcombe that's ok
22:58 Drakonstein They were 600k yesterday evening
22:58 cholcombe it's finding more files to churn through
22:59 cholcombe well i guess it could be bad if the heal daemon isn't doing anything and the count is just growing
22:59 cholcombe you said you killed the gluster processes on the 3rd replica right?
23:00 Drakonstein gluster 3.5.1
23:00 cholcombe ok cool
23:00 Drakonstein Yeah, gluster is completely off on the 3rd replica
23:00 Drakonstein so this is just between these 2
23:01 cholcombe i think it would be best to start up gluster on the 3rd replica but keep traffic away from it
23:01 cholcombe so the self heal daemon can repair things
23:01 Drakonstein by my estimations, it would take roughly 3 days to finish repairing it
23:02 cholcombe yeah i've had some clusters go a week or 2 before finishing repairs
23:02 Drakonstein ~400GB of information
23:02 cholcombe but they had to repair a 50TB of data
23:02 Drakonstein In a cloud infrastructure with block storage drives :/
23:02 cholcombe you're using the block feature in gluster?
23:02 cholcombe how's that working for you?  i've been curious about it
23:03 Drakonstein We don't do searches on the drive.  rewrite our application to know the location of all of the files it needs
23:03 Drakonstein With that, it works quite well
23:03 cholcombe i see
23:04 cholcombe yeah that's prob ideal
23:04 cholcombe using the gluster api?
23:06 [7] look at this... http://pastie.org/10224201
23:06 [7] fresh gluster installation
23:07 plarsen joined #gluster
23:07 [7] IMO that's a blocker
23:07 cholcombe that's quite interesting
23:07 cholcombe you forgot to rebalance though
23:08 [7] IIUC rebalance wouldn't do anything as it the distribute width didn't change
23:09 [7] but even then this is a situation that IMO may never ever happen (unless you specify a force argument to a potentially dangerous recovery command, none of which was the case here)
23:09 cholcombe what do you have to lose at this point?  it's already broken
23:10 [7] so what do you suggest? add a rebalance after both add-bricks?
23:10 [7] just rebalance start?
23:10 cholcombe well def a fix layout first
23:11 cholcombe yeah i think after replica 2 rebalance with fix layout
23:11 cholcombe and then again after replica 3
23:12 [7] guess I can also reproduce this with replica2, so I'm not even going to 3 this time
23:12 cholcombe go for it.  you have a quick repo
23:12 cholcombe repro*
23:12 [7] yeah, just waiting for all those sockets to time out after stopping glusterd
23:13 [7] otherwise the volume start will fail... (another such thing that really shouldn't happen)
23:17 [7] bwahahaha... http://pastie.org/pastes/10224209/text
23:17 [7] this is behaving all the way erratic
23:17 cholcombe fail sauce
23:17 [7] the rebalance actually made it worse
23:18 cholcombe :D
23:18 [7] but I'm fairly sure that was just random behavior
23:18 [7] every ls gets different results :P
23:18 cholcombe how about swapping the fix layout for just rebalance start?
23:19 cholcombe if that doesn't work that seems like a bug to me
23:20 [7] I'd even consider a brief inconsistent behavior between the add-brick and rebalance a serious issue
23:20 [7] this is the kind of thing that e.g. suddenly crashes a bunch of qemu VMs
23:20 cholcombe are you mounting with fuse or nfs?
23:20 [7] fuse
23:20 cholcombe ok
23:20 cholcombe anything interesting in your fuse logs?
23:20 cholcombe i'd imagine it's freaking out
23:21 [7] (just for testing, production use case is mostly libgfapi or whatever that thing is called)
23:21 cholcombe right
23:21 cholcombe gfapi would be fuse like without the penalty
23:21 [7] that's what I expected
23:22 [7] heh, just rebalance start, and even more folders went missing ;)
23:22 cholcombe http://www.gluster.org/pipermail/g​luster-users/2013-June/013347.html
23:23 cholcombe hmm interesting so this is a supported feature
23:24 [7] yes - since 3 major releases
23:24 [7] so I'd have expected most bugs to be found by now ;)
23:24 cholcombe you're running 3.6?
23:24 [7] 3.7.1
23:24 cholcombe ah
23:25 cholcombe are you using afrv2? just curious
23:25 B21956 joined #gluster
23:25 [7] no idea what that is
23:26 cholcombe ok i think that's not finished yet
23:26 cholcombe i'm looking on their wiki
23:26 cholcombe i'd shoot an email to the mailing list with your results.
23:26 cholcombe i'm not sure what's wrong
23:27 [7] http://review.gluster.org/#/c/6010/ suggests that it has landed
23:27 theron joined #gluster
23:28 cholcombe neat
23:29 [7] wooop
23:29 [7] it has healed the bricks!
23:29 cholcombe wow they nuked 10,000 lines of code with that patch
23:29 cholcombe oh cool so the self heal daemon knows how to fix this crap
23:29 cholcombe it just takes a few mins
23:29 * [7] retries to see if that was just a glitch
23:29 [7] the second rebalance (after going to replica3) seems to have triggered it - but it didn't with fix-layout
23:30 [7] but I guess that's just coincidence
23:30 cholcombe yeah i'm not sure
23:30 cholcombe yeah try again and see what happens
23:30 cholcombe the self heal code has come a long way.  it's pretty solid now
23:38 [7] wtf http://pastie.org/pastes/10224220/text
23:38 [7] that second rebalance does seem to trigger something - but after it completes there's still quite a bunch of folders missing?
23:43 [7] nah, it's just timing
23:43 cholcombe is your heal count zero now?
23:43 [7] replacing the rebalance by a sleep of the same duration has similar results
23:43 cholcombe i see
23:43 cholcombe so your first rebalance did it
23:43 [7] no, I removed both
23:44 cholcombe oh
23:44 [7] just the sleeps did it - however I waited for like half an hour in previous tests, and there it didn't help
23:44 [7] so I guess I need to wait between the switch to replica2 and to replica3 or something
23:44 Jandre joined #gluster
23:44 cholcombe yeah until it settles
23:45 [7] that should really be blocking any offending operations until it has settled then - or be able to handle them
23:45 cholcombe you could enable barrier
23:45 cholcombe while it's doing it's thing
23:46 [7] I still consider it a bug that I have an inconsistent view of the fs during that operation
23:47 [7] and I need a way to figure out whether that self-healing is finished before triggering further operations
23:53 rwheeler joined #gluster
23:54 kkeithley1 joined #gluster
23:55 cholcombe [7]: there's a vol info heal command for that
23:56 cholcombe it should be at or near zero when it's done
23:56 [7] hahaha
23:56 gildub joined #gluster
23:56 [7] that keeps showing zero all the time
23:56 [7] or wait... it suddenly starts showing stuff after the move to replica3, but not after the move to replica 2, even though there's definitely self-heal activity going on at that time
23:57 [7] let me run another test, monitoring that more closely
23:59 cholcombe ok

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary