Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-11-15

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 cholcombe joined #gluster
00:01 haomaiwa_ joined #gluster
00:12 cyberbootje joined #gluster
00:14 ctria joined #gluster
00:36 mlhamburg1 joined #gluster
00:36 mlhamburg_ joined #gluster
00:54 ahino joined #gluster
01:01 haomaiwang joined #gluster
01:07 daMaestro joined #gluster
02:00 samikshan joined #gluster
02:01 haomaiwa_ joined #gluster
02:25 frozengeek joined #gluster
02:33 frozengeek hej all, i'm wondering what the best way is of retiring a brick in a destributed replicated volume, without introducing a SPOF on the other brick in the pair.
03:01 haomaiwang joined #gluster
03:02 harish joined #gluster
03:07 Mr_Psmith joined #gluster
03:10 gildub_ joined #gluster
03:15 cyberbootje joined #gluster
03:42 Lee1092 joined #gluster
04:01 haomaiwang joined #gluster
04:01 [7] joined #gluster
04:17 DV joined #gluster
04:33 harish joined #gluster
04:46 abyss^ joined #gluster
04:47 eljrax joined #gluster
04:47 linagee joined #gluster
04:50 sloop joined #gluster
04:58 XpineX joined #gluster
04:58 gildub_ joined #gluster
04:58 m0zes joined #gluster
04:58 TheSeven joined #gluster
04:59 sesa joined #gluster
04:59 ahino joined #gluster
05:00 papamoose joined #gluster
05:01 haomaiwang joined #gluster
05:07 ackjewt joined #gluster
05:09 DV joined #gluster
05:29 hos7ein joined #gluster
05:38 DV joined #gluster
06:01 haomaiwa_ joined #gluster
07:01 zhangjn joined #gluster
07:01 haomaiwang joined #gluster
07:06 zhangjn joined #gluster
07:27 nangthang joined #gluster
07:46 zhangjn joined #gluster
07:57 LebedevRI joined #gluster
07:59 tomatto joined #gluster
08:01 haomaiwa_ joined #gluster
08:33 kovshenin joined #gluster
08:41 Telsin joined #gluster
08:52 ahino joined #gluster
09:01 haomaiwang joined #gluster
09:14 beeradb joined #gluster
09:48 zhangjn joined #gluster
10:01 haomaiwang joined #gluster
10:52 kovshenin joined #gluster
11:01 haomaiwang joined #gluster
11:05 zhangjn joined #gluster
11:42 deniszh joined #gluster
11:55 ekuric joined #gluster
12:01 haomaiwa_ joined #gluster
12:20 RedW joined #gluster
12:34 gem joined #gluster
12:42 sakshi joined #gluster
12:56 ro_ joined #gluster
13:00 RameshN joined #gluster
13:01 haomaiwa_ joined #gluster
13:03 ahino1 joined #gluster
13:12 kshlm joined #gluster
13:12 kaushal_ joined #gluster
13:19 ro_ Anyone happen to be around? have a gluster volume that's gone unresponsive and I'm at a loss on how to troubleshoot it
13:55 Mr_Psmith joined #gluster
14:01 haomaiwang joined #gluster
14:10 mhulsman joined #gluster
14:32 dmnchild joined #gluster
14:32 dmnchild left #gluster
14:32 dmnchild joined #gluster
14:47 mhulsman joined #gluster
15:01 haomaiwa_ joined #gluster
15:06 ahino joined #gluster
15:17 jwd joined #gluster
15:25 cholcombe joined #gluster
15:29 shyam joined #gluster
15:36 DV__ joined #gluster
15:38 DV joined #gluster
16:01 haomaiwang joined #gluster
16:04 mhulsman joined #gluster
16:08 JoeJulian ro_: Start with the client log
16:11 bennyturns joined #gluster
16:15 gluster-user joined #gluster
16:15 gluster-user hello dears,
16:16 gluster-user hello dears, I'm stuck with the replace-brick bug, and not able to recover the cluster. https://bugzilla.redhat.co​m/show_bug.cgi?id=1124744 any ideas how to get out of the situation?
16:16 glusterbot Bug 1124744: high, unspecified, ---, bugs, NEW , glusterd : "heal full" command failed  with "Commit failed on <hostname>. Please check log file for details."
16:16 gluster-user btw, the heal full command didn't work
16:17 gluster-user the volume status is okay, but the data is not accessible on client server
16:22 JoeJulian gluster-user: If you're experiencing that bug, I would expect a reasonable workaround to be simply restarting all glusterd. I don't know what a client server is.
16:23 gluster-user JoeJulian: the client server is apache server mounting the volume. I've did restart, nothing works
16:23 gluster-user JoeJulian: the client server has ls: reading directory .: Input/output error
16:25 JoeJulian Ok, for clarity and since we're only interested in the behavior of gluster and not apache, let's simply refer to it as client, since we also have servers under the gluster subject.
16:25 JoeJulian Does the client log have an error?
16:29 gluster-user JoeJulian:okay no problem,  http://pastebin.com/w1s3UyRd
16:29 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
16:33 JoeJulian Well, assuming you followed the same cause path as that bug report and assuming you've restarted your glusterd management daemons, I can think of two solutions: One, remount your gluster volume. Two, change a volume setting that affects the client (performance.quick-read for instance) which would force a graph reload.
16:36 JoeJulian If you did the latter, you could then reset that option back to normal safely.
16:36 gluster-user JoeJulian: I wonder about something, the apache is able to deliver files from the cluster ( i'm not sure if it's from the replicated data of from the master), but the ls command is not working
16:37 gluster-user JoeJulian: I did another mounting point on the client and it's the same error : ls: reading directory .: Input/output error
16:41 JoeJulian gluster-user: Odd are, apache has already cached it and you're still within your cache timeout.
16:41 JoeJulian same connection error?
16:44 bennyturns joined #gluster
16:45 gluster-user JoeJulian: yes. how is it possible to create new volume and copy the data from the faulty one?
16:46 JoeJulian You would delete the volume, delete the volume-id extended attribute and the .glusterfs tree, recreate the volume with the bricks in the exact same order.
16:48 gluster-user JoeJulian: how to delete the volume-id extended attribute ? i see i scary idea right?
16:51 gluster-user JoeJulian:  I mounted the the volume on the same gluster server and it worked, the ls also works. does it help somewhere?
16:53 JoeJulian If it works somewhere, then it should work everywhere.
16:54 JoeJulian Did you try unmounting and mounting again?
16:56 gluster-user yes, it says, transport error. probably because I killed the glusterfs process that was catching the mount point. rebooting the whole server again to see
17:02 haomaiwa_ joined #gluster
17:04 arielb joined #gluster
17:04 gluster-user JoeJulian: really weird the same issue. the ls is not working on the client
17:04 JoeJulian iptables?
17:05 kovshenin joined #gluster
17:05 JoeJulian If it works locally, but you get a transport error remotely, perhaps you have a firewall issue.
17:05 gluster-user no, the iptables is clear
17:07 JoeJulian It gives you a transport error connecting to an ip address and port. Is that the right address and port (you can see the brick port in "gluster volume status")
17:07 JoeJulian If yes, can you telnet to it?
17:08 gluster-user which port number?
17:09 gluster-user gluster volume status is everything is online, but there is at the button:  Replace brick    fb2d763d-2e7e-48db-85aa-d5150958d153      completed
17:10 gluster-user I believe the client is going to the second server in the pool that has the second brick
17:14 JoeJulian Does the address and port you're getting an ETRANS for even exist in gluster volume status?
17:16 Chinorro joined #gluster
17:16 [7] joined #gluster
17:17 gluster-user I've exported the directory that I've mounted locally on the server by nfs and mounted the directory on the apache server it worked. something in the glusterfs client on the apache
17:18 JoeJulian yep
17:18 JoeJulian It's either trying to connect to the old port for the formerly replaced brick, like the bug, or it's trying to connect to the new port and is getting blocked.
17:19 gluster-user http://pastebin.com/CYtG442T
17:19 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
17:21 gluster-user how can I figure out which port number, I will add it to the mounting string. I see lots of ports on netstat
17:21 JoeJulian mounting string? No.
17:22 JoeJulian http://pastebin.com/w1s3UyRd
17:22 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
17:22 JoeJulian Line 1.
17:22 gluster-user I use this method to mount the client http://www.gluster.org/community/documentation/i​ndex.php/Gluster_3.2:_Manually_Mounting_Volumes
17:22 JoeJulian Oh for the love of pete.
17:22 JoeJulian Stop doing that.
17:23 JoeJulian @rtfm
17:23 glusterbot JoeJulian: Read the fairly-adequate manual at https://github.com/gluster/glusterfs/tre​e/master/doc/admin-guide/en-US/markdown
17:23 JoeJulian hmm, need to update that.
17:23 JoeJulian http://gluster.readthedocs.org/en/latest/
17:23 glusterbot Title: Gluster Docs (at gluster.readthedocs.org)
17:24 JoeJulian @forget rtfm
17:24 glusterbot JoeJulian: The operation succeeded.
17:24 JoeJulian @learn rtfm as Read the fairly-adequate manual at https://gluster.readthedocs.org/en/latest/
17:24 glusterbot JoeJulian: The operation succeeded.
17:25 gluster-user oh adam, I didn't know about that :(
17:26 JoeJulian So, "mount -t glusterfs store1.xx.com:external-volume /mountpoint"
17:27 gluster-user JoeJulian: this is exactly what I'm doing
17:28 JoeJulian Oh, ok. Sorry, I saw manual and I thought it was the old volfile document. I jumped to a conclusion erroneously.
17:28 gluster-user oh no, so is there anything I can do on the client ?
17:28 gluster-user the nfs worked fine, i guess no problem in the connectivity
17:29 JoeJulian So that mount connects to a server on 24007 and retrieves the volume definition. It then connects to each brick as shown in volume status on the port specified there.
17:30 JoeJulian I also re-read your error and it looks more like it's getting a tcp rst so the problem might be coming from the server. Check the brick log on store1.
17:32 gluster-user on the brick log on store1 is nothing much, I've repeated W [server-resolve.c:419:resolve_anonfd_simple] 0-server: inode for the gfid (366606cd-9008-44de-a486-065d4cee7373) is not found. anonymous fd creation failed
17:32 gluster-user lots of those
17:33 JoeJulian I'd probably resort to wireshark at this point if it was me.
17:33 gluster-user hmm .. its too deep for me :)
17:33 JoeJulian blame your network guys.
17:34 gluster-user lool
17:34 kovshenin joined #gluster
17:34 JoeJulian But seriously, talk to them. It sounds more and more like a network problem to me.
17:34 gluster-user they are on the same network switch, can't be
17:35 gluster-user it just happened after I did the fkn replace-brick
17:37 gluster-user this option should really be removed
17:38 JoeJulian No, replace-brick is useful and had a good purpose. You just found the best way not to use it. ;)
17:38 JoeJulian It's more for replacing a brick with one from a different host.
17:44 gluster-user dam, i have created another volume on store1 and mounted on the client and everything is fine!
17:44 gluster-user something wrong with that volume
17:46 gluster-user but why the hell it works locally on the same server
17:50 gluster-user I will create another volume, and mount it locally and copy data with rsync ... I got tired of this
18:01 gluster-user JoeJulian:Thank you!  I gonna leave it copying for now. thanks for your time
18:01 haomaiwa_ joined #gluster
18:12 plarsen joined #gluster
18:13 ahino joined #gluster
18:14 mhulsman joined #gluster
18:22 kovshenin joined #gluster
19:01 haomaiwang joined #gluster
19:10 mlncn joined #gluster
19:27 daMaestro joined #gluster
19:30 ctria joined #gluster
19:56 plarsen joined #gluster
19:56 gluster-user joined #gluster
20:01 haomaiwa_ joined #gluster
20:12 Mr_Psmith joined #gluster
20:21 cyberbootje joined #gluster
20:23 cholcombe joined #gluster
20:37 ahino joined #gluster
20:40 skylar1 joined #gluster
20:58 cyberbootje joined #gluster
21:01 haomaiwa_ joined #gluster
21:17 daMaestro joined #gluster
21:23 harish joined #gluster
21:36 gildub_ joined #gluster
21:46 TheSeven joined #gluster
21:48 skylar1 joined #gluster
21:52 frozengeek joined #gluster
21:52 mjrosenb Is there a way to ask gluster to migrate all linktos to the correct brick?
21:54 JoeJulian mjrosenb: that's what a rebalance is. "force" if you want it to go ahead an migrate even if the target brick is more full than the source.
21:56 mjrosenb does a rebalance also recalculate the percentage that gets sent to each brick + move files that are now on the wrong brick?
21:57 mjrosenb I'd actually prefer to do it one file at a time for now, since I've made some changes to the gluster souce, and I don't really trust anything to work correctly right now.
21:58 JoeJulian mjrosenb: yeah, it sets the hash mask allocation based on the number of dht subvolumes.
21:59 harish joined #gluster
21:59 mjrosenb does it take into accound how full each of the volumes are?
22:01 haomaiwang joined #gluster
22:17 skylar joined #gluster
22:26 prg3 joined #gluster
23:01 haomaiwa_ joined #gluster
23:14 marlinc joined #gluster
23:24 skylar1 joined #gluster
23:27 delhage joined #gluster
23:35 daMaestro joined #gluster
23:41 mlncn joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary