Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-08-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 Banko gotcha
00:01 Banko will stopping the self heal daemon do the trick or is that crazy?
00:01 Banko and then restarting it
00:04 jcastillo joined #gluster
00:05 nzero joined #gluster
00:05 JoeJulian It might
00:06 JoeJulian I haven't ever taken the time to figure out where it keeps the state to determine if it's in the middle of a full heal.
00:07 Banko gotcha ill try to keep the full heal going for as long as i can
00:07 Banko hopefully we don't run out of space
00:18 bennyturns joined #gluster
00:22 JoeJulian Banko: I would be comfortable doing a rebalance fix-layout. That would make your space available without moving any files.
00:40 victori joined #gluster
00:41 cyberswat joined #gluster
00:44 Peppard joined #gluster
00:57 Banko ahh gotcha, thank you JoeJulian
01:14 suliba joined #gluster
01:39 jcastill1 joined #gluster
01:46 jcastillo joined #gluster
02:01 nangthang joined #gluster
02:35 julim joined #gluster
03:07 Lee1092 joined #gluster
03:10 aaronott joined #gluster
03:51 newdave joined #gluster
03:55 cyberswat joined #gluster
04:01 TheSeven joined #gluster
04:45 jwd joined #gluster
04:48 jwaibel joined #gluster
05:06 jcastill1 joined #gluster
05:11 jcastillo joined #gluster
05:16 jcastillo joined #gluster
05:48 maveric_amitc_ joined #gluster
05:52 papamoose joined #gluster
05:59 jwd joined #gluster
06:55 jiffin joined #gluster
07:37 jiffin joined #gluster
07:55 newdave joined #gluster
07:55 autoditac joined #gluster
08:10 RameshN joined #gluster
08:28 prg3 joined #gluster
08:29 LebedevRI joined #gluster
08:36 m0zes joined #gluster
09:37 haomaiwa_ joined #gluster
09:50 aaronott joined #gluster
10:18 Gnomethrower joined #gluster
10:24 squaly joined #gluster
10:25 Gnomethrower Hi there
10:25 Gnomethrower I've got a machine using almost all its RAM for GlusterFS operations
10:26 Gnomethrower there's barely any I/O happening on the box
10:38 Gnomethrower http://i.imgur.com/mjWhYs5.png
11:14 aaronott joined #gluster
11:57 cyberswat joined #gluster
12:36 Mr_Psmith joined #gluster
13:00 maveric_amitc_ joined #gluster
13:04 hagarth joined #gluster
13:15 Lee1092 joined #gluster
13:18 haomaiwa_ joined #gluster
13:26 papamoose joined #gluster
13:51 sage___ joined #gluster
13:56 R0ok_ joined #gluster
14:01 haomaiwa_ joined #gluster
14:34 nishanth joined #gluster
14:34 deniszh joined #gluster
14:46 deniszh joined #gluster
14:59 jcastill1 joined #gluster
15:05 jcastillo joined #gluster
15:09 ahab joined #gluster
15:31 calisto joined #gluster
15:33 bhubble3 joined #gluster
15:49 nsoffer joined #gluster
15:50 deniszh joined #gluster
15:50 newdave joined #gluster
15:56 pcaruana joined #gluster
15:56 plarsen joined #gluster
16:01 cholcombe joined #gluster
16:01 mikedep333 joined #gluster
16:05 michaeljk joined #gluster
16:07 hagarth joined #gluster
16:12 calisto joined #gluster
16:23 Pupeno joined #gluster
16:32 mikedep333 joined #gluster
16:42 hagarth joined #gluster
16:42 mikedep333 Hi. I just upgraded Gluster on my Fedora 21 64-bit machine from 3.5.5-2.fc21.x86_64 (provided by Fedora) to 3.7.3 from download.gluster.org. Now, I can no longer mount the 1 volume it had mounted before. The 2 gluster servers are running 3.7.2 (about to update) & 3.7.3.
16:42 mikedep333 http://fpaste.org/250568/44716514/
16:42 glusterbot Title: #250568 Fedora Project Pastebin (at fpaste.org)
16:47 mikedep333 bbl
16:51 JoeJulian mikedep333: You're not the first person to report trouble connecting with 3.7.3 to 3.7.2 servers. There shouldn't have been any changes to the rpc, but that doesn't preclude a bug. Bump the client down to 3.7.2 until after you've upgraded your servers.
17:14 calavera joined #gluster
17:24 TheCthulhu3 joined #gluster
17:26 plarsen joined #gluster
17:26 Pupeno joined #gluster
17:29 TheCthulhu joined #gluster
17:40 dusmant joined #gluster
17:58 elico joined #gluster
18:06 calisto joined #gluster
18:44 mikedep333 JoeJulian, I updated the 3.7.2 server to 3.7.3, and restarted the glusterd service. That seems to have fixed it.
18:44 mikedep333 should this be reported as a bug or issue?
18:46 mikedep333 although, the log file on the client is still full of rpc-clnt.c:362:saved_frames_unwind errors
18:46 mikedep333 *new errors
18:47 mikedep333 (the same error message, but reported during/after the successful mount)
18:53 mikedep333 my bad, both servers were running 3.7.2. I updated the other one to 3.7.3, unmounted and mounted on the client, and there are no more error messages being produced.
19:20 plarsen joined #gluster
19:21 JoeJulian mikedep333: It would be helpful if you could file a bug report. That's got to be a bug.
19:21 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
19:21 mikedep333 will do.
19:40 mikedep333 JoeJulian, reported. Thank you.
19:40 mikedep333 https://bugzilla.redhat.co​m/show_bug.cgi?id=1249339
19:40 glusterbot Bug 1249339: unspecified, unspecified, ---, bugs, NEW , Gluster 3.7.3 client fails to mount volume on 3.7.2 servers
19:45 mator joined #gluster
19:52 Mr_Psmith joined #gluster
20:06 plarsen joined #gluster
20:23 uebera|| joined #gluster
20:25 julim joined #gluster
20:27 DV joined #gluster
20:32 cholcombe joined #gluster
20:51 michaeljk Just testing GlusterFS 3.7.3 on Debian Jessie, with 2 nodes and 4 bricks on each. Configured as distributed replicated volume (replica 2). Our test-script writes 10.000 files (80K size) with random content to the volume which is mounted with NFS (no native GlusterFS client). If we stop the glusterfsd on Node1 while our script is writing the files, all goes on normaly until it finishes. But if we disconnect one node (e.g. p
20:51 michaeljk ull the network cable or reboot), client operation immediatly stops. No reads and writes possible, even a "ls" hangs. Is this normal behaviour or do we miss something in the configuration?
20:56 Banko michaeljk, is the node you nfs mounted, the same node you pulled the cable from ?
20:56 Banko because if so the NFS export runs per node, and if say you mounted node1, and you pulled the cable on node1, everything that mounted node1 will fail
20:56 michaeljk no, we have a third server which acts as test-client
20:57 Banko are you doing the ls on the test-client ?
20:57 Banko and which node does the test-client mount ?
20:57 michaeljk ok, so here's the structure:
20:58 michaeljk 2 servers for glusterfs with each 4 bricks. 1 server as client. "mount -t nfs gfs1.local:/data /mnt/test". Client hangs if we reboot gfs1.local
20:59 michaeljk if we shut down glusterfsd on gfs1, all works like before
21:01 Banko now i'm not exactly sure, because I don't use NFS that often, but everytime I did, it takes a while to know that the connection is down
21:01 Banko im not 100% sure if the network.ping-timeout setting applies to NFS however
21:03 Banko but if you want high availability on the NFS exports, i would suggest using some sort of CARP setup across all your nodes, so that if a node goes down the virtual ip just switches over to the other server
21:03 Banko that might cause less down time
21:03 michaeljk I just tested to mount the volume from gfs2.locl (the second machine) and reboot the first server while writing the files - but ist still the same
21:03 Banko oh, then i have no idea =[
21:04 Banko but that doesn't sound right, at least the gluster client itself doesn't have that issue which is what we primarily use
21:05 michaeljk I configured NFS because we are using many small files up to 100K (the native client would be slower according to some information in the web, but I didn't tried it yet)
21:07 Banko wait when you mounted the volume on gfs2.local did you also mount using gfs2.local as the source ?
21:09 michaeljk no, I'm always mounting from the third machine, never directly on the glusterfsd nodes
21:11 michaeljk but I can mount it with "mount -t nfs gfs1.local:/volume" or with "mount-t nfs gfs2.local:/volume" - the result is the same, if I reboot one glusterfsd node while writing on the volume, the client mount hangs
21:12 l0uis do you have quorum enabled?
21:12 michaeljk as soon as the glusterfsd-node is back online, client access is possible again (sometimes I need to remount on the client)
21:13 l0uis are you sure you have the brick replication spanning nodes?
21:13 l0uis i.e., brick 1 and 2 on node 1 should not be a pair. order matters when you specify them when creating the volume
21:14 michaeljk gluster volume get data all |grep quorum
21:14 michaeljk cluster.quorum-type                     none
21:14 michaeljk cluster.server-quorum-type              off
21:14 michaeljk the command to create the volume was:
21:14 michaeljk gluster volume create data replica 2 transport tcp gfs1.local:/data/brick1/gfs gfs2.local:/data/brick1/gfs gfs1.local:/data/brick2/gfs gfs2.local:/data/brick2/gfs gfs1.local:/data/brick3/gfs gfs2.local:/data/brick3/gfs gfs1.local:/data/brick4/gfs gfs2.local:/data/brick4/gfs
21:18 l0uis well Im out of ideas, sorry :)
21:19 l0uis as long as you aren't rebooting the node the client mounted, i would expect it to work
21:19 l0uis but obviously if you mount gfs1.local and reboot gfs1.local, the mount won't work. but if you mount gfs2.local and reboot gfs1.local, I think that should work.
21:27 michaeljk I'm just testing if the problem persists when using the native client ("mount -t glusterfs")...
21:30 michaeljk some behaviour.. as soon as one glusterfsd-node is not reachable in the network, the client hangs :(
21:30 michaeljk err same
21:31 l0uis you sure you're not pulling the client cable? :)
21:33 michaeljk yes :) I can reproduce it if I just enter "reboot" on one of the glusterfsd nodes. As soon as the network is down on the node, the client hangs. When the server node comes up again, communication starts again (not always, but most of the time)
21:35 l0uis how long are you waiting? i wonder if the client is waiting for network.ping-timeout
21:35 l0uis it is 42 seconds
21:36 michaeljk a reboot is faster normaly... but I can test it, wait... :)
21:36 l0uis I believe this thread is relevant: https://lists.gnu.org/archive/html/​gluster-devel/2010-03/msg00019.html
21:37 glusterbot Title: [Gluster-devel] ping timeout (at lists.gnu.org)
21:41 michaeljk 1m6.378s - could be 42sec because I shut down the node manually in the SSH shell
21:41 l0uis so it works after 1m6s ?
21:42 michaeljk yes, at least a "time ls -lh" shows the directory after that time
21:42 michaeljk this is with the native client... just testing again with nfs mount...
21:43 l0uis should be the same.
21:43 l0uis the nfs daemon is just a gluster client on the server side (i believe)
21:48 michaeljk yes.. same thing, 1m15.896s with nfs mount
21:49 michaeljk so this behaviour is normal? or should the write operations continue even if one node goes down?
21:50 l0uis it's normal
21:51 l0uis i believe it is a trade off to avoid spurious split brains o rsomething. there are more details in that thread.
21:57 michaeljk ok, thank you for your help :)
21:57 l0uis np
21:58 Telsin joined #gluster
22:03 michaeljk joined #gluster
22:22 plarsen joined #gluster
22:48 ron-slc joined #gluster
22:52 Lee_ joined #gluster
22:58 cyberswat joined #gluster
23:15 mkzero joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary