Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-11-15

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 jkroon joined #gluster
00:08 plarsen joined #gluster
00:11 Amdintel joined #gluster
00:12 Javezim Anyone know how I can get rid of this error? - http://paste.ubuntu.com/23478016/
00:12 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
00:12 Javezim tried 'service glusterfs-server restart' on all nodes
00:13 Javezim Is there a process I can kill on all nodes to allow me to run this again?
00:24 yalu joined #gluster
00:37 vinurs joined #gluster
00:52 rafi joined #gluster
01:04 ankitraj joined #gluster
01:11 p7mo_ joined #gluster
01:14 kz joined #gluster
01:31 zat joined #gluster
01:36 kz I’m looking to setup a glusterfs pool that will be replicated across a few KVM hypervisors and I want the hosts to pull from their own replication as otherwise it will saturate the network. According to http://serverfault.com/questions/644381/does-the-glusterfs-client-prefer-local-lan-nodes-over-remote-lan-nodes I should be looking for the read-subvolume option (unfortunately the link provided is broken). I can’t seem to find
01:36 kz any documentation on this or the other options available for the /etc/glusterfs/datastore.vol configuration file. Does anyone know where I can find this? Thanks in advance for your help.
01:36 glusterbot Title: Does the GlusterFS "client" prefer local LAN nodes over remote LAN nodes? - Server Fault (at serverfault.com)
01:46 calisto joined #gluster
02:06 calisto joined #gluster
02:08 derjohn_mobi joined #gluster
02:18 haomaiwang joined #gluster
02:18 ankitraj joined #gluster
02:21 Alghost joined #gluster
02:28 ankitraj joined #gluster
02:29 kramdoss_ joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:55 Lee1092 joined #gluster
03:04 rafi joined #gluster
03:12 haomaiwang joined #gluster
03:17 magrawal joined #gluster
03:25 shubhendu joined #gluster
03:31 daMaestro joined #gluster
03:32 leafbag joined #gluster
03:33 haomaiwang joined #gluster
03:33 nathwill joined #gluster
03:34 nathwill joined #gluster
03:35 nathwill joined #gluster
03:36 nathwill joined #gluster
03:37 nathwill joined #gluster
03:41 nbalacha joined #gluster
03:46 Gnomethrower joined #gluster
03:54 Jacob843 joined #gluster
03:55 rafi joined #gluster
03:57 jiffin joined #gluster
04:00 calisto joined #gluster
04:01 atinm joined #gluster
04:01 rafi joined #gluster
04:04 itisravi joined #gluster
04:09 RameshN joined #gluster
04:16 satya4ever joined #gluster
04:20 Jacob843 joined #gluster
04:20 jkroon joined #gluster
04:20 nishanth joined #gluster
04:24 buvanesh_kumar joined #gluster
04:25 rafi joined #gluster
04:27 rafi joined #gluster
04:33 ankitraj joined #gluster
04:44 prasanth joined #gluster
04:58 ndarshan joined #gluster
05:01 annettec joined #gluster
05:03 kdhananjay joined #gluster
05:06 annettec joined #gluster
05:06 itisravi joined #gluster
05:08 kkeithley joined #gluster
05:09 shdeng joined #gluster
05:12 skoduri joined #gluster
05:15 apandey joined #gluster
05:19 ppai joined #gluster
05:25 karthik_us joined #gluster
05:27 sanoj joined #gluster
05:28 ndarshan joined #gluster
05:38 poornima_ joined #gluster
05:38 nathwill joined #gluster
05:39 haomaiwang joined #gluster
05:39 ppai joined #gluster
05:47 hgowtham joined #gluster
05:48 shdeng joined #gluster
05:51 ndarshan joined #gluster
05:52 ankitraj joined #gluster
05:53 shdeng joined #gluster
05:54 buvanesh_kumar_ joined #gluster
06:02 Bhaskarakiran joined #gluster
06:02 Muthu joined #gluster
06:14 ndarshan joined #gluster
06:14 satya4ever joined #gluster
06:22 edong23 joined #gluster
06:26 msvbhat joined #gluster
06:31 itisravi joined #gluster
06:33 prth joined #gluster
06:35 skoduri_ joined #gluster
06:38 edong23 joined #gluster
06:39 kdhananjay joined #gluster
06:39 nathwill joined #gluster
06:40 ashiq joined #gluster
06:41 hchiramm joined #gluster
06:42 ppai joined #gluster
06:45 Saravanakmr joined #gluster
06:48 atinm joined #gluster
06:49 tomaz__ joined #gluster
06:49 kdhananjay joined #gluster
06:53 mhulsman joined #gluster
06:55 buvanesh_kumar joined #gluster
06:56 panina joined #gluster
07:02 prth joined #gluster
07:05 k4n0 joined #gluster
07:09 kramdoss_ joined #gluster
07:11 hchiramm joined #gluster
07:17 jtux joined #gluster
07:18 itisravi_ joined #gluster
07:27 apandey joined #gluster
07:31 devyani7 joined #gluster
07:37 irated joined #gluster
07:41 prth joined #gluster
07:45 hackman joined #gluster
07:46 skoduri joined #gluster
07:46 atinm joined #gluster
07:55 k4n0 joined #gluster
08:09 witsches joined #gluster
08:18 jri joined #gluster
08:20 susant joined #gluster
08:22 kotreshhr joined #gluster
08:30 renout_away joined #gluster
08:35 kramdoss_ joined #gluster
08:38 fsimonce joined #gluster
08:38 riyas joined #gluster
08:39 renout_away joined #gluster
08:50 karthik_us joined #gluster
08:53 flying joined #gluster
09:03 skoduri joined #gluster
09:03 ahino joined #gluster
09:13 renout joined #gluster
09:14 Slashman joined #gluster
09:26 witsches joined #gluster
09:26 aravindavk joined #gluster
09:27 aravindavk joined #gluster
09:27 ketarax Hmm.  running a disperse test volume, I seem to be getting weird disk usage;
09:28 ketarax the folder size is (i
09:28 ketarax I'm PRETTY sure) 1G.  "du -sh" in the folder gives me 25G; "sudo du -sh" says 1.5G, and finally "sudo du -sh . " seems to give the right size.
09:30 ketarax googling, there's something that i believe should be fixed (by fresher versions); i'm running gluster 3.6.9 and (mostly) kernels around 3.13.0 or so
09:33 panina joined #gluster
09:35 derjohn_mobi joined #gluster
09:41 nathwill joined #gluster
09:43 xavih ketarax: you shouldn't use disperse with 3.6.x. It doesn't work well. I recommend using latest 3.7.x or 3.8.x
09:47 ketarax okay, thanks.
09:52 Marbug Hi I'm still havign problems with getting gluster-nfs up, now The brick won't even connect anymore to the other nodes :/ Anyway, what I found out was that /proc/net/rpc don't exist on the running host, but on the new it exists there is nfsd, nfs4 and auth dirs, or may they not be there?
09:56 kdhananjay joined #gluster
10:00 Muthu joined #gluster
10:00 msvbhat joined #gluster
10:19 zat1 joined #gluster
10:21 ndevos Marbug: right, make sure you are not using any of the kernel nfs (client or server) bits on a gluster server
10:22 ndevos Marbug: only the rpcbind service should be started, the rest is taken care of by gluster
10:23 Marbug at first it was, but I can't seem to find any of the server process running
10:24 Marbug nfslock is running, but this is also running on the other hosts :/
10:24 Marbug I'm really out of options here :/
10:24 Marbug btw, is there a reason why glusterfsd won't start ?
10:24 ndevos nlslock (well, rpc.statd) should get started by gluster/nfs, if there is a service, it should be disabled
10:25 ndevos glusterfsd is just a binary, and used for many different tasks, you would need to check the log for the task to see what the error is
10:28 Marbug nfslock is creating an rpc.statd service
10:28 ndevos yes, and rpc.statd is expected to get started by gluster/nfs
10:29 Marbug well that is the problem too, the logs arn't me telling much, it just says the brick is disconnected from gluster
10:29 Marbug and
10:29 ndevos nfslock might also load the lockd kernel module, and that will give conflicts with gluster/nfs
10:29 Marbug strange thing that it's running on all my other nodes
10:30 cloph_away joined #gluster
10:30 Marbug http://apaste.info/sBKyN
10:30 Marbug multiple days struggle with the same issue isn't something normal :/
10:32 ndevos I would not know what the disconnects cause, maybe there is something in the logs of the bricks, nfs or self-heal-daemon?
10:34 linex01 joined #gluster
10:37 linex01 Hi All, i'm using GlusterFS version 3.7.11. After intitial geo-replication session, sometimes after day or 2. One of the brick fails to replicated and is stuck at history crawl state.
10:38 linex01 Anyone seen this issue before? http://www.gluster.org/pipermail/gluster-users/2016-November/029129.html
10:38 glusterbot Title: [Gluster-users] GlusterFS geo-replication brick KeyError (at www.gluster.org)
10:38 msvbhat joined #gluster
10:38 ashiq_ joined #gluster
10:49 hgowtham joined #gluster
10:51 kotreshhr left #gluster
10:52 Wizek joined #gluster
11:05 Philambdo joined #gluster
11:11 ndarshan joined #gluster
11:14 rafi joined #gluster
11:18 satya4ever_ joined #gluster
11:24 k4n0 joined #gluster
11:24 satya4ever joined #gluster
11:32 Saravanakmr linex01, one related bug is here and fix is in - http://www.gluster.org/pipermail/bugs/2016-June/061785.html
11:32 glusterbot Title: [Bugs] [Bug 1348086] New: [geo-rep]: Worker crashed with "KeyError: " (at www.gluster.org)
11:37 bfoster joined #gluster
11:38 witsches joined #gluster
11:40 haomaiwang joined #gluster
11:42 nathwill joined #gluster
11:43 calisto joined #gluster
11:44 bfoster joined #gluster
12:04 ahino joined #gluster
12:04 panina joined #gluster
12:11 Saravanakmr left #gluster
12:16 haomaiwang joined #gluster
12:19 Caveat4U joined #gluster
12:21 jkroon joined #gluster
12:27 witsches joined #gluster
12:28 calisto joined #gluster
12:30 owitsches joined #gluster
12:30 Marbug ok found the problem, lo wasn't open i nthe firewall >_<
12:31 Javezim Hey we're having a major problem
12:31 Javezim half our bricks are offline
12:32 Javezim and even restarting glusterfs and even the machine, the brick processes don't start
12:37 Javezim Tried doing a start force - https://bugzilla.redhat.com/show_bug.cgi?id=963169 still no go
12:37 glusterbot Bug 963169: medium, medium, ---, kparthas, CLOSED NOTABUG, glusterd : 'gluster volume start <volname> force' is unable to start brick process and it fails but 'gluster volume stop' followed by 'gluster volume start' starts that brick process
12:38 kshlm Javezim, Did you check the brick logs?
12:38 owitsches joined #gluster
12:38 Javezim There aren't any updated logs because the processes aren't running
12:38 Javezim last logs are from a while ago
12:39 kshlm Hmm, bricks dying even before they write any logs? That's strange.
12:40 kshlm Next step would be to check glusterd logs.
12:41 satya4ever joined #gluster
12:43 nathwill joined #gluster
12:43 Javezim Hmm I did an Errors only - http://paste.ubuntu.com/23480154/
12:43 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
12:44 Javezim "No Route To Host" is interesting
12:44 Javezim They can ping each other fine
12:44 Javezim No firewall rules been changed
12:45 P0w3r3d joined #gluster
12:46 kshlm If you can restart glusterd, could you restart one with trace logs enabled?
12:47 kshlm If you're on fedora/centos with systemd, edit /etc/sysconfig/glusterd and set 'LOG_LEVEL=TRACE'
12:47 kshlm Javezim, ^
12:47 Javezim kshlm I ammmm on Ubuntu
12:48 Javezim gluster volume set gv0 diagnostics.brick-log-level trace
12:48 Javezim That's right yeah?
12:50 Muthu joined #gluster
12:50 kshlm Javezim, That doesn't set the log level for glusterd.
12:51 kshlm Ubuntu has upstart right?
12:52 Javezim kshlm I don't thin so
12:52 Javezim #glusterd --log-level WARNING
12:52 Javezim https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Configuring_the_Log_Level.html
12:52 glusterbot Title: 15.4. Configuring the Log Level (at access.redhat.com)
12:52 Javezim So I assume glusterd --log-level TRACE
12:53 kshlm That is if you want to launch glusterd directly, instead of using the system service manager to launch it.
12:53 kshlm ... this works well enough
12:54 Javezim > http://paste.ubuntu.com/23480185/
12:54 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
12:54 johnmilton joined #gluster
12:56 Javezim kshlm Might help but, On Monday we added an Arbiter to the current volume, has been self-healing metadata for a few days, just started getting this issue today though
12:58 p7mo joined #gluster
12:59 nbalacha joined #gluster
13:01 Javezim This is just after restarting glusterfs service - http://paste.ubuntu.com/23480217/
13:01 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:04 kshlm Javezim, Could you also do a 'volume start <vol> force' and give the glusterd logs including it.
13:04 kshlm I want to check what's causing bricks to not start.
13:05 prth joined #gluster
13:06 ahino joined #gluster
13:06 Javezim Done and done :) - http://paste.ubuntu.com/23480232/
13:06 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:11 kshlm Javezim, Still no relevant info. The paste doesn't have the volume start logs I'm looking for.
13:11 kshlm To be absolutely sure we get the logs we want.
13:13 kshlm Stop glusterd, cleanup/rotate the old glusterd log.
13:13 kshlm Start glusterd with trace logs again.
13:13 kshlm Wait for a little while for connections to settle, wait for peer status to give a Peer in Cluster, conected for all peers.
13:13 kshlm Run start.
13:14 kshlm Give logs.
13:17 Javezim I think this should be it now
13:17 Javezim http://paste.ubuntu.com/23480269/
13:17 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:17 Javezim kshlm looks like a lot more data in that one
13:19 Javezim Did it again just in case - http://paste.ubuntu.com/23480273/
13:19 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:19 owitsches joined #gluster
13:21 owitsches joined #gluster
13:21 Javezim This one is after a volume start foce - http://paste.ubuntu.com/23480280/
13:21 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:23 kshlm Javezim, Taking a look now.
13:25 owitsches joined #gluster
13:26 owitsches joined #gluster
13:26 prth joined #gluster
13:29 kshlm Javezim, Does the node you're running commands on have any bricks for the volume?
13:30 Javezim Yes
13:30 Javezim 3
13:31 kshlm Okay. In that case, the last logs you pasted have been snipped just after the part I want.
13:32 skoduri joined #gluster
13:33 Javezim I actually think I may have figured it
13:34 kshlm Javezim, Cool!
13:34 kshlm So what happened/is happening?
13:34 Javezim When I did a gluster peer status, one of my servers had "Other Names" and then a random DNS Entry
13:34 Javezim No idea where it got it from
13:35 Javezim So I commented it out in /var/lib/glusterd/peers
13:35 Javezim restarted service
13:35 Javezim and brick processes came back
13:35 kshlm Awesome!
13:36 kshlm But I can't think of why that would cause bricks to not start.
13:36 Javezim Me neither
13:36 kshlm Did the bricks start everywhere?
13:42 plarsen joined #gluster
13:42 shubhendu joined #gluster
13:44 Javezim kshlm one machine is missing all its /var/lib/glusterd/vols/ files though
13:44 Javezim any idea if we can sync these from another machine?
13:44 d0nn1e joined #gluster
13:46 kshlm Javezim, try 'gluster volume sync <hostname of another peer> all'
13:47 bfoster joined #gluster
13:47 kshlm But ensure that there is absolutely nothing else in /var/lib/glusterd/vols/
13:47 atinm joined #gluster
13:48 Javezim @kshlm Any idea what pulls data into the /var/lib/glusterd/peers files?
13:48 Javezim Like that weird DNS Entry keeps coming bacl
13:48 Javezim Like that weird DNS Entry keeps coming back
13:49 kshlm It keeps pulling it from the other peers everytime you start.
13:49 kshlm You'll need to edit it out everywhere, stop all glusterds, and then start them again.
13:50 kshlm Don't start before you stop all glusterds, or the entry will be pulled back again.
13:53 unclemarc joined #gluster
13:54 ndk_ joined #gluster
14:03 Javezim @kshlm Hmm  did that but it keeps pulling it back somehow
14:08 abyss^ JoeJulian: please give me a sign when you will be here, thanx
14:11 ankitraj joined #gluster
14:12 Javezim @kshlm 5th times a charm,  that sync command is failing with ;Connection failed. Please check if gluster daemon is operational.; can I just copy paste these files from another server?
14:13 kshlm Javezim, Looks like glusterd is stopped on that server.
14:13 Javezim Can't start it
14:13 Javezim Just died instantly
14:13 kshlm Logs?
14:14 Javezim http://paste.ubuntu.com/23480531/
14:14 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
14:15 arc0 joined #gluster
14:16 kshlm Javezim, The /var/lib/glusterd/vols directory is not complete. So restore is failing.
14:16 kshlm You can copy over /var/lib/glusterd/vols from another machine.
14:22 jkroon joined #gluster
14:29 Javezim @kshlm Thanks :) All but 1 Machine back online now
14:29 Javezim Glusterd failing to start again
14:29 Javezim Logs bit different - http://paste.ubuntu.com/23480636/
14:30 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
14:30 kshlm Javezim, line 13 in the paste.
14:30 annettec joined #gluster
14:31 kshlm You cannot comment out stuff in /var/lib/gluster/peers
14:31 kshlm You need to delete the line.
14:33 Javezim Okay yeah that was a problem, we deleted that weird dns from all entries except that one where it was commented out. Deleted it now
14:34 Javezim Stiall failing though - http://paste.ubuntu.com/23480652/
14:34 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
14:34 Javezim Confirmed not the issue where missing /var/lib/glusterd/vols/gv0 files
14:34 Javezim They are all there
14:37 shyam joined #gluster
14:37 kshlm Javezim, One last time 'glusterd --log-level DEBUG', some other config file is not right.
14:38 kshlm If all config files are right, initlization should have at least succeded. (line 13 again)
14:39 Javezim > http://paste.ubuntu.com/23480678/ There you are
14:39 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
14:40 derjohn_mobi joined #gluster
14:42 Javezim Bit more after a restart - http://paste.ubuntu.com/23480689/
14:42 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
14:44 nathwill joined #gluster
14:47 kshlm Javezim, One of the files in /var/lib/gluster/peers could be empty.
14:49 Javezim @kshlm Okay just checked all of them, they all have content (And more importantly the correct content)
14:51 kshlm GlusterD is having trouble parsing a peer file (line 478 in the last paste).
14:51 haomaiwang joined #gluster
14:52 vbellur joined #gluster
14:52 kshlm So one of the files might be not all right.
14:53 arc0 joined #gluster
15:03 aravindavk joined #gluster
15:04 Muthu joined #gluster
15:05 ndarshan joined #gluster
15:08 Javezim Just gone through them all, can't see any issues
15:09 Javezim same as the other bricks that are currently online
15:13 panina joined #gluster
15:18 Javezim Just appears to be the same thing over and over - http://paste.ubuntu.com/23480837/
15:18 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
15:19 qazar joined #gluster
15:21 atinm joined #gluster
15:21 qazar hello, quick question..  i have some boxes with glusterfs client to 3.7.5, is this client compatible to glusterfs server 3.8 branch ?  thanks !
15:21 xavih joined #gluster
15:22 Caveat4U joined #gluster
15:24 nathwill joined #gluster
15:29 ira joined #gluster
15:39 ndarshan joined #gluster
15:44 farhorizon joined #gluster
15:45 farhoriz_ joined #gluster
15:57 Javezim Getting following errors when trying to start glusterfs-server - http://paste.ubuntu.com/23480837/
15:57 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
15:57 Javezim See alot of things saying that it could be resolving hostnames incorrectly but everything seems okay
15:57 Javezim anyone else had experience with it?
16:00 jns left #gluster
16:05 qazar hello, quick question..  i have some boxes with glusterfs client v3.7.5, is this gluster fs client compatible to glusterfs server v3.8.x branch ?   in other words could a client v3.7.x connect to  a server v3.8.x thanks !
16:08 hchiramm joined #gluster
16:11 kotreshhr joined #gluster
16:16 farhorizon joined #gluster
16:21 Muthu joined #gluster
16:21 sbulaage joined #gluster
16:35 wushudoin joined #gluster
16:36 JoeJulian abyss^: I'm here. Do you have what I asked for?
16:37 JoeJulian qazar: Yes, in fact upgrading servers before clients is the recommended process. I would still work at upgrading those clients though. There are several known bugs in that older 3.7 version.
16:41 shubhendu joined #gluster
16:49 ankitraj joined #gluster
16:51 squizzi_ joined #gluster
16:59 squizzi_ joined #gluster
17:03 annettec joined #gluster
17:06 skoduri joined #gluster
17:14 qazar @joejulian, ok great.. we will look forward upgrading the clients.. but we need to ask our OS package provider to update them (in order to stay supported by them).
17:14 JoeJulian Which OS?
17:15 qazar @joejulian,  Oracle Linux..
17:15 JoeJulian Er, more accurately, which distro?
17:15 JoeJulian Ugh
17:16 qazar @joejulian, currently, they don't provide any glusterfs server version.. they only providing the glusterfs client v3.7.5 (latest from them)
17:16 JoeJulian Can't you still use the CentOS Gluster SIG?
17:16 qazar @joejulian, no.. we are running Oracle DB applications.. FOr support reasons, they want us to use Oracle Linux
17:17 qazar @joejulian, but i noticed they are already bundling fuse lib + module with their OS.. Is this could be an issue ?
17:17 JoeJulian Yeah, but I /think/ you can still install the centos-release-gluster rpm and install from there.
17:18 ndevos qazar: I very much doubt Oracle supports the glusterfs-client and fuse module that comes with their OS
17:18 qazar @joejulian, i noticed they are already bundling fuse lib + module with their OS.. Is this could be an issue ?
17:19 qazar ndevos, we are using their own Oracle  glusterfs client package from their repo.
17:19 JoeJulian I also think that the version you're looking at is the one they repackage from RHEL which, I believe, was the RHGS client, not entirely compatible with upstream. I could be wrong though.
17:19 ndevos qazar: in order to use the 3.8.x gluster servers, you will need an updated glusterfs client - and the packages from the centos storage sig would be the way to go
17:20 ndevos qazar: well, you can ask them to update their glusterfs client packages to a newer 3.7.x version too, but I doubt they'll do that
17:20 qazar ndevos,  Oracle support wont support us if we aren't using their "approved" packages
17:20 JoeJulian Then run gluster in a container and use whatever distro you want inside that container.
17:20 qazar ndevos, do you know if the fuse they are "pre" bundling could cause issues with the glusterfs client ?
17:21 JoeJulian unlikely
17:21 ndevos qazar: does that mean Oracle supports the Gluster client? that would mean they have to understand how it works *and* be able to provide updates/patches - I *really* doubt that
17:22 qazar @joejulian, http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/fuse-2.9.4-1.0.1.el6.x86_64.rpm
17:22 JoeJulian So many hateful things I want to type...
17:22 qazar @joejulian this package is already installed.. and the fuse module is already loaded
17:22 ndevos qazar: no, that should not be a problem, but there are certain kernels that Oracle provides and they have a fuse module that does not work well with (any?) Gluster version
17:22 JoeJulian I'm not a big Oracle fan.
17:23 qazar ndevos, this package is already installed.. and the fuse module is already loaded  (fuse 2.9.4)
17:23 qazar ~]# modinfo fuse filename:       /lib/modules/2.6.39-400.284.1.el6uek.x86_64/kernel/fs/fuse/fuse.ko alias:          devname:fuse alias:          char-major-10-229
17:23 ndevos qazar: sure, but it is well possible that mounting fails, or gives weir errors... I dont remember exactly what the issue was
17:24 JoeJulian OL is mostly just a repackaged and rebranded RHEL, with a few kernel customizations that have been known to break things.
17:24 qazar ndevos, mount is failing right now.. we are currently scratching our head to figure why...
17:24 kpease_ joined #gluster
17:24 JoeJulian What's the full gluster client version?
17:24 qazar ---> Package glusterfs-fuse.x86_64 0:3.7.5-19.el6 will be installed --> Processing Dependency: glusterfs = 3.7.5-19.el6 for package: glusterfs-fuse-3.7.5-19.el6.x86_64 --> Processing Dependency: glusterfs-client-xlators = 3.7.5-19.el6 for package: glusterfs-fuse-3.7.5-19.el6.x86_64
17:24 glusterbot qazar: -'s karma is now -360
17:25 qazar glusterfs-fuse-3.7.5-19.el6.x86_64
17:25 ndevos qazar: there could be two reasons: a. glusterfs client needs an update, b. use non "uek" kernel
17:25 qazar installation works without any issues.. but mount is failing..
17:25 qazar ndevos, we can't use uek kernel, this is required by the Oracle DB application
17:26 qazar ndevos, we need to stick with uek kernel
17:26 ndevos I'm pretty sure you need an newer glusterfs client to connect to 3.8 gluster servers
17:26 qazar ndevos, how we could check what is the problem with the client mount ?
17:26 JoeJulian dmesg and client logs.
17:27 qazar ndevos, at first we think acl issues, but now i have  confirmed, there's no ACL iptables and neither SELinux running.
17:27 qazar @joejulian, where are the client logs located ?
17:28 ndevos qazar: if the logs mention ENODEV, the kernel (or fuse module) does not work with the glusterfs packages that you have installed (needs kernel replacement)
17:28 JoeJulian logs are in /var/log/glusterfs
17:28 qazar clients logs also ?
17:29 ndevos qazar: maybe it is easier to mount over NFS? that is a protocol that Oracle should support and test
17:31 bowhunter joined #gluster
17:31 JoeJulian Otherwise, I'd strongly recommend taking this up with Oracle support. Looks like they've had problems with uek kernels and fuse filesystems since at least 2012.
17:32 qazar ndevos, they only do support NFSv3...  and to bring HA to NFSv3 for us  is a tedious process..
17:33 qazar ndevos, do you think we need to uninstall their fuse & module package ?  dbfs isn't using it right now.. (and we wont use it)
17:33 ndevos qazar: you need the fuse module to use glusterfs, if NFS is not an option
17:34 qazar ndevos, BTW< Oracle pushing to use dbfs instead of glusterfs..
17:34 ndevos I've never heard of dbfs
17:34 prth joined #gluster
17:34 JoeJulian https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob_fs.htm#ADLOB45943
17:34 glusterbot Title: Introducing the Oracle Database File System (at docs.oracle.com)
17:34 qazar ndevos, could the pre-bundled fuse module incompatible (on the client side) with glusterfs-3.8.5 server from centos sig ?
17:35 skylar joined #gluster
17:35 JoeJulian Of course they are. Anything to keep you locked in to paying license fees.
17:36 qazar should be uninstall the fuse module from Oracle ? And let the glusterfs client package install them ?
17:36 JoeJulian The fuse module is a kernel module. There are none packaged with gluster.
17:36 qazar But, i notice they are normally installed by the glusterfs  package
17:36 JoeJulian glusterfs-fuse is the client, not the kernel module.
17:37 qazar @Joejulian, in our case, when we installed the glusterfs client, fuse module was already loaded.. and we confirmed package was already there... but client installation worked very well without any dependencies issues or warnings
17:37 qazar so far, only mount is failing..
17:37 qazar so we need to look inside : /var/log/glusterfs ?
17:38 qazar Oracle glusterfs client could piggyback the existing fuse module ?
17:38 JoeJulian That's usually the first place to look, yes.
17:41 abyss^ JoeJulian: will be here about 20:00? I will put my son to bed and then we will be able to try to fix the split-brain. Thank you.
17:41 abyss^ *will you be
17:41 JoeJulian Which TZ are we assuming?
17:41 arc0 joined #gluster
17:42 mhulsman joined #gluster
17:42 abyss^ JoeJulian: CET :)
17:43 JoeJulian Yep, I'll be around.
17:43 msvbhat joined #gluster
17:44 abyss^ JoeJulian: thank you.
17:46 sage_ joined #gluster
17:49 ahino joined #gluster
17:52 haomaiwang joined #gluster
17:54 qazar @joejulian,  do you know which TCP/UDP ports need to be open for allowing glusterfs client -> glusterfs server communication ?
17:56 ivan_rossi left #gluster
17:56 JoeJulian @ports
17:56 glusterbot JoeJulian: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
17:57 qazar ok great thanks
17:59 qazar is the gluster server perfoming some callbacks ? or the session from the client is always outgoing traffic ?
17:59 ashiq joined #gluster
17:59 JoeJulian The client opens the tcp sessions which remain open.
18:00 JoeJulian But the servers also operate clients for healing, etc, that need to connect to other servers.
18:05 shaunm joined #gluster
18:12 shyam joined #gluster
18:15 vbellur joined #gluster
18:16 circ-user-aoCw3 joined #gluster
18:20 portante joined #gluster
18:21 qazar @joejulian,
18:21 qazar here's the client logs output :
18:21 qazar [2016-11-15 18:17:12.797108] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-shadowrepvol-client-1: Connected to shadowrepvol-client-1, attached to remote volume '/bricks/shadow_brick2/brick'. [2016-11-15 18:17:12.797145] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-shadowrepvol-client-1: Server and Client lk-version numbers are not same, reopening the fds
18:21 glusterbot qazar: This is normal behavior and can safely be ignored.
18:21 qazar [2016-11-15 18:17:12.801513] I [fuse-bridge.c:4965:fuse_thread_proc] 0-fuse: unmounting /dbreports [2016-11-15 18:17:12.801746] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7aa1) [0x7fbaafd7aaa1] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fbab115e175] -->/usr/sbin/glusterfs(cleanup_and_exit+0x71) [0x7fbab115dca1] ) 0-: received signum (15), shutting down [2016-11-15 18:17:12.801766] I [fuse-bridge.c:5
18:21 glusterbot qazar: ('s karma is now -167
18:21 JoeJulian @paste
18:21 glusterbot JoeJulian: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
18:22 qazar @joejulian, clients are incompatible ?
18:22 JoeJulian I see nothing that suggests that yet.
18:23 JoeJulian Paste up the whole log someplace and give me the link.
18:23 qazar @joejulian, why it's not mounting the volume then ?
18:23 Jacob843 joined #gluster
18:23 JoeJulian received signum (15), shutting down
18:25 ashiq_ joined #gluster
18:27 nathwill_ joined #gluster
18:28 portante joined #gluster
18:28 qazar @joejulian, please look: http://pastebin.com/z1MfQHZU
18:29 kotreshhr left #gluster
18:29 k4n0 joined #gluster
18:30 JoeJulian Because it received an unmount from fuse. Try your mount using "bash -x /usr/bin/mount.glusterfs". See if that offers some clue.
18:33 qazar ok
18:33 abyss^ JoeJulian: https://gist.github.com/anonymous/416d43cae4c7e0d3b33f318c7a230bc8
18:33 glusterbot Title: glusterfs · GitHub (at gist.github.com)
18:34 abyss^ JoeJulian: I get attr and both are the same
18:34 JoeJulian abyss^: yes, there is no split-brain there.
18:34 JoeJulian So where are you getting that they are split-brain?
18:34 JoeJulian abyss^: also, do you only have 2 servers?
18:37 qazar @joejulian  /usr/bin/mount.glusterfs binary doesn't exist
18:37 JoeJulian Ok, wherever your distro has that script.
18:38 abyss^ JoeJulian: no. OK I thought if first brick is replica second I should have more than two instances of this directories... Ok I going to look for different attr and I'll give a sign.
18:38 JoeJulian abyss^: directories are created on all bricks.
18:38 abyss^ JoeJulian: ok I see.
18:41 abyss^ JoeJulian: I should look for difference in particular place? For example in gluster.dht? Because on different brick not replica I have everything the same except dht: trusted.glusterfs.dht=0x00000001000000007ffffcaa999996ed
18:41 JoeJulian That's the way it should be.
18:41 shyam joined #gluster
18:42 abyss^ but on replica of this brick is the same
18:42 JoeJulian So going back to: where are you getting the information that tells you they are split-brain?
18:43 abyss^ JoeJulian: gluster volume heal saas_bookshelf info split-brain
18:43 abyss^ I see this directory among other thing
18:43 qazar @joejulian ../sbin/mount.glusterfs is the location
18:43 JoeJulian abyss^: Note the timestamps in that output.
18:45 abyss^ errr, but there's no timestamp
18:45 JoeJulian paste up the output of "gluster volume heal saas_bookshelf info split-brain"
18:46 abyss^ https://gist.github.com/anonymous/9cfa82a712fd87d7bf215c536aa51d70 - there was much more but mostly I fix by your script on blog :)
18:46 glusterbot Title: split-brain · GitHub (at gist.github.com)
18:47 abyss^ and here is the problem because directories are to large to just delete them
18:47 flying joined #gluster
18:48 abyss^ because before it will delete the files will replicate on replica and I can remove directory;)
18:49 JoeJulian abyss^: let's see that same getfattr output for nowaera/app/ne-bookshelf-yb.ydp.eu on gluster-saas-3-prd and gluster-saas-4-prd
18:50 JoeJulian Sorry, more specifically gluster-saas-3-prd:/ydp/shared5/gfs/nowaera/app/ne-bookshelf-yb.ydp.eu and gluster-saas-3-prd:/ydp/shared5/gfs/nowaera/app/ne-bookshelf-yb.ydp.eu
18:51 shyam joined #gluster
18:51 qazar + warn 'Mount failed. Please check the log file for more details.' + echo 'Mount failed. Please check the log file for more details.' Mount failed. Please check the log file for more details.
18:52 JoeJulian qazar: That's the results of the failure. What was the cause?
18:52 qazar ..
18:52 ankitraj joined #gluster
18:52 qazar + '[' 0 -ne 0 ']' ++ stat -c %i /dbreports + inode= + '[' 1 -ne 0 ']'
18:53 qazar that's the test it's doing
18:54 JoeJulian qazar: So, when the fuse filesystem is mounted, the inode for the mount point *should* be inode 1. Looks like your fuse makes it inode 0, which is odd.
18:55 JoeJulian You could play with modifying that test and see if it actually works.
18:55 qazar ok
18:55 abyss^ JoeJulian: there is difference! https://gist.github.com/anonymous/425d13c2761a9faf57a704a31290d3df
18:55 glusterbot Title: split-brai2 · GitHub (at gist.github.com)
18:56 abyss^ in request you gave me twice the same server I assumed you meant gluster-saas-4-prd not 3, yes?:)
18:58 JoeJulian abyss^: Aha! (You're right. I meant to change that.)
18:58 farhoriz_ joined #gluster
18:58 abyss^ ok JoeJulian it clear for me how to find split-brain I just must look at output of heal split-brain (I'm not sure why I didn't see that:/)
18:59 JoeJulian So I see two problems. 1 is that client-9 says it has changes for client-8 and client-8 has changes for client-9. Neither one has a matching trusted.glusterfs.dht.
18:59 farhori__ joined #gluster
18:59 abyss^ but how to fix it?;) I suppose now I have change that dht to the same value?
19:00 JoeJulian What I would do is reset the trusted.afr non-zero attribute from one of them (setfattr -n trusted.afr.sass_bookshelf-cleint-9 -v 0x0 $dirname). This will not fix the glusterfs.dht allocation but you can fix that later.
19:01 JoeJulian Once you have the split-brains healed, run "rebalance fix-layout" to fix the glusterfs.dht allocations.
19:01 abyss^ errr I lost;) what client-9?;)
19:01 JoeJulian https://gist.github.com/anonymous/425d13c2761a9faf57a704a31290d3df line 5
19:01 glusterbot Title: split-brai2 · GitHub (at gist.github.com)
19:01 JoeJulian er, line 6
19:02 abyss^ oh, ok;)
19:02 abyss^ I didn't notice the difference;)
19:03 abyss^ I should do this setfattr on client side or server side?
19:03 JoeJulian Those are bricks 9 and 10 (client-N where N starts numbering from 0). Each pair of bricks will have their own client-N.
19:04 sage_ joined #gluster
19:04 vbellur joined #gluster
19:05 flyingX joined #gluster
19:06 abyss^ JoeJulian: I will reset client-9 but I should reset client-8 on gluster-4 I suppose?
19:07 abyss^ so I should do setfattr -n trusted.afr.sass_bookshelf-cleint-9 -v 0x0 $dirname
19:07 abyss^ on gluster 3
19:07 abyss^ and setfattr -n                    trusted.afr.sass_bookshelf-cleint-8 -v 0x0 $dirnam on gluster 4 ?
19:07 JoeJulian No, just change one side. Let self-heal fix the other.
19:10 abyss^ JoeJulian: I get:
19:10 abyss^ setfattr -n trusted.afr.saas_bookshelf-client-9 -v 0x0 /ydp/shared5/gfs/nowaera/app/ne-bookshelf-yb.ydp.eu/
19:10 abyss^ bad input encoding
19:11 annettec joined #gluster
19:11 JoeJulian maybe you need all the 0's. I've always done it that way, but I don't see why you should have to.
19:12 leafbag joined #gluster
19:12 leafbag quick question... if i see the file attr `trusted.glusterfs.dht.linkto="bigdata2-client-14"` i know that bigdata2 is the name of our volume, how do i determine what client-14 is?
19:14 abyss^ JoeJulian: works :)
19:14 annettec1 joined #gluster
19:15 JoeJulian leafbag: similar to what I was just saying moments ago. client-N starts numbering from 0. So it's actually Brick 15.
19:17 abyss^ JoeJulian: OK I get it how to fix. I just have to zero afr.x-client-x from where I have to keep the directory and command rebalance will rebalance this file to replica, right?
19:17 JoeJulian correct
19:17 qazar hello
19:17 glusterbot qazar: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
19:17 qazar what is the client, glusterfs or glusterfs-fuse ?
19:17 qazar client package.. i mean
19:17 JoeJulian depends on the distro
19:17 qazar centos
19:17 abyss^ JoeJulian: you're great:) Thank you very much. Now I know how to fix rest of directories:)
19:17 JoeJulian EL based distros it's glusterfs-fuse
19:18 JoeJulian abyss^: good luck. Get some sleep. :)
19:18 qazar is that normal that glusterfs-fuse requires package glusterfs ?
19:18 qazar Error: Package: glusterfs-fuse-3.8.5-1.el6.x86_64 (/glusterfs-fuse-3.8.5-1.el6.x86_64)            Requires: glusterfs(x86-64) = 3.8.5-1.el6
19:18 qazar i just want to install the client part
19:18 JoeJulian I think so. I've been running Arch for a while so I haven't been paying that much attention to the EL packaging.
19:19 abyss^ JoeJulian: but I don't get one thing... If it start from 0, so client-9 is brick 8, but on brick 8 we haven't got any issue with split-brain...
19:19 JoeJulian no, client-9 is brick 10
19:19 abyss^ we have an issue on brick9 and brick10, yes (for example), right?;)
19:20 JoeJulian Right, we were just looking at bricks 9 and 10 which were client-8 and client-9.
19:20 abyss^ oook;) brick 1 = client=0, brick 2 = client=1 etc... I lost;)
19:20 JoeJulian I wish they'd numbered them consistently, but it would be bad to change it now.
19:21 abyss^ oohhh Ok, I'm stupido!;)
19:21 abyss^ never mind :/
19:21 abyss^ shuuut, to much time in front of computer I suppose ;)
19:21 abyss^ JoeJulian: thank you very, very much :)
19:22 JoeJulian You're welcome. :)
19:24 panina joined #gluster
19:27 nishanth joined #gluster
19:32 farhorizon joined #gluster
19:38 qazar @joejulian,  mounting is working well if we are removing the umount command from the /sbin/mount.glusterfs script
19:38 qazar @joejulian, i have also tested the centos fuse client, with the exact same results
19:39 mhulsman joined #gluster
19:39 qazar @joejulian (latest 3.8.5)  is creating same behaviors
19:39 qazar workaround is to modify the /sbin/mount.glusterfs script
19:40 qazar @joejulian, any idea why ? stat test command isn't returning same results under Oracle vs Centos ?
19:41 haomaiwang joined #gluster
19:41 riyas joined #gluster
19:50 panina joined #gluster
19:53 calisto1 joined #gluster
19:58 JoeJulian qazar: because oracle likes to make things work in non-standard ways to lock you in.
20:02 qazar @joejulian, both version returning the same thing....  Oracle glusterfs client 3.7.4 vs Centos glusterfs client 3.8.5.   So the issue is the stat command ?  Fuse module ?
20:02 farhorizon joined #gluster
20:04 qazar Alternate question, Is there a way to change the server names (here server1 & server2) once the test-volume replicat has been set..  gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2
20:05 qazar we just renamed our servers.. and the callbacks aren't going fine since then. we need to put workarounds inside the hosts files
20:07 JoeJulian qazar: I think you should be able to: replace-brick $old_hostname:$brickpath $new_hostname:$brickpath commit force
20:12 skylar joined #gluster
20:18 leafbag JoeJulian: thanks for the info on client # vs brick # -- I've tried to verify this with a number of files and found a few that do not match. One says bigdata2-client-13 and the data is actually living on brick-13 instead of 14
20:19 leafbag could this be the result of something gone wrong with a rebalance?
20:19 JoeJulian Sure
20:19 panina joined #gluster
20:19 leafbag Any ideas on a fix? This is part of a much larger problem
20:20 JoeJulian Why are your symptoms?
20:20 JoeJulian Why? What! I need more coffee.
20:21 leafbag here's the email i sent to gluster users yesterday
20:21 leafbag http://www.gluster.org/pipermail/gluster-users/2016-November/029143.html
20:21 glusterbot Title: [Gluster-users] Gluster File Abnormalities (at www.gluster.org)
20:21 leafbag err, this morning
20:21 leafbag I too need moar coffee
20:24 qazar @joejulian, could i perform this online ? or i must stop volume, and ask clients to umount ?
20:25 JoeJulian Should work live. Just make sure you've probed by the new hostnames first so the peer info has the new hostnames as well.
20:28 DoubleJ joined #gluster
20:30 qazar @joejulian, how do i probe ?  it's the same physical server, only the hostname has changed...  ping is working fine for me at this time
20:30 JoeJulian Do you have more than one server?
20:30 dnorman joined #gluster
20:32 DoubleJ http://pastebin.com/5WN5S63B - 99% of latency is caused by fop WRITE, but disk util is at most 1-2% according to iostat
20:32 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
20:33 DoubleJ http://paste.ubuntu.com/23482364/
20:33 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
20:33 qazar @joejulian, yes.  replication..  but the client only using 1 ip for now..
20:33 qazar right now, i have tried the replace-brick..
20:33 qazar Brick may be containing or be contained by an existing brick and i'm getting
20:33 qazar Brick may be containing or be contained by an existing brick
20:34 qazar i can't just rename apparently..
20:34 JoeJulian Darn
20:34 qazar the server node..
20:34 JoeJulian I was hoping "force" would do it.
20:34 qazar it need a new volume ?
20:34 arpu joined #gluster
20:34 qazar to rename the server hostname ?
20:36 JoeJulian Well, if you're going to stop the volume anyway, you could replace the hostname in the files under /var/lib/glusterd as long as glusterd is stopped.
20:37 JoeJulian DoubleJ: Looks like 99% of fops are write, too.
20:38 DoubleJ JoeJulian: yes, and that's causing most of the latency. but why so?
20:38 DoubleJ the disks are almost idle
20:39 DoubleJ I also experience high CPU sage of the gluster processes, I'm wondering if the problems are related
20:40 JoeJulian If I did 100 things that all take 1 second and 99 foo and 1 bar, foo is going to take up 99% of my latency - so that doesn't jump out at me.
20:42 DoubleJ hm, my log is also full of "... has not responded in the last 42 seconds, disconnecting."
20:42 JoeJulian 105us avg latency. Most writes seem to be in the 128byte range. Add ~40 bytes for a TCP header, another few bytes for RPC, that's a lot of network overhead.
20:43 JoeJulian That's a problem though.
20:43 DoubleJ all servers are connected via 1gbe
20:43 JoeJulian What version?
20:43 DoubleJ 3.8.5
20:43 DoubleJ i had no problems while running 3.4.2, at least no disconnects - they started appearing when upgrading to 3.8.5
20:43 JoeJulian Was just going to ask that question.
20:44 DoubleJ did the new version add network overhead?
20:44 DoubleJ the workload did not change
20:44 JoeJulian I'm going to be quiet for a little while. I noticed something odd when I upgraded from 3.7 to 3.8. Something used up a lot of cpu for several hours, but I kind-of ignored it after it went away.
20:45 JoeJulian I'm going to dig through some source and see if there's something that happens as part of an upgrade that isn't documented.
20:47 DoubleJ I'm als running gluster 3.6.9 on another cluster with the same workload, the servers are only connected via 100 mbit  - but no disconnects
20:48 DoubleJ no problem with 3.4.2, none with 3.6.9 on an even slower connection, but loads of disconnect  3.8.5... It's hard to believe that there's not a regression of some kind with the new version
20:56 zat1 joined #gluster
21:02 R0ok_ joined #gluster
21:05 mhulsman joined #gluster
21:07 elastix joined #gluster
21:13 farhorizon joined #gluster
21:14 farhorizon joined #gluster
21:15 farhoriz_ joined #gluster
21:15 DoubleJ (also, on the 3.6.9 cluster i'm currently running a rebalance, and even that does not clog the network connection/cause disconnects)
21:29 haomaiwang joined #gluster
21:35 R0ok_ joined #gluster
21:39 eKKiM joined #gluster
21:43 leafbag My understanding is that the 1000 perm (stub) files report the actual size of the file but are size zero on disk. We've seen a number of stub files that are reporting the same size on disk as the actual file. Is there a way for us to differentiate the broken stubs from the good ones? We're trying to come up with a way to figure out which files on t
21:43 leafbag he cluster have gotten FUBAR.
21:48 P0w3r3d joined #gluster
21:48 siel joined #gluster
22:08 derjohn_mobi joined #gluster
22:17 zat1 joined #gluster
22:20 kpease_ joined #gluster
22:38 bluenemo joined #gluster
22:47 vbellur joined #gluster
22:51 P0w3r3d joined #gluster
23:00 annettec1 joined #gluster
23:05 dnorman joined #gluster
23:10 david___ joined #gluster
23:11 P0w3r3d joined #gluster
23:11 david___ hi all, what is the best upgrade path to go from 3.5 to 3.7 ? I have 5 servers, with 4 volumes ( distributed , distributed-replica)
23:11 david___ thanks in advance
23:18 haomaiwang joined #gluster
23:30 JoeJulian leafbag: The dht.linkto files should be size 0.
23:32 caitnop joined #gluster
23:33 JoeJulian david___: Upgrade servers first, then upgrade clients.
23:33 P0w3r3d joined #gluster
23:35 david___ hi JoeJulian, yes, but should I go directly from 3.5 to 3.7 ?
23:35 david___ ( on server side )
23:35 JoeJulian Should work. Test first, of course.
23:35 david___ we can't really shutdown gluster volumes.
23:36 JoeJulian Well your distributed volumes without replica are going to lose connection to the files on those bricks when you upgrade those. Without replication there's no way around that.
23:37 david___ yes, we can shutdown distributed volume - as it is for internal use
23:37 david___ i'm just worry about the files attribute/ self healing if I jump from 3.5 to 3.7
23:38 david___ or any other unforeseen issues
23:38 JoeJulian Oh, right...
23:38 david___ I saw not for 3.7, It needs client to be on at least v3.6
23:38 david___ (note )
23:38 Marbug_ joined #gluster
23:39 JoeJulian https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.0.md#known-issues
23:39 glusterbot Title: glusterfs/3.7.0.md at release-3.7 · gluster/glusterfs · GitHub (at github.com)
23:40 JoeJulian The allow-insecure settings are required to allow the <3.7 clients to connect to a >=3.7 server
23:41 david___ Ok, so I should configure rpc-auth-allow-insecure before upgrading one server after the other
23:43 P0w3r3d joined #gluster
23:44 JoeJulian Yes
23:44 JoeJulian and server.allow-insecure on
23:44 JoeJulian both of them
23:55 DoubleJ JoeJulian: did you find something?
23:55 JoeJulian No. There's got to be something though.
23:56 DoubleJ atm, I don't know where to look next - using iftop, the network link is somewhere around 5-20 mbit/s
23:57 DoubleJ after 1 day of uptime, one of the glusterfs processes is at 14 cpu hours...
23:58 JoeJulian Yeah, it's obviously something happening on the server itself. Short of finding it in the source, I just don't know where else to look. I'm also going to ask on the mailing lists.
23:59 JoeJulian I did give it a little attempt when I saw it here at home, but that's right when my daughter broke her ankle so I got tied up and by the time things settled down again it was done with whatever it was doing.
23:59 DoubleJ on one of the servers, I even get "INFO: task dw.out:3995 blocked for more than 120 seconds." messages in the syslog
23:59 DoubleJ where dw.out is a process writing to the  gluster mount

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary