Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-10-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:34 tannerb3 I'm having an issue where my brick was unmounted and gluster restarted, and it looks like gluster setup a new brick/allocated a section of the DHT to this brick. I'm getting all sorts of errors on client machines. "no subvolume for has (value)", and "Lookup of ... on <volume>-client-1 fiailed [Transport endpoint is not connected]
00:38 tannerb3 I tried mounting the volume locally on one of the servers, the mount log is continuously spamming 0-farmcommand-client-1: connection to 10.2.3.67:49155 failed (Connection refused); disconnecting socket
00:38 tannerb3 The brick says it is listening on port 49155, however lsof -i tcp doesn't show anything listening
01:12 susant joined #gluster
01:24 rastar joined #gluster
01:43 baber joined #gluster
01:53 rastar joined #gluster
01:55 ilbot3 joined #gluster
01:55 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:02 gospod3 joined #gluster
02:43 itisravi joined #gluster
02:47 lalatenduM joined #gluster
02:56 rafi joined #gluster
02:58 itisravi joined #gluster
03:03 itisravi__ joined #gluster
03:23 ic0n joined #gluster
03:30 itisravi joined #gluster
03:34 psony joined #gluster
03:35 leifmadsen joined #gluster
03:35 leifmadsen joined #gluster
03:38 itisravi joined #gluster
03:39 ppai joined #gluster
03:42 rafi2 joined #gluster
03:43 itisravi__ joined #gluster
03:49 nbalacha joined #gluster
03:53 itisravi joined #gluster
03:55 atinm joined #gluster
03:56 kramdoss_ joined #gluster
03:58 itisravi__ joined #gluster
04:02 dominicpg joined #gluster
04:03 farhorizon joined #gluster
04:14 fenikso joined #gluster
04:15 flomko joined #gluster
04:16 ndevos joined #gluster
04:16 ndevos joined #gluster
04:23 siel joined #gluster
04:24 Humble joined #gluster
04:25 daMaestro joined #gluster
04:30 apandey joined #gluster
04:47 susant joined #gluster
04:58 atinmu joined #gluster
05:01 skumar joined #gluster
05:04 karthik_us joined #gluster
05:08 apandey_ joined #gluster
05:16 yosafbridge joined #gluster
05:17 xavih joined #gluster
05:23 sanoj joined #gluster
05:26 kdhananjay joined #gluster
05:26 karthik_us joined #gluster
05:30 prasanth joined #gluster
05:32 Prasad joined #gluster
05:34 marbu joined #gluster
05:40 skumar joined #gluster
05:41 apandey__ joined #gluster
05:41 Prasad_ joined #gluster
05:44 Prasad joined #gluster
05:47 atinmu joined #gluster
05:56 kotreshhr joined #gluster
06:00 Saravanakmr joined #gluster
06:03 msvbhat_ joined #gluster
06:03 msvbhat joined #gluster
06:07 rwheeler joined #gluster
06:16 jtux joined #gluster
06:20 psony joined #gluster
06:21 portante joined #gluster
06:21 ndk_ joined #gluster
06:24 lalatenduM joined #gluster
06:26 bfoster joined #gluster
06:30 poornima_ joined #gluster
06:31 aravindavk joined #gluster
06:35 social joined #gluster
06:38 skumar joined #gluster
06:46 Humble joined #gluster
06:53 skoduri joined #gluster
06:58 [diablo] joined #gluster
06:59 itisravi joined #gluster
07:03 fsimonce joined #gluster
07:05 ivan_rossi joined #gluster
07:07 apandey joined #gluster
07:08 itisravi joined #gluster
07:18 itisravi joined #gluster
07:21 rouven joined #gluster
07:22 itisravi__ joined #gluster
07:30 arif-ali joined #gluster
07:39 _KaszpiR_ joined #gluster
07:47 jkroon joined #gluster
07:56 psony joined #gluster
08:01 arif-ali joined #gluster
08:10 buvanesh_kumar joined #gluster
08:14 ThHirsch joined #gluster
08:16 Wizek__ joined #gluster
08:27 Saravanakmr joined #gluster
08:44 itisravi joined #gluster
08:46 buvanesh_kumar joined #gluster
08:47 jkroon_ joined #gluster
08:54 buvanesh_kumar joined #gluster
08:57 jkroon_ joined #gluster
09:00 msvbhat joined #gluster
09:00 msvbhat_ joined #gluster
09:14 kotreshhr1 joined #gluster
09:29 skumar joined #gluster
09:36 dorvan joined #gluster
09:36 dorvan hi all
09:38 dorvan i have a problem during compilation related to "make[4]: [Makefile:1000: libglusterfs_la-inode.lo] Error 1 "; that end with: "/usr/include/sys/sysmacros.h:57:45: error: attempt to use poisoned "system""
09:39 dorvan sys-cluster/glusterfs-3.10.1 with sys-libs/glibc-2.25
09:40 skumar_ joined #gluster
09:40 dorvan glibc flag set: audit multilib nscd profile rpc systemtap -caps -debug -gd -hardened -selinux -suid -vanilla
09:43 MrAbaddon joined #gluster
09:50 buvanesh_kumar joined #gluster
10:05 kotreshhr joined #gluster
10:09 dorvan seems related to a deprecation message section at line 50 in /usr/include/sys/sysmacros.h
10:10 buvanesh_kumar joined #gluster
10:14 skumar__ joined #gluster
10:23 shyam joined #gluster
10:32 susant joined #gluster
10:36 buvanesh_kumar joined #gluster
10:41 Saravanakmr joined #gluster
10:50 plarsen joined #gluster
10:54 ndevos dorvan: was this you? https://bugzilla.redhat.com/show_bug.cgi?id=1494654
10:54 glusterbot Bug 1494654: high, unspecified, ---, bugs, NEW , Failure to compile glusterfs with glibc 2.25, exempt sys/sysmacro.h from pragma poisoning.
10:56 ndevos dorvan: it has a patch attached, you could try it out :)
11:07 v12aml joined #gluster
11:26 skumar_ joined #gluster
11:27 Shu6h3ndu joined #gluster
11:33 kkeithley joined #gluster
11:46 kramdoss_ joined #gluster
11:59 rouven joined #gluster
12:13 dominicpg joined #gluster
12:15 nh2 joined #gluster
12:18 shyam joined #gluster
12:25 owlbot joined #gluster
12:25 rouven joined #gluster
12:40 atinmu joined #gluster
12:48 dcyrille18 joined #gluster
12:53 dcyrille18 Hi all, I come to you about a big problem with GlusterFS. I've been test and reproduce this problem with versions 3.5.2, 3.8.8 and 3.10.6 on two nodes installed on Debian 8.9.
12:55 dcyrille18 I search to start and mount 5 bricks on node boot (nodes are servers and clients) but on two starts out of ten, the starting script fails !
12:56 dcyrille18 The script is started by systemD, one minute after machine boot so I think that the network is available.
12:57 dcyrille18 The launched script aims to start glusterd and then starts the five mounts in RO mode.
12:58 dcyrille18 When the script failed, it is generally due to the "Transport endpoint is not connected" error message on brick logs.
12:58 dcyrille18 I see on internet some similar errors without the resolution errata.
12:59 dcyrille18 Do anybody can help me to resolve it ?
13:00 cdrcdr18 joined #gluster
13:02 ppai joined #gluster
13:06 Asako joined #gluster
13:07 Asako Good morning.  I'm having an issue with geo-replication on one of my slave nodes.  Logs show errors like this.  [2017-10-04 03:01:03.269879] E [repce(/var/mnt/gluster/brick2):209:__call__] RepceClient: call failed on peer   call=22261:140408242493248:1507086063.27        metho
13:07 Asako d=history error=ChangelogException
13:07 Asako any idea how to fix this?
13:09 dominicpg joined #gluster
13:09 Asako do I need to resync the entire volume?
13:13 Asako kind of hate to waste 230 GB of bandwidth
13:15 ppai joined #gluster
13:17 Asako You can enforce a full sync of the data by erasing the index and restarting GlusterFS Geo- replication.  Where is the index?  And how do I delete it?
13:18 Asako the docs really suck sometimes :|
13:29 susant joined #gluster
13:30 Shu6h3ndu joined #gluster
13:31 atinmu joined #gluster
13:34 msvbhat joined #gluster
13:35 skylar joined #gluster
13:35 msvbhat_ joined #gluster
13:36 atinmu joined #gluster
13:44 07IABBWGM joined #gluster
13:44 3NAABJUIG joined #gluster
13:48 baber joined #gluster
13:54 _KaszpiR_ joined #gluster
13:56 farhorizon joined #gluster
13:59 Klas dcyrille18: sounds a bit like a race condition, like sometimes the server starts first, sometimes the client
13:59 Klas does the client depend on the server being up in the systemd script?
14:00 jiffin joined #gluster
14:05 hmamtora joined #gluster
14:05 hmamtora_ joined #gluster
14:06 dcyrille18 Hi Klas, I resolved the problem. I add to my systemD script a function waiting for volume started after glusterd daemon started. Thank's a lot.
14:08 nbalacha joined #gluster
14:12 hmamtora joined #gluster
14:12 hmamtora_ joined #gluster
14:21 atinmu joined #gluster
14:21 pioto joined #gluster
14:23 shyam joined #gluster
14:25 dorvan ndevos: thanks for the link. I have to see.
14:26 atinmu joined #gluster
14:27 jstrunk joined #gluster
14:30 Asako I'm also seeing some inconsistency in geo-replication status
14:30 Asako master shows that the slave is in CHANGELOG_CRAWL, slave shows no geo-replication sessions found
14:30 Asako logs show this.  0-management: master not found, while handling geo-replication options
14:42 _KaszpiR_ joined #gluster
14:46 cloph that slave shows no session is normal, unless you  tier  it to yet another location
14:49 Asako strange, my other slave shows all sessions
14:49 _KaszpiR_ joined #gluster
14:50 cloph maybe you think of something different with "showing all sessions" - for me "gluster volume geo status" doesn't report any active sessions on slave (as expected, as I don't think it was different for me)
14:51 Asako gluster volume geo-replication status definitely shows different output on each slave
14:53 Asako paste.fedoraproject.org/paste/ohokZDa5XDZROSURmvsaCA replication seems to be working but I don't know why they show different status
14:54 Asako everything is running gluster 3.12 now
14:55 msvbhat joined #gluster
14:55 msvbhat_ joined #gluster
14:58 shyam joined #gluster
15:00 skumar joined #gluster
15:00 susant joined #gluster
15:03 nh2 joined #gluster
15:06 cloph might be that  your  srv3 is in the same cluster as the master
15:06 leifmadsen joined #gluster
15:06 leifmadsen joined #gluster
15:08 kramdoss_ joined #gluster
15:08 Asako they're all the same
15:08 Asako srv3 and srv4 are both slaves of srv1
15:09 Asako srv2 is a replica
15:09 foster joined #gluster
15:09 wushudoin joined #gluster
15:11 atinmu joined #gluster
15:13 ppai joined #gluster
15:13 Asako srv3 also keeps showing this message in the glusterd.log file.  0-management: Using passed config template
15:16 atinmu joined #gluster
15:18 cloph being slave or not doesn't tell something about  being in the same cluster though. also what type the volumes are should not matter.
15:25 vbellur joined #gluster
15:28 vbellur joined #gluster
15:29 vbellur joined #gluster
15:37 weller hi, I have a gluster volume exported with ctdb. when I fuse mount, everything seems fine. the same thing with vfs_gluster makes powerpoint freeze/hang on first file save (~5 minutes?), but then subsequent saving works fine. since a mounted gluster volume exported by path works, I assume this might be a gluster issue!? thanks in advance for hints! :)
15:39 v12aml left #gluster
15:43 vbellur joined #gluster
15:44 vbellur joined #gluster
15:44 vbellur joined #gluster
15:45 vbellur joined #gluster
15:46 vbellur joined #gluster
15:46 kramdoss_ joined #gluster
15:49 dlambrig joined #gluster
15:55 jkroon joined #gluster
15:56 rafi joined #gluster
16:03 gem joined #gluster
16:21 aravindavk joined #gluster
16:29 baber joined #gluster
16:37 ivan_rossi left #gluster
16:37 gem joined #gluster
16:42 Saravanakmr joined #gluster
16:44 Acinonyx joined #gluster
16:52 baber joined #gluster
16:52 vbellur joined #gluster
17:03 msvbhat_ joined #gluster
17:03 msvbhat joined #gluster
17:11 jiffin joined #gluster
17:19 buvanesh_kumar joined #gluster
17:44 _KaszpiR_ joined #gluster
17:44 marlinc joined #gluster
17:46 plarsen joined #gluster
17:47 buvanesh_kumar joined #gluster
17:50 victori joined #gluster
17:50 kpease joined #gluster
17:57 victori joined #gluster
18:03 rouven joined #gluster
18:22 Brainspackle joined #gluster
18:32 vbellur joined #gluster
18:34 buvanesh_kumar joined #gluster
18:35 baber joined #gluster
18:41 ahino joined #gluster
18:46 vbellur joined #gluster
18:57 omie888777 joined #gluster
18:59 Humble joined #gluster
19:10 jiffin joined #gluster
19:28 shyam left #gluster
19:33 baber joined #gluster
19:34 vbellur joined #gluster
19:36 Wayke91 joined #gluster
19:36 rouven joined #gluster
19:38 omie888777 joined #gluster
19:42 Wayke91 hi all, I'm having a spectacular problem with duplicate empty files on my volume.  The volume heal info is backlogged with over 6000 files, some legitimate paths/names, but most are gfid.  I'm not seeing any split-brain issues, metaata or otherwise, according to Gluster.  It's a distributed-replicated between 2 nodes and an arbiter totalling 12 bricks on 3.12.  After a few hours, the Number of entries counter hasn't
19:42 Wayke91 decreased at all.  The self heal log doesn't appear to have anything, neither does glusterd.log.  The only thing I can see that has anything that remotely looks like it's doing something is the log for the local fuse mount, but it hasn't written anything for almost an hour.  Any pointers on where to start correcting this?
19:47 madm1ke joined #gluster
19:47 |R Trying to setup TLS on 3.12, I'm using a real CA for /etc/ssl/glusterfs.ca, is that a problem or can this be used instead of concatenating all certificates?
19:50 madm1ke can somebody help me figure out why my glusterfs servers won't start any more? https://paste.debian.net/plainh/47416e58
19:50 madm1ke I checked the peer lists and all hostnames resolve properly
19:51 madm1ke in fact the only machine where the server starts is the one mentioned in the log (io-1)
19:51 vbellur1 joined #gluster
19:54 vbellur joined #gluster
19:54 vbellur1 joined #gluster
19:55 Wayke91 <madm1ke>: On this host with the issue, when you do a gluster peer status, does the peer io-1 show any "Other names"?
19:55 snehring joined #gluster
19:55 madm1ke can't run that: "Connection failed. Please check if gluster daemon is operational.
19:57 madm1ke I am on version 3.5.2 btw
19:57 Wayke91 so your peers aren't using the fqdn it looks like to me, how are the names being resolved, by host entries?
20:01 xaban joined #gluster
20:03 xaban Hi, I am new to Gluster, I did read some documentation but still hard to understand the replicated/distributed part. I have 4 servers with each 2 bricks, I want to have 1 volume with all bricks together, within this volume I want to have a replica size of 3, distributed over all servers; so 1 replica per server. Is this possible?
20:07 lcami1 joined #gluster
20:10 Wayke91 I don't think you'll be able to have replica 3 across 4 nodes, you'd end up with a server's worth of storage not able to be used in the volume
20:12 xaban But I need that storage :) ..and I really want 3 replica's without having 2 replica's on the same server.
20:13 Wayke91 Then you'd need 2 other servers.  What you might consider is an arbiter and then do replica 2 + arbiter
20:13 xaban Hmm, let me Google about arbiter; don't know what it is. I am limited to 4 servers, unfortunately..
20:15 Wayke91 Well, the arbiter holds empty copies of your volume with metadata and can be the tie-breaker, it's used to help prevent split-brain.  As such, it doesn't need the capacity of the storage nodes.  Mine is running as a small VM, but it could be anything really, and old retired server, pc, etc.
20:16 madm1ke Wayke91: i have a local dns caching server that all peers are using in their resolv.conf
20:16 lcami1 hi everyone maybe someone can help me out with this: 6-node gluster setup EC(4+2), when the gluster shared storage volume is created it is clearly shown in "gluster volume info" as a 1X3 replica.
20:17 lcami1 the shared volume is mounted from all nodes so a brick exists in all nodes and not just the 3 bricks described in the gluster volume info output...
20:19 Wayke91 the volume being mounted doesn't necesarrily mean that you have bricks for that volume on your node - what does gluster volume status show?  Does it show bricks on all 6 nodes?
20:20 lcami1 Apart from that according to https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-managing_red_hat_storage_volumes-shared_volume the volume can be a replica 2 or replica 3 depending on the number of nodes which are online in the cluster when this option is enabled
20:20 glusterbot Title: 11.8. Setting up Shared Storage Volume - Red Hat Customer Portal (at access.redhat.com)
20:23 lcami1 but in fact a brick is created on each node since in fstab a mount option is inserted for every node and every node mounts the volume locally
20:23 lcami1 volume status shows only 3 nodes and 3 bricks, a  brick for each node...
20:24 Wayke91 and how many peers to you get in gluster peer status?
20:24 lcami1 6
20:24 Wayke91 ok, so it sounds to me like you had 3 nodes to begin with, added 3 more and did gluster peer probe <new servers>, but didn't add their bricks to the volume
20:25 lcami1 i would agree exactly with your reasoning but that is not the case
20:25 lcami1 i probed all the peers as a first step
20:26 lcami1 then i ran "gluster volume set all cluster.enable-shared-storage enable"
20:26 lcami1 this created the special shared brick on all nodes and the entry in fstab
20:26 Wayke91 oooh ok now I see
20:27 lcami1 gluster volume info shows brick on the first 3 nodes
20:27 lcami1 i do not have the servers on to give you exact iutput but I will be working on this tomorrow again and can give you more info if you can help me out on this
20:31 Wayke91 unfortunately I don't have any experience with the shared storage feature, all of mine have been done by manually adding bricks like an animal so I personally wouldn't be of much help, but there are a ton of people on here that come and go
20:31 lcami1 I also have another question on the nfs-ganesha HA cluster, where on the same 6-node setup I enabled 3 nodes to be part of an HA NFS-Ganesha setup, failing one node and failover of VIP and services works fine, but the failback does not happen.... IP's remain on the node to which the failover happened
20:33 Wayke91 madm1ke: Sorry, just now caught your reply.  I found this that looks like it could make sense for you https://stackoverflow.com/questions/23119791/glusterfs-server-will-not-start-after-reboot.  You might try adding io-1 without domain suffix to your host file as a test and see if glusterd starts after that
20:33 glusterbot Title: ubuntu - GlusterFS Server Will Not Start After Reboot - Stack Overflow (at stackoverflow.com)
20:35 madm1ke Wayke91: I thought about that already actually and tried exactly that. Did not help :(
20:40 vbellur joined #gluster
20:42 Wayke91 Oh man
20:44 farhorizon joined #gluster
20:45 lcami1 madm1ke:, are you using IB in your setup as there are a number of rdma failed events in the log?
20:46 major joined #gluster
20:46 lcami1 if gluster does not start i would go with the low level troubleshooting but usually gluster is not that difficult to have it running
20:47 lcami1 it gives a lot of details on what is going on and how it is setup
20:50 madm1ke lcami1: no IB involved
20:51 madm1ke I found some inconsistencies in /var/lib/glusterd/peers regarding hostnames
20:51 madm1ke I think that's the reason
20:54 lcami1 yes sorry did not notice that you were just referencing the link
20:54 lcami1 glad youo have a lead on the problem :-)
20:55 madm1ke servers are starting again.. I guess I fucked up with the initial peer probe somehow
21:02 Vaelatern joined #gluster
21:03 madm1ke thanks for being persisent about the host resolution, Wayke91  :)
21:07 lcami1 left #gluster
21:07 lcami1 joined #gluster
21:11 lcami1 left #gluster
21:20 squarebracket joined #gluster
21:21 squarebracket is there a way to configure the RPC for volume profile stats data to only pull out the stats for a single brick/server? i'm not talking via CLI options, but the actual RPC dict -- trying to program a collectd plugin
21:50 plarsen joined #gluster
21:57 Wayke91 Well, incase anyone is reading I either made my problem better or much worse.  I did a tree listing for my entire volume, and it seems to have cleared up the gfid entries from the volume heal entries, but it now seems that every single file in the volume needs healing
21:58 farhorizon joined #gluster
23:06 farhorizon joined #gluster
23:08 gladoslies joined #gluster
23:47 jbrooks joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary