Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2018-01-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:25 ic0n joined #gluster
01:31 ic0n joined #gluster
01:50 drymek joined #gluster
02:05 ic0n joined #gluster
02:59 ilbot3 joined #gluster
02:59 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:12 ic0n joined #gluster
03:27 ic0n joined #gluster
03:44 psony joined #gluster
03:45 ic0n joined #gluster
04:01 ic0n joined #gluster
04:44 krk joined #gluster
05:01 rouven joined #gluster
05:03 ic0n joined #gluster
05:04 rouven joined #gluster
05:09 rouven joined #gluster
05:27 plarsen joined #gluster
05:28 Shu6h3ndu joined #gluster
05:30 psony joined #gluster
05:31 ic0n joined #gluster
05:39 psony_ joined #gluster
05:46 psony joined #gluster
06:20 ic0n joined #gluster
06:29 msvbhat joined #gluster
07:01 psony joined #gluster
07:01 msvbhat joined #gluster
07:18 psony joined #gluster
07:43 ACiDGRiM joined #gluster
07:46 msvbhat joined #gluster
07:58 psony joined #gluster
07:59 psony joined #gluster
08:28 msvbhat joined #gluster
08:45 illwieckz joined #gluster
09:23 drymek joined #gluster
09:38 psony|brb joined #gluster
10:07 msvbhat joined #gluster
10:20 msvbhat joined #gluster
10:42 buvanesh_kumar joined #gluster
13:07 drymek joined #gluster
13:28 armyriad joined #gluster
13:41 msvbhat joined #gluster
14:17 drymek joined #gluster
14:41 rouven joined #gluster
14:46 rouven_ joined #gluster
14:47 msvbhat joined #gluster
14:49 rouven joined #gluster
15:41 Vapez joined #gluster
15:49 Megaf joined #gluster
15:50 ThHirsch joined #gluster
15:50 drymek joined #gluster
16:05 rouven joined #gluster
16:08 ic0n joined #gluster
16:33 ic0n joined #gluster
16:58 ic0n joined #gluster
17:15 ic0n joined #gluster
17:28 ic0n joined #gluster
17:42 Acinonyx_ joined #gluster
17:48 Acinonyx joined #gluster
17:58 ic0n joined #gluster
18:02 plarsen joined #gluster
18:08 brettnem joined #gluster
18:13 ic0n joined #gluster
18:37 decayofmind joined #gluster
18:37 ic0n joined #gluster
18:52 Acinonyx_ joined #gluster
19:06 ic0n joined #gluster
19:45 ic0n joined #gluster
19:54 Teraii joined #gluster
20:23 ACiDGRiM joined #gluster
20:33 drymek joined #gluster
20:47 DV joined #gluster
21:03 ic0n joined #gluster
21:11 Vapez joined #gluster
21:11 Acinonyx joined #gluster
21:17 ACiDGRiM Anyone online to answer a quick question about healing a Distributed-Replicate volume?
21:25 ic0n joined #gluster
21:29 Acinonyx joined #gluster
21:35 Acinonyx_ joined #gluster
21:47 ic0n joined #gluster
21:50 Acinonyx joined #gluster
21:59 Acinonyx_ joined #gluster
22:16 kale glusterfs process is running with 100% cpu, is this because i use ssl?
22:18 nh2[m] kale: not necessarily, I also use SSL and don't have that problem right now
22:19 kale my gluster-client is sending two to replicas and an arbiter. i am unpacking a large tarball
22:20 kale there is no iowait
22:26 ic0n joined #gluster
22:27 ACiDGRiM Are you able to mount the volume without SSL to isolate?
22:30 kale seems adding more cpus moved theads to other cpus
22:31 kale so glusterfs process now goe beyond 100% cpu
22:32 ACiDGRiM Could also be data load, currently I'm syncing a new node on a 14TB volume and am seeing 200% real load
22:32 ACiDGRiM no SSL involved
22:34 ACiDGRiM if you're using modern CPU's with AES offload, your SSL overhead should be minute
22:34 protoporpoise joined #gluster
22:35 kale it an i7, 870, 2.93 GHz, i do not see an aes cpuflag
22:35 protoporpoise Hey Gluster people, quick question - what's the best way of seeing what volumes, connections or gluster server components are using all the memory up on a gluster host? (ignoring top, htop etc...)
22:36 protoporpoise context: 3 node cluster (1 arbiter), been running fine since 3.13 came out, all of a sudden yesterday all the memory was gobbled up, even though no changes had been made to the cluster or the clients for 4+ days
22:37 protoporpoise htop just shows glusterd as the culprit using 9.7GB reserved and 10.6GB virtual out of the total 11.5GB available
22:37 protoporpoise usually glusterd only uses about 1-2GB at most
22:41 ACiDGRiM kale: is this a VM on that CPU?, sounds like your hypervisor isn't exposing that CPU feature
22:42 protoporpoise glusterd.log just full of the usual 0-management: RPC_CLNT_PING notify failed messages and the normal cannot open shared object file for the nfs xlator
22:42 ACiDGRiM kale: https://ark.intel.com/products/41315/Intel-Core-i7-870-Processor-8M-Cache-2_93-GHz
22:42 glusterbot Title: Intel® Core™ i7-870 Processor (8M Cache, 2.93 GHz) Product Specifications (at ark.intel.com)
22:42 masber joined #gluster
22:43 ACiDGRiM kale: nope, doesn't have AES-NI on the die, you'll have to turn SSL off
22:44 kale ACiDGRiM: adding an extra cpu made the disk my bottleneck, so i am happy for now
22:45 ACiDGRiM kale: ok I assumed when you added an extra cpu it was a VM. glad it's resolved
22:46 ACiDGRiM protoporpoise: sorry I can't help, I'm just a guy trying to get my own answer here
22:46 kale it is a vm, i'm on a mission to make my home server redundant. glusterfs is a part of that :-)
22:47 ACiDGRiM heh, same here, careful of vm sans, I moved to physical due to expandability
22:49 protoporpoise ACiDGRiM: no probs, I'll just leave IRC open for a while and see if anyone notices it and has an idea, I just try avoiding mailing lists at all costs 😂
22:50 ACiDGRiM If it saves you some headache, setup pacemaker/corosync and an active VIP between both nodes. also don't try to use nfs4.1 for vm storage
22:50 ACiDGRiM not to lecture, but I had been trying to get it work the way I thought it should for the last few years, and finally tried the best practice and feel like a dummy
22:50 kale ACiDGRiM: i let the gluster client handle it
22:51 _KaszpiR_ kale so you need 3 boxes, then
22:51 _KaszpiR_ at least
22:51 ACiDGRiM if you're using KVM, then cool! I'm using vSphere, with it's own headaches
22:52 ACiDGRiM _KaszpiR_: eh, the arbitor can be a VM with SSD backend, That's what I do at least
22:52 kale _KaszpiR_: guess you are right, but i will stay on two. right now i am on only one box, once i get my old server moved to that i can use both boxes for kvm/libvirt
22:53 _KaszpiR_ on one of those nodes?
22:54 ACiDGRiM I use 2 physical gluster nodes with a VM arbitor on the same hypervisor I run vcenter/pfsense
22:55 kale ACiDGRiM: i am writing this as i go along uptime.kallenberg.dk, still a work in progress, but i will get there eventually
22:55 ACiDGRiM kale: highly recommend when you go to 2 nodes, spring for some kind of 3rd for arbitor, splitbrain is not worth cheaping out
22:56 kale ACiDGRiM: i guess i could use some low end harware for the arbiter. it does not seem to use that many resources.
22:57 ACiDGRiM no it doesn't, just back it with an SSD, should be fine for home lab, my 14TB array only uses about 20GB on the arbiter
22:58 ACiDGRiM hell, a laptop would do it if you're trying to slap it together with found hardware
22:59 kale i will put it on my todo list, which i quite too long already
23:00 ACiDGRiM :) have fun, my lab is pretty much built and it's just maintaining it. It was more fun building it tbh
23:01 kale are you running dns/www/ldap/mail on it?
23:03 _KaszpiR_ ACiDGRiM having arbiter on the one of the nodes it belongs to is really bad idea
23:03 _KaszpiR_ especially if you have only 2 data nodes
23:03 protoporpoise ++ agreed
23:04 protoporpoise note that the arbiter node when you create it will use as much memory / network as the 'real' nodes though
23:04 _KaszpiR_ better to set up some super cheap third bare metal
23:04 protoporpoise just a VM is fine
23:04 ic0n joined #gluster
23:05 protoporpoise it does use as much memory though which seems pretty crazy, the clients all communicate to the arbiter as if it was a 'real' node too, which means its slightly less efficient
23:05 _KaszpiR_ but with two nodes then vm is on one of them, so you have two scenarios - bare metal without arbiter goes down and storage works, and the other is total collapse cause one bare metal and vm on it dies and the remaining node is in minority
23:06 protoporpoise oh sorry i didn't realise you had no other storage
23:07 _KaszpiR_ and from my experience gluster works better with 5+ nodes
23:08 protoporpoise I bet that's true, our three node (one being replica only) performance is /really/ bad for writes, even with 100% SSD, low latency and high throughput networking.
23:09 protoporpoise if it was for anything other than web-app attachment storage I'd be doing a distributed replicate or something like setup w/ 5 nodes
23:10 _KaszpiR_ maybe tune ssd?
23:11 kale protoporpoise: my arbiter does not receive the same level of data as the replicas
23:11 protoporpoise SSDs are running well, we get 40-60K random 4k write IOP/s per VM and 300-600MB/s
23:11 protoporpoise kale: I know it won't /store/ the data, but surely it gets lots of network traffic etc..?
23:12 kale nope
23:12 protoporpoise wow really
23:12 protoporpoise maybe I do have a problem
23:12 kale i notices that i had to add the following to my .vol file :"option arbiter-count 1" in the "volume replica" section
23:14 protoporpoise hmm, I don't like editing .vol files manually, maybe there's a gluster cli option that should be set
23:15 protoporpoise is that in volname.tcp-fuse.vol ?
23:15 kale i placed it in /etc/glusterfs/home.vol
23:15 protoporpoise what is home.vol? is that the same as glusterd.vol ?
23:16 kale my volume is called home, so i thought home.vol would be a good name for a file describing how to mount that volume
23:16 protoporpoise oh ok, I see, I'm assuming that's debian? centos keeps the .vol's under /var/lib/glusterd/bla
23:17 kale err thats not a daemon, thats a client. but yes if the are changed frequently, then they would go under /var
23:17 protoporpoise I can't even find option arbiter-count in the docs anywhere other than when you create a volume
23:18 kale protoporpoise: yes i had to create /var/lib/glusterd/ manually to add the secure-access file which enabled ssl, did not know the .vol files were meant to go here as well
23:19 protoporpoise if I try and set it on a volume, gluster complains saying it's not a valid option: "volume set: failed: option : arbiter-count does not exist"
23:20 kale protoporpoise: if you mount the volume manually and then check the logs, in my cas /var/log/glusterfs/home.vol then you will se what it was mounted with, and in there that option is. i noticed it by coincidence
23:21 protoporpoise we're using the official kubernetes client to mount the volumes which I believe uses libgfapi, but yeah I'll take a look thanks for pointing me to this
23:23 protoporpoise wow I think there is a problem - "Number of Bricks: 1 x 3 = 3"
23:23 protoporpoise I think that should be 1x2 + 1
23:24 protoporpoise and none of the bricks have (arbiter) after their location
23:25 kale where do you get that info? i could see what i have
23:25 protoporpoise gluster volume info dev_static
23:25 protoporpoise where dev_static is the volume name
23:25 kale Number of Bricks: 1 x (2 + 1) = 3
23:26 protoporpoise gluster 3.13?
23:26 kale 3.8.8-1
23:26 kale debian stretch
23:27 protoporpoise wow that's really old gluster
23:27 protoporpoise it might be different in newer versions
23:29 kale does your arbiter create files of size 0 ?
23:30 protoporpoise https://paste.fedoraproject.org/paste/HX6pHjS3mnWnkfEb4~5k9A
23:30 protoporpoise yeah it does
23:30 glusterbot Title: gluster-vol-example - Modern Paste (at paste.fedoraproject.org)
23:30 protoporpoise they're purely metadata files
23:30 protoporpoise so it's like the arbiter node is acting correctly, but perhaps providing the wrong info to clients
23:31 kale my arbiter create the files with all of their content until i used that arbiter option on the client
23:31 protoporpoise weird
23:31 protoporpoise maybe a bug in that old version of gluster
23:31 protoporpoise because that shouldn't be the case
23:32 kale i have this for my arbiter "Brick3: gluster07:/srv/home/brick (arbiter)"
23:33 protoporpoise hmmm,
23:34 ACiDGRiM Most of my clients access gluster via nfs-ganesha since I only want to expose certain subdirectories or in the case of vSphere only supports nfs. I'm on 3.13.1, and the Arbiter VM only has 4GB of ram and doesn't see near the resource demand of the physical nodes.
23:35 ACiDGRiM Also, the arbiter is on a hypervisor that doesn't use gluster, just local storage
23:37 ACiDGRiM protoporpoise: have you rebooted the node with the memory leak?upgraded your kernel?
23:37 protoporpoise restarting glusterd.service flushes the memory
23:37 protoporpoise and it's happy after
23:37 protoporpoise but i wonder for how long
23:38 ACiDGRiM could be one of those issues where working around is less effort than investigating :)
23:38 ic0n joined #gluster
23:38 protoporpoise haha yeah i know what you mean, bloody memory leaks
23:38 ACiDGRiM I can't tell you exactly because I rebooted both nodes last night, but have been on 3.13 for a while and hadn't noticed any ballooning
23:39 ACiDGRiM :( sad to see 73 day uptime vanish because I reconfigured the bond and something broke
23:48 ACiDGRiM has anyone had an issue where "gluster heal volume glvol info" reports that the volume is not a replicate volume?
23:48 ACiDGRiM when in fact it is
23:50 ACiDGRiM Volume Name: storage3
23:50 ACiDGRiM Type: Distributed-Replicate
23:50 ACiDGRiM gluster volume heal storage3 info
23:50 ACiDGRiM Volume storage3 is not of type replicate/disperse
23:50 ACiDGRiM Volume heal failed.
23:58 ic0n joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary