Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-11-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 marc_888 Do you know if the above commands are the correct way to add a new brick to a replicated cluster?
00:03 plarsen joined #gluster
00:07 msvbhat joined #gluster
00:23 vbellur joined #gluster
00:23 msvbhat joined #gluster
00:40 yato_ joined #gluster
01:19 anthony25_ joined #gluster
01:21 major joined #gluster
01:26 decayofmind joined #gluster
01:27 wistof joined #gluster
01:28 ThHirsch joined #gluster
01:28 BitByteNybble110 joined #gluster
01:28 owlbot joined #gluster
01:28 malevolent joined #gluster
01:28 NuxRo joined #gluster
01:28 mk-fg joined #gluster
01:28 amye joined #gluster
01:28 abyss^ joined #gluster
01:28 pdrakeweb joined #gluster
01:28 javi404 joined #gluster
01:28 sage__ joined #gluster
01:28 Tartifle joined #gluster
01:28 arif-ali joined #gluster
01:28 tg2 joined #gluster
01:28 yosafbridge joined #gluster
01:28 samppah joined #gluster
01:28 scc joined #gluster
01:28 ingard joined #gluster
01:33 decayofmind joined #gluster
01:35 yosafbridge joined #gluster
01:49 vandb_ joined #gluster
01:52 vandb__ joined #gluster
01:55 prasanth joined #gluster
02:02 gospod3 joined #gluster
02:18 vandb__ joined #gluster
02:30 gyadav joined #gluster
02:56 ilbot3 joined #gluster
02:56 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:06 aravindavk joined #gluster
03:36 jeffspeff joined #gluster
03:40 DV joined #gluster
03:40 [Kid] joined #gluster
03:40 [Kid] so, i see references to iSCSI and Gluster, but is it possible to use Fiber Channel with Gluster?
03:41 [Kid] basically present a LUN to a linux server and then build gluster on top of that
03:42 aravindavk joined #gluster
03:42 [Kid] really, the only problem i can forsee is multipathing
03:43 [Kid] and how gluster handles that. i.e., i have a linux server that shows 4x 500G drives. They are all the same LUN, but gluster does gluster know that?
03:43 [Kid] or do i have the ubuntu host handle multipathing and just treat each LUN as a drive in the eyes of gluster?
03:56 PatNarciso_ [Kid], what type of gluster volume are you setting up?
03:58 [Kid] i am not sure yet
03:58 [Kid] i am new to gluster
03:58 [Kid] i want to use it for kubernetes
03:59 PatNarciso_ imo: whatever LUN setup provides the lowest IO.  which I would imagine is some sort of LVM, presenting a single disk to gluster.
04:00 psony joined #gluster
04:01 [Kid] yeah, it would be presenting the single disk to gluster if the linux host handles multipathing
04:01 vbellur joined #gluster
04:01 [Kid] and then i would just create a LUN per host?
04:01 [Kid] maybe even multiple?
04:02 [Kid] although i think multiple would be overkill since the SAN is already handling some fault tolerance on a single LUN
04:02 PatNarciso_ I'm trying to get an idea of your end goal.  how many hosts are you considering?
04:07 [Kid] i have 8 at one site and 6 at another
04:07 [Kid] will be separate gluster clusters
04:07 [Kid] end goal would be to use fiber channel with kubernetes
04:08 [Kid] since i can't do that natively, i need a file system on top of the LUNs i present to the OS
04:08 [Kid] gluster fills that need
04:08 [Kid] or i hope it does
04:09 PatNarciso_ I know very little able kubernetes, other than the google I just performed... the 8 hosts, I'm guessing they need access to files within a shared volume?
04:09 PatNarciso_ s/able/about
04:12 [Kid] yes, they need to be a shared volume
04:14 PatNarciso_ gotcha.  there are a few options for gluster volumes, and depending on what you're looking (performance, redundancy, etc), the volume setup will be different.
04:15 PatNarciso_ the reason why this is important now in your setup, is... I'm unsure about how much redundancy your current LUN setup (or setups?) provide.
04:17 PatNarciso_ deff take a look at the quick start: http://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/
04:17 glusterbot Title: Architecture - Gluster Docs (at docs.gluster.org)
04:20 purpleidea joined #gluster
04:20 purpleidea joined #gluster
04:21 [Kid] i have redundancy from the SAN level
04:24 ronrib joined #gluster
04:27 Humble joined #gluster
05:03 om2 joined #gluster
05:04 bobloblian joined #gluster
05:08 int-0x21 reate-export-ganesha.sh seems to be missing for the 3.12 package in centos
05:22 int-0x21 Does anyone use ganesha with 3.12 in centos ?
05:48 purpleidea joined #gluster
05:48 purpleidea joined #gluster
05:57 ronrib joined #gluster
06:15 TBlaar joined #gluster
06:36 jtux joined #gluster
06:39 xavih joined #gluster
06:52 mbukatov joined #gluster
06:56 wushudoin joined #gluster
07:12 skoduri joined #gluster
07:30 [diablo] joined #gluster
07:41 blue left #gluster
07:47 int_0x21 joined #gluster
07:52 ivan_rossi joined #gluster
08:05 kharloss joined #gluster
08:20 ronrib_ joined #gluster
08:34 marc_888 joined #gluster
08:37 buvanesh_kumar joined #gluster
09:10 ronrib__ joined #gluster
09:25 ronrib_ joined #gluster
09:56 marc_888 joined #gluster
10:02 _KaszpiR_ joined #gluster
10:20 TBlaar joined #gluster
10:27 buvanesh_kumar joined #gluster
10:43 nbalacha joined #gluster
10:52 map1541 joined #gluster
11:08 bfoster joined #gluster
11:15 tacoboy joined #gluster
11:30 rafi joined #gluster
11:36 susant joined #gluster
11:38 MrAbaddon joined #gluster
11:38 rastar joined #gluster
11:45 rastar joined #gluster
11:51 rastar joined #gluster
12:32 shyam joined #gluster
12:44 skumar joined #gluster
12:57 kdhananjay joined #gluster
12:59 phlogistonjohn joined #gluster
13:16 ant0nio joined #gluster
13:19 ant0nio Hey All. Im using gluster for over a year, in a 2 node cluster. I was using 3.8 and last weekend i upgraded to 3.10. I'll be upgrading it to 3.12.
13:19 ant0nio in  the mean time, i noticed the following message:
13:20 ant0nio Staging of operation 'Volume Heal' failed on localhost : Command not supported. Please use "gluster volume heal volume1 info" and logs to find the heal information
13:20 ant0nio its being logged on /var/log/glusterfs/glusterd.log, every 10 minutes or so
13:21 Humble joined #gluster
13:22 ant0nio all the brick for the volume are online, i dont see any obvious message that could lead to the reason why this is happening
13:23 ant0nio Any idea where i should look first?
13:28 jkroon joined #gluster
13:30 tacoboy joined #gluster
13:32 msvbhat joined #gluster
13:37 DV joined #gluster
13:48 jkroon has anyone seen load average of >600 during heal operations?
13:50 jiffin joined #gluster
13:53 jkroon can I safely kill the self-heal daemons for a while to prevent that healing process?
14:06 psony joined #gluster
14:16 jkroon i think i found a problem with flock() on a fuse mounted gluster volume.
14:17 jkroon I had to execute gluster volume start <volname> force in order to bring two bricks back online (is there another way?)
14:17 jkroon after this, a bunch of locks won't release and as a result I've got around 800 flock() processes currently not releasing.
14:18 jkroon since they are in uninteruptible wait it makes the load average climb, and some people are freaking out :)
14:19 pladd joined #gluster
14:19 jkroon from other systems I know that executing an rm on the lock file will allow things to go on again, what I don't know is how long the flock() calls will hang around.
14:19 jkroon glusterfs 3.12.2
14:19 sanoj joined #gluster
14:20 _dist joined #gluster
14:22 gyadav joined #gluster
14:35 psony joined #gluster
14:36 lkoranda joined #gluster
14:39 skumar joined #gluster
14:41 hmamtora joined #gluster
14:41 hmamtora_ joined #gluster
14:48 vandb__ joined #gluster
14:51 skylar1 joined #gluster
14:54 jkroon can't find anything with google either ... will have to set up a test system somewhere to try and reproduce this.
15:01 jstrunk joined #gluster
15:16 phlogistonjohn joined #gluster
15:18 farhorizon joined #gluster
15:23 JoeJulian jkroon: I would expect the maximum would be 30 minutes (frame_timeout).
15:33 jiffin joined #gluster
15:44 marbu joined #gluster
15:45 om2 joined #gluster
15:45 jkroon JoeJulian, 30 minutes before those flock's should die off their own accord?
15:46 vbellur joined #gluster
15:46 jkroon and hi btw, been a while.
15:49 timotheus1_ joined #gluster
15:52 baber joined #gluster
16:03 kpease joined #gluster
16:05 kpease_ joined #gluster
16:08 JoeJulian jkroon: that's my guess at a maximum.
16:15 snehring joined #gluster
16:20 _dist left #gluster
16:26 plarsen joined #gluster
16:27 aravindavk joined #gluster
16:47 atinm joined #gluster
16:51 hmamtora joined #gluster
16:51 hmamtora_ joined #gluster
17:10 rastar joined #gluster
17:32 ivan_rossi left #gluster
17:34 Humble joined #gluster
17:57 rafi joined #gluster
18:08 wushudoin joined #gluster
18:08 wushudoin joined #gluster
18:21 farhorizon joined #gluster
18:22 _KaszpiR_ joined #gluster
18:27 mrcirca_ocean Hello, i have a question... how can i see if glusterfs works with tls protocol?
18:32 JoeJulian My inclination is to suggest you configure it and see - but I suspect I may be misinterpreting your question.
18:33 jkroon joined #gluster
18:33 mrcirca_ocean JoeJulian: to see what?
18:34 mrcirca_ocean i open wireshark and i dont see tls protocol
18:34 jkroon JoeJulian, those exact same flock()s are still uninterruptable wait.  looks like a reboot to clear them :$
18:35 JoeJulian jkroon: That's odd. That would have to be a problem at the filesystem level then. Do you have zombies?
18:35 jkroon none.
18:35 JoeJulian mrcirca_ocean: Ah, ok, I see. I was misunderstanding.
18:35 jkroon it triggered when I did a gluster volume start <volname> force
18:36 JoeJulian Even so... if you kill glusterfsd there should be no filesystem locks.
18:36 jkroon well, what happens with a fuse() process that believes it owns the lock already?
18:37 JoeJulian fuse?
18:37 JoeJulian I thought we were on a server.
18:37 jkroon so let's for a moment assume that some process issued flock() prior to the bricks being restarted (start ... force, from what I can deduce kills all the bricks and brings up new ones)
18:37 jkroon the lock file resides on glusterfs.
18:37 jkroon so the flock is on a file that resides on glusterfs, mounted via fuse.
18:38 jkroon sorry if I did not explain this properly.
18:38 JoeJulian No problem. I've been there. It seems so obvious when you've been looking at it for hours.
18:39 jkroon indeed.  thanks for understanding.
18:40 jkroon mrcirca_ocean, what are you trying to see/do?
18:40 mrcirca_ocean if connection between 2 nodes are encrypted
18:40 mrcirca_ocean i do all settings about SSL
18:41 jkroon i've never used ssl on glusterfs (it's slow enough as it is, guess I'd just use IPSec ... but that's me)
18:41 mrcirca_ocean ok cool
18:41 mrcirca_ocean So i am testing it
18:42 jkroon perfect, tcpdump/wireshark really should show you if it sees SSL protocol from the start.  I'm not sure it would look for the SSL/TLS ClientHello signature on all ports by default though.
18:43 mrcirca_ocean i am seeing only tcp packets
18:43 jkroon ... only another TB of data to transfer for this heal.
18:43 jkroon mrcirca_ocean, SSL would reside inside those.
18:47 jkroon eesh, at current rate this heal is going to take 5 days ...
18:50 jkroon JoeJulian, have you ever seen anything like that?  also, on bootup it looks like my bricks doesn't start up by default.
18:52 JoeJulian jkroon: first, I'm not too happy about *all* the bricks restarting on a start..force. That seems like a bug. The lock status would normally be retained and synced up to the restarted brick(s) from bricks that were still up.
18:53 jkroon how can i request it to only start bricks that aren't started?
18:53 rafi joined #gluster
18:53 JoeJulian jkroon: To clear the client locks, if you couldn't umount (you couldn't if the locks were held) you could still kill the client application (glusterfs) which will kill the fuse mount.
18:53 jkroon the start..force is a nice way to restart all bricks post-update, but it really is misnamed.
18:53 JoeJulian jkroon: good question. That's what it used to do and what I would expect.
18:54 jkroon JoeJulian, fuser is indicating that the uninteruptable processes has files open on the mount, so no go :)
18:54 JoeJulian Restarting glusterd will do it.
18:54 JoeJulian jkroon: Right, but killing the glusterfs application that provides the userspace component of fuse kills the mount and any locks.
18:54 jkroon it does?  my experience indicates that that merely restarts glusterd, but not the brick, nor the shd processess.
18:55 jkroon ah yes, kill the FUSE process, so that the filesystem errors out, resulting in IO errors.  that could work.
18:55 jkroon ok, just waiting for all workloads to be migrated (will take about half an hour), then I can try that.
18:56 JoeJulian restaring glusterd has always restarted missing bricks for me. Not sure I've checked that that's still true with 3.12.
18:56 jkroon (I use /proc/ to determine which processes are using no-longer-on-disk binaries and libraries and thus should be restarted, and it always moans about glusterfsd processes until I start..force.
18:57 jkroon ah *missing* bricks, that could work, i'll give that a shot if I bump into it again.
18:57 jkroon i actually think I found the cause for the bricks not coming up on boot.
18:57 JoeJulian Oh, good.
18:57 jkroon boot order is wrong ... net, glusterd, netmount, dns
18:58 JoeJulian I have resolved as a requirement for glusterd.
18:58 jkroon so i moved dns to after net, but before glusterd.  will need to reboot to confirm ...
18:58 JoeJulian oh, boot ordering. how quaint. ;)
18:59 jkroon well, missing dependency in glusterd startup script really :)
19:13 Gambit15 joined #gluster
19:15 map1541 joined #gluster
19:16 skylar1 joined #gluster
19:20 jkroon whoop JoeJulian - you've just saved my ass yet again.
19:20 JoeJulian yay!
19:21 jkroon killing the right glusterfs process releases all the stuck flocks.
19:24 timotheus1_ joined #gluster
19:42 jkroon JoeJulian, you said that you reckon start..force restarting all bricks is probably a bug - what's your reasoning behind that?  I don't consider it a bad idea, but I do think it probably needs some better control to enable flock's to transit in some way perhaps ... but that kind of thing is hard, to say the least.
19:43 jkroon moving sockets (as per php-fpm being a champion of that on upgrade/restart) is one thing, moving structured data is another ...
19:43 jkroon in the case of replica, obviously we can restart one at a time and have them basically transit the data between live nodes, but even on non-replica bricks the process should still work.
19:51 JoeJulian It seems to me that blindly restarting all the servers loses state, can potentially corrupt heals, interrupt the end-user experience, etc.
20:03 skylar1 joined #gluster
20:07 jkroon well, i did specify force.
20:07 jkroon but I do think that start (without force) should perhaps check for non-started bricks, or perhaps another restart option should be provided.  agreed though at state loss.
20:08 jkroon this is going to become a late night again ... now to trigger the migrate to the other host again ... a process trivally taking two hours.
20:14 jkroon i'm sure there must be a way to avoid the state loss, may require some work to pack the state into a file or something for re-use when the new brick daemon starts.
20:14 IRC-Source_77981 joined #gluster
20:16 bjuanico joined #gluster
20:21 bjuanico I added 2 new bricks to a replicated volume (now its 2x2 replicated distributed) and when i mount it from a client using NFS, it reports only the space of the original 1x2 bricks instead of the new total space, when mounting with gluster its reported ok
20:21 bjuanico any ideas?
20:31 JoeJulian bjuanico: what if you mount with fuse?
20:35 bjuanico mounting with glusterfs.fuse reports the correct space
20:35 bjuanico nfs shows only the size of the 2 original replicated bricks
20:41 JoeJulian I seem to remember seeing that a long time ago... Can't find a bug report though.
20:41 bjuanico me neither, been crawling for a while
20:41 JoeJulian Are you running an old version?
20:41 bjuanico nope, 3.12.1
20:43 JoeJulian Well, I'd file a bug report. The Red Hat devs aren't really working on the native nfs anymore, but facebook has taken over continued development.
20:43 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
20:43 JoeJulian Or there's ganesha
20:44 bjuanico thanks anyway
20:45 bjuanico :)
20:47 bjuanico ill probably move to ganesha
20:47 bjuanico thats what we did on our other storage, but im curious about this behavior
20:48 JoeJulian I remember it being a bug, and there being some valid reason why, but I don't remember what that wasl.
20:52 farhorizon joined #gluster
20:59 farhorizon joined #gluster
21:03 farhorizon joined #gluster
21:03 jcall joined #gluster
21:04 farhorizon joined #gluster
21:05 portante joined #gluster
21:07 ndk_ joined #gluster
21:24 major joined #gluster
21:50 msvbhat joined #gluster
21:55 bluenemo joined #gluster
22:04 map1541 joined #gluster
22:47 map1541 joined #gluster
22:49 masber joined #gluster
22:53 bturner_ joined #gluster
22:58 timotheus1_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary