Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2018-01-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 jobewan joined #gluster
00:06 shellclear joined #gluster
00:07 shellclear_ joined #gluster
00:15 gospod2 joined #gluster
00:18 e1z0 joined #gluster
00:23 ron-slc joined #gluster
00:35 jobewan joined #gluster
00:42 jri joined #gluster
01:20 orngchkn joined #gluster
01:20 orngchkn Hopefully someone can point me in the right direction to answer this question:
01:20 gospod2 joined #gluster
01:21 orngchkn How does GlusterFS distribute load for Windows machines that have mounted a share via NFS?
01:22 orngchkn We built a cluster with six machines. I mounted the shared volume with Windows using NFS (v3) using a specific machine's IP address. But it's mysterious to me how Windows will know to connect to different IP addresses where the distributed data may reside.
01:23 javi404 joined #gluster
01:26 caitnop joined #gluster
01:34 arpu joined #gluster
01:46 rouven_ joined #gluster
02:01 gospod3 joined #gluster
02:18 atinm joined #gluster
02:21 atinm_ joined #gluster
02:26 gospod3 joined #gluster
02:33 kotreshhr joined #gluster
02:41 ompragash joined #gluster
03:04 ilbot3 joined #gluster
03:04 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:09 Vishnu_ joined #gluster
03:11 Vishnu__ joined #gluster
03:26 gyadav joined #gluster
03:26 msvbhat joined #gluster
03:29 gyadav_ joined #gluster
03:31 gospod3 joined #gluster
03:34 gyadav joined #gluster
03:38 Saravanakmr joined #gluster
03:51 psony joined #gluster
04:03 atinm joined #gluster
04:10 itisravi joined #gluster
04:12 ppai joined #gluster
04:12 plarsen joined #gluster
04:21 jobewan joined #gluster
04:24 psony|afk joined #gluster
04:25 MrAbaddon joined #gluster
04:26 apandey joined #gluster
04:32 ppai_ joined #gluster
04:37 gospod3 joined #gluster
04:43 jri joined #gluster
04:48 jiffin joined #gluster
04:50 poornima joined #gluster
04:54 sunny joined #gluster
05:01 ompragash_ joined #gluster
05:06 ompragash__ joined #gluster
05:10 gospod4 joined #gluster
05:16 hgowtham joined #gluster
05:16 kotreshhr joined #gluster
05:17 Prasad joined #gluster
05:17 protoporpoise joined #gluster
05:18 karthik_us joined #gluster
05:28 aravindavk joined #gluster
05:30 ompragash joined #gluster
05:33 cmdpancakes joined #gluster
05:33 skumar joined #gluster
05:34 cmdpancakes hello gluster friends! i've been struggling recently a bit with my 3.12.4 cluster and maybe someone can provide insight...long story short, i have 13 hosts with 12x 42T bricks each, everything was cool up until 6.3PB on the mounted volume, but as soon as i probed the 14th peer, the space dropped down to 5.9PB
05:35 cmdpancakes this is the second time i've rebuild this cluster from scratch with the same issue...everytime i get to 6-7PB, eventually something happens and the reported size of the volume decreases dramatically...all the bricks on each host appear healthy in the logs
05:36 cmdpancakes is 12 bricks per host too many? does gluster not support these larger sizes?
05:42 gospod4 joined #gluster
05:45 itisravi cmdpancakes: Is it a plain distribute volume?
05:45 ndarshan joined #gluster
05:46 itisravi Number of bricks must not make any difference in free space reported.
05:46 cmdpancakes itisravi: yep, pure distributed
05:46 Shu6h3ndu joined #gluster
05:47 cmdpancakes it seems like when i added another peer, that ports were over written or something and the space reported to the client was lowered
05:47 cmdpancakes but im just guessing
05:47 itisravi Are you using `df -h ` on the client to calculate the space?
05:47 cmdpancakes yep
05:47 cmdpancakes and it was up to 6.3, probed another peer and it fell back to 5.9P
05:48 poornima joined #gluster
05:48 itisravi this  new peer does not have any bricks yet?
05:48 cmdpancakes correct
05:49 itisravi this is weird. does gluster peer status show everything as connected?
05:49 msvbhat joined #gluster
05:49 cmdpancakes yes, everything looks healthy
05:49 cmdpancakes all 12 glusterfsd processes are on each host that is already in the cluster
05:50 cmdpancakes volume info shows 155 connected 42T bricks
05:50 itisravi hmm okay
05:50 cmdpancakes that should be ~6.5P give or take rounding
05:50 cmdpancakes last night i had over 8.8PB of bricks added and only 1.3PB was being shown with df to a client
05:51 itisravi what about running df on the bricks themseves and aggregating?
05:51 itisravi (just to see that the bricks themselves show the correct free space)
05:52 cmdpancakes each brick is around 44201747968K, total that would be 6851270935040k df reports 6410419993728
05:52 cmdpancakes yeah on the hosts they seem fine
05:52 itisravi beats me
05:52 cmdpancakes is there any way to see on the client end which bricks it is being displayed?
05:53 cmdpancakes i know that sorta defeats the purpose of gluster, but it seems outside of volume info and status which all show healthy, thats all there is
05:53 itisravi no it just sends the stat call to all bricks and sums it up.
05:53 itisravi I wonder if the client got disconnected from a few bricks.
05:54 cmdpancakes thats what it feels like
05:54 cmdpancakes is it possible to have too many bricks to a client?
05:54 itisravi no it should work fine.
05:54 cmdpancakes ok cool
05:54 itisravi cmdpancakes: how about temporarily mounting the volume on one of the bricks and doing a df on that mount?
05:56 cmdpancakes you mean just mounting the whole volume on one of the bricks as a client?
05:56 itisravi yes, just on one of the brick nodes.
05:57 cmdpancakes i'll give it a whirl
05:58 cmdpancakes also, is there any way on the server end to see how large the volume should be?
05:58 itisravi nothing other than manually adding the df outputs I think.
05:59 cmdpancakes gotcha
05:59 varshar joined #gluster
05:59 cmdpancakes updating the auth.allow now
05:59 cmdpancakes also, most of the gluster commands time out with this many bricks it seems
06:00 itisravi hmm.
06:00 cmdpancakes it seems its better to have fewer larger bricks than many smaller ones
06:00 cmdpancakes do you have any advice on how to break up larger volumes like this?
06:02 itisravi Maybe having multiple volumes with fewer bricks is  a better choice from an administration point of view.
06:03 cmdpancakes yeah we were thinking about potentially breaking it up...it was nice to have a single volume but it seems we have probably out grown it
06:04 cmdpancakes we have 11PB total which used to work with 3.7.20 actually...then a host restarted one day and things just came crashing down
06:04 itisravi There's also the  brick multiplexing feature that reduces the number of brick processes running on a node.
06:05 cmdpancakes mounting the volume on a brick also shows 5.9P
06:05 itisravi this is a fuse mount right?
06:06 cmdpancakes yep
06:06 itisravi okay, one last thing is to check using the .meta folder on the mount and see if the client is connected to all bricks.
06:06 itisravi hold on
06:07 itisravi `cat /fuse_mnt/.meta/graphs/active/<volname>-client-i/private|grep connected` should show connected=1
06:08 itisravi where i = 0,1,2...n-1, n is the number of bricks.
06:08 cmdpancakes ah, let me check that
06:08 cmdpancakes our volume has a lot of subfolders so it takes awhile to enumerate sometimes
06:09 psony|afk joined #gluster
06:12 cmdpancakes i have 155 bricks, and 155 clients all with connected = 1
06:12 cmdpancakes i didn't know about this .meta folder though, i'll poke around more at it
06:12 itisravi okay, then I guess its some bug in aggregating at the DHT layer.
06:13 itisravi you should file a bug.
06:13 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
06:13 orngchkn joined #gluster
06:13 skumar_ joined #gluster
06:14 cmdpancakes ok awesome, thanks for the chat itisravi !
06:15 itisravi cmdpancakes: no prob!
06:15 cmdpancakes but overall this number of bricks and size of cluster seems reasonable?
06:16 itisravi sounds okay to me but I'm a dev, so I don't  use as many bricks while testing but I think we have setups where we have 100 bricks
06:17 cmdpancakes yep thats fair...you're response wasn't "whoa man, thats a lot of bricks" so that makes sense :P
06:17 itisravi see http://lists.gluster.org/pipermail/gluster-users/2017-March/030180.html
06:17 glusterbot Title: [Gluster-users] Maximum bricks per volume recommendation (at lists.gluster.org)
06:18 itisravi some feedback from users.
06:19 itisravi looks like Serkan has 1500 bricks  on his volume :)
06:20 cmdpancakes haha ah nice! looks like we aren't that out of the ordinary then
06:21 cmdpancakes thanks again itisravi
06:21 itisravi welcome
06:34 MrAbaddon joined #gluster
06:34 kdhananjay joined #gluster
06:38 kdhananjay joined #gluster
06:43 xavih joined #gluster
06:43 kdhananjay joined #gluster
06:44 kdhananjay joined #gluster
06:48 gospod4 joined #gluster
06:49 kramdoss_ joined #gluster
06:51 orngchkn Sorry if someone answered this after I disconnected, but: How does GlusterFS distribute load for Windows machines mounting the GlusterFS volume with NFS (at a single IP address / machine name)?
06:52 _KaszpiR_ orngchkn it's not
06:54 _KaszpiR_ orngchkn that's why I used consul and samba client
06:54 orngchkn Is this true: When the Windows machine writes to the mounted volume, the files will be distributed by Gluster to various bricks but then when trying to read them some of the files won't be present (because they're on other machines in the Gluster cluster)?
06:54 _KaszpiR_ https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Consul/
06:54 glusterbot Title: Consul integration - Gluster Docs (at gluster.readthedocs.io)
06:54 orngchkn Thanks for the link
06:55 _KaszpiR_ well, connection from windows server to the gluster client will be a single link, then gluster clinet will connect to multiple gluster data nodes on its own
06:56 orngchkn (Not knowing how Consul works:) Will the network traffic all go through that single Consul end-point? Or will the network traffic be load-balanced among the cluster?
06:56 _KaszpiR_ I think the only way to make it 'multiple widows to multiple nfs servers' is to make sure nfs dns server address wil be resolved to ip pool
06:57 _KaszpiR_ consul just wokrs as dns translator
06:57 msvbhat joined #gluster
06:57 _KaszpiR_ it just helps to resolve single dns name into the ip addresses
06:57 _KaszpiR_ and by default it returns random ip of the healthy service per dns request
06:58 orngchkn Once it resolves the IP does the Samba client on the Windows box continue using just the one resolved address? Or does it continue to ask Consul and be balanced?
06:59 _KaszpiR_ consul is not contacted util till dns cache is valid
06:59 orngchkn (Apologies if this is all RTFM)
07:01 orngchkn I see this "the way Windows share works is that it will connect to one of the samba servers, if this server die then transfers are aborted, and we must retry operation, but watch out for delay." so maybe that answered my question
07:01 _KaszpiR_ in our case consul was only contacted in the moment that samba client was initiating connection or when when connection was established an samba server died (connection lost to server, but then samba makes retry connection, you still have to write logic to retry operation on file read/copy failure, though)
07:02 orngchkn To paraphrase you: Once a samba client connected to a particular server, it always stayed associated with that server (until the server went away).
07:02 orngchkn Is that right?
07:02 sunny joined #gluster
07:03 _KaszpiR_ AFAIR yest
07:03 _KaszpiR_ *yes
07:03 _KaszpiR_ until samba connection is connected
07:04 _KaszpiR_ I don't remember if samba closes connections to servers after some time of inactivity
07:04 _KaszpiR_ I introduced consul to avoid dealing with virtual ips and we did not havea control over official dns server
07:05 _KaszpiR_ you probably also should read about ctdb
07:05 _KaszpiR_ afair rackspace has an article about gluster+samba in HA
07:05 _KaszpiR_ and nfs
07:05 _KaszpiR_ with VIP pool
07:06 _KaszpiR_ https://support.rackspace.com/how-to/glusterfs-high-availability-through-ctdb/
07:06 glusterbot Title: Explore high availability of GlusterFS through CTDB (at support.rackspace.com)
07:06 _KaszpiR_ CTDB as explained in this article provides highly available NFS and CIFS services across GlusterFS replica servers. However, it does not load balance connections. To prevent the interfaces from being saturated on any of the GlusterFS servers, you can configure your solution with a round-robin DNS or WINS (or even hosts) for the CTDB-defined IP addresses.
07:07 _KaszpiR_ so yeah, they sued rr-DNS/WINS, I used consul for that
07:08 _KaszpiR_ *used (lol sued)
07:09 _KaszpiR_ of course using consul forces you to have control over samba clients
07:09 _KaszpiR_ and this may be problematic or impossible in certain situaltions
07:11 skumar_ joined #gluster
07:12 _KaszpiR_ orngchkn one more thing, you can easily convert/expand this setup from smb to nfs
07:14 orngchkn So consul can be the endpoint for NFS (with the same limitations, I imagine, of getting assigned to a single server when mounting the share)?
07:14 _KaszpiR_ and it would be possible to avoid consul on windows nodes and just reconfigure their main dns server in such way that it would ask consul server pool for specific dns suffix
07:15 poornima_ joined #gluster
07:15 _KaszpiR_ for simplicty consul just works as dns server and nothig else
07:16 _KaszpiR_ consul as service is not the endpoint for smb/nfs connections
07:16 _KaszpiR_ treat it just like smart bind9/named or windows dns server
07:17 orngchkn Sounds like I've got some reading to do. :) Thanks for pointing me in a direction
07:18 _KaszpiR_ let say that consul is ta derivateve of zookeeper or etcd, if you know those
07:19 _KaszpiR_ but has some cool features ;)
07:20 _KaszpiR_ well more like registrator+etcd+confd
07:28 msvbhat joined #gluster
07:29 skumar__ joined #gluster
07:32 [diablo] joined #gluster
07:45 jtux joined #gluster
07:51 rafi joined #gluster
07:53 gospod4 joined #gluster
07:57 mlhess joined #gluster
08:05 ivan_rossi joined #gluster
08:06 jri joined #gluster
08:26 jri joined #gluster
08:33 hgowtham joined #gluster
08:35 hgowtham joined #gluster
08:45 mbukatov joined #gluster
08:46 fsimonce joined #gluster
08:48 itisravi joined #gluster
08:51 uebera|| Hi. How come https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.10/+packages lists v3.10.9 GA ("Published 2018-01-04") but apt[-get] will not update 3.10.8-ubuntu1~xenial1 on Ubuntu Xenial?
08:51 glusterbot Title: Packages in “glusterfs-3.10” : glusterfs-3.10 : “Gluster” team (at launchpad.net)
08:54 buvanesh_kumar joined #gluster
08:57 ahino joined #gluster
08:59 gospod4 joined #gluster
09:06 vbellur joined #gluster
09:09 ompragash joined #gluster
09:15 ahino joined #gluster
09:20 ronrib_ joined #gluster
09:26 sanoj joined #gluster
09:27 omark joined #gluster
09:43 msvbhat joined #gluster
09:45 sanoj joined #gluster
09:54 pa9tv joined #gluster
10:02 psony|afk joined #gluster
10:04 gospod4 joined #gluster
10:14 MrAbaddon joined #gluster
10:19 itisravi__ joined #gluster
10:20 kramdoss_ joined #gluster
10:32 itisravi joined #gluster
10:40 Acinonyx joined #gluster
10:50 sunnyk joined #gluster
10:50 apandey_ joined #gluster
10:50 gyadav_ joined #gluster
10:50 Prasad_ joined #gluster
10:50 kramdoss__ joined #gluster
10:51 kdhananjay1 joined #gluster
11:00 poornima_ joined #gluster
11:06 kettlewe_ joined #gluster
11:08 Acinonyx joined #gluster
11:09 gospod4 joined #gluster
11:13 Vishnu_ joined #gluster
11:32 shellclear joined #gluster
11:37 ivan_rossi left #gluster
11:41 bfoster joined #gluster
11:51 sunny joined #gluster
12:06 social joined #gluster
12:15 gospod4 joined #gluster
12:18 kotreshhr left #gluster
12:20 skumar_ joined #gluster
12:23 ThHirsch joined #gluster
12:26 ndarshan joined #gluster
12:49 Humble joined #gluster
13:07 kramdoss_ joined #gluster
13:08 phlogistonjohn joined #gluster
13:09 rwheeler joined #gluster
13:16 ahino joined #gluster
13:20 gospod4 joined #gluster
13:27 shyam joined #gluster
13:38 social joined #gluster
13:44 shellclear_ joined #gluster
13:45 buvanesh_kumar joined #gluster
13:51 shyam joined #gluster
14:00 shyam joined #gluster
14:12 msvbhat joined #gluster
14:23 Rakkin_ joined #gluster
14:26 gospod4 joined #gluster
14:31 voidm joined #gluster
14:37 ahino1 joined #gluster
14:45 msvbhat joined #gluster
14:46 phlogistonjohn joined #gluster
14:46 skylar1 joined #gluster
14:48 buvanesh_kumar joined #gluster
14:51 jstrunk joined #gluster
14:52 aravindavk joined #gluster
15:31 gospod4 joined #gluster
15:34 kpease joined #gluster
15:37 Asako_ The message "C [MSGID: 101184] [inode.c:223:__is_dentry_cyclic] 0-meta-autoload/inode: detected cyclic loop formation during inode linkage. inode (00000000-0000-0000-0000-000000000001) linking under itself as brick" repeated 3 times.  How can I see what is causing this?
15:40 gyadav_ joined #gluster
15:41 Humble joined #gluster
15:48 kdhananjay joined #gluster
15:49 Asako_ Also, how do I see what is causing gluster commands to time out?
15:51 ic0n_ joined #gluster
15:51 shyam left #gluster
15:55 ppai joined #gluster
15:55 shyam joined #gluster
16:00 jiffin joined #gluster
16:15 shellclear_ joined #gluster
16:17 plarsen joined #gluster
16:26 vbellur joined #gluster
16:37 gospod4 joined #gluster
17:04 UNM-HPC joined #gluster
17:11 Rakkin_ joined #gluster
17:12 UNM-HPC I have a question related to setting up gluster replica in two nodes with 2 bricks ?
17:14 kpease joined #gluster
17:17 Asako_ UNM-HPC: should be pretty simple
17:19 Asako_ I think I broke gluster
17:20 Asako_ var/mnt/gluster/brick2/inode: detected cyclic loop formation during inode linkage. inode (00000000-0000-0000-0000-000000000001) linking under itself as brick
17:20 Asako_ this shows up in the brick logs multiple times
17:21 ompragash left #gluster
17:35 armyriad joined #gluster
17:36 jiffin joined #gluster
17:42 gospod4 joined #gluster
17:44 msvbhat joined #gluster
17:45 UNM-HPC thanks for the responde , Our goal is to be able to have a replicated system between node 1 and 2 (A bricks) and add an additional 2 bricks (B bricks)  from the 2 nodes. so we can have a total of 28tb replicated mode.
17:45 UNM-HPC At this  I was able to create the replica nodes between node 1 and 2 (brick A) but I’ve not been able to add to the replica together, Gluster switches to distributed replica   when i add it with only 14Tb.
17:58 jiffin joined #gluster
18:02 jiffin1 joined #gluster
18:18 jri joined #gluster
18:32 ahino joined #gluster
18:34 pdrakeweb joined #gluster
18:39 tom[] joined #gluster
18:43 _nixpanic joined #gluster
18:43 _nixpanic joined #gluster
18:43 UNM-HPC left #gluster
18:48 gospod4 joined #gluster
18:49 s34n joined #gluster
18:49 s34n hello
18:49 glusterbot s34n: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answe
18:50 s34n I'm trying to build a clustered fs across a ragtag collection of old servers.
18:51 s34n Each server a different amount of disk space.
18:51 s34n Some servers have multiple disks, most have just one
18:53 s34n When I have more than one disk, is it best to join them up with lvm before I hand them to gluster, or is it best to keep them individual?
18:54 s34n When I have only one disk, is it best to give gluster its own partition? Or should I just let gluster carve up folders on the system partition?
18:56 s34n How smart is gluster about distributing data across disks so that a single failure isn't catastrophic?
19:02 illwieckz joined #gluster
19:02 aronnax joined #gluster
19:03 MrAbaddon joined #gluster
19:06 kettlewell joined #gluster
19:11 jbrooks joined #gluster
19:16 crag joined #gluster
19:22 pasqualeiv joined #gluster
19:46 msvbhat joined #gluster
19:53 gospod4 joined #gluster
20:37 Asako_ s34n: if you have replica enabled it shouldn't matter if a drive dies
20:54 Asako_ is the "brick" directory created automatically?
20:54 major joined #gluster
20:59 gospod4 joined #gluster
21:08 ThHirsch joined #gluster
21:27 naisanza joined #gluster
22:04 gospod4 joined #gluster
22:34 zerick joined #gluster
22:49 msvbhat joined #gluster
22:54 anthony25 joined #gluster
23:09 gospod4 joined #gluster
23:51 major joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary