Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-10-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 plarsen joined #gluster
00:57 bEsTiAn joined #gluster
01:07 shyam joined #gluster
01:54 shyu joined #gluster
01:56 ilbot3 joined #gluster
01:56 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:59 bEsTiAn joined #gluster
02:58 bEsTiAn joined #gluster
03:21 Wizek_ joined #gluster
03:33 masber joined #gluster
03:40 nbalacha joined #gluster
03:47 itisravi joined #gluster
03:49 apandey joined #gluster
03:52 prasanth joined #gluster
03:59 bEsTiAn joined #gluster
04:22 karthik_us joined #gluster
04:24 apandey_ joined #gluster
04:28 sanoj joined #gluster
04:32 aravindavk joined #gluster
04:38 karthik_us joined #gluster
04:42 rafi joined #gluster
04:45 jkroon joined #gluster
04:45 apandey__ joined #gluster
04:46 Saravanakmr joined #gluster
04:50 poornima joined #gluster
04:54 sanoj joined #gluster
05:00 bEsTiAn joined #gluster
05:03 BlackoutWNCT Hey guys, I've got an issue with the gluster client after doing an upgrade yesterday. The client mount point is not showing the full capacity of the cluster (cluster is 300TB, mount shows 60TB) but it appears to be able to access all data from all bricks.
05:04 BlackoutWNCT gluster version is 3.12.2, OS is ubuntu 17.10
05:04 BlackoutWNCT All clients have also been updated to 3.12.2
05:06 ndarshan joined #gluster
05:08 rafi2 joined #gluster
05:10 rouven joined #gluster
05:10 aravindavk joined #gluster
05:20 xavih joined #gluster
05:21 aravindavk joined #gluster
05:23 ndarshan joined #gluster
05:25 rouven joined #gluster
05:29 ThHirsch joined #gluster
05:29 rouven joined #gluster
05:35 kramdoss_ joined #gluster
05:41 Humble joined #gluster
05:42 skumar joined #gluster
05:44 omie888777 joined #gluster
05:45 rouven joined #gluster
05:46 shruti` joined #gluster
05:47 msvbhat joined #gluster
05:49 rouven joined #gluster
05:51 poornima joined #gluster
05:54 omie888777 joined #gluster
05:55 rouven joined #gluster
05:57 kdhananjay joined #gluster
05:59 rouven joined #gluster
06:00 bEsTiAn joined #gluster
06:02 omie88877777 joined #gluster
06:05 rouven joined #gluster
06:08 omergerd2019 joined #gluster
06:09 rouven joined #gluster
06:12 omie88877777 joined #gluster
06:14 susant joined #gluster
06:24 msvbhat joined #gluster
06:25 rouven joined #gluster
06:29 jtux joined #gluster
06:29 rouven joined #gluster
06:36 kramdoss_ joined #gluster
06:37 skoduri joined #gluster
06:43 apandey__ joined #gluster
06:45 nishanth joined #gluster
06:45 rouven joined #gluster
06:46 Shu6h3ndu joined #gluster
06:47 ppai joined #gluster
06:48 ivan_rossi joined #gluster
06:48 ivan_rossi left #gluster
06:49 rouven joined #gluster
06:51 Saravanakmr joined #gluster
06:57 fsimonce joined #gluster
07:00 rouven joined #gluster
07:01 bEsTiAn joined #gluster
07:04 rouven joined #gluster
07:05 jkroon joined #gluster
07:11 [diablo] joined #gluster
07:17 poornima joined #gluster
07:50 rouven joined #gluster
07:59 rouven joined #gluster
08:00 _KaszpiR_ joined #gluster
08:00 ppai joined #gluster
08:04 arif-ali joined #gluster
08:07 buvanesh_kumar joined #gluster
08:11 panina joined #gluster
08:15 rouven joined #gluster
08:15 ahino joined #gluster
08:24 rouven joined #gluster
08:30 rouven joined #gluster
08:34 rouven joined #gluster
08:38 BlackoutWNCT ok so, I've answered my own question from before. Basically it looks like I've got this issue...
08:38 BlackoutWNCT https://www.spinics.net/lists/gluster-users/msg32822.html
08:38 glusterbot Title: Re: upgrade to 3.12.1 from 3.10: df returns wrong numbers - Gluster Users (at www.spinics.net)
08:38 BlackoutWNCT My question now is, what is the process for modifying the .vol files?
08:38 BlackoutWNCT And do I have to edit each of the brick .vol files?
08:45 rouven joined #gluster
08:47 panina joined #gluster
08:49 msvbhat joined #gluster
08:49 rouven joined #gluster
08:57 sunny joined #gluster
09:01 Saravanakmr joined #gluster
09:06 nishanth joined #gluster
09:07 MrAbaddon joined #gluster
09:08 prasanth joined #gluster
09:09 jiffin joined #gluster
09:10 panina joined #gluster
09:10 rouven joined #gluster
09:11 rafi joined #gluster
09:12 kramdoss_ joined #gluster
09:17 rafi joined #gluster
09:19 rouven joined #gluster
09:21 rastar joined #gluster
09:27 legreffier joined #gluster
09:35 Saravanakmr joined #gluster
09:45 rouven joined #gluster
09:58 rouven joined #gluster
10:01 rafi joined #gluster
10:01 rouven joined #gluster
10:06 rouven joined #gluster
10:16 rouven joined #gluster
10:19 rastar joined #gluster
10:21 rouven joined #gluster
10:23 jkroon joined #gluster
10:37 map1541 joined #gluster
11:09 Gestahlt joined #gluster
11:09 Gestahlt Hi
11:09 glusterbot Gestahlt: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer
11:10 Gestahlt I have a question regarding a HA setup. We have a new datacenter and already between our 2 datacenters a glusterfs setup (1 GFS Server each datacenter, means 2 GFS Servers which run mirrored)
11:10 Gestahlt I hate you glusterbot
11:11 Gestahlt Ok anyway, i tried to set up 4 GFS server for the new datacenter and it struck me, that it does not really offer more / better reliability than 2.
11:11 Gestahlt I was testing failover until 3 servers were down and 1 was left
11:12 Gestahlt I noticed half of the written data was gone and 50% of writes were unable to perform
11:12 Gestahlt so i just readded another server and still only 50% of writes and data were working
11:13 Gestahlt I kinda noticed / understood that the setup is not really fully replicated
11:13 Gestahlt is this correct?
11:15 kotreshhr joined #gluster
11:16 cloph 2 peers in a cluster is bad thing, as you can never tell whether it is just a network communication issue between the two servers (and clients may still be able to connect to both machines) or it one peer is down.
11:17 cloph three or more peers are recommended/give better quorum support in this regard.
11:17 Klas that's why arbiters are a wonderful thing =)
11:18 cloph 4 indeed doesn't give that much over 3, as it would be again even number, and to get proper quorum more than half should be up, i.e. you can only use one host (unless you don't care about this and configure quorum differently)
11:25 Gestahlt yeah i also thought of that
11:26 Gestahlt my first setup was with 3, but then again i dont know how it handled replicas in this case
11:26 Gestahlt I had by accident a distributed setup? and i was not sure if it was the right way to do it with glusterfs
11:27 Gestahlt My goal is a fully replicated setup... no matter how many nodes, i always want full replica
11:27 Gestahlt means the file has to exist on every node
11:27 Gestahlt i can go for 3 and for 5, no problem
11:27 Gestahlt but what would be the correct setup then to achieve my goal?
11:29 cloph if you didn't add a "replica x" to your volume create command, you did create a distributed volume.
11:30 jkroon joined #gluster
11:31 cloph Klas: mentioned arbiter, that would be replica 2 (two full copies) + one metadata copy - this is a nice way to have quorum/easy recovery when one host goes down, without the overhead of a third copy (i.e. the third host doesn't need to fullfill the same diskspace requirements, can instead have a smaller disk). And it won't have to send a third copy of the file via the network.
11:32 jjabs_ joined #gluster
11:32 jjabs_ cloph: sorry i got disconnected
11:32 Klas it's also the normal solution in a quurum
11:32 cloph replica 5 (5 full copies) seems overkill/not much benefit from my POV
11:32 jjabs_ what was the last thing you guys wrote regarding replicas?
11:32 Klas also, remember that you can have several arbiters on the same server if you want
11:32 jjabs_ cloph: Maximum HA.. so 5 is ok
11:32 Klas cloph: replica 3 is really the only situation I see as viable with gluster, with or without arbiter
11:33 cloph "mentioned arbiter, that would be replica 2 (two full copies) + one metadata copy - this is a nice way to have quorum/easy recovery when one host goes down, without the overhead of a third copy (i.e. the third host doesn't need to fullfill the same diskspace requirements, can instead have a smaller disk). And it won't have to send a third copy of the file via the network."
11:34 jjabs_ So, when i create a 3 node cluster, i need to specify replica 3, is this correct?
11:34 cloph yes and no.
11:34 cloph peers don't need to be part of a volume :-)
11:34 cloph so if you want a volume with three full copies on the 3 bricks, then yes, specify replica 3
11:34 mattmcc Speaking of replicas.. I had one machine of a replica 2 volume die a month or so ago. Blew the brick away, volume went back to replica 1. Made a new node, created a new brick, added to volume, and that second brick still doesn't have a full copy of the volume.
11:35 mattmcc It's currently about 58gb vs the 83gb the volume actually has.
11:36 cloph mattmcc: what does heal status report? did the heal-daemon crash? get new files synced properly?
11:36 mattmcc If I pick out a file that's missing from the second brick and try to read it through the volume, then the file gets synced over.
11:38 mattmcc And, yeah, new files are fine.
11:39 mattmcc Actually.. That may not be the case anymore, checking.
11:39 cloph then a gluster volume heal volname full  should do the trick
11:39 Wizek_ joined #gluster
11:40 mattmcc No, a new file doesn't appear on the second brick. :/
11:42 mattmcc Ick. Probable reason: http://dpaste.com/0H93J7V
11:42 glusterbot Title: dpaste: 0H93J7V (at dpaste.com)
11:42 cloph gluster volume status volname reports the bricks as up and the self-heal daemon running?
11:43 cloph yeah.
11:46 mattmcc Alright, looks like heal is back in business. Sorry, lame oversight.
12:03 msvbhat joined #gluster
12:12 plarsen joined #gluster
12:38 nbalacha joined #gluster
12:42 hoerup joined #gluster
12:47 shyam joined #gluster
12:48 hoerup hi - if i deploy 3 vms across 3 DC's that are physically located within 20 km's of each other so they have a very low latency (max 1 ms) - is it safe to use ordinary gluster or should i use geo-replicated gluster =
12:49 hoerup ?
13:08 plarsen joined #gluster
13:15 poornima joined #gluster
13:19 kotreshhr left #gluster
13:20 alvinstarr joined #gluster
13:21 cloph hoerup: if you mean whether gluster can cope with that: yes, no prob. But of course latency can add up...
13:22 hoerup out of couriosity how large latency can ordinary gluster cope with
13:23 buvanesh_kumar joined #gluster
13:25 hoerup ... I mean, there must be a line somewhere at which point you should switch to geo-replication instead of ordinary gluster
13:25 cloph that can be pretty large, but then the "I think that host is down" kicks in (but I think that is multiple seconds)
13:25 hoerup ahh
13:26 cloph whether to use geo-rep or not also depends on performance, of course if you have big latency, performance suffers, as it has to wait for the bricks to report their status until it can hand stuff over to the client.
13:27 hoerup my use case is for a sysadmin document server where I must have at least one full copy at each site in a worst-case scenario
13:28 hoerup and even though colleages are discussing ceph for another setup - i would really need these 3 servers to be self-contained
13:32 uebera|| joined #gluster
13:32 uebera|| joined #gluster
13:33 MeltedLux joined #gluster
13:37 TBlaar joined #gluster
13:38 cloph for that you should be fine even with some higher latency, so no problem afaict
13:46 shyam joined #gluster
13:49 susant joined #gluster
14:04 nbalacha joined #gluster
14:22 phlogistonjohn joined #gluster
14:23 Klas hoerup: just remember that the client writes to all servers
14:23 Klas so those paths needs to be clear
14:36 tom[] joined #gluster
14:38 MeltedLux joined #gluster
14:49 nbalacha joined #gluster
14:50 hmamtora joined #gluster
14:55 wccropper joined #gluster
14:55 farhorizon joined #gluster
14:56 wccropper Does anyone have experience with geo-replication failing on lots of files?
14:58 Shu6h3ndu joined #gluster
14:59 sunny joined #gluster
15:02 wushudoin joined #gluster
15:03 wushudoin joined #gluster
15:13 [o__o] joined #gluster
15:16 msvbhat joined #gluster
15:23 MeltedLux joined #gluster
15:25 kpease_ joined #gluster
15:27 panina joined #gluster
15:28 cloph wccropper: not on the number of files per se, but gluster had a bug when syncing symlinks that made it fail. And it is pretty slow/inefficient when it needs to catch up and there are lots of changes
15:29 cloph https://bugzilla.redhat.com/show_bug.cgi?id=1431081 was fixed recently \o/
15:29 glusterbot Bug 1431081: high, unspecified, ---, khiremat, ASSIGNED , symlinks trigger faulty geo-replication state (rsnapshot usecase)
15:29 cloph wccropper: is the status faulty? Or is it just slow?
15:30 wccropper the status was faulty. what I am seeing is files with different gfid on the dest and they fail to sync.
15:36 skylar1 joined #gluster
15:38 kpease joined #gluster
15:43 plarsen joined #gluster
15:54 ThHirsch joined #gluster
15:56 ron-slc joined #gluster
15:57 MeltedLux joined #gluster
16:04 ThHirsch joined #gluster
16:06 rwheeler joined #gluster
16:10 apandey joined #gluster
16:31 msvbhat joined #gluster
16:46 msvbhat joined #gluster
16:49 atinm joined #gluster
16:51 buvanesh_kumar joined #gluster
17:04 NoctreGryps joined #gluster
17:11 kpease_ joined #gluster
17:15 ahino joined #gluster
17:37 anthony25 joined #gluster
17:41 msvbhat joined #gluster
17:46 dlambrig joined #gluster
17:48 [diablo] joined #gluster
17:50 janlam7 joined #gluster
17:56 gbox joined #gluster
18:00 gbox Anyone on that uses tiering?  I’m wondering if the examples with SSD and NVMe hot tiers represent the most viable use case.  What if the hot tier is just raid0 or raid10 on an app server?  Local I/O that’s 2x faster without network latency has to beat jbod or raid6 on another box?  But does the tiering overhead make that less optimal?
18:10 _KaszpiR_ joined #gluster
18:15 cyberbootje joined #gluster
18:26 jkroon joined #gluster
18:33 kpease joined #gluster
18:33 jkroon_ joined #gluster
18:34 dlambrig @gbox tiering overhead is not really significant. Assuming your workload is steady state (you hot data is not changing much), having RAID0 or RAID10 on the hot tier and EC on the slow tier may be worth trying.
18:38 [diablo] joined #gluster
18:39 CrackerJackMack joined #gluster
18:40 gbox dlambrig: thanks that’s very reassuring!
18:43 CrackerJackMack anyone been having issues with rebalancing a distributed volume on gluster 3.12.2, ZoL 6.5.6 and FALLOCATE errors?  I added a new zfs brick (on a new peer) and can't get anything to rebalance to it
18:44 CrackerJackMack asking generally because I can't seem to find anything regarding fallocate errors at all
18:45 CrackerJackMack also not exactly sure how to ask for help with this specific issue so any pointers are welcome
18:46 XpineX joined #gluster
18:48 snehring CrackerJackMack: I have no idea how realistic an option this is for you, but are you able to try a more recent version of zfs on linux?
18:48 snehring that's all I've got
18:51 CrackerJackMack smejromg wasn't sure that was needed as noted in https://github.com/zfsonlinux/zfs/issues/326: As of 0.6.4 the FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE behavior of fallocate(2) is supported.
18:52 CrackerJackMack But I couldn't make sense of the gluster codebase enough to see what flags were being passed to fallocate, not to mention the gluster api has it's own abstracted fallocate for clients
18:53 snehring it's a bit hard to follow yeah
18:55 panina joined #gluster
18:57 CrackerJackMack there are a lot of layers and components so I appreciate your guidance.  doesn't look like there is an official ZoL > 0.6.5
18:58 CrackerJackMack apparently 0.7.x branch still has a lot of little integration bugs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871619#100
18:58 glusterbot Title: #871619 - zfs-dkms: Please package ZFS 0.7.0 - Debian Bug report logs (at bugs.debian.org)
19:01 atinm joined #gluster
19:01 [diablo] joined #gluster
19:06 baber joined #gluster
19:06 anthony25_ joined #gluster
19:10 snehring 0.7.3 is the current latest release
19:10 snehring 0.6.5.11 is the latest in the 0.6.5 branch
19:11 CrackerJackMack ah, I should mention I'm using ubuntu 16.04 LTS
19:12 CrackerJackMack instead of CentOS/RHEL.  I believe if I were using the rpm variant I'd be on 0.7.x already.  "For Ubuntu 15.10 and newer, ZFS packages are provided by the distribution."
19:12 snehring shouldn't really matter as long as you're alright with installing their debs or whatever, disclaimer: I don't run ubuntu, and never had with zfs
19:13 snehring no docs for it though, so ubuntu may just be stuff with what it has
19:14 snehring s/stuff/stuck
19:14 CrackerJackMack I did find, https://github.com/gluster/glusterfs/blob/release-3.12/xlators/storage/posix/src/posix.c#L1015-L1040 finally.  looks like gluster is using supported flags to fallocate
19:14 glusterbot Title: glusterfs/posix.c at release-3.12 · gluster/glusterfs · GitHub (at github.com)
19:15 CrackerJackMack well, odd thing is the volume works, just rebalance doesn't
19:15 CrackerJackMack using native fuse mounts things are swimmingly stable
19:16 CrackerJackMack [2017-10-23 18:48:35.933841] E [MSGID: 115074] [server-rpc-fops.c:1929:server_fallocate_cbk] 0-media-server: 32665: FALLOCATE 1 (c23771a3-4a9d-4e58-88ea-81697e53a1e7), client: willow-11636-2017/10/23-18:48:11:134015-media-client-2-0-0, error-xlator: media-posix [Operation not supported]
19:16 snehring https://github.com/gluster/glusterfs/issues/219
19:17 glusterbot Title: FALLOCATE support with EC · Issue #219 · gluster/glusterfs · GitHub (at github.com)
19:17 snehring seems like that shouldn't apply to you
19:17 CrackerJackMack I saw that too, but not using EC
19:17 CrackerJackMack yeah
19:17 vbellur joined #gluster
19:26 snehring any outstanding gluster updates anywhere?
19:26 snehring seems unlikely with a new peer
19:33 [diablo] joined #gluster
19:34 CrackerJackMack all nodes 3.12.2 and have been rebooted to ensure latest zfs module, kernel, and daemon
19:38 CrackerJackMack when I get the volume diagnostics.log-level which log would that show up in? glusterd?
19:41 CrackerJackMack oh looks like is shows up in the rebalance log
19:45 CrackerJackMack okay, it's not specific to the new brick.  It happens (less often) on the other nodes as well
20:02 farhoriz_ joined #gluster
20:33 CrackerJackMack gone through and checked the build logs on the PPA server https://launchpadlibrarian.net/341826080/buildlog_ubuntu-xenial-amd64.glusterfs_3.12.2-ubuntu1~xenial2_BUILDING.txt.gz and verified that FALLOC_FL_KEEP_SIZE should be used if HAVE_LINKAT is defined.
20:34 CrackerJackMack upgraded all the logs to trace and I don't see anything new that we didn't already suspect
20:36 CrackerJackMack hrm, but it doesn't look like FALLOC_FL_KEEP_SIZE is defined in any fnctl.h on xenial
20:38 CrackerJackMack oh.. https://github.com/gluster/glusterfs/blob/601b6547f2c53651b88a0560a41d702db06c0d1c/libglusterfs/src/syscall.c#L668-L683
20:38 glusterbot Title: glusterfs/syscall.c at 601b6547f2c53651b88a0560a41d702db06c0d1c · gluster/glusterfs · GitHub (at github.com)
20:39 CrackerJackMack if mode is passed when using posix_fallocate nuclear-explsion.gif
20:42 CrackerJackMack I guess it's bug report time
20:44 bluenemo joined #gluster
20:57 CrackerJackMack narrowed it down, specifically related to ZFS
20:58 CrackerJackMack root@sulley:/gluster/media# touch test; fallocate -n -l 1G test
20:58 CrackerJackMack fallocate: fallocate failed: keep size mode is unsupported
21:02 snehring nice
21:19 cyberbootje joined #gluster
21:29 baber joined #gluster
21:34 farhorizon joined #gluster
21:39 farhoriz_ joined #gluster
21:52 Acinonyx joined #gluster
21:55 MrAbaddon joined #gluster
22:56 baber joined #gluster
23:18 msvbhat joined #gluster
23:28 plarsen joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary