Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-11-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:11 johnnyNumber5 joined #gluster
00:20 marbu joined #gluster
00:21 cholcombe joined #gluster
00:22 sadbox joined #gluster
00:37 Rivitir12x joined #gluster
00:46 hackman joined #gluster
00:51 Klas joined #gluster
00:58 johnnyNumber5 joined #gluster
01:00 shdeng joined #gluster
01:00 johnnyNumber5 joined #gluster
01:03 luizcpg joined #gluster
01:20 jermudgeon_ joined #gluster
01:21 jermudgeon_ left #gluster
01:37 haomaiwang joined #gluster
01:45 johnnyNumber5 joined #gluster
02:04 thwam joined #gluster
02:13 haomaiwang joined #gluster
02:17 arpu joined #gluster
02:20 johnnyNumber5 joined #gluster
02:22 aj__ joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:56 d0nn1e joined #gluster
03:04 johnnyNumber5 joined #gluster
03:16 Gnomethrower joined #gluster
03:25 johnnyNumber5 joined #gluster
03:43 Lee1092 joined #gluster
03:45 riyas joined #gluster
04:11 riyas_ joined #gluster
04:22 mchangir joined #gluster
04:35 Alghost joined #gluster
05:38 cvstealth joined #gluster
05:40 ahino joined #gluster
05:51 PaulCuzner joined #gluster
06:03 jkroon joined #gluster
06:17 ahino joined #gluster
06:30 Javezim Running a Replica 3 Arbiter 1 on Ubuntu, Bricks are RAIDz2 with ZFS Formatted. Can anyone recommend any other Options that could be configured to improve read/writes? - http://paste.ubuntu.com/23410337/
06:30 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
06:57 skoduri joined #gluster
06:57 riyas joined #gluster
07:16 rouven joined #gluster
07:40 prth_ joined #gluster
07:56 kxseven joined #gluster
08:09 juo joined #gluster
08:09 mhulsman joined #gluster
08:20 pcdummy joined #gluster
08:37 pfactum joined #gluster
08:39 Gnomethrower joined #gluster
08:39 mhulsman1 joined #gluster
08:43 flying joined #gluster
08:44 riyas joined #gluster
08:50 panina joined #gluster
08:54 social joined #gluster
08:57 mhulsman joined #gluster
08:57 mchangir joined #gluster
09:05 kramdoss_ joined #gluster
09:14 panina joined #gluster
09:26 kramdoss_ joined #gluster
09:34 mchangir joined #gluster
09:46 nix0ut1aw joined #gluster
10:16 hackman joined #gluster
10:18 Debloper joined #gluster
10:22 mhulsman joined #gluster
10:30 Gnomethrower joined #gluster
10:30 aj__ joined #gluster
10:35 B21956 joined #gluster
10:46 bfoster joined #gluster
10:51 luizcpg joined #gluster
10:51 bfoster joined #gluster
10:57 skoduri joined #gluster
11:01 B21956 joined #gluster
11:04 luizcpg joined #gluster
11:10 jkroon joined #gluster
11:52 Gnomethrower joined #gluster
11:52 haomaiwang joined #gluster
12:01 johnmilton joined #gluster
12:03 ira joined #gluster
12:09 arc0 joined #gluster
12:26 Ryllise joined #gluster
12:27 Ryllise Re: https://www.gluster.org/pipermail/gluster-users/2015-September/023484.html does anyone know if adding an Arbiter to an already existing volume is possible yet?
12:27 glusterbot Title: [Gluster-users] AFR arbiter volumes (at www.gluster.org)
12:38 nage joined #gluster
12:49 pfactum Ryllise, yes, with 3.8+
12:49 Ryllise Okay so does the Metadata get automatically sent to the Arbiter for the existing data
12:49 Ryllise or does it only apply for new files?
12:49 Ryllise + is there documentation on how to add an Arbiter to an existing Brick?
12:49 post-factum Ryllise, new brick is being healed automatically for all the data
12:50 post-factum type "gluster volume add-brick" and you'll see usage help
12:50 post-factum iirc, you need to do gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/shitty_brick
12:51 Ryllise Because I currently have an 11 brick replicated cluster, I'd want to create 11 Folders on the Arbiter machine and have the metadata replicate to the bricks folder on the arbiter machine
12:51 post-factum yes
12:52 post-factum you will add all 11 bricks with 1 command
12:53 Ryllise So if I say did gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/shitty_brick, would that create a folder for each bricks metadata?
12:54 mhulsman1 joined #gluster
12:55 MadPsy using the PPA on Ubuntu 16.04 I'm getting a dependency problem.. is anyone aware of this: nfs-ganesha : Depends: libntirpc1 but it is not installable
12:55 MadPsy libntirpc1.3 exists but not 1
12:55 MadPsy (gluster 3.8)
12:55 Ryllise https://launchpad.net/~gluster/+archive/ubuntu/libntirpc
12:56 glusterbot Title: libntirpc : “Gluster” team (at launchpad.net)
12:56 MadPsy thanks!
13:17 fcoelho joined #gluster
13:18 squizzi joined #gluster
13:21 jkroon joined #gluster
13:24 ahino joined #gluster
13:24 mhulsman joined #gluster
13:26 post-factum Ryllise, no, you create folder first
13:26 post-factum 11 folders in fact
13:27 alvinstarr I get http://pastebin.centos.org/56781/ just before replication fails.
13:28 Ryllise So then I need to do the command 11 times?
13:28 Ryllise gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick1
13:28 Ryllise gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick2
13:28 Ryllise etc
13:29 post-factum no, all 11 in one line
13:29 Ryllise ahhhh gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick1 gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick2 gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick3 etc
13:29 Ryllise actually woops
13:29 Ryllise gluster volume add-brick replica 3 arbiter 1 new.server.xxx:/brick1 new.server.xxx:/brick2 new.server.xxx:/brick3
13:30 Ryllise etc etc
13:36 post-factum yes
13:38 gem joined #gluster
13:41 glua joined #gluster
13:42 Rivitir12x joined #gluster
13:59 haomaiwang joined #gluster
14:04 jsfrerot joined #gluster
14:06 skylar joined #gluster
14:09 jsfrerot Hi, I'm using glusterfs 3.8.5 and to be honest, i'm very new to glusterfs. I have setup a 2 servers cluster and i'm mounting with gluster client. Now, the problem I have is with performance. I'm trying to copy about 20k small files (less than 4k) to the glusterfs, but it's very very slow. I've looked on google for performance tuning but nothing has improved significantly the performance. I'm looking for
14:09 jsfrerot help to fix this issue if possible.
14:11 marbu joined #gluster
14:12 lkoranda joined #gluster
14:13 Rivitir jsfrerot, sounds like your having the same issues as I am. Is only your write speed effected or both read and write?
14:16 jsfrerot Rivitir: read is also slow
14:17 jsfrerot basicly, the IO is almost not used, neither the network, but cpu is quite used for ~ 50%.
14:20 Rivitir Like you, I have many small files. And so far performance has been horrible. My read performace improved when I mounted with directio option enabled
14:21 Rivitir I have not yet found a solution for write speed. It is taking up to 4 sec per file. Doesn't matter if its 300k or the file is 1.1mb. this is on a local GB switch
14:22 jsfrerot Rivitir: yep, hope we have a solution for that, if not, i'll start looking at other distributed filesystem.
14:24 Rivitir I know gluster can have good speed for small files. Unfortunately I have not yet found the solution. Like you I have asked this question here, hopefully someone can help us.
14:25 mhulsman joined #gluster
14:25 Rivitir Something you can try is turning write-behind off. Hasn't helped me, but may help you.
14:27 Lee1092 joined #gluster
14:27 Rivitir jsfrerot, are you running on Centos 7?
14:28 jsfrerot Rivitir: will try write-behind, and yes centos7
14:30 Rivitir jsfrerot: I also get better performance by mounting gluster from another server, instead of on the node. You may want to try that.
14:32 plarsen joined #gluster
14:34 jsfrerot Rivitir: i did mount from another server, it doesn't improve perf for me
14:39 jsfrerot Rivitir: write-behind didn't improve perf.
14:39 mss joined #gluster
14:40 Rivitir jsfrerot, can you share your volume status and info?
14:40 Rivitir @paste
14:40 glusterbot Rivitir: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
14:48 amye joined #gluster
14:53 ahino joined #gluster
14:58 Jules- joined #gluster
15:04 wushudoin joined #gluster
15:04 wushudoin joined #gluster
15:08 farhorizon joined #gluster
15:08 alvinstarr If on the initial copy of data when setting up a geo-replicated volume the link fails will the copy start from the beginning again?
15:16 shyam joined #gluster
15:19 JoeJulian alvinstarr: My educated guess would be no, but I haven't looked at code to verify that guess.
15:20 JoeJulian ~mount server | Rivitir, jsfrerot
15:20 glusterbot Rivitir: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.
15:29 hagarth joined #gluster
15:30 squizzi joined #gluster
15:32 Rivitir JoeJulian, like jsfrerot I am seeing slow speeds on small files and high CPU. Do you know what we can do to resolve this?
15:33 ira joined #gluster
15:34 Rivitir On a volume with 34k files (all files under 1mb) it takes 10 seconds to run a simple ls.
15:34 ahino joined #gluster
15:35 JoeJulian Sounds like something's broken. Check your client log?
15:35 JoeJulian Is this a replicated volume? What's the latency like between the client and servers?
15:36 Rivitir all my servers on on the same subnet, in same cabinet. Gig switch connecting them.
15:37 Rivitir ping time latency is showing 0.197ms
15:39 Rivitir JoeJulian, I am seeing the following in my gluster log of my client. Gluster volume name is xvweb.: http://termbin.com/p3m2
15:39 prth_ joined #gluster
15:41 JoeJulian Wow, that's a lot of xattr removal attempts. What's the server logs say about any of those?
15:43 Rivitir I am accessing that gluster mount through Samba. To copy I had to use robocopy with the copy DT flags. It would fail on attributes if I didn't.
15:44 pcaruana joined #gluster
15:44 JoeJulian Are you using the gfapi vfs?
15:44 Rivitir JoeJulian, I'm not seeing anything in the logs so far. Any log I should focus on?
15:45 JoeJulian I'd just look for the timestamp
15:49 Rivitir JoeJulian, my glustershd.log: http://termbin.com/48yy my etc-glusterfs-glusterd log: http://termbin.com/12u6
15:49 Rivitir that is what is edited today
15:50 JoeJulian That's the self-heal daemon.
15:50 JoeJulian The brick servers are under bricks/
15:52 Rivitir I see, here is the brick log for xvweb volume. Note that I just rebooted them a bit ago. http://termbin.com/juzj
15:58 JoeJulian Have you tried testing gluster without samba to see if you can isolate the issue?
16:01 Rivitir JoeJulian, honestly I have not. All my connections are coming from Windows servers. To bad their isn't a fuse client for it.
16:02 JoeJulian Too bad windows doesn't have fuse.
16:10 MadPsy joined #gluster
16:10 MadPsy joined #gluster
16:10 nigelb joined #gluster
16:10 rideh joined #gluster
16:10 JPaul joined #gluster
16:15 Rivitir JoeJulian, I think you are right. It must be Samba. I just copied 171 files all about 3k in size and it took about 19 seconds using scp
16:17 kpease joined #gluster
16:18 JoeJulian Rivitir: try playing with disabling oplocks
16:18 JoeJulian I had good luck with that in a previous life.
16:26 Rivitir JoeJulian, thanks for the tip, I will. And thank you for all your help.
16:27 snehring joined #gluster
16:42 [diablo] joined #gluster
16:46 jkroon joined #gluster
16:55 MadPsy Any ideas why this file might be missing with gluster 3.8 / ganesha:  nfs-ganesha-config.service: Failed at step EXEC spawning /usr/libexec/ganesha/nfs-ganesha-config.sh: No such file or directory
17:00 JoeJulian What distro?
17:02 MadPsy ubuntu 16.04 - there isn't even a /usr/libexec/ concept in debian I don't think
17:03 JoeJulian kkeithley: ^
17:05 JoeJulian MadPsy: I would use whatever tool your distro uses to see where (or if) the ganesha package installs that script and add a drop-in to override the ExecStart with wherever that script actually is.
17:06 MadPsy I couldn't find the script anywhere
17:06 MadPsy *find /)
17:07 MadPsy guess I could list the package contents
17:08 JoeJulian Right. I just don't know the tools for that distro. I know it's pkg-something
17:08 JoeJulian Create an issue at https://github.com/nfs-ganesha/nfs-ganesha/issues
17:08 glusterbot Title: Issues · nfs-ganesha/nfs-ganesha · GitHub (at github.com)
17:08 MadPsy will do ta
17:09 JoeJulian The source for that script is at https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/scripts/nfs-ganesha-config.sh
17:09 glusterbot Title: nfs-ganesha/nfs-ganesha-config.sh at next · nfs-ganesha/nfs-ganesha · GitHub (at github.com)
17:09 JoeJulian I see they also have /etc/sysconfig in there, another RHEL specific reference.
17:11 MadPsy yup
17:19 jsfrerot Rivitir: here is the info and stats for my volume: http://pastebin.com/Tp4vpRzU
17:19 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
17:24 jsfrerot @paste
17:24 glusterbot jsfrerot: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
17:24 jsfrerot here is the paste ising fpaste: http://paste.fedoraproject.org/467475/47802104/
17:24 glusterbot Title: #467475 • Fedora Project Pastebin (at paste.fedoraproject.org)
17:25 derjohn_mob joined #gluster
17:34 devyani7_ joined #gluster
17:43 nage joined #gluster
17:48 derjohn_mob joined #gluster
17:50 Gambit15 joined #gluster
17:50 Gambit15 joined #gluster
17:53 hackman joined #gluster
18:06 nage joined #gluster
18:26 kpease joined #gluster
19:21 derjohn_mob joined #gluster
19:26 squizzi_ joined #gluster
19:27 PaulCuzner joined #gluster
19:35 farhoriz_ joined #gluster
19:44 farhorizon joined #gluster
19:47 plarsen joined #gluster
19:53 ira joined #gluster
19:54 farhorizon joined #gluster
19:58 plarsen joined #gluster
20:01 mhulsman joined #gluster
20:13 PatNarciso @JoeJulian in your previous life -- what kind of performance were ya able to get out of your samba setup?  I'm topping out at ~40MB/s for a 1G file copy w samba on gluster fuse; full 100MB/s via cp on gluster fuse.  (I read your comment above, I'll experiement with oplocks...)
20:15 JoeJulian PatNarciso: I didn't benchmark. I did just enough to prevent people from emailing me with complaints.
20:17 PatNarciso right right... that's an important step.
20:46 ahino joined #gluster
20:57 derjohn_mob joined #gluster
21:00 mhulsman joined #gluster
21:00 farhoriz_ joined #gluster
21:14 MidlandTroy joined #gluster
21:21 panina joined #gluster
21:24 derjohn_mob joined #gluster
21:26 farhorizon joined #gluster
21:27 Jules-_ joined #gluster
21:41 farhorizon joined #gluster
21:48 PsionTheory joined #gluster
21:53 d0nn1e joined #gluster
22:00 shaunm joined #gluster
22:11 masber joined #gluster
22:14 masuberu joined #gluster
22:19 farhorizon joined #gluster
22:30 elastix joined #gluster
23:03 cliluw joined #gluster
23:13 Rivitir12x joined #gluster
23:16 plarsen joined #gluster
23:24 prth__ joined #gluster
23:38 Marbug_ joined #gluster
23:59 Alghost_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary