Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-08-03

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 http_GK1wmSU joined #gluster
00:02 http_GK1wmSU left #gluster
00:11 daMaestro joined #gluster
00:20 kjackal joined #gluster
00:28 baojg joined #gluster
00:31 cliluw joined #gluster
00:52 vbellur joined #gluster
00:53 vbellur joined #gluster
00:53 vbellur joined #gluster
00:54 vbellur joined #gluster
00:54 vbellur1 joined #gluster
00:55 vbellur joined #gluster
00:55 fassl__ joined #gluster
00:56 vbellur1 joined #gluster
01:09 MadPsy joined #gluster
01:09 MadPsy joined #gluster
01:13 vbellur joined #gluster
01:18 vbellur joined #gluster
01:29 baojg joined #gluster
01:41 vbellur joined #gluster
01:41 vbellur joined #gluster
01:49 guhcampos joined #gluster
01:51 ilbot3 joined #gluster
01:51 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:04 prasanth joined #gluster
02:08 baber joined #gluster
02:30 baojg joined #gluster
02:46 GlusterJoe joined #gluster
02:47 GlusterJoe hello all, have a few questions about geo-replication if anyone is willing to help
02:47 ppai joined #gluster
03:09 ankitr joined #gluster
03:18 masuberu joined #gluster
03:35 riyas joined #gluster
03:39 kramdoss_ joined #gluster
03:53 dominicpg joined #gluster
04:05 Shu6h3ndu joined #gluster
04:17 atinm joined #gluster
04:33 nbalacha joined #gluster
04:34 jiffin joined #gluster
04:35 baojg joined #gluster
04:38 karthik_us joined #gluster
04:42 ankitr joined #gluster
04:44 skumar joined #gluster
04:45 buvanesh_kumar joined #gluster
04:50 jkroon joined #gluster
04:59 rastar joined #gluster
05:04 skoduri joined #gluster
05:05 Saravanakmr joined #gluster
05:13 msvbhat joined #gluster
05:17 poornima joined #gluster
05:21 ndarshan joined #gluster
05:26 Electrometro joined #gluster
05:28 mb_ joined #gluster
05:28 PotatoGim joined #gluster
05:33 ankitr joined #gluster
05:34 hgowtham joined #gluster
05:38 baojg joined #gluster
05:45 mbukatov joined #gluster
05:49 ashiq joined #gluster
05:51 [diablo] joined #gluster
05:54 ankitr joined #gluster
05:57 prasanth joined #gluster
06:01 susant joined #gluster
06:02 Prasad joined #gluster
06:10 kotreshhr joined #gluster
06:11 Saravanakmr joined #gluster
06:13 msvbhat joined #gluster
06:13 MikeLupe joined #gluster
06:14 ankitr joined #gluster
06:15 aravindavk joined #gluster
06:15 kdhananjay joined #gluster
06:18 bowhunter joined #gluster
06:24 jtux joined #gluster
06:27 major joined #gluster
06:39 baojg joined #gluster
06:41 ankitr joined #gluster
06:56 msvbhat joined #gluster
07:01 arif-ali joined #gluster
07:01 jkroon joined #gluster
07:05 apandey joined #gluster
07:25 fsimonce joined #gluster
07:26 kramdoss_ joined #gluster
07:27 prasanth joined #gluster
07:40 baojg joined #gluster
07:44 arif-ali joined #gluster
07:45 ankitr joined #gluster
07:59 sona joined #gluster
08:10 rastar joined #gluster
08:12 ankitr joined #gluster
08:20 kramdoss_ joined #gluster
08:21 prasanth joined #gluster
08:27 ankitr joined #gluster
08:34 msvbhat joined #gluster
08:41 baojg joined #gluster
08:53 ankitr joined #gluster
08:55 MikeLupe joined #gluster
08:56 jiffin1 joined #gluster
09:12 jiffin1 joined #gluster
09:16 rdanter Hi all, I have a 2 node gluster with replication which has a really high CPU load during startup of an application which reads/writes quite a few files initially. CPU load is ~80% for several minutes before settling back down. I have used perf to capture what is going on and wonder if anyone can explain what is happening, perf call trace here: https://pastebin.com/PsYCWUG2
09:16 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
09:17 rdanter If we can get an idea of what is happening at this time we may be able to do something to avoid it or spread the load
09:19 rdanter Since it seems glusterbot does not like pastebin... https://paste.fedoraproject.org/paste/TR7HC685H2zaQK~wOALrzw
09:20 glusterbot Title: glusterfsd profile - Modern Paste (at paste.fedoraproject.org)
09:20 ThHirsch joined #gluster
09:21 msvbhat joined #gluster
09:22 MikeLupe joined #gluster
09:25 sona joined #gluster
09:42 MikeLupe joined #gluster
09:43 baojg joined #gluster
09:52 mb_ joined #gluster
09:56 ashiq joined #gluster
10:21 sona joined #gluster
10:44 baojg joined #gluster
10:56 MrAbaddon joined #gluster
11:15 h4rry joined #gluster
11:17 ppai joined #gluster
11:43 shyam joined #gluster
11:45 baojg joined #gluster
11:48 skoduri joined #gluster
11:49 baojg joined #gluster
11:52 baber joined #gluster
12:02 msvbhat joined #gluster
12:02 rafi joined #gluster
12:09 kotreshhr joined #gluster
12:13 Saravanakmr joined #gluster
12:21 Brs joined #gluster
12:39 kjackal joined #gluster
12:41 edong23 joined #gluster
12:48 ppai joined #gluster
12:55 msvbhat joined #gluster
12:55 p7mo joined #gluster
12:55 kramdoss_ joined #gluster
13:09 Intensity joined #gluster
13:12 rwheeler joined #gluster
13:13 jiffin1 joined #gluster
13:16 mbukatov joined #gluster
13:20 ic0n joined #gluster
13:20 vbellur joined #gluster
13:20 buvanesh_kumar joined #gluster
13:21 skylar joined #gluster
13:25 guhcampos joined #gluster
13:34 shyam joined #gluster
13:37 nh2 joined #gluster
13:38 nh2 given that gluster's SSL cert is global, is it impossible to mount 2 glusterfs that require different certs?
13:43 vbellur joined #gluster
13:45 vbellur joined #gluster
13:45 vbellur1 joined #gluster
13:54 prasanth joined #gluster
13:59 dominicpg joined #gluster
14:01 baojg joined #gluster
14:03 kdhananjay joined #gluster
14:03 vbellur joined #gluster
14:06 skoduri joined #gluster
14:21 msvbhat joined #gluster
14:25 jiffin joined #gluster
14:26 kdhananjay joined #gluster
14:33 baojg joined #gluster
14:42 jstrunk joined #gluster
14:56 farhorizon joined #gluster
15:01 shaunm joined #gluster
15:06 wushudoin joined #gluster
15:14 kramdoss_ joined #gluster
15:15 MrAbaddon joined #gluster
15:17 atinm joined #gluster
15:18 kpease joined #gluster
15:26 vbellur joined #gluster
15:28 marbu joined #gluster
15:31 rafi3 joined #gluster
15:35 rwheeler joined #gluster
15:36 msvbhat joined #gluster
15:41 nh2 joined #gluster
15:46 susant joined #gluster
15:54 aravindavk joined #gluster
15:57 kraynor5b joined #gluster
15:58 kraynor5b hey everyone, I'm still having big problems with my replicated volume
16:00 kraynor5b There was a large file that said "Possibly undergoing healing" for just over 24 hours, but the files were very similar in size and consistency
16:00 kraynor5b I stopped the volume and started it again, now I see the following
16:00 kraynor5b https://paste.fedoraproject.org/paste/nmNF7xCLtzcgQRvTSlkUnw
16:00 glusterbot Title: gluster v heal info - Modern Paste (at paste.fedoraproject.org)
16:00 snehring joined #gluster
16:04 kraynor5b what do I do at this point?
16:04 leni1 joined #gluster
16:04 kraynor5b I'm not sure how integral my files are now
16:04 prasanth joined #gluster
16:13 Shu6h3ndu joined #gluster
16:13 kramdoss_ joined #gluster
16:30 rastar joined #gluster
16:33 farhorizon joined #gluster
16:34 JoeJulian rdanter: readdirp. Huge directory?
16:37 JoeJulian kraynor5b: Is this replica 3?
16:37 kraynor5b the command was run on replica 3, correct
16:38 JoeJulian Well that doesn't look bad then. I see no split-brain.
16:38 kraynor5b it seems to have corrected itself following my restart of the volume and little freak out above ^^
16:38 JoeJulian You can see high cpu usage and very little data transfer on huge files.
16:38 JoeJulian It only transfers data if the hash match for a section fails.
16:39 rafi1 joined #gluster
16:40 rdanter JoeJulian: the total content is about 3GB spread over a few 10's of directories and files from 512b up to a few hundred MB, mostly the files are on the smaller side
16:41 JoeJulian Size of the file doesn't matter in a directory read, only the number of entries.
16:42 victori joined #gluster
16:42 rdanter there are about 1200 files + dirs in total
16:43 JoeJulian That doesn't sound all that large. I wonder if there's a lot of lookups of files that don't exist.
16:43 JoeJulian @php
16:43 glusterbot JoeJulian: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
16:43 glusterbot JoeJulian: --fopen-keep-cache
16:43 JoeJulian Try the "negative-timeout" mount option.
16:44 JoeJulian The "HIGH" is a number in seconds.
16:44 rdanter what sort of number constitutes high?
16:44 JoeJulian That's use-case specific.
16:45 rdanter ok, we can do some experiments, rough idea where to start??
16:45 JoeJulian 60?
16:45 JoeJulian 300?
16:45 JoeJulian 1 billion!!! <puts pinky to edge of mouth>
16:46 rdanter :)
16:46 rdanter ok, we shall have a play :)
16:46 JoeJulian Let me know what you find.
16:46 rdanter will do, thank you
16:48 farhorizon joined #gluster
16:57 jkroon joined #gluster
17:01 farhorizon joined #gluster
17:01 rwheeler joined #gluster
17:08 Humble joined #gluster
17:09 skoduri joined #gluster
17:17 h4rry joined #gluster
17:24 ankitr joined #gluster
17:27 jiffin joined #gluster
17:31 jiffin joined #gluster
17:35 farhorizon joined #gluster
17:42 jiffin joined #gluster
17:45 sona joined #gluster
17:46 kraynor5b joined #gluster
17:47 om2 joined #gluster
17:48 kraynor5b1 joined #gluster
18:05 farhorizon joined #gluster
18:20 plarsen joined #gluster
18:54 Jacob843 joined #gluster
18:57 masber joined #gluster
19:00 farhoriz_ joined #gluster
19:05 freephile joined #gluster
19:54 bowhunter joined #gluster
20:04 om2 joined #gluster
20:04 freephile new to gluster, setting up on AWS using Ansible, but that hangs on peer probe
20:05 freephile it seems to be a permissions issue because if I use the command line and
20:05 freephile gluster peer probe 10.0.50.165 (as meza-ansible, or centos) it will fail:
20:06 freephile ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Permission denied)
20:06 freephile ERROR: failed to open logfile /var/log/glusterfs/cli.log
20:06 freephile whereas if I use sudo on the command line, it will succeed
20:06 freephile sudo gluster peer probe 10.0.50.165
20:06 freephile peer probe: success. Host 10.0.50.165 port 24007 already in peer list
20:06 JoeJulian Gluster needs to run as root for a number of reasons, the least of which is access to the log files.
20:07 freephile With that info, I changed my ansible play to include become: true and become_user:root
20:07 freephile but it still hangs on the same issue
20:08 freephile admitedly this may be more of an ansible problem, but I'm just wondering what I'm doing wrong; or if anyone has suggestions
20:09 JoeJulian If it's hanging, my guess would be some expectation of user input (not sure why there would be). One sec where I find an apparently undocumented flag to try...
20:09 freephile fyi, here's my ansible role https://github.com/enterprisemediawiki/meza/blob/master/src/roles/gluster/tasks/main.yml
20:09 glusterbot Title: meza/main.yml at master · enterprisemediawiki/meza · GitHub (at github.com)
20:10 freephile actually, it doesn't 'hang'... it fails with fatal: [10.0.50.161]: FAILED! => {"changed": false, "failed": true, "msg": "failed to probe peer 10.0.50.165 on 10.0.50.161"}
20:12 JoeJulian Any clues in /var/log/glusterfs/cli.log or glusterd.log on 161?
20:13 fcami joined #gluster
20:17 freephile @joejulian nothing stands out.  I'm running it again.
20:20 freephile [2017-08-03 20:18:17.076960] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
20:20 freephile [2017-08-03 20:18:17.085661] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
20:20 freephile [2017-08-03 20:18:17.093915] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.0.50.161 24007
20:20 freephile [2017-08-03 20:18:17.102900] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.0.50.165 24007
20:20 freephile all normal
20:21 JoeJulian Well then I would have to presume the msg comes from the ansible module.
20:22 vbellur joined #gluster
20:23 freephile https://pastebin.com/prLJnngw cli.log normal too (i think)
20:23 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
20:24 JoeJulian Aha
20:24 JoeJulian "already in peer list" != "peer in cluster"
20:24 JoeJulian They should have written the module around the xml output.
20:26 JoeJulian So it's an ansible bug. https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/gluster_volume.py#L305
20:26 glusterbot Title: ansible/gluster_volume.py at devel · ansible/ansible · GitHub (at github.com)
20:36 freephile @joejulian: suggestion for a work-around? Actually, looking at the code you referenced, the option is for "bricks" and my ansible role specified "brick"; so I changed that and I'm running it again.
20:37 freephile (I agree that the module should have been written around xml output instead of string parsing and grepping)
20:39 freephile but no luck
20:41 JoeJulian If it was me, I'd just fix ansible and run with my own hacked version while I file an issue upstream and (if I have time) submit a patch.
20:41 JoeJulian I wouldn't have time though because I wouldn't do it unless I redid the whole thing to use the xml output. :)
20:42 JoeJulian Just like I did with salt.
20:46 freephile This role works on a monolithic environment, and I notice that the ansible module doesn't probe localhost
20:47 freephile but supposedly this role is supposed to work on a multi-node environment
20:48 freephile and this specific play is word-for-word the same as Jeff Geerling's role https://www.jeffgeerling.com/blog/simple-glusterfs-setup-ansible
20:48 glusterbot Title: Simple GlusterFS Setup with Ansible | Jeff Geerling (at www.jeffgeerling.com)
20:57 bowhunter joined #gluster
21:03 Humble joined #gluster
21:19 h4rry joined #gluster
21:20 vbellur joined #gluster
21:21 vbellur joined #gluster
21:21 shyam joined #gluster
21:21 vbellur joined #gluster
21:22 vbellur joined #gluster
21:23 vbellur joined #gluster
21:23 vbellur joined #gluster
21:24 vbellur joined #gluster
21:24 farhorizon joined #gluster
21:42 vbellur joined #gluster
21:42 vbellur joined #gluster
21:43 vbellur joined #gluster
21:44 vbellur joined #gluster
21:45 vbellur joined #gluster
21:49 wushudoin joined #gluster
21:50 farhorizon joined #gluster
21:52 vbellur joined #gluster
21:57 vbellur joined #gluster
22:40 vbellur joined #gluster
23:07 loadtheacc joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary