Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-12-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 NuxRo joined #gluster
00:08 zhangjn joined #gluster
00:09 zhangjn joined #gluster
00:09 dlambrig joined #gluster
00:17 EinstCrazy joined #gluster
00:21 delhage joined #gluster
00:37 harish joined #gluster
00:42 delhage joined #gluster
00:50 al joined #gluster
00:52 theron joined #gluster
00:56 al joined #gluster
00:59 EinstCrazy joined #gluster
00:59 zhangjn joined #gluster
00:59 al joined #gluster
01:06 Iouns joined #gluster
01:14 JoeJulian lapy, don't follow any gluster 3.1 documentation.
01:15 JoeJulian also, read-local is the default. Look at "gluster volume set help" for other options.
01:15 nangthang joined #gluster
01:15 JoeJulian oops, missed that. You found them.
01:16 JoeJulian 37 seconds? Are you replicating across the globe?
01:25 lpabon joined #gluster
01:39 zhangjn joined #gluster
01:46 haomaiwa_ joined #gluster
01:47 Lee1092 joined #gluster
01:53 julim joined #gluster
02:01 haomaiwa_ joined #gluster
02:12 hagarth_ joined #gluster
02:16 harish joined #gluster
02:25 nangthang joined #gluster
02:36 Humble joined #gluster
02:49 sakshi joined #gluster
02:49 calavera joined #gluster
02:51 calavera_ joined #gluster
03:01 Peppaq joined #gluster
03:06 calavera joined #gluster
03:20 zhangjn joined #gluster
03:37 aravindavk joined #gluster
03:42 kdhananjay joined #gluster
03:44 nbalacha joined #gluster
03:49 atinm joined #gluster
03:54 kanagaraj joined #gluster
03:55 ppai joined #gluster
04:06 dgbaley joined #gluster
04:08 itisravi joined #gluster
04:19 arcolife joined #gluster
04:20 vmallika joined #gluster
04:21 bharata-rao joined #gluster
04:23 arcolife joined #gluster
04:23 beeradb_ joined #gluster
04:25 ramteid joined #gluster
04:27 skoduri joined #gluster
04:29 RameshN joined #gluster
04:30 jiffin joined #gluster
04:31 nehar joined #gluster
04:39 archit_ joined #gluster
04:40 hgowtham joined #gluster
04:40 Manikandan joined #gluster
04:41 vmallika joined #gluster
04:42 ashiq joined #gluster
04:48 Guest54275 joined #gluster
04:49 ndarshan joined #gluster
04:50 yuri_ joined #gluster
04:51 empyrean Hey all, I have a problem... I'm not sure how to describe it... but here's the single line of error, out of a multitude of them `0-shared-dht: Failed to get hashed subvol for <gfid:00000000-0000-0000-0000-000000000000>`.
04:51 empyrean it seems like the file has been renamed, removed, or something.
04:55 yuri_ more to add to @empyrean, that error occurs only on one of the nodes in cluster of two
04:55 gem joined #gluster
04:55 RameshN joined #gluster
05:01 hgowtham joined #gluster
05:02 vimal joined #gluster
05:06 deepakcs joined #gluster
05:08 haomaiwang joined #gluster
05:11 kotreshhr joined #gluster
05:11 haomai___ joined #gluster
05:13 Bhaskarakiran joined #gluster
05:19 skoduri joined #gluster
05:19 pppp joined #gluster
05:27 Apeksha joined #gluster
05:29 JoeJulian empyrean: Are you writing files directly to the bricks?
05:29 empyrean erm nope not that i know of
05:29 JoeJulian ... also... is anything not actually working?
05:29 empyrean the standard operation is to write through nfs
05:31 haomaiwa_ joined #gluster
05:32 zhangjn joined #gluster
05:32 empyrean apart from nfs.log bloating up to hundreds of gigabytes, which is a hideous problem, i would think things are working
05:33 rafi joined #gluster
05:34 empyrean we're getting "GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) " errors which seems to be pointing to nfs
05:34 empyrean not gluster per se
05:35 JoeJulian that error line above, can you paste the one complete line, please? That string exists in several places.
05:36 raghu joined #gluster
05:36 empyrean sure
05:39 empyrean https://bpaste.net/show/5e7346d39fe5 < here's the line above plus some of the preceding onees, they seem to get printed out as a block.
05:39 glusterbot Title: show at bpaste (at bpaste.net)
05:40 JoeJulian Oh, wow! 3.3.1?
05:41 empyrean ...yeah..
05:44 JoeJulian Anything at all preceeding that first error?
05:46 gildub joined #gluster
05:49 JoeJulian Well that's interesting. I have no idea where that structure gets loaded. nor why the parent parameter would be 0.
05:50 JoeJulian My guess would be that a symlink under .glusterfs on some brick isn't pointing to the correct parent.
05:50 empyrean right
05:50 empyrean well, https://bpaste.net/show/53e56a8162f5
05:50 glusterbot Title: show at bpaste (at bpaste.net)
05:50 JoeJulian There's another bug I remember where a brick would have a directory instead of a symlink for .glusterfs/00/00/00000000-0​000-0000-0000-000000000000
05:51 empyrean hm
05:51 empyrean symlink you say?
05:52 JoeJulian should be a symlink
05:52 spalai joined #gluster
05:52 newdave joined #gluster
05:53 empyrean file ffff19c9-dbf3-46bb-9322-096eb449509e
05:53 empyrean ffff19c9-dbf3-46bb-9322-096eb449509e: symbolic link in a loop
05:53 empyrean something like that?
05:55 empyrean so, i probably should search for files which if are symlinks, stat them, and if are directories, scream
05:56 JoeJulian should not be in a loop. symlinks under .glusterfs should point to directories.
05:56 JoeJulian lrwxrwxrwx 1 root root 63 Oct 21 16:11 .glusterfs/85/a7/85a769b3-8​d3b-4122-a56d-a91744a7d8c5 -> ../../c2/2b/c22b8668-691d-4a0​b-8f62-3f02bda369ef/Atlantic Starr
05:56 JoeJulian for instance
05:56 empyrean so anything but a loop? #steeplearningcurve
05:57 JoeJulian where c22b8668... is the parent directory's gfid to the 'Atlantic Starr' directory.
05:57 anil joined #gluster
05:57 JoeJulian You know about how gluster uses ,,(extended attributes) I hope.
05:57 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://pl.atyp.us/hekafs.org/index.php/​2011/04/glusterfs-extended-attributes/
05:58 JoeJulian Once you get that sussed, consider upgrading. That hasn't happened in years.
05:58 rjoseph joined #gluster
05:59 JoeJulian Good luck. I'm heading to bed.
06:00 empyrean sure, thanks a lot for your help!!
06:08 overclk_ joined #gluster
06:14 rjoseph joined #gluster
06:16 calavera joined #gluster
06:19 Norky joined #gluster
06:20 Manikandan joined #gluster
06:22 kotreshhr joined #gluster
06:22 DV joined #gluster
06:27 atalur joined #gluster
06:30 Humble joined #gluster
06:33 Saravana_ joined #gluster
06:34 poornimag joined #gluster
06:54 ramky joined #gluster
06:54 atalur joined #gluster
06:57 zhangjn joined #gluster
07:18 ashiq joined #gluster
07:18 gem joined #gluster
07:19 jtux joined #gluster
07:20 purpleidea joined #gluster
07:28 jtux joined #gluster
07:28 spalai joined #gluster
07:30 kshlm joined #gluster
07:35 mobaer joined #gluster
07:43 Manikandan joined #gluster
07:44 rjoseph joined #gluster
07:46 kotreshhr joined #gluster
07:46 zhangjn joined #gluster
07:55 hgowtham joined #gluster
07:57 sripathi joined #gluster
08:06 mhulsman joined #gluster
08:07 atalur joined #gluster
08:22 RitBit joined #gluster
08:27 [Enrico] joined #gluster
08:28 harish_ joined #gluster
08:28 zhangjn joined #gluster
08:33 gem joined #gluster
08:33 deniszh joined #gluster
08:33 ashiq joined #gluster
08:37 ppai joined #gluster
08:38 sripathi joined #gluster
08:46 morse joined #gluster
08:48 ivan_rossi joined #gluster
08:48 fsimonce joined #gluster
08:53 afics joined #gluster
08:55 zhangjn joined #gluster
08:55 hgowtham joined #gluster
08:58 ctria joined #gluster
09:00 morse joined #gluster
09:07 gem joined #gluster
09:08 uebera|| joined #gluster
09:08 Norky joined #gluster
09:09 Slashman joined #gluster
09:10 deepakcs joined #gluster
09:14 Bhaskarakiran joined #gluster
09:14 Saravana_ joined #gluster
09:16 arcolife joined #gluster
09:20 atalur joined #gluster
09:22 kotreshhr joined #gluster
09:23 Manikandan joined #gluster
09:28 Bhaskarakiran joined #gluster
09:32 glafouille joined #gluster
09:36 overclk joined #gluster
09:37 Norky joined #gluster
09:40 F2Knight joined #gluster
09:41 atalur joined #gluster
09:48 glafouille joined #gluster
09:50 archit_ joined #gluster
09:52 ashiq joined #gluster
09:58 bluenemo joined #gluster
10:01 Manikandan joined #gluster
10:03 spalai joined #gluster
10:05 ppai joined #gluster
10:07 newdave joined #gluster
10:07 mhulsman1 joined #gluster
10:09 Larsen_ joined #gluster
10:11 zhangjn joined #gluster
10:22 zhangjn joined #gluster
10:25 jrm16020 joined #gluster
10:27 zhangjn joined #gluster
10:27 kotreshhr joined #gluster
10:28 Saravana_ joined #gluster
10:28 mlncn joined #gluster
10:34 blu_ joined #gluster
10:44 jwd joined #gluster
10:46 newdave joined #gluster
10:47 paratai_ joined #gluster
10:48 mdavidson joined #gluster
10:48 xavih joined #gluster
10:55 frakt joined #gluster
10:56 rich0dify joined #gluster
11:03 zhangjn joined #gluster
11:05 jrm16020 joined #gluster
11:10 mhulsman joined #gluster
11:13 kkeithley1 joined #gluster
11:20 spalai joined #gluster
11:32 zhangjn joined #gluster
11:42 itisravi joined #gluster
11:44 poornimag joined #gluster
11:47 rwheeler joined #gluster
11:48 kotreshhr joined #gluster
11:48 jrm16020 joined #gluster
11:49 shyam joined #gluster
11:50 ccoffey_ joined #gluster
11:51 chirino joined #gluster
11:54 Humble joined #gluster
12:00 Manikandan joined #gluster
12:02 EinstCrazy joined #gluster
12:02 primusinterpares joined #gluster
12:04 ccoffey_ Is there anyway I can identify which brick is which? They were mislabelled and I'm not confident the fstab is correct. Is there a way to relate the xattr to the definitions with /var/lig/glusterd ?
12:09 mhulsman1 joined #gluster
12:11 chirino joined #gluster
12:11 spalai left #gluster
12:12 shyam joined #gluster
12:13 rwheeler joined #gluster
12:16 nehar joined #gluster
12:19 wiza joined #gluster
12:20 Bhaskarakiran joined #gluster
12:21 spalai joined #gluster
12:22 wiza Got little problem with 3.7.5 which seems to be hoarding memory compared to old versions (on CentOS 7). It lead to split-brain, solved it mostly except cache directory. That I renamed, recreated and now trying to remove the old one which has split-brain in it. Is it correct way to just rm -rf it on both nodes?
12:27 ccoffey_ @wiza, is this much use? https://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/ ?
12:28 jmarley joined #gluster
12:28 devilspgd joined #gluster
12:28 ira joined #gluster
12:29 EinstCrazy joined #gluster
12:32 wiza ccoffey_: might be useful with the files, once I get to those
12:32 wiza thanks
12:34 kotreshhr left #gluster
12:34 EinstCra_ joined #gluster
12:35 wiza shit, something is really wrong, doing source-brick healing: Lookup failed on /service/cache_old/sendgrid_ips:Input/output error
12:36 wiza and I tried to use both nodes as source
12:36 plarsen joined #gluster
12:37 zhangjn joined #gluster
12:41 RedW joined #gluster
12:45 SOLDIERz joined #gluster
12:47 atalur joined #gluster
12:48 d-fence joined #gluster
12:48 MessedUpHare joined #gluster
12:54 kdhananjay joined #gluster
12:54 pppp joined #gluster
12:58 kdhananjay joined #gluster
13:02 d0nn1e joined #gluster
13:09 rafi Manikandan++
13:09 glusterbot rafi: Manikandan's karma is now 3
13:10 Manikandan rafi, :-)
13:10 Humble joined #gluster
13:10 mhulsman1 joined #gluster
13:12 ndarshan joined #gluster
13:12 lorddoskias joined #gluster
13:12 lorddoskias hello
13:13 glusterbot lorddoskias: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
13:13 lorddoskias why is gluster using trusted attributes?
13:14 Philambdo left #gluster
13:17 rjoseph joined #gluster
13:19 shubhendu joined #gluster
13:20 theron joined #gluster
13:21 wiza ok, removed directory from both bricks, solved that. Now I have few files with seems to only have gfids...
13:21 atinm joined #gluster
13:28 wiza and I'm happy to just delete those files
13:35 shaunm joined #gluster
13:40 B21956 joined #gluster
13:42 wiza ok, sorted that one out too
13:43 lorddoskias left #gluster
13:45 unclemarc joined #gluster
13:46 wiza now I have one split-brain file and on source-brick split-brain heal it says Lookup failed on ... Input/output error
13:48 jiffin joined #gluster
13:48 atinm joined #gluster
13:58 wiza now I only have one directory in split-brain
13:59 wiza with no files in it
14:02 wiza nvm, it seemed that that was something it could fix itself :)
14:02 d0nn1e joined #gluster
14:06 wiza btw. anyone else hitting higher than normal memory usage on 3.7.6 compared to earlier versions?
14:06 mobaer joined #gluster
14:06 wiza said earlier that I'm using 3.7.5 but nope, .6
14:08 bennyturns joined #gluster
14:10 mhulsman joined #gluster
14:21 spalai left #gluster
14:28 Park joined #gluster
14:30 arcolife joined #gluster
14:31 hamiller joined #gluster
14:35 haomaiwang joined #gluster
14:40 mhulsman1 joined #gluster
14:44 overclk joined #gluster
15:08 skoduri joined #gluster
15:10 Ericle_ joined #gluster
15:13 kotreshhr joined #gluster
15:13 harish_ joined #gluster
15:20 maserati joined #gluster
15:22 papamoose joined #gluster
15:29 tapoxi joined #gluster
15:32 tapoxi hi everyone, now that i've started storing a ton of data on gluster its decided to slow down, i have 1 brick replicated across 4 azs in amazon us-east, suggestions on how to make it faster?
15:35 theron joined #gluster
15:35 adama joined #gluster
16:00 ndevos tapoxi: that really depends on how you access your data, i.e. if you do a lot of directory listings, things will start to slow down if the number of files in a directory increase
16:01 tapoxi ndevos: I noticed the issue when I tried to delete a few directories
16:02 jbrooks joined #gluster
16:02 ndevos tapoxi: deleting a directory would involve crawling the directory tree and delete each file/dir one by one, it'll take more time when the directories grow
16:03 hamiller joined #gluster
16:03 [o__o] joined #gluster
16:04 tapoxi ndevos: what's the bottleneck there, network bandwidth between gluster nodes?
16:05 ndevos tapoxi: directory listing is an expensive operation, the results from multiple bricks need to get merged into one that can get passed back to the client - or something like that
16:06 Park joined #gluster
16:06 tapoxi ndevos: ok, thanks. I guess it won't hit my actual use case then
16:06 corretico joined #gluster
16:07 ndevos tapoxi: I think shyam and others were looking into improving the performance of directoty listings
16:08 ndevos tapoxi: normal access/read/write should not do those indeed, most applications are smart enough to just open the file they need, without doing a directory listing
16:09 tapoxi ndevos: also, the .glusterfs directory should clean itself up as the links to the files it cares about are deleted, right?
16:09 F2Knight joined #gluster
16:09 ndevos tapoxi: yes, it should
16:10 tapoxi ndevos: great, thanks again
16:10 DV joined #gluster
16:12 ccoffey_ Can you identify which ext filesystem should be which brick ?
16:24 ndevos ccoffey_: I dont know, one of the dht developers might be able to answer that question, best bet is to send an email to gluster-users@gluster.org
16:25 wushudoin joined #gluster
16:27 ccoffey_ Thanks ndevos
16:28 chirino joined #gluster
16:29 muneerse2 joined #gluster
16:31 kshlm joined #gluster
16:33 ccoffey_ wait, just remembered the bricks are mirrors of another set so I can just compare the files form 1 set to anothr
16:33 hagarth_ joined #gluster
16:36 ndevos ccoffey_: yeah, that could work :)
16:37 bluenemo joined #gluster
16:42 ccoffey_ hmmm, I don't think it matters now, looks like all 5 bricks have everything in lost and found. Will just need to redo everythign on this host :(
16:44 ndevos ccoffey_: lost+found is only used by fsck, if things land there, something went really wrong
16:49 lapy win 3
16:50 EinstCrazy joined #gluster
16:50 [o__o] joined #gluster
16:51 jiffin joined #gluster
16:51 deepakcs joined #gluster
16:51 theron joined #gluster
16:57 ccoffey_ when something went wrong at ext level
16:57 ccoffey_ gluster wasn't at fault at all
16:58 Manikandan joined #gluster
16:59 tapoxi left #gluster
17:03 muneerse joined #gluster
17:09 muneerse2 joined #gluster
17:11 ivan_rossi left #gluster
17:17 calavera joined #gluster
17:25 bennyturns joined #gluster
17:27 rwheeler joined #gluster
17:28 mhulsman joined #gluster
17:28 atrius joined #gluster
17:33 jiffin joined #gluster
17:38 kotreshhr joined #gluster
17:38 kotreshhr left #gluster
17:52 EinstCrazy joined #gluster
17:55 jiffin1 joined #gluster
18:05 mobaer joined #gluster
18:06 muneerse joined #gluster
18:12 hagarth_ joined #gluster
18:15 [o__o] joined #gluster
18:16 dlambrig joined #gluster
18:24 amye- joined #gluster
18:28 Rapture joined #gluster
18:28 calavera joined #gluster
18:31 _feller joined #gluster
18:44 dlambrig joined #gluster
18:45 Park hi, anyone knows if the volume object created by libgfapi is thread safe?
18:46 JoeJulian From what I know of what makes things thread safe, it should be. But I'm no expert on that.
18:48 JoeJulian So, Park, about that 128k read on seek. Do you have the strace for the application that's using gluster and creating that read? I'd like to see if I can duplicate it in C (or python preferably).
18:51 Park JoeJulian, yes, how can I send the whole output to you?
18:53 Park And for why I asked if volume object created by libgfapi is thread safe,  I got segfault when using the same volume in multiple threads env,  the backtrace from the core:  https://pastebin.mozilla.org/8853531
18:53 glusterbot Title: Mozilla Pastebin - collaborative debugging tool (at pastebin.mozilla.org)
18:53 JoeJulian me@joejulian.name
18:54 JoeJulian file a bug report for that segfault and also ask on the gluster-devel mailing list.
18:54 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
18:56 cornfed78 joined #gluster
18:59 JonathanD joined #gluster
19:02 Park JoeJulian, re BZ#:  I will.   Not sure why I can't send out the mail yet,  but can I send the strace out to you by other way?  no idea about how to yet though, :-)
19:04 skylar joined #gluster
19:04 cliluw joined #gluster
19:05 OregonGunslinger joined #gluster
19:11 JoeJulian I didn't know your email was broken.
19:13 Park wait for a while, I'm uploading to a box.
19:16 cholcombe joined #gluster
19:25 cornfed78 joined #gluster
19:36 julim joined #gluster
19:41 klaxa joined #gluster
19:51 calavera joined #gluster
20:02 theron joined #gluster
20:03 ira joined #gluster
20:09 mobaer joined #gluster
20:29 maserati|work joined #gluster
20:31 maserasdssd joined #gluster
20:32 mhulsman joined #gluster
20:41 EinstCrazy joined #gluster
20:53 coredump joined #gluster
21:12 calavera joined #gluster
21:26 theron_ joined #gluster
21:37 onlyanegg joined #gluster
21:46 mlncn joined #gluster
21:47 gildub joined #gluster
22:01 skylar joined #gluster
22:40 EinstCrazy joined #gluster
22:46 mhulsman joined #gluster
23:26 delhage joined #gluster
23:51 Rapture joined #gluster
23:55 delhage joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary