Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-03-10

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 major I guess it is time to fire up this btrfs-snapshot tree onto some nodes and see what sort of demons it summons
00:18 kramdoss_ joined #gluster
00:27 mlg9000 joined #gluster
00:33 XpineX joined #gluster
01:00 baber joined #gluster
01:08 shdeng joined #gluster
01:22 kramdoss_ joined #gluster
02:05 daMaestro joined #gluster
02:06 vinurs joined #gluster
02:18 d0nn1e joined #gluster
02:28 derjohn_mob joined #gluster
02:40 Wizek_ joined #gluster
02:41 sanoj|afk joined #gluster
02:41 om2 joined #gluster
02:46 om2 joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:54 kramdoss_ joined #gluster
02:55 kraynor5b joined #gluster
03:03 Gambit15 joined #gluster
03:05 shdeng joined #gluster
03:06 nh2 joined #gluster
03:19 atinm joined #gluster
03:19 baber joined #gluster
03:20 nh21 joined #gluster
03:32 magrawal joined #gluster
03:37 niknakpaddywak joined #gluster
03:42 nbalacha joined #gluster
03:56 gyadav_ joined #gluster
03:56 xMopxShell joined #gluster
04:00 jerrcs_ joined #gluster
04:01 BitByteNybble110 joined #gluster
04:04 itisravi joined #gluster
04:11 BitByteNybble110 joined #gluster
04:13 susant joined #gluster
04:18 arpu joined #gluster
04:19 Shu6h3ndu joined #gluster
04:19 skumar joined #gluster
04:23 susant left #gluster
04:27 nbalacha joined #gluster
04:37 kdhananjay joined #gluster
04:44 papna joined #gluster
04:49 Jacob843 joined #gluster
04:51 nbalacha joined #gluster
04:54 ashiq joined #gluster
05:02 karthik_us joined #gluster
05:08 ankush joined #gluster
05:10 daMaestro joined #gluster
05:14 aravindavk joined #gluster
05:15 oajs joined #gluster
05:17 ndarshan joined #gluster
05:18 apandey joined #gluster
05:23 ankush joined #gluster
05:24 buvanesh_kumar joined #gluster
05:25 ppai joined #gluster
05:26 hgowtham joined #gluster
05:26 kotreshhr joined #gluster
05:27 apandey_ joined #gluster
05:38 rafi joined #gluster
05:40 jiffin joined #gluster
05:42 riyas joined #gluster
05:50 Saravanakmr joined #gluster
05:57 apandey joined #gluster
06:02 ashiq joined #gluster
06:08 susant joined #gluster
06:13 mhulsman joined #gluster
06:21 daMaestro joined #gluster
06:21 sanoj joined #gluster
06:25 sbulage joined #gluster
06:25 Prasad joined #gluster
06:25 rafi1 joined #gluster
06:27 susant joined #gluster
06:39 ankitr joined #gluster
06:39 atinm_ joined #gluster
06:42 sbulage joined #gluster
06:43 Philambdo joined #gluster
06:45 atinmu joined #gluster
06:49 sanoj joined #gluster
06:52 Klas StormTide: hmm, interesting, since I will try to do basically that next week or maybe today
06:56 Karan joined #gluster
06:58 rafi1 joined #gluster
06:59 msvbhat joined #gluster
07:06 kdhananjay1 joined #gluster
07:17 ankush joined #gluster
07:24 jtux joined #gluster
07:33 msvbhat joined #gluster
07:42 RameshN joined #gluster
07:43 anbehl joined #gluster
07:44 kdhananjay joined #gluster
07:50 msvbhat joined #gluster
07:50 kdhananjay joined #gluster
07:53 kdhananjay joined #gluster
08:03 kdhananjay joined #gluster
08:06 mbukatov joined #gluster
08:06 kdhananjay joined #gluster
08:13 Humble joined #gluster
08:19 kdhananjay joined #gluster
08:24 sanoj joined #gluster
08:33 kdhananjay joined #gluster
08:36 kdhananjay joined #gluster
08:37 derjohn_mob joined #gluster
08:38 nishanth joined #gluster
08:45 kdhananjay joined #gluster
08:48 sona joined #gluster
08:51 fsimonce joined #gluster
08:53 RameshN joined #gluster
08:56 k4n0 joined #gluster
08:59 Seth_Karlo joined #gluster
08:59 Seth_Karlo joined #gluster
09:04 ankitr joined #gluster
09:04 rafi1 joined #gluster
09:05 pulli joined #gluster
09:05 RameshN joined #gluster
09:08 armyriad joined #gluster
09:10 ankitr joined #gluster
09:11 DaKnOb joined #gluster
09:18 sona joined #gluster
09:18 Saravanakmr joined #gluster
09:18 MadPsy has anyone seen this with Gluster 3.8.8 / native NFS - 'rm: cannot remove '/blah': Directory not empty"'
09:18 vinurs joined #gluster
09:19 ppai joined #gluster
09:20 samppah_ iipee 94.199.61.151
09:21 ashiq joined #gluster
09:21 samppah_ bleh.. please disregard that :)
09:23 nbalacha MadPsy, is your dir empty?
09:24 nbalacha MadPsy, you can sometimes see this if the dir is not present on the hashed subvom
09:24 nbalacha *subvol
09:24 MadPsy nbalacha, nope but rm was called with '-r' and it only happens randomly
09:24 nbalacha MadPsy, racing dir fops in the same dir?
09:25 MadPsy the volume has this run about 1000 a day and maybe occurs twice or so but it's enough to cause a headache as it's an automated script
09:25 MadPsy nbalacha, sorry what to you mean :)
09:25 MadPsy do*
09:26 nbalacha MadPsy, if you are performing rm -rf on dir1, is there any fop happening inside that dir, from another client say, at the same time?
09:27 MadPsy nbalacha, hmmm that's possible I guess although the files are being read by a web server so I wouldn't expect it to hold the file open
09:27 nbalacha MadPsy, does repeating the rm -rf succeed?
09:27 jwd joined #gluster
09:28 MadPsy nbalacha, that's what I'm thinking of trying - I don't normally check the exit status of an 'rm' given i've never seen it fail before but thinking that might be worth trying
09:31 flying joined #gluster
09:45 RameshN joined #gluster
09:46 rjoseph joined #gluster
09:48 Ashutto joined #gluster
09:48 ankitr joined #gluster
09:48 Ashutto Hello
09:48 glusterbot Ashutto: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an
09:48 Ashutto I have a strange issue
09:48 cloph MadPsy: the obvious: sure the directory is empty (and doesn't have hidden files, like .foobar)?
09:49 Ashutto it seems that every now and then, files get duplicated on my gluster (distributed replicated with arbiter node). I found that the suspected duplication is due to files went on more bricks that they should (files appears on 5 or 6 nodes instead of 2 + 1 meta)
09:50 Ashutto i'm using gluster 3.10.0
09:50 MadPsy cloph, it won't be empty but that shouldn't matter - 'rm -r' handles that 99.9% of the time
09:51 MadPsy cloph, e.g. 'mkdir -p dir1/dir2 ; touch dir1/dir2/.test ; rm -r dir1'
09:51 nbalacha MadPsy, I think rm -rf * probably will not work the same way
09:52 cloph sure, but could be that some other process did something in between..
09:52 nbalacha Ashutto, can you provide the ls -l output  of such files on all the bricks they are on?
09:52 Ashutto sure
09:52 nbalacha Ashutto, I want to check if they are linkto files
09:53 Ashutto nope
09:53 Ashutto they are common files
09:53 nbalacha Ashutto, not sure if you know what linkto files are in gluster?
09:53 om2 joined #gluster
09:53 Ashutto do you mean the .glusterfs pointer?
09:54 nbalacha Ashutto, not quite
09:54 nbalacha linkto files are internal dht files that have the same name and gfid as the original
09:54 nbalacha but are 0 bytes
09:54 Ashutto ok, l'ill simply provide the ls output then, sorry :)
09:54 nbalacha ls -l please
09:54 nbalacha I need to see the permissions
09:55 rafi2 joined #gluster
09:55 Ashutto ah ok, i thought that the 0 bytes files were that one on the arbiter
09:56 Ashutto do you have a preferred nopaste?
09:57 nbalacha Ashutto, no.
09:59 shruti` joined #gluster
10:00 MadPsy nbalacha, cloph - FYI it looks suspiciously like this https://bugzilla.redhat.com/show_bug.cgi?id=960910 :)
10:00 glusterbot Bug 960910: high, high, ---, sabansal, CLOSED DEFERRED, "rm -rf" failed to remove directory complained "directory not empty" from fuse mount
10:00 nbalacha MadPsy, We fixed most of those issues
10:01 ashiq joined #gluster
10:01 nbalacha MadPsy, can you check if the rm -rf  still fails?
10:01 nbalacha MadPsy, actually before you try that, can you check if the dir actually is empty?
10:02 Ashutto joined #gluster
10:02 MadPsy nbalacha, the directory is gone now - it's a web app so the errors I get back are actually from PHP and have been fixed manually (it breaks the website when it happens) so I guess the answer is 'yes it can be deleted'
10:03 Ashutto nbalacha, https://nopaste.me/view/a2fec593
10:03 glusterbot Title: Untitled - Nopaste.me (at nopaste.me)
10:04 nbalacha MadPsy, so you see any errors in the logs?
10:04 nbalacha and by manually, was it just another rm -rf ?
10:04 MadPsy nbalacha, gluster's logss ?
10:04 nbalacha Ashutto, ---------T 2 root root   0 Mar  9 17:11 homes03/safe/brick0/corsera/baseroot/w​ebroot/includes2013/_variables-dev.rb
10:04 glusterbot nbalacha: -------'s karma is now -10
10:04 nbalacha this is a linkto file
10:04 MadPsy nbalacha, yeah another rm would have fixed it, no other magic involved
10:05 nbalacha MadPsy, yes, specifically the gluster NFS server used to mount the volume
10:05 Ashutto nbalacha, ok. what does it mean to me? (please feel free to link to any documentation that you feel appropriate)
10:05 nbalacha MadPsy, then it is highly unlikely to be the BZ you pasted above
10:06 MadPsy nbalacha, kk - I presume that remained broken indefinitely
10:06 nbalacha MadPsy, yes. A second rm -rf would not have fixed that
10:06 MadPsy kk
10:06 nbalacha Ashutto, https://joejulian.name/blog​/dht-misses-are-expensive/
10:06 glusterbot Title: DHT misses are expensive (at joejulian.name)
10:07 nbalacha Ashutto, they usually show up if your file has been renamed
10:08 Ashutto it is possible...but they remain there undefinetly? i was unable to rm -rf that directory as that files were undeletable...
10:08 nbalacha Ashutto, they should not
10:08 nbalacha rm -rf should delete them
10:09 nbalacha how many are there
10:09 nbalacha also, do you see 2 entries for the same file when you run rm -rf?
10:09 Ashutto at the moment 1.2gb of occupied space composed of small files ( ~ 2kb)
10:09 nbalacha sorry - ls
10:09 Ashutto yes
10:09 Ashutto you can see that in the nopaste
10:09 nbalacha that should not happen either
10:09 nbalacha can you please check the xattrs on those files?
10:09 Ashutto sure
10:10 Ashutto every brick or should one suffice ?
10:10 nbalacha every brick
10:12 Ashutto nbalacha, https://nopaste.me/view/b7dbe87c
10:12 glusterbot Title: file xattr - Nopaste.me (at nopaste.me)
10:14 nbalacha Ashutto, the xattrs look ok to me
10:14 Ashutto i'm not an expert, but they look "clean" to me too
10:17 Ashutto how whould you fix a situation like this? is there a way to achieve a subfolder rebalance (my volume is a 1.5 T of really small files... a full rebalance would take forever)
10:17 nbalacha Ashutto, what are the problems you are currently facing
10:17 nbalacha having these linkto files on the bricks should not cause issues
10:17 nbalacha and what version of gluster are you running
10:17 Ashutto at the moment i'm unable to delete a folder which contains those files
10:18 Ashutto 3.10.0
10:18 nbalacha Ashutto, what is the error you get?
10:18 Ashutto "directory is not empty"
10:19 Ashutto i don't know if that causes other issues in the long run
10:19 nbalacha Ashutto, if you run ls -l on the mountpoint in the dir you are trying to delete, what does it return?
10:19 Ashutto both files
10:20 Ashutto it seems rm (rm -f, rm -rf) is unable to delete those files
10:20 msvbhat joined #gluster
10:20 nbalacha only 2 files?
10:20 Ashutto all the duplicated files
10:21 Ashutto (there are more dupes)
10:21 nbalacha how many in total?
10:21 MrAbaddon joined #gluster
10:21 Ashutto do you need subfolders too?
10:21 nbalacha if there are subfolders something else is wrong
10:22 nbalacha itisravi, you there?
10:22 MadPsy nbalacha, I don't see anything at the time the 'rm' failed but here you go none the less: https://paste.ubuntu.com/24150871/
10:22 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
10:22 Ashutto 2 files in the directory (2 + 2)
10:24 nbalacha MadPsy, those are AFR errors. I'm going to ask itisravi to take a look
10:24 nbalacha itisravi, there?
10:24 MadPsy ah balls - that's the wrong PHP log (that's another example) same error, different timestamp, should have been 21:13:55 :)
10:27 nbalacha ok, is it the same log messages?
10:27 nbalacha I'm assuming the gfid is diff?
10:28 cloph https://bugzilla.redhat.co​m/show_bug.cgi?id=1431081 - would be great if someone could poke the right people about this one - it's a pain in the butt for us
10:28 glusterbot Bug 1431081: high, unspecified, ---, bugs, NEW , symlinks trigger faulty geo-replication state (rsnapshot usecase)
10:30 MadPsy nbalacha, yeah diff gfid - i'll just repaste so it looks better: https://paste.ubuntu.com/24150901/
10:30 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
10:30 nbalacha MadPsy, can you send this info to the gluster-users mailing list>
10:31 MadPsy nbalacha, sure thing
10:31 nbalacha One of the AFR folks can take a look at it
10:31 nbalacha I don't know that component, sorry
10:31 MadPsy nbalacha, that's fine, thanks
10:32 jkroon joined #gluster
10:32 nbalacha Ashutto, can you send an email to gluster-users?
10:32 nbalacha I think we will need more ppl to look into it
10:32 nbalacha also, if you can send the gluster logs, we can see if there are any error messages there
10:34 Ashutto nbalacha, ok...last email i sent got ignored btw :(
10:36 nbalacha Ashutto, sorry about that. Can you let me know the subject?
10:36 itisravi nbalacha: yup
10:36 Ashutto ignore that, it referred a problem we solved
10:36 Ashutto i'll prepare a new email with that
10:36 Ashutto what logs to you need? (they are quite big...)
10:37 nbalacha are you using fuse
10:37 Ashutto fuse on the writing nodes, ganesha-nfs on the reading one
10:37 nbalacha ah ok
10:37 itisravi MadPsy: Is this replica 2?
10:37 Ashutto ganesha doesn't expose the double file error
10:37 nbalacha Ashutto, which node are you running the rm -rf from?
10:38 Ashutto fuse
10:38 Ashutto nfs is exported read-only
10:38 nbalacha Ashutto, can you run an rm -rf on the dir on which it is failing and send the logs for that op?
10:38 nbalacha this would be the fuse mount logs
10:38 Ashutto btw...i read that nfs should not be used for writing... is that right?
10:39 Ashutto (i mean, gluster nfs/ganesha)
10:39 nbalacha Ashutto, I will ask the ganesha folks to reply to that
10:39 Ashutto Thanks
10:39 nbalacha jiffin, are you there?
10:40 jiffin nbalacha: yus
10:41 nbalacha jiffin, Can you reply to Ashutto ?
10:42 nbalacha on a question on ganesha
10:42 itisravi nbalacha: Those are gfid split-brain errors. If a dir has files in gfid split-brain, the directory cannot be `rm -rf`d.
10:42 jiffin Ashutto: if u are using fuse mount , then it is not recommend to use nfsclients via ganesha
10:43 jiffin Ashutto: IMO if ur application does not worry about locks then it is okay to fuse and ganesha parallely
10:43 Ashutto jiffin, why? we achieve a much higher throughput using nfs than fuse (we have our documentroot on gluster, we need bitrate)
10:43 Ashutto ok
10:44 Ashutto jiffin, we use ganesha to export nfs in RO mode, locks are negligible in our setup (web server)
10:44 jiffin Ashutto: okay
10:45 Ashutto may you point me to a document that says not to use nfs in RW and fuse, particularly dealing about locks ?
10:46 jiffin Ashutto: nope, our ganesha and gluster integration doc is bit outdated
10:46 MadPsy itisravi, yup replica 2
10:46 jiffin Ashutto: I am planning to update ASAP
10:46 Ashutto jiffin, damn... i need to source that information
10:47 jiffin reference doc http://gluster.readthedocs.io/en/latest/
10:47 glusterbot Title: Gluster Docs (at gluster.readthedocs.io)
10:47 Ashutto or my boss is going to use nfs in rw
10:47 jiffin Ashutto: http://gluster.readthedocs.io/en/latest/Administra​tor%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
10:47 glusterbot Title: Configuring NFS-Ganesha server - Gluster Docs (at gluster.readthedocs.io)
10:47 chawlanikhil24 joined #gluster
10:48 RameshN joined #gluster
10:48 itisravi MadPsy: right. so like I was telling nbalacha, rmdir could not have succeded on a directory just because there was a second attempt. The gfid split-brain need to have been resolved for that to happen.
10:48 itisravi s/on a directory/ on that directory.
10:49 nbalacha MadPsy, how do you resolve this when it happens?
10:49 MadPsy itisravi, is that auto fixed though? I've not done anything manually
10:50 itisravi MadPsy: no gfid split-brain requires manual intervention to resolve. There is work upstream to do it automatically though. karthik_us is working on it.
10:50 itisravi MadPsy: wonder if a sys-admin did anything?
10:50 MadPsy itisravi, if I look in that dir now - there's no directory with that name - I'm the only one with access (apart from PHP itself)
10:51 itisravi Interesting.
10:52 msvbhat joined #gluster
10:54 skumar joined #gluster
11:01 nbalacha itisravi, can you also take a look at Ashutto's problem?
11:03 itisravi nbalacha: sure
11:03 rafi2 joined #gluster
11:16 skumar joined #gluster
11:19 skumar_ joined #gluster
11:26 nh2 joined #gluster
11:30 itisravi nbalacha: Interestingly, both the actual file and the link-to file are in the same brick for 2 bricks.
11:36 nbalacha itisravi, ?
11:38 nbalacha how is that possible
11:39 itisravi nbalacha: that is what https://nopaste.me/view/a2fec593# is suggesting :)
11:39 glusterbot Title: Untitled - Nopaste.me (at nopaste.me)
11:39 itisravi nbalacha: oh no I'm sorry
11:39 itisravi nbalacha: the paths are different.
11:39 nbalacha ok
11:40 nbalacha I am more concerned abt why the subdirs are not getting deleted
11:41 aravindavk joined #gluster
11:44 sbulage joined #gluster
11:46 Seth_Karlo joined #gluster
11:49 itisravi Ashutto: Does the getfattr output of the parent dir . i.e. "includes2013" show any non zero value for any of the 'trusted.afr' attributes?
12:20 MrAbaddon joined #gluster
12:21 rafi1 joined #gluster
12:30 nh2 joined #gluster
12:38 sbulage joined #gluster
12:41 d0nn1e joined #gluster
12:52 kotreshhr left #gluster
12:54 Ashutto itisravi, no such attribute is set in any brick
13:09 msvbhat joined #gluster
13:15 RameshN joined #gluster
13:16 kpease_ joined #gluster
13:23 RameshN joined #gluster
13:23 ashiq joined #gluster
13:30 kramdoss_ joined #gluster
13:32 MrAbaddon joined #gluster
13:34 ashka hi, sorry for a question about an old version (3.5.2). I have an issue on a distribute gluster, bricks will disconnect in a chain and create "transport endpoint is not connected" errors on clients, with errors like 'readv on brick1-ip:49152 failed (Connection reset by peer)'. If I follow the chain (in brick1's logs brick2-ip is referenced with the same error, and so on), I'll end up on a brick that has no error in the log. Is this a network
13:34 ashka issue? the network I am on should be very reliable
13:41 neofob joined #gluster
13:42 ira joined #gluster
13:42 neofob hi, just wonder if anyone has used glusterfs to store graphite whisper data
13:43 shyam joined #gluster
13:43 baber joined #gluster
13:59 unclemarc joined #gluster
14:03 derjohn_mob joined #gluster
14:05 jiffin joined #gluster
14:07 Philambdo joined #gluster
14:17 plarsen joined #gluster
14:19 arif-ali joined #gluster
14:19 uebera|| joined #gluster
14:21 shyam joined #gluster
14:21 lolka joined #gluster
14:29 skylar joined #gluster
14:32 sona joined #gluster
14:33 nbalacha joined #gluster
14:38 aardbolreiziger joined #gluster
14:41 bwerthmann joined #gluster
14:45 ashiq joined #gluster
14:45 oajs joined #gluster
14:45 aravindavk joined #gluster
14:51 saali joined #gluster
14:53 atinm_ joined #gluster
15:01 shyam joined #gluster
15:05 fre joined #gluster
15:09 oajs joined #gluster
15:23 buvanesh_kumar joined #gluster
15:38 hybrid512 joined #gluster
15:41 Asako joined #gluster
15:42 Asako Hello.  What is the procedure for adding a new slave on a geo replicated volume?
15:42 Asako The docs don't seem to mention how to add a new slave
15:42 cloph you create a new connection/session, it is not different to adding the first slave
15:43 Asako ok.  Do the slave volume names have to be different?
15:43 cloph no, volume names can be the save, but obviously you cannot use the very same volume as a slave for different master volumes.
15:44 cloph but as long they are different volumes, no problem.
15:44 farhorizon joined #gluster
15:44 Asako and it's a replicated volume, do I need to run geo-replication commands on both masters?
15:45 Asako it's a new volume on a new host.  I just can't get the replication session to start right.
15:45 Asako [resource(/var/mnt/gluster/brick2):238:logerr] Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory
15:48 MrAbaddon joined #gluster
15:49 atinm_ joined #gluster
16:02 wushudoin joined #gluster
16:06 baber joined #gluster
16:10 Philambdo1 joined #gluster
16:10 Asako Starting geo-replication session between gv0 & gluster-srv4::slavevol has been successful
16:10 Asako but when I check the status it says Faulty
16:12 Philambdo joined #gluster
16:16 kkeithley StormTide, et al.: I've updated the debian pkgs on download.gluster.org and the ubuntu pkgs in the PPA. Should address the issues with georep and events. Please give them a try and let me know if you see any issues.
16:18 Asako hmm, apparently there's a step missing in the docs
16:18 Asako ran gluster volume geo-replication gv0 gluster-srv4::slavevol config remote_gsyncd /usr/libexec/glusterfs/gsyncd and now it works
16:19 nishanth joined #gluster
16:20 Asako and now it's syncing
16:20 Asako I really shouldn't have to dig through mailing list archives to make stuff work...
16:35 shyam1 joined #gluster
16:47 MrAbaddon joined #gluster
16:53 shyam joined #gluster
16:55 Gambit15 joined #gluster
16:56 logan_ joined #gluster
16:58 kotreshhr joined #gluster
17:04 MrAbaddon joined #gluster
17:13 sona joined #gluster
17:17 vbellur joined #gluster
17:17 Gambit15 joined #gluster
17:33 plarsen joined #gluster
17:41 akester joined #gluster
17:46 akester I'm having some issues mounting a volume after upgrading to 3.10.  Getting SETVOLUME on remote-host failed [Permission denied] in the logs.  Would anyone have any insight as to what troublehsooting to try?
17:48 DaKnOb joined #gluster
17:55 unclemarc joined #gluster
18:06 bwerthmann joined #gluster
18:26 shyam1 joined #gluster
18:28 oajs_ joined #gluster
18:28 plarsen joined #gluster
18:35 baber joined #gluster
18:49 ketarax i think i have a (or is 'the') CPU consumption issue ... running 3.7.17 (have to, for a while at least), and all my nodes have 10-40 gluster tasks, with #cpus at 100% and the rest consuming, well, the rest.  one node was down for periods during a time that the volume was being populated, however if there's any balancing that should be done, I don't see how it should take weeks, and also i'm not really seeing any balancing occurin
18:51 ketarax the nodes are connected via 1GB, but they're only transferring about 50-1000kB/s for the cpu consumption; it also appears most of everything that's going on with the bricks is reading
18:52 ketarax i don't have updatedb running on nodes (nor the clients that use the volume), and _AFAIK_ no other such "hidden" tasks that should be using the volume so.
18:53 ketarax any ideas will be appreciated.  also, and i know i maybe SHOULD'VE thought about this more before setting things up, but how's upgrading glusterfs in general?  can i go to 3.8?  3.9?  even 3.10?  what about the procedure -- i guess i ought to bring the volume offline before running the upgrades on nodes?
18:54 ketarax the nodes have 3 or 4 CPU's each
19:00 vbellur joined #gluster
19:01 Philambdo joined #gluster
19:05 oajs joined #gluster
19:12 kotreshhr left #gluster
19:32 niknakpaddywak joined #gluster
19:48 MrAbaddon joined #gluster
19:55 XpineX joined #gluster
19:57 zachfoo joined #gluster
19:57 derjohn_mob joined #gluster
19:57 zachfoo does anyone here work for a gluster consultancy that provides SLAs (and isn't Redhat)?
19:58 zachfoo most of the gluster partners on gluster.org are in europe, and they are few and far between...
19:59 oajs joined #gluster
20:00 shyam joined #gluster
20:09 neofob joined #gluster
20:09 mhulsman joined #gluster
20:10 neofob left #gluster
20:14 MrAbaddon joined #gluster
20:18 Asako https://gluster.org/ shows an error in Firefox, btw
20:20 misc yeah, we need to move to letsencrypt, but that requires a upgrade of the server (iirc) cause taht's still rhel 6
20:20 misc so we just push people to use www.gluster.org
20:31 baber joined #gluster
20:35 Asako it doesn't redirect until you accept the cert though
20:44 balacafalata joined #gluster
20:44 h0Ss joined #gluster
20:47 h0Ss joined #gluster
20:48 h0Ss-boss joined #gluster
20:50 h0Ss joined #gluster
20:50 h0Ss-boss joined #gluster
21:10 MrAbaddon joined #gluster
21:13 jwd joined #gluster
21:19 plarsen joined #gluster
21:23 StormTide kkeithley: will take a look... do i need to undo all my symlinking n stuff?
21:25 StormTide kkeithley: like is there a diff or anything so i can see what did/didnt change before i upgrade here...
21:25 kharloss joined #gluster
21:36 jwd joined #gluster
21:41 Munger joined #gluster
21:44 MrAbaddon joined #gluster
21:45 Munger Hi. I've been experimenting with GlusterFS, initially using a two node replicated cluster providing webroot for a WordPress site load-balanced over two locations linked on a VPN (80Mb broadband). Not surprisingly I'm encountering performance issues as the replication is synchronous. Does anyone have any suggestions that could help improve performance of GlusterFS or suggest a suitable alternative?
21:48 shyam joined #gluster
21:52 StormTide Munger: running php7 with opcache on and set to only look at files on server restart?
21:53 StormTide Munger: if not, thats your first optimization step (get all the reads for the source code on hit to go away)
21:55 Munger Yep. I'm already doing that, and have a varnish cache in front of nginx and Cloudflare upstream,
21:55 StormTide so wheres it giving you performance problems then? just volume of rarely used data?
21:55 Munger HAProxy is on EC2
21:57 StormTide php is really greedy about stat() calls which are hard on gluster.. most of which can be eliminated php side of things... but if thats not doing it, try the new 3.9+ small file performance options ... they're working well for me in a similar setup..
21:57 Munger My guess is that because I'm using a VPN on a broadband connection, I'm basically using a method designed for replication on a LAN rather than async geo-replication. i.e. unrealistically wanting the best of both worlds
21:58 Munger Do you have a link documenting those options?
21:58 StormTide Munger: its all about expectation, but usually if its overloading, its probably because your php is making too many stat() calls....
21:58 StormTide or you just have a lot of traffic going for imagery thats cold-cache or something
21:58 StormTide Munger:  hrm theres a link on the blog somewhere but i can paste the options im using, sec
21:59 cloph @search dht
21:59 glusterbot cloph: There were no matching configuration variables.
21:59 cloph @whatis dht
21:59 glusterbot cloph: I do not know about 'dht', but I do know about these similar topics: 'dd'
21:59 cloph https://joejulian.name/blog​/dht-misses-are-expensive/
21:59 glusterbot Title: DHT misses are expensive (at joejulian.name)
22:00 Munger I suspect there is an element of poor PHP config. I'm only testing and haven't ported all the config options from my live site. It certainly isn't traffic as I don't have any on the test site.
22:01 StormTide https://paste.fedoraproject.org/paste/SVu​KbHiMjnN0cqa0VS8qVF5M1UNdIGYhyRLivL9gydE= are the options im using
22:01 glusterbot Title: Untitled - Modern Paste (at paste.fedoraproject.org)
22:01 Munger Bookmarked.
22:01 StormTide http://blog.gluster.org/2016/10/gluste​r-tiering-and-small-file-performance/ <-- talks a bit more about those options
22:01 glusterbot StormTide: <'s karma is now -28
22:01 glusterbot Title: Gluster tiering and small file performance | Gluster Community Website (at blog.gluster.org)
22:02 cloph (the dht link was for Munger btw, explains why things are slow with typical php apps)
22:03 StormTide Munger: if you're seeing a lot of fileio traffic from a wordpress site its usually that you dont have opcache enabled properly or you're trying to log apache/nginx logs to the shared drive (write on each hit) ....
22:04 StormTide as php (at least since 7 and opcache) has pretty good fileio avoidance options...
22:05 Munger I will read that as soon as I get a quiet moment. Thanks. As you can guess, I'm very new to clustering in general. the only clustered directories are /var/wwe and whatever directories in /etc are appropriate to each VM. PHP, nginx, Persona, varnish etc each have a dedicated VM
22:05 StormTide http://php.net/manual/en/op​cache.configuration.php#ini.opcache.validate-timestamps might be helpful
22:05 glusterbot Title: PHP: Runtime Configuration - Manual (at php.net)
22:05 oajs joined #gluster
22:06 Munger (Percona most definitely doesn't use Gluster)
22:06 StormTide turning it off will stop php from going to the disk with each hit for the apps source code... but then you have to reload your server or write a clear cache script to update your site code...
22:07 Munger Yeah. I wrote a WordPress plugin to invalidate cache for edited articles.
22:07 StormTide thats db cache... different. this is the index.php etc... like the actuall app source which is usually read from disk
22:08 Munger Ah, yes. you're right.
22:08 StormTide if you're hitting those files with each hit over something like gluster (esp with a vpn) you're gonna have a bad time.
22:08 baber joined #gluster
22:10 Munger opcache is pretty good for that. My live site is squeezed into a t2-small instance on EC2 and holds up pretty well, but when we get a surge in traffic it groans a bit
22:12 StormTide yah, so just watch out for whats being read/written per hit, and you should be able to eliminate those reads for anything but a cold-cache-hit... and then if thats still too slow, look at the small file perf options.
22:12 Munger Anyway, thanks for the pointers. Very much appreciated. I'll have a read and a tinker and no doubt will be bugging you again.
22:13 Munger Probably not the right question to ask here, but do you think Gluster is the best option for my requirements, or should I also try other solutions?
22:14 Munger I don't want to waste your time if I'm barking up the wrong tree.
22:14 StormTide depends how much you need the self-healing/fault tolerance features really. we selected it for user content but are rsync replicating the site source code for example...
22:15 Munger Ah, so only wp-content is replicated?
22:15 StormTide (we didnt want a sync delay from time a user uploaded a picture to time it was visible on site for example)
22:15 Munger Gotcha
22:16 Munger That's certainly something I can look at. The source tends to be much less dynamic.
22:16 StormTide kinda right tool for whatever job
22:17 StormTide we also put a hdd cache in front of the user content so that anything accessed in the last 7 days or so caches on the app server's ssd if its been requested before
22:17 Munger I put the whole webfoot on cluster in an attempt to keep it in sync with the database. Each node has a local copy of that replicated with Percona
22:17 StormTide but we have unique identifiers for each image for example, so you have to know if you're chaning/editing files and so forth
22:18 StormTide Munger: look for where the webserver is logging, if you're appending to the access log and that log is on gluster, that'll be a problem
22:18 Munger I think the varnish VM does a pretty good job and images are 90& served by Cloudflare
22:18 Munger No. The logs are local
22:19 Munger Just webfoot and /etc/nginx on Gluster.  php7-fpm has webfoot and /etc/php and so on
22:19 Munger *webroot
22:19 StormTide .htaccess lookup disabled?
22:19 Munger Yep
22:19 Munger nginx
22:20 StormTide got me, you'll have to figure it out with lsof n such
22:21 Philambdo joined #gluster
22:21 Munger You've given me a few things to look at. I'll certainly consider more carefully which bits of webroot get glustered
22:22 nh2 joined #gluster
22:25 StormTide yah, just keep in mind if you turn on the small file performance options that it will eat a bunch of client memory
22:27 StormTide 6463252 5.738g   6808 S   1.3 18.3  74:11.76 glusterfs  =  VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  ... on one of my app servers using the small file perf options on a 500 or so gb small file volume
22:33 Munger Memory (for once) isn't an issue. One of the reasons I split each component into it's own VM is to allow me to allocate resources as required. I have three Proliant servers in a XenServer pool at each site, each boasting 96GB of RAM. Each has dual quad-core Xeons.
22:33 Munger The broadband is the bottleneck, hence the load-balancing
22:34 StormTide then yah give em a try... its working really nice for me. Eliminate a ton of the repetive calls for directory stats n stuff.
22:34 Munger Will do ;)
22:35 Munger Also, do you know of a cheap electricity company? Those Proliants were cheap as chips but are hungry :D
22:41 oajs joined #gluster
23:15 bwerthmann joined #gluster
23:32 MrAbaddon joined #gluster
23:45 bwerthma1n joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary