Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-09-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 wadeholler joined #gluster
00:46 wadeholler joined #gluster
00:48 shdeng joined #gluster
00:56 Alghost joined #gluster
01:32 Lee1092 joined #gluster
01:38 sandersr joined #gluster
01:40 wadeholler joined #gluster
01:46 derjohn_mobi joined #gluster
01:48 ilbot3 joined #gluster
01:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:50 muneerse2 joined #gluster
01:57 dlambrig joined #gluster
02:07 sandersr joined #gluster
02:27 kovshenin joined #gluster
02:27 kramdoss_ joined #gluster
02:30 Alghost joined #gluster
02:33 Alghost joined #gluster
02:34 Alghost_ joined #gluster
02:40 aspandey joined #gluster
02:46 plarsen joined #gluster
03:02 cliluw joined #gluster
03:02 Gambit15 joined #gluster
03:05 kdhananjay joined #gluster
03:16 poornima joined #gluster
03:32 magrawal joined #gluster
03:46 itisravi joined #gluster
03:47 LiftedKilt joined #gluster
03:54 itisravi joined #gluster
03:57 sanoj joined #gluster
04:07 atinm joined #gluster
04:08 kshlm joined #gluster
04:10 nbalacha joined #gluster
04:13 aspandey joined #gluster
04:19 riyas joined #gluster
04:28 shubhendu joined #gluster
04:36 aspandey_ joined #gluster
04:37 glustin joined #gluster
04:38 zerick joined #gluster
04:38 loadtheacc joined #gluster
04:40 shdeng joined #gluster
04:43 twisted` joined #gluster
04:43 ndk_ joined #gluster
04:43 Larsen_ joined #gluster
04:45 jiffin joined #gluster
04:46 ashiq joined #gluster
04:52 XpineX joined #gluster
04:55 ppai joined #gluster
05:03 karthik_ joined #gluster
05:04 nbalacha joined #gluster
05:05 ndarshan joined #gluster
05:05 scubacuda joined #gluster
05:06 PotatoGim joined #gluster
05:08 ndarshan joined #gluster
05:11 RustyB joined #gluster
05:12 poornima joined #gluster
05:14 fyxim joined #gluster
05:14 AppStore joined #gluster
05:14 tyler274 joined #gluster
05:15 telius joined #gluster
05:15 hgowtham joined #gluster
05:15 davidj joined #gluster
05:19 Gnomethrower joined #gluster
05:22 Bhaskarakiran joined #gluster
05:23 prasanth joined #gluster
05:25 an_ joined #gluster
05:30 bkunal joined #gluster
05:34 karnan joined #gluster
05:36 an__ joined #gluster
05:38 kshlm joined #gluster
05:43 satya4ever joined #gluster
05:44 skoduri joined #gluster
05:44 an_ joined #gluster
05:46 RameshN joined #gluster
05:48 ankitraj joined #gluster
05:48 jiffin joined #gluster
05:48 mhulsman joined #gluster
05:52 aravindavk joined #gluster
05:54 mhulsman joined #gluster
05:58 prth joined #gluster
05:58 ic0n_ joined #gluster
05:58 msvbhat joined #gluster
05:59 an_ joined #gluster
06:04 Saravanakmr joined #gluster
06:06 an_ joined #gluster
06:09 jwd joined #gluster
06:10 devyani7 joined #gluster
06:19 jiffin joined #gluster
06:19 burn_ joined #gluster
06:21 mlhess- joined #gluster
06:21 jtux joined #gluster
06:21 juhaj joined #gluster
06:22 rideh joined #gluster
06:30 aspandey joined #gluster
06:32 devyani7 joined #gluster
06:34 karnan joined #gluster
06:35 aravindavk_ joined #gluster
06:37 Muthu_ joined #gluster
06:39 om how fast is volume healing supposed to be?
06:39 om I have a volume with 105 GB data and it's been 3 days while only 35 GB has replicated
06:40 om we have 10 Gbps and 1 Gbps links
06:40 om not the bottleneck...
06:41 om when I run gluster volume heal info I see what is healing... quite slow
06:41 om and iftop shows slow connections for this
06:43 hgowtham joined #gluster
06:45 atinm joined #gluster
06:46 Klas we heal about 500 GB per day, virtual machines over 10 Gbps link, 2+1 setup
06:47 Klas for us, the bottleneck seems to be the arbiter, for some reason (load is very high on it)
06:47 Klas are you still stuck with incomplete data then?
06:47 Klas how bloody annoying =(
06:56 skoduri joined #gluster
07:00 derjohn_mobi joined #gluster
07:00 aravindavk joined #gluster
07:01 ashiq joined #gluster
07:05 jtux joined #gluster
07:08 k4n0 joined #gluster
07:11 mhulsman1 joined #gluster
07:13 rastar joined #gluster
07:19 an_ joined #gluster
07:19 msvbhat joined #gluster
07:19 atinm joined #gluster
07:26 kovshenin joined #gluster
07:26 jri joined #gluster
07:27 jri joined #gluster
07:30 fsimonce joined #gluster
07:36 ivan_rossi joined #gluster
07:37 Alghost joined #gluster
07:40 an_ joined #gluster
07:41 aspandey joined #gluster
07:44 tomaz__ joined #gluster
07:51 Philambdo joined #gluster
07:52 valkyr1e joined #gluster
07:56 deniszh joined #gluster
07:57 msvbhat joined #gluster
07:58 Philambdo1 joined #gluster
07:59 Pupeno joined #gluster
08:02 Philambdo joined #gluster
08:05 an_ joined #gluster
08:06 arcolife joined #gluster
08:08 Klas ehm, I'm currently experimenting with a high number of files in same directory (10 000), an ls seems to take about 16 seconds, which sounds reasonable. However, ls | wc -l, for some reason, takes 0.2 seconds (with correct info)
08:08 Klas how can this be?
08:13 hackman joined #gluster
08:16 an_ joined #gluster
08:22 itisravi Klas:  Writing to stdout takes a lot of time. `ls >/dev/null ` would take less time too.
08:24 Klas but, why would gluster be the limiting factor here?
08:25 Klas in both cases, the limiting factor should be all the stats, which should be necessary both when piping and not, or am I thinking incorrectly?
08:26 Jacob843 joined #gluster
08:28 Klas om: replace-brick seems to work correctly for me, btw, finally had time to test it
08:28 Klas heal is the wrong way to do it
08:35 om oh
08:35 om can you send me that link?
08:39 Klas https://joejulian.name/blog/replacin​g-a-glusterfs-server-best-practice/
08:39 glusterbot Title: Replacing a GlusterFS Server: Best Practice (at joejulian.name)
08:39 an_ joined #gluster
08:40 Klas gluster volume replace-brick ${volname} ${oldserver}:${oldbrickpath} ${newserver}:${newbrickpath} start
08:40 Klas I changed "start" to "commit force"
08:40 Klas otherwhise everything seemed correct
08:40 Klas start didn't seem like an option anymore
08:40 Klas oh, look, force might've been unneccessary, I will try that next time
08:41 hgowtham joined #gluster
08:42 itisravi Klas: not sure I follow. Why do you think gluster is the limiting factor?
08:43 Klas itisravi: ls on 10k files doesn't take 16 seconds for my system
08:44 ramky joined #gluster
08:44 Klas otoh, now it takes way shorter than 16 seconds on the gluster share as well
08:44 Klas 00.34 seconds
08:47 itisravi on local filesystem there's no network involved right
08:47 itisravi that's prolly due to caching.
08:48 Klas exactly my point.
08:49 Klas /glustershare # ls = takes 16
08:49 Klas /glustershare # ls | wc -l = takes 0.2
08:49 Klas in both cases, the limiting factor should be network/glusterfs
08:52 itisravi My point was if you are piping the output to wc, there is no writing to the terminal invovled, hence it would always be faster.
08:55 kenansulayman joined #gluster
08:57 karnan joined #gluster
08:59 klaas joined #gluster
08:59 Champi joined #gluster
09:00 eryc joined #gluster
09:00 Saravanakmr joined #gluster
09:00 satya4ever joined #gluster
09:00 pkalever joined #gluster
09:00 XpineX joined #gluster
09:00 prth joined #gluster
09:03 sandersr joined #gluster
09:03 shaunm joined #gluster
09:03 ebbex_ joined #gluster
09:03 frakt joined #gluster
09:03 rjoseph joined #gluster
09:03 The_Ball joined #gluster
09:03 gvandeweyer joined #gluster
09:03 jesk joined #gluster
09:03 javi404 joined #gluster
09:03 Anarka joined #gluster
09:03 ItsMe`` joined #gluster
09:03 kevc joined #gluster
09:03 wistof joined #gluster
09:03 kovshenin joined #gluster
09:04 lkoranda joined #gluster
09:04 eMBee joined #gluster
09:12 an_ joined #gluster
09:13 ankitraj joined #gluster
09:22 ira joined #gluster
09:26 Pupeno joined #gluster
09:26 [diablo]_ joined #gluster
09:29 [diablo] joined #gluster
09:40 an_ joined #gluster
09:40 decay joined #gluster
09:43 ahino joined #gluster
09:52 itisravi joined #gluster
09:56 an_ joined #gluster
09:59 an_ joined #gluster
10:04 k4n0 joined #gluster
10:08 hgowtham joined #gluster
10:09 ndarshan joined #gluster
10:09 msvbhat joined #gluster
10:14 Pupeno joined #gluster
10:14 robb_nl joined #gluster
10:26 RameshN joined #gluster
10:32 shyam joined #gluster
10:39 arcolife joined #gluster
10:40 B21956 joined #gluster
10:41 [diablo] joined #gluster
10:45 hchiramm joined #gluster
10:52 RameshN joined #gluster
11:02 tomaz__ joined #gluster
11:03 Pupeno joined #gluster
11:10 Gnomethrower joined #gluster
11:18 Pupeno joined #gluster
11:24 om is there a way to make gluster parse read requests before it checks with all bricks?
11:29 prth_ joined #gluster
11:35 jkroon joined #gluster
11:45 Pupeno joined #gluster
11:46 armin joined #gluster
11:51 an_ joined #gluster
11:55 squizzi joined #gluster
12:04 [diablo] joined #gluster
12:04 k4n0 joined #gluster
12:06 Pupeno joined #gluster
12:07 mhulsman joined #gluster
12:09 ramky joined #gluster
12:11 marbu joined #gluster
12:13 d0nn1e joined #gluster
12:18 [diablo] joined #gluster
12:23 Klas itisravi: well, yes, but not 80 times faster
12:23 Klas om: that sounds like the direct opposite way it's supposed to work
12:24 Klas gluster values integrity of data way over speed
12:24 Klas (but, granted, I can see why reads would be less sensitive)
12:25 unclemarc joined #gluster
12:27 itisravi Klas: did you unmount, drop_caches etc for each run?
12:28 Klas itisravi: nope, but every time I ran ls it came up about 16 seconds, every time i piped to wc, it tok about 0.2 seconds
12:28 Klas maybe ten times each?
12:29 itisravi hmm
12:32 Klas exactly =P
12:33 Klas it doesn't matter that much either, it just seems weird
12:33 Klas now it goes way faster btw, so I might have encountered something strange along the way
12:34 Klas now the difference is between 0.3 and 0.6, THAT seems reasonable
12:36 prth joined #gluster
12:56 jiffin1 joined #gluster
12:58 kdhananjay1 joined #gluster
13:01 TvL2386 joined #gluster
13:06 muneerse joined #gluster
13:11 ashiq joined #gluster
13:15 rastar joined #gluster
13:20 bluenemo joined #gluster
13:25 mbukatov joined #gluster
13:27 dgossage joined #gluster
13:29 ic0n joined #gluster
13:29 lkoranda joined #gluster
13:30 ic0n joined #gluster
13:31 baojg joined #gluster
13:37 an_ joined #gluster
13:37 shubhendu joined #gluster
13:39 kshlm joined #gluster
13:42 aravindavk joined #gluster
13:44 ndarshan joined #gluster
13:48 mhulsman joined #gluster
13:49 rastar joined #gluster
13:51 skylar joined #gluster
13:51 msvbhat joined #gluster
13:54 emitor_uy joined #gluster
13:57 dnunez joined #gluster
13:58 emitor_uy joined #gluster
13:59 om Klas: what did you do to get the read times to be faster?
14:00 om I have the same problem with ls
14:00 Klas nothing
14:00 Klas it just got better =P
14:00 Klas but it's on shared storage, so, well, might vary for different reasons
14:03 om How are you supposed to replace a brick when it's gone?
14:03 om the old-brick and new-brick are the same path and name
14:03 om how does that command make sense?
14:04 om gluster volume replace-brick ${volname} ${oldserver}:${oldbrickpath} ${newserver}:${newbrickpath} start
14:04 om the old and new server has the same IP and dns
14:04 om the old and new brick has the same path
14:04 om I rebuilt the broken server in place...
14:05 om how long did it take to recover with that cmd??  gluster volume replace-brick ${volname} ${oldserver}:${oldbrickpath} ${newserver}:${newbrickpath} start
14:05 marbu joined #gluster
14:05 om Klas: ^^
14:05 kovsheni_ joined #gluster
14:07 kdhananjay1 dgossage: There?
14:08 dgossage I am
14:09 plarsen joined #gluster
14:11 mhulsman joined #gluster
14:14 an_ joined #gluster
14:16 shyam joined #gluster
14:16 rwheeler joined #gluster
14:20 kpease joined #gluster
14:21 skoduri joined #gluster
14:31 ashiq joined #gluster
14:32 amye joined #gluster
14:33 k4n0 joined #gluster
14:38 Lee1092 joined #gluster
14:41 [o__o] joined #gluster
14:42 shaunm joined #gluster
14:55 emitor_uy hi, somebody have experienced very bad performance with gluster 3.7?
14:57 post-factum emitor_uy: sure, depends on workload/usage pattern
14:57 riyas joined #gluster
14:57 emitor_uy I get less than 30MB/s on a replica 2 volume configuration and 11MB/s on replica 3 arbiter 1 volume configuration
14:57 post-factum version?
14:57 emitor_uy local disk write at 300MB/s
14:58 emitor_uy 3.7.13
14:58 post-factum update to 3.7.15 first
14:58 post-factum there were some fixes regarding this issue
14:59 emitor_uy oh, ok
15:04 k4n0 joined #gluster
15:06 emitor_uy do you recomend to upgrade to 3.8?
15:07 emitor_uy instead to 3.7.15
15:07 JoeJulian I wish glusterbot still announced new and closed bugs.
15:07 JoeJulian I used to be on top of every bug status...
15:10 plarsen joined #gluster
15:10 kramdoss_ joined #gluster
15:13 emitor_uy I would like to use the gluster volume as a vm's storage for oVirt
15:14 wushudoin joined #gluster
15:15 emitor_uy I'm testing it with the gluster 3.7.13 which is the one on the repos, maybe it will be better to upgrade directly to 3.8 instead of 3.7.15. I don't see any bug fix related to bad performance in the 3.7.15 release.
15:16 robb_nl joined #gluster
15:18 Pupeno joined #gluster
15:32 prth joined #gluster
15:34 JoeJulian Did you really look at all 2110 pull requests since 3.7.13? That's even more dedication than I have.
15:34 Pupeno joined #gluster
15:36 JoeJulian Hmm, maybe that's wrong. Seem no matter which versions I compare in github is shows 2110 commits.
15:42 baojg joined #gluster
15:42 Klas om: I just did it with 1000 1 MB files, it took less than 20 minutes
15:43 Klas and, well, I suppose you could replace the brick with itself, if empty, not certain. I would probably create a new brick on the same server at least.
15:45 JoeJulian lots of leaks fixed in 3.7.15 from .14. That's nice.
15:50 Pupeno joined #gluster
15:51 JoeJulian Klas: The difference between "ls" and "ls | $anything" is that once you've piped it, ls knows it's not tty output so it does not read the attributed of all the file entries because it's no longer trying to color code the entries.
15:52 JoeJulian s/attributed/attributes/
15:52 glusterbot JoeJulian: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:52 Klas ah, great, that does change stuff
15:52 Bhaskarakiran joined #gluster
15:52 JoeJulian @restart
15:52 Klas thanks for this =)
15:52 JoeJulian You're welcome.
15:53 Klas a lot less to stat since a lot of attributes aren't parsed =)
15:56 glusterbot joined #gluster
15:58 Pupeno joined #gluster
16:02 muneerse joined #gluster
16:05 muneerse2 joined #gluster
16:08 mbukatov joined #gluster
16:09 skoduri joined #gluster
16:09 Philambdo joined #gluster
16:11 hchiramm joined #gluster
16:11 jiffin joined #gluster
16:13 squizzi_ joined #gluster
16:22 raymond-thor joined #gluster
16:23 raymond-thor Hello i am trying to follow the instructions for setting up glusterfs-server on rhel. The problem is it looks like everything has moved to SIG :( Do i have to add Centos extras repo to my rhel 7 box just to get the gluster repo?
16:23 ivan_rossi left #gluster
16:24 raymond-thor I am not seeing an easy way to add the repo anymore for RHEL users.
16:26 jiffin joined #gluster
16:27 ndarshan joined #gluster
16:28 Gambit15 joined #gluster
16:34 JoeJulian Perhaps this will help? https://wiki.centos.org/SpecialInter​estGroup/Storage/gluster-Quickstart
16:34 glusterbot Title: SpecialInterestGroup/Storage/gluster-Quickstart - CentOS Wiki (at wiki.centos.org)
16:39 jiffin joined #gluster
16:45 glustin joined #gluster
16:46 kpease joined #gluster
16:46 om Klas: 20 minutes for 1 GB???  That's horrible
16:48 Klas om: for 10, in small segments =P
16:48 Klas and I said less, didn't measure
16:48 Klas was probably less than 10 minutes in reality
16:53 JoeJulian om: Look at gluster, or any other piece of software you can use for that matter, as a tool in your arsenal. If the tool fits, use it. If it doesn't, use a different tool. It'll help you avoid the intense feelings associated with those pieces of software and prevent you from feeling horror.
16:54 om It's not bad
16:55 om just thinking we might need to see if another software will work for replicating data between data centers in separate continents
16:56 JoeJulian For eventually consistent global data availability, I don't think you can really beat swift at this point. Minio is getting there, but doesn't have distribution ready yet.
16:57 JoeJulian But you're never going to have instant global replication until someone creates quantum entangled switches.
16:58 JoeJulian Or, even better, quantum entangled drives.
16:58 JoeJulian Store data in qubits instantly replicated anywhere in the universe...
17:01 om JoeJulian: I'm having this issue
17:01 om I had to rebuild a brick server.
17:01 om How do I add the new server brick (that was rebuilt in-place) to a volume?
17:02 om I added it, and it's joined,
17:02 om then started heal but that's taking forever
17:02 om I keep get rejected from gluster volume replace-brick
17:02 JoeJulian I promise, it will finish before the end of time.
17:02 om gluster volume replace-brick gv_sftp0 i5-p-fshare-102.uk02.kexpress.net i5-p-fshare-101.uk02.kexpress.net force
17:02 om Usage: volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> {commit force}
17:03 om root@i5-p-fshare-101:/home/ubuntu#
17:03 om gluster volume replace-brick gv_sftp0 i5-p-fshare-102.uk02.kexpress.net i5-p-fshare-101.uk02.kexpress.net force
17:03 om Usage: volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> {commit force}
17:03 om says usage is wrong
17:03 JoeJulian Why would you replace brick?
17:03 JoeJulian and the usage *is* wrong, you only listed a server. A brick path is $server:$path.
17:03 JoeJulian But you can't replace-brick in-place.
17:04 JoeJulian You just start...force and that tells gluster to ignore the fact that there's no volume-id xattr and it creates it and starts the brick.
17:04 shyam joined #gluster
17:04 raymond-thor @JoeJulian I have been to that wiki page  'yum install centos-release-gluster' does not work as that repo is only in the centos extras repos
17:04 om gluster volume replace-brick gv_sftp0 i5-p-fshare-102.uk02.kexpress.net:/gluster/brick i5-p-fshare-101.uk02.kexpress.net:/gluster/brick commit
17:04 om Usage: volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> {commit force}
17:04 raymond-thor the centos extras is not a repo that should be added to a base rhel 7 system
17:04 JoeJulian raymond-thor: I would contact my RH support if I was the guy paying for it.
17:05 JoeJulian I have only used free stuff, never paid for RHEL so I don't know what deficiencies they have.
17:06 om no matter how I run replace-brick, it fails... returns USAGE: message
17:06 raymond-thor I will contact support however I don't know how they can help me since the move to the SIG basically excludes the rhel users.
17:07 JoeJulian om: Note the end of the command you said, "commit" where the example says "commit force".
17:08 om volume replace-brick: failed: Brick: not available. Brick may be containing or be contained by an existing brick
17:08 JoeJulian There you go.
17:08 om so now, I have to forfeit the 50% heal to be able to recreate this brick?
17:09 JoeJulian I have no idea what that means.
17:09 om not sure what to do...  Let heal take it's course, (50% done in 3 days), or kill the brick and do a replace-brick
17:09 JoeJulian You have to get the data from the replica that has it onto the one that does not.
17:10 om right
17:10 om so 2 options:  wait another 3 days for heal to finish OR destroy this brick and start over with replace-brick
17:10 om dunno whats better
17:11 om also, all glusterfs clients are showing the new volume fs.... so it is missing all the data on the other replica's
17:11 JoeJulian "I had to rebuild a brick server. How do I add the new server brick (that was rebuilt in-place) to a volume?" This is the source of my confusion because this does not involve replace-brick.
17:11 om Klas: was having this issue too
17:12 Klas JoeJulian: incorrectly using heal to try to replace-brick leaves the files which are not yet healed unaccessible on the clients
17:12 om Ok, sorry... I wrote incorrectly...  I wanted to know the best practice to do that... a link
17:13 om but the new brick is already joined to the volume now
17:13 Klas that is fully logical
17:14 Klas me and om both fucked up respectively, the question is if you should give up the heal or if you can use what you've already done
17:14 om So... What should I do now...?  Stop the heal process and replace-brick (hoping that will work and be faster than the heal)  OR continue to heal for the next 3 days
17:14 JoeJulian Ok, let me rephrase to see if I'm getting this...
17:14 Klas I believe you can use what you already have, cause replace-brick on an old version of the brick was very fast
17:15 Klas (I'm cooking dinner, so I might miss stuff)
17:15 JoeJulian You have a replica subvolume with one brick that is not fully healed (call it brick B). You want to replace B with C but if you call replace-brick B C the clients can no longer access the files on A that had not healed to B (or C).
17:15 om Klas explained this correctly
17:16 om Something like that...
17:16 om I have a replica 4
17:16 JoeJulian 4! holy cow, you must really have some unreliable hardware.
17:16 om I rebuilt 1 brick server from scratch and added it back to the cluster and volume.
17:17 om Then all the data on the 3 bricks stops showing on the mounts
17:17 om only the healed data shows on gluster client
17:18 om the healed data onto the new brick
17:18 JoeJulian Ok! got it. What version is this?
17:18 om 3.7.14
17:18 JoeJulian Do you happen to know of Klas is on the same version?
17:18 om yep it's the same version
17:19 JoeJulian Let me toss together a set of containers to see if I can duplicate that.
17:19 JoeJulian And that's every client, right?
17:19 om You will
17:19 om yep
17:19 om So question is... What to do now?
17:19 JoeJulian ~pasteinfo | om
17:19 glusterbot om: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
17:20 om https://dpaste.de/fZ3i
17:20 glusterbot Title: dpaste.de: Snippet #378662 (at dpaste.de)
17:21 om as a work around, I disabled nfs on gluster, installed nfs kernel server, and exported the brick from one gluster server to all the servers that need the fs mounted.
17:21 om I had no choice
17:22 JoeJulian Ah, so this is only for NFS?
17:22 JoeJulian Not fuse?
17:22 om and remounted all as NFS, instead of gluster client
17:22 om No, it's for everything
17:22 Klas JoeJulian: I only tried FUSE
17:22 JoeJulian Ok, I see. You're just writing directly to the brick.
17:22 JoeJulian You realize you're likely to lose data, right?
17:22 om I was using gluster client, but used NFS kernel server to work around this issue for now to stay on line
17:23 om I have no idea
17:23 msvbhat joined #gluster
17:23 om all the servers needing that fs are now mounting from one nfs kernel server running of one brick with all the data
17:23 JoeJulian Anyway, try setting cluster.read-subvolume-index to point to a replica subvolume that's healthy.
17:23 om I had no choice
17:24 JoeJulian Klas: Try setting cluster.read-subvolume-index to point to a replica subvolume that's healthy.
17:24 JoeJulian om: Try setting cluster.read-subvolume-index to point to the replica subvolume that you've been writing to.
17:24 JoeJulian om: that just *might* save you.
17:25 om how?
17:25 JoeJulian om: I'm not positive, but I think that even affects self-heal so it might heal the updated data to the rest of your replica.
17:26 JoeJulian om: you might have to heal-full after your other brick is healthy.
17:27 om how do I set cluster.read-subvolume-index ?
17:27 JoeJulian see gluster volume set help
17:31 om doesn't take the server brick
17:33 mhulsman joined #gluster
17:33 om ok did it
17:33 om now what?
17:33 om gonna heal faster or something/?
17:33 Philambdo joined #gluster
17:34 om heal rate is like 500KB per second
17:34 om for 60 GB .... another 3 days
17:34 om did I do something wrong?
17:34 om JoeJulian: ?
17:35 om I guess the issue Klas and I had is a bug...
17:35 om gluster client showing brick data on new brick that is missing the data from the other replicas...'
17:35 om If this were
17:36 om if this were not the case, meh, I would not care, but this bug renders glusterfs difficult to rebuild a brick and stay online with other replicas
17:37 JoeJulian It does appear that way, yes. I've never seen it but I've never done more than replica 3.
17:37 JoeJulian With replica 3, I've wiped bricks many times and healed them back with no problem.
17:37 om this is replica 4
17:37 JoeJulian I know that.
17:37 post-factum replica 4? but why?
17:37 post-factum it is so expensive
17:38 om 2 in each dc
17:38 JoeJulian oh holy shit
17:38 post-factum oh ye
17:38 rafi joined #gluster
17:38 JoeJulian What's the latency between dc?
17:38 om negligible
17:38 post-factum in msecs plz
17:39 om about 90 ms
17:40 post-factum mwhahaha ghrm meow
17:40 JoeJulian that's a long way from negligible.
17:40 om well, it's what I have to work with
17:40 post-factum it is *wrong* usage
17:40 om far from 500 ms
17:40 JoeJulian So you're talking about 270ms in chatter between heal blocks.
17:40 JoeJulian Just to handle locking.
17:40 om yea, I know
17:41 om I was warned not to do this, and I told higher ups this is not a good idea, but they pushed on it
17:41 om now I'm stuck with the issues
17:41 JoeJulian Been there, done that. I sympathize.
17:41 post-factum run, forrest, run. really
17:41 om Will suggest some other distributed fs for the future, but now I am with this
17:41 post-factum georep instead, no?
17:42 JoeJulian probably
17:42 om would AFS be better?
17:43 om what is used by cloud distributed fs and con's?
17:43 post-factum no, 90 ms for any netfs is a nightmare
17:43 om cdn
17:43 post-factum for *sync* fs
17:43 JoeJulian cdn
17:43 post-factum georep is for cdn
17:43 JoeJulian swift for async cdn makes more sense, imho.
17:43 post-factum in fact, anything async
17:44 post-factum maybe, cron+rsync would be ok as well
17:44 om not sure what to recommend at this point
17:44 om yea, cron and rsync
17:45 om but they don't want to hear that
17:45 om they want to be 'cloud'
17:45 post-factum cloud they said, no issues they said
17:45 post-factum what is cloud then?
17:45 om cloud fs, auto-replication, auto-DR, auto-everything
17:46 post-factum auto-rsync, why not. lsync/clsync
17:46 post-factum custom fanotify-based solution, why not
17:47 post-factum who the hell cares what is behind clouds
17:47 om cloud this and that
17:47 om it's all on HW
17:47 om gimmick
17:47 JoeJulian jbr will probably make it more possible to do this.
17:47 om some nebulous gas holding up the apps and servers?
17:48 JoeJulian nebulous sounds like a good name for an app.
17:48 om hey so, I guess I'm stuck waiting for heal...
17:48 om probably already exists
17:48 JoeJulian I believe so, yes. Did the setting I suggested allow fuse mounts to see all the files?
17:48 Klas JoeJulian: now you made want to try more stuff with heal on other versions, and we where just starting to lock down =P
17:49 JoeJulian Klas: do you have the same latency problem?
17:49 robb_nl joined #gluster
17:49 Klas nope, no latency issues
17:49 Klas the nodes are in the same subnet and everything
17:50 JoeJulian Ok, good. That'll be easier to diagnose.
17:50 Klas I can check what the latency is
17:50 om JoeJulian: when you said swift, do you mean the Openstack swift?
17:50 JoeJulian And you have replica 4?
17:50 JoeJulian om: yes
17:50 Klas JoeJulian: 2+1
17:50 JoeJulian So this isn't the same thing at all.
17:50 JoeJulian but your client still isn't seeing the files that haven't healed?
17:51 Klas correct
17:51 Klas 0.2-0.5 ms latency or so
17:51 JoeJulian What if, from the client, you try to stat a missing file?
17:51 Klas hmm
17:51 Klas I only tried ls and so forth
17:52 Klas I believe it's rebuilt almost all data now, so hard to try again
17:52 Klas 1739899472 vs 1648045692, spread over MANY file
17:52 Klas s
17:52 JoeJulian fyi, om, "nfs.disable: off" is unnecessary. That's the default.
17:53 Klas and now wife is pulling me away
17:53 Klas I'll experiment tomorrow if I have time =)
17:53 Klas probably in the afternoon UTC+2
17:53 JoeJulian Good luck. UTC+7 here.
17:53 post-factum UTC+2? sounds like you are somewhere around
17:54 post-factum JoeJulian: -7 or +7?
17:54 om damn, I thought that was on
17:54 JoeJulian oops, -7
17:54 JoeJulian time machine problems today
17:54 post-factum JoeJulian: I thought you are in Bangalore :D
17:54 JoeJulian lol, I'm in Seattle.
17:54 om I gotta go to sleep.  Been up 20 hours, working
17:54 post-factum i know
17:54 post-factum but that +7
17:54 om for 12
17:54 om thanks for the help all
17:54 om ttyl
17:55 JoeJulian I think BLR is +0.5, iirc.
17:55 post-factum oh, BLR is  +5:30
17:55 post-factum okay
17:56 JoeJulian There you go...
17:56 JoeJulian I knew there was a half hour in there somewhere.
17:57 post-factum JoeJulian: going to move to Brno in 2 weeks
17:58 JoeJulian Oh, wow. I might be able to get that beer sometime then. There seems to be plenty of foss conventions there.
17:58 JoeJulian Bangkok! There's a +7 city.
17:58 post-factum JoeJulian: yup, you'll definitely get one from me once you are there
17:59 post-factum JoeJulian: and once everything goes well for me in RH
17:59 JoeJulian *in* RH? You're going to work for them?
18:00 post-factum JoeJulian: correct. they've hired me
18:00 JoeJulian Congratulations! That's a good gig.
18:00 JoeJulian I don't know a single person that regrets working there.
18:00 post-factum JoeJulian: thanks. that was long road started in March (!)
18:00 misc post-factum: oh, hired on what position ?
18:01 post-factum kernel support engineer
18:01 misc nice
18:01 misc well, I might go in Brno in November
18:01 yosafbridge joined #gluster
18:02 post-factum with a ping from ndevos to deal with gluster too sometimes
18:02 misc but support is cool, people told you learn a lot
18:03 post-factum misc: that is what i want — learn kernel better
18:03 post-factum it appeared my C/gdb skills are sufficient for that
18:03 misc (and brno is pretty nice city too)
18:03 misc (almost moved there)
18:03 post-factum almost?
18:04 post-factum you're in DE, right?
18:04 misc nope, in Paris
18:04 misc afaik, that's part of France since more than 60 years
18:05 post-factum :D sorry
18:07 post-factum misc: so, why almost?
18:11 msvbhat joined #gluster
18:18 robb_nl joined #gluster
18:23 jwd joined #gluster
18:26 bdashrad left #gluster
18:32 misc post-factum: got another position who was not office based
18:32 misc so I decided to stay in France
18:37 post-factum misc: sounds reasonable
18:39 hackman joined #gluster
18:49 dgossage joined #gluster
18:58 deniszh joined #gluster
18:59 rafi joined #gluster
18:59 prth_ joined #gluster
19:05 derjohn_mobi joined #gluster
19:28 kkeithley joined #gluster
19:38 emitor_uy Hi, I've just installed gluster 3.8.3 and I still see a really bad performance, here are the details https://thepb.in/p/BBf9cjV3EWJ9JtW
19:38 glusterbot Title: TheP(aste)B.in - For all your pasting needs! (at thepb.in)
19:38 emitor_uy How could I troubleshoot this?
19:40 post-factum should one stop benchmarking glfs with dd?
19:42 emitor_uy sorry, I didn't get it
19:44 kkeithley dd is a really bad benchmark. For pretty much anything.
19:45 emitor_uy what do you recomend? I'm trying with fio and the results are bad too
19:46 post-factum real workload would do the thing
20:01 emitor_uy But I have another installation of gluster 3.5.2 into similar servers that this values are much better so I asume that something is wrong
20:02 glustin joined #gluster
20:13 dgossage have you tried using 3.5.2 on the suspect servers to confirm same results then?  I wouldn't trust any i/o tests that tried to tell me I was writing at 2.2GB/s locally though.
20:17 emitor_uy Ok, I'm agree that dd is not good to benchmark. This is the test logs on the gluster 3.8.3 using fio https://thepb.in/p/BBf9cjV3E5A9lHW
20:17 glusterbot Title: TheP(aste)B.in - For all your pasting needs! (at thepb.in)
20:22 BitByteNybble110 joined #gluster
20:31 JoeJulian Comparing (any) clustered filesystem to a local filesystem is like comparing apples and orchards.
20:32 JoeJulian http://joejulian.name/blog/dont-get​-stuck-micro-engineering-for-scale/
20:32 glusterbot Title: Don't Get Stuck Micro Engineering for Scale (at joejulian.name)
20:36 shyam joined #gluster
20:37 B21956 joined #gluster
20:54 Klas post-factum: swede
20:55 post-factum Klas: n
20:59 Klas I mean that I am =P
21:01 k4n0 joined #gluster
21:02 arcolife joined #gluster
21:02 post-factum ah oh uh ok
21:03 Philambdo joined #gluster
21:12 shyam joined #gluster
21:28 snehring joined #gluster
21:34 R0ok_ joined #gluster
21:55 ashiq joined #gluster
22:13 john51 joined #gluster
22:33 hagarth joined #gluster
22:43 jobewan joined #gluster
22:57 arcolife joined #gluster
23:10 jkroon joined #gluster
23:39 shyam joined #gluster
23:47 plarsen joined #gluster
23:55 masber joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary