Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-07-20

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:25 rotbeard joined #gluster
00:26 edwardm61 joined #gluster
00:38 nangthang joined #gluster
00:44 topshare joined #gluster
01:01 topshare joined #gluster
01:07 topshare_ joined #gluster
01:13 gildub joined #gluster
01:18 glusterbot News from newglusterbugs: [Bug 1206539] Tracker bug for GlusterFS documentation Improvement. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1206539>
01:18 vmallika joined #gluster
01:33 harish joined #gluster
01:47 ilbot3 joined #gluster
01:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:50 nangthang joined #gluster
01:50 topshare joined #gluster
02:17 bharata-rao joined #gluster
02:22 topshare joined #gluster
02:43 Lee1092 joined #gluster
02:47 kshlm joined #gluster
02:47 topshare joined #gluster
03:17 TheSeven joined #gluster
03:17 sakshi joined #gluster
03:19 glusterbot News from newglusterbugs: [Bug 1242708] fuse/fuse_thread_proc : The fuse_graph_sync function cannot be handled  in time after we fix-layout. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1242708>
03:35 topshare joined #gluster
03:49 itisravi joined #gluster
03:50 shubhendu joined #gluster
04:02 atinm joined #gluster
04:06 meghanam joined #gluster
04:07 schandra joined #gluster
04:14 schandra joined #gluster
04:16 sripathi joined #gluster
04:18 hagarth joined #gluster
04:18 vimal joined #gluster
04:22 nbalacha joined #gluster
04:25 kdhananjay joined #gluster
04:25 yazhini joined #gluster
04:26 kdhananjay joined #gluster
04:29 Jitendra joined #gluster
04:31 rotbeard joined #gluster
04:34 ppai joined #gluster
04:34 topshare joined #gluster
04:47 gem joined #gluster
04:47 ramteid joined #gluster
04:51 itisravi joined #gluster
04:55 Jitendra joined #gluster
04:56 ndarshan joined #gluster
05:02 lezo joined #gluster
05:06 twisted` joined #gluster
05:06 pppp joined #gluster
05:13 aravindavk joined #gluster
05:22 deepakcs joined #gluster
05:25 Bhaskarakiran joined #gluster
05:27 RameshN joined #gluster
05:37 tg2 joined #gluster
05:38 soumya joined #gluster
05:39 hgowtham joined #gluster
05:40 anil_ joined #gluster
05:41 Lee1092 joined #gluster
05:41 fyxim joined #gluster
05:42 gildub joined #gluster
05:42 frankS2 joined #gluster
05:42 billputer joined #gluster
05:46 vmallika joined #gluster
05:46 maZtah joined #gluster
05:48 Pintomatic joined #gluster
05:56 karnan joined #gluster
05:58 saurabh_ joined #gluster
05:58 smohan joined #gluster
06:02 Saravana_ joined #gluster
06:04 R0ok_ joined #gluster
06:05 overclk joined #gluster
06:06 spalai joined #gluster
06:07 dusmant joined #gluster
06:08 atalur joined #gluster
06:14 hagarth joined #gluster
06:14 anil_ joined #gluster
06:15 ndarshan joined #gluster
06:16 raghu joined #gluster
06:16 topshare joined #gluster
06:17 Manikandan joined #gluster
06:28 maveric_amitc_ joined #gluster
06:31 RedW joined #gluster
06:34 jtux joined #gluster
06:39 soumya joined #gluster
06:39 scubacuda joined #gluster
06:41 scubacuda joined #gluster
06:42 nangthang joined #gluster
06:45 jiffin joined #gluster
06:48 nsoffer joined #gluster
06:56 Manikandan joined #gluster
07:01 [Enrico] joined #gluster
07:02 atalur joined #gluster
07:05 jwd joined #gluster
07:07 jwaibel joined #gluster
07:07 kshlm joined #gluster
07:21 ndarshan joined #gluster
07:22 Manikandan joined #gluster
07:31 kshlm joined #gluster
07:33 topshare joined #gluster
07:44 Lee1092 joined #gluster
07:46 fyxim joined #gluster
07:46 kokopelli joined #gluster
07:47 mbukatov joined #gluster
07:56 atalur joined #gluster
07:58 autoditac joined #gluster
08:01 topshare joined #gluster
08:04 lezo joined #gluster
08:05 ajames-41678 joined #gluster
08:11 frankS2 joined #gluster
08:11 saurabh_ joined #gluster
08:13 itisravi joined #gluster
08:14 Trefex joined #gluster
08:18 arthurh joined #gluster
08:19 Pintomatic joined #gluster
08:29 hagarth joined #gluster
08:33 Pupeno joined #gluster
08:33 dusmant joined #gluster
08:33 Pupeno joined #gluster
08:39 RameshN joined #gluster
08:43 pcaruana joined #gluster
08:48 Trefex joined #gluster
08:53 atalur joined #gluster
08:54 deniszh joined #gluster
08:54 topshare joined #gluster
09:00 topshare joined #gluster
09:11 kdhananjay joined #gluster
09:14 atalur_ joined #gluster
09:14 rjoseph joined #gluster
09:15 kaushal_ joined #gluster
09:16 vmallika joined #gluster
09:19 michal_ joined #gluster
09:21 michal_ hi, I am trying to setup gluster volume on 3 servers (replica 3) with infidiband and rdma transport. If I do it with only 2 servers (replica 2), everything works as expected. However, with 3 servers I always fail. Gluster version is 3.7 (latest) and all servers are fedora 20.
09:21 topshare joined #gluster
09:22 michal_ I vaguely remember, that there were some problems with rdma transport in the history. Could it be that this is a bug (e.g. rmda does work only on even number of servers or so...)?
09:23 Slashman joined #gluster
09:23 Philambdo joined #gluster
09:23 michal_ basicaly - are there people with positive experience using similar setup?
09:24 kdhananjay joined #gluster
09:24 ndevos not that I know, and my favorite rdma guy (rafi) isnt online at the moment
09:24 hchiramm rastar_afk, ^^^^
09:25 michal_ hm, with our configuration, tcp transport is no problem. That part always worked.
09:25 hchiramm michal_, any specific error you noticed with 3 servers ?
09:25 ndevos oh, right, rastar knows about rdma too
09:26 michal_ the error was "Commit failed on server2"
09:28 hchiramm hmm.. something glusterd folks can look at
09:30 hchiramm michal_, give us some time, we will revert on this.
09:30 michal_ ok
09:31 michal_ I will try to investigate
09:31 michal_ but I can already confirm that on two servers (i.e. replica 2) I can create and start volume with rdma transport.
09:32 hchiramm ok.. thanks for this isolation part..
09:32 hchiramm michal++
09:32 glusterbot hchiramm: michal's karma is now 1
09:34 michal_ also I am going to check every server solo - single volume on single server using rdma - I want to be sure that all the underlaying functionality is ok.
09:35 michal_ it will take some time though.
09:36 soumya joined #gluster
09:45 atinm joined #gluster
09:49 ajames-41678 joined #gluster
09:55 rjoseph joined #gluster
10:02 maveric_amitc_ joined #gluster
10:14 krishna___ joined #gluster
10:14 krishna___ left #gluster
10:15 uebera|| joined #gluster
10:15 michal_ aha, I have discovered missing library on my 2nd server (out of those 3 to compose gluster volume) - I am going to verify it, but so far it seems that the problem was completely on my side ...
10:15 michal_ sorry for the fuss.
10:18 ndevos oh, would be good to know if thats it, indeed
10:19 ndevos aaaah, there is a fly in my room and its driving me nuts!
10:19 * ndevos bets everyone really appreciates that piece of information
10:20 csim can we add it to the newsletter ?
10:24 Romeor there is a little (4 months) baby in my flat, that started to go over to the ultra sound and this really driving CRAZY
10:36 dusmant joined #gluster
10:38 ndevos Romeor: I think I was now able to kick the fly out, you could try that with the baby too?
10:39 ndevos csim: you can add anything to the etherpad, the volunteer that is going to post it might sensor some things out though
10:40 ndevos uh, not sensor, censur?
10:40 csim censor
10:40 csim or sensur :)
10:41 ndevos ah, so close!
10:47 ira joined #gluster
10:51 pppp joined #gluster
10:58 Norky joined #gluster
11:06 ekuric joined #gluster
11:09 kovshenin joined #gluster
11:10 ekman joined #gluster
11:17 shubhendu joined #gluster
11:18 ndarshan joined #gluster
11:20 glusterbot News from newglusterbugs: [Bug 1244721] glusterd: Porting left out log messages to new logging API <https://bugzilla.redhat.co​m/show_bug.cgi?id=1244721>
11:25 ndarshan joined #gluster
11:28 dusmant joined #gluster
11:38 LebedevRI joined #gluster
11:41 overclk joined #gluster
11:41 autoditac joined #gluster
11:44 [Enrico] joined #gluster
11:51 pppp joined #gluster
11:51 shubhendu joined #gluster
11:56 julim joined #gluster
11:57 soumya joined #gluster
11:59 smohan_ joined #gluster
12:08 plarsen joined #gluster
12:11 nangthang joined #gluster
12:11 overclk joined #gluster
12:13 nsoffer joined #gluster
12:13 jtux joined #gluster
12:14 hchiramm joined #gluster
12:15 ppai joined #gluster
12:18 rjoseph joined #gluster
12:19 jrm16020 joined #gluster
12:21 jiffin joined #gluster
12:27 pppp joined #gluster
12:32 B21956 joined #gluster
12:35 autoditac joined #gluster
12:40 unclemarc joined #gluster
12:41 jtux joined #gluster
12:44 jarrpa joined #gluster
12:54 cleong joined #gluster
12:59 DV__ joined #gluster
13:08 bene joined #gluster
13:09 spalai left #gluster
13:10 nsoffer joined #gluster
13:12 jmarley joined #gluster
13:14 Pupeno joined #gluster
13:14 Pupeno joined #gluster
13:18 dgandhi joined #gluster
13:23 DV__ joined #gluster
13:27 mpietersen joined #gluster
13:27 georgeh-LT2 joined #gluster
13:29 hagarth joined #gluster
13:30 hamiller joined #gluster
13:34 mpietersen joined #gluster
13:34 aaronott joined #gluster
13:37 smohan joined #gluster
13:40 aravindavk joined #gluster
13:46 shyam joined #gluster
13:55 cyberswat joined #gluster
14:00 rwheeler joined #gluster
14:03 nsoffer joined #gluster
14:08 cyberswat joined #gluster
14:08 squizzi joined #gluster
14:09 bene2 joined #gluster
14:10 _dist joined #gluster
14:13 wushudoin joined #gluster
14:19 [Enrico] joined #gluster
14:20 dusmant joined #gluster
14:23 haomaiwa_ joined #gluster
14:25 cyberswat Could really use some help.  I have a replicated volume I lost a brick on. Self-heal will not repair the new brick as far as I can tell.  I've run volume heal full a few times followed by  find -exec stat {} \; from mounted clients.  The bricks still look unbalanced.  However, I can verify that new files get replicated as they should.  What can I look at to help me balance these bricks and ensure they are healthy?
14:26 jarrpa left #gluster
14:32 smohan_ joined #gluster
14:35 mpietersen joined #gluster
14:36 pdrakewe_ joined #gluster
14:36 ajames-41678 joined #gluster
14:36 shyam cyberswat: Do the logs show anything from a self heal perspective? Either the client or the self-heal logs? (which version of gluster is this? although I am no self heal expert, it may help others answer)
14:39 liewegas_ joined #gluster
14:39 jdossey joined #gluster
14:39 lanning_ joined #gluster
14:39 Marqin_ joined #gluster
14:40 d-fence_ joined #gluster
14:40 sadbox_ joined #gluster
14:41 _dist cyberswat: is it possible any of the files were written to locally?
14:41 atrius_ joined #gluster
14:42 ultrabizweb joined #gluster
14:42 sankarshan joined #gluster
14:42 squaly joined #gluster
14:42 wushudoin joined #gluster
14:42 B21956 joined #gluster
14:42 cyberswat @shyam which logs should I look at? It's version 3.7.2.
14:42 Telsin joined #gluster
14:43 perpetualrabbit joined #gluster
14:43 xoritor joined #gluster
14:43 lyang0 joined #gluster
14:43 cyberswat _dist: I do not believe so ... this volume has been mounted from a web server behind automation ... afaik nobody has accessed them directly until one of the bricks failed.
14:44 [o__o] joined #gluster
14:44 shyam cyberswat: client mount logs (say fuse mount, then /var/log/glusterfs/<mountpoint path>.log)
14:45 jermudgeon joined #gluster
14:45 monotek1 joined #gluster
14:46 shyam cyberswat: and, /var/log/glusterfs/glustershd.log on the brick nodes (I would start with these)
14:47 anoopcs joined #gluster
14:49 cyberswat @shyam https://gist.github.com/cyb​erswat/cb7de52ea456df89c77f
14:49 cyberswat I do not see anything there that looks problematic
14:50 virusuy joined #gluster
14:51 cyberswat I have a new healthy volume running ... does it make sense to simply transfer the files off the known good brick to the new volume?
14:54 _maserati joined #gluster
15:01 autoditac joined #gluster
15:06 jwaibel hi there
15:07 jwaibel is there a benefit to have more than 2 nodes in a cluster?
15:07 jwaibel would that have any drawbacks?
15:10 sankarshan_ joined #gluster
15:10 _maserati as in mirror or striping ?
15:11 jwaibel mirror
15:11 jwaibel i need a fault tolerance setup for distributed config files and shared data
15:12 _maserati when you're doing writes, every gluster server has to perform the write before you get an ACK that the write has been completed. I'm not sure if gluster will multiplex the read though which could have significant read performance increase
15:12 vimal joined #gluster
15:12 jwaibel so for the config files that would not harm me. those do not change that often
15:12 _maserati but if performance isn't really what you're looking for, and you keep an eye on your two nodes, that should be just fine
15:14 jwaibel what would happen when the network between both nodes goes away and still files get added/changed on both sides. will that be resolved somehow?
15:15 [Enrico] joined #gluster
15:15 _maserati i beleive gluster will resolve those. but i'm gonna have to defer you to someone with more knowledge than I in this irc room
15:16 jwaibel you will have more experience than me. :-)
15:18 _maserati like i said, I -beleive- but not 100% sure, as long as they are replicated bricks, they will be able to self heal any disparity
15:18 shyam cyberswat: The log files look pretty normal, did you attempt a stat on a file that you know has not healed, and checked the log files for any activity? or is this the log file during all the "find -exec stat..." commands?
15:20 bennyturns joined #gluster
15:21 cyberswat @shyam I believe the latter to be true.  I have not identified a specific file that is missing in the new brick.  I'm seeing about 85,000 files missing based on the output of  find . -type f | wc -l from the mount points
15:24 shyam cyberswat: If this is a log of the latter, I would defer the analysis to AFR/replicate experts here. The find command, was executed on the brick directory? Could you you exclude .../<brick dir>/.glusterfs in that case and see of the count matches? Want to eliminate any stale files in the gluster internal directory as the cause for the discrepency.
15:26 Bhaskarakiran joined #gluster
15:29 cyberswat @shyam Thank you for the help you have provided.  The total count while excluding the .glusterfs directories is 820710 on the long standing brick and 737382 on the newer one.
15:33 scuttle|afk joined #gluster
15:37 mpietersen joined #gluster
15:38 shyam cyberswat: Well this about ends what I can think of ATM, suggest you try to find *a* file that is missing between the bricks, that may help others prod for more data, and also post this to users ML for attention.
15:41 cyberswat @shyam thanks again ... I just submitted the redhat consulting form as well to start that conversation.
15:45 harish joined #gluster
15:56 scubacuda joined #gluster
16:14 cholcombe joined #gluster
16:15 adzmely joined #gluster
16:19 bennyturns joined #gluster
16:19 bennyturns joined #gluster
16:21 calavera joined #gluster
16:30 soumya joined #gluster
16:51 gem joined #gluster
16:55 pppp joined #gluster
17:02 Rapture joined #gluster
17:02 calisto joined #gluster
17:03 kampnerj joined #gluster
17:04 bene2 joined #gluster
17:08 edong23 joined #gluster
17:14 jobewan joined #gluster
17:15 nage joined #gluster
17:31 smohan joined #gluster
17:32 calavera joined #gluster
18:00 rotbeard joined #gluster
18:03 md2k hi all
18:04 nsoffer joined #gluster
18:04 md2k i set 'cluster.data-self-heal, cluster.entry-self-heal. cluster.metadata-self-heal ' to off, but still can see in logs on servers who mounts the gluster volume that self-heal performed for files.
18:13 mpietersen joined #gluster
18:13 md2k but looks like this self-heal triggered only for newly created files, so maybe it is correct
18:14 xoritor so i am testing and have  blown away my glusterfs and installed ceph
18:14 xoritor right now i get 2-6x better performance using glusterfs
18:15 jrm16020 joined #gluster
18:26 JoeJulian xoritor: yeah, that's what I've seen as well.
18:27 mpietersen joined #gluster
18:34 cholcombe anyone good with Gluster RPC?  I have some questions about the struct members
18:34 xoritor i do get great numbers from rados bench on reads though
18:34 xoritor JoeJulian, are you in #ceph?
18:34 JoeJulian I am
18:34 xoritor did you see the numbers i put in there?
18:35 xoritor thats using ipoib
18:35 JoeJulian yep
18:37 JoeJulian Overall, I'm not very impressed with ceph. The only thing it has going for it is the procedural replica placement vs gluster's predetermined placement.
18:37 Rapture joined #gluster
18:38 xoritor it is a PTIA to setup/maintain
18:38 JoeJulian Yep
18:38 xoritor but then gluster can be a PITA to maintain too at times ;-)
18:39 xoritor not to setup mind you
18:39 xoritor but when something does not "heal"
18:39 xoritor or rebalance fails with no reason why
18:39 xoritor or
18:39 xoritor or
18:39 xoritor lol
18:40 xoritor at least the messages are not nearly as cryptic
18:40 JoeJulian Well, the only heal that fails (unless something's very wrong elsewhere) is split-brain. Now even that can be handled from the cli?
18:40 JoeJulian s/?/.
18:41 xoritor you dont anything nearly this cryptic with gluster....
18:41 xoritor 2015-07-20 13:15:37.537252 7f1240647700  0 -- 10.0.0.4:0/1004658 >> 10.0.0.4:6789/0 pipe(0x7f122c006d70 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f122c00b010).fault
18:41 JoeJulian yeah, not quite as cryptic.
18:41 xoritor too many monitors down
18:41 xoritor heh
18:42 xoritor why not just say "hey idiot... i  dont have enough monitors, make sure they are running"
18:44 deniszh joined #gluster
18:44 fsimonce joined #gluster
18:47 hgowtham joined #gluster
18:53 dbruhn joined #gluster
18:56 wushudoin joined #gluster
19:12 dbruhn_ joined #gluster
19:17 mpietersen joined #gluster
19:36 mpietersen joined #gluster
19:39 mpietersen joined #gluster
19:39 calavera joined #gluster
19:41 calisto joined #gluster
19:52 mpietersen joined #gluster
20:02 pdrakeweb joined #gluster
20:05 MugginsM joined #gluster
20:11 MugginsM so, I have a 2-server gluster setup with multiple bricks replicated between them. I want to add a third node, to make a 3-way replica, any guides on doing that? I can't find much
20:12 MugginsM (long term goal being replace the 2 nodes with 2 OS upgraded nodes)
20:14 JoeJulian @lucky gluster 2 servers to 3 servers
20:14 glusterbot JoeJulian: https://www.digitalocean.com/community/tu​torials/how-to-create-a-redundant-storage​-pool-using-glusterfs-on-ubuntu-servers
20:14 JoeJulian no... (sigh)
20:14 JoeJulian MugginsM:
20:15 JoeJulian https://joejulian.name/blog/how-to-expand-g​lusterfs-replicated-clusters-by-one-server/
20:15 MugginsM thanks, I think that just still describes 2-replicas across 3 servers?
20:16 MugginsM is there a way to get 3 copies of each brick?
20:16 JoeJulian Oh, well if you just want replica 3, set that during add-brick, ie. gluster volume add-brick foo replica 3 server3:/brick
20:17 JoeJulian also...
20:17 JoeJulian @lucky replication do's and don'ts
20:17 glusterbot JoeJulian: https://joejulian.name/blog/glus​terfs-replication-dos-and-donts/
20:18 MugginsM ok so that's a valid thing to do, and my "add-brick: failed: Operation failed" is some other problem? :)
20:18 JoeJulian yep
20:18 JoeJulian check the glusterd log(s)
20:19 MugginsM the end goal is replace the two old servers with two new ones, but we're under quite heavy load 24x7 and dropping to one copy of the data while it's replicating the new ones is a bit iffy
20:19 JoeJulian excellent
20:21 MugginsM ah, server 3 is holding its own lock again, sigh
20:21 * JoeJulian rolls his eyes.
20:28 MugginsM so much for rsyncing peers while not properly awake :)
20:32 MugginsM ok the error from the add-brick request is blank
20:32 MugginsM that'll be fun to google
20:44 deniszh joined #gluster
20:48 MugginsM ah well, lets hope replace-brick works :)
20:48 MugginsM gonna take aaaaages
20:52 JoeJulian Shouldn't be any different success than add-brick.
20:52 JoeJulian What's add-brick's error?
20:53 MugginsM blank, "....0-management:  replica-count is 3 "
20:53 MugginsM then "....0-management: "
20:54 MugginsM this was from gluster 3.4.5
20:56 JoeJulian moar detail. I can't look at the source with that.
20:57 badone joined #gluster
20:59 MugginsM pastebin.com/Wv7xKvwC
20:59 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
20:59 MugginsM fairy nuff glusterbot
21:00 MugginsM hmm, actually that's quite a lot of E's
21:01 dgandhi joined #gluster
21:10 JoeJulian MugginsM: which server was that, the one you issued the cli command from, or the new brick server?
21:12 MugginsM the new one
21:12 MugginsM validate replica count failing
21:15 MugginsM so the volume is 7x2 bricks
21:15 MugginsM I want to make it 7x3 I guess
21:15 MugginsM can it do it one brick at a time?
21:16 MugginsM (lots of bricks because until recently max disk size on AWS was 1TB)
21:18 JoeJulian Ah, ok, no. You have to do all 7.
21:19 JoeJulian And I don't have an advised way to reduce the brick count. Especially with 3.4.
21:20 MugginsM we're currently on Ubuntu 10,04 with 3.4.5.  My plan was to get them onto Ubuntu 14.04 with 3.4.5 then upgrade that to 3.6 or 3.7
21:23 MugginsM can redo bricks on the new machines, but not keen on trying to backport 3.6 to lucid
21:23 MugginsM is there a way to do all bricks at once?
21:24 MugginsM turn it all off, edit the vol files.
21:26 MugginsM ok so a replace brick should work ok?
21:26 MugginsM because the replica count stays same?
21:26 bennyturns joined #gluster
21:32 * MugginsM runs off to a meeting, back later
21:40 JoeJulian @later tell MugginsM Are these EBS volumes?
21:40 glusterbot JoeJulian: The operation succeeded.
21:41 nsoffer joined #gluster
21:42 marcoceppi joined #gluster
21:44 calavera joined #gluster
22:07 mribeirodantas joined #gluster
22:12 chirino joined #gluster
22:12 indigoblu joined #gluster
22:14 dijuremo joined #gluster
22:14 xoritor anyone running on fedora 22?
22:15 MugginsM joined #gluster
22:15 dijuremo Hi again, coming back to seek some help with performance issues
22:16 dijuremo Is it better to just send e-mail to the mailing list?
22:16 xoritor dijuremo, what is your issue?
22:16 xoritor my issue with performance is that it is never as fast as i want it to be
22:16 xoritor ever....
22:16 indigoblu what configuration?
22:18 indigoblu with beefy enough hardware and sequential read/writes I have always bottlenecked on the network.
22:18 dijuremo So I have glusterfs running on two nodes and have a third node for quorom but no bricks
22:18 dijuremo Each server has Areca raid controller and 12 SAS 7200rpm drives
22:19 dijuremo I seem to be having a horrible performance acting on lots of small files...
22:19 indigoblu what gluster version?
22:19 dijuremo For example, doing and ls -lR of a few thousand files is super slow
22:19 dijuremo 3.6.4
22:19 indigoblu FWIW I think there are some readahead tunable that got better with small files in 3.5 and later
22:20 dijuremo I tried mounting with direc-io=disable and the performance got even worse
22:20 dijuremo Not sure what tunables I need...
22:21 dijuremo # gluster volume info export | grep perf
22:21 dijuremo performance.cache-size: 256MB
22:21 dijuremo performance.io-thread-count: 32
22:21 calavera joined #gluster
22:21 dijuremo That is what I tried so far
22:21 indigoblu err
22:21 indigoblu looks like there was some changes in 3.7 as well
22:22 dijuremo So if I do ls -lR on the mounted brick... I get:
22:22 dijuremo time ( ls -lR .winprofile.V2 | wc )
22:22 dijuremo 3799   24797  206200
22:22 dijuremo real 0m3.311s
22:23 dijuremo But if I do it on the mounted gluster volume, it takes an order or magnitude higher...
22:24 indigoblu all timely single threaded meta-data lookups I think
22:24 dijuremo time ( ls -lR /home/miguel/.winprofile.V2/ | wc )
22:24 dijuremo 3799   24797  211568
22:24 dijuremo real 0m19.651s
22:25 dijuremo This time was not so bad cause of cache, since I just had recently ran the other one in the brick...
22:25 dijuremo The issue is that there is so much delay that it takes some users with roaming profiles so long to just log in and out of computers...
22:26 indigoblu I only use gluster for large dumb storage and qemu/kvm backed stores.  Both cases low in metadata ops...but lets see
22:26 dijuremo Both servers are connected via 10gbps for gluster and separate 10 gbps connections for the IPs serving samba
22:27 cleong joined #gluster
22:27 dijuremo My large file performance is excellent, I can max out reads and writes to the client's nics which are gigabit
22:27 dijuremo I am using vfs gluster module on samba, so cannot blame fuse...
22:28 indigoblu I have an 8 node / 12 disk array chock full with data and ls ops have always been slow
22:28 dijuremo I know ls also usually is aliased to sort, so I have also tried using find....
22:29 indigoblu If I wanted small file performance I would use Ceph
22:29 dijuremo Well, I sort of need a good compromise...
22:30 dijuremo This is for people working with revit files which are usually hundreds of megabytes...
22:31 dijuremo In the past, I ran drbd/heartbeat and that worked very well. Switched to gluster when I ran into issues with the file system I was running on top of drbd
22:32 indigoblu have you read this thread? http://www.gluster.org/community/documentati​on/index.php/Features/Feature_Smallfile_Perf
22:35 dijuremo No, I had not, will get on it!
22:39 dijuremo Ok, so I guess I will have to upgrade to 3.7.x to test since some of those features are implemented there...
22:39 topshare joined #gluster
22:45 JoeJulian xoritor: nothing's ever as fast as I want it to be.
22:46 JoeJulian dijuremo: what kernel are you using?
22:48 JoeJulian indigoblu: ceph is useless for small files. cephfs is not production ready, block devices are useless as a clustered filesystem, and what's left is object storage - which, again, is not really all that performant.
22:50 JoeJulian btw, dijuremo, hundreds of megabytes are not small files.
22:50 topshare joined #gluster
22:51 JoeJulian small files are files in which the self-heal overhead (a few bytes done at lookup, basically 1 network rtt to both bricks) is a significant portion of the iop.
22:53 JoeJulian If it's "open, write a few bytes, close", then yeah, that performance will see overhead. If you can leave the file open you should be able to do your iops at wire speed.
22:58 dijuremo @JoeJulian, I referenced hundreds of megabytes on the files where I get good performance...
22:58 dijuremo I can fill up the NICs on reads and writes with those files...
22:58 dijuremo My problem is with roaming profiles, IE Cookie files anyone?
22:59 dijuremo The machines are both CentOS 7 with the latest kernel:
22:59 dijuremo [root@ysmha01 ~]# uname -a
22:59 dijuremo Linux ysmha01.ad.ysmdesign.com 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
22:59 JoeJulian Ugh, cookie files suck. I hate microsoft.
22:59 dijuremo But then again any other type of small files from adobe, autodesk, etc...
23:00 JoeJulian The way I handled that was to eliminate IE and exclude the ie profile from the roaming profiles.
23:00 dijuremo I know I can run cron jobs to erase the cookie files, but still with a couple thousand files on a standalone server it was super fast..
23:00 dijuremo I am looking into settings in GPO to clear and lower maximum cache sizes for all browsers, firefox/IE/Chrome
23:01 JoeJulian I made my users use firefox and put the profile on their user directory. No offline availability, but they were never offline anyway.
23:01 JoeJulian I did the same for the "My documents" folder.
23:01 dijuremo Well, firefox on network share could also backfire on you... lots of sqlite files can cause a lot of i/o
23:02 dijuremo I have folder rediretion for everything, except the roaming profile part...
23:02 JoeJulian It met ,,(Joe's performance metric)
23:02 glusterbot nobody complains.
23:02 JoeJulian I also excluded most adobe caches from the roaming profile.
23:02 JoeJulian They're pigs.
23:02 dijuremo I tried folder redirection of roaming profiles, some stupid badly written apps would not run, and firefox was so awfully slow
23:02 dijuremo Right, we have adobe, autodesk, web browser and many more caches..
23:03 dijuremo I will have to keep looking into those things...
23:03 dijuremo come out with a compromise on what users really need to roam and what they do not...
23:03 dijuremo And I guess give 3.7.x a try...
23:03 JoeJulian Most things that need caches do it for performance reasons, not for state. Since you're seeing negative performance, no reason to sync those.
23:04 dijuremo But these files get copied once on login and then once on log off, the reason I do it is so the users have the same experience regardless of the computer they use..
23:05 dijuremo Does anybody have any idea of what version of gluster will Red Hat release for RHEL 7/ Red Hat Storage 3.1?
23:05 JoeJulian Right, but if they're logging on to pandora, the time it takes to load the swf from cache vs from pandora.com (only once per machine) is pretty insignificant.
23:06 dijuremo I sort of wanted to keep this install as much vanilla as possible so I did on purpose not update to 3.7.x cause i think RHEL7 will have the 3.6.x series of gluster...
23:06 JoeJulian So far, RHS has taken bits of pre-release and put their own (competing) version numbers to it.
23:06 dijuremo And I am also running everything for this 30 user office in these two servers... including ovirt
23:07 JoeJulian nice
23:07 dijuremo So I run actually 3 gluster volumes in the servers...
23:08 dijuremo One volume for the self hosted engine and vm storage on top of ssds, and one with hdds for the file server
23:10 dijuremo See you guys later, gotta drive now...
23:10 JoeJulian later
23:10 social joined #gluster
23:23 plarsen joined #gluster
23:42 aaronott joined #gluster
23:44 cyberswat joined #gluster
23:53 cyberswat joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary