Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-06-25

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:21 Ark joined #gluster
00:28 georgeh|workstat joined #gluster
00:29 dtrainor joined #gluster
01:08 glusterbot New news from resolvedglusterbugs: [Bug 987555] Glusterfs ports conflict with qemu live migration <https://bugzilla.redhat.com/show_bug.cgi?id=987555>
01:11 georgeh|workstat joined #gluster
01:13 siel joined #gluster
01:33 georgeh|workstat joined #gluster
01:38 glusterbot New news from resolvedglusterbugs: [Bug 764655] NetBSD port <https://bugzilla.redhat.com/show_bug.cgi?id=764655>
02:08 harish joined #gluster
02:10 RioS2 joined #gluster
02:14 theron joined #gluster
02:30 gmcwhistler joined #gluster
02:33 baojg joined #gluster
02:36 rameez joined #gluster
02:37 rameez hello team
02:37 rameez requesting your help on a geo replication issue.
02:37 rameez i have two datacenters where the geo-replication is implemented.
02:38 rameez from A to B it is working as expected.
02:38 rameez but from B to A to (different set of volumes), it is spawning additional seessions.
02:39 rameez there are required Active sessions, But equal number of sessions goes in to defunct status.
02:40 Bullardo joined #gluster
02:40 theron joined #gluster
02:53 churnd joined #gluster
02:55 bharata-rao joined #gluster
02:56 haomaiw__ joined #gluster
02:56 baojg joined #gluster
03:07 sjm joined #gluster
03:14 vpshastry joined #gluster
03:15 theron_ joined #gluster
03:27 rejy joined #gluster
03:29 kdhananjay joined #gluster
03:30 gmcwhistler joined #gluster
03:31 RioS2 joined #gluster
03:47 shubhendu__ joined #gluster
03:49 atinm joined #gluster
03:55 sauce joined #gluster
03:57 kanagaraj joined #gluster
03:57 prasanthp joined #gluster
03:58 davinder15 joined #gluster
04:05 gmcwhist_ joined #gluster
04:13 itisravi joined #gluster
04:15 Pupeno joined #gluster
04:15 gildub joined #gluster
04:16 nishanth joined #gluster
04:22 baojg joined #gluster
04:22 ndarshan joined #gluster
04:27 RameshN joined #gluster
04:31 spandit joined #gluster
04:36 bala joined #gluster
04:38 vpshastry joined #gluster
04:40 ppai joined #gluster
04:46 rjoseph joined #gluster
04:48 vimal joined #gluster
04:56 lalatenduM joined #gluster
05:01 kshlm joined #gluster
05:02 baojg joined #gluster
05:03 Pupeno joined #gluster
05:04 nbalachandran joined #gluster
05:04 baojg joined #gluster
05:14 hagarth joined #gluster
05:16 psharma joined #gluster
05:18 aravindavk joined #gluster
05:23 kdhananjay joined #gluster
05:24 ppai joined #gluster
05:28 saurabh joined #gluster
05:32 Ark joined #gluster
05:45 Pupeno joined #gluster
05:49 aravindavk joined #gluster
05:51 bala1 joined #gluster
05:52 vkoppad joined #gluster
05:55 ws2k33 joined #gluster
05:55 hagarth joined #gluster
05:56 Pupeno joined #gluster
05:58 nhayashi joined #gluster
05:59 prasanthp joined #gluster
05:59 Philambdo joined #gluster
06:00 mbukatov joined #gluster
06:03 Pupeno joined #gluster
06:10 Pupeno joined #gluster
06:11 raghu` joined #gluster
06:19 pureflex joined #gluster
06:27 Pupeno joined #gluster
06:30 _polto_ joined #gluster
06:33 karnan joined #gluster
06:33 firemanxbr joined #gluster
06:35 rastar joined #gluster
06:35 ctria joined #gluster
06:38 shylesh__ joined #gluster
06:42 psharma_ joined #gluster
06:45 psharma__ joined #gluster
06:52 psharma_ joined #gluster
06:53 pureflex joined #gluster
06:57 aravindavk joined #gluster
06:58 hagarth joined #gluster
07:03 prasanthp joined #gluster
07:03 overclk joined #gluster
07:04 hflai joined #gluster
07:04 eseyman joined #gluster
07:04 ndarshan joined #gluster
07:07 vpshastry joined #gluster
07:08 kevein joined #gluster
07:12 ppai joined #gluster
07:17 keytab joined #gluster
07:18 hflai joined #gluster
07:20 hagarth joined #gluster
07:25 Swaroop joined #gluster
07:26 Swaroop Hello All, I need some help in setting up glusterfs with peer replication
07:26 Swaroop I would like to have the peers replicate each other instead of clients replicating data to peers
07:26 Swaroop Had anyone has this kind of setup?
07:29 ktosiek joined #gluster
07:32 fsimonce joined #gluster
07:37 glusterbot New news from newglusterbugs: [Bug 957917] gluster create volume doesn't cleanup after its self if the create fails. <https://bugzilla.redhat.com/show_bug.cgi?id=957917>
07:47 ppai joined #gluster
07:49 baojg joined #gluster
07:49 hflai joined #gluster
07:49 baojg joined #gluster
07:54 vpshastry joined #gluster
07:58 vpshastry joined #gluster
08:00 spandit joined #gluster
08:04 wgao joined #gluster
08:04 pasqd hello, how can i check if the heal process is done? without checking each brick's size?
08:05 pureflex joined #gluster
08:05 _polto_ joined #gluster
08:06 ricky-ti1 joined #gluster
08:07 glusterbot New news from newglusterbugs: [Bug 1112980] NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses <https://bugzilla.redhat.com/show_bug.cgi?id=1112980>
08:14 baojg joined #gluster
08:19 Sunghost joined #gluster
08:21 pureflex joined #gluster
08:23 Sunghost Hello, i have a question for debian wheezy <-> jessie and glusterfs. I actually run wheezy but i need the last xfsprogs which are only since jessie in repo.
08:25 Sunghost I installed gluster via own source list file, like it is described in the howto on the downloadfolder of gluster for debian. i have installed version 3.5.1beta2
08:26 Sunghost My question is can i upgrade from wheezy to jessie without problems or have i change the source file?
08:27 Sunghost Version in Jessie is 3.5.0-1 for amd64. When will latest 3.5.1 available?
08:27 fraggeln Sunghost: when its done ;)
08:27 fraggeln best anwser ever.. :)
08:28 Sunghost Yes quiet good answer ;)
08:30 Sunghost My question was more on timing of both packages - so came the package on the gluster downloadserver much earlier as on debian repo or nearly at the same time.
08:31 Sunghost So if it came at the same time, i could use the debian repos, but if the time between the releases differntiate much, than i whould use the extra source file.
08:34 dusmant joined #gluster
08:34 hagarth joined #gluster
08:35 meghanam joined #gluster
08:37 Slashman joined #gluster
08:37 meghanam_ joined #gluster
08:48 Sunghost Any infos to this timing of debian repo and upload on glusterfs fileserver?
08:52 ndarshan joined #gluster
08:55 overclk joined #gluster
08:59 liquidat joined #gluster
09:01 overclk joined #gluster
09:02 nishanth joined #gluster
09:03 spandit joined #gluster
09:03 karnan joined #gluster
09:07 overclk joined #gluster
09:15 tomased joined #gluster
09:17 ndarshan joined #gluster
09:22 nishanth joined #gluster
09:25 dusmant joined #gluster
09:26 nthomas joined #gluster
09:33 davinder15 joined #gluster
09:41 nthomas_ joined #gluster
09:48 Ark joined #gluster
09:53 nishanth joined #gluster
09:57 swebb joined #gluster
10:10 hagarth joined #gluster
10:14 gildub joined #gluster
10:20 aravindavk joined #gluster
10:22 pureflex joined #gluster
10:27 _polto_ joined #gluster
10:43 Nightshader2 joined #gluster
10:54 tdasilva joined #gluster
10:55 RameshN joined #gluster
11:05 tdasilva_ joined #gluster
11:07 coreping joined #gluster
11:09 77CAAJHTE joined #gluster
11:09 6JTAAI72K joined #gluster
11:10 edward1 joined #gluster
11:11 baojg joined #gluster
11:12 baojg joined #gluster
11:17 ppai joined #gluster
11:20 nshaikh joined #gluster
11:22 aravindavk joined #gluster
11:25 LebedevRI joined #gluster
11:27 keds joined #gluster
11:31 partner that topic was slightly discussed last night
11:31 partner as a user i of course assume all the releases to be built, otherwise i'm not sure what's the point of building anything
11:32 partner but i also know how the stuff works, volunteers doing the packaging and stuff so can't ask too much from them either
11:38 glusterbot New news from newglusterbugs: [Bug 1002907] changelog binary parser not working <https://bugzilla.redhat.com/show_bug.cgi?id=1002907> || [Bug 1113066] DHT: Log new layout of directory generated during directory self healing <https://bugzilla.redhat.com/show_bug.cgi?id=1113066>
11:41 simulx joined #gluster
11:43 fraggeln partner: the buildserver can build the packages as well? ;)
11:43 fraggeln I know both bamboo and jenkins has good support for building both .rpm's and .deb's
11:44 partner i would assume the project does not have such but individuals build packages wherever
11:46 diegows joined #gluster
11:52 kkeithley Our jenkins in running on CentOS 6. It'll build rpms in mock for el5 and el6. It has problems building Fedora and el7 rpms. I haven't seen any sign that it'll build .debs, but I may not have looked in the right place. Or maybe it'll do that if it's running on debian or ubuntu.
11:53 kkeithley That aside, we build gluster for Fedora and epel in the Fedora koji build system. That works pretty good, and the Fedora rpms go straight into Fedora.
11:55 kkeithley We'd like to build .debs for Debian and Ubuntu ourselves, and I believe we're getting closer to doing that. Until then we're still relying on volunteers who do that on their own time.
11:56 fraggeln kkeithley: I think you need a jenkins-slave running debian to build packages, but im not sure.
11:57 kkeithley Our jenkins is pretty outdated too. Once we get it updated to the latest-and-greatest it may do some of those things, or do them better.
11:59 fraggeln just wget the war-file and let tomcats autodeploy do the rest ;)
11:59 fraggeln jenkins is a pain in the ass some times, I have been running one for almost 2 years now, every now and then something breaks.
11:59 fraggeln jobs not showing up and stuff like that
12:00 fraggeln oh, you are running 1.515
12:00 fraggeln yea, it could use some upgrading.
12:04 kkeithley yep, it's in the works
12:05 fraggeln btw, green balls ftw ;)
12:05 fraggeln thats the default plugin you always install ;)
12:08 tdasilva joined #gluster
12:11 calum_ joined #gluster
12:12 fraggeln kkeithley: how much volontary work is done in relation to the work that redhat employees do?
12:17 kkeithley hard to say.
12:17 kkeithley because I don't know, and it's difficult to track
12:18 itisravi joined #gluster
12:20 fraggeln anyway, I like the product, its really nice, keep up the good work.
12:20 kkeithley thanks
12:21 fraggeln we get awesome performance on large files, now we just need to optimize it for smaller files.
12:21 kkeithley yeah, we hear that a lot. ;-)
12:22 fraggeln but if we manage to get good enough performance we will switch to gluster for our main website.
12:22 pureflex joined #gluster
12:23 kkeithley are you using PHP by any chance?
12:23 fraggeln and imho, that is really cool.
12:23 Nightshader joined #gluster
12:23 fraggeln well, we use php for some sites, but the main page runs escenic
12:23 kkeithley okay
12:23 deepakcs joined #gluster
12:24 fraggeln but, the webfronts will have local ssd-disks for scaled images, so it will not be that much load on the gluster I hope
12:27 pureflex joined #gluster
12:31 fraggeln kkeithley: is it possible to fix the "issues" with performance when dealing with small files, or is it impossible dueto the design?
12:35 fraggeln or is the solution to switch to infiniband between the servers? :)
12:35 kanagaraj joined #gluster
12:44 velladecin joined #gluster
12:46 coredump joined #gluster
12:48 sputnik13 joined #gluster
12:49 sputnik13 joined #gluster
12:49 aravindavk joined #gluster
12:51 sroy joined #gluster
12:53 kkeithley IIRC, jdarcy got a big increase in small file performance by only sending lookups to one server instead of all of them. Maybe we'll see that in 3.6.
12:55 deeville joined #gluster
12:55 fraggeln last question for the day, is it possible to use RoCE with gluster?
12:56 fraggeln regarding lookups, is that the client that does that or the servers itself? (in version 3.5)
12:57 theron joined #gluster
12:58 theron joined #gluster
12:59 bennyturns joined #gluster
12:59 kkeithley clients do lookups, generally
13:00 kkeithley gluster will use IBoIB. I haven't used RoCE myself, so I'm not sure. I'd think if RoCE looks like an ethernet/TCP-IP then GlusterFS would "just work", but can't say that with any authority
13:00 chirino joined #gluster
13:01 japuzzo joined #gluster
13:01 kkeithley IPoIB
13:03 lpabon joined #gluster
13:03 julim joined #gluster
13:07 fraggeln kkeithley: thank you very much for your time, have a very good day/morning/night o what best suits your timezone :)
13:08 glusterbot New news from newglusterbugs: [Bug 1101111] [RFE] Add regression tests for the component geo-replication <https://bugzilla.redhat.com/show_bug.cgi?id=1101111>
13:09 tty00 btw, have anyone of you tried to decrease the NIC queue/buffers so the NIC is flushing packets to the wire faster?
13:19 Ark joined #gluster
13:28 mjsmith2 joined #gluster
13:31 baojg joined #gluster
13:32 primechuck joined #gluster
13:44 fsimonce` joined #gluster
13:44 gmcwhistler joined #gluster
13:45 burnalot joined #gluster
13:52 nshaikh joined #gluster
13:53 fsimonce joined #gluster
13:53 Kedsta joined #gluster
14:02 marbu joined #gluster
14:03 theron joined #gluster
14:09 mortuar joined #gluster
14:10 jag3773 joined #gluster
14:16 hagarth joined #gluster
14:17 sjm joined #gluster
14:18 wushudoin joined #gluster
14:20 jezier_ hi.. we have couple of messages in logs like [server-rpc-fops.c:1739:server_stat_cbk] 0-storage-server: 2415179: STAT (null) (08521fc9-1824-415c-9972-eae65ac19c43) ==> (No such file or directory)... it looks like race condition.. file creation happens on one node and open/read on another node... when I verify after a couple of minutes, this file exists... how can we fix this? (ubuntu 12.04, glusterfs3.4.3)
14:21 hybrid512 joined #gluster
14:24 mjsmith2 joined #gluster
14:27 ultrabizweb joined #gluster
14:30 doc|holliday joined #gluster
14:31 doc|holliday is it safe to connect to cluster running 3.4.2 with clients running 3.4.4?
14:32 doc|holliday my clients (mount via fuse) keep leaking memory with 3.4.2 and echo 2 > .../drop_caches didn't help
14:33 doc|holliday want to try 3.4.4
14:34 lmickh joined #gluster
14:36 ndk joined #gluster
14:40 _polto_ joined #gluster
14:48 Guest41106 joined #gluster
14:49 Guest41106 Hi guys, I'm getting memory leak on glusterfs 3.5.0beta3 built on Feb 11 2014 20:45:24
14:49 Guest41106 and the kernel kills the process after it eats 7GB
14:50 fsimonce joined #gluster
14:52 Guest41106 quit
14:52 Guest41106
14:58 harish joined #gluster
15:02 kdhananjay joined #gluster
15:04 Ark joined #gluster
15:04 zaitcev joined #gluster
15:05 jdarcy joined #gluster
15:06 ndk` joined #gluster
15:10 mbukatov joined #gluster
15:11 gts joined #gluster
15:11 JustinClift *** Gluster Community Meeting time is now in #gluster-meeting on irc.freenode.net ***
15:16 lpabon joined #gluster
15:28 sage__ joined #gluster
15:28 _polto_ joined #gluster
15:30 dberry joined #gluster
15:30 dberry joined #gluster
15:39 shubhendu__ joined #gluster
16:04 lalatenduM joined #gluster
16:05 rwheeler joined #gluster
16:25 mjsmith2 joined #gluster
16:25 Mo_ joined #gluster
16:27 fsimonce joined #gluster
16:32 bala joined #gluster
16:34 jobewan joined #gluster
16:34 wushudoin joined #gluster
16:42 lmickh joined #gluster
16:43 sjm joined #gluster
16:43 zerick joined #gluster
16:46 coredump joined #gluster
16:47 sputnik13 joined #gluster
16:48 jag3773 joined #gluster
17:09 rwheeler joined #gluster
17:11 dusmant joined #gluster
17:15 dusmant joined #gluster
17:18 Matthaeus joined #gluster
17:19 daMaestro joined #gluster
17:22 sjm joined #gluster
17:31 lmickh joined #gluster
17:34 coredump joined #gluster
17:37 qdk joined #gluster
17:51 sjm joined #gluster
17:52 MacWinner joined #gluster
17:58 davinder15 joined #gluster
17:59 vpshastry joined #gluster
18:03 ramteid joined #gluster
18:12 _dist joined #gluster
18:16 mjrosenb joined #gluster
18:21 sjm joined #gluster
18:51 sjm joined #gluster
18:53 lmickh joined #gluster
18:57 dtrainor joined #gluster
18:59 dtrainor joined #gluster
19:02 n0de In Gluster 3.2.4, was there no .glusterfs/ directoy ?
19:06 semiosis sounds right
19:06 semiosis simpler times
19:10 rotbeard joined #gluster
19:12 sjm joined #gluster
19:20 calum_ joined #gluster
19:20 n0de Hi semiosis, this is the gluster in which I have a failed raid6 on for one of my bricks
19:21 semiosis do you have a question?
19:22 n0de yes, so the brick in question consists of software 4 x raid6 in a raid0
19:23 n0de one of those raid6s has failed and I will need to wipe it
19:23 n0de everything is mounted still and works, just the files on that failed raid6 will spit out i/o errors
19:23 n0de prior to wiping the failed raid6, I want to make sure I have the correct plan in place in order to avoid loosing any data.
19:25 n0de Volume type is Distributed-Replicate, it feels like if I simply fix the broken raid6, gluster will heal the files on it's own?
19:26 Paul-C joined #gluster
19:28 lpabon_ joined #gluster
19:36 bene2 joined #gluster
19:41 dtrainor joined #gluster
19:41 fraggeln is there any good way to optimize the clients? like sysctl-stuff or tcp-offloading stuff?
19:43 sjm joined #gluster
19:44 theron joined #gluster
19:50 tdasilva joined #gluster
19:51 semiosis fraggeln: is that really your bottleneck?
19:52 semiosis n0de: well what i would do is: kill the brick daemon for the badd brick.  firewall the port for the bad brick so nothing can connnect to it.  replace the drive/raid/whatever with a new one, formatted & mounted to the same place.  restart glusterd to respwn the brick daemon.  unfirewall the port
19:53 semiosis n0de: now you'll have an empty brick, and as files are accessed they will be healed on demand, in the background.
19:53 semiosis since this is pre-3.3 you'll need to stat each file to make it heal.  i recommend using the ,,(targeted self heal) strategy which I wrote just for this situation (though it's not needed since 3.3)
19:53 glusterbot https://web.archive.org/web/20130314122636/http://community.gluster.org/a/howto-targeted-self-heal-repairing-less-than-the-whole-volume/
19:54 dtrainor joined #gluster
19:54 semiosis summary of that is: use find on the good brick to stat files through a local mount
19:54 semiosis see article for details
19:54 semiosis s/local mount/local client mount/
19:54 glusterbot What semiosis meant to say was: summary of that is: use find on the good brick to stat files through a local client mount
19:55 n0de semiosis: I will do that, thank you for your help
19:55 semiosis yw
19:56 n0de Can I skip the firewalling step if I just stop gluster, fix raid, and then restart it?
19:56 n0de it == gluster
19:56 semiosis you need to stop the volume, gluster volume stop $volname
19:56 semiosis if you want to do that
19:56 semiosis it will stop all brick daemons on all servers
19:56 semiosis cutting off clients
19:56 n0de I rather not stop the volume
19:56 semiosis right
19:57 semiosis so the other way is to kill the brick daemon process & firewall the port (in case it gets restarted prematurely)
19:59 n0de got it
19:59 dtrainor joined #gluster
19:59 n0de looks like its port 9001 for glusterfs and port 8036 for glusterd
20:02 semiosis that doesnt sound right
20:03 semiosis should be something around 24009 or a bit above
20:03 semiosis ,,(ports)
20:03 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
20:04 n0de hrm, that is not what I am seeing
20:05 n0de on my gluster 3.4.2 setup, I see glusterd is on 28166 and glusterfs is on various 282xx ports
20:05 chirino joined #gluster
20:05 semiosis possible you're looking at process ids?
20:06 semiosis please pastie.org the output of 'sudo netstat -anp | grep gluster'
20:06 n0de crap, I am my apologies
20:06 semiosis lol
20:06 n0de did that thing again of being human :)
20:07 n0de glusterd == 24007 and glusterfs == 38465, 38466,38467
20:08 semiosis see ,,(processes)
20:08 glusterbot The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal).
20:08 semiosis you want to kill the 'glusterfsd' (brick daemon) for the bad brick, and firewall it's port, which will be 24009 or higher
20:09 semiosis when it's running its command line will clearly identify the brick path & the port number, use 'ps axwww | grep glusterfsd'
20:10 semiosis to see the full command line for all the brick daemons
20:14 sjm joined #gluster
20:15 n0de # ps axwww | grep glusterfsd
20:15 n0de 10892 pts/0    S+     0:00 grep --colour=auto glusterfsd
20:15 n0de got nothing it seems
20:16 n0de I see glusterfsd running on other bricks though
20:16 n0de the brick with the failed raid6 does not have glusterfsd running
20:18 bennyturns joined #gluster
20:19 n0de I assume the firewalling is done so that any new writes will be recorded as not successful on the brick I am working on?
20:20 semiosis without a brick daemon there's nothing to write to.  the firewall is a safeguard in case that brick daemon get started before you're ready for it, for example if you restart the system
20:23 sroy joined #gluster
20:23 fraggeln semiosis: yes, I think it is, since we have a shitload of small files :)
20:24 semiosis fraggeln: i'm still not convinced
20:24 fraggeln I am.
20:24 dtrainor joined #gluster
20:24 fraggeln 3 servers, gbit network, if I copy 1 4gb file, I get wirespeed.
20:25 semiosis whats the latency from client to the servers
20:25 semiosis ?
20:25 semiosis average rtt?
20:25 fraggeln they are in the same switch, so I guess its pretty low, cant check atm, im not connected to the vpn
20:25 _dist JoeJulian: when you're around just wondering about that vm healing issue I think you said it was on your list to look into :)
20:26 semiosis fraggeln: ok same switch should be <1ms
20:27 fraggeln 3 days of testing.
20:27 fraggeln :)
20:27 semiosis that's still likely to be the bottleneck though, context switches are ~1us
20:28 semiosis roughly speaking
20:29 fraggeln the problem is the workload, its a shitload of small files
20:29 semiosis how small?
20:29 fraggeln between 10k and 5mb
20:29 fraggeln around 90.000 of them ;)
20:29 JoeJulian _dist: Tried to duplicate it again yesterday...
20:31 fraggeln semiosis: we are a newssite, so we have a shitload of pictures ;)
20:31 semiosis JoeJulian: got any quick small-file optimization tips for fraggeln?
20:31 semiosis istr something about a scheculer recommendation?
20:31 fraggeln semiosis: we talked yesterday about it
20:31 semiosis we did?
20:32 fraggeln naa, me and JoeJulian
20:32 * semiosis goes to logs
20:32 semiosis ohhh
20:32 semiosis ok, still want to read that log tho
20:32 zaitcev left #gluster
20:32 fraggeln I did get a big performance-boost wen adding more bricks to the volume.
20:32 fraggeln when..
20:32 fraggeln version is 3.5, on debian wheezy.
20:33 semiosis i'll have 3.5.1 for wheezy available tonight
20:33 fraggeln oh!
20:33 semiosis dont ask whats new in 3.5.1, i dont know :)
20:33 fraggeln ill read the changelog later on, no worries.
20:34 semiosis let me know!
20:34 fraggeln Im gonna go and fetch my laptop, so I can hook the vpn up, and get the current config for the volume.
20:34 fraggeln brb
20:38 primechuck joined #gluster
20:38 primechuck joined #gluster
20:39 fraggeln semiosis: this is my config for the volume atm, http://fpaste.org/113242/72876214/
20:39 glusterbot Title: #113242 Fedora Project Pastebin (at fpaste.org)
20:40 fraggeln servers 1 and 3 have local disks, 6 sas-disks in raid10
20:40 fraggeln server 2 has a 8gbit FC-lun attached.
20:40 semiosis wow lots of options reconfigured
20:40 semiosis do those do anything for you?
20:40 fraggeln some dell compellent-thingy
20:40 fraggeln semiosis: like I said, 3 days of testing :)
20:41 semiosis neat
20:41 fraggeln I have tried a shitload of different options
20:41 fraggeln boxes themselvs are dual quadcores with 32gb ram
20:41 fraggeln so, it should not be HW-related.
20:46 SpeeR I'm having some problems cleaning up a split-brain problem, I have this file on 2 brick, I've remove the file, and the link-file, then stat the file.
20:46 SpeeR on my next heal info, it's back in the list
20:48 n0de semiosis: I will not be writing to the volume while the raid is getting fixed, nor will any of my clients.
20:48 n0de But will firewall regardless as a precaution
20:49 JoeJulian SpeeR: is it "back" in the list, or is it just "still" in the list? The list is a log with timestamps. As long at it's not newly in the list you're probably good.
20:50 SpeeR it disappears from the list every few seconds
20:50 SpeeR then comes back
20:50 JoeJulian If it disappears, that's a bug.
20:50 JoeJulian wait
20:50 theron joined #gluster
20:50 JoeJulian the "info split-brain" list or just the "info" list?
20:51 SpeeR just info
20:51 JoeJulian Oh. That probably just means it's being actively changed.
20:51 SpeeR ahh ok, so I should rely more on the info split-brain?
20:52 SpeeR ohh yeh, I see it's timestamped
20:53 SpeeR ok, that makes a lot more sense... so a file that is active can show up there and disappear from there
20:53 _dist JoeJulian: were you not able to duplicate it?
20:56 fraggeln semiosis: http://fpaste.org/113251/40372975/ <-- some "performance" messures
20:56 glusterbot Title: #113251 Fedora Project Pastebin (at fpaste.org)
20:56 _dist If you find that's the case, I don't mind setup a new cluster that has the issue and giving you VPN access to
21:02 semiosis fraggeln: can you ditch the raid altogether and try just plain old disks for bricks?
21:03 semiosis you already have gluster replica 3, why bother with raid?!
21:03 semiosis seems like it's just going to slow you down
21:03 fraggeln semiosis: its hw-raid, should not be a problem.
21:03 fraggeln and, there is no io-wait on the servers
21:03 semiosis worth a try
21:04 semiosis ok ok
21:04 semiosis suit yourself
21:04 * semiosis is biased against raid
21:04 fraggeln sorry, I was wrong, 0.2 wa,
21:04 semiosis a ha!
21:05 semiosis i mean, how could it *not* get in the way? :P
21:05 fraggeln bite me ;)
21:06 fraggeln not sure if that raidcontroller support jbod tho.
21:07 _dist semiosis: implemented correctly raid could actually speed things up. Personally I don't use HW raid but SW, however, I'd also rather replace a disk in my raid set than a brick in gluster as well.
21:07 fraggeln real6m45.909s <-- that was the time that it took to do a copy of the testworkload to local disks on one of the clients
21:07 _dist but, truthfully I never considered using multiple bricks on the same server as a type of sw raid
21:07 fraggeln _dist: i use raid10 atm, 6 disks
21:07 semiosis i like knowing if i have to get data off a disk it's as easy as mounting it
21:08 _dist yeap that's fair, especially with HW raid you can run into scenarios where md can't mount it
21:08 fraggeln _dist: thats why you have replicas, and a good backup-strategy
21:08 semiosis +1
21:08 fraggeln imho raid != backup.
21:09 fraggeln ahh, fuck, look at the time, I need to get my beautysleep.
21:09 _dist fraggeln: I totally agree, raid and gluster replica is just HA, not a backup. If you're using raid you just need to understand that your entire replica volume will be as slow as your slowest brick (raid set)
21:10 _dist (assuming there is no network bottleneck of course)
21:10 fraggeln _dist: yea, but disk-speed is not really the issue here, I think.
21:10 sjm joined #gluster
21:12 _dist I'd expect a raid10 of 6 would give you the read speed of 3 disks and the write speed of 2 disks. Your volume would never be faster than that, 1gbe would be a bottleneck, 2gbe probably not
21:12 _dist (depends on your disks of course)
21:12 fraggeln _dist: I max out 1gbit when I do a read or write of 1 large file
21:12 fraggeln my issue is when I try to handle a shitload of small files :)
21:13 semiosis fraggeln: you dont get any iowait when you have your small file activity storm?
21:13 _dist I'd test locally first, what is a shitload ? :)
21:14 fraggeln _dist: 40k small files packed into 8gb
21:14 fraggeln in test-folder
21:14 fraggeln but, in real world, when/if we go live, it will be like 900k files
21:15 _dist Well, replica bricks will help that, what kind of test are you doing, how are you mounting the volume?
21:15 fraggeln usingthe glusterfs client
21:15 semiosis @options
21:15 glusterbot semiosis: See config options and their defaults with 'gluster volume set help' and see also this page about undocumented options: http://goo.gl/mIAe4E
21:17 semiosis idk how but there might be some way to disable replication checking on the client
21:17 semiosis leave it up to the self heal daemon, hopefully
21:17 semiosis that should help quite a bit
21:17 fraggeln sounds risky ;)
21:17 fraggeln but, its worth a try.
21:17 semiosis for your read-heavy workload, shouldnt be too risky
21:17 fraggeln since I will not have that many writes
21:17 semiosis oh btw, did you use noatime,nodiratime on your brick mounts?
21:17 semiosis that's important too
21:18 fraggeln semiosis: I thought that noatime didnt really have any affect on xfs
21:18 _dist fraggeln: do your tests represent what's going to be going on in prod?
21:18 semiosis hmm, might not
21:18 semiosis idk
21:18 fraggeln _dist: yes, mostly reads.
21:19 fraggeln when someone is crawling our site, it goes through all links, and then it needs to resize files that are not in our cache, and that hurts :/
21:19 fraggeln and for some reason, people tend to do that alot :(
21:20 julim joined #gluster
21:21 fraggeln and the cms creates a metadatafile next to each picture, with worthless info, and that metadata-file needs to be read eveytime the picture needs to be scaled.
21:21 * fraggeln dont like Escenic.....
21:23 _dist fraggeln: This is older, but it may help
21:23 _dist http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/
21:23 glusterbot Title: Optimizing web performance with GlusterFS (at joejulian.name)
21:24 fraggeln _dist: I did read that one yesterday.
21:24 fraggeln doesnt really tell how to optimize glusterfs, just how to prevent using it, with the help of varnish,apc,memcached and stuff like that.
21:24 _dist I think the reverse proxy is a good idea in your case?
21:25 JoeJulian fraggeln: did you see the "dht misses are expensive" page?
21:25 JoeJulian Might have useful info too.
21:25 fraggeln JoeJulian: didnt read that one, ill have a look.
21:25 JoeJulian @lucky dht misses are expensive
21:25 glusterbot JoeJulian: http://joejulian.name/blog/dht-misses-are-expensive/
21:26 fraggeln _dist: we have enough caching ;)
21:26 fraggeln we are swedens biggest newspaper online ;)
21:26 JoeJulian nice
21:27 JoeJulian I've been wondering.
21:27 fraggeln Wondering what?
21:28 _dist JoeJulian: would it help if I gave you a test cluster with the vm heal issue to work with? I could probably set that up in an hour or so, (not today though)
21:28 JoeJulian fraggeln: wondering who you were.
21:29 fraggeln oh.
21:29 JoeJulian _dist: sure
21:29 fraggeln operationsmanager at Aftonbladet
21:29 fraggeln thats me
21:29 _dist ok, I'll see if I can have that setup before the end of the week and get you creds to get in :)
21:31 semiosis "The king stopped Aftonbladet from being printed and banned it. This was answered by starting the new newspaper "Det andra Aftonbladet" (The second Aftonbladet), which was subsequently banned, followed by new versions named in similar fashion until the newspaper had been renamed 26 times, after which it was allowed by the king."
21:31 semiosis from wikipedia
21:31 semiosis that's awesome
21:32 fraggeln ;)
21:32 JoeJulian lol
21:33 _dist alright, I'm outta here :) ttyl
21:33 JoeJulian o/
21:33 JoeJulian fraggeln: Ok, so what is the goal that's not being met? Let's try phrasing it into something quantifiable.
21:34 JoeJulian because, imho, "small files" and "too slow" are not.
21:37 _polto_ joined #gluster
21:37 _polto_ joined #gluster
21:53 _polto_ joined #gluster
21:53 _polto_ joined #gluster
22:03 RicardoSSP joined #gluster
22:03 RicardoSSP joined #gluster
22:12 JoeJulian fraggeln: I hope you didn't read that as adversarial. I just ask so I have some idea of a target to help you shoot for.
22:20 jcsp1 joined #gluster
22:37 sjm left #gluster
22:57 tdasilva joined #gluster
23:03 theron joined #gluster
23:06 primechuck joined #gluster
23:17 lpabon joined #gluster
23:19 lpabon joined #gluster
23:22 _polto_ joined #gluster
23:22 _polto_ joined #gluster
23:27 gildub joined #gluster
23:32 Ark joined #gluster
23:37 sjm joined #gluster
23:54 theron joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary