Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-05-26

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 maduser joined #gluster
00:38 yinyin_ joined #gluster
00:52 plarsen joined #gluster
00:52 gdubreui joined #gluster
01:00 primechuck joined #gluster
01:29 theron joined #gluster
01:50 haomaiwang joined #gluster
02:04 bala joined #gluster
02:14 Scott6 joined #gluster
02:32 ceiphas__ joined #gluster
03:01 primechuck joined #gluster
03:27 kshlm joined #gluster
03:38 shubhendu joined #gluster
03:50 itisravi joined #gluster
04:05 RameshN joined #gluster
04:15 ppai joined #gluster
04:16 meghanam joined #gluster
04:24 shubhendu joined #gluster
04:30 dusmant joined #gluster
04:33 kdhananjay joined #gluster
04:35 cyber_si joined #gluster
04:37 spandit joined #gluster
04:38 rahulcs joined #gluster
04:39 davinder6 joined #gluster
04:49 bala joined #gluster
04:49 ramteid joined #gluster
04:56 ndarshan joined #gluster
05:01 primechuck joined #gluster
05:03 bala joined #gluster
05:05 haomaiwang joined #gluster
05:06 vpshastry joined #gluster
05:08 fidevo joined #gluster
05:14 _Bryan_ joined #gluster
05:14 nishanth joined #gluster
05:25 lalatenduM joined #gluster
05:33 prasanthp joined #gluster
05:36 hagarth joined #gluster
05:41 itisravi_ joined #gluster
05:42 hagarth1 joined #gluster
05:42 itisravi_ joined #gluster
05:43 nshaikh joined #gluster
05:43 davinder6 joined #gluster
05:46 RameshN joined #gluster
05:48 haomaiwa_ joined #gluster
05:48 prasanthp joined #gluster
05:52 prasanthp joined #gluster
05:57 itisravi joined #gluster
05:57 hagarth joined #gluster
06:02 hagarth1 joined #gluster
06:03 ravindran1 joined #gluster
06:06 ProT-0-TypE joined #gluster
06:09 bala1 joined #gluster
06:12 prasanthp joined #gluster
06:14 ricky-ti1 joined #gluster
06:17 psharma joined #gluster
06:18 kanagaraj joined #gluster
06:21 hagarth joined #gluster
06:23 davinder7 joined #gluster
06:24 sjm joined #gluster
06:31 zerick joined #gluster
06:32 kshlm joined #gluster
06:33 meghanam joined #gluster
06:35 vimal joined #gluster
06:46 rahulcs joined #gluster
06:47 kaushal_ joined #gluster
06:52 rahulcs_ joined #gluster
06:58 tjikkun joined #gluster
07:00 ctria joined #gluster
07:00 Pupeno joined #gluster
07:02 primechuck joined #gluster
07:05 Philambdo joined #gluster
07:05 ramteid joined #gluster
07:15 hybrid512 joined #gluster
07:23 keytab joined #gluster
07:25 ProT-0-TypE joined #gluster
07:40 edward1 joined #gluster
07:41 Thilam joined #gluster
07:46 prasanthp joined #gluster
07:47 fsimonce joined #gluster
07:49 ngoswami joined #gluster
07:49 borreman_dk joined #gluster
08:00 ktosiek joined #gluster
08:04 glusterbot New news from newglusterbugs: [Bug 1101111] [RFE] Add regression tests for the component geo-replication <https://bugzilla.redhat.com/show_bug.cgi?id=1101111> || [Bug 1094815] [FEAT]: User Serviceable Snapshot <https://bugzilla.redhat.com/show_bug.cgi?id=1094815>
08:04 ravindran1 left #gluster
08:14 ceiphas__ joined #gluster
08:14 rgustafs joined #gluster
08:15 andreask joined #gluster
08:22 cyberbootje joined #gluster
08:25 mbukatov joined #gluster
08:38 Slashman joined #gluster
08:48 davinder9 joined #gluster
08:49 monotek joined #gluster
09:03 primechuck joined #gluster
09:04 glusterbot New news from newglusterbugs: [Bug 1101138] meta-data split-brain prevents entry/data self-heal of dir/file respectively <https://bugzilla.redhat.com/show_bug.cgi?id=1101138>
09:07 liquidat joined #gluster
09:08 davinder11 joined #gluster
09:25 glusterbot New news from resolvedglusterbugs: [Bug 851451] meta-data split-brain prevents entry/data self-heal of dir/file respectively <https://bugzilla.redhat.com/show_bug.cgi?id=851451>
09:42 davinder11 joined #gluster
09:46 keytab joined #gluster
09:47 aravindavk joined #gluster
10:25 hagarth joined #gluster
10:32 itisravi joined #gluster
10:49 olisch joined #gluster
10:58 tdasilva joined #gluster
11:03 edward1 joined #gluster
11:03 davinder11 joined #gluster
11:04 primechuck joined #gluster
11:12 rahulcs joined #gluster
11:15 bala1 joined #gluster
11:17 vikumar joined #gluster
11:19 olisch joined #gluster
11:20 DV joined #gluster
11:24 ppai joined #gluster
11:28 harish joined #gluster
11:44 diegows joined #gluster
12:06 1JTAASMP2 joined #gluster
12:08 cyberbootje joined #gluster
12:09 nshaikh joined #gluster
12:14 rastar joined #gluster
12:15 mmorsi1 joined #gluster
12:35 lalatenduM joined #gluster
12:39 sroy_ joined #gluster
12:43 andreask joined #gluster
12:56 harish joined #gluster
13:01 ctria joined #gluster
13:05 primechuck joined #gluster
13:08 wushudoin joined #gluster
13:09 hagarth joined #gluster
13:15 davinder11 joined #gluster
13:16 wushudoin joined #gluster
13:19 rgustafs joined #gluster
13:23 vpshastry joined #gluster
13:28 vpshastry joined #gluster
13:34 dusmant joined #gluster
13:54 liquidat joined #gluster
14:01 monotek left #gluster
14:08 deeville joined #gluster
14:11 rahulcs joined #gluster
14:12 sroy_ joined #gluster
14:13 deeville does anyone here have experience of ZFSonlinux + gluster?
14:17 Scott6 joined #gluster
14:21 ilbot3 joined #gluster
14:21 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
14:27 yinyin_ joined #gluster
14:37 vimal joined #gluster
14:45 getup- joined #gluster
14:46 getup- hello, when we add new pairs of bricks to gluster in a replica set of 2, will those bricks always be replicas of eachother? Is there some way to check who's a replica of who?
14:49 ndevos getup-: yes, when you add more replica-pairs, each pair will become a subvolume for the distribution logic, and both bricks of the pair will have the same contents
14:49 ndevos getup-: there is a tool called lsgvt on forge.gluster.org, that can be used to visualize the layout of the volumes
14:50 getup- ndevos: cool, sounds good, i'll give it a go
14:51 ndevos getup-: basically, its this script https://forge.gluster.org/lsgvt/lsgvt/blobs/raw/master/lsgvt
14:51 getup- ndevos: that's brilliant, thanks
14:52 ndevos getup-: I've also put it in an rpm, you can get it through http://copr.fedoraproject.org/coprs/devos/gluster-addons/
14:52 glusterbot Title: devos/gluster-addons Copr (at copr.fedoraproject.org)
14:52 lmickh joined #gluster
14:56 yinyin_ joined #gluster
14:58 karimb joined #gluster
15:02 getup- ndevos: we're a debian company most of the times, but thanks anyway :)
15:09 vpshastry joined #gluster
15:10 Thilam left #gluster
15:13 Thilam joined #gluster
15:13 Thilam I, is there someone who can help me with a quota issue ?
15:18 Thilam when I define a quota on a folder and when I write stuff in it, it takes to much time to block the write once the quota is reached
15:19 plarsen joined #gluster
15:21 liquidat joined #gluster
15:28 vpshastry joined #gluster
15:40 kanagaraj joined #gluster
15:52 harish joined #gluster
16:16 andreask joined #gluster
16:26 rotbeard joined #gluster
16:27 wushudoin joined #gluster
16:29 bala joined #gluster
16:45 sjm joined #gluster
16:45 recidive joined #gluster
16:53 gmcwhistler joined #gluster
16:56 Scott6 left #gluster
17:18 lalatenduM joined #gluster
17:39 calum_ joined #gluster
17:53 rgustafs joined #gluster
17:54 ramteid joined #gluster
18:04 Pupeno joined #gluster
18:09 karimb joined #gluster
18:15 bala joined #gluster
18:40 karimb joined #gluster
18:48 zerick joined #gluster
18:51 karimb joined #gluster
18:52 daMaestro joined #gluster
19:00 Pupeno_ joined #gluster
19:01 theron joined #gluster
19:02 semiosis :O
19:05 ry joined #gluster
19:07 semiosis JoeJulian: you around?
19:10 y4m4 joined #gluster
19:12 rahulcs joined #gluster
19:12 liammcdermott joined #gluster
19:15 liammcdermott I've been getting into Gluster and learning as I go, I've got a Gluster cluster running but tried to cheap out and run glusterfsd on the same servers as our Web heads.
19:15 liammcdermott Would I be right in thinking this was a tremendously bad idea?
19:17 semiosis liammcdermott: I'm concerned by the terms you're using, "run glusterfsd," what version of glusterfs are you using?
19:17 liammcdermott Sorry, I'll check my terms, hang on...
19:17 semiosis liammcdermott: see ,,(glossary)
19:17 glusterbot liammcdermott: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
19:18 semiosis just worried you might have followed one of the many old/outdated HOWTOs hanging around the internets
19:19 liammcdermott Running glusterfs-server 3.4.2, glusterfsd is simply the name of the process as I see it in top
19:19 semiosis ok you're good
19:19 semiosis 3.4.3 is the latest in the 3.4 series though, you might want to upgrade
19:19 liammcdermott Phew, thanks for checking, and I can see why, I noticed the proliferation of bad HOWTO's
19:19 karimb joined #gluster
19:20 semiosis anyway, to your question, lots of people do run their storage & web/app clusters on the same hardware
19:21 liammcdermott semiosis, ah, thanks that's good to know.
19:21 semiosis however
19:21 semiosis you need to take some precautions
19:22 semiosis first of all, where do writes come from?  i.o.w. are your web server processes read-only, or do all the web servers write to the gluster volume?
19:23 semiosis there's a possibility for split-brain if your servers lose contact with each other but continue writing into their local part of the gluster volume
19:24 semiosis two common ways to prevent this are: 1, make the web/app processes use a read-only mount, if possible, and only do writes from a special writeable client mount point, or 2, use quorum on the volume, which is possible with replica 2 but best with replica 3
19:24 liammcdermott The Web server processes can all write to the gluster, interesting.
19:25 liammcdermott semiosis, woah, okay thanks. This is really helpful information.
19:25 liammcdermott What happens for us is: whenever we do a rollout of a new version of our code it creates a new directory + complete code checkout, then moves all the site-specific files to that new directory...
19:25 liammcdermott That can be up to 500MB of files being moved, so glusterfs and glusterfsd use all the CPU on the web heads
19:26 semiosis hmmm, moves
19:26 liammcdermott Could that be due to split-brain, or is it just because we're doing a lot of writes?
19:26 semiosis that *might* be fine, but you should be aware of how move/rename works on glusterfs...
19:26 semiosis well, before i go into that, can you please ,,(pasteinfo)
19:26 glusterbot Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:27 liammcdermott I'll do that now.
19:27 plarsen joined #gluster
19:28 liammcdermott https://dpaste.de/7Pq2
19:28 glusterbot Title: dpaste.de: Snippet #269430 (at dpaste.de)
19:28 liammcdermott semiosis, ^^
19:29 liammcdermott I tried to keep things pretty vanilla. :)
19:29 semiosis ah, replica 3, thats good for quorum!  also, no distribution, that's good for renames, i think
19:30 liammcdermott Is there a special setting we need for quorum, or is it just something that happens?
19:30 semiosis as far as I know move/rename on gluster doesnt actually move data, but instead creates a special "linkfile" in the volume that points the new name to the old name, where teh data still resides
19:31 semiosis but in a pure-replicate volume (without distribution) that may not be the case.  you could look into your bricks to find out
19:31 liammcdermott There's a lot of copying involved in the process too
19:32 semiosis you need to enable quorum, see "gluster volume set help" for available options
19:32 liammcdermott Okay, thanks.
19:32 semiosis yw
19:33 liammcdermott We were about to move gluster onto two servers by itself, I take it it would be better to put it on 3 (for quorum)
19:33 liammcdermott ?
19:34 liammcdermott And (if you don't mind another question, please) is it usual during periods of lots of move/copy operations for the gluster server (and client) process to take up as much as 80% CPU?
19:36 semiosis quorum will turn a client mount read-only if it can't write to a majority of the replicas, so with replica 2 any outage/partition will turn the client(s) read-only
19:36 semiosis you should of course test all this out and not just take my word for it
19:36 semiosis i dont even use quorum myself!
19:37 semiosis as for the cpu load, i really dont know, my workload is write-once-read-many, so i dont do copy/move
19:38 liammcdermott This is what top was saying: http://i.imgur.com/67xfU8u.png
19:38 liammcdermott But yeah, no worries if you're not sure. I guess if Gluster needs lots of CPU, then fair enough!
19:38 sroy_ joined #gluster
19:39 liammcdermott semiosis, oh you run the gluster ppa!
19:40 liammcdermott That's cool, now we can upgrade.
19:40 liammcdermott semiosis, also just in case we get really stuck, do you know of any Gluster consultants (who would also be willing to work on Ubuntu)?
19:41 semiosis yep, i'm the ,,(ppa) guy
19:41 glusterbot The official glusterfs packages for Ubuntu are available here: 3.4 stable: http://goo.gl/u33hy -- 3.5 QA: http://goo.gl/Odj95k -- introducing QEMU with GlusterFS 3.4 support: http://goo.gl/7I8WN4
19:42 semiosis bug me here if you find problems with the packages, or for any feedback really
19:42 liammcdermott semiosis, well thanks very much for taking that work on.
19:42 semiosis my pleasure
19:43 semiosis i started doing it because i needed debs when there were none (up to date)
19:43 liammcdermott Scratching your own itch, eh? :)
19:43 semiosis right
19:45 semiosis as for consulting, sorry I'm not up to speed on that, and I'm not available for hire right now
19:45 semiosis maybe JoeJulian would know, but he doesnt seem to be around at the moment
19:46 semiosis ah i need to update that
19:46 liammcdermott semiosis, thanks for the tip, I may ping him later. :)
19:47 karimb joined #gluster
19:47 semiosis he'll see this sooner or later, if you're still in the channel he'll mention you if he has anything to add
19:48 liammcdermott Great, and if he has anything to add about high CPU during lots of copies/moves that would be great, too!
19:48 semiosis @forget ppa
19:48 glusterbot semiosis: The operation succeeded.
19:49 semiosis @learn ppa as The official glusterfs packages for Ubuntu are available here: 3.4 stable: http://goo.gl/u33hy -- 3.5 stable: http://goo.gl/cVPqEH -- introducing QEMU with GlusterFS 3.4 support: http://goo.gl/7I8WN4
19:49 glusterbot semiosis: The operation succeeded.
19:58 karimb joined #gluster
20:05 liammcdermott semiosis, thinking about it, the vast majority of those heavy copy/move ops happen on the one server...
20:07 liammcdermott Do you have any big jobs, like copying lots of files, creating/extracting a massive tarball, in your workload (even if it's done on the one server, then copied out to the others)?
20:08 sputnik13 joined #gluster
20:13 edward1 joined #gluster
20:13 semiosis i have photo/video processing apps write into the volume from their local scratch space, then web/app servers read the files to serve them to the web
20:14 edward1 joined #gluster
20:15 liammcdermott semiosis, interesting, how much CPU do you find Gluster needs when a large photo/video is written to the volume?
20:15 liammcdermott (if you don't mind me asking)
20:15 semiosis if we get a large batch of uploads the workers could be writing ~50-100 files into the volume in parallel, on rare occasions this could spike to 100s, maybe 1000s, of parallel files being created, mostly JPEGs in the 0.1 to 1 MB range
20:16 semiosis tbh i use ,,(joes performance metric) and don't usually pay attention to the numeric load metrics
20:16 glusterbot I do not know about 'joes performance metric', but I do know about these similar topics: 'Joe's performance metric'
20:16 wushudoin joined #gluster
20:16 semiosis yeah, ,,(Joe's performance metric)
20:16 glusterbot nobody complains.
20:17 liammcdermott Sounds similar to the files we're writing when the site is copied over to a new version of our codebase.
20:17 semiosis i've scaled up to about 50 worker threads, each writing dozens of files at a time, and the gluster servers (two) dont seem to notice
20:18 liammcdermott semiosis, you don't look at the %CPU at all?
20:20 liammcdermott That is a pretty good performance metric though, and I wholeheartedly endorse its usage. :)
20:20 semiosis really haven't had to look at cpu util in a long time
20:23 liammcdermott That does lead me to worry that we're doing something wrong... I wonder why we'd get such high %CPU?
20:23 semiosis my workload is pretty much ideally suited for glusterfs, it's quite a good match for us!
20:24 semiosis high cpu isn't necessarily bad... i would say an idle cpu is a waste of $$$
20:25 semiosis if the performance is within your tolerance then dont worry about some silly numbers
20:25 liammcdermott Well yes, but it's bringing the web site down. :)
20:25 liammcdermott People are complaining! :(
20:25 liammcdermott :)
20:25 semiosis ahhh
20:26 liammcdermott So what we're planning to do is move Gluster onto its own servers.
20:26 wushudoin left #gluster
20:26 liammcdermott Which sounds like we'd have something similar to your setup.
20:26 semiosis tell me more about your setup.... what kind of network connects the servers?  what kind of disks?  what kind of web app?
20:27 karimb joined #gluster
20:29 liammcdermott It's an Amazon EC2 VPC, the disks are whatever Amazon gives you by default (though our new servers are going to be high IOPS/fast network), the app is a simple Drupal blog (PHP & MySQL) with Varnish in front... that gets hammered; sometimes by 90,000 simultaneous users.
20:29 semiosis ok, varnish is good, you're on the right track.  i hope/assume you're using glusterfs FUSE mounts, not NFS, right?
20:30 semiosis there's a few things you can do to optimize PHP for glusterfs
20:30 semiosis see ,,(php)
20:30 glusterbot (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
20:30 glusterbot --fopen-keep-cache
20:30 liammcdermott Hell yeah, we started out with proper NFS and it was a disaster, so I don't even want to touch the protocol! :)
20:31 liammcdermott The fewest simultaneous users we get is ~400 and I think that's part of the problem, whenever we roll out a new version of our code (with all the copying and moving) the caches are also invalidated, so Gluster is not only busy writing, PHP is going ape doing reads.
20:31 semiosis you should optimize your php include path, put the most common path first (like the path to your framework) this will reduce the number of lookups when PHP tries to include/require a file
20:32 semiosis if drupal/your app supports it, you should use autoloading and eliminate all the require/include calls in the code.  most modern php frameworks do this (i'm not sure about drupal)
20:33 semiosis but even if your framework/app dont behave well (if they have to do tons of require/include) you can still use APC to mitigate the performance problems
20:33 liammcdermott Drupal makes heavy use of autoloading... in the next version. :)
20:34 liammcdermott Yes, we have APC thanks, pretty much a requirement for Drupal to run decently. The other stuff is new to me though, so that's wonderfully useful information.
20:34 semiosis check out the stat option in APC, you can use APC to forcibly cache php code and prevent lookups (stat calls) to the filesystem.  this will give a major perf boost, but at the cost that you'll need to reload (or maybe restart) apache whenever your php src files change
20:35 liammcdermott semiosis, pretty sure I turned that on when we were using NFS, I'll double-check though, good call.
20:39 karimb joined #gluster
20:41 semiosis liammcdermott: oh another tip is to mount your glusterfs bricks with 'noatime,nodiratime' since you most likely don't need atimes
20:42 liammcdermott semiosis, good point, not sure if we're doing that.
20:42 semiosis @php 2
20:42 glusterbot semiosis: I do not know about 'php 2', but I do know about these similar topics: 'php'
20:42 semiosis @php #2
20:42 glusterbot semiosis: I do not know about 'php #2', but I do know about these similar topics: 'php'
20:43 semiosis those mount options in part 2 of ,,(php) are new to me, but if glusterbot says it, then it's worth trying
20:43 glusterbot (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
20:43 glusterbot --fopen-keep-cache
20:43 liammcdermott I think we're doing all or most of the things mentioned in JoeJulian's blog post, so I'll try moving Gluster onto its own servers and change those mount options.
20:45 semiosis today is a us holiday so seems like not many people are around, but hang out here for a couple days to get more input from others
20:45 semiosis i'm going afk for a while.  good luck!
20:45 liammcdermott semiosis, thanks very much!
20:45 semiosis yw!
21:06 karimb joined #gluster
21:35 karimb joined #gluster
21:39 wushudoin joined #gluster
21:46 karimb joined #gluster
21:55 karimb joined #gluster
21:58 theron joined #gluster
22:10 edward2 joined #gluster
22:25 karimb joined #gluster
22:58 Ark joined #gluster
23:02 fidevo joined #gluster
23:35 primechuck joined #gluster
23:43 MugginsM joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary