Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-05-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:13 yinyin_ joined #gluster
00:33 elico cyberbootje: I am not claiming anything but how do you see the status of the file?
00:35 lyang0 joined #gluster
00:37 vpshastry joined #gluster
00:45 plarsen joined #gluster
01:06 haomaiwang joined #gluster
01:09 harish joined #gluster
01:20 cyberbootje gluster volume heal volume1 info split-brain
01:23 bala joined #gluster
01:31 Franklu joined #gluster
01:47 elico cyberbootje: thanks. and...
01:47 elico the basic thing is to decide what file is the up-to-date and that one that you will manually decide to be the real one.
01:54 elico basically then you will need to rsync from one node to the other or scp or something similar (rsync preferred).
01:54 elico After copying the *main* file to all the nodes stat the file on the mounted NFS or glustefs FS.
01:58 elico another option probably would be to erase the defected or wrong copy of the files on the other side of the split-brain and then stat the file on the mounted FS.
01:59 elico note that I am not an expert and partially sure about it.
01:59 elico There is a video that explains couple things about the self-healing but not split-brain as far as I can tell.
02:08 haomaiwang joined #gluster
02:09 haomaiwa_ joined #gluster
02:10 haomaiwang joined #gluster
02:46 cogsu joined #gluster
02:48 Franklu Hi gluster experts, We saw gfid mismatch issue in our deployment, and our apache httpd processes are all hung in D state when accessing the files in the directory which has gfid-mismatch issue.
02:48 Franklu The httpd processes are even unkillable using "kill -9"
02:54 elico Franklu: and how is the gluster looks like?
02:54 elico how many nodes? etc..
03:22 MeatMuppet joined #gluster
03:23 SFLimey joined #gluster
03:29 rjoseph joined #gluster
03:31 badone_ joined #gluster
03:35 n0de_ joined #gluster
03:56 rahulcs joined #gluster
03:59 haomai___ joined #gluster
04:04 haomaiwa_ joined #gluster
04:05 vpshastry joined #gluster
04:05 vpshastry left #gluster
04:19 yinyin_ joined #gluster
04:36 hagarth joined #gluster
05:29 rahulcs joined #gluster
05:42 haomaiwa_ joined #gluster
05:43 SFLimey joined #gluster
05:46 Licenser joined #gluster
06:39 davinder joined #gluster
06:45 rjoseph joined #gluster
06:52 Debolaz2_ joined #gluster
06:58 vpshastry joined #gluster
06:59 edward joined #gluster
07:48 MrAbaddon joined #gluster
07:49 ThatGraemeGuy joined #gluster
08:03 edward joined #gluster
08:22 ilbot3 joined #gluster
08:22 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
09:42 vpshastry joined #gluster
10:26 haomaiwa_ joined #gluster
10:47 mgarcesMZ joined #gluster
11:11 scuttle_ joined #gluster
11:11 basso joined #gluster
11:12 decimoe joined #gluster
11:14 sprachgenerator joined #gluster
11:17 DV joined #gluster
11:18 social joined #gluster
11:18 tryggvil joined #gluster
11:51 DV joined #gluster
11:54 vpshastry joined #gluster
12:13 edward joined #gluster
12:49 tycho_ joined #gluster
12:52 davinder joined #gluster
12:59 vpshastry left #gluster
13:26 vpshastry joined #gluster
13:26 vpshastry left #gluster
13:39 haomaiwa_ joined #gluster
13:49 vpshastry joined #gluster
13:49 vpshastry left #gluster
13:49 plarsen joined #gluster
13:55 haomaiw__ joined #gluster
13:55 leo_ joined #gluster
13:56 leo joined #gluster
13:57 leo_ joined #gluster
13:59 glusterbot New news from newglusterbugs: [Bug 1086782] Add documentation for the Feature: oVirt 3.2 integration <https://bugzilla.redhat.co​m/show_bug.cgi?id=1086782>
14:02 haomaiwa_ joined #gluster
14:15 leo_ joined #gluster
14:18 haomaiw__ joined #gluster
14:21 kshlm joined #gluster
14:25 leo_ joined #gluster
14:58 vpshastry joined #gluster
15:30 haomaiwa_ joined #gluster
15:36 EWDurbin joined #gluster
15:57 diegows joined #gluster
16:08 sprachgenerator joined #gluster
16:25 glusterbot New news from resolvedglusterbugs: [Bug 962089] gluster native client write performance slower than normal <https://bugzilla.redhat.com/show_bug.cgi?id=962089>
16:32 XpineX joined #gluster
16:41 TvL2386 joined #gluster
16:56 vpshastry left #gluster
16:59 surabhi joined #gluster
17:00 qdk joined #gluster
17:23 rotbeard joined #gluster
17:25 glusterbot New news from resolvedglusterbugs: [Bug 921817] autogen should warn if pkg-config missing <https://bugzilla.redhat.com/show_bug.cgi?id=921817>
17:40 DV joined #gluster
17:48 scuttle_ joined #gluster
17:58 rahulcs joined #gluster
18:15 surabhi joined #gluster
18:54 rahulcs_ joined #gluster
19:17 dusmant joined #gluster
19:27 rotbeard joined #gluster
19:56 theron joined #gluster
20:18 d-fence joined #gluster
21:04 badone joined #gluster
21:15 jonnyno5 joined #gluster
21:15 jonnyno5 hello
21:15 glusterbot jonnyno5: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
21:16 jonnyno5 I need some help for using caching options in gluster clients as webservers
21:16 jonnyno5 please any advice?
21:16 jonnyno5 some advice?
21:22 mjsmith2 joined #gluster
21:23 jonnyno5 some support please
21:24 ghenry joined #gluster
21:24 ghenry joined #gluster
21:27 jiffe98 joined #gluster
21:38 theron joined #gluster
21:44 fidevo joined #gluster
21:54 d-fence joined #gluster
22:00 jonnyno5 someone knows about caching performance of gluster fs clients in a webserver enviroment?
22:00 jonnyno5 how to tunning gluster at reads-level
22:41 jonnyno5 helloooo?
22:46 ndevos ~php | jonnyno5
22:46 glusterbot jonnyno5: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
22:46 glusterbot --negative-timeout=HIGH --fopen-keep-cache
22:46 ndevos jonnyno5: ^ should give you some ideas on what kind of tuning you are looking for
22:48 jonnyno5 increase performance at reads-level
22:49 jonnyno5 my enviroment is 2 replicated nodes
22:49 jonnyno5 with 4 webserver with gluster client fs mounted
22:50 jonnyno5 a lot of apache process respawn due to low glusterfs serponse from briks
22:50 ndevos some users have better performance when mounting over nfs, maybe that is an option to try?
22:51 jonnyno5 nfs can read with concurrency from 2 replicated briks and works with failover ?
22:53 ndevos replication is not like a RAID-0, it does not really speed up reading
22:53 jonnyno5 i mount with gluster native client and failover and each node serves deliver half of conten requested by each client now
22:54 jonnyno5 i want to define cache
22:54 elico jonnyno5: what is the issue? slow read?
22:54 jonnyno5 optins
22:54 ndevos do they? that is interesting... I never noticed that
22:54 jonnyno5 slow read yes
22:55 ndevos is the contents changing often? maybe you can go the nfs+fscache route (there is not fscache for fuse)
22:55 jonnyno5 content may change each 60 secs
22:55 elico What is the size of the glusterfs volume? how many files?
22:56 ndevos and the size of the files?
22:56 jonnyno5 will be refres time for cache
22:56 jonnyno5 size is 10 GB
22:56 jonnyno5 2 replicated volumes of 10 GB
22:56 elico Then use NFS...
22:57 jonnyno5 mounted in /var/www in n websevers with gluster client
22:57 jonnyno5 NFS can do failover?
22:57 elico if you need failover then use a virtual-ip for the nfs..
22:58 elico there are couple options you should use while mounting the mount but it is better then glusterfs due to lots of things.
22:58 jonnyno5 virtual-ip with gluster and nfs client works?
22:58 elico yes
22:58 ndevos you have two storage server, use two floating-ips (redundancy), and mount using rrdns (distribution of load)
22:59 elico I would not use rrdns..
22:59 ndevos you can use pacemaker for the floating/virtual IPs, or ctdb
22:59 jonnyno5 in amazon EC2 could work?
22:59 elico (just my private idea not anything else)
22:59 ndevos elico: the combination of 2 x vip and rrdns makes it interesting :)
23:00 jonnyno5 amazon ELB is rrdns
23:00 jonnyno5 i can use it
23:00 elico where do you use this storage at?
23:00 elico on EC2?
23:00 ndevos with EC2 you need an elastic IP (I think?)
23:00 jonnyno5 yes for nodes
23:01 elico then you will need to be creative with all this thing since VIP is kind of ..well not the default by AWS if i'm not wrong.
23:01 ndevos not sure if there are scripts for ctdb/pacemaker that migrate virtual ip inside the amazon cloud...
23:01 jonnyno5 client webservers enviroment is autoscaled from 2 at 6 machines
23:01 jonnyno5 depending load average rules
23:02 ndevos nice!
23:02 elico what do you want to benefit from the volume eventually?
23:02 jonnyno5 but i have read issues because
23:02 jonnyno5 when a apache process request file of node
23:03 jonnyno5 an another
23:03 jonnyno5 an another
23:03 jonnyno5 an another
23:03 jonnyno5 there are a queue
23:03 jonnyno5 of reads
23:03 jonnyno5 and the respawn a lot of apache process and my server goes overloaded
23:04 jonnyno5 but nodes have not load
23:04 elico Then it might be another issue.
23:04 jonnyno5 i have aneckbotle in clients side
23:04 elico you can try first to see if nfs helps you..
23:04 elico and you can always switch the service from one node to the other manually.
23:04 jonnyno5 but i see some translator for enable caching
23:05 elico where?
23:05 jonnyno5 but i am so confused how works
23:05 elico (link)
23:05 jonnyno5 https://www.gluster.org/community/documen​tation/index.php/Translators/performance
23:05 ndevos I think it sounds more like a limitation in the fuse client...
23:05 glusterbot Title: Translators/performance - GlusterDocumentation (at www.gluster.org)
23:06 jonnyno5 what do you think about caching options?
23:06 jonnyno5 in client side?
23:06 elico it's for "write-behind" which is not a read...
23:06 jonnyno5 please go down
23:07 theron joined #gluster
23:07 jonnyno5 in document
23:07 ndevos if you have a test environment, and preferable can write patches in C, you could test with https://forge.gluster.org/m​od_proxy_gluster/pages/Home
23:07 glusterbot Title: Apache Module mod_proxy_gluster - Home - Open wiki - Gluster Community Forge (at forge.gluster.org)
23:07 elico "A sample IO-Cache config:"  ?
23:07 jonnyno5 yes
23:08 ndevos that should setup more client connections inside the apache process, it should improve performance
23:08 elico well you can verify your issue outside of apache...
23:09 elico What machine is it?
23:09 jonnyno5 ubuntu 12.04
23:09 jonnyno5 12.04.3 LTS
23:09 jonnyno5 server of course
23:10 elico size of the machine by ram vpu and disk?
23:10 jonnyno5 2 vcores with 2,6 Ghz
23:10 jonnyno5 and 4 GB of mem
23:11 jonnyno5 disk 10 GB in briks
23:11 elico OK and this is in php?
23:11 jonnyno5 system (/) 8 GB
23:11 jonnyno5 yes
23:11 jonnyno5 apache
23:12 elico I assume it's not 10GB of php right?
23:12 jonnyno5 not
23:12 jonnyno5 is a wordpress
23:12 elico ok
23:12 elico and you need the gluster for uploads?
23:12 jonnyno5 maybe 500 MB actual size of data
23:12 elico Hoo
23:12 jonnyno5 mostly reads
23:13 jonnyno5 1-5 posts weekly
23:13 jonnyno5 and some images
23:13 elico and the posts are in the mysql....
23:13 jonnyno5 yes
23:13 jonnyno5 another server
23:14 elico so what do you want\need to store on the glusterfs?
23:14 elico images and files?
23:14 jonnyno5 wordpress content
23:14 jonnyno5 yes
23:14 mjsmith2_ joined #gluster
23:14 jonnyno5 not host files
23:14 jonnyno5 only is a blog
23:14 jonnyno5 famous blog
23:15 elico famous or not we are talking about reads and other stuff...
23:15 jonnyno5 i have this issues when newsletter is sended
23:15 jonnyno5 apache goes crazy requesting small files to gluster
23:15 jonnyno5 gluster nodes have not load
23:15 elico and do you afraid of using nfs?
23:16 jonnyno5 apache overload my client machines
23:16 elico just for the read caching..
23:16 jonnyno5 i need caching
23:16 elico but it's another issue..
23:16 elico it might be that it's not a glustefrs level issue but rather apache load issue.
23:16 elico You need to profile the issue and understand it...
23:17 jonnyno5 i need increase reads of gluster clients
23:17 jonnyno5 i checked whit commands
23:17 jonnyno5 times
23:17 elico I have 2 glusterfs nodes with 2.5 TB and the only limit is that it has 1Gbps nics and it needs to replicate at the same time on the only 1x1 Gbps nic
23:18 jonnyno5 if i have cache
23:18 jonnyno5 y will have 0.006
23:18 jonnyno5 sec
23:18 jonnyno5 if not
23:18 elico well you can try a test run of nfs mount for a few days to see if it goes better....
23:18 jonnyno5 0.031
23:18 jonnyno5 sec
23:18 elico then decide on the strategy..
23:18 elico it's still not that much
23:19 jonnyno5 yes i understand you idea
23:19 jonnyno5 thanks a lot but a like to test some cache configs before
23:19 elico if in both nfs and glusterfs you still have the issue you might having another issue.
23:19 ndevos is wordpress not written in ,,(php)?
23:19 glusterbot (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
23:19 glusterbot --fopen-keep-cache
23:20 elico nfs is the only reasonable one.. there is not cache in glustefs..
23:20 elico ndevos: yes..
23:20 jonnyno5 NFS have native cache
23:20 jonnyno5 can be the solution
23:20 elico yes this and also you will have another thing in it..
23:20 ndevos so, jonnyno5, have you tried the options explained in that ^^ blog post?
23:21 elico the replication will happen on the gluster nics rather the service nodes.
23:23 jonnyno5 i know that yes
23:23 elico have you tested basic apache settings? I had the chance of seeing couple websites that moves so slow and the service goes up and uses 8GB for no reason due to miss-configuration.
23:23 jonnyno5 thanks a lot but i guess why does it appear a cache config for gluster what that means?
23:23 elico also you can use the mount with "noatime"
23:24 jonnyno5 in internet i refer
23:24 diegows joined #gluster
23:25 elico you can try the settings but I do not think it's the issue in hands at all due to the existence of thousands of setups which still works with no cache at all.
23:30 elico jonnyno5: also there are couple good wordpress cache plugins..
23:31 jonnyno5 it already have optimized that mi wordpress have an 80/100 punctuation in google insights analytics
23:31 jonnyno5 i have a varnish cache
23:31 jonnyno5 and w3 total cache
23:32 jonnyno5 js css html minification and compresion enabled
23:32 jonnyno5 and etc
23:32 jonnyno5 w3 total cache is a wordpress plugin
23:32 jonnyno5 works fine
23:33 jonnyno5 the processes queue is a consecuence of low reads of gluster client config side
23:34 elico what commands and what tests did you run to verify it?(curios)
23:34 jonnyno5 i am trying nfs
23:34 jonnyno5 time ls
23:34 jonnyno5 for example
23:34 elico ls is not a test...
23:35 elico you need to access a specific file(just a note)
23:35 jonnyno5 i tested du command to
23:35 jonnyno5 and find command
23:35 elico same thing as ls
23:35 jonnyno5 3  or 4 finds commands makes CPU at 100%
23:36 elico because this is what it's suppose to do..
23:36 jonnyno5 with concurrency finds
23:36 jonnyno5 maybe
23:37 elico on the mounting nodes.. right?
23:37 jonnyno5 in clients
23:37 elico yes clients\mounting the same thing.
23:37 elico try to do something else..
23:38 jonnyno5 can you help me?
23:38 elico yes
23:38 jonnyno5 thanks a lot
23:38 elico try reading a file a 50Kb file.
23:38 jonnyno5 which command?
23:38 elico using "time cat /var/www/upload/X/X/XY >/dev/null"
23:39 elico a specific file..
23:41 elico for me it takes a 700MB file to run at 8-14 secs
23:42 elico when I load the disks it takes a bit more.
23:42 elico And when I have used nfs it was quite speedy but almost the same.
23:42 jonnyno5 real0m0.007s user0m0.000s sys0m0.004s
23:43 elico what file size?
23:43 jonnyno5 51 KB
23:43 jonnyno5 i am testing with one node and one client now
23:43 elico OK
23:44 jonnyno5 in production enviroment
23:44 jonnyno5 real0m0.005s
23:44 jonnyno5 user0m0.000s
23:44 jonnyno5 sys0m0.000s
23:44 jonnyno5 2 nodes
23:44 jonnyno5 2 clients now
23:44 elico this sounds more like it..
23:45 jonnyno5 wich time i should to have ?
23:45 elico when you have load you can do this test and see what speed it runs at.
23:45 elico it can vary...
23:45 jonnyno5 i checked that
23:46 jonnyno5 not output
23:46 elico it should have around 0.000-9 for small files
23:46 jonnyno5 system wait and wait and wait
23:46 jonnyno5 until i restart apache
23:46 elico what wait?
23:46 elico then it's another issue..
23:46 jonnyno5 command output
23:46 elico hoo
23:47 elico so you have apache running and blocks the read?
23:48 jonnyno5 apache begin to request files
23:48 jonnyno5 and generates queue
23:49 jonnyno5 sorry for my english
23:49 jonnyno5 i am not english
23:49 elico it's ok. pretty good..
23:49 elico and can you see something in the glusterfs logs at the client side?
23:51 jonnyno5 sorry i dont see that
23:51 elico tail -f /var/log/glustefs/X.log
23:52 elico In any case I suggest that you will post in the mailing list a nice post regarding the issue you suspect and what you have considered until now...
23:52 jonnyno5 but enviroment now is autoscaled when apache cant serve web page autoscale rules dow the servers and instances 2 servers more
23:52 jonnyno5 y lost logs of last machines
23:53 elico I did not understood the last sentence..
23:54 jonnyno5 sorry, client logs are in overloaded machines that have been terminated by amazon rules of autoscaled enviroment
23:54 jonnyno5 when amazon terminated means
23:54 jonnyno5 eliminated
23:54 jonnyno5 and i have no logs
23:54 elico why was that done?
23:54 jonnyno5 then autoscale ups two identical machines to another
23:55 jonnyno5 nodes not being autoscaled
23:55 elico well you need to see if there is something in the logs at that time.
23:55 jonnyno5 only apache webservers with gluster fs clients
23:55 jonnyno5 if CPU of apaches is at 100% is autoscaled
23:55 elico but yet how many requests per second do you have on the server?
23:56 jonnyno5 not much
23:56 elico how many is not much?
23:56 jonnyno5 60 ?
23:56 elico really nothing..
23:57 elico you should see at the time what happens on the machine... what is using the cpu etc.
23:57 elico htop can help you with that.
23:57 jonnyno5 thats what i am so confused
23:57 elico top..
23:57 elico atop
23:58 elico You first need to detect the basic cause of the issue and eliminate the basic service issue.
23:58 jonnyno5 apache proceses stay stucked waiting for gluster
23:58 elico how do you see that it's waiting for gluster?
23:59 jonnyno5 because i cant acces to files when it happens
23:59 jonnyno5 vol is mount
23:59 jonnyno5 ed
23:59 elico access like "vi /var/www/file" ?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary