Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-28

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 cyberbootje joined #gluster
00:09 sarkis joined #gluster
00:11 bazzles joined #gluster
00:16 ninkotech__ joined #gluster
00:16 ninkotech_ joined #gluster
00:20 mattappe_ joined #gluster
00:33 MacWinner joined #gluster
00:39 ninkotech__ joined #gluster
00:42 fyxim joined #gluster
00:42 aurigus_ joined #gluster
00:42 kkeithley joined #gluster
00:51 ilbot3 joined #gluster
00:51 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
00:51 xavih_ joined #gluster
00:56 ninkotech joined #gluster
00:57 baoboa joined #gluster
00:58 sputnik13 anyone using gluster with openstack?
00:59 sputnik13 does it matter whether you have a single volume for multiple services (cinder, nova, glance) vs a dedicated volume for each service?
01:00 xymox joined #gluster
01:01 _VerboEse joined #gluster
01:02 johnmwilliams__ joined #gluster
01:05 nueces joined #gluster
01:06 elyograg thanks to whoever it was tht pointed me at linux kernel tuning parameters.  I had done a couple of those already, but now I have a few more.
01:07 crazifyngers joined #gluster
01:08 kkeithley joined #gluster
01:08 sulky joined #gluster
01:08 johnmwilliams__ joined #gluster
01:09 ninkotech__ joined #gluster
01:10 xymox joined #gluster
01:10 ninkotech_ joined #gluster
01:12 bazzer joined #gluster
01:13 dbruhn joined #gluster
01:14 mtanner_ joined #gluster
01:18 YazzY joined #gluster
01:20 askb joined #gluster
01:20 askb joined #gluster
01:21 ninkotech_ joined #gluster
01:24 ninkotech joined #gluster
01:26 nixpanic_ joined #gluster
01:26 nixpanic_ joined #gluster
01:27 dbruhn_ joined #gluster
01:27 shapemaker joined #gluster
01:27 ninkotech__ joined #gluster
01:28 bazzles joined #gluster
01:29 jiqiren_ joined #gluster
01:29 tzi0m joined #gluster
01:29 smithyuk1 joined #gluster
01:29 kkeithley joined #gluster
01:29 kkeithley joined #gluster
01:31 vincent_vdk joined #gluster
01:31 larsks joined #gluster
01:31 askb joined #gluster
01:32 tomased joined #gluster
01:41 tokik joined #gluster
01:41 jmarley joined #gluster
01:41 jmarley joined #gluster
01:42 bala joined #gluster
01:50 jskinner_ joined #gluster
01:53 jskinner_ joined #gluster
01:59 REdOG drop half the replica all vm's crash....whoopsie
02:24 bazzles joined #gluster
02:27 hagarth joined #gluster
02:30 prasanth joined #gluster
02:35 hagarth1 joined #gluster
02:36 rahulcs joined #gluster
02:37 askb joined #gluster
02:42 haomaiwa_ joined #gluster
02:50 cjanbanan joined #gluster
02:54 JoeJulian YazzY: There is no primary node.
02:55 hagarth joined #gluster
02:56 Alex There is only your mind?
03:01 chirino_m joined #gluster
03:01 nightwalk joined #gluster
03:22 spandit joined #gluster
03:26 bazzles joined #gluster
03:28 bharata-rao joined #gluster
03:30 d3O joined #gluster
03:31 shubhendu joined #gluster
03:40 itisravi joined #gluster
03:41 spandit joined #gluster
03:44 spandit joined #gluster
03:49 ira joined #gluster
03:54 hagarth joined #gluster
03:57 sarkis joined #gluster
03:59 dusmant joined #gluster
04:02 shylesh joined #gluster
04:06 mattappe_ joined #gluster
04:13 sprachgenerator joined #gluster
04:13 bazzles joined #gluster
04:15 eryc joined #gluster
04:15 CheRi joined #gluster
04:18 satheesh2 joined #gluster
04:21 Guest48886 joined #gluster
04:23 kdhananjay joined #gluster
04:24 SuperYeti joined #gluster
04:24 ppai joined #gluster
04:24 SuperYeti Hey Guys. Does anyone have any experience getting php applications running well against a gluster replication set? :)
04:25 SuperYeti we've got it running ok, have set apc.stat=0
04:25 SuperYeti the application cache is set on a local drive
04:25 SuperYeti but it's still 3x slower than when we run the app from local disk
04:31 rfortier joined #gluster
04:33 chirino joined #gluster
04:36 larsks|alt joined #gluster
04:36 elyograg SuperYeti: have you seen Joe Julian's blog post on it?  http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/
04:37 glusterbot Title: Optimizing web performance with GlusterFS (at joejulian.name)
04:37 rfortier2 joined #gluster
04:37 shubhendu joined #gluster
04:37 SuperYeti no I haven't
04:38 SuperYeti oh yes
04:38 SuperYeti already read that :)
04:39 SuperYeti have done everything in there, still not quite there
04:39 SuperYeti when the webserver makes calls to load the php application, we're seeing like 60% cpu usage spikes from glusterfs process
04:40 elyograg I don't write PHP apps, and I don't have the apps that I use on gluster.
04:40 SuperYeti and apc doesn't look like it's missing anything
04:40 kkeithley joined #gluster
04:41 _tziOm joined #gluster
04:41 vincent_vdk joined #gluster
04:42 rjoseph joined #gluster
04:42 larsks joined #gluster
04:42 shylesh joined #gluster
04:43 crazifyngers joined #gluster
04:44 jiqiren joined #gluster
04:47 toki joined #gluster
04:47 kanagaraj_ joined #gluster
04:47 sahina joined #gluster
04:47 chirino joined #gluster
04:47 kkeithley joined #gluster
04:49 shylesh_ joined #gluster
04:49 shubhendu_ joined #gluster
04:57 sputnik13 joined #gluster
05:00 itisravi_ joined #gluster
05:01 prasanth joined #gluster
05:04 chirino_m joined #gluster
05:04 cp0k joined #gluster
05:06 mibby- joined #gluster
05:07 crazifyngers_ joined #gluster
05:08 haomai___ joined #gluster
05:08 SachinPandit joined #gluster
05:09 shubhendu_ joined #gluster
05:11 prasanth_ joined #gluster
05:14 mohankumar joined #gluster
05:14 mibby joined #gluster
05:14 crazifyngers_ joined #gluster
05:14 dusmant joined #gluster
05:15 davinder joined #gluster
05:15 SachinPandit joined #gluster
05:15 bazzles joined #gluster
05:15 cp0k joined #gluster
05:15 prasanth joined #gluster
05:16 chirino joined #gluster
05:22 prasanth joined #gluster
05:28 Slasheri joined #gluster
05:28 Slasheri joined #gluster
05:28 smithyuk1 joined #gluster
05:28 CheRi joined #gluster
05:28 bala joined #gluster
05:28 cyberbootje joined #gluster
05:29 crazifyngers joined #gluster
05:29 askb joined #gluster
05:29 mohankumar joined #gluster
05:38 saurabh joined #gluster
05:38 chirino joined #gluster
05:38 jporterfield joined #gluster
05:43 cp0k hey guys, any idea how I can fix this error?
05:43 cp0k [2014-02-28 05:42:45.640577] E [glusterd-utils.c:2334:glus​terd_compare_friend_volume] 0-: Cksums of volume th-tube-storage differ. local cksum = -350699831, remote cksum = 1539331206
05:43 cp0k ( I am still stuck on adding new peers to an existing gluster) :(
05:44 borreman_dk joined #gluster
05:45 mohankumar__ joined #gluster
05:45 cp0k my understanding is that I may need to run 'gluster volume sync  host ip' to fix this up
05:45 cp0k ?
05:49 askb joined #gluster
05:50 CheRi joined #gluster
05:50 cp0k https://bugzilla.redhat.com/show_bug.cgi?id=949625
05:50 glusterbot Bug 949625: unspecified, medium, ---, kaushal, ASSIGNED , Peer rejected after upgrading
05:50 cp0k explains my issue 1:1
05:51 lman482 joined #gluster
05:52 badone joined #gluster
05:52 cjanbanan joined #gluster
05:52 askb joined #gluster
05:53 aravindavk joined #gluster
05:55 mohankumar__ joined #gluster
05:59 nithind1988 joined #gluster
06:01 vpshastry joined #gluster
06:02 haomaiwang joined #gluster
06:04 cp0k I'll mess with it more tomorrow, seems everyone is sleeping right about now. Talk to you guys later
06:05 nithind1988 left #gluster
06:05 hagarth joined #gluster
06:06 lalatenduM joined #gluster
06:08 bazzles joined #gluster
06:08 lman4822 joined #gluster
06:10 lalatenduM_ joined #gluster
06:10 mohankumar__ joined #gluster
06:12 Philambdo joined #gluster
06:13 dusmant joined #gluster
06:14 Philambdo joined #gluster
06:15 rastar joined #gluster
06:17 d3O joined #gluster
06:18 prasanth joined #gluster
06:18 badone joined #gluster
06:29 mohankumar__ joined #gluster
06:36 sputnik13 joined #gluster
06:36 sputnik1_ joined #gluster
06:37 sputnik13 joined #gluster
06:40 rahulcs joined #gluster
06:40 vimal joined #gluster
06:45 rahulcs joined #gluster
06:48 nithind1988 joined #gluster
06:50 d3O joined #gluster
06:54 d3O_ joined #gluster
06:59 kdhananjay joined #gluster
07:00 hagarth joined #gluster
07:02 bazzles joined #gluster
07:07 wica joined #gluster
07:14 ngoswami joined #gluster
07:16 kdhananjay joined #gluster
07:20 ekuric joined #gluster
07:21 jtux joined #gluster
07:30 prasanth joined #gluster
07:36 stickyboy I can't get fuse mounts to mount at boot on CentOS 6.4/6.5... Gluster is version 3.4.2.  I've tried with _netdev and without.
07:40 rgustafs joined #gluster
07:56 rahulcs joined #gluster
07:56 ctria joined #gluster
07:58 bazzles joined #gluster
08:03 eseyman joined #gluster
08:08 xavih joined #gluster
08:08 cjanbanan joined #gluster
08:09 ndevos stickyboy: hmm, _netdev and enabling the netfs service should do it...
08:09 nithind1988 left #gluster
08:16 hagarth joined #gluster
08:22 stickyboy ndevos: Hmm, netfs service is enabled. :\
08:22 stickyboy I noticed recently two of my clients fail to mount the gluster mounts at boot.
08:23 gdubreui joined #gluster
08:23 andreask joined #gluster
08:24 cjanbanan joined #gluster
08:32 lalatenduM joined #gluster
08:38 prasanth joined #gluster
08:38 hybrid512 joined #gluster
08:40 vravn joined #gluster
08:42 deepakcs joined #gluster
08:45 dusmant joined #gluster
08:47 partner i wonder if it was fixed for debian already, need to try it out somewhere
08:48 ThatGraemeGuy joined #gluster
08:49 bala1 joined #gluster
08:49 fsimonce joined #gluster
08:50 shubhendu_ joined #gluster
08:50 stickyboy partner: The mounting issue?
08:51 hagarth joined #gluster
08:55 kanagaraj_ joined #gluster
08:55 gdubreui joined #gluster
08:56 bazzles joined #gluster
08:58 partner yeah, there was some issues at least with squeeze, haven't got time to test it with newest version yet (ref http://joejulian.name/blog/glusterfs-volumes-​not-mounting-in-debian-squeeze-at-boot-time/)
08:58 glusterbot Title: GlusterFS volumes not mounting in Debian Squeeze at boot time (at joejulian.name)
08:59 ndevos stickyboy: you may find something in the mount logs (/var/log/glusterfs/<mntpoint>.log)
09:00 ndevos stickyboy: if the info there does not help, you can add the option log-level=DEBUG in /etc/fstab
09:05 khushildep joined #gluster
09:07 Pavid7 joined #gluster
09:08 doekia joined #gluster
09:09 qdk joined #gluster
09:11 doekia hi
09:11 glusterbot doekia: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:12 glusterbot New news from newglusterbugs: [Bug 1049981] 3.5.0 Tracker <https://bugzilla.redhat.co​m/show_bug.cgi?id=1049981>
09:12 doekia I need some good advize on performances  gfs for apache2/php
09:13 crashmag joined #gluster
09:13 doekia currently I have 3 nodes mounting a share/replicated volume all systems are SSD + 1GB network in between them
09:15 doekia machines are Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz 8 cpu thread, glusterfs is taking 43% + 35% cpu use
09:15 andreask glusterfs ... gfs is something different ;-)
09:16 doekia sorry it is  glusterfs
09:17 ndevos ~php | doekia
09:17 glusterbot doekia: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
09:17 glusterbot --negative-timeout=HIGH --fopen-keep-cache
09:18 doekia @ndevos ?? sorry what do you mean?
09:19 ndevos doekia: that is the advise for running php on glusterfs
09:20 doekia ok got it ... will look at that again ...
09:21 doekia any recipe concerning glusterfs based farm of server ... yes again php based code (e-commerce)
09:22 ndevos doekia: that blog post is pretty good
09:24 wica joined #gluster
09:25 rastar joined #gluster
09:25 X3NQ joined #gluster
09:28 keytab joined #gluster
09:36 stickyboy ndevos: Ok, lemme look.  johnmark is calling me out on twitter ;)
09:36 stickyboy Now I must dig deeper.
09:38 liquidat joined #gluster
09:38 stickyboy ndevos: Could be:  [2014-02-28 06:43:15.935153] E [socket.c:2157:socket_connect_finish] 0-glusterfs: connection to 192.168.5.28:24007 failed (No route to host)
09:39 ndevos stickyboy: sounds like a possible cause, yes
09:39 stickyboy So netwrok wasn't up apparently.  Which is insane, of course, as networking works fine...
09:39 stickyboy I could reboot the client again and see if it comes up.  For giggles.
09:40 stickyboy It's lunch time soon anyways, so I could pass by the server room on the way back ;)
09:41 stickyboy netfs          0:off1:off2:off3:on4:on5:on6:off
09:41 stickyboy So _netdev should be working...
09:41 * ndevos just wanted to ask that
09:42 mattappe_ joined #gluster
09:42 ndevos stickyboy: in case network comes up really slowly, you can add a delay setting in /etc/sysconfig/network-scripts/ifcfg-*
09:44 stickyboy Ok, do you know the configsetting off hand?
09:44 ndevos LINKDELAY=30 would cause 30 seconds delay
09:48 stickyboy ndevos: Cool.  Seems CentOS docs are always from CentOS 5 whenever I search. ;)
09:49 ndevos stickyboy: I normally use the RHEL docs, or in this case: grep DELAY /etc/sysconfig/network-scripts/ifup*
09:50 shubhendu_ joined #gluster
09:50 kanagaraj_ joined #gluster
09:52 dusmant joined #gluster
09:52 stickyboy ndevos: Sometimes I get lucky and stumble on some recent-ish Fedora docs.
09:53 bala joined #gluster
09:55 bazzles joined #gluster
09:58 stickyboy ndevos: Trying again without LINKDELAY and with _netdev... as a control. :D
10:00 ndevos stickyboy: there is also backup-volfile-servers as mount option, it takes a list (separeted by :) of hostnames to try in case the 1st failed
10:00 ndevos 'No route to host' could mean that glusterd is not available on that particular storage server at that time, even if network is up and running
10:04 stickyboy ndevos: Yah, I'm using backupvolfile-server= in my mount options.
10:05 ndevos stickyboy: okay, that suggests that the network is indeed not ready yet when netfs runs
10:14 stickyboy ndevos: LINKDELAY=30 helped.
10:15 bazzer joined #gluster
10:16 stickyboy I will blog about this.
10:16 stickyboy To get it "out there".
10:19 shubhendu_ joined #gluster
10:21 kanagaraj_ joined #gluster
10:21 ndevos stickyboy: nice to know, you may want to change the delay, 30 seconds might be pretty long
10:22 bala joined #gluster
10:22 stickyboy ndevos: Yah.
10:23 satheesh2 joined #gluster
10:24 al joined #gluster
10:24 rossi_ joined #gluster
10:27 spandit joined #gluster
10:33 senor_puck joined #gluster
10:35 senor_puck hey, is it possible to connect a gluster client to multiple volumes?
10:35 stickyboy senor_puck: Definitely.  I have 4 gluster mounts on my client.   /homes, /export/data, /export/apps, etc.
10:36 DV_ joined #gluster
10:38 senor_puck thanks stickyboy, will it work even if the volumes are on the same server?
10:39 stickyboy senor_puck: Same server as the client?  Or if the client is mounting many volumes from one server?
10:39 senor_puck the latter
10:40 stickyboy senor_puck: Definitely that will work.
10:40 stickyboy Here's an excerpt from my /etc/fstab on CentOS client:  https://gist.githubusercontent.com/alano​rth/99d4d7a9d935af040407/raw/ae71002a540​6303d2228391252e819a5202eb21e/fstab.txt
10:41 senor_puck Excellent, I will have to have another try. I was trying to set this up in a production environment but I couldnt have too much downtime. I will try it again in a test environment
10:42 stickyboy senor_puck: No problem.  I'm here running in production, so I can help. :D
10:43 davinder2 joined #gluster
10:44 mattappe_ joined #gluster
10:46 CheRi joined #gluster
10:49 diegows joined #gluster
11:03 gmcwhistler joined #gluster
11:08 shubhendu_ joined #gluster
11:08 bala joined #gluster
11:09 rossi_ joined #gluster
11:15 rahulcs joined #gluster
11:16 rjoseph joined #gluster
11:24 mohankumar__ joined #gluster
11:38 mohankumar__ joined #gluster
11:45 mattappe_ joined #gluster
11:46 mohankumar__ joined #gluster
11:46 RameshN joined #gluster
11:53 wica joined #gluster
12:02 prasanth joined #gluster
12:04 calum_ joined #gluster
12:07 psyl0n joined #gluster
12:08 harish_ joined #gluster
12:13 itisravi_ joined #gluster
12:14 P0w3r3d joined #gluster
12:19 sputnik13 joined #gluster
12:20 edward joined #gluster
12:23 nightwalk joined #gluster
12:27 vorakl joined #gluster
12:29 TvL2386 joined #gluster
12:32 bharata_ joined #gluster
12:33 Pavid7 joined #gluster
12:35 ira_ joined #gluster
12:42 wrale-josh joined #gluster
12:45 wrale-josh hi.. i'm new to gluster and fio (which i'm using to bechmark).. I'm trying to compare a gluster volume against a known benchmark, so I am looking at this page: http://scalability.org/?p=1856 (among others).. anyway, can anyone give me a good idea of how to calculate iodepth (aggregated depth)  for gluster volumes?
12:45 wrale-josh *aggregated queue depth
12:46 cjanbanan joined #gluster
12:46 tokik joined #gluster
12:55 vorakl left #gluster
13:13 mattapperson joined #gluster
13:13 khushildep joined #gluster
13:14 hagarth joined #gluster
13:17 lalatenduM joined #gluster
13:19 rgustafs joined #gluster
13:20 pvh_sa joined #gluster
13:31 davinder joined #gluster
13:35 rgustafs joined #gluster
13:36 hybrid512 joined #gluster
13:46 dewey joined #gluster
14:03 cfeller joined #gluster
14:05 cjanbanan joined #gluster
14:07 jskinner_ joined #gluster
14:10 jmarley joined #gluster
14:10 jskinner joined #gluster
14:15 khushildep joined #gluster
14:17 ninkotech joined #gluster
14:17 ninkotech_ joined #gluster
14:18 giulivo joined #gluster
14:19 giulivo JoeJulian, not sure if you're still around but in case you remember the issue I raised yesterday with brick name too long... I did some more attempts and the problem doesn't seem to be the volume name but the actual path of the brick which can't be longer than 32chars (not for me at least)
14:21 giulivo so to sum it up "gluster volume create very_long_name IP:/path/to/brick" works as long as "path/to/brick" is shorter than 32chars
14:21 ctria joined #gluster
14:24 jag3773 joined #gluster
14:26 khushildep joined #gluster
14:29 B21956 joined #gluster
14:36 cfeller joined #gluster
14:39 davinder joined #gluster
14:40 bennyturns joined #gluster
14:42 dbruhn joined #gluster
14:47 ninkotech joined #gluster
14:47 ninkotech_ joined #gluster
14:48 sroy joined #gluster
14:54 khushildep joined #gluster
14:55 Slash_ joined #gluster
15:01 georgeh|workstat joined #gluster
15:01 kshlm joined #gluster
15:03 doekia someone to point me to fstab entry to set this  --attribute-timeout=HIGH --entry-timeout=HIGH  --negative-timeout=HIGH --fopen-keep-cache - or the appropriate mount syntax
15:10 primechuck joined #gluster
15:10 ctria joined #gluster
15:15 mattappe_ joined #gluster
15:17 FrodeS doekia: thought about googling it?
15:17 doekia I deed
15:22 Pavid7 joined #gluster
15:23 rpowell joined #gluster
15:25 sprachgenerator joined #gluster
15:25 wushudoin| joined #gluster
15:25 ndevos giulivo: that is interesting, I did not know there was such a small limit on the brick-path
15:26 ndevos giulivo: if that is on a recent version, can you file a bug for that?
15:26 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
15:26 giulivo ndevos, I will yes, thanks for pinging
15:26 ndevos giulivo: np, thanks for filing the bug!
15:30 jobewan joined #gluster
15:31 bugs_ joined #gluster
15:33 failshell joined #gluster
15:36 zerick joined #gluster
15:37 plarsen joined #gluster
15:40 msp3k1 joined #gluster
15:41 kaptk2 joined #gluster
15:43 lman4821 joined #gluster
15:49 xavih joined #gluster
15:50 lmickh joined #gluster
15:55 sarkis joined #gluster
16:06 wrale i'm setting up an gluster + ovirt cluster, where both are on every node.  i'm also going to be using the hosted engine option.  i have a question about volume replication.  will one of the replicas be written locally by default or must i tune this in somehow?  doing so would effectively bring compute to the storage, as the saying goes, right? (six nodes)
16:06 sputnik13 joined #gluster
16:08 elyograg wrale: the gluster client connects to all the bricks and writes to them all at the same time.  If you're mounting NFS, then the NFS server is a gluster client.
16:09 elyograg s/writes the them all/writes to all appropriate replicas/
16:09 glusterbot elyograg: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
16:09 wrale elyograg:  thanks.. i'm hoping to use a gluster-native client..
16:10 wrale elyograg: i have two bricks per each of six nodes.. is there any policy i need to consider for replica locality?
16:10 andreask joined #gluster
16:11 elyograg no.  because of the way that the gluster client works, preferring local bricks doesn't get you anything -- it has to talk to all the replicas anyway.
16:11 wrale elyograg: sounds good.. that is helpful.. thank you
16:14 theron joined #gluster
16:20 pvh_sa joined #gluster
16:20 RameshN joined #gluster
16:23 ndk joined #gluster
16:25 kaptk2 joined #gluster
16:26 jag3773 joined #gluster
16:34 msp3k1 Will using a higher replication count help to protect against failed heals and split brains?
16:37 semiosis msp3k1: quorum will prevent split brain.  best used with replica 3
16:37 semiosis loss of quorum causes clients to turn read-only, so replica 3 allows them to continue writing when one brick or server goes down
16:38 daMaestro joined #gluster
16:45 vpshastry joined #gluster
16:45 msp3k1 With replication 3 and cluster.quorum-type auto, if one brick or server goes down, will it automatically heal by taking the best 2-out-of-3?
16:45 msp3k1 (When the downed brick/server comes back up, that is)
16:48 semiosis "best 2-out-of-3" doesnt really make sense to me.  if you want to know how glusterfs determines which way to heal, read the ,,(extended attributes) article
16:48 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
16:54 RayS joined #gluster
16:57 kshlm joined #gluster
16:58 cp0k joined #gluster
16:58 cp0k Hey guys, I think I finally narrowed down the issue I am having with probing new peers after my upgrade to Gluster 3.4.2 - https://bugzilla.redhat.com/show_bug.cgi?id=949625
16:58 glusterbot Bug 949625: unspecified, medium, ---, kaushal, ASSIGNED , Peer rejected after upgrading
16:59 cp0k from the link above it states "Gluster v3.3 brings some changes to the volfiles and some options. So, on upgrade from 3.2 to 3.3, these files need to have been regenerated. If the upgrade had been done via rpms, the regeneration should have happened correctly. Since the regeneration hasn't happened, you either did a source install or the rpms you used were faulty (in which case please provide details on the rpms)."
16:59 cp0k which goes on to say that I need to follow the upgrade instructions shown here - http://vbellur.wordpress.com/2012/​05/31/upgrading-to-glusterfs-3-3/
17:01 davinder joined #gluster
17:02 sputnik13 joined #gluster
17:02 cp0k however the instructions I followed for the upgrade were - http://vbellur.wordpress.com/2013/​07/15/upgrading-to-glusterfs-3-4/
17:03 cp0k so now is it just a matter of killing glusterd, running "glusterd --xlator-option *.upgrade=on -N" and restarting it back up ?
17:09 doekia someone to point me to fstab entry to set this  --attribute-timeout=HIGH --entry-timeout=HIGH  --negative-timeout=HIGH --fopen-keep-cache - or the appropriate mount syntax - please
17:11 dbruhn joined #gluster
17:14 msp3k1 What I mean by best 2 out of three is this: The extended attributes article defines split-brain as when the xattrs for the same path have different values for each other.  But it doesn't say what  the behavior would be in a replication 3 setup where the xattrs for a path match on two peers but differ on the third.  Is that still considered a split-brain scenario that must be handled by the admin by hand, or does gluster decide on it's own that the two tha
17:15 xavih_ joined #gluster
17:16 Pavid7 joined #gluster
17:17 cjanbanan joined #gluster
17:18 japuzzo joined #gluster
17:19 andreask joined #gluster
17:19 mattappe_ joined #gluster
17:19 xavih joined #gluster
17:28 ColPanik_ joined #gluster
17:31 ColPanik_ this is probably an easy one.  If I'm using SAMBA to share out a gluster volume to Windows host, should oplocking be disabled?
17:31 ColPanik_ as a followup, is that documented anywhere?
17:37 a1 joined #gluster
17:37 fsimonce` joined #gluster
17:41 Mo_ joined #gluster
17:44 glusterbot New news from newglusterbugs: [Bug 1071411] gluster cli : Incorrect grammar in the "Usage" of "rebalance" command <https://bugzilla.redhat.co​m/show_bug.cgi?id=1071411>
17:47 Pavid7 joined #gluster
17:47 ndk joined #gluster
17:48 hagarth joined #gluster
17:48 harish_ joined #gluster
17:49 jobewan joined #gluster
17:50 askb joined #gluster
17:50 wushudoin| joined #gluster
17:50 bennyturns joined #gluster
17:50 edward1 joined #gluster
17:50 msp3k joined #gluster
17:50 japuzzo joined #gluster
17:50 doekia joined #gluster
17:50 sroy__ joined #gluster
17:50 23LAAD20U joined #gluster
17:50 xavih joined #gluster
17:50 giulivo joined #gluster
17:50 zerick joined #gluster
17:50 failshell joined #gluster
17:50 cp0k joined #gluster
17:50 theron joined #gluster
17:50 rpowell joined #gluster
17:50 B21956 joined #gluster
17:50 edward1 joined #gluster
17:50 bennyturns joined #gluster
17:50 wushudoin| joined #gluster
17:50 askb joined #gluster
17:51 hybrid512 joined #gluster
17:51 ngoswami joined #gluster
17:51 andreask joined #gluster
17:51 andreask joined #gluster
17:52 liquidat joined #gluster
17:55 Matthaeus joined #gluster
17:56 marcoceppi joined #gluster
17:56 marcoceppi joined #gluster
17:56 cfeller joined #gluster
17:59 ColPanik_ joined #gluster
17:59 wushudoin| joined #gluster
17:59 failshel_ joined #gluster
18:00 ngoswami joined #gluster
18:01 B219561 joined #gluster
18:04 Slash_ joined #gluster
18:05 plarsen joined #gluster
18:05 DV_ joined #gluster
18:06 xavih joined #gluster
18:08 bala joined #gluster
18:09 wrale JoeJulian:  First of all thanks for writing your do's and don't article.  Secondly, what do you suppose would be a wise replica value for a production-data six node cluster with two bricks per node? (converged 10GbE, if that's any matter)
18:09 davinder joined #gluster
18:09 wrale JoeJulian: Seems that 3 would be good, but I'm hoping to confirm that with you, as per: "Most admins that I've spoken with use a simple replica count of 2. Ed Wyse & Co, Inc. has fewer servers so a replica count of 3 is warranted as the likelihood of simultaneous failures within the replica set is higher."
18:10 bazzles joined #gluster
18:10 giulivo joined #gluster
18:10 gmcwhistler joined #gluster
18:10 wushudoin| joined #gluster
18:12 daMaestro joined #gluster
18:17 lmickh joined #gluster
18:20 RayS joined #gluster
18:23 wrale elyograg: the NUFA option listed here seems to provide the prefer-local-write about which i asked earlier.. the page also says "NOTE: When NUFA is enabled, client mounts fail outside the trusted storage pool. " which is surely a trade-off.. http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented
18:23 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
18:25 SuperYeti joined #gluster
18:26 SuperYeti Morning guys, any gluster php experts here, could give me some tuning advise or assist with pointing me in the correct direction for tuning?
18:26 SuperYeti I've followed all the guides, and best practices I could, but my page response times are still like 600MS instead of about 150MS on local disk, looking for what else I can tune to try and improve that
18:26 semiosis SuperYeti: ,,(php)
18:26 glusterbot SuperYeti: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
18:26 glusterbot --negative-timeout=HIGH --fopen-keep-cache
18:26 vpshastry joined #gluster
18:27 SuperYeti I followed joejulian's
18:27 SuperYeti apparently APC still stat's if you are using relative includes
18:27 SuperYeti which the framework Symfony2 we're using does. I'm not sure if we are able to change that.
18:28 SuperYeti It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
18:29 SuperYeti so i'm currently using fstab like the following to mount my share: fohlderfs01:/sites /mnt/sites glusterfs defaults,_netdev,backupvolfile-server=fohlderfs02 0 0
18:29 SuperYeti what would I want to change?
18:29 semiosis did you tune your include path?
18:30 SuperYeti like in php?
18:31 semiosis thats right.  path most likely to have the file (i.e. framework path) should be first
18:31 semiosis least likely should be last
18:31 SuperYeti let me check
18:31 edward1 joined #gluster
18:31 zerick joined #gluster
18:31 sputnik13 joined #gluster
18:31 bugs_ joined #gluster
18:31 edward1 joined #gluster
18:32 semiosis there also may be alternative class loaders for symfony.  i know ZF2 lets you build a static class map which, when combined with APC, should give an ~80% performance boost compared to dynamic class loading (searching the include path) iirc
18:32 semiosis generally speaking, i think you can get more out of tuning php & your framework/app then tuning glusterfs
18:33 diegows joined #gluster
18:33 semiosis s/then/than/
18:33 SuperYeti is mounting fuse with glusterfs worth looking into as well?
18:33 glusterbot What semiosis meant to say was: generally speaking, i think you can get more out of tuning php & your framework/app than tuning glusterfs
18:33 semiosis SuperYeti: i only use fuse/native glusterfs mounts, not nfs
18:34 semiosis also you should use a front end cache for static assets, varnish is good, apache mod_cache is easy
18:34 SuperYeti Ok
18:35 SuperYeti fohlderfs01:/sites /mnt/sites glusterfs defaults,_netdev,backupvolfile-server=fohlderfs02 0  <- so is that using NFS, or fuse/native mount?
18:35 SuperYeti sorry for the stupid questions, still learning about gluster heh
18:36 zerick joined #gluster
18:36 doekia native + php is actually a dead-end for me ... 6s to generated php based page
18:37 vimal joined #gluster
18:37 doekia and I'm facing issue not able to use nfs for what ever unknown reason to date
18:37 semiosis doekia: you could tune your app/framework/apache to get better performance
18:38 doekia claims about unknown proto or nfs version unsupported
18:38 semiosis you need the right mount options for ,,(nfs)
18:38 glusterbot To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
18:38 doekia not really ... it is a stock app
18:38 doekia tried it to no avail
18:39 semiosis doekia: you can tune APC to not stat files, even for a stock app.  will need to restart apache after code deploys though
18:40 vpshastry left #gluster
18:40 doekia once I understand how thoses unducumented stuff work I will certainly be able to go a step further
18:41 doekia 6s is not only due stat() ... something is apparently pretty bad in my config
18:41 doekia and I can't figure-out what
18:41 [o__o] joined #gluster
18:42 semiosis doekia: optimize your include path
18:42 semiosis as i explianed above
18:43 doekia 2 days!!! I have followed good step by step as a copy cat ... system is up but behave poorly... now each time I try do adjust something it is more hours and crashes
18:43 doekia documentation is outdated to 3.1
18:43 doekia I feel miserable
18:43 owenmurr joined #gluster
18:47 wrale would creating a file (formatted with say ext4) inside your gluster mount be an optimization to those doing small files, supposing locking isn't a concern?
18:48 doekia Duno but I love your out of the box thinking
18:48 bala joined #gluster
18:50 wrale lol. it's a truly unoriginal idea, though.. virtualization (e.g. openstack, ovirt) use this method, instead of just chrooting inside distributed and _shared_ filesystems
18:50 wrale well, when it's configured and working correctly
18:51 wrale thanks though.. i much need confidence in my present endeavor
18:51 wrale (building a gluster + ovirt cluster) ..lol
18:54 SuperYeti semiosis found a way to optimize the autoloader and php include_path
18:54 SuperYeti got it from 700 to 350MS response
18:54 * SuperYeti highfives semiosis
18:54 sputnik13 joined #gluster
18:58 doekia I try that soon tomorrow and let you know ... this will save my life for my app which is having a 6s compared to its usual 700ms
18:59 SuperYeti 700ms is still really slow, you need to do some php optimizations
18:59 doekia This is a huge e-commerce app
18:59 SuperYeti we're loading a full stack Symfony2 framework and have it hovering around 200ms
18:59 wrale I read in the ovirt users mailing list that I should enable quorum on the gluster side of things.. I can't seem to figure out how to do this globally.  Is this something that only gets set on a per-volume basis?
18:59 SuperYeti i jiust now got it down to 350ish on gluster
19:00 doekia thru loopback mount?
19:00 wrale Latency.. A lot of that could be the network.. What is the backbone here?
19:00 SuperYeti EC2
19:01 wrale Have deep pockets?
19:01 doekia got 1gb/s between nodes (verified)
19:01 SuperYeti so should be 1gig for me as well
19:01 wrale try dedicated tenancy if so
19:01 doekia backbone here is onlineSAS
19:01 * SuperYeti cries
19:01 SuperYeti lol
19:01 wrale lol
19:02 doekia my galera works like a charme but gluster sluggish
19:02 SuperYeti i was looking at doing placement groups
19:02 SuperYeti then you get 10gig backbone
19:02 SuperYeti but then the whole app sits on one availability zone
19:02 SuperYeti and then I also have to re-create and re-ip all instances
19:02 wrale perhaps you could try fewer gluster nodes + more provisioned iops.. (if you know what i mean)
19:03 SuperYeti I'm only running 2 gluster nodes currently
19:03 wrale ah.. okay
19:03 SuperYeti with 3 volumes
19:03 SuperYeti but provisioned iops could help
19:03 doekia 3 nodes sounded just enough to start... we are servincing from a single node actually
19:03 SuperYeti i still think its network related
19:03 doekia 1 volume 3 bricks distributed/replicated is my design so far
19:04 SuperYeti because when we run off just EBS, its 1/2 the response time, and the only thing different would be the overhead from gluster
19:04 wrale lol. try a ramdisk
19:04 wrale (half joke)
19:04 SuperYeti lol
19:05 doekia physical is SSD already so pretty close to ram
19:05 * wrale makes the kinda sorta head gesture
19:05 SuperYeti I think we can live with where I've got it to now, with some more reverse proxy caching
19:06 wrale caching in front of the web server i hope..
19:06 wrale not involved with gluster
19:06 SuperYeti yes
19:06 wrale cool
19:06 SuperYeti move some static assets to there
19:06 SuperYeti that's next on the list, and also utilizing the SF2 caching
19:06 SuperYeti that requires a bit of re-engineering in the app to support secure access to cached resources though, so not happening today
19:06 SuperYeti wanted to get as much as I can out of the underlying stuff before we ventured down that path
19:07 doekia purpose here was to make a small cluster for starting customer
19:07 wrale you might look up the intel does lustre (not gluster) on ec2 blog post/paper... you'd need more nodes, though
19:07 wrale (small files might work better there)
19:08 doekia ec2 is not an option here ... too much latency and not native language support
19:08 wrale also, not used it but xtreemfs is interesting
19:09 cp0k joined #gluster
19:09 doekia rsync between nodes was a good approach but does not scale good and ... latency to reflect changes
19:09 wrale doekia: just realized there are two discrete cases here, rather than you guys really working on the same thing.. cool..
19:10 SuperYeti lol
19:10 doekia @wrale ... exact
19:10 wrale :) .. i'm thinking about gluster and ovirt.. for the most part.. no lie
19:10 wrale multithreading isn't multi-core.. said another way
19:10 wrale *hyperthreading
19:10 wrale ha
19:10 wrale case in point
19:11 doekia After so many hour trying to make gluster behave properly I now have to provide some delivery
19:11 doekia Budget is already locked ...
19:11 doekia I do not expected such ...
19:11 wrale doekia: how many servers providing gluster?  replication? striping? introspective (servers are consumers)?
19:11 doekia 3 server gluster server
19:12 doekia 2 more gluster client read-only
19:12 wrale i read that an even number of gluster servers should always be used for easy quorum
19:12 wrale not sure if it applies here.. i went to six from five for this reason
19:13 wrale read-only OS on those?
19:13 wrale like PXE boot?
19:13 doekia Intended design was to fallback to a single node during disaster scenario
19:13 wrale you need drdb, not gluster, i bet
19:13 wrale do you have IPMI (baseboard management) or otherwise a way to kill nodes programmatically?
19:13 wrale (fencing)
19:14 lalatenduM joined #gluster
19:14 wrale specifically, drdb + floating ip nfs might be good for you
19:14 wrale if it's an emergency
19:14 doekia again the initial plan counted on rsync to replicate ... but due incapacity to scale up of this design I chosoed to go network shared fs hence glusterfs
19:15 wrale oh .. the age-old scaling.. hmmm
19:15 SuperYeti ok I gotta run guys, thanks for the help again
19:15 wrale is your client a self-professed Web 55.0 company?  a startup wishing for mega scale clientele in the future?
19:15 doekia we are just in between small to big ... too big to continue on a single node
19:16 doekia not big enough to go infiniband san based fs
19:16 wrale yeah.. i think looking at xtreemfs wouldn't be a bad idea.. not sure about consistency there though
19:16 doekia sorry I don't know what 55.0 acronym means
19:17 doekia the compagny is doing 500 order/day growing by 20% every month
19:17 wrale sorry.. i meant.. does the company expect everything to be horizontally scalable up front, with no wont for vertical scale
19:18 wrale i see
19:19 doekia the apps is database (mysql) intensive ... and every thing was sitting on single node to date...
19:19 doekia the new design is load-balancer + media server + farm
19:20 wrale oh i see.. mysql.. i think you mentioned galera.. interesting stuff.. i don't know if you would benefit from hosting galera on the gluster cluster.. i'd have to read/think about that...
19:20 doekia lb+ms are on similar nodes in resilient pattern 1 up for lb on for media under normal situation
19:21 doekia galera is the new db, was mysql ...
19:21 wrale galera master-master?
19:21 doekia yes
19:21 wrale no raid?
19:21 doekia sure everything is SSD raid based
19:21 doekia raid 1
19:22 doekia hard
19:22 wrale hmm.  how many ssds? one probably?
19:22 wrale *per side of raid 1
19:22 doekia one exact
19:23 wrale i think you could do a lot worse than lvm on drdb on two servers with space for more disks
19:23 wrale as in.. it's scalable
19:23 wrale with a little work when the scale occurs
19:24 wrale i would recommend daily backups to external hard drive or tape if possible
19:24 doekia I haven't see any good lvm tied with good network for my area
19:24 doekia france
19:24 nikk anyone know if glusterfs can do write blocking?  aka when you lose a brick then prevent any writes to the volume until the brick comes back?
19:25 sputnik13 joined #gluster
19:25 wrale nikk: might be in here.. read something about blocking http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented
19:25 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
19:25 doekia don't worry backup is no prob... each node makes a copy then pushes it to a san based ftp solution (from the hoser) and a second replicat is done on a remote location
19:26 nikk i'll check it out, thanks
19:26 wrale nikk: np
19:26 doekia ok I have to go ... about time to fetch some food
19:26 wrale doekia: good luck
19:26 doekia ;-)
19:31 * JoeJulian shudders at the mention of drbd.
19:31 sarkis joined #gluster
19:32 wrale lol.. agreed, but he was talking about latency on very small files.. *shrugs*
19:33 zerick joined #gluster
19:34 lalatenduM nikk, there is quorum for this kind of situation http://www.gluster.org/community/documen​tation/index.php/Features/Server-quorum
19:34 glusterbot Title: Features/Server-quorum - GlusterDocumentation (at www.gluster.org)
19:34 nikk interesting
19:34 * wrale agrees
19:34 nikk i've never seen that
19:34 mattapperson joined #gluster
19:35 wrale is there any fencing involved?
19:35 DV__ joined #gluster
19:36 lalatenduM wrale, are you asking me :)?
19:36 wrale indeed..
19:36 JoeJulian Only if you're Errol Flynn.
19:36 rotbeard joined #gluster
19:37 criticalhammer joined #gluster
19:37 lalatenduM wrale, yes I think . came across this https://access.redhat.com/site/documentation/en​-US/Red_Hat_Storage/2.0/html/Administration_Gui​de/sect-User_Guide-Managing_Volumes-Quorum.html
19:37 glusterbot Title: 10.9. Configuring Server-Side Quorum (at access.redhat.com)
19:38 criticalhammer Hi, has anyone here have any experience with nic bonding? Are there any gotchyas to consider?
19:38 wrale JoeJulian: i posted these earlier, but reposting here: JoeJulian:  First of all thanks for writing your do's and don't article.  Secondly, what do you suppose would be a wise replica value for a production-data six node cluster with two bricks per node? (converged 10GbE, if that's any matter).....<wrale> JoeJulian: Seems that 3 would be good, but I'm hoping to confirm that with you, as per: "Most admins that I've spoken with use a simple replica c
19:38 wrale ount of 2. Ed Wyse & Co, Inc. has fewer servers so a replica count of 3 is warranted as the likelihood of simultaneous failures within the replica set is higher."
19:38 JoeJulian criticalhammer: The only occasions we've seen anything in here about it were people having trouble with it. I assume it must be successful sometimes, but nobody comes to complain about that.
19:39 wrale <lalatenduM>: thanks, reading.
19:39 georgeh|workstat joined #gluster
19:39 * wrale needs a better irc client than xchat.. suggestion welcome..
19:40 JoeJulian wrale: How many servers to you expect to have down at any one time? Do you have an SLA (or OLA) that you have to perform to? From there it's just math.
19:40 JoeJulian I use xchat.
19:40 wrale JoeJulian: well, quorum for the layers above requires three (which is expected).. (zookeeper + mesos and the like)
19:41 wrale well, not three.... three and one failure... i guess
19:41 wrale oh geez.. let me start over..
19:41 pvh_sa joined #gluster
19:42 lalatenduM JoeJulian, on a different context , have you came across this https://access.redhat.com/site/documentati​on/en-US/Red_Hat_Storage/2.1/html-single/A​dministration_Guide/index.html#chap-Admini​stration_Guide-Performance_Enhancements
19:42 glusterbot Title: Administration Guide (at access.redhat.com)
19:44 lalatenduM JoeJulian, I am talking about chapter "Configuring Red Hat Storage for Enhancing Performance"
19:46 wrale The IBM says: "On five-node cluster, you would need three to go down before the ZooKeeper service stops functioning." http://www.ibm.com/developer​works/library/bd-zookeeper/  ...
19:46 glusterbot Title: ZooKeeper fundamentals, deployment, and applications (at www.ibm.com)
19:46 jag3773 joined #gluster
19:47 wrale JoeJulian: therefore, i can lose probably three of six, because the sixth node won't have a zookeeper, i suppose (odd numbers for that)
19:47 JoeJulian lalatenduM: I hadn't read that. Useful stuff there so far...
19:48 wrale Red hat docs are awesome, usually..
19:48 wrale if sometimes light on details
19:48 wrale I like their Docbook layouts
19:49 lalatenduM JoeJulian, yup, thinking of sending it to gluster-users, not sure if it will help everybody
19:49 lalatenduM wrale, it least it will help to understand GlusterFS :)
19:49 lalatenduM in our case
19:49 wrale lalatenduM: agreed
19:50 JoeJulian Hrm, they left off changing the scheduler.
19:50 JoeJulian deadline or noop are better.
19:51 Matthaeus joined #gluster
19:55 ColPanik_ sorry for the re-ask.  Should I disable samba oplocks if I'm using it solely as a bridge for windows clients to access gluster?
19:57 wrale JoeJulian: No SLA, but the cluster is being constructed to have the best uptime possible.. dual power rails backed by dual rows of cabs of batteries backed by dual diesel generators ( i think the generators are dual)..
19:57 wrale (possible with what we have).. (small state university compute cluster)
19:59 JoeJulian http://www.eventhelix.com/realtimemantra/faul​thandling/system_reliability_availability.htm
19:59 glusterbot Title: System Reliability and Availability Calculation (at www.eventhelix.com)
19:59 semiosis wrale: Konversation
19:59 wrale JoeJulian: thanks.. will read
19:59 wrale semiosis: excellent.. will look into that too
19:59 wrale (thanks)
20:01 wrale JoeJulian: so zookeeper will be in parallel.. not in series.. *thinking*
20:02 semiosis @later tell SuperYeti ^5
20:02 glusterbot semiosis: The operation succeeded.
20:06 davinder joined #gluster
20:08 Matthaeus joined #gluster
20:10 zaitcev joined #gluster
20:14 glusterbot New news from newglusterbugs: [Bug 985957] Rebalance memory leak <https://bugzilla.redhat.com/show_bug.cgi?id=985957>
20:17 pvh_sa joined #gluster
20:18 wrale JoeJulian: so, after consideration, the plan is never to allow more than two of six servers to go offline... Therefore, my replica must be at least three.. Any more might mean congestion and unnecessarily lost space (supposing imaginary SLA of four servers always up holds true)... Does this sound correct?
20:18 wrale (And any less than replica three could get caught being the two down)
20:19 wrale Hmm.. but this assumes instant self healing.. errr :(
20:20 wrale (Because I could be doing a rolling kernel upgrade -- two servers at a time -- and stupidly stop the servers trying to catch up to the loss)?
20:21 JoeJulian Sounds correct to me.
20:21 wrale Seems I will need to closely monitor any healing before/during kernel upgrades..
20:22 pvh_sa wrale, what you're saying raises an interesting question: is there an easy way to see how close a volume is to critical failure? possibly something could be written that could parse the output of gluster volume status ?
20:23 wrale pvh_sa:  i appreciate this.. i wish to know the same
20:24 JoeJulian That's what monitoring software is for.
20:25 pvh_sa JoeJulian, well what's good for monitoring the state of gluster ?
20:25 JoeJulian I use icinga, semiosis I believe uses nagios. I'm sure there are others.
20:25 wrale My plan is to use Sensu, which uses nagios plugins
20:26 wrale icinga is based on nagios, correct?
20:26 JoeJulian yes
20:26 wrale Cool..
20:26 JoeJulian I just like the UI better.
20:26 semiosis kinda stopped bothering with nagios
20:26 wrale ditto on both nagios comments.. amen
20:26 semiosis it's still there, but it never tells me anything useful
20:27 SuperYeti joined #gluster
20:27 wrale i stupidly configured it to tell me when printers are down... one person hordes their network printer and disconnects it from the network every time i connect it... *shakes fist at usb*
20:28 JoeJulian I want a way of preventing glusterfsd from stopping, at least via init/systemd/upstart unless the volume heal info is empty.
20:28 SuperYeti Help, I just broke my volumes, issues a gluster volume stop volumename, now when i try to start with gluster volume start volumename i get the following error: volume start: images: failed: Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /mnt/storage/images. Reason : No data available
20:28 JoeJulian SuperYeti: mount /mnt/storage/images.
20:28 wrale JoeJulian: maybe runit?
20:29 SuperYeti JoeJulian, nope
20:29 JoeJulian SuperYeti: Did you format that filesystem? Why isn't the trusted.glusterfs.volume-id on that brick?
20:29 SuperYeti i have no idea, i ran the command from the gluster node
20:30 JoeJulian there is not gluster node.
20:30 JoeJulian @glossary
20:30 glusterbot JoeJulian: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
20:30 SuperYeti ok
20:30 SuperYeti i ran the stop command from a server
20:30 SuperYeti server1 in this instance
20:31 SuperYeti so i need to stop it on server2 as well before i restart it?
20:31 JoeJulian Nope
20:31 JoeJulian Wait... yep
20:31 JoeJulian er
20:31 JoeJulian what the?
20:31 JoeJulian If it's stopped, it should be stopped on all servers.
20:31 JoeJulian From the one command.
20:32 SuperYeti right, just checked, it is indeed stopped on both servers
20:32 SuperYeti just tried starting from server2
20:32 SuperYeti same error
20:33 JoeJulian When you created the volume, an ,,(extended attribute) trusted.glusterfs.volume-id was created at the brick root. That error is telling you that it's missing. Most commonly that's because the filesystem that held the brick isn't mounted.
20:33 glusterbot I do not know about 'extended attribute', but I do know about these similar topics: 'extended attributes'
20:33 SuperYeti https://bugzilla.redhat.com/show_bug.cgi?id=991084
20:33 glusterbot Bug 991084: high, unspecified, ---, vbellur, NEW , No way to start a failed brick when replaced the location with empty folder
20:33 JoeJulian One other possibility would be that the brick filesystem wasn't mounted when you created the volume and now you've mounted over the actual brick.
20:33 SuperYeti that seems like it's the same error
20:34 JoeJulian @alias "extended attributes" "extended attribute"
20:34 glusterbot JoeJulian: Error: This key has more than one factoid associated with it, but you have not provided a number.
20:34 SuperYeti the bricks roots a sub-directory of a mounted EBS volume
20:34 SuperYeti the brick's root is a sub-directory*
20:34 SuperYeti and it is indeed mounted, i see all the content there
20:34 SuperYeti the .glusterfs folder is there
20:34 JoeJulian @extended attributes
20:34 glusterbot JoeJulian: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
20:35 JoeJulian @alias "extended attributes" 1 "extended attribute"
20:35 glusterbot JoeJulian: (alias [<channel>] <oldkey> <newkey> [<number>]) -- Adds a new key <newkey> for factoid associated with <oldkey>. <number> is only necessary if there's more than one factoid associated with <oldkey>. The same action can be accomplished by using the 'learn' function with a new key but an existing (verbatim) factoid content.
20:35 JoeJulian @alias "extended attributes" "extended attribute" 1
20:35 glusterbot JoeJulian: The operation succeeded.
20:36 SuperYeti hmm attr wasn't installed
20:36 SuperYeti i wonder if that caused it
20:36 JoeJulian no
20:37 JoeJulian The functions that glusterfs uses to read those are kernel functions.
20:37 SuperYeti "getfattr -m .  -d -e hex /mnt/storage/images" gives me "trusted.gfid=0x00000000000000000000000000000001"
20:38 SuperYeti which I have no clue what it means lol
20:40 JoeJulian Was this volume created with a prior version of glusterfs?
20:40 SuperYeti yes
20:40 SuperYeti i had originally created it with the version in the repo
20:40 JoeJulian Ah, interesting...
20:41 SuperYeti then you guys helped me upgrade gluster with the ppa
20:41 msp3k left #gluster
20:41 JoeJulian Right. Let me see if there's an easy answer in the source.
20:41 SuperYeti wicked
20:41 SuperYeti thanks
20:41 SuperYeti s/repo/universe
20:45 JoeJulian I'm not sure if this will fix that, but could you try "glusterd --xlator-option *.upgrade=on -N"
20:46 Matthaeus joined #gluster
20:50 wrale JoeJulian: any thoughts on Distributed Replicated vs. Replicated, in the context of the six node cluster i mentioned?  I can't seem to understand how I should handle having two bricks per server.. Can I expect gluster to handle that, supposing I include both bricks for every node in the volume creation?
20:52 SuperYeti so shut down gluster first, then run that option?
20:52 JoeJulian I distribute across multiple bricks per server, ie "replica 3 server{1,2,3}:/brick1 server{1,2,3}:/brick2"
20:52 JoeJulian SuperYeti: yes.
20:52 SuperYeti librdmacm: couldn't read ABI version.
20:52 SuperYeti librdmacm: assuming: 4
20:52 SuperYeti CMA: unable to get RDMA device list
20:53 SuperYeti was the response from that command, is that good or bad?
20:53 JoeJulian That's fine.
20:53 JoeJulian Just means you're not using rdma.
20:53 SuperYeti do i need to run that on both servers?
20:53 JoeJulian yes
20:53 SuperYeti ok now start gluster service and try to restart volumes?
20:54 JoeJulian yes
20:54 SuperYeti so above you mention you run multiple bricks on each server, what does that help with out of curiosity
20:54 SuperYeti in case i'm rebuilding these lol
20:54 jmarley joined #gluster
20:54 jmarley joined #gluster
20:55 elyograg each of my gluster storage servers has eight bricks, each of which is 5TB.
20:56 social any dev around, I have some questions about storage.owner-uid storage.owner-gid and why they aren't updated dynamically on chown of root gfid?
20:56 wrale must be backed by SAN or raid.. unless drives that big are economical
20:56 wrale (SAN+raid) or just raid..
20:56 JoeJulian SuperYeti: If smartmon registers errors on a disk, I can kill just that brick, replace it, and heal just that one brick.
20:57 khushildep joined #gluster
20:57 SuperYeti ah, so each brick is on its own physical device in the server?
20:57 JoeJulian yes
20:57 SuperYeti ok, doesn't apply to me then heh
20:57 SuperYeti so uh, i guess since i'm still in POC i should just nuke these volumes, and recreate them?
20:57 elyograg twelve 4TB drives in the server, two RAID 5 arrays.  Each of those is split into four pieces with LVM.  This paradigm lets us create 5TB bricks no matter how big the disks in the server are.  with 1TB drives, each raid array would yield one 5TB brick .. and so on.
20:58 JoeJulian SuperYeti: If you want. I take it that it didn't work.
20:58 SuperYeti no bueno :(
20:58 wrale elyograg: that is interesting.. i would have guessed superior performance without RAID, but perhaps the hardware striping is indeed better (faster)?
20:59 _dist joined #gluster
20:59 elyograg it gets us two things: consistent brick size and the ability to handle drive failures by simply replacing the drive, no filesystem or gluster work necessary.
20:59 JoeJulian SuperYeti: You can, or you can just do this after setting $vol and $brick:
20:59 JoeJulian setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/$vol/info|cut -d= -f2|tr -d '-') $brick
20:59 wrale elyograg: ah.. fascinating..
21:00 SuperYeti too late, stuff deleted lol
21:00 SuperYeti but i will know if it happens again
21:00 JoeJulian It won't. :D
21:00 SuperYeti oh well, this way i'm starting with a volume built with the newest version, it has to be "more better"
21:00 SuperYeti lol
21:02 wrale Just read in a doc: "The number of bricks should be a multiple of the replica count for a distributed replicated volume".. that's no fun.. two bricks per, replica 3 == fail?
21:05 wrale JoeJulian: in response to your replica setup, you must have two servers, because otherwise, you'd probably want your third replica elsewhere, correct?  (four servers for "good quorum" would be the next step up?)... does this sound right?
21:08 wrale And just to be sure semantics are getting in the way.. When we say replica 3, when mean there are destinations, not four.. (because replica 3 doesn't mean original + 3 replicas, it means three altogether?)
21:08 pvh_sa with two servers, do you get redunancy? i.e. can the volume still be used if one of the servers goes down? that doesn't seem to work in my experiments
21:08 wrale s/are/aren't
21:09 wrale pvh_sa: seems that would be a split brain situation... (?) http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/
21:09 glusterbot Title: GlusterFS Split-Brain Recovery Made Easy (at joejulian.name)
21:10 [o__o] joined #gluster
21:10 wrale dunno.. feeling quite stupid, as usual.. :)
21:11 elyograg wrale: replica 3 does indeed mean that there are three total copies.
21:12 elyograg pvh_sa: that's what's supposed to happen.  works for me on my replica 2 system.  If you're mounting gluster via NFS, you need a floating IP that moves when there's a failure.
21:13 pvh_sa wrale, ok but split-brain is different to what I'm talking about. I'm talking about 'volume create blah0 replica 2 store1:/opt/glusterfs/part1 store2:/opt/glusterfs/part2'
21:13 pvh_sa and then store2 goes down....
21:14 pvh_sa *fires up test setup to see*
21:14 primechuck joined #gluster
21:16 elyograg pvh_sa: that worked in all my tests, and seems to work in production as well.
21:16 wrale pvh_sa: i think split brain is the outcome of losing one of two nodes, but this shouldn't be fatal
21:16 wrale elyograg: much thanks
21:18 pvh_sa elyograg, how do you mount the volume? because I do what I described just now... and then mount -t glusterfs head:replica1 /mnt (where head is neither store1 nor store2)
21:19 pvh_sa ah its working now... just took a while to complete in one case...
21:20 wrale I swear, every time I try to build an HPC, I need to fix a printer.. :)..
21:22 pvh_sa wrale, anyway, split-brain is different... from what I understand its caused by 1) brickA goes down, 2) brickA comes up again (so now healing starts) 3) brickB goes does (before healing is complete) 4) something is written to brickA, 5) brickB comes up
21:23 mtanner_ joined #gluster
21:23 pvh_sa so now brickA and brickB both have changes that the other doesn't have... its like when a git repo has multiple heads...
21:23 wrale pvh_sa: i see.. that makes sense
21:23 elyograg pvh_sa: is 'head' part of the cluster?  it would be part of the output from 'gluster peer status' on store1 and store2 if it is.
21:23 elyograg can the client actually successfully get to store1 and store2?  the client downloads the volume file from whichever machine you mount, then connects directly to the brick servers.
21:25 pvh_sa elyograg, yeah head is part of the cluster - I guess in production I need to take things a bit further and have failover for the mount somehow (in case 'head' is down)
21:27 elyograg pvh_sa: a shared IP address with software as simple as ucarp would take care of high availability for mounting.  for operation, the client machine will have to be able to directly talk to gluster ports for bricks on store1 and store2 at all times.
21:28 elyograg heartbeat or pacemaker work well too.  whatever you're familiar with.
21:28 wrale Needed to gracefully bring down a gluster cluster of any size.. Can I do that say, via parallel ssh?
21:29 wrale *If i... gee
21:29 Alex joined #gluster
21:29 wrale Also supposing the writing stopped and all was healed that can be
21:29 semiosis pvh_sa: ,,(rrdns) for ha mounting
21:29 glusterbot pvh_sa: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
21:31 elyograg rrdns works too.  personally, I prefer the shared IP address.  I'm doing my mounting via NFS, though.  it's *required* for NFS high availability.
21:32 semiosis right
21:32 _dist left #gluster
21:32 Dga joined #gluster
21:32 xymox joined #gluster
21:33 lkoranda joined #gluster
21:33 ctria joined #gluster
21:35 pvh_sa elyograg, and I see there is also backupvolfile-server that does failover
21:35 badone joined #gluster
21:37 wrale This document (cont.): https://access.redhat.com/site/documentation/en​-US/Red_Hat_Storage/2.0/html/Administration_Gui​de/sect-User_Guide-Managing_Volumes-Quorum.html
21:37 glusterbot Title: 10.9. Configuring Server-Side Quorum (at access.redhat.com)
21:37 wrale Says this: "By default, the quorum is met if the percentage of active nodes are more than 50% to the total storage nodes."
21:37 wrale Does this mean I do or do not have to configure quorum?  If there is a default value that I agree with, must I do anything?
21:38 wrale It seems there is binary on/off switch preempting the percentage...
21:38 elyograg if you have at least three replicas, you can use quorum.  You just have to enable it, I think it's a volume option.
21:38 wrale Oh, i see.
21:39 wrale elyograg: it give the example: "gluster volume set all cluster.server-quorum-ratio percentage%".. all here means, i take it, "all existing at this time".. as opposed to "exist or will ever exist"?
21:39 wrale s/give/gives
21:40 xymox joined #gluster
21:40 elyograg there's an idea in the works for arbiter nodes that would allow quorum on two-replica volumes (when there are other nodes, that is).  As far as I know, it hasn't actually made it into the codebase.
21:40 elyograg http://www.gluster.org/pipermail/glu​ster-users/2012-November/034747.html
21:40 glusterbot Title: [Gluster-users] Avoid Split-brain and other stuff (at www.gluster.org)
21:41 wrale Rather..Do 'gluster volume set all' commands apply to future volumes?
21:41 elyograg that's a really good question.  didn't know you could even do that.
21:41 wrale Glad you agree.. I'm baffled
21:43 semiosis the PPAs clearly warn people not to email me for support, yet every couple weeks, another email.  why am I surprised?  if they can't even follow my simple contact preferences, they can't be expected to set up a cluster filesystem either
21:44 semiosis ,,(meh)
21:44 glusterbot I'm not happy about it either
21:50 social ok I understand how xlators get updated if volume option changes but so far I didn't find whether xlator can change volume option :/ but it seems it should be
21:53 xavih_ joined #gluster
21:53 semiosis volume options are xlator options are volume options
21:53 mattapp__ joined #gluster
21:53 rpowell1 joined #gluster
21:53 khushildep joined #gluster
21:54 doekia joined #gluster
21:54 DV__ joined #gluster
21:54 RayS joined #gluster
21:54 hagarth joined #gluster
21:54 askb joined #gluster
21:58 social semiosis: to explain I'd love to add some volume options and update them from nfs and posix translators, for starters I'd like to update on for example posix_do_chown the brick-uid and brick-gid but I'm lame in C :/
21:58 glusterbot New news from resolvedglusterbugs: [Bug 839595] Implement a server side quorum in glusterd <https://bugzilla.redhat.com/show_bug.cgi?id=839595>
21:59 wrale I would be very happy to know how to set the default quorum type to server for future volumes.... Any ideas?
22:02 wrale "Though the quorum ratio is configured at the trusted storage pool level, you can choose to or either not to enforce quorum on a particular volume by setting cluster.server-quorum-type volume option."..Sounds great, but what about future volumes? https://access.redhat.com/site/documentation/en​-US/Red_Hat_Storage/2.0/html/Administration_Gui​de/sect-User_Guide-Managing_Volumes-Quorum.html
22:02 glusterbot Title: 10.9. Configuring Server-Side Quorum (at access.redhat.com)
22:03 theron_ joined #gluster
22:04 wushudoin joined #gluster
22:05 wrale "Quorum can now be enforced from the Trusted Storage Pool." a war of words: https://forge.gluster.org/gluster-docs-pro​ject/pages/GlusterFS_34_Release_Notes#Quor​um+enforcement+from+Trusted+Storage+Pool
22:05 glusterbot Title: Gluster Docs Project - GlusterFS_34_Release_Notes - Open wiki - Gluster Community Forge (at forge.gluster.org)
22:09 wrale maybe i can just put 'cluster.server-quorum-type server' in my glusterd.vol file to make the setting persistent?
22:11 jag3773 joined #gluster
22:11 mattapperson joined #gluster
22:11 hagarth joined #gluster
22:11 sroy__ joined #gluster
22:11 hybrid512 joined #gluster
22:11 japuzzo joined #gluster
22:12 JoeJulian wrale: no. That configuration file is for the management daemon configuration.
22:12 sputnik13 joined #gluster
22:12 JoeJulian not for volumes
22:13 jag3773 joined #gluster
22:13 zerick joined #gluster
22:13 wrale JoeJulian: How can we say we always want quorum? (I'm concerned about future volumes)
22:13 georgeh|workstat joined #gluster
22:13 JoeJulian Use puppet?
22:13 ColPanik joined #gluster
22:13 badone joined #gluster
22:14 wrale JoeJulian: I see.  Thanks
22:14 JoeJulian Would be a good feature request: a "defaults" volume or something.
22:14 JoeJulian file a bug
22:14 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
22:15 zerick joined #gluster
22:19 Matthaeus joined #gluster
22:36 mattappe_ joined #gluster
22:37 sroy_ joined #gluster
22:37 jobewan joined #gluster
22:37 jesse joined #gluster
22:38 hybrid512 joined #gluster
22:38 wrale Done..
22:39 hagarth1 joined #gluster
22:39 wrale https://bugzilla.redhat.co​m/show_bug.cgi?id=1071487
22:39 glusterbot Bug 1071487: unspecified, unspecified, ---, kparthas, NEW , Always default to cluster.server-quorum-type server?
22:45 diegows joined #gluster
22:45 glusterbot New news from newglusterbugs: [Bug 1071487] Always default to cluster.server-quorum-type server? <https://bugzilla.redhat.co​m/show_bug.cgi?id=1071487>
22:46 georgeh|workstat joined #gluster
22:46 ColPanik joined #gluster
22:49 sputnik13 joined #gluster
22:51 zerick joined #gluster
22:59 glusterbot New news from resolvedglusterbugs: [Bug 1069840] GlusterFS rpm compilation fails on CentOS 5.x <https://bugzilla.redhat.co​m/show_bug.cgi?id=1069840>
23:01 cjanbanan joined #gluster
23:03 Matthaeus joined #gluster
23:07 khushildep joined #gluster
23:08 fidevo joined #gluster
23:15 ekis_isa joined #gluster
23:21 y4m4 joined #gluster
23:31 * wrale wishes for a Google Sheets template that can easily calculate GlusterFS volume capacity based on Distributed, Replicated and so on..
23:40 khushildep_ joined #gluster
23:40 badone joined #gluster
23:40 ColPanik__ joined #gluster
23:43 sputnik13 joined #gluster
23:43 fidevo joined #gluster
23:43 fidevo joined #gluster
23:45 wrale Interesting thread: http://openstack.redhat.com/forum/discussion/1​41/openstack-on-glusterfs-data-locality-qs/p1
23:45 glusterbot Title: OpenStack on GlusterFS - data locality Qs - RDO (at openstack.redhat.com)
23:53 jporterfield joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary