Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-09-02

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 haomaiwa_ joined #gluster
00:23 haomaiwa_ joined #gluster
00:25 plarsen joined #gluster
00:30 haomaiwang joined #gluster
00:43 haomaiwang joined #gluster
01:14 jmarley joined #gluster
01:23 recidive joined #gluster
01:24 vimal joined #gluster
01:39 haomai___ joined #gluster
01:47 harish_ joined #gluster
01:54 gildub joined #gluster
01:57 haomaiwa_ joined #gluster
02:02 haomaiwa_ joined #gluster
02:14 elico joined #gluster
03:00 Lilian joined #gluster
03:15 haomai___ joined #gluster
03:16 kumar joined #gluster
03:20 kshlm joined #gluster
03:28 haomaiwa_ joined #gluster
03:42 hchiramm joined #gluster
03:51 shubhendu joined #gluster
03:52 haomaiw__ joined #gluster
03:55 rjoseph joined #gluster
03:57 ndarshan joined #gluster
04:03 RameshN joined #gluster
04:06 XpineX joined #gluster
04:06 nbalachandran joined #gluster
04:27 meghanam joined #gluster
04:27 meghanam_ joined #gluster
04:42 Rafi_kc joined #gluster
04:42 rafi1 joined #gluster
04:43 atinmu joined #gluster
04:44 anoopcs joined #gluster
04:49 ramteid joined #gluster
04:50 Lilian joined #gluster
04:53 kdhananjay joined #gluster
04:57 hagarth joined #gluster
04:59 jiffin joined #gluster
05:23 haomaiwang joined #gluster
05:27 nshaikh joined #gluster
05:28 haomaiwa_ joined #gluster
05:39 raghu` joined #gluster
05:43 haomaiwang joined #gluster
05:46 Philambdo joined #gluster
05:48 deepakcs joined #gluster
06:00 RaSTar joined #gluster
06:02 kanagaraj joined #gluster
06:13 cristov_mac joined #gluster
06:29 getup- joined #gluster
06:32 lalatenduM joined #gluster
06:46 itisravi joined #gluster
06:53 ctria joined #gluster
07:03 rjoseph1 joined #gluster
07:04 dusmant joined #gluster
07:08 bala joined #gluster
07:11 Thilam joined #gluster
07:11 Thilam hi
07:11 glusterbot Thilam: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
07:16 saurabh joined #gluster
07:16 mhoungbo joined #gluster
07:16 deepakcs joined #gluster
07:20 nbalachandran joined #gluster
07:21 atinmu joined #gluster
07:24 hagarth joined #gluster
07:25 deepakcs joined #gluster
07:27 bala joined #gluster
07:39 hchiramm joined #gluster
07:45 haomaiwa_ joined #gluster
07:47 Pupeno joined #gluster
07:51 tru_tru joined #gluster
07:57 fsimonce joined #gluster
07:59 rjoseph joined #gluster
08:00 atinmu joined #gluster
08:03 nbalachandran joined #gluster
08:05 pkoro joined #gluster
08:05 liquidat joined #gluster
08:06 hybrid512 joined #gluster
08:16 hagarth joined #gluster
08:25 glusterbot New news from newglusterbugs: [Bug 1136221] The memories are exhausted quickly when handle some special rpc message <https://bugzilla.redhat.com/show_bug.cgi?id=1136221>
08:28 ppai joined #gluster
08:31 harish_ joined #gluster
08:35 Slashman joined #gluster
08:47 Guest39717 joined #gluster
08:48 peema joined #gluster
08:50 ThatGraemeGuy joined #gluster
08:50 anands joined #gluster
09:00 dmyers1 joined #gluster
09:07 aravindavk joined #gluster
09:08 mbukatov joined #gluster
09:30 NigelSUK joined #gluster
09:36 NigelSUK morning :)
09:37 atalur joined #gluster
09:37 dusmant joined #gluster
09:37 dmyers joined #gluster
09:37 dmyers morning nigelSUK
09:38 dmyers i'm new to gluster. does anyone know if i need a minimum EC2 instance type for client/server requirements for gluster?
09:38 NigelSUK hey dmyers im new to and also running on EC2 :D
09:39 dmyers i saw that ec2 says the cheaper plans are low network perforance and os i wasnt sure if it would be problematic or if anyone else had some
09:39 dmyers suggestions*
09:39 dmyers sweet!
09:40 NigelSUK yeah the lower spec instances have low to moderate performance on the network, try a m1 large / small
09:40 dmyers the workload is mostly small files for hosting php apps
09:41 dmyers NigelSUK: I gotcha, kind of what i was thinking. would that mean both the client and servers ideally should match?
09:41 NigelSUK dmyers similar thing here, i'm still testing so just using small cheaper instances, havent worked out the exact spec of the live cluster instances yet
09:42 NigelSUK you wouldnt need the same for the client, but you dont want to go too small as the net performance will just be awful.
09:43 dmyers ah very cool. yeah i'm mostly a dev, but trying to get into more sysadmin/operations/devops so i wasn't sure if it mattered. i guess the client would cache it, but the server should be able to respond to a fetch pretty fast so i guess that makes sense
09:45 NigelSUK :) what distro are you planning on using ?
09:45 dmyers i was actually not sure on that either
09:46 dmyers i'm most familiar with ubuntu, but have worked with aws ami a bit and centos
09:46 NigelSUK ahh, well i dont know what's best, im still a newbie myself, but the guys here helped me get set up with ubuntu 14.04 and so far after a few niggles it seems ok
09:46 dmyers oh that's good to hear then!
09:47 NigelSUK semiosis and samppah were extremely helpful.
09:47 dmyers are you also working with small files? i was reading a bit about tuning gluster, but what i read was a few years ago and not sure if it's still needed
09:48 dmyers very awesome. this is a pretty large community which is another reason i wanted to use it. :)
09:49 NigelSUK :) we wil be using small files yup, apparently thats not gluster's strong point, but i'm not disuaded by it.
09:51 dmyers oh i gotcha
09:53 dmyers do you know of a good way to monitor the cluster's health? I'm sort of new to that too ha.
09:54 NigelSUK thats what im looking up right now
09:54 NigelSUK there's the cli commands, info / status etc
09:55 glusterbot New news from newglusterbugs: [Bug 1136296] Option to specify a keyfile needed for Geo-replication create push-pem command. <https://bugzilla.redhat.com/show_bug.cgi?id=1136296>
09:56 dmyers ah good point. i heard of nagios before, but i havent used it. i found this old documentation page about the commands if it helps. http://gluster.org/community/documentation/index.php/Gluster_3.2:_Monitoring_your_GlusterFS_Workload
09:56 atinmu joined #gluster
09:57 NigelSUK ah that will help, thanks!
09:58 NigelSUK i'm looking for a nice failover solution, sussed out how to select a server at boot but need to figure out how to switch to a different server if it goes offline during normal operation
09:59 qdk joined #gluster
09:59 dmyers i was wondering about that too
10:00 NigelSUK im hoping someone here may have a better idea than me, ive never dealt with HA stuff before
10:00 dmyers at boot when you mount a gluster server? is it supposed to be the master one or does it not matter? i wonder if you could throw ha proxy into it
10:00 NigelSUK you may find this helpful though, specifically the fstab volume config file.. http://www.jamescoyle.net/how-to/439-mount-a-glusterfs-volume
10:01 NigelSUK dmyers my understanding is the volume info is grabbed from whatever server you mount, and a server selected from that
10:01 dmyers oh i see
10:04 NigelSUK i'm sure im not explaining it right but i've only had 1 cup of coffee so far lol
10:05 dmyers ha, i'm still up, my sleep schedule is backwards.
10:05 NigelSUK oh god lol
10:05 dmyers good link i didn't know about the volume config file
10:05 NigelSUK nor me until yesterday, was happy to find that !
10:06 dmyers do you know how many gluster servers you plan on running? i'm trying to decide how many to use myself
10:06 NigelSUK we have 2 planned at the moment, and 3 clients
10:06 NigelSUK it's only a medium sized web cluster, any more than 2 would be overkill i think
10:07 NigelSUK as to how it will perform on AWS, not to sure.
10:07 dmyers oh nice, that's what i was thinking of starting out with too
10:10 NigelSUK start small, and scale up
10:13 SmithyUK joined #gluster
10:14 dmyers yea totally, im on a startup budget too hah which is why i was also wondering on the networking on ec2
10:15 NigelSUK i know that feeling all to well
10:23 hchiramm gluster, who is who page is in its first version..
10:23 hchiramm http://www.gluster.org/community/documentation/index.php/Who_is_Who
10:24 hchiramm please feel free to edit and pass comments.
10:24 hchiramm hopefully it will help some one :)
10:25 glusterbot New news from newglusterbugs: [Bug 1136312] geo-rep mount broker setup has to be simplified. <https://bugzilla.redhat.com/show_bug.cgi?id=1136312>
10:33 atinmu joined #gluster
10:34 MattAtL joined #gluster
10:35 mbukatov joined #gluster
10:35 glusterbot New news from resolvedglusterbugs: [Bug 928656] nfs process crashed after rebalance during unlock of files. <https://bugzilla.redhat.com/show_bug.cgi?id=928656>
10:36 ricky-ti1 joined #gluster
10:40 dmyers NigelSUK: do you know of a perferred standard path for mounting in ubuntu? a guide on gluster just used the main dir like /storage-pool, but i thought most sysadmins used something else standard
10:41 NigelSUK dmyers some of the guides ive seen tend to use /gluster/data
10:43 dmyers ah okay cool, i didn't know if you could use /mnt/ or if that was automatic by OS or not, at my old job they used /export/ i think it's a centos thing or sun solaris
10:45 peema In my head /mnt is the /usr/local of disk mounts.
10:46 peema Tempted to set up autofs to mount under /gfs
10:47 dmyers ohh interesting peema. i'm kind of new to sysadmin stuff other than just apache/php and googling didnt really say if it was reserved by os or my queries were bad
10:49 dmyers nvm i found this http://www.pathname.com/fhs/pub/fhs-2.3.html#MNTMOUNTPOINTFORATEMPORARILYMOUNT
10:49 glusterbot Title: Filesystem Hierarchy Standard (at www.pathname.com)
10:53 edward1 joined #gluster
10:55 qdk joined #gluster
10:56 meghanam_ joined #gluster
10:58 meghanam joined #gluster
10:58 elico joined #gluster
11:00 dmyers does anyone know the best practice to make sure users/groups are setup across hosts (gluster server and clients)?
11:03 elico dmyers: On what level? for administration or another level?
11:03 plarsen joined #gluster
11:04 NigelSUK LDAP maybe?
11:04 dmyers well im new to sysadmin sort of so i didnt know if there was an easy way (passwd file?) and if the server has to have the users or if i can just have the clients set up right
11:05 dmyers gluster server has to be in sync i mean
11:06 elico dmyers: in sync, what should be in sync? for permissions of files inside the bricks or for admin access?
11:06 dmyers oops i mean permissions for folders and files
11:07 dmyers inside the bricks
11:07 elico well it depends but it's being done on the client side of glusterfs.
11:08 elico if I change ownership on a mounted volume it changes the whole cluster permissions.
11:10 dmyers ah that make sense. i plan on having several web servers mounted as glusterfs clients with users writing to files and having to keep them in sync across the web servers as it scales
11:11 todakure joined #gluster
11:11 elico OK but take in account that Glusterfs do not work well with lots of small files..
11:11 elico it will work but if you need lots of read\write access for small files it will probably move slow sometimes.
11:12 dmyers oh hmm
11:12 NigelSUK elico is that a permenant issue with glusterfs or is it something that can be improved ?
11:13 dmyers that's interesting, i thought there were some things that could be tuned?
11:13 dmyers like maybe http://gluster.org/community/documentation/index.php/Translators/performance/stat-prefetch
11:15 ninkotech__ joined #gluster
11:15 marbu joined #gluster
11:15 RameshN joined #gluster
11:16 dmyers this post about performance is 4 years old, but i had found it looking into tuning it (havent started testing yet) http://www.sirgroane.net/2010/03/tuning-glusterfs-for-apache-on-ec2/
11:16 dmyers io-cache and io-threads and stat-prefetch looked interesting for small files
11:16 elico dmyers: NigelSUK: There are options to tune it up and seek for a reasonable response time but the main destination for Glusterfs is lots of Big files and not Lots of Small files.
11:17 elico I have tried to use GlusterFS witnout tuning as a Dovecot backend and it was really slow(using two nodes with one disk on each).
11:19 dmyers ah hm i gotcha, i'm hoping to reduce those reads with opcode caching in my php so that would help some and maybe caching, for the most part my files won't change much (wordpress)
11:19 elico I can try to run a test again but the last results was so low that I went back to a simple MDAM RAID as a basic disk failure redundency.
11:20 recidive joined #gluster
11:20 elico dmyers: Wordpress ontop of glusterfs is wrong by nature as far as I can tell
11:21 elico if you mind waiting couple minutes(10-20) I will be back then after a good wash.
11:21 dmyers ah alright cool
11:22 elico (maybe we can even run a live test here :D )
11:24 bala joined #gluster
11:38 ira joined #gluster
11:44 recidive joined #gluster
11:47 edwardm61 joined #gluster
11:48 elico dmyers: PM ping *
11:49 LebedevRI joined #gluster
11:57 anands left #gluster
11:58 dusmant joined #gluster
12:00 ndevos *reminder:* Gluster bug triage meeting in #gluster-meeting starting *now*
12:02 elico ndevos: can i get in?
12:02 ndevos elico: sure!
12:02 elico thanks! I'm there
12:04 todakure joined #gluster
12:06 jiku joined #gluster
12:08 Thilam joined #gluster
12:16 plarsen joined #gluster
12:18 B21956 joined #gluster
12:21 B21956 joined #gluster
12:22 diegows joined #gluster
12:24 jdarcy joined #gluster
12:25 glusterbot New news from newglusterbugs: [Bug 1136349] DHT - remove-brick - data loss - when remove-brick with 'start' is in progress, perform rename operation on files. commit remove-brick, after status is 'completed' and few files are missing. <https://bugzilla.redhat.com/show_bug.cgi?id=1136349>
12:26 jiffin joined #gluster
12:34 mdavidson joined #gluster
12:40 RameshN joined #gluster
12:48 aravindavk joined #gluster
12:49 ninkotech joined #gluster
12:49 ninkotech_ joined #gluster
12:52 LHinson joined #gluster
12:53 NigelSUK joined #gluster
12:54 RameshN joined #gluster
12:57 kanagaraj joined #gluster
12:57 _NiC joined #gluster
12:58 todakure_ joined #gluster
13:00 _Bryan_ joined #gluster
13:01 todakure_ ,,ctdb
13:01 LHinson1 joined #gluster
13:01 todakure_ how to use bot ? kkkk
13:02 JoeJulian @ctdb
13:02 glusterbot JoeJulian: http://www.gluster.org/community/documentation/index.php/CTDB
13:06 bennyturns joined #gluster
13:06 asku1 left #gluster
13:07 mick27 joined #gluster
13:07 todakure_ :D
13:09 getup- joined #gluster
13:10 elico I have tried in the past to use GlusterFS as a backend for dovecot.
13:10 elico There were couple issues with it but I never tried to do something about it by tuning or something else.
13:10 elico The issues were similar for both NFS and FUSE mount.
13:10 elico Any directions on it>
13:11 skippy joined #gluster
13:16 ndevos elico: I'm not sure, I've not used it myself like that, but there are others that seem to use glusterfs as an email-storage
13:16 ndevos elico: maybe sending an email to gluster-users@gluster.org gets some useful reactions
13:16 elico I am seeking for their suggestions.
13:17 elico I have tried it twice already in the past.
13:17 tdasilva joined #gluster
13:18 msmith joined #gluster
13:18 ndevos well, glusterfs isnt very well suitable for small files, big files are a much better use-case...
13:18 ndevos it should work, but just dont expect very high performance
13:18 JoeJulian The paid version of dovecot has features that would work better.
13:18 mojibake joined #gluster
13:19 elico JoeJulian: what???
13:20 JoeJulian It's been forever since I looked at that...
13:20 shubhendu joined #gluster
13:21 kdhananjay joined #gluster
13:21 JoeJulian they had some hybrid mbox/maildir format iirc.
13:21 elico Ho
13:22 hagarth joined #gluster
13:23 dusmant joined #gluster
13:31 Bnch joined #gluster
13:32 julim joined #gluster
13:33 Bnch JoeJulian: okay, imagine 2 folders in sync with no xattr's and no .glusterfs dir, how to make a replica 1 brick out of it ?
13:33 LHinson joined #gluster
13:34 JoeJulian The "best" way would be to wipe the second one and let gluster create all the attributes and let it handle the sync. I understand that's not the direction you would like to go with it, however.
13:35 bennyturns joined #gluster
13:36 JoeJulian The problem arises that gluster was not designed to have data on the bricks when you create a volume. By stripping all the xattrs and .glusterfs directories, you've essentially done just that. You have a bunch of unmanaged files and directories on two bricks.
13:36 Bnch JoeJulian: i had a little blackout 2nite :D
13:37 JoeJulian Can probably be made to work though.
13:38 Bnch in fact, when i use nfs mount and lookup the files , it starts to assign gfid brick and dht xattrs
13:38 JoeJulian To begin with, I would just start one single brick, mount it, and walk the tree from that mount.
13:38 JoeJulian right
13:38 Bnch i did that THREE times now with other brick
13:38 Bnch *bricks
13:38 Bnch brick->bricked->NEW VOLUME
13:39 Bnch JoeJulian: the "move directory" thing is really critical as i see now
13:40 JoeJulian there's a bunch of critical bits...
13:42 Bnch when "input/output error" arised on a specific folder, i tried to rename the directory from the working mount, in fact the result was a directory in that directory bot linked to the same folder but with different inode numbers, also in the .glusterfs the corresponding "link" was a folder (i never saw any .glusterfs/*/*/$FILE stat'ing as dir)
13:42 JoeJulian After walking the tree with one brick up, since you don't seem to want to resync a blank brick, I would make a script to copy the gfid and trusted.dht attributes from the "up" brick to the "down" one. I'd probably also make sure that files are files and directories are directories.
13:42 Bnch *both
13:43 Bnch rsync with X^^
13:43 Bnch and --dry-run
13:43 JoeJulian You don't want the trusted.afr
13:43 Bnch ^^thx
13:43 Bnch is there a hack to create a replica level 1
13:43 JoeJulian Copying the afr would leave then split-brain.
13:44 JoeJulian No. Everyone I know of just uses gluster.
13:44 Bnch JoeJulian: is there a hack to create a replica level 1 and stack it up , just add the seconde pre-synced brick ?
13:45 JoeJulian Why does pre-sync matter? I've never gotten anyone to answer that one.
13:45 Bnch redundance
13:45 JoeJulian But you're down while creating that pre-sync, right?
13:45 Bnch the data are already there,
13:46 Bnch glusterfs-server is shut down are volumes are stop when using tools other then gluster
13:46 Bnch in fact the "pre-sync" is a result of a gluster volume cleared out
13:47 Bnch there are several other (untouched) replica 2 vols
13:48 Bnch they are still in sync and have all xattrs
13:48 JoeJulian Ok, you have a brand new hard drive. You want to create a replica of one single brick of data. You add that brick (setting the replica count) and gluster replicates it, or, you rsync the data from the single drive across the same network connection from and to the same hardware. Where's the advantage?
13:48 jmarley joined #gluster
13:48 Bnch the data is already there
13:48 Bnch the advantage is not moving several TB's over a Gbit connection
13:49 JoeJulian So you're rsyncing locally?
13:49 JoeJulian Tennis-shoe network?
13:49 Bnch nope, rsync is just use to see if the files match all attributes, md5sum to  verify content
13:50 JoeJulian There's no difference there. Gluster does that too.
13:50 JoeJulian Ok, never mind that. I was just curious.
13:52 Bnch JoeJulian: (and if, i would have 2 brand new hard disks deployed in 2 servers and rsync into a fuse mount ;) )
13:53 JoeJulian make sure to save yourself some headache with that and use --inplace.
13:53 NigelSUK joined #gluster
13:53 ndevos Thilam: are you looking into that backport?
13:53 Bnch JoeJulian: you mean any operation, also moving into a fuse mount ?
13:54 JoeJulian So if you have the same files and directories, they all have matching gfid and the directories all have matching trusted.dht, you should be in a pretty good place.
13:54 ron-slc joined #gluster
13:54 Bnch ^^ THX^10
13:54 raghu joined #gluster
13:55 JoeJulian Bnch: Totally. Otherwise rsync creates a tempfile, with a filename that hashes to some random brick. Once the file is renamed it no longer hashes to that brick and now is sub-optimally located for the dht lookup.
13:56 gmcwhistler joined #gluster
13:57 Bnch JoeJulian: yeah it complained very often ... "file removed while syncing" or so ..
13:57 goo__ joined #gluster
13:57 JoeJulian which sounds like you have connectivity issues.
13:59 anoopcs joined #gluster
13:59 Bnch JoeJulian: nope, i migrated from that servers directly into a new fuse mounted directory..due to a hard disk fail one disk is supplied via nbd , i would have seen ATA errors in syslog )
13:59 jobewan joined #gluster
14:00 Bnch JoeJulian: its that rsync issue , rsync creates the tempfile and removes it , gluster spits a log that the file was removed before syncing (the tempfile)
14:00 JoeJulian It wouldn't have been syncing if it had written to both replicas from the fuse client, it would have already been synced.
14:00 simulx joined #gluster
14:01 JoeJulian That implies that the fuse client was having issues being connected to a brick.
14:01 LHinson joined #gluster
14:02 Bnch JoeJulian: may it be that the fuse mount tends to reach the hostnames instead of the fqdn's ?
14:02 aravindavk joined #gluster
14:03 Bnch JoeJulian: in fact i also watched the network traffic to be simultaneous
14:03 JoeJulian depends on how your volume is configured. However the bricks are named is how the client will try to find them.
14:03 Bnch JoeJulian: fqdn'ed
14:06 jobewan joined #gluster
14:13 tdasilva joined #gluster
14:18 kanagaraj joined #gluster
14:20 andreask joined #gluster
14:21 wushudoin joined #gluster
14:26 dusmant joined #gluster
14:28 aravindavk joined #gluster
14:29 mojibake It was brought up by someone last week that they were setting up Volumes for each website client. Originally I was thinking I was going to have one big volume. But it seems it would make more sense to have a volume for each site, as the security and quotas could be set. Any thoughts or advice regarding having less than 100 volumes? Versus one big volume and trying to handle permissions for each site? Is there a limit to new of Volumes?
14:31 pdrakeweb joined #gluster
14:31 JoeJulian Just memory management. Make sure you tune down the performace.cache-size to keep your servers from overrunning your ram.
14:31 mojibake EDIT: Is there a limit to number of volumes?
14:32 andreask left #gluster
14:32 mojibake setting performance.cache on the volume is for the server and the client? or just the server, or just the client?
14:33 JoeJulian For 60 bricks on a little 32Gb server, I set mine to 8MB which somehow seems to get multiplied to the point where that uses most of the ram.
14:33 JoeJulian I set it for the volume.
14:33 mojibake OK.. I will need to do some testing. Thank you for the advice.
14:35 xleo joined #gluster
14:36 kkeithley JoeJulian, re: your BZ https://bugzilla.redhat.com/show_bug.cgi?id=1127140.  You say you saw the leak in 3.4.4, wondered if it was a dupe of 1112844, which was fixed in 3.4.5. Are you still seeing the leak in 3.4.5?
14:36 glusterbot Bug 1127140: unspecified, unspecified, 3.4.6, gluster-bugs, NEW , memory leak
14:39 JoeJulian kkeithley: Yes, just got back last night from vacation. I'll update the bug later today.
14:39 kkeithley okay, thanks
14:43 RameshN joined #gluster
14:44 Norky joined #gluster
14:45 mojibake JoeJulian: Just confirming performance.cache-size, looks like default is 32mbs, and you lowered to 8mbs for your volumes. http://gluster.org/community/documentation/index.php/Gluster_3.2:_Tuning_Volume_Options
14:50 mojibake JoeJulian: Did you adjust performance.io-thread-count  at all?
14:52 lmickh joined #gluster
14:53 NigeyUK| joined #gluster
14:53 lkthomas_ left #gluster
14:57 bala joined #gluster
15:01 luckyinva joined #gluster
15:02 luckyinva can anyone here explain to me how to enable security on the volumes?  I can’t seem to find any documentation on how to apply it
15:03 kdhananjay joined #gluster
15:04 coredump joined #gluster
15:14 monotek joined #gluster
15:15 siel joined #gluster
15:17 luckyinva can anyone here explain to me how to enable security on the volumes?  I can’t seem to find any documentation on how to apply it
15:19 nueces joined #gluster
15:20 dberry joined #gluster
15:20 dberry joined #gluster
15:22 mick27 joined #gluster
15:25 meghanam joined #gluster
15:25 recidive joined #gluster
15:27 meghanam_ joined #gluster
15:32 dtrainor joined #gluster
15:32 mhoungbo joined #gluster
15:34 kanagaraj joined #gluster
15:37 msbrogli joined #gluster
15:38 msbrogli Hi. I’m having some performance issues with my replication. I read about the translators, but I don’t know how can I list which translators are actually being used in my volume. Can anyone help me?
15:40 semiosis what kind of performance issues?
15:40 msbrogli It is very slow when dealing with many small files (like git repositories).
15:41 msbrogli I’m trying to check which translators are being used in the server and the client, but I don’t know how. I already did a lsof and checked the opened libs, but I don’t know if all are in use and in which order.
15:43 msbrogli I don’t know if checking the xlators is the right way, but as far as I could understand from doc it seems to be.
15:44 semiosis unfortunately small file performance is usually limited mostly by network latency
15:44 bala joined #gluster
15:44 semiosis what kind of network connects the client to the servers?
15:45 msbrogli A gigabit local ethernet network.
15:45 NigeyUK| can anyone see a problem with this volfile? calling it from fstab and i just get 2failed to mount" error in boot.log :|
15:45 msbrogli I’m using the default config of gluster. Is it good for small files?
15:45 NigeyUK| http://pastebin.com/m2FRYh8q
15:45 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
15:45 semiosis NigeyUK|: no fuse xlator
15:46 semiosis NigeyUK|: but why mess with volfiles?  whats wrong with mount -t glusterfs server:volname...?
15:46 NigeyUK| oh this is for the web server clients
15:47 msbrogli semiosis: I created the volumes using the cmd line. How can I export the equivalent vol files of them?
15:47 NigeyUK| vol config add a bit of redundancy
15:47 semiosis NigeyUK|: where'd you get that idea?  for mounting redundancy use ,,(rrdns)
15:47 glusterbot NigeyUK|: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
15:48 NigeyUK| semiosis got the idea from here .. http://www.jamescoyle.net/how-to/439-mount-a-glusterfs-volume
15:48 peema left #gluster
15:48 semiosis NigeyUK|: well that's just bogus
15:49 semiosis NigeyUK|: it didnt even work when you followed the directions
15:49 NigeyUK| heh it does seem to be slightly non functioning :|
15:51 msbrogli semiosis: how can I check which xlators are in use? And how can I change the xlators of a volume?
15:51 msbrogli Both in server and client side.
15:51 toordog-work joined #gluster
15:52 semiosis msbrogli: the only recommended way to change settings is through ,,(options)
15:52 glusterbot msbrogli: See config options and their defaults with 'gluster volume set help'; you can see the current value of an option, if it has been modified, with 'gluster volume info'; see also this page about undocumented options: http://goo.gl/mIAe4E
15:52 toordog-work Hi, I would need some general behavorial information about gluster.   We have around 6tb of music file for streaming and I'd like to have a bigger understanding of how gluster behave in different context and what is usually ideal context.
15:53 NigelSUK joined #gluster
15:56 semiosis toordog-work: ask away
15:58 MattAtL joined #gluster
16:01 recidive joined #gluster
16:01 Pupeno_ joined #gluster
16:02 msbrogli semiosis: Do you have any hint to discover what is the cause of performance issues? For instance, in the local storage a git diff is performed in 0.4 seconds and in the mounted server it takes almost 18 seconds.
16:04 semiosis like i said before, most often it's network latency
16:05 msbrogli I’m monitoring the network and it is absolutely fine (the maximum rate is 70Mbps). I already stressed it using netcat and it reaches 650Mbps without any problem.
16:05 semiosis you might get better performance with an NFS client due to caching, but you lose redundancy....
16:06 semiosis msbrogli: that's throughput, not latency.  whats the max ping time from client to either server?
16:06 semiosis whenever a file is accessed the client makes several round trips to each replica to check file replication integrity
16:06 msbrogli The maximum latency to both storages is 0.524ms. Is it so bad?
16:07 semiosis for small files that check dominates the time to read the file
16:07 semiosis if your git diff has a lot of files to check that adds up
16:08 msbrogli I guess I never had any performance issue with NFS because of the caching, right? Can I increase caching in gluster to increase performance? Another way to go is to use quickread xlator - I checked and it is loaded in the client side, but I’m not sure if it is in use.
16:09 jbrooks joined #gluster
16:10 semiosis not that i know of
16:12 JoeJulian mojibake: No, I didn't adjust io-thread-count.
16:13 msbrogli semiosis: In this repository server I never have used a local storage. I already used NFS, DRBD and iSCSI and all of them worked fine. I know that I have a replica of my files in two storages and gluster must perform some checks, but does it make sense to slow down this much? Isn’t it “just” read from both places and check the xattrs?
16:13 anoopcs joined #gluster
16:14 mojibake JoeJulian: Thank you.
16:16 recidive joined #gluster
16:17 bala joined #gluster
16:17 bennyturns joined #gluster
16:18 semiosis msbrogli: i'm out of ideas, sorry i dont have the magic answer you're looking for.
16:18 hagarth joined #gluster
16:18 semiosis msbrogli: for what it's worth, i decided to not use glusterfs for my git repo storage, because it was too slow :/
16:20 JoeJulian luckyinva: http://blog.gluster.org/author/zbyszek/
16:20 luckyinva Thanks Joe
16:20 msbrogli semiosis: thanks for your help and your time. :)
16:20 semiosis yw. good luck
16:20 NigelSUK semiosis does gluster continuously check the status / health of the bricks? say for example fs02 went offline, would it pick that up by default, or do i need some kind of heartbeat script to do that for me?
16:21 semiosis NigelSUK: the fuse client is always connected to all bricks in the volume.  if one goes down (and there's a replica of it still online) the client just marks the brick unavailable & continues working with the rest.
16:22 semiosis depending on the nature of the outage, the client may hang for ,,(ping-timeout) before marking the brick unavailable & continuing
16:22 glusterbot The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
16:22 semiosis graceful restart won't cause ping-timeout.  pulling a network cable will.
16:23 NigelSUK ahh right, oki, i'll do some reading on this then, seems silly to make the boot mounting redundant if there's no checking if a server just dies through its normal operation.
16:26 doo joined #gluster
16:27 JoeJulian Hehe, semiosis, I commented too. :D
16:29 _Spiculum Does anyone have any glusterfs consultants that they recommend?
16:30 _Spiculum we're having various issues that I'd prefer to bring in an experienced expert to come in and help with our environment
16:31 _Spiculum http://www.hastexo.com/knowledge/storage-io/glusterfs lost their glusterfs guy
16:31 glusterbot Title: GlusterFS | www.hastexo.com (at www.hastexo.com)
16:31 nbalachandran joined #gluster
16:31 semiosis _Spiculum: pm
16:39 neofob joined #gluster
16:39 and` joined #gluster
16:40 tdasilva joined #gluster
16:42 juhaj joined #gluster
16:43 kmai007 joined #gluster
16:44 kmai007 hey yall, last week i added a +2bricks to my 4brick gluster....ran rebalance, and its still running...my heart is aching on how long its taking
16:45 kmai007 its about 3T
16:45 kmai007 should I be worried?
16:48 JoeJulian kmai007: what version?
16:48 kmai007 glusterfs3.4.2-1
16:48 JoeJulian yeah, it's not going to finish.
16:48 kmai007 hot damn
16:48 JoeJulian go to 3.4.5
16:48 kmai007 do you suggest 3.5.2?
16:48 kmai007 ok, 3.4.5 it is
16:49 kmai007 i'll stop it and "upgrade"
16:49 JoeJulian or that. I haven't made up my mind on 3.5 yet.
16:49 kmai007 JoeJulian: while you've been out, i recently went prod 2 weeks ago
16:49 kmai007 all success
16:49 kmai007 thanks to everyone on #gluster and mailing-list
16:49 JoeJulian awesome!
16:49 JoeJulian Did you blog/tweet/whatever?
16:49 kmai007 now its all about documentation, and trying to hand this off to the next sucker
16:50 kmai007 hahahahha
16:50 kmai007 i don't social blog/tweet
16:50 JoeJulian kmai007: Please do. It's one really easy way to give back.
16:50 kmai007 sure where do i start? how about google+ ?
16:50 kmai007 i already have an account
16:51 JoeJulian sure, that'd be great
16:51 kkeithley we like it when people say nice things about Gluster. ;-)
16:51 JoeJulian btw... blogging/social media is how you become known as an expert...
16:51 JoeJulian that's when the big bucks start rolling in. ;)
16:51 kmai007 man u do not want me to be an expert
16:51 kmai007 hahahhaha, ok as long as there is some $$$ left over from what you claim
16:52 kmai007 ;-)
16:52 JoeJulian Nothing makes you an expert faster than explaining to other people.
16:52 kmai007 yehi've tried to be a voice here when you're out
16:52 kmai007 i know _dist is a great voice
16:52 kmai007 so is semosis?
16:53 JoeJulian Need to get the johnmark, JustinClift, or equivalent to rekindle the community appreciation program.
16:54 NigelSUK joined #gluster
16:55 * semiosis needs moar t-shirts
16:55 semiosis seriously though, i need some for my students now!
16:55 kmai007 https://plus.google.com/107393626046519351298/posts/ZzyR6srfZHU
16:57 glusterbot New news from newglusterbugs: [Bug 1109613] gluster volume create fails with ambiguous error <https://bugzilla.redhat.com/show_bug.cgi?id=1109613>
17:00 skippy I published this late Friday, thought folks might like to peek at it: https://github.com/covermymeds/puppet-gluster
17:00 glusterbot Title: covermymeds/puppet-gluster · GitHub (at github.com)
17:00 skippy the purpleidea puppet module did a little too much for my tastes.
17:00 JoeJulian btw... I'm not going to Paris. My talk was not accepted.
17:01 LHinson joined #gluster
17:02 PeterA joined #gluster
17:04 semiosis boo
17:05 PeterA Anyone got a chance to look into the following bugs?
17:05 PeterA https://bugzilla.redhat.com/show_bug.cgi?id=1037511
17:05 glusterbot Bug 1037511: high, unspecified, ---, vbellur, NEW , Operation not permitted occurred during setattr of <nul>
17:05 PeterA https://bugzilla.redhat.com/show_bug.cgi?id=1030200
17:05 glusterbot Bug 1030200: medium, unspecified, ---, rhs-bugs, NEW , DHT : file rename operation is successful but log has error 'key:trusted.glusterfs.dht.linkto error:File exists' , 'setting xattrs on <old_filename> failed (File exists)'
17:05 PeterA https://bugzilla.redhat.com/show_bug.cgi?id=1016478
17:05 glusterbot Bug 1016478: high, high, ---, rgowdapp, CLOSED ERRATA, quota: dir quota limit mismatches in calculation with actual amount of data in it
17:06 nage joined #gluster
17:08 JoeJulian PeterA: best is to either follow up on the bug itself, or email the assignee directly. We don't know anything more than what bugzilla says.
17:09 MacWinner joined #gluster
17:09 PeterA ic
17:09 PeterA will do thanks!
17:11 PeterA two of those are still in New status and no one assigned yet…
17:11 JoeJulian For those, email gluster-devel@gluster.org
17:11 PeterA thanks
17:12 purpleidea skippy: more info?
17:12 luckyinva can someone point me to where the auth.allow file gets created ?
17:12 purpleidea skippy: also, anything extra you don't want it to do is optional... typically the defaults are sane too... anything that doesn't fit that bill, let me know
17:15 diegows joined #gluster
17:15 deedlee joined #gluster
17:23 skippy purpleidea: I realize that the extra stuff is optional.  It felt a little overwhelming working past the bits I knew I didn't need.
17:23 dtrainor joined #gluster
17:24 skippy purpleidea: also, doing all that extra stuff is a big divergence from our current purpose-built approach to Puppet modules.  Small parts, loosely coupled; and all that.
17:24 skippy it's not a slight against your work.  It's just not a good fit for our needs.
17:24 dtrainor joined #gluster
17:25 JoeJulian purpleidea: http://goo.gl/lXicgD
17:26 skippy lawl
17:27 kmai007 so remind me again, for an upgrade, server first, then clients, or clients first, then server?
17:27 kmai007 i 4getz
17:27 JoeJulian clients first
17:27 kmai007 gracias amigo
17:27 anoopcs joined #gluster
17:28 tdasilva joined #gluster
17:28 recidive joined #gluster
17:28 zerick joined #gluster
17:30 anoopcs joined #gluster
17:30 luckyinva when setting gluster volume set auth.allow ipadd   is there a way to list the allowed entries?  Can someone explain to me where the data is stored or how to view it?
17:31 kmai007 gluster volume info <volume>
17:31 kmai007 should display the feature of auth.allow and its clients
17:32 kmai007 i don't recall where its dynamically stored...somewhere in the .vol file
17:32 luckyinva thanks
17:33 JoeJulian .vol files are dynamically generated. The actual settings that convert to that are in the .info
17:34 luckyinva great thank you for the answer and your time
17:34 lmickh joined #gluster
17:34 RameshN joined #gluster
17:34 LHinson joined #gluster
17:37 kmai007 should I upgrade all my storage bricks
17:37 kmai007 before i run a heal check ?
17:38 LHinson1 joined #gluster
17:39 JoeJulian yes
17:40 kmai007 thanks JoeJulian
17:40 a1 joined #gluster
17:41 rwheeler joined #gluster
17:44 PeterA1 joined #gluster
17:50 Rafi_kc joined #gluster
17:50 dtrainor joined #gluster
17:51 aravindavk joined #gluster
17:51 kmai007 on gluster3.4.5 does the heal info output dynamically update? i know in prior version glusterd had to be restarted to have that updated....?
17:52 dtrainor joined #gluster
17:54 NigeyUK| joined #gluster
17:58 neofob left #gluster
18:02 kmai007 alright, fingers crossed, i've updated from glusterfs3.4.2-1 -> glusterfs.3.4.5; restarted all my gluster nodes, and started the "rebalance"
18:03 atrius joined #gluster
18:03 NuxRo joined #gluster
18:05 anoopcs1 joined #gluster
18:05 pasqd joined #gluster
18:05 kmai007 the rebalance logs looks like its chugging along, with no apparent errors
18:08 anoopcs joined #gluster
18:12 anoopcs1 joined #gluster
18:12 Rafi_kc joined #gluster
18:15 anoopcs joined #gluster
18:21 qdk joined #gluster
18:24 rotbeard joined #gluster
18:25 anoopcs1 joined #gluster
18:26 msbrogli joined #gluster
18:26 PeterA1 is there a way to sync the quota calculation on a directory??
18:26 PeterA1 i already tried to do a find .
18:27 julim joined #gluster
18:32 guntha joined #gluster
18:43 toordog-work joined #gluster
18:56 luckyinva gluster on 3.5.2 - on rhel 7 - doesn’t appear to stop and restart properly  - I have ensured killall is in the correct location but when I grep for glust after executing a stop I see gluster still up, is this a known issue?
19:00 skippy is it actively serving a brick?
19:00 ThatGraemeGuy joined #gluster
19:00 fubada joined #gluster
19:01 fubada hi anyone experience jbd2 process using high io while gluster is on? rhel 6.5, 2.6.31
19:01 fubada gluster 3.3.1
19:02 luckyinva yes it is
19:03 fubada ?
19:03 fubada is there a known issue or workarounsd
19:03 luckyinva @skippy yes it is actively serving
19:04 toordog joined #gluster
19:05 toordog what is the current production release version for CentOS?
19:08 Pupeno joined #gluster
19:15 ThatGraemeGuy joined #gluster
19:15 JoeJulian fubada: Should mean that it's actively writing to disk. Self-healing something maybe?
19:16 skippy luckyinva: i'm still learning Gluster myself; but it seems that stopping the glusterd service does not stop actvie bricks.
19:16 fubada JoeJulian: its a mirrored setup with 2 boxes, and they are both going really high oin jbd2
19:16 fubada gluster 3.3.1, im going to try upgrading to latest 3.5
19:16 JoeJulian luckyinva: it's been bugged and supposedly fixed, at least in master. I completely disagree with the concept of forcibly stopping all your bricks just to restart the management daemon, but if that's what you want, check bugzilla for it.
19:17 fubada are there any special steps needed for 3.3.1->3.5.x upgrade
19:17 JoeJulian @upgrade notes
19:17 glusterbot JoeJulian: I do not know about 'upgrade notes', but I do know about these similar topics: '3.3 upgrade notes', '3.4 upgrade notes'
19:17 JoeJulian @3.4 upgrade notes
19:17 glusterbot JoeJulian: http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4/
19:17 fubada ty
19:17 JoeJulian hmm, no 3.5 upgrade notes?
19:20 skippy JoeJulian: if `service glusterd stop` stops the management daemon, do you have a prescribed way to stop bricks?
19:24 ThatGraemeGuy joined #gluster
19:28 JoeJulian skippy: I just kill
19:29 JoeJulian "pkill -f gluster" will kill anything with gluster in the command line.
19:29 kmai007 kill -9 `ps -ef|grep gluster |awk '{print $2}'`
19:29 kmai007 or that
19:29 kmai007 lol
19:30 Danishman joined #gluster
19:31 fubada JoeJulian: i upgraded one gluster member to 3.5.1 from 3.3.1, started glusterd, running heal info returns 5 entries
19:34 fubada running volume heal <volume> returns Launching heal operation to perform index self heal on volume splunk_mbdl has been unsuccessful
19:35 kmai007 i was informed taht you should update all your gluster bricks
19:35 kmai007 then run the heal
19:35 kmai007 i updated today
19:36 kmai007 from 3.4.2-1 -> 3.4.5 and my method was to update each storage node,, then client, then i ran the heal against the volume
19:36 kmai007 and I got no errors
19:36 fubada ty
19:37 fubada ill do heal again when my second node is also 3.5.1
19:37 kmai007 make sure in your upgrade, you had no glusterfs processes running
19:38 fubada i chkconfig off and reboot the box
19:38 kmai007 JoeJulian: the rebalance on glusterfs3.4.5 appears to be working well, I can see the xfs mounted filesystem grow allot fast on my newer bricks
19:39 kmai007 oh ok, you normally don't have to reboot your server, but that method will also be a clean update
19:40 gomikemike question about geo-replication setup
19:40 JoeJulian kmai007: Also, the client crash bug when rebalancing has been fixed in that version. Something you'll be happy to not encounter.
19:40 kmai007 thank goodness, i justin sent me the link to the github release notes
19:41 gomikemike i have a 2 node glusterfs setup on aws e, i want to have it sync with an identical setup in aws w, is geo-replication the right idea for this?
19:41 kmai007 but it didnt' mention that
19:43 fubada JoeJulian: is it normal for 'gluster volume heal splunk_mbdl info
19:43 fubada to take a long time
19:43 JoeJulian only when it's really slow, ie busy.
19:43 fubada Launching heal operation to perform index self heal on volume splunk_mbdl has been unsuccessful
19:43 fubada actual heal fails
19:43 fubada how do i debug this
19:44 gomikemike onther question, i created the volume without replication, can i add replication to an already created volume?
19:44 kmai007 are the volumes up?  when you run gluster volume status <volume>
19:44 gildub joined #gluster
19:45 purpleidea skippy: yeah that's fine, i guess that means i can't give you maintainership of real puppet-gluster :P
19:45 fubada http://fpaste.org/130485/96870711/
19:45 glusterbot Title: #130485 Fedora Project Pastebin (at fpaste.org)
19:45 kmai007 are the storage filesystems (xfs) mounted?
19:45 fubada and yes volumes appear to be up
19:45 JoeJulian fubada: Try killing the self-heal daemons (pkill -f glustershd) and then restarting glusterd.
19:46 kmai007 http://fpaste.org/ your volume status output here <----
19:46 skippy purpleidea: :(  What, I can't contribute to multiple similar projects?
19:46 glusterbot kmai007: <--'s karma is now -1
19:46 glusterbot Title: New paste Fedora Project Pastebin (at fpaste.org)
19:46 kmai007 what is karma?
19:47 JoeJulian gomikemike: yes. You have to add as many bricks as you already have, ie a 1 brick volume changing to replica 2, add 1 brick. A 2 brick volume changing to replica 2 add 2 bricks. A 3 brick volume changing to replica 3 add 6 bricks.
19:47 skippy it's a measure of how popular something is, kmai007.  You add two plusses to increases, and two minusses to decrease.
19:47 skippy gluster++
19:47 glusterbot skippy: gluster's karma is now 1
19:47 fubada JoeJulian: thanks that seems to be resolved, healing launches, but heal info still hangs ans takes forever
19:47 JoeJulian glusterbot--
19:47 glusterbot JoeJulian: glusterbot's karma is now 6
19:48 kmai007 kmai007+++++++
19:48 glusterbot kmai007: kmai007+++++'s karma is now 1
19:48 JoeJulian fubada: It'll probably get better after it finishes the crawl.
19:48 JoeJulian Yes, ok, glusterbot's regex treatment of pluses and minuses is flawed. I'm not going to bother fixing it. :P
19:49 fubada JoeJulian: im trying to rerun heal and its again reporting unsuccessful
19:49 fubada should I repear your suggestion?
19:49 JoeJulian doubtful.
19:49 JoeJulian It's probably busy healing already.
19:50 gomikemike JoeJulian: urgh...so i'm better off recreating the volume since i have no plans on adding more instances to my gluster setup
19:51 JoeJulian You can remove-brick down to the size you want to start with.
19:51 gomikemike oh, wait...i do have 1 brick only per volume. I brick per host in the cluster
19:52 peema joined #gluster
19:52 gomikemike so all i need to do is add 1 brick per host. but what would hte command to add the replicat look like
19:52 semiosis JoeJulian++--
19:52 glusterbot semiosis: JoeJulian's karma is now 12
19:52 glusterbot semiosis: JoeJulian++'s karma is now -1
19:52 fubada JoeJulian: i dont think its healing and its still taking forever to show heal status
19:52 fubada what to do?"
19:52 JoeJulian check the logs
19:53 JoeJulian gomikemike: gluster volume add-brick $vol replica 2 $new_server:$brick_path
19:54 gomikemike this is what i used to create it (just fyi) volume create NAME-vol transport tcp awsglstuq02:/mnt/bricks/lv-NAME awslxglstuq01:/mnt/bricks/lv-FNRW force
19:54 gomikemike JoeJulian: ok, thats easy enough, thxs
19:55 JoeJulian that shows two bricks, not 1, so you'd have to add 2 bricks.
19:55 gomikemike well, 1 brick on each host
19:55 gomikemike so, i would add 1 brick per host. (for a total of 2 new bricks)
19:55 JoeJulian yes
19:56 gomikemike cool
19:56 JoeJulian And you'll want to reverse the order of servers when you do that so they don't replicate to themselves.
19:57 gomikemike now, im going to create this exact setup in aws w, is geo replication the right thing for keeping this 2 glusterfs in sync
19:57 msbrogli joined #gluster
19:57 Pupeno_ joined #gluster
19:57 gomikemike ok, got it (about reversing the order for the addition of the replicas)
20:02 ninkotech__ joined #gluster
20:03 semiosis JoeJulian++--++--++--
20:03 glusterbot semiosis: JoeJulian++--++--'s karma is now 1
20:03 glusterbot semiosis: JoeJulian++--++--++'s karma is now -1
20:03 semiosis i just had to know
20:09 elico joined #gluster
20:13 julim joined #gluster
20:30 ThatGraemeGuy joined #gluster
20:35 julim joined #gluster
20:35 fubada JoeJulian: I need some help getting the cluster back to a clean state could you help
20:37 kmai007 fubada: what state is it in now?
20:37 fubada one sec paste coming
20:37 luckyinva I got the following http://blog.gluster.org/author/zbyszek/ and followed it
20:38 luckyinva im concerned there is an issue in the instruction
20:38 luckyinva openssl req -new -x509 -key gluster.key -subj /CN=Anyone -out glusterfs.pem
20:38 luckyinva if gluster.key vs glusterfs.key is expected on that command
20:38 fubada kmai007: http://fpaste.org/130506/40969030/
20:39 glusterbot Title: #130506 Fedora Project Pastebin (at fpaste.org)
20:39 luckyinva when using that gluster.key that file in not referenced anywhere else is the instruction
20:39 skippy filenames dont matter, luckyinva
20:39 skippy as long as you use the same key filename in every step
20:39 skippy (havent read the linked page yet)
20:40 luckyinva i did just that to work around the error and i ended up with ssl errors
20:40 skippy oh, yes, you're correct.  that appears to be a typo.
20:40 skippy you should use -key glusterfs,key
20:40 kmai007 fubada: what does gluster peer status show?
20:41 kmai007 i'm not sure why its not showing tcp endpoint connection
20:42 _dist joined #gluster
20:43 ricky-ticky joined #gluster
20:43 a1 joined #gluster
20:43 fubada kmai007: they are both connecred
20:43 fubada number of peers: 1 on each
20:44 kmai007 and on those servers the xfs filesystem is mounted ?
20:45 fubada kmai007: yes
20:45 fubada i use ext4 but yes
20:45 kmai007 hmmmm....
20:45 semiosis yuck ext4
20:46 al joined #gluster
20:46 fubada kmai007: can ya help me debug deeper
20:46 kmai007 from gls002 please show me the gluster volume status output
20:47 fubada kmai007: http://fpaste.org/130510/96908541/
20:47 glusterbot Title: #130510 Fedora Project Pastebin (at fpaste.org)
20:50 kmai007 what happens if you try to mount that volume to a client against that storage server?  Does it get the vol file?
20:50 fubada trying
20:52 fubada kmai007: yes i can
20:53 kmai007 show me your gluster volume peer status output please
20:53 fubada you  mean peer status?
20:53 kmai007 right
20:53 calum_ joined #gluster
20:53 kmai007 gluster peer status
20:53 kmai007 sorry
20:54 fubada http://fpaste.org/130513/96912551/
20:54 glusterbot Title: #130513 Fedora Project Pastebin (at fpaste.org)
20:54 fubada kmai007: that is from botj
20:54 fubada both**
20:57 kmai007 what does this log file say on gls002:/var/log/glusterfs/glustershd.log
20:59 fubada [2014-09-02 20:58:50.020338] E [socket.c:2161:socket_connect_finish] 0-splunk_shp-client-0: connection to 10.95.35.18:49153 failed (Connection timed out)
20:59 fubada ;/
20:59 fubada is there anew port needed in 3.5
20:59 fubada ?
20:59 kmai007 netstat -anp|grep <port> for your volume
20:59 kmai007 from the status output
21:00 fubada so i dont have any of these ports open
21:01 fubada between gls001 and gls002, is this a new feature in 3.5.1
21:01 fubada 3.5.x
21:01 kmai007 what did u upgrade from?
21:01 fubada 3.3.1
21:02 JoeJulian @ports
21:02 kmai007 hmmm....3.40 had new ports
21:02 glusterbot JoeJulian: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up since 3.4.0 (24009 & up previously). (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
21:02 kmai007 @ports fubada
21:02 kmai007 damnit i dunno how to do that
21:02 fubada ii think I should just allow  all
21:02 fubada between bricks
21:02 kmai007 that would be  a quick test
21:05 fubada ok
21:05 fubada working
21:06 kmai007 bingo!
21:11 fubada kmai007: so now I just keep running the heal command
21:11 fubada until no entries is 0?
21:11 kmai007 no
21:12 kmai007 heal info ?
21:12 fubada yea i see some entries
21:12 fubada 4 and 2
21:12 kmai007 i don't know if heal output itself is dynamic in 3.5
21:12 kmai007 but you can see over time.....
21:13 kmai007 i'm not running 3.5 yet
21:13 kmai007 my focus was on heal info split-brain
21:13 luckyinva This is error I get after trying to enable ssl across my volumes when following http://blog.gluster.org/author/zbyszek/  http://pastie.org/9522431
21:14 luckyinva I did correct the typo in the instructions but I landed the errors regardless
21:19 fubada kmai007: on the clients i cant chdir into anything mounted
21:20 kmai007 remount it on the clients
21:20 kmai007 are you using FUSE or gNFS?
21:20 fubada i tried
21:20 fubada nfs
21:20 fubada im getting supported nfs version is not supported
21:20 fubada and all used to work
21:22 fubada gluster:/splunk_shp /appdata/splunk_shp nfs defaults,_netdev,proto=tcp,vers=3 0 0
21:22 fubada wrong for 3.5?
21:22 kmai007 it looks right, but the NFS port is different now
21:22 kmai007 its dynamic it can be a range
21:23 kmai007 manually mount it
21:23 kmai007 mount -t nfs server:/vol /mnt
21:23 elico joined #gluster
21:23 Lilian joined #gluster
21:24 fubada mount -t nfs gluster:/splunk_shp /mnt
21:24 fubada mount.nfs: requested NFS version or transport protocol is not supported
21:24 glusterbot fubada: make sure your volume is started. If you changed nfs.disable, restarting your volume is known to work.
21:25 kmai007 well, thats not helpful, can you pastebin the output of mount -vvv -t nfs gluster:/splunk_shp /mnt
21:28 fubada http://fpaste.org/130520/09693308/
21:28 peema joined #gluster
21:28 glusterbot Title: #130520 Fedora Project Pastebin (at fpaste.org)
21:28 fubada tbhanks kmai007
21:33 fubada kmai007: any clue there/
21:35 luckyinva I found my issue, I created keys on each host vs taking the key and replicating out to hte remaining hosts.. silly noob mistake
21:35 kmai007 luckyinva: good job, i'm not using SSL so i couldn't help ya
21:36 kmai007 fubada: i guess you can try "mount -vvv -t nfs -o vers=3 <server>:/vol /mntpoint
21:36 luckyinva appreciated regardless
21:37 fubada kmai007: doesnt likem iot
21:37 fubada it
21:37 kmai007 show me your command
21:37 kmai007 or output
21:38 fubada http://fpaste.org/130523/93907140/
21:38 glusterbot Title: #130523 Fedora Project Pastebin (at fpaste.org)
21:39 kmai007 is "gluster" a cname or some sorts?
21:39 kmai007 i thought your logs said gs002
21:39 fubada yes its a cname
21:40 kmai007 what happens if you just directly specify a server name
21:40 fubada same
21:40 uebera|| joined #gluster
21:41 fubada ok so I restarted glusterd
21:41 fubada on both gls001 and gls002
21:42 fubada i can mount the nfs share
21:42 fubada but I cant read anything in/out of it
21:42 fubada correction, one of the two top level folders in the share
21:42 fubada throws io errors
21:43 fubada i think theres some split brain, heal issue
21:43 kmai007 io like with ???????????????????? ????????????????????????
21:43 kmai007 yeh thats the spirit
21:43 fubada ls: cannot open directory etc: Input/output error
21:43 kmai007 yeh,
21:43 fubada how do I fix this
21:43 kmai007 so split-brain is right
21:43 kmai007 @splitbrain
21:43 glusterbot kmai007: I do not know about 'splitbrain', but I do know about these similar topics: 'split-brain'
21:43 kmai007 @split-brain
21:43 glusterbot kmai007: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/
21:44 andreask joined #gluster
21:44 kmai007 its a fun exercise if you're not pressed for time
21:44 fubada ha
21:45 fubada http://fpaste.org/130525/94334140/
21:45 glusterbot Title: #130525 Fedora Project Pastebin (at fpaste.org)
21:46 kmai007 yep gotta clean those up
21:46 kmai007 use this link http://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/
21:46 glusterbot Title: GlusterFS Split-Brain Recovery Made Easy (at joejulian.name)
21:46 kmai007 or if you have access to redhat you can follow https://access.redhat.com/solutions/193843
21:46 glusterbot Title: How to recover a file from a split-brain on a Red Hat Storage volume ? - Red Hat Customer Portal (at access.redhat.com)
21:47 fubada splitmount is run on the gluster servers?
21:47 kmai007 yes
21:47 fubada i dont have ot
21:47 fubada it
21:48 fubada any idea where it comes from
21:48 kmai007 you have to download it
21:48 kmai007 from that blog
21:48 kmai007 follow it
21:55 fubada JoeJulian: what do to if "/" is split brain
21:55 fubada as the path
21:57 fubada JoeJulian: splitmount is not working for me
21:58 fubada says it mounted successully but theres nothing in dirs r1, r2
21:59 andreask joined #gluster
22:01 andreask joined #gluster
22:10 Pupeno joined #gluster
22:10 fubada Anyone here to help me fix: http://fpaste.org/130528/69582314/
22:10 glusterbot Title: #130528 Fedora Project Pastebin (at fpaste.org)
22:13 peema joined #gluster
22:14 semiosis fubada: ,,(split brain)
22:14 glusterbot fubada: I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
22:14 semiosis fubada: ,,(split-brain)
22:14 glusterbot fubada: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/
22:14 fubada semiosis: im split on directories
22:15 fubada not files
22:15 fubada i have no idea how to fix that
22:15 fubada and splitmount wont work for me, although command oputput says it mounted
22:17 Pupeno_ joined #gluster
22:17 ninkotech_ joined #gluster
22:18 ninkotech joined #gluster
22:22 fubada what does this even mean http://fpaste.org/130529/69653314/
22:22 glusterbot Title: #130529 Fedora Project Pastebin (at fpaste.org)
22:25 sputnik13 joined #gluster
22:29 dmyers joined #gluster
22:29 dmyers joined #gluster
22:35 recidive joined #gluster
22:41 plarsen joined #gluster
22:53 JoeJulian fubada: Just set the trusted.afr.* to 0x0000000000000000 (however many places it is) on all but one brick.
22:54 Pupeno joined #gluster
22:56 blubberdi joined #gluster
22:56 fubada JoeJulian: please clarify
22:56 fubada i have 2
22:57 JoeJulian On one of the servers: getfattr -m trusted.afr $brick_path
22:58 fubada ok
22:58 fubada thats it?
22:58 JoeJulian nope
22:58 JoeJulian That's just how to get the names that I don't have.
22:59 fubada http://fpaste.org/130538/98723140/
22:59 glusterbot Title: #130538 Fedora Project Pastebin (at fpaste.org)
22:59 fubada not sure what to do with this
23:01 fubada JoeJulian: i tried to use splitmount, it reported no issues but nothing was mounted
23:01 fubada im thinking about blowing a brick away to fix this
23:01 fubada is there a better way?
23:04 JoeJulian fubada: yes, that's been reported re: splitmount. I was finally able to reproduce it once, but I'm still not sure how. I'm going to re-write part of it to, hopefully, cure that in the future.
23:04 fubada ok :)
23:05 fubada JoeJulian: do you have a min to help me resolve the directory split brain?
23:06 fubada http://fpaste.org/130541/14096991/
23:06 glusterbot Title: #130541 Fedora Project Pastebin (at fpaste.org)
23:06 JoeJulian fubada: echo trusted.afr.splunk_mbdl-client-{0,1} | xargs -I{} setfattr -n {} -v 0x000000000000000000000000 /appdata/bricks/splunk_mbdl/
23:07 fubada ok
23:07 fubada JoeJulian:  on both ?
23:08 JoeJulian one should be sufficient.
23:08 JoeJulian both probably would be fine, too.
23:08 fubada ok
23:08 fubada nothing happened
23:09 fubada gluster volume heal splunk_mbdl info still shows entries
23:17 Pupeno_ joined #gluster
23:18 fubada 2014-09-02 21:41:24 /
23:18 fubada thats my split-brain
23:19 fubada ;/
23:19 JoeJulian That was two and a half hours ago.
23:19 fubada 2014-09-02 23:18:37 /
23:19 fubada http://fpaste.org/130546/40969996/
23:19 glusterbot Title: #130546 Fedora Project Pastebin (at fpaste.org)
23:20 JoeJulian Was 23:18 before or after you set the attributes?
23:20 fubada everytime i run 'info split-brain' it adds another item
23:20 fubada after
23:20 * JoeJulian grumbles
23:20 JoeJulian on both servers: getfattr -m . -d -e hex /appdata/bricks/splunk_mbdl/
23:21 fubada ok
23:21 fubada http://fpaste.org/130547/14097001/
23:21 glusterbot Title: #130547 Fedora Project Pastebin (at fpaste.org)
23:24 fubada http://fpaste.org/130548/00226140/ another dir in splunk_mbdl thats having issues, here the trusted.gfid is different between gls001 and gls002
23:24 glusterbot Title: #130548 Fedora Project Pastebin (at fpaste.org)
23:24 JoeJulian d'oh!
23:24 JoeJulian I goofed.
23:25 JoeJulian my echo was wrong.
23:25 fubada ;)
23:25 JoeJulian on gls002: setfattr -x 'trusted.afr.splunk_mbdl-client-0 trusted.afr.splunk_mbdl-client-1'
23:25 JoeJulian on gls002: setfattr -x 'trusted.afr.splunk_mbdl-client-0 trusted.afr.splunk_mbdl-client-1' /appdata/bricks/splunk_mbdl/
23:25 fubada done
23:27 fubada same on gls001?
23:27 JoeJulian no
23:28 glusterbot New news from newglusterbugs: [Bug 1126831] Memory leak in GlusterFs client <https://bugzilla.redhat.com/show_bug.cgi?id=1126831>
23:28 JoeJulian setfattr -n trusted.afr.splunk_mbdl-client-0 -v 0x000000000000000000000000 /appdata/bricks/splunk_mbdl/ ; setfattr -n trusted.afr.splunk_mbdl-client-1 -v 0x000000000000000000000000 /appdata/bricks/splunk_mbdl/
23:28 JoeJulian on one of the servers.
23:29 fubada done
23:29 fubada on gls001
23:30 JoeJulian That should have cured it and also now if you do the getfattr on both, they should both show all 0x000000000000000000000000
23:30 fubada unfortunately same
23:31 JoeJulian can't be
23:31 JoeJulian It's not possible...
23:31 fubada http://fpaste.org/130553/09700709/
23:31 glusterbot Title: #130553 Fedora Project Pastebin (at fpaste.org)
23:32 JoeJulian Ah, but gls001 is still all zeros.
23:33 JoeJulian Ok... (I wish I could figure out how it gets in this state) on 002: setfattr -n trusted.afr.splunk_mbdl-client-1 -v 0x000000000000000000000000 /appdata/bricks/splunk_mbdl/
23:33 fubada done
23:33 fubada can this be done recursive?
23:34 fubada theres also that log002 directory
23:34 JoeJulian no
23:34 JoeJulian well, yes, but only using find.
23:34 fubada ok
23:34 fubada better, but log002 on gls002 is still showing up
23:35 fubada ok
23:35 fubada healed
23:35 JoeJulian yay
23:35 fubada now my other brick, basically same issue
23:35 fubada for many dirs
23:35 fubada can I just repeat the process
23:36 fubada http://fpaste.org/130554/97009731/
23:36 glusterbot Title: #130554 Fedora Project Pastebin (at fpaste.org)
23:36 JoeJulian Pranith told me that's not supposed to happen and that it's supposed to fix itself. I wish he was on this hemisphere...
23:36 fubada files too
23:38 xoritor joined #gluster
23:38 JoeJulian Yes you can for directories: find /appdata/bricks/splunk_mbdl/ -type d -exec setfattr -n trusted.afr.splunk_mbdl-client-0 -v  0x000000000000000000000000 {} \; -exec setfattr -n trusted.afr.splunk_mbdl-client-1 -v  0x000000000000000000000000 {} \;
23:38 xoritor would it be better to have 3 replicated 4 TB bricks in a server or maybe setup software raid?
23:39 xoritor ie... 3 4TB HDDs each as a brick (maybe using NUFA)
23:39 fubada Joe and for files?
23:39 fubada can I also run the same?
23:40 xoritor or software raid and maybe (doubtful) gain some I/O ... I am not sure there would be much gain in using the software raid of any level
23:40 JoeJulian fubada: note that client-0 is gls001 and client-1 is gls002. The non-zero means there's pending writes for that server. Set them to all zero on one server, and just reset the one for itself on the server whose data you want to keep.
23:41 JoeJulian ie to keep the data on gls002 reset both on gls001 and only client-1 on gls002.
23:41 JoeJulian xoritor: sometimes.
23:42 fubada ok im resetting both on gls002
23:43 xoritor JoeJulian, i am thinking it may be better to add more servers with more bricks each and maybe setup nufa
23:43 JoeJulian Does that still exist?
23:44 xoritor 3 servers 3 4TB HDDs each with 4 1 GB (bonded) nics
23:45 xoritor with a replica count of 12
23:45 JoeJulian Hmm. I thought ALL those old translator pages were marked as no longer valid.
23:45 xoritor ;-)
23:45 xoritor is nufa no good anymore?
23:46 xoritor nufa =! noworky?
23:46 JoeJulian You haven't written your own vol files for years.
23:46 fubada JoeJulian: what can I do for files/
23:46 JoeJulian it might still exist, and may even still work.
23:46 fubada same?
23:46 xoritor but but but..... but but but.... you mean i cant stack volumes anymroe?
23:46 JoeJulian fubada: yes.
23:47 * xoritor cries in his beer
23:47 JoeJulian xoritor:  you may, but you do lose the ability to change volumes live.
23:47 xoritor ;-)
23:47 xoritor im ok
23:48 JoeJulian xoritor: I think you also lose proactive self-heal.
23:48 JoeJulian trying to think what else you've missed in your time under a rock... ;)
23:50 xoritor hehehehe
23:50 xoritor hey i have not been under a rock... just thinking back
23:50 xoritor i am current
23:50 xoritor 3.5.2
23:50 JoeJulian Ah, ok.
23:51 xoritor but i have been using it since way back
23:51 xoritor in the stone age
23:51 JoeJulian Figured, since you and I may be the only two people in the channel that know what nufa even is.
23:52 xoritor is there a way to do nufa now?
23:52 xoritor or does it "just do it"
23:52 xoritor ?
23:52 JoeJulian What are you trying to accomplish?
23:52 eshy joined #gluster
23:52 xoritor i dont know
23:52 JoeJulian lol
23:52 xoritor just thinking
23:53 xoritor replicated files mainly
23:53 xoritor no data loss
23:53 xoritor zero data loss
23:53 xoritor lots of places for it to be
23:53 xoritor lots of disks, hosts, etc...
23:53 firemanxbr joined #gluster
23:54 JoeJulian More interested in fault tolerance then.
23:54 xoritor ie... 10 servers 3 bricks each with replica count 30
23:54 xoritor yes
23:54 xoritor much much more focused on that
23:54 JoeJulian replica count 30 is excessive, unless you're using it for an on-demand service or something.
23:55 xoritor H/A VMs mainly
23:55 JoeJulian @reliability calculation
23:55 JoeJulian @whatthehelldidicallthatfactoid
23:55 JoeJulian @calculation
23:55 glusterbot JoeJulian: I do not know about 'calculation', but I do know about these similar topics: 'sla calculations'
23:55 JoeJulian @sla calculation
23:55 glusterbot JoeJulian: I do not know about 'sla calculation', but I do know about these similar topics: 'sla calculations'
23:55 xoritor but yea 100% availability at all times if i can get it ;-)
23:55 JoeJulian @sla calculations
23:55 glusterbot JoeJulian: Calculate your system reliability and availability using the calculations found at http://www.eventhelix.com/realtimemantra/faulthandling/system_reliability_availability.htm . Establish replica counts to provide the parallel systems to meet your SLA requirements.
23:55 xoritor yea
23:55 JoeJulian it's math.
23:56 xoritor i have done most of them
23:56 xoritor but not that one
23:56 JoeJulian @alias "sla calculations" "reliability calculations"
23:56 glusterbot JoeJulian: The operation succeeded.
23:57 xoritor funny thing is that i have most of the redundancy in place already
23:57 xoritor power from multiple grids
23:58 xoritor multiple pdu's
23:58 xoritor multiple switches
23:58 JoeJulian Point being, anything above replica 3 is probably beyond what you're probably looking for.
23:58 xoritor multiple etc...
23:58 JoeJulian probably.
23:58 xoritor hmm
23:59 JoeJulian (apparently my fingers like typing that word a lot)
23:59 xoritor lol
23:59 xoritor mine line typing ...... no
23:59 xoritor dont know why

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary