Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-04-10

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 gdubreui joined #gluster
00:11 harish joined #gluster
00:17 andreask joined #gluster
00:18 andreask joined #gluster
00:18 andreask joined #gluster
00:33 andreask joined #gluster
00:37 hchiramm__ joined #gluster
00:52 yinyin_ joined #gluster
01:00 vpshastry joined #gluster
01:00 jmarley joined #gluster
01:00 jmarley joined #gluster
01:14 bala joined #gluster
01:16 mtanner_ joined #gluster
01:18 RyanD- joined #gluster
01:19 edong23_ joined #gluster
01:19 aurigus_ joined #gluster
01:21 the-me_ joined #gluster
01:22 Uguu joined #gluster
01:22 divbell_ joined #gluster
01:22 sticky_afk joined #gluster
01:22 atrius` joined #gluster
01:23 Slasheri joined #gluster
01:23 Slasheri joined #gluster
01:23 atrius joined #gluster
01:23 stickyboy joined #gluster
01:23 Abrecus joined #gluster
01:24 Dasberger joined #gluster
01:25 Oneiroi joined #gluster
01:25 Jakey_ joined #gluster
01:37 chirino_m joined #gluster
01:49 ilbot3 joined #gluster
01:49 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:52 yinyin_ joined #gluster
02:02 baojg joined #gluster
02:09 VerboEse joined #gluster
02:10 knfbny joined #gluster
02:33 hagarth joined #gluster
02:40 ceiphas_ joined #gluster
02:41 rastar joined #gluster
02:56 bharata-rao joined #gluster
03:01 nightwalk joined #gluster
03:14 glusterbot New news from newglusterbugs: [Bug 1076625] file disappeared in the heterogeneity architecture computer system(arm and intel) <https://bugzilla.redhat.com/show_bug.cgi?id=1076625>
03:15 gmcwhistler joined #gluster
03:19 zaitcev joined #gluster
03:20 kshlm joined #gluster
03:23 gmcwhistler joined #gluster
03:35 meghanam joined #gluster
03:35 meghanam_ joined #gluster
03:35 kanagaraj joined #gluster
03:36 Ark joined #gluster
03:46 itisravi joined #gluster
03:46 haomaiwa_ joined #gluster
03:52 cyberbootje joined #gluster
03:55 rastar joined #gluster
03:56 baojg joined #gluster
04:02 ndarshan joined #gluster
04:11 ppai joined #gluster
04:12 kumar joined #gluster
04:13 dusmant joined #gluster
04:14 misuzu joined #gluster
04:16 yinyin_ joined #gluster
04:18 deepakcs joined #gluster
04:20 rjoseph joined #gluster
04:22 atinm joined #gluster
04:25 an joined #gluster
04:32 hagarth joined #gluster
04:34 shubhendu joined #gluster
04:38 baojg hi all
04:43 spandit joined #gluster
04:44 Jakey_ joined #gluster
04:45 meghanam_ joined #gluster
04:45 meghanam joined #gluster
04:51 ppai joined #gluster
04:54 sahina joined #gluster
04:55 kdhananjay joined #gluster
05:02 mkzero joined #gluster
05:02 rastar joined #gluster
05:06 RameshN joined #gluster
05:06 primechuck joined #gluster
05:08 ravindran1 joined #gluster
05:10 bharata-rao joined #gluster
05:11 benjamin_____ joined #gluster
05:12 prasanth_ joined #gluster
05:17 davinder joined #gluster
05:25 rastar joined #gluster
05:27 bala1 joined #gluster
05:27 raghu joined #gluster
05:28 baojg joined #gluster
05:32 ravindran1 left #gluster
05:35 baojg_ joined #gluster
05:36 vpshastry joined #gluster
05:37 Philambdo joined #gluster
05:37 prasanth_ joined #gluster
05:41 bala1 joined #gluster
05:50 haomaiwang joined #gluster
05:50 nightwalk joined #gluster
05:51 lalatenduM joined #gluster
05:54 ppai joined #gluster
06:01 deepakcs joined #gluster
06:15 glusterbot New news from newglusterbugs: [Bug 1086097] [RFE] Allow the usage of the wildcard character '*' to the options "nfs.rpc-auth-allow" and "nfs.rpc-auth-reject" <https://bugzilla.redhat.com/show_bug.cgi?id=1086097>
06:21 jtux joined #gluster
06:23 psharma joined #gluster
06:25 wgao joined #gluster
06:27 an joined #gluster
06:31 bharata-rao joined #gluster
06:35 wgao_ joined #gluster
06:41 chirino joined #gluster
06:45 glusterbot New news from newglusterbugs: [Bug 1084508] read-ahead not working if open-behind is turned on <https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
06:50 andreask joined #gluster
06:52 kumar joined #gluster
06:55 vimal joined #gluster
06:56 dusmant joined #gluster
06:58 ekuric joined #gluster
07:06 ctria joined #gluster
07:11 saurabh joined #gluster
07:12 vsa glusterfs 3.4.2
07:12 vsa Hi all. when using geo-replication, does anybody see high usage swap space?
07:14 nshaikh joined #gluster
07:18 keytab joined #gluster
07:30 samppah vsa: iirc someone mentioned that there was bug that was fixed in 3.4.3
07:30 samppah unfortunately i'm not familiar with this issue :(
07:32 itisravi joined #gluster
07:33 hagarth joined #gluster
07:33 fsimonce joined #gluster
07:34 shubhendu joined #gluster
07:34 prasanth_ joined #gluster
07:34 rjoseph joined #gluster
07:42 dusmant joined #gluster
07:42 chirino_m joined #gluster
07:45 sac`away joined #gluster
07:46 Durzo i cant find release notes for 3.4.3, even the announce to gluster-users simply lists the download url and nothing more
07:48 Durzo found it by guessing the url based on 3.4.2 release notes. completely unlinked from gluster.org (3.4.3 gets no mention of even being released. worst. site. ever.)
07:49 Durzo if anyone cares: https://github.com/gluster/glusterfs/blob/release-3.4/doc/release-notes/3.4.3.md
07:49 glusterbot Title: glusterfs/doc/release-notes/3.4.3.md at release-3.4 · gluster/glusterfs · GitHub (at github.com)
07:59 davinder joined #gluster
08:14 ngoswami joined #gluster
08:15 RameshN joined #gluster
08:16 hagarth joined #gluster
08:19 haomaiwa_ joined #gluster
08:21 haomai___ joined #gluster
08:23 Philambdo joined #gluster
08:24 _NiC joined #gluster
08:29 nightwalk joined #gluster
08:29 T0aD joined #gluster
08:46 Pavid7 joined #gluster
08:48 Calum joined #gluster
08:51 saravanakumar joined #gluster
08:55 harish joined #gluster
09:07 codeitagile joined #gluster
09:12 Pavid7 joined #gluster
09:18 Philambdo joined #gluster
09:32 kdhananjay joined #gluster
09:33 Slash_ joined #gluster
09:44 davinder joined #gluster
09:50 primechu_ joined #gluster
10:00 eseyman joined #gluster
10:21 pk joined #gluster
10:23 hagarth joined #gluster
10:28 ngoswami joined #gluster
10:30 lalatenduM joined #gluster
10:33 hybrid512 joined #gluster
10:40 kdhananjay joined #gluster
10:44 kanagaraj joined #gluster
10:59 lpabon joined #gluster
11:00 kkeithley1 joined #gluster
11:03 Pavid7 joined #gluster
11:04 primechuck joined #gluster
11:17 kanagaraj joined #gluster
11:18 dusmant joined #gluster
11:21 ppai joined #gluster
11:21 ndarshan joined #gluster
11:22 sahina joined #gluster
11:23 shubhendu joined #gluster
11:24 Pavid7 joined #gluster
11:38 ira_ joined #gluster
11:39 harish joined #gluster
11:51 brodiem joined #gluster
11:55 RicardoSSP joined #gluster
11:55 RicardoSSP joined #gluster
12:12 pk left #gluster
12:12 itisravi joined #gluster
12:15 user_42 joined #gluster
12:17 user_42 Hi All! I still have problems connecting a client (centos 6.5) to a master (centos 6.5). gluster version on both sides is 3.4.2. With another master (same os and versions) it is working. both masters are listening on the same ports except the working server is listening on 49152, 49153, 49155 and *:ipcserver while the non working is not listening on these but additonally on *:nas. the installation procedure was exactly the same on both
12:18 glusterbot New news from newglusterbugs: [Bug 1086228] Remove-brick: File permission (setuid) changes after migration of the file <https://bugzilla.redhat.com/show_bug.cgi?id=1086228>
12:21 deepakcs joined #gluster
12:25 Ark joined #gluster
12:35 benjamin_____ joined #gluster
12:41 meghanam joined #gluster
12:41 meghanam_ joined #gluster
12:52 Pavid7 joined #gluster
12:53 John_HPC joined #gluster
12:59 Oneiroi joined #gluster
13:00 bennyturns joined #gluster
13:02 B21956 joined #gluster
13:03 meghanam joined #gluster
13:05 harish joined #gluster
13:05 tdasilva joined #gluster
13:12 nshaikh joined #gluster
13:29 theron joined #gluster
13:33 japuzzo joined #gluster
13:34 lalatenduM_ joined #gluster
13:34 dbruhn joined #gluster
13:35 dbruhn morning gents
13:35 plarsen joined #gluster
13:40 plarsen joined #gluster
13:43 user_42 joined #gluster
13:48 mgarcesMZ joined #gluster
13:48 mgarcesMZ hi there
13:51 Pavid7 joined #gluster
13:53 lmickh joined #gluster
13:53 ctria joined #gluster
13:53 hagarth joined #gluster
13:59 rpowell joined #gluster
13:59 vpshastry joined #gluster
14:01 vpshastry left #gluster
14:01 primechuck joined #gluster
14:02 user_42 joined #gluster
14:02 user_42 sorry to ask again. i got disconnected. has anybody had the problem of glusterfs not listening on ports  49152, 49153, 49155 ? what could i do?
14:04 diegows joined #gluster
14:08 jobewan joined #gluster
14:11 LoudNoises joined #gluster
14:11 B21956 joined #gluster
14:15 _Bryan_ joined #gluster
14:18 ndevos @later tell user_42 type @ports in #gluster to learn more about why some servers use other ports than other servers
14:18 glusterbot ndevos: The operation succeeded.
14:22 vpshastry joined #gluster
14:22 B21956 joined #gluster
14:23 B21956 joined #gluster
14:23 wushudoin joined #gluster
14:25 rpowell left #gluster
14:26 lalatenduM_ joined #gluster
14:27 hybrid512 joined #gluster
14:28 hagarth joined #gluster
14:30 hybrid512 joined #gluster
14:38 nikk_ joined #gluster
14:43 deepakcs joined #gluster
14:45 Andyy2 @ports
14:45 glusterbot Andyy2: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
14:50 jmarley joined #gluster
14:55 tjikkun_work joined #gluster
14:55 vpshastry left #gluster
14:56 jag3773 joined #gluster
14:59 jbrooks joined #gluster
15:01 cyberbootje joined #gluster
15:05 primechuck joined #gluster
15:19 glusterbot New news from newglusterbugs: [Bug 1086303] 30-min posix_acl_access timeouts <https://bugzilla.redhat.com/show_bug.cgi?id=1086303>
15:25 vipulnayyar joined #gluster
15:27 benjamin_____ joined #gluster
15:32 ircolle joined #gluster
15:41 chirino joined #gluster
15:43 primechuck joined #gluster
15:51 sijis is there a proper way to re-ip a glusternode? i re-ip the server but i cant start the service on it
15:52 social sijis: you didn't use DNS when probing your peers?
15:53 social it's whey better to do so. but anyway now what error you get in logs?
15:53 sijis social: i did ues dns
15:54 social sijis: amazon? peer probing peers from one node? the node not starting up is second one?
15:56 sijis social: nope. vmware in-house.
15:56 sijis social: http://paste.fedoraproject.org/93236/71453761/
15:56 glusterbot Title: #93236 Fedora Project Pastebin (at paste.fedoraproject.org)
15:57 sijis i haven't tried reprobing. but on gfs-12, i tried a simple gluster volume info and got 'gluster daemon not operational'. i tried starting up glusterd and that failed
15:58 sijis gfs-12 is the one i re-ip'ed
15:59 sijis i haven't touched the other node gfs-11 at all
16:08 sijis let me swap the old ip back and see if it starts up
16:10 primechuck joined #gluster
16:10 jag3773 joined #gluster
16:12 7JTAAA1DS joined #gluster
16:13 sijis social: so i re-ip it back to the original ip and glusterd start's right up
16:17 kaptk2 joined #gluster
16:19 sijis social: any thoughts?
16:22 vipulnayyar joined #gluster
16:23 Mo__ joined #gluster
16:23 gmcwhistler joined #gluster
16:30 uebera|| joined #gluster
16:31 sputnik13 joined #gluster
16:32 sputnik13 joined #gluster
16:34 primechuck joined #gluster
16:34 sijis so for a re-ip of a node, its better to remove the brick, remove it as a peer, then re-ip the server and then add as peer and brick?
16:35 sijis i figured, since i used dns to create it, i should have been able to re-ip the server without ill effects.
16:35 an joined #gluster
16:37 dbruhn If you used DNS this shouldn't be an issue at all
16:38 dbruhn do you have something screwing with your resolution, like a rouge hosts file?
16:41 sijis dbruhn: nothing odd. i tkae a few mins for dns to propegate. i did check the hosts file too. i didn't try to restart the service until the gfs-11 (node1) was able to ping it with new ip by name
16:42 dbruhn when you do a peer status what is the report?
16:42 dbruhn do it from each server
16:44 sijis dbruhn: i just changed the ip on the 2nd node to its new ip. i'll tell you in a sec here
16:46 sijis ha! its up on node2. screwed up hosts file. i had 10.1.20.82wq as the ip :)
16:46 dbruhn good to know
16:47 ircolle left #gluster
16:47 sijis yeah. i was starting to look at teh /var/lib/gluster stuff. but there no ip mentioned in anything there
16:51 jag3773 joined #gluster
16:52 cleverfoo joined #gluster
16:52 sijis dbruhn: thanks
16:52 cleverfoo hey folks, I’m in the middle of a production heal event and could use some help
16:52 cleverfoo things look a little strange
16:53 cleverfoo is there any way to get a better sense on how far along the heal process is?
16:53 cleverfoo I’ll totally buy a beer in exchange for some help here
16:53 cleverfoo :)
16:54 cleverfoo gluster volume heal VOLNAME info is pretty useless
16:56 sijis cleverfoo: i cna't say i've done healing
16:56 sijis but.. in looking at the docs
16:57 dbruhn hearings are always running in the background
16:57 zerick joined #gluster
16:57 sijis if you do volume heal <vol> info it should give u a number
16:57 dbruhn unless you stop them
16:57 cleverfoo @dbruhn we triggered a heal full
16:57 sijis i'm guessing if you try it again in a few minutes, hopefully that number changes?
16:57 dbruhn What was your purpose for the full?
16:58 cleverfoo we lost a brick
16:58 cleverfoo and had to add new one
16:58 dbruhn I hate to say this, but I just watch the fill level of the brick using df
16:58 cleverfoo @dbruhn really?
16:59 dbruhn The last time I did it, that's how I went about it. Not 100% sure if it's the best way.
16:59 dbruhn I also after I was done started a stat on everything on the volume
16:59 brodiem I don't think it really "knows" what it's heal progress is does it, until it finishes checking each file?
16:59 sijis newbie question --- if you lose a brick and re-add another one, you have to run heal full?
16:59 brodiem *checking/healing each file
16:59 dbruhn the heal repopulates the bricks when it finds the files missing
17:00 cleverfoo @sijis yup AFAIK
17:00 dbruhn in this case anyway
17:01 sijis does that matter if its distributed or replicated?
17:01 cleverfoo @dbruhn is there a way to throttle it up/down?
17:01 cleverfoo my heal has been going for 3 days now
17:01 cleverfoo hence my concern
17:01 dbruhn clever, if you stat the files it will go faster than the heal
17:01 brodiem how much disk space is used on the brick being healed vs the others?
17:02 dbruhn I am not 100% sure if there is a way to increase the priority
17:02 d3vz3r0 joined #gluster
17:02 dbruhn the stat triggers a heal on the file
17:02 d3vz3r0 just a stat will trigger a heal?
17:02 dbruhn on a single file
17:03 cleverfoo @dbruhn on the new one is about 2TB versus 12TB on the old
17:03 dbruhn you can use the find command in conjunction with the stat command to stat every file in the file system
17:03 d3vz3r0 is that becuase of the atime being updated?
17:03 d3vz3r0 that it triggers a heal?
17:03 dbruhn it's a slow process cleverfoo, it took about 3/4 days for a brick to repopulate for me with a full heal at 2.3TB on 15k SAS and 40GB Infiniband
17:03 dbruhn d3vz3r0, not sure
17:04 dbruhn be back in a few
17:04 * sijis thinks damn.. i only have 9G volumes
17:04 d3vz3r0 dbruhn: thanks
17:04 cleverfoo yup @dbruhn thanks
17:04 dbruhn my volume for that system is 26TB
17:04 dbruhn brb
17:05 cleverfoo btw @d3vz3r0 and I work together
17:06 cleverfoo in case folks are wondering
17:06 d3vz3r0 we are a team
17:06 sijis boy.. we struggle providing 2TB for some teams :/ ..
17:06 cleverfoo yup
17:06 sijis heck, 1TB
17:06 cleverfoo we have 1PB on gluster @sijis
17:06 cleverfoo actually 2PB if you want ot be technical
17:06 sijis i'm feeling sad
17:06 cleverfoo LOL
17:07 cleverfoo so there’s not a heal “make fast” option anywhere?
17:07 sijis we don't have any san that has anything close to that.
17:07 cleverfoo I know there’s limitations everywhere
17:07 cleverfoo but gluster is definately not pushing the box
17:07 lpabon joined #gluster
17:08 primechu_ joined #gluster
17:10 cleverfoo @sijis well, we don’t use a SAN, open compute hardware actually bit.ly/R6nCDr
17:12 cleverfoo it’s pretty awesome
17:12 John_HPC Is there any fast way to change ownership/permissions across all files on a gluster system? I have 150million files across 160TB of data.
17:12 jiffe98 so it seems one of my volumes is healing which seems to be causing the clients to be very slow and load average is through the roof
17:12 cleverfoo @John_HPC a script?
17:12 John_HPC Thats what I may have to do
17:13 cleverfoo @jiffe98 welcome to my world
17:13 semiosis cleverfoo: would you mind talking to our community organizer, johnmark, about your epic gluster deployment?
17:13 sijis chown b:b -R dir/  ... go home and sleep
17:13 semiosis i'm sure he would be very interested
17:13 cleverfoo sure thing @semiosis
17:13 cleverfoo but the word “epic" scares me
17:14 semiosis 2PB is pretty epic, imho
17:14 jiffe98 can I stop the self healing?
17:15 sijis cleverfoo: so you use this io. service for all your stuff?
17:15 cleverfoo so @dbruhn is the stat supposed to be done on the empty brick or the full one?
17:15 cleverfoo because it’s no worky
17:16 jiffe98 the self heal also seems to be incredibly slow, volume heal info shows the same entries for long periods of time and the files aren't very large
17:16 knfbny joined #gluster
17:16 semiosis cleverfoo: if you know only one brick was down then the best way to watch progress is to look at the files on that brick.  when it is as full as its replica then heal is probably done
17:17 cleverfoo yeah it’s far from it
17:17 cleverfoo and it doesn’t appear to be moving
17:17 cleverfoo almost like the healing stopped
17:17 semiosis cleverfoo: you can tune the healing process by adjusting how many files are healing in parallel.  you can also change the heal alorithm.
17:18 semiosis depending on your use case you might find the "full" algorithm to perform better than the default, "diff" alg
17:18 cleverfoo oh wait a minute… @semiosis any documentation on that?
17:18 semiosis the heal alg is documented in the online help, run 'gluster volume set help'
17:18 semiosis the background heal count is in ,,(undocumented options)
17:18 glusterbot Undocumented options for 3.4: http://www.gluster.org/community/documentation/index.php/Documenting_the_undocumented
17:19 semiosis cluster.background-self-heal-count
17:19 semiosis brb
17:19 cleverfoo @semiosis reading..
17:20 kaptk2 joined #gluster
17:21 cleverfoo @semiosis can the heal process algorithm be changed on the fly?
17:22 LoudNoises is there a way to find out which brick a particular filename will be written to?
17:24 John_HPC semiosis: that command is rather helpful. I see Option: storage.owner-uid and Option: storage.owner-gid; which may help me with my chown issues.
17:25 zaitcev joined #gluster
17:28 semiosis cleverfoo: yes, volume option changes are dynamic
17:28 semiosis John_HPC: cool
17:29 semiosis LoudNoises: you can see where an existing file is using ,,(pathinfo)
17:29 cleverfoo awesome @semiosis I totally owe you a beer
17:29 glusterbot LoudNoises: find out which brick holds a file with this command on the client mount point: getfattr -d -e text -n trusted.glusterfs.pathinfo /client/mount/path/to.file
17:29 LoudNoises yea semiosis i'm hoping to get that for a filename i haven't made yet
17:29 semiosis LoudNoises: but idk any way to find out for a potential file
17:29 cleverfoo @semiosis I’m raf@io.com if you want me to make good on that beer
17:29 semiosis cleverfoo: hah, thanks
17:29 LoudNoises okay, i may be able to touch the file then overwrite it - it should go to the same place right?
17:30 semiosis not sure where you're located but i'll be in SF next week for redhat summit, if you're nearby we could grab a beer
17:30 d3vz3r0 @semiosis,I'm in sf
17:30 semiosis LoudNoises: yes
17:31 semiosis d3vz3r0: cool.  check this out, on sunday: http://www.meetup.com/Gluster/San-Francisco-CA/1126032/
17:31 glusterbot Title: Gluster Meetup - San Francisco, CA | Apr 13, 2014 - Meetup (at www.meetup.com)
17:31 cleverfoo @semiosis I’m not gonna be able to make it but @d3vz3r0 works with me and can make good on that offer and he can tell you all about our multiple PB’s of gluster storage
17:32 semiosis awesome
17:32 semiosis you should come to the hackathon, lots of gluster people will be there
17:32 semiosis http://www.eventbrite.com/e/data-liberate-an-open-source-storage-hackathon-and-mixer-registration-11055809223
17:32 glusterbot Title: Data Liberate: an Open Source Storage Hackathon and... Registration, San Francisco - Eventbrite (at www.eventbrite.com)
17:32 semiosis free!
17:33 d3vz3r0 semiosis, unfortuantely I will be out of town this weekend :(
17:33 semiosis d'oh!
17:34 d3vz3r0 yea, bad timing
17:34 d3vz3r0 but i'll see if another person on the team here can go
17:34 jiffe98 heal finished and everything seems to have returned back to normal
17:34 d3vz3r0 thanks
17:35 jiffe98 I am using gluster to not have a single point of failure but gluster was just a single point of failure
17:39 meghanam joined #gluster
17:41 edong23_ any performance tuning tips for gluster on centos 6.5?
17:42 edong23_ this is truely morbid...
17:54 jag3773 joined #gluster
17:55 MeatMuppet joined #gluster
18:04 dbruhn @later tell cleverfoo the stat is done from the mount not from one of the bricks.
18:04 glusterbot dbruhn: The operation succeeded.
18:05 dbruhn or d3vz3r0 if you want to pass that on
18:05 dbruhn edong23_, what are you looking for?
18:11 d3vz3r0 dbruhn, thanks
18:11 d3vz3r0 that makes sense now, through the fuse driver allows gluster to understand that the file is being stated
18:11 d3vz3r0 on the brink mount, it would have no idea
18:12 dbruhn indeeed
18:15 rjoseph joined #gluster
18:16 jiku joined #gluster
18:16 brodiem hey I have a question about how bandwidth distribution works. If you mount a volume using FUSE, the client with the mount can write files directly to the corresponding node, is that right (since it would get that metadata about the cluster)? What if you use NFS? All of the b/w would be forwarded to whatever hostname you used when mounting?
18:17 dbruhn brodiem, NFS is a front to the gluster client on the servers, so the distribution happens after the client communicates with the NFS server.
18:18 brodiem that's what I figured
18:18 rshade98 joined #gluster
18:19 brodiem just doing some testing here for an implementation, and I read quite a few stories of people having to use NFS because of slow file listing
18:20 dbruhn NFS is faster for file system stat functions for sure. You can battle it a bit by ensuring your latency is low from your servers
18:20 dbruhn What's your use?
18:21 gojk joined #gluster
18:21 brodiem the first use case will just be log archival (~100TB)
18:21 brodiem is there any recommended number of inodes to keep within any single directory?
18:22 dbruhn It would only be limited by what's going on under gluster
18:24 brodiem It would be pretty heavily write-oriented, just want to make sure it'll still be useful when someone needs to go traversing files to look for something
18:24 meghanam_ joined #gluster
18:25 edong23_ dbruhn: its just... massively slow performance on gluster...
18:25 edong23_ about 1/10th the performance of nfs
18:25 edong23_ maybe less
18:25 brodiem edong23_: file listings you mean?
18:25 edong23_ write performance
18:26 edong23_ read is better, but still about 1/4th of nfs
18:26 brodiem even large files?
18:26 edong23_ yep
18:26 brodiem wow
18:26 dbruhn edong23_, how many gluster servers, and what kind of transport between the gluster serves and clients?
18:26 dbruhn seems like maybe you have an issue from the client side connecting to all the peers or something along those lines
18:28 edong23_ 2 gluster servers
18:28 edong23_ infiniband between (ipoib, not rdma)
18:28 dbruhn clients?
18:28 edong23_ im testing with 1 client
18:28 dbruhn is your client infiniband as well?
18:28 brodiem so if one were to use NFS.. maybe the best way to have some level of bandwidth separation would be to have clients mount using a hostname that has multiple A records for each of the gluster nodes?
18:29 edong23_ yep
18:29 edong23_ that was to dbruhn, not you brodiem
18:29 brodiem ah
18:29 edong23_ you solution might be right, but i dont know
18:30 dbruhn brodiem, you are correct a lot of people are using rrdns to spread the load
18:30 brodiem the thing I hate about NFS is the blocking issues that can happen if there's a break between client/server. It can make the client really unstable when you have to try and force unmount it
18:30 dbruhn edong23_, are you comparing raw NFS performance to gluster, or are you comparing gluster NFS to gluster?
18:31 dbruhn brodiem, any reason you can't use the fuse client?
18:31 edong23_ brodiem: i believe you can set the timeout at mount time
18:31 brodiem I'd prefer to, just seem to be a lot of people that report abysmal performance using it (no tests of my own though)
18:32 dbruhn I use it, how many files are you planning on cramming into a directory?
18:32 edong23_ dbruhn: both?
18:32 edong23_ initially, i tested nfs
18:32 edong23_ i know gluster will be slower, but i didnt htink this much
18:32 brodiem dbruhn: if the issue is solely about number of files in directories, then we could manage that easily enough
18:32 edong23_ i had 7MB/s write performance inside a vm
18:32 edong23_ native, i get 1.3GB/s write performance
18:33 edong23_ and over the nfs client on another machine (gluster nfs) it is 66MB/s
18:33 dbruhn brodiem, gluster is purely a product of what it's built on, so if you had a x files in a directory and ls'ed it and it was slow on raw disk it's going to be worse on gluster
18:34 dbruhn I've worked with a lot of large NAS systems and they all tend to suffer from these things.
18:34 rwheeler joined #gluster
18:34 brodiem dbruhn: ah ok makes sense. The people reporting those issues made it sound like it was only an issue using fuse (maybe NFS was faster because of caching)
18:34 dbruhn ifs is faster because of caching
18:34 dbruhn s/ifs/nfs/
18:34 glusterbot What dbruhn meant to say was: nfs is faster because of caching
18:35 dbruhn brodiem, the best thing you can do is determine what is acceptable, and them design to it.
18:35 dbruhn s/them/then/
18:35 glusterbot What dbruhn meant to say was: brodiem, the best thing you can do is determine what is acceptable, and then design to it.
18:37 brodiem yup, this first project won't be all that sensitive to any performance standards so it'll be a good way to gauge it moving forward
18:37 dbruhn edong23_, having a little issue tracking with you, maybe you can detail your tests little more. Native? is that raw to the direct attached disk?
18:37 edong23_ yes
18:37 edong23_ native as in local on the machine
18:37 brodiem thanks for the tips
18:37 dbruhn brodiem, I am running about 1.6/1/7 million files per TB on my system and it's not unusable.
18:38 edong23_ i have to head back to work
18:38 edong23_ talk about it later
18:38 dbruhn kk
18:38 Pavid7 joined #gluster
18:39 brodiem dbruhn, thanks we should be well under that
18:43 dbruhn also, one thing that helps a lot with thinks like listing directories is reducing latency via using things like Infiniband, or 10GB
18:43 dbruhn Do you already have a budget for the project?
18:44 B21956 joined #gluster
18:47 andreask joined #gluster
18:48 brodiem druhn, well we have several storage boxes that are currently standalone that we're just going to re-use (about 20T arrays on each). We figured gigabit links would be sufficient for this use case, with the option to bond up to 4 gigabit links per node if we needed to.
18:49 brodiem the second use case for more sensitive data will a replica 2 + geo-rep, but nothing is really scoped out for it yet
18:50 ThatGraemeGuy joined #gluster
18:51 gdavis33 joined #gluster
18:51 _dist joined #gluster
18:56 dbruhn brodiem, the problem with bonding is that it provides more throughput, but doesn't really provide better latency so small IOPS like directory listing doesn't improve
18:58 gdavis33 also lacp bonding is connection based so one to one communication does not see any benefit, only many to one
19:01 jiku joined #gluster
19:03 brodiem I suppose I'll just have to see how we grow into it. It's a pretty gradual migration and we won't just be flipping a switch, so that's good
19:05 brodiem is ~80% link capacity usually when latency starts becoming apparent usually?
19:07 dbruhn you lose 20% to overhead
19:09 brodiem good to know
19:09 brodiem that's still true writing a large file?
19:10 dbruhn it's protocol overhead
19:11 dbruhn part of what makes jumbo frames faster for a lot of people is that it reduces protocol overhead, and back and forth for the smaller frames
19:12 kmai007 joined #gluster
19:12 kmai007 can anybody tell me what this means?
19:12 kmai007 [2014-04-09 16:11:50.713370] W [quota.c:2167:quota_fstat_cbk] 0-dyn_cfu-quota: quota context not set in inode (gfid:e872838f-5d55-49c7-9aef-f2cc6b6c5c69)
19:13 brodiem dbruhn: ah I guess I was thinking 20% overhead as compared to scp'ing the same file for example
19:14 kmai007 i look find the inum associated with it from the .glusterfs/e8/72/e872* but i don't get the exact file or dir. when i find that -inum
19:14 brodiem well scp was a bad example, but probably get the idea
19:14 kmai007 i was rsyncin'g data over glusterNFS
19:15 siel joined #gluster
19:15 dbruhn brodiem, I am just talking about ethernet and TCP/IP in general. It's always safe to say you lose 20% of your bandwidth to layer 2/3 protocol overhead
19:15 John_HPC joined #gluster
19:15 brodiem gotcha
19:16 dbruhn kmai007, does this look like your issues
19:16 dbruhn https://bugzilla.redhat.com/show_bug.cgi?id=812230
19:16 glusterbot Bug 812230: low, medium, ---, vshastry, POST , Quota: quota show wrong value and log full of "quota context not set in inode"
19:17 failshell joined #gluster
19:25 kmai007 dbruhn: my quota is not full, but my logs stopped logging that when I unmounted the glusterNFS
19:26 kmai007 has anybody ever seen this?  [2014-04-09 16:29:38.305935] W [client-rpc-fops.c:2398:client3_3_rename_cbk] 0-dyn_cfu-client-0: remote operation failed: Cannot allocate memory
19:29 chirino joined #gluster
19:34 nightwalk joined #gluster
19:34 dbruhn_ joined #gluster
19:38 dbruhn joined #gluster
19:42 edong23_ ok dbruhn
19:42 edong23_ i have some minutes
19:42 edong23_ if you do
19:42 dbruhn I'll be of help if I can
19:42 rahulcs joined #gluster
19:42 edong23_ ok
19:42 edong23_ ill tell you the setup and how im testing
19:42 edong23_ you tell me what you think could be an issue
19:42 dbruhn perfect
19:43 edong23_ i have 2 gluster servers  both identical
19:43 edong23_ 3.4ghz haswell with 12GB ram
19:43 edong23_ 20 1TB drives
19:43 edong23_ linux md raid 10 offset 3 copies
19:43 edong23_ gives me 1.3GB/s with dd test
19:43 dbruhn ok
19:43 edong23_ about 1.0GB/s average with bonnie++
19:43 edong23_ local, on the md device
19:44 edong23_ i have IB QDR cards which are in ipoib mode for now ( ihave to transistion) at 10gb/s
19:44 edong23_ my clients are vmware
19:44 edong23_ which only supports nfs
19:45 edong23_ but i have a linux machine as well to test gluster vs gluster nfs
19:45 dbruhn ok
19:45 edong23_ nfs only to these machines (without gluster) i get about 900MB/s throughput
19:45 edong23_ nfs spikes the cpu to about 35% across all cores
19:45 edong23_ but never saturated
19:45 edong23_ with gluster client, i get 100MB/s
19:46 edong23_ with nfs client connected to gluster, i get 50-60MB/s
19:46 edong23_ and inside a vm, i get 7MB/s
19:47 dbruhn inside the vm are you using gluster fuse? or NFS
19:48 edong23_ no no, sorry
19:48 edong23_ i should explain that better
19:48 edong23_ the disk for the vm is on gluster
19:48 edong23_ which is being connected to from vmware with nfs client
19:48 dbruhn ahh ok
19:48 edong23_ and it is the disk i am writting too
19:48 edong23_ i know there is overhead with each of thses...
19:48 edong23_ but.. this is phenominally bad...
19:48 dbruhn so here is a question, what does your memory and cpu look like with each layer?
19:49 dbruhn one thing I know about MD is that if it is resource starved at all, it gets extremely slow
19:49 brodiem wow that's all over the place
19:49 edong23_ md is fine
19:49 edong23_ trust me
19:49 edong23_ md with nfs and i am perfectly fine
19:49 edong23_ only gluster uses cpu
19:49 edong23_ and its alot
19:49 edong23_ but not all
19:49 edong23_ i still have plenty of resourse
19:49 edong23_ i havent noticed gluster using much memory
19:51 edong23_ when i was using nfs (not gluster nfs), i would be able to toast 2Gb/s on the infiniband link, without much cpu usage from nfs and get great througput from inside a vm
19:51 edong23_ inside a vm, i would get about 650MB/s
19:56 dbruhn makes sense
19:57 dbruhn edong23_, how many bricks per server?
20:01 edong23_ dbruhn: phone call
20:01 edong23_ just a sec
20:01 cleverfoo joined #gluster
20:01 dbruhn no worries
20:02 edong23_ 1 brick per server
20:03 edong23_ dbruhn: just the 1 brick, it is 7TB
20:04 edong23_ 6
20:04 edong23_ 5.5, apparently
20:04 edong23_ dbruhn: and its raid 10, so very little cpu involved
20:16 Matthaeus joined #gluster
20:18 chirino joined #gluster
20:23 gmcwhist_ joined #gluster
20:23 pk joined #gluster
20:30 churnd joined #gluster
20:31 churnd o... got a q... trying to set up glusterfs on centos 6.5 & noticing most tutorials recommend installing the gluster repos, but i see that the default centos repos have some gluster packages in them
20:32 churnd i realize asking here may be a bit biased but just wondering which i should use
20:33 edong23_ churnd: i htink the default repos only have gluster client tools
20:33 edong23_ no server tool.
20:33 edong23_ but i could be wrong
20:33 churnd edong23_ that makes sense, i was thinking the same since there's no "gluster-server" package in the default repos
20:34 _dist churnd: there might be a glusterfs-server, but it's usually better to use the official repos
20:35 churnd yeah typo there... did "yum search gluster" & didn't see a server package
20:43 plarsen joined #gluster
20:46 dbruhn back
20:46 dbruhn yeah use the gluster repo's the cent's are the downstream of the red hat and are client only
20:47 dbruhn edong23_, are you seeing anything weird in your logs while running your tests?
20:47 dbruhn any of the gluster logs specifically?
20:47 edong23_ lemme check
20:48 Ark joined #gluster
20:53 opacus_jon joined #gluster
21:05 badone joined #gluster
21:06 churnd so i added the gluster repo, now it's complaining about conflicts with the base repo
21:07 churnd i am using yum priorities
21:08 Staples84 joined #gluster
21:09 Staples84 joined #gluster
21:09 churnd http://pastebin.com/Lht8HD2u
21:09 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
21:34 sputnik1_ joined #gluster
21:35 edong23_ dbruhn: no...
21:39 Pavid7 joined #gluster
21:47 dbruhn edong23_, sorry I'm not sure what could be causing the issue.
21:48 dbruhn what are you configuring your volume as?
21:49 edong23_ 2 replicas
21:49 edong23_ dbruhn: im still digging
21:51 ksingh1 joined #gluster
21:51 ksingh1 hello gluster geeks
21:51 ksingh1 http://gluster.org/community/documentation/index.php/GlusterFS_Concepts#Plus_and_Minus
21:51 glusterbot Title: GlusterFS Concepts - GlusterDocumentation (at gluster.org)
21:52 ksingh1 are these positive and negative points about gluster
21:52 dbruhn edong23_, one thing to keep in mind is a replica will double the bandwidth on the writes
21:52 dbruhn not that, that explains the entire drop in speed
21:53 ksingh1 i am confused as its openly mentioned in documentation ? is these for latest version
21:53 ksingh1 dbruhn : please help
21:53 edong23_ dbruhn: yeah... like i said, i expect SOME overhead
21:54 edong23_ but not a 97% drop in speed
21:54 edong23_ but.. there could be some infiniband stuff going on here
21:54 edong23_ digging to that now
21:54 cleverfoo joined #gluster
21:54 dbruhn ksingh1, what are you asking
21:55 ksingh1 i am very new to gluster , was reading about it on its documentation
21:55 ksingh1 so is this http://gluster.org/community/documentation/index.php/GlusterFS_Concepts#Plus_and_Minus
21:55 glusterbot Title: GlusterFS Concepts - GlusterDocumentation (at gluster.org)
21:55 ksingh1 the latest information
21:55 ksingh1 i mean pros and cons of gluster
21:55 dbruhn You want to know what the pro's and cons are?
21:56 ksingh1 yes
21:56 ksingh1 especially whare are the limitations in gluster
21:56 ksingh1 i know its a very rich storage system but there should be some limitations to this
21:57 ksingh1 can we post documentation link in IRC , because as soon as i put a documentation url , i got message from glusterbot
21:57 dbruhn Yeah I saw the doc you linked
21:58 ksingh1 ok great , so can i consider these points for my understanding ?
21:58 ksingh1 those are mentioned in link
21:58 dbruhn I am about to walk out the door for the day, and have a meeting I need to get to another meeting. If you want to wait for others to respond to your questions you can, or I will be back in here tomorrow around 9AM US Central
21:59 ksingh1 just yes or no  , if you can answer
22:00 kmai007 it will always be depends
22:00 kmai007 thats what i've gotten
22:00 kmai007 while being here
22:00 gmcwhist_ joined #gluster
22:01 dbruhn You should get the 3.3.1 admin guide, there are advantages and disadvantages to different configuration types
22:01 dbruhn so the question is to broad for a simple yes or no
22:03 gmcwhis__ joined #gluster
22:06 iamben_tw joined #gluster
22:09 cleverfoo joined #gluster
22:29 chirino joined #gluster
22:29 sputnik13 joined #gluster
23:03 social sijis: sorry my connection went down. thus I thought you did gluster peer probe and havent reverse record or so, so the second gluster peer saved the other node by IP, this can be seen in /var/lib/glusterd/peers/ and you can mitigate it by calling gluster peer probe from both sides
23:03 diegows joined #gluster
23:22 MeatMuppet Hey guys.  I work with cleverfoo and d3vz3r0.  We've been watching a healing operation after a failed brick.   We're down to one file reported in heal info but it seems to be stuck.  gluster reports it to be split brain, so I deleted the stale copy and stat'd it at the fuse mount.  the file has not been recreated on that brick and gluster reports it to still be split-brain but only shows the one remaining copy.   Can anyone suggest next steps?
23:23 MeatMuppet cleverfoo says to offer beer.  :)
23:24 Matthaeus There's another file you have to delete.
23:24 Matthaeus On the brick, in a .directory
23:24 MeatMuppet in .glusterfs?  Got that.
23:25 Matthaeus Does it still show up in lsof?
23:26 MeatMuppet yes, as deleted.  we're discussing a graceful restart of gluster.
23:27 Matthaeus That'll be necessary.
23:32 chirino joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary