Camelia, the Perl 6 bug

IRC log for #gluster, 2012-10-29

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 UnixDev joined #gluster
00:19 UnixDev joined #gluster
00:36 hagarth joined #gluster
01:05 UnixDev joined #gluster
01:08 foster_ joined #gluster
01:17 kevein joined #gluster
01:41 UnixDev_ joined #gluster
01:45 yeming joined #gluster
02:50 sgowda joined #gluster
03:02 dmachi1 joined #gluster
03:09 bharata joined #gluster
03:17 chacken joined #gluster
03:18 chacken left #gluster
03:20 chacken1 joined #gluster
03:29 bulde1 joined #gluster
03:37 MK_FG joined #gluster
03:41 shylesh joined #gluster
03:41 shylesh_ joined #gluster
03:46 sripathi joined #gluster
03:46 nodots1 left #gluster
03:55 ika2810 joined #gluster
04:09 MK_FG Hi. I've just setup a few glusterfs nodes and copied some stuff there, now trying to access it...
04:09 MK_FG And getting ls making "getdents" calls in an infinite loop on a few paths
04:10 MK_FG What can be wrong there?
04:12 vpshastry joined #gluster
04:13 MK_FG glusterfs 3.3.1, --debug shows endless " D [afr-common.c:704:afr_get_call_child] 0-core-replicate-1: Returning 0, call_child: 1, last_index: -1" lines with different id's before message
04:15 MK_FG getdents() seem to return 21, 3, 1, 1, 1 entries in the loop (strace seem to omit the actual entries)
04:17 MK_FG Um, yeah, and the nodes in question are in "3 x 2 = 6" arrangement
04:25 vikumar joined #gluster
04:28 VeggieMeat joined #gluster
04:34 faizan joined #gluster
04:50 faizan joined #gluster
04:52 Humble joined #gluster
04:54 deepakcs joined #gluster
05:10 sgowda MK_FG: are you running on ext4 backend?
05:10 MK_FG Yes
05:10 MK_FG sgowda, Is it expected to be a problem?
05:11 sgowda MK_FG: the recommended back-end fs for glusterfs is xfs
05:12 sgowda MK_FG: bug #838784 is already filed for similar issue
05:12 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=838784 high, high, ---, sgowda, ASSIGNED , DHT: readdirp goes into a infinite loop with ext4
05:12 MK_FG sgowda, Thanks!
05:13 MK_FG Indeed it looks quite similar to what I have here
05:13 chacken1 is there a preferred OS to run this on?  I'm assuming CentOS because the project is supported by Red Hat..
05:14 sgowda MK_FG: you are welcome
05:16 MK_FG Do such troubles with ext4 as a backend-fs come up often?
05:18 MK_FG Hm, nevermind, guess I'll try xfs anyway
05:18 hagarth joined #gluster
05:27 raghu joined #gluster
05:30 seanh-ansca joined #gluster
05:42 sripathi1 joined #gluster
05:42 bulde joined #gluster
05:49 yeming joined #gluster
05:50 chacken3 joined #gluster
05:52 mdarade1 joined #gluster
05:57 bharata joined #gluster
05:59 hagarth joined #gluster
06:06 bharata bulde, wrt your recent mail on libvirt-gluster documentation, when you say "above definition still holds good", not clear which one you mean
06:07 bharata bulde, mine or harsh's ?
06:07 bulde bharata: harsh's : +            <td> gluster </td> +            <td> a server running glusterd daemon </td>
06:08 bulde or, you want him to make it, "a server in the storage pool"
06:08 bulde ?
06:10 bharata bulde, ok will ask him change to that
06:12 bharata bulde, so the server should necessarily have glusterd running
06:18 bulde yes
06:19 bharata bulde, "a server in the storage pool" needs more qualification since we need to define what a storage pool is...
06:19 bharata bulde, can we have a more simpler description :) ?
06:21 bulde "a gluster server part of the cluster?"
06:22 bulde "a gluster server" is obviously the one which runs 'glusterd'
06:23 bharata bulde, somehow tie definition that with volume (and hence the VM image that sits on the volume) ?
06:24 bulde "gluster server which has volume definition" (as you said earlier) still makes sense then
06:24 bulde as of today, that is true on all the machines which are part of storage pool :-)
06:25 bharata bulde, ok
06:25 bharata bulde, so then the definition I have in QEMU source/documentation "'server' specifies the server where the volume file specification for
06:25 bharata the given volume resides." is hopefully not incorrect
06:25 bulde yep
06:26 bharata bulde, use the same in libvirt then ?
06:26 bulde its not incorrect for sure, but i just wanted to make sure you are aware that 'volume resides on all the servers which are having glusterd (and part of cluster)'
06:27 bharata bulde, ok
06:27 bulde sure.. he can use that, i guess, thats better for libvirt community to understand (without gluster's behavioral knowledge)
06:28 harshpb joined #gluster
06:28 deepakcs joined #gluster
06:37 vimal joined #gluster
06:47 bala joined #gluster
06:47 bharata joined #gluster
06:48 MK_FG left #gluster
06:49 guigui3 joined #gluster
06:53 deepakcs joined #gluster
06:56 lkoranda joined #gluster
06:57 ramkrsna joined #gluster
06:57 ramkrsna joined #gluster
06:59 shireesh joined #gluster
07:07 ngoswami joined #gluster
07:20 atrius_away joined #gluster
07:28 bala joined #gluster
07:37 ekuric joined #gluster
07:43 hagarth joined #gluster
07:49 quillo joined #gluster
07:56 puebele1 joined #gluster
08:03 sripathi joined #gluster
08:06 ctria joined #gluster
08:19 Tarok joined #gluster
08:24 kevein joined #gluster
08:36 sripathi joined #gluster
08:38 dobber joined #gluster
08:53 ngoswami joined #gluster
08:56 crashmag joined #gluster
08:57 tjikkun_work joined #gluster
08:59 Triade joined #gluster
09:01 crashmag joined #gluster
09:01 ctria joined #gluster
09:05 vpshastry joined #gluster
09:11 sripathi joined #gluster
09:24 sgowda joined #gluster
09:29 saz joined #gluster
09:31 overclk joined #gluster
09:38 Nr18 joined #gluster
09:45 pkoro joined #gluster
09:45 TheHaven joined #gluster
09:54 samkottler joined #gluster
09:54 hagarth joined #gluster
09:58 shireesh joined #gluster
10:00 Alpinist joined #gluster
10:00 guigui3 joined #gluster
10:02 Nr18 joined #gluster
10:07 ramkrsna joined #gluster
10:08 vpshastry joined #gluster
10:10 lng joined #gluster
10:11 lng Hi! How do I upgrade from 3.3.0 to 3.3.1 on Debian?
10:38 TheHaven joined #gluster
10:42 shireesh joined #gluster
11:05 tryggvil joined #gluster
11:19 ika2810 left #gluster
11:27 davdunc joined #gluster
11:27 davdunc joined #gluster
11:30 rcheleguini joined #gluster
11:31 guigui3 joined #gluster
11:54 snowboarder04 joined #gluster
11:56 sgowda joined #gluster
12:01 unalt joined #gluster
12:03 Tarok joined #gluster
12:11 balunasj joined #gluster
12:12 mdarade1 left #gluster
12:12 morse joined #gluster
12:39 Alpinist joined #gluster
12:42 Melsom joined #gluster
12:44 stefanha joined #gluster
12:44 JoeJulian chacken3: gluster tries to be distro agnostic.
12:45 stefanha Will commits on glusterfs.git/master after v3.3.0qa39 be released in v3.4?  (Assuming they are not fixes for point releases)
12:45 stefanha I'm looking at the glfs.h API which has been introduced recently and doesn't seem to be available in a release or distro packages yet.
12:46 stefanha IIUC it will come out in v3.4
12:46 JoeJulian ~apt repo | lng
12:46 glusterbot JoeJulian: Error: No factoid matches that key.
12:46 JoeJulian ~ppa repo | lng
12:46 glusterbot lng: semiosis' Launchpad PPAs have 32 & 64-bit binary packages of the latest Glusterfs for Ubuntu Lucid - Precise. http://goo.gl/DzomL (for 3.1.x) and http://goo.gl/TNN6N (for 3.2.x) and http://goo.gl/TISkP (for 3.3.x). See also @upstart
12:47 JoeJulian stefanha: If there hasn't been a release-3.4 branch then yes. Changes in master will become 3.4.
12:48 stefanha JoeJulian: thanks
12:48 stefanha There's no release-3.4 yet
12:48 JoeJulian I didn't expect so, but I haven't looked in a week so I didn't want to say with certainty. :)
12:49 ramkrsna joined #gluster
12:52 niv joined #gluster
12:54 niv Hi. We're having a bit of a problem with a 4-node gluster 3.3.0 setup. On one of the clients (a web server with moderate load, vhosts on a glusterfs mount) we're seeing a high cpu load for 'glusterfs'. Where would we start diagnosing and tuning it?
12:54 bulde1 joined #gluster
12:55 JoeJulian logs
12:55 niv on the gluster servers?
12:56 JoeJulian glusterfs would the the client process, so /var/log/glusterfs/mount-path.log
12:57 niv oh, right
12:57 chacken3 JoeJulian: thank you
12:57 JoeJulian You're welcome.
12:57 niv I already checked that, there's some errors about remote operation failed: stale nfs file handle, but nothing to indicate the high load
12:59 JoeJulian Is the mount being heavily used?
12:59 niv it's very read-heavy (reading small php files at a rate of about 5-15/second) - but that shouldn't be it, right?
13:00 JoeJulian Doesn't sound like it. 1 file every 5 seconds isn't very heavy.
13:00 niv 15/second at peak
13:01 niv hmmh
13:01 JoeJulian ack, read that backwards... still not fully awake
13:01 niv no worries
13:01 niv I've already tried parsing the profiler output but that didn't really mean all that much to me
13:02 niv i've tweaked read-ahead to be off and that didn't do anything either :)
13:02 niv so i'm kind of stuck being a gluster noob
13:03 Nr18_ joined #gluster
13:04 JoeJulian Of course, 15 php files per second is commonly more like 1500 with about 15000 lookup failures in many php apps...
13:04 JoeJulian @php
13:04 glusterbot JoeJulian: php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details.
13:04 puebele joined #gluster
13:04 niv thanks *clicks*
13:07 gbrand_ joined #gluster
13:08 chacken3 so in conclusion, use Python ;)
13:08 niv i'd rather not use anything at all. pesky users. ;) but that's not going to float
13:09 niv basic question though: running a 4-brick glusterfs with replication over all 4 nodes is fine though?
13:14 HeMan joined #gluster
13:15 shireesh joined #gluster
13:20 JoeJulian @afr
13:20 glusterbot JoeJulian: For some do's and don'ts about replication, see http://joejulian.name/blog/glust​erfs-replication-dos-and-donts/
13:22 cbehm ... speaking of replication, i've been trying to run a replace-brick and it crashes about 40-50 minutes into the replace. after that, it can't resume the replace-brick. brick.log initially has complaints about missing translator options and then eventually that it can't get the remote brick port
13:23 niv thanks, i'll read up
13:23 cbehm it only gets to the errors about the port after i've manually tweaked the volume config for the "pump" translator to include the missing options
13:23 puebele1 joined #gluster
13:24 cbehm niv: i run a 3-node replication setup and it generally does fine. i'm not sure if our usage scenarios are the same (not php and native fuse clients, not nfs) but it works well otherwise
13:29 JoeJulian cbehm: Which version?
13:46 samkottler joined #gluster
13:53 harshpb joined #gluster
13:57 shireesh joined #gluster
14:05 chouchins joined #gluster
14:12 stopbit joined #gluster
14:13 Nr18 joined #gluster
14:18 harshpb joined #gluster
14:21 chouchin_ joined #gluster
14:22 cbehm JoeJulian: 3.3 (not 3.3.1)
14:24 wushudoin joined #gluster
14:28 bennyturns joined #gluster
14:29 harshpb joined #gluster
14:32 manik joined #gluster
14:33 rwheeler joined #gluster
14:34 ppradhan joined #gluster
14:35 copec joined #gluster
15:02 dmachi joined #gluster
15:07 noob2 joined #gluster
15:08 noob2 i have a little split brain delima.  metadata split brain and i have no idea which node is correct.  one has a slightly newer version than the other one by .001 seconds.  The newer one is half the size of the older one.  which do i blow away :D ?
15:08 noob2 i'm not sure which file owns this metadata.  gluster just gave me a gfid and no file reference
15:09 tmirks joined #gluster
15:11 sag47 joined #gluster
15:11 sgowda joined #gluster
15:12 JoeJulian noob2: I would search for the inode. ls -i $gfidfile ; find $brickpath -inum $inodenumber
15:12 noob2 ok let me have a look
15:14 noob2 hopefully it's a log file or something easy that i can just blow away
15:14 JoeJulian Or something that was already deleted.
15:15 noob2 exactly
15:15 noob2 Joe have you seen this before?  It's in my nfs log a few times: [2012-10-27 18:19:54.634279] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 9d25ee9d, GETATTR: NFS: 10006(Error occurred on the server or IO Error), POSIX: 117(Structure needs cleaning)
15:16 sgowda left #gluster
15:16 JoeJulian Yes. No idea what it means though.
15:16 noob2 i have a coworker performing some intense rsync migrations to the gluster so i suspect it could be related to that
15:17 JoeJulian The error text doesn't exist in the source as anything more than a comment on a #define
15:18 noob2 oh ok
15:18 noob2 guess it's NFS specific then
15:18 JoeJulian #define GF_ERROR_CODE_UCLEAN         117     /* Structure needs cleaning */
15:18 noob2 lol
15:18 noob2 i saw something on redhat bugs that looked possibly similar
15:18 noob2 https://bugzilla.redhat.com/show_bug.cgi?id=806761
15:18 glusterbot Bug 806761: high, high, 3.3.0beta, sgowda, VERIFIED , NFS: unable to delete directories after disk failure
15:19 noob2 i haven't had any disk failures though
15:20 JoeJulian http://xfs.org/index.php/XFS_FAQ#Q:_I_s​ee_applications_returning_error_990_or_.22Structure_needs_cleaning.22.2C_what_is_wrong.3F
15:20 glusterbot Title: XFS FAQ - XFS.org (at xfs.org)
15:20 faizan joined #gluster
15:20 tryggvil joined #gluster
15:20 noob2 this looks bad
15:20 s0| joined #gluster
15:21 noob2 so they're saying i need to unmount the brick and perform an xfs repair
15:22 s0| so I stopped a 3.3 gluster volume , then I deleted it. but now when I try and make a new volume it says that my path is already part of a volume .... how do I clean this up completely ?
15:23 noob2 you need to remove the gluster metadata from the volume
15:23 noob2 let me look up the command i have for that
15:24 noob2 http://fpaste.org/t8cD/
15:24 glusterbot Title: Viewing Paste #247507 (at fpaste.org)
15:24 s0| noob2, thanks. I am not seeing that command in the admin guide.
15:24 noob2 s0| try that
15:24 JoeJulian EUCLEAN is a default value assigned to a local_t for either dht or afr so I guess is the local->op_errno isn't set by some other error but an operation fails, that would be the default.
15:24 noob2 substitute your brick location
15:25 noob2 Joe, think i should take some proactive action on this and unmount/repair?
15:25 JoeJulian s0|: If you wouldn't paraphrase, glusterbot would have automatically linked you to the article. :D
15:26 s0| JoeJulian, I don't understand.
15:26 JoeJulian http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
15:26 glusterbot Title: GlusterFS: {path} or a prefix of it is already part of a volume (at joejulian.name)
15:26 JoeJulian or a prefix of it is already part of a volume
15:26 glusterbot JoeJulian: To clear that error, follow the instructions at http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
15:26 noob2 yes those are better instructions
15:27 cbehm JoeJulian: Sorry I missed your question earlier about version (re: my replace-brick problem). I'm running version 3.3 (not 3.3.1)
15:27 JoeJulian noob2: Unless something else looks broken, I would go ahead and take the 'W' as just that.
15:27 noob2 ok
15:27 JoeJulian cbehm: I haven't had good luck with rebalance before 3.3.1, personally, and I avoided it as much as possible.
15:28 noob2 not that i've noticed.  i have ovirt hooked up to the gluster now as it's vm storage.  it seems to be working fine
15:28 cbehm JoeJulian: thanks - maybe we'll be able to update to 3.3.1 and have better luck then
15:28 JoeJulian I haven't had a single rebalance failure since upgrading though.
15:28 noob2 is 3.3.1 worth upgrading from 3.3.0-6?
15:28 cbehm I realize I can look this up, but is 3.3.1 an online update from 3.3?
15:29 JoeJulian IMHO, very much worth it.
15:29 JoeJulian @repo
15:29 glusterbot JoeJulian: I do not know about 'repo', but I do know about these similar topics: 'repository', 'yum repo', 'yum repository', 'git repo', 'ppa repo', 'yum33 repo', 'yum3.3 repo'
15:29 JoeJulian hmm... should clean that up a bit... which distro do you want, cbehm?
15:29 noob2 cool.  i'll have to kick off a rolling restart when i have some down time
15:30 pkoro joined #gluster
15:30 noob2 yeah looks like a ton of patches in 3.3.1
15:30 cbehm We'd be upgrading via the RPMs that came from the main gluster site
15:32 noob2 will gluster be ok with me stopping the gluster daemon, upgrading and then putting him back into the cluster?
15:32 cbehm Our gluster servers are running CentOS 6, so I believe we'd be trying to update from the epel-6/x86_64 3.3.1 RPM
15:32 JoeJulian @yum repo
15:32 glusterbot JoeJulian: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
15:37 tc00per left #gluster
15:38 noob2 would i need to use this: glusterd –xlator-option *.upgrade=on -N for a 3.3.0 to 3.3.1 upgrade?
15:38 tc00per joined #gluster
15:39 JoeJulian No
15:39 noob2 ok
15:39 noob2 so this is a piece of cake then
15:41 kkeithley cbehm: Using my fedorapeople.org repo, all you have to do is install the repo file in /etc/yum.repos.d/, then yum will automatically fetch the RPMs from the correct repo, i.e. epel-6 in your case.
15:42 copec joined #gluster
15:42 cbehm kkeithley: i'll have to download them manually, no public network for the boxes in question, but i suspect we'll be doing that upgrade once we're out of the main business day
15:45 kkeithley okay
15:47 kkeithley I suppose, for situations like this, there'd be some value in providing a tar file of the repo. Then you could just clone the repo inside on a server in your private network.
15:48 kkeithley s/inside on a server in your private network/on a server inside your private network/
15:48 glusterbot kkeithley: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:49 kkeithley I suppose, for situations like this, there'd be some value in providing a tar file of the repo. Then you could just clone the repo on a server inside your private network.
15:49 kkeithley glusterbot ftw
15:50 * johnmark is just hoping his power survives the day intact
15:50 johnmark wind's starting to whip outside
15:52 DaveS joined #gluster
15:56 seanh-ansca joined #gluster
16:09 copec joined #gluster
16:11 steven_ quit
16:19 bdperkin joined #gluster
16:28 Mo__ joined #gluster
16:30 bulde1 joined #gluster
16:34 ctria joined #gluster
16:35 cbehm johnmark: yeah weather starting to get a bit crazy
16:39 kkeithley be glad you don't live in the eastern seaboard or CT!
16:40 H__ bad time to visit NYC for me
16:44 nhmlap joined #gluster
16:45 hagarth joined #gluster
16:47 s0| http://pastebin.com/TZPN9iVV
16:47 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
16:48 daMaestro joined #gluster
16:50 s0| http://dpaste.org/STjEx/
16:50 glusterbot Title: dpaste.de: Snippet #212030 (at dpaste.org)
16:50 s0| what am I doing wrong. ... seems I can create a gluster mount, but it can't wrtie anything other than empty files to it.
16:51 JoeJulian Gah! Saw someone else with that recently... Let's see if I can find it.
16:52 guigui3 joined #gluster
16:56 guigui3 left #gluster
16:58 cbehm kkeithley: yeah i'm hearing that's going to be pretty rough. i'm not too concerned here for more than inconvenience here (DC)
17:00 JoeJulian s0|: Is that 3.3.0?
17:01 kkeithley Yeah, my friend in the Reston VA area didn't seem too especially put out. (Tho' his wife's name is Sandy too. ;-))
17:01 s0| JoeJulian, glusterfs-server-3.3.1-1.el6.x86_64
17:04 s0| JoeJulian: this is just a test env. so if you want more details for a BZ I can keep it around for a bit, otherwise I am just going to shoot it all in the head and try to re-create stuff from scratch.
17:04 JoeJulian s0|: Might want to add yourself to bug 861423. I don't see a resolution.
17:04 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=861423 medium, medium, ---, vshastry, ASSIGNED , Glusterfs mount on distributed striped replicated volume fails with E [stripe-helpers.c:268:stripe_ctx_handle] <volname> Failed to get stripe-size
17:05 s0| JoeJulian: thanks I will add my self to that BZ. going to shoot this setup and try again.
17:06 y4m4 joined #gluster
17:18 Nr18 joined #gluster
17:22 JoeJulian Looks like bug 832694 is going to stop being a problem, at least in rpm distros. Not sure if this made it into upstream yet.
17:22 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=832694 medium, unspecified, rc, law, ON_QA , ESTALE error text should be reworded
17:34 harshpb joined #gluster
17:38 s0| what kind of performance should I expect with gluster? I ran bonnie on a six node , replicate 3 node and I got about 1/5 of what any single gets to it's attached disk array.  is this about what I should expect ?
17:39 s0| I know there are tuning options, just wondering if I should expect to be able to get "line speed of network" or if there is anyway to calculate what I should be able to get, and then verify.
17:43 harshpb joined #gluster
17:48 jdarcy s0|: Is "1/5 of what any single gets to it's attached disk array" not nearly network line speed?  What kind of array, and what kind of network?
17:49 jdarcy s0|: In general, what performance you will get depends a lot on what kind of I/O you're doing.  Worst case is small synchronous ops (including metadata ops) from a single thread.  Best is large async ops from many threads.
17:50 harshpb joined #gluster
17:51 JoeJulian joined #gluster
17:52 jdarcy Also, bonnie is just about the worst benchmark ever - mostly measures libc overhead and memcpy speed. not disk I/O.
17:54 jdarcy I seriously used to *build* it but not run it as part of a filesystem stress test.
17:54 s0| jdarcy these are old junk boxes I am testing on (hp dl 360 g4p) . I can get about 50MB/S writes 80MB/S reads on them as reported by iostat -xm or bonnie or dd or fio ...  , I got about 18MB/s with the six node cluster.
17:56 jdarcy s0|: How many dd/fio threads?
17:56 s0| 4
17:56 s0| I am wondering if I should lower the gluster thread count from 16 to 4 since this boxes are only 4 core
17:57 jdarcy s0|: You're tweaking the io-thread count?  You shouldn't need to; it does a pretty good job auto-tuning that.  Also, four I/O threads seems a bit low.  Many people find it easy to saturate GigE with 15-20 threads and medium-to-large requests.
17:58 s0| I am not saturating gigE with it at 16 currently.
17:59 s0| seems to be doing about 60% of line speed as a aggravate on any given node when looking at it via iftop
18:00 s0| basically I want to replace our nfs + automount for /home dirs with a gluster cluster, but so far I can't get the POC to have performance anywhere close to what I would expect to see , even with old hardware and it's too slow for use as a home dir for 20+ devs.
18:00 jdarcy s0|: So you're seeing 75MB/s/node at the network level but only 18MB/s total at the application?
18:01 s0| jdarcy: in a nut shell . yes.  I also have been seeing lots of network traffic to a single node in my setup but almost no cpu or disk utilization on that node.
18:02 s0| underlying file system on all the systems is ext4 (not xfs). I wouldn't think that would make large difference but maybe.
18:02 jdarcy s0|: Are your clients mounting via NFS, or native protocol?
18:03 s0| native gluster client.
18:03 faizan joined #gluster
18:04 jdarcy Are you saying that network traffic seems concentrated on one node?  That shouldn't be the case with native protocol.
18:05 UnixDev joined #gluster
18:05 UnixDev is nlm on by default in 3.3 and if so, what function does it serve?
18:06 s0| it seems to do all it's work with the other nodes then one node just keeps sending data and forth for minutes/hours longer than the rest.  I may have a bad node , I tried to punt it from the cluster with a reboot but I was too nice and the box won't go down. now all the nodes are still talking to it, but I can't ssh into it. and ilo isn't working so I am waiting for someone in the dc to walk...
18:06 s0| ...over and hit it over the head
18:07 nhmlap s0|: might want to try using something like collectl or iostat to watch the disk stats on the nodes while you test your writes.
18:08 jdarcy s0|: Well, the symptoms you describe don't match anything I know about.  Perhaps there is something going on with that one node, and see if the results change with it factored out.
18:08 nhmlap s0|: might provide clues if one of the disks is really backed up.
18:08 jdarcy UnixDev: NLM is the NFS lock manager.  I think it's on by default for 3.3, but not sure.
18:08 s0| they aren't reporting a long queue depth or anything in the cciss driver
18:09 UnixDev jdarcy: what happens to locks when there is a network failure?
18:09 s0| and speed reports via iostat -xm 1 are junk.
18:09 s0| i.e. I don't see anything at 100% util nor do I see anything of higher than maybe 10MB/s for reads / writes.
18:09 UnixDev jdarcy: also, how does gluster handle syncing of large files, like 500gig ones that change inside regularly ?
18:10 jdarcy UnixDev: I'm not sure of the exact NFS lock semantics.
18:10 s0| how do I kick a node out. I don't want to replace it, I just want to fail it down ?
18:11 jdarcy UnixDev: Do you mean syncing via gsyncd?  With AFR, no syncing is necessary; all writes are already done to both replicas.
18:11 jdarcy s0|: In general, "gluster peer detach..." but not sure what it does if there are bricks assigned to that node.
18:12 s0| jdarcy, shouldn't gluster deal with a node failure ?
18:13 jdarcy s0|: It can deal with a node failure, but it can't necessarily deal with a misbehaving node.
18:13 jdarcy s0|: If a node is accepting requests but then spinnings its wheels while processing them, that's a very hard thing for any distributed system to deal with.
18:13 UnixDev jdarcy: ok, so thats while all nodes are working normally. when you have a failure of one, what happens then? and fallback? I'm guessing auto heal. If there heal alg is set to diff, how would it handle this while new writes while healing in the case of 500 gig files?
18:16 jdarcy UnixDev: During self-heal, new writes will be applied to both replicas as in normal operation.  If they're in the already-healed region then we're done.  If they're in the not-yet-healed region then they'll be propagated or not depending on the self-heal algorithm.
18:17 UnixDev jdarcy: one thing that worries me about self-heal is the lack of some progress indicator. The heal list is fine for dealing with multiple small files, but when syncing large 500+ gig files its scary.
18:18 perler joined #gluster
18:18 jdarcy UnixDev: Indeed.  Not much I can really say about that, I guess, until we get to ordered-async replication which will (among other things) provide better visibility into that process.
18:19 UnixDev jdarcy: how far out do you think that may be?
18:20 jdarcy UnixDev: Depends how much time I spend on IRC.  ;)
18:20 UnixDev lol
18:20 UnixDev depends how many split brains we have :P
18:20 jdarcy UnixDev: Seriously, it'll be months.  I have some good infrastructure for it, but haven't really had a chance to sink my teeth into it yet.
18:23 UnixDev jdarcy: are there any other xlators planned? maybe something like iscsi?
18:25 stopbit joined #gluster
18:27 jdarcy UnixDev: Well, there's the BD translator from IBM, which integrates directly with qemu.  iSCSI has come up a few times but if it's on the road map at all it's near the end.
18:30 harshpb joined #gluster
18:31 s0| jdarcy, it won't let you detach. to answer your question about what it does if there are bicks assinged to the node.
18:33 perler hello, is someone willing to have a look at an issue using glusterfs inside amazon's AWS. this is the problem: https://forums.aws.amazon.com​/thread.jspa?messageID=394162
18:33 glusterbot Title: AWS Developer Forums: glusterfs in ubuntu 12.04 on EC2 ... (at forums.aws.amazon.com)
18:37 jdarcy perler: What happened when you tried with 3.3?
18:37 jdarcy perler: The first thing I'd look at would be security-group (firewall) settings, make sure the nodes can talk to one another.
18:38 perler i didn't as I try to avoid compiling stuff on a production server, so I tried to use the repository
18:38 perler security groups setting: http://screencast.com/t/zKYn1hVnfJ0e
18:38 glusterbot Title: 2012-10-29_1938 - perler's library (at screencast.com)
18:39 perler ah.. does glusterfs use UDP?
18:39 harshpb joined #gluster
18:39 jdarcy I don't think we use UDP anywhere.  Even the management connections used for peer probes are TCP.
18:39 perler so, this rule should do it?
18:40 jdarcy perler: Should.  What happens on each node if you do "gluster peer status"?
18:40 harshpb joined #gluster
18:42 perler https://gist.github.com/3975620
18:42 glusterbot Title: gluster AWS Gist (at gist.github.com)
18:43 perler but i just realize that the second peer (c002) has the wrong hostname set internally
18:43 perler let me fix this..
18:43 jbrooks joined #gluster
18:44 jdarcy Ah.  That could cause a lot of confusion.
18:44 perler is it enough to clean the /etc/glusterfs/peers directory?
18:45 perler after setting the correct hostname?
18:45 UnixDev joined #gluster
18:45 jdarcy Probably better to do "gluster peer detach" if you can.  Messing with the /etc/glusterfs directories directly is not recommended.
18:46 perler too late ;)
18:46 perler let me reboot the instances and check the status
18:47 perler ok, this looks good to me: root@c001:~# gluster peer status No peers present
18:47 bennyturns joined #gluster
18:48 Tarok joined #gluster
18:49 perler ok, i updated the gist: https://gist.github.com/3975620
18:49 glusterbot Title: gluster AWS Gist (at gist.github.com)
18:49 perler so it looks from both sides
18:50 perler but this is still not woring:
18:50 perler root@c001:~# gluster volume create datastore replica 2 transport tcp c001.patsplanet.com:/export/data c002.patsplanet.com:/export/data/ Host c001.patsplanet.com not a friend
18:51 jdarcy Try probing the other way, using the name.
18:52 jdarcy 3.2 was pretty stupid about names and addresses.  3.3 is no genius, but it's better.
18:53 perler what do you mean by "probing the other way, using the name." ?
18:54 jdarcy perler: From c002, gluster peer probe c001.patsplanet.com
18:54 perler ok, did it
18:54 perler now on both sides the status looks good:
18:54 jdarcy Generally you need to probe both ways for it to be consistent about using names instead of addresses.
18:55 perler root@c001:~# gluster peer status Number of Peers: 1  Hostname: c002.patsplanet.com Uuid: 45507ecb-503c-4e53-9044-b35793e16173 State: Peer in Cluster (Connected)
18:55 perler root@c002:~# gluster peer status Number of Peers: 1  Hostname: c001.patsplanet.com Uuid: a86118f6-51e8-4ace-82fa-6637bf1195fc State: Peer in Cluster (Connected)
18:55 perler but still:
18:55 perler root@c001:~# gluster volume create datastore replica 2 transport tcp c001.patsplanet.com:/export/data c002.patsplanet.com:/export/data/ Host c001.patsplanet.com not a friend
18:55 jdarcy Hm.
18:56 perler ok, is there an ubuntu 12.04 packeage of 3.3 available?
18:57 perler ok, got it..
18:57 perler https://launchpad.net/~semiosis​/+archive/ubuntu-glusterfs-3.3
18:57 glusterbot Title: ubuntu-glusterfs-3.3 : semiosis (at launchpad.net)
18:57 perler let me give it a try
19:03 perler still nothing
19:03 perler I added both peers from either side
19:04 perler root@c001:~# gluster peer probe c002.patsplanet.com Probe successful root@c001:~# gluster peer status Number of Peers: 1  Hostname: c002.patsplanet.com Uuid: 9517a9b7-d924-4f92-8cf0-e8edd73d5d69 State: Peer in Cluster (Connected) root@c001:~# gluster volume create datastore replica 2 transport tcp c001.patsplanet.com:/export/data c002.patsplanet.com:/export/data Host c001.patsplanet.com not a friend
19:05 perler in AWS you have an external and an internal IP - does this ring a bell?
19:05 perler right now, I'm using the external IPs/hostnames
19:06 jdarcy You don't have any aliasing or split-horizon DNS kinds of things going on, do you?
19:09 perler i'm not sure what amazon does internally, but the hostnames (c001/c002) are resolving identically on both ends
19:09 perler although the IP is different:
19:09 perler root@c002:~# host c002.patsplanet.com c002.patsplanet.com has address 54.247.71.196 root@c002:~# ifconfig eth0      Link encap:Ethernet  HWaddr 12:31:3b:06:09:e4           inet addr:10.235.10.18  Bcast:10.235.11.255  Mask:255.255.254.0           inet6 addr: fe80::1031:3bff:fe06:9e4/64 Scope:Link           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           RX packets:1248 errors:0 dropped:0 overruns:0 frame:0           TX pa
19:11 jdarcy Could you put the results of hostname/uname, ifconfig -a, and any local DNS modifications into a gist?
19:11 perler sure.. give  me a sec
19:12 perler ok
19:12 perler not neccessary
19:12 perler I just used the internal IPs on both sides and it's working
19:13 perler not ideal, but I will get along with it.. but maybe the devs can have a look at this behavour and find a solution for using the external IP on AWS, I can imagine I'm not the only one on this platform whoi wants to use AWS
19:14 cbehm perler: that will break if the AWS internal IPs change though. when i've used gluster in aws (not extensive) i've created host file entries that the gluster servers could use
19:14 perler what did you put into the hosts?
19:15 cbehm just in /etc/hosts - e.g. <internal aws ip> c001.patsplanet.com
19:15 perler ah, ok.. but this will change when I create a new instance, so this is also not really fool proof
19:16 jdarcy That's why there's a whole market for provisioning tools to do this kind of customization.
19:16 perler ;)
19:16 * jdarcy is still wondering why the external IPs won't work.
19:16 cbehm jdarcy: AWS instances can't talk to each other by their external IPs
19:16 jdarcy Doesn't really matter, you're better off using the internal ones anyway, but still curious.
19:17 jdarcy cbehm: It seems like they can talk in some limited way, or else the peer probes wouldn't have worked either.
19:17 perler the use the internal routes I suppose..
19:17 perler y
19:18 cbehm one way or the other dealing with AWS ip issues is a pain ;)
19:18 perler that's what I learned today ;)
19:18 perler but I;m happy that it works now, so that i can test the infamous gluster. thanks all!
19:18 jdarcy cbehm: No argument there.
19:19 cbehm i ended up creating a cloudformation + cloudinit setup that uses a private DNS server and autoregisters the gluster servers so they can find each other easily
19:19 * jdarcy is pretty happy with Rackspace.  Or Linode.  Or Storm on Demand.
19:19 ctria joined #gluster
19:20 jdarcy That reminds me, I need to run some tests on Rackspace's new block storage.
19:24 johnmark hrm. just had first power outage - surprised it came back
19:24 jdarcy Power outrage.
19:25 johnmark jdarcy: exactly :)
19:25 johnmark I figured I was out for the night
19:26 dmachi joined #gluster
19:42 copec joined #gluster
19:47 Triade joined #gluster
19:49 Triade1 joined #gluster
19:58 Technicool joined #gluster
19:58 gbrand_ joined #gluster
20:20 hchiramm_ joined #gluster
20:22 foster joined #gluster
20:28 nodots joined #gluster
20:45 lh joined #gluster
20:45 lh joined #gluster
21:04 nhmlap hrm, isn't rackspace's block storage basically just iscsi to a netapp?
21:13 tryggvil_ joined #gluster
21:17 a2 johnmark, jdarcy: good luck handling the hurricane.. hope it dies out sooner!
21:17 nodots left #gluster
21:19 rodlabs_ joined #gluster
21:23 nhmlap Yeah, didn't really think about that.  I imagine it's getting pretty hectic.
21:27 dmachi1 joined #gluster
21:54 nodots1 joined #gluster
22:11 koodough joined #gluster
22:24 ppradhan left #gluster
22:32 andreask joined #gluster
22:35 johnmark a2: thanks :)
22:36 johnmark a2: so far, it's fine. just a lot of wind and rain. as long as we dont' drive, it's not a problem
22:58 sazified joined #gluster
23:03 blendedbychris joined #gluster
23:03 blendedbychris joined #gluster
23:14 z00dax johnmark: i thought you lived out in California ?
23:14 JoeJulian He thought the weather was too boring so he moved to Boston
23:15 z00dax right, fair point
23:18 dmachi joined #gluster
23:29 z00dax btw, lots of my issues are down to the fact that most of my operations are walking the filesystem tree, enumerating / checking / hashing every file as it goes
23:30 z00dax and i am not sure if the solution to that is to reduce bricks + increase size of each brick. -OR- increase bricks, reduce size of each brick
23:35 JoeJulian z00dax: interesting question. Intuitively I would expect an increased number of bricks to help, but I haven't any data to support or refute that.
23:36 z00dax I might get access to about 10 physical machines, wth reasonable storage ( 4x sata's and 8 cores on each machine ) for a period of ~ a month
23:36 z00dax would make for an interesting test bed
23:36 z00dax when i say 'might', i mean they have been ordered..
23:37 JoeJulian Yeah it would. Hehe, that's usually a good likelihood.
23:47 kevein joined #gluster
23:58 tryggvil joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary