Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 bronaugh JoeJulian: ok, so back to what I started with here... you seemed to imply that one could glusterfs-ify any volume (ie: make it a storage brick) and use the data on that volume as part of a glusterfs brick. am I correct in my interpretation?
00:03 bronaugh ie: can one just glusterfs any old thing and then use the data on it transparently?
00:04 elyograg bronaugh: once you do that, ALL access to that data must be through the client mount.
00:04 bronaugh elyograg: so no screwing around behind glusterfs' back, got that.
00:05 elyograg you can cause problems if you modify the brick directly, especially once you add more bricks to go distributed/replicated.
00:05 bronaugh and if you decide gluster's not for you, how easy is it to back out?
00:06 elyograg wipe the .glusterfs directory and optionally the xattrs added to the rest of the files.
00:06 bronaugh great.
00:07 bronaugh yeah, definitely going to have to try this out.
00:07 bronaugh anyone doing glusterfs on zfs yet?
00:08 elyograg i think some people are.  the primary reason you might want to would be snapshots, but they are not available to the client.
00:09 bronaugh that and mdraid is a shit :)
00:09 bronaugh (and xfs sucks, and and and)
00:09 elyograg gotta catch a train.
00:09 SpeeR I did a test with zfs for dedupe, but it reuires so much ram
00:09 bronaugh np; thanks for the info.
00:10 bronaugh SpeeR: ZFS dedup is a nightmare.
00:10 SpeeR heh yeh
00:10 bronaugh SpeeR: my understanding is that you basically -have- to have an L2ARC device.
00:10 bronaugh then it can work OK
00:10 bronaugh but yeah. 320 bytes of overhead per block.
00:11 bronaugh something like 2.5GB of RAM per TB of disk.
00:11 bronaugh or 2.5GB of L2ARC device, which is a bit more palatable.
00:12 SpeeR yeh, I think I read 2GB, made it unusable for my needs
00:12 SpeeR for 1TB
00:12 bronaugh yeah
00:13 bronaugh transparent compression's nice, though, as are a number of other features.
00:13 Kins Which filesystem has the least overhead? I'm not really well versed in unix filesystems.
00:14 guest2012 Hello! Anyone knows what could be the cause for "data movement attempted from node with higher disk space to a node with lesser disk space"?
00:14 bronaugh Kins: what kind of overhead?
00:15 khushildep joined #gluster
00:16 Kins Any kind. I'm not really sure what the most efficient setup for my use case is (keeping web servers synced)
00:16 bronaugh just use ext4.
00:16 bronaugh it's fine.
00:17 Kins I used xfs because thats what the quickstart had. Would ext4 work better? (maybe it will solve this damn input/output error)
00:17 m0zes @ext4 | bronaugh and Kins
00:17 m0zes @ext4
00:17 glusterbot m0zes: Read about the ext4 problem at http://goo.gl/PEBQU
00:17 bronaugh Kins: I would recommend against xfs if you do not have a >16TB filesystem.
00:18 m0zes it may or may not affect your kernel, but it has caused headaches for many.
00:18 bronaugh we run it. it is its own little nightmare.
00:18 JoeJulian -1 bronaugh
00:19 bronaugh JoeJulian: here's why XFS sucks. in the middle of a write, pull power to the machine.
00:19 bronaugh JoeJulian: now power the machine back on and take a peek at the contents of the file being written to.
00:19 bronaugh JoeJulian: oh look, it's all \0
00:20 JoeJulian Don't pull the power. ;)
00:20 SpeeR haha I saw that coming
00:20 bronaugh JoeJulian: yeah you don't always have the option of "don't have a kernel panic"
00:21 bronaugh and kernel panic -> same behaviour.
00:21 m0zes have battery backed raid controllers
00:21 JoeJulian Compared to just plain ol' not working (ext4) or causing the kernel panic (btrfs, zfs) though...
00:21 bronaugh ext4 hasn't caused us any trouble.
00:22 bronaugh the tools completely and utterly suck if you have a >16TB FS though.
00:22 bronaugh m0zes: that doesn't save you.
00:22 JoeJulian And you're using a kernel that doesn't have the most recent ext4 code.
00:22 bronaugh m0zes: the problem is that if xfs gets interrupted at the wrong time, it believes that a file may be inconsistent, so it fills it with \0
00:22 JoeJulian btw, have you filed a bug against xfs for that?
00:23 bronaugh JoeJulian: it's been true for 10 years.
00:23 bronaugh as far as I know, it's a well known issue.
00:23 JoeJulian Has there been a bug filed?
00:23 _Bryan_ is there a way to tell what clients have a volume moutned?
00:23 * m0zes has never had an issue with xfs...
00:23 m0zes and it had been much more performant in my workloads.
00:23 JoeJulian I'm not saying it doesn't happen, hell, it might even be by design, just curious though.
00:23 bronaugh m0zes: I've also had XFS corrupt itself to the point where the filesystem causes NUL pointer derefs until you run xfs_repair
00:24 bronaugh m0zes: that should not be able to happen.
00:24 bronaugh (that was about 10 years ago)
00:25 m0zes the design stems from assumptions made about the hardware. sgi boxes had battery backed everything. you *almost* couldn't lose power in the middle of a transaction.
00:25 bronaugh hm.
00:25 bronaugh NUL problem might be fixed as of 2.6.22
00:25 bronaugh that'd be nice.
00:26 bronaugh http://sandeen.net/wordpress/computers/xfs​-does-not-null-files-and-requires-no-flux/
00:26 glusterbot <http://goo.gl/TdaSi> (at sandeen.net)
00:27 bronaugh m0zes: better perf than ext4?
00:27 bronaugh we all know ext3 sucked :) but ext4 has been pretty good in my experience.
00:28 bronaugh I think it's the one filesystem that -hasn't- lost data for me or friends of mine :)
00:28 m0zes <insert joke about reiser4/>
00:28 bronaugh funny story about that...
00:29 bronaugh I was running reiser4 for a brief period of time.
00:29 Gilbs left #gluster
00:29 bronaugh unpacked some java crap, tried running it. no dice; weird screwups. meanwhile, friend running some other FS had no problems. so I'm really wondering what's going on...
00:29 bronaugh I look in the directory listing. a whole subtree is completely missing.
00:30 bronaugh so yeah. suffice to say that's the last time I tried reiser4.
00:34 JoeJulian When we were having dinner at Red Hat summit, I was sitting across from Ric Wheeler, Senior Manager & Architect RHEL Kernel File System Group for Red Hat. He was pretty strongly suggesting xfs over ext4 even before that issue came up. From the source-code standpoint, xfs is a much more robust system and ext4 is riddled with kludgy workarounds.
00:36 * JoeJulian wishes there were chat logs for irl conversations. :D
00:36 bronaugh hah
00:36 bronaugh still, with testing, you can make even pretty kludgy code work KO
00:36 bronaugh OK*
00:37 bronaugh how's gluster's code? :P
00:37 kwevers joined #gluster
00:37 m0zes the only reason I ever saw to run ext4, was that in-place migration to btrfs would be possible... at some point. it isn't a horrible filesystem, it just pissed me off enough with the tools not handling block devices bigger than 16TB
00:37 bronaugh m0zes: ... yup. that's why most of our volumes are XFS.
00:38 bronaugh ext4 was also the first FS to get TRIM support, which is important to us.
00:38 bronaugh not that it works very well.
00:38 bronaugh (but it doesn't work very well on any filesystem I've tested)
00:38 bronaugh one ends up having to call fstrim ~weekly to have continued usability without I/O stalls.
00:39 * m0zes has a couple 75TB volumes that can be expanded to ~180TB if the need arises....
00:39 bronaugh shrug. 3 30TB volumes, 1 20TB volume.
00:40 m0zes ssds would be nice to have, they tend to cost a lot when you want 60TB of enterprise grade ones.
00:40 bronaugh part of why we're interested in gluster. that and NFS' severe suck.
00:40 * m0zes has a boss with plans bigger than budgets.
00:41 JoeJulian bronaugh: Actually, from my reading, gluster's code seems pretty clean. I can't imagine Avati, Bulde, or Csaba letting a hack go through. They're very strict about fixing problems not working around them (from what I've seen).
00:41 bronaugh JoeJulian: I like hearing that.
00:42 JoeJulian Which is why an ext4 fix didn't make it into 3.3.1. It wasn't good enough.
00:43 * m0zes is upgrading from 3.2.7 to 3.3.1 tomorrow morning. crossing fingers that I remembered all the systems that have glusterfs clients on them.
00:43 bronaugh m0zes: yeah 60T worth would cost a bit too much.
00:45 m0zes he wants hierarchical storage. there is nothing out there, that I've found, to do that without paying lots of money in software licensing fees. so he thinks we can do it all with bcache or flashcache. it makes me very nervous.
00:45 bronaugh hierarchical storage meaning...?
00:46 bronaugh ah. nm.
00:46 bronaugh it is what I thought it is.
00:46 bronaugh you can do this with L2ARC + ZFS
00:46 bronaugh zfsonlinux is treating us fairly well
00:46 m0zes l2arc + ZFS is almost the same thing as bcache and any other linux FS
00:47 bronaugh shrug. perhaps, but better integrated.
00:47 bronaugh and psosibly better tested.
00:48 bronaugh nice to see that non-ZFS people are catching up with ZFS. I like options.
00:51 nueces joined #gluster
00:57 yinyin joined #gluster
01:07 bronaugh ok, so another question.
01:08 bronaugh got glusterfs set up in our test env.
01:08 JoeJulian ... that's a statement. ;)
01:08 bronaugh noticing that we can't make it rebind to an IP without taking the filesystems offline.
01:08 bronaugh so is there a way to do this?
01:09 JoeJulian 3.3.0?
01:09 bronaugh ie: we'd like to bind both ethernet interface and infiniband interface.
01:09 JoeJulian Ah, ok
01:09 bronaugh 3.2.7 because that was what was super easy.
01:10 bronaugh so yeah. how does one do this?
01:10 JoeJulian It should be listening on 0.0.0.0 so as long as the hostname resolves to whichever you want /that/ client to connect via, you should be good.
01:11 JoeJulian Unless I don't know anything about IB.
01:11 JoeJulian ... which I don't.
01:11 bronaugh heh.
01:12 JoeJulian But the way I understand it to work, if you've set it up tcp,rdma you should be able to resolve to whichever you want by hostname.
01:12 bronaugh yeah, that makes sense.
01:18 nightwalk joined #gluster
01:22 anyone9 joined #gluster
01:30 glusterbot New news from newglusterbugs: [Bug 876214] Gluster "healed" but client gets i/o error on file. <http://goo.gl/eFkPQ>
01:32 khushildep_ joined #gluster
01:33 anyone9 Guys? What would be the best use case for GlusterFS?
01:36 semiosis anyone9: imho scientific datasets and media are the best fit
01:36 semiosis anyone9: but people use glusterfs for all kinds of stuff... home directories, web sites, vm images, the list goes on & on
01:37 bronaugh ok, so just transitioned from 3.2.7 to 3.3.1
01:37 bronaugh stopped glusterd, waved the following dead chicken: glusterd --xlator-option *.upgrade=on -N
01:37 bronaugh started glusterd again
01:37 bronaugh No volumes present
01:39 chirino joined #gluster
01:43 JoeJulian ... By what method did you make that transition?
01:44 berend joined #gluster
01:44 berend Trying to install semiosis ubuntu glusterfs package for lucid, but get: glusterfs-common: Depends: libssl1.0.0 but it is not installable
01:44 berend is this ppa known to work?
01:45 khushildep joined #gluster
01:45 JoeJulian It's known... I think that's as far as we've gotten with it yet.
01:45 berend so should not use it?
01:45 bronaugh JoeJulian: installed new packages from glusterfs.org?
01:45 JoeJulian He just built it yesterday with the caveat that it needs testing.
01:46 berend just did then :-)
01:46 JoeJulian semiosis: ^
01:46 bronaugh JoeJulian: then followed instructions at http://vbellur.wordpress.com/2012/​05/31/upgrading-to-glusterfs-3-3/
01:46 glusterbot <http://goo.gl/qOiO7> (at vbellur.wordpress.com)
01:46 semiosis o_O
01:46 anyone9 semiosis, so would that be more for HIGH throughput storage than IOPS (like databases and very small files)... anything that can tolerate some latency in a production env?
01:47 berend semiosis: libssl1.0.0 not available on lucid: http://packages.ubuntu.com/​search?keywords=libssl1.0.0
01:47 JoeJulian bronaugh: rpm based?
01:47 glusterbot <http://goo.gl/gV4zt> (at packages.ubuntu.com)
01:47 bronaugh JoeJulian: no, debian based.
01:47 JoeJulian well there's your problem.... ;)
01:48 JoeJulian Did you move /etc/glusterd to /var/lib/glusterd?
01:48 bronaugh ... nope.
01:48 bronaugh if that's it, then great.
01:48 JoeJulian That's it.
01:49 bronaugh ok, great. back up and running.
01:49 bronaugh where is this documented?
01:49 bronaugh oh, blah.
01:49 bronaugh it -is- documented there; I just missed it.
01:50 bronaugh ok, well, good. that was fairly painless.
01:51 yinyin joined #gluster
01:52 anyone9 semiosis, thanks anyway ;)
01:53 JoeJulian Apart from poking fun at the deb packaging as I'm want to do, the reason I actually expected that problem is that the rpm actually moves that for you as part of the upgrade.
01:53 semiosis berend: i'll fix that right now, thanks!
01:54 berend semiosis: let me know when I can do apt-get update and retest
01:54 JoeJulian anyone9: certainly, iops over network latency is always going to be worse than local. Add in the necessary checks for partition fault tolerance and those get expensive.
01:55 kevein joined #gluster
01:59 semiosis berend: uploaded to launchpad... within a few minutes it will be scheduled for building, which could happen right away or take hours, depending on launchpad
01:59 semiosis i'll keep an eye on it and update in a few
01:59 berend thanks
01:59 semiosis yw, and thank you for reporting the issue
02:00 bfoster_ joined #gluster
02:01 anyone9 JoeJulian Thanks for that insight. We're looking to migrate some production VMs over glusterFS and we're looking at all possibilities. ;) But if you could run some low0
02:01 kkeithley1 joined #gluster
02:01 jdarcy_ joined #gluster
02:01 anyone9 *low-IO stuff on it it would be real nice. Because the tech is awesome!
02:02 anyone9 thanks
02:04 semiosis builds should be started in an hour or less, if the times on launchpad are to be trusted
02:04 JoeJulian anyone9: qemu-kvm recently added support for direct volume access that helps a lot with that.
02:05 JoeJulian anyone9: Also direct volume support is going to be added to postgres
02:05 bdperkin_ joined #gluster
02:05 jack_ joined #gluster
02:06 dblack joined #gluster
02:07 semiosis JoeJulian: lol @ it's known
02:07 JoeJulian :)
02:08 semiosis anyone9: i'd say it's more for mult-reader and/or multi-writer workloads, which databases traditionally are not (they would be master/slave)
02:08 kshlm|AFK joined #gluster
02:08 kshlm|AFK joined #gluster
02:09 semiosis newer "NoSQL" style databases like mongo etc can do multi-master and imho there's no reason to use glusterfs under them because they do their own cluster storage
02:13 semiosis JoeJulian: i have mixed feelings about doing the upgrade stuff in the package... most importantly, it's hard and idk how.  but then i post-rationalize and think that people could still mess up their cluster even with that, so those people would be screwed either way
02:13 bdperkin joined #gluster
02:14 anyone9 JoeJulian, Oh yeah! I googled it on the side... the speed is a lot *faster* -Can't wait for 3.4
02:14 semiosis ,,(3.3 upgrade notes)
02:14 glusterbot http://goo.gl/qOiO7
02:16 bronaugh Mount failed. Please check the log file for more details.
02:16 bronaugh which log file is it referring to here?
02:17 JoeJulian semiosis: I get it. I wondered about it too, but rpm updates have a fairly simple script process so you can tell if it's a update or an install, and I just moved the directory if it existed. If the destination directory exists, that would fail so I felt pretty safe about it. That was before Kaleb started managing it too.
02:18 semiosis bronaugh: client log, /var/log/glusterfs/mount-point-path.log
02:18 JoeJulian It's also the last one listed in ls -ltr /var/log/glusterfs ;)
02:19 bronaugh uh, no such file.
02:20 bronaugh [2012-12-18 18:19:52.525851] E [rdma.c:4417:tcp_connect_finish] 0-skateboard0-client-0: tcp connect to 10.0.230.41:24008 failed (Connection refused)
02:20 JoeJulian did you start your volume?
02:20 bronaugh yes, it's started.
02:21 JoeJulian Dammit... http://nzbmatrix.com/the_end.html
02:21 glusterbot Title: The End (at nzbmatrix.com)
02:22 torrancew JoeJulian: nzbtv.net
02:22 torrancew I had the same sad realization a week back, they're holding me over for now, but nowhere near as good
02:22 bronaugh it seems glusterd is not listening on 24008 even though the client expects that.
02:22 bronaugh and in spite of the fact that Transport-type: tcp,rdma
02:22 bronaugh argh. I think this is going to have to wait for tomorrow.
02:22 torrancew (and it's not limited to tv, despite the name)
02:22 JoeJulian bronaugh: glusterfsd is the ,,(process) that should be listening there.
02:22 glusterbot bronaugh: I do not know about 'process', but I do know about these similar topics: 'processes'
02:22 JoeJulian Try restarting glusterd.
02:22 JoeJulian @processes
02:22 glusterbot JoeJulian: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
02:23 bronaugh JoeJulian: yeah well; nothing with gluster in its name is listening on 24008
02:23 anyone9 semiosis, thanks for the tip. ;)
02:23 semiosis anyone9: yw
02:23 semiosis bronaugh: did you follow ,,(3.3 upgrade notes) ?
02:23 glusterbot bronaugh: http://goo.gl/qOiO7
02:24 bronaugh semiosis: yup.
02:24 bfoster joined #gluster
02:25 JoeJulian torrancew: They're not one of the built-in's for sickbeard though. :(
02:25 torrancew JoeJulian: pm?
02:25 JoeJulian Oops, was going to take that to pm...
02:25 torrancew np
02:25 kkeithley joined #gluster
02:26 Kins Sickbeard is still working for me, so one of my alternative providers must still be working fine.
02:26 bronaugh anyhow, gotta go.
02:26 bronaugh thanks for all the help, folks :)
02:28 jdarcy joined #gluster
02:29 JoeJulian When all TV is outlawed, only outlaws will watch TV. ;)
02:53 semiosis builds on launchpad still haven't started.  i'm outta here though.  later all
02:53 semiosis if anyone wants to keep an eye on them watch these pages...
02:53 semiosis amd64: https://launchpad.net/~semiosis/+archi​ve/ubuntu-glusterfs-3.3/+build/4073291
02:53 glusterbot <http://goo.gl/t8r8E> (at launchpad.net)
02:54 semiosis i386: https://launchpad.net/~semiosis/+archi​ve/ubuntu-glusterfs-3.3/+build/4073292 (but who uses that anyway)
02:54 glusterbot <http://goo.gl/11O5d> (at launchpad.net)
02:55 berend semiosis: i386? that's me....
03:00 anyone9 Awww... Is it possible that I sometimes get an error "Invalid argument " when I'm creating files over a Stripped/Replicated volume on CentOS 6.3 - 3.3.1. The thing is, If I try many times the file gets created....
03:01 sunus joined #gluster
03:03 anyone9 My problem was SELinux
03:03 anyone9 Man, I feel like a newbie ;)
03:04 * anyone9 mutes
03:04 anyone9 nope. It was not SELinux
03:17 saz joined #gluster
03:27 berend semiosis: new built works perfectly
03:27 berend at least it installs :-)
03:27 berend will upgrade a 3.2 cluster with it over the coming days.
03:30 saz_ joined #gluster
03:31 bulde__ joined #gluster
04:02 bharata joined #gluster
04:22 m0zes joined #gluster
04:27 anyone9 https://bugzilla.redhat.com/show_bug.cgi?id=861423 any idea if it will be fixed someday?
04:27 glusterbot <http://goo.gl/pFESl> (at bugzilla.redhat.com)
04:27 glusterbot Bug 861423: medium, medium, ---, vshastry, ASSIGNED , Glusterfs mount on distributed striped replicated volume fails with E [stripe-helpers.c:268:stripe_ctx_handle] <volname> Failed to get stripe-size
04:27 anyone9 wow! this bot is amazing ;)
04:32 semiosis you probably dont really want ,,(stripe)
04:32 glusterbot Please see http://goo.gl/5ohqd about stripe volumes.
04:32 semiosis the bot pulls in any url's title but can also look up bugs like bug 861423 :)
04:32 glusterbot Bug http://goo.gl/pFESl medium, medium, ---, vshastry, ASSIGNED , Glusterfs mount on distributed striped replicated volume fails with E [stripe-helpers.c:268:stripe_ctx_handle] <volname> Failed to get stripe-size
04:32 anyone9 haaaa!
04:32 anyone9 I see ;)
04:33 anyone9 but why don't I not really want Striping?
04:34 semiosis joe's article glusterbot linked to above explains it somewhat
04:34 anyone9 I'll go and read it then.
04:34 m0zes the *only* use for stripe, afaics is if your individual files will be larger than your bricks.
04:37 semiosis there's one other (rare) use case which is a workload with many clients accessing a few large files in different places
04:37 semiosis i've heard some hpc workloads are like that but idk what they are specifically
04:39 semiosis "Stripe was largely developed for HPC pre and post processing jobs (large number of clients reading / writing same file)." at the end of this page: http://supercolony.gluster.org/piperma​il/gluster-devel/2012-May/007987.html
04:39 glusterbot <http://goo.gl/pMZtL> (at supercolony.gluster.org)
04:40 semiosis author of that statement, AB, is a founder of gluster
04:40 isomorphic joined #gluster
04:41 lanning every file starts at the first brick, and stripes through the rest of the bricks.
04:42 anyone9 m0zes, seems like what you're saying is true in regard to what was written about Stripe volume.
04:42 lanning so the IO will generally be first brick heavy and trailing off the later bricks
04:42 anyone9 semiosis, seems like I should be using distributed/replicated! Thanks again for this insight!
04:43 semiosis yw
04:43 vpshastry joined #gluster
04:49 rastar joined #gluster
04:53 yinyin joined #gluster
04:54 Kins joined #gluster
05:00 yinyin joined #gluster
05:06 hagarth joined #gluster
05:16 mohankumar joined #gluster
05:23 bulde joined #gluster
05:24 sgowda joined #gluster
05:32 mohankumar joined #gluster
05:41 mdarade joined #gluster
06:06 shylesh joined #gluster
06:06 shylesh_ joined #gluster
06:09 raghu joined #gluster
06:22 ngoswami joined #gluster
06:23 mdarade left #gluster
06:33 yinyin joined #gluster
06:42 vimal joined #gluster
06:45 rgustafs joined #gluster
06:56 grade joined #gluster
06:58 xim1 joined #gluster
07:00 ekuric joined #gluster
07:04 sunus joined #gluster
07:18 Nevan joined #gluster
07:22 jtux joined #gluster
07:24 yinyin joined #gluster
07:34 ekuric joined #gluster
07:37 guigui1 joined #gluster
07:38 vpshastry1 joined #gluster
07:40 mdarade1 joined #gluster
07:40 mdarade1 left #gluster
07:50 mooperd joined #gluster
07:52 rudimeyer_ joined #gluster
07:53 ctria joined #gluster
08:00 rudimeyer_ joined #gluster
08:04 shireesh joined #gluster
08:05 ekuric joined #gluster
08:06 ekuric joined #gluster
08:11 dobber joined #gluster
08:12 jtux joined #gluster
08:12 gbrand_ joined #gluster
08:15 Ali joined #gluster
08:15 Ali hi members
08:16 Ali Any one have experience
08:16 Ali DO any one have experience on csync2??
08:25 Ali helloo
08:25 Ali any one who can help me on this
08:25 Ali ?
08:36 mdarade1 joined #gluster
08:37 jmara joined #gluster
08:44 xim1 left #gluster
08:45 * ndevos has no idea what csync2 is, so his experience level is non-existent
08:57 ekuric left #gluster
09:04 mdarade1 left #gluster
09:06 ramkrsna joined #gluster
09:07 greylurk joined #gluster
09:17 mohankumar joined #gluster
09:17 vpshastry joined #gluster
09:24 greylurk joined #gluster
09:26 DaveS joined #gluster
09:33 tryggvil joined #gluster
09:33 yinyin joined #gluster
09:36 bida joined #gluster
09:43 bauruine joined #gluster
09:55 jmara joined #gluster
09:58 puebele1 joined #gluster
10:03 hackez joined #gluster
10:07 ekuric joined #gluster
10:23 greylurk joined #gluster
10:30 pkoro joined #gluster
10:34 morse joined #gluster
10:37 x4rlos Remind me GB: prefix of it is already part of a volume
10:37 x4rlos volume create: gv0: failed: /srv/kvm/test-20G or a prefix of it is already part of a volume
10:37 glusterbot x4rlos: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
10:39 bulde joined #gluster
10:43 hackez joined #gluster
10:47 manik joined #gluster
10:50 tryggvil joined #gluster
10:55 mooperd joined #gluster
10:55 x4rlos did anyone get virtualisation set up across gluster?
11:14 mooperd joined #gluster
11:22 Norky joined #gluster
11:26 ramkrsna joined #gluster
11:30 blues-man joined #gluster
11:30 blues-man hello everybody
11:40 Norky I have a volume made of 4 bricks, distributed x2, replicated. It's got a lot of data in it. I want to change the brick mount points on the servers.  What's the right way to do this?
11:42 Norky can I mount the brick (XFS) on both mount points, then do a "volume brick-replace"?
11:42 Norky err, replace-brick*
11:43 passie joined #gluster
11:51 morse joined #gluster
11:54 passie left #gluster
11:58 passie joined #gluster
12:01 passie left #gluster
12:16 blues-man joined #gluster
12:17 vimal joined #gluster
12:18 hagarth joined #gluster
12:29 greylurk joined #gluster
12:30 clag_ joined #gluster
12:43 edward1 joined #gluster
12:53 kwevers left #gluster
13:01 _br_ joined #gluster
13:08 _br_ joined #gluster
13:13 mooperd joined #gluster
13:37 Staples84 joined #gluster
13:40 wushudoin joined #gluster
13:44 khushildep joined #gluster
14:04 guest2012 joined #gluster
14:05 Norky joined #gluster
14:07 Ali left #gluster
14:12 rgustafs joined #gluster
14:14 theron joined #gluster
14:18 ekuric joined #gluster
14:30 khushildep joined #gluster
14:39 passie joined #gluster
14:48 rwheeler joined #gluster
14:50 stopbit joined #gluster
14:59 plarsen joined #gluster
15:08 stopbit joined #gluster
15:09 duerF joined #gluster
15:31 Staples84 joined #gluster
15:54 sunus joined #gluster
16:01 passie left #gluster
16:13 daMaestro joined #gluster
16:16 lh joined #gluster
16:16 lh joined #gluster
16:37 cbehm joined #gluster
16:37 nueces joined #gluster
16:40 ndevos joined #gluster
16:42 m0zes joined #gluster
16:45 _Bryan_ ok so I have hosed myself on a production gluster....the files seem to be in sync but there is a split brain issue left on one of the mirror sets...I know that I used to have a commadn to remove all the attrs so I could resync them and clean this up but I cannot find it...does anyone know of a link to this information?
16:46 ndevos joined #gluster
16:52 zaitcev joined #gluster
16:56 eightyeight after a cluster of 3 peers has been created, is it possible to add a 4th and 5th peer? if so, how?
16:58 zaitcev joined #gluster
17:01 isomorphic joined #gluster
17:08 m0zes probe the new peers from the existing cluster. then you could issue gluster volume <add-brick> statements
17:12 khushildep joined #gluster
17:12 Norky joined #gluster
17:19 JoeJulian Hey all.
17:20 hagarth joined #gluster
17:24 JoeJulian x4rlos: What do you mean by "virtualisation set up across gluster"? I run VMs using a gluster volume for the images.
17:26 JoeJulian Norky: I'm doing that myself. The way I do it is to kill the ,,(processes) for that one brick (I run multiple bricks per server), change the mount point, then replace-brick...commit force. This makes the volume change and starts up the new brick service pointing to the new mountpoint and all is good.
17:26 glusterbot information.
17:26 glusterbot Norky: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
17:26 abyss^ joined #gluster
17:27 * JoeJulian raises an eyebrow at glusterbot
17:29 JoeJulian _Bryan_: I think the safest is probably to determine which xattr you want cleared and find -exec setfattr -n trusted.afr.whatever -v 0x000000000000000000000000 {} \;
17:30 _Bryan_ JoeJulian: thanks man...
17:32 semiosis but you might not get it
17:32 semiosis glusterbot can tell you a udp joke
17:33 Mo___ joined #gluster
17:33 jdarcy Then the tachyon leaves the bar.
17:35 chouchins joined #gluster
17:36 nullck joined #gluster
17:47 puebele joined #gluster
17:56 jdarcy A tachyou walks into a bar and has a drink.
17:57 hagarth high latency?
17:58 jdarcy No, it was just a really fast tachyon.
18:01 * JoeJulian waited patiently for that one. :D
18:05 21WAAE5LI joined #gluster
18:05 torrancew Morning, glusterfolks. Can anyone tell me if gluster supports nfs volumes as bricks?
18:07 jdarcy IIRC, NFSv3 doesn't have the xattr support we need, so no.
18:07 21WAAE5LI I'm having trouble touching a file on a gluster mount. I get an error: touch: cannot touch `x': No such file or directory
18:07 21WAAE5LI I'm in a directory which is in a fuse mounted gluster volume.
18:07 21WAAE5LI Any ideas?
18:07 jdarcy torrancew: Also, ick.  ;)
18:09 jdarcy schmidmt_: That sounds pretty weird.
18:10 schmidmt_ I'm not sure where to even start looking.
18:10 semiosis log files
18:10 semiosis are always a good place to start looking
18:11 semiosis in this case, client & brick logs
18:11 chirino joined #gluster
18:11 jdarcy Unfortunately, an errno like ENOENT isn't much to go on.
18:12 jdarcy Often it's good to add a marker to the log and then attempt the failing operation, to see what's showing up after the marker.
18:12 schmidmt_ The client logs shows:
18:12 schmidmt_ [dht-common.c:416:dht_lookup_dir_cbk] 0-TEST_VOLUME-dht: /ashdev/pipeline/region/us/feeds/bcm/ma​tch/collect/6d/ee/split-21.amazon.com: gfid different on TEST_VOLUME-replicate-1
18:13 jdarcy Well, there we go.
18:13 schmidmt_ What does that mean?
18:13 semiosis schmidmt_: what version of glusterfs?
18:14 jdarcy There are several similar messages.  Let me take a quick look so I know which one we're dealing with.
18:14 * JoeJulian cringes waiting for schmidmt_'s answer...
18:14 schmidmt_ glusterfs 3.3.0 built on May 31 2012 11:16:29
18:14 * JoeJulian relaxes.
18:14 torrancew jdarcy: ick indeed
18:15 torrancew if you knew the whole project, you may vomit - I know I want to
18:15 JoeJulian torrancew: What I did with the nas drives I had was to open them up and take out the hard drives and put them into the servers.
18:16 schmidmt_ Any idea how to rectify the issue?
18:17 torrancew JoeJulian: I'll just go ahead and get this out of the way:
18:17 torrancew We have a netapp, but $boss doesn't want to use it as frontline storage
18:17 torrancew so he requested a solution wherein we use local disks, and real-time replicate to a netapp volume
18:17 jdarcy schmidmt_: OK, so we sent a lookup request to all bricks, either because it's a directory or because it wasn't in the first place we looked (via hashing).  We got multiple responses back, and the GFID in one of those responses didn't match.  That means either the file itself or the parent directory was created twice, each time as though it didn't already exist.
18:17 torrancew so that when local disks die, we can pivot to netapp (yes, this is obscenely ludicrous)
18:17 JoeJulian hehe
18:18 torrancew his suggestion was DRBD, but obviously a block solution doesn't play well with NFS, due to the whole 'FS' bit... so that's when I turned to Gluster
18:18 torrancew I'll investigate NFSv4 support on our netapp
18:18 * jdarcy puts on the helmet with the big chin strap, to keep his head from exploding.
18:19 torrancew jdarcy: I wish mine hadn't already blown up
18:19 * JoeJulian looks for the screws on netapp boxes...
18:19 torrancew I've been trying to consume enough liquor to forget this whole incident upon getting home at night, but this head cold makes that trickier
18:19 jdarcy schmidmt_: First step is to look at the file and the parent directory on the bricks, see exactly where the mismatch occurs.  You can use "getfattr -d -e hex -m . $file_or_dir" for that.
18:19 JoeJulian The liquor cures colds too.
18:20 schmidmt_ jdarcy: thanks  btw.
18:20 jdarcy torrancew: So you're going to have multiple clients replicating between their respective local disks and the same NetApp?
18:21 torrancew jdarcy: in theory, yes, though with dedicated netapp vols per client
18:21 torrancew I'm basically hoping I can prove this won't work, though
18:21 torrancew because wtf is the point of dropping 6 digits on a NetApp if you don't trust it?
18:21 semiosis torrancew: just use geo-replication to sync from glusterfs to the netapp's nfs
18:21 Kins JoeJulian, did you get the msg I sent you?
18:22 jdarcy torrancew: Well, it's ugly, and likely to be slow, but the real killer is the lack of xattr support.
18:22 semiosis torrancew: you could've had redundancy for just one more digit
18:22 JoeJulian Not here...
18:22 torrancew semiosis: we have redundant heads, just not redundant appliances
18:22 torrancew the whole thing started when we hit some OnTap bug that caused both heads to reboot within a short time
18:23 semiosis JoeJulian: i pm'd you yesterday too... maybe your znc is messed up
18:23 torrancew (this occurred just before I started at $thisGig)
18:23 JoeJulian It's probably on my office desktop.
18:23 JoeJulian reconnecting to znc doesn't replay pms.
18:23 obryan joined #gluster
18:25 schmidmt_ jdarcy: We have a 4 host x 2 brick config. 2 show one gfid and 2 show a different gfid.
18:25 Kins Yeah, I hate that.
18:26 Kins I swear it didn't always lose your PMs.
18:26 Kins Atleast they added multi-network support to users!
18:26 torrancew semiosis: can you point me to a link to chase your geo-replication theory down?
18:27 semiosis uhh ,,(rtfm) i guess
18:27 glusterbot Read the fairly-adequate manual at http://goo.gl/E3Jis
18:27 semiosis haven't used geo-rep yet myself :)
18:28 torrancew ok, thought you were speaking from experience, currently perusing the geo-rep docs
18:32 jdarcy schmidmt_: Basically, you need to decide which pair you want.
18:32 jdarcy schmidmt_: If it's the file, unless you know how to merge the contents, you'll just need to delete/archive the other.
18:33 schmidmt_ If it doesn't matter, would there be an issue with removing the directory?
18:33 jdarcy schmidmt_: That would resolve the GFID mismatch.  Up to you whether that's an acceptable solution.
18:33 schmidmt_ jdarcy: More importantly, what could cause this issue?
18:33 y4m4 joined #gluster
18:34 jdarcy schmidmt_: GFID mismatches are practically always (maybe just always) caused by things being created while bricks where they already exist are down.
18:34 jdarcy schmidmt_: So it exists on A+B, but both are down (!) so a second separate copy gets created on C+D.
18:36 schmidmt_ jdarcy: I'm set the volume to enforce a quorum. Would that prevent issues if the client is using NFS?
18:36 jdarcy schmidmt_: There's a different flavor of GFID mismatch within a replica pair, if just one is down, but that would lead to a different symptom than you describe (GFID=xxx on one brick, GFID=yyy on another, absent on the remainder).
18:37 jdarcy schmidmt_: Now that is an interesting question.  I strongly suspect that the quorum options don't get set in the NFS volfile, so quorum enforcement won't be done.  Let me check, since that would be my fault.
18:38 * jdarcy knows that NFS volfiles get built separately from regular client/server volfiles, and doesn't remember changing that code.
18:41 jdarcy Hey neat, it does build the NFS volfile correctly (i.e. with quorum enforcement turned on) but obviously that's not preventing the problem in this case.
18:42 semiosis jdarcy: istr trying to create files when a distrib. subvol was missing and for files that hashed to that volume glusterfs would not create them at all
18:43 semiosis it would not place them on another subvol
18:43 jdarcy semiosis: ISTR that being fixed, perhaps by yours truly.
18:44 semiosis :)
18:45 jdarcy And yet, I just tried it on a two-way DHT volume and half of my file creates failed.  Hm.
18:46 schmidmt_ jdarcy: What do you believe would be the best course of action to prevent this?
18:47 semiosis afaict we still dont know what to prevent
18:50 nueces joined #gluster
18:52 jdarcy schmidmt_: The first thing I'd do would be to verify the theory by looking at how often bricks get disconnected (or go down) and if that seems likely to be the true cause.
18:53 schmidmt_ The situation where this occurred was when many client were doing many file operations on what are mostly the same files. It seems the servers may get loaded and become unresponsive to some clients.
18:54 schmidmt_ As far as I can tell the bricks never went down.
18:57 jdarcy schmidmt_: Right.  I have another bug right now that seems to be related to certain clients becoming disconnected from certain bricks, also under load, but no one ever completely went down.
18:58 schmidmt_ We have a fantastic load so if I can test something to help out let me know.
19:00 y4m4 joined #gluster
19:01 jdarcy I love Grumpy Cat.  http://memegenerator.net/instance/32167626
19:01 glusterbot Title: WHEN LIFE GIVES YOU LEMONS... DIE - Grumpy Cat 1 | Meme Generator (at memegenerator.net)
19:02 andreask joined #gluster
19:03 jdarcy schmidmt_: You could try looking in the *client* logs for signs of frequent disconnect/reconnect cycles.
19:04 andreask left #gluster
19:06 * elyograg burns your house down.  with the lemons.
19:06 nhm elyograg: the goggles, they do nothing!
19:10 schmidmt_ jdarcy: we'll test it out again and I'll monitor the logs. I'll get back to you.
19:17 andreask joined #gluster
19:17 andreask left #gluster
19:20 bauruine joined #gluster
19:20 khushildep joined #gluster
19:27 mooperd joined #gluster
19:29 * m0zes found this bug helpful today: https://bugzilla.redhat.com/show_bug.cgi?id=782285
19:29 glusterbot <http://goo.gl/61CT0> (at bugzilla.redhat.com)
19:29 glusterbot Bug 782285: medium, low, future, rtalur, ASSIGNED , [FEAT] volume set 'nfs-transport-type' should be included in 'volume set help'
19:31 m0zes when upgrading from 3.2.7 to 3.3.1 with several tcp,rdma volumes, the nfs server decided it would fail to connect using rdma. forcing it to tcp, and nfs works for those volumes again.
19:31 bronaugh m0zes: ok, that's -exactly- what we're experiencing.
19:31 gbrand_ joined #gluster
19:31 bronaugh it seems it doesn't bind port 24008 on the server.
19:31 bronaugh which is required for RDMA
19:47 Slydder joined #gluster
19:48 Slydder hey all
19:48 Slydder anyone here running gluster on a dell dr4000?
19:55 Slydder how about on a C6220?
19:58 bronaugh why is the hardware even relevant?
20:01 Slydder because our company buys only from dell. policy is a bitch sometimes.
20:02 bronaugh considering gluster just sits on top of a filesystem, I'd say reliability is going to be about the same as the filesystem.
20:07 Slydder I already run gluster on a few servers (2 of them are ARM based Qnap servers privately owned) so it's not a question of whether I want to use it or not or how reliable it is. I am still looking for exact specs for the C6220. I have only been able to find chipset info on the site.
20:08 Slydder found it. AMD opterons
20:09 Slydder ah that was the C6140. the c6220 are xeon based. either way a good deal.
20:14 bronaugh *cough*
20:15 bronaugh http://www.supermicro.com/products/c​hassis/4U/847/SC847E26-R1K28JBOD.cfm
20:15 glusterbot <http://goo.gl/bDQv2> (at www.supermicro.com)
20:15 bronaugh but I know you're stuck with Dell.
20:15 bronaugh if I had to guess, thuogh, the Supermicro solution will be less than half the price.
20:16 bronaugh fwiw, Seagate's also selling their 4TB drives in an "enterprise" version now, so you might consider those.
20:16 bronaugh (and Hitachi's been doing same for a year or so)
20:19 Slydder yeah. my last qnap buy was with the hitachi's. almost got my underwear moist when I made that order. lol
20:19 bronaugh haha
20:20 bronaugh I'm not really sold on the whole "enterprise" drive thing. paying twice as much for a possibly marginally lower failure rate seems silly to me.
20:21 Slydder yeah well they go with dell because on-site support can be there inside 15 minutes. our DC is like right next door to the dell shop in hamburg
20:21 bronaugh *shrug* guess that makes sense.
20:21 Slydder so if a part fails we can get it quicker than normal.
20:21 Slydder otherwise we wouldn't be with them.
20:22 bronaugh yup. fair enough.
20:23 Slydder so. have made my choice. going with the C2100. up to 28 TB is enough for this one and keeps me below my 6000 Euro cost prediction. ;)
20:24 bronaugh heh.
20:24 bronaugh we can get about twice that for 6000eur here in Canada using Supermicro hw + desktop drives.
20:24 bronaugh actually more like 2.5x that.
20:25 bronaugh and for 7500eur or so we can do 105TB
20:26 JoeJulian http://www.asus.com/Server_Work​station/Servers/RS720QAE6RS12/
20:26 glusterbot <http://goo.gl/YpBCF> (at www.asus.com)
20:27 bronaugh they'd be more confidence inspiring if they could spell :P
20:27 bronaugh but yeah. fun box.
20:27 bronaugh about the same density as 7U 14-blade chassis.
20:27 JoeJulian Perhaps, but the servers that I have bought are outstanding.
20:28 bronaugh we've looked at their boards in the past.
20:28 bronaugh IIRC the lack of sufficient PCIe slots was a blocker.
20:28 bronaugh I don't have anything overly negative to say about Asus, though. their consumer stuff seems to have quite a low failure rate.
20:30 JoeJulian I'd always dismissed them in the past as bleeding edge and unstable, but their case designs are really well thought out. ipmi included (I suppose they probably all do that now, but comparing to the last time I bought servers...)
20:30 JoeJulian And they've been fast, reliable, and energy efficient.
20:30 bronaugh IPMI ... yeah.
20:30 bronaugh how good is their implementation?+
20:30 JoeJulian I don't need a windows box...
20:31 JoeJulian It's better than any of the supermicros or the tyan boxes we have.
20:31 bronaugh hm
20:31 bronaugh I am not impressed with either Supermicro or Tyan's IPMI implementations.
20:31 bronaugh they seem to randomly fuck off when you need them.
20:32 bronaugh ie: not very reliable.
20:32 bronaugh and the tools to interact with them are shiver inducing.
20:33 bronaugh really wish they'd stick SFF8087 connectors on their boards.
20:34 glusterbot New news from newglusterbugs: [Bug 884280] distributed volume - rebalance doesn't finish - getdents stuck in loop <http://goo.gl/s4xvj>
20:46 y4m4 joined #gluster
20:57 gbrand__ joined #gluster
20:59 gbrand_ joined #gluster
20:59 GLHMarmot joined #gluster
21:52 GLHMarmot joined #gluster
21:53 duerF joined #gluster
22:09 inodb_ joined #gluster
22:30 badone joined #gluster
22:32 tryggvil joined #gluster
22:50 saz_ joined #gluster
23:05 rodlabs joined #gluster
23:19 hattenator joined #gluster
23:27 inodb_ left #gluster
23:56 m0zes_ joined #gluster
23:58 m0zes joined #gluster
23:59 Kins joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary