Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-22

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 bronaugh _Scotty: ok, read backlog.
00:03 bronaugh _Scotty: so here's what I did; I actually deleted the volume. however, it didn't actually disappear; it put glusterfs into a bad state where the server couldn't start.
00:04 bronaugh I removed the directory and pid-file in /var/lib/glusterd/vols and that fixed it. but now, it doesn't want to create a new volume using that brick because it claims it's already part of a brick.
00:04 bronaugh there's no attributes on the directory, no .gluster directory in the brick, and only one mention of it in the config -- in the NFS config.
00:05 _Scotty bronaugh: did you check the xattrs
00:05 bronaugh on what?
00:06 bronaugh I checked them on the root inode of the filesystem..
00:06 _Scotty the brick.
00:06 bronaugh er, the brick.
00:06 _Scotty yeah, the root.
00:06 _Scotty what's your brick mountpint
00:06 _Scotty *mountpoint
00:06 bronaugh getfattr /datasets/0099 returns nothing.
00:07 bronaugh and yes, that's the mount point.
00:07 _Scotty what's the error you're getting
00:07 _Scotty the specific error
00:07 bronaugh root@skateboard:/var/lib/glusterd# gluster volume create datasets-0099 transport tcp,rdma skateboard-ib:/datasets/0099/
00:07 bronaugh /datasets/0099 or a prefix of it is already part of a volume
00:07 glusterbot bronaugh: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
00:08 _Scotty HA! Glusterbot strikes again.
00:08 bronaugh root@skateboard:/var/lib/glusterd# setfattr -x trusted.glusterfs.volume-id /datasets/0099/
00:08 bronaugh setfattr: /datasets/0099/: No such attribute
00:08 bronaugh oho. but there was a gfid...
00:09 bronaugh why wasn't the gfid listed by getfattr?
00:10 _Scotty Okey, try: getfattr -m . -d /datasets/0099 from skateboard
00:10 _Scotty what's the output?
00:10 _Scotty AND
00:10 _Scotty what's the output of getfattr -m . -d /datasets
00:10 bronaugh wtf.
00:10 bronaugh that works.
00:11 bronaugh root@skateboard:/var/lib/glusterd# getfattr -m /datasets/0099/. -d /datasets/0099
00:11 bronaugh that doesn't.
00:11 _Scotty yeah, you have to give it a match.
00:11 _Scotty yeah
00:11 _Scotty because,
00:11 bronaugh ok nm.
00:11 _Scotty the -m flag is a regex match
00:11 bronaugh I get it now.
00:11 _Scotty on the particular variable.
00:11 bronaugh why the hell doesn't -d work?
00:11 _Scotty -m . means give me everything
00:11 bronaugh it's supposed to dump all attributes according to the man page.
00:11 _Scotty you still need to give it a match.
00:11 _Scotty yeah the man page is wrong.
00:11 bronaugh fuck :/
00:12 _Scotty lol
00:12 _Scotty Been there, done that, *facepalm*'d it
00:12 bronaugh can you put something about this in what you've written up? because that's a serious gotcha
00:12 bronaugh in fact, there's a lot of serious gotcha's here.
00:12 bronaugh not liking it :/
00:12 _Scotty whatcha mean?  what's wrong with the zfs on gluster docs?
00:13 _Scotty I mean, I can add a wiki page on xattrs, I suppose.
00:13 bronaugh nothing. there's an overall lack of any coherent docs on what to do when things go wrong with gluster, though.
00:13 _Scotty ah.
00:13 bronaugh and there's obviously some minefield-like problems.
00:13 _Scotty xyzzy
00:14 bronaugh for instance, what I did: I did a glusterfs volume delete (not smart) from a client node.
00:14 bronaugh this claimed to have failed.
00:14 _Scotty I, much like you, seem to be a minesweeper.
00:14 _Scotty hrm.
00:14 bronaugh however, it didn't actually fail. it partly succeeded, and left the glusterd config in a broken state which prevented glusterd from starting.
00:14 bronaugh that's not OK.
00:14 _Scotty yeah, I only run volume deletes from the server side
00:15 bronaugh operations like this should be atomic.
00:15 _Scotty in theory yes.
00:15 bronaugh it's pretty simple, really.
00:15 bronaugh some operations, you know will succeed. you save those for last.
00:15 bronaugh the other operations, you accomplish by trying to -move- things, not delete them.
00:15 _Scotty I didn't write it.  I'm just a lowly l-user.
00:16 bronaugh if the moves fail, you roll back the transaction by moving things back.
00:16 bronaugh it can probably be done better than the way I'm describing.
00:16 bronaugh but atomicity of operations here is kind of important.
00:16 bronaugh failure to have that means the system is a bomb waiting to go off.
00:16 _Scotty atomicity is important, yes, but it really isn't scalable?
00:17 bronaugh these are big operations. anything you're doing with a volume is a big operation.
00:17 bronaugh that means, infrequent.
00:17 bronaugh performance doesn't matter on those so long as they don't involve insanely long wait times.
00:17 bronaugh this operation happened quickly, because all it involves is deleting a few files and extended attributes.
00:19 bronaugh anyhow... operations like this need to be safe.
00:19 _Scotty ah.
00:19 bronaugh they should never leave things in such a broken state that the server can't start.
00:19 _Scotty I'd recommend filing a feature request, or filing a bug.
00:19 bronaugh this smells bad.
00:19 bronaugh this smells like people coding for the happy case and not thinking about what happens when things go sideways.
00:20 _Scotty I don't know how many folks have in their use cases of creating then deleting and re-recreating volumes to account for a transport change; which in theory shouldn't be required client-side.
00:20 _Scotty sounds like an edge case
00:20 _Scotty but you should file a bug, anyhow
00:20 glusterbot http://goo.gl/UUuCq
00:21 _Scotty Kudos, glusterbot.  I was wondering when you'd kick in.  :-)
00:29 bronaugh heh.
00:31 ersatz-i joined #gluster
00:31 crushkill left #gluster
00:34 _Scotty Sorry I can't be more help in this case; I have my own bugs I'm trying to get addressed before I can roll this out to production.
00:35 bronaugh yeah, no worries.
00:35 bala joined #gluster
00:37 ersatz-i I could use some help. I'm trying to convert my current cluster(LAMP nodes) adding load balancing & having a headache trying to absorb everything on distributed file systems, drbd, HA, etc. In my potential (ideal) use case potentially the same section of the same file could be getting written to on all 4 nodes at the same time -- it seems like a distributed file system + drbd is the best way
00:37 ersatz-i to go about this problem? is this the path I should be looking into? Thanks #gluster. :)
00:37 bronaugh well; gluster ensures that they're all written at the same time...
00:37 bronaugh it doesn't use drdb to achieve this.
00:39 ersatz-i That's what I'm trying to understand. Gluster is a distributed file system? say I create four nodes install the same LAMP stack on each (e.g client htdocs in /var/www on all 4 nodes), how would I implement gluster? use gluster on each node and the data gets distributed across all 4 nodes kind of like the way raid works?
00:40 ersatz-i if that's the case, what happens if 3/4 nodes go down, etc.
00:40 bronaugh well; think of "distribute" as "raid0"
00:40 bronaugh but per-file.
00:40 _Scotty you're looking for replicate.
00:41 bronaugh and "replicate" as "raid1"
00:41 _Scotty right on, bronaugh.
00:41 bronaugh so if you want -all- the same data to be on the same nodes, use replicate.
00:41 JoeJulian ersatz-i: do you need the file stored in 4 unique places, or do you just need the file to be accessible from them?
00:41 bronaugh if you want raid10 equiv, use distribute + replicate
00:42 twx_ bronaugh: i.e. replica=2 but with >2 nodes, amrite?
00:42 ersatz-i JoeJulian: I technically just 'need' each node to have the same data, how I accomplish this is totally up in the air -- but it seems distributed file systems are the only way to avoid SPOF.? :)
00:42 bronaugh twx_: not having set it up yet, you probably know more than me about the exact details :)
00:42 bronaugh twx_: I'm still trying to get the full overview straight.
00:42 ersatz-i :)
00:43 twx_ coz I was curious regarding distribute vs stripe really, stripe actually striped the file into several pieces in a traditional manner ?
00:43 twx_ and then as you said, distribute places files around various nodes
00:43 bronaugh twx_: stripe = block striping.
00:43 twx_ but how is that calculated
00:43 bronaugh distribute = file striping
00:43 twx_ ahh..
00:43 bronaugh distribute distributes files by a combination of a hashed filename and a node id.
00:44 bronaugh stripe stripes files afaik in a round-robin fashion as raid0 does.
00:44 JoeJulian See http://joejulian.name/blog​/dht-misses-are-expensive/ to play with hash calculation.
00:44 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
00:44 twx_ yeah, so basically distribute != some form of concatenation
00:44 glusterbot New news from newglusterbugs: [Bug 889630] gluster volume delete is not atomic <http://goo.gl/duZ9q>
00:44 twx_ which is good
00:45 bronaugh twx_: think of bricks when using distribute as overlays.
00:45 bronaugh put all the bricks together, the overlays cover all the files.
00:45 bronaugh distribute + replicate = overlays cover all files n times.
00:46 JoeJulian distribute is an amalgamation of filesystem name spaces.
00:46 twx_ yup, got that already, was just curious about the diff between stripe/distrib really
00:46 JoeJulian @stripe
00:46 glusterbot JoeJulian: Please see http://goo.gl/5ohqd about stripe volumes.
00:46 bronaugh yeah. just diff between block-level and file-level striping across volumes really.
00:47 ersatz-i My situation currently, I have db1.mydomain.tld, ww1, mx1, ns1, ns2, managing everything/sites/apps from ww1 via ispconfig. I switched to a new VPS provider where I can implement load balancers as I want without added charge, so I figured (having absolutely no HA on my own-end, though each vps is itself redundant at the provider), I could be making better use of my resources going with 4 nodes
00:47 ersatz-i all running LAMP+mail+dns. Trying to understand the data part has been a headache. :)
00:48 JoeJulian I wouldn't even use the word striping for distribute, personally. Too many bad experiences putting stripes back together to try to reassemble lost data.
00:48 bronaugh JoeJulian: eh; really, though, how is it significantly different? if you lose a brick that contains a bunch of files, you're going to lose 1/n of the files.
00:48 twx_ not many scenarios where you'd want to stripe that I can think of really
00:48 _Scotty FWIW, the only case I'd consider using striping or stripe + replicate would be to create a volume to host HDFS data.
00:49 JoeJulian bronaugh: Because when you lose a drive from a stripe set you lose 1/n of ALL your files.
00:49 bronaugh JoeJulian: it's slightly better than raid456, where you lose 1/n of the blocks... but you still lose data.
00:49 twx_ _Scotty: the reason for that? performance or the architecture of hdfs ?
00:50 _Scotty twx_: performance, mainly.  At least how we use HDFS.  Large files, as in 20GB files.
00:50 twx_ I c.
00:50 ersatz-i If there are any suggestions I'd be highly appreciative. I could go with unison or rsync methods but it seems like if two people write the same file one two different nodes the changes are lost implementing rsync or unison etc to keep nodes in sync?
00:50 twx_ not really familiar with hdfs
00:50 bronaugh ersatz-i: yes, you are liable to have split-brain problems with unison/rsync.
00:51 _Scotty twx_: it's ugly.  based on a reference paper on Google's BigTable.  ugh.
00:51 bronaugh ersatz-i: you still can get into them with gluster, but it's far less likely.
00:51 _Scotty ersatz-i: if you use replicate, there's no need to use rsync or HA, really.
00:52 ersatz-i What happens in a cluster with 4 web nodes all formatted with a distributed file system/gluster when one node goes offline?
00:52 _Scotty ersatz-i: you lose the data on that one node.
00:52 bronaugh ersatz-i: depends if you're using distribute or replicate.
00:52 _Scotty ersatz-i: or at least, access to the data stored on that node.
00:52 ersatz-i ahh!
00:52 twx_ if 4 nodes = 4 bricks, and you have replica=2 you loose one brick, you still have your data
00:53 bronaugh ersatz-i: if you're using replicate, after a few seconds the remaining nodes continue to work just fine.
00:53 bronaugh ersatz-i: if you're using distribute, the files stored on that node go missing.
00:53 _Scotty right on, bronaugh.
00:53 ersatz-i twx: I understand now, you can use gluster to replicate AND/OR distribute?! I didn't understand.
00:53 JoeJulian Also, "formatted" is incorrect as the actual filesystem used for the brick would be a block-level filesystem like xfs.
00:53 bronaugh ersatz-i: yeah. that's exactly right.
00:53 ersatz-i that makes perfect sense.
00:53 bronaugh ersatz-i: what JoeJulian said.
00:53 bronaugh you can stick gluster on anything.
00:54 ersatz-i ahh :)
00:54 bronaugh you can also back gluster out of anything (which is especially easy if you're just using replicate).
00:54 JoeJulian Although, be careful with ,,(ext4)
00:54 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
00:54 bronaugh but don't mess with files under glusterfs.
00:54 bronaugh it will get angry
00:54 bronaugh -once- things are part of a glusterfs volume, do not screw with the underlying fs. mount it as gluster and screw with the mounted copy.
00:54 _Scotty bronaugh: oh so true.
00:54 JoeJulian +1
00:54 twx_ making it sound like that black smoke monster in Lost
00:55 bronaugh _Scotty: haven't actually done that; but heard enough dire warnings.
00:55 _Scotty hahahahahahaha!!!!
00:55 bronaugh (plus it makes sense to me)
00:55 _Scotty bronaugh: lived to tell the tale.
00:55 _Scotty twx_: but it's just the island security system.  right?? lol
00:56 twx_ yeah its just your distributed filsystem / data governance system
00:56 twx_ it WILL get angry
00:56 twx_ :)
00:56 _Scotty lol
00:56 ersatz-i so if I use gluster on a 4 node web cluster I can use replicate to insure all nodes are in-sync up to the second/few seconds? when one node dies, using replicate, I'm assuming it will be braught back up to speed when it comes back online? .. my last question is it possible to actually replicate "/" entirely across nodes?
00:57 twx_ you'd probably want to go for replica=2, and mount using nfs, it sounds like.
00:57 _Scotty ersatz-i: what do you mean entirely across nodes
00:57 JoeJulian Oh, bronaugh, you're on Vancouver Island?
00:57 bronaugh ersatz-i: regarding resync after node rejoins, I'm not sure.
00:57 bronaugh JoeJulian: yup.
00:57 JoeJulian Cool, Seattle area here.
00:57 bronaugh nice.
00:57 _Scotty ersatz-i: replica=2 ensures a single file is replicated on two of the nodes out of the total number of nodes you have
00:58 twx_ yes, sorry for not pointing that out
00:58 bronaugh ersatz-i: if you have n bricks, and replica=n, then that's guaranteed.
00:58 _Scotty ersatz-i: to follow on what bronaugh is saying, if you have 10 bricks, replica=2 means that any given file will exist on 2 of those 10 bricks
00:59 ersatz-i Well, if I'm trying to make a cluster of four ispconfig lamp servers, I have to make sure literally everything on the file system is kept the same on all nodes, not just certain directories, and the way ispconfig installs, it's writing everything to certain directories, if I have to mount the gluster share/fs some place else like /data I'd have to go through each PHP file for the control panel
00:59 ersatz-i and change where it installs to, etc? sorry, a bit newbish about this distributed FS stuff! :)
00:59 twx_ (but still be accessible from all of em)
00:59 bronaugh _Scotty, JoeJulian: I've been meaning to ask another question. we may have multiple bricks per node. in that case, is there any way to partition bricks into sets to ensure that your shit gets replicated to physically separate hw?
00:59 bronaugh and/or is that done by default?
00:59 twx_ ersatz-i: not familiar with ispconfig, but is it possible to use a single shared nfs mountpoint for installing it ?
00:59 twx_ across diff nodes
00:59 twx_ or at least the data that _needs_ to be shared
01:00 _Scotty ersatz-i: if you're, ah, just mounting glusterfs as a client from the server, it won't matter. you don't need *exact* replicas of all four servers; that's just wasted space.  technically.
01:00 JoeJulian bronaugh: replica sets are built in the order listed when you create the volume, so for replica 2 each pair form a replica set.
01:00 bronaugh ersatz-i: you probably don't want replica=n though... it will kill perf as n increases.
01:00 _Scotty bronaugh: yup yup
01:00 bronaugh JoeJulian: so there's explicit joining of the replica sets, then.
01:01 _Scotty ersatz-i: every increase in number kills performance because it has to make sure writes are committed on "n" nodes for replica=n
01:01 JoeJulian yes
01:01 bronaugh ok, so what happens if a replica brick disappears?
01:01 bronaugh is that where self-healing kicks in?
01:01 _Scotty bronaugh: you still have access to your data
01:01 _Scotty bronaugh: self-healing kicks in when the missing node comes back online, IIRC
01:01 ersatz-i twx_ that's a good question & what I'm trying to figure out in my head. It's a bit tough to know exactly what needs to exist replicated across nodes and what doesn't. Ispconfig does all its client related stuff in /var/www, and it's CP related stuff in SQL. So technically I could probably just replicate /var/www and use an SQL solution to replicate the SQL database. But that leaves me with
01:01 ersatz-i the issue of having to install and manage mail/configs etc on each node individually?
01:02 ersatz-i (just trying to make sure I'm understanding all of this really).
01:02 _Scotty ersatz-i: fwiw, any distributed filesystem is not your pick if you are running a database
01:02 bronaugh ersatz-i: regarding configuring machines in batches, look at puppet; or just go cloudy.
01:04 JoeJulian I run mysql using innodb on a gluster volume. One single mysql server, but should I need to fail that server over, the data's safe and immediately available.
01:04 bronaugh JoeJulian: so does glusterfs take any action to ensure data continues to be replicated at a set level?
01:04 ersatz-i I want to set-up a shared hosting env for my own personal stuff & close family, I have a re-seller account with a 'cloud vps" provider, have 8 gigs of RAM 2 TB hd space & can make as many nodes(VPS's) & load balancers as i want/need with the resources. That's what got me into researching all of this. :)
01:04 _Scotty JoeJulian: how do you handle the small file accesses inherent in MySQL?
01:05 bronaugh JoeJulian: I thought it would work somewhat differently tbh. I thought that glusterfs would simply ensure that data would be replicated twice on -some- volume.
01:05 ersatz-i (looking up cloudy & puppet)
01:05 bronaugh instead of an entire volume being mirrored.
01:05 JoeJulian bronaugh: Yes, if a brick is missing, the files that get changed are marked as dirty. When the brick returns, those attributes trigger self-heal
01:05 bronaugh ersatz-i: I was using "cloudy" as a joke way of referencing doing shit using a cloud computing service.
01:06 ersatz-i ahh. I thought you might be. :)
01:06 _Scotty lol
01:06 JoeJulian _Scotty: I don't, I use innodb's monolithic storage. All the tables are spread over 8 files (which distributes nicely).
01:06 _Scotty JoeJulian: I'm with bronaugh on that one.
01:06 _Scotty JoeJulian: ah.
01:07 JoeJulian bronaugh, _Scotty: Which is why I do replica 3.
01:07 _Scotty egads
01:07 ersatz-i Perhaps I shouldn't even be worrying about this as any VPS I create is already on a "cloud" VPS provider's cloud & completely redundant, just would be cool to make my own little cloud beneath that for learning. :)
01:07 glusterbot ersatz-i: You've given me 5 invalid commands within the last minute; I'm now ignoring you for 10 minutes.
01:07 JoeJulian hehe
01:07 _Scotty lol
01:07 ersatz-i wtf. lol
01:08 _Scotty glusterbot ftw
01:08 twx_ let's say I create a 2node setup with replica=2, can I just add node by node after that ?
01:08 twx_ and how does gluster handle it if so, while the nodes are an uneven number
01:08 _Scotty twx_: you're going to need to add two nodes at a time for that volume
01:08 twx_ and I guess we should say bricks and not nodes btw
01:08 _Scotty twx_: err, bricks
01:08 twx_ :)
01:08 ersatz-i thanks guys, looking into using NFS, as well as glusterFS manual on using replicate.
01:09 ersatz-i err "gluster".
01:09 _Scotty twx_: so if you have replica=2 or stripe=2, you need to add 2 bricks at a time to that volume
01:09 JoeJulian Yes
01:09 bronaugh JoeJulian: anyhow. so let's say a storage brick is taken down and a server monkey drops it on the floor, which screws the brick completely. you then replace the disks and bring it back up. what steps are necessary to bring the now-unreplicated volume back online?
01:09 twx_ so, followup question: can i change the replica count
01:09 JoeJulian bronaugh: gluster volume heal $vol full
01:09 twx_ from 1->2
01:09 _Scotty bronaugh: how screwed is screwed?
01:10 twx_ on an already created volume
01:10 bronaugh _Scotty: 3 disks out of a raid6 set have failed; so the data's gone.
01:10 twx_ or would I have to delete the volume and then recreate it with the new replica setting
01:10 _Scotty bronaugh: because, I keep spare chassis on hand and swap drives when a massive HW failure occurs I just swap disks.
01:10 bronaugh JoeJulian: yes, but that's necessarily after performing several other steps: presumably, removing the old replica volume, and adding the new one.
01:10 _Scotty bronaugh: aha!  well, you should follow this:
01:10 _Scotty bronaugh: http://www.gluster.org/2012/10/replac​ing-a-glusterfs-server-best-practice/
01:10 glusterbot <http://goo.gl/qKnXI> (at www.gluster.org)
01:10 JoeJulian bronaugh: As long as you mount the new formatted drive in the same place, that's it.
01:11 bronaugh JoeJulian: so gluster'll stick its identifiers back on the volume? that seems unlikely.
01:11 * JoeJulian disconnects "brick" from "server".
01:11 _Scotty bronaugh: specifically, replace-brick
01:11 bronaugh ok. "brick", not volume. but yeah.
01:11 _Scotty yup yup
01:11 JoeJulian bronaugh: I've done it many times.
01:11 bronaugh ok, just checking.
01:12 JoeJulian Not the dropping on the floor bit, but replacing drives.
01:12 _Scotty lol
01:12 bronaugh yeah I was just giving that as a pathological case.
01:12 _Scotty I prefer the BOFH method of a hammer i mean "magnetic field realignment tool"
01:12 bronaugh PSU failure causing unreliable behaviour is more likely.
01:12 ersatz-i Thank you all, completely set me on the right path @ google-fu & found a tut on ispconfig+glusterFS.
01:12 _Scotty ersatz-i: welcome!
01:12 JoeJulian I've got an old Teac 1Gig with 3 40cal holes through it.
01:13 _Scotty JoeJulian: I prefer double-ought buckshot, m'self
01:13 JoeJulian No, bullets penetrate and leave this really awesome exit wound.
01:13 bronaugh hahah
01:13 _Scotty JoeJulian: Only with hollow points.  That's charming, that is
01:14 JoeJulian If you'd ever had the occasion to support systems that used that drive, you'd have cheered.
01:14 _Scotty JoeJulian: Once ounce sabot slugs out of a 12-guage leave an interesting exit pattern.  as in, the back panel of the hard drive is missing.
01:14 _Scotty JoeJulian: no worse than deathstars.
01:14 _Scotty *one ounce
01:14 JoeJulian Yeah, they had a higher failure rate.
01:14 bronaugh Teac didn't make particularly great drives in my experience.
01:15 JoeJulian ... though at least the data could be recovered.
01:15 bronaugh I did have one drive made by ... that bloody Indian manuf, whoever it was.
01:15 bronaugh early 90s, they were making hard drives.
01:15 bronaugh QC was apparently -very- hit and miss.
01:15 JoeJulian I was working for a computer store at the time. People would bring in their dead computer, I would take it in the back room, remove the drive, then WHACK it against the counter to unstick the heads.
01:15 wushudoin joined #gluster
01:16 bronaugh ew :/
01:16 _Scotty Oh, the days of Quantum.  I remember freezing those to get the disintegrated bearings to re-form and running them long enough to copy data off.
01:16 JoeJulian +1
01:16 bronaugh modern drives have off-disk head parking ... heh.
01:16 _Scotty oh yeah, the infamous WD20EARS.
01:16 JoeJulian That teac was off-disk, but it would just get stuck there.
01:17 bronaugh huh.
01:17 _Scotty JoeJulian: the hammer is a wonderful attitude adjustment tool suitable for head parking and users.  lol
01:17 JoeJulian hehe
01:17 bronaugh done the freezer treatment on a WD800JB I owned to get it to calibrate.
01:17 bronaugh worked.
01:18 * JoeJulian is old enough that he used to get paid 8 hours to format mfm drives.
01:18 bronaugh I avoided MFM like the plague.
01:18 JoeJulian That's all their was at the time.
01:18 bronaugh almost everything I dealt with was IDE. MFM drives got the "sledgehammer in the alley" treatment.
01:18 JoeJulian Followed quickly by RLLs.
01:19 bronaugh had one 340M ESDI drive come by.
01:19 _Scotty lol!!!!
01:19 JoeJulian My first hard drive was 10M.
01:19 _Scotty oh dear lord
01:19 _Scotty this is taking me down memory lane
01:19 bronaugh I know this is a weird thing to say, but it had the most awesome spinup noise ever. sounded much like the THX sound.
01:20 JoeJulian Ah, yeah.
01:20 _Scotty lol
01:20 bronaugh first comp: 286-16, 40M WD93044-A, 1M RAM
01:20 _Scotty the old 8G seagate drives with "seashield" had that spinup too
01:20 _Scotty bronaugh: first comp: VIC-20
01:20 bronaugh yup. you're older than me. congrats :)
01:20 _Scotty lol
01:20 twx_ lol
01:21 bronaugh I was 10 when we got that computer in 1991
01:21 _Scotty NOW I need another beer
01:21 JoeJulian The vic 20 taught me that I had a unique talent and set me on my career path in 7th grade.
01:21 twx_ my first home box was a 466mhz celeron
01:21 twx_ so go ahead, have 2 beers scotty
01:21 _Scotty JoeJulian: me too.  Oh, the code I used to copy out of magazines
01:21 bronaugh I started screwing around with programming in about 1994...
01:22 bronaugh also, my lulz for the day: http://newsthump.com/2011/12/23/sal​es-of-any-old-shit-expected-to-treb​le-as-men-start-christmas-shopping/
01:22 glusterbot <http://goo.gl/o9wSo> (at newsthump.com)
01:23 JoeJulian I actually still have an original IBM PC (5150)
01:24 bronaugh by the time I was screwing around with hardware (1994 or so) I considered XTs and below to be junk
01:25 bronaugh even 286es were pretty marginal at that point.
01:27 _Scotty heh
01:27 _Scotty had 286's in the computer lab and I had a 486 at home
01:27 _Scotty 486DX actually.
01:28 bronaugh fancy... heh.
01:28 JoeJulian This isn't the Scotty that I know is it?
01:28 _Scotty JoeJulian: Who is the Scotty you know? lol
01:29 JoeJulian Actually, I know a couple whose experiences could parallel yours.
01:29 _Scotty JoeJulian: I'm at JHU, so if you're in Seattle sounds doubtful. :)
01:30 _Scotty JoeJulian: but it's a small world anyway.
01:30 JoeJulian I wouldn't put it past either Scotty that I know to come in here and mess with me.
01:30 _Scotty bronaugh: I still posit the C-64 was one of the best machines of all time.
01:30 JoeJulian But you're not him, obviously. You'd know if you were.
01:31 _Scotty lol
01:31 bronaugh I posit that I last used a C-64 in grade 3
01:31 bronaugh and then only briefly.
01:31 _Scotty JoeJulian: true.  :)
01:31 JoeJulian Actually, the Amega would pretty awesome.
01:31 JoeJulian s/would/was/
01:31 glusterbot What JoeJulian meant to say was: Actually, the Amega was pretty awesome.
01:31 * m0zes first pc was a p2-slot with 384MB of mem.
01:31 bronaugh I used a lot of Apple IIc's but I can't say I really used them in much depth.
01:31 _Scotty I loved the Amiga.  I couldn't afford it.
01:31 _Scotty Apple IIc's boot sound was the sound to beat.
01:31 JoeJulian The 68000 processors were so much faster than the Intel of the same generation.
01:32 m0zes although my brother had a c64. it was awesome and i never got to touch it.
01:32 _Scotty though I confess I set an entire server room's linux boot sound to the super mario theme song.  Rebooting the HPC grid was a thing of BEAUTY
01:32 JoeJulian lol
01:32 _Scotty JoeJulian: yup yup! they were.
01:33 bronaugh oh, the joys of playing music on the PC squeaker...
01:33 _Scotty I mean, who gets monophonic game music amid the roar of HVAC?  *I DO*
01:33 JoeJulian Hehe, this is the most chat we've had in this channel on a Friday in a long time.
01:33 _Scotty lol
01:33 _Scotty bronaugh: Hahaha!  You can get stereo in a rack row.
01:33 m0zes these are quite a bit better than my p2 was: http://ganglia.beocat.cis.ks​u.edu/?c=Beocat&amp;h=mage01
01:33 glusterbot <http://goo.gl/B7py3> (at ganglia.beocat.cis.ksu.edu)
01:33 bronaugh I remember how excited I was when I found out that a certain mod player would actually do a halfway reasonable job of playing back mods on PC squeaker.
01:33 _Scotty bronaugh: I'm just going to ask you to trust me on that.
01:34 _Scotty MOD!!!! that was the absolute shiznit
01:34 _Scotty Right along with getting a soundblaster-16 to play DOOM
01:34 bronaugh yup. had a Vibra16c
01:34 bronaugh because I'm cheap :)
01:34 * JoeJulian considers Scotty's trick just to mess with the floor techs at the colo.
01:34 _Scotty * hehe
01:35 m0zes most servers don't have speakers anymore, do they?
01:35 bronaugh JoeJulian: if you -really- want to mess with them, you should play doom rocket sounds at random locations.
01:35 JoeJulian hehe
01:35 * _Scotty thinks his trick was the best frickin' trick to play.  EVAR.
01:35 bronaugh m0zes: they most definitely do. overheat alarms, etc.
01:36 _Scotty bronaugh: ever have a tech look at a rack of boxes all showing green or blue, randomly beeping, to find the offender?  lol
01:36 bronaugh nope :)
01:36 m0zes hrm, never attempted to use them. not that you could hear *anything* in my server room when my nodes boot up.
01:36 bronaugh I don't have techs to play games with :/
01:36 _Scotty HAR!  It was pirate day.
01:36 JoeJulian Our CoLo has microphones in every rack so they could be really fun to mess with.
01:36 bronaugh HA
01:36 _Scotty m0zes: clearly, you need to have super mario blaring out of every pc speaker
01:36 _Scotty JoeJulian: want scripts?!
01:37 bronaugh JoeJulian: you need webcams and voice synthesis.
01:37 bronaugh I recommend festival.
01:37 JoeJulian 200 servers whispering at 2am could really freak someone out.
01:37 bronaugh :P
01:37 * _Scotty has his servers say "I'm watching you."
01:38 JoeJulian "kill him..."
01:38 _Scotty JoeJulian: ROFL
01:38 bronaugh LOL
01:38 * m0zes needs children laughing creepily...
01:38 JoeJulian lol
01:38 _Scotty hmm.
01:39 JoeJulian "The voices! THE VOICES!!!"
01:39 _Scotty whispered words sound believable.  children laughing is clearly digitized through the speaker.  c'mon!!! lol
01:39 * m0zes thinks this: http://www.thinkgeek.com/product/c427/ just coming out the pc speaker.
01:39 _Scotty "Was that the HVAC noise or was that... no...."
01:39 glusterbot Title: ThinkGeek :: The ThinkGeek EvilTron (at www.thinkgeek.com)
01:39 _Scotty Now, if you can combine "kill him" with all of the front LEDs going to red, that's entertainment right there, that is.
01:40 _Scotty A room full of HAL.
01:40 JoeJulian Oh, my...
01:40 _Scotty Just a thought...
01:40 _Scotty lol
01:40 _Scotty Scaring technicians is the highlight of my day, some days.  lol
01:41 bronaugh lol
01:41 _Scotty and why is it that I always get blamed when something hinky happens in the server room?  I just do not understand... lol
01:42 _Scotty End users are a different story.  I play the Montgomery Scott rule on that.  Deliver above and beyond.
01:42 _Scotty I get at least two bottles of scotch a year from the users!
01:42 JoeJulian m0zes: Those graphs look pretty nicely boring.
01:42 _Scotty Last time was The Balvenie 12 year DoubleWood
01:42 m0zes JoeJulian: yeah, but 80 *real* cores and 1TB of mem in a single box are really nice.
01:43 JoeJulian holy cow
01:43 JoeJulian You could design the space shuttle with a computer like that... ;)
01:43 _Scotty ????
01:43 _Scotty what are you packing over there, mazes?
01:44 _Scotty *m0zes.  sorry, autocorrect struck again.
01:44 bronaugh quad opteron I'm guessing?
01:45 m0zes 10-core xeon
01:45 JoeJulian Funny story, I was talking with one of the Shuttle pilots who was giving a talk about the space program and he mentioned some completely wrong processor specs so I did some quick googling and my Tegra2 processor on my cell phone has 1000 times the processing power of the Cray I.
01:46 JoeJulian The Cray I, of course, was used to model the flight characteristics of the space shuttle.
01:46 _Scotty JoeJulian: most of the processors taken into space are 2+ generations back because they have to be hardened against interstellar radiation. e.g., cosmic rays.  bit flipping sucks during retentry.
01:46 bronaugh m0zes: cute.
01:46 _Scotty *re-entry.
01:46 twx_ what does one use 80 real cores of that magnitude for in a single box
01:46 twx_ for
01:47 JoeJulian _Scotty: Yep
01:47 bronaugh huh; guess AMD doesn't yet have 20-core.
01:47 _Scotty not much I can tell ya, unless it's compute and not memory intensive.  most of my cores sit idle because everybody likes to run 120GB-of-RAM consuming jobs.
01:47 bronaugh twx_: it's a way to light money on fire to support legacy applications not written to make good use of MPI, most of the time.
01:47 twx_ kinda solid argument
01:48 _Scotty *DESPISE* Java.  esp. Java 6.  Which, in a compute grid, stinks.  Did you know that it mallocs 30% of the machine's INSTALLED RAM, just because?  It completely ignores the heap size for that malloc.
01:48 twx_ might be teh wrong way of tackling the problem with the crappy legacy app but .. :)
01:50 bronaugh I'm learning to be better about leveraging cluster @UVic
01:50 bronaugh because you can't just build your own. hard to come up with the $$$ to buy 3000 cores.
01:50 bronaugh even if you don't get to use them all.
01:50 JoeJulian left #gluster
01:50 JoeJulian joined #gluster
01:51 _Scotty bronaugh: as a seasoned cluster administrator, all i can tell you is that just do your best to please the majority of your users.  you will NOT please them all. it may hurt, but that's just the way it is...
01:51 JoeJulian Exactly... see ,,(Joe's performance metric)
01:51 glusterbot nobody complains.
01:51 _Scotty HAHAHAHAHAHAHAHAHA
01:51 _Scotty that is AWESOME
01:51 bronaugh burying them in a shallow grave has that effect, yes :)
01:52 _Scotty bronaugh: I prefer in a 50-gallon drum deep in the ocean.  Er.....
01:52 bronaugh heh.
01:53 _Scotty So back to commodore 64's!!!
01:54 _Scotty that wasn't an awkward silence at all.  lol
01:54 bronaugh meh; just looking up other stuff.
01:54 bronaugh surprised AMD hasn't done a 20-core chip yet.
01:54 _Scotty I used mine to add video titles on VHS.  lol
01:54 _Scotty meh
01:54 _Scotty more cores doesn't always buy you what you think.  there's context switching to take into account.
01:55 _Scotty I'd take more RAM and faster cores over more cores.
01:55 bronaugh depends on the workload.
01:55 _Scotty the faster jobs finish, the more you can run through.  base queuening theory.
01:55 bronaugh right now Intel's schooling them in per-chip benchmarks.
01:55 _Scotty in my case, they are high memory compute-intensive jobs.
01:55 bronaugh so it's all kind of moot.
01:55 _Scotty yeah
01:55 bronaugh Xeon E5-2620 is a great chip for the $$$
01:55 _Scotty Intel is pretty bangin' these days.  AMD does well to keep them on their toes with innovation.
01:55 bronaugh $400 for a very fast 6-core chip? sure, I'll take two.
01:56 _Scotty I ALSO like Intel's southbridge architecture.  Blows AMD out of the water.  Especially in the storage arena...
01:56 bronaugh couldn't care less about Intel's storage.
01:56 _Scotty no no
01:56 _Scotty I mean their southbridge.
01:56 _Scotty if you go with AMD architecture, you will be limited to 1GB/s.  I can guarantee this.
01:56 _Scotty Intel does not have that limitation.
01:56 bronaugh 1GB/sec what?
01:56 bronaugh per..?
01:57 _Scotty total HDD throughput.  It's a PCI limitation with their architecture.
01:57 bronaugh most of the boards I've played with (Intel boards) you're limited to 500M/sec for all 6 sata ports aggregate.
01:57 _Scotty I have two identical supermicro systems, except one is running Intel and the other an AMD chipset.  The Intel chipset smokes AMD.  By a factor of almost 4:1
01:58 _Scotty what boards are you using???
01:58 bronaugh talking about older stuff, mostly.
01:58 bronaugh pre-i7
01:58 _Scotty I'm getting almost 1.8G/s out of an Intel E3-1200v2
01:58 _Scotty ah okey
01:59 _Scotty well, technically, I'm getting closer to 4GB/s
01:59 _Scotty but that's a benchmarking issue, methinks.
01:59 bronaugh probably
01:59 bronaugh some software has some pretty ridiculous limits.
01:59 _Scotty realized throughput out of link-agged 10Gb is 1.8GB/s.
02:00 bronaugh yeah we pull more than that over QDR infiniband.
02:00 bronaugh over TCP/IP
02:00 bronaugh you need multiple threads to pull off more than ~1.2GB/sec but yeah.
02:00 _Scotty If I run two 10Gb controllers off the two PCI busses, I get closer to 4GB/s
02:00 bronaugh you're talking aggregate of both uplink and downlink I presume.
02:01 _Scotty yeah
02:01 _Scotty but it's still real, demonstrated performance
02:01 _Scotty and again, with AMD it blows at 1GB/s.
02:01 _Scotty I tried every config I could think of, talked to engineers... oh, nada.
02:01 _Scotty I start talking to Intel, and they say try one of these boards.... and.. well, for storage I haven't looked back.
02:02 bronaugh ... huh.
02:02 _Scotty For raw compute, i.e. most cores for most RAM, I use AMD.
02:02 bronaugh surprised that the IO sucks that hard.
02:02 _Scotty but at least in my environment computing has changed.  They need more RAM per core, so the next buy I'm going back to Intel for that.
02:03 _Scotty yeah AMD's IO really, really stinks.  It really surprised me.  But if you look at AMD they covered it up pretty well.
02:03 _Scotty Even squashed reports on it.
02:03 twx_ feels like intel been pulling further and further away from amd
02:03 _Scotty They knew it.
02:03 twx_ lately
02:03 bronaugh yeah; we've only bought Intel for our servers.
02:03 bronaugh and recently bought a crapload more desktop hw... all Intel.
02:03 _Scotty coolbeans
02:03 bronaugh there was a much stronger argument for AMD on the desktop side.
02:03 bronaugh (A10 stuff is actually pretty decent)
02:04 bronaugh but in the end, driver quality matters, and Intel seems to have that sorted out.
02:04 twx_ dont they use a shitload of power ?
02:04 _Scotty define shitload of power
02:04 _Scotty no more than AMD in my experience
02:04 twx_ i meant the other way around, regarding the a10's
02:05 bronaugh actually, yes. they would have.
02:05 _Scotty i've got servers with dual-redundant 1100W power supplies drawing 700 watts combined.  AMD & Intel in the server arena consume around the same power.
02:05 bronaugh I didn't really look up any benchmarks.
02:05 twx_ was looking around on something similar few months back cant remember the specs but I think the amd cpus was something like 120W idle and intel more like 65-70
02:05 bronaugh but Ivy Britches seems to be considerably better on power consumption.
02:05 twx_ sorry not idle but avg
02:05 bronaugh http://www.techspot.com/review​/580-amd-a10-5800k/page8.html
02:05 glusterbot <http://goo.gl/VFoNF> (at www.techspot.com)
02:05 bronaugh the idle is similar for both
02:06 bronaugh power consumption under load is nearly double.
02:06 _Scotty ah.
02:06 _Scotty well
02:06 _Scotty I can tell you, it depends on the motherboard design as well
02:06 _Scotty as for instance
02:06 _Scotty SAME CPU and config in a SuperMicro and Dell server... the SuperMicro draws 30% more power.
02:07 _Scotty and that adds up over a year.
02:07 bronaugh though to be fair, the GPU in the A10 is nearly twice as fast as well.
02:07 bronaugh _Scotty: that's odd. which chassis, which board?
02:07 twx_ same psu config too? :)
02:08 _Scotty bronaugh: Dell PowerEdge R815 vs SuperMicro 2042
02:08 bronaugh ok, I'll revise my statement regarding A10 GPU. it's not nearly twice as fast -- it's more than twice as fast.
02:08 _Scotty well, the supermicros come with 1400 watt power supplies, the dells with 1100.
02:08 _Scotty but if it's the same processor i'd expect the same power draw.
02:08 _Scotty such is not the case.
02:08 _Scotty same ram, same 10Gb card, everything.
02:08 _Scotty supermicro drew 30% more.
02:08 bronaugh how much average power?
02:09 _Scotty 910 watts versus 700.
02:09 bronaugh hm.
02:09 _Scotty full bore.
02:09 _Scotty same hard drives.
02:09 _Scotty same everything.
02:09 bronaugh and benchmark results were the same?
02:09 _Scotty yup!
02:09 bronaugh ok. just checking :)
02:09 _Scotty same power conservation settings, too. :)
02:09 _Scotty as in, NO power conservation.  full bore.
02:10 bronaugh good to hear you controlled for confounding vars. too few people do.
02:10 _Scotty yeah, i've found that :D
02:10 bronaugh that PSU should be more than 90% efficient.
02:10 _Scotty meh
02:10 bronaugh the Dell clearly can't be more than 100% efficient...
02:11 _Scotty SuperMicro is 80+ gold
02:11 bronaugh so either a) the test report is a lie.
02:11 bronaugh or b) the motherboard's power control circuitry sucks.
02:11 _Scotty bronaugh: leaning towards b)
02:11 bronaugh could be either.
02:11 bronaugh we're running those 1400W PSUs
02:11 bronaugh they aren't very good.
02:11 _Scotty well
02:12 _Scotty i think it's a PSU design issue
02:12 bronaugh ie: we've had at least one of (2 PSUs + PDU) fail to filter power adequately, resulting in flaky behaviour.
02:12 _Scotty the supermicro PSU's haven't been very good to me.  I routinely replace one or two every 2 months
02:12 _Scotty the dell's I haven't had to change, yet.  knock on wood.
02:12 _Scotty and supermicro's no-advance-replacement policy BLOWS
02:13 _Scotty Dell doesn't give me that kind of grief
02:13 _Scotty and it's the same price per server
02:13 bronaugh yeah, it's not for us.
02:13 _Scotty and Dell gave me 3 year next day on-site, so... well, it was kind of a no-brainer to start buying more dell
02:13 bronaugh the last time we tried to get Dell to give us a quote, they gold-plated it and came in at twice the price as us building a Supermicro box.
02:13 bronaugh so we didn't call them back.
02:14 _Scotty ah.
02:14 bronaugh plus, last I checked they don't have any really entertaining storage chassis
02:14 _Scotty well, my university buys a lot of dell so they give us a significant cost reduction, in line with supermicro
02:14 bronaugh Supermicro has the 36-disk beast.
02:14 bronaugh yeah uni here too, but cost reductions seem to be pretty pathetic.
02:15 _Scotty not really.  though Dell does have the 4U-60 drive beast.
02:15 _Scotty powervault 3260 i think??
02:15 bronaugh 60 drives in 4U? how are they pulling that off?
02:15 _Scotty lol
02:15 _Scotty TRAYS
02:15 _Scotty you literally disconnect an entire tray of disk to replace one drive
02:15 bronaugh jesus.
02:15 _Scotty I said, cool, but F- that
02:15 bronaugh that's worse than Backblaze.
02:15 _Scotty Check out Quanta
02:16 _Scotty they have a 4U 60-drive beast that slides completely out and all drives are top loading
02:16 _Scotty they also have a really cool 12-drive 1U unit.  that's what I'm evaulating.
02:16 bronaugh ... huh.
02:16 _Scotty *evaluating
02:16 _Scotty quantaqct.com
02:17 bronaugh wow.
02:17 bronaugh I'm actually impressed by that.
02:17 _Scotty yeah... it's impressive
02:17 _Scotty BUT
02:17 _Scotty you still have to weigh attaching frontends to that disk
02:17 _Scotty so
02:17 _Scotty in my case
02:17 _Scotty it's more performant and cost-efficient for the 1U 12-drive nodes
02:17 _Scotty particularly for gluster
02:18 bronaugh hmm
02:18 _Scotty think about it.  60 drives on a 6Gb backplane.  you're going to saturate the bus.  PCI can only pump so much
02:18 bronaugh yeah we've been buying Supermicro SC847A
02:18 bronaugh 36-disk 4U chassis w/2U space for server.
02:18 _Scotty fine for storage-centric solutions, but not performance
02:18 _Scotty you want performance + storage, 12-drive 1U is the boat to set sail on
02:19 bronaugh ... we called it "medusa" for a reason. 9 SFF8087 cables does give it that appearance.
02:19 _Scotty LOL
02:19 _Scotty true enough
02:19 _Scotty so
02:19 _Scotty I use the SC847 with a 45-drive jbod myself
02:19 bronaugh $1500/chassis
02:19 _Scotty but performance blows
02:19 bronaugh cheap :)
02:19 bronaugh good to know that the JBOD variant sucks.
02:19 _Scotty yup, cheap.  for a reason ;)
02:19 bronaugh are you using -E26?
02:19 _Scotty I don't recall
02:19 bronaugh ok, they've got 2-3 variants of the 45-disk JBOD
02:20 bronaugh I'm sure each has a different SAS expander.
02:20 _Scotty probably
02:20 bronaugh what kind of cabling connects the box up?
02:20 _Scotty sas cabling.  to an external lsi 9211.
02:21 bronaugh how many lanes?
02:21 _Scotty IIRC 2 lanes.  4 lanes sucked worse than 2.
02:21 bronaugh weird.
02:21 _Scotty yeah
02:21 _Scotty their internal wiring diagram blew too.
02:21 _Scotty couldn't make it simple, oh no.
02:21 bronaugh so 8 ports worth of cables then. so that should pull 6*8=~5GB/sec
02:21 bronaugh if it doesn't pull that, the expanders suck.
02:22 _Scotty no here I am with a freakin guide made by somebody in IKEA translated to a foreign language back to english.
02:22 bronaugh 600*8 rather
02:22 _Scotty oh hell no
02:22 _Scotty 1GB.
02:22 _Scotty and that's sequential data.
02:22 bronaugh ok so shitty expanders.
02:22 bronaugh very shitty.
02:22 _Scotty yup
02:22 bronaugh good to know.
02:22 _Scotty coupled with an AMD board
02:22 _Scotty and you're hosed.
02:22 bronaugh did you test with an Intel board?
02:23 bronaugh ie: maybe this is an AMD bottleneck.
02:23 _Scotty No, I was given a box of parts and told to build a 300TB array.
02:23 bronaugh want to rule that out.
02:23 _Scotty everything was purchased before i started.
02:23 bronaugh ok. so it's actually -likely- that the botteneck was the board.
02:23 _Scotty it was just sitting nice and boxed up
02:23 _Scotty i'd say *maybe*
02:23 bronaugh it lines up pretty well
02:23 _Scotty it bothers me that SuperMicro would try to sell AMD for storage
02:23 _Scotty knowing it blows
02:23 _Scotty so my question back to that is, what else are they covering up
02:23 bronaugh Supermicro is just another Asian supplier. rule there is "caveat emptor"
02:24 _Scotty i'd say it's as likely they have a shitty expander, and the AMD chipset masked it
02:24 bronaugh you go over what they're selling with a fine toothed comb and cherry pick the good products.
02:24 _Scotty lol I wasn't the emptor in this case.  lol
02:24 bronaugh yup.
02:24 bronaugh any chance you can test on Intel?
02:25 bronaugh (I totally understand if you can't)
02:25 _Scotty negative, i've ditched supermicro for pretty much any new equipment purchases.  i will never buy one of those things again if i can help it.  even for the compute nodes that were again purchased before me, we have a 15% failure rate
02:25 _Scotty Dell is sitting at a big fat 0% for 5 years.
02:25 bronaugh damn :/ that sucks.
02:25 bronaugh it also lines up with what we've seen, sadly.
02:26 _Scotty cheap for a reason, says I
02:26 _Scotty most likely they re-sell whatever didn't pass QC as an ODM for other vendors.
02:26 bronaugh so over how many years is that 15% failure rate
02:27 _Scotty 18 months.
02:27 bronaugh ok.
02:30 _Scotty so you're seeing that kind of failure rate too?
02:30 _Scotty makes me feel not as bad, i suppose, those a little upset at supermicro for letting that kind of crap out of the factory
02:30 _Scotty *tho
02:32 bronaugh don't have a large enough sample size.
02:32 bronaugh I -can- say that their older 900W PSU design seems to be robust.
02:32 _Scotty Quanta's S100 12U 1U server is the bee's knees.
02:32 bronaugh at least from our experience with a limited sample size.
02:32 _Scotty yeah, all we have are their 1400W designs
02:32 bronaugh basically our n is too small to draw any significant conclusions.
02:33 bronaugh how have you had PSUs fail?
02:33 bronaugh what was the syndrome?
02:33 _Scotty uh, randomly going to amber light instead of green.  with and without smoking.
02:34 _Scotty and, 25% of the time, they smoke motherboard caps.
02:34 bronaugh sketchy.
02:34 _Scotty nothing like wandering into a server room smelling of burnt phenolic.
02:35 bronaugh did they always nail machines as they went down, or did the redundancy sometimes save you from that?
02:35 _Scotty redundancy saved me 75% of the time.
02:35 _Scotty but, um, not what I had in mind for a redundant power supply.  just sayin'
02:35 bronaugh well; that's a good statistic.
02:35 bronaugh and no. it's not.
02:35 _Scotty if it decides to shit itself i'd expect it not to fry my motherboard.
02:36 _Scotty ya know, something like diodes and VRs and the like to keep that from happening.
02:37 bronaugh there's a lot of ways to fry shit :/
02:37 bronaugh that's the problem.
02:37 _Scotty but, typically the $500 part will fry itself to save the 50c part.
02:38 bronaugh yeah. undervoltage can kill shit too :?
02:38 _Scotty in either case, I'd have more fun shagging a cactus.
02:38 bronaugh overvoltage is the obvious bad case.
02:38 _Scotty lol
02:39 _Scotty well, undervoltage can fry shit IF it's allowed to.  typically fuses kick in at that point because it draws to much current.  OR, the device itself will stop functioning with insufficient driving voltage.
02:39 _Scotty *too
02:39 bronaugh yes; the VRM can pull a lot of clever
02:39 bronaugh one of those clever things being "shutting down".
02:39 _Scotty lol like old Pentiums issuing "halt" on overtemp
02:39 _Scotty ah yes
02:40 bronaugh guessing caps around CPU failed?
02:40 bronaugh trying to diagnose which voltage rail went awry here.
02:41 _Scotty yup yup
02:41 bronaugh because with those PSUs, it's one big god damned blade connector delivering 110A or so of +12V
02:41 bronaugh the PDU does the step down.
02:41 bronaugh (so it's a single point of failure... good job guys)
02:41 _Scotty YUP
02:41 _Scotty something mysteriously (or not so) missing from the Dell PSUs
02:41 bronaugh but it sounds like PSU just let massive overvoltage through.
02:42 _Scotty yeah.
02:42 _Scotty it was a spike that smoked it roundly.
02:42 bronaugh probably wouldn't take that much overvoltage either.
02:42 bronaugh likely 14V would do it.
02:42 _Scotty i should say, that the entire server room gets finely conditioned voltage (within .1V) from an oversized UPS
02:43 _Scotty so if it were an input voltage issue, my server room should have gone down in a blaze of glory
02:43 _Scotty such was not the case.
02:43 _Scotty and metrics from the UPS and other PSU's did not indicate issues
02:43 _Scotty it's a fundamental supermicro design issue in my opinion
02:43 bronaugh I'd say that's actually likely.
02:44 bronaugh but also likely limited to that PSU design.
02:44 bronaugh ie: it's particularly shitty.
02:44 _Scotty perhaps, i'd agree on that.
02:44 _Scotty yup
02:44 bronaugh anyhow, gotta run; but it's been great chatting, and thanks for all the info on Supermicro stuff. very good to have.
02:45 _Scotty sure thing!  thanks for your input too!
02:45 _Scotty cheers, mate.
02:45 _Scotty and Happy Holidays!  Merry Christmas, and the like.
02:45 bronaugh thanks :
02:45 bronaugh :)
03:41 _Scotty left #gluster
04:38 bala joined #gluster
04:44 bauruine joined #gluster
05:19 y4m4 joined #gluster
05:26 mooperd joined #gluster
05:49 berend joined #gluster
05:58 bala joined #gluster
07:27 nightwalk joined #gluster
08:43 mooperd joined #gluster
08:45 weplsjmas left #gluster
10:02 ccorsani joined #gluster
10:38 milos_ joined #gluster
11:57 DaveS joined #gluster
12:33 mooperd joined #gluster
12:36 Humble joined #gluster
13:15 raven-np joined #gluster
14:43 chirino joined #gluster
15:29 robo joined #gluster
15:59 juhaj joined #gluster
15:59 juhaj I thought I'd create a nice redundant filesystem by using two glusterfs bricks at geographically separated locations, but it turns out the filesystem is horribly slow. Apparently because glusterfs always contacs the distant server instead of the closer one.
15:59 juhaj Is there anything that can be done about it?
16:03 m0zes juhaj: it is by design. the client does the replication it writes and does integrity checks on both. 3.4 *should* (afaics) have multi-master georeplication that might be useful for you when it comes out.
16:08 juhaj Hm... could geo-replication be used even now or is it one-way only?
16:08 juhaj Actually, even that might be acceptable ... perhaps
16:11 chirino joined #gluster
16:22 juhaj Hm... there's something wrong with that glusterfs anyway: before I asked here, I tried mounting it from the other server and now I notice that one file gives: "ls: cannot access <file>: Input/output error" when mounted from that server but works ok from the other!
16:22 juhaj Any idea what's wrong?
17:02 robo joined #gluster
18:10 robo joined #gluster
18:44 mooperd joined #gluster
19:33 _Scotty joined #gluster
19:34 _Scotty 'ello all
19:38 _Scotty anyone have experience with directories containing a large number (~1 million) of files, and the performance with a gluster fuse client doing an 'ls'?  Specifically, how long it'll take?
19:39 _Scotty I recall reading something that the client keeps the state of the filesystem, so performing directory listings on a filesystem containing millions of files can be slow.  Is that true?
19:43 PaulC2 joined #gluster
19:44 H__ it will be slow, yes
19:45 H__ traversing 1M directories on my production boxen takes 2 days.
19:53 _Scotty just doing an ls takes 2 days??
19:54 H__ a find
19:54 _Scotty ah ok.
19:55 _Scotty i'm moving from NFS to gluster, and my filesystem contains around 1b files.  there are a few directories containing around a million files, most only have a few thousand.
19:56 _Scotty i didn't think a find would be greatly improved.
19:56 _Scotty i was more concerned with end users doing an 'ls' and how quickly it would return.  I'd want performance to be the same or faster than what i have now.
19:58 H__ what do you have now ?
19:59 _Scotty two opensolaris servers, and zfs for the block storage.  as soon as my demo hardware arrives i'll be evaluating centos 6.3, zfs on linux, and gluster for the distributed filesystem
20:00 _Scotty i've already posted the gluster + zfs article on gluster.org.  It works fine in a virtual env.
20:00 H__ i think zfs will outperform gluster, but then again zfs is not a distributed filesystem
20:00 _Scotty but when my demo hw arrives i'll be testing it with real data.
20:01 H__ i'm very interested in the results you find :)  (and i assume others here too)
20:05 _Scotty There has definitely been a lot of interest already :-)  A few other folks have made blog posts about it, but I'm doing this for a production env.
20:05 _Scotty everything in my limited testing has gone extremely well.
20:05 cicero my experience with volume of replica 2 is that it's significantly slower than an nfs ls, when no. files in a dir are large
20:06 _Scotty cicero: thanks.
20:06 semiosis that's everyone's experience unfortunately
20:06 semiosis its probably even worse with ls -l than just ls
20:06 cicero yeah
20:06 _Scotty I'm planning on setting replica=2 for the home directory volume, but that's going to be a very very small volume.  10TB or so.
20:06 _Scotty all other volumes will be straight distribute
20:06 * semiosis &
20:07 _Scotty and on the zfs backing store, it will be all RAID10
20:12 _Scotty does anyone know if tiered storage is still on the roadmap?
20:15 samppah that subject pops up every now and then but haven't heard anything about it for quite long time
20:15 samppah it's not in for 3.4 at least
20:15 samppah http://www.gluster.org/community/d​ocumentation/index.php/Features34
20:15 glusterbot <http://goo.gl/4MvOh> (at www.gluster.org)
20:16 samppah it may be easier to handle that in filesystem or using raid controller or something like flashcache
20:18 _Scotty well, i was thinking of just creating a volume containing only slower, old storage bricks and periodically moving files to that volume.
20:18 _Scotty it would just be nice if it were transparent, is all
20:18 samppah that's true
20:52 tryggvil joined #gluster
21:17 robo joined #gluster
21:20 juhaj Any idea why one file on gluster gives input/output errors when the fs is mounted from server A but not when mounted from server B? Obviously, the fs is replicated across A and B
21:29 _Scotty what does the log file show
21:29 _Scotty look under /var/log/gluster
21:31 _Scotty ouch, just got bit by zfs on linux for the first time by a nice year-old bug.  Apparently deleting directories containing files with xattrs causes the zfs_iput_taskq process to use 100% CPU.  Oh, and the storage servers slow to a crawl.
21:31 _Scotty 5 minutes to unzip the linux 2.4 kernel tarball on Gluster, 30 minutes+ to delete it.  youch.
21:32 _Scotty not that it matters much, I suppose - users don't generally delete data and clean up after themselves.  lol
21:33 H__ heheh :)
21:34 _Scotty so, just for kicks and grins, I'm going to try gluster on freebsd to see how it fares.
21:34 H__ gluster uses xattr heavily, so that bug needs fixing if you want to use zfs as native host fs
21:34 _Scotty H__: indeed.
21:35 H__ which zfs implementation are you using exactly ?
21:35 _Scotty centos 6.3, zfs on linux rc12
21:35 _Scotty i have yet to get rc13 to build correctly.
21:36 H__ i mean which zfs code base, which zfs roject
21:36 H__ (i've only used zfs on freebsd)
21:37 _Scotty H__: www.zfsonlinux.org
21:38 _Scotty H__: Yeah, freebsd is nice bc zfs is built into the kernel
21:38 _Scotty H__: I've used ZFSGuru a lot
21:40 H__ do you know which zfs version is zfsonlinux.org using ?
21:43 _Scotty v28
21:44 H__ ah, good
21:45 _Scotty v28 - 5
21:47 duerF joined #gluster
21:57 paulc2 joined #gluster
22:00 Staples84 joined #gluster
22:22 paulc2 joined #gluster
22:26 paulc2 joined #gluster
22:27 paulc2 joined #gluster
23:06 jabrcx joined #gluster
23:14 raven-np joined #gluster
23:15 jabrcx Hello.  Just swung by for some piece of mind.  When I do "gluster volume delete MYVOLUME" (after stopping it), all the data in my underlying xfs fileystems should be left intact, right?
23:16 tryggvil joined #gluster
23:17 jabrcx hah, s/piece/peace/, tis the season
23:26 _Scotty jabrcx: that's correct
23:38 jabrcx _Scotty: thanks.  I'm extra paranoid, and will probably set xfs read-only between stopping and deleting.  Hopefully it won't interfere with the deletion (I'm guessing gluster leaves behind its metadata in all the xattrs anyways)
23:39 _Scotty jabrcx: you can set xfs to read-only but it's not necessary.  yes, gluster only stores xattrs on the files and directories; it doesn't remove any files from the underlying store when you delete a gluster volume.
23:42 jabrcx _Scotty: glad to hear it.  Very nice feature of gluster, being able to add/remove at will like this.  Thanks again.
23:42 _Scotty yw
23:53 chirino joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary