Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-07

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:24 jiqiren joined #gluster
00:46 glusterbot New news from newglusterbugs: [Bug 870256] Samba "store dos attributes" feature doesn't work with GlusterFS. <http://goo.gl/aJxru>
01:14 samppah joined #gluster
01:29 Bullardo joined #gluster
02:04 Bullardo joined #gluster
02:25 sunus joined #gluster
02:29 ika2810 joined #gluster
02:30 pranithk joined #gluster
02:37 pranithk joined #gluster
02:42 ika2810 left #gluster
02:44 UnixDev is it recommend not to mount localhost:/share through nfs on a brick?
02:44 UnixDev what about through fuse?
03:03 bharata joined #gluster
03:18 bharata joined #gluster
03:25 shylesh joined #gluster
03:25 shylesh_ joined #gluster
03:37 ika2810 joined #gluster
03:44 Bullardo joined #gluster
03:45 m0zes joined #gluster
04:33 sunus hi, can someone donwload this qemu-glusterfs video for me? my connection sucks.. http://linuxfoundation.ubica​st.tv/videos/permalink/254/
04:33 glusterbot <http://goo.gl/wgS79> (at linuxfoundation.ubicast.tv)
04:36 vshastry joined #gluster
04:47 mdarade1 left #gluster
04:55 ngoswami joined #gluster
05:05 mdarade1 joined #gluster
05:06 kshlm joined #gluster
05:06 kshlm joined #gluster
05:12 ramkrsna joined #gluster
05:18 faizan joined #gluster
05:31 bala joined #gluster
05:34 bulde joined #gluster
05:43 bala joined #gluster
05:45 sunus joined #gluster
05:52 pranithk joined #gluster
05:54 guigui1 joined #gluster
05:55 jays joined #gluster
06:06 sripathi joined #gluster
06:22 rgustafs joined #gluster
06:45 raghu joined #gluster
07:03 ankit9 joined #gluster
07:08 sripathi joined #gluster
07:30 pkoro joined #gluster
07:32 lkoranda joined #gluster
07:34 pkoro Hi everyone, we are experiencing the following situation on one of our glutser servers. The server serves nfs beyond cluster and the nfs.log file (within /var/log/glusterfs) quickly takes up all available disk space on the node. Any idea what we may do to reduce logging of ifs service?
07:35 pkoro version and release of gluster server are 3.3.0-1.el6
07:35 pkoro os is SL6.1
07:39 ankit9 joined #gluster
07:40 JoeJulian pkoro: Assuming there's some sort of error that's causing that to fill up, fix the error. If you're not using nfs, turn it off. Decrease the client-log-level (not really recommended, but it's still a possibility). Also 3.3.1 is a bit safer. See the ,,(yum repo).
07:40 glusterbot pkoro: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
07:43 pkoro JoeJulian: Well there is no error with the service and we need the nfs for some old OS clients . I will try the client-log-level as you suggest thanks! Btw where can I find documentation on the upgrade procedure you suggest?
07:46 JoeJulian Not sure if there is any. Yum upgrade glusterfs-server. gluster volume heal $vol info (wait for it to be empty) then do the next server.
07:47 JoeJulian After all the servers are done, upgrade the clients. You'll have to unmount and mount to use the new version.
07:53 vshastry joined #gluster
08:00 Nr18 joined #gluster
08:01 shireesh joined #gluster
08:02 hagarth joined #gluster
08:09 andreask joined #gluster
08:17 glusterbot New news from newglusterbugs: [Bug 872703] sticky-pointer with no trusted.dht.linkto after a replace-brick commit force, heal full migration <http://goo.gl/m3MHd>
08:20 mohankumar joined #gluster
08:23 badone joined #gluster
08:24 vshastry joined #gluster
08:25 TheHaven joined #gluster
08:25 ekuric joined #gluster
08:26 badone joined #gluster
08:31 dobber joined #gluster
08:36 bulde1 joined #gluster
08:43 bulde joined #gluster
08:47 sshaaf joined #gluster
09:00 vshastry joined #gluster
09:00 Triade joined #gluster
09:22 puebele1 joined #gluster
09:28 webwurst joined #gluster
09:38 SteveCooling NFS client seems to be much happier to cache some things than the glusterfs client. What's up with that?
09:42 rgustafs SteveCooling, That is correct, I heard there where plans in introducing more caching to the glusterfs client
09:42 rgustafs SteveCooling, When are you experiencing it?
09:43 hagarth joined #gluster
09:44 DaveS_ joined #gluster
09:54 puebele joined #gluster
10:03 ctria joined #gluster
10:10 sripathi joined #gluster
10:16 overclk joined #gluster
10:24 guigui1 joined #gluster
10:27 tryggvil joined #gluster
10:30 atrius joined #gluster
10:47 faizan joined #gluster
10:59 Tarok joined #gluster
11:15 SteveCooling rgustafs: read performance is dramatically better. 4 parallel threads reading 4000 files of 100K in random order
11:16 SteveCooling but i'm going to go LACP on the nodes and client to get 2gbps network speeds. hope that will help.
11:17 rgustafs SteveCooling, I'm no expert but I heard about this before. I'm sure someone else here is more qualified to help you
11:45 raghu` joined #gluster
11:52 mm_bzf joined #gluster
11:52 hagarth joined #gluster
11:56 _ilbot joined #gluster
11:56 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
12:05 balunasj joined #gluster
12:24 gbrand_ joined #gluster
12:26 mdarade1 left #gluster
12:37 Psi-Jack Hmmm
12:37 Psi-Jack GlusterD itself provides the NFS server, correct? I shouldn't need to do anything else?
12:38 Psi-Jack Besides have the nfs-client installed on the clients, to use nfs?
12:38 jays left #gluster
12:44 samppah Psi-Jack: correct.. it should work with tcp and nfs version 3
12:44 kkeithley glusterd is just the control path. When you start (export) a volume, you'll get a glusterfsd — the native/fuse server, and a glusterfs — the nfs server.
12:44 Psi-Jack Hmmm. All I'm getting is mount system call failed, trying to mount it via NFS/.
12:45 kkeithley do you have an iptables firewall?
12:45 Psi-Jack No.
12:45 kkeithley what's the client
12:45 kkeithley ?
12:45 Psi-Jack Ubuntu 10.04, server, Ubuntu 10.04
12:45 Psi-Jack nfs-client, standard stuff for the client.
12:46 kkeithley mount -o proto=tcp,vers=3?
12:46 Psi-Jack Yep
12:46 kkeithley and the glusterfs process is running on the server?
12:46 Psi-Jack Even trying port= what the port's showing on the gluster volume status
12:46 Psi-Jack Yes
12:47 Psi-Jack NFS Server on localhost38467Y9449
12:48 glusterbot New news from resolvedglusterbugs: [Bug 828701] rename failures with multiple clients <http://goo.gl/0EYL2>
12:48 ndevos do you have glusterd running on the nfs-client? if so, you may want to try adding "nolock" to the mount options
12:48 Psi-Jack showmount --all just says, RPC: Unable to receive.
12:48 Psi-Jack ndevos: No, I don't.
12:49 Psi-Jack Storage is not on the clients. ;)
12:49 ndevos well, you wouldnt be the first :)
12:49 Psi-Jack heh
12:49 ndevos so, if showmount shows an error, you may not have portmapper/rpcbind running?
12:49 ndevos does rpcinfo show some useful output?
12:50 Psi-Jack portmap is running, rpc.statd is running.
12:50 ndevos on server and client?
12:50 Psi-Jack Yep, rpcinfo shows running RPC.
12:50 Psi-Jack But, not on the server side no.
12:51 Psi-Jack Client, yes, server, no.
12:51 kkeithley needs to be on the server
12:51 Psi-Jack What needs to be on the server?
12:51 ndevos portmap (or rpcbind)
12:51 Psi-Jack Kinda right back where we started with this. I asked this earlier. ;)
12:51 kkeithley rpc.{mountd,nfs,nlockmgr}
12:52 Psi-Jack So, nfs-common
12:52 * ndevos doesnt know ubuntu
12:52 kkeithley I wasn't on-line until 20 min ago
12:53 kkeithley gluster's nfs provides those. You should see those in an rpcinfo
12:53 Psi-Jack Hehe
12:54 kkeithley on the server
12:54 Psi-Jack kkeithley: Interesting, because, I don't. Unless I install nfs-common.
12:54 kkeithley rpcbind installed?
12:54 Psi-Jack I just get connection refused.
12:54 kkeithley and running?
12:54 Psi-Jack That's part of nfs-common
12:54 kkeithley and you're sure about the iptables firewall?
12:54 Psi-Jack 100%, all our firewalls are Cisco.
12:55 kkeithley on your ubuntu machine. I'm pretty sure — like Fedora — the iptables firewall is enabled by default.
12:55 khushildep joined #gluster
12:55 Psi-Jack It's not enabled by default, ever.
12:57 Psi-Jack Aha
12:57 ndevos after installing rpcbind, the glusterfs process that is the nfs-server should be restarted so that it registers itself in the portmaper
12:57 Psi-Jack Now that I installed nfs-common AND restarted glusterd, it's working.
12:58 Psi-Jack ndevos: Heh, yeah, just found out, just before you mentioned that. ;)
12:58 ndevos you should now also see a glusterfs process that writes an nfs.log
12:58 Psi-Jack So, the answer to my original question is: No, you need to install portmap. ;)
12:59 ndevos ah, 13:37 < Psi-Jack> GlusterD itself provides the NFS server, correct? I shouldn't need to do anything else?
13:00 kkeithley interesting that the Ubuntu .debs don't have a dependency on that
13:01 Psi-Jack Yeah.. semiosis is to blame. ;)
13:01 kkeithley Fedora rpms don't either
13:01 Psi-Jack Should at least be a suggests. ;)
13:02 kkeithley and by extension, neither do the debian .debs.
13:02 ekuric joined #gluster
13:03 manik joined #gluster
13:16 lh joined #gluster
13:16 lh joined #gluster
13:20 manik joined #gluster
13:24 mohankumar joined #gluster
13:28 Psi-Jack Hmmm okay yeah.. NFS methods were 2s slower than GlusterFS methods. :/
13:30 UnixDev_ joined #gluster
13:33 rgustafs joined #gluster
13:37 HeMan joined #gluster
13:37 ramkrsna joined #gluster
13:44 gbrand_ joined #gluster
13:44 FU5T joined #gluster
13:49 guigui1 joined #gluster
14:05 ramkrsna joined #gluster
14:06 bennyturns joined #gluster
14:06 duerF joined #gluster
14:06 duerF w
14:09 plarsen joined #gluster
14:12 Psi-Jack ...
14:12 Psi-Jack Wow, semiosis. You completely killed the lucid repo?
14:24 edward1 joined #gluster
14:38 21WAANUMM joined #gluster
14:55 manik joined #gluster
14:56 puebele joined #gluster
14:56 manik joined #gluster
15:07 stopbit joined #gluster
15:17 UnixDev_ left #gluster
15:17 UnixDev_ joined #gluster
15:20 kevein joined #gluster
15:25 dstywho joined #gluster
15:26 wushudoin joined #gluster
15:27 plarsen joined #gluster
15:29 deckid joined #gluster
15:31 dstywho joined #gluster
15:34 semiosis Psi-Jack: lucid is a release not a repo... what do you mean?
15:35 Psi-Jack Depecated the upstarted-* repos to ubuntu-* and don't have a lucid branch within it.
15:36 semiosis yep
15:38 semiosis hmm
15:40 Psi-Jack Considering that glusterfs's own download site has no deb packages for it, and the README points to your repos, I'd suggest, personally, to not drop lucid, until LTS is up.
15:40 Psi-Jack Otherwise, you're slapping a whole bunch of people in the face.
15:42 semiosis btw, the glusterfs-server package does suggest nfs-common
15:43 Psi-Jack hmm, odd, it does... aptitude's supposed to default to install suggested packages...
15:47 semiosis i suppose glusterfs-server could depend on rpcbind
15:47 semiosis might be a good idea
15:47 ndevos or portmap, depending on the distribution...
15:48 semiosis ndevos: talking about my ubuntu packages specifically
15:48 ndevos ah, no idea what they use there :)
15:49 semiosis but in general, are you +1-ing the idea of having glusterfs' server package depend on the distro's rpcbind/portmap?  seems reasonable
15:49 ndevos yes, the nfs servers is started by default, so imho it should be working by default as well
15:50 ika2810 joined #gluster
15:50 semiosis well when you put it that way
15:53 semiosis Psi-Jack: re: supporting lucid for as long as LTS... this is not an ubuntu Main package, and I don't work for Canonical, so I'm not inclined to necessarily support lucid
15:54 semiosis Psi-Jack: there's a MIR for glusterfs though, idk what the status is lately -- last time i checked it seemed "stalled" imo
15:55 manik joined #gluster
15:57 semiosis kkeithley: re: ubuntu firewall... there is an iptables wrapper called UFW (uncomplicated firewall) which afaict is included but disabled by default on ubuntu server
15:58 semiosis kkeithley: though if it were enabled it would install a default ruleset
15:58 semiosis kkeithley: it's little more than a wrapper around iptables btw
15:59 semiosis kkeithley: so imho if someone has enabled ufw they should be hip enough to know it's blocking things
16:01 rodlabs joined #gluster
16:01 wN joined #gluster
16:04 wushudoin joined #gluster
16:04 khushildep joined #gluster
16:06 hagarth1 joined #gluster
16:18 semiosis ndevos: Psi-Jack: ok I've made nfs-common a dependency (formerly a suggestion) of glusterfs-server.  uploading to launchpad now.
16:19 chacken3 joined #gluster
16:23 gmcwhistler joined #gluster
16:24 lkoranda joined #gluster
16:39 ndevos semiosis: can you file a bug against the build component for that? we should probably include that in the rpm packages too
16:39 glusterbot http://goo.gl/UUuCq
16:40 * ndevos checked the .spec and it is not a requires atm
16:40 kkeithley ndevos: are you sure that one of the existing Requires or somewhere down its dependency chain doesn't already address that?
16:41 khushildep joined #gluster
16:41 ndevos kkeithley: I dont think so, we're just lucky that nfs-utils gets installed on most systems and that pulls rpcbind/portmapper in
16:41 * kkeithley checked the spec too, but doesn't have an easy way to check the dependency chain
16:43 ndevos kkeithley: none of the requires are related to rpc or nfs, there is no reason for them to require rpcbind/portmapper
16:43 kkeithley well, it's easy enough to test. I'll just load up a pristine f17 vm and see what we get, out of the box
16:47 semiosis ndevos: i'm hesitant to file a bug for something that doesn't affect me & that i can't actually reproduce since i dont use rh/fedora
16:47 glusterbot http://goo.gl/UUuCq
16:48 blendedbychris joined #gluster
16:48 blendedbychris joined #gluster
16:49 ndevos semiosis: ah, I understand - I'll try to remind myself to do that
16:50 semiosis doing my part tho committing the fix to debian :)
16:51 ndevos :)
16:52 semiosis spinning up a new debian vm to see if nfs-common is included by default (doubt it) and confirm the change works like it did on ubuntu (should)
16:53 JoeJulian yum erase portmap (on el5) doesn't try to erase glusterfs-server for whatever that information is worth.
16:54 ndevos right, thats a fair test, el6 uses rpcbind and I think it'll react the same
16:54 kkeithley that confirms that glusterfs-server doesn't Require: portmap or nfs-utils
16:55 kkeithley which we know it doesn't, not directly
16:56 semiosis nice graph search problem... is there a dependency path from the base system packages to portmap
16:56 semiosis idk the right tool for that job tho :/
16:57 kkeithley minimum f17 install does not include rpcbind by default.
16:58 kkeithley and installing gluster did not pull it in
16:59 kkeithley next question maybe, is does the fedora server install — which is probably a fairer test — install rpcbind
17:00 ndevos kkeithley: I dont really care about the typical installations, imho gluster should have a working nfs-server after installing the packages
17:03 kkeithley I didn't say (or write) anything that's in conflict with that. I just want to understand where the discrepancy lies.
17:06 khushildep joined #gluster
17:15 blendedbychris joined #gluster
17:15 blendedbychris joined #gluster
17:15 chouchins joined #gluster
17:17 H__ is it normal that a rebalance status command only gives 'rebalance step ... in progress' on *the same* box as where it was started ?
17:23 JoeJulian H__: What's the status of the other servers? What version?
17:27 chouchins any suggestions on backing up millions of small files totaling nearly 30TB on glusterfs?  We're on 10GB network which works great on the large files but the small files slow this waaaay down.
17:28 daMaestro joined #gluster
17:30 chouchins looks like in most situation with a nonclustered storage arrangement people are just taking disk images for small files.  Obviously that won't help here.
17:30 jdarcy chouchins: My #1 suggestion is that you don't back up those files *as individual files*.  Combine them into some sort of archive that can be streamed to/from GlusterFS.
17:31 chouchins tar first and then move to backup server?
17:32 jdarcy chouchins: Something like that.  It gives GlusterFS a workload that it's best at, while retaining the advantage of clustered storage.
17:33 seanh-ansca joined #gluster
17:35 seanh-ansca joined #gluster
17:37 Psi-Jack Hmmm, so iperf shows we're definitely able to get ~900 Mbit/s constant throughput, but glusterfs is only ever using about ~106 Kbit/s throughput.
17:40 Mo__ joined #gluster
17:42 ika2810 left #gluster
17:43 faizan joined #gluster
17:45 mrobbert joined #gluster
18:08 layer3 joined #gluster
18:18 JoeJulian Psi-Jack: I would look at your application, your cpu load, memory usage, disk throuput, etc. In sheer throughput tests, I can always max out my network.
18:18 chandank|work joined #gluster
18:18 Psi-Jack Yeah, sheer throughput, I can get excellent results, I'm trying to get GlusterFS to meet or exceed it.
18:19 mnaser joined #gluster
18:19 JoeJulian That's what I'm saying.
18:19 Psi-Jack iperf shows Network performance is optimal, but actually specifically monitoring glusterfs itself, it's not getting that throughput at all,.
18:19 mnaser Is running gluster on applications servers to create replicated storage between application servers OK or it has to be on separate machines?
18:19 JoeJulian mnaser: It's pretty common, actually.
18:20 Psi-Jack Heh yeah.
18:20 mnaser JoeJulian: thanks, a lot of people mentioning it as "use these servers to get storage" so yeah, thanks
18:21 Psi-Jack I'm ALMOST tempted to try that approach, use Gluster to replicate, but use the local physical data not via glusterfs mount, though I hate that idea, since it will result in local differences, specifically with tmp stuff.
18:22 Psi-Jack Actual Disk I/O is excellent. Easily get 200 MB/sec out of it.
18:24 JoeJulian So where's your bottleneck? If your network's fast enough, your disk is fast enough, and other people are able to get near wirespeed transfers out of GlusterFS, what's different about either your configuration or your testing method?
18:25 chouchins is it safe to remove a directory on each brick themselves or does that risk corruption?
18:26 Psi-Jack JoeJulian: That's the thing. GlusterFS itself isn't able to get anywhere near the performance.
18:26 Psi-Jack And that's what I'm trying to find out.
18:26 Psi-Jack heh
18:27 mnaser is there a way to tell gluster with replication to try and store a replica on the local server as well so disk reads can happen with minimal network io or not really?
18:28 Psi-Jack I'm putting together a smaller test environment now, to see if I get any different results. Not nearly as powerful of an infrastructure, but it should be able to help me pinpoint out the bottlenecks.
18:30 JoeJulian chouchins: I wouldn't. (but I have)
18:30 JoeJulian mnaser: Not yet.
18:30 chouchins its taking hours to delete this dir with the millions of small files :)  Thinking it has to be faster on the bricks
18:30 mnaser JoeJulian: the yet gives a bit of hope, is that something that's supposed to happen soon?
18:32 JoeJulian yes. I believe it's already in gerrit for 3.4. I could be wrong on the progress, but I know it's up there on the list.
18:35 nightwalk joined #gluster
18:42 wN joined #gluster
18:43 Psi-Jack Basically, I'm following this guide here: http://www.gluster.org/community/d​ocumentation/index.php/QuickStart    to create a 2-node gluster server setup,.
18:43 glusterbot <http://goo.gl/OEzZn> (at www.gluster.org)
18:43 Psi-Jack Simple replica 2 setup.
18:43 stat1x joined #gluster
18:45 stat1x I am having a helluva time trying to fix a split-brain situation .. I've had more success in the past. Getting I/O errors for just a couple directories. The only thing I can see is that the trusted.afr.xxx-client-n attr differs on these replicas. Should those be the same?
18:45 stat1x gfid is already the same
18:46 stat1x The client ids are the same, but the values differ
18:47 stat1x Using v3.2
19:02 jack joined #gluster
19:06 nueces joined #gluster
19:08 JoeJulian stat1x: The trusted.afr should be all zeros. If they're not, that specifies which brick that file has pending operations for. If two bricks think they have pending operations for each other, that's ,,(split-brain). The cure is:
19:08 glusterbot stat1x: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
19:14 raghu joined #gluster
19:15 stat1x So, since I'm using 3.2, I should remove those attrs?
19:17 JoeJulian No, that just means that the .glusterfs tree portion of that fix is unnecessary.
19:21 DaveS_ joined #gluster
19:22 stat1x I see
19:24 andreask joined #gluster
19:26 DAVES_ joined #gluster
19:30 nightwalk joined #gluster
19:33 stat1x I guess I'm not at that depth yet; I can't tell what files are causing the split brain. I already deleted all files that were created on the "bad brick" while it was out of the cluster
19:34 stat1x Usually, just that works to fix the split brain, but this time it isn't
19:35 JoeJulian stat1x: With 3.3 you would just do "gluster volume heal $vol info split-brain". With 3.2 you'll have to look in your client logs.
19:35 sshaaf joined #gluster
19:38 stat1x I'm getting a lot of "cannot create linkfile file for XXX", but for files that are identical: content and xattrs
19:39 stat1x same gfid
19:40 stat1x Also getting "found anomalies in /directory. holes=1 overlaps=0"
19:42 stat1x If I can at least "zero out" the "bad brick" so that the files are retained and the self-heal is as fast as possible, then that's what I need to do
19:42 stat1x This is a massive directory with hundreds of thousands of files
19:47 JoeJulian stat1x: None of those are split-brain errors. Did you do the ,,(repair)?
19:47 glusterbot stat1x: http://goo.gl/uA812
19:48 stat1x Not fully. Was trying to deal with split brain first because I'm also getting "subvolumes have unrecoverable errors" ... "split-brain possible, no source detected"
19:48 stat1x Self-heal is triggering, but writes to the directory are failing
19:49 H__ JoeJulian: the other server says that no rebalance is active. This is version 3.2.5
19:52 stat1x And yeah... Input/output errors
19:52 JoeJulian Don't the "split-brain possible" lines show which files they're reporting?
19:53 blendedbychris joined #gluster
19:53 blendedbychris joined #gluster
19:53 JoeJulian H__: I've never had good luck doing rebalances prior to 3.3.1
19:53 H__ *gulp*
19:54 stat1x No, only the "cannot create linkfile file" give filenames, and they're very old files which appear to be properly replicated
19:54 stat1x The parent directory is giving errors like "/directory returned -1 (Input/output error)"
19:55 stat1x Also getting a lot of "no subvolume for hash" errors. It's a disaster
20:05 badone joined #gluster
20:07 purpleidea joined #gluster
20:11 bfoster joined #gluster
20:12 mohankumar joined #gluster
20:30 gcbirzan joined #gluster
20:30 gcbirzan if I have a volume that is exposed by only one host and that host had a network fluke, is it possible to recover the volume on a remote host, without remounting>?
20:33 stat1x JoeJulian: you are the man.
20:34 stat1x I set the afr xattrs for the replica to zero on the "bad brick" and I think it's fully self-healing now
20:36 stat1x Awesome. Thank you so much.
20:38 nightwalk joined #gluster
20:41 JoeJulian You're welcome.
20:43 semiosis Psi-Jack: what are you using to generate the load for your throughput measurements?  if dd, are you setting bs?  try bs=1M
20:43 Psi-Jack semiosis: Bonnie++, for one. :)
20:45 semiosis oh ok, not familiar enough to give advice on that one
20:46 H__ afaik bonnie will still only use 1 file for it's tests. need a whole tree of em to test gluster properly IMO. maybe there's an option for that ?
20:46 Psi-Jack Heh, hmm.
20:47 Psi-Jack When I do dd if=/dev/urandom of=bench.data bs=1M count=4096, I can definitely see throughput ramped up to 5.72 MBps.
20:47 gcbirzan Use /dev/null
20:47 gcbirzan /dev/urandom is slow
20:47 semiosis /dev/zero
20:47 JoeJulian urandom has it's own bottlnecks.
20:47 gcbirzan er, yeah
20:47 gcbirzan sorry
20:47 gcbirzan also, the gluster fuse client is slooooow, so if you really need speed, you might consider mounting the same volume more than once
20:48 Psi-Jack Yeah, I wanted to test actual data throughput, not negligable data.
20:48 gcbirzan make a random data file, and dd from that I guess
20:48 Psi-Jack Pardon? Same volume more than once?
20:49 Psi-Jack Yeah, /dev/zero's pushing 28 MBps, so far.
20:49 gcbirzan depending on your workload, you can double your trhoughput
20:50 semiosis +1 multiple dd processes per client, and multiple clients, should both get you more aggregate throughput
20:50 semiosis multiple client *machines* even moreso
20:50 gcbirzan yeah, but hey
20:50 Psi-Jack I tried the NFS client method too, and it was actually worse.
20:50 Psi-Jack Than the fuse client method.
20:50 gcbirzan but we've found that doing two writes per client goes from 300 to 320
20:51 JoeJulian gcbirzan: In situations where that's effective, I would be interested in knowing if setting performance.client.io-threads on a single mount is as effective.
20:51 gcbirzan JoeJulian: hm. I can check
20:51 Psi-Jack What's the default performance.client.io-threads?
20:51 JoeJulian off
20:51 JoeJulian It's boolean
20:51 Psi-Jack Ooooh.. ;)
20:52 gcbirzan that'd be actually cool if it worked
20:52 gcbirzan though, it'd be even cooler if we could use it for one day without having a catastrophic failure :P
20:52 JoeJulian Yeah, that would be a bonus.
20:53 JoeJulian What kind of catastrophes are you having?
20:53 semiosis next you'll want egg in your beer /cc kkeithley
20:53 JoeJulian :D
20:53 JoeJulian The egg makes it healthy, right?
20:54 semiosis if by healthy you mean taste bad, probably
20:55 JoeJulian Well it's got protein that way. It's like a protein shake without the blender.
20:57 gcbirzan JoeJulian: basically, any kind of problems with any of the hosts exposing the volumes breaks it
20:57 gcbirzan like, now, we had a network issue, I needed to remount the volume to be able to access it. if the host had died, I'd have files I couldn't heal ever
20:57 gcbirzan so we'd delete them and recopy them
20:57 semiosis could you clarify "exposing" and "breaks" ?
20:58 JoeJulian gcbirzan: Are you on 3.3.1? I know of a bug in .0 where client's wouldn't reconnect to a lost server.
20:58 semiosis gcbirzan: and what version of glusterfs are you using?
20:58 semiosis yeah, that
20:59 gcbirzan (glusterfs-3.3.0-8.fc17.x86_64) we have pairs of hosts that each have 2 volumes each (though, doesn't matter how many), in replcation. if any of the hosts dies (reboot?), we need to manually fix the files by copying it off the brick, deleting it from the volume, copying it back
20:59 gcbirzan that's when we're lucky
20:59 gcbirzan sometimes, clients need to reconnect in order to realise the file has been self healed
20:59 JoeJulian Yeah, that bug was fixed too.
20:59 Psi-Jack Hmmm, interesting. performance.io-thread-count set to 60 has improved it.
21:00 gcbirzan sometimes, reading a file works fine, writing blocks indefinitely
21:00 gcbirzan that's when we need to remove the file from the volume, and not even that doesn't work sometimes, and we have to remount the volume
21:00 sshaaf joined #gluster
21:01 JoeJulian gcbirzan: I recommend upgrading. 3.3.1 sounds like it'll fix most of what ails you.
21:04 nightwalk joined #gluster
21:05 gcbirzan yeah, I'll try that. but, at this point, we have a problem every day, and it's getting rather old. especially the remounting stuff
21:06 JoeJulian I would also fix that network problem. ;)
21:09 gcbirzan that would definitely help me get some sleep
21:09 Mo__ joined #gluster
21:13 khushildep joined #gluster
21:17 layer3 joined #gluster
21:25 z00dax so, replace-brick is supposed to really work right ?
21:25 JoeJulian 3.3.1 worked great.
21:25 z00dax and not lose files, in random - yea 3.3.1
21:25 z00dax I've lost about 45k files
21:25 z00dax ~ 100G worth ( give or take a few )
21:26 nightwalk joined #gluster
21:26 JoeJulian You did replace-brick ... start, watch replace-brick...status until it said it was complete, then replace-brick...commit
21:26 z00dax gluster volume replace-brick glusterfs-srv cats:/glusterfs-bricks/srv majha:/glusterfs-bricks/srv3 start
21:26 z00dax ( wait for that to complete )
21:26 z00dax gluster volume replace-brick glusterfs-srv cats:/glusterfs-bricks/srv majha:/glusterfs-bricks/srv3 commit
21:26 z00dax when its done
21:27 z00dax what did i get wrong ?
21:27 JoeJulian Did you ever do status in between?
21:27 z00dax yes, i waited for the first one to get to complete before doing the commit
21:27 JoeJulian Ok. Just wanted to make sure I understood correctly.
21:27 JoeJulian ~pasteinfo | z00dax
21:27 glusterbot z00dax: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:28 z00dax the replace-brick ran for a few hours
21:28 rodlabs joined #gluster
21:29 z00dax http://dpaste.org/EkwOY/
21:29 glusterbot Title: dpaste.de: Snippet #212708 (at dpaste.org)
21:30 JoeJulian hmm, straight distribute, huh... Have you confirmed that the files are actually missing out of the srv3 directory?
21:30 z00dax yup
21:30 z00dax and if I mount the /glusterfs-bricks/srv2 in cats, they are in there
21:31 z00dax well, a large number of them are, I've not done a complete compare - that would take hours
21:31 JoeJulian Well, short of digging through logs I don't know what to tell you about them being missing. Might as well just rsync them from cats though to fix the problem.
21:32 z00dax rsync takes ~ 20 hrs to workout what files it needs to sync over :(
21:32 JoeJulian If you want to post the migration log somewhere, I'll take a look and see if I see anything obvious.
21:32 z00dax can i just rsync them into /glusterfs-bricks/srv2 directly ?
21:32 z00dax i know what the answer is going to be :D
21:33 JoeJulian gluster volume replace-brick glusterfs-srv cats:/glusterfs-bricks/srv majha:/glusterfs-bricks/srv3 ... don't you mean srv3?
21:33 z00dax yes, sorry - srv3
21:34 JoeJulian As long as it's brick to brick (and you don't screw up and put them on srv2) for a distribute only volume you should be pretty safe.
21:35 z00dax yeah, Brick2 -> srv3, and Brick3 -> srv2
21:35 z00dax its just the order they were pulled back in
21:36 z00dax the Brick3 move, a while back went fine
21:42 z00dax so, looking at /glusterfs-bricks/srv/ and srv2 and srv3 .... should i see files replicated into multiple bricks ?
21:42 z00dax there are atleast a couple of files that seem to exist on multiple bricks
21:42 z00dax whats up with that
21:43 forest joined #gluster
21:43 JoeJulian There should only be one with the size of more than 0. The others should be size zero and mode 1000.
21:43 forest left #gluster
21:43 JoeJulian They should have a trusted.gluster.dht.linkto attribute.
21:44 z00dax aha, yes, size=0
21:44 z00dax so what causes some files to be like this and others to not
21:45 JoeJulian I'll write something up about that, but it has to do with the dht hash lookup technique.
21:46 JoeJulian A rebalance will clear those up.
21:48 z00dax ok, a rebalance before spreading these bricks around again
21:58 Daxxial_1 joined #gluster
22:00 hattenator joined #gluster
22:05 badone_ joined #gluster
22:08 tg2 joined #gluster
22:08 lkoranda joined #gluster
22:19 andreask joined #gluster
22:36 andreask hi ... what is the "timeout" paramter for geo-replication setup?
22:40 stefanha joined #gluster
22:45 TSM2 joined #gluster
23:04 nightwalk joined #gluster
23:46 mrobbert left #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary