Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 stat1x_ joined #gluster
00:08 nobodydontknow joined #gluster
00:11 badone_ joined #gluster
00:12 stat1x_ I'm 3.2.5 that's heavily used over wan. I need to delete an unused, huge directory of small files (50+ GB). Will it cause the clients to all hang if I `rm -rf` the directory on a client? Can I delete the raw /export files instead, or is that dangerous?
00:14 jbrooks joined #gluster
00:18 jjnash joined #gluster
00:20 jjnash left #gluster
00:23 ultrabizweb joined #gluster
00:25 Deformative joined #gluster
00:37 jclift thisisdave: Um, when you talk about the volume name, what are you meaning?  That's something I've not had to do yet.
00:37 jclift thisisdave: it the .rdma-fuse bit
00:37 jclift s/it/ie/
00:37 glusterbot What jclift meant to say was: Deformative: That seems to have volume level operations logged into ie
00:37 jclift glusterbot: STFU
00:38 * jclift wishes glusterbot wouldn't try and fix s/// stuff.  It just confuses things
00:39 ultrabizweb joined #gluster
00:40 Deformative Lawl.
00:57 forest joined #gluster
01:00 thisisdave jclift: it was the only way I could get the client to mount via rdma
01:00 jclift thisisdave: k.  I'm just not understanding what you mean... did you manually have to create/adjust .vol file or something?
01:01 thisisdave nope, just in the mount command
01:01 jclift thisisdave: Interesting.  Do you still have the command handy?  Would like to see the full thing, just so I "get it". :)
01:02 jclift Maybe it's a v3.3.x thing.
01:02 jclift Or maybe I've just been luck so far. :)
01:02 jclift s/luck/lucky/
01:05 jclift ... or not...
01:09 thisisdave jclift: in my case it was `mount -t glusterfs host:/volume.rdma-fuse /mount/point`
01:09 jclift thisisdave: Thanks.  That makes it clear.
01:09 jclift I've not had to do that with stuff I've worked with yet.
01:09 * jclift didn't know that syntax even existed. :D
01:10 thisisdave jclift: of course. :-) just dealing with bringing one of my two server nodes up (as it's being stubborn with zfs--damned its lack of a 'brick' dir!)
01:11 jclift That lack of brick dir thing should only be a problem with gluster 3.4 and stuff built from git.  Guess you're trying that out now...
01:11 thisisdave jclift: nor did I... I did find a blog post specifying the .rdma suffix but it didn't work; experimented with the .rdma-fuse and it did the trick.
01:11 jclift Cool
01:12 bala joined #gluster
01:12 forest joined #gluster
01:12 jclift thisisdave: Btw, if it turns out that lack of a brick dir is a blocker to Gluster working on ZFS, then file a bug (or tell me to).  The code which does it could be reverted if there's sufficient reason, etc. :)
01:12 glusterbot http://goo.gl/UUuCq
01:13 thisisdave jclift: zfs mount was commented out in rc.local so gluster was writing to the mount dir ... I'm pretty sure that's the main/only justification for the brick-dir best practice...
01:16 jclift On a completely random note, this looks like fun.  Want one, even though I'd be over it in like 2 mins: http://www.maplin.co.uk/x4-mini-quad-copter-685087
01:16 jclift :)
01:16 glusterbot <http://goo.gl/KojsB> (at www.maplin.co.uk)
01:30 forest joined #gluster
01:40 kevein joined #gluster
01:56 bala joined #gluster
01:58 portante joined #gluster
02:04 hagarth joined #gluster
02:07 rajesh joined #gluster
02:19 thisisdave jclift: we make our own here. ;-)
02:19 jclift :)
02:23 y4m4 joined #gluster
02:27 bala joined #gluster
02:36 bharata joined #gluster
02:43 [MDL]Matt joined #gluster
02:53 glusterbot New news from newglusterbugs: [Bug 962619] glusterd crashes on volume-stop <http://goo.gl/XXzSY>
03:09 forest joined #gluster
03:19 mohankumar joined #gluster
03:37 jag3773 joined #gluster
03:42 DEac- joined #gluster
03:52 badone_ joined #gluster
03:52 NeonLicht joined #gluster
03:52 icemax joined #gluster
03:52 fleducquede joined #gluster
03:52 ehg joined #gluster
03:52 Peanut joined #gluster
03:52 shanks joined #gluster
03:52 cyberbootje1 joined #gluster
03:52 efries_ joined #gluster
03:52 js_ joined #gluster
03:52 pull joined #gluster
03:52 Dave2 joined #gluster
03:52 juhaj joined #gluster
03:52 twx joined #gluster
03:52 mrEriksson joined #gluster
03:53 20WACNT4Q joined #gluster
03:53 bharata-rao joined #gluster
03:53 nightwalk joined #gluster
03:53 isomorphic joined #gluster
03:53 GLHMarmot joined #gluster
03:53 larsks joined #gluster
03:53 jclift joined #gluster
03:53 madd joined #gluster
03:53 ccha joined #gluster
03:53 fxmulder joined #gluster
03:53 Cenbe joined #gluster
03:53 Goatbert joined #gluster
03:53 mriv joined #gluster
03:53 snarkyboojum_ joined #gluster
03:54 itisravi joined #gluster
03:54 itisravi joined #gluster
03:54 31NAAR84N joined #gluster
03:55 badone_ joined #gluster
03:55 NeonLicht joined #gluster
03:55 icemax joined #gluster
03:55 fleducquede joined #gluster
03:55 ehg joined #gluster
03:55 Peanut joined #gluster
03:55 shanks joined #gluster
03:55 cyberbootje1 joined #gluster
03:55 efries_ joined #gluster
03:55 js_ joined #gluster
03:55 pull joined #gluster
03:55 Dave2 joined #gluster
03:55 juhaj joined #gluster
03:55 twx joined #gluster
03:55 mrEriksson joined #gluster
04:01 forest joined #gluster
04:05 Deformative How do I make sure gluster can only be mounted from the intranet?
04:16 hagarth Deformative: you can use auth-allow settings on a volume or block port 24007 on all storage server for clients outside your intranet.
04:17 Deformative Thanks.
04:18 Deformative Any advice for additional changes I should make to improve performance?
04:18 Deformative http://www.fpaste.org/19532/71615514/
04:18 glusterbot Title: #19532 Fedora Project Pastebin (at www.fpaste.org)
04:19 Deformative Or some common customizations?
04:19 hagarth depends on your workload. what do you plan to use this volume for?
04:21 vpshastry joined #gluster
04:25 Deformative hpc data.
04:25 Deformative And the /home directory.
04:25 Deformative But most of hte users just keep a bunch of data there.
04:26 Deformative For their own simulations.
04:26 Deformative hagarth,
04:27 hagarth Deformative: the default should be good enough for most use cases.
04:29 atrius joined #gluster
04:29 Deformative hagarth, Thanks.
04:30 Deformative Do you know where I can learn about diagnosing bottlenecks?
04:30 Deformative Like figure out why some things may be slow.
04:31 aravindavk joined #gluster
04:49 shireesh joined #gluster
04:50 CheRi joined #gluster
05:01 Deformative What are the properties of rdma?
05:07 * ferringb stretches
05:07 ferringb Deformative: fast
05:08 Deformative ferringb, Then why is tcp default?
05:08 ferringb cause most people don't have infiband hardware (which is what rdma is)
05:08 sgowda joined #gluster
05:09 Deformative Oh.
05:09 Deformative I didn't know it couldn't be done over ethernet.
05:10 ferringb :)
05:10 balunasj|mtg joined #gluster
05:11 Deformative Is there a tool to test what type of bandwidth I can get on my current network?
05:11 Deformative I know ping is for latency.
05:13 rastar joined #gluster
05:15 ferringb yes
05:15 ferringb sec
05:16 ferringb iperf
05:18 bulde joined #gluster
05:18 Deformative Cool.
05:18 Deformative Trying now.
05:18 Deformative What is a reasonable latency and bandwidth for a gluster network?
05:30 Deformative I am seeing .6ms; 94.7 Mbits/sec, pretty sure I have a shitty network switch.
05:32 Deformative I suspect my nics aren't too great either.
05:38 vpshastry1 joined #gluster
05:39 sonne joined #gluster
05:44 ricky-ticky joined #gluster
05:45 lalatenduM joined #gluster
05:50 bala joined #gluster
05:50 pkoro joined #gluster
05:50 vpshastry joined #gluster
05:59 satheesh joined #gluster
06:00 satheesh1 joined #gluster
06:06 psharma joined #gluster
06:09 arusso- joined #gluster
06:09 arusso joined #gluster
06:10 jcastle joined #gluster
06:10 johnmark joined #gluster
06:11 sticky|away joined #gluster
06:11 stickyboy joined #gluster
06:11 waldner joined #gluster
06:11 waldner joined #gluster
06:15 ferringb Deformative: Gb switch == best case around 111MB/s, which is the limits
06:16 ferringb (assuming you've not bonded ports)
06:17 thommy_ka joined #gluster
06:17 Deformative ferringb, Aren't most modern switches better than Gb?
06:17 vimal joined #gluster
06:18 thommy joined #gluster
06:19 Deformative There is a linksys sitting not plugged in that says "1000 gbit switch"
06:22 glusterbot New news from resolvedglusterbugs: [Bug 802671] NFS/DHT: rename fails under load <http://goo.gl/dSVKi>
06:22 thessy joined #gluster
06:23 jtux joined #gluster
06:24 fidevo joined #gluster
06:24 vshankar joined #gluster
06:27 thessy joined #gluster
06:28 ngoswami joined #gluster
06:30 raghu joined #gluster
06:31 rastar joined #gluster
06:31 thessy_ joined #gluster
06:35 romero joined #gluster
06:36 thessy__ joined #gluster
06:36 clag_ joined #gluster
06:40 thessy__ joined #gluster
06:45 thommy_ka joined #gluster
06:46 andreask joined #gluster
06:49 StarBeast joined #gluster
06:49 mohankumar joined #gluster
06:49 thommy joined #gluster
06:50 ekuric joined #gluster
06:52 ctria joined #gluster
06:54 thommy joined #gluster
06:58 thessy joined #gluster
07:03 thessy_ joined #gluster
07:07 thessy_ joined #gluster
07:11 thessy__ joined #gluster
07:14 ramkrsna joined #gluster
07:14 ramkrsna joined #gluster
07:16 thessy__ joined #gluster
07:19 rgustafs joined #gluster
07:20 thessy__ joined #gluster
07:26 vpshastry joined #gluster
07:28 dobber_ joined #gluster
07:46 StarBeast joined #gluster
08:09 jbrooks joined #gluster
08:10 vpshastry joined #gluster
08:25 spider_fingers joined #gluster
08:27 zetheroo1 joined #gluster
08:36 zetheroo1 we have run into an issue with our use of Gluster. We have 2 hosts servers each hosting a brick which is part of a replica 2. We run VM's from this gluster. Problem is that if one brick goes offline/down the other brick becomes read-only making the VM's running from it totally useless. Is there any solution that would enable the running brick to remain readable and writable - and then when the other brick is back online there to be a way to get the two
08:37 andreask zetheroo1: native gluster mount?
08:37 frakt your message got split so we cant see all of it.. it ends with "when the other brick is back online there to be a way to get the two
08:37 frakt "
08:37 zetheroo1 ok sorry
08:38 zetheroo1 Is there any solution that would enable the running brick to remain readable and writable - and then when the other brick is back online there to be a way to get the two bricks in sync again?
08:38 andreask it should do so after 42s
08:38 andreask .. and then auto-heal ...
08:39 zetheroo1 andreask: it should do what after 42s?
08:39 andreask zetheroo1: allow to write
08:39 zetheroo1 andreask: interesting ... is that a default setting or something that we have to enable?
08:39 andreask default setting
08:40 zetheroo1 ok, but even in this case it is disruptive to the VM's
08:40 zetheroo1 is there a way to disable the bricks from entering read-only mode?
08:41 ferringb zetheroo1: offhand, it should remain writable depending on your xlater stack
08:42 ferringb that 42s bit is in reference to the ping-timeout when the client gives up on a brick
08:43 ferringb as for getting back in sync, self-heal handles that; self-heal-daemon being enabled adds a proactive scanner for it
08:46 vpshastry1 joined #gluster
08:50 aravindavk joined #gluster
08:54 spider_fingers joined #gluster
08:58 lyang0 joined #gluster
09:02 zetheroo1 ferringb: but it's not remaining writable ... that's the thing ... is falls into read-only mode and then the VM's are useless until they are all rebooted
09:04 Oneiroi joined #gluster
09:15 aravindavk joined #gluster
09:25 ricky-ticky joined #gluster
09:29 realdannys1 joined #gluster
09:47 zetheroo1 is there anyway to disable a brick from going into read-only mode?
09:48 samppah zetheroo1: is the brick in read only mode or only the vm's running in it?
09:50 zetheroo1 samppah: the VM's are because the brick went into this mode .. and although the brick may recover after 42sec's the VM's remain stuck as read-only.
09:50 zetheroo1 so it seems I need to be able to disable the bricks from ever going to read-only mode
09:50 samppah zetheroo1: you may want to decrease ping-timeout value.. gluster volume set volname network.ping-timeout 10
09:50 samppah for example that sets it to 10 seconds
09:51 zetheroo1 ok, but even if the brick becomes read-only for just 10 seconds the VM's will likely still be affected
09:51 aravindavk joined #gluster
09:52 samppah zetheroo1: i think it should be around 30s before vm's are set into read only state
09:52 samppah i'm not sure if there is some value inside vm that can be adjusted
09:53 zetheroo1 so there is no way to stop the read-only mode entirely?
09:54 vpshastry1 joined #gluster
10:02 samppah zetheroo1: afaik gluster doesn't set it to read only mode.. if server goes down it blocks io for 42 seconds (by default) to see if server comes back up
10:03 zetheroo1 and if the timeout is set to 0 seconds!?
10:03 ekuric left #gluster
10:03 samppah that's not recommended
10:03 ekuric joined #gluster
10:05 the-me joined #gluster
10:06 zetheroo1 so basically gluster is not usable as a high-availability option ...
10:07 zetheroo1 the only way it's useful to us is if the VM's are uninterrupted when one brick fails
10:08 samppah is 10 seconds too long in your case?
10:10 zetheroo1 well I don't see why the VM's would not be affected by the gluster going read-only for 10 seconds ...
10:11 zetheroo1 but in case I want to try it out ... where would I change the setting?
10:14 StarBeas_ joined #gluster
10:15 LordErr joined #gluster
10:16 aravindavk joined #gluster
10:16 LordErr hi. im looking for some documentation to explain the memory status for a given volume. What fordblks etc represents
10:17 LordErr please point me in the right direction.
10:18 Goatbert joined #gluster
10:22 nightwalk joined #gluster
10:22 isomorphic joined #gluster
10:30 samppah zetheroo1: you can set it to 10 seconds running following command on serverside: gluster volume set volname network.ping-timeout 10
10:31 zetheroo1 samppah: ok, thanks. And why is it not a good idea in my case to put that to 0 seconds?
10:35 edward1 joined #gluster
10:36 samppah zetheroo1: after that timeout it considers server dead and doesn't send io requests to it unil it's up again.. after it's up again it has to reset file locks etc
10:37 samppah if the value is too low then even short disconnects may cause server to go into offline state
10:38 bala joined #gluster
10:41 deepakcs joined #gluster
10:59 manik joined #gluster
11:04 hagarth joined #gluster
11:04 andreask joined #gluster
11:09 psharma joined #gluster
11:12 lpabon joined #gluster
11:15 psharma joined #gluster
11:20 aravindavk joined #gluster
11:21 the-me joined #gluster
11:21 vpshastry1 joined #gluster
11:21 CheRi joined #gluster
11:22 nobodydontknow joined #gluster
11:24 shireesh joined #gluster
11:26 yinyin joined #gluster
11:40 CheRi joined #gluster
11:42 CheRi joined #gluster
11:44 atrius_ joined #gluster
11:48 balunasj joined #gluster
11:48 forest joined #gluster
11:50 psharma joined #gluster
12:03 jthorne joined #gluster
12:05 aliguori joined #gluster
12:07 marcinm joined #gluster
12:11 mmarcin hi all, I have a question re glusterfs, perhaps you can help me
12:11 mmarcin I have 2 servers with 3 gluster volumes on each working in replication mode
12:11 mmarcin filesystems are ext4
12:12 mmarcin also, gluster replicates not the partitions itself but logical volumes
12:13 mmarcin my question is - can I use lvextend to resize logical volumes which are being replicated
12:15 jthorne joined #gluster
12:18 manik joined #gluster
12:20 zetheroo1 does the ext4 filesystem have a default IO timeout period?
12:21 mmarcin how can I check that?
12:23 mmarcin in fstab the ext4 filesystems are defined to mount with defaults
12:33 Norky joined #gluster
12:33 pkoro mmarcin: we have done this on our installation without any problem. lvresize, xfs_grow and you are there I believe.
12:33 tziOm joined #gluster
12:33 pkoro oops!
12:33 pkoro I just noticed you are using ext4.
12:33 pkoro sorry
12:34 pkoro it should work with that also, but I cannot say that with confidence.
12:40 bala joined #gluster
12:41 mmarcin pkoro: I've already done this twice and seems that everything is working fine, but I'm bit worried if this is ok to do so
12:42 mmarcin also, have you also been using glusterfs for replication?
12:43 pkoro yes, one replicated volume. resized both bricks.
12:43 mmarcin ok, and no glusterfsd restart was required in your case right?
12:44 hagarth @channelstats
12:44 glusterbot hagarth: On #gluster there have been 143204 messages, containing 6085803 characters, 1020846 words, 4152 smileys, and 525 frowns; 878 of those messages were ACTIONs. There have been 53908 joins, 1687 parts, 52252 quits, 19 kicks, 146 mode changes, and 5 topic changes. There are currently 196 users and the channel has peaked at 217 users.
12:49 bet_ joined #gluster
12:49 social__ hmm interesting, if I set brick-sys-log-level to WARN and brick-log-level to NONE instead of getting all logs into syslog I get no logs as brick-log-level NONE disables logging completely
12:53 mmarcin also guys, one more question - is it safe to restart glusterfsd on one machine in replication mode, when there are services using the stores? will the data sync properly after glusterfsd will start back?
12:55 zetheroo1 so it would seem to me that there is no way of making sure that VM's running off a gluster will continue to run properly despite a brick going offline
12:56 lalatenduM joined #gluster
12:57 mohankumar joined #gluster
12:57 mmarcin I'm quite new to gluster so perhaps I'm asking obvious questions
12:58 Norky mmarcin, I've done brick reszingin while glusterfs is online
12:59 mmarcin Norky: ext4? any issues after?
13:00 Norky the servers noticed larger bricks/volumes withotu my having to do anything, but a couple of the nodes did not immediately notice the larger volume (cached fsdata I assume), I might have had to do an unmount/remount on those two (out of 40 or so) clients
13:00 Norky no, xfs
13:01 Norky but the difference between ext3/4 and xfs should have no bearing on that particularl question
13:01 mmarcin ok, thx - in umount/remount safe or do the services using the storage have to be stopped prior to that?
13:02 mmarcin can there be any sync issues after that?
13:02 Norky err, I did the remount on two *clients*, there are no permanent daemons running on the clients that use glusterfs volumes
13:03 Norky I di nothing to the servers (other than actually resizing the LV and growing the xfs)
13:03 mmarcin ok
13:03 vpshastry joined #gluster
13:03 vpshastry left #gluster
13:04 nixpanic joined #gluster
13:04 nixpanic joined #gluster
13:07 yinyin joined #gluster
13:07 nobodydontknow joined #gluster
13:12 pkoro mmarcin: no, no restart of services was retired. sorry for the late reply
13:12 pkoro required
13:13 mmarcin ok, thx, I'll give it a try
13:16 Norky if you're worried about the consequences,, do it first on s test volume or some test servers....
13:19 mmarcin Norky: it was already done twice in the past and nothing happened, volumes started after the server was rebooted, but I'm just wondering if I don't do anything wrong
13:21 dewey joined #gluster
13:23 vpshastry joined #gluster
13:23 plarsen joined #gluster
13:24 vpshastry left #gluster
13:25 madnexus2 joined #gluster
13:27 statix_ joined #gluster
13:36 dewey joined #gluster
13:40 theron joined #gluster
13:51 rwheeler joined #gluster
13:53 andrewjsledge joined #gluster
13:54 jclift joined #gluster
14:00 thommy_ka joined #gluster
14:01 bugs_ joined #gluster
14:02 joelwallis joined #gluster
14:03 vpshastry joined #gluster
14:06 statix_ Using 3.2.5 over WAN. I have a very large directory within a heavily used volume, which is ~65GB of small web files. The folder is no longer needed, is not used, and must be deleted. If I just try to `rm -rf` it from a client, will it cause the clients to all hang? Could I delete the files directly from the /export directories? I cannot take the volume offline
14:08 kkeithley_ It should not cause clients to hang. Slow down probably, but not hang. Delta any bugs of course, YMMV.
14:08 deepakcs joined #gluster
14:08 je23 joined #gluster
14:09 zetheroo1 if one of two bricks in a replica 2 gluster goes down the brick remaining on will go to read-only mode ... is there any way of stopping this from happening?
14:10 kkeithley_ If you're concerned about performance impact, I'd rename the subdir to something invisible and maybe up a couple directories and then delete the files during the off hours.
14:10 awheeler_ joined #gluster
14:11 statix_ That's the other side of it: I'm not sure how intensive a mv of such a large directory would be
14:12 GLHMarmot joined #gluster
14:12 kkeithley_ As long as it's on the same fs, it should be instantaneous and cheap.
14:14 kkeithley_ In the same subdir, e.g. rename from reallybigdir to .reallybigdir should be a no cost, low overhead operation.
14:15 plarsen joined #gluster
14:19 yinyin joined #gluster
14:25 statix_ Ugh.. so scared of this
14:27 kkeithley_ And no, you should (almost) never touch files on the bricks directly. Always do it from a client.
14:28 kkeithley_ This shouldn't be scary. What's the brick fs mount point? What's the big subdir path relative to the mount point?
14:29 rastar joined #gluster
14:29 statix_ The bricks are basically /export/topdir/DIR_TO_DELETE; clients mount /topdir. There's about 12 clients
14:30 statix_ All of /export are on the same FS on the bricks
14:31 statix_ All FUSE clients
14:32 forest joined #gluster
14:32 itisravi joined #gluster
14:32 kkeithley_ okay, forget moving DIR_TO_DELETE up (down) in the tree. I'd just rename it to .DIR_TO_DELETE (notice the dot) and then start to delete files during off-peak times.
14:32 hchiramm_ joined #gluster
14:34 kkeithley_ If you're still uncomfortable, wait a bit and see if anyone else reacts to my suggestion before you pull the trigger.
14:34 statix_ What'
14:34 kkeithley_ Once the west coast wakes up and gets on-line
14:34 statix_ What's the point of the rename?
14:35 kkeithley_ no point per se. It just removes the dir from normal visibility.
14:36 kkeithley_ Then nobody is likely to write files in it, accidentally or otherwise
14:37 spider_fingers left #gluster
14:42 statix_ I see. So with a command like `mv` or `rm -rf` on a directory like that, will the client essentially issue the command to all of the bricks, and wait for them to return. Or will there be other sorts of "gluster" stuff going on.. may be an odd question
14:43 dbruhn joined #gluster
14:44 kkeithley_ yes, the client issues file-ops, e.g. unlink, which are relayed to the glusterfsd on the server, i.e. the bricks.
14:44 kkeithley_ and the glusterfsd unlinks (deletes) the file in the brick.
14:45 jthorne joined #gluster
14:45 statix_ Ok, just did a rename and it was almost instantaneous.. that gives me some confidence
14:45 kkeithley_ Not sure what you mean by gluster stuff going on. There's no geo-rep in 3.2.5, so there's nothing else going on.
14:45 kkeithley_ right
14:46 kkeithley_ directory and file renames are "cheap".  Removing 65GB of files will take a while.
14:47 statix_ Gonna go for the full delete
14:47 statix_ By "gluster stuff", I mean I hope that wouldn't trigger like a full directory scan or something, for integrity check or whatever
14:49 statix_ I guess the concern is that I wouldn't want a large operation like that to affect any parent/sibling directories
14:50 statix_ Command is running, other clients are not hanging
14:50 dbruhn I am having a weird issue on a couple systems when trying to delete files I get this error "Directory not empty (/mnt/ENTV02EP/root/data/1/71/88/2/1/1)"
14:50 kkeithley_ No, nothing like that. The rm -rf is going to descend into the directory you're deleting and do opendir/readdir (into any subdirs too) and that's going to take a while.
14:51 dbruhn even though it is empty from the client perspective
14:52 statix_ Doesn't seem to be loading down the bricks at all, but it's taking a while
14:52 kkeithley_ statix_: yes, it's going to take a while.
14:53 kkeithley_ dbruhn: and what about from the server perspective, is there anything there?
14:54 statix_ This is going as well as I could have hoped *knock on wood* .. very encouraging
14:55 T0aD joined #gluster
14:59 dbruhn kkeithley: You mean on the bricks?
14:59 kkeithley_ yes
15:03 dbruhn kkeithley: the bricks have data under them
15:03 dbruhn at the corresponding locations
15:12 kkeithley_ what version of gluster, and what are the filenames on the bricks?
15:13 pvradu joined #gluster
15:14 dbruhn 3.3.1 on both systems exhibiting the issue
15:14 dbruhn and there is a directory under 4 bricks called 92
15:16 statix_ The delete of the directory does seem to have triggered a self-heal on that directory (it was pretty corrupt)
15:17 dbruhn one of the systems is a 1GB system and the other is QDR IB system
15:17 T0aD left #gluster
15:17 Deformative joined #gluster
15:20 lalatenduM joined #gluster
15:21 forest joined #gluster
15:24 psharma joined #gluster
15:24 zetheroo1 left #gluster
15:24 brosner joined #gluster
15:25 kkeithley_ I think I'd wait for the self heal to finish and then try deleting it again (on the client)
15:25 kkeithley_ Or did the self heal already finish?
15:26 dbruhn That's what I keep needing to do, but I am in the process of deleting millions of files, and this keeps popping up.
15:26 dbruhn I have 8 jobs running that should be an automated process, and the jobs are stalling out on my application layer.
15:26 dbruhn because of the issue
15:27 dbruhn I am just trying to get to the root what might be going on to correct the issue in general
15:27 daMaestro joined #gluster
15:28 kkeithley_ understood
15:28 dbruhn Would turning quorum on with self healing help preemp this issue in the future?
15:38 statix_ kkeithley: self-heal is ongoing; not sure if it's actually deleting anything yet
15:38 andreask joined #gluster
15:42 statix_ It is deleting. Looks to be doing meta-data self-heals on individual files and getting a whole lot of "file not found"s
15:44 Norky is it normal and expected for a largish (22TiB) distrivuted-replicated volume to fill up two (a replica pair) of its bricks while the other two are 'only' 90% full? (aggregate 95% full)
15:44 itisravi_ joined #gluster
15:44 kkeithley_ Yeah, anything that touches a file is going to trigger a self heal check. When you're just deleting I suppose it can be annoying.
15:44 Norky http://fpaste.org/19663/
15:44 glusterbot Title: #19663 Fedora Project Pastebin (at fpaste.org)
15:45 kkeithley_ Norky: I would say it's not abnormal. It's obviously not optimal.
15:47 Norky would a rebalance fix it?
15:47 Norky at leat, fix the immediate problem of files that happen to be the full bricks getting E_NOSPACE
15:48 Norky s/to be/to be on/
15:48 glusterbot What Norky meant to say was: at leat, fix the immediate problem of files that happen to be on the full bricks getting E_NOSPACE
15:48 kkeithley_ A rebalance will fix it if you add another replica pair. All by itself, no.
15:48 lalatenduM joined #gluster
15:51 theron joined #gluster
15:52 Norky okay, going back to gluster basic, file distribution (the decision on which part (brick or replica sub-volume) of a distributed vol to place a file) is done by a hashing function, yes?
15:52 Norky this is in the docs, I'd like to check my understanding of the docs
15:53 kkeithley_ correct
15:53 Norky a hash of what, exactly?
15:54 Norky and presumably all clients will be using the same function to write/read a given file from the correct brick/server?
15:55 kkeithley_ The file name.  (Files with similar, short names will usually hash to the same value, so it's not uncommon when you have a lot of files with short, similar names to fill up one brick before the others.)
15:55 kkeithley_ also correct.
15:56 bala joined #gluster
15:56 Norky that hash will obviously be affected by number of sub-volumes, which is why a rebalance will only really do anything major if you change the number of sub-volumes?
15:57 Norky ahh, so presumably, if you rename a file, it'll have a different hash and so might now be expected to reside on a different brick/sub-vol?
16:00 hagarth joined #gluster
16:01 Norky actually, that makes a lot of sense for this particular case, the volume holds a number of CFD files, where a given user project/directory will have a number (maybe up to 20) of multi-GB files of very nearly the same name
16:02 kkeithley_ correct again. Unfortunately I don't know if rename and rebalance will result in the file being moved.
16:03 Deformative joined #gluster
16:04 kkeithley_ IIRC, when you rename, resulting in a new hash, the next attempt to read by the new name will silently fail. Gluster will find it and create a metadata link from where it thinks it should be to where it really is. The next attempt to read will find the metadata link and go directly to the file.
16:06 jclift hagarth__ johnmark: Any interest in publishing meeting minutes for the 3.4 readiness thing to gluster-devel or similar?
16:07 forest joined #gluster
16:09 statix_ .. thanks for the help kkeithley
16:12 Mo__ joined #gluster
16:17 Norky this makes perfect sense, the dir the user was working in when they hit the problem has 12 60GB files on one brick pair while the other pair has only 3, not enough to accoutn for the overall disparity, but there will be plenty of other directories with similarly alike files
16:17 Norky thank you very much for your time/patience kkeithley_ :)
16:22 hagarth joined #gluster
16:23 realdannys1 joined #gluster
16:26 theron joined #gluster
16:26 Deformative joined #gluster
16:28 jag3773 joined #gluster
16:47 Deformative joined #gluster
16:50 forest joined #gluster
17:03 forest joined #gluster
17:06 joelwallis joined #gluster
17:07 nightwalk joined #gluster
17:07 cicero would there be a speed boost if i mounted /brick/brick1/.glusterfs on ssd?
17:07 cicero as an example
17:20 vpshastry joined #gluster
17:21 bulde joined #gluster
17:26 lpabon joined #gluster
17:34 manik joined #gluster
17:34 mrEriksson Anyone mounting gluster volumes under SLES 11 (SP2) ? I'm having problems getting the system to mount the gluster volumes after the network has been initiated
17:35 zaitcev joined #gluster
17:36 joelwallis joined #gluster
17:38 semiosis mrEriksson: pastie.org your client log file
17:39 forest joined #gluster
17:40 mrEriksson semiosis: Sure, but I'm pretty sure that the problem is that the glusters are mounted before the network is initiated at bootup, if I just mount -a, the volumes mounts just fine
17:40 semiosis fair enough
17:40 tg3 yeah it happens that gluster tries to mount before the network is up
17:40 semiosis have you added _netdev to the options in fstab?
17:41 mrEriksson The _netdev option seems to have been removed in sles :( Seems to be a hardcoded list of filesystems that it considers to be "network filesystems"
17:41 tg3 i think we ended up using an init script
17:41 semiosis mrEriksson, tg3: the log would confirm if that is the cause of this problem
17:41 tg3 yeah true
17:41 mrEriksson Sure, hold on :)
17:43 semiosis mrEriksson: if in fact sles has a hard coded list of network filesystems, then you really should file a bug with the sles people who maintain that list
17:43 glusterbot http://goo.gl/UUuCq
17:43 semiosis no glusterbot
17:43 semiosis not now
17:43 tg3 loll
17:43 tg3 overzealous glusterbot
17:44 mrEriksson 0-software-client-0: server 172.31.20.21:24013 has not responded in the last 42 seconds, disconnecting.
17:45 mrEriksson Also, preceeded by lots of Network is unreachable-messages :)
17:45 mrEriksson Problem confirmed I guess?
17:45 mrEriksson tg3:  So you have seen the same with sles?
17:46 mrEriksson /etc/init.d/boot.localfs defines the following:
17:46 mrEriksson typeset -r netfs=nfs,nfs4,smbfs,cifs,afs,ncpfs
17:47 cicero ahh good ole ncpfs
17:47 tg3 lol
17:47 tg3 yeah thats probably what it is
17:47 tg3 i saw it on centos
17:47 mrEriksson And then seems to ignore those filesystems on the first run, so it seems to be quite hardcoded :P
17:48 tg3 sry was on ubuntu
17:48 tg3 centos was fine
17:49 tg3 gentoo was fine
17:49 mrEriksson Yeah, centos isn't an issue :)
17:49 tg3 i'll grab the init script we have
17:49 tg3 you can use it :)
17:49 tg3 just remove the mount from fstab so it doesn't hold up the boot
17:49 mrEriksson Thanks :) Still don't like that solution though
17:50 tg3 well, the solution is to have the OS understand it
17:50 mrEriksson I don't mind custom initscripts, but to mount a filesystem, that's just wrong
17:51 tg3 http://pastie.org/private/sntcwn8jhj2gwcilcyagg
17:51 glusterbot Title: Private Paste - Pastie (at pastie.org)
17:51 tg3 dirty
17:51 tg3 but it works
17:51 tg3 ignore the iprouts ;D
17:52 forest joined #gluster
17:53 mrEriksson Thanks :) This is still wrong though :-)
17:55 stigchristian joined #gluster
17:56 romero_ joined #gluster
17:57 tg3 i think this assumes the record in fstab
18:06 wgao joined #gluster
18:08 plarsen joined #gluster
18:11 dbruhn For the slow mount issue, I just run a mount command in my rc.local with a 30 second wait in front of it to make sure the fstab/gluster components are all ready
18:12 forest joined #gluster
18:23 Deformative joined #gluster
18:25 isomorphic joined #gluster
18:39 neofob left #gluster
18:40 mrEriksson tg3: I ended up doing a script that parses fstab and mounts all glusters after initlevel has been reached, that way, I can still manager the gluster mountpoints via fstab, just that I prevent rc from mounting them with noauto
18:40 mrEriksson Seems to work quite well
18:45 semiosis mrEriksson: are you installing from a package?
18:45 badone_ joined #gluster
18:45 mrEriksson Installing what?
18:45 semiosis glusterfs
18:45 mrEriksson yeah
18:45 semiosis which package?
18:45 mrEriksson found some sles-packages on opensuses buildfactory
18:46 semiosis please open a bug report with the package maintainer or whoever is responsible for it
18:46 semiosis it's clearly a packaging bug
18:47 semiosis save someone else from having the same problem
18:47 mrEriksson Well, it's an unsupported package
18:47 mrEriksson And I don't really see any decent way of solving this
18:48 mrEriksson Not by using fstab anyways
18:48 semiosis i'm sure the suse package maintainers would like to help... having a glusterfs package that works well would be good for everyone in the sles community
18:49 mrEriksson I found a couple of lustre-guys who have filed bugreports for the same issue to suse over a couple of years without success
18:50 semiosis bummer
18:50 mrEriksson Yeah
18:50 semiosis why use that distro then?
18:50 semiosis if the community isn't helpful
18:50 tg3 copmany mandate probably
18:50 semiosis yikes
18:50 tg3 now's your chance to push RHEL :D
18:50 semiosis tg3: ubuntu actually :D
18:51 semiosis but i'm just a community guy, i dont represent any of these companies
18:51 mrEriksson Would be very simple to solve the issue though, but it is an issue for suse, not the package maintainer
18:51 mrEriksson I actually prefer suse to rhel :)
18:52 mrEriksson If I get some spare time, I'll probably try to implement a more permanent solution for this and see if it would be accepted upstream
18:55 theron joined #gluster
19:05 Deformative joined #gluster
19:13 semiosis mrEriksson: that would be really cool, thanks
19:15 mrEriksson Don't thank me yet :-)
19:21 piotrektt joined #gluster
19:22 NcA^ can anyone point me in the right direction here... I'm seeing a slew of logs come into glustershd.log every 10 minutes. Logs are over 5Gb now. All entries are for metadata self-heal on different gfid's
19:23 brosner joined #gluster
19:24 portante joined #gluster
19:30 Jasson joined #gluster
19:32 je23 joined #gluster
19:32 Jasson Hello, I'm looking for a little advice with a gluster 3.3 install, and trying to get a client to mount a gluster volume as read only.
19:33 Jasson It seems to be working over NFS with our smaller shares, but we have an 18TB share that doesn't work correctly over NFS, and attempting to mount with the ro option fails to mount as a glusterfs connection.
19:33 semiosis Jasson: bug 853895 -- read only mount is broken, since 3.3.0 :(
19:33 glusterbot Bug http://goo.gl/xCkfr medium, medium, ---, csaba, ON_QA , CLI: read only glusterfs mount fails
19:34 semiosis 3.4.0 should work
19:34 semiosis when it's released
19:34 semiosis maybe 3.3.2 as well, idk the status of the patch
19:34 semiosis afk
19:34 Jasson how stable is 3.4?  And can I just upgrade my inplace 3.3 servers and clients to 3.4?
19:37 je23 hi, I am having problems with gluster 3.2. Specifically the php function is_dir is returning false for directories which exist. Has anyone experienced anything like that before?
19:37 tg3 ls -l on the volume itself shows the directory
19:38 tg3 and you can descend into it?
19:38 tg3 try to do a dirlist inside php in the parent directory and see the records
19:38 pvradu joined #gluster
19:38 piotrektt joined #gluster
19:39 kkeithley_ Doesn't look like read-only is fixed in 3.3.2. Not yet anyway.
19:39 je23 yes, the directory is there, and the progam saves a file into the directory, but it has code to create dir if it is not present and it is trying to do that every time since I moved from an nfs share to a gluster share
19:39 Jasson is read-only working in 3.4?
19:39 kkeithley_ Yes, it's fixed in 3.4.0
19:39 tg3 ah but if you read a file from within it, that works?
19:40 jclift Jasson: We're holding a test day for 3.4 on friday.  It's probably a bad idea to try using it straight in production though without testing it a bunch with your environment.
19:40 * jclift points out he's the cautious type with real data :)
19:40 kkeithley_ read-only fixed in 3.4.0 in commit 702b2912970e7cc19416aff7d3696d15977efc2f
19:40 piotrektt_alpha joined #gluster
19:40 tg3 i saw some ready only stuff in the 3.3.1 source, but i guess it wasn't fully implemented.
19:41 Jasson we're in a state where we have had to move large amounts of data to a new storage server and were hoping to move to 3.3, but now we need read only working to finish the migration.  Since it works in NFS mode, we missed that it was broken.
19:41 Jasson but NFS won't read correctly the 18TB share.
19:41 tg3 we could mount 1 brick as read only, and 1 as normal, and every write/create would go to the writeable brick, reads went to both
19:41 tg3 but if you tried to delete a file after it wouldn't work even though it was on the write brick
19:43 y4m4 joined #gluster
19:43 jclift kkeithley: Looking at that commit, should it be backported to 3.3?
19:44 jclift It doesn't look all that complex a piece of code...
19:44 piotrektt joined #gluster
19:45 je23 Okay, if I try to list the directory contents of a gluster share with php nothing is shown, if I do it with a local path it shows the files
19:45 mrEriksson je23: As the same user?
19:45 jclift mrEriksson: Good point.
19:45 mrEriksson :P
19:46 jclift :)
19:46 mrEriksson I've recently had a similar issue :)
19:46 je23 yes, I tried both these as the root user
19:46 mrEriksson je23: So the php-script isn't executed via a webserver?
19:47 je23 it is but I am trying to debug it from the command line, and its doing the same unusual behavior
19:47 jclift je23: Is SELinux (or equivalent) enabled on the box having the dir read issue?
19:47 forest joined #gluster
19:47 jclift ... just coz that's worth checking.
19:49 je23 It was, which is embarrassing I should have checked that, but I've put it into permissive mode and still no joy
19:49 y4m4 joined #gluster
19:50 mrEriksson je23: Have you tried your php-script on a non-glustered directory?
19:51 je23 its literally  php -r "var_dump(scandir('/gluster'));"
19:51 je23 from the prompt, and when I try that on /home it list the directories
19:52 jclift je23: What's the output of gluster peer status, and gluster volume info ?
19:53 Jasson Question on the 3.4 and read only status.  Is that a client side bug?  Could I install 3.4 on the client and still read a share from a 3.3.1 server?
19:53 piotrektt joined #gluster
19:54 je23 Number of Peers: 1
19:54 je23 Hostname: 192.168.4.165
19:54 je23 Uuid: 51cc78b0-ce9e-4436-8a6d-ed69bc3f35e0
19:54 je23 State: Peer in Cluster (Connected)
19:54 krishna_ joined #gluster
19:55 je23 Volume Name: gv1
19:55 je23 Type: Replicate
19:55 je23 Volume ID: 02ce3521-2965-42cc-90c2-a05ea10edef5
19:55 je23 Status: Started
19:55 je23 Number of Bricks: 1 x 2 = 2
19:55 je23 Transport-type: tcp
19:55 je23 Bricks:
19:55 je23 Brick1: 192.168.4.164:/export/sdc1/brick
19:55 je23 Brick2: 192.168.4.165:/export/sdc1/brick
19:56 piotrektt joined #gluster
19:59 realdannys1 joined #gluster
20:02 jclift dbruhn: Do you have an IB test environment you could run some Gluster tests on this friday?
20:03 dbruhn What kind of test?
20:03 jclift dbruhn: We're putting together a "Gluster 3.4 RDMA Test Day" specifically so we can shake out bugs and find any weirdnesses left in 3.4 code base before it goes GA
20:04 Jasson Can a 3.4 client connect to a 3.3 server?
20:04 jclift dbruhn: So, it'd be both boring standard volume lifecycle creation stuff (creating new volumes, etc), plus also trying out some of the 3.4 specific new features
20:04 semiosis je23: please use pastie.org or similar for multiline pastes
20:04 jclift Jasson: No, don't do that
20:05 dbruhn Right now both of my RDMA systems are in production, but I just ordered 10 servers for a non gluster system and was thinking about ordering an IB switch of ebay. I will be out at the cloudstack convention and my test environment machines won't be here till like thursday though.
20:05 dbruhn of next week
20:05 jclift Jasson: Gluster's not yet at a stage where different versions of things will nicely interoperate with each other.  Should be ok in the near future though (maybe 3.4 onwards, maybe 3.5)
20:05 nueces joined #gluster
20:06 Jasson yeah, I suspected that was the case.  Thanks
20:06 jclift dbruhn: k, you'll miss the test day on friday then.  That being said, if you're willing to run through stuff when you get them next week, that's still very welcome. :)
20:06 jclift dbruhn: Out of curiosity, what sort of switch are you looking for?
20:06 jclift The Voltaire 4036's have a good rep, and can be had cheaply
20:06 dbruhn Let em see if I can get some used IB stuff, that environment is going to be purely a dev test environment forever.
20:07 dbruhn right now i am using the mellenox IS5030
20:07 dbruhn I saw a 5025 on ebay last night for 1500
20:07 semiosis NcA^: maybe pastie.org a sample of the logs?  hard to say whats going on (or if there's even a problem) without seeing the logs.  did you have some kind of event which would require healing?  server down, network partition, etc?
20:07 jclift dbruhn: Heh I have a 5025 sitting next to me right at the moment (borrowed, I have to give it back)
20:07 jclift But, they didn't send me any cables, so it's not used atm. :(
20:08 joelwallis joined #gluster
20:08 dbruhn have you used it to run IP o IB at all?
20:08 jclift dbruhn: I'm using a Voltaire ISR 9024D-M instead at the moment.
20:08 dbruhn I have like 3 spare cables right now is all, and I left my three spare cards sitting in our vegas datacenter
20:08 jclift Mainly because it can run fanless
20:08 NcA^ semiosis: I'll paste up part of the logs. At the moment, it's almost exactly every 10 minutes. 3 servers running replication 3.
20:09 jclift So, I haven't even tried hooking the 5025 up to anything.  It's QDR and all my personal stuff here is DDR.
20:09 jclift The Mellanox guys are looking into finding some spare cables for me...
20:09 * jclift thinks it's kind of weird they didn't send any with the switch and cards
20:10 jclift dbruhn: Why the question about IPoIB?
20:10 dbruhn does the 5025 have the subnet manager built in?
20:10 jclift No
20:10 jclift It's a completely dumb switch
20:10 jclift Doesn't even do SNMP, no remote CLI or anything
20:10 dbruhn I am thinking about using gluster under cloudstack, and the RDMA stuff is less stable than TCP/IP
20:10 dbruhn I get random disconnects and what not
20:10 jclift So, you can't even retrieve temp info from the 5025
20:11 dbruhn but in my environment IO is king so I am dealing RDMA at the moment over IP
20:11 jclift dbruhn: Yeah.  There's a bunch of new RDMA specific code that's gone into 3.4, which is one of the reasons we want to shake out bugs
20:11 jclift I'm finding it more stable so far than 3.3 and older.
20:11 dbruhn That's awesome, I really want to participate now.
20:11 jclift But still hitting strange performance things in some scenarios.
20:12 jclift eg: Good write speeds, strangely slow read speeds (slower than IPoIB)
20:12 * jclift wants to get that sorted of course
20:12 dbruhn My environment is 99.5% random read I/O with small files? *sigh*
20:14 je23 The only unusual thing I can find is that one of the client machines which created the dir is a 64bit machine and has the i386 packages installed, is that likely to cause issues
20:14 jclift Heh.  Well the read performance difference is like 500MB/s write, 80MB/s read (with a 2 node stripe using SSD backend)
20:14 jclift dbruhn: So, that's definitely a bug somewhere :)
20:14 jclift dbruhn: eg, something that needs fixing
20:14 dbruhn Holy shit, that's a big deal.
20:15 plarsen joined #gluster
20:15 * jclift wonders if the code paths are somehow sending the read data over the (actual) ethernet nic or something
20:15 jclift Will look into that properly later
20:16 dbruhn My two IB systems are 100% RDMA
20:17 NcA^ semiosis: http://pastie.org/private/fetasvkbbitxodvfiwpw
20:17 glusterbot Title: Private Paste - Pastie (at pastie.org)
20:18 NcA^ semiosis: that's just a small sample of what gets output, there's thousands of lines that are similar
20:18 dbruhn Is HP rebranding the Mellenox switches as their own?
20:18 jclift dbruhn: Yep
20:18 jclift dbruhn: Aha, this should help: http://www.ebay.com/itm/ws/eBayISAPI​.dll?ViewItem&amp;item=261223469914
20:18 jclift :)
20:18 glusterbot <http://goo.gl/MyuwM> (at www.ebay.com)
20:19 semiosis NcA^: maybe you have a client disconnected from one of the replica bricks, so shd kicks in to heal
20:19 dbruhn Trying to play hardware catch up here, does that have the subnet manager, and can all of the switches do IPoIB?
20:20 dbruhn the IPoIB is purely done on the software layer in the client machines right?
20:21 jclift dbruhn: Those switches can definitely do IPoIB.
20:21 NcA^ semiosis: client, such as a server mounting via the FUSE client?
20:22 jclift dbruhn: The IP layer is encapsulated inside the IB layer.
20:22 semiosis client, such as "mount -t glusterfs ...."
20:23 dbruhn That's what I thought, I had a dell guy telling me that most of the IB switches weren't able to be used for IP it kind of threw me off
20:23 jclift dbruhn: The Dell guy was probably meaning the IB switches can't be used for _native_ IP.
20:23 dbruhn I will say he was totally a windows admin too?. so? yeah
20:23 jclift dbruhn: One of the things I learned about the Mellanox ConnectX cards when I got them from eBay, is they can run in either native IB mode, or native 10GbE mode
20:24 dbruhn I have the ConnectX 3 cards in all of my servers right now
20:24 dbruhn whats a good cheap qdr IB card to find used?
20:24 jclift And the native 10GbE mode worked fine when cabled back to back to my other boxes.  But, by DDR switch doesn't talk native 10GbE mode.
20:24 jclift s/by/my/
20:24 glusterbot What jclift meant to say was: And the native 10GbE mode worked fine when cabled back to back to my other boxes.  But, my DDR switch doesn't talk native 10GbE mode.
20:25 * jclift shoots glusterbot
20:25 jclift dbruhn: I'm not sure there is such a thing as a cheap QDR card atm.  Depending on your definition of cheap.
20:25 jclift dbruhn: DDR cards are very dirt cheap though.
20:25 dbruhn well like $100 or less range per card
20:26 jclift DDR cards are like $40 each, dual port
20:26 jclift QDR ones are still a couple of hundred
20:26 jclift dbruhn: You can see why I setup my dev/test lab with DDR gear. :)
20:26 dbruhn are all of the ddd cards that weird transceiver connection instead of the qsfp+ connection?
20:26 dbruhn yeah for sure
20:26 jclift "weird transceiver connection" == CX4
20:27 jclift The cheaps one definitely are
20:27 jclift CX4-CX4 cables are pretty cheap too.  US$14 each (from china, so add on postage)
20:28 jclift Haven't yet found a cheap source of QDR-QDR nor FDR-FDR cables though
20:28 jclift Nor CX4-QDR/FDR either
20:28 rcoup joined #gluster
20:29 dbruhn is gluster 100% agnostic to DDR vs QDR/FDR?
20:29 jclift dbruhn: Yep.  It doesn't care.
20:30 dbruhn I am wondering if I can get into 10 cards, and a ddd switch for less than a grand
20:30 dbruhn s/ ddd / ddr
20:30 NcA^ semiosis: this is running a drupal/php site, more specifically, the uploaded shared files which are infrequently accessed and cached in varnish, seems there's other mention of users with similar issues when php is touching / creating the files
20:31 semiosis NcA^: mention where?
20:32 NcA^ semiosis: mailing list
20:32 semiosis [citation needed] :)
20:32 NcA^ semiosis: http://gluster.org/pipermail/glus​ter-users/2013-March/035662.html
20:32 glusterbot <http://goo.gl/V7ULH> (at gluster.org)
20:32 NcA^ was hunting down that tab ;)
20:32 jclift dbruhn: Cheapo switch: http://www.ebay.com/itm/181150402796
20:32 semiosis thx
20:32 glusterbot Title: HP Voltaire ISR 9024 D Grid Switch 9024D ISR9024D | eBay (at www.ebay.com)
20:33 jclift dbruhn: ideally you'd go the 9024D-M (the "D" meaning "DDR" and the "M" meaning it's a managed switch with subnet manager, etc)
20:33 jclift dbruhn: Postage might be the biggest pita atm
20:33 semiosis NcA^: that sounds like a different problem
20:33 dbruhn lol
20:33 dbruhn I see that
20:33 semiosis NcA^: are you seeing that kind of thing in your client log files?
20:35 NcA^ semiosis: nope, not that exactly
20:35 NcA^ just what was in the paste
20:35 semiosis NcA^: then i think its safe to say that user's problem is unrelated to yours
20:35 semiosis lots of people use php over glusterfs without either of these problems
20:35 NcA^ semiosis: yeah, was driving myself crazy and going down the wrong rabbit-hole
20:35 semiosis so no connection there
20:35 NcA^ kk
20:36 NcA^ semiosis: this is the other thing I saw: http://pastie.org/private/l6mpwfou842krzme9fn5uw
20:36 glusterbot <http://goo.gl/mhIiN> (at pastie.org)
20:36 NcA^ some stale indexes
20:36 semiosis "remote operation failed" -- whenever i see that i go looking for corresponding messages in brick logs
20:36 semiosis the other side of the "remote operation"
20:37 semiosis NcA^: can you pastie.org 'gluster volume status' please
20:37 NcA^ sure
20:38 semiosis also check brick logs for anything suspicious
20:38 NcA^ semiosis: http://pastie.org/private/yjas6letzj6mr9welzleig
20:38 glusterbot <http://goo.gl/0gPXn> (at pastie.org)
20:40 NcA^ one of the bricks is full of "no suchfile or directory" errors
20:40 semiosis progress
20:41 semiosis is there anything wrong with the mount for that brick?
20:42 NcA^ hm, not overtly, this was hanging about in the messages log
20:43 NcA^ "Failed to get IB devices"
20:45 semiosis do you have any IB devices?
20:45 je23 left #gluster
20:46 NcA^ nope
20:46 jclift dbruhn: Oh, there is such a thing as a cheap QDR card.  The QDR Mezzanine cards for a specific model of Dell server are always cheap. (and very popular with home server builders)
20:46 semiosis NcA^: then i wouldn't worry about not finding any :)
20:46 NcA^ cheers :P
20:47 jclift dbruhn: But, that's about it.  Weirdly, all of the cheap DDR IB switches located in the US seem to have dried up atm.  Voltaire, Flextronics, etc.  Maybe take a look again in a few days, as it varies heaps.
20:47 NcA^ looks like I might have found the start date. Brick log first starts reporting remote operation failed
20:47 semiosis NcA^: i'm running out of ideas... if i were in your position, i would just kill that brick's process and restart it (either by restarting glusterd on the host, or with 'gluster volume start $volname force')
20:47 dbruhn looks like these can be had for about $120sh  a card MHQH19-XTC
20:48 NcA^ semiosis: I'll give that a shot. Thanks a lot for your help, learned quite a bit about gluster logs today!
20:48 semiosis yw, let us know when you find out what the issue was
20:50 jclift dbruhn: That's weird, I'm not seeing any results on searches for that.
20:50 jclift dbruhn: Maybe my country settings are wrong.  Paste me a URL?
20:51 dbruhn http://www.cedarpc.com/product/18​580/Mellanox-MHQH19-XTC-Adapter/
20:51 glusterbot <http://goo.gl/O2Q40> (at www.cedarpc.com)
20:51 dbruhn They have 8 of them
20:51 dbruhn and one xtr version for the same price
20:52 jclift Interesting.  That would work with the 4036 switch very well.  QDR cables are still pretty expensive though.
20:52 jclift dbruhn: But, my budget is very different from most
20:52 jclift i.e. home user :D
20:54 dbruhn Totally understood, mine isn't much better. I have been running in production since 3.1, and had been testing since the early 2 releases. All i have is VM's for testing, so my RDMA stuff is a black hole to me.
20:54 dbruhn my budget isn't that awesome
20:54 lpabon joined #gluster
20:56 Deformative joined #gluster
20:59 jclift dbruhn: I've been getting cables from this guy (3 lots): http://www.ebay.co.uk/itm/251200441924
20:59 glusterbot Title: New Infiniband 10GBs 4X CX4 to CX4 Cable 1M/3.3FT SAS M/M | eBay (at www.ebay.co.uk)
20:59 jclift I do "make offer" and he accepts US$14
21:01 social__ https://www.youtube.com/watch?v=H3fSpllkUtA
21:01 glusterbot Title: Psí vojáci - Žiletky - YouTube (at www.youtube.com)
21:01 social__ err, wrong channel soory
21:02 sprachgenerator joined #gluster
21:03 sprachgenerator a random question, running 3.4 I lost a node, rebuilt it using the same hostname, however the brick isn't replicating data from it's mate and it's showing as offline in volume status for all volumes (however volume info, peer status, etc is all there)
21:03 sprachgenerator any suggestions on getting this to replicate from it's made and be listed online?
21:03 sprachgenerator a replace-brick commit gives a staging failed on localhost error
21:04 tziOm joined #gluster
21:04 kkeithley_ jclift: re: Looking at that commit, should it be backported to 3.3? It doesn't look all that complex a piece of code...
21:05 jclift kkeithley_: Yeah, just wondering... :)
21:05 kkeithley_ yeah, perhaps
21:05 jclift kkeithley_: People seem to ask about r/o mode a decent amount.  No need to force them to 3.4 for just that...
21:05 kkeithley_ yup
21:09 semiosis sprachgenerator: on the replaced server, does gluster volume info show the volume(s)?
21:09 sprachgenerator yes
21:09 sprachgenerator volume heal states self-heal daemon not running
21:10 sprachgenerator 3.4.0beta2
21:10 semiosis sprachgenerator: restart glusterd, that should spawn any missing processes
21:10 semiosis sprachgenerator: also... iptables?
21:10 sprachgenerator I have restarted glistered as well
21:10 sprachgenerator made sure manually kill any that didn't die off
21:12 sprachgenerator isn't the self heal daemon the one responsible for recreating the files?
21:13 semiosis i suppose so
21:14 semiosis check the glusterd log file for signs why the self heal daemon may not be starting
21:14 tziOm So when is the pnfs part implemented?
21:14 tziOm ;)
21:14 yinyin joined #gluster
21:15 tziOm I say fsck the gluster fuse and focus pnfs
21:16 sprachgenerator it would appear that the script defaults with it off?
21:16 semiosis the script?
21:16 sprachgenerator -/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0beta2 (/usr/sbin/glusterfs -s localhost --volfile-id whisper --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option
21:16 sprachgenerator *replicate*.readdir-failover=off --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=339280d7-b4​d3-414b-be84-e8e4638f3751 --socket-file /var/lib/glusterd/vols/whisper/rebalance/​339280d7-b4d3-414b-be84-e8e4638f3751.sock --pid-file /var/lib/glusterd/vols/whisper/rebalance/​339280d7-b4d3-414b-be84-e8e4638f3751.pid -l /var/log/glusterfs/whisper-rebalance.log)
21:16 sprachgenerator startup
21:16 sprachgenerator this is in the log entry
21:16 semiosis next time please use pastie.org for long pastes like that
21:17 sprachgenerator sure thing - it kinda rubbed me the wrong way
21:26 forest joined #gluster
21:29 NcA^ semiosis: still filling the logs up, but I think I may have something... somehow, one of the gluster nodes is running 3.3.1.1, whereas the other two are on 3.3.0.1
21:34 neofob joined #gluster
21:35 dbruhn jclift: I think I am going to try and order some DDR IB gear next week. I am out till then anyway, that way I can test the stuff too. I will load my full application stack on it and see what I can break
21:39 penglish1 joined #gluster
21:39 penglish1 Hiya - I'm adding a whole bunch of bricks to a "distributed replicate" volume
21:40 penglish1 Can someone confirm that this looks correct, and I'll end up adding 3-pair of replicated systems
21:40 penglish1 sudo gluster volume add-brick data tyr10.3tier.com:/brick1 tyr11.3tier.com:/brick1 tyr12.3tier.com:/brick1 tyr13.3tier.com:/brick1 tyr14.3tier.com:/brick1 tyr15.3tier.com:/brick1
21:40 penglish1 "data" is the name of the volume
21:40 penglish1 (yes, not very creative)
21:41 penglish1 Everybody is asleep
21:41 penglish1 I'm reasonably confident. :-)
21:45 jiqiren ah! :( gluster meetup for tomorrow go canceled!
21:45 tziOm gluster is canceled?
21:46 dbruhn looks right penglish1, assuming you intend 10 to replicate to 11, 12 to 13, and 14 to 15?
21:46 penglish1 dbruhn: yes exactly
21:46 penglish1 thanks for the confirmation
21:47 dbruhn no problem
21:49 penglish1 woo! 268TB
21:49 jclift dbruhn: Cool. :)
21:50 penglish1 some people skydive for a rush. I add bricks to my gluster volume
21:50 dbruhn penglish1: What are you doing on your 268TB system?
21:51 jclift jiqiren: The gluster meetup in mountain view?
21:52 penglish1 dbruhn: http://www.3tier.com/en/about/press-​releases/3tier-selected-nrel-update-​national-wind-integration-dataset/
21:52 glusterbot <http://goo.gl/fL4dw> (at www.3tier.com)
21:52 penglish1 apparently this will be some fairly groundbreaking science for the next few years in this area
21:53 penglish1 (so says NREL)
21:53 jclift Cool. :)
21:54 dbruhn neat!
21:54 penglish1 beats spying on people, IMHO. :-D
21:56 dbruhn totally
21:58 rjoseph joined #gluster
21:59 penglish1 I guess there are OTHER fun big data storage problems out there too...
21:59 penglish1 While I have your attention - the node on which I started the rebalance seems to know there's a rebalance going on.. specifically it is "fixing layout"
21:59 penglish1 but the other nodes don't seem to know about the rebalance
22:00 penglish1 penglish@tyr9:~$ sudo gluster volume rebalance data status
22:00 penglish1 rebalance step 1: layout fix in progress: fixed layout 571
22:00 dbruhn YOu need to run the rebalance
22:00 dbruhn sorry gotta run
22:00 penglish1 dbruhn: np, thanks for your help
22:07 joel joined #gluster
22:08 joel joined #gluster
22:08 joeljojojr joined #gluster
22:08 bet_ joined #gluster
22:10 isomorphic joined #gluster
22:11 nightwalk joined #gluster
22:12 joelwallis joined #gluster
22:15 jiqiren jclift: yea, the one in MV (walked away from computer - sorry for delay in responding)
22:15 jclift jiqiren: Damn.  I just saw a confirmation email from one of the proposed speakers that he could attend. :(
22:15 jclift jiqiren: Any idea why it was cancelled?
22:17 jiqiren jclift: nope, just got an email from john walker
22:18 Deformative joined #gluster
22:18 joeljojojr I have a 2 server replicated-distributed setup. (/data1 /data2 on each server.) One of the /data2 mirrored disks died, so I replaced it. Somehow the remaining /data2 disk became inactive. I ended up with inconsistent data. Through various rsync panicking, I am now in a position where I just want any files on EITHER disk in the mirror to be mirrored back to the other. Would deleting the .glusterfs directory on both sides of the mirror (then restarting) give
22:18 joeljojojr what I want, or would that be even more catastrophic?
22:18 badone joined #gluster
22:19 jiqiren joeljojojr: replicated-distributed with 4 bricks on 2 servers doesn't seem right
22:20 joeljojojr Each server has /data1 and /data2. data1 is mirrored between the 2 servers, and data2 is mirrored between the two servers. They are then combined into 1 distributed volume.
22:25 forest joined #gluster
22:29 genesis66 joined #gluster
22:29 cicero it sounds like more catastrophic
22:29 cicero if you deleted .glusterfs
22:30 cicero you should definitely make a copy somewhere, immediately
22:30 cicero in case you lose all the gluster data
22:30 cicero why are they combined into a distributed volume?
22:33 genesis66 hi, we setup 3 physical servers and 1 volume (replica 3) so that each server get a replica of the data. Data are published to end-user using samba running on virtual machines. One physical server is now offsite and connecting to the 2 others via a vpn. then smb browsing get dramastically slow when this "node" is "connected" on gluster... then normal when "disconnected"...any ideas would be very helpfull
22:43 yinyin joined #gluster
22:48 fcami joined #gluster
22:48 arusso joined #gluster
22:50 joeljojojr The problem with the copy is that we have millions of files on the cluster. Takes forever to backup and this is production (I know, not ideal, and I have been trying to make geo-replication work well for a long time, but it's been problematic.) Would deleting the .glusterfs directory cause all files to be deleted? (Basically, I had 4 disks and 2 servers. This allowed me to have both redundancy, and a larger volume.)
22:51 premera joined #gluster
22:54 __Bryan__ joined #gluster
22:59 yinyin joined #gluster
23:16 clag_ joined #gluster
23:37 bulde joined #gluster
23:38 joeljojojr Is there any way to tell my replicated volume to ignore the .glusterfs grids? I just want everything that is currently on the brick to be marked as good and replicate to the other brick.
23:38 joeljojojr gfids, not grids.
23:39 semiosis wipe out the bad brick?
23:40 joeljojojr Problem is that my .glusterfs directory on the "good" brick is messed up due to some bad decisions. The bad brick was already cleared. As a result, good files exist on the "good" brick, but it won't replicate them because they don't have a gfid in the .glusterfs directory.
23:46 semiosis hmmm
23:46 semiosis have you tried 'gluster volume heal $volname force' ?
23:50 joeljojojr No. I haven't. I can't find what that is supposed to do.
23:58 semiosis i might have that command wrong, maybe it was full not force, not sure
23:59 semiosis how about the article about healing ,,(split brain) on 3.3?
23:59 glusterbot I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
23:59 semiosis ,,(split-brain)
23:59 glusterbot (#1) To heal split-brain in 3.3, see http://goo.gl/FPFUX ., or (#2) learn how to cause split-brain here: http://goo.gl/Oi3AA
23:59 semiosis did you read that?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary