Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 yinyin joined #gluster
00:19 hagarth joined #gluster
00:20 genewitch so i can only add one peer
00:21 genewitch then any other peer i try to probe says peer on host glu4 port 0 already in peer list
00:21 genewitch if i detach the peer i can probe any other peer and have it succeed, but i cannot then add another peer.
00:21 genewitch I dunno what causes this :-(
00:24 genewitch so i went to glu2 and peer probe glu1, that worked, but it says glu3 and glu4 are part of another cluster. let me massh rm -rf /var/lib/glusterd/*
00:30 genewitch that worked.
00:30 genewitch Hooray!
00:38 genewitch okay i have 4 machines with 4 drives each, i want stripe and replicated, bricks are /glu/brick[1-4] on each box. I am reading that distributed striped replicated volumes aren't supported unless you're doing mapreduce, so what should i do?
00:39 genewitch each drive is 350GB, i want to end up with ~3TB of total storage space when i am done
00:39 hagarth joined #gluster
01:04 robo joined #gluster
01:07 yinyin joined #gluster
01:14 d3O joined #gluster
01:15 d3O joined #gluster
01:50 jbrooks joined #gluster
01:59 JoeJulian genewitch: Probe from inside the trusted peer group to outside. A machine can't just claim to be a peer and join itself to your trusted peer group.
02:16 doc|holliday left #gluster
02:17 d3O joined #gluster
02:24 Supermathie Does the glusterfs-ext4 bug only trigger on NFS shares?
02:25 Supermathie Oh wait, no... just the distribute translator?
02:30 jbrooks joined #gluster
03:09 Supermathie Happy to report that Discourse is now using GlusterFS in prod :)
03:13 saurabh joined #gluster
03:36 hagarth joined #gluster
03:41 shylesh joined #gluster
03:44 d3O joined #gluster
03:44 sgowda joined #gluster
03:55 vshankar joined #gluster
03:59 hchiramm_ joined #gluster
04:00 itisravi joined #gluster
04:03 jules_ joined #gluster
04:17 vpshastry joined #gluster
04:27 piotrektt joined #gluster
04:29 mohankumar joined #gluster
04:30 raghu joined #gluster
04:31 hchiramm_ joined #gluster
04:41 sgowda joined #gluster
04:43 vpshastry joined #gluster
04:46 hagarth joined #gluster
04:55 vpshastry1 joined #gluster
05:06 lalatenduM joined #gluster
05:08 hchiramm_ joined #gluster
05:13 shireesh joined #gluster
05:21 bala joined #gluster
05:38 vimal joined #gluster
05:44 rotbeard joined #gluster
05:47 JoeJulian Woot!
05:49 samppah ?
05:50 JoeJulian Responding about 2:40 late to Supermathie
05:50 samppah :O
05:53 bala joined #gluster
06:02 Nevan joined #gluster
06:07 guigui1 joined #gluster
06:10 Supermathie JoeJulian: :)
06:21 satheesh joined #gluster
06:22 vpshastry joined #gluster
06:23 Supermathie So yeah... as far as configuration errors go... setting 9000 MTU on one node and 1500 MTU on the other node does pretty bad things to gluster.
06:24 JoeJulian Hmm, it shouldn't. Maybe it's switch dependent.
06:25 Supermathie Naw, at the network level that'll mess anything up.
06:25 JoeJulian It should get negotiated at the tcp setup.
06:26 samppah Supermathie: what kind of bad things?
06:27 Supermathie JoeJulian: Don't think that would get negotiated then. I'm not totally sure though.
06:28 Supermathie samppah: lots of files getting out of sync
06:28 samppah oh, so it's not working at all?
06:28 Supermathie Maybe it wasn't that... still getting lots of:
06:28 Supermathie [2013-04-16 02:28:25.140129] W [client3_1-fops.c:707:client3_1_truncate_cbk] 0-gv0-client-10: remote operation failed: Permission denied
06:29 Supermathie [2013-04-16 02:28:25.140294] W [client3_1-fops.c:707:client3_1_truncate_cbk] 0-gv0-client-11: remote operation failed: Permission denied
06:29 Supermathie [2013-04-16 02:28:25.140787] W [nfs3.c:889:nfs3svc_truncate_cbk] 0-nfs: b7ac9614: /fleming1/db0/ALTUS_flash/archivelog/2013_​04_16/.o1_mf_1_1035__1366093663121884_.arc => -1 (Permission denied)
06:29 Supermathie [2013-04-16 02:28:25.147221] E [nfs3.c:3408:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.10.3:64443) gv0 : 29260dff-2f1c-4cf8-8007-a62f9f190cd9
06:30 vshankar joined #gluster
06:33 Supermathie [2013-04-16 02:33:38.954023] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gv0-replicate-2: no active sinks for performing self-heal on file <gfid:53c40c8c-b549-4e2f-9a55-e2a3c88edde2>
06:34 Supermathie is that... can't talk to the brick?
06:35 Nagilum_ "gluster peer status" would tell you
06:36 Supermathie All status is OK. But:
06:37 Supermathie after running some tests, I now have 78 files that need healing. Before, it was 0.
06:37 JoeJulian Do you still have that one server hitting 100%?
06:37 bala joined #gluster
06:37 Supermathie It's better now, only up to 50%. I think a lot of the I had earlier was the MTU issue.
06:38 Supermathie I'm rerunning my tests without DNFS turned on - making sure my changes didn't mess things up
06:39 Supermathie It's hitting 40% (I also turned off the stats collection)
06:39 vshankar joined #gluster
06:40 Supermathie That particular brick holds the users tablespace, so it's getting a lot of traffic.
06:40 Supermathie One file... *that* I should stripe.
06:41 JoeJulian I wonder if packet fragmentation is disabled. It should be.
06:41 Bonaparte left #gluster
06:42 Supermathie Yeah, there's something about the sequence of Oracle's DNFS calls that is messing up gluster - not sure which end is to blame.
06:42 Supermathie You need to hit a router to get fragmentation
06:47 aravindavk joined #gluster
06:51 bulde joined #gluster
06:54 Supermathie Ah well, I'll hammer on this tomorrow.
06:57 ngoswami joined #gluster
07:01 ctria joined #gluster
07:02 ollivera joined #gluster
07:04 andreask joined #gluster
07:10 ujjain joined #gluster
07:17 rb2k joined #gluster
07:22 bala joined #gluster
07:23 tjikkun_work joined #gluster
07:23 d3O joined #gluster
07:26 hchiramm_ joined #gluster
07:44 zykure|uni joined #gluster
07:50 bala joined #gluster
07:55 dobber_ joined #gluster
07:58 dobber joined #gluster
08:06 spider_fingers joined #gluster
08:08 hchiramm_ joined #gluster
08:33 itisravi joined #gluster
08:34 rastar joined #gluster
08:48 joehoyle- joined #gluster
09:10 fleducquede joined #gluster
09:25 ricky-ticky joined #gluster
09:48 saurabh joined #gluster
09:59 edward1 joined #gluster
10:23 puebele1 joined #gluster
10:29 mariusp_ joined #gluster
10:32 samppah any estimate for 3.4 beta?
10:32 samppah having problems with rebalance on 3.4 alpha2
10:37 satheesh joined #gluster
10:41 Unidentified4773 joined #gluster
10:43 puebele1 joined #gluster
11:02 duerF joined #gluster
11:12 vrturbo joined #gluster
11:13 GLHMarmot joined #gluster
11:15 vrturbo when using transport method RDMA , is the connection brick to brick and brick to client RDMA ?
11:22 H__ samppah: what rebalance problems ?    ( ~shiver~ )
11:23 itisravi joined #gluster
11:26 samppah H__: i add 2 bricks to replicated volume and start rebalance after that.. it seems to migrate couple files to new bricks and after that it stops
11:27 samppah rebalance status shows "in progress
11:27 samppah and this is in rebalance log on another node: [2013-04-16 10:14:23.376106] W [glusterfsd.c:970:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7fc9c39b590d] (-->/lib64/libpthread.so.0(+0x7851) [0x7fc9c4001851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40523d]))) 0-: received signum (15), shutting down
11:28 itisravi joined #gluster
11:28 vpshastry2 joined #gluster
11:30 itisravi joined #gluster
11:33 kkeithley1 joined #gluster
11:39 vrturbo joined #gluster
11:44 flrichar joined #gluster
11:46 hagarth joined #gluster
11:56 dustint joined #gluster
12:00 guigui1 joined #gluster
12:04 glusterbot New news from newglusterbugs: [Bug 950024] replace-brick immediately saturates IO on source brick causing the entire volume to be unavailable, then dies <http://goo.gl/RBGOS>
12:04 hchiramm__ joined #gluster
12:07 chirino joined #gluster
12:08 bet_ joined #gluster
12:25 puebele2 joined #gluster
12:40 jurrien_ joined #gluster
12:43 jurrien_ Hi, I'm trying to solve a split-brain situation but as far as I can see this is always a manual action. Is this correct? Is there also a way to automate this for example the last file written is always leading or something?
12:44 77CAAYFZV joined #gluster
12:45 sjoeboo_ joined #gluster
12:57 manik joined #gluster
12:57 vpshastry joined #gluster
13:03 jskinner_ joined #gluster
13:04 jdarcy joined #gluster
13:04 jag3773 joined #gluster
13:06 jskinner_ So I am still having issues getting my NFS client to talk to gluster without specifying nfsvers=3. I am thinking about just mounting the volume on the gluster box, and exporting it out with NFS on the box instead of gluster. What do you guys think about doing that?
13:08 robo joined #gluster
13:10 Norky that sounds possible
13:11 Norky the real answer is just mount on the client with vers=3
13:11 jskinner_ I know - but this NFS share is for cloudstack
13:11 jskinner_ and I dont have any control over how it mounts the share
13:12 jskinner_ and it is not passing vers=3
13:12 ndevos jskinner_: you could set the default in /etc/nfsmount.conf
13:12 jdarcy You *can* re-export a GlusterFS volume using the in-kernel NFSv4 server, but IIRC there might have been some extra steps involved.
13:13 jskinner_ the client that mounts the share does not have that file
13:13 ndevos and, you will want to read /usr/share/doc/fuse-2.8.3/README.NFS
13:13 jskinner_ its a very minimal debian squeeze box
13:13 ndevos ah, I dont know what nfs-utils carries that configfile
13:14 jdarcy ndevos: Have I ever complimented you on your encyclopedic knowledge?
13:14 ndevos jdarcy: Not that I can remember!
13:14 jskinner_ ive been working on this for the past day and its driving me nuts. If I had more control over the debian box I would be golden.
13:15 jdarcy ndevos: OK.  Dude, you're like a walking encyclopedia.  How awesome.  ;)
13:15 ndevos \o/ you made my day, jdarcy
13:15 kkeithley| He's European, IIRC over there  it's encyclopaedic
13:16 jdarcy kkeithley: I actually typed it that way at first, then un-corrected myself.
13:16 kkeithley| ;-)
13:16 kkeithley| that's your inner Kiwi at work
13:17 jdarcy I still have to fight the urge to type "our" instead of "or" at the end of some words.
13:17 kkeithley| how about practise and defense?
13:18 jdarcy kkeithley: Practice and defense.  I never said I was consistent.  ;)
13:19 jskinner_ @ndevos, I don't appear to have that fuse doc on my servers
13:20 hagarth joined #gluster
13:20 jdarcy Also, a-loo-mi-num and loo-tenant, not al-you-minium and left-tenant.
13:20 ndevos jskinner_: http://paste.fedoraproject.org/7646/11839913
13:20 kkeithley| http://review.gluster.org/#/c/4840/ looks like we're finally coming around to the idea of running nfs on the regular ports. Back in http://review.gluster.org/#/c/2809/ it got shot down
13:20 glusterbot Title: #7646 Fedora Project Pastebin (at paste.fedoraproject.org)
13:20 glusterbot Title: Gerrit Code Review (at review.gluster.org)
13:20 jdarcy "Lieutenant" was the only spelling word I ever got wrong in school.  :(
13:22 jskinner_ ah ok, thanks.
13:22 kkeithley| English is tough stuff. http://www.frivolity.com/teatime/Songs​_and_Poems/english_is_tough_stuff.html
13:22 glusterbot <http://goo.gl/MCeHF> (at www.frivolity.com)
13:23 jskinner_ hmmm
13:23 jskinner_ kkeithleyl - can I manually change the NFS port for Gluster?
13:24 kkeithley| yes — if you're willing to recompile the source.
13:25 jskinner_ hmm
13:25 jskinner_ ok, well I've got some options now lol.
13:25 jskinner_ Is that change for 3.4?
13:26 ndevos "gluster volume set MYVOL nfs.port 2049" should work
13:26 jskinner_ if that works
13:26 jskinner_ I am going to go home for the day
13:26 jskinner_ lol
13:27 kkeithley| ndevos, the walking encyclopaedia FTW.
13:27 * ndevos knows it works in 3.3
13:27 kkeithley| you are awesome
13:29 * ndevos opens the index and looks for coffee
13:29 kkeithley| I was sitting here thinking it ought to be an option——
13:29 jskinner_ dude, you are awesome
13:29 jskinner_ that worked like a charm
13:29 kkeithley| and it is
13:30 jskinner_ I feel so much better about this day now lol
13:31 jurrien_ Does ndevos maybe have a answer to my question? :)
13:32 gbrand_ joined #gluster
13:34 ndevos jurrien_: yes: no, because not always the last writes are the imortant criteria to decide what the 'good' copy is
13:34 glusterbot New news from newglusterbugs: [Bug 895528] 3.4 Alpha Tracker <http://goo.gl/hZmy9> || [Bug 918917] 3.4 Beta1 Tracker <http://goo.gl/xL9yF>
13:35 jurrien_ But I wanted to be that way?
13:35 jurrien_ +if
13:35 puebele1 joined #gluster
13:36 jurrien_ I'm building a proof of concept with gluster for the home dirs on the linux workstations
13:37 jurrien_ I'm know testing split brain situations and I was wondering if there where self healing tools
13:37 ndevos you probably could script that, but how would you handle directories?
13:37 jurrien_ Merge them I guess
13:38 rgustafs joined #gluster
13:38 jurrien_ Also looking at the possibility to use each work station as a gluster host with a local partition as a brick
13:38 ndevos and the xattrs, permissions and owernship?
13:42 jurrien_ How do you mean ndevos ?
13:43 ndevos jurrien_: well, when you have a split brain on a directory, it is likely that xattrs, permissions, ownership or other attributes of that directory have changed
13:44 ndevos if you want to script a split-brain resolver, you will have to decide which is the 'good' dir, the mtime will have changed when something inside the dir was changed, so thats not a good item to check
13:44 jurrien_ Well they are home dirs... So after /home/<username> you should know who the owner is. And if permissions are incorrect then that's a manual action for the user I guess
13:45 jurrien_ But I understand that this is a problem where the solution is different for each admin
13:45 ndevos sure, but someone will need to resolve the split-brain on the brick(s) - and users can have shared directories as well
13:46 ndevos probably logging what is done to resolve the split-brain and be able to review the changes would be helpful
13:48 ndevos I guess you understand now why there is no 'default' option to resolve split-brains, it can be dangerous to delete files and/or clear directories - rather be safe and have a sysadmin decide
13:49 jurrien_ but for example in a raid 1 config of disks. Ok this is a complete other method of data. But if disk are out of sync the system desides that one the disk is more up to date then the other.
13:49 jurrien_ of +handeling data
13:50 jurrien_ I'm not saying that this should be build :-)
13:50 jurrien_ Such feature
13:50 ndevos yes, but it is also much less likely that those two disks get changed independently from each other - I think
13:51 jurrien_ Offcourse
13:52 jurrien_ But thanks for the answers. :-)
13:53 ndevos well, I think a script that pick the file that has been updated last, merges directories and fixes the split-brain in a general way could be helpful
13:53 ndevos if you decide to write something, I'm sure others would be interested in it - even if it is for analyzing the split-brain without doing any actions
13:53 guigui3 joined #gluster
13:54 jurrien_ How much is it a performance hit to make each desktop a gluster host with a brick and replicate that brick to each other workstation and mount /home on each workstation to the localhost:/volume ?
13:54 jurrien_ If I'm going to write such script I will share it!
13:54 jurrien_ You got my word.
13:54 puebele1 joined #gluster
13:54 nueces joined #gluster
13:55 * jurrien_ supporter of the open source community :-)
13:56 ndevos :-)
13:56 mariusp_ joined #gluster
14:00 hybrid5123 joined #gluster
14:04 glusterbot New news from newglusterbugs: [Bug 918917] 3.4 Alpha3 Tracker <http://goo.gl/xL9yF> || [Bug 952693] 3.4 Beta1 Tracker <http://goo.gl/DRzjx>
14:11 aliguori joined #gluster
14:12 mariusp joined #gluster
14:14 spider_fingers left #gluster
14:21 daMaestro joined #gluster
14:31 lh joined #gluster
14:31 lh joined #gluster
14:31 theron joined #gluster
14:34 sjoeboo_ joined #gluster
14:35 andreask joined #gluster
14:40 bugs_ joined #gluster
14:44 manik joined #gluster
14:56 jikz_ joined #gluster
14:56 jikz joined #gluster
14:57 balunasj joined #gluster
14:58 jikz joined #gluster
15:09 kkeithley1 joined #gluster
15:10 doc|holliday joined #gluster
15:11 doc|holliday newb problem, I created volume with 1 brick on server1, then I peer probe server2 and try to add-brick from server2, but it says 'Host server2 not connected'
15:12 doc|holliday peer status shows that it's connected
15:13 doc|holliday on server2 peer status also says that server1 is connected (although hostname shows ip address instead of server name)
15:13 theron joined #gluster
15:35 glusterbot New news from newglusterbugs: [Bug 949406] Rebalance fails on all the nodes when glusterd is down on one of the nodes in the cluster <http://goo.gl/Q8dyW>
15:40 ndevos ~hostnames | doc|holliday
15:40 glusterbot doc|holliday: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
15:43 chirino joined #gluster
15:46 doc|holliday well, I detached the peer and probed it again and now under state tsays "Peer in Cluster", whereas earlier it was saying "Accepted peer request", which googling has revealed was not what I wanted to see
15:46 _pol joined #gluster
15:47 doc|holliday but now tsays that {path} or a prefix of it is already part of a volume
15:47 glusterbot doc|holliday: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
15:47 _pol joined #gluster
15:50 doc|holliday glusterbot: ssup
15:50 glusterbot doc|holliday: I do not know about 'ssup', but I do know about these similar topics: 's3'
15:50 doc|holliday :)
15:56 Supermathie Do sequence diagrams exist of the process for sync and async writes for NFS writes and/or GlusterFS client writes?
15:57 jclift_ joined #gluster
16:02 _pol joined #gluster
16:03 _pol joined #gluster
16:06 doc|holliday ok that worked. thanks ndevos
16:06 doc|holliday and thanks glusterbot, if you care :)
16:06 ndevos you're welcome, doc|holliday
16:07 hagarth joined #gluster
16:08 doc|holliday am I correct in assuming that with pure distributed system overhead/delay on reads/writes to/from the cluster is minimal?
16:10 semiosis please define terms overhead, delay, and minimal ;)
16:13 doc|holliday I have a network with 20 servers, 300+ clients writing video streams to servers and 100+ streams reading said streams
16:14 doc|holliday currently I am using in-house software to determine where each client is supposed to write or read the streams from
16:15 doc|holliday I'd like to replace it with glusterfs and treat all servers as one cluster
16:16 doc|holliday by overhead/delay I mean accessing the cluster vs directly accessing the server (via NFS or CIFS)
16:16 doc|holliday semiosis: ^
16:17 Supermathie doc|holliday: I think you might want to read up on the translators: http://www.gluster.org/community/d​ocumentation/index.php/Translators - they're responsible for directing/balancing reads/writes (right? :) )
16:17 glusterbot <http://goo.gl/APZ49> (at www.gluster.org)
16:19 * doc|holliday goes to read up on xlators
16:20 semiosis that might be a distraction :)
16:21 semiosis doc|holliday: i'd recommend trying things to see what works best
16:22 semiosis yours sounds like a good use case for glusterfs
16:22 semiosis s/good/great/
16:22 semiosis glusterbot: meh
16:22 glusterbot semiosis: I'm not happy about it either
16:22 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
16:24 doc|holliday semiosis: I am prepping my server and client images now and will try it tonight or tomorrow :)
16:24 semiosis doc|holliday: furthermore, imho overhead/delay shouldn't be a primary concern. can your hardware deliver the performance required to do the job or not? does glusterfs offer other benefits over the alternatives?  thats how i look at it
16:27 ctria joined #gluster
16:27 Supermathie semiosis: On that note, if I've got a single 3.2TB gluster brick and want to configure gluster to be able to drive maximum IO to it, where should I look for tuning knobs? So far I've found performance.io-thread-count.
16:28 doc|holliday last question (hopefully): is there an option to encrypt data between the servers and the clients? or do I need to use some other solution for that?
16:29 semiosis Supermathie: very rarely (in my experience) do glusterfs configuration options have any impact on performance.  the one exception is the self heal algorithm, for my use case full is way faster than diff.
16:30 semiosis doc|holliday: some work was done on encryption both at-rest and in-flight on ,,(hekafs) but that fork hasn't been reintegrated into glusterfs yet
16:30 glusterbot doc|holliday: CloudFS is now HekaFS. See http://hekafs.org or https://fedoraproject.org/wiki/Features/CloudFS
16:35 Supermathie semiosis: Pretty sure what I'm working on is an atypical configuration. Working with high performance SSDs and a ioScale card.
16:35 semiosis Supermathie: if you've got a single brick, why are you even using glusterfs?
16:36 Supermathie semiosis: I'm testing out that configuration along with others. Also: replication
16:36 Mo____ joined #gluster
16:44 shylesh joined #gluster
16:47 barking|spiders joined #gluster
16:49 barking|spiders left #gluster
16:53 jclift_ joined #gluster
16:54 doc|holliday semiosis: thank you sir. I will go ahead and try gluster by itself and maybe later pass it through openvpn
16:59 jdarcy joined #gluster
17:00 Supermathie doc|holliday: what about setting up ipsec between the nodes?
17:05 glusterbot New news from newglusterbugs: [Bug 874498] execstack shows that the stack is executable for some of the libraries <http://goo.gl/NfsDK>
17:07 doc|holliday Supermathie: I believe ipsec is more complicated to set up. I've done a few openvpn setups already and felt more comfortable with it. but maybe I should look at it again
17:07 JoeJulian openswan (ipsec) is actually pretty easy to set up, just complicated to actually understand.
17:08 Supermathie doc|holliday: machines on same layer2?
17:08 doc|holliday Supermathie: yes
17:12 Supermathie doc|holliday: IPv6? :)
17:12 doc|holliday Supermathie: no
17:14 manik1 joined #gluster
17:14 Supermathie awwwwwwww http://www.tldp.org/HOWTO/Linux+IPv6-HOWT​O/chapter-encryption-authentication.html
17:14 glusterbot <http://goo.gl/aHUVy> (at www.tldp.org)
17:15 Supermathie Although, I will admit, existing documentation really REALLY sucks.
17:16 doc|holliday Supermathie: this is genius
17:16 doc|holliday once I get my gluster running, I will give it a try
17:17 chirino joined #gluster
17:20 doc|holliday JoeJulian: I guess what scared me off from openswan was the Gentoo HowTo, which seemed very complicated. I guess this will be my next option after ipv6
17:20 JoeJulian Yeah, learning ipv6 is time better spent. We need to get there eventually.
17:21 doc|holliday true that :)
17:21 Supermathie Good IPv6 resource: http://ipv6.he.net/certification/
17:21 glusterbot Title: Hurricane Electric IPv6 Certification (at ipv6.he.net)
17:21 JoeJulian +1
17:21 Supermathie Also, free IPv6 connectivity :)
17:23 ujjain2 joined #gluster
17:23 gbrand_ joined #gluster
17:24 doc|holliday ok thanks guys, time for me to get my hands dirty. ttyl
17:39 sjoeboo_ joined #gluster
17:43 sa__ joined #gluster
17:46 tyl0r joined #gluster
17:47 lh joined #gluster
17:47 lh joined #gluster
18:09 semiosis sa__: in channel, please
18:09 semiosis [13:44] <sa__> any idea when 3.4 for ubuntu will be available on launchpad ppa ?
18:11 semiosis sa__: i have been having some trouble with launchpad, sorry for the delay.  i do intend to get the qa/alpha/beta releases up there, but have not had the time to sort out the issue
18:13 semiosis i have engaged #launchpad for help, fwiw
18:14 sa__ thanks for the update
18:14 semiosis yw
18:17 genewitch how do i see how much total space i am exporting without attaching a client?
18:18 joehoyle joined #gluster
18:19 genewitch also if i have more clients doing writes will i acheive higher overall throughput, or should i just use 1 or 2 clients
18:30 awickhm joined #gluster
18:31 stderr joined #gluster
18:31 Guest15043 go a question for you guys, setup a basic 2 node gluster setup with 2 replicated bricks, trying to mount it on a windows box but i cant write to it
18:32 Guest15043 ive tried googling but failed, where can i fix the permissions?
18:36 genewitch Woooooot, 3.3TB
18:36 genewitch that looks right :-)
18:38 joehoyle joined #gluster
18:38 genewitch wow, doing an ls takes a while :-(
18:39 _ilbot joined #gluster
18:39 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
18:41 Guest15043 is there an /etc/exports file specifically for gluster?
18:43 genewitch what
18:43 genewitch there are ACLs
18:43 genewitch POSIX or something
18:43 Guest15043 out of the box setup, cant write to my newly created volume from a windows box, anyone know why?
18:44 Guest15043 via cifs or nfs
18:44 JoeJulian Guest15043: the volume name is the export there is no "exports" file.
18:44 Guest15043 yes, but for options
18:44 JoeJulian via cifs... have you installed and configured samba to share the volume?
18:44 Guest15043 i see the volume name, i can mount it
18:44 Guest15043 but again, i cannot write to it
18:45 Guest15043 well, thats kinda what I'm cludgey on... do i need to mount the gluster share locally to the linux box and THEN share it out via cifs or nfs?
18:45 JoeJulian I'm guessing it's the standard posix permissions.
18:45 Guest15043 i guess i was under the assumption I could directly mount it without having to do that
18:45 genewitch is it normal for a gluster client box to have 50% cpu on the gluster process?
18:46 JoeJulian Guest15043: For samba, yes. For nfs, no.
18:46 Guest15043 ok, so nfs mount... how do i configure the write permissions then?
18:46 genewitch the client isn't doing anything, according to the logs
18:46 JoeJulian genewitch: Mine are all sitting around 2%.
18:46 JoeJulian genewitch: Are your bricks ,,(ext4)?
18:46 glusterbot genewitch: (#1) Read about the ext4 problem at http://goo.gl/xPEYQ or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
18:47 genewitch JoeJulian: I just mounted the gluster mount, and touched a file. i watched the file replicate, but the client process is still sitting at 50%
18:47 genewitch and that's 25% user and 25% sys
18:48 genewitch JoeJulian: OH. they might be ext4.
18:48 genewitch let me check the script
18:49 genewitch JoeJulian: recommendation? xfs? ext3?
18:49 genewitch butterfuss
18:51 Guest15043 any idea why i cant write to the nfs share from a windows server?
18:53 JoeJulian Guest15043: Yes
18:53 JoeJulian [11:45] <JoeJulian> I'm guessing it's the standard posix permissions.
18:53 Guest15043 yeah, ive already tried that
18:53 Guest15043 didnt change anything
18:53 y4m4 joined #gluster
18:55 JoeJulian Can you mount the volume using fuse locally and write to it using the windows user?
18:57 Guest15043 i can mount it locally on the linux box and write just fine
18:59 JoeJulian Sounds like your volume's installed correctly then and works, which brings us to windows and the extent of my knowledge. I haven't, nor do I ever intend to, use windows with nfs.
19:00 JoeJulian Since your company's willing to spring for MS licensing, have you considered a ,,(commercial) purchase?
19:00 glusterbot Commercial support of GlusterFS is done as Red Hat Storage, part of Red Hat Enterprise Linux Server: see https://www.redhat.com/wapps/store/catalog.html for pricing also see http://www.redhat.com/products/storage/ .
19:01 Guest15043 we've tried DFS but it seemed flakey
19:01 JoeJulian Here in the IRC channel, we're ,,(volunteers)
19:01 glusterbot A person who voluntarily undertakes or expresses a willingness to undertake a service: as one who renders a service or takes part in a transaction while having no legal concern or interest or receiving valuable consideration.
19:01 Guest15043 so im better off going with CIFs then?
19:02 JoeJulian That's what I do and it meets my ,,(performance metric).
19:02 glusterbot I do not know about 'performance metric', but I do know about these similar topics: 'Joe's performance metric'
19:02 JoeJulian @'Joe's performance metric
19:02 JoeJulian @Joe's performance metric
19:02 glusterbot JoeJulian: nobody complains.
19:02 * JoeJulian sighs... way too much work for the punch line.
19:04 genewitch heh
19:11 sefz joined #gluster
19:14 sefz hi, what is considered to be a better practice, for using gluster on amazon with EBS drives... using each of them as a brick, (let's say 3 machines with 4 ebs each) and create a gluster volume with 12 bricks
19:14 sefz or creating a logical raid volume (madm) and using only 3 big bricks ?
19:15 sefz i have terrible performance using gluster with 3 server and 12 ebs drives... writing/reading is stuck at 20mb/s
19:16 semiosis i've expressed my opinion on that enough to have a ,,(canned ebs rant) :)
19:16 glusterbot http://goo.gl/GJzYu
19:17 genewitch okay i reformatted everything to xfs
19:17 genewitch only took 20 minutes
19:18 JoeJulian That's what... 40 bricks if you typed the commands for each one individually?
19:21 sefz so semiosis, you basically encourage the use of the ephemereal space for glusterfs?
19:21 semiosis i what?!
19:21 sefz on the canned ebs rant page
19:22 semiosis i dont say that anywhere
19:23 semiosis and it wasnt even your question, which i was responding to, about striping ebs volumes
19:23 * semiosis mystified
19:25 genewitch JoeJulian: 16 bricks and i had to reboot to clear mtab and delete /var/lib/glusterd and whatnot. also i had to commit the script to do it, which is a PITA on the dev box i have
19:25 genewitch 20 minutes isn't too too bad
19:25 genewitch sefz: i'm using ephemeral for gluster
19:25 genewitch right nao
19:25 genewitch 3.1TB stripe 2 replica 2
19:26 sefz which instance size?
19:26 semiosis sefz: how are you testing performance anyway?  getting that 20mb/s with dd?
19:27 sefz yes
19:27 sefz big files / 1GB with 1mb blocks
19:27 sefz maybe i should not even try with m1.small.. but anyway the cpu is around 2% when starting dd
19:28 semiosis dd is an awful benchmark
19:28 sefz is there something better?
19:28 semiosis not an awful load generator if used appropriately
19:28 sefz i also tried creating a random file with dd , and manually time cp file /gluster/
19:28 sefz takes same time
19:29 semiosis how about doing 10x at a time
19:29 semiosis do you get 10x the performance?
19:29 semiosis s/performance/throughput/
19:29 glusterbot What semiosis meant to say was: do you get 10x the throughput?
19:29 Supermathie Someone really ought to put together a nice fio load generator to test performance... it is on my list :)
19:29 failshell joined #gluster
19:30 sefz didn't try... but
19:30 failshell hello. what's a good backup strategy for a glusterfs cluster?
19:30 semiosis is there a good benchmark?  not in my opinion :)
19:30 sefz what if I want to read a big file fast?
19:30 genewitch failshell: geo-replication?
19:30 genewitch sefz: stripe
19:30 JoeJulian No, not ,,(stripe)
19:30 glusterbot Please see http://goo.gl/5ohqd about stripe volumes.
19:30 sefz ok, what if it's already stripe-replicated^^ ?
19:31 genewitch stripe isn't fast reads?
19:31 genewitch My worldview is collapsing
19:31 semiosis sefz: ebs optimized instances & ebs provisioned iops
19:31 genewitch ebs is for non-engineers
19:31 genewitch afk
19:31 JoeJulian genewitch: Excellent! Now we're making progress.
19:31 sefz did two tests with dd, one in local EBS brick, got 50mb/s, one in the glusterfs with 12 bricks, got 12mb/s .
19:31 sefz sorry, 20mb/s
19:32 semiosis comparing a distributed cluster filesystem to a local block device isn't helpful
19:34 _NiC joined #gluster
19:34 sefz ok but, something is wrong with the 20mb/s speed i'm getting... i don't understand if it's due to m1.small, due to misconfiguration, or something else
19:34 semiosis stripe won't help a single reader to read faster (unless that reader is reading different parts of the file at the same time)
19:34 sefz or bad practice...
19:35 semiosis the first is easy to test, try with larger instances
19:36 semiosis i settled on m1.large
19:36 sefz ok, trying right now..
19:36 semiosis going beyond that didnt get me any better performance (two years ago when i did all the experiments)
19:36 sefz usually, the default configuration is sufficient, right?
19:36 sefz i mean, on volume creation...
19:36 semiosis things may be different now though, there have been lots of changes in ec2 since then
19:36 sefz am I supposed to tweak chunk size/stripe size and so on?
19:36 semiosis i use defaults
19:37 semiosis and i dont use stripe, recommend against it
19:37 semiosis unless you've already exhausted all other options
19:37 sefz i know but it's a big requirement (they may have file of several TB)
19:39 andreask joined #gluster
19:44 ricky-ticky joined #gluster
19:45 edward1 joined #gluster
19:47 sjoeboo_ joined #gluster
19:51 edward1 joined #gluster
19:58 _pol joined #gluster
20:00 77CAAYIXX joined #gluster
20:00 CROS_ joined #gluster
20:02 dustint joined #gluster
20:05 CROS_ left #gluster
20:06 sjoeboo_ joined #gluster
20:06 failshell i changed a node's ip. now, it shows up as being disconnected. but i configured my cluster to only use FQDNs.
20:06 failshell not sure what's going on here
20:07 gbrand_ joined #gluster
20:14 H__ failshell: do all other nodes show that peer as fqdn too or do some have its old IP address ?
20:15 failshell H__: old IP
20:15 failshell I basically need to move the cluster to a new subnet
20:16 semiosis failshell: ,,(hostnames)
20:16 glusterbot failshell: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
20:16 H__ ah, i was wondering if that helped
20:17 failshell its not updating it
20:18 failshell i now have a new peer, with a 0000000 uuid
20:18 failshell and the old one is still there disconnected
20:19 semiosis restart glusterds
20:19 semiosis (?)
20:21 failshell nope
20:24 failshell ah it seems that reprobing put that peer in limbo
20:24 failshell after removing it and restarting glusterds
20:24 failshell it fixed itself
20:26 rb2k joined #gluster
20:27 manik joined #gluster
20:28 jdarcy joined #gluster
20:40 semiosis failshell: sweet
20:40 failshell well, im not sure what worked
20:40 semiosis probing it by name then restarting the glusterds, of course :)
20:59 failshell semiosis: weird thing though, 2 nodes out of 4 i didnt have to do that
20:59 semiosis yep
20:59 semiosis seen that happen
20:59 semiosis idk why
20:59 semiosis sometimes glusterd needs to be restarted to notice a change, other times it just notices
21:00 failshell oh well, its moved to the right place
21:00 failshell now: backups
21:01 _pol joined #gluster
21:07 jdarcy joined #gluster
21:09 failshell clients only see half the storage now ...
21:09 failshell but gluster peer status and gluster volume info seems fine
21:13 failshell so you need to probe all nodes
21:34 rwheeler joined #gluster
21:47 sefz joined #gluster
21:49 nueces joined #gluster
21:55 fidevo joined #gluster
22:06 joehoyle joined #gluster
22:11 badone joined #gluster
22:13 joehoyle- joined #gluster
22:18 sjoeboo_ joined #gluster
22:41 genewitch does gluter not handle filenames that start with a period?
22:41 semiosis i've heard of people running homedirs on glusterfs, so that sounds unlikely
22:41 genewitch i am trying to rsync to it and .php.conf.<random> fails every time
22:41 genewitch saying invalid argument
22:42 semiosis use --inplace when rsyncing to glusterfs, it's better to avoid renames if you can
22:42 semiosis idk why its failing though
22:42 genewitch inplace
22:42 semiosis check logs if you really want to know
22:42 genewitch let me goo that
22:43 genewitch which logs?
22:43 semiosis glusterfs client logs.  are you using fuse or nfs client?
22:43 genewitch now it's failing on ~/.svn/all-wcprops
22:44 genewitch fuse
22:44 semiosis then /var/log/glusterfs/the-client-mount-point.log
22:44 semiosis feel free to pastebin it
22:44 semiosis ahem, pastebin.com
22:44 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
22:44 genewitch i'm rsycning from a remote server to a server with gluster mounted on fuse like rsync /blah/ glu-client:/DR/blah
22:44 semiosis there we go
22:44 genewitch ok
22:45 semiosis pastie.org, github gist, whathaveyou
22:46 genewitch it's large
22:47 semiosis just the end of it, with the failures for your present situation
22:47 semiosis last 100 lines should be enough
22:47 genewitch oh i see
22:48 genewitch http://bpaste.net/show/91959
22:48 glusterbot Title: Paste #91959 at spacepaste (at bpaste.net)
22:49 genewitch that's the lines it spits out when i rsync
22:50 semiosis genewitch: ,,(pasteinfo) please
22:50 glusterbot genewitch: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:50 semiosis or bpaste if you like
22:50 semiosis i dont care
22:50 semiosis just not in channel :)
22:52 genewitch http://bpaste.net/show/91960
22:52 glusterbot Title: Paste #91960 at spacepaste (at bpaste.net)
22:52 semiosis ehh ,,(stripe)
22:52 glusterbot Please see http://goo.gl/5ohqd about stripe volumes.
22:52 semiosis do you really need that?
22:53 semiosis the files you've mentioned so far (php, svn) are very much *not* the kind of thing stripe was made for
22:53 genewitch i don't know
22:53 genewitch ok
22:53 semiosis if you dont know you really need it, then you don't need it
22:53 genewitch i don't really need gluster either, i just think if it works it'll give us an excuse to use it instead of netapps
22:53 genewitch so i should change replica to 4?
22:53 semiosis haha
22:54 JoeJulian Do you plan on having 3 servers fail at the same time?
22:54 genewitch 2 out of 4 can
22:54 genewitch easily
22:54 semiosis i moved from a direct attached raid array to the cloud thanks to glusterfs.  huge win for me.  netapp was too expensive so i needed an alternative to keep growing
22:54 JoeJulian If it's that easy, then yeah. I'd probably do 4.
22:54 genewitch so last time i did this i did replica 2 but all the LVM mountpoints were RAID
22:55 genewitch i should go read how to do this
22:55 genewitch :-D
22:55 semiosis love it when people say EC2 is expensive.  they should get a netapp quote from rackspace
22:56 genewitch i hear "ec2 is expensive" all the time, usually followed by "but we don't want to reserve just yet"
22:56 semiosis but i digress
22:56 genewitch if i detach all the volumes
22:57 genewitch gluster volume delete, rather
22:57 genewitch should i reformat the discs and wipe /var/lib/glusterd?
22:57 JoeJulian Hrm... I just had something weird happen and I'm not even entirely sure what.
22:57 JoeJulian @which brick
22:57 glusterbot JoeJulian: To determine on which brick(s) a file resides, run getfattr -n trusted.glusterfs.pathinfo $file through the client mount.
22:59 genewitch massh glu{1-100} -e locate "file" :-p
22:59 JoeJulian trusted.glusterfs.pathinfo="(<DISTRIBUTE:home-dht> (<REPLICATE:home-replicate-2> <POSIX(/var/spool/glusterfs/c_home):e​wcs7:/var/spool/glusterfs/c_home/ewp> <POSIX(/data/glusterfs/home/c):ewcs10​.ewcs.com:/data/glusterfs/home/c/ewp> <POSIX(/var/spool/glusterfs/c_home):ewcs2.e​wcs.com:/var/spool/glusterfs/c_home/ewp>))"
22:59 JoeJulian ^ much more useful.
23:02 genewitch if i have four servers and two can fail, should i make replica 2 or 3?
23:02 semiosis 3 and i would recommend not doing replica 3 over 4 servers, it gets messy
23:03 semiosis but is probably possible
23:09 genewitch @space
23:10 genewitch @total space
23:10 genewitch :-(
23:31 JoeJulian Ah, that's what happened. Argh. "XFS (dm-7): xfs_iunlink_remove: xfs_inotobp() returned error 22."
23:34 JoeJulian semiosis: btw... have I mentioned lately how much I love having all my logs in one place? Thanks, whack, for logstash.
23:36 joehoyle joined #gluster
23:42 a2 JoeJulian, did you just hit the xfs_inotobp() error?
23:44 JoeJulian Yes
23:44 a2 what operations were you doing?
23:45 JoeJulian Heh, too many to list. These are home directories so I don't even know which client it was.
23:45 a2 hmph!
23:45 a2 what server OS?
23:45 JoeJulian centos 6.4 kernel 2.6.32-279.el6.x86_64
23:45 a2 ok
23:50 sjoeboo_ joined #gluster
23:52 tru_tru JoeJulian: 6.4 kernel are 2.6.32-358 series
23:52 JoeJulian Well this machine hasn't been rebooted in a while
23:52 JoeJulian Which is why I specified
23:52 tru_tru you are still running the genuine 6.3 kernel :P
23:54 JoeJulian a2: I don't suppose any of this is of any use? http://fpaste.org/mEx2/
23:54 glusterbot Title: Viewing Paste #292663 (at fpaste.org)

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary