Camelia, the Perl 6 bug

IRC log for #gluster, 2013-02-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 clusterflustered can i buy you a beer JoeJulian?
00:14 clusterflustered is this how a one does a backup as well? or is this only for offsite replication?
00:16 johnmark clusterflustered: that or if you want to replicate across a network that is unreliable
00:16 lh joined #gluster
00:21 cicero hey, you gotta get in line to buy JoeJulian a beer
00:21 cicero i owe JoeJulian and semiosis roughly a keg
00:35 andreask1 joined #gluster
00:37 JoeJulian clusterflustered: Heh, sorry was off having fun on gluster-users... Come to Cascadia IT Conference!
00:38 clusterflustered is that on the west coast
00:38 clusterflustered ?
00:39 amccloud "Request received from non-privileged port. Failing request" How do privileges work?
00:40 JoeJulian clusterflustered: yes, seattle in March.
00:41 JoeJulian amccloud: To ensure it's a root user trying to connect to the ports, the default only accepts connections from ports <= 1024.
00:42 JoeJulian http://casitconf.org/casitconf13/
00:42 glusterbot Title: Cascadia IT Conference 2013 (at casitconf.org)
00:43 JoeJulian amccloud: That's manageable through setting the option rpc-auth-allow-insecure
00:44 clusterflustered 15 and 16th is no goodfor me :(  im a wanna be guitar player, and i acutally get to play a show both those nights :(
00:46 tomsve joined #gluster
00:46 amccloud JoeJulian: Why would mount.glusterfs be using a port outside of that range?
00:50 JoeJulian I think it's because they were all in use.
00:54 JoeJulian clusterflustered: There's gigs out there??? Looks a bit sparsely populated...
00:56 clusterflustered that's why they let folks like me play. this area doesn't cultivate the arts so much, so there isn't much competition. during sundance, we all play.
00:56 amccloud JoeJulian: Does everything look fine here? https://gist.github.com/5e32b0c6bc4ac0c14c34
00:56 glusterbot Title: gist:5e32b0c6bc4ac0c14c34 (at gist.github.com)
00:56 JoeJulian Well I hope to make a circuit of all our stores this summer some time. Maybe you can come down to Idaho Falls when I'm ou tthere.
00:57 clusterflustered i love idaho.
00:57 JoeJulian I tolerate it.
00:57 JoeJulian amccloud: Yep, looks right.
00:57 clusterflustered any excuse i can find to get there, i go.  great folks, great paragliding, great camping and hiking, great back country skiing. love it.
00:59 amccloud JoeJulian: The error i'm seeing currently. https://gist.github.com/91b3c6cbb027d363b93a
00:59 glusterbot Title: gist:91b3c6cbb027d363b93a (at gist.github.com)
01:00 JoeJulian which one's .3?
01:01 amccloud prod-web01.dwp.net
01:02 JoeJulian is glusterd running on web01?
01:02 bauruine joined #gluster
01:02 amccloud yes
01:03 x4rlos2 joined #gluster
01:04 semiosis mmmm beer
01:06 JoeJulian check /var/log/glusterfs/etc-glusterd.vol.log on web01 and see if there's any clues there as to why the client's showing "Transport endpoint is not connected"
01:06 badone joined #gluster
01:07 semiosis JoeJulian: what log?
01:08 amccloud JoeJulian: Oh that's where I got "Request received from non-privileged port. Failing request" from.
01:13 JuanBre joined #gluster
01:15 JoeJulian amccloud: Are you mounting from a non-root user?
01:16 amccloud JoeJulian: Nope, root.
01:16 JoeJulian semiosis: Do you know, could this be a vagrant thing?
01:16 semiosis wasnt paying attention, scrolling back
01:16 amccloud JoeJulian: I've had it working in Vagrant before.
01:16 amccloud I'm not sure what has changed.
01:18 semiosis can you pastie the log with the error please?
01:18 semiosis and give me a brief recap of the issue
01:21 JoeJulian essentially he's trying to mount a volume but glusterd's refusing the connection from a non-privileged port.
01:21 JoeJulian amccloud: nat?
01:22 amccloud semiosis: all the info aggregated into one gist https://gist.github.com/49035faf263afe3eab2c
01:22 glusterbot Title: prod-app01.log (at gist.github.com)
01:22 semiosis JoeJulian: could be nat.  looks like that's the default networking mode for virtualbox
01:23 amccloud semiosis: JoeJulian This is the routing table https://gist.github.com/ff9a1cf21597cf019e28
01:23 glusterbot Title: gist:ff9a1cf21597cf019e28 (at gist.github.com)
01:24 semiosis amccloud: run 'tcpdump port 24007 and see if there's SNAT/Masquerading happening
01:24 semiosis s/24007/24007'/
01:24 semiosis s/24007/24007\'/
01:24 semiosis meh
01:24 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
01:24 glusterbot What semiosis meant to say was: amccloud: run 'tcpdump port 24007' and see if there's SNAT/Masquerading happening
01:25 amccloud Oh
01:25 amccloud hmm
01:25 amccloud bluster is probably listening on eth0
01:25 amccloud gluster*
01:25 amccloud eth2 is 10.1.10.0
01:26 amccloud could that be problematic?
01:26 semiosis gluster listens on all interfaces
01:26 semiosis so, how about that nat?
01:26 semiosis i think JoeJulian is on to something
01:26 semiosis listening on a specific interface wouldn't explain the port differences
01:27 semiosis imho nat is the best explanation
01:27 amccloud oh
01:28 amccloud JoeJulian: https://gist.github.com/2fbe7c675025d737f670 looks like you nailed it
01:28 glusterbot Title: gist:2fbe7c675025d737f670 (at gist.github.com)
01:29 JoeJulian If these people ever catch up to me, I'm going to get SOO drunk....
01:29 semiosis hahaha
01:29 semiosis amccloud: you can run that same tcpdump on both client & server at the same time to be completely sure the source port is changing in flight
01:30 semiosis or you could probably look at the connection tracking table in the vm host
01:31 semiosis solution of course is to switch the virtualbox network from nat to something else
01:32 semiosis what i like for my vms is a bridge connecting the VMs to the host, then host acting as gateway
01:36 polfilm joined #gluster
01:39 kevein joined #gluster
01:46 amccloud semiosis: JoeJulian: thanks for your help. I'll play with this some more in the morning,
01:46 semiosis yw
01:55 ShaunR joined #gluster
02:03 y4m4 joined #gluster
02:09 JuanBre joined #gluster
02:09 polenta joined #gluster
02:22 nueces joined #gluster
02:23 fedora joined #gluster
02:37 bharata joined #gluster
03:09 amccloud joined #gluster
03:12 DWSR joined #gluster
03:12 DWSR joined #gluster
03:24 hagarth joined #gluster
03:42 Shdwdrgn joined #gluster
03:59 mohankumar joined #gluster
04:01 lala joined #gluster
04:12 pai joined #gluster
04:15 mmakarczyk__ joined #gluster
04:16 mohankumar joined #gluster
04:19 Humble joined #gluster
04:20 arusso joined #gluster
04:20 randomcamel joined #gluster
04:21 xymox joined #gluster
04:22 jjnash left #gluster
04:22 vpshastry joined #gluster
04:23 masterzen joined #gluster
04:35 sripathi joined #gluster
04:36 shireesh joined #gluster
04:37 shireesh joined #gluster
04:47 deepakcs joined #gluster
04:48 Qten left #gluster
05:15 shylesh joined #gluster
05:18 Humble joined #gluster
05:22 sripathi joined #gluster
05:22 overclk joined #gluster
05:24 andrewbogott joined #gluster
05:36 Humble joined #gluster
05:37 sripathi joined #gluster
05:41 sripathi1 joined #gluster
05:46 bulde1 joined #gluster
05:55 bala joined #gluster
06:02 jbrooks joined #gluster
06:27 pai joined #gluster
06:28 badone joined #gluster
06:28 sripathi joined #gluster
06:41 sripathi joined #gluster
06:44 ramkrsna joined #gluster
06:44 ramkrsna joined #gluster
06:44 raven-np joined #gluster
06:46 sripathi1 joined #gluster
06:49 pai joined #gluster
06:50 bala joined #gluster
06:54 raven-np joined #gluster
06:55 Nevan joined #gluster
07:07 vimal joined #gluster
07:07 sripathi joined #gluster
07:09 rgustafs joined #gluster
07:11 tomsve joined #gluster
07:12 badone joined #gluster
07:13 puebele1 joined #gluster
07:16 jtux joined #gluster
07:30 Gilbs1 joined #gluster
07:30 Gilbs1 Anyone on for some late night troubleshooting?
07:32 puebele joined #gluster
07:33 tomsve joined #gluster
07:40 sripathi joined #gluster
07:42 ekuric joined #gluster
07:49 sripathi joined #gluster
07:50 ngoswami joined #gluster
07:50 bulde joined #gluster
07:54 jjnash joined #gluster
07:55 jtux joined #gluster
07:56 hagarth joined #gluster
07:59 Joda joined #gluster
08:04 guigui1 joined #gluster
08:07 JuanBre joined #gluster
08:10 jtux joined #gluster
08:12 lh joined #gluster
08:12 lh joined #gluster
08:13 theron joined #gluster
08:20 JuanBre joined #gluster
08:24 tjikkun_work joined #gluster
08:28 sripathi1 joined #gluster
08:31 jbrooks joined #gluster
08:37 JuanBre joined #gluster
08:49 duerF joined #gluster
08:52 raghu joined #gluster
08:54 shireesh joined #gluster
08:57 ngoswami joined #gluster
08:58 lh joined #gluster
08:59 sripathi joined #gluster
09:01 andreask joined #gluster
09:03 Joda joined #gluster
09:03 chirino joined #gluster
09:11 davis_ joined #gluster
09:17 x4rlos When i copy a large(ish) file: 650MB across to a mirrored gluster (2 nodes). Performing an ls from another node seems to take about a hundred years. This considered normal?
09:18 bauruine joined #gluster
09:18 samppah hmm nope.. don't think so.. did you copy it to gluster mount point?
09:19 x4rlos yes. My database copies its log files there, and i'm just copying across the main archive now.
09:19 samppah hmm
09:19 ndevos that does not sound right to me
09:20 x4rlos actually, now it's come back pretty fast
09:20 x4rlos update: If you do an ls -alh, it takes a long time. A standard ls will be back pretty fast.
09:20 samppah is that only file in that directory?
09:21 x4rlos Wonder if its something to do with timestamp or size.
09:21 samppah ls -alh triggers self heal
09:21 x4rlos aaaaaah.
09:21 x4rlos of course. That explains it.
09:22 chirino joined #gluster
09:24 Staples84 joined #gluster
09:26 vpshastry joined #gluster
09:26 Humble joined #gluster
09:26 DaveS_ joined #gluster
09:34 x4rlos Still learning :-)
09:35 Azrael808 joined #gluster
09:38 manik joined #gluster
09:50 mgebbe joined #gluster
09:54 hagarth joined #gluster
09:59 JuanBre joined #gluster
10:01 vijaykumar joined #gluster
10:01 killermike joined #gluster
10:02 vpshastry joined #gluster
10:03 killermike left #gluster
10:07 bala joined #gluster
10:13 DaveS_ joined #gluster
10:13 shireesh joined #gluster
10:13 frakt joined #gluster
10:13 grzany joined #gluster
10:13 portante joined #gluster
10:13 gmcwhistler joined #gluster
10:13 foster joined #gluster
10:13 JoeJulian joined #gluster
10:15 mjrosenb joined #gluster
10:22 hagarth joined #gluster
10:36 andreask joined #gluster
10:44 manik joined #gluster
10:46 lala joined #gluster
10:59 hybrid512 joined #gluster
11:03 ngoswami_ joined #gluster
11:12 duerF joined #gluster
11:13 andreask joined #gluster
11:20 davis_ Hello, I have 4 nodes acting as a 2x2 distributed-replicate cluster. I extended the volume by adding additional bricks on each of the 4 nodes, as I found a RH manual that said that was the way to extend volumes. I've since found that it's much simpler/neater to extend the existing single brick, and now I'd like to migrate the "additional" bricks' data back to the "extended" brick to reclaim the space. Is this possible?
11:21 w3lly joined #gluster
11:24 cyberbootje joined #gluster
11:29 cyberbootje joined #gluster
11:39 rastar joined #gluster
11:50 chirino joined #gluster
12:08 Joda joined #gluster
12:32 luckybambu joined #gluster
12:32 theron joined #gluster
12:32 samppah @options
12:32 glusterbot samppah: http://goo.gl/dPFAf
12:34 andreask joined #gluster
12:35 glusterbot New news from newglusterbugs: [Bug 906763] SSL code does not use OpenSSL multi-threading interface <http://goo.gl/JXYw6>
12:53 chirino joined #gluster
12:57 designbybeck___ joined #gluster
13:01 luckybambu joined #gluster
13:10 andrei joined #gluster
13:11 andrei hello guys. I've got a question on geo replication
13:11 andrei let's say i've got two servers - primary and secondary used a s a geo-replicated server
13:12 dustint joined #gluster
13:12 andrei the primary server has a bunch of 2TB files which are VM disks
13:12 Staples84 joined #gluster
13:13 andrei these files are frequently changed, however, the majority of the changes are not significant and they do not change the entire 2TB file.
13:13 andrei does the geo-replication mechanism copy the entire file across even if there is a small change to a file?
13:13 andrei or does it copy only the bits that have been changed since the last sync?
13:13 raven-np joined #gluster
13:14 dustint joined #gluster
13:17 Humble joined #gluster
13:17 ramkrsna joined #gluster
13:18 rastar joined #gluster
13:35 duerF joined #gluster
13:37 andreask andrei: it uses rsync ... so it only syncs the changes
13:41 stopbit joined #gluster
13:42 bitsweat left #gluster
13:49 Humble joined #gluster
14:00 bashtoni joined #gluster
14:00 bashtoni How can I set the tcp bind address in gluster 3.3?
14:07 duerF joined #gluster
14:12 jack joined #gluster
14:12 rwheeler joined #gluster
14:16 aliguori joined #gluster
14:18 Staples84 joined #gluster
14:21 duerF joined #gluster
14:48 ramkrsna joined #gluster
14:48 JuanBre joined #gluster
14:53 jack joined #gluster
14:54 bashtoni Does anyone know how I can set the bind address in gluster 3.3?
14:54 x4rlos bashtoni: patience dude :-)
15:06 _benoit_ Is there a way to do server failover when using qemu-kvm gluster protocol ?
15:11 vijaykumar joined #gluster
15:24 vijaykumar joined #gluster
15:25 SEJeff_work joined #gluster
15:25 SEJeff_work Does glusterfs support infiniband's raw libverbs, or is it only IPoIB?
15:26 SEJeff_work Other enterprise cluster filesystems allow access via raw libverbs
15:26 bugs_ joined #gluster
15:26 wushudoin joined #gluster
15:30 m0zes SEJeff_work: the rdma support is libverbs iirc. you don't have to setup ipoib if you don't want it.
15:30 SEJeff_work Ok beautiful
15:30 * m0zes was using rdma in the 3.2 line, haven't gotten it working in the 3.3 line yet.
15:31 SEJeff_work beautiful
15:31 SEJeff_work We have this super fast fabric, and don't really want to enable ipoib for gluster
15:31 m0zes it was considered a tech preview for the 3.2 and 3.3 lines, i think the plan is production-ready in 3.4
15:32 SEJeff_work Good to know
15:32 m0zes yeah, ipoib has *so* much overhead.
15:33 SEJeff_work We're considering playing with gluster to move off of another big enterprise cluster fs
15:33 SEJeff_work and see 0 reason in testing gluster if it doesn't at least support our fabric
15:33 SEJeff_work thankyou
15:33 m0zes no problem :)
15:33 Norky I'm using Red HAt Storage, which includes glusterfs 3.3
15:34 Norky I could not get RDMA support to work - the pieces are there, but it did not work for me
15:34 jdarcy SEJeff_work: So which other enterprise cluster FS are you using?  The only open-source RDMA-capable alternatives I know of are PVFS, Lustre, and ourselves.
15:34 Norky I beleive some people have
15:35 SEJeff_work gpfs :)
15:35 SEJeff_work I'd most likely be using the RH storage piece
15:35 jdarcy SEJeff_work: Heard good things about it, wish I could evaluate it for myself.  ;)
15:35 SEJeff_work as the native client is RHEL only, right?
15:35 m0zes SEJeff_work: it is all opensource.
15:36 jdarcy m0zes: AFAIK we *currently* require IPoIB for initial connection setup, then it's verbs after that.  There's a patch in the works to get rid of the IPoIB dependency.
15:36 SEJeff_work jdarcy, Are you a gluster developer for redhat?
15:36 jdarcy <redhat>The native client is *supported* on RHEL only.</redhat>  It works elsewhere, even on non-Linux platforms.
15:37 SEJeff_work m0zes, The sales guy said the redhat storage native client was RHEL only
15:37 SEJeff_work Ah gotcha, we didn't get that from the sales guy. Good to know.
15:37 jdarcy SEJeff_work: Affirmative.
15:37 Norky the FUSE-based native glusterfs (as opposed to NFS) client is OSS, and available pre-compiled for many distros
15:37 Norky sorry, was a bit late
15:37 SEJeff_work That makes so much more sense. Thanks
15:37 * m0zes runs it *all* in gentoo :)
15:37 jdarcy It's what we're here for.  That and funny cat pictures.
15:38 SEJeff_work jdarcy, When will the IPoIB dependency be removed?
15:38 SEJeff_work I can't sell that to management until there is 0 IPoIB dep
15:38 SEJeff_work rough timeline-wise
15:38 jdarcy We're trying to get that done for 3.4, but it's getting kind of late in the game.  The patches are pretty difficult to review/test due to shortage of either personnel or hardware.
15:39 jdarcy I would expect it to be in master within a few months, in an upstream release maybe six, in RHS later than that.
15:40 SEJeff_work jdarcy, Anything I can do to push sales to push product management?
15:40 SEJeff_work Or is that mostly futile?
15:40 SEJeff_work As a RH customer
15:41 jdarcy I don't think there's much we can do to accelerate it.  Maybe if someone were to offer some time on IB-equipped hardware and a person to help get stuff configured.
15:41 jdarcy I say that because the problem isn't lack of will (which sales/PM can influence).  It's more lack of resources.
15:42 SEJeff_work Do you work with jeremy eter?
15:42 SEJeff_work I believe he just got in some Mellanox IB switches
15:43 jdarcy Name doesn't ring a bell.  Big company.  :(
15:43 SEJeff_work He is in performance engineering. perhaps not.
15:43 SEJeff_work Last name might be spelled wrong.
15:44 jdarcy I run into a *lot* of people in my role, and I remember function names better than people names.  Sad but true.
15:44 SEJeff_work jdarcy, Ask him, he has mellanox ib switches: http://www.linkedin.com/in/jeremyeder
15:44 glusterbot Title: Jeremy Eder | LinkedIn (at www.linkedin.com)
15:45 semiosis :O
15:51 chouchins joined #gluster
15:52 Humble joined #gluster
15:52 andrewbogott joined #gluster
15:56 plarsen joined #gluster
15:56 andrei SEJeff_work: if you are still around, i could tell you about my experience with gluster over rdma
15:56 andrei it worked for me out of the box with centos
15:57 andrei however, I was not really impressed with the speed using rdma
15:57 andrei as I've realised that the problem is with fuse on the client side
15:57 andrei it is just way too slow to properly support IB
15:58 andrei I was only getting around 700-800mb/s per client
15:58 andrei however, the server side was pretty fast
16:02 jdarcy andrei: That's the main reason for it.  Clients usually either have lower demands or are bottlenecked elsewhere, but the reduced server load often means you can have fewer servers.
16:03 jdarcy It's a bit different in HPC, but that's not really our primary market.  Elsewhere it's all about cost.
16:03 andrei jdarcy: I would not make this assumption
16:03 jdarcy It's not an assumption.  It's a fact based on observing hundreds of customers.
16:03 andrei usually rdma clients have the power to handle high throughput
16:04 andrei and limiting the clients to the fuse max throughput is dissapointing, at least it was for me
16:04 andrei as i've got pretty fast clients that run vms that need high throughput
16:04 andrei and I am limited to sharing 700mb/s bandwidth between 20-30vms
16:04 andrei which is extremely slow
16:05 andrei comparing to what the potential is
16:05 andrei and I am sure there is a large number of companies switching to IB for their virtualisation needs
16:06 jdarcy andrei: I'm sorry to hear that, but that doesn't change the general rule.  For every user who has IB-equipped clients, there are dozens who have GigE (not even 10GbE).
16:06 andrei so, my vote goes for scrapping fuse )))
16:06 jdarcy andrei: So we should stop all other development and bug fixing for a year, and jettison several important features, so we can put it all in the kernel?
16:06 andrei jdarcy: yeah, I absolutely agree, IB has a very small share )))
16:07 hybrid512 joined #gluster
16:07 andrei and the mainstream users are likely to be ethernet
16:07 hagarth joined #gluster
16:08 andrei it was mainly my assumption that if you support rdma it is supported to it's full potential on both sides
16:08 andrei not just from the server side
16:08 andrei and bottleneck the clients with fuse
16:09 jdarcy andrei: It's all about tradeoffs.  A superfast kernel client would be nice, everybody likes to play King Of The Hill, but it turns out that it's not the most beneficial idea for users.
16:10 jdarcy Maybe I'm just spoiled, though.  I spent three years working on a fabric many times faster than IB.  The novelty has worn off for me.
16:10 andrei )))
16:10 andrei perhaps
16:10 andrei it was my first experience with IB and I am still learning
16:12 jdarcy If I have to choose between better IB support and better replication protocols plus encryption plus erasure coding, IB goes to the bottom of the list and probably stays there until we can hire more developers.
16:13 jdarcy Just as an example.  Those aren't the actual tradeoffs, just an illustraion.
16:13 andrei jdarcy: sure, i totally agree
16:14 andrei do you know of lustre has the same client side performance bottlenecks?
16:14 andrei or they don't use fuse at all?
16:14 jdarcy BTW, what is the current performance difference between IPoIB and raw verbs?  I used to do verbs and at that time it was several times faster than IPoIB, but I've heard others report that IPoIB is often hard to beat nowadays.
16:15 andrei jdarcy: I am not sure what the figures are for glusterfs ipoib vs rdma
16:15 jdarcy andrei: Lustre doesn't use FUSE.  They require a patched kernel and they have a fragile single metadata server instead.  ;)
16:15 andrei i've tested nfs ipoib vs rdma
16:15 andrei and the difference wasn't really huge in terms of speed
16:15 andrei around 700mb/s vs 900mb/s
16:16 andrei that's with nfs
16:16 jdarcy I spent three years trying to keep Lustre usable on that fast hardware I mentioned.  The problems I had with their MDS crashing under even moderate load could fill a book.
16:16 andrei that is on a 40gbit/s IB
16:16 andrei jdarcy: yeah, that doesn't sound good )))
16:17 jdarcy Is that because the network hardware isn't being used effectively, or because the servers are waiting for disks?
16:17 andrei i was using ramdisks for testing
16:17 hateya joined #gluster
16:17 partner speaking of networking.. what would you recommend for setup if i for example have 4 Gbit nics on the storage servers, does it make any sense to bond few towards the clients and two towards the storage networks other servers? i haven't seen much documents around this topic
16:18 jdarcy Ah.  Yeah, I do that too when I want to focus on the network side.
16:18 andrei and the ib_read|send_bw was around 4 gb/s
16:19 jdarcy partner: Split front-end and back-end networks are likely to be in the next release.  You can kinda-sorta cobble something together outside of Gluster using DNS or iptables tricks, but it's neither pretty nor complete.
16:20 andrei jdarcy: by the way, do you know when 3.3.2 is out?
16:20 jdarcy http://hekafs.org/index.php/2013/01/sp​lit-and-secure-networks-for-glusterfs/
16:20 glusterbot <http://goo.gl/nufzN> (at hekafs.org)
16:21 partner thanks
16:21 andrei jdarcy: do you know anyone who has successfully used glusterfs + cloudstack?
16:21 Norky jdarcy, is that an issue for iSCSI or other over-network block protocols?
16:21 andrei I am interested in picking their brains on the performance improvements
16:22 jdarcy andrei: Not off the top of my head, but one of our advisory-board members is a Cloudstack big shot at Citrix, so I'm pretty sure he'd know.
16:23 jdarcy Norky: Is which an issue?  The split-network stuff?
16:23 Norky certainly I understand the problem where intra-server traffic is over a 'back end' network, but if all gluster traffic is over one network, and the other network is for purely iSCSI (or ATAoE or whatever)
16:23 hybrid5121 joined #gluster
16:23 Norky yes, the split-network stuff
16:25 raven-np joined #gluster
16:25 jdarcy Norky: From the GlusterFS perspective, we'll just use the addresses and networks that match the server names you give when you create volumes etc.  Whether those are the same as your client or storage networks is kind of up to you.
16:25 Norky ...true
16:25 Norky I shoudl have worked that out myself
16:26 Norky ty
16:26 jdarcy Hey, we do everything we can to make this stuff non-obvious.  Glad to see it's working.  ;)
16:26 partner :)
16:28 partner at least for me there is just gazzillion things to figure out, i have a list of 100+ questions for myself to answer, lots of reading and googling around, and few here too as you are excellent source for providing pointers to pages i've managed to miss
16:28 andrewbogott I'm suffering a pretty serious gluster meltdown -- can someone walk me through a diagnosis of the problem?
16:29 andrewbogott This is probably a consequence of an upgrade to 3.3.1 from a few days ago.  Right now accessing volumes from the clients either fails or is incredibly slow.
16:30 andrewbogott log files are full of 'connection attempt failed (Connection refused)'
16:30 chouchin_ joined #gluster
16:31 jdarcy Firewall problem?
16:31 jdarcy Can you check what ports the clients are trying to connect to, and then check those against what the servers are listening on and what's open via iptables>
16:32 amccloud joined #gluster
16:32 andrewbogott Unlikely that it's a firewall problem since the problem is intermittent rather than total
16:33 andrewbogott But lemme see if I can confirm
16:36 jdarcy When I run my own company, monitoring IRC is going to be a part of every support engineer's regular assignment.
16:36 glusterbot New news from newglusterbugs: [Bug 906832] chown system calls() performed on object creation after being renamed into place <http://goo.gl/RR0wr> || [Bug 906838] chown() used even when an open file descriptor is available <http://goo.gl/UbqXW> || [Bug 906839] Consider using xattr methods on already open file descriptors where possible <http://goo.gl/ERtoF>
16:38 sashko joined #gluster
16:38 andrewbogott Here's another example of the failure:  # gluster volume start bots-home force
16:38 andrewbogott Operation failed on labstore4.pmtpa.wmnet
16:39 andrewbogott Sometimes I get that for labstore4, sometimes for labstore2 (there are four nodes altogether)
16:39 jdarcy So sometimes the glusterfsd daemons don't even start?
16:40 andrewbogott It /looks/ like they start… I can see glusterfsd processes in ps
16:41 andrewbogott Although, hm, I see /two/ running on labstore4 which seems like one too many
16:46 jdarcy Ah, maybe the old ones are still in the way?
16:47 jdarcy You might want to try manually killing all glusterfsd processes on a server, then going the "volume start xxx force" again.
16:53 SEJeff_work jdarcy, You're forgetting one big thing
16:53 SEJeff_work IB is very expensive
16:53 SEJeff_work Therefor, IB users normally will PAY
16:54 Norky IB's usually cheaper than 10GbE
16:54 SEJeff_work The same can not always be said for a lot of the users who use 1ge networks
16:54 SEJeff_work Not sure where you heard that
16:54 SEJeff_work I've supported both in prod for much of my career
16:55 SEJeff_work But my experience is focused primarily in low latency automated market making (financial) tech
16:55 SEJeff_work andrei, ping
16:55 andrei online ))
16:56 SEJeff_work andrei, I'm interested in  your experiences with gluster over an ib fabric
16:56 SEJeff_work if you don't mind
16:56 jdarcy Norky: Certainly used to be the case.  At equivalent speed IB has generally been much cheaper, and at same price it has generally been much faster.  Unfortunately, a lot of people compare fastest Ethernet vs. 3x as fast IB and see a different price picture.
16:56 andrei not at all
16:56 andrei what are you after? anything in particular?
16:56 SEJeff_work Just your experience and conclusions. You mentioned the fuse client was slower
16:56 lala joined #gluster
16:57 SEJeff_work I know that they've enabled direct io in fuse in the linux kernel relatively recently
16:57 andrei yeah, the server side was working perfectly well
16:57 andrei and pretty fast
16:57 andrei however, the clients were working at around 700-800mb/s max
16:57 SEJeff_work What speed were you running your fabric at?
16:57 andrei i've been testing it about 4 months ago with 3.3.0 gluster and latest centos
16:58 SEJeff_work 40G+?
16:58 andrei 40gbit/s ib
16:58 SEJeff_work Yeah that is pretty miserable
16:58 SEJeff_work What type of HBAs?
16:58 SEJeff_work *HCAs
16:59 SEJeff_work andrei, Also, what kernel were you using?
16:59 andrei i was using mellanox on the servers
16:59 andrei and qlogic on the clients
16:59 andrei the latest updated kernel from centos 6.3
16:59 SEJeff_work qlogic cards aren't that great
16:59 SEJeff_work Ok so likely not with some of the latest greatest bits of fuse
16:59 andrei yeah, but pure rdma tests were working okay
16:59 SEJeff_work There has been gobs of work going into fuse in the upstream kernel. Some of it gets backported to RHEL
17:00 andrei i was getting around 4gb/s with rdma tests
17:00 SEJeff_work Ok thats more like what is expected
17:00 andrei i can't try it now as we've switched to nfs instead
17:01 andrei what are you going to use it for?
17:01 SEJeff_work sending packets :)
17:01 andrei )))
17:01 andrei nice one1
17:01 andrei i mean that kind of data do you have?
17:01 SEJeff_work I work in finance. it isn't secret that we need some of the fastest tech
17:01 andrei i am using it for virtualisation
17:02 SEJeff_work messaging amongst clusters for a trading infrastructure
17:02 SEJeff_work thats about all I can say
17:02 andrei and I found that glusterfs wasn't working well with kvm
17:02 SEJeff_work yeah we dont have any virt in the env
17:02 andrei ah, well, if you don't use it for virtualisation, you should be okay
17:02 SEJeff_work But redhat owns glusters and qumranet, so that will get better I'm sure
17:03 andrei true
17:03 andrei anyway, my experience with setting up glusterfs and rdma was pretty good and easy
17:03 SEJeff_work thanks for the info
17:03 andrei it's too bad the virtualisation wasn't working as well as I expected
17:04 andrei i've pretty much followed the docucmentation and all worked like a charm
17:04 SEJeff_work Have you tried out sheepdog?
17:04 andrei nope
17:04 JoeJulian Mostly what I've seen is that if you're not seeing the performance you're expecting, it seems to usually be driver related.
17:04 andrei what is it?
17:04 andrei JoeJulian: i think in my case it was the fuse issue
17:04 SEJeff_work Distributed storage for kvm
17:05 andrei i've read a lot from people that fuse sucks for high throghput stuff
17:05 SEJeff_work cluster filesystem built for fault tolerant vm clusters using kvm
17:05 andrei JoeJulian: oh, do you mean the virtualisation performance?
17:05 SEJeff_work A lot of that is old knowledge
17:05 andrei yueah
17:05 SEJeff_work As upstream fuse gets a LOT of love
17:05 SEJeff_work After they merged direct io support, they did a lot of perf work
17:05 JoeJulian rdma performance generally
17:07 andrei JoeJulian: the server side performance was working well. I could easily get 1.2-1.4GB/s with two clients doing about 700mb/s max
17:07 andrei sejeff: thanks I will check it out
17:08 andrei perhaps there are improvements afterall )))
17:08 nb-ben joined #gluster
17:08 andrei JoeJulian: however, I was not able to get a single client over 700-800mb/s max
17:09 andrei and i've read that it's the fuse issue
17:09 nb-ben Hi, I currently would like to create a volume with one server, but the "gluster volume create" command requires I enter a brick, do I simply specify the local brick? (entering the WAN accessible host)
17:10 andrei so, I was ready to accept 700-800mb/s per single client, however, after putting virtualisation on top it all went horribly wrong
17:10 andrei performance wise
17:10 andrei my vms were only doing about 50mb/s on a mountpoint that was capable of 700-800mb/s
17:10 andrei that was a huge blow )))
17:11 JoeJulian nb-ben: Sure.
17:12 JoeJulian Are you using 3.4 and the new qemu direct interface?
17:12 JoeJulian andrei: ^
17:12 JoeJulian That avoids fuse entirely.
17:13 chouchins joined #gluster
17:13 nb-ben JoeJulian, if my servers use node1.domain.com, node2.domain.com etc... given I only have one server, do I need to add the server to itself as a peer in order to use it as the brick for the command?
17:13 JoeJulian no
17:13 nb-ben I could use the WAN domain without having it listed as a peer?
17:15 Norky yes
17:15 JoeJulian Whatever the hostnames resolves to is what the client is going to try to connect to. Lan/wan, doesn't matter. The server listens on all.
17:15 nb-ben alright
17:15 JoeJulian This works well for you as split-horizon dns can allow you to connect to whichever (lan/wan) you choose.
17:16 nb-ben what does split-horizon mean :D
17:17 andreask joined #gluster
17:18 elyograg nb-ben: generally it's used for making names resolve to public IPs when accessed externally and internal IPs when accessed internally.
17:19 nb-ben I see :P yeah I knew that just didn't understand the term split-horizon
17:23 semiosis @lucky split horizon dns
17:23 glusterbot semiosis: http://en.wikipedia.org/wiki/Split-horizon_DNS
17:24 nb-ben thank you semiosis
17:25 semiosis yw
17:32 guigui1 left #gluster
17:35 vpshastry joined #gluster
17:35 vpshastry left #gluster
17:39 theron joined #gluster
17:43 luckybambu joined #gluster
17:45 Gilbs joined #gluster
17:51 zaitcev joined #gluster
17:54 balunasj joined #gluster
17:57 dustint joined #gluster
18:07 amccloud joined #gluster
18:10 nueces joined #gluster
18:17 ramkrsna joined #gluster
18:17 ramkrsna joined #gluster
18:22 avati_ joined #gluster
18:24 andrewbogott joined #gluster
18:29 amccloud joined #gluster
18:33 jbrooks joined #gluster
18:38 sashko joined #gluster
18:49 w3lly joined #gluster
19:09 bauruine joined #gluster
19:13 vijaykumar left #gluster
19:18 sjoeboo joined #gluster
19:20 dustint joined #gluster
19:41 sashko joined #gluster
19:51 sashko joined #gluster
19:58 johnmark joined #gluster
19:59 nueces joined #gluster
20:04 elyograg kkeithley: are you around? or anyone else who knows something about glusterfs-swift.
20:08 JoeJulian I know some
20:08 nueces joined #gluster
20:09 elyograg i'm having the same problem with 3.3.1-8 on centos that i did on fedora 18.
20:11 elyograg http://www.fpaste.org/ZNue/
20:11 glusterbot Title: Viewing glusterfs-swift 3.3.1-8 on CentOS by elyograg (at www.fpaste.org)
20:12 elyograg after install, i moved all the conf files ending in -gluster to the same names without that suffix. then i added a user and my memcached servers to the proxy config, and tried to start it.
20:13 puebele joined #gluster
20:13 nb-ben if I specify "replica 2" for a volume, but have 5 bricks supporting it, will every file have a copy somewhere on a different server? or should I always specify replica as the same amount of bricks?
20:13 nb-ben because I keep seeing that people only specify replica as the same amount of bricks
20:13 elyograg the file that it can't find is there on 3.3.1-2.
20:13 samppah @eager locking
20:14 samppah @options
20:14 glusterbot samppah: http://goo.gl/dPFAf
20:14 elyograg nb-ben: replica should be however many copies of your data you want around.
20:15 nb-ben elyograg: thank you
20:15 elyograg each copy should be on a different server.
20:15 elyograg unless you're doign a proof of concept.
20:15 nb-ben what do you mean by proof of concept?
20:15 elyograg a testbed, not production.
20:16 nb-ben oh I see, that was directed to the different server thing
20:16 elyograg yes.
20:18 w3lly joined #gluster
20:52 daMaestro joined #gluster
21:10 Bunxed joined #gluster
21:10 sjoeboo joined #gluster
21:10 Bunxed Hi - I need some help with gluster and brick data synchronization
21:10 Bunxed Basically I have a volume configured with replica = 2 - it currently has 2 bricks
21:11 Bunxed It just happned that one of the bricks keeps dying - cause we took a power hit
21:11 Bunxed After I redid the brick - mkfs.xfs
21:12 Bunxed I ran a gluster rebalance fix-layout start
21:12 Bunxed then I did a gluster rebalance data-migrate start
21:12 Bunxed But the data never was recopied from the other brick
21:12 Bunxed I am running Gluster 3.2.3
21:12 Bunxed Any ideas?
21:12 m0zes Bunxed: replica isn't handled by the rebalance operation.
21:12 JoeJulian @repair
21:12 elyograg rebalance is for a distributed volume.  best bet would be to stat every file from the client.
21:12 glusterbot JoeJulian: http://goo.gl/uA812
21:13 JoeJulian ~repair | Bunxed
21:13 glusterbot Bunxed: http://goo.gl/uA812
21:13 Bunxed Hmm - repair is not good
21:13 JoeJulian Bunxed: btw, 3.3 has proactive self-healing so in that situation after you replaced the brick you would have issued, "gluster volume heal $vol full" and it would have rebuilt it.
21:14 Bunxed I have severall hundreds of millions of files
21:14 Bunxed 3.3 - hmm
21:14 Bunxed OK - here is a side effect
21:14 Bunxed I then did as follows
21:14 Bunxed volume remove-brick broken:brick
21:14 Bunxed volume remove-brick volume broken:brick
21:15 Bunxed volume add-brick volume broken:brick
21:15 JoeJulian 3.2 would still require the repair.
21:15 JoeJulian How many bricks in your volume?
21:15 Bunxed then I ran a volume rebalance volume data-migrate
21:15 Bunxed all the data was updated
21:15 Bunxed This was in a small test environment
21:15 Bunxed Before trying on the real large environment
21:15 Bunxed Is this OK?
21:16 Bunxed JoeJulian - 2 bricks in volume - 87TB each brick
21:16 JoeJulian rebalance is for mapping file hashes to different bricks within the distribute translator.
21:17 JoeJulian Since you only have 2 bricks, ,,(targeted self heal) isn't going to help you any. You're going to have to walk the tree one-way of another.
21:17 glusterbot http://goo.gl/E3b2r
21:17 Bunxed JoeJulian: ok - I will try the find
21:17 Bunxed See how it works out
21:18 Bunxed JoeJulian: walk the tree?
21:18 Bunxed You mean doing a find?
21:18 Bunxed Could I simply just do an rsync?
21:18 Bunxed From mount point to the client?
21:18 Bunxed where client is not having glusterd running?
21:18 Bunxed while rsync is running
21:19 JoeJulian Why would an rsync be better?
21:20 JoeJulian You have a client. You perform a lookup() on a filename. The client contacts both servers and checks to see if they're both sane. One's not. The client copies the necessary data from the sane to the not-sane.
21:21 JoeJulian vs rsync... The rsync client contacts a server. Copies all the data from the server to the client, then writes the data to both servers. That's 150% as much data transfer.
21:21 Bunxed Oh - got it
21:22 Bunxed Well
21:22 elyograg JoeJulian: i didn't know the client did that copy.  for some reason i thought the servers worked that out.  i never asked, though.
21:22 Bunxed I meant to copy directly to the brick
21:22 Bunxed But I see your point
21:22 Bunxed OK - that helps
21:22 Bunxed Let me try
21:22 JoeJulian elyograg: With 3.3, the self heal daemon (essentially a client) does that from one server to another.
21:23 elyograg that would be better.  especially on my setup where I have a dedicated network card for server to server communications, clients go through a different one.
21:24 JoeJulian elyograg: Just make sure you have split-horizon hostname resolution so your servers resolve the private network for the other servers.
21:25 elyograg JoeJulian: understood.  actually, i'm using hosts files on the servers.  only two servers so far, and even if this ramps up I don't expect more than a dozen or two, easy enough to keep that many up to date.  DNS resolves to the client side.
21:26 elyograg if i forget to update a server's hosts file, it'll still work via dns.
21:28 kombucha joined #gluster
21:29 Bunxed JoeJulian: neither command of find worked from the client
21:29 Bunxed the files are not synched
21:30 Bunxed Do I need to do a rebalance fix-layout first?
21:32 elyograg Bunxed: your volume isn't distributed, so no.
21:34 Bunxed no it is mirrored
21:34 JoeJulian Bunxed: Check your clieng log.
21:34 Bunxed OK
21:34 JoeJulian s/clieng/client/
21:34 glusterbot What JoeJulian meant to say was: Bunxed: Check your client log.
21:42 y4m4 joined #gluster
21:47 Bunxed JoeJulina - the logs don't say anything that I think can help me
21:48 Bunxed JoeJulian:  I think I know why the find does not work
21:48 Bunxed For some reason
21:48 Bunxed After I did the mkfs.xfs on the broken volume
21:48 Bunxed and simply restarted glusterd on the broken brick server
21:48 Bunxed The brick does not seem to really be part of the original volume
21:49 Bunxed it is as if the signature is not the same
21:58 sashko joined #gluster
22:00 Gilbs left #gluster
22:08 bauruine joined #gluster
22:26 amccloud joined #gluster
22:27 RobertLaptop joined #gluster
22:47 sjoeboo joined #gluster
23:19 hattenator joined #gluster
23:24 raven-np joined #gluster
23:25 clusterflustered ok you fellers im installing gluster on4 servers for a test
23:26 clusterflustered can i run 'gluster peer probe host1 host2 host3 host4' ?
23:26 clusterflustered or do i need to do it individually for all hosts/
23:27 elyograg clusterflustered: when I try, I get: Usage: peer probe <HOSTNAME>
23:28 elyograg this is 3.3.1
23:29 olem joined #gluster
23:30 clusterflustered do i need to do this as root?
23:30 clusterflustered or sudo?
23:30 elyograg i believe so.  i always have.
23:31 clusterflustered http://xkcd.com/149/
23:31 glusterbot Title: xkcd: Sandwich (at xkcd.com)
23:32 elyograg clusterflustered: very common thing that happens to me: http://xkcd.com/979/
23:32 glusterbot Title: xkcd: Wisdom of the Ancients (at xkcd.com)
23:32 puebele3 joined #gluster
23:33 clusterflustered hahhaahahaaaaa
23:35 clusterflustered it seems the basic tutorial im following is leaving a little out
23:35 clusterflustered Connection failed. Please check if gluster daemon is operational.
23:36 clusterflustered i see in my init.d folder i have glusterd and flusterfsd.  how and in what order should i start these
23:38 elyograg only glusterd needs starting.  glusterfsd only gets run with 'stop' and it should happen automatically whenever it needs to.
23:40 clusterflustered awesome thanks
23:42 sashko joined #gluster
23:48 clusterflustered so, two questions, my next step is to do a volume create, and is recommending a path of /export/brick1  .  with 4 servers can i use replica 3? and will it create this exports folder on its own?
23:49 partner replica of 3 will get your data to 3 servers ie. all having the same data
23:50 partner path can be whatever but make sure your disk for the brick is mounted there
23:51 clusterflustered so i have 4 servers, replica 2 would then make 2 copies of everthing? is there a page i can learn about the replica numbers
23:52 clusterflustered (replica)
23:52 clusterflustered GLUSTERBOT I NEED YOU. replica
23:52 partner replica is just a copy of a brick X times N (the number you define)
23:54 elyograg if you choose replica 2, then each pair of bricks on the create/add commandline (in order) will comprise a replica set.
23:54 clusterflustered ok, so replica 2 on a 4 server cluster will provide me 2x the speed of a single file server, but also redundancy?  if i do replica 1, that would be the fastest, but least redundant, correct?
23:55 partner gluster volume create myreplica replica 2 server1:/export/brick1 server2:/export/brick1
23:55 partner clusterflustered: no
23:55 clusterflustered i think im misunderstanding the term brick.
23:55 elyograg @glossary
23:55 glusterbot elyograg: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
23:56 partner brick is the mountpoint, the disk you are going to make available for the volume
23:57 partner i'm sorry but i am going to refer to RH docs again, they have all the nice pictures and stuff..
23:57 partner https://access.redhat.com/knowledge/docs/en​-US/Red_Hat_Storage/2.0/html/Administration​_Guide/chap-User_Guide-Setting_Volumes.html
23:57 glusterbot <http://goo.gl/Ax7wx> (at access.redhat.com)
23:58 clusterflustered so since our servers are single array servers, (1 logical disk) im going probably going to want 1 brick per server.
23:58 clusterflustered rgr will read thanks
23:59 partner clusterflustered: one logical disk can be still partitioned into multiple smaller ones. it would be assumably the /dev/sda disk.
23:59 elyograg clusterflustered: here's my volume info.  http://www.fpaste.org/O0AV/
23:59 glusterbot Title: Viewing gluster volume info by elyograg (at www.fpaste.org)

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary