Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:01 _ilbot joined #gluster
01:01 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
01:13 jdarcy joined #gluster
01:23 glusterbot New news from newglusterbugs: [Bug 919953] Namespace clash for TMP_MAX in gluster code on OSX <http://goo.gl/2I0hE>
01:40 sahina joined #gluster
01:42 Ryan_Lane joined #gluster
01:50 merickson joined #gluster
01:52 kevein joined #gluster
01:52 moterpent joined #gluster
01:54 moterpent joined #gluster
01:55 moterpent Does anyone happen to know if ssh is required for geo-replication or can native gluster ports be used instead?
02:07 neofob i just made a vagrant box that has gluster installed; do people deploy gluster with virtualbox|virtual machines ?
02:09 bala joined #gluster
02:11 klaxa hmm we do, also it's used as an example in the getting started guides
02:12 klaxa wait...
02:12 klaxa my answer didn't match your quesiton i think
02:13 fidevo moterpent, as far I tested geo-replication, ssh is required
02:15 moterpent fidevo: Interestingly I just got it to work by specifying the geolocation like so:  gluster volume geo-replication volumename gluster://slavehost:/slavevol start
02:16 moterpent fidevo: Looks like ssh is the default if you just specify "slavehost:/slavevol"
02:16 fidevo moterpent, the problem is, if you are replicating from New York to LA, how would you guarantee security between them ?
02:17 moterpent fidevo: ipsec ptp tunnels between locations
02:17 fidevo moterpent++
02:17 moterpent fidevo: But yes, I see the point of security inside the LAN.
02:17 fidevo moterpent, yep, I would keep ssh + ipsec ptp tunnels
02:17 fidevo just in case :-)
02:18 moterpent fidevo: Has it been your experience that writing to a master volume is very very slow?
02:18 moterpent fidevo: I seem to max out at 2.5 - 3 MB/s
02:18 fidevo moterpent, from slave?
02:19 moterpent fidevo: Writing on the master.
02:19 moterpent fidevo: So, for example, on the master gluster volume doing a "dd if=/dev/zero of=out.1 bs=1024 count=10000"
02:20 fidevo moterpent, just to make sure, you are writing that from a mounted point on Clinet_X
02:21 moterpent fidevo: Aye.  Two mount points.  One is the "brick" (ie. ext4 filesystem that the glusterfs volume is created from) -- that is as fast as can be.  The other mount point is the mounted glusterfs.  that is very slow.
02:22 moterpent It is my understanding that one always wants to write the mounted glusterfs in order for the replication/distribution etc to go off properly.
02:23 fidevo moterpent, hmm, never tried that, only going through the glusterfs mounted point
02:23 moterpent Entirely possibly that my interpretation of all of this is off.  Have only been experimenting for a day or so.
02:23 fidevo I would ask to test your network using netcat
02:24 fidevo just to see what is going to be speed besides applications
02:24 moterpent fidevo: The "master" server doesn't have any other servers/peers participating.  Just a single filesystem on server 1 georeplicating to to a second server (server2).  Both are independent and not shared peers with one another.
02:25 moterpent fidevo: The whole idea being to replicate files offsite for DR/backup purposes.
02:25 moterpent ...hope that makes sense.
02:25 fidevo moterpent, yep, I meant to say to test Server->Client Networking
02:26 fidevo moterpent, what kind of volume are you using?
02:26 moterpent The replication is going off without issue.  I'm just surprised that a local write to server1's glusterfs mount is so slow even prior to any replication going on.
02:27 moterpent Or perhaps replication is going on while it is writing.  I don't know how that works.
02:28 moterpent If it is truely that slow, and I'm not just being dumb, it probably is not reasonable to use.  Having hundreds of users writing to a cifs that is then back-ended by gluster and only being able to write at 2.5MB/s is probably a no go.  :(
02:29 moterpent Will have to continue to experiment.  No doubt I'm missing something.
02:33 moterpent Appears that the block size has huge implications on the performance.  By changing from 1K BS to 1M BS I go from 2.5MB/s to 125MB/s.  Still slower than native but a gigantic improvement.
02:34 moterpent So it would appear that throughput is good but lots of little IO's have big performance hit.
02:38 _pol joined #gluster
02:41 _pol joined #gluster
02:47 Humble joined #gluster
02:49 bharata joined #gluster
02:54 stopbit joined #gluster
03:04 jdarcy joined #gluster
03:05 bala joined #gluster
03:10 bulde joined #gluster
03:11 Humble joined #gluster
03:11 sgowda joined #gluster
03:24 yinyin joined #gluster
03:40 vshankar joined #gluster
03:43 shylesh joined #gluster
03:44 sgowda joined #gluster
03:46 hagarth joined #gluster
03:50 sgowda joined #gluster
03:51 sgowda joined #gluster
03:53 glusterbot New news from newglusterbugs: [Bug 919007] After some time, transfer is slow and all writes are 4kb. Re-opening fds brings back fast transfer <http://goo.gl/9rIAH>
04:04 sripathi joined #gluster
04:13 raghu` joined #gluster
04:18 tg2 joined #gluster
04:23 Humble joined #gluster
04:31 yinyin joined #gluster
04:44 gluslog joined #gluster
04:52 vpshastry joined #gluster
05:08 deepakcs joined #gluster
05:10 yinyin joined #gluster
05:14 aravindavk joined #gluster
05:25 jbrooks joined #gluster
05:26 ramkrsna joined #gluster
05:26 ramkrsna joined #gluster
05:29 yinyin joined #gluster
05:30 lalatenduM joined #gluster
05:32 bala joined #gluster
05:35 vshankar joined #gluster
05:46 kkeithley1 joined #gluster
05:48 bala joined #gluster
05:51 ramkrsna joined #gluster
05:53 neofob left #gluster
05:54 glusterbot New news from newglusterbugs: [Bug 919990] Crash in server_rpc_notify <http://goo.gl/5YxNH>
06:02 rastar joined #gluster
06:05 bulde joined #gluster
06:14 puebele joined #gluster
06:24 glusterbot New news from newglusterbugs: [Bug 917686] Self-healing does not physically replicate content of new file/dir <http://goo.gl/O8NMq>
06:28 bala joined #gluster
06:37 sripathi joined #gluster
06:46 stickyboy joined #gluster
06:48 bala joined #gluster
06:50 rgustafs joined #gluster
06:50 vimal joined #gluster
06:56 test joined #gluster
06:56 ramkrsna joined #gluster
07:07 displaynone joined #gluster
07:14 Philip_ joined #gluster
07:17 jtux joined #gluster
07:21 sripathi joined #gluster
07:22 Nevan joined #gluster
07:23 bulde joined #gluster
07:25 xymox joined #gluster
07:27 shireesh joined #gluster
07:27 stickyboy I'm just setting up a new GlusterFS deployment on CentOS 6.3... gonna update to CentOS 6.4 to see if there are any issues.  Anyone tried yet?
07:28 xymox joined #gluster
07:29 satheesh joined #gluster
07:31 ramkrsna joined #gluster
07:36 ThatGraemeGuy joined #gluster
07:38 sgowda joined #gluster
07:41 msvbhat joined #gluster
07:41 yinyin joined #gluster
07:44 ngoswami joined #gluster
07:44 bala joined #gluster
07:50 raghu` joined #gluster
07:53 ekuric joined #gluster
07:58 dobber_ joined #gluster
08:01 mohankumar joined #gluster
08:02 y4m4 joined #gluster
08:02 jtux joined #gluster
08:06 kshlm joined #gluster
08:06 kshlm joined #gluster
08:14 mooperd joined #gluster
08:17 Nevan joined #gluster
08:19 test_ joined #gluster
08:20 shireesh joined #gluster
08:20 Humble joined #gluster
08:31 hagarth joined #gluster
08:33 msmith_ joined #gluster
08:43 vimal joined #gluster
08:43 Staples84 joined #gluster
08:44 shireesh joined #gluster
08:45 andreask joined #gluster
08:49 gbrand_ joined #gluster
08:50 kkeithley1 joined #gluster
08:53 sripathi joined #gluster
08:54 tryggvil joined #gluster
08:56 tjikkun_work joined #gluster
09:01 manik joined #gluster
09:02 sripathi1 joined #gluster
09:09 Norky joined #gluster
09:13 rotbeard joined #gluster
09:19 bulde joined #gluster
09:24 rastar joined #gluster
09:25 bulde joined #gluster
09:30 kkeithley1 joined #gluster
09:32 sripathi joined #gluster
09:34 sripathi joined #gluster
09:34 mooperd joined #gluster
09:41 jdarcy joined #gluster
09:43 johndesc1 joined #gluster
09:47 bulde joined #gluster
09:49 johndesc1 hi !
09:49 johndesc1 sorry for the probably 1000th troll but why do you advertise everywhere a support of CIFS but in fact, it is juste a reexport with a third party samba server ?
09:50 johndesc1 in my test environment with natif gluster we get a rough 60mbps data write rate and with that kind of hack, it drops to 20mbps…
09:54 masterzen joined #gluster
09:55 bulde1 joined #gluster
09:55 rastar joined #gluster
09:58 kkeithley_blr How is a re-export using Samba not Samba/CIFS? That aside, we are working on writing a Gluster VFS plug-in for Samba that should substantially improve performance.
09:58 kkeithley_blr These things don't happen over night though.
10:00 johndesc1 nice to know :)
10:02 johndesc1 what do you mean with "using Samba not Samba/CIFS"? I'm using the samba server from my distro.
10:03 kkeithley_blr I'm not sure what you're asking.
10:05 aravindavk joined #gluster
10:05 johndesc1 I'm mainly wondering why CIFS is advertised everywhere as kind of "intergrated", like NFS, but in fact we lose a lot of throughput by re-exporting with stock CIFS server a gluster mountpoint.
10:05 kkeithley_blr You ask why we advertise support of Samba (CIFS, there is no CIFS, according to Microsoft and the Samba team). We support NFS the same way. The only difference is, we wrote the NFS server.
10:07 kkeithley_blr I don't know about "advertised everywhere." Yes, I'm sure it's on the gluster web site somewhere.
10:08 johndesc1 here: http://www.gluster.org/about/ it is together with native and NFS
10:08 glusterbot Title: About | Gluster Community Website (at www.gluster.org)
10:09 johndesc1 (BTW your NFS support is quite good but sadly we have windows destop too here…)
10:10 kkeithley_blr Okay, well, it is what it is today. Tomorrow it'll be quite a bit better. It'll still be the Samba server though, just better integrated than it is today.
10:10 robos joined #gluster
10:11 johndesc1 do you mean that there is already samba specific tweaks already in the code? maybe I've missed something…
10:13 johndesc1 as far as I know currently samba is only reexported without interaction outside dropping the files in a directory that -luckly- is a gluster mount
10:14 hybrid512 joined #gluster
10:14 hybrid512 joined #gluster
10:15 clag_ joined #gluster
10:15 johndesc1 so there is *no* samba support glusterfs-wise IMHO ;)
10:18 rastar joined #gluster
10:18 stickyboy kkeithley_blr: Is it ok to use your repos on CentOS 6.4?
10:20 kkeithley_blr I don't equate what's on the page you mentioned with '''advertised everywhere as a kind of "integrated" like NFS.'''  You can run an NFS server (ours) over your Gluster storage. You can run a Samba server (not ours) over your Gluster storage. That's all that page says.
10:21 kkeithley_blr stickyboy: Should be.
10:22 johndesc1 what bothers me is the mounting-reexporting… that don't happen with NFS
10:22 kkeithley_blr Actually, yes it does.
10:22 stickyboy kkeithley_blr: Cheers.
10:23 kkeithley_blr The glusterfs process is a fuse client with an NFS server on the top of the translator stack.
10:24 johndesc1 oh
10:24 yinyin joined #gluster
10:25 johndesc1 (sorry AFK lunch)
10:41 jclift_ joined #gluster
10:42 hagarth joined #gluster
10:43 shireesh joined #gluster
10:47 nueces joined #gluster
11:02 duerF joined #gluster
11:04 yinyin joined #gluster
11:07 sripathi joined #gluster
11:14 mooperd_ joined #gluster
11:18 Rydekull joined #gluster
11:26 ThatGraemeGuy joined #gluster
11:36 stickyboy Anyone using CentOS with bricks on ext4 volumes larger than 16TB?
11:39 kkeithley_blr I'd be worried about fsck times. And don't forget the ext4 bug.
11:39 kkeithley_blr sorry if you know abut these already
11:39 kkeithley_blr sorry for repeating
11:41 rwheeler joined #gluster
11:44 yinyin joined #gluster
11:48 malevolent joined #gluster
11:51 manik joined #gluster
11:54 deepakcs joined #gluster
11:54 adocampo left #gluster
11:57 jdarcy joined #gluster
11:57 hagarth joined #gluster
12:03 balunasj joined #gluster
12:04 Norky joined #gluster
12:05 johndesc1 (back) so if NFS and samba work almost the same, I may have a setup problem that cut the performances so much, is it a well known problem or just for me ?
12:06 johndesc1 I saw some threads on the internetz but nothing really helpful
12:06 stickyboy Hmm, anyone using ext4 bricks on volumes larger than 16TB?
12:07 H__ stickyboy: almost I'm at 15T :)
12:07 H__ stickyboy: why ? what's the question ?
12:09 stickyboy H__: CentOS 5/6's e2fsprogs don't support volumes larger than 16TB.
12:09 stickyboy Grr.
12:09 H__ oh you mean bricks of that size
12:09 H__ i thought you meant volumes of 16T
12:10 H__ as in gluster volumes
12:11 stickyboy Yah, bricks.  As in, underlying raw storage.
12:11 H__ stickyboy: which version e2fsprogs is that. (and do you know in which version larger is supported?)
12:12 H__ I use 2T bricks.
12:13 lpabon joined #gluster
12:16 Staples84 joined #gluster
12:17 stickyboy H__: Apparently e2fsprogs 1.42.x has support for larger-than-16TB volumes.
12:17 stickyboy Yah, I'm using a 29TB brick.  ext4.
12:18 stickyboy I compiled my own e2fsprogs into /usr/local, which worked, but on startup fsck uses the /sbin/fsck* before searching the PATH.
12:18 edward1 joined #gluster
12:18 stickyboy So it dies.
12:19 H__ I remember that when i used redhat (old old OLD userland), went to ubuntu because of it and am quite happy of it
12:21 H__ 1.42.5 here
12:22 stickyboy Yah
12:22 stickyboy I find it hard to believe I'm the only one who has ever used large volumes in CentOS 6.
12:23 H__ well, it's generally recommended not to keep single filesystems of that size, it's just Too Long to restore in case of disaster
12:23 H__ not to mention a full fsck will take days
12:24 stickyboy True.
12:25 stickyboy That's actually a really valid point (the fsck one), and it would solve my e2fs problems anyways.
12:26 H__ why are your bricks so large ? (I use gluster to scale out)
12:26 stickyboy H__: They just are :)  I hadn't thought about it.
12:27 stickyboy I figured since it was abstracted at the gluster level it didn't matter what was underneath.
12:27 andreask joined #gluster
12:29 nueces joined #gluster
12:30 ThatGraemeGuy joined #gluster
12:31 johndesc1 about SMB/CIFS support, that's also misleading : http://www.gluster.org/community/document​ation/index.php/Getting_started_overview
12:31 glusterbot <http://goo.gl/Lq4vp> (at www.gluster.org)
12:32 Nagilum H__: fsck on ext4 is quite fast though
12:35 displaynone joined #gluster
12:36 stickyboy Nagilum: I have a 29TB brick.  It's probably sane to break it up anyways. :D
12:36 jclift_ stickyboy: What happens if you make the /sbin/fsck* stuff non-executable?
12:37 * jclift_ wonders if it might then find your own compiled stuff
12:37 jclift_ Prob not... but an interesting idea to check, just in case. :)
12:37 stickyboy jclift_: Hmmm, good question.  I assume any mods to /sbin/fsck would get overwritten on subsequent e2fsprogs updates.
12:37 Nagilum stickyboy: yeah, I don't think the codepaths for ext4>16TB are well tested already
12:38 jclift_ stickyboy: It's completely possible to blacklist the e2fsprogs rpm though, so it doesn't get updated.
12:38 jclift_ Pretty ugly approach but. ;)
12:38 Nagilum jclift_: simply replace the fsck with a recent version
12:38 Philip_ joined #gluster
12:39 jclift_ Yeah.  It's CentOS (ie not a supportable issue thing), so doing an srpm compile of newer e2fsprogs would prob be the go unless you're unhappy about it.
12:39 stickyboy jclift_: Yah.  hehe.
12:40 stickyboy I think I lose nothing by breaking my storage up into 3 * 10TB bricks.
12:40 stickyboy Then if I get hit by a beer truck when crossing the road, the next sysadmin who comes along won't have some funny hacks to deal with ;)
12:41 jclift_ :)
12:43 robos joined #gluster
12:50 bulde joined #gluster
12:51 aliguori joined #gluster
12:53 jdarcy joined #gluster
12:57 bennyturns joined #gluster
12:59 mooperd joined #gluster
13:02 Staples84 joined #gluster
13:05 hagarth joined #gluster
13:06 ThatGraemeGuy joined #gluster
13:29 deepakcs joined #gluster
13:31 dustint joined #gluster
13:33 bulde joined #gluster
13:37 lh joined #gluster
13:38 msmith_ Having an odd issue with gluster 3.3.1.  I have 6 servers, 1 brick per server (10 drives raid-6 xfs formatted) in a 2x replication setup (3x2) and mounting with NFS.  For this one file, it exists on server 1&2 as a link (getfattr shows both servers reporting the same values for trusted.gfid and trusted.glusterfs.dht.linkto), does not exist on server 3&4, and the real file exists on server 5&6 (getfattr reports the same gfid on both se
13:38 msmith_ rvers, they have the same size, tstamps and data).  The problem is that when I try and access the file through the mount, I get "Remote I/O error", and a directory listing shows the file as the following "-????????? ? ?                ?      ?            ? dovecot.quota"
13:39 msmith_ and, NFS is *not* mounted through localhost, each mounts from one of the other servers
13:42 Staples84 joined #gluster
13:44 sjoeboo So, I've got some quotas basically being 100% ignored (just pinged the mailing list too)
13:44 sjoeboo any idea where to start digging?
13:45 sjoeboo quotas are enabled, and limits set, of course, they just seemingly are not being adheared to.
13:46 stickyboy sjoeboo: Eek!  Sorry, can't help.
13:47 stickyboy Just curious, what Linux distro / gluster version?
13:47 sjoeboo centos 6.3, 3.3.1
13:55 rwheeler joined #gluster
13:58 ThatGraemeGuy joined #gluster
13:59 plarsen joined #gluster
14:01 stickyboy sjoeboo: Damn, I'm deploying on the same platform and was expecting to use quotas...
14:01 lh joined #gluster
14:01 sjoeboo yeah, not sure why this is happening
14:01 sjoeboo i see a KB answer saying a remount client side is needed, quick test shows that works
14:01 sjoeboo but thats not good for us
14:02 neofob joined #gluster
14:02 sjoeboo so some sort of refresh of the client w/o a remount is needed...but we'll see...
14:05 Norky joined #gluster
14:06 lpabon joined #gluster
14:11 Humble joined #gluster
14:14 ThatGraemeGuy__ joined #gluster
14:22 wushudoin joined #gluster
14:23 vpshastry joined #gluster
14:24 lh joined #gluster
14:24 lh joined #gluster
14:27 wN joined #gluster
14:30 meshugga joined #gluster
14:30 stopbit joined #gluster
14:31 arusso joined #gluster
14:31 Cenbe joined #gluster
14:31 arusso joined #gluster
14:31 morse joined #gluster
14:31 bdperkin joined #gluster
14:31 jbrooks joined #gluster
14:31 thekev joined #gluster
14:32 tjikkun_ joined #gluster
14:35 mohankumar joined #gluster
14:36 dbruhn joined #gluster
14:36 dbruhn Is anyone around here running 3.3.1 with RDMA
14:36 kshlm|AFK joined #gluster
14:38 Skunnyk joined #gluster
14:40 dbruhn I have this issue where I can build a volume using RDMA, but then I can't mount the volume via RDMA and I am not seeing anything in the logs to give me an indication as to even look.
14:41 bugs_ joined #gluster
14:42 dblack joined #gluster
14:48 Staples84 joined #gluster
14:50 xavih dbruhn: have you searched in brick logs ?
14:52 dbruhn I am looking at it, but I am not seeing anything that seems to have to do with the mount. At least from what I can tell.
14:53 jdarcy joined #gluster
14:53 jdarcy_ joined #gluster
14:55 lh joined #gluster
14:58 venkatesh joined #gluster
14:58 venkatesh left #gluster
14:59 xavih dbruhn: I've had some troubles connecting RDMA bricks, but I've always seen at least one error in one log
15:00 robo joined #gluster
15:00 xavih dbruhn: there seems to be a problem between the listening port of the brick and the port that the client tries to connect to
15:01 dbruhn Is this the error for the volume? E [rdma.c:4604:tcp_connect_finish] 0-ENTV03-client-7: tcp connect to  failed (Connection refused)
15:02 xavih dbruhn: yes
15:02 dbruhn what did you do to get around it/
15:05 xavih dbruhn: I'm not sure if it is the best way to do it, but it's fast
15:05 xavih dbruhn: When you start the volume and you try to mount it, all bricks are started
15:06 dbruhn Ok?
15:06 xavih dbruhn: then you look at brick log and there it should be the port number it is listening
15:06 xavih dbruhn: I think this is only shown in debug mode though
15:07 xavih dbruhn: then you add the option listen-port to the client with the value shown in the log
15:07 xavih dbruhn: it's dirty, but it worked for me
15:07 xavih dbruhn: probably a bug should be opened
15:08 dbruhn Do you remember right off how to turn on the debug logging, and then what you did to add the listen-port
15:09 wN joined #gluster
15:09 dbruhn and I am confused.. RDMA doesn't have traditional ports because it's not running tcp?
15:11 daMaestro joined #gluster
15:12 dbruhn It's also weird that my bricks are communicating with each other over RDMA, but I can't mount them.
15:16 lh joined #gluster
15:16 displaynone joined #gluster
15:20 xavih dbruhn: the debug logging can be enable with 'gluster volume set <volname> diagnostics.brick-log-level debug'
15:21 sjoeboo so, does anyone know a way, aside from remounting, to get a gluster client to refresh the vol info?
15:21 sjoeboo specifically, for a newly created quota limit
15:21 xavih dbruhn: sorry, it's not 'listen-port', it's 'remote-port'
15:22 xavih dbruhn: I'm not sure if there is a way to modify it from gluster command line
15:22 xavih dbruhn: I modified it manually in the volume file
15:24 torbjorn1_ I'm wondering if there are any news or status regarding the 3.4.0 release ? .. AFAIK, the planned schedule is to release every 6 months ?
15:25 dbruhn xavih,
15:26 dbruhn can you past bin a copy of your config file for me
15:28 jdarcy_ joined #gluster
15:30 _br_ joined #gluster
15:33 _br_ joined #gluster
15:35 _br_ joined #gluster
15:35 xavih dbruhn: http://pastebin.com/mFFPebVc
15:35 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
15:36 xavih dbruhn: however it's not recommended to modify volfiles manually
15:37 _br_ joined #gluster
15:37 dbruhn understood
15:38 xavih dbruhn: in my case bricks were listening on port 24010. I'm not sure if yours will be the same
15:39 dbruhn is it the same on every brick
15:39 jdarcy_ joined #gluster
15:39 xavih dbruhn: on another test, the port was 24012
15:40 bitsweat left #gluster
15:40 dbruhn xavih: Are you on 3.3.1?
15:41 dbruhn I just noticed this after turning on debugging is why
15:42 dbruhn W [options.c:782:xl_opt_validate] 0-ENTV03-server: option 'listen-port' is deprecated, preferred is 'transport.rdma.listen-port', continuing with correction
15:43 xavih dbruhn: it's not a problem, it is defined in a file, however it seems more difficult to change
15:43 xavih dbruhn: it's better to change 'remote-port' on client
15:43 dbruhn my clients and nodes are on the same machine is why I ask
15:43 xavih dbruhn: yes, 3.3.1
15:44 hagarth joined #gluster
15:45 xavih dbruhn: in subdirectory 'bricks' of the volume definition, you will find the ports that each brick listens from
15:45 dbruhn yep 24009
15:45 dbruhn how do you set it on the client?
15:45 dbruhn sorry for so many questions
15:46 xavih dbruhn: add the option 'remote-port' on the client volfile, inside each of the protocol/client xlators
15:46 Gugge joined #gluster
15:47 xavih dbruhn: anyway, are you trying to use rdma for testing purposes or to achieve better speed ?
15:48 dbruhn Better speed
15:48 dbruhn I have some I/O intensive stuff that I am looking to use it for
15:49 xavih dbruhn: in that case I would use IPoIB, and configure the bricks using tcp
15:49 dbruhn I have a QDR IB setup, and am at 99% random read loads
15:49 Nagilum is there a way to enable transport.socket.nodelay on an active gluster?
15:50 xavih dbruhn: rdma xlator supports inet-sdp address family (that I think should be faster than TCP)
15:50 dbruhn xavih, from what I can tell going from RDMA to IPoIB takes response latency from 1us to 3us
15:50 xavih dbruhn: however it has a bug that makes impossible to configure it
15:50 xavih dbruhn: so I think it's better to go with tcp
15:51 xavih dbruhn: yes, it's slower a bit, but I haven't found anything faster
15:52 xavih dbruhn: maybe someone who knows better the rdma implementation of gluster could help you better
15:52 dbruhn does the port change everytime?
15:52 dbruhn the only other person in here I have been able to find running IB is m0zes and he went back to 1gbE
15:52 xavih dbruhn: It didn't change while I was doing the tests, however I can't promise anything
15:53 dbruhn did you do any reboots through your testing?
15:53 xavih I don't remember well. Maybe once
15:53 rotbeard joined #gluster
15:54 semiosis ,,(processes)
15:54 glusterbot the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
15:54 semiosis you can see what port each brick is listening on by looking at the glusterfsd processes with ps/pgrep
15:54 semiosis also ,,(ports)
15:54 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
15:55 dustint joined #gluster
15:56 dbruhn semiosis when you create a volume does it never changes that port then? at least what I got from that?
15:57 dbruhn sorry poorly worded
15:57 dbruhn it sets the port and don't change it after that
15:58 semiosis 24007 & 24008 are constant, thats where glusterd listens for mgmt traffic from other glusterds and clients trying to mount
15:58 msmith_ Having another odd issue with gluster 3.3.1.  I have 6 servers, 1 brick per server (10 drives raid-6, xfs formatted) in a 2x replication setup (3x2) and mounting with NFS.  For this one file, it exists on server 1&2 as a link (getfattr shows both servers reporting the same values for trusted.gfid and trusted.glusterfs.dht.linkto), does not exist on server 3&4, and the real file exists on server 5&6 (getfattr reports the same gfid on b
15:58 msmith_ oth servers, they have the same size, tstamps and data).  The problem is that when I try and access the file through the mount, I get "Remote I/O error", and a directory listing shows the file as the following "-????????? ? ?                ?      ?            ? dovecot.quota"
15:58 msmith_ nfs is remote mounted
15:59 semiosis then there's a counter for brick processes (glusterfsd) which begins at 24009 and goes up for each brick on the server.  it's never decremented.
15:59 shylesh joined #gluster
15:59 dbruhn interesting, so the fact that the cluster is forming and the peers all seem to be happy means that is working
16:01 semiosis msmith_: what about the trusted.afr* xattrs?  ideally they should be all zero.  are they?
16:03 msmith_ trusted.afr.mail-client-4 and trusted.afr.mail-client-5 are all zero on servers 5 and 6.
16:06 mitzip joined #gluster
16:08 semiosis msmith_: pastie/gist your client log file
16:09 satheesh joined #gluster
16:10 _pol joined #gluster
16:18 bdperkin joined #gluster
16:19 JoeJulian http://global3.memecdn.com/Monday_o_93314.jpg
16:20 semiosis :O
16:21 msmith_ I'm not sure which log you are asking about.  the glusterfs/nfs.log?
16:21 semiosis msmith_: yeah that's a good one
16:21 xymox joined #gluster
16:23 Staples84 joined #gluster
16:28 msmith_ here's a 60 second window of the nfs.log from when that file went crazy.  http://pastie.org/private/rjforlwmnxejzjfq6l9w
16:28 glusterbot Title: Private Paste - Pastie (at pastie.org)
16:39 mooperd joined #gluster
16:42 Mo____ joined #gluster
16:48 _pol joined #gluster
16:50 mooperd joined #gluster
16:57 _pol joined #gluster
17:01 mynameisbruce joined #gluster
17:01 mynameisbruce_ joined #gluster
17:03 mynameisbruce_ left #gluster
17:28 tryggvil joined #gluster
17:29 Ryan_Lane joined #gluster
17:33 50UACDB44 joined #gluster
17:34 mohankumar joined #gluster
17:35 * jclift_ looks at http://www.ebay.co.uk/itm/251237955048 and wishes. :D
17:35 glusterbot Title: NEW MIS5025Q Mellanox Infiniband switch 36 port 2.88 Tb/s IS5025 QDR 40Gb/s | eBay (at www.ebay.co.uk)
17:35 stat1x joined #gluster
17:45 msmith_ semiosis: did that pastie come through and is that what you were looking for?
17:45 semiosis yeah sorry got distracted
17:46 semiosis not sure about that log, i would try stopping & starting the volume & unmounting/remounting the client(s)
17:46 semiosis shot in the dark
17:46 sjoeboo_ joined #gluster
17:47 johnmark jclift_: heh heh
17:47 johnmark make a bid :)
17:48 jclift_ johnmark: Well, the problem is if I _win_ the bid.  No food for me for the next few months. ;D
17:48 jclift_ Probably not the right trade off to make. :D
17:52 JoeJulian johnmark: fixing a failing kernel update on 200 systems over 25 locations... hopefully we can talk after I get this handled.
17:53 tru_tru joined #gluster
17:58 johnmark JoeJulian: ok. ouch :(
18:03 jdarcy__ joined #gluster
18:04 jdarcy_ joined #gluster
18:05 andreask joined #gluster
18:05 displaynone joined #gluster
18:07 msmith_ semiosis: that cleared up the problem this time, but that's not a very viable solution for a production mail storage
18:08 semiosis msmith_: iirc you were messing around with brick data to fix a split brain, is that right?
18:08 semiosis msmith_: or someone else?
18:09 semiosis msmith_: it's normal to have to do this kind of "restart" when mucking around with brick data, although hopefully you never have to do this kinda thing
18:11 msmith_ not me.  i had problems last week with localhost nfs.  mid friday I wiped and recreated the bricks/volume, then started feeding mail to it.  a few hours later it started having this issue.
18:11 semiosis hmm, ok
18:13 msmith_ and, i'm using dovecot proxy, so each user mailbox access is directed to a specific mount host, so it shouldn't be because i'm accessing the same files from multiple hosts.
18:15 GLHMarmot joined #gluster
18:18 manik joined #gluster
18:20 _br_ joined #gluster
18:24 sjoeboo no to keep pinging the chan, but anyone has any glustetr quota troubleshooting experience?
18:24 sjoeboo and/or a way to get clients to refresh quota info (not sizes, but whole new quota), without remounting?
18:25 johnmark sjoeboo: heya. I don't know that quotas are "hard" - at least, they didn't used to be
18:26 johnmark jdarcy_: ^^^
18:26 sjoeboo johnmark: what do you mean "hard" ? was in, enforced? or..."hard" to get working? :-)
18:26 johnmark haha... as in enforced
18:27 johnmark although the latter may also be true ;)
18:27 jdarcy Well, they can't be hard quotas because of the problems with dividing up between bricks (e.g. spurious ENOSPC because one brick's full while other's still have space).
18:27 jdarcy Still, they should be (and are) enforced as long as we know about them.
18:27 johnmark jdarcy: ah, ok. how does one use them? I sense a topic we need to address in the docs
18:28 jdarcy In 3.5 quota enforcement might move to the server BTW.  Doing it on the client is inherently insecure, plus the problem of clients not picking up changes without remount.
18:29 _pol joined #gluster
18:30 sjoeboo jdarcy: so thats the current state in 3.3.1 then ?That for clients to see new quotas on a given volume, it needs to be remounted?
18:30 _pol joined #gluster
18:30 sjoeboo no other way to get clients to refresh that data(some sort of volfile refresh?)
18:31 jdarcy sjoeboo: It might be possible to re-fetch the volfile by sending a SIGHUP to the glusterfs process, but I'm not sure if that's going to be any less disruptive - or even work at all, since I don't know of any tests for it.
18:31 jdarcy sjoeboo: There's code there to do that, so it might work, but best to test on an isolated setup first.
18:31 sjoeboo hrm, okay. i might be able to test...basically make a new quota, put it over with a "Stale" client, and then HUP that proc and see whats what..
18:31 semiosis sjoeboo: setting a volume option should trigger a client refresh
18:32 semiosis client-log-level to info for example
18:32 jdarcy semiosis: I'm not sure a quota change qualifies as a volume option in that sense.
18:32 sjoeboo semiosis: but adding a quota wouldn't qualify as that sort of event eh?
18:33 sjoeboo if the client option bit is true....that actually might be good. we;ve script most of our provisioning for storage, so fliping a log-level option quickly back and forth, if it does teh trick, would work for us...
18:33 semiosis idk for sure but assuming it's not, changing some *other* option like log level does trigger some kind of client config refresh
18:33 semiosis still not sure whether it's enough of a refresh to recognize a new quota, but worth trying
18:33 jdarcy Yep, worth a shot.
18:36 sjoeboo cool, i'll test a bit...
18:37 sjoeboo well, so, i can 100% confirm the control case of a brand new quota not being enforced if the client already has teh volume mounted and left as such
18:41 disarone joined #gluster
18:41 sjoeboo flipiing a volume option back and forth, also nothing so far
18:42 sjoeboo sending a HUP to glusterfs on the client didn't do it either
18:42 sjoeboo but, a umount/mount did
18:42 johnmark sjoeboo: ok. which was the worst case, right?
18:43 sjoeboo yeah
18:43 johnmark great
18:44 sjoeboo our goal was a single, large volume, with multiple quotas on it for individual groups/labs. if we could mount subvols w/ native gluster maybe this would be avoided, since we wouldn't mount the subvol until the quota is in place, so it would be there from the start, but, i know mounting subvolumes/folders isn't possible, only via nfs
18:48 johnmark sjoeboo: hrm. this sounds like one of our target use cases for 3.5
18:48 johnmark which doesn't help you now...
18:48 sjoeboo ha, no, it doesn't
18:49 johnmark sjoeboo: but basically, implementing multi-tenancy as jdarcy did with hekafs
18:51 _pol_ joined #gluster
18:53 _pol_ joined #gluster
18:55 _pol_ joined #gluster
18:56 disarone joined #gluster
18:57 rhys_ joined #gluster
18:58 mohankumar joined #gluster
18:58 rhys_ so this guide shows mounting gluster from /etc/glusterfs/glusterfs.vol http://www.howtoforge.com/high-availability-stor​age-with-glusterfs-3.0.x-on-debian-squeeze-autom​atic-file-replication-across-two-storage-servers but there is no mention of this in the 3.3 admin guide
18:58 glusterbot <http://goo.gl/CaGky> (at www.howtoforge.com)
18:59 rhys_ i don't see anything in /usr/share/doc/glusterfs-client/ either
19:00 semiosis rhys_: that's ancient, >2 year old instruction
19:00 semiosis ,,(rtfm)
19:00 glusterbot Read the fairly-adequate manual at http://goo.gl/E3Jis
19:00 semiosis things have changed a lot since 3.0
19:01 JoeJulian Ah, howtofail.com
19:01 semiosis lol
19:01 johnmark jdarcy: speaking of... perhaps we can enlist sjoeboo and friends to help us implement multi-tenancy
19:01 rhys_ alright. then a problem, how do I mount via a local file description? I have two servers in a cluster, connected by a bonded 10gE connections. They reboot at different times. they each are the gluster server and clients.
19:01 sjoeboo johnmark: we're already trying it, so I'm certainly good with "really" doing it and hleping!
19:02 johnmark sjoeboo: cool :)
19:02 semiosis rhys_: mount -t glusterfs localhost:volname /mount/point
19:02 johnmark jdarcy: ^^^^ we found a suc^H^H^Hvolunteer!
19:02 rhys_ ahh.
19:03 semiosis rhys_: but beware the possibility of split brain if they both write to the volume.  you should strongly consider using quorum -- which will force the clients read-only when one of the two servers is down.
19:03 johnmark semiosis: ah, good to know. I always just used the local machine's IP address. that looks simpler
19:03 rhys_ semiosis, i thought that was on by default
19:04 semiosis johnmark: possibly 127.0.0.1 is better than localhost, not sure.  ideally they both work well
19:04 semiosis rhys_: quorum is off by default in 3.3.x
19:04 semiosis rhys_: ,,(split brain)
19:04 glusterbot rhys_: I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
19:04 rhys_ we've discussed getting 3 so we can have reboot one and not have writes blocked
19:04 semiosis rhys_: ,,(split-brain)
19:04 glusterbot rhys_: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
19:05 JoeJulian johnmark: Uh-oh... I should probably get a warning for that one.
19:06 rhys_ semiosis, also splitbrain is not actually that big of an issue for us, since the only files that exist on it are VM images, and they can only have 1 VM writing to the image at a time.
19:07 semiosis rhys_: see glusterbot's link #1, split-brain "in time" may still be possible with alternating server failures
19:08 rhys_ semiosis, until we get Proxmox's fenced up and running with our PDUs, I think thats a chance we can take.
19:15 manik joined #gluster
19:16 johnmark JoeJulian: :)
19:16 johnmark JoeJulian: you mean well :)
19:39 JoeJulian johnmark: I'll apologize later. I'm sure it's partly stress talking.
19:39 sjoeboo so, no real solution to get clients to see new quotas but to remount eh? hrm...
19:51 xymox joined #gluster
19:52 johnmark sjoeboo: these are the moments where I look at the ground, shuffle my feet, and ignore the question
19:52 sjoeboo johnmark: ha, yeah, don't blame you
20:20 20WAB3KIT joined #gluster
20:20 20WAB3KIT whats the link for a bug submission
20:21 semiosis file a bug
20:21 glusterbot http://goo.gl/UUuCq
20:22 bdperkin joined #gluster
20:36 tryggvil joined #gluster
20:37 mtanner_ joined #gluster
20:40 furkaboo_ joined #gluster
20:40 frakt_ joined #gluster
20:40 20WAB3KIT Thanks semiosis, with help from xavih I got the RDMA stuff working, and entered a bug.
20:41 semiosis nice!
20:41 cicero_ joined #gluster
20:43 20WAB3KIT As far as I can tell without digging much deeper with RDMA the bricks aren't handing the port information back, while the clients trying to connect it just stalls, had to modify the brick settings for each of the clients with the correct port information and it started working :/
20:45 hagarth joined #gluster
20:47 Rydekull joined #gluster
20:47 Rydekull_ joined #gluster
20:47 tqrst if I kill a brick's glusterfsd process, is there any chance that the glusterd process might respawn it?
20:47 tqrst I just want to make sure my brick will stay dead while I do some maintenance
20:48 DWSR joined #gluster
20:48 semiosis tqrst: glusterd will spawn any missing glusterfsds when it starts up
20:49 tqrst semiosis: but not while it's running, right?
20:49 semiosis tqrst: so if you restart glusterd your brick process will come back.  you could also use iptables to drop traffic on the brick's port
20:49 ultrabizweb joined #gluster
20:49 semiosis tqrst: a 'gluster volume start <VOLNAME> force' from any server will spawn the missing processes too
20:50 tqrst semiosis: ok, thanks
20:50 semiosis those are the only ways i am aware of to cause glusterd to (re)spawn glusterfsd brick export daemons
20:54 tqrst and assuming I'm backing up the full contents of a brick to another hard drive, is this sufficient, assuming both are xfs? rsync -axlvXS /my/brick/mountpoint /myotherpartition
20:54 tqrst guess I should also include hard links with -H?
20:56 JoeJulian yep
20:57 glusterbot New news from newglusterbugs: [Bug 920332] Mounting issues <http://goo.gl/TYQNi>
20:59 mooperd joined #gluster
21:07 Rydekull joined #gluster
21:16 cyberbootje joined #gluster
21:21 dbruhn__ joined #gluster
21:23 Philip_ joined #gluster
21:42 awheeler_ joined #gluster
21:56 masterzen joined #gluster
22:10 hagarth joined #gluster
22:11 Humble joined #gluster
22:13 vigia joined #gluster
22:14 andres833 joined #gluster
22:24 * JoeJulian wishes he had the luxury of staging.
22:25 JoeJulian ... and time to review release notes.
22:25 JoeJulian Though one of my issues wasn't covered in the release notes.
22:28 glusterbot New news from newglusterbugs: [Bug 920369] Compilation failure on OSX due to missing pthread.h include <http://goo.gl/1flja>
22:42 Humble joined #gluster
22:44 hagarth joined #gluster
22:53 hattenator joined #gluster
22:58 glusterbot New news from newglusterbugs: [Bug 920372] Inconsistent ./configure syntax errors due to improperly quoted PKG_CHECK_MODULES parameters <http://goo.gl/ylPvK>
23:07 mitzip what's the best way to do quotas when your primary client is a windows box?
23:08 JoeJulian format and install linux! ;)
23:08 mitzip I agree! :-)
23:09 mitzip I want to use a gluster volume as datastorage for windows backups... I'm thinking of seeing if I can use the honor system and have windows manage how much space it's backups take up...
23:12 JoeJulian Since the quota system is directory based, just make each windows box backup into a separate directory.
23:17 duerF joined #gluster
23:28 bennyturns joined #gluster
23:30 mitzip JoeJulian: thanks!
23:46 mooperd joined #gluster
23:48 manik joined #gluster
23:57 tyl0r joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary