Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-08-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 sputnik13 joined #gluster
00:15 tyrok_laptop left #gluster
00:26 Pupeno joined #gluster
00:27 sputnik13 joined #gluster
00:31 gildub joined #gluster
00:49 bala joined #gluster
01:37 sputnik13 joined #gluster
01:43 sputnik13 joined #gluster
01:55 bit4man joined #gluster
01:55 bala joined #gluster
01:58 plarsen joined #gluster
01:59 harish_ joined #gluster
02:02 haomaiwa_ joined #gluster
02:08 overclk joined #gluster
02:13 bala joined #gluster
02:14 haomai___ joined #gluster
02:18 haomaiwa_ joined #gluster
02:20 overclk_ joined #gluster
02:27 purpleidea joined #gluster
02:33 haomai___ joined #gluster
02:49 tyrok_laptop joined #gluster
02:58 mjrosenb joined #gluster
03:01 RioS2 joined #gluster
03:19 RameshN joined #gluster
03:20 itisravi joined #gluster
03:48 kumar joined #gluster
03:52 kanagaraj joined #gluster
03:57 shubhendu_ joined #gluster
04:09 coredump joined #gluster
04:12 ndarshan joined #gluster
04:21 nthomas joined #gluster
04:31 Rafi_kc joined #gluster
04:32 nbalachandran joined #gluster
04:35 spandit joined #gluster
04:38 kshlm joined #gluster
04:40 marcoceppi joined #gluster
04:41 ppai joined #gluster
04:46 bmikhael joined #gluster
04:47 dusmant joined #gluster
04:48 hagarth joined #gluster
04:49 anoopcs joined #gluster
04:51 bmikhael Is there a way to implement memcached with GlusterFS???
05:00 ramteid joined #gluster
05:02 jiffin joined #gluster
05:07 itisravi joined #gluster
05:10 prasanth_ joined #gluster
05:19 prasanth_ joined #gluster
05:19 ppai joined #gluster
05:20 deepakcs joined #gluster
05:23 bala joined #gluster
05:28 azar joined #gluster
05:29 kdhananjay joined #gluster
05:30 edward1 joined #gluster
05:33 gildub joined #gluster
05:39 azar hello everybody, I want to know more about 'gluster test framework', I mean, I want to know what the tests purpose are. but some of them do not have any comments in their test file. for example test named 'bug-888752' said that its purpose is:
05:39 azar # Check if xml output is generated correctly for volume status for a single brick
05:39 azar # present on another peer and no async tasks are running.
05:39 azar but others not. how should I find out about these tests purposes??
05:44 tokern3 joined #gluster
05:45 azar ?
05:49 hagarth azar:  most test units have a bug ID as part of the file name
05:50 hagarth for example bug-888752 would refer to bz 888752
05:50 hagarth bug 888752
05:50 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=888752 unspecified, medium, ---, kaushal, CLOSED CURRENTRELEASE, "volume status" for single brick fails if brick is not on the server where peer command was issued.
05:50 hagarth thanks glusterbot!
05:50 cristov_macbook joined #gluster
05:51 hagarth azar: you can refer to the bug or associated patch for determining why a particular test case was written
05:52 azar hagarth: Thanks for your help :-)
05:52 karnan joined #gluster
05:54 cristov_macbook joined #gluster
05:56 bmikhael joined #gluster
05:56 rastar joined #gluster
05:57 dusmant joined #gluster
05:58 bmikhael_ joined #gluster
05:58 cris_mac joined #gluster
06:01 bmikhael_ any help for memcached with Gluster
06:05 bmikhael joined #gluster
06:05 hagarth bmikhael: what about memcached and gluster - how do you plan to integrate them?
06:06 cris_mac hi #gluster... could you tell me about linux kernel tuning for glusterfs ?
06:08 Eric_HOU joined #gluster
06:09 vshankar joined #gluster
06:10 raghu joined #gluster
06:14 vpshastry joined #gluster
06:15 lalatenduM joined #gluster
06:21 hagarth azar: np, feel free to let us know if you need more help
06:22 spandit joined #gluster
06:24 sahina joined #gluster
06:25 Pupeno joined #gluster
06:37 nshaikh joined #gluster
06:39 roeme^scs joined #gluster
06:41 roeme^scs hello #gluster!
06:42 meghanam joined #gluster
06:43 meghanam_ joined #gluster
06:43 roeme^scs a quick question for a replicated server setup: assuming servers A,B, and clients FUSE-mounting through, how are new mounts done if server A goes down?
06:43 roeme^scs *through A
06:44 roeme^scs essentially, we would like to have a behaviour as per the doc at http://www.gluster.org/community/documentation/index.php/NFS_Replication_and_Virtual_IP_Migration , but if possible without NFS.
06:44 glusterbot Title: NFS Replication and Virtual IP Migration - GlusterDocumentation (at www.gluster.org)
06:49 hchiramm roeme^scs, checking ur query
06:49 roeme^scs thx
06:49 SpComb roeme^scs: afaict `glusterfs nas.example.com:/bar` works correctly if the DNS name resolves to multiple hosts
06:49 SpComb roeme^scs: i.e. it will using any available one
06:50 andreask joined #gluster
06:50 hchiramm roeme^scs, yep.. it wil be serving from available server..
06:50 roeme^scs SpComb: so the client code will check all RR's?
06:50 roeme^scs that's would be great
06:51 roeme^scs this means we don't have to worry about virtual IP's and the like.
06:51 roeme^scs hchiramm: just to confirm, this is for _new_ mounts after the server has gone away, right?
06:51 hchiramm roeme^scs, I am confirming this thought..
06:52 roeme^scs maybe this would be useful to document.
06:53 roeme^scs (..or i just didn't find it so far, with the recent rework of the website)
06:55 hchiramm roeme^scs, native client (glusterfs-fuse) knows about all the existing nodes once volume is mounted, so can contact available server.
06:55 ctria joined #gluster
06:56 hchiramm roeme^scs, so u dont have to worry abt "VIPs" if u are with native client ..
06:57 edward1 joined #gluster
06:57 roeme^scs hchiramm: this i know, i am trying to determine the behaviour of "mount -t glusterfs serverA:/storage" where serverA is down, but replicated serverB is up.
06:58 roeme^scs it's expected to not work, however:
06:58 roeme^scs with the info SpComb provided, this can be made to work given a domain name serverC containing two A RR's resolving to both serverA and serverB.
07:00 hchiramm roeme^scs, ah.. I see ur mainly looking on new mounts ..
07:01 hchiramm before the mount glusterfs-fuse is not aware about the nodes
07:02 hchiramm so need to use the option for providing backup(secondary) node to mount, in case primary fails
07:02 roeme^scs yes.
07:06 hchiramm iirc ,  'backup-volfile-servers' is the option which helps u on that
07:07 ndevos roeme^scs: /sbin/mount.glusterfs is a script, you can verify the exact option there (I think it differs in some versions)
07:08 mbukatov joined #gluster
07:09 mbukatov joined #gluster
07:14 roeme^scs thanks.
07:14 hchiramm roeme^scs, np!
07:14 hchiramm yeah that option differs in some versions..
07:16 roeme^scs apparently, /sbin/mount.glusterfs is not around on a stock 6.5 centos with EPEL. hrm.
07:16 ndevos roeme^scs: no, it is part of the glusterfs-fuse package
07:17 hchiramm semiosis, ping
07:17 glusterbot hchiramm: Please don't naked ping. http://blogs.gnome.org/markmc/2014/02/20/naked-pings/
07:17 ndevos glusterbot++
07:17 glusterbot ndevos: glusterbot's karma is now 6
07:18 hchiramm ndevos, for that naked ping ? :)
07:18 dusmant joined #gluster
07:18 ndevos hchiramm: hehe, yes :)
07:18 roeme^scs oops, forgot that rpm. thanks, ndevos.
07:19 hchiramm ndevos, :)
07:21 nbalachandran joined #gluster
07:22 keytab joined #gluster
07:34 glusterbot New news from newglusterbugs: [Bug 1126734] Writing data to a dispersed volume mounted by NFS fails <https://bugzilla.redhat.com/show_bug.cgi?id=1126734>
07:36 fsimonce joined #gluster
07:41 edward1 joined #gluster
07:43 prasanth_ joined #gluster
07:52 ppai joined #gluster
07:59 nbalachandran joined #gluster
08:00 keytab joined #gluster
08:01 prasanth_ joined #gluster
08:05 doekia joined #gluster
08:05 doekia_ joined #gluster
08:05 doekia_ left #gluster
08:06 dusmant joined #gluster
08:10 ghghz joined #gluster
08:10 ghghz # rm -rf ok2
08:10 ghghz rm: cannot remove `ok2': Stale file handle
08:10 ghghz any idea?
08:18 hchiramm ghghz, which version of glusterfs ?
08:18 hchiramm its on fuse mount right ?
08:18 ghghz 3.5.0
08:19 hchiramm if its on non production, can u try the mount with --use-readdirp=no option ?
08:19 ghghz in logs I see this
08:19 ghghz remote operation failed: Stale file handle
08:19 hchiramm yep..
08:19 ghghz what does this option?
08:19 hchiramm because I see this .. https://bugzilla.redhat.com/show_bug.cgi?id=1041109
08:19 glusterbot Bug 1041109: urgent, unspecified, ---, csaba, NEW , structure needs cleaning
08:20 ghghz but in points to version 3.4.2
08:20 ghghz s/in/it
08:21 hchiramm https://bugzilla.redhat.com/show_bug.cgi?id=1041109#c18
08:21 glusterbot Bug 1041109: urgent, unspecified, ---, csaba, NEW , structure needs cleaning
08:21 hchiramm I think it applies to 3.5.x as well.
08:21 liquidat joined #gluster
08:23 * hchiramm afk ..
08:25 Eric_HOU left #gluster
08:38 richvdh joined #gluster
08:43 ghghz hchiramm: still having the same issue..
08:43 doekia joined #gluster
08:54 ghghz but it's interesting, it is valid only for directories
08:54 ghghz with files everything is OK
08:59 vikumar joined #gluster
09:05 hchiramm ghghz, was away for lunch
09:06 hchiramm iic, the issue is partially resolved .. Isnt it :)
09:06 prasanth_ joined #gluster
09:07 ghghz hchiramm: no, the same was before
09:07 hchiramm hmmm.. let me check
09:20 hchiramm kshlm, any idea whats going on ( stale file handle ) here  ^^ ?
09:26 kshlm hchiramm, I don't know about the fs part of gluster so much.
09:26 hchiramm kshlm, oh..ok.. nw.. ghghz, if possible, can u open a bug ?
09:26 kshlm But this seems to be some caching effect.
09:26 hchiramm yep. I though use-readdi* can help me..
09:26 ghghz hchiramm: sure, where should I open it?
09:26 hchiramm bugzilla.redhat.com
09:27 ghghz okey
09:27 hchiramm https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS ghghz
09:27 glusterbot Title: Log in to Red Hat Bugzilla (at bugzilla.redhat.com)
09:27 hchiramm though/thought
09:28 ndevos ghghz: normally a stale filehandle indicates that the file on the server side has been removed/renamed, and the client still has it cached
09:28 ghghz yes, but I cleared the inode/dentry cache
09:28 prasanth_ joined #gluster
09:28 ndevos ghghz: you should check on the bricks that hold the file, of the file is really there (and on all replicas)
09:28 ghghz how to check?
09:29 ghghz quickly :)
09:29 ndevos ghghz: maybe the file is still open? that would prevent dropping it from the caches
09:30 ndevos ghghz: you need to log in on the storage servers and check for the file in the filesystem that is used as a brick
09:30 ghghz does gluster has any inode/dentry cache itself?
09:31 ndevos something like it yes, but I dont think you can (or need to) flush it
09:32 ndevos you can tune it with some mount options, like attribute-timeout, see 'glusterfs --help'
09:34 ghghz entry-timeout i think this is for dentry cache
09:34 ghghz Time after which a dentry will timeout in the dentry cache.
09:34 ndevos yeah, that one
09:34 ghghz "in the dentry cache" leads to linux dentry cache or gluster? :)
09:35 ghghz hm, it's 1 second by default
09:35 ndevos I think it gets passed on to the fuse layer, so that would be the Linux dcache
09:35 ndevos but, I can be wrong ;)
09:37 ghghz hm, nevermind, will open as a bug ;-)
09:39 ndevos ghghz: when you so, include the status of the file on the bricks, and test if you can remove the file from a different mountpoint of the same volume
09:40 ppai joined #gluster
09:40 ndevos ghghz: also, when you have located the file on the bricks, execure 'getfattr -ehex -m. -d $path_to_file' so that the extended attributes can get checked
09:41 ndevos s/execure/execute/
09:41 glusterbot What ndevos meant to say was: ghghz: also, when you have located the file on the bricks, execute 'getfattr -ehex -m. -d $path_to_file' so that the extended attributes can get checked
09:42 ghghz will check this soon
09:48 karnan joined #gluster
09:49 nthomas joined #gluster
09:50 dusmant joined #gluster
09:50 ghghz ndevos, hchiramm one isterresting part
09:50 ghghz http://p.defau.lt/?84g0fb8_yR2arz8RKE9DWw
09:50 glusterbot Title: default paste at 2014-08-05 09:49:56 (at p.defau.lt)
09:50 ghghz Why it points only client-22,23,5
09:50 ghghz should I check there something?
09:51 ghghz hm, how to know what is "0-backup-hostinger-client-22"
09:52 ndevos ghghz: it looks like /server50/u616925024/files (or one of its parent dirs) are out-of-sync, ,,(split-brain)
09:52 glusterbot ghghz: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/
09:53 ghghz ndevos: but I use distrubuted, not replicated
09:53 ndevos ghghz: because that directory (or a parent) does no exist on all bricks, the file you want to remove gets different errors from the bricks
09:53 ndevos ghghz: distributed requires all directories to be located/replicated on all bricks
09:57 ghghz # gluster volume heal backup-hostinger
09:57 ghghz Volume backup-hostinger is not of type replicate
10:01 ghghz # getfattr -ehex -m. -d /backup/server50/u586648739/files/ok
10:02 ghghz returns nothing
10:02 ghghz empty
10:02 Slashman joined #gluster
10:05 ghghz ndevos: another question, how to know what is behind "0-backup-hostinger-client-X" ?
10:13 ndevos ghghz: you can find that in one of the .vol files under /var/lib/glusterd/vols/$VOLNAME/*.vol
10:13 harish joined #gluster
10:14 ghghz really, thank you
10:18 ghghz ndevos: thank you, solved
10:19 ndevos ghghz: nice :)
10:19 ghghz my xfs parition on one client was broken
10:20 ndevos oh, yeah, things like that would do it - what version are you running?
10:20 hchiramm 3.5.1
10:20 ghghz yep, somewhere 3.5
10:21 ndevos hmm, why would the brick process not get killed then? there is a health-checker in the brick process that should cause a failure when the filesystem goes unresponsive
10:21 ghghz interessting :)
10:21 ndevos ghghz: what xfs error did you find, and how are you mounting the filesystem?
10:22 ghghz Input/Output error
10:22 ghghz did fully unmount and remount
10:22 ghghz went ok
10:22 ndevos xfs does a fsck when mounting, so thats good
10:23 ghghz anyway it's odd, why it doesn't warn about such things trying to access that file
10:23 ndevos normally an "Input/Output error" would definitely cause the brick process to kill itself
10:24 ghghz understood
10:24 ndevos it probably did warn you in the logs of the brick, somewhere on the storage server under /var/log/gluster/bricks/
10:24 ghghz will double check it now
10:25 hchiramm ghghz, please feel free to open a bug if required..
10:26 kaushal_ joined #gluster
10:26 ghghz yeah, it warn about that.. [2014-08-05 10:25:32.807855] I [server-rpc-fops.c:154:server_lookup_cbk] 0-backup-hostinger-server: 48111: LOOKUP (null) (98d7a451-75f1-43d5-8d33-25e3f44f4514) ==> (Stale NFS file handle)
10:26 ghghz client returns the same to gluster what gets from NFS
10:35 glusterbot New news from newglusterbugs: [Bug 1126801] glusterfs logrotate config file pollutes global config <https://bugzilla.redhat.com/show_bug.cgi?id=1126801> || [Bug 1126802] glusterfs logrotate config file pollutes global config <https://bugzilla.redhat.com/show_bug.cgi?id=1126802>
10:39 JonathanD joined #gluster
10:41 ira joined #gluster
10:42 giannello joined #gluster
10:42 kkeithley1 joined #gluster
10:42 LebedevRI joined #gluster
10:42 SmithyUK Hi all, was doing some exploit testing on some of my servers yesterday and managed to send so many requests (404s) that glusterd used all 16 CPUS and filled 48GB of mem and 80GB of swap effectively locking me out of the server meaning I needed to reboot (900 days uptime nooo!). I'm hosting a gluster mount using httpd in order to serve data. Any ideas or suggestions as to how I can make this better? Thankfully I spotted it pretty quickly but in future I m
10:42 SmithyUK ight not be so lucky!
10:45 torbjorn__ SmithyUK: just a suggestion, but you might limit the number of concurrent requests you have going, so as not to overload the storage layer ?
10:47 SmithyUK torbjorn__: I have had a little play with Apache settings to try this, I haven't yet built up the guts to try another pen test! Cheers for the suggestion
10:49 ndarshan joined #gluster
10:55 B21956 joined #gluster
10:57 gildub joined #gluster
11:05 nthomas joined #gluster
11:05 dusmant joined #gluster
11:06 karnan joined #gluster
11:14 torbjorn__ SmithyUK: You're probably aware, but the setting MaxClients regulates how many connections Apache will try to answer at the same time .. any clients coming after that has reached the ceiling will receive 503 responses
11:21 kshlm joined #gluster
11:24 rtalur_ joined #gluster
11:25 diegows joined #gluster
11:27 tyrok_laptop I have a two node replicated Gluster volume running on caching RAID controllers on both bricks.  Testing the filesystems on each brick with Bonnie++, I get a per-character block latency of ~15ms and block write latencies of ~195ms.  Each brick has 4x 1GbE connections bonded with round robin so the load is spread over multiple physical NICs.
11:27 glusterbot tyrok_laptop: Bonnie's karma is now 1
11:27 tyrok_laptop If I mount the volume on one of them and run Bonnie++ on that volume, I get per-character latencies of ~650ms and block write latencies of ~30s.  I have yet to see the network links saturate, but generally am CPU-bound on one or the other of the bricks (they seem to alternate having a single glusterfsd thread pegged at 100% CPU).  Any ideas?
11:27 glusterbot tyrok_laptop: Bonnie's karma is now 2
11:30 tyrok_laptop I also tried mounting and testing from a different server than the bricks themselves.  Said other server has two bonded 1GbE NICs.  Same behavior and almost exactly the same latency.  dd'ing from this server from /dev/zero gets me 62.3MB/s on Gluster whereas when I dd zeroes from the bricks themselves directly to their filesystems (before setting them up as bricks), it was at about 280MB/s.
11:33 tyrok_laptop When I run Bonnie++ from this other server, both bricks normally have a single glusterfsd process at around 55% of one CPU core with intermittent times where one will go to 100% and the other to about 3%, and they alternate as to which one's pegged.  When pegged, network traffic on both bricks goes to near 0Mbps.
11:33 glusterbot tyrok_laptop: Bonnie's karma is now 3
11:34 ghghz ndevos, hchiramm https://bugzilla.redhat.com/show_bug.cgi?id=1126823 what I missed reporting this?
11:34 glusterbot Bug 1126823: unspecified, unspecified, ---, gluster-bugs, NEW , Remove brick if(op_errno == ESTALE)...
11:35 glusterbot New news from newglusterbugs: [Bug 1126814] rdma: transport tcp,rdma volume can't be mounted using tcp <https://bugzilla.redhat.com/show_bug.cgi?id=1126814> || [Bug 1126817] rdma: glusterd 'transport rdma' volume listens on port 24007, s/b 24008 <https://bugzilla.redhat.com/show_bug.cgi?id=1126817> || [Bug 1126823] Remove brick if(op_errno == ESTALE)... <https://bugzilla.redhat.com/show_bug.cgi?id=1126823>
11:36 tyrok_laptop I should note that in both cases I'm mounting using the Gluster native system and not NFS.
11:41 torbjorn__ tyrok_laptop: have you tried mounting using NFS and seeing if that affects throughput or latency in any way ?
11:42 tyrok_laptop torbjorn__: I have not yet.  That will be my next test.  But that is far less desirable for me because of the lack of automatic failover.
11:45 torbjorn__ tyrok_laptop: I hear you, and I'm in the same situation, more or less .. I haven't tried doing the math, but if you're looking at sync-ed writes like me, that throughput might be the theoretical maximum due to latency and having to wait for each write to be acknowledged before issuing the next one
11:46 tyrok_laptop torbjorn__: I'm currently testing using async in the mount parameters and write caching on the bricks.  Currently trying to get the performance to a place I'm comfortable with before adding safety features back in.
11:52 tyrok_laptop NFS mounting from the other server gives me 87.7MB/s on dd from /dev/zero.  Running Bonnie++ currently, and I'm seeing the same kind of alternating pegged brick behavior I was seeing earlier.
11:52 glusterbot tyrok_laptop: Bonnie's karma is now 4
11:59 B21956 joined #gluster
11:59 richvdh joined #gluster
12:01 Slashman_ joined #gluster
12:05 glusterbot New news from newglusterbugs: [Bug 1126831] Memory leak in GlusterFs client <https://bugzilla.redhat.com/show_bug.cgi?id=1126831>
12:05 harish_ joined #gluster
12:21 roeme^scs joined #gluster
12:22 rwheeler joined #gluster
12:26 bene2 joined #gluster
12:35 glusterbot New news from newglusterbugs: [Bug 1126837] Performance degradation in SMB over VFS when compared to FUSE <https://bugzilla.redhat.com/show_bug.cgi?id=1126837>
12:40 skippy joined #gluster
12:41 tyrok_laptop torbjorn__: Bonnie++ just finished.  Looking at about 19ms per-character latency using NFS and 97s block write latency.
12:41 glusterbot tyrok_laptop: Bonnie's karma is now 5
12:44 skippy where can I find more details about the consequences of a lower network.ping-timeout ?  I see the reports that "reconnect is expensive", but that's not very detailed.
12:48 ppai joined #gluster
12:59 nbalachandran joined #gluster
13:01 lalatenduM joined #gluster
13:02 marcoceppi joined #gluster
13:03 bennyturns joined #gluster
13:06 julim joined #gluster
13:09 purpleidea joined #gluster
13:11 skippy where can I find more details about the consequences of a lower network.ping-timeout ?  I see the reports that "reconnect is expensive", but that's not very detailed.
13:12 skippy ,,ping
13:12 glusterbot skippy: Please don't naked ping. http://blogs.gnome.org/markmc/2014/02/20/naked-pings/
13:12 skippy err...
13:12 skippy the bot has some shortcut to the ping timeout issue; but I don't recall what it is. :(
13:14 JordanHackworth joined #gluster
13:14 marcoceppi joined #gluster
13:14 marcoceppi joined #gluster
13:15 kkeithley_ ,,(ping)
13:15 glusterbot I do not know about 'ping', but I do know about these similar topics: 'ping-timeout'
13:15 skippy ,,ping-timeout
13:15 skippy thanks kkeithley_
13:15 kkeithley_ ,,(ping-timeout)
13:15 glusterbot The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
13:15 dusmant joined #gluster
13:15 mbukatov joined #gluster
13:15 Pupeno joined #gluster
13:15 overclk joined #gluster
13:15 kumar joined #gluster
13:15 tyrok_laptop joined #gluster
13:15 R0ok_ joined #gluster
13:15 the-me joined #gluster
13:15 foster joined #gluster
13:15 dreville joined #gluster
13:15 pdrakewe_ joined #gluster
13:15 semiosis joined #gluster
13:15 m0zes joined #gluster
13:15 AaronGr joined #gluster
13:15 theYAKman joined #gluster
13:15 saltsa joined #gluster
13:15 jezier_ joined #gluster
13:15 d-fence_ joined #gluster
13:15 sspinner joined #gluster
13:15 jcsp1 joined #gluster
13:15 cyberbootje joined #gluster
13:15 Nowaker joined #gluster
13:15 eryc joined #gluster
13:15 morse_ joined #gluster
13:15 sauce joined #gluster
13:15 eshy joined #gluster
13:15 marcoceppi joined #gluster
13:15 marcoceppi joined #gluster
13:15 tyrok_laptop1 joined #gluster
13:16 skippy yet, if one of my Gluster servers dies, I don't really want my clients to block all Gluster I/O for 42 seconds... that rather defeats the purpose of a distributed replicated volume, I think.
13:16 chirino joined #gluster
13:19 torbjorn__ tyrok_laptop1: I'm afraid I don't really know what a reasonable number would be, although I'm quite interested in finding a procedure for finding one .. is the Bonnie test a char+flush, so it's basically a sequential write on a synced file ? .. if you're not worried about the sync bit, maybe you can enable the write-behind xlator to get better performance ?
13:20 tyrok_laptop1 torbjorn__: Don't know how Bonnie++ is doing character/block writes, but I do already have the write-behind translator enabled and configured.
13:20 glusterbot tyrok_laptop1: Bonnie's karma is now 6
13:20 torbjorn__ skippy: on that issue, are you doing VM hosting ? .. I'm still on the 42 second default, and on failovers, actively writing VM's will go to "my block device failed, so I won't write to  the filesystem anymore" .. you wouldn't happen to know how long the kernel waits for the block device before going to that mode ?
13:21 skippy torbjorn__: I'm not (currently) doing VM hosting.  I'm looking to replace a NAS with something more resilient, so this will be for regular file I/O.
13:21 skippy I don't really want my applications blocking for 42 seconds; but I also don't want to dig a big hole for myself with too short of a timeout...
13:22 torbjorn__ tyrok_laptop1: are you seeing any differences if you toggle the write-behind xlator on or off ? .. If I'm remembering correctly, there was a version of Gluster that would disable a bunch of xlators of the file seemed to be opened with O_DIRECT or something like that
13:23 torbjorn__ skippy: I have the impression that dis- and re-connects are very rare, and usually related to "real" problems, be that hardware or software .. I'm considering moving my timeout down to a couple of seconds, but I'm unsure how small it has to be to avoid the "I lost my block device!" kernel situation
13:23 tyrok_laptop1 torbjorn__: I have not tried it without it.
13:24 torbjorn__ skippy: my logs haven't got any reconnects in them that aren
13:24 torbjorn__ skippy: my logs haven't got any reconnects in them that aren't actual problems, at least
13:24 partner joined #gluster
13:24 torbjorn__ tyrok_laptop1: might consider giving that a go, to ensure you're actually getting the effect as intended from your configuration
13:25 skippy I don't expect my network to be jittery, and I don't expect my servers to be flapping.  So I'd rather clients keep talking to whatever servers are available in the event of a real problem...
13:25 torbjorn__ tyrok_laptop1: or maybe increase the write-behind xlator cache, if you can stomach the possibly-lost-writes situation
13:25 torbjorn__ skippy: IMHO you'd be fine with a timeout of a second or two .. YMMV, obviously
13:25 dreville joined #gluster
13:26 JordanHackworth joined #gluster
13:26 skippy torbjorn__: that's my impression, but I'd like to make sure I'm not doing something obviously dumb.  I'm new to Gluster, and the available documentation is painfully thin in places. :(
13:27 torbjorn__ skippy: indeed it is, it's been trial by fire on my part a couple of times .. and I'm doing VM hosting, so it hurts a bit when stuff doesn't work
13:27 torbjorn__ that being said, Gluster works very well once a workable configuration is in place
13:30 marcoceppi joined #gluster
13:30 marcoceppi joined #gluster
13:32 tokern3 left #gluster
13:34 ninthBit joined #gluster
13:35 ninthBit amazon's virtual computers sometimes are given with no swap.  how important is a swap with a gluster server or with the glusterfs-client (fuse) ?
13:36 SpComb how can I share a gluster volume as read-only to certain hosts?
13:37 skippy mount them ro from the guest, SpComb ?
13:37 plarsen joined #gluster
13:37 skippy mount -t glusterfs -o ro server:/vol /mnt
13:37 bit4man joined #gluster
13:37 SpComb as in, I do not trust those hosts enough to given them rw access
13:38 SpComb I guess there's no per-host options like that in glusterfsd?
13:49 lalatenduM joined #gluster
13:50 tyrok_laptop1 SpComb: IIRC there are tunable settings for access control.  You'd have to check the docs for them, though.  I don't have a direct link.
13:50 skippy SpComb: https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_managing_volumes.md#tuning-options
13:50 glusterbot Title: glusterfs/admin_managing_volumes.md at master · gluster/glusterfs · GitHub (at github.com)
13:50 skippy nothing per-client.  it's either "gets access" or "does not get access".  doesnt control read/write
13:51 SpComb only thing I can find is http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#auth.allow
13:51 glusterbot Title: Gluster 3.2: Setting Volume Options - GlusterDocumentation (at gluster.org)
13:52 SpComb yeah, doesn't seem like its possible to export read-only to specific hosts
13:52 skippy you could use NFS, and export a read-only mount to the clients.  or you could mount them read-only on the client.
13:53 skippy but if you don't trust the client to  begin wiht, the latter option probably won't work
13:53 lalatenduM joined #gluster
13:59 SpComb shame, it would make glusterfs more useful when crossing trust boundries
14:06 marcoceppi joined #gluster
14:07 mortuar joined #gluster
14:11 wushudoin joined #gluster
14:12 marcoceppi joined #gluster
14:12 marcoceppi joined #gluster
14:12 JordanHackworth joined #gluster
14:15 kshlm joined #gluster
14:33 hagarth joined #gluster
14:40 morse joined #gluster
14:43 tdasilva joined #gluster
14:53 lalatenduM joined #gluster
14:55 nueces joined #gluster
14:56 kanagaraj joined #gluster
14:58 bennyturns joined #gluster
14:59 tdasilva_ joined #gluster
15:01 wushudoin joined #gluster
15:10 rotbeard joined #gluster
15:17 bala joined #gluster
15:22 firemanxbr joined #gluster
15:24 firemanxbr joined #gluster
15:28 chirino joined #gluster
15:33 sputnik13 joined #gluster
15:34 sputnik13 joined #gluster
15:36 bala joined #gluster
15:36 theron joined #gluster
15:39 edward1 joined #gluster
15:41 dtrainor joined #gluster
15:42 dtrainor joined #gluster
15:42 sputnik13 joined #gluster
15:43 sputnik13 joined #gluster
15:46 tyrok_laptop1 torbjorn__: Getting closer, I think.  I get *much* better throughput by limiting my I/O threads to 1.  Any more than that (even 2), and I can watch on a brick while I get one thread taking 100% CPU and another 100% in I/O wait on a different core.
15:49 mortuar joined #gluster
15:50 Peter2 joined #gluster
15:55 lpabon_test joined #gluster
15:57 bmikhael joined #gluster
16:01 coredump joined #gluster
16:01 lpabon joined #gluster
16:02 lpabon joined #gluster
16:06 glusterbot New news from newglusterbugs: [Bug 1126932] Random crashes while writing to a dispersed volume <https://bugzilla.redhat.com/show_bug.cgi?id=1126932>
16:19 bala joined #gluster
16:23 systemonkey joined #gluster
16:24 ramteid joined #gluster
16:34 vipulnayyar joined #gluster
16:43 zerick joined #gluster
16:44 theron joined #gluster
16:46 theron joined #gluster
16:47 XpineX joined #gluster
16:54 ctria joined #gluster
16:59 ninkotech joined #gluster
17:07 ninkotech_ joined #gluster
17:07 ninkotech joined #gluster
17:10 huleboer joined #gluster
17:15 huleboer joined #gluster
17:21 ninkotech_ joined #gluster
17:22 jbrooks joined #gluster
17:29 prasanth_ joined #gluster
17:31 huleboer joined #gluster
17:33 recidive joined #gluster
17:35 ninkotech_ joined #gluster
17:36 ricky-ti1 joined #gluster
17:37 ninkotech__ joined #gluster
17:44 richvdh joined #gluster
17:44 edward1 joined #gluster
17:52 geewiz joined #gluster
17:54 geewiz Hi! We're running into a memory leak with Gluster 4.4. To which version should we upgrade?
17:56 JoeJulian That would depend on where the memory is leaking from and whether or not the leak has been found and fixed.
17:57 geewiz How would we investigate this?
17:58 geewiz Sorry, it's gluster 3.4.4.
17:58 JoeJulian memory leaks can be found using valgrind. git logs can tell you what's been patched between versions.
17:59 JoeJulian fwiw, I have a leak in 3.4.4 also, and I think the only memory leak that was fixed in 3.4.5 is for nfs.
17:59 B21956 joined #gluster
18:00 JoeJulian You could /try/ 3.5.2, but I obviously wouldn't guarantee that a leak not found would be fixed there.
18:00 geewiz This bug report would fit our symptoms quite well: http://review.gluster.org/#/c/5392/
18:00 glusterbot Title: Gerrit Code Review (at review.gluster.org)
18:02 theron joined #gluster
18:02 JoeJulian You can trigger a dump (make sure the /var/run/gluster directory exists) with pkill -USR1 glusterfs
18:04 JoeJulian In that, you can see which structure has the most memory allocated.
18:04 theron joined #gluster
18:04 geewiz Would that negatively affect operation? IoW, can I do that while in production?
18:05 JoeJulian You can do that in prod.
18:05 geewiz What size will that dump have?
18:05 nueces joined #gluster
18:06 geewiz Does it correlate with the amount of RAM gluster is using?
18:08 JoeJulian No, depends on open fd and locks. Just checked one of my compute nodes, and the client dump took 222k.
18:09 geewiz Okay, no worries then.
18:09 recidive joined #gluster
18:09 geewiz So, I assume the dump is plain text.
18:09 JoeJulian yes
18:10 geewiz When I have identified the structure, how do I proceed?
18:11 JoeJulian file a bug
18:11 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
18:12 JoeJulian especially if you can cause the leak
18:12 JoeJulian valgrind reports are especially useful in that case as well.
18:12 NuxRo http://blog.gluster.org/2014/02/migrating-vms-from-windows-server-2008-r2-to-windows-server-2012-r2/ <- WTF
18:12 NuxRo has gluster been aquired by M$? :)
18:13 NuxRo *acquired
18:13 geewiz Would it make sense to try a 3.5 client to see if it improves things?
18:13 JoeJulian Blog aggregation fail.
18:14 JoeJulian s/aggregation/syndication/
18:14 glusterbot What JoeJulian meant to say was: Blog syndication fail.
18:14 JoeJulian Too many simultaneous conversations...
18:16 JoeJulian geewiz: If you have a client you can try that with, sure!
18:16 geewiz 3.5.x clients work with a 3.4.4 server?
18:16 JoeJulian yes
18:17 JoeJulian But you have to let me know your results.
18:17 geewiz Brilliant, will do. Thanks a lot! (BTW, your blog saved my bacon last week.)
18:20 geewiz A volume wouldn't start because the root xattr was missing. I restored it from the info file.
18:23 semiosis NuxRo: that blog post seems quite off topic!
18:29 XpineX joined #gluster
18:31 NuxRo semiosis: no kidding :)
18:31 NuxRo not even related to linux
18:47 CyberMage joined #gluster
18:48 CyberMage Hey guys n gals.  Just upgraded one server running a single mirrored brick from 3.3 to 3.4.  Now it refuses to play nice with the rest of the 3.3 cluster.  Getting "Peer Rejected (Connected)".
18:49 CyberMage Can someone point me in the right direction here?  All googled out.
18:51 CyberMage The freshly upgraded server shows all other peers as "Peer Rejected" and all the other peers show each other as "Peer in Cluster" but the upgraded server as "Peer Rejected (Connected)"
18:57 semiosis NuxRo: http://blog.gluster.org/2013/04/grilling-the-perfect-steak/
18:57 semiosis WTAF?
18:57 ninthBit the gluster volume replace-brick command.  after it has completed and replace-brick commit is executed will gluster update the replication links from the old to the new brick for the other bricks that had replication references? is this question clear?
18:58 semiosis glusterbot: ping
18:58 glusterbot pong
18:58 semiosis glusterbot: where's my title?!
18:58 zerick joined #gluster
19:02 CyberMage So has anyone done a rolling upgrade from 3.3 to 3.4?
19:08 XpineX joined #gluster
19:16 ninthBit what does gluster do about a volume still online and available during the replace-brick command running?  does it write files to this brick?  if so which brick the new one or the replication target?  what happens during the commit exactly?
19:19 skippy when a client uses the native protocol to read files, does the client perform the hashing to find the correct volume, or does the client talk to a server to get that info?
19:19 skippy s/volume/brick(s)/
19:19 glusterbot What skippy meant to say was: when a client uses the native protocol to read files, does the client perform the hashing to find the correct brick(s), or does the client talk to a server to get that info?
19:20 JoeJulian @lucky dht
19:20 glusterbot JoeJulian: http://en.wikipedia.org/wiki/Dihydrotestosterone
19:20 JoeJulian damn
19:21 JoeJulian @lucky distributed hash translation
19:21 glusterbot JoeJulian: http://en.wikipedia.org/wiki/Distributed_hash_table
19:21 JoeJulian skippy: ^
19:21 JoeJulian Also http://joejulian.name/blog/dht-misses-are-expensive/
19:21 glusterbot Title: DHT misses are expensive (at joejulian.name)
19:24 semiosis JoeJulian: gluster's "DHT" isn't really a distributed hash table
19:25 skippy i get that the client can hash the file.  Since the client is using the native protocol, does it have all the data it needs to calculate the hash for the bricks?  Or must the client query a server for that?
19:25 semiosis skippy: the fuse client knows how to compute the hash & goes directly to the brick for a file.  in the odd case where the file is really somewhere other than where it should be, the client may find a "linkfile" pointing to another brick, or else it will poll all the bricks looking for the file, which is why negative lookups are expensive iirc
19:25 skippy awesome.  thank you.
19:25 semiosis skippy: best case scenario it has all the info it needs, but there are also those other cases i mentioned
19:26 skippy sure.
19:26 semiosis @negative lookup
19:26 semiosis something's been written about that, let me find it
19:26 semiosis @php
19:26 glusterbot semiosis: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH --fopen-keep-cache
19:27 skippy just preparing some basic info for my colleagues. i dont need to get into the weeds of the protocol yet.
19:27 semiosis ok
19:27 CyberMage @semiosis should I be able to do a rolling upgrade from 12.04 and your PPA to 14.04 and your PPA?  Or am I missing a step?
19:27 semiosis CyberMage: i really dont know. personally, i never trust rolling upgrades even when told it should work
19:28 CyberMage lol nice
19:28 CyberMage Unfortunately I use gluster because I have 24x7 access to the data from around the world and it's all mission critical, so downtime is... hard.
19:29 CyberMage The two versions definitely aren't liking each other for me though.
19:29 ninthBit would there not be a unit test or integration test that validates rolling/hot upgrades?
19:30 semiosis CyberMage: i'd guess that if you were keeping the gluster versions the same, only upgrading the ubuntu release one server at a time, it just might work
19:30 semiosis CyberMage: but mixing gluster versions is always a gamble
19:30 semiosis sadly
19:32 CyberMage that's life I guess.  It appears I have no real option for upgrading Gluster than downtime though.  My other concern is even if I do the downtime will these servers magically talk after I do the ugprades?  Guess I need to do some more tests with my two test servers.
19:32 semiosis CyberMage: what versions of gluster are you working with?
19:32 semiosis what do you want to upgrade to?
19:32 CyberMage 3.3 PPA and 3.4 PPA
19:32 semiosis hmm
19:32 CyberMage trying to move to 3.4 one at a time
19:32 CyberMage I'll get full version #s...
19:33 semiosis there was a time 3.3 & 3.4 could interop, as indicated by ,,(3.4 upgrade notes) but idk if that still works since the newer 3.4 releases :/
19:33 glusterbot http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4/
19:33 CyberMage 3.3.2-ubuntu1~precise2 to 3.4.4-ubuntu1~trusty1
19:33 CyberMage Well those bastages need to fix the release notes - that's the only reason I tried this
19:34 semiosis also note 3.4.5 is out i just havent had a chance to package it yet :(
19:34 CyberMage I'm fine staying a bit behind (as you can see by just now upgrading Ubuntu) as long as things work :-)
19:35 ninthBit semiosis: i would call you a slacker but i am a consumer of your repository :)
19:35 CyberMage I wonder, can I downgrade the one server back to 3.3.2 while still on Trusty?
19:35 CyberMage Until I can schedule downtime?
19:35 semiosis ninthBit: yeah, it's not like i have anything else to do
19:36 semiosis CyberMage: idk
19:36 CyberMage Just think, without tying him up on IRC we'd probably have bleeding edge builds in the PPA ;-)
19:40 qdk joined #gluster
19:40 JoeJulian Didn't the 3.3->3.4 migration require the servers be upgraded first? I seem to recall that.
19:43 CyberMage @JoeJulian if so the release notes don't say that.  In fact they say you can do rolling-no-downtime upgrades as promised the last time they broke compatibility.
19:44 ninthBit i have a distributed replicated volume.  would the replace-brick start->commit update the brick replica pairing to the new brick? say if the replaced brick was a replica.  would the primary brick be linked to the new replica brick with that command or is there further configuration required to setup the replica brick pairing?
19:44 JoeJulian Actually, the upgrade instructions semiosis posted earlier say just that.
19:46 JoeJulian ninthBit: the "preferred" method from the developers is to skip the start, wait, commit process and just go straight to "commit force" and allow replication to populate the new brick.
19:46 CyberMage mmm seems fuse-utils doesn't exist in 14.04 so can't just install the dpk's
19:46 JoeJulian fwiw, I'm not sure the non-deprecated process ever worked 100% right anyway.
19:52 ninthBit JoeJulian: how does using commit force compare to the instructions in your blog for replacing a GlusterFS Server: best practice that use the start/wait/commit approach?  That is kind of what I am doing is going to be replacing a server but everything is new. new server, hostname, brick, ip.. etc.  i am thinking i will join it to the cluster and then replace the bricks from the target to the new.
19:54 JoeJulian I'm completely unhappy with the decision to deprecate the start/wait/commit approach, but in some recent attempts to use that, it caused all the bricks that were to be affected by the change to hang. I think they've stopped testing that method and so we're stuck with the commit force.
19:55 CyberMage @semiosis for reference I took a new server (not the one I need to work yet) and downgraded to  3.3.2-ubuntu1~raring2 (running on Trusty) and had it join the cluster and it seems to be talking fine.
19:58 gildub joined #gluster
20:00 halfinhalfout joined #gluster
20:05 firemanxbr joined #gluster
20:05 ninthBit JoeJulian: i can understand the frustration.  i am a bit frustrated with the online documentation at gluster.org.  getting 404 on links and trying to determine if hits on 3.1 and 3.2 really are current.  Then there is the github repo for documentation. my solution is to collect and gather what i find and read it all to flush out what might be the correct method method for what i want to do.
20:07 ninthBit jojulian: my google foo actually found a post to gluster-users that talks about the change in replace-brick.  had to use google cache
20:07 JoeJulian Anything that's 404 was probably pointing at a knowlege base website that I think went out of business.
20:08 ninthBit joejulian: but it was in the cloud? that is supposed to last forever right.. ha ha ha... oh man
20:09 JoeJulian If you try to replace-brick...start with 3.4.3+ it'll tell you that the command is deprecated and suggest commit force.
20:09 halfinhalfout semiosis: know when glfsheal binary might be part of the ubuntu 3.5.1 PPA? ref: https://bugzilla.redhat.com/show_bug.cgi?id=1113778
20:09 glusterbot Bug 1113778: medium, unspecified, ---, pkarampu, ASSIGNED , gluster volume heal info keep reports "Volume heal failed"
20:10 ninthBit oh good. i am somewhere in 3.4 version.. i probably would have found that once i got started :)
20:11 CyberMage @semiosis scratch that - it's refusing to start the NFS server on Trusty running the older PPA 3.3.2
20:13 bene2 joined #gluster
20:23 systemonkey joined #gluster
20:23 semiosis halfinhalfout: http://goo.gl/qZTngS
20:24 Pupeno joined #gluster
20:25 JoeJulian Here's an interesting question... "I'm testing performance and when I write a file that's 400Gb, after I exceed all the caching I get blah blah blah."  Is that even a valid statement? Isn't the whole purpose of caching to improve performance? Who cares how slow you can make it unless your use case actually makes it that slow?
20:25 ninthBit is there a gluster command to output the brick replica pairings?  i want to use this to double check what gluster is reporting when i attempt this replace brick
20:28 halfinhalfout semiosis: thx
20:29 ninthBit the low tech method was to write a large file and see what disks the files appear on and then check its expected mirror..... :P
20:32 ninthBit i guess i could go with the replica pattern and use that when viewing the gluster volume info all output ?
20:36 skippy left #gluster
20:37 semiosis JoeJulian: makes most sense when comparing hardware performance, say between two hdd/ssds
20:39 cmtime joined #gluster
21:02 Pupeno joined #gluster
21:06 fignew joined #gluster
21:22 chirino joined #gluster
21:28 diegows hi
21:28 glusterbot diegows: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
21:28 diegows ok glusterbot
21:29 diegows I have an issue with my two server cluster that I think it's related to a out of space issue that you had
21:29 diegows we had
21:29 diegows # gluster volume start gv0
21:29 diegows volume start: gv0: failed: Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /data/brick1/gv0. Reason : No data available
21:29 diegows getfattr on the bricks returns nothing
21:30 diegows I'm reading a document about how to recover a failed server, but it talks about replacing a new server with the same name AFAIK
21:30 diegows but I don't want to do that, I have both server with the data there, I want to recover them
21:30 diegows shall "Brick Restoration - Replace Crashed Server" work in this case?
21:31 diegows 8 TB of data, recover from backup is a big problem here, takes days
21:33 dberry joined #gluster
21:44 diegows fixed following that procedure :)
22:14 RobertLaptop joined #gluster
22:14 mibby joined #gluster
22:15 dtrainor joined #gluster
22:15 tdasilva_ joined #gluster
22:24 jbrooks joined #gluster
22:33 Pupeno joined #gluster
22:36 coredump joined #gluster
22:45 coredump joined #gluster
22:50 diegows any idea why this could be failing?
22:50 diegows setfattr -n security.NTACL -v foo testfile
22:50 diegows on a gluster mounted volume
22:50 diegows look like it's an issue with that prefix, other attrs are written fine
22:53 sputnik13 joined #gluster
23:02 theron joined #gluster
23:30 plarsen joined #gluster
23:30 bit4man joined #gluster
23:41 JoeJulian @mount server
23:41 glusterbot JoeJulian: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
23:44 JoeJulian file a bug
23:44 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
23:47 JoeJulian bug 986429
23:47 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=986429 medium, medium, ---, fharshav, CLOSED CURRENTRELEASE, Backupvolfile server option should work internal to GlusterFS framework
23:58 azalime joined #gluster
23:59 azalime Hi guys, i'm having a nasty issue with glusterfs not exporting volumes through nfs
23:59 azalime it works for a single volume but not for other volumes
23:59 azalime any idea?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary