Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-02-17

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 theron joined #gluster
00:18 theron joined #gluster
00:26 _polto_ joined #gluster
00:33 MugginsM joined #gluster
00:56 coredump joined #gluster
00:58 genial left #gluster
01:19 badone_ joined #gluster
01:28 MugginsM joined #gluster
01:54 tessier joined #gluster
01:56 tessier joined #gluster
02:07 _polto_ joined #gluster
02:10 chirino joined #gluster
02:10 tessier joined #gluster
02:19 plarsen joined #gluster
02:26 bala joined #gluster
02:28 mdavidson joined #gluster
02:32 sprachgenerator joined #gluster
02:34 harish joined #gluster
02:49 ilbot3 joined #gluster
02:49 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:50 joseki joined #gluster
02:51 joseki i'm looking (like many) to move from nfs to gluster and had a few questions i couldn't find good answers to. i'm curious, with the nfsv3 "emulation mode", can i still use something like cachefilesd for local cache?
02:52 bharata-rao joined #gluster
02:52 joseki also, looking to start with two machines in distributed model (each machine has a raid6 volume), is it okay to also run code on one of the boxes that mounts the share locally and uses it like a normal local storage device?
02:58 nishanth joined #gluster
02:58 theron joined #gluster
03:17 cornusammonis joined #gluster
03:40 kanagaraj joined #gluster
03:51 itisravi joined #gluster
03:56 atinmu joined #gluster
03:56 RameshN joined #gluster
03:56 hagarth joined #gluster
04:01 shubhendu joined #gluster
04:06 rejy joined #gluster
04:08 jackdpeterson joined #gluster
04:08 nbalacha joined #gluster
04:11 jackdpeterson Hello all, experiencing some nasty gluster 3.6.2. Simply removing files is causing weirdness -- Input Output errors on NFS clients to the gluster 3.6.2 server
04:12 rafi joined #gluster
04:21 jackdpeterson the errors include 'nfs server timed out' -- even though it's responding perfectly fine. then mounts die after rming these files in certain directories
04:23 schandra joined #gluster
04:33 rjoseph|afk joined #gluster
04:33 anoopcs joined #gluster
04:37 Manikandan joined #gluster
04:37 Manikandan_ joined #gluster
04:40 schandra joined #gluster
04:41 hagarth joined #gluster
04:42 gem joined #gluster
04:42 kshlm joined #gluster
04:43 prasanth_ joined #gluster
04:53 suliba joined #gluster
04:53 ndarshan joined #gluster
04:55 Manikandan joined #gluster
04:55 jackdpeterson looking into this further, it looks like it may be self-heal related. The directory that is having issues is continuing to indicate that it is 'possibly undergoing self-heal'
05:04 JordanHackworth joined #gluster
05:07 JordanHackworth joined #gluster
05:15 lalatenduM joined #gluster
05:15 spandit joined #gluster
05:17 maveric_amitc_ joined #gluster
05:22 ppai joined #gluster
05:23 bala joined #gluster
05:27 prasanth_ joined #gluster
05:29 kumar joined #gluster
05:30 meghanam joined #gluster
05:30 hagarth joined #gluster
05:30 jiffin joined #gluster
05:33 rafi1 joined #gluster
05:33 karnan joined #gluster
05:37 jiffin1 joined #gluster
05:43 nhayashi joined #gluster
05:46 hchiramm joined #gluster
05:47 nbalacha joined #gluster
05:47 jiffin joined #gluster
05:52 rafi joined #gluster
05:52 bharata_ joined #gluster
05:58 atalur joined #gluster
06:08 kdhananjay joined #gluster
06:11 jiffin1 joined #gluster
06:11 overclk joined #gluster
06:14 rafi1 joined #gluster
06:14 anil joined #gluster
06:22 itpings joined #gluster
06:22 itpings hey guys
06:23 rafi joined #gluster
06:24 hagarth joined #gluster
06:25 nbalacha joined #gluster
06:31 raghu joined #gluster
06:35 TvL2386 joined #gluster
06:37 nshaikh joined #gluster
06:39 kdhananjay joined #gluster
06:42 glusterbot News from newglusterbugs: [Bug 1193298] [RFE] 'gluster volume help' output could be sorted alphabetically <https://bugzilla.redhat.com/show_bug.cgi?id=1193298>
06:46 rjoseph|afk joined #gluster
06:59 mbukatov joined #gluster
07:08 kumar joined #gluster
07:15 LebedevRI joined #gluster
07:22 _polto_ joined #gluster
07:22 anrao joined #gluster
07:28 jtux joined #gluster
07:36 [Enrico] joined #gluster
07:49 ntt joined #gluster
07:50 ntt Hi. I have a replice 2 glusterfs installation. Is there a way to see clients connected through nfs? (gluster volume status <vol-name> clients shows only clients connected with the native gluster client)
07:52 ntt my glusterfs version is 3.4.1
08:01 kovshenin joined #gluster
08:03 kdhananjay joined #gluster
08:16 ndevos ntt: you should be able to use the 'showmount' command against all your NFS-servers for that
08:17 ntt ndevos: thank you. It works!
08:18 ndevos ntt: just remember that tracking NFS-clients is always a little difficult, the result does not need to be 100% correct
08:19 ndevos ntt: an NFS-client can reboot without doing an unmount, and a storage server can reboot and likely looses the knowledge of which clients were mounted
08:22 harish joined #gluster
08:22 jkroon i've got a scenario where both knfsd and gluster-nfs is trying to run on the same system - I'm guessing this is a bad idea?  I don't need the gluster nfs instance, but trying to disable it with gluster volume set VOLNAME nfs.disable on fails with volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again - ideas?
08:23 jkroon and no, we can't find any client mounting this with nfs ...
08:29 _polto_ joined #gluster
08:30 ntt ndevos: in my case showmount is enough.... thank you
08:37 fsimonce joined #gluster
08:44 kdhananjay joined #gluster
08:54 soumya_ joined #gluster
09:03 [Enrico] joined #gluster
09:03 [Enrico] joined #gluster
09:06 Norky joined #gluster
09:14 _polto_ joined #gluster
09:14 shubhendu joined #gluster
09:16 ricky-ticky joined #gluster
09:22 ninkotech joined #gluster
09:22 ninkotech_ joined #gluster
09:22 karnan joined #gluster
09:24 shaunm joined #gluster
09:25 soumya__ joined #gluster
09:28 ntt I have a corrupted test-volume  (replica 2). In /export/brick1 i have only the .glusterfs folder and du -sh shows me 1.5GB size. Is there a way to recover this volume? I've tried with "gluster volume heal test-volume" but this not works (a lot of no su file or folder error in brick1.log)
09:37 deepakcs joined #gluster
09:40 meghanam joined #gluster
09:46 rjoseph|afk joined #gluster
09:47 Slashman joined #gluster
10:00 nshaikh joined #gluster
10:01 atalur joined #gluster
10:03 SOLDIERz joined #gluster
10:13 glusterbot News from newglusterbugs: [Bug 1193388] Disperse volume: Failed to update version and size (error 2) seen during delete operations <https://bugzilla.redhat.com/show_bug.cgi?id=1193388>
10:20 atalur joined #gluster
10:22 atalur_ joined #gluster
10:24 shubhendu joined #gluster
10:41 ricky-ticky1 joined #gluster
10:48 social joined #gluster
10:50 anoopcs joined #gluster
10:54 Manikandan joined #gluster
10:56 T0aD joined #gluster
10:58 ndarshan joined #gluster
11:06 meghanam joined #gluster
11:21 kkeithley1 joined #gluster
11:23 social joined #gluster
11:24 social joined #gluster
11:31 ndevos REMINDER: in 30 minutes, we will have a Gluster Community Bug Triage meeting in #gluster-meeting
11:32 ricky-ticky joined #gluster
11:38 ganymed joined #gluster
11:38 soumya__ joined #gluster
11:38 rjoseph|afk joined #gluster
11:43 ganymed Hi, I'm just a private individual, but as my hobby I like to check out enterprise solutions. So I have a little datacenter. Now my question: Do I undestand right that I can use GlusterFS instead of a storage area network?
11:45 kkeithley1 Yes, you can use glusterfs to create your own scale out SAN.
11:46 calum_ joined #gluster
11:47 ganymed keeithley1 , cool! So I will try it out, thank you!
11:49 ganymed by the way, gluster.org seams to contain some dead links. Is it helpful when I report them somewhere?
11:55 ntt I have a question about peers. In a replica 2 configuration, "gluster peer status" on server 1 shows Hostname: server2, but on server2 the same command shows me the ip (10.100.0.1). Why? I'm doing something wrong? (/etc/hosts file is ok on server1 and server2)
11:55 ntt gluster version 3.6.2
12:01 ndevos REMINDER: Gluster Community Bug Triage meeting starts *now* in #gluster-meeting
12:02 liquidat joined #gluster
12:02 meghanam joined #gluster
12:05 Debloper joined #gluster
12:08 social joined #gluster
12:13 glusterbot News from newglusterbugs: [Bug 1191423] upgrade to gluster 3.6 <https://bugzilla.redhat.com/show_bug.cgi?id=1191423>
12:13 keithley_ ganymed: you can report broken links in #gluster-dev
12:14 keithley_ ntt: did you probe by IP or by hostname?
12:14 ntt keithley: i do not remember.... is there a way to change and use the hostname ?
12:15 ntt keithley_: i do not remember.... is there a way to change and use the hostname ?
12:15 ntt what happens if i detach the peer and the probe again?
12:16 keithley_ yes, you can detach and probe again.
12:17 ndevos hostnames | ~ntt
12:17 ndevos hmm, ,,(hostnames)
12:17 glusterbot Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
12:17 ntt ok. But i have another similar question. I have a replica 2 and i want to use a dedicated network (fibre) for replication between servers. So i have server1 -> 10.0.0.1 and server2 -> 10.0.0.2, but i want mount my volume from another network (192.168.254.0/24). Obviously, gluster server have another nic connected to 192.168.254.0/24
12:18 itisravi_ joined #gluster
12:19 ntt on the client i need to resolve "server1" and "server2" ?
12:19 keithley_ ntt: look for JoeJulian's blog posts about using split-horizon DNS I think he calls it.
12:19 ntt ok...
12:21 meghanam joined #gluster
12:22 ntt keithley_ : have you a link for this post?
12:23 keithley_ ,,(dns)
12:23 glusterbot I do not know about 'dns', but I do know about these similar topics: 'rrdns'
12:23 keithley_ ntt: not at hand. google!
12:24 ntt i know... but i do not found it....
12:25 ntt i have to go.... thank you ( keythley_ , ndevos)
12:28 owlbot joined #gluster
12:30 calisto joined #gluster
12:32 meghanam joined #gluster
12:36 ira joined #gluster
12:40 nbalacha joined #gluster
12:43 glusterbot News from newglusterbugs: [Bug 1191497] Stripe translator wrongly calculating vector size and count <https://bugzilla.redhat.com/show_bug.cgi?id=1191497>
12:43 glusterbot News from newglusterbugs: [Bug 1193474] Package libgfapi-python for its consumers <https://bugzilla.redhat.com/show_bug.cgi?id=1193474>
12:44 shaunm joined #gluster
13:01 anoopcs joined #gluster
13:04 spandit joined #gluster
13:05 genial joined #gluster
13:10 anrao joined #gluster
13:13 glusterbot News from newglusterbugs: [Bug 1193482] Implement setuid(), setgid() wrapper apis in libgfapi-python <https://bugzilla.redhat.com/show_bug.cgi?id=1193482>
13:16 rjoseph|afk joined #gluster
13:22 hagarth joined #gluster
13:24 anoopcs joined #gluster
13:25 spandit joined #gluster
13:40 bennyturns joined #gluster
13:42 rjoseph|afk joined #gluster
13:53 prasanth_ joined #gluster
13:54 theron joined #gluster
13:56 _polto_ joined #gluster
13:56 wkf joined #gluster
14:02 tdasilva joined #gluster
14:09 virusuy joined #gluster
14:10 shaunm__ joined #gluster
14:17 atalur_ joined #gluster
14:23 ricky-ti1 joined #gluster
14:24 huleboer joined #gluster
14:25 ildefonso joined #gluster
14:26 theron joined #gluster
14:27 morse joined #gluster
14:29 georgeh-LT2 joined #gluster
14:29 georgeh-LT2_ joined #gluster
14:31 georgeh-LT2 joined #gluster
14:33 kovshenin joined #gluster
14:35 soumya__ joined #gluster
14:41 morse joined #gluster
14:41 bala joined #gluster
14:42 dgandhi joined #gluster
14:47 drma joined #gluster
14:54 bene_in_BLR joined #gluster
14:54 etcp joined #gluster
14:59 harish joined #gluster
15:00 B21956 joined #gluster
15:00 calisto joined #gluster
15:01 drma joined #gluster
15:02 wushudoin joined #gluster
15:04 etcp left #gluster
15:09 neofob joined #gluster
15:12 elico joined #gluster
15:14 glusterbot News from newglusterbugs: [Bug 1187456] Performance enhancement for RDMA <https://bugzilla.redhat.com/show_bug.cgi?id=1187456>
15:17 soumya__ joined #gluster
15:26 shubhendu joined #gluster
15:27 jobewan joined #gluster
15:27 plarsen joined #gluster
15:33 rjoseph|afk joined #gluster
15:35 RicardoSSP joined #gluster
15:42 lmickh joined #gluster
15:45 SOLDIERz joined #gluster
15:47 SOLDIERz joined #gluster
16:11 _polto_ joined #gluster
16:13 jackdpeterson Hey gluster folks -- is anyone available too chat regarding self-heal issues and gluster AND nfs client jamming up due to what appear to be self-heal related issued? The original issue had to do with a split-brain. We resolved all of the 'split-brain' GFIDs and things seemed normal for about 24 hours. We now have one directory that, when modified, causes NFS clients to jam up with input/output errors, and glusterFS mount-type will just hang (clie
16:15 jackdpeterson Then, on the gluster server side of things, we continue to have 637 heal-failed etnries. That's even after a full heal. index heals continue to report that same number
16:16 jackdpeterson Server ... GlusterFS V: 3.6.2 - Centos 6.x - Hosted on AWS (VPC). Clients: Ubuntu 14.04, al updated (verified as 3.6.2).
16:16 ndevos jackdpeterson: I suspect that the directory has a different gfid on the bricks - that would 'normally' cause such behavior
16:17 ndevos jackdpeterson: could it be that several clients try to create the same directory(-structure)?
16:17 jackdpeterson the directory was deleted as were the GFIDs
16:18 ndevos ah, well, I think the heal command shows a log, healed files might still be displayed - at least that was the case with older versions
16:19 jackdpeterson so on both bricks -- at least if you were to browse through the mount points you wouldn't see it. That being said, I'm guessing that there are like some dependent gfids? or something along those lines for files/folders nested below that can't be resolved. But I'm not sure how to clean that up
16:20 ndevos well, "dependent gfids" is a little how NFS works, the NFS-client does an OPENDIR an it gets a file-handle, that filehandle is actually a structure that encodes the GFID for the directory
16:20 jackdpeterson According to the Redhat Docs I'm guessing it's what they are calling 'Directory Entry split-brain'. but I can't be certain. The steps to clean up that either aren't helpful because the directory is gone so there isn't anything else to getfattr on
16:21 ndevos when the file-handle is used to list files in the directory, the bricks are instructed to 'open' the GFID-symlink (under the .glusterfs directory) and will pass the contents back to the nfs-server
16:22 jackdpeterson okay, we have a bunch of file or directory not found in NFS
16:22 jackdpeterson with respect to GFIDs
16:22 ndevos yes, "directory entry split-brain" would be a name for it
16:22 ndevos you would need to check the gfid on the brick (not through the nfs/glusterfs mountpoint)
16:23 ndevos ,,(getfattr)
16:23 glusterbot ndevos: Error: No factoid matches that key.
16:23 jackdpeterson right, but if the directory is deleted how can I get the GFID in question?
16:23 ndevos something like this would show the gfid of the dir: getfattr -ehex -m. -d /path/to/brick/containing/the/dir
16:23 shaunm__ joined #gluster
16:24 ndevos oh, well, if the directory is deleted, you cant really figure out the GFID
16:24 ndevos was the directory deleted from the bricks, or through a Gluster managed mountpoint?
16:25 jackdpeterson right, we deleted the associated gfid at that point in time to resolve split brain. but that didn't solve the issue in this case. It solved the issue for 99% of the other split brains. in this case it left us with this tree of nastiness. The GFIDs and the directory were removed directly on the gluster servers as clients would hang when trying to do anything w/ that directory
16:25 ndevos if it was deleted on the bricks, and not through glusterfs/fuse, or nfs, you may need to restart the brick process - it could have some caches that now dont make any sense
16:26 jackdpeterson is there a way to ensure that it'll come up safely?
16:26 jackdpeterson (it's a production system)
16:26 ndevos could you explain what you mean with "deleted the associated gfid"?
16:27 jackdpeterson cd he/xv/ rm -f hexvalue-gdhfgskfghdgksfhkgkjsfdkgdskdfh
16:28 ndevos right, so depending on what you deleted, it was either a hard-link for a file, or a sym-link for a directory
16:28 ndevos if you deleted the sym-link for a directory, the xattr with the gfid is still set in the original directory
16:31 jackdpeterson how would i find that xattr? it's not in /mnt/bricks/storage5j/etc/etc/etc/folder/in/question
16:33 ndevos the xattr should be readable with the getfattr command above
16:35 ndevos and you would need to compare the trusted.gfid xattr from all the bricks, and figure out which one has the wrong gfid
16:36 jackdpeterson mm... if I were to perform a ls -l assuming my pwd is "/mnt/bricks/storage5j/etc/etc/etc/folder/in"
16:36 jackdpeterson the directory 'question' is missing
16:36 jackdpeterson therefore .. performing getfattr will say, 'no such file or directory'
16:37 ndevos maybe the parent directory of the missing one has the issue?
16:37 jackdpeterson I'm open to the possibility.
16:37 jackdpeterson So, going full circle -- how would I go about cleaning that sucker out safely/consistently
16:38 jackdpeterson that directory (the parent one) has a trusted.gfid of 0x67fb .....etc
16:39 ndevos I think you should be able to follow the last part of https://github.com/gluster/glusterfs/blob/release-3.6/doc/split-brain.md
16:39 jackdpeterson so if I rm -f that gfid on one of the replica 2 servers... self-heal should do (insert magic here)
16:39 ndevos as in https://github.com/gluster/glusterfs/blob/release-3.6/doc/split-brain.md#fixing-directory-entry-split-brain
16:40 jackdpeterson last step -- how does one determine if there are hard links present?
16:41 ndevos you can check with 'ls -l', the 2nd column in the output is the number of links
16:41 jackdpeterson in the gfid folder
16:41 jackdpeterson ?
16:42 ndevos yes, that works, or in the path on the brick
16:42 jackdpeterson I do see there's one link poiinting to another ... gfid path?
16:43 ndevos uhh, pointing to a gfid-path? I dont know about that, sounds bogus to me
16:44 jackdpeterson well, what i'm saying is that it's a directory that's relatively placed in the gfid region (.glusterfs/c6/7d/long-hex-value/folder-in-question
16:44 glusterbot News from resolvedglusterbugs: [Bug 1190138] Debian-based distributions do not have /usr/libexec, hard-coded paths to /usr/libexec are broken <https://bugzilla.redhat.com/show_bug.cgi?id=1190138>
16:45 edualbus joined #gluster
16:46 ndevos oh, well, yes, (long-hex-value=gfid) .glusterfs/c6/7d/long-hex-value is a symlink
16:47 ndevos if it is a symlink indeed, than all is fine, if it is not a symlink, its wrong, or I'm not up to date
16:48 jackdpeterson mm... now that's done I'm getting a cannot create directory -- input/output error
16:48 jackdpeterson client-side
16:49 jackdpeterson after doing the ls -al on that directory to trigger self-heal
16:51 ndevos did you trigger a self-heal on the parent directory too?
16:51 jackdpeterson yeah
16:52 jackdpeterson so this appears to be extremely destructive -- NFS clients just hosed themselves across the board. outage
16:52 jackdpeterson same issue we encountered last night -- anytime a self-heal happens... NFS clients get knocked out
16:53 ndevos hmm, I'm not sure how that can happen - unless the nfs-clients were holding on to a file-handle with the gfid that got deleted
16:54 ndevos if the nfs-server can not resolve the gfid anymore, it would return ESTALE to the clients
16:54 gem joined #gluster
16:55 jackdpeterson ... which would cause the whole mount to go away?
16:55 jackdpeterson because in this case the mounts go away and clients do not self-heal / restore
16:56 jackdpeterson they then complain nfs.mount system call failed
16:56 jackdpeterson nfs: server (IP) not responding, timed out
16:56 jackdpeterson rebooting clients brings them back to operational and mountable
16:56 jackdpeterson -- it's horrible --
16:57 ndevos bleah, yeah, that is *very* nasty
16:57 ndevos the /var/log/glusterfs/nfs.log might have some useful notes?
16:58 jackdpeterson remote operation failed: transport endpoint is not connected
16:58 jackdpeterson permission denied ... gfid
16:58 jackdpeterson and now the server in question ... brick isn't online
16:58 jackdpeterson wtf
16:58 jackdpeterson weird
16:59 jackdpeterson how do i successfully restart the brick service?
16:59 ndevos you can restart the glusterd service, or do a 'gluster volume start $VOLUME force' to start the missing processes
17:00 jackdpeterson still offline (gluster volume pod1 start force
17:00 anoopcs joined #gluster
17:00 ndevos when you removed the gfid links and all, was the brick process for that filesystem still running?
17:01 ndevos it's the other way around: gluster volume start pod1 force
17:01 jackdpeterson looks like it took a minute
17:01 jackdpeterson need to do this for each volume
17:04 theron joined #gluster
17:06 jackdpeterson good news -- that one folder in question is now editable. horribly bad news is that we experienced a major outage w/o a really good explanation -- because those things (self-heal) should be within normal operating parameters of gluster
17:07 ndevos okay, at least some success...
17:07 ndevos I think the problem on the side of the NFS-client is, that the file-handle (GFID based) became invalid
17:08 calisto joined #gluster
17:08 jackdpeterson we still have the exact same count of heal failed entried though
17:08 ndevos self-heal is considered a 'normal' operation, but manual resolving of split-brains is not
17:09 ndevos I do not know how the "heal failed" is tracked, but I think it is a log, and only will clear the old entries after some time
17:12 jackdpeterson Any info that I can grab concerning NFS hosing itself when these heals take place?
17:14 ndevos I'd start with: exact time/date, the nfs.log, and possibly output from the nfs-client with some rpcdebug enabled and a tcpdump taken on the nfs-server with "-i any -s 0" to include all the bits
17:14 ndevos but well, logging all that before the issue starts, and have the errors captured is the most important
17:15 ndevos if you have an idea how to setup an environment for reproducing, that would be welcome too, and you can file a bug with those details
17:15 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
17:16 jackdpeterson it looks like there were a number of locks on the directory in question. How can I identify who client 0xa9bc10 would be? this is in /var/log/bricks/brick-in-question.log that had the issues
17:17 PeterA joined #gluster
17:18 ndevos the brick does not know about the nfs-clients, the nfs-server is the client the bricks would see
17:20 jackdpeterson hmm okay
17:23 ndevos the client 0xa9bc10 that you see, could be the lk_owner that the nfs-server associated with a particular nfs-client, but I do not think tracking that back is doable
17:29 jobewan joined #gluster
17:29 jiffin joined #gluster
17:36 genial left #gluster
17:37 maveric_amitc_ joined #gluster
17:41 jackdpeterson Hmm, I think wer're hitting some kind of performance bottleneck of *some kind*. we have tons of small files that are periodically cleared out. when that happens it looks like some nfs clients are getting knocked off as welll
17:41 jackdpeterson the two gluster servers are both pushing around 150-400 Mbit/s) according to nload
17:42 jackdpeterson -- does anyone have any experience with large quantities of file deletions and NFS clients getting hosed?
17:43 * ndevos has not heard about that before
17:44 jackdpeterson nothing server side (log wise) as far as I can tell is indicating any failures
17:44 jackdpeterson volume status indicates everything is up
17:44 CyrilPeponnet hey any skilled guy in geo-rep around ?
17:44 glusterbot News from newglusterbugs: [Bug 1135548] Error in quick start: start volume and specify mount point <https://bugzilla.redhat.com/show_bug.cgi?id=1135548>
17:45 ricky-ticky1 joined #gluster
17:55 _Bryan_ joined #gluster
17:57 jackdpeterson @Ndevos - getting more NFS related errors - 10006(error occurred on the server or IO error) POSIX: 14(bad address), as well a flurry of other errors. like gfid and type mismatch
17:59 ndevos jackdpeterson: that all sounds as if a a file-handle became unexpectedly invalid - were the gluster processes restarted (including the nfs-server) that could have any of those gfids cached in memory?
18:17 lalatenduM joined #gluster
18:20 davetoo joined #gluster
18:20 Rapture joined #gluster
18:21 bene2 joined #gluster
18:34 awerner joined #gluster
18:35 MacWinner joined #gluster
19:01 free_amitc_ joined #gluster
19:04 m0zes joined #gluster
19:05 dgandhi joined #gluster
19:15 glusterbot News from newglusterbugs: [Bug 1193636] [DHT:REBALANCE]: xattrs set on the file during rebalance migration will be lost after migration is over <https://bugzilla.redhat.com/show_bug.cgi?id=1193636>
19:28 dgandhi joined #gluster
19:46 johnbot Upgraded gluster from 3.5.0 to 3.6.2 few days ago and received an error while attempting to remove a brick. "One or more nodes do not support the required op-version. Cluster op-version must atleast be 30600." I assume this is happening since an older version of gluster created the volume?
19:46 glusterbot johnbot: set the desired op-version using ''gluster volume set all cluster.op-version $desired_op_version''.
19:47 johnbot Do I just need to increment the op-version to something greater than what it is or something specific for 3.6.2
19:55 lmickh joined #gluster
19:55 daMaestro joined #gluster
20:02 daMaestro joined #gluster
20:06 keithley_ ,,(op-version)
20:06 glusterbot The operating version represents the RPC and translator capabilities required to accommodate the volume settings ( http://gluster.org/community/documentation/index.php/OperatingVersions ). To allow older version clients to connect to newer servers, reset any volume options that require the newer op-version.
20:12 _polto_ joined #gluster
20:22 CyrilPeponnet 0-glusterfs-fuse: 67819: /.gfid/83f9cd6c-a2d9-4702-a7c5-eb3739b2f943 => -1 (Operation not permitted) any hint ?
20:26 kovshenin joined #gluster
20:36 tessier joined #gluster
20:54 MugginsM joined #gluster
21:03 theron joined #gluster
21:07 _polto_ joined #gluster
21:22 badone_ joined #gluster
21:39 badone_ joined #gluster
21:47 _polto_ can I stripe between 3 almost same size disks ? On one of the hdd I have a system particion, can I use the 97% partition in a stripe with 2 full size disks ? as I understand it will work but providing 97% of the global space, not more. ? Could someone confirm ?
21:48 _polto_ joined #gluster
21:48 social joined #gluster
21:49 _br_ joined #gluster
21:57 rotbeard joined #gluster
22:01 _br_ joined #gluster
22:16 SOLDIERz joined #gluster
22:17 badone_ joined #gluster
22:18 bene2 joined #gluster
22:27 JoeJulian _polto_: Should be correct.
22:35 _br_ joined #gluster
22:38 _polto_ JoeJulian: I tried a second time, but I get errors.. I am not sure it's related...
22:39 _polto_ [2015-02-17 22:34:47.059462] I [MSGID: 109036] [dht-common.c:6222:dht_log_new_layout_for_dir_selfheal] 0-data-dht: Setting layout of /footage/yolll with [Subvol_name: data-stripe-0, Err: -1 , Start: 1407397652 , Stop: 2814001543 ], [Subvol_name: data-stripe-1, Err: -1 , Start: 2814001544 , Stop: 4294967295 ], [Subvol_name: data-stripe-2, Err: -1 , Start: 0 , Stop: 1407397651 ],
22:48 badone_ joined #gluster
22:49 JoeJulian There's no error there.
22:50 _polto_ JoeJulian: a message full of "Err" is not an error ?
22:50 JoeJulian According to the priority, it's an " I "nfo
22:52 _polto_ JoeJulian: I just recreated the volume without the smaller disk, so I now have replica 2 stripe 5 instead if stripe 6 as before, I still have the same problem: directory creation is extreemly slow
22:53 JoeJulian Don't create directories.
22:53 _polto_ time mkdir /data/footage/3 --> real0m4.859s
22:53 JoeJulian Problem solved.
22:53 _polto_ JoeJulian: sure, thanks !
22:54 JoeJulian tbqh, there are only a very small handfull of people that have any experience with stripe and I'm not one of them. I've not heard anyone complain about directory creation being slow before.
22:55 _polto_ I do need to stripe
23:01 davetoo any ideas how I can troubleshoot this:  System contains 1 snapshot(s).
23:01 davetoo Do you still want to continue and delete them?  (y/n) y
23:01 davetoo snapshot delete: failed: Pre Validation failed on xxxx-01-08. Please check log file for details.
23:01 davetoo I'm stepping into this after somebody else set it up (and am new to glusterfs)
23:01 davetoo its
23:01 davetoo a 14-node x 2-brick/node volume,
23:02 davetoo four of the nodes are missing that snapshot directory
23:02 davetoo am getting tons of "Brick foo:/var/run/gluster/snaps/bar/bricknn/data has disconnected from glusterd"
23:04 _br_ joined #gluster
23:06 jjahns i've messed around with stripe
23:07 jjahns i actually have a 30-node cluster with 4x brick/node
23:07 khanku joined #gluster
23:08 jjahns we had smaller drives and i did striping on these and write performance increased
23:08 jjahns doesnt do anything for read though
23:09 _polto_ jjahns: thanks for sharing ! I manly need write speed. Strange that read speed is not better in stripe ..
23:10 jjahns if you dont do replica,
23:10 _polto_ I do replica 2 stripe 6 distribute 3
23:11 jjahns i see better performance with replica 3 on read
23:12 _polto_ jjahns: yeah, but at cost of more space used by replicas...
23:13 jjahns of course, but more tolerable of failures
23:13 _polto_ yep
23:13 _polto_ I can not afford it yet.. :/
23:14 jjahns i am more interested in SSD caching
23:14 jjahns :)
23:15 _polto_ do anyone have experience with data-classification ?
23:15 _polto_ http://www.gluster.org/community/documentation/index.php/Features/data-classification
23:18 SOLDIERz joined #gluster
23:26 RicardoSSP joined #gluster
23:26 RicardoSSP joined #gluster
23:29 _br_ joined #gluster
23:32 gildub joined #gluster
23:33 jobewan joined #gluster
23:34 plarsen joined #gluster
23:37 _br_ joined #gluster
23:53 n-st joined #gluster
23:57 _polto_ is it possible to switch from replica 2 to replica 3 live by adding nodes with needed number of bricks ?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary