Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-07-24

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 cyberbootje joined #gluster
00:17 Peter4 anyone experienced wrong quota usage reported?
00:20 Peter4 I copied 600G of files to a NFS shared gluster export
00:20 Peter4 later it showed 1.4 TB used
00:21 Peter4 after I rm -rf * on that export, quota still shows 769G used
00:31 JoeJulian @which brick
00:31 glusterbot JoeJulian: To determine on which brick(s) a file resides, run getfattr -n trusted.glusterfs.pathinfo $file through the client mount.
00:31 JoeJulian (that's for me)
00:31 Peter4 ic
00:32 Peter4 joejulian: have u seen the quota issue before?
00:34 Peter4 anyone here using quota with NFS?
00:40 Peter4 JoeJuilian: Did u hear anything from Varun regarding to the du vs df issue?
00:45 NuxRo Peter4: I use quota with NFS, haven't noticed this particular problem yet
00:45 NuxRo I'm on 3.4.0
00:45 Peter4 i am on 3.5.1 and i had that problem on 3.4.3 too
00:46 Peter4 i have that problem on some of my exports
00:46 Peter4 i have features.quota-deem-statfs: on
00:47 Peter4 and share directories as NFS export from gluster
00:49 bennyturns joined #gluster
00:51 Peter4 how can i tell what file still open on gluster volume?
00:52 Peter4 maybe gluster think those fd in the export still open??
00:52 Peter4 how does quotad calculate the usage?
00:56 JoeJulian Peter4: Do you have a set of instructions I can follow to reproduce the issue you're seeing?
00:57 Peter4 it happens kinda random
00:57 Peter4 i created around 60 exports
00:57 RioS2 joined #gluster
00:57 Peter4 only 4 seeing these
00:59 Peter4 like last time i mentioned to you
00:59 Peter4 seeing the df and du different
01:00 Peter4 and i wonder if quota think the file descriptor still open?
01:08 JoeJulian I thought you could repro it every time. I thought I recalled you saying you mount the parent directory for these quota'd subdirectories and mv a file from one subdirectory to another.
01:20 Peter4 if that directory already have quota messed up then yes
01:21 Peter4 same mv file will make it worser and worser....
01:21 Peter4 the question is how did it start get messed up....
01:22 Peter4 and that's how i noticed a daily job write files to the same dir in the NFS keep getting increase in doubling the usage
01:45 harish joined #gluster
01:48 theron joined #gluster
01:53 Peter4 NuxRo: do you have a replica on your volume?
02:32 recidive joined #gluster
03:02 bharata-rao joined #gluster
03:16 glusterbot New news from newglusterbugs: [Bug 1122762] /usr/sbin/glusterfs don't have option to use backup-volfile-servers <https://bugzilla.redhat.co​m/show_bug.cgi?id=1122762>
03:38 atinmu joined #gluster
03:43 kanagaraj joined #gluster
03:44 shubhendu joined #gluster
03:54 itisravi joined #gluster
04:06 gildub joined #gluster
04:20 ramteid joined #gluster
04:32 anoopcs joined #gluster
04:39 glusterbot New news from resolvedglusterbugs: [Bug 764655] NetBSD port <https://bugzilla.redhat.com/show_bug.cgi?id=764655>
04:40 jiffin joined #gluster
04:40 spandit joined #gluster
04:42 Rafi_kc joined #gluster
04:59 saurabh joined #gluster
05:04 meghanam__ joined #gluster
05:04 meghanam joined #gluster
05:11 prasanth_ joined #gluster
05:11 kumar joined #gluster
05:13 haomaiwang joined #gluster
05:16 nishanth joined #gluster
05:18 ppai joined #gluster
05:28 caiozanolla joined #gluster
05:30 gildub joined #gluster
05:32 lalatenduM joined #gluster
05:40 _NiC joined #gluster
05:44 kdhananjay joined #gluster
05:45 aravindavk joined #gluster
05:53 sputnik13 joined #gluster
05:58 caiozanolla joined #gluster
06:06 ricky-ti1 joined #gluster
06:08 raghu joined #gluster
06:09 tom[] joined #gluster
06:12 RameshN joined #gluster
06:12 vpshastry joined #gluster
06:17 dusmant joined #gluster
06:26 psharma joined #gluster
06:37 Peter1 joined #gluster
06:37 Peter1 my gluster still crashing on bricks
06:37 Peter1 http://pastie.org/9416690
06:37 glusterbot Title: #9416690 - Pastie (at pastie.org)
06:38 Peter1 0-rpcsvc: rpc actor failed to complete successfully
06:38 Peter1 how could a brick die like this?
06:40 Philambdo joined #gluster
06:42 Peter1 etc-gluster log
06:43 Peter1 [2014-07-24 05:53:28.911318] E [glusterd-utils.c:4124:gluster​d_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/6b5e8ccc26ea6cdfe54005da41e57062.socket err:
06:43 Peter1 whta does that means?
06:43 Peter1 i feel something wrong with my gluster but just dun know what
06:43 Peter1 more volume getting issue with the space not right by quota
06:43 Peter1 is that something relate to the heal and memory leak that keep crashing system?
06:44 keytab joined #gluster
06:45 Peter1 how can i tell if healing still running?
06:45 Peter1 i m on ubuntu 3.5.1 and gluster volume heal $vol info does not exit
06:45 Peter1 exist
06:47 glusterbot New news from newglusterbugs: [Bug 1122794] Uninterruptible processes when accessing a file <https://bugzilla.redhat.co​m/show_bug.cgi?id=1122794>
06:50 JoeJulian file a bug
06:50 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
06:53 ctria joined #gluster
07:03 ekuric joined #gluster
07:03 dusmant joined #gluster
07:06 andreask joined #gluster
07:09 kanagaraj joined #gluster
07:13 R0ok_ joined #gluster
07:17 dusmant joined #gluster
07:17 glusterbot New news from newglusterbugs: [Bug 1122807] [enhancement]: Log a checksum of the new client volfile after a graph change. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1122807>
07:20 aravindavk joined #gluster
07:26 LebedevRI joined #gluster
07:30 Peter1 i keep getting setxattr and getxattr error from gluster
07:30 Peter1 how do i enable xattr on xfs on ubuntu?
07:31 JoeJulian That's probably for files that don't exist. That's usually what causes that.
07:32 JoeJulian xattrs are enabled by default. user_attr is the only one that's optional.
07:58 FooBar Always nice... people asking questions... and then logging off the next minute ;)
08:07 aravindavk joined #gluster
08:08 deepakcs joined #gluster
08:13 dusmant joined #gluster
08:18 glusterbot New news from newglusterbugs: [Bug 1122834] Issues reported by Coverity static analysis tool <https://bugzilla.redhat.co​m/show_bug.cgi?id=1122834>
08:23 JoeJulian FooBar: Heh, and I always forget to check. Oh well, maybe someone else will find it useful information.
08:24 siel joined #gluster
08:24 siel joined #gluster
08:31 liquidat joined #gluster
08:34 abyss__ JoeJulian: ofcourse :) Once someone asked about something and left you anwered and then I realized that is I needed;)
08:34 FooBar JoeJulian: :)
08:35 abyss__ *then you answered ;)
08:37 JoeJulian hehe
08:38 Philambdo joined #gluster
08:43 sage joined #gluster
08:48 vpshastry joined #gluster
08:50 shubhendu joined #gluster
08:54 dusmant joined #gluster
08:54 RameshN joined #gluster
08:54 nishanth joined #gluster
09:06 demonicsage joined #gluster
09:06 demonicsage hey guys, anyone around? :)
09:07 demonicsage just wondering about one thing
09:11 nightlydev joined #gluster
09:11 nightlydev Hi everyone
09:11 demonicsage gluster storage on SSD (at DigitalOcean), what are the implications of putting all WordPress files in gluster?
09:11 demonicsage I mean WordPress is "read" heavy, not really write heavy to the IO.
09:13 demonicsage i have a recommendation to put only wp-contents/themes and wp-uploads to shared gluster storage (since they contain user uploaded media files), and keep the rest on local file system
09:14 kshlm joined #gluster
09:14 nightlydev I have a 8 nodes glusterfs cluster (4 replica / 4 distributed ). I'm using glusterfs to store OpenVZ containers. It's running good but write IO performance for small files is very bad. If I use asynchronimous replication (GeoReplication), can I espect to have better performance ?
09:18 swebb joined #gluster
09:21 R0ok_ joined #gluster
09:24 shubhendu joined #gluster
09:26 nishanth joined #gluster
09:27 dusmant joined #gluster
09:29 JoeJulian ~php | demonicsage
09:29 glusterbot demonicsage: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
09:29 glusterbot --fopen-keep-cache
09:31 demonicsage thanks, mate!
09:31 Rafi_kc joined #gluster
09:31 demonicsage i think i will just put theme files and media uploads in gluster storage and the rest on local file system
09:31 demonicsage WordPress uses tons of PHP files, small ones :D
09:32 vpshastry joined #gluster
09:33 gehaxelt Is the network throughput limited somehow? Have replica 2 servers in France/Germany, but I only get about 10mbit/s throughput?
09:34 cultavix joined #gluster
09:38 demonicsage @gehaxelt can you test your two servers using iperf?
09:38 gehaxelt demonicsage, will do. Give me a sec
09:38 demonicsage sure do
09:40 gehaxelt hmm, it's only about 10mbit :(
09:40 gehaxelt Strange. Kimsufi/OVH has a 100mbit connection. Strato should have one too...
09:40 gehaxelt Then it's not gluster. Sorry :)
09:41 demonicsage which host are you with?
09:42 gehaxelt see above. Kimsufi/OVH @France (RBX) and Strato @Germany (not sure with DC).
09:42 gehaxelt Okay
09:42 gehaxelt Germany <> NL gives me 80mbit/s
09:42 demonicsage Hrmph
09:43 demonicsage Could be an issue with Kimsufi/OVH
09:43 gehaxelt France <> NL gives me 80mbit/s too
09:43 demonicsage Doing geo-replication?
09:43 gehaxelt So it seems like some routing/network problem between ovh <> strato?
09:44 gehaxelt Nope, no geo replication.
09:44 demonicsage Yeah, looks that way
09:44 demonicsage Might want to talk to the guys from OVH
09:45 hchiramm joined #gluster
09:45 gehaxelt Yeah, it's not that urgent/a big problem.
09:45 gehaxelt Using that for some daily backups, so it doesn't matter if it takes 10 minutes or 1 hour to sync.
09:45 gehaxelt I was just wondering if I did something wrong.
09:46 demonicsage ah no, don't think gluster has an issue with network throughput
09:47 gehaxelt okay, so one last question: In a replica X setup, all data is written to all nodes at the same time, right?
09:47 gehaxelt So that should limit the throughput of all nodes to the slowest connection?
09:51 RameshN joined #gluster
09:58 T0aD joined #gluster
10:05 harish_ joined #gluster
10:09 Philambdo joined #gluster
10:09 Alexw joined #gluster
10:10 hchiramm joined #gluster
10:10 Alexw Hello, I am trying to get a mirrored file system with two servers. But when I created the new volume the status shows the following: Number of Bricks: 1 x 2 = 2
10:10 Alexw Should that not be Number of Bricks: 2
10:10 Alexw Because if I place something on server one in the folder nothing happens on the other server
10:10 Alexw It did create the folder at the second server
10:11 Alexw Any idea what to do next?
10:12 social joined #gluster
10:13 gehaxelt Alex, did you set the "replica "-count to 2 ?
10:14 Alexw yes
10:14 gehaxelt Are the 491xx-ports open in the firewall on both servers?
10:14 Alexw Both servers have full access to each other
10:14 gehaxelt hmm
10:15 Alexw "netstat -tap | grep glusterfsd" shows 10 connections and 2 listenings
10:15 Alexw (I got two volumes)
10:15 gehaxelt So you mounted the glusterfs-volume on the one server, put some files into it and nothing happened on the other server?
10:15 Alexw 2 LISTEN and 10 ESTABLISHED
10:16 Alexw I only started it
10:16 Alexw And went to the folder I specified
10:16 gehaxelt Ah, I think you have to mount it to put some files into it.
10:17 Alexw But there is already a folder there
10:17 gehaxelt mkdir /mnt/glusterfs && mount -t glusterfs Host1:/volname /mnt/glusterfs
10:17 Alexw So that folder is just a dummy?
10:17 Alexw And need to be mounted from
10:17 atinmu joined #gluster
10:18 gehaxelt Afaik, the brick-folders are only for glusterfs-server to manage the files and not for the "clients" to use.
10:18 Alexw Ah ok
10:18 Alexw That clears things up
10:18 gehaxelt :)
10:19 Alexw Is there any harm if I created a new volume at: /dev/www
10:19 Alexw and then mount it from there to /var/www
10:20 gehaxelt I don't think so.
10:20 Alexw Ok
10:20 Alexw I like organizing :)
10:20 gehaxelt ;)
10:20 Alexw Since all "mount points" are at /dev
10:23 demonicsage hey gehaxelt, sorry, i got busy
10:24 demonicsage hmm that i m not too sure. i m still new to glusterfs too, but it makes logical sense
10:24 gehaxelt demonicsage, no problem :)
10:24 gehaxelt Okay, thanks. I'm still new too.
10:37 demonicsage gotta go mate. will be back soon
10:37 demonicsage ciao!
10:37 gehaxelt cu
10:43 burn420 joined #gluster
11:00 nullck_ joined #gluster
11:15 Alexw hey
11:15 Alexw Now I'm unable to mount it :(
11:15 Alexw Mount failed. Please check the log file for more details.
11:16 TvL2386 joined #gluster
11:16 Alexw I am not sure where to find the "log file"
11:18 shireesh joined #gluster
11:33 diegows joined #gluster
11:46 ppai joined #gluster
11:56 gehaxelt Alex, take a look in /var/log/glusterfs
11:57 shubhendu joined #gluster
12:01 hagarth joined #gluster
12:02 hagarth1 joined #gluster
12:02 marbu joined #gluster
12:06 RameshN joined #gluster
12:07 kdhananjay joined #gluster
12:07 chirino joined #gluster
12:10 glusterbot New news from resolvedglusterbugs: [Bug 874554] cluster.min-free-disk not having an effect on new files <https://bugzilla.redhat.com/show_bug.cgi?id=874554>
12:13 xleo joined #gluster
12:16 dusmant joined #gluster
12:16 bala joined #gluster
12:18 giannello joined #gluster
12:30 edward1 joined #gluster
12:37 theron joined #gluster
12:39 tdasilva joined #gluster
12:50 glusterbot New news from newglusterbugs: [Bug 1122930] Create a document on how self-heal-daemon works <https://bugzilla.redhat.co​m/show_bug.cgi?id=1122930>
12:53 kkeithley ,,(bug)
12:53 glusterbot I do not know about 'bug', but I do know about these similar topics: 'fileabug'
12:53 Lee- Does glusterfs actually care if a partition is created on the block device it is utilizing for it's data storage?
12:55 kkeithley glusterfs only needs a file system (or subdir in a file system).  It doesn't care if you created partitions or used lvm to create the file system(s) on.
12:55 kkeithley ,,(fileabug)
12:55 glusterbot Please file a bug at http://goo.gl/UUuCq
12:58 recidive joined #gluster
12:59 overclk joined #gluster
13:02 rjoseph joined #gluster
13:03 mbukatov joined #gluster
13:03 bennyturns joined #gluster
13:07 sjm joined #gluster
13:10 julim joined #gluster
13:15 _Bryan_ joined #gluster
13:16 dusmant joined #gluster
13:18 harish_ joined #gluster
13:19 mojibakeumd joined #gluster
13:21 kkeithley1 joined #gluster
13:23 bala1 joined #gluster
13:25 mojibakeumd Newb mojibakeumd back online today. Have a few best practices questions. 1. What is a good naming practice for the exports created on the server nodes for bricks. I am thinking for bricks for a replica server1 /export/brick1 and server2 /export/brick1. This way "brick1" could be easily associated with being replicas of each other. However for a defaul distributed volume. server1 /export/brick1a and server2 /export/brick1b. This way, I know they ar
13:25 mojibakeumd of each other.. Anythoughts, advice? Maybe I have the concepts of bricks and volumes, confused.
13:25 bene2 joined #gluster
13:29 mojibakeumd 2. If I am looking to mount the volumes on clients for purposes of httpd content. Is it advisable to mount to /var/www, or should I mount to something like /mnt/gfs/shared-volume, and make symbolic links to the mount point? Additionally, can I, or is it advisable to mount to a deeper directory in the volume if I only want the client mount to have access to a subdirectory. Like /mnt/gfs/shared-volume/var/www/site1.
13:31 recidive joined #gluster
13:34 tdasilva joined #gluster
13:37 lalatenduM joined #gluster
13:37 Lee- Is the only way to rename a brick to use volume replace-brick? I found commands for renaming volume, but not brick. I'm using 3.5.1.
13:40 bala joined #gluster
13:41 plarsen joined #gluster
13:51 Lyfe joined #gluster
13:52 recidive joined #gluster
13:54 gehaxelt Lee-, I think so. Wanted to do the same a couple of days ago. I finally used replace brick. Alternatively you could run a remove-brick followed by a add-brick
13:59 Lee- gehaxelt, I tried replace-brick, but it seems to have broken something. doing replace-brick status, voolume status, and volume info all hang for a couple minutes and then return absolutely nothing (no errors, no messages, just goes back to the prompt). I'm just going to recreate my bricks and volumes. i'm still in testing phases of this -- not production yet
14:00 gehaxelt hmm, did you commit the action?
14:00 gehaxelt That was my fault first, which has lead to some random behaviour...
14:03 Lee- oh I used the "start" operation, not "commit". that's probably the issue. I read the "gluster help" and then read a guide and the guide didn't mention commit. I guess I'll try the rename again after I do my next tests
14:04 lmickh joined #gluster
14:13 Lee- I think the replace-brick hosed this system. I tried a volume delete and it doesn't appear to be deleting. I went back to do the commit operation on the replace-brick and it seems to do nothing as well. So I can't commit my old operation and I can't seem to delete the volume either
14:14 simulx joined #gluster
14:16 gehaxelt Lee-, I had exactly the same problems...
14:17 Lee- oh yeah this it totally messed up now. I have 2 servers, 1 volume that was replicated. On the one server if I run "gluster peer status" it says: "Connection failed. Please check if gluster daemon is operational.". This makes me think that the gluster daemon has failed on server1, however when I run "gluster peer status" on server2, I get the same data I'd expect when things are running normally (it lists server1 and says that it is in cluster and connected)
14:17 Lee- State: Peer in Cluster (Connected)
14:17 Lee- So server2 says everything is OK and server1 thinks things are not OK
14:18 Lee- I was doing tests with gluster 3.2 a few days ago (it's what my linux distro included) and ran in to some strange behavior. I was told to upgrade to 3.5, so I did and now I get this. I really want to use gluster, but it seems a bit too easy to break.
14:19 gehaxelt How I fixed it (maybe not the best solution): Stop Glusterfs on Server 2. Detach Server 2 on Server 1. (I think I had to run remove-bricks before). Then I reinstalled glusterfs on Server 2 and removed the configs from /var/lib/glusterfs. Then gave Server 2 a new hostname through /etc/hosts. Then installed glusterfs again. Peered again. Ran "add-bricks" on Server 1. Ran "volume heal" to restore/copy the data.
14:19 gehaxelt So it was a bit of a mess.
14:20 Lee- I'm doing tests on EC2 instances, so I'm just going to terminate these and create fresh VMs
14:20 gehaxelt ;)
14:20 Lee- It seems like it works fine as long as I don't try to make any changes (like replacing a brick)
14:20 Lee- I likely won't be doing that once this stuff is in production.
14:21 gehaxelt Lee-, yeah. Don't touch a running system lol
14:22 mortuar joined #gluster
14:23 Lee- From what I gather NFS is preferred for small files and the native method for everything else. What I read was a bit outdated. Is this still the general recommendation?
14:24 gehaxelt Not sure. I'm just using the glusterfs
14:24 Lee- Basically I have a large website that utilizes PHP and has a directory for user content (like images). I was planning on using gluster for only the images and since the PHP and other code (css, js) is only updated via subversion, I do not need this on gluster
14:24 xleo joined #gluster
14:25 Lee- the images, css, js end up getting served via CDN anyway, but my concern is a failure for files that aren't cached or if a user uploads a file -- gluster seems to be a good solution for this.
14:25 gehaxelt (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
14:25 gehaxelt <glusterbot> --fopen-keep-cache
14:25 glusterbot Title: Optimizing web performance with GlusterFS (at joejulian.name)
14:25 gehaxelt Lee-, maybe that will help...
14:25 Lee- I actually use apc.stat=0 and apc.cstat=0, so it doesn't reread cached files until I explicitly clear the APC cache (or restart the web server).
14:26 gehaxelt And yeah, gluster seems to be useful for your usecase.
14:26 Lee- My PHP files are not going to be served via gluster -- only user generated content like images and such
14:28 gehaxelt okay :)
14:28 anoopcs joined #gluster
14:28 wushudoin joined #gluster
14:28 LebedevRI joined #gluster
14:32 cultavix joined #gluster
14:33 Lee- actually I'm thinking maybe I should just do the PHP code via gluster, but mount it via NFS and keep the user content with glusterfs. Not sure if this is possible, but I shall test
14:35 Alexw I am back
14:35 Alexw I finally got GlusterFS working
14:35 Alexw But when one server is down the other server server have a corrupted file system
14:35 Alexw And when the first server is back up everthing is fine again
14:36 kumar joined #gluster
14:37 Alexw And the info command also states this: Number of Bricks: 1 x 2 = 2
14:37 Alexw I think that should be: Number of Bricks: 2
14:38 ndk joined #gluster
14:39 gehaxelt Alex, if you run a replica 2 volume, the number is fine. I haven't understood that too, but it seems to be correct.
14:43 theron joined #gluster
14:44 Alexw I guess I was wrong
14:44 Alexw It works just fine
14:44 Alexw Never mind
14:44 FooBar Number of Bricks: 15 x 3 = 45
14:44 Alexw thanks anyway :)
14:44 FooBar 15 volumes with replica 3
14:45 deepakcs joined #gluster
14:45 theron_ joined #gluster
14:46 bene2 joined #gluster
14:56 theron joined #gluster
14:56 bennyturns joined #gluster
15:00 daMaestro joined #gluster
15:00 theron joined #gluster
15:02 gehaxelt Is it possible to limit the traffic to one of the replica-nodes?
15:03 gehaxelt I'd like to add a server with a limited amount of bandwith. Writing should be allowed, but reading tasks should be run from the other unlimited nodes.
15:03 kanagaraj joined #gluster
15:06 Lee- gehaxelt, certainly I'm still new to gluster, but I read that it determines the client to read from based on first to respond, so doing a simple bw limit might be bad -- your bw limited server may end up being the first to respond, but then respond slowly
15:07 Lee- s/client to read from/server to read from/
15:07 glusterbot Lee-: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:09 B21956 joined #gluster
15:09 gehaxelt Lee-, yeah, that's why I'm asking...
15:10 gehaxelt I thought maybe there's some kind of switch to set a priority or something...
15:11 Lee- with regards to priorities I ran across this page recently -- http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented    -- it has a few references to setting priorities, but I got the impression they were more soft priorities, which wouldn't allow for a strict "do not read from"
15:11 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
15:11 gehaxelt okay, thanks I'll read through that.
15:12 jobewan joined #gluster
15:12 jbrooks joined #gluster
15:18 Slashman joined #gluster
15:19 klaxa|work joined #gluster
15:21 glusterbot New news from newglusterbugs: [Bug 1123004] Automounter maps with 'localdir -fstype=glusterfs host:/remote/dir' fails <https://bugzilla.redhat.co​m/show_bug.cgi?id=1123004>
15:22 sickness anyone already tried the ec xlator? =_)
15:22 recidive joined #gluster
15:22 mortuar joined #gluster
15:26 dusmant joined #gluster
15:36 ctria joined #gluster
15:36 ekuric joined #gluster
15:39 klaxa|work left #gluster
15:39 ndk` joined #gluster
15:40 jiffe98 well it seems if I stick with ubuntu 12.10 rather than 14.04 I can still re-export a local gluster 3.4.2 import
15:47 Lee- More strange behavior with gluster. If I delete the volume and try to create a new volume that utilizes one of those bricks, I get a message that the brick is in a volume, however I can't remove the brick from the volume since the volume is deleted.
15:48 nishanth joined #gluster
15:50 kkeithley_ gluster writes xattrs to the top-level directory of the brick. If you try to reuse the brick in a new volume, gluster will see them and refuse. You can delete the xattrs, or mkfs the volume to get rid of them, or if you followed "best practices" and created the brick in a subdir of the volume, you can `rm -rf $subdir`
15:51 deeville joined #gluster
15:51 jbrooks Hi all -- just a sanity check -- when you access a replicated gluster cluster through nfs, you're only ever talking directly to the host you nfs mount, right?
15:54 bennyturns jbrooks, afaik yes
15:55 jbrooks bennyturns: thx
15:55 Lee- I understand that there is old meta data associated with the brick. The odd behavior stems from the unclear messages or terminology in use. The message states that the brick is associated with a volume. If this is the case, then remove-brick should allow me to remove the brick from the volume, but it does not allow this.
15:56 Peter3 joined #gluster
15:56 Peter3 E [marker.c:2542:marker_removexattr_cbk] 0-sas02-marker: No data available occurred while creating symlinks
15:56 Peter3 what does this means?
15:56 bennyturns Lee-, there are xsattrs that dont get cleaned ip
15:56 jbrooks This is my understanding, but I've had trouble finding a doc page or something on the web clearly stating it
15:56 theron joined #gluster
15:57 bennyturns Lee-, there is a script that will clean them iirc?
15:57 bennyturns Lee-, I can't remember as I jsut call my cleanup script every time
15:58 Peter3 how can we tell is our filesystem takes xattrs ?
15:58 Peter3 i m on ubuntu gluster 3.5.1 with xfs
15:58 Peter3 i keep getting getattr and setattr errors
15:58 Lee- I understand that there are stale xattrs. The issue here is that the message clearly states that the brick is part of a volume. If this is the case, then the "gluster remove-brick" command should allow the removal. It does not. Therefore, either the message needs to be changed to something more clear or the remove-brick command should be modified to handle this case.
15:59 Lee- It's like: "you can't do X because of Y, go use the command to address Y", then you go use the command to address Y and it says "nope!"
15:59 bennyturns Lee-, ahh! may be worth filing an RFE?  make remove brick clean up stale xattrs as well?
16:08 jiffe98 so I have a directory that if I find | wc -l through the fuse mount I get 407 files but looking at the gluster servers themselves theres over 67000 files
16:08 sputnik13 joined #gluster
16:12 _Bryan_ joined #gluster
16:13 morse_ joined #gluster
16:16 dtrainor joined #gluster
16:16 dtrainor 'morning.  I keep running in to this, but there's no data in the volume.  Nothing on bricks, nothing in .gluster, either:  [2014-07-24 16:02:50.596601] E [afr-self-heal-common.c:233:a​fr_sh_print_split_brain_log] 0-vol-cinder-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 1 ] [ 1 0 ] ]
16:18 Lee- jiffe98, how many files in the ".glusterfs" directory on your gluster server?
16:20 daMaestro joined #gluster
16:22 vu joined #gluster
16:22 bennyturns dtrainor, I saw that in earlier builds but not for a long time, what gluster version you running?
16:23 dtrainor 3.5.1
16:23 dtrainor I think I *may* see it... I think cinder created a dir inside of the mount point where the gluster volume was mounted prior to the gluster volume being mounted
16:23 dtrainor gonna do a test
16:23 haomaiw__ joined #gluster
16:29 Peter4 joined #gluster
16:31 theron_ joined #gluster
16:36 dtrainor Alright.  I can't make this work.  Out of ideas.
16:41 dtrainor Does anything fishy happen when I have a single volume mounted twice on the same system?
16:42 dtrainor I do believe so.  I unmounted the existing volume and no more split-brain errors.
16:45 dtrainor that's a lie.  fixing this couldn't be that easy.
16:46 Peter3 joined #gluster
16:48 Peter4 joined #gluster
16:49 bennyturns dtrainor, are you looking to RCA or just recover from SB?   You can fix the split brian by just resetinng the xattr
16:50 dtrainor RCA would be nice but since there's nothing on here I can just blow it all away.  I did that a few times, removed all files in volumes and bricks and .gluster dirs, still get the error
16:50 dtrainor oh i can?
16:50 bennyturns dtrainor, ya I have the exact command in a bz, searching for it now
16:50 dtrainor me too
16:51 bennyturns dtrainor, https://bugzilla.redhat.com/show_bug.cgi?id=913208
16:51 glusterbot Bug 913208: is not accessible.
16:52 bennyturns iirc all you need to do is set these to 0:
16:52 bennyturns b
16:52 bennyturns [root@storage-qe04 ~]# getfattr -d -e hex -m "trusted.afr." /bricks/healtest_brick1/
16:52 bennyturns getfattr: Removing leading '/' from absolute path names
16:52 bennyturns # file: bricks/healtest_brick1/
16:52 bennyturns trusted.afr.healtest-client-​2=0x000000000000000200000000
16:52 bennyturns trusted.afr.healtest-client-​3=0x000000000000000000000000
16:52 dtrainor ah ha, i see it.
16:52 bennyturns dtrainor, I'll fire up my self heal tests that I hit this with in the past on 3.5
16:53 bennyturns to see if I can repro
16:53 dtrainor word.  i'm oging to try it again too
16:53 bennyturns you shouldn't ever get split brain on /
16:53 dtrainor right, that's weird.
16:53 dtrainor though it appears in my /var/log/glusterfs/var-lib-cinder-mnt​-4e82803995f082229cd734d5f62552e8.log log which hints that it applies only to this volume
16:53 dtrainor so i'm confused
16:54 JoeJulian Yes, in a broad sense: "getfattr -m trusted.afr -d -e hex $brick_root" then "setfattr -n $trusted-afr-from-previous -v 0x000000000000000000000000 $brick_root"
16:55 dtrainor trusted.afr.vol-cinder-client​-0=0x000000000000000100000000
16:55 dtrainor trusted.afr.vol-cinder-client​-1=0x000000000000000000000000
16:55 JoeJulian And on your other brick it'll be client-1 that's non-zero.
16:56 dtrainor right
16:58 JoeJulian set 'em both to zero and give them the bird.
16:59 andreask joined #gluster
16:59 bennyturns :)
16:59 JoeJulian (the bird is optional, but it usually helps)
16:59 dtrainor mandatory
17:01 zerick joined #gluster
17:02 dtrainor awesome.  that was it.  thank you.
17:02 bennyturns woot!
17:03 dtrainor is there a know reason for it?
17:03 bennyturns JoeJulian++
17:03 glusterbot bennyturns: JoeJulian's karma is now 7
17:03 dtrainor er, for why that happens
17:03 dtrainor thanks for the blog post that you had already linked me to in the past, JoeJulian++ ....
17:03 glusterbot dtrainor: JoeJulian's karma is now 8
17:03 bennyturns dtrainor, IMHO its something race-y happening when brick processes go offline / come online
17:04 dtrainor sounds about right
17:04 bennyturns dtrainor, but we weren't able to find the smoking gun at the time and the BZ got closed.  I haven't seen it for a year
17:05 JoeJulian I have no idea why that happens. Same think, it's so seldom it's hard to find a root cause.
17:05 JoeJulian s/think/thing/
17:05 glusterbot What JoeJulian meant to say was: I have no idea why that happens. Same thing, it's so seldom it's hard to find a root cause.
17:05 dtrainor haha
17:06 Peter3 joined #gluster
17:06 dtrainor now to get back to my original problem ....
17:06 cultavix joined #gluster
17:08 dtrainor https://access.redhat.com/solutions/193843
17:08 glusterbot Title: How to recover a file from a split-brain on a Red Hat Storage volume ? - Red Hat Customer Portal (at access.redhat.com)
17:09 JoeJulian @split-brain
17:09 glusterbot JoeJulian: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
17:09 JoeJulian Though test that first. I've had reports that it may not be working with 5.1
17:09 dtrainor ahh
17:10 JoeJulian well. I've had A report.
17:12 JoeJulian Not sure why it would fail. The RPC, though extended, shouldn't have changed for what I'm doing.
17:17 ndk`` joined #gluster
17:17 tdasilva joined #gluster
17:18 Peter3 my brick keep crashing under load
17:18 Peter3 http://pastie.org/9416690
17:18 glusterbot Title: #9416690 - Pastie (at pastie.org)
17:19 Peter3 anyone had similar issue??
17:19 Peter3 E [rpcsvc.c:533:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
17:20 Peter3 also keep getting error like this:
17:20 Peter3 E [marker.c:2542:marker_removexattr_cbk] 0-sas02-marker: No data available occurred while creating symlinks
17:20 Peter3 can use create symlink on a gluster nfs share?
17:21 JoeJulian Peter3: Ooh, that looks like a promising source for your quota mismatch
17:22 Peter3 ya!!!
17:22 Peter3 but what could cause this? and what should we do to resolvE?
17:23 Peter3 and it keep saying removexattr
17:23 Peter3 i have been getting all kinds of xattr related error
17:23 vu joined #gluster
17:23 Peter3 i m on ubuntu 12.04 with xfs
17:23 Peter3 how do i know if xattr is enabled??
17:24 JoeJulian Go to https://botbot.me/freenode/gluster/ and see if anybody answered that question even though you logged back off.
17:24 glusterbot Title: Logs for #gluster | BotBot.me [o__o] (at botbot.me)
17:24 eryc joined #gluster
17:25 bennyturns Peter3, I usually just try to set an xarr on a file to see if its enabled.
17:25 JoeJulian Ok, the readv failed and subsequent rpc errors are caused by the brick crash.
17:26 Peter3 hmmm so where should i look into why brick crash?
17:27 JoeJulian trusted.*, system.*, and security.* extended attributes (I think there's one more iirc) can only be set by root. Unless you mount with user_attr I don't think you can set other random xattrs.
17:27 Peter3 u mean setfattr ?
17:28 JoeJulian But most (if not all) filesystems that support xattrs enable them by default.
17:28 JoeJulian And gluster will fail to create a brick if that brick's filesystem does not support xattrs.
17:30 Peter3 https://access.redhat.com/documentation/en-US/Red​_Hat_Storage/2.1/html/Administration_Guide/chap-A​dministration_Guide-Performance_Enhancements.html
17:30 glusterbot Title: Chapter 11. Configuring Red Hat Storage for Enhancing Performance (at access.redhat.com)
17:30 Peter3 i was trying to following this to create xfs on ubuntu
17:30 JoeJulian Peter3: What was the bug id of the crash you filed with the core file?
17:30 Peter3 but mkfs.xfs size=512 failed
17:30 JoeJulian mkfs.xfs -i size=512
17:31 Peter3 ya -i
17:31 Peter3 failed
17:31 Peter3 ubuntu can said can only do 256
17:31 * JoeJulian raises an eyebrow.
17:31 bennyturns thats odd
17:31 JoeJulian Nope, I use 512 on my bricks.
17:31 Peter3 u on ubuntu?
17:31 JoeJulian Unfortunately.
17:32 bennyturns Peter3, it shouldn't matter unless you have alot of app setting xattrs on things.  most peeps don't hit problem with the default inode size
17:32 Peter3 ya i tho so too but i have been getting hits on lots of error regarding xattr
17:33 JoeJulian It was theorized that it would be more efficient, but apparently when Red Hat's performance testers checked, it didn't really matter.
17:33 Peter3 E [marker.c:2482:marker_setattr_cbk] 0-sas02-marker: Operation not permitted occurred during setattr of /TrafficPrDataFc01/TrafficCost//mbq​/costdtl_mbq_20140724062338_038.bcp
17:33 bennyturns JoeJulian, here is the official from the xfs guys:
17:33 bennyturns Recommendation: ( from first meeting ) for large files, it doesn't matter. If average file size is small, then if using Swift/Gluster go with 256-byte inode size, since Swift will generate additional metadata that won't fit in 512-byte inode anyway, otherwise use 512-byte inode size as we do today.   Test Swift/Gluster with ssbench or other Swift benchmark.
17:35 bennyturns JoeJulian, it made a big different when lots of xattrs were set:
17:35 bennyturns This filesystem parameter has a significant (40%) impact on XFS small-file performance when extended attributes are used, but less of an effect with Gluster (RHS 2.0U4) on top of XFS. In addition, it is not clear with OpenStack Swift, SELinux, samba, and other Gluster services adding their own extended attributes, that inode size of 512 or even 1024 is sufficient most of the time. When xattr data overflows all 3 sizes, then 256-byte inode p
17:35 bennyturns erforms best. Data indicates that pure native Gluster works best with inode size of 512. Another factor: wasted space in xattr extents, which are always 4 KB. allocsize=4k option is unnecessary if default 256-byte size is used.
17:36 Peter3 so my default mkfs.xfs should work in ubuntu?
17:36 bennyturns Peter3, yep.
17:38 XpineX joined #gluster
17:42 mojibake joined #gluster
17:43 Peter3 and gluster been running stable until this monday
17:43 qdk joined #gluster
17:43 Peter3 i disabled and renabled quota on the volume
17:43 Peter3 and we have been problematic since then
17:44 Peter3 it generated a large heal on volume
17:44 Peter3 and since then nfs and brick keep breaking
17:44 mojibakeumd @JoeJulian... Saw your comment about https://botbot.me/freenode/gluster/. Is irc.gnu.org, and irc..freenode.net tied into same IRC system? Maybe that is why my nick was taken. LOL.
17:45 Peter3 and all those relink issue
17:45 vshankar joined #gluster
17:45 Peter3 just strange and more space issue with quota on the replica 2 volume
17:46 bennyturns Peter3, are you going to continue to use quota?  just curious why you disable / re enabled it?
17:46 Peter3 cuz i got a quota error
17:46 Peter3 when ever i did a volume quota list
17:46 Peter3 i got error like can not do quota on a gfid
17:46 Peter3 my last resort was disable quota and reenable it
17:46 Peter3 and it goes away
17:47 Peter3 but the reenable quota generated a massive healing
17:47 bennyturns hrm thats really strange I enable / disable quots in tests about 100 times a day :/
17:47 Peter3 how many files in ur volume
17:47 Peter3 and is ur volume replicated?
17:47 bennyturns Peter3, it depends on file sizes / numbers and yes its replicated
17:47 Peter3 my volume has more then 4000k files
17:47 bennyturns nothing under 50 GB
17:48 Peter3 and has around 4TB data
17:48 Peter3 and it's replicated
17:49 JoeJulian mojibake: yes, irc.glu.org and irc.freenode.net are the same.
17:49 Peter3 and when i reenable quota, brick start crashing
17:49 Peter3 getting rpc error
17:49 Peter3 and also quota filled up on exports
17:49 Peter3 and getting healing error
17:50 Peter3 and like a chain effect then entire gluster crashed
17:50 Peter3 and seems like it generated tons of lost links and gfids
17:50 bennyturns Peter3, maybe you should run the cleanup xattr script for cleaning up after quota is disabled
17:50 Peter3 and feels like i m screwed
17:51 Peter3 where is the cleanup xattr??
17:51 bennyturns looking
17:51 mojibake JoeJulian: Thank you.. I am now myself.
17:51 Peter3 THANKS!!!
17:51 JoeJulian I was also considering rm -rf the markers tree under .glusterfs while quota was disabled.
17:52 bennyturns Peter3, scripts are in the gluster source, /glusterfs/extras
17:52 daMaestro joined #gluster
17:52 Peter3 joeJulian: where is the markets tree?
17:52 Peter3 which one?
17:52 Peter3 i m in .glusterfs but dun know which one
17:53 bennyturns Peter3, here is the doc on it https://access.redhat.com/documentation/en-U​S/Red_Hat_Storage/2.1/html/Administration_Gu​ide/chap-User_Guide-Dir_Quota-Disable.html
17:53 glusterbot Title: 13.8. Disabling Quotas (at access.redhat.com)
17:53 _dist joined #gluster
17:53 Peter3 ok let me try it on my dev system
17:54 Peter3 which one is the marker tree??
17:54 Peter3 i see this
17:54 Peter3 http://pastie.org/9418111
17:54 glusterbot Title: #9418111 - Pastie (at pastie.org)
17:55 JoeJulian try ls -la
17:55 JoeJulian I thought I remembered it being named something obvious
17:56 Peter3 http://pastie.org/9418122
17:56 glusterbot Title: #9418122 - Pastie (at pastie.org)
17:57 bennyturns I wonder if the xattrs got 1/2 cleaned up when quota was disabled here
17:58 JoeJulian Where's indices? Did that go away with 3.5? replaced with changelog I suppose
17:59 Peter3 hmm  not sure
17:59 Peter3 bennyturns: how can i tell if xattrs got cleaned up?
18:00 Peter3 i see tons of goodies scripts
18:00 Peter3 backend-cleanup.sh ?
18:00 Peter3 clear_xattrs.sh ?
18:00 Peter3 disk_usage_sync.sh ??
18:00 bennyturns Peter3, run this on one fo your files getfattr -m . -d -e hex
18:00 bennyturns you should see something like:
18:00 bennyturns trusted.glusterfs.quota.limit-set=​0x0000003200000000ffffffffffffffff
18:01 Peter3 run on a gfs mount?
18:01 Peter3 or from the gluster node?
18:01 Peter3 or against the brick?
18:01 bennyturns Peter3, on the brick
18:01 Peter3 O!
18:01 Peter3 OK!
18:01 bennyturns run on a dir not a file
18:02 Peter3 so i am going to run on one of the gluster server's brick subdirectory?
18:02 bennyturns Peter3, check the dir you set the quota on
18:03 * bennyturns can't find the script.  I tought it was in /usr/libexec/glusterfs/quota/quota-remove-xattr.sh
18:04 Peter3 i see these
18:04 Peter3 http://pastie.org/9418138
18:04 glusterbot Title: #9418138 - Pastie (at pastie.org)
18:04 bennyturns Peter3, same, still looking.
18:04 bennyturns Peter3, you still quota xattrs on the dirs though>?
18:05 Peter3 http://pastie.org/9418141
18:05 glusterbot Title: #9418141 - Pastie (at pastie.org)
18:05 Peter3 i got this from the getattr
18:05 Peter3 YES
18:05 bennyturns ya they still there
18:05 Peter3 i have quota enabled
18:05 JoeJulian https://github.com/gluster/glu​sterfs/tree/release-3.4/extras
18:05 glusterbot Title: glusterfs/extras at release-3.4 · gluster/glusterfs · GitHub (at github.com)
18:05 Peter3 i am going to run against my prods now
18:06 JoeJulian So that was removed from master
18:06 Peter3 u mean the clear_xattrs.sh ??
18:07 diegows joined #gluster
18:07 bennyturns the one I am thinking of was called /usr/libexec/glusterfs/quota/quota-remove-xattr.sh
18:07 bennyturns maybe something has changed?
18:08 Peter3 i just noticed something!!!!
18:08 Peter3 on prod which the dirs having quota issue
18:08 Peter3 trusted.glusterfs.quota.size=0x0000003f73c52e00
18:08 Peter3 this is different on each gluster node
18:09 Peter3 while they are the same on dev
18:09 bennyturns ya my thinking was:
18:09 Peter3 are they suppose to be the same for the same dir?
18:09 bennyturns you disable, the cleanp start
18:09 bennyturns you renable while the cleanup is happening
18:09 recidive joined #gluster
18:09 bennyturns weird stuff happens
18:09 Peter3 ah.....
18:09 cultav1x joined #gluster
18:09 Peter3 yes!!!!
18:09 Peter3 ls
18:09 Peter3 opps
18:10 Peter3 is the trusted.glusterfs.quota.size suppose to be the same on all the nodes?
18:10 bennyturns it makes sense, I wonder how we are cleaning up the quota xattrs now a days
18:10 Peter3 err...
18:11 JoeJulian No, you shouldn't even see that attribute on the client nodes.
18:11 Peter3 i see this on the server node
18:11 JoeJulian 'cause you said all the nodes.... :P
18:12 Peter3 opps sorry :P
18:12 bennyturns Peter3, when you re set the quota did you use a different limit?
18:12 Peter3 nope
18:12 Peter3 same
18:12 Peter3 trusted.afr.sas04-client-2=​0x000000000000000000000000
18:12 Peter3 trusted.afr.sas04-client-3=​0x000000000000000000000000
18:12 Peter3 what are these?
18:12 bennyturns cheangelofgs
18:12 Peter3 i see for same dir, some server node has
18:13 Peter3 what's that?
18:13 bennyturns how afr tracks changes to replicas
18:13 Peter3 ic
18:13 bennyturns if those are different the file is split brain
18:14 Peter3 on another volume
18:14 Peter3 which quota was not touched
18:14 Peter3 i also noticed the same issue on a dir
18:15 Peter3 that the trusted.glusterfs.quota.size is different on each server nodes
18:15 daMaestro joined #gluster
18:16 bennyturns Peter3, k, maybe its different on each subvolume or somehting
18:16 bennyturns Peter3, here is the old script that was removed https://github.com/gluster/glusterfs/blob​/release-3.4/extras/quota-remove-xattr.sh
18:16 glusterbot Title: glusterfs/extras/quota-remove-xattr.sh at release-3.4 · gluster/glusterfs · GitHub (at github.com)
18:16 JoeJulian correction: of those are different the file *may be* split brain. If they are conflicting, that's SB.
18:17 bennyturns JoeJulian, thats what I was meaning :)
18:17 JoeJulian :)
18:18 JoeJulian bug 969461
18:18 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=969461 high, high, ---, rgowdapp, NEW , RFE: Quota fixes
18:18 Peter3 i should run that script when quota disabled
18:18 JoeJulian @meh
18:18 glusterbot JoeJulian: I'm not happy about it either
18:18 JoeJulian yes, with quota disabled.
18:18 Peter3 cool
18:19 bennyturns Peter3, thats what I am thinking
18:19 JoeJulian http://review.gluster.org/6369
18:19 glusterbot Title: Gerrit Code Review (at review.gluster.org)
18:19 JoeJulian setfattr -n "glusterfs.quota-xattr-cleanup" -v 1 <path/to/file>
18:20 JoeJulian Interesting
18:20 Peter3 that's new right? :P
18:20 bennyturns yepo!
18:20 JoeJulian So you can just disable quota, then do a "find $client_mount -exec setfattr -n "glusterfs.quota-xattr-cleanup" -v 1 {}\;"
18:20 JoeJulian Yes, that's new in 3.5
18:21 Peter3 oh so i can do that now?!
18:21 JoeJulian yes
18:21 Peter3 should i run the find on each gluster server node?
18:22 Peter3 or from a gfs client?
18:22 JoeJulian No. On a client mount
18:22 JoeJulian hence the psuedo-variable $client_mount
18:22 JoeJulian That'll have to be a fuse mount
18:22 JoeJulian not nfs
18:23 JoeJulian (I think)
18:23 Peter3 when i try to do a getfattr
18:23 Peter3 getfattr -m . -d -e hex devsas01
18:23 Peter3 i got nothing from fuse client
18:24 Peter3 i do that on a gfs client node
18:24 JoeJulian That's correct.
18:24 Peter3 but i can still run the setattr??
18:24 JoeJulian yes
18:25 JoeJulian The quota translator listens for that attribute being set and performs the xattr clean-up task on the brick replicas.
18:25 vu joined #gluster
18:25 bennyturns thats pretty cool, way better than runningthat script
18:28 * bennyturns is gonna grab some food real quick bbiab
18:30 Philambdo joined #gluster
18:30 Peter3 i just kicked off quota disable
18:30 Peter3 notices this
18:30 Peter3 http://pastie.org/9418196
18:30 glusterbot Title: #9418196 - Pastie (at pastie.org)
18:30 Peter3 on the server node
18:30 Peter3 so seems like it's actually running that on the server
18:31 JoeJulian Heh, so it does.
18:31 Peter3 ya on all the node
18:31 Peter3 on all the server nodes
18:32 JoeJulian That seems like overkill, but whatever.
18:32 Peter3 and looks like gonna take forever
18:33 sputnik13 joined #gluster
18:35 caiozanolla joined #gluster
18:35 Peter3 while the setfattr still running
18:35 Peter3 http://pastie.org/9418203
18:35 glusterbot Title: #9418203 - Pastie (at pastie.org)
18:35 Peter3 getting these
18:41 JoeJulian It's just a warning (" W ") so unless there's a problem afterward, I would ignore it.
18:44 Peter3 just done running
18:45 ekuric joined #gluster
18:45 JoeJulian make sure it's done on all your servers
18:45 Peter3 yes
18:46 JoeJulian Overheard and this makes my brain melt: "Sounds like someone needs to leverage the power of distributed results-driven scalable private-cloud big-data infrastructure as a service."
18:46 Peter3 lol
18:46 JoeJulian Well then if it's all done I'd say re-enable and see if it's fixed.
18:46 Peter3 so i assume the clean up is done and i can reenable quota?
18:47 Peter3 ya this is my dev system
18:47 Peter3 i will run this on my prod after this
18:47 JoeJulian Yay for dev!
18:47 Peter3 the enable is super quick
18:47 JoeJulian fyi: bug 1123067
18:47 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1123067 unspecified, unspecified, ---, gluster-bugs, NEW , User is not informed, nor is there a way to check if completed, the quota xattr cleanup after disabling quota
18:47 Peter3 o it's crawling again
18:48 JoeJulian Right. has to build the quotas
18:48 Peter3 o man....
18:48 Peter3 3.5.2 bug? :P
18:49 JoeJulian And they wonder why I'm still on 3.4
18:49 Peter3 yes i wonder too
18:50 JoeJulian Because they changed a bunch of things, rewrote from scratch, etc. and there are always bugs associated with that. Usually the .1 is relatively safe depending on your SLA's etc.
18:51 Peter3 i wonder if the quota renable would fix this?
18:51 Peter3 E [marker.c:2542:marker_removexattr_cbk] 0-sas02-marker: No data available occurred while creating symlinks
18:51 JoeJulian Now I wait for feedback from the people that are willing to test the .2 release. If there's nothing major, then I start to get more confident.
18:52 Peter3 hope the .2 is more stable
18:52 Peter3 it's been my nightmare here
18:52 JoeJulian But then there's things like quota that maybe 1% of the userbase use.
18:52 glusterbot New news from newglusterbugs: [Bug 1123067] User is not informed, nor is there a way to check if completed, the quota xattr cleanup after disabling quota <https://bugzilla.redhat.co​m/show_bug.cgi?id=1123067>
18:52 Peter3 i think it's all started after i disabled the quota….
18:53 JoeJulian So unless you guys test the betas and file bugs, those bugs are going to stay in releases.
18:53 Peter3 lesson learn
18:54 JoeJulian Sorry, didn't mean to make it sound like a lesson. I'm just reflecting on the process.
18:54 Peter3 totally!! appreciate for helping!!
18:55 JoeJulian We (the community) would be happy to have more testers though.
18:55 Peter3 i have test systems and willing to test 3.5.2
18:55 Peter3 i would like to pick up the testing process too
18:55 Peter3 esp if it can be jenkins those steps
18:55 _dist anyone know where I can get the deb for glusterfs-dbg 3.4.2-1 ?
18:56 _dist (debian wheezy)
18:58 JoeJulian http://download.gluster.org/pub/gluster/gluster​fs/3.4/3.4.2/Debian/apt/pool/main/g/glusterfs/
18:58 glusterbot Title: Index of /pub/gluster/glusterfs/3.4/3.4.2/​Debian/apt/pool/main/g/glusterfs (at download.gluster.org)
18:58 JoeJulian oh, -1...
18:58 _dist http://download.gluster.org/pub/gluster/glu​sterfs/3.4/LATEST/old/Debian/apt/pool/main/​g/glusterfs/glusterfs-dbg_3.4.4-1_amd64.deb
18:58 _dist just found it :)
18:58 JoeJulian no idea there.
18:58 _dist oh wait, damn that's 4.4
18:58 JoeJulian That's 3.4.4
18:59 * _dist has a feeling he'll have to compile it
19:00 JoeJulian I think they pulled the -1 packages right away due to some critical bug.
19:00 _dist oh? any idea what the bug was?
19:00 JoeJulian Something about formatting the root filesystem...
19:00 JoeJulian Nah! I'm just kidding.
19:00 _dist haha, I'll read the release notes
19:01 _dist I can't imagine a serious bug, I've been running it for months
19:01 _dist but (who knows!)
19:01 JoeJulian People were running heartbleed for months too...
19:01 _dist well I'm certain 3.4.2* has bugs :)
19:02 JoeJulian :)
19:02 JoeJulian I know at least one that will crash clients when doing a rebalance.
19:02 JoeJulian That's fixed in 3.4.5
19:02 Peter3 E [server-rpc-fops.c:1076:server_unlink_cbk] 0-sas02-server: 20391: UNLINK /image-11032923594200.jpg (fc474b45-1c40-4e16-a359-d0aea2c​23b42/image-11032923594200.jpg) ==> (Permission denied)
19:02 JoeJulian ... I'm starting to worry about semiosis.
19:03 Peter3 how do i get a permission denied on gluster log?
19:03 _dist haven't seen him in a while
19:04 JoeJulian yeah, and he's not responding on hangouts
19:04 JoeJulian @seen semiosis
19:04 glusterbot JoeJulian: semiosis was last seen in #gluster 5 days, 21 hours, 32 minutes, and 4 seconds ago: <semiosis> randomish
19:04 Peter3 vacation?
19:05 * JoeJulian shrugs
19:06 _dist I can only find -1 for ubuntu
19:07 _dist I'm using proxmox (cause afaik it's the only ui that works with libgfapi) but sometimes I get stuck guests. I'm get gdb up on that node but it's missing the debug packages. Unfortunately it thinks the glusterfs-dbg should be for 3.2.7 even though -client is @ 3.4.2-1 so I was hoping to just get the deb...
19:08 JoeJulian Try the -2. It might have just been a packaging thing.
19:08 _dist *I'm trying to get gdb up... (I swear sometimes words I type just don't show up)
19:08 _dist already did, refuses to install due to mismatch version on client
19:08 JoeJulian lame
19:08 _Bryan_ Can someone point me to the doc for gluster 3.2.5 on setting up a node in the cluster to be a nfs gateway....but not part of the storage nodes.
19:09 _Bryan_ I cant seem to find the link I had sved
19:09 JoeJulian _Bryan_: Just add it as a peer.
19:09 _dist JoeJulian: exactly, this is that kind of annoying setup issue that makes me mad :)
19:09 JoeJulian It doesn't have to have any bricks.
19:09 _Bryan_ right you prove it so it is part of the cluster....but if I remember correctly I had to change somethingin the startup script to start the nfs daemon on just that one and not on all gluster nodes
19:10 JoeJulian Ah, well I've not seen anything about that.
19:11 JoeJulian I suppose you could kill the glusterfs instance for nfs on all the servers but the one you don't want it dead on.
19:11 _dist _Bryan: as lon as you don't install the server on there it should have a port conflict (if that's what you mean)
19:11 _Bryan_ since there are issues with performance and stability in 3.2.5 with enabling the NFS server...there was a howto that described this...to add an additional node that basically jsut acted like a gateway
19:11 _Bryan_ to be clear I am only discussing the internal gluster nfs daemon..
19:16 sputnik13 joined #gluster
19:19 Peter3 on prod, when i disable quota on a volume
19:19 Peter3 i got this
19:19 Peter3 .2014-07-24 19:15:19.837116] E [glusterd-utils.c:4124:gluster​d_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/e472f475caa7b04ff9d3cc9465979ed9.socket error: No such file or directory
19:20 Peter3 how would that related?
19:21 _Bryan_ Found my answer....you have to start gluster, peer probe it....then stop gluster and modify the init.d script o that it only starts the gluster nfs daemon then install the client and mount the gluster volume.....then you can use the gluster nfs daemon to export a remotely mounted gluster volume
19:22 JoeJulian _Bryan_: As an aside, is there some reason you don't just use a more stable GlusterFS version?
19:22 _Bryan_ There were quite a few people who worked on this awhile ago..
19:23 _Bryan_ Lets say I have ALOT of gluster nodes in many volumes and a significantly more clients....to upgrade I have to do it all at once..since 3.3+ is not backward companible to 3.2.5 clients.....and this is a HUGH undertaking that we jsut have not made it to yet
19:24 _Bryan_ basically right now it is the daemons I know vs the daemons I do not...the long term goal is to upgrade..but we have to completely redo my performance tuning so we know how to setup the new version...and then figure out a way to upgrade everything without causing downtime.
19:26 JoeJulian That's another thing you'll see. 3.3 had some major performance improvements over 3.2
19:33 bennyturns Peter3, howd it go?
19:33 * bennyturns is back
19:33 magicrobotmonkey joined #gluster
19:34 magicrobotmonkey i have kind of a strange problem
19:34 magicrobotmonkey i have a volume mounted on two different hosts
19:34 _Bryan_ yep...
19:34 magicrobotmonkey and one particular dir has different permssions on both mounts
19:34 _Bryan_ just a BIG step...
19:35 bennyturns magicrobotmonkey, are they replicated?
19:35 magicrobotmonkey no
19:35 bennyturns I mean are you using a replicated volume
19:36 JoeJulian Different permissions? Do tell.
19:36 * JoeJulian is betting on same permissions, different owners.
19:37 _pol joined #gluster
19:37 magicrobotmonkey heh, yes good point, JoeJulian
19:37 magicrobotmonkey it is different owners
19:37 magicrobotmonkey but the uids are the same
19:37 magicrobotmonkey and its root on one
19:37 bennyturns ahh!
19:38 JoeJulian ownership on filesystems is stored as uid and gid. That just means your passwd and group tables are not the same between clients.
19:38 bennyturns magicrobotmonkey, look in the /etc/passwd on the clients, I bet the number is different
19:38 magicrobotmonkey yes i'm aware. they are the same
19:38 JoeJulian hmm...
19:39 magicrobotmonkey wait
19:39 magicrobotmonkey it just changed back
19:39 magicrobotmonkey wait a tic
19:40 magicrobotmonkey ok
19:40 magicrobotmonkey if im in the directory and i ls -la ../ is right
19:41 magicrobotmonkey but outside its wrong
19:41 SpComb symlinks?
19:41 sputnik13 joined #gluster
19:42 rwheeler joined #gluster
19:42 JoeJulian Also, add -i to your ls to ensure you're looking at the same inode either way.
19:43 JoeJulian I bet SpComb is right, you're dereferencing a symlink differently
19:44 magicrobotmonkey yea im suspecting a hardlink
19:44 _Bryan_ Just being around you guys helped me remember...  8-)
19:44 JoeJulian That happens to me all the time.
19:45 JoeJulian As I'm describing my problem in some IRC channel or another, I suddenly realize what I didn't look at.
19:45 magicrobotmonkey they're both the same inode
19:47 JoeJulian use fpaste.org and show us
19:48 JoeJulian in fact, show us the stat command both ways
19:49 magicrobotmonkey http://fpaste.org/120570/06231337/
19:49 glusterbot Title: #120570 Fedora Project Pastebin (at fpaste.org)
19:49 magicrobotmonkey its the segway dir
19:49 magicrobotmonkey (sorry for my fancy prompt)
19:50 JoeJulian That's really wierd.
19:51 magicrobotmonkey http://fpaste.org/120571/14062314/
19:51 glusterbot Title: #120571 Fedora Project Pastebin (at fpaste.org)
19:51 magicrobotmonkey thats stats
19:51 JoeJulian Oh, I bet it's different on different bricks.
19:51 JoeJulian a split-brain directory. Never seen it legitimately.
19:52 magicrobotmonkey wow
19:52 JoeJulian yep. segway is a file on one of your bricks.
19:52 JoeJulian instead of a directory
19:52 JoeJulian well, maybe not
19:53 JoeJulian I should think before I speak occasionally
19:53 magicrobotmonkey well i dont know enough about introspecting gluster, but how can i find out which brick something is on?
19:53 JoeJulian ~pasteinfo | magicrobotmonkey
19:53 glusterbot magicrobotmonkey: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:55 magicrobotmonkey http://fpaste.org/120574/40623170/
19:55 glusterbot Title: #120574 Fedora Project Pastebin (at fpaste.org)
19:57 JoeJulian So on each server, just ls -ld /lsi_jbrick*/research/rparulkar/segway and make sure they're all the same.
19:58 magicrobotmonkey interesting
19:58 magicrobotmonkey ok
19:59 JoeJulian If they're all correct, then I'm stumped.
20:00 magicrobotmonkey they are not
20:00 JoeJulian whew
20:00 magicrobotmonkey in fact, on the first server, only 1 is right
20:01 JoeJulian The first thing I would try is, from the client, change the perms/owner away from what you want to something else entirely, then change them back and see if that fixes what you're seeing on the servers.
20:02 magicrobotmonkey k
20:02 magicrobotmonkey it only changed it on the one that was right
20:03 magicrobotmonkey the root ones stayed root
20:03 JoeJulian "getfattr -m . -d -e hex" for one of the entries that's wrong and the one that's right.
20:05 magicrobotmonkey uh us that supposed to output something?
20:05 JoeJulian Do it as root
20:05 magicrobotmonkey ah right
20:07 magicrobotmonkey http://fpaste.org/120577/06232431/
20:07 glusterbot Title: #120577 Fedora Project Pastebin (at fpaste.org)
20:10 vu joined #gluster
20:10 Peter3 my quota disbale crawl seems hang
20:11 Peter3 the quota-crawl.log hasn't been update for hour
20:11 JoeJulian magicrobotmonkey: gluster volume rebalance bluthton fix-layout start
20:12 magicrobotmonkey is this going to take any data offline?
20:12 JoeJulian no
20:13 JoeJulian They should all have trusted.glusterfs.dht and that should fix that missing attribute.
20:13 magicrobotmonkey ok
20:14 magicrobotmonkey how can i monitor the progress of that?
20:14 magicrobotmonkey (sorry for all the questions, i've inherited this cluster and haven't gotten to know it yet)
20:15 bennyturns gluster volume rebalance $volname status
20:18 bennyturns Peter3, this is new, maybe there is a problem with it?
20:19 bennyturns Peter3, I am gonna spin up an env and try it real quick
20:19 Peter3 for the find quota crawl, i just did a strace and seems like it still running
20:19 Peter3 cool thanks!
20:19 Peter3 i tried the disbale and reenable on my dev and works
20:19 Peter3 but now seems stalled on the prod
20:20 Peter3 but the /usr/bin/find . -exec /usr/bin/setfattr -n glusterfs.quota-xattr-cleanup -v 1 {} \ ; seems still running
20:20 Peter3 on the other hand, i seeing the volume that i had quota disable and reenabled getting tons of error on the brick log
20:21 magicrobotmonkey it seems like its working, thanks, JoeJulian and bennyturns
20:21 bennyturns JoeJulian++ again :)
20:21 glusterbot bennyturns: JoeJulian's karma is now 9
20:21 JoeJulian magicrobotmonkey: Thanks for a fun puzzle.
20:21 Peter3 http://pastie.org/9418378
20:21 glusterbot Title: #9418378 - Pastie (at pastie.org)
20:22 Peter3 bennyturns and JoeJulian: maybe a recap on my story
20:22 Peter3 i try to fix a quota issue and disable and reenabled one of my volume
20:23 Peter3 and now the volume seems having issue including performance and quota
20:23 Peter3 i am now disabling quota on another volume that had quota on and still running the find
20:24 Peter3 but on the quota side, we see the df vs du issue happening randomly on all volumes
20:24 theron joined #gluster
20:26 theron joined #gluster
20:26 rotbeard joined #gluster
20:28 theron_ joined #gluster
20:29 vu joined #gluster
20:33 bala joined #gluster
20:44 xleo joined #gluster
20:45 calum_ joined #gluster
20:46 JoeJulian Hey, bennyturns, do you happen to know for certain. I've been working under the assumption that as soon as the first server says the write is done, the client tells fuse the write is done, even if the write is still outstanding for one or more other replicas. Is that correct?
20:46 _dist JoeJulian: is there an easy way to strip all gluster xattrs from a file ?
20:47 bennyturns JoeJulian, I don't know for sure but that is how I understand it
20:49 JoeJulian _dist: There's no "setfattr -n * -x", no. I would do it in python. Walk the tree, read the xattr, remove it.
20:50 JoeJulian QAE doen't know the order of ops... <sigh>
20:52 JoeJulian You know what... I don't think that's true. If you're writing to a locally mounted replica on one of the servers. Start a series of small writes to it. Down the other server, the writes get significantly faster.
20:55 kkeithley1 joined #gluster
20:56 andreask joined #gluster
21:11 andreask joined #gluster
21:19 plarsen joined #gluster
21:21 daMaestro joined #gluster
21:30 sjm left #gluster
21:32 rjoseph joined #gluster
21:57 _pol joined #gluster
21:57 bala joined #gluster
21:58 _pol joined #gluster
22:15 Peter3 the quota crawl runs forever
22:15 Peter3 and when i stopped the volume, the crawl stopped
22:16 Peter3 and unlike yesterday when we try to stop a quota enabled volume and hang, it did not hang this time
22:24 daMaestro joined #gluster
22:32 _pol joined #gluster
22:43 Pupeno joined #gluster
22:48 cogsu_ joined #gluster
22:54 Nowaker joined #gluster
23:00 jiffe98 man gluster is just killing me lately, so many problems
23:00 jiffe98 I can cat a file but I can't touch or rm it
23:00 jiffe98 says it doesn't exist
23:17 rjoseph joined #gluster
23:32 systemonkey joined #gluster
23:36 KORG joined #gluster
23:42 bene2 joined #gluster
23:48 Peter3 bennyturns: how does it is take for u to stop and restart quota?
23:52 Peter3 anyone know about this error?
23:52 Peter3 E [nfs3.c:301:__nfs3_get_volume_id] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5​.1/xlator/nfs/server.so(nfs3_fsstat+0x1be) [0x7f045c9d6d7e] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1​/xlator/nfs/server.so(nfs3_fsstat_reply+0x3b) [0x7f045c9d665b] (-->/usr/lib/x86_64-linux-gnu/glus​terfs/3.5.1/xlator/nfs/server.so(n​fs3_request_xlator_deviceid+0x78) [0x7f045c9c9078]))) 0-nfs-nfsv3: invalid argument: xl
23:52 glusterbot Peter3: ('s karma is now -5
23:52 glusterbot Peter3: ('s karma is now -6
23:52 glusterbot Peter3: ('s karma is now -7
23:56 hagarth joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary