Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-08-26

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 jonb joined #gluster
00:10 haomaiwa_ joined #gluster
00:17 jmarley joined #gluster
00:22 gildub_ joined #gluster
00:29 stopbyte joined #gluster
00:42 Zhang joined #gluster
00:47 stopbyte i'm running into an issue with one volume (out of 7), using distributed-replicate (2 bricks each on 2 glusterd hosts), regularly not reporting some files in ls output on a client as well as regularly showing duplicate entries for other files
00:48 stopbyte the issue was initially noticed while running 3.6.1 (the first version we had installed in this environment), but we just upgraded to 3.6.5 and it still occurs
00:50 stopbyte we've tried clearing all the extended attributes, to no effect, and even just wiped out everything under the individual brick/ directories (including .glusterfs/) while the volume was stopped and clients were disconnected, then copied files back into the newly-emptied volume (over gluster, not directly into the bricks) -- but the same files end up being invisible on the client (even though they are present
00:50 stopbyte on the bricks)
00:53 stopbyte this is on centos 6.6 using the official RPMs, for whatever that's worth
00:53 srepetsk joined #gluster
00:54 stopbyte rather, the official gluster.org RPMs, not centos-supplied RPMs
01:06 shyam joined #gluster
01:10 DV_ joined #gluster
01:10 haomaiwa_ joined #gluster
01:33 Lee1092 joined #gluster
01:38 mckaymatt joined #gluster
01:45 gildub_ joined #gluster
02:06 gildub_ joined #gluster
02:10 haomaiwa_ joined #gluster
02:19 d-fence joined #gluster
02:23 kperkowski joined #gluster
02:26 kperkowski Hello all. I just installed a 2 brick system with one volume as a replica. When starting this custom app on the local disk it takes 6 seconds to start. However on the gluster volume it's taking 55 seconds to complete the service status command. Running the profile command on the volume showed me alot of latency on LOOKUP fop's. Any ideas on how to speed this up>
02:26 kperkowski *?
02:27 kperkowski The application using ruby on rails and a mysql db
02:29 kperkowski Just a FYI. Running centos 7 w/ glusterfs 7.3
02:33 PaulCuzner joined #gluster
02:42 auzty joined #gluster
02:44 mpietersen joined #gluster
02:44 harish joined #gluster
02:47 nangthang joined #gluster
02:51 gildub joined #gluster
03:08 elico joined #gluster
03:09 kdhananjay joined #gluster
03:10 haomaiwa_ joined #gluster
03:13 maveric_amitc_ joined #gluster
03:17 kotreshhr joined #gluster
03:18 vmallika joined #gluster
03:22 sakshi joined #gluster
03:26 kdhananjay joined #gluster
03:27 mjrosenb so, what is the .glusterfs directory used for?
03:28 TheSeven joined #gluster
03:35 dlambrig joined #gluster
03:39 baojg joined #gluster
03:41 woakes070048 joined #gluster
03:42 nbalacha joined #gluster
03:45 hchiramm_home joined #gluster
03:45 m0zes joined #gluster
03:46 raghug joined #gluster
03:47 atinm joined #gluster
03:48 shubhendu joined #gluster
03:52 kotreshhr left #gluster
03:53 baojg joined #gluster
03:54 s_ joined #gluster
03:55 ndarshan joined #gluster
03:55 gildub joined #gluster
04:00 nishanth joined #gluster
04:02 itisravi joined #gluster
04:03 kanagaraj joined #gluster
04:05 haomaiwa_ joined #gluster
04:07 skoduri joined #gluster
04:08 RameshN_ joined #gluster
04:10 haomaiwa_ joined #gluster
04:16 bharata-rao joined #gluster
04:17 gem joined #gluster
04:26 anil joined #gluster
04:26 yazhini joined #gluster
04:30 woakes070048 how can I remove all files of a brick?
04:30 kkeithley1 joined #gluster
04:31 neha joined #gluster
04:31 mjrosenb rm -rf?
04:33 woakes070048 I have done that before and it lead to a split brain situation
04:35 deepakcs joined #gluster
04:39 hgowtham joined #gluster
04:49 gildub joined #gluster
04:55 jiffin joined #gluster
05:05 yazhini joined #gluster
05:06 cliluw joined #gluster
05:07 kotreshhr joined #gluster
05:10 haomaiwa_ joined #gluster
05:13 ndevos mjrosenb: http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
05:13 glusterbot Title: What is this new .glusterfs directory in 3.3? (at joejulian.name)
05:14 * ndevos knows glusterbot has a keyword for that link, but just doesnt remember what it was
05:14 Bhaskarakiran joined #gluster
05:14 vimal joined #gluster
05:14 hagarth joined #gluster
05:16 meghanam joined #gluster
05:16 mjrosenb yeah, I have that problem with bots all the time.
05:17 mjrosenb so, it looks like nfs isn't enabled for my volume, but it gives a somewhat mysterious error when I try to turn it off.
05:17 aravindavk joined #gluster
05:18 overclk joined #gluster
05:19 mjrosenb ndevos: for a dht system, it sounds like it won't use the .glusterfs directory at all.  Is this correct?
05:19 skoduri joined #gluster
05:20 Philambdo joined #gluster
05:21 atalur joined #gluster
05:21 mjrosenb ok, I am /so/ confused
05:22 mjrosenb this is the second time I had some insurmountable issue with gluster
05:22 mjrosenb then came back the next day
05:22 mjrosenb and it was gone.
05:26 ramky joined #gluster
05:27 atalur joined #gluster
05:27 deniszh joined #gluster
05:28 raghu joined #gluster
05:38 pppp joined #gluster
05:43 poornimag joined #gluster
05:45 rjoseph joined #gluster
05:49 Manikandan joined #gluster
05:49 ashiq joined #gluster
05:57 harish joined #gluster
05:57 dusmant joined #gluster
05:58 jwd joined #gluster
06:04 jordie joined #gluster
06:08 ppai joined #gluster
06:09 Saravana_ joined #gluster
06:10 haomaiwang joined #gluster
06:12 kkeithley_blr @.glusterfs
06:12 glusterbot kkeithley_blr: For information about the .glusterfs directory tree, see http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
06:12 kkeithley_blr ndevos: ^^^
06:15 hagarth joined #gluster
06:19 skoduri joined #gluster
06:20 jtux joined #gluster
06:25 harish joined #gluster
06:29 barius_ joined #gluster
06:30 sakshi joined #gluster
06:32 barius_ hi everyone, I'm quite new to gluster and have a few questions about it
06:33 kkeithley1 joined #gluster
06:34 barius_ I have set up a 50-node gluster cluster, and each time I peer probe a new server into the pool, it might take a long time, e.g. 10 or more minutes, for glusterd to complete writing /var/lib/glusterd/peers
06:35 ramky joined #gluster
06:36 barius_ And when I do a lot of peer probes in a row, the command line tool will time out
06:37 barius_ Does my relatively large node count have anything to do with this problem? Or did I missed something?
06:38 poornimag joined #gluster
06:38 yazhini joined #gluster
06:39 maveric_amitc_ joined #gluster
06:39 ndevos atinm: maybe a question for you? ^
06:40 atinm barius_, your understanding is correct
06:41 arcolife joined #gluster
06:41 atinm barius_, for a peer probe in a n nodes cluster there is an overhead of having n X n exchanges, given n is large it might take more than 120 secs which is the default cli time out and command will time out
06:42 atinm barius_, this is one of the reason why we want to switch to GlusterD 2.0, refer to http://www.gluster.org/community/documentati​on/index.php/Features/thousand-node-glusterd
06:42 barius_ oops, I get it, is there any work-around for this for now? because I might add more nodes into my pool
06:46 atinm barius_, actually the nodes will get added even though the command times out
06:46 weijin joined #gluster
06:46 atinm barius_, to prove it you could run a gluster peer status and see whether the newly added node in part of the list or not
06:52 hagarth hmm, maybe fsync()s in glusterd store are being expensive for barius
06:52 hagarth @channelstats
06:52 glusterbot hagarth: On #gluster there have been 399391 messages, containing 15248746 characters, 2508358 words, 9018 smileys, and 1289 frowns; 1790 of those messages were ACTIONs.  There have been 182908 joins, 4569 parts, 178690 quits, 29 kicks, 2299 mode changes, and 8 topic changes.  There are currently 284 users and the channel has peaked at 287 users.
06:53 yazhini joined #gluster
06:53 deniszh joined #gluster
06:54 hchiramm itisravi++ thanks !
06:54 glusterbot hchiramm: itisravi's karma is now 4
06:55 dlambrig joined #gluster
06:56 nangthang joined #gluster
07:09 Zhang joined #gluster
07:10 haomaiwa_ joined #gluster
07:10 baojg joined #gluster
07:11 weijin joined #gluster
07:18 atinm joined #gluster
07:19 kkeithley1 joined #gluster
07:21 eMBee i read about a recommendation that local system and gluster disks should be separate because local disk operations could slow gluster down. but with a set of 5x6TB disks that is not possible. is that really a concern?
07:21 Xtreme dont bother
07:21 Xtreme this chat is dead
07:22 hagarth eMBee: It is a general recommendation. would advise running your workload and determine the behavior/impact.
07:22 hagarth Xtreme: hmm, I don't think so :)
07:23 Xtreme hagarth, well, i tried asking help many times. and i think you are the first one to reply me EVER
07:23 sac joined #gluster
07:24 hagarth Xtreme: I would suggest asking for help in a slightly more polite tone. I was sifting through yesterday's chat logs and your questions/comments did not seem very positive.
07:24 Zhang joined #gluster
07:24 eMBee well, the servers is dedicated for gluster, so it's only glusters own workload
07:25 hagarth eMBee: what kind of applications would be accessing storage?
07:25 Xtreme hagarth, well, coz yesterday was 5th or 6th time i guess
07:25 hagarth eMBee: apart from gluster's logging and state maintenance, there wouldn't be other activities on the system disks.
07:26 Xtreme anyways, i am shifting to rsync
07:26 eMBee hmm, more dificult to describe. the data is lots of (smaller) pictures
07:26 hagarth Xtreme: sure, happy to help in case you want to project your problem again.
07:26 eMBee pictures are uploaded and downloaded, but files are rarely (if ever) changed
07:26 Xtreme :) okay.
07:27 meghanam joined #gluster
07:27 hagarth eMBee: can /var be a different partition at all?
07:27 Xtreme i have two servers. S1 and S2. S1 is master, S2 is slave.
07:28 Xtreme issue 1: whenever i reboot S1 and S2, my /storage-pool gets empty. i dont know why.
07:28 eMBee sure, i can have as many partitions as i like, but they will all be on the same disk. because i can only create a raid accross the whole set of disks
07:28 Xtreme i have to stop volume1
07:28 Xtreme start volume1 (forcefully)
07:28 Xtreme and then remount. then it starts working
07:28 Xtreme else it just doesnt work. it remains empty.
07:29 eMBee right now i have one partition for / one for swap and one for gluster
07:29 hagarth Xtreme: hang on, are you using geo-replication between S1 and S2 ?
07:29 eMBee on each disk. / is a raid 1 (so i got 5 copies)
07:30 eMBee can't really do it any other way, lest i get disks with different amounts of free space
07:30 hagarth eMBee: different partitions can help as you wouldn't want your data partitions to be affected by gluster or system logging in /var.
07:31 eMBee ok, but you are talking about partitions now, that is no problem. i was wondering about having a /var partition on the same disk as the gluster partition
07:32 Xtreme hagarth, armm, what is geo replication? one sec
07:32 hagarth eMBee: that should be fine
07:32 hagarth Xtreme: are you using a replica volume?
07:32 eMBee ok then, thanks
07:32 eMBee a completely different question: raid or no raid for gluster?
07:33 Xtreme hagarth, yup.
07:33 Xtreme i followed this tutorial https://www.digitalocean.com/community/tutorials​/how-to-use-haproxy-as-a-layer-4-load-balancer-f​or-wordpress-application-servers-on-ubuntu-14-04
07:33 glusterbot Title: How To Use HAProxy As A Layer 4 Load Balancer for WordPress Application Servers on Ubuntu 14.04 | DigitalOcean (at www.digitalocean.com)
07:33 eMBee if i use raid then i get one brick per server, otherwise i get 5 bricks
07:33 eMBee with one brick per server i have to add two servers each time i need more space
07:34 hagarth Xtreme: is /storage-pool your mount point?
07:35 eMBee we are talking about a small cluster here, right now our diskspace needs are less than 10TB, one server would be big enough. we want gluster mainly for the added redundancy
07:36 eMBee and for the ease of adding more space by adding servers
07:36 eMBee space use only grows a few TB per year
07:36 hagarth eMBee: raid would provide protection against disk failures. Normally it is recommended to have 2-way replication with raid and 3-way replication without raid.
07:37 eMBee yes, i was planning for 3-way replication
07:38 hagarth eMBee: then raid is not something that would be essential for you
07:42 fsimonce joined #gluster
07:44 hagarth bbiab
07:46 eMBee with two servers, i'd have 2 bricks on one server, and one brick on the second servers. then when i add a 3rd server, i could rebalance to have one brick on each. etc...
07:46 jcastill1 joined #gluster
07:46 eMBee with raid i'd have to add 2 servers each time i need more space
07:46 _Bryan_ joined #gluster
07:46 Xtreme hagarth, yup
07:47 eMBee the raid method is a lot easier though, as i would not have to reconfigure existing servers, only add new ones
07:51 jcastillo joined #gluster
07:55 poornimag joined #gluster
07:56 Slashman joined #gluster
08:05 skoduri joined #gluster
08:06 xiu joined #gluster
08:07 barius2333 joined #gluster
08:08 barius2333 atinm, many thanks to your answer. I had problems with my internet so sorry for late reply.
08:09 barius2333 now I use subprocesses to do peer probing in parallel, which seems to partially solve the time out problem.
08:10 barius2333 all I need to do is to happily use the current glusterd version and wait as happily for the mighty 2.0 version :P
08:10 haomaiwa_ joined #gluster
08:19 ctria joined #gluster
08:20 atinm barius2333, I am glad that you got to it working
08:21 RedW joined #gluster
08:22 barius2333 Yes, I'm impressed by the throughput that GlusterFS offers, thanks again.
08:27 LebedevRI joined #gluster
08:28 kshlm joined #gluster
08:30 hagarth Xtreme: how does your volume configuration look like?
08:31 RameshN_ joined #gluster
08:31 Alex-31 joined #gluster
08:31 karnan joined #gluster
08:43 Trefex joined #gluster
08:47 nishanth joined #gluster
08:47 poornimag joined #gluster
08:48 dusmant joined #gluster
08:55 meghanam joined #gluster
09:08 poornimag joined #gluster
09:10 haomaiwa_ joined #gluster
09:12 Zhang joined #gluster
09:15 ashiq joined #gluster
09:15 dusmant joined #gluster
09:16 atinm joined #gluster
09:18 spalai joined #gluster
09:18 spalai left #gluster
09:19 Zhang joined #gluster
09:26 scubacuda_ joined #gluster
09:26 Peppaq joined #gluster
09:28 LebedevRI joined #gluster
09:29 ctria joined #gluster
09:29 frankS2 joined #gluster
09:30 badone__ joined #gluster
09:33 dusmant joined #gluster
09:37 kanagaraj joined #gluster
09:39 Manikandan joined #gluster
09:43 DaMs-_- joined #gluster
09:44 Manikandan_ joined #gluster
09:44 R0ok_ joined #gluster
09:47 kdhananjay joined #gluster
09:50 gem joined #gluster
09:55 pppp joined #gluster
09:56 atinm joined #gluster
09:57 pppp joined #gluster
09:58 ramky joined #gluster
09:59 kkeithley1 joined #gluster
10:01 kkeithley1 semiosis: > kkeithley: sounds good to me, i would just advise against merging it into the glusterfs source (which had been done in the past)
10:02 kkeithley_blr semiosis: > let me know how to proceed
10:02 kkeithley_blr semiosis: well, to clarify, in the past there were packaging  files in the source tree for various distros, but that was too hard to  work with.
10:02 kkeithley_blr semiosis: > so debian ignored them and did their own, as did I
10:03 gem joined #gluster
10:03 kkeithley_blr No, we're not doing that. I.e. not merging it into glusterfs source. It's a separate repo: glusterfs-git.  We were hoping to just 'move' your repo to glusterfs to make it easier for us to add contributors. We went ahead and created our own repo because I did not see your response.
10:03 Manikandan_ joined #gluster
10:03 kkeithley_blr semiosis: ^^^
10:04 kkeithley_blr semiosis: it's a separate repo: glusterfs-debian
10:05 kkeithley_blr semiosis: our volunteer package builders will use that repo from now on.
10:07 vmallika joined #gluster
10:09 poornimag joined #gluster
10:10 haomaiwa_ joined #gluster
10:11 hchiramm joined #gluster
10:12 kkeithley_blr semiosis: and we will continue to use the launchpad ~gluster PPA
10:16 Zhang joined #gluster
10:40 spalai joined #gluster
10:45 atalur joined #gluster
11:01 kotreshhr left #gluster
11:01 poornimag joined #gluster
11:02 Bhaskarakiran joined #gluster
11:14 ndarshan joined #gluster
11:21 mpietersen joined #gluster
11:23 gem joined #gluster
11:23 firemanxbr joined #gluster
11:23 EinstCrazy joined #gluster
11:27 nishanth joined #gluster
11:32 SeerKan` joined #gluster
11:32 SeerKan` Hi guys, If I mount a volume from server1 with server2 as backup with the fuse mount, I understand that once server1 is down it will use server2 automatically. But what happens when server1 is back ? will it start automatically use server1 even if it doesn't have the latest data or keep using server2 until it goes down and then go back to server1 ?
11:34 EinstCrazy you mean you mount as a option backup-server add in the command?
11:35 SeerKan` yes, from what I understand when you moun using the fuse gluster mount, you can mount "server1" and add "server2" as a backup option to fallback if server1 is not responding anymore
11:37 SeerKan` mount -t glusterfs -obackup-volfile-servers=<server2>: <server3>:...:<serverN> <server1>:/<volname> <mount_point>
11:37 overclk joined #gluster
11:37 EinstCrazy well,this option is only useful for manager, I mean use to notify for the configure change
11:37 SeerKan` it doesn't fallback automatically ?
11:38 jcastill1 joined #gluster
11:39 Bonaparte joined #gluster
11:40 EinstCrazy well, I don't notice this, but the data doesn't benefit from this option
11:40 Bonaparte How does gluster prioritise auth.allow and auth.reject volume options?
11:43 jcastillo joined #gluster
11:43 yazhini joined #gluster
11:43 yazhini joined #gluster
11:45 dastar joined #gluster
11:48 meghanam REMINDER: Weekly gluster community meeting will start in ~15 minutes
11:48 jrm16020 joined #gluster
11:51 chirino joined #gluster
11:51 RedW joined #gluster
11:55 raghu joined #gluster
11:56 meghanam_ joined #gluster
12:06 purpleidea joined #gluster
12:08 ira joined #gluster
12:08 Zhang joined #gluster
12:08 arcolife joined #gluster
12:10 EinstCrazy joined #gluster
12:15 hagarth joined #gluster
12:18 jtux joined #gluster
12:19 pdrakeweb joined #gluster
12:19 DV joined #gluster
12:23 unclemarc joined #gluster
12:24 ws2k3_ joined #gluster
12:34 jcastill1 joined #gluster
12:36 weijin joined #gluster
12:37 harish_ joined #gluster
12:40 rwheeler joined #gluster
12:43 jcastillo joined #gluster
12:45 klaxa|work joined #gluster
12:49 EinstCrazy joined #gluster
12:56 overclk joined #gluster
12:58 aravindavk joined #gluster
13:01 jcastillo joined #gluster
13:04 spalai left #gluster
13:08 patryck meghanam_: where is this WGZCM you are referring to?
13:09 DaMs-_- Hello there
13:10 jiffin patryck: meeting usually happens on #gluster-meeting at freenode on wednesday at 5.30pm IST
13:11 patryck ack, thx
13:11 hgowtham joined #gluster
13:12 DaMs-_- since a server crash I've a brick that looks connected from both side -- I can see both brick from both side. But when requesting a volume heal Vol info it ends by "Status: Transport endpoint is not connected"
13:13 DaMs-_- and of course number of entries is growing .... otherwise is not interesting
13:13 DaMs-_- 1 brick replicate
13:16 ndevos patryck: you just missed it, maybe you can join next week :)
13:19 Trefex joined #gluster
13:19 cholcombe joined #gluster
13:22 julim joined #gluster
13:23 Alex-31 joined #gluster
13:24 Zhang joined #gluster
13:25 RameshN_ joined #gluster
13:29 B21956 joined #gluster
13:31 spcmastertim joined #gluster
13:31 overclk joined #gluster
13:36 DV joined #gluster
13:41 dgandhi joined #gluster
13:43 shyam joined #gluster
13:43 harold joined #gluster
13:44 patryck ndevos: CET already told me, thx ;)
13:45 mpietersen joined #gluster
13:52 rwheeler joined #gluster
13:53 mckaymatt joined #gluster
14:04 shyam joined #gluster
14:07 spcmastertim joined #gluster
14:10 Alex-31 joined #gluster
14:13 mpietersen joined #gluster
14:22 haomaiwa_ joined #gluster
14:25 B21956 joined #gluster
14:26 overclk joined #gluster
14:27 DV_ joined #gluster
14:31 DV joined #gluster
14:32 Twistedgrim joined #gluster
14:32 mpietersen joined #gluster
14:33 jrm16020 joined #gluster
14:33 jrm16020 joined #gluster
14:34 mckaymatt joined #gluster
14:34 mpieters_ joined #gluster
14:35 Twistedgrim1 joined #gluster
14:35 mckaymatt joined #gluster
14:38 stopbyte running gluster 3.6.5 on centos 6.6 in a two node distributed-replicate cluster and we're seeing on the client side some files not appearing in directory listings, while other files show up twice in the output (most files are fine)
14:39 stopbyte we've tried clearing all the extended attributes, as well as just wiping out everything (including .glusterfs/) on the bricks and starting clean, but the same issue occurs
14:39 jobewan joined #gluster
14:41 stopbyte the files are copied from a local temp dir on the client into the gluster volume mountpoint - right after that copy completes, 'ls' will show most of the files, but not all of them - then, is an 'rm *' is done, all the visible files are deleted, but a bunch of previously-invisible ones appear, and in that set of files some are displaying twice
14:41 stopbyte fwiw, the gluster cluster has 7 volumes, but we have only seen this behavior on one of the volumes
14:41 stopbyte the bricks are all ext4
14:42 stopbyte any thoughts on things to look for in the logs, or additional troubleshooting steps we can take? the gluster logs on both client and server side aren't showing any warnings or errors
14:42 hagarth stopbyte: do bricks from other volumes run on ext4 too?
14:43 stopbyte yes, all 7 volumes are 2x2 distributed replicate with ext4 on the bricks
14:46 Zhang joined #gluster
14:48 hagarth stopbyte: is there a pattern for the missing files on the mount point? are they seen on bricks?
14:48 Zhang joined #gluster
14:49 Zhang joined #gluster
14:50 wushudoin joined #gluster
14:50 stopbyte when looking directly at the bricks, everything looks correct - the files are all there, and they are distributed and replicated as expected
14:51 stopbyte as for a pattern - there's not one i can discern
14:51 stopbyte however, it's always the same files that end up being invisible and the same ones that are duplicated
14:51 stopbyte even after we delete everything and copy the files again to a fresh volume
14:52 stopbyte the copy, btw, is just a straight up 'cp' - from a local device to the gluster mount on the client
14:53 mpietersen joined #gluster
14:53 hagarth shyam: any thoughts on this one? ^^
14:54 hagarth stopbyte: was rebalance done on this volume at any point in time?
14:54 DV joined #gluster
14:55 mpietersen joined #gluster
14:55 wushudoin joined #gluster
14:55 stopbyte we did do a rebalance/fix-layout on the volume, but that did not clear up the issue
14:58 topshare joined #gluster
14:58 topshare joined #gluster
15:01 topshare joined #gluster
15:03 topshare joined #gluster
15:04 aravindavk joined #gluster
15:04 topshare joined #gluster
15:06 shyam stopbyte: When you create a fresh volume and do this cp it still turns up with the problem, right? When you state fresh volume, you do clear out the brick path of any remaining files, right (rm -rf /path/to/brick/directory)? (trying to eliminate rebalance/fix-layout with this quesiton)
15:07 Zhang joined #gluster
15:10 natarej joined #gluster
15:10 _maserati joined #gluster
15:10 haomaiwa_ joined #gluster
15:15 zhangjn joined #gluster
15:16 zhangjn joined #gluster
15:17 Alex-31 joined #gluster
15:19 zhangjinnan joined #gluster
15:20 zhangjn joined #gluster
15:21 mjrosenb [2015-08-26 15:05:42.050130] E [fuse-bridge.c:2065:fuse_open_resume] 0-glusterfs-fuse: 529: OPEN 5ad1819d-caff-4d27-9199-dca9599ae70a resolution failed
15:21 dlambrig joined #gluster
15:22 mjrosenb is that hex string a file that is in the .glusterfs directory?
15:22 zhangjn joined #gluster
15:23 zhangjn joined #gluster
15:23 stopbyte shyam: yes, when we attempted to clear the volume, we did an rm -rf on everything inside the brick directory, including the .glusterfs/ subdir
15:24 stopbyte then we recreated the subdirectories the app needs, from the client side via gluster, and attempted to copy the files again to see if the issue persisted
15:25 mjrosenb [2015-08-26 15:04:30.362810] W [fuse-bridge.c:2159:fuse_readv_cbk] 0-glusterfs-fuse: 348: READ => -1 (Operation not permitted)
15:25 shyam mjrosenb: yes, that is the GFID of the file, and should be in the .glusterfs directory on the brick
15:26 zhangjn joined #gluster
15:26 shyam stopbyte: How were the directories created? From a single client? (you already state that it was created using a gluster mount, so that is good) or from multiple clients?
15:26 mjrosenb ok, so .glusterfs is actually used for dht volumes, not just replicated volumes?
15:26 zhangjn joined #gluster
15:26 shyam mjrosenb: yes
15:27 mjrosenb blast.
15:27 mjrosenb this has not always been the case
15:27 stopbyte there is actually only one client system that connects to this particular volume, so the mkdirs were run from there only
15:27 shyam mjrosenb: Well it is used in the backend by the POSIX xlator to create the hard/soft links inside .glusterfs, so is independent of DHT or AFR
15:27 mjrosenb perhaps I can not upgrade to 3.7.3
15:27 zhangjn joined #gluster
15:27 mjrosenb oh, it can use soft links?
15:28 zhangjn joined #gluster
15:29 zhangjn joined #gluster
15:30 shyam stopbyte: ok, could you fpaste the output of "getfattr -d -e hex -m . <object>" for the following objects from all the bricks, from a good file object (i.e one that is seen on the client), and its parent directory object, from a bad file object (i.e one that is on the brick, but not seen from the mount) and its parent directory object. To see if there are any layout issues here that are causing a problem. I would probably need the log file
15:30 shyam stopbyte: better yet send a mail with all the details :)
15:31 shyam mjrosenb: It contains hard links to files and soft links to directories, it cannot just be replaced using a soft link, also I am unsure what you are attempting to do. This is an internal gluster directory maintained by gluster brick processes, so please tread with caution.
15:32 anil joined #gluster
15:36 plarsen joined #gluster
15:39 stopbyte shyam: will do - i'll collect that info shortly and send it to the mailing list
15:40 mjrosenb shyam: so, my brick is using zfs, and I set it up before 3.2, and used submounts for many directories, mostly to ease administration
15:41 mjrosenb this gave me some headaches when I upgraded to 3.3, but it more or less worked
15:41 mjrosenb just lots of hardlinks failed to be created
15:41 mjrosenb now it looks like it is actively not working.
15:42 mjrosenb and when I said replace hardlinks with softlinks, I meant editing the source to use soft links rather than hard links, as a configurable option.
15:48 d-fence joined #gluster
15:54 natarej_ joined #gluster
15:58 Vitaliy|3 joined #gluster
15:59 Vitaliy|3 Hi, need help. Have a 2x2 setup with 4 bricks and trying to remove two briks. Operation fails. Looking at the logs I see rebalance crashes. This is OpenSuse 13.2. Gluster 3.7.3
15:59 Vitaliy|3 Any pointers?
16:12 Vitaliy|2 joined #gluster
16:14 bennyturns joined #gluster
16:19 Rapture joined #gluster
16:20 overclk joined #gluster
16:25 Vitaliy|2 anyone? At least how do I do it manually? Have to change IPs of those 2 bricks.
16:27 shyam mjrosenb: a lot of operations on the brick process work against the GFID 'hard' link for the file. Say the setattr RPC, so this impacts the named file as well (as it is a hard link), to change this relationship would mean, following the softlink in code given a GFID and then operating on the named file. This is just one aspect of it, there are possibly more that I am missing. So if you are thinking of changing the code, I would suggest get
16:42 kshlm joined #gluster
16:45 squizzi joined #gluster
16:49 tg2 Has anybody run into these errors on their storage server brick logs? E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
16:49 tg2 symptom is that a client can't connect to the brick so it shows as if the brick is not in the volume
16:49 tg2 it is accompanied by (after): E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.​0(_gf_log_callingfn+0x186)[0x7f714537fda6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.​6.2/xlator/protocol/server.so(get_fram​e_from_request+0x2e0)[0x7f713b15d9a0] (--> /usr/lib/x86_64-linux-gnu/glusterfs/​3.6.2/xlator/protocol/server.so(serv​er3_3_lookup+0x8b)[0x7f713b17617b] (--> /usr/lib/x86_64-linux-gnu/libgfrp​c.so.0(rpcsvc_handle_rpc_call+0x2
16:49 tg2 99)[0x7f7145151979] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.​0(rpcsvc_notify+0x16b)[0x7f7145151fbb] ))))) 0-server: invalid argument: client
16:49 glusterbot tg2: ('s karma is now -102
16:49 glusterbot tg2: ('s karma is now -103
16:49 glusterbot tg2: ('s karma is now -104
16:49 glusterbot tg2: ('s karma is now -105
16:49 tg2 :(
16:49 glusterbot tg2: ('s karma is now -106
16:54 Vitaliy|2 joined #gluster
16:54 str8edge joined #gluster
16:55 jrm16020 joined #gluster
16:55 str8edge Hi. How do I list all of the options for a volume. gluster volume info <volname> only shows the reconfigured options.
16:57 _maserati_ joined #gluster
16:58 tg2 also spotted in etc glusterd log -> readv on /var/run/<brickid>.socket failed (invalid argument)
17:00 Vitaliy|2 Any help with trying to remove / reballance / heal  bricks?
17:00 Vitaliy|2 Nothins seems to be working
17:01 Vitaliy|2 Rebalance crashes. Heal - says see log, nothing informative in the log: "Commit failed on host"...
17:05 anoopcs str8edge, gluster volume get <VOLNAME> all
17:16 PatNarciso joined #gluster
17:16 PatNarciso j #gluster
17:17 PatNarciso Happy Wednesday #gluster.
17:19 jwd joined #gluster
17:20 PatNarciso whats going on with readthedocs?  plenty of 404's; and missing assets on load... hmm.
17:34 mjrosenb shyam: your message cut off after 'I would suffest gett'
17:35 amye joined #gluster
17:42 shortdudey123 I am doing some Gluster testing.  I have a 3x2 distributed replicated cluster.  When i firewall off a host to see what happens when a host just vanishes, it takes 37 seconds before the Gluster mount becomes accessable again.  This due to the network timeout setting?  How can I remove this large timeout?
17:51 rwheeler joined #gluster
17:51 Vitaliy|2 Anyone? How can I troubleshoot rebalance? Or run it manually? Or whatever? I must move bricks and they have 1/2 of the volume data!
17:54 htrmeira joined #gluster
17:55 shyam mjrosenb: "... I would suggest getting some pointers by posting to the devel list, so that you do not miss anything."
17:56 mjrosenb ahh.
17:57 mjrosenb these operations didn't always go through the .gluster directory, do you have any idea when they started?
17:57 mjrosenb maybe I can downgrade to the laster version before this started.
17:57 shyam mjrosenb: No, I am unsure when this started
17:58 mjrosenb I'll go spelunking when I get home
17:58 mjrosenb JoeJulian: you happen to know?
18:08 paratai joined #gluster
18:13 deniszh joined #gluster
18:22 Twistedgrim joined #gluster
18:23 spcmastertim joined #gluster
18:24 Twistedgrim1 joined #gluster
18:32 Gill joined #gluster
18:47 diegows joined #gluster
18:54 aravindavk joined #gluster
18:55 purpleidea joined #gluster
19:01 a2 joined #gluster
19:30 B21956 joined #gluster
19:43 amye joined #gluster
20:04 woakes07004 joined #gluster
20:27 RedW joined #gluster
20:29 _maserati joined #gluster
20:30 plarsen joined #gluster
20:32 cholcombe joined #gluster
20:39 autoditac_ joined #gluster
20:40 jrm16020 joined #gluster
20:50 woakes070048 joined #gluster
20:52 royadav joined #gluster
20:52 Trefex joined #gluster
20:53 royadav @all: I am unable to debug a gluster issue, it complains about volume add-brick: failed: operation failed..
20:54 royadav :all: gluster.log shows that readv failed (Not data available)
20:55 royadav @all: could anyone please tell me as how to debug this issue
20:56 techsenshi joined #gluster
20:59 RedW joined #gluster
20:59 mrrrgn joined #gluster
20:59 royadav @RedW: hi
21:00 RedW hi
21:00 glusterbot RedW: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
21:01 royadav @redW: I am unable to debug a gluster issue, it complains about volume add-brick: failed operation failed..
21:01 royadav @RedW: gluster.log shows that readv failed (No data available)
21:01 royadav @RedW: I am not able to debug this issue
21:02 RedW hmm
21:03 RedW too bad I'm not very experienced at gluster
21:04 royadav @RedW: I see, do you know anyone who may be? I have joined this IRC channel for the first time
21:09 RedW royadav, maybe check permissions of bricks locations?
21:19 badone__ joined #gluster
21:40 rideh joined #gluster
21:56 cholcombe joined #gluster
22:02 CyrilPeponnet hey guys, running 3.6.5 I notice this gluster vol heal myvol info heal-failed Command not supported.
22:02 CyrilPeponnet but the help still show me volume heal <VOLNAME> [{full | statistics {heal-count {replica <hostname:brickname>}} |info {healed | heal-failed | split-brain}}]
22:03 CyrilPeponnet actually only split-brain is working
22:03 CyrilPeponnet Is there something broken or the help is not up to date ?
22:04 dijuremo2 joined #gluster
22:06 dijuremo2 Guys, since updgrading gluster from 3.6.4 to 3.7.3 samba is now creating several core dumps... anybody seen this? Is there a fix for it?
22:08 dijuremo2 core.10018.1440573555.dump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/smbd'
22:10 Vitaliy|2 @dijuremo2 did you reboot? I've noticed just restarting glusterfsd isn't enough.
22:10 Vitaliy|2 just a guess
22:11 dijuremo2 Yes, I had to reboot both servers
22:11 cholcombe joined #gluster
22:11 dijuremo2 I tried every single way of adding server.allow.insecure and the rpc-auth-allow-insecure options to get 3.6.4 to work with 3.7.3 to *NO* aval....
22:12 dijuremo2 So I had to stop all servers, updated, also applied some kernel updates and rebooted all servers
22:13 Vitaliy|2 yeah same here, had to upgrade all clients to 3.7.3 too.
22:13 _maserati joined #gluster
22:14 Vitaliy|2 But not using samba, can't tell much.
22:14 Vitaliy|2 More having issues with trying to remove bricks - doesn't work, rebalance process crashes.
22:15 dijuremo2 I am also having problems with small file performance... samba roaming profiles are extremely slow... but that was present before in 3.6.4 and still in 3.7.3
22:16 plarsen joined #gluster
22:16 Vitaliy|2 yup, "performance.stat-prefetch: on" helps a bit but only on repeated access
22:17 dijuremo2 So on the brick itself:
22:17 dijuremo2 time ( find .winprofile.V2 -type f | wc )
22:17 dijuremo2 2142    2893  304869
22:17 dijuremo2 real    0m4.326s
22:17 dijuremo2 user    0m0.016s
22:17 dijuremo2 sys     0m0.046s
22:18 dijuremo2 On fuse mounted gluster:
22:18 dijuremo2 time ( find .winprofile.V2 -type f | wc )
22:18 dijuremo2 2142    2893  304869
22:18 dijuremo2 real    0m15.347s
22:18 dijuremo2 user    0m0.022s
22:18 dijuremo2 sys     0m0.084s
22:19 Vitaliy|2 ouch
22:19 dijuremo2 These are the only options I have:
22:19 dijuremo2 performance.io-thread-count: 32
22:19 dijuremo2 nfs.disable: on
22:19 dijuremo2 cluster.server-quorum-type: server
22:19 dijuremo2 performance.cache-size: 256MB
22:19 dijuremo2 server.allow-insecure: on
22:19 dijuremo2 cluster.server-quorum-ratio: 51%
22:19 dijuremo2 And for users with a lot more files (stupid IE cookies) it takes over a minute
22:21 dijuremo2 Actually, that result of 15ish seconds was very fast only because I had run the same thing on the bricks before...
22:21 dijuremo2 So some caching is in play
22:28 dijuremo2 time ( find .winprofile.V2 -type f | wc )
22:28 dijuremo2 12971   25290 1273724
22:28 dijuremo2 real    1m21.674s
22:28 dijuremo2 user    0m0.120s
22:28 dijuremo2 sys     0m0.285s
23:02 cholcombe joined #gluster
23:09 gildub joined #gluster
23:18 mjrosenb joined #gluster
23:22 mjrosenb so, does a definite list of things that .glusterfs is used for exist?
23:23 diegows joined #gluster
23:25 shortdudey123 I am doing some Gluster testing.  I have a 3x2 distributed replicated cluster.  When i firewall off a host to see what happens when a host just vanishes, it takes 37 seconds before the Gluster mount becomes accessable again.  This due to the network timeout setting?  How can I remove this large timeout?
23:45 mjrosenb shortdudey123: what do the logs say while this is happening?
23:48 shortdudey123 mjrosenb: [2015-08-26 17:43:08.212198] E [socket.c:2276:socket_connect_finish] 0-chef-client-1: connection to 172.23.28.92:24007 failed (Connection timed out)
23:49 shortdudey123 [2015-08-26 17:47:28.068004] I [client-handshake.c:1413:sele​ct_server_supported_programs] 0-chef-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
23:49 shortdudey123 [2015-08-26 17:47:28.084182] I [client-handshake.c:1200:client_setvolume_cbk] 0-chef-client-1: Connected to chef-client-1, attached to remote volume '/mnt/glusterfs/xvdf1/brick'.
23:49 shortdudey123 [2015-08-26 17:47:28.084198] I [client-handshake.c:1210:client_setvolume_cbk] 0-chef-client-1: Server and Client lk-version numbers are not same, reopening the fds
23:49 shortdudey123 [2015-08-26 17:47:28.100302] I [client-handshake.c:188:client_set_lk_version_cbk] 0-chef-client-1: Server lk version = 1
23:49 glusterbot shortdudey123: This is normal behavior and can safely be ignored.
23:51 shortdudey123 mjrosenb: more current logs http://pastebin.com/pf4xcX6s
23:51 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
23:55 zhangjn joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary