Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-03-12

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:10 doekia_ joined #gluster
00:13 vpshastry joined #gluster
00:14 kam270 joined #gluster
00:15 philv76 joined #gluster
00:24 kam270 joined #gluster
00:27 elyograg if I kill the self-heal deamon, will it auto-restart?
00:27 elyograg it's chewing up 2.6GB of RAM on a couple of my brick servers (not for the same volume we've been dealing with, JoeJulian )
00:28 kkeithley1 joined #gluster
00:29 elyograg kkeithley1: are you actually here or just autojoining?
00:31 sroy joined #gluster
00:31 doekia_ anyone knows about a good munin plugins for glusterfs-server 3.4.2?
00:32 vpshastry joined #gluster
00:33 kam270 joined #gluster
00:34 elyograg didn't even know what munin was.  i'm sure you can find the same things I can with google, though.
00:35 tokik joined #gluster
00:38 JoeJulian elyograg: no, it doesn't. To restart it you have to restart glusterd.
00:38 elyograg ok.
00:39 elyograg restarting glusterd should be a completely safe thing to do, I would expect.  as long as I don't stop glusterfsd or kill any of those processes.
00:41 elyograg i can probably kill the nfs server, too, as this is not the machine that NFS clients talk to.  they actually can't because of network isolation.
00:42 elyograg the nfs process is using more memory than I would expect as well, 194mb.
00:42 elyograg not enough to really worry me, so perhaps I'll leave that alone.
00:43 kkeithley_ elyograg: yes and yes
00:43 kam270 joined #gluster
00:54 dcmbrown joined #gluster
00:55 masterzen joined #gluster
00:55 sticky_afk joined #gluster
00:55 HeisSpiter joined #gluster
00:56 sac`away joined #gluster
00:57 Ark joined #gluster
00:58 fyxim_ joined #gluster
00:58 sulky_ joined #gluster
00:58 stickyboy joined #gluster
00:58 yosafbridge joined #gluster
00:58 Joe630 joined #gluster
00:58 larsks joined #gluster
00:58 johnmwilliams__ joined #gluster
00:59 Ark Hello all, I have a distributed replicated volume on 4 bricks of identical size. The first pair of  bricks are getting more data then the other set. These have been working flawlessly over the last 1.5 months and fix layout was run long ago. I did a rebalance and it did not move any of the data. Ideas?
00:59 JonnyNomad joined #gluster
00:59 delhage joined #gluster
00:59 edong23 joined #gluster
00:59 neoice joined #gluster
01:00 slappers joined #gluster
01:00 Amanda joined #gluster
01:00 sputnik13 joined #gluster
01:02 elyograg Ark: are your files (or at least some of them) really large?  if they are, you've probably run into a situation where the distributed hash algorithm is not well balanced.  If the really big files have the same filename but are in different subdirectories, that's another explanation - only the end filename is used for hashing.
01:03 kostyrevaa1 joined #gluster
01:03 zerick joined #gluster
01:04 jurrien_ joined #gluster
01:04 kostyrevaa1 good day!
01:04 kostyrevaa1 could anybody help me to stop  geo-replication?
01:04 kostyrevaa1 I use
01:04 kostyrevaa1 glusterfs-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 glusterfs-fuse-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 glusterfs-geo-replication-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 glusterfs-libs-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 glusterfs-cli-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 glusterfs-server-3.4.2-1.el6.x86_64
01:04 kostyrevaa1 gluster volume geo-replication distrepl ex436-node05.atlas-2.ru::geo-slave-vol status
01:04 kostyrevaa1 NODE                 MASTER               SLAVE                                              STATUS
01:04 kostyrevaa1 --------------------------------------------------​-------------------------------------------------
01:04 kostyrevaa1 ex436-node01.sc.local distrepl             ex436-node05.atlas-2.ru::geo-slave-vol             OK
01:04 kostyrevaa1 gluster volume geo-replication distrepl ex436-node05.atlas-2.ru::geo-slave-vol stop
01:04 kostyrevaa1 geo-replication command failed
01:04 Kins joined #gluster
01:04 bennyturns joined #gluster
01:05 Peanut joined #gluster
01:05 cyberbootje joined #gluster
01:06 fsimonce joined #gluster
01:06 wrale joined #gluster
01:09 kkeithley joined #gluster
01:09 bala joined #gluster
01:10 bfoster joined #gluster
01:11 Leolo joined #gluster
01:14 dcmbrown joined #gluster
01:14 robo joined #gluster
01:19 JMWbot joined #gluster
01:19 velladecin joined #gluster
01:25 fyxim_ joined #gluster
01:33 Leolo I have a question
01:33 Leolo (obviously)
01:34 Leolo in the discussion of removing the rebalancing stuff from replace-brick, I see them talk about not using a mount point for the brick, but a directory under the mount point.  I was wondering why.
01:45 Leolo hmmm "We're going to define the actual brick in the /brick directory on that filesystem. This helps by causing the brick to fail to start if the xfs filesystem isn't mounted."
01:48 andreask joined #gluster
01:50 glusterbot New news from newglusterbugs: [Bug 1063832] add-brick command seriously breaks permissions on volume root <https://bugzilla.redhat.co​m/show_bug.cgi?id=1063832>
01:55 ira joined #gluster
02:09 Ark elyograg: I am backing up 600gb+ ibdata files for mysql, so yes really big. Thank you for your responce.
02:23 jag3773 joined #gluster
02:51 haomaiwang joined #gluster
02:52 hagarth joined #gluster
02:54 cp0k joined #gluster
02:56 bharata-rao joined #gluster
02:59 Ark joined #gluster
03:06 Leolo gluster volume create test1 replica 2 gl1:/data/glusterfs/test1/brick1/brick gl2:/data/glusterfs/test1/brick2/brick
03:06 Leolo volume create: test1: failed: /data/glusterfs/test1/brick1/brick or a prefix of it is already part of a volume
03:06 glusterbot Leolo: To clear that error, follow the instructions at http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/ or see this bug https://bugzilla.redhat.com/show_bug.cgi?id=877522
03:08 Leolo volume create: test1: failed
03:08 Leolo Now I get ^^
03:11 Leolo I see nothing useful in the logs
03:18 Leolo damnit, iptables had reset because I didn't save it before rebooting
03:20 haomaiwa_ joined #gluster
03:21 ricky-ti1 joined #gluster
03:23 Leolo clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)
03:37 haomaiwa_ joined #gluster
03:37 RameshN joined #gluster
03:40 haomai___ joined #gluster
03:42 itisravi joined #gluster
03:44 shubhendu joined #gluster
03:50 gmcwhistler joined #gluster
03:55 dcmbrown joined #gluster
03:57 zerick joined #gluster
03:57 yosafbridge joined #gluster
04:13 tokik joined #gluster
04:14 ajha joined #gluster
04:17 saurabh joined #gluster
04:19 vpshastry joined #gluster
04:19 ndarshan joined #gluster
04:24 kdhananjay joined #gluster
04:24 satheesh joined #gluster
04:32 deepakcs joined #gluster
04:34 satheesh2 joined #gluster
04:34 bazzles joined #gluster
04:36 vimal joined #gluster
04:42 nshaikh joined #gluster
04:50 bala joined #gluster
04:53 neurodrone__ joined #gluster
04:53 ravindran joined #gluster
04:58 spandit joined #gluster
05:00 ppai joined #gluster
05:01 mohankumar joined #gluster
05:03 hagarth joined #gluster
05:07 dusmant joined #gluster
05:09 cyber_si joined #gluster
05:14 shylesh joined #gluster
05:21 glusterbot New news from newglusterbugs: [Bug 955548] adding host uuids to volume status command xml output <https://bugzilla.redhat.com/show_bug.cgi?id=955548> || [Bug 1075417] Spelling mistakes and typos in the glusterfs source <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075417>
05:22 KORG joined #gluster
05:28 fidevo joined #gluster
05:28 prasanth joined #gluster
05:30 pk1 joined #gluster
05:30 itisravi joined #gluster
05:38 mohankumar joined #gluster
05:50 psharma joined #gluster
05:52 raghu joined #gluster
05:54 benjamin_____ joined #gluster
06:21 dkorzhevin joined #gluster
06:29 dkorzhevin joined #gluster
06:33 dkorzhevin joined #gluster
06:33 rahulcs joined #gluster
06:40 KORG joined #gluster
06:41 dorko joined #gluster
06:52 kostyrevaa joined #gluster
07:17 ngoswami joined #gluster
07:17 lalatenduM joined #gluster
07:21 ravindran joined #gluster
07:32 Philambdo joined #gluster
07:48 ekuric joined #gluster
07:51 Pavid7 joined #gluster
07:53 glusterbot New news from resolvedglusterbugs: [Bug 849630] client_t implementation <https://bugzilla.redhat.com/show_bug.cgi?id=849630>
07:56 rahulcs joined #gluster
08:00 eseyman joined #gluster
08:06 Pavid7 joined #gluster
08:10 benjamin_____ joined #gluster
08:12 keytab joined #gluster
08:23 muhh joined #gluster
08:26 harish joined #gluster
08:29 cjanbanan joined #gluster
08:31 X3NQ joined #gluster
08:38 ricky-ticky1 joined #gluster
08:40 muhh joined #gluster
08:43 jtux joined #gluster
08:46 neoice joined #gluster
08:48 ctria joined #gluster
08:50 rahulcs joined #gluster
08:50 RameshN joined #gluster
08:53 dusmant joined #gluster
08:54 ndarshan joined #gluster
09:04 vpshastry joined #gluster
09:15 al joined #gluster
09:16 rahulcs joined #gluster
09:20 ctria joined #gluster
09:21 andreask joined #gluster
09:27 mohankumar joined #gluster
09:30 liquidat joined #gluster
09:33 YazzY joined #gluster
09:33 mohankumar joined #gluster
09:34 ricky-ticky joined #gluster
09:34 Norky joined #gluster
09:36 lalatenduM @g4s
09:36 glusterbot lalatenduM: I do not know about 'g4s', but I do know about these similar topics: 'git', 'gmc'
09:37 lalatenduM @learn gluster-swift-docs as https://github.com/gluster/gluste​r-swift/tree/master/doc/markdown
09:37 glusterbot lalatenduM: The operation succeeded.
09:38 lalatenduM @swift
09:38 glusterbot lalatenduM: (#1) http://edwyseguru.wordpress.com/2011/08/25/unifie​d-file-and-object-storage-on-glusterfs-3-1-howto/, or (#2) See Kaleb's Quick and Dirty UFO howto at http://repos.fedorapeople.org/re​pos/kkeithle/glusterfs/HOWTO.UFO
09:38 lalatenduM @gluster
09:38 glusterbot lalatenduM: I do not know about 'gluster', but I do know about these similar topics: '.glusterfs', 'Can I use glusterfs on an i386 platform', 'What is the difference between Gluster Storage Platform (GSP) and GlusterFS?', 'What ports does glusterfs use for nfs?', 'gluster-swift-docs', 'python-gluster'
09:43 calum_ joined #gluster
09:44 rgustafs joined #gluster
09:49 prasanth joined #gluster
09:49 RameshN joined #gluster
09:49 ndarshan joined #gluster
09:50 dusmant joined #gluster
09:52 glusterbot New news from newglusterbugs: [Bug 1075488] nfs-ganesha: renaming directory into existing file fails with incorrect error. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075488>
09:58 diegows joined #gluster
10:13 tokik joined #gluster
10:21 tryggvil joined #gluster
10:22 glusterbot New news from newglusterbugs: [Bug 1075506] Dict leak in quota_lookup when quota is disabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075506>
10:27 ajha joined #gluster
10:28 ctria joined #gluster
10:40 rahulcs joined #gluster
10:52 ajha joined #gluster
10:58 harish joined #gluster
10:58 vimal joined #gluster
11:02 lpabon joined #gluster
11:06 tdasilva joined #gluster
11:11 rahulcs joined #gluster
11:15 latha joined #gluster
11:16 harish joined #gluster
11:18 rahulcs joined #gluster
11:26 mohankumar joined #gluster
11:29 vpshastry1 joined #gluster
11:34 mohankumar joined #gluster
11:35 rahulcs joined #gluster
11:41 rahulcs joined #gluster
11:41 B21956 joined #gluster
11:53 rahulcs joined #gluster
11:55 ppai joined #gluster
12:02 mohankumar joined #gluster
12:09 gmcwhistler joined #gluster
12:12 Ark joined #gluster
12:15 kdhananjay joined #gluster
12:15 rossi_ joined #gluster
12:15 rahulcs joined #gluster
12:16 dusmant joined #gluster
12:22 deepakcs joined #gluster
12:22 andreask joined #gluster
12:28 yinyin joined #gluster
12:36 neurodrone__ joined #gluster
12:38 ppai joined #gluster
12:39 rahulcs joined #gluster
12:41 rfortier1 joined #gluster
12:48 getup- joined #gluster
12:49 junaid joined #gluster
12:50 tdasilva left #gluster
12:52 pk1 joined #gluster
12:57 sroy joined #gluster
13:01 rfortier joined #gluster
13:03 rfortier joined #gluster
13:03 jmarley joined #gluster
13:06 mohankumar__ joined #gluster
13:19 FarbrorLeon joined #gluster
13:20 benjamin_____ joined #gluster
13:20 dusmant joined #gluster
13:22 shyam joined #gluster
13:23 glusterbot New news from newglusterbugs: [Bug 1075611] [FEAT] log: enhance gluster log format with message ID and standardize errno reporting <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075611>
13:27 andreask joined #gluster
13:29 jtux joined #gluster
13:29 chirino joined #gluster
13:30 japuzzo joined #gluster
13:37 theron joined #gluster
13:40 chirino joined #gluster
13:43 aravindavk joined #gluster
13:45 jobewan joined #gluster
13:49 mohankumar__ joined #gluster
13:53 rfortier1 joined #gluster
13:54 burn420 joined #gluster
13:55 kam270_ joined #gluster
13:56 itisravi joined #gluster
14:00 plarsen joined #gluster
14:01 shylesh joined #gluster
14:04 kaptk2 joined #gluster
14:04 yinyin joined #gluster
14:09 seapasulli joined #gluster
14:10 jdarcy joined #gluster
14:11 jdarcy Hey, strangers.
14:11 lalatenduM hei jdarcy :)
14:11 pk1 jdarcy: Hey Jeff ;-)
14:12 kkeithley_ @yum
14:12 glusterbot kkeithley_: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
14:15 wushudoin joined #gluster
14:18 robo joined #gluster
14:19 tdasilva joined #gluster
14:22 morsik left #gluster
14:29 rahulcs joined #gluster
14:32 mohankumar__ joined #gluster
14:36 rpowell joined #gluster
14:37 ricky-ticky1 joined #gluster
14:40 zaitcev joined #gluster
14:41 jobewan joined #gluster
14:42 Psi-Jack joined #gluster
14:43 RayS joined #gluster
14:43 hagarth joined #gluster
14:44 NeatBasis joined #gluster
14:46 robo joined #gluster
14:50 rfortier joined #gluster
14:50 lmickh joined #gluster
14:51 Psi-Jack I'm trying to figure out what exactly the glusterfs bd stuff does and how it works.
14:53 sks joined #gluster
14:53 zaitcev joined #gluster
14:54 vkoppad joined #gluster
14:55 vkoppad joined #gluster
14:55 bchilds joined #gluster
14:56 kkeithley_ Gluster community meeting in five minutes over in #gluster-meeting on freenode
14:56 hagarth joined #gluster
14:57 Psi-Jack Oh, well, then..
14:59 benjamin_____ joined #gluster
14:59 jdarcy Psi-Jack: The BD translator lets you map a logical volume into the GlusterFS namespace, so its contents can be accessed without going through a local file system.
15:00 Psi-Jack Hmm, interesting.
15:00 shubhendu joined #gluster
15:03 criticalhammer joined #gluster
15:03 calum_ joined #gluster
15:08 jobewan joined #gluster
15:09 ndk joined #gluster
15:13 ndk` joined #gluster
15:13 philv76 joined #gluster
15:15 rpowell1 joined #gluster
15:16 ndk`` joined #gluster
15:19 rwheeler_ joined #gluster
15:21 burn420 have you guys seen any issues with quota and large files?
15:22 jmarley joined #gluster
15:22 jmarley joined #gluster
15:30 ctria joined #gluster
15:33 Pavid7 joined #gluster
15:34 vpshastry joined #gluster
15:34 failshell joined #gluster
15:40 rossi_ joined #gluster
15:42 robo joined #gluster
15:42 rpowell joined #gluster
15:53 rpowell1 joined #gluster
15:53 rpowell2 joined #gluster
15:55 jmarley joined #gluster
15:55 jmarley joined #gluster
15:56 rpowell joined #gluster
15:57 bala joined #gluster
16:00 hybrid512 joined #gluster
16:00 rpowell1 joined #gluster
16:00 hybrid512 joined #gluster
16:01 philv76 joined #gluster
16:07 avati joined #gluster
16:09 calum_ joined #gluster
16:09 rfortier joined #gluster
16:10 jag3773 joined #gluster
16:18 kmai007 joined #gluster
16:19 kmai007 hey yall, what ports do i need to let the firewall admin know to open to be able to mount gluster as NFS ?
16:20 lalatenduM @ports
16:20 glusterbot lalatenduM: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
16:20 lalatenduM @ports| kmai007
16:20 lalatenduM @ports | kmai007
16:20 semiosis ~ports | kmai007
16:20 glusterbot kmai007: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
16:20 kmai007 lalatenduM: thank you
16:21 kmai007 glusterbot: thank you
16:21 glusterbot kmai007: you're welcome
16:21 andreask joined #gluster
16:21 lalatenduM kmai007, :)
16:29 Mo_ joined #gluster
16:30 rfortier joined #gluster
16:31 rfortier joined #gluster
16:33 kkeithley1 joined #gluster
16:35 rpowell joined #gluster
16:36 kmai007 can anyone provide me some technical reads on how gluster-fuse works?
16:37 kmai007 I need to prove to my management that there are trade-offs between traditional NFS and gluster-fuse
16:37 kmai007 or display to them i have a deeper understanding
16:38 kkeithley1 joined #gluster
16:39 Leolo roughly : glusterfs is faster, NFS has caching that make it better for small file reads
16:39 Leolo but that's probably not enough for you
16:39 kmai007 yeh i can't just say
16:40 kmai007 our implementation is for web content, and all our monitors have showed an increase in latency
16:40 kmai007 when switching from NFS -> gluster-FUSE
16:40 lalatenduM kmai007, fuse gives better performance with more nodes I think
16:40 kmai007 and reads is where the bulk of the workload is going to be
16:41 lalatenduM kmai007, what type of volume u r using
16:41 kmai007 distr-repl. 2 x 4 = 8
16:42 rpowell1 joined #gluster
16:42 kmai007 Volume Name: dyn_cfu
16:42 kmai007 Type: Distributed-Replicate
16:42 kmai007 Volume ID: dc440289-13cc-47fe-9dfd-59df26bc33b6
16:42 kmai007 Status: Started
16:42 kmai007 Number of Bricks: 2 x 4 = 8
16:42 kmai007 Transport-type: tcp
16:42 kmai007 Bricks:
16:42 kmai007 Brick1: omhq1140:/export/dynamic/coldfusion
16:42 kmai007 Brick2: omdx1c5d:/export/dynamic/coldfusion
16:42 kmai007 Brick3: omhq11ad:/export/dynamic/coldfusion
16:42 kmai007 Brick4: omdx1781:/export/dynamic/coldfusion
16:42 kmai007 Brick5: omhq1c56:/export/dynamic/coldfusion
16:42 purpleidea ~paste | kmai007
16:42 glusterbot kmai007: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
16:43 kmai007 gotcha
16:43 lalatenduM purpleidea, you r fast:)
16:43 purpleidea lalatenduM: lucky timing
16:43 lalatenduM kmai007, seems fine
16:43 Leolo .... omhq1140 is one hell of a hostname
16:43 gmcwhistler joined #gluster
16:44 lalatenduM kmai007, u r using a distribute-replicate volume
16:44 lalatenduM I call it disrep :)
16:44 Leolo kmai : web content is very often a lot of small files.
16:46 sputnik13 joined #gluster
16:47 Leolo and it seems to me that it would be trivial to switch to the gluster-native NFS, to see if you that improves latency
16:51 satheesh joined #gluster
16:51 pk1 left #gluster
16:54 ndevos kmai007: ,,(php) might have some tricks for you, but I think it advises to mount over nfs instead glusterfs-fuse
16:54 glusterbot kmai007: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH
16:54 glusterbot --negative-timeout=HIGH --fopen-keep-cache
17:00 burn420 joined #gluster
17:01 Pavid7 joined #gluster
17:02 seapasulli joined #gluster
17:02 RayS joined #gluster
17:06 chalkfiles joined #gluster
17:08 vpshastry joined #gluster
17:08 JoeJulian ndevos: never!
17:09 ndevos JoeJulian: no?
17:09 * ndevos isnt into the website business and does not really care
17:09 JoeJulian Anything I write is probably more biased toward redundancy and uptime guarantees.
17:10 JoeJulian ... and consistency
17:11 ndevos hehe, that has my preference too :)
17:15 FarbrorLeon joined #gluster
17:18 brosner joined #gluster
17:24 Matthaeus joined #gluster
17:29 satheesh1 joined #gluster
17:34 robo joined #gluster
17:37 Leolo there's also that apples vs orchard analogy
17:37 vpshastry joined #gluster
17:41 bchilds_ joined #gluster
17:50 elyograg We've been loading some data onto gluster.  Unfortunately it's being done with the wrong ownership and permissions.  the recursive chmod and chown will take *days* to execute because there are millions of directories and files.  Can I do chown/chmod on the backend instead. once everything's settled down and there's no read or write activity on the volume?
17:53 nullck joined #gluster
17:53 nullck_ joined #gluster
17:56 elyograg so what I'd like to do is chown -R user:group /bricks/*/volname/$DIRPATH
17:56 elyograg and a similar chmod.
17:57 elyograg hours instead of days.
17:58 rpowell joined #gluster
18:07 calum_ joined #gluster
18:11 tdasilva joined #gluster
18:16 Leolo joined #gluster
18:40 getup- joined #gluster
18:41 Elico joined #gluster
18:49 jag3773 joined #gluster
18:52 robo joined #gluster
18:54 JoeJulian elyograg: Should work fine as long as you do it on each replica. Only do it for the actual files, not dht-linkto.  If the file is accessed by a client during this process, it may (I think 'will', but I'm hedging) determine the file is split-brain if they differ at that moment.
18:54 JoeJulian Once they no longer differ, it should clear that though.
18:56 elyograg I wasn't going to touch .glusterfs.  I need directories and files to be right, though.
18:57 elyograg I wonder if it might be a good idea to stop the self-heal daemon until after it's done, though.
18:58 ndk joined #gluster
18:58 JoeJulian Sound like a good idea.
18:58 elyograg your comment about doing it on each replica may mean that I need to do one brick at a time on both servers, rather than just chmod them all with one command.
18:58 JoeJulian And the dht-linkto are in the regular directory tree. They're the same filename but mode 1000 and 0 bytes.
18:58 elyograg oh, I see.
18:59 JoeJulian So yeah, probably not chmod with one command.
18:59 elyograg I *hope* that there aren't any of those - this is a brand new volume.
18:59 JoeJulian How was it populated?
18:59 sroy joined #gluster
18:59 elyograg I'll check, though.  find is fast on xfs.
18:59 elyograg oh, there *will be some.  first bunch of rsyncs were done without --inplace.
19:00 JoeJulian What about mounting the volume from several clients and doing the change on smaller subsets but from multiple sources?
19:00 awheeler_ joined #gluster
19:00 andreask joined #gluster
19:00 robos joined #gluster
19:01 tdasilva joined #gluster
19:01 elyograg if they won't stomp on each other and just cause thrashing, that could work.  Would two or three mounts on each machine count as 'multiple clients' in that?
19:03 JoeJulian I think so. I can't imagine that the network would be the bottleneck.
19:04 elyograg me either.  it's all gig.
19:04 robo joined #gluster
19:05 JoeJulian It's all rpc calls so small packets.
19:05 JoeJulian Makes me wonder if a threaded python script might be faster than the command line utility...
19:07 elyograg any tools would be appreciated.  it's not your itch, so I don't expect you to scratch it, though. :)
19:07 elyograg I can see how the problem is interesting.
19:08 JoeJulian I think fuse is single threaded though. Might just stack up commands.
19:08 elyograg well, I have eight gluster servers.
19:09 elyograg so i can get 8 native mounts on separate machines. :)
19:09 misuzu joined #gluster
19:09 misuzu anyone ever seen an issue where gluster's samba shares are sharing the root of the node's filesystem instead of the root of the gluster volume?
19:10 elyograg if all of fuse works on a single thread, that might explain a few things.  because the source drives are NTFS, which is also fuse.
19:11 elyograg fuse.sourceforge.net says "By default FUSE filesystems run multi-threaded."
19:12 tryggvil joined #gluster
19:12 JoeJulian I was just about to paste that myself.
19:31 rahulcs joined #gluster
19:34 _dist joined #gluster
19:37 Matthaeus joined #gluster
19:40 Staples84 joined #gluster
19:52 criticalhammer Whats considered best practice to heal a split brain issue?
19:56 JoeJulian Pick a good one, nuke the other.
19:57 JoeJulian Well, best is don't get split-brained.
20:01 MacWinner joined #gluster
20:03 Matthaeus Have an odd number of nodes, sacrifice the partition that has the fewest members.
20:16 criticalhammer Whats the best way to reduce split brain?
20:17 semiosis criticalhammer: use quorum & replica 3
20:22 uebera|| joined #gluster
20:27 foobar77 joined #gluster
20:27 foobar77 hi, are gluster gurus hiding here?
20:28 semiosis hi
20:28 glusterbot semiosis: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
20:28 semiosis foobar77: ^^
20:29 XpineX joined #gluster
20:29 lalatenduM semiosis, :), glusterbot never spares
20:30 foobar77 OIC, I am looking for some clarity on volume heal in gluster 3.4. I have just replaced a brick and manually re-created the trusted volume file on it. heal status shows nothing healed, and nothing split-brain. New files written to the volume are appearing on the brick, existing files are not present
20:30 wushudoin left #gluster
20:32 kmai007 elyograg: hey your suggestion worked yesterday on the storage nodes, on getting rid of the (deleted) glusterd processes
20:32 _dist foobar77: if the self heal daemon is up (you can check with gluster volume status), it takes a while, something like every 5 min to start a crawl, you can trigger one manually by typing "gluster volume heal vol"
20:33 foobar77 I see a craw in progress, so I am just impatient? ;)
20:33 _dist foobar77: crawls can take a very long time, something I would expect to rsync in 10min might take 24 hours to self heal
20:34 _dist semiosis: that's normal right? I always assumed self-heal was set to be cautious so not to tank performance of the volume, but never asked
20:34 semiosis me?
20:34 foobar77 ok, but my volume is just for testing and contains only 2 files, one is around 250MB and the other just a few bytes
20:34 kmai007 guys when i run 'perf top' on rhel6, i see this on my fuse client
20:34 kmai007 1.58%  libglusterfs.so.0.0.0.#prelink#.lgvzZR (deleted)  [.] 0x000000000003a705
20:34 semiosis _dist: idk really
20:34 kmai007 the delete part concerns me....
20:35 _dist foobar77: there is a self-heal daemon log you can check to see what's going on, that's not normal. Sometimes I've had to type "gluster volume heal vol full" to get it to go, by default it only does diff
20:35 _dist foobar77: One other thing, you HAVE to have written those two files via gluster client, otherwise the xattr data won't be there and it won't heal
20:37 elyograg kmai007: that looks like it's a temporary file created perhaps by the linking process.  Seems kind odd, but it's not likely to be the actual file.
20:38 foobar77 _dist, what happened was the flies were originally placed by fuse client. Then I dd'd one of the bricks with /dev/zero (deliberately, just so see if nything funky happened) since that destroyed the filesystem, I recreated it and placed the brick back online.
20:38 kmai007 well when i search for it this is what i get
20:38 serencus joined #gluster
20:38 foobar77 I see in glustershd... 0-testvol-replicate-0: Could not create the task for 0 ret -1
20:38 kmai007 pasteme http://fpaste.org/84822/94656704/
20:38 glusterbot Title: #84822 Fedora Project Pastebin (at fpaste.org)
20:39 kmai007 blah its a prelink so i'll just chalk it up to the ignore
20:40 kmai007 elyograg: updating the client's glusterfs-fuse package, do you recommend umounting the gluster volume?
20:40 chirino joined #gluster
20:43 elyograg you'll have to unmount it and remount it before it will begin using the new version.
20:45 foobar77 _dist, a full heal does the job but is that not a bad idea if this was a big volume?
20:55 qdk joined #gluster
21:04 robo joined #gluster
21:09 prometheanfire joined #gluster
21:11 rpowell1 joined #gluster
21:12 prometheanfire gluster with two nodes (replica of 1?
21:15 FarbrorL_ joined #gluster
21:15 Ark joined #gluster
21:15 prometheanfire sound logical?
21:16 elyograg if you only have one copy of your data, as soon as a server goes down, you lose access to that data. if the failure is related to the data, it's just gone.
21:18 prometheanfire replica of 1 meaning that it's mirrored across two nodes
21:20 robo joined #gluster
21:21 foobar77 replica 2 to create it on 2 nodes
21:22 elyograg yes ... replica 1 means there's only one copy.  replica 2 means there are two.
21:25 foobar77 which is a little bit misleading imo, replica 1 sounds like it means is 1 replica of the original data, hence 2 copies
21:26 elyograg if it were a situation where there were such a thing as a 'master' ... that would make sense.  but in a true cluster, there is no master, only replicas.
21:26 Matthaeus Replica 0 is using /dev/null as a brick.
21:26 Matthaeus The write speeds are unbelievable!
21:27 rossi__ joined #gluster
21:27 prometheanfire ok
21:27 JoeJulian lol
21:28 prometheanfire combine it with the read speeds on /dev/zero and you just revolutionized computing
21:28 foobar77 elyograg thanks, that actually does make sense
21:29 elyograg there is no Dana, only Zuul.
21:29 foobar77 is there a reason why there is a requirement for manual changing of xattrs when things go wrong, surely there should be a toolset for these things or did I miss something?
21:30 Matthaeus elyograg: See, that's a replica 2 but using two bricks on one node.  Not recommended.
21:30 * foobar77 wants to tell gluster, in this split-brain situation, just use the copy from the brick on server N
21:31 * prometheanfire hates split brain
21:31 prometheanfire so much
21:31 * foobar77 caused it deliberately and is now very confused
21:31 JoeJulian http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/
21:31 glusterbot Title: GlusterFS Split-Brain Recovery Made Easy (at joejulian.name)
21:31 Matthaeus foobar77: Delete the file (and all hardlinks to that file) on the node you want to fail.
21:32 prometheanfire left #gluster
21:32 elyograg I have eight bricks per server.  i don't put both replicas on the same machine, though.  http://fpaste.org/84837/13946599/
21:32 glusterbot Title: #84837 Fedora Project Pastebin (at fpaste.org)
21:32 yinyin joined #gluster
21:33 FarbrorLeon joined #gluster
21:34 foobar77 Matthaeus, I thought if I induced a split brain situation, when running info split-brain I would see the same file shown on 2 servers but seemingly, I have the same file listed 3 times on the same server
21:35 Matthaeus foobar77: You're going to need to be a lot more specific than that.  Also, telling us how you intentionally caused the split brain may help.
21:37 failshel_ joined #gluster
21:37 elyograg split-brain is actually a *log*, not a current list.  can't seem to find the bug on it.
21:38 elyograg bug 864963
21:38 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=864963 low, medium, ---, vsomyaju, ASSIGNED , Heal-failed and Split-brain messages are not cleared after resolution of issue
21:40 foobar77 Matthaeus, I have a small iso  replicated to 3 nodes. On node 1 itself, echo "bad news" > my.iso, (I know you wouldn't ordinarily do something like this but I am in the process of see how I break things and then how I go about fixing them from there), http://pastebin.com/DmaWZm5h
21:40 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
21:41 elyograg are you doing that right to the brick?  writing directly to bricks will cause problems that gluster is not prepared to deal with.
21:41 * foobar77 tries again http://fpaste.org/84839/
21:42 elyograg all access to gluster volumes needs to happen via a mount, normally fuse, nfs, or samba vfs.
21:42 foobar77 elyograg yes, I am , I know this is bad but could happen right? Jus ttrying to get to grips with what breaks and how badly when I do things I am not supposed to
21:43 * JoeJulian mumbles something about "couldn't happen on my servers..."
21:44 foobar77 beause of course, it's not beyond the realms of possibility that a stressed out, sleep deprived sysadmin does something silly in the middle of the night
21:44 JoeJulian @split-brain
21:44 glusterbot JoeJulian: To heal split-brain in 3.3+, see http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/ .
21:45 JoeJulian @forget split-brain
21:45 glusterbot JoeJulian: The operation succeeded.
21:45 JoeJulian @learn split-brain as To heal split-brain, use splitmount. http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/
21:45 glusterbot JoeJulian: The operation succeeded.
21:46 elyograg sysadmins who are not in charge of gluster itself should not have access to the actual gluster bricks.  There are so many things that a tired sysadmin can do that can't be rolled back ... you can't plan for them all.
21:46 JoeJulian @learn split-brain as For additional information, see this older article http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
21:46 glusterbot JoeJulian: The operation succeeded.
21:46 foobar77 I figured my split-brain info would tell me that node1 has a different copy to nodes 2 & 3
21:46 elyograg how did you let gluster know that it even needed to check for that?
21:47 elyograg what happened via the mount when you did that?
21:48 JoeJulian I don't want gluster to have a democratic election to determine which data to lose. That's an human-only decision.
21:48 foobar77 elyograg, sorry not understanding, are you asking what the client now sees after making the change directly on the brick?
21:48 elyograg yes.
21:49 foobar77 bad things
21:49 foobar77 :(
21:49 foobar77 it looked ok but gives me I/O error when trying to access the file
21:49 JoeJulian Yep, that's split-brain. Good gluster.
21:50 kmai007 gluster NFS supports only NFSv3 ?
21:50 cjanbanan joined #gluster
21:50 JoeJulian kmai007: correct.
21:50 JoeJulian @nfs
21:50 glusterbot JoeJulian: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
21:51 Matthaeus foobar77: What JoeJulian says.  Gluster somewhat assumes that you're only accessing/modifying data through the mount, and that the bricks themselves are sacrosanct.
21:51 semiosis s/somewhat assumes/requires/
21:52 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
21:52 semiosis glusterbot: thx
21:52 glusterbot semiosis: you're welcome
21:52 JoeJulian I was waiting for that ... :D
21:52 kmai007 JoeJulian:
21:53 elyograg split brain can happen when only accessing through the mount, but something unusual is probably going on when it happens, like network outages, system load so high that things don't respond, etc.
21:53 JoeJulian And you can mitigate that with quorum features.
21:53 kmai007 can a client have a filesystem mount from a true NFS source, and have gluster NFS mounted?
21:54 foobar77 I do understand why you dont want to modify files directly on the brick and I would not normally do such a thing but I dont understand why it is a big problem to rectify, surely if you have 2 nodes that contain the same file content and a third node that was modified independently. I guess ultimately when I finish my report the conclusion will be dont go there, it is interesting to break things in a safe environment though :)
21:54 kmai007 so i have a client that has FUSE mounts, and 1 NFS mount, and i'm trying to mount gluster NFS
21:54 JoeJulian both are "true" but yes. You can mount a gluster nfs source with the "tcp,vers=3" mount options, and an nfsd source without those options.
21:55 Matthaeus foobar77: Gluster doesn't do a byte-for-byte comparison of all copies of a file on every read.
21:56 Matthaeus foobar77: It uses xattrs to keep track of which nodes have been updated and which haven't, and assumes that if the xattrs say all files are in sync, then all files are in sync.
21:56 JoeJulian "I dont understand why it is a big problem to rectify, surely if you have 2 nodes that contain the same file content and a third node that was modified independently." Which one's right? The one with, perhaps, bitrot due to a drive error or a recently fired sysadmin? The two that are the same? What if the change was intentional but missed two replicas due to netsplit?
21:56 kmai007 yck netsplit
21:57 JoeJulian (that admin was fired for writing to the brick, btw...)
21:57 semiosis ha
21:58 foobar77 Which one is right? The one I tell gluster is right after I work it out? ;)
21:58 avati joined #gluster
21:58 JoeJulian Why is why I keep pointing you to my blog.
21:58 Matthaeus JoeJulian: You made me snort air out my nose.
21:59 foobar77 yeah sure, I will definitely digest that
21:59 JoeJulian hehe
21:59 kmai007 JoeJulian: when proceeding with gluster NFS, i see from the mount -v output that it tries all NFS ports to gluster storage, then only connects with 2049, is that the logic? it tries it all?
21:59 kmai007 @NFS
21:59 glusterbot kmai007: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
22:00 kmai007 @PORTS
22:00 glusterbot kmai007: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
22:02 JoeJulian kmai007: I don't know. I think that's an nfs client thing.
22:03 kmai007 so my internal client connects to the gluster storage via NFS protocol.....
22:04 kmai007 checking the brick nfs.log says this http://fpaste.org/84844/66181213/
22:04 glusterbot Title: #84844 Fedora Project Pastebin (at fpaste.org)
22:04 kmai007 confusing....
22:04 kmai007 so its mounted, but the logs leads me to believe otherwise.
22:05 kmai007 thats internal, i cannot even get it to mount in my DMZ
22:06 rahulcs joined #gluster
22:06 kmai007 i can mount glusterfs all day outside my firewall, just not gluster NFS
22:06 JoeJulian 100003 is nfs. 4 is version 4.
22:06 avati JoeJulian /msg ?
22:07 sputnik13 joined #gluster
22:08 kmai007 JoeJulian: you did it again, ok so now the logs correlate
22:08 getup- joined #gluster
22:12 foobar77 JoeJulian++ the information on your blog and splitmount is awesome, thank you
22:12 JoeJulian You're welcome.
22:13 johnmark left #gluster
22:13 johnmark joined #gluster
22:14 rahulcs joined #gluster
22:14 Pavid7 joined #gluster
22:14 nage joined #gluster
22:14 nage joined #gluster
22:14 Elico joined #gluster
22:15 ctria joined #gluster
22:18 johnmark @chanstats
22:19 johnmark oh no... where's glusterbot? :(
22:20 velladecin Gents, I'm trying to 'create' a split-brain but the damn file always heals itself. I reboot server1, make changes to file on server2, reboot server2, when server1 comes back up I make changes to file there. Now, once I managed to get split-brain messages in the logs, and Input/Output error when accessing the file. BUT, after a while it fixed itself... all the other times I don't even get anything in logs. I want to test manual recovery of split-b
22:20 johnmark glusterbot: hi
22:20 glusterbot johnmark: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
22:20 rpowell1 left #gluster
22:20 johnmark glusterbot: @chanstats
22:20 johnmark @channelstats
22:20 glusterbot johnmark: On #gluster there have been 272684 messages, containing 11110831 characters, 1840493 words, 6995 smileys, and 944 frowns; 1453 of those messages were ACTIONs. There have been 114109 joins, 3315 parts, 110931 quits, 24 kicks, 206 mode changes, and 7 topic changes. There are currently 196 users and the channel has peaked at 239 users.
22:20 johnmark perfect...
22:20 seapasulli left #gluster
22:21 JoeJulian "after a while it fixed itself..." ... um... no? That can't happen....
22:21 * JoeJulian boggles
22:22 JoeJulian johnmark: what ever happened to getting a copy of all our data from community.gluster.org?
22:22 kmai007 JoeJulian: on my client when i execute 'showmount -e <gluster_node>' i get this
22:22 kmai007 clnt_create: RPC: Program not registered
22:23 foobar77 JoeJulian, happens all the time ;)
22:23 rpowell joined #gluster
22:25 johnmark JoeJulian:  I have it. in an mbox file :(
22:25 JoeJulian Cool, send it to me so I can pick out useful chunks and make them available again.
22:26 robo joined #gluster
22:26 johnmark JoeJulian: ah, lovely. I can do that
22:36 Ark joined #gluster
22:41 velladecin yes, that's what I was thought too "self-fix - cannot happen" but Gluster 3.4.2 is just too good :) I'll keep trying, but the sequence of actions that I take (described above) is about right, right?
22:41 JoeJulian Yes, that's one way of doing it.
22:42 velladecin that would suggest there's another way too..?
22:42 JoeJulian Another would be to mount the client on both servers and block them from each other, pull the cable or use iptables.
22:43 velladecin ic, more less the same thing without the reboots
22:43 JoeJulian yep
22:43 sroy_ joined #gluster
22:43 JoeJulian effectively creating yourself a netsplit.
22:44 velladecin I'll give that a go as the reboot didn't work for me too well, thanks
22:49 driver_ joined #gluster
22:52 rossi__ joined #gluster
22:54 robo joined #gluster
22:58 tryggvil joined #gluster
23:04 gmcwhistler joined #gluster
23:06 Joe630 hey guys
23:06 Joe630 re out conversation yerday about roundrobin dns
23:06 Joe630 looks like you don't have to do that to get failover
23:06 Joe630 when using mount.glusterfs
23:06 Joe630 is that the case?
23:07 Matthaeus Glusterfs will fail on mount if the address specified in the mount command is unavailable, but once it's mounted (using the glusterfs client, that is), it'll be fault tolerant.
23:08 Joe630 ah!  thats why that suggestion was made, many thanks for the clarity
23:09 chirino joined #gluster
23:13 Joe630 one more question
23:13 Joe630 if I write directly to the brick, should that error?  are there consistency checks?
23:15 Matthaeus Joe630: don't write directly to the brick.
23:15 Matthaeus Ever.
23:17 JoeJulian except when... ;)
23:19 Joe630 That's fine advice.
23:19 Joe630 What happens if you do?
23:21 JoeJulian https://encrypted-tbn0.gstatic.com/​images?q=tbn:ANd9GcQsZk4BIATtKe1tEk​wbk0FODZMP1qjhSyOpLI5VqNOdmRnX0b2Z
23:21 JoeJulian That leads to undefined behavior by it's very definition.
23:21 Joe630 is it like dividing by zero
23:22 JoeJulian I'm hoping to make Matthaeus snort again...
23:22 Joe630 https://i.chzbgr.com/maxW500/7051425792/hC915AD3F/
23:22 Matthaeus I snorted.
23:23 JoeJulian Joe630: Read up on ,,(extended attributes) and peruse my blog before you think about touching the bricks.
23:23 glusterbot Joe630: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
23:23 rpowell1 joined #gluster
23:23 sputnik13 joined #gluster
23:24 Joe630 thanks JoeJulian
23:25 Joe630 a accidently the whole filesystem.
23:25 Joe630 I accidently the whole filesystem.
23:32 failshell joined #gluster
23:32 failshel_ joined #gluster
23:35 seapasulli_ joined #gluster
23:36 ninkotech__ joined #gluster
23:37 mbukatov joined #gluster
23:38 ShanghaiScott joined #gluster
23:38 ThatGraemeGuy joined #gluster
23:38 tg2 joined #gluster
23:40 chirino joined #gluster
23:43 * foobar77 needs sleep now, thank you very much all for your assistance
23:44 yinyin joined #gluster
23:57 robo joined #gluster
23:58 kminooie joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary