Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-11-17

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:34 phak joined #gluster
00:54 topshare joined #gluster
01:05 plarsen joined #gluster
01:09 hamcube joined #gluster
01:21 calum_ joined #gluster
01:24 gildub joined #gluster
01:25 rjoseph joined #gluster
01:27 lyang0 joined #gluster
01:53 harish joined #gluster
01:59 glusterbot New news from newglusterbugs: [Bug 1161903] Different client can not execute "for((i=0;i<1000;i++));do ls -al;done" in a same directory at the sametime <https://bugzilla.redhat.co​m/show_bug.cgi?id=1161903>
02:03 wgao joined #gluster
02:09 hagarth joined #gluster
02:17 haomaiwang joined #gluster
02:36 kshlm joined #gluster
02:43 hamcube joined #gluster
02:48 LebedevRI joined #gluster
03:07 clare joined #gluster
03:21 bharata-rao joined #gluster
03:53 MugginsM joined #gluster
03:54 itisravi joined #gluster
03:56 RameshN joined #gluster
03:59 ndarshan joined #gluster
04:03 atinmu joined #gluster
04:09 rafi1 joined #gluster
04:10 Rafi_kc joined #gluster
04:12 nishanth joined #gluster
04:12 smallbig_ joined #gluster
04:15 nbalachandran joined #gluster
04:26 spandit joined #gluster
04:37 anoopcs joined #gluster
04:45 shubhendu joined #gluster
04:45 anoopcs joined #gluster
04:47 jiffin joined #gluster
04:56 atalur joined #gluster
04:56 kumar joined #gluster
05:07 pp joined #gluster
05:13 meghanam joined #gluster
05:14 lalatenduM joined #gluster
05:27 hagarth joined #gluster
05:38 atalur joined #gluster
05:49 ndarshan joined #gluster
05:52 ramteid joined #gluster
06:09 overclk joined #gluster
06:22 nshaikh joined #gluster
06:30 glusterbot New news from newglusterbugs: [Bug 1141940] Mount -t glusterfs never completes and all file-system commands hang <https://bugzilla.redhat.co​m/show_bug.cgi?id=1141940>
06:30 bharata-rao joined #gluster
06:31 ndarshan joined #gluster
06:31 atalur joined #gluster
06:36 unwastable joined #gluster
06:37 unwastable anybody familiar with gluster + samba
06:49 rjoseph joined #gluster
06:49 bala joined #gluster
06:50 mator joined #gluster
06:59 bala joined #gluster
07:00 glusterbot New news from newglusterbugs: [Bug 1153569] client connection establishment takes more time for rdma only volume <https://bugzilla.redhat.co​m/show_bug.cgi?id=1153569>
07:05 atinmu joined #gluster
07:07 ctria joined #gluster
07:10 hagarth unwastable: let us know your problem with gluster + samba, rastar_afk or somebody might be able to help.
07:17 rjoseph joined #gluster
07:20 deepakcs joined #gluster
07:20 ndarshan joined #gluster
07:28 shubhendu joined #gluster
07:33 Fen2 joined #gluster
07:43 rjoseph joined #gluster
07:45 ndarshan joined #gluster
07:47 Philambdo joined #gluster
07:48 Slydder joined #gluster
07:51 RameshN joined #gluster
07:55 [Enrico] joined #gluster
07:56 fsimonce joined #gluster
08:12 kanagaraj joined #gluster
08:19 RameshN joined #gluster
08:21 d4nku joined #gluster
08:43 elico joined #gluster
08:46 saurabh joined #gluster
08:50 SOLDIERz joined #gluster
08:58 ekuric joined #gluster
09:01 Slashman joined #gluster
09:01 d4nku joined #gluster
09:04 kumar joined #gluster
09:05 liquidat joined #gluster
09:10 ProT-0-TypE joined #gluster
09:14 meghanam joined #gluster
09:17 lalatenduM joined #gluster
09:19 Norky joined #gluster
09:21 deniszh joined #gluster
09:25 d4nku joined #gluster
09:29 johndescs joined #gluster
09:31 karnan joined #gluster
09:43 d4nku joined #gluster
09:46 johndescs_ joined #gluster
09:52 bala joined #gluster
09:57 nshaikh joined #gluster
09:57 shubhendu joined #gluster
09:58 cultavix joined #gluster
09:58 cultavix joined #gluster
10:05 d4nku joined #gluster
10:11 SmithyUK joined #gluster
10:12 SmithyUK Hi all, having some issues with GlusterFS 3.6.1. After it has been running for around an hour gluster commands cease to work.  with the following error: [2014-11-17 10:10:21.818686] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/c17c278617345958e75a372b836b1e5d.socket failed (Invalid argument)
10:12 SmithyUK [2014-11-17 10:10:22.564092] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking failed on 10.35.13.77. Please check log file for details.
10:12 SmithyUK [2014-11-17 10:10:22.564435] E [glusterd-syncop.c:1602:gd_sync_task_begin] 0-management: Locking Peers Failed.
10:13 SmithyUK This was not an issue in our QA environment but is now causing issues in our production environment. Any help is appreciated
10:13 SmithyUK The message "I [MSGID: 106006] [glusterd-handler.c:4257:__g​lusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2014-11-17 10:08:09.669540] and [2014-11-17 10:10:06.801652]
10:13 SmithyUK Seeing that a lot too but NFS is set to disabled
10:18 shubhendu joined #gluster
10:20 ira joined #gluster
10:24 geaaru joined #gluster
10:29 mator volume nfs disabled or system (linux kernel) nfs server?
10:29 geaaru hi, is there a way to force a geo-replication session with a non-root user ? thanks in advance
10:33 shubhendu_ joined #gluster
10:34 d4nku joined #gluster
10:45 meghanam joined #gluster
10:45 meghanam_ joined #gluster
11:07 SmithyUK mator: volume nfs disabled
11:08 SmithyUK mator: I'm having so many different issues I'm not sure what's related anymore, this is from trying to start a rebalance
11:08 SmithyUK [2014-11-17 11:05:23.297978] E [glusterd-utils.c:10190:glusterd​_volume_rebalance_use_rsp_dict] (--> /usr/lib64/glusterfs/3.6.1/xlator/mgm​t/glusterd.so(glusterd_volume_rebalan​ce_use_rsp_dict+0xe2)[0x2ac264aac1d2] (--> /usr/lib64/glusterfs/3.6.1/xlator/mgmt/glusterd.s​o(__glusterd_commit_op_cbk+0x6f2)[0x2ac264ab52d2] (--> /usr/lib64/glusterfs/3.6.1/xlator/mgmt/glusterd.​so(glusterd_big_locked_cbk+0x60)[0x2ac264ab26f0] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_h
11:08 SmithyUK andle_reply+0xa0)[0x2ac25c550f40] (--> /usr/lib64/libgfrpc.so.0(rpc_cln​t_notify+0x164)[0x2ac25c5511b4] ))))) 0-: Assertion failed: (GD_OP_REBALANCE == op) || (GD_OP_DEFRAG_BRICK_VOLUME == op)
11:08 SmithyUK [2014-11-17 11:05:23.298029] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : 00000000-0000-0000-0000-000000000000
11:08 glusterbot SmithyUK: ('s karma is now -45
11:08 glusterbot SmithyUK: ('s karma is now -46
11:08 glusterbot SmithyUK: ('s karma is now -47
11:08 glusterbot SmithyUK: ('s karma is now -48
11:08 SmithyUK [2014-11-17 11:05:23.298056] E [glusterd-op-sm.c:6566:glusterd_op_sm] 0-management: Unable to get transaction's opinfo
11:08 glusterbot SmithyUK: ('s karma is now -49
11:11 ramteid joined #gluster
11:12 mator SmithyUK, i wonder why do you see nfs related problems, if volume nfs is disabled...
11:13 mator SmithyUK, if it's timeout issue, do you have probably some kind of stateful firewall between gluster hosts ?
11:13 mator just a thought
11:15 SmithyUK mator: no firewall enabled between hosts
11:15 SmithyUK I have the same setup in 3 sites but this site just seems to have loads of issues
11:15 SmithyUK This is mine too https://bugzilla.redhat.co​m/show_bug.cgi?id=1151384
11:15 glusterbot Bug 1151384: high, unspecified, ---, bugs, NEW , Rebalance fails to complete - stale file handles after 202,908 files
11:21 kshlm SmithyUK, The nfs logs are benign and are not real errors. GlusterD is still attempting to connect to the glusternfs server even when it should not.
11:21 kshlm We need to fix that.
11:21 SmithyUK kshlm: okay, thanks for that. I'm still seeing this bug in 3.6.1 https://bugzilla.redhat.com/show_bug​.cgi?format=multiple&amp;id=1102656
11:21 glusterbot Title: Full Text Bug Listing (at bugzilla.redhat.com)
11:22 SmithyUK And my rebalance bug also
11:23 kshlm Are all your commands failing with 'Failed to acquire lock'(something along this line) message?
11:23 kshlm Not just rebalance.
11:23 SmithyUK They are now yeah
11:24 SmithyUK But even when I can start a rebalance I get a bunch of stale file handle errors and it exits after roughly 200k files scanned on each host
11:25 meghanam joined #gluster
11:25 meghanam_ joined #gluster
11:25 kshlm Rebalance migration failures are a little out of my domain, but I can help you with your command execution failures.
11:26 kshlm A log said 'Locking failed on *.77' .
11:26 calisto joined #gluster
11:26 SmithyUK It happens with multiple hosts yeah
11:26 kshlm Can you look at the glusterd logs of that peer and see if you have a log along the lines 'Lock held by <UUID>'?
11:27 SmithyUK yep, got plenty of those
11:27 kshlm Are the UUIDs different?
11:27 SmithyUK all held by the same host by the looks of it
11:28 kshlm Can you do a 'gluster pool list' and identify the peer?
11:28 SmithyUK hmm just did
11:28 SmithyUK its not in there
11:29 kshlm That's suspicious.
11:29 SmithyUK phantom uuid
11:29 kshlm I assume you didn't detach any peer.
11:29 kshlm ?
11:29 SmithyUK no i didnt
11:30 SmithyUK i did however remove a brick some time ago
11:30 kshlm Okay.
11:30 SmithyUK perhaps it didnt go correctly
11:30 shubhendu joined #gluster
11:31 SmithyUK not listed in $ ll /var/lib/glusterd/peers/
11:32 kshlm The local peer (the one where you ran the `ll` command) is not present in /var/lib/glusterd/peers and in the output of 'gluster peer status'
11:33 SmithyUK oh that's it then
11:33 kshlm 'gluster pool list' should show all the peers including the localhost.
11:33 SmithyUK yeah this is the local peer ip
11:33 kshlm But you said it doesn't have it either.
11:34 SmithyUK oh, that's my mistake. it does have it there
11:34 SmithyUK i ran status the first time
11:34 kshlm Oh cool. So it's the local peer holding the lock then :)
11:34 SmithyUK its the one i attempted to start the rebalance from so i imagine something to do with running that command
11:35 kshlm Yup most likely.
11:35 kshlm Can you paste the log snippet for glusterd around the time of running the rebalance command?
11:35 kshlm !fpaste
11:35 kshlm Isn't glusterbot here?
11:36 hagarth @fpaste
11:36 SmithyUK he told me off earlier, give me a few secs and ill get that
11:36 hagarth @seen glusterbot
11:36 glusterbot hagarth: I have not seen glusterbot.
11:36 kshlm !paste
11:36 kshlm @paste
11:36 glusterbot kshlm: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
11:36 kshlm Ah!
11:36 * ndevos hands glusterbot a mirror
11:38 kshlm SmithyUK, To recover from you current inability to run gluster commands, you can restart glusterd on the current peer.
11:38 kshlm This should cause the locks on the remaining peers to get cleared, when they have a disconnect.
11:39 kshlm s/have a/see the/
11:39 glusterbot What kshlm meant to say was: This should cause the locks on the remaining peers to get cleared, when they see the disconnect.
11:39 cultav1x joined #gluster
11:40 SmithyUK http://fpaste.org/151400/16224406/
11:40 glusterbot Title: #151400 Fedora Project Pastebin (at fpaste.org)
11:41 kshlm SmithyUK, The logs seem to indicate that the local peer somehow forgot that it itself it the lock owner!
11:42 kshlm "[2014-11-17 11:05:21.359521] E [glusterd-utils.c:148:glusterd_lock] 0-management: Unable to get lock for uuid: 245f9645-0023-4f51-94d9-0b83d422373e, lock held by: 245f9645-0023-4f51-94d9-0b83d422373e"
11:42 SmithyUK oh wow, that doesnt sound healthy
11:42 kshlm This seems similar to an issue we are tracking, I'm searching for the bug-id.
11:44 kshlm 1154635
11:45 kshlm Why isn't glusterbot working again?
11:45 kshlm https://bugzilla.redhat.co​m/show_bug.cgi?id=1154635
11:45 glusterbot Bug 1154635: high, unspecified, ---, amukherj, POST , glusterd: Gluster rebalance status returns failure
11:46 shubhendu joined #gluster
11:46 kshlm A fix for the problem has been merged into the master branch. It could be backported to 3.6.x.
11:47 SmithyUK Cool, any idea when the bug was introduced?
11:47 ramteid joined #gluster
11:47 kshlm The bug should have been present all along since glusterd debuted.
11:48 liquidat joined #gluster
11:48 kshlm But slight changes to the locking framework in 3.6 caused it to pop up.
11:48 kshlm s/slight/pretty big/
11:48 glusterbot What kshlm meant to say was: But pretty big changes to the locking framework in 3.6 caused it to pop up.
11:48 SmithyUK Righto, well thanks for checking into that for me. I suppose I'll wait for the next bunch of RPMs and try again!
11:49 kshlm SmithyUK, Cool! Did restarting glusterd help?
11:49 SmithyUK It did, yep, I'll just be careful about what commands I run for a while
11:49 soumya joined #gluster
11:51 kshlm SmithyUK, Cool! I'm heading home now. If you've anymore questions, I'll be back online in ~45 minutes.
11:51 SmithyUK kshlm: Awesome, thanks again!
11:51 hagarth kshlm++
11:51 glusterbot hagarth: kshlm's karma is now 1
11:55 meghanam joined #gluster
11:55 meghanam_ joined #gluster
11:57 SOLDIERz joined #gluster
11:57 ramteid joined #gluster
11:58 MrAbaddon joined #gluster
11:58 dusmant joined #gluster
12:01 glusterbot New news from newglusterbugs: [Bug 1164768] Write strings to a file by O_APPEND mode (echo "strings" >> /mountpoint/file.txt) is abnormal <https://bugzilla.redhat.co​m/show_bug.cgi?id=1164768>
12:06 diegows joined #gluster
12:21 RameshN joined #gluster
12:38 dusmant joined #gluster
12:39 Inflatablewoman joined #gluster
12:42 calisto joined #gluster
12:44 hagarth joined #gluster
12:51 diegows joined #gluster
13:01 Slashman_ joined #gluster
13:01 glusterbot New news from newglusterbugs: [Bug 1164775] Glusterd segfaults on gluster volume status ... detail <https://bugzilla.redhat.co​m/show_bug.cgi?id=1164775>
13:07 dusmant joined #gluster
13:08 breun joined #gluster
13:09 harish joined #gluster
13:11 breun Hello. I have a small GlusterFS cluster running, but I was wondering whether POSIX file locking is enabled by default. I read the FAQ entry at http://gluster.org/community/documentation/index.p​hp/GlusterFS_Technical_FAQ#How_is_locking_handled.3F but it doesn't have a lot of information.
13:12 breun I see I have /usr/lib64/glusterfs/3.6.1/xl​ator/features/posix-locks.so, but I don't see a process on my system that has it loaded.
13:15 Fen1 joined #gluster
13:19 rjoseph joined #gluster
13:19 ekuric left #gluster
13:20 diegows joined #gluster
13:33 edward1 joined #gluster
13:35 SteveCooling joined #gluster
13:37 rjoseph joined #gluster
13:37 diegows joined #gluster
13:38 itisravi joined #gluster
13:38 sjohnsen joined #gluster
13:50 cygoonda joined #gluster
13:52 lpabon joined #gluster
13:55 B21956 joined #gluster
13:56 plarsen joined #gluster
14:04 virusuy joined #gluster
14:04 virusuy joined #gluster
14:05 cygoonda joined #gluster
14:07 tdasilva joined #gluster
14:08 cultav1x joined #gluster
14:09 meghanam_ joined #gluster
14:09 meghanam joined #gluster
14:11 itisravi joined #gluster
14:14 virusuy Morning gents, i'm planning an upgrade to my distributed-replicated gluster from 3.4.0 to 3.5.2
14:14 virusuy but i wouldn't upgrade all my gluster member at the same time
14:15 virusuy the question is, Is there any 'compatibility matrix' to see if i can have my gluster working with nodes using those 2 versions ?
14:16 bennyturns joined #gluster
14:19 ricky-ticky joined #gluster
14:24 dmyers joined #gluster
14:26 bene joined #gluster
14:33 ctria joined #gluster
14:42 _Bryan_ joined #gluster
14:44 itisravi joined #gluster
14:45 SOLDIERz_ joined #gluster
14:50 Slash__ joined #gluster
15:03 plarsen joined #gluster
15:03 dblack joined #gluster
15:06 _Bryan_ joined #gluster
15:08 julim joined #gluster
15:08 _dist joined #gluster
15:13 calisto joined #gluster
15:14 NuxRo hi guys, any quick instructions for fixing split brain for gluster 3.4.0 ?
15:14 NuxRo @split-brain
15:14 glusterbot NuxRo: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
15:15 NuxRo well, there you go :)
15:15 NuxRo all heil glusterbot
15:24 Fuzion joined #gluster
15:24 Fuzion Hello Guys
15:24 Fuzion I would like to know if it's possible to speak to the xlator changelog with libgfapi to dump the journal log ?
15:25 meghanam_ joined #gluster
15:25 meghanam joined #gluster
15:27 Fuzion if not, is there a way to get logs in a file from the changelog xlator ?
15:28 Fuzion my point is to purge some content in my varnish instances when a file is updated
15:30 pradeepto left #gluster
15:30 topshare joined #gluster
15:49 nszceta joined #gluster
15:50 nszceta how does glusterfs deal with random node sizes? like lets say I have 3 quotas, each of which is on a separate VM. 10, 20, and 30 GB.
15:51 jbrooks joined #gluster
15:54 bviktor joined #gluster
15:54 bviktor left #gluster
15:57 nbalachandran joined #gluster
15:57 jobewan joined #gluster
15:59 rwheeler joined #gluster
16:01 _dist nszceta: Three bricks with different sizes (in a replica volume for example) it just doens't care. The first one that fills will have some kind of failure (specifically I don't know) but I suspect the brick would go offline, or possibly worse since I've never  tested it.
16:05 calisto joined #gluster
16:06 social joined #gluster
16:11 Fen1 joined #gluster
16:12 lmickh joined #gluster
16:12 rotbeard joined #gluster
16:17 coredump joined #gluster
16:32 bala joined #gluster
16:34 Ramereth joined #gluster
16:40 doo joined #gluster
16:41 jhc76 joined #gluster
16:44 bennyturns joined #gluster
16:44 cultav1x joined #gluster
16:46 bala joined #gluster
16:53 RameshN joined #gluster
16:53 cultav1x joined #gluster
16:54 haomaiwa_ joined #gluster
16:54 daMaestro joined #gluster
16:55 nszceta _dist: in such a failure mode, is the data on that particular node that failed readable
16:55 hagarth joined #gluster
16:59 nishanth joined #gluster
16:59 _dist nszceta: If everything works as I'd expect the worst scenario I imagine would happen would be that the volume would be unavailable for 42 seconds to all clients (the default timeout). However, I've not tested this so it's possible that the volume may need to be told the brick is offline manully. It's a simple test, just make a two-brick volume one with 10MB and one with 100MB, then write 30MBs to it via gluster-fuse
16:59 _dist and see what happens.
17:01 _dist nszceta: In practice we've always made our replicate brick underlying FS the same as all others in the same replica voume, and with more space than we imagine we'll use. In my experience gluster handles most "bad" stuff with grace, but I just haven't tested this one.
17:03 bala joined #gluster
17:05 nszceta thanks _dist
17:07 sputnik13 joined #gluster
17:08 nszceta I think this can be avoided with some sort of disk space usage monitoring anyway
17:08 nszceta is that what you do _dist
17:09 JoeJulian It will work, but once the smallest brick is full, further creation of files will cause those files to be created on other bricks. If you grow a file that's on a full brick, your client will receive ENOSPC.
17:11 mojibake joined #gluster
17:12 rjoseph joined #gluster
17:19 nszceta makes sense thanks JoeJulian
17:22 PeterA1 joined #gluster
17:22 JoeJulian nszceta: What some of us have done is use lvm to carve all the disks into bricks of the same size, then just have multiple bricks per machine to allow an even distribution.
17:23 nszceta interesting concept
17:23 nszceta I would have thought that slows things down a lot
17:41 elico joined #gluster
17:45 jhc76 If I add additional gluster brick to existing cluster which is almost completely full, is gluster smart enough to fill the newly added brick instead of choking the pre-existing bricks? I'm using distributed mode.
17:48 JoeJulian No
17:49 JoeJulian In fact, to make glusterfs use the newly added brick, you need to at least run a rebalance...fix-layout to set the masks used for dht. A full rebalance (if successful) is how you would handle moving files to balance things out.
17:50 cfeller joined #gluster
17:53 rafi1 joined #gluster
17:55 ckotil joined #gluster
17:58 if-kenn joined #gluster
18:00 jhc76 oh wow... more complicated then I had hoped. thank JoeJulian. Is there a document on the rebalancing?
18:00 jhc76 thanks*
18:03 JoeJulian gluster volume help :)
18:04 JoeJulian https://github.com/gluster/glusterfs/blob/​master/doc/admin-guide/en-US/markdown/admi​n_managing_volumes.md#rebalancing-volumes
18:04 glusterbot Title: glusterfs/admin_managing_volumes.md at master · gluster/glusterfs · GitHub (at github.com)
18:05 NuxRo JoeJulian: I think this has better behaviour starting with 3.6, ie filling the more empty brick
18:05 jhc76 JoeJulian: Cool. Thx!
18:06 PeterA joined #gluster
18:06 JoeJulian NuxRo: I'll have to do some testing. So far I still don't trust rebalance to either finish or not crash my clients.
18:09 lalatenduM joined #gluster
18:10 zerick joined #gluster
18:11 NuxRo JoeJulian: never done a rebalance and don't plan on it either :)
18:11 NuxRo if i could keep adding servers and gluster to prefer filling the emptier ones, then I am happy (for my use case)
18:12 NuxRo of course, at some point one of them will fill up and it will suck
18:12 if-kenn I have Googled quite a bit to find a definitive answer to this one, but have not come up with concrete numbers.  Are there any specifications for the recommended network throughput and minimal acceptable latency for Gluster replication?
18:13 jmarley joined #gluster
18:14 JoeJulian Not usually. It's all use-case dependent.
18:15 JoeJulian Obviously, cold storage would need much less bandwidth and could accept higher latency than, say, media streaming or image stores.
18:17 topshare joined #gluster
18:20 JoeJulian @ports
18:20 glusterbot JoeJulian: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up since 3.4.0 (24009 & up previously). (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
18:25 jackdpeterson joined #gluster
18:27 edwardm61 joined #gluster
18:29 sputnik13 joined #gluster
18:37 warci joined #gluster
18:38 warci hi guys, i'm planning on setting up a POC to check how gluster performs for RHEV, but everyone keeps telling me this is a bad idea. Is it really?
18:40 warci even a red hat guy said i had to go for ceph, but we're using gluster already and i'd love it if we could keep one storage solution...
18:47 sputnik13 joined #gluster
18:48 JoeJulian warci: Red Hat's selling a product for which they bought people that could support ceph. As a paid product, it's in their best interest to sell that support. In most tests that I've heard about, gluster performs better than ceph if you're using libgfapi.
18:49 warci Thanks for a very clear answer JoeJulian :)
18:49 warci tbh, i really like gluster, so i'm a bit partial, i really want to get it to work
18:50 warci you've restored my hope
18:50 JoeJulian I've been a Gluster supporter for years, so I am too. But my $dayjob has me learning ceph so I'm working very hard on becoming an expert at both so I can give balanced opinions.
18:52 warci nice... it's such a bummer, it's still early days for both technologies, but now i'm forced to choose one to demo, so we can use it in prod down the line
18:52 B21956 joined #gluster
18:54 JoeJulian They each have compelling features, but the ones that are unique to ceph are in the process of being developed in GlusterFS, so for today and looking toward the next two years, I'd choose gluster.
18:56 warci yeah, tbh i like the fundamental concept of gluster a lot more, and i don't see why they can't provide the features that make ceph compelling atm
18:56 warci and gluster now already runs my production quite smoothly, so i have faith :)
18:57 nszceta nice
18:57 warci anyway, gotta run for now, but many thanks for your input
18:57 JoeJulian any time
18:59 rafi1 joined #gluster
19:04 jiffin joined #gluster
19:11 sputnik13 joined #gluster
19:21 jhc76 warci: I love gluster. it works seemlessly with ctdb/samba vfs. it really kicks lama's ass!
19:27 skippy what is the default value of cluster.server-quorum-type?  If I haven't explicitly set it, how do I see what it's value is?
19:34 sputnik13 joined #gluster
19:37 semiosis jhc76: s/kicks/whips/
19:37 JoeJulian skippy: gluster volume set help
19:38 jhc76 semiosis: it's been awhile since I had winamp.
19:38 jhc76 memory gets bit fuzzy
19:38 semiosis yeah, winamp jumped the shark
19:39 jhc76 at least I got the context right :D
19:39 skippy thanks JoeJulian !
19:39 jhc76 instead of saying s/whips/rapes that'll be bad
19:40 semiosis haha
19:42 nszceta I miss Winamp
19:43 semiosis i miss milkdrop
19:44 jhc76 I used stare at winamp's visualization for hours. I wasn't even on anything.
19:46 jhc76 https://www.youtube.com/watch?v=QFZBTYFTeto
19:46 glusterbot Title: Milkdrop 2.0 Winamp Visualizations - YouTube (at www.youtube.com)
19:56 skippy do folks here recommend nobarriers on bricks hosted on hardware RAID?
20:01 glusterbot New news from resolvedglusterbugs: [Bug 859248] Mount fails when the Gluster Server has an IPv4 and IPv6 address <https://bugzilla.redhat.com/show_bug.cgi?id=859248>
20:13 ricky-ticky joined #gluster
20:18 MrAbaddon joined #gluster
20:31 elico joined #gluster
20:37 MugginsM joined #gluster
20:43 Paul-C joined #gluster
20:49 _br_ joined #gluster
20:58 calisto joined #gluster
20:59 sputnik13 joined #gluster
21:01 coredump joined #gluster
21:03 sputnik13 joined #gluster
21:05 SOLDIERz joined #gluster
21:06 _br_ joined #gluster
21:14 jhc76 #rsync
21:16 kr0w joined #gluster
21:17 kr0w Anyone on that could help me resolve a replace-brick error I am getting? I would be fine with just aborting it but it won't let me
21:17 sickness rsync++
21:17 glusterbot sickness: rsync's karma is now 1
21:17 sickness eheh :)
21:26 mator joined #gluster
21:28 chirino joined #gluster
21:36 yoavz Hi, I have 2 servers serving the same volume (2 replicas). What's the best way to mount this volume on a third server in a way that'll be resilient to failures of any specific node of the two?
21:38 semiosis the fuse client does HA automatically.  ,,(mount srever)
21:38 glusterbot I do not know about 'mount srever', but I do know about these similar topics: 'mount server'
21:38 semiosis ,,(mount server)
21:38 glusterbot (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.
21:47 yoavz semiosis, I just got it working with this fstab line: 192.168.70.41:gluster_covers /mnt/covers glusterfs defaults,_netdev,backupvol​file-server=192.168.70.42 0 0
21:47 yoavz checked failover and it works
21:47 yoavz any reason why it shouldn't work?
21:50 semiosis magic
21:51 semiosis what distro are you on?
21:51 semiosis does that backupvolfile-server even work?  where'd you get that from?
21:52 sputnik13 joined #gluster
21:53 kr0w Can anyone help me with a replace-brick error? It seems very similar to rz___ in this log, but I can't see what the fix was: http://irclog.perlgeek.de/gluster/2012-10-05
21:53 glusterbot Title: IRC log for #gluster, 2012-10-05 (at irclog.perlgeek.de)
21:53 badone joined #gluster
21:53 velladecin joined #gluster
21:53 kr0w JoeJulian: was the one to help him.
21:54 ricky-ticky1 joined #gluster
21:54 Egidijus joined #gluster
21:54 ildefonso joined #gluster
21:55 yoavz semiosis, CentOS 6.5
21:56 sputnik13 joined #gluster
21:59 JoeJulian kr0w: looks like rz never said whether or not that fixed it.
21:59 kr0w Ah
21:59 JoeJulian kr0w: As you could see from that, we postulated that they key was removing the ,,(extended attributes) related to replace-brick from the brick root.
21:59 glusterbot kr0w: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
22:00 kr0w Well, I did kill all the servers as you said there. But it has not fixed it for me. I was trying to do a strange setup as described here: http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/. But I have decided that it is not a good setup for us and just want to abort it
22:00 ildefonso hi all! has anybody observed files "coming back from the dead", ie: reappearing after being deleted?
22:00 JoeJulian ildefonso: depends where you're deleting them from.
22:00 ildefonso using gluster client (mount -t glusterfs ...)
22:01 kr0w JoeJulian: yes I looked there and there didn't appear to be any attrs out of the ordinary as in his pasty
22:01 kr0w pastie*
22:01 JoeJulian hrm... check for something related in /var/lib/glusterd/vols/$volume_name
22:01 kr0w Ok
22:02 sputnik13 joined #gluster
22:03 kr0w JoeJulian: getfattr didn't pull anything on that dir. The new brick isn't listed there either.
22:03 JoeJulian anything?
22:03 JoeJulian Are you doing it as root?
22:04 kr0w I am
22:04 kr0w It gave a blank response
22:04 JoeJulian oh, you're saying to xattrs in /var/lib/glusterd/vols/$volume_name... No, I mean anything in the files there. I don't know what those files do if the replace-brick fails.
22:04 kr0w [root@proxmox1 15:02:44 ~]#getfattr -m . -d -e hex /var/lib/glusterd/vols/ha_gluster/
22:04 ildefonso JoeJulian, details: 3 test servers (server1, server2, server3), created one brick on each server, created a volume with "replica 3", and the three bricks.
22:04 kr0w [root@proxmox1 15:04:16 ~]#
22:04 kr0w I see
22:05 kr0w There is no information in there about the brick I was trying to replace. but let me dig a little more
22:05 ildefonso JoeJulian, on server1, mounted the volume from self (mount -t glusterfs server1:/test1 /mnt/gluster), of course, brick is on another directory, /gluster/brick1/
22:05 ildefonso went there, created a new file (touch file1)
22:05 ildefonso shutdown both other two servers (server2 and server3)
22:06 ildefonso then deleted the file
22:06 ildefonso after that, shutdown server1 too (all servers down)
22:06 ildefonso started server2 and server3 (no server1)
22:06 ildefonso mounted glusterfs on server2 from self (mount -t glusterfs server2:/test1 /mnt/)
22:06 kr0w JoeJulian: there is an rbstate file in there. It shows the src, destination, and status (1) of the replace-brick command it appears.
22:06 ildefonso created a new file, file2
22:06 ildefonso started server1... and file1 came back!
22:07 ildefonso the file will remain deleted if I don't create file2, and the change will get properly replicated to server2 and server3 when server1 comes back.
22:07 JoeJulian kr0w: With all glusterd's stopped, I'd mv that file to /tmp on all the servers and start glusterd again.
22:07 sage_ joined #gluster
22:08 ildefonso however, it looks like creating file2 has some effect, and makes file1 deletion change to be ignored or something.
22:09 gburiticato joined #gluster
22:09 JoeJulian ildefonso: interesting. I assume that when you create the file in a split-brain directory situation, the two agreeing servers beat the odd-man-out.
22:09 rastar_afk joined #gluster
22:09 JoeJulian I'd rather have that happen than data loss, of course.
22:09 JoeJulian File a bug report.
22:09 kr0w JoeJulian: Before I do that, there is also this in the vol file for the server I was doing the replace on: http://pastebin.com/sDC2tTzV
22:09 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
22:10 ildefonso JoeJulian, no, the file was created with all servers up.
22:10 ildefonso it was deleted with two servers down, and one up.
22:10 ildefonso the last file (file2) was created with two up, and one down.
22:10 ildefonso but it made file1 come back.
22:10 JoeJulian yeah, I got that. Classic split-brain-over-time.
22:11 kr0w JoeJulian: here is the fpaste one: http://fpaste.org/151583/14162622/
22:11 glusterbot Title: #151583 Fedora Project Pastebin (at fpaste.org)
22:11 velladecin joined #gluster
22:12 ildefonso JoeJulian, ok, will create a bug report, thanks!
22:13 troublesome joined #gluster
22:13 sputnik13 joined #gluster
22:14 JoeJulian kr0w: Back up that directory. Delete all the .vol files and the rbstate. Start glusterd on just that one server.
22:14 JoeJulian See if it recreates the vol files.
22:14 JoeJulian I think it's supposed to.
22:14 kr0w Ok
22:14 Egidijus hello all, i have two gluster servers with a brick on each of them, and a client (with gluster client installed), i can mount the storage from the client via this "mount.glusterfs gluster01:/glustervol /mnt/glusterfs" but i have no luck mounting the storage via .vol file from the client. please help
22:14 Egidijus http://fpaste.org/151585/62624771/
22:15 glusterbot Title: #151585 Fedora Project Pastebin (at fpaste.org)
22:15 JoeJulian Mounting from vol files hasn't been supported since 3.0.
22:15 Egidijus :O
22:15 Egidijus i am on 3.5.1
22:15 Egidijus okay
22:16 Egidijus could somebody please point me in the direction of fault tolerant mounting?
22:16 JoeJulian @rrdns
22:16 glusterbot JoeJulian: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
22:16 Egidijus okay
22:16 Egidijus thank you
22:17 ildefonso ok, that was another question that was crawling in my mind.
22:17 * ildefonso takes note of that blog post.
22:19 JoeJulian ildefonso: not the entire blog. I published a lot of things that I've learned here while answering questions.
22:20 JoeJulian s/not/note/
22:20 glusterbot What JoeJulian meant to say was: ildefonso: note the entire blog. I published a lot of things that I've learned here while answering questions.
22:23 kr0w JoeJulian: Well, I dont seem to be getting the errors I was before. But it still isn't letting me add a brick to the volume
22:24 JoeJulian kr0w: So that worked on one server. Make sure the other servers have an exact copy of that vol directory. They have to be in sync or no volume changes will be accepted.
22:24 kr0w JoeJulian: Ok
22:24 JoeJulian and glusterd should not be running when
22:24 JoeJulian manipulating the /var/lib/glusterd tree
22:26 kr0w Right
22:26 kr0w On the server where I am manipulating ti
22:30 kr0w JoeJulian: I think I know why the command is failing. ha_gluster volume exists on the third server. The third server is where the new brick is coming from.
22:30 kr0w But it isn't in the volume info
22:33 glusterbot New news from newglusterbugs: [Bug 1021998] nfs mount via symbolic link does not work <https://bugzilla.redhat.co​m/show_bug.cgi?id=1021998>
22:34 sickness so anyone tried glusterfs server brick on cygwin? =_)
22:36 Egidijus hi all
22:37 Egidijus instead of rrdns, could i use virtual IP load balancing for fault tolerant mounting?
22:38 kr0w JoeJulian: This may be noob, but I can't seem to remove that volume on that server
22:39 JoeJulian Like I say, if /var/lib/glusterd/vols is not in sync among all the servers, no volume operations will succeed. There should be an error in one of the glusterd logs that matches what I'm saying.
22:39 JoeJulian Egidijus: no point. It won't load balance and it would just be another thing to break.
22:39 kr0w JoeJulian: My bad. I got it added once I removed the fattrs. (It already had ones linking it to a volume). Thanks JoeJulian!
22:40 JoeJulian Yay
22:40 Egidijus sigh
22:40 JoeJulian sickness: I think I heard of someone doing that. Some things won't work without fuse though.
22:41 JoeJulian Egidijus: There's also a mount option for choosing other fail-over servers: backup-volfile-servers
22:43 Egidijus thank you!
22:44 hamcube joined #gluster
22:50 jmarley joined #gluster
22:53 andreask joined #gluster
22:59 sickness JoeJulian: even if you just want the server brick running? (I want the server not the client on win32)
23:00 JoeJulian sickness: yes. the server uses the client for various operations, self-heal, rebalance, etc.
23:01 sputnik13 joined #gluster
23:01 JoeJulian If you absolutely have to run on a windows machine, there's always virtualbox or hyper-v...
23:02 sickness oh, now I understand...
23:02 sickness nah, just wanted to somewhat use the vastly unused space on win32 workstations disks :P
23:03 sickness apart from win32
23:03 sickness did someone succeed to build glusterfs on openbsd? =_)
23:08 semiosis sickness: there's an effort to run it on netbsd & os x
23:08 semiosis sickness: havent heard much about openbsd though
23:11 Paul-C left #gluster
23:21 joevartuli joined #gluster
23:23 joevartuli left #gluster
23:32 if-kenn joined #gluster
23:48 sputnik13 joined #gluster
23:53 chirino joined #gluster
23:55 sickness semiosis: tnx

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary