Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-12-03

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 tessier_ Do the gluster rpms from gluster.org not provide an init script or something to start gluster?
00:05 tessier_ Ah, it did on my CentOS 6 machine but not on my CentOS 5 machine.
00:07 JoeJulian It didn't on CentOS 5???
00:08 tessier_ Doh. Nevermind. It failed to install at all on this machine due to gpg issues.
00:08 tessier_ error: glusterfs-server-3.6.1-4.el5: Header V4 RSA/SHA1 signature: BAD, key ID 4ab22bb3
00:08 tessier_ Ah...not gpg issues. A hash issue. I've seen this before. Something changed between 5 and 6 regarding rpm hashes.
00:10 JoeJulian yes
00:11 JoeJulian I wonder who's building el5. I'll find out and let them know about that.
00:11 semiosis must be nice to have so many people building packages for your distro
00:12 semiosis dont take that for granted :)
00:15 JoeJulian Hehe
00:15 JoeJulian I'm going to start building archlinux packages...
00:16 JoeJulian Whoever is doing it now is behind and doesn't answer emails.
00:19 gildub joined #gluster
00:35 glusterbot News from newglusterbugs: [Bug 1169999] Debian 7 wheezy, /sbin/mount.glusterfs fails to automount and print help <https://bugzilla.redhat.co​m/show_bug.cgi?id=1169999>
00:43 tessier_ hmm... volume create: glustertest: failed: /export/glustertest/brick is already part of a volume
00:43 JoeJulian @path or prefix
00:43 glusterbot JoeJulian: http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
00:43 tessier_ The first time I created it I had /export/glustertest/brick on / on one of my systems by accident because I forgot to mount the intended disk at /export
00:44 tessier_ JoeJulian: Shouldn't I also just be able to delete the volume at this point since it has nothing in it? But when I try that I get: volume delete: glustertest: failed: Volume glustertest does not exist
00:47 JoeJulian Sounds like you already deleted the volume. These are safeguards that prevent you from creating a broken volume. They're crappy safeguards with no UI for overriding them.
00:47 tessier_ ah....that wasn't exactly the issue. I didn't create the brick dir in the newly mounted volume.
00:47 tessier_ volume create: glustertest: success: please start the volume to access data
00:47 tessier_ success! :)
00:47 JoeJulian If it were me at the stage you're currently at, I would unmount my bricks, re-mkfs them, and mount them again.
00:48 JoeJulian Or that
00:51 tessier_ hm...so I'm at the end of http://www.gluster.org/community/documen​tation/index.php/Getting_started_wrap_up and I have two bricks setup on two different machines and a volume started...so how do I put some test data into the volume from my third machine which is part of the cluster but does not contain the bricks?
00:54 * tessier_ reads the User Guide about setting up glusterfs client, that seems to be what I need next.
00:57 tessier_ Success! :)
01:08 topshare joined #gluster
01:18 FrankLu joined #gluster
01:27 bala joined #gluster
01:34 tessier_ Hmm...So I have 5 pairs of identical disk servers (10 total) for hosting bricks with varying numbers of disks in them. Some pairs have different performance profiles such as having huge 7200 RPM disks while others have smaller 15k rpm disks. Presumably I should make each of these different classes of disk different gluster volumes? If I make each physical disk in a server a brick presumably gluster is smart enough to not only replicate to differen
01:38 FrankLu The used size from quota list command is empty when quota feature is enabling
01:38 y4m4 joined #gluster
01:38 FrankLu How could I re-collect the used size ?
01:38 FrankLu Just disable it first then re-enable it?
02:08 ira joined #gluster
02:14 haomaiwa_ joined #gluster
02:48 calisto joined #gluster
03:22 RameshN joined #gluster
03:30 meghanam_ joined #gluster
03:31 meghanam joined #gluster
03:42 RameshN joined #gluster
03:54 maveric_amitc_ joined #gluster
03:55 kanagaraj joined #gluster
03:57 unwastable joined #gluster
03:58 unwastable can someone help me to understand, why can't gluster work with NFSv2?
03:58 bala joined #gluster
04:00 shubhendu joined #gluster
04:00 atinmu joined #gluster
04:06 itisravi joined #gluster
04:06 kdhananjay joined #gluster
04:14 RameshN joined #gluster
04:17 nbalacha joined #gluster
04:24 ryao joined #gluster
04:28 JoeJulian ~brick order | tessier_
04:28 glusterbot tessier_: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
04:33 JoeJulian unwastable: NFSv2 support was never written. I haven't even heard an interest in it existing until now.
04:34 kumar joined #gluster
04:35 nishanth joined #gluster
04:36 bala joined #gluster
04:37 anoopcs joined #gluster
04:39 rjoseph joined #gluster
04:41 rafi1 joined #gluster
04:54 cleo_ joined #gluster
04:56 cleo_ could anyone tell me how to get resources infornation like cpu, memory ...from glusterfs??
04:57 cleo_ is there specific glusterfs commands?
05:10 ppai joined #gluster
05:22 shubhendu joined #gluster
05:26 maveric_amitc_ joined #gluster
05:34 hagarth joined #gluster
05:38 DV joined #gluster
05:39 kdhananjay joined #gluster
05:40 atinmu joined #gluster
05:46 ramteid joined #gluster
05:48 XpineX joined #gluster
05:49 poornimag joined #gluster
05:56 saurabh joined #gluster
06:03 aravindavk joined #gluster
06:05 soumya joined #gluster
06:11 overclk joined #gluster
06:20 dusmant joined #gluster
06:22 rotbeard joined #gluster
06:23 Philambdo joined #gluster
06:33 nbalacha joined #gluster
06:39 atalur joined #gluster
06:47 raghu` joined #gluster
06:55 ctria joined #gluster
06:56 nshaikh joined #gluster
07:04 jiku joined #gluster
07:06 glusterbot News from resolvedglusterbugs: [Bug 1049727] Dist-geo-rep : volume won't be able to stop untill the geo-rep session is deleted. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1049727>
07:15 SOLDIERz_ joined #gluster
07:15 anoopcs joined #gluster
07:15 DV joined #gluster
07:24 jiffin joined #gluster
07:27 nbalacha joined #gluster
07:38 ghenry joined #gluster
07:38 ghenry joined #gluster
07:38 DV joined #gluster
07:44 LebedevRI joined #gluster
07:45 hagarth joined #gluster
07:47 Fen1 joined #gluster
07:49 RameshN joined #gluster
07:52 ricky-ticky1 joined #gluster
07:55 jaank joined #gluster
08:17 LebedevRI joined #gluster
08:21 T0aD joined #gluster
08:24 kovshenin joined #gluster
08:31 bala joined #gluster
08:36 glusterbot News from newglusterbugs: [Bug 1170075] [RFE] : BitRot detection in glusterfs <https://bugzilla.redhat.co​m/show_bug.cgi?id=1170075>
08:38 cmtime joined #gluster
08:41 Slashman joined #gluster
08:42 liquidat joined #gluster
08:42 cristov_mac joined #gluster
08:42 vimal joined #gluster
08:42 topshare joined #gluster
08:43 R0ok_ joined #gluster
08:46 cristov__ joined #gluster
08:49 dusmant joined #gluster
08:50 hagarth joined #gluster
08:54 LebedevRI joined #gluster
08:59 meghanam joined #gluster
08:59 meghanam_ joined #gluster
09:13 Slashman joined #gluster
09:15 Slashman joined #gluster
09:16 ninkotech joined #gluster
09:16 ninkotech_ joined #gluster
09:29 SOLDIERz__ joined #gluster
09:40 calum_ joined #gluster
09:41 Fen1 Hi ! :) Can we install Red Hat Storage Console on CentOS 7 to be used with glusterFS ?
09:42 ndevos Fen1: you probably should install the oVirt version of the RHS-Console
09:43 Fen1 i don't want to use ovirt
09:44 atalur joined #gluster
09:44 Fen1 i will use proxmox for virtualization and installing ovirt will be confusing i think
09:52 nbalacha joined #gluster
09:56 _shaps_ joined #gluster
09:56 ndevos Fen1: RHS-Console is oVirt, you want the oVirt/storage component, or whatever it is called
10:01 harish joined #gluster
10:06 dusmant joined #gluster
10:10 hagarth joined #gluster
10:10 drankis joined #gluster
10:10 soumya joined #gluster
10:11 DV joined #gluster
10:18 ndevos Fen1: I think ovirt-engine is the part you need, it should have a 'storage only' mode
10:19 Fen1 ndevos : great thx ndevos ! :)
10:20 Anuradha joined #gluster
10:22 ndevos Fen1: ah, since oVirt-3.2 the option is there, see http://www.ovirt.org/OVirt_​3.2_release_notes#Installer
10:28 elico joined #gluster
10:30 aravindavk joined #gluster
10:38 harish joined #gluster
10:44 frankS2 joined #gluster
10:45 calum_ joined #gluster
10:45 harish joined #gluster
10:49 meghanam_ joined #gluster
10:49 meghanam joined #gluster
10:49 Anuradha joined #gluster
11:06 social joined #gluster
11:10 thb_ joined #gluster
11:12 thb_ Hi, can I bother anybody with my weird 8-node distributed/replicated gluster problems?
11:14 diegows joined #gluster
11:24 plarsen joined #gluster
11:27 ndevos thb_: just ask your question, maybe someone can help you out
11:29 sharknardo joined #gluster
11:32 XpineX joined #gluster
11:32 thb_ I've got one node that's much higher cpu load than the rest
11:33 thb_ While seven more or less cruise along at 10-20% usage, this one goes up to 100% on all 12 cores under load.
11:33 atalur joined #gluster
11:35 thb_ Peer status seems o.k. though
11:35 calisto joined #gluster
11:36 thb_ According to "gluster volume profile" that one node has ~60% of it's latency in FXATTROP
11:36 meghanam joined #gluster
11:36 meghanam_ joined #gluster
11:40 thb_ The underlying raid seems to be o.k., though.
11:40 thb_ Any chance that I could trace what's causing this?
11:47 saurabh joined #gluster
11:55 sharknardo joined #gluster
11:57 aravindavk joined #gluster
11:59 jdarcy joined #gluster
12:00 plarsen joined #gluster
12:00 topshare joined #gluster
12:02 bene2 joined #gluster
12:04 feeshon joined #gluster
12:09 SOLDIERz__ joined #gluster
12:12 aravindavk joined #gluster
12:18 julim joined #gluster
12:24 DJClean joined #gluster
12:29 itisravi_ joined #gluster
12:32 anoopcs joined #gluster
12:33 edwardm61 joined #gluster
12:33 lpabon joined #gluster
12:35 kshlm joined #gluster
12:49 delhage joined #gluster
12:50 RameshN joined #gluster
13:00 ctria joined #gluster
13:02 B21956 joined #gluster
13:02 bennyturns joined #gluster
13:03 calisto joined #gluster
13:03 misko_ semiosis: any success regarding qemu-block.so? thanks
13:04 Fen1 joined #gluster
13:06 B21956 left #gluster
13:08 hagarth joined #gluster
13:08 kanagaraj joined #gluster
13:12 dusmant joined #gluster
13:15 ppai joined #gluster
13:40 SOLDIERz__ joined #gluster
13:45 topshare joined #gluster
13:46 nbalacha joined #gluster
13:55 SOLDIERz__ joined #gluster
13:59 tdasilva joined #gluster
13:59 hagarth joined #gluster
14:12 virusuy joined #gluster
14:14 johnnytran joined #gluster
14:24 mator joined #gluster
14:26 ctria joined #gluster
14:30 tdasilva joined #gluster
14:34 calisto joined #gluster
14:39 kshlm joined #gluster
14:45 ricky-ticky joined #gluster
14:47 Fen2 joined #gluster
14:49 asku joined #gluster
14:57 bene joined #gluster
14:57 Fen1 joined #gluster
14:59 ricky-ticky joined #gluster
15:01 prasanth_ joined #gluster
15:04 jmarley joined #gluster
15:10 shubhendu joined #gluster
15:13 [Enrico] joined #gluster
15:13 HIve1 joined #gluster
15:14 kovshenin joined #gluster
15:16 HIve1 hello having an issue with a rebalance on 3.4.3 it is completed or says in the log it is completed but it is still running and I cannot stop it. http://pastebin.com/BXEL5yGb
15:16 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
15:16 HIve1 http://fpaste.org/156232/14176197/
15:17 kovshenin joined #gluster
15:17 tdasilva joined #gluster
15:18 [Enrico] joined #gluster
15:21 wushudoin joined #gluster
15:21 jobewan joined #gluster
15:24 hagarth joined #gluster
15:29 Slashman joined #gluster
15:31 johnnytran joined #gluster
15:38 glusterbot News from newglusterbugs: [Bug 1170254] Fix mutex problems reported by coverity scan <https://bugzilla.redhat.co​m/show_bug.cgi?id=1170254>
15:40 bala joined #gluster
15:40 Telsin joined #gluster
15:43 jobewan joined #gluster
15:45 lmickh joined #gluster
15:51 meghanam joined #gluster
15:51 meghanam_ joined #gluster
15:57 Telsin left #gluster
16:00 jcsp_ joined #gluster
16:01 jcsp_ left #gluster
16:01 soumya joined #gluster
16:03 prasanth_ joined #gluster
16:07 johnnytran joined #gluster
16:17 bene joined #gluster
16:27 coredump has anyone encountered errors like this https://gist.github.com/cor​edump/aacbdaf8873c244e22fb
16:34 rwheeler joined #gluster
16:35 chirino joined #gluster
16:36 feeshon joined #gluster
16:39 kanagaraj joined #gluster
16:39 harish joined #gluster
16:41 maveric_amitc_ joined #gluster
16:43 kovshenin joined #gluster
16:44 kaushal_ joined #gluster
16:50 chirino joined #gluster
17:04 chirino joined #gluster
17:08 PeterA joined #gluster
17:14 harish joined #gluster
17:24 kovshenin joined #gluster
17:36 rohit joined #gluster
17:36 _Bryan_ joined #gluster
17:36 rohit left #gluster
17:46 nshaikh joined #gluster
17:58 ultrabizweb joined #gluster
18:14 diegows joined #gluster
18:20 daMaestro joined #gluster
18:29 chirino joined #gluster
18:57 Hive2 joined #gluster
18:57 free_amitc_ joined #gluster
19:01 ricky-ticky joined #gluster
19:34 rwheeler joined #gluster
19:39 m0ellemeister joined #gluster
19:54 _dist joined #gluster
20:09 nshaikh left #gluster
20:14 PeterA a brick keep crashing on 3.5.2....
20:14 PeterA any clue?
20:14 PeterA http://pastie.org/9758925
20:15 PeterA when tried to do the gluster volume start <volume> force
20:15 PeterA the brick keep throwing out this error
20:15 PeterA this started happen 30mins ago….been running smooth for last 3 months
20:16 chirino joined #gluster
20:17 JoeJulian coredump: There are no errors in that gist.
20:18 JoeJulian Same with your paste, PeterA.
20:18 rotbeard joined #gluster
20:18 JoeJulian The crash log might be more telling.
20:18 diegows joined #gluster
20:19 PeterA http://pastie.org/9758938
20:19 PeterA brick log?
20:20 DV joined #gluster
20:20 PeterA noticed this:
20:20 PeterA [2014-12-03 20:15:46.590692] W [socket.c:522:__socket_rwv] 0-sas03-quota: readv on /tmp/quotad.socket failed (No data available)
20:21 PeterA [2014-12-03 20:15:47.655796] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-sas03-quota: failed to submit rpc-request (XID: 0x1 Program: Quota enforcer, ProgVers: 1, Proc: 1) to rpc-transport (sas03-quota)
20:21 PeterA [2014-12-03 20:15:47.655813] W [quota-enforcer-client.c:187​:quota_enforcer_lookup_cbk] 0-sas03-quota: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
20:22 PeterA JoeJulian: are you talking about the brick log or which crash log?
20:23 JoeJulian That looks like the quota daemon is not running
20:23 PeterA root     21678     1  0 12:19 ?        00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/e31b90aa87baf489abd4f59b002c6fa0.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off
20:23 PeterA root     23937 19591  0 12:23 pts/2    00:00:00 grep --color=auto quota
20:23 PeterA but it is.....
20:24 PeterA i already tried to bounced glusterfs-server
20:24 PeterA let me do it again
20:24 JoeJulian As for what log, I was basing that on "a brick keep crashing on 3.5.2...." so I thought there was a crash.
20:24 PeterA it keep dying
20:25 JoeJulian stop glusterfs-server, kill 21678, then start it again.
20:25 JoeJulian I'm not sure that stopping glusterfs-server stops the the self-heal, nfs, or quota daemons.
20:26 PeterA i just did
20:26 PeterA but that brick still not starting :(
20:26 PeterA same warnings from brick log and that brick just die....
20:27 JoeJulian Hang on... "option transport.socket.connect-path /tmp/quotad.socket" but it looks like it should be "/var/run/e31b90aa87baf489abd4f59b002c6fa0.socket"
20:28 PeterA ??
20:28 JoeJulian Is there even a /tmp/quotad.socket ?
20:29 PeterA yes
20:29 PeterA root@glusterprod006:/tmp# ls -l quotad.socket
20:29 PeterA srwxr-xr-x 1 root root 0 Dec  3 12:25 quotad.socket
20:29 JoeJulian hmm, ok..
20:29 JoeJulian I've never looked at the quota code, and it's changed a lot from little I did know...
20:30 PeterA we have other volume with quota which working fine
20:30 PeterA only one brick not able to start up…
20:30 PeterA happen to be the prod one :(
20:30 JoeJulian Of course...
20:30 PeterA what should i do ?
20:31 JoeJulian I'm looking...
20:31 PeterA i tried the service stop then pkill gluster then start…no lunch...
20:31 PeterA no luck
20:31 PeterA nor lunch
20:33 JoeJulian What about the quotad log? Any clue in there?
20:33 PeterA [2014-12-03 20:33:46.068082] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-sas03-client-5: changing port to 49154 (from 0)
20:33 PeterA [2014-12-03 20:33:46.068539] E [socket.c:2161:socket_connect_finish] 0-sas03-client-5: connection to 10.40.12.78:49154 failed (Connection refused)
20:34 coredump JoeJulian: right, just warnings, but still a permission denied error
20:35 PeterA somehting messed up on rpc?
20:36 JoeJulian What's 10.40.12.78 ?
20:36 JoeJulian The machine that's failing, or a different one?
20:36 PeterA that's the ip of the node
20:36 PeterA yes the physical node ip
20:36 PeterA of the gluster node
20:37 JoeJulian You're just trying to make me have a conniption fit, aren't you...
20:37 PeterA sory
20:39 PeterA wonder what could cause the connection refuse
20:40 jackdpeterson joined #gluster
20:41 JoeJulian Oh... Try stopping glusterfs-server, killing the process for quotad, deleting /tmp/quotad.socket, and trying again.
20:41 jackdpeterson is GlusterFS v 3.5.2 compatible with cachefilesd when clients are mounting glusterFS via NFSv3?
20:41 PeterA ok
20:41 JoeJulian I don't know anything about cachefilesd, but I do know that the kernel FSCache does get used with NFS mounts.
20:43 PeterA just did….still no luck :(
20:45 jackdpeterson I guess my question is directed at the general idea of fiddling with those xattrs -- has anyone had any major blowups as a result of implementing FSCache/cachefilesd
20:45 PeterA still getting quotad connection refused :(
20:46 semiosis jackdpeterson: https://botbot.me/freenode/g​luster/search/?q=cachefilesd
20:47 jackdpeterson @semiosis -- TY!
20:48 semiosis yw!
20:48 PeterA help my brick still down :(
20:48 semiosis jackdpeterson: there's not much there, but maybe it's helpful to you
20:48 PeterA googling and seems someone hit the same?
20:48 PeterA http://gluster.org/pipermail/glust​er-users.old/2014-June/017574.html
20:49 jackdpeterson `tis all good -- Slydder's comment on 2014-09-11 actually gives me hope :-)
20:49 semiosis PeterA: iptables
20:49 PeterA iptables off
20:49 PeterA the system been running smooth for 3months
20:49 PeterA just have this happened...
20:49 JoeJulian bug 1089470
20:49 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1089470 urgent, urgent, ---, kdhananj, CLOSED CURRENTRELEASE, SMB: Crash on brick process during compile kernel.
20:49 semiosis if you only knew how many times people say 'iptables off' then later it turns out iptables NOT off
20:50 semiosis what causes connection refused?  iptables set to REJECT, or no process bound to port, or ip conflict
20:51 JoeJulian semiosis: unix socket... /tmp/quotad.socket
20:51 semiosis oh i was looking at [15:33] <PeterA> [2014-12-03 20:33:46.068539] E [socket.c:2161:socket_connect_finish] 0-sas03-client-5: connection to 10.40.12.78:49154 failed (Connection refused)
20:51 JoeJulian I don't know how, with the listening daemon running and attacked to the unix socket, it's even possible to get a connection refused error.
20:51 JoeJulian s/attacked/attached/
20:51 glusterbot What JoeJulian meant to say was: An error has occurred and has been logged. Check the logs for more informations.
20:52 PeterA ufw status
20:52 PeterA Status: inactive
20:52 JoeJulian "iptables -S" is how I make sure.
20:53 PeterA iptables -S
20:53 PeterA -P INPUT ACCEPT
20:53 PeterA -P FORWARD ACCEPT
20:53 PeterA -P OUTPUT ACCEPT
20:53 semiosis ok ok so maybe not iptables ;)
20:54 PeterA so what would be our next clue ?:)
20:55 JoeJulian lsof -p $(cat /var/lib/glusterd/quotad/run/quotad.pid)
20:55 JoeJulian Look for /tmp/quotad.socket
20:55 PeterA glusterfs 12060 root    7u  unix 0xffff880fd6cd7b00      0t0 119498088 /tmp/quotad.socket
20:56 JoeJulian As an emergency solution, disable quotas?
20:56 PeterA root@glusterprod006:/var/log/glusterfs# lsof -p $(cat /var/lib/glusterd/quotad/run/quotad.pid) | grep socket
20:56 PeterA glusterfs 12060 root  mem    REG                8,3    80568   1448338 /usr/lib/x86_64-linux-gnu/gluster​fs/3.5.2/rpc-transport/socket.so
20:56 PeterA glusterfs 12060 root    6u  unix 0xffff880fd8501800      0t0 119475726 /var/run/e31b90aa87baf489abd4f59b002c6fa0.socket
20:56 PeterA glusterfs 12060 root    7u  unix 0xffff880fd6cd7b00      0t0 119498088 /tmp/quotad.socket
20:56 PeterA glusterfs 12060 root    9u  unix 0xffff881fd8bd6900      0t0 119498089 /tmp/quotad.socket
20:56 PeterA glusterfs 12060 root   10u  unix 0xffff880fd5e8aa00      0t0 119498018 /tmp/quotad.socket
20:56 PeterA glusterfs 12060 root   36u  unix 0xffff881fd6eaec00      0t0 119498091 /var/run/e31b90aa87baf489abd4f59b002c6fa0.socket
20:56 PeterA glusterfs 12060 root   37u  unix 0xffff881fd6eae600      0t0 119498092 /tmp/quotad.socket
20:56 PeterA we have tried the disable quota process takes forever
20:56 PeterA and hang the volume
20:56 PeterA as the amount of files
20:57 JoeJulian paste /var/log/glusterfs/quotad.log
20:58 PeterA very large now....
20:58 JoeJulian ok, well stop glusterfs-server, kill quotad, truncate the log, then start it again and cause the failure.
20:59 calisto joined #gluster
21:01 semiosis don't paste in channel.  use pastie.org, gist.github.com, etc
21:01 PeterA https://gist.github.com/ano​nymous/8b92f16c17b2df7ff149
21:01 calisto joined #gluster
21:04 PeterA line 588 seems the 1st warning related to the error
21:05 PeterA strange is why only quota fail only on that particular volume?
21:05 PeterA we have 4 volumes and only this one fail for quota…
21:13 PeterA semiosis and JoeJulian, still here?
21:18 JoeJulian Doing work, too...
21:19 PeterA thanks
21:19 PeterA also getting this
21:19 PeterA http://pastie.org/9759075
21:19 PeterA from etc-glusterd
21:22 _dist joined #gluster
21:23 jobewan joined #gluster
21:24 PeterA how come getting this
21:24 PeterA E [rpcsvc.c:547:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
21:24 PeterA seems like this?
21:24 PeterA https://bugzilla.redhat.co​m/show_bug.cgi?id=1161025
21:24 glusterbot Bug 1161025: high, unspecified, ---, kkeithle, ASSIGNED , Brick process crashed after failing to send a RPC-Reply, client_t related?
21:25 JoeJulian Do you have "crash" in any of your logs?
21:26 johnnytran joined #gluster
21:26 JoeJulian So far, what I'm seeing, is some failure to start, resulting in the brick shutting down. But if there's the string, "crash" anywhere, that might be a clue.
21:26 PeterA yes
21:27 PeterA http://pastie.org/9759100
21:27 PeterA the brick log
21:28 PeterA said time of crash....
21:30 JoeJulian Ah fudge...
21:30 JoeJulian 1122120
21:30 JoeJulian bug 1122120
21:30 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1122120 urgent, unspecified, ---, bugs, MODIFIED , Bricks crashing after disable and re-enabled quota on a volume
21:30 andreask joined #gluster
21:31 PeterA http://pastie.org/9759111
21:32 PeterA the gdb core
21:32 JoeJulian The crash is happening in exactly the right place to match that bug.
21:33 PeterA except this time i did not disable or reenable the quota
21:33 PeterA fix on 3.5.3 ??/
21:33 JoeJulian That's what it says.
21:35 PeterA is that worthy to retry a reboot?
21:35 johnnytran joined #gluster
21:41 PeterA when i tried to do a volume start force, i am getting these from etc-gluster log
21:41 PeterA https://gist.github.com/ano​nymous/3206af832898952a5032
21:42 PeterA wondering why it keep complaint about the permission of the socket file
21:42 JoeJulian do you have any client mounts on the server?
21:42 PeterA clinet are mounting over nfs
21:43 PeterA gluster nfs
21:43 JoeJulian crap. Are any of the clients mounting from *this* server?
21:43 PeterA no
21:43 JoeJulian whew
21:43 PeterA :)
21:44 JoeJulian stop glusterfs-server ; pkill glusterfs ; start glusterfs-server
21:45 PeterA done
21:46 PeterA brick still crash after restart :(
21:48 PeterA same error
21:49 _Bryan_ joined #gluster
21:50 JoeJulian The crash still happens at mq_loc_fill_from_name+0x89
21:52 PeterA ??
21:55 daMaestro joined #gluster
21:55 calum_ joined #gluster
21:56 JoeJulian Sorry, yes. That was meant to have a ? at the end.
21:57 PeterA what does that mean? :)
21:57 PeterA sorry i am not familiar with the symbols
21:57 badone joined #gluster
21:57 JoeJulian You said is the same error. The error you posted had that string in it.
21:58 PeterA right ic
22:00 PeterA what else can we try at this point?
22:01 PeterA i checked the /var/run/ and the file permissions and looks the same as other nodes....
22:01 PeterA wonder why it keep getting the rpc connection refuse on the port
22:02 PeterA E [socket.c:2161:socket_connect_finish] 0-sas03-client-5: connection to 10.40.12.78:49154 failed (Connection refused)
22:07 JoeJulian because it crashed.
22:07 PeterA so brick crash cuz of quota not able to connect?
22:08 PeterA or quota not able to connect because the brick crashed?
22:11 JoeJulian I'm guessing the latter.
22:11 JoeJulian Don't forget... I didn't write any of this...
22:15 PeterA what should I do now?
22:17 PeterA is 3.5.3 out for ubuntu?
22:18 badone joined #gluster
22:20 tessier_ gluster for HA Maildir mailbox storage with millions of files: Good or bad idea?
22:21 tessier_ I understand this is often worst case performance scenarios for things like this.
22:24 tessier_ Looks like there will be some volume options I will want to tweak at the very least.
22:29 JoeJulian Oh, I thought you had already tried that, PeterA.
22:30 JoeJulian Is 3.5.3 out, semiosis?
22:30 PeterA i am on 3.5.2
22:30 JoeJulian tessier_: Right. Bad for performance.
22:30 semiosis for trusty, yes, https://launchpad.net/~gluster​/+archive/ubuntu/glusterfs-3.5
22:30 PeterA 3.5.3 still not appear for ubuntu
22:30 semiosis what release are you using PeterA?
22:30 PeterA precise
22:30 PeterA 12.04
22:30 semiosis ok
22:31 semiosis sorry for the delay
22:31 semiosis i need to publish that one
22:31 PeterA thanks
22:31 semiosis yw
22:31 PeterA i m running out of idea how to fix this
22:32 tessier_ JoeJulian: That is very unfortunate. I was hoping that maybe I had found the solution to my email scaling problems.
22:32 tessier_ Gluster may well solve other problems such as VM backing though. My work in that area is successful so far. Just lamenting that Xen won't talk directly to a file in gluster and instead insists on an iscsi target be setup and looped over localhost.
22:33 JoeJulian The solution to email scaling is, a new email backend that uses object stores and probably involves hadoop.
22:35 tessier_ JoeJulian: My thoughts exactly.
22:36 JoeJulian I'm actually really surprised that nobody has done that yet. Seems like a natural need.
22:36 tessier_ JoeJulian: Unfortunately, the imap servers and delivery agents need to be rewritten.
22:36 tessier_ Exactly.
22:36 badone joined #gluster
22:36 tessier_ You can't even interface existing imap/lda's through fuse because that's exactly the problem we already have with gluster.
22:37 JoeJulian imap would be, essentially, load balancers. They would query the database for the object lookup, then fetch the object from the store.
22:38 JoeJulian Additionally, an add-on feature that should then be added to the imap protocol is a way to pass the object url to the client and let the client and let the client fetch the object directly.
22:39 tessier_ If you want to partition based on username you can use something like perdition or nginx imap proxies and basically DHT it.
22:39 samppah afaik, dovecot has object storage plugin but it's commercial
22:39 JoeJulian yes
22:39 PeterA regarding to the brick crash….what else can I do besides waiting for 3.5.3?
22:40 JoeJulian Integrate it with sphynx for natural language searches and you'd have a hell of a distributable mail system.
22:40 tessier_ The Dovecot Object Storage PlugIns are available for:
22:40 tessier_ - Amazon S3
22:40 tessier_ - Windows Azure
22:40 tessier_ - Dropbox
22:40 tessier_ Not exactly what I have in mind. :)
22:40 JoeJulian semiosis: will probably have it built before you could do anything else to fix it.
22:41 JoeJulian Looks like it has something to do with mixing quotas with self-heal.
22:44 PeterA i am thinking if reboot would help :)
22:45 JoeJulian According to the details in the bug report, it won't help. You might also lose any other bricks that *are* up and running on that server.
23:01 gildub joined #gluster
23:05 PeterA wonder what would trigger the crash
23:05 PeterA as we can not even able to bring the brick back up
23:13 PeterA I noticed the glustershd on another node for the same volume stuck
23:13 PeterA so i bounced the servervice
23:13 PeterA and the brick came back up…
23:17 andreask joined #gluster
23:27 JoeJulian Oh, that's interesting.
23:27 JoeJulian Good job, PeterA.
23:34 PeterA ya interesting...
23:35 diegows joined #gluster
23:38 harish joined #gluster
23:40 daMaestro joined #gluster
23:45 PeterA now i have these heal-failed entries that can't get rid of....
23:46 JoeJulian They're just log entries. If you must clear them, restart all glusterd (and maybe glustershd).
23:46 PeterA how to restart glustershd only?
23:47 bala joined #gluster
23:49 JoeJulian pkill -f glustershd
23:49 JoeJulian then restart glusterfs-server
23:49 PeterA let me try
23:49 PeterA but should restart glusterfs-server will restart everything?
23:50 PeterA and even only one node reporting heal-failed
23:50 PeterA i have to restart all all node to clear the heal-failed entries?
23:58 PeterA do i have to restart glusterfs-server on all node at once or a rolling restart?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary