Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-06-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:52 nangthang joined #gluster
00:58 nangthang joined #gluster
01:03 harish joined #gluster
01:09 xavih joined #gluster
01:10 malevolent joined #gluster
01:37 ira joined #gluster
01:39 Gill joined #gluster
01:42 nangthang joined #gluster
01:51 lyang0 joined #gluster
02:25 tessier JoeJulian: I can make a tcp connection to port 24007. So it isn't blocked and glusterd is running. But when I do a gluster peer probe I still get: peer probe: failed: Probe returned with unknown errno 107
02:33 DV joined #gluster
02:53 hagarth joined #gluster
03:01 Folken joined #gluster
03:02 overclk joined #gluster
03:18 malevolent joined #gluster
03:18 xavih joined #gluster
03:47 shubhendu__ joined #gluster
03:47 spandit joined #gluster
03:54 maveric_amitc_ joined #gluster
03:55 bennyturns joined #gluster
03:57 itisravi joined #gluster
03:58 [7] joined #gluster
03:58 sakshi joined #gluster
04:01 atinmu joined #gluster
04:03 gem joined #gluster
04:04 atinmu joined #gluster
04:06 sripathi joined #gluster
04:14 RameshN joined #gluster
04:15 bharata-rao joined #gluster
04:16 nbalacha joined #gluster
04:21 dusmant joined #gluster
04:23 poornimag joined #gluster
04:27 vimal joined #gluster
04:28 yazhini joined #gluster
04:32 kanagaraj joined #gluster
04:39 kshlm joined #gluster
04:42 anil joined #gluster
04:45 saurabh_ joined #gluster
04:48 ramteid joined #gluster
04:51 ppai joined #gluster
04:52 zeittunnel joined #gluster
04:55 rafi joined #gluster
05:03 schandra joined #gluster
05:09 mat1010 left #gluster
05:17 paulc_AndChat joined #gluster
05:18 kdhananjay joined #gluster
05:18 pppp joined #gluster
05:18 Manikandan joined #gluster
05:19 PaulCuzner joined #gluster
05:19 hgowtham joined #gluster
05:20 raghu` joined #gluster
05:20 Manikandan_ joined #gluster
05:21 rgustafs joined #gluster
05:21 jiffin joined #gluster
05:27 soumya joined #gluster
05:28 ashiq joined #gluster
05:38 xavih joined #gluster
05:38 malevolent joined #gluster
05:47 zeittunnel joined #gluster
05:50 nangthang joined #gluster
05:53 deepakcs joined #gluster
05:55 ProT-0-TypE joined #gluster
05:58 glusterbot News from newglusterbugs: [Bug 1229563] Disperse volume: Failed to update version and size (error 2) seen during delete operations <https://bugzilla.redhat.com/show_bug.cgi?id=1229563>
05:59 dusmant joined #gluster
06:00 karnan joined #gluster
06:03 Gill joined #gluster
06:04 hagarth joined #gluster
06:15 rgustafs joined #gluster
06:18 twisted` joined #gluster
06:22 dusmant joined #gluster
06:24 soumya joined #gluster
06:24 atalur joined #gluster
06:27 maZtah joined #gluster
06:28 glusterbot News from newglusterbugs: [Bug 1206539] Tracker bug for GlusterFS documentation Improvement. <https://bugzilla.redhat.com/show_bug.cgi?id=1206539>
06:28 nangthang joined #gluster
06:28 aravindavk joined #gluster
06:35 haomaiwa_ joined #gluster
06:38 aravindavk joined #gluster
06:38 aravindavk joined #gluster
06:43 dusmant joined #gluster
06:49 Jandre joined #gluster
06:56 rafi joined #gluster
07:02 lexi2 joined #gluster
07:04 nbalacha joined #gluster
07:05 jmcantrell joined #gluster
07:05 Trefex joined #gluster
07:11 arcolife joined #gluster
07:11 kdhananjay joined #gluster
07:20 fyxim joined #gluster
07:25 frankS2 joined #gluster
07:26 dusmant joined #gluster
07:26 LebedevRI joined #gluster
07:27 Pintomatic joined #gluster
07:29 lezo joined #gluster
07:30 samsaffron___ joined #gluster
07:35 al joined #gluster
07:36 billputer joined #gluster
07:38 smohan joined #gluster
07:46 soumya joined #gluster
07:46 anrao joined #gluster
07:52 Slashman joined #gluster
07:55 haomaiw__ joined #gluster
08:05 [Enrico] joined #gluster
08:05 semajnz joined #gluster
08:11 itisravi joined #gluster
08:15 aravindavk joined #gluster
08:17 semajnz joined #gluster
08:20 haomaiwa_ joined #gluster
08:22 tessier joined #gluster
08:40 c0m0 joined #gluster
08:54 vimal joined #gluster
08:54 ashiq ndevos++
08:54 glusterbot ashiq: ndevos's karma is now 19
08:56 liquidat joined #gluster
09:01 dusmant joined #gluster
09:03 Jandre joined #gluster
09:04 rjoseph joined #gluster
09:08 maveric_amitc_ joined #gluster
09:12 shubhendu__ joined #gluster
09:22 itisravi_ joined #gluster
09:24 eljrax joined #gluster
09:35 autoditac joined #gluster
09:35 gildub joined #gluster
09:35 eljrax Hey, I ran yum update last night, and got 3.7.1, and when I restarted a brick, I see this: [posix.c:1341:posix_mkdir] 0-gvolume1-posix: mkdir (/.trashcan/internal_op): gfid (00000000-0000-0000-0000-000000000006) is already associated with directory (/var/lib/glusterfs/gvolume1/.glusterfs/00/00/00000000-0000-0000-0000-000000000005/internal_op). Hence, both directories will share same gfid and this can lead to inconsistencies.
09:35 eljrax I really don't like to read the word "inconsistencies" in my storage logs :)
09:36 eljrax What's this even about?
09:37 eljrax features.trash is off, if that matters?
09:41 kshlm joined #gluster
09:42 ira joined #gluster
09:49 ndevos eljrax: that definitely looks like something related to trash, anoopcs or jiffin might know
09:51 jiffin eljrax: its just a misleading error in log
09:53 jiffin eljrax: fix for that send upstream: http://review.gluster.org/#/c/10387
09:53 eljrax Ah ok, so I can rest easy :) Thanks
09:57 jiffin eljrax: np
10:04 badone_ joined #gluster
10:15 javi404 joined #gluster
10:15 RameshN joined #gluster
10:29 glusterbot News from newglusterbugs: [Bug 1229639] build: fix gitclean target <https://bugzilla.redhat.com/show_bug.cgi?id=1229639>
10:35 tjikkun joined #gluster
10:36 kkeithley2 joined #gluster
10:36 tjikkun hey guys, gluster fails to start on boot for me: http://fpaste.org/230279/43384617/
10:36 tjikkun any idea what might be causing something like this, or how to troubleshoot further?
10:40 anoopcs tjikkun, What does the logs say?
10:41 nsoffer joined #gluster
10:47 atinmu joined #gluster
10:51 firemanxbr joined #gluster
10:52 rjoseph joined #gluster
10:56 bene2 joined #gluster
10:56 dusmant joined #gluster
11:00 tjikkun anoopcs: I updated to 3.7.1 to be sure, the error I get is http://fpaste.org/230290/84759614/
11:00 rafi1 joined #gluster
11:01 tjikkun I see some lookup failure, so I'll try to add the hosts in /etc/hosts
11:05 arcolife joined #gluster
11:10 atinmu joined #gluster
11:12 shubhendu__ joined #gluster
11:12 rjoseph joined #gluster
11:12 anoopcs tjikkun, You are right.. It's related to host name resolution, I guess.
11:37 Trefex1 joined #gluster
11:39 hagarth joined #gluster
11:40 gildub joined #gluster
11:49 B21956 joined #gluster
11:53 diegows joined #gluster
11:56 atinmu REMINDER: Gluster Community Bug Triage meeting starting in another 5 minutes in #gluster-meeting
11:59 zeittunnel joined #gluster
12:00 soumya joined #gluster
12:01 harish joined #gluster
12:22 mkzero joined #gluster
12:39 glusterbot News from resolvedglusterbugs: [Bug 1214772] gluster xml empty output volume status detail <https://bugzilla.redhat.com/show_bug.cgi?id=1214772>
12:42 psilvao Dear people: What is the difference between writing in a brick vs write a gluster  volume?
12:47 ndevos psilvao: writing directly to the brick prevents gluster from knowing about the write, so it can not do the replication or distribution and other fancy things
12:49 wkf joined #gluster
12:49 psilvao ndevos: thanks and whats happens if you remove a file inside the brick?, gluster knows about this operation?
12:54 ndevos psilvao: no, you should always access the contents through a gluster mount point (either fuse or nfs)
12:55 psilvao Thnaks, we were making a conceptual error.
12:55 ndevos psilvao: gluster maintains a .glusterfs/ directory on the brick, that has soft- and hard-links so that filehandles can be used to access the contents (nfs requires that)
12:56 ndevos psilvao: if you delete files on the bricks, those links will still be there, and in case of a hard-link, the space will not be free'd yet
12:59 glusterbot News from newglusterbugs: [Bug 1203739] Self-heal of sparse image files on 3-way replica "unsparsifies" the image <https://bugzilla.redhat.com/show_bug.cgi?id=1203739>
13:02 bene2 joined #gluster
13:03 rafi joined #gluster
13:06 hagarth joined #gluster
13:07 julim joined #gluster
13:10 Trefex joined #gluster
13:10 pppp joined #gluster
13:15 plarsen joined #gluster
13:19 hamiller joined #gluster
13:22 dusmant joined #gluster
13:24 jcastill1 joined #gluster
13:25 Folken joined #gluster
13:27 dgandhi joined #gluster
13:29 jcastillo joined #gluster
13:30 Gill joined #gluster
13:31 Twistedgrim joined #gluster
13:40 bennyturns joined #gluster
13:40 ppai joined #gluster
13:46 aaronott joined #gluster
13:51 diegows joined #gluster
13:59 rjoseph joined #gluster
14:00 glusterbot News from newglusterbugs: [Bug 1221941] glusterfsd: bricks crash while executing ls on nfs-ganesha vers=3 <https://bugzilla.redhat.com/show_bug.cgi?id=1221941>
14:00 glusterbot News from newglusterbugs: [Bug 1221866] DHT Layout selfheal code should log errors <https://bugzilla.redhat.com/show_bug.cgi?id=1221866>
14:00 DV__ joined #gluster
14:01 jcastill1 joined #gluster
14:03 rwheeler joined #gluster
14:05 R0ok_ joined #gluster
14:06 jcastillo joined #gluster
14:17 tjikkun anoopcs: with host in /etc/hosts the issue is gone :)
14:17 maveric_amitc_ joined #gluster
14:23 jeffrin joined #gluster
14:24 jeffrin hello all
14:26 deepakcs joined #gluster
14:27 rafi joined #gluster
14:30 julim joined #gluster
14:32 bennyturns joined #gluster
14:47 RameshN joined #gluster
15:05 NuxRo hi JoeJulian. can splitmount be used on production volumes or must the volume be inactive when I am doing this?
15:13 shubhendu__ joined #gluster
15:14 CyrilPeponnet @ndevos hey ! Do you think it's worth the try to set rmtab file to /dev/null? Joe ans I were wondering.
15:16 CyrilPeponnet Regarding the performance issue I have with nfs clients and multi (un)mount.
15:19 hamiller joined #gluster
15:19 ndevos CyrilPeponnet: I dont know how it would behave... it might replace /dev/null with a real file, I think (it uses standard gf_store_* functions for writing the file)
15:22 CyrilPeponnet hmm let me check that on a test env
15:23 firemanxbr_ joined #gluster
15:24 CyrilPeponnet you're right
15:24 CyrilPeponnet :( too bad
15:28 NuxRo is there a way to resolve a gfid to a filename without using "find" on inodes? it's taking forever
15:30 CyrilPeponnet @NuxRo not that I know, I got the same issue... resolving hardlink take a while...
15:30 NuxRo oh boy, I seem to have hundreds, gonna take a while :(
15:32 DV joined #gluster
15:32 NuxRo oh, only now I realise, I have to do this on all servers hosting bricks for my replicas, right?
15:32 NuxRo cause the gfid/file might be on any of them
15:36 ndevos NuxRo: yes, that is correct :-/
15:37 ndevos well, you could do it on only one of the replica pairs, no need to do it on all of them
15:37 CyrilPeponnet if replicated the gfid are consistent between bricks
15:38 NuxRo well, I have 4 bricks, replica x2, so I'd need to run it at least on 2
15:38 ndevos yes, the gfid should be the same on all replica pairs
15:38 ndevos each replicated file/dir should have the same gfid
15:39 ndevos I think that is even the case when geo-replication is used
15:41 NuxRo aha, good
15:41 NuxRo any reason a file with same timestamp, size and md5sum would be in split brain?
15:42 NuxRo I mean, both replicas are identical by the looks of it
15:43 CyrilPeponnet @ndevos you're right (when it works :p)
15:47 maveric_amitc_ joined #gluster
15:48 bennyturns joined #gluster
15:48 soumya joined #gluster
15:49 ndevos NuxRo: could be permissions or extended attributes
15:51 ndevos NuxRo: http://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ shows how to check
15:58 NuxRo thanks ndevos
15:59 cholcombe joined #gluster
15:59 adzmely joined #gluster
16:18 atalur joined #gluster
16:40 n-st joined #gluster
16:40 glusterbot News from resolvedglusterbugs: [Bug 1220347] Read operation on a file which is in split-brain condition is successful <https://bugzilla.redhat.com/show_bug.cgi?id=1220347>
16:43 NuxRo Would "cp -a -n" would be more efficient than "rsync" when moving files between volumes?
16:44 CyrilPeponnet don't think so
16:46 jiffin joined #gluster
16:46 NuxRo meh, looks like cp is also invoking stat() ..
16:47 JoeJulian cpio
16:47 JoeJulian maybe
16:47 JoeJulian everything's going to call fstat because they all need to check if they're copying a directory, file, or symlink.
16:48 JoeJulian (or any other file type, of course)
16:48 NuxRo oh, makes sense
16:48 NuxRo then they'd all be more or less pretty slow
16:49 NuxRo with rsync I can easily resume interrupted transfers at least
16:49 JoeJulian true
16:50 NuxRo copying from a /brick replica into a mounted volume should be safe, right?
16:50 JoeJulian yes
16:50 CyrilPeponnet yep
16:50 NuxRo I think this would be the fastest
16:50 JoeJulian You won't (obviously) be able to copy the extended attributes.
16:50 CyrilPeponnet it is :)
16:50 georgeh-LT2 joined #gluster
16:50 NuxRo yeah, i am not interested in those
16:50 NuxRo it's just some bulk storage
16:50 NuxRo cheers
16:53 NuxRo JoeJulian: back to my initial question, can splitmount be used on live volumes?
16:54 shubhendu__ joined #gluster
16:54 JoeJulian yes
16:57 NuxRo thanks
16:59 JoeJulian tessier: check all glusterd. Check that tcp connection to 24007 each way. If selinux is enforcing, check the audit.log.
17:00 glusterbot News from newglusterbugs: [Bug 1229808] Read-replica-selection options are not having the desired effect <https://bugzilla.redhat.com/show_bug.cgi?id=1229808>
17:06 fsimonce joined #gluster
17:29 malevolent joined #gluster
17:30 xavih joined #gluster
17:38 mikemol So, I added a brick to a volume, commanded a rebalance, now the glusterd process crashes. Pretty much the moment I start it.
17:38 mikemol https://p.6core.net/p/kznEX7Ugc8C19EAolBA5QTyN
17:45 JoeJulian version?
17:46 mikemol 3.7.1
17:47 mikemol https://p.6core.net/p/cwi92dmQ0c8X1AzgXe0T81q1
17:48 mikemol Looks like it's only happening on the brick the data should be migrating two. (I'm expanding a replica-1 volume from one brick to two)
17:48 mikemol s/migrating two/migrating to/
17:48 glusterbot What mikemol meant to say was: An error has occurred and has been logged. Please contact this bot's administrator for more information.
17:53 atinmu joined #gluster
17:53 JoeJulian Ah, there it is... I hate bugzilla sometimes.
17:53 JoeJulian mikemol: bug 1227677
17:53 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1227677 high, unspecified, ---, spalai, ASSIGNED , Glusterd crashes and cannot start after rebalance
18:01 glusterbot News from newglusterbugs: [Bug 1229825] Add regression test for cluster lock in a heterogeneous cluster <https://bugzilla.redhat.com/show_bug.cgi?id=1229825>
18:06 mikemol Hm. So this is completely reproducible for me; uninstalling and reinstalling the packages doesn't fix the problem, and I couldn't find any cruft under /var or /etc to clean out.
18:07 autoditac joined #gluster
18:08 JoeJulian And you saw that bug reference, right?
18:15 atinmu mikemol, are you talking about 1227677 ?
18:15 mikemol atinmu, JoeJulian yes.
18:15 JoeJulian that's what I said... is this thing on?
18:15 * JoeJulian blows on the keyboard.
18:15 atinmu mikemol, if yes then I am afraid to say we have a known issue in rebalance in 3.7.1
18:16 atinmu mikemol, rebalance is broken in 3.7.1
18:16 atinmu mikemol, the fix will be available in 3.7.2 which is probably in another week's time
18:17 mikemol Well, that's, ah, less than heartening. My volume will probably be full by the end of the week.
18:18 * mikemol read through the bug, saw someone mention a workaround, but didn't see any further information apart from confusion whether a compiler optimization was causing the problem.
18:19 mikemol OIC, the 'REVIEW' comments. I haven't dealt with RH bugzilla in a while.
18:21 mikemol So, probably next week, then?
18:25 * mikemol does what he can to slow down his data growth.
18:38 ira joined #gluster
18:43 chirino joined #gluster
18:55 Rapture joined #gluster
19:07 firemanxbr joined #gluster
19:11 dgandhi joined #gluster
19:12 nsoffer joined #gluster
19:12 kkeithley_ @ports
19:12 glusterbot kkeithley_: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up since 3.4.0 (24009 & up previously). (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
19:13 paulc_AndChat joined #gluster
19:20 codex joined #gluster
19:20 rafi joined #gluster
19:23 bene2 joined #gluster
19:28 rotbeard joined #gluster
19:35 chirino_m joined #gluster
19:48 Trefex joined #gluster
19:56 rwheeler joined #gluster
19:57 premera joined #gluster
19:57 wkf_ joined #gluster
20:10 wkf joined #gluster
20:13 Trefex1 joined #gluster
20:17 DV joined #gluster
20:24 jmcantrell joined #gluster
20:24 fyxim joined #gluster
20:32 Gill joined #gluster
20:39 badone_ joined #gluster
21:10 badone__ joined #gluster
21:34 abyss_ joined #gluster
22:01 PeterA joined #gluster
22:02 PeterA i just got a brick crash and not able to bring it up :(
22:02 PeterA http://pastie.org/10232083
22:02 PeterA what would that means? [quota-enforcer-client.c:187:quota_enforcer_lookup_cbk] 0-sas03-quota: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
22:08 PeterA noticed this E [socket.c:2161:socket_connect_finish] 0-sas03-client-5: connection to 10.40.12.78:49154 failed (Connection refused)
22:10 JoeJulian Nobody's looked at that bug since March?!?!?!
22:10 PeterA what bug?
22:11 PeterA how should i fix that?
22:11 JoeJulian bug 1203433
22:11 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1203433 urgent, unspecified, ---, bugs, NEW , Brick/glusterfsd crash when tried to rm a folder with IO error
22:11 JoeJulian yours
22:11 PeterA ya :(
22:11 PeterA but also seems like this time is the quotad not able to connect
22:11 PeterA when the brick try to start it just crash due to rpc
22:11 JoeJulian The crash is in the same spot.
22:12 PeterA hmm
22:12 cyberbootje joined #gluster
22:12 PeterA but last time was due to a file got access
22:12 PeterA this time the brick not even coming back :(
22:13 JoeJulian the "connection refused" is the down brick.
22:13 PeterA u mean the quota?
22:14 PeterA hm…..so how can i bring the brick back?
22:14 badone_ joined #gluster
22:15 Jandre joined #gluster
22:18 JoeJulian I'm looking at the source for some clue how to answer that
22:20 JoeJulian That crash was fixed in 3.5.3
22:21 PeterA what is the cause of the crash?
22:22 JoeJulian Looks like a null or uninitialized pointer
22:22 PeterA trigger by?
22:23 PeterA wonder if we can just bring up the brick for now
22:23 JoeJulian https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/marker/src/marker-quota.c#L176-L182
22:24 JoeJulian That check wasn't there in 3.5.2
22:25 PeterA is they due to some files that got moved?
22:25 JoeJulian bug 1144315
22:25 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1144315 urgent, urgent, ---, kdhananj, CLOSED CURRENTRELEASE, core: all brick processes crash when quota is enabled
22:25 PeterA er….quota....
22:27 JoeJulian afaict, if you disable readdirp you should be able to start without crashing.
22:27 Jandre_ joined #gluster
22:27 PeterA readdirp ?
22:28 Jandre joined #gluster
22:28 JoeJulian hmm, can't disable it volume-wide. Guess it's just a mount option.
22:29 JoeJulian Probably would fail due to shd and nfs anyway.
22:29 PeterA wonder if we can trace which file cause it?
22:32 glusterbot News from newglusterbugs: [Bug 1203433] Brick/glusterfsd crash when tried to rm a folder with IO error <https://bugzilla.redhat.com/show_bug.cgi?id=1203433>
22:32 edwardm61 joined #gluster
22:35 PeterA http://pastie.org/10232121
22:35 PeterA getting these from brick log
22:35 PeterA wonder if we can try to remove those file
22:42 PeterA so it's a bug on the quota that crash the glusterfsd ?
22:45 PeterA http://pastie.org/10232132
22:45 PeterA noticed this
22:45 PeterA lock error??
23:02 glusterbot News from newglusterbugs: [Bug 1229914] glusterfs self heal takes too long following node outage <https://bugzilla.redhat.com/show_bug.cgi?id=1229914>
23:02 PeterA JoeJulian, would that be possible to force up the brick?
23:06 gildub joined #gluster
23:10 JoeJulian no idea. Since there's a fix, if it were me, I'd just implement the fix.
23:11 JoeJulian Otherwise you're just going to hit it again.
23:12 PeterA should i go 3.5.4 or 3.5.3?
23:15 JoeJulian based on the bug fixes and your use of nfs, I would absolutely go for 3.5.4.
23:16 PeterA ok
23:35 Rapture joined #gluster
23:37 PeterA gluster 3.5.4 for ubuntu is out??
23:39 plarsen joined #gluster
23:39 PeterA can we get the build for 3.5.4 available for ubuntu?
23:42 maveric_amitc_ joined #gluster
23:43 ira joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary