Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2015-06-01

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:48 badone_ joined #gluster-dev
01:48 ilbot3 joined #gluster-dev
01:48 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
03:43 ppai joined #gluster-dev
03:44 itisravi joined #gluster-dev
03:47 shubhendu joined #gluster-dev
03:53 kanagaraj joined #gluster-dev
04:13 kshlm joined #gluster-dev
04:23 spandit joined #gluster-dev
04:36 soumya joined #gluster-dev
04:40 sakshi joined #gluster-dev
05:00 Manikandan joined #gluster-dev
05:01 hgowtham joined #gluster-dev
05:01 ashiq joined #gluster-dev
05:12 kdhananjay joined #gluster-dev
05:16 pppp joined #gluster-dev
05:16 Manikandan joined #gluster-dev
05:19 gem joined #gluster-dev
05:19 anekkunt joined #gluster-dev
05:23 ashishpandey joined #gluster-dev
05:27 rafi joined #gluster-dev
05:36 hagarth joined #gluster-dev
05:37 vimal joined #gluster-dev
05:44 jiffin joined #gluster-dev
05:52 kdhananjay joined #gluster-dev
05:52 Gaurav_ joined #gluster-dev
06:01 atalur joined #gluster-dev
06:07 overclk joined #gluster-dev
06:10 overclk joined #gluster-dev
06:16 aravindavk joined #gluster-dev
06:17 nbalacha joined #gluster-dev
06:23 ashiq joined #gluster-dev
06:37 rgustafs joined #gluster-dev
06:38 deepakcs joined #gluster-dev
06:54 sas_ joined #gluster-dev
06:57 rjoseph joined #gluster-dev
07:00 overclk joined #gluster-dev
07:00 nkhare joined #gluster-dev
07:04 ppai joined #gluster-dev
07:06 kdhananjay1 joined #gluster-dev
07:14 overclk joined #gluster-dev
07:23 ppai joined #gluster-dev
07:43 anekkunt joined #gluster-dev
07:58 ashiq joined #gluster-dev
07:58 gem joined #gluster-dev
08:01 ndevos ping aravindavk, overclk: can peer_add_secret_pub work on RHEL5? it only gets installed when geo-rep is enabled, glusterfind needs that too?
08:13 anekkunt ndevos,  http://review.gluster.org/#/c/10996/  is ready for merge
08:14 anekkunt ndevos,  could you merge this patch
08:15 aravindavk ndevos: now glusterfind needs that. It was existing file so it should work with RHEL5
08:16 ndevos aravindavk: hmm, okay, thanks!
08:17 ndevos anekkunt: you have an empty line too much in the %changelog, could you fix that?
08:17 anekkunt ndevos, k ..
08:24 anekkunt ndevos,  I have sent the patch (http://review.gluster.org/#/c/10996/) by removing  extra new lines ... could you merge it .. previous patch passed all regression test
08:25 ndevos anekkunt: done!
08:26 anekkunt ndevos, Thanks
08:31 pranithk joined #gluster-dev
08:35 hgowtham joined #gluster-dev
08:43 saurabh_ joined #gluster-dev
08:43 ashiq joined #gluster-dev
08:44 Manikandan joined #gluster-dev
08:45 anekkunt joined #gluster-dev
08:49 shubhendu joined #gluster-dev
08:50 overclk ndevos, aravindavk is the person for that :)
08:52 gem joined #gluster-dev
08:52 krishnan_p joined #gluster-dev
08:53 kdhananjay joined #gluster-dev
08:57 aravindavk overclk: ndevos any issues? sorry I missed the conversation
08:58 ndevos aravindavk, overclk: no issues, just fixing the build for el5, peer_add_secret_pub was not installed and that caused building the RPMs to fail
08:59 atinmu joined #gluster-dev
09:04 ashiq joined #gluster-dev
09:05 Manikandan joined #gluster-dev
09:06 aravindavk ndevos: ok
09:08 hagarth joined #gluster-dev
09:09 krishnan_p hagarth, I sent the email on -devel ML announcing pushing of 3.7.1 release tag. Does it look OK?
09:11 krishnan_p hagarth, Would it make sense to merge http://review.gluster.org/10959 without the NetBSD regression build, given that many such builds aborted over this weekend? This patch is a cherry-pick of an identical patch on master which passed both Linux and NetBSD regressions.
09:11 hagarth krishnan_p: looks good to me
09:13 soumya joined #gluster-dev
09:13 krishnan_p hagarth, thanks.
09:14 krishnan_p hagarth, Vijaikumar's quota.conf backward compatibility fix has also been merged to release3-7. So, we are almost there :)
09:14 aravindavk krishnan_p: can this go in http://review.gluster.org/#/c/11016/ Don't know the regression status
09:15 hagarth krishnan_p: yes that plus a bunch of ec & bitrot patches, makes 3.7.1 quite good  :)
09:16 anekkunt joined #gluster-dev
09:17 krishnan_p aravindavk, is this fixing an issue that makes it worthwhile to delay the pushing of 3.7.1 tags?
09:18 krishnan_p hagarth, yep. First thing is to alleviate a known annoying issue. Rest is optional, if we intend to release 3.7.2 in 2 weeks.
09:18 hagarth krishnan_p: right
09:18 rastar_afk joined #gluster-dev
09:20 aravindavk krishnan_p: merged in master. spec file issue, glusterfind will not work without georep rpms without this patch
09:21 hchiramm_ joined #gluster-dev
09:22 * ndevos starts to hit the [submit] button
09:26 krishnan_p ndevos, which submit button exactly? :)
09:26 ndevos krishnan_p: the one in Gerrit for some changes that are ready for 3.7.1 :)
09:27 krishnan_p ndevos, go for it :)
09:27 ndevos krishnan_p: OK! \o/
09:27 krishnan_p aravindavk, if its ready to merge, merge it.
09:29 krishnan_p The idea behind pushing 3.7.1 soon is to ensure users have a usable version of 3.7.1 where they can test the new features that have come in this release. Without the fix for volume-status bug, the user experience can be (rightfully) annoying. There was another quota backward compatibility bug too!
09:30 krishnan_p These must be addressed. If your favorite component has these class of issues, please fix them. Don't wait for me :)
09:30 krishnan_p ndevos, I thought you would be busy 'grape-farming' ;-)
09:30 atinmu krishnan_p, ha ha
09:31 atinmu :)
09:31 ndevos krishnan_p: :P
09:31 ndevos but at least someone read the email!
09:31 atinmu ndevos, I am yet to go through your proposal though :)
09:31 krishnan_p I am itching to reply to. I like the idea. I want to think a little more about it.
09:32 ndevos krishnan_p: was it you who suggested to use something like linux/kref before?
09:32 ndevos atinmu: dont spend too much time about the grapes ;-)
09:33 krishnan_p ndevos, while we are at it, I am tempted to disallow gf_ref_alloc, since this could lead to ref living outside the ref'd object.
09:33 krishnan_p ndevos, I have been building castles in the air with that idea, yes :)
09:34 ndevos krishnan_p: right, I'm not sure why I added gf_ref_alloc(), my nfs changes don't need it either
09:34 krishnan_p ndevos, I still don't know the consequences of completely disallowing it. But having a function to do that, if we won't need it most of the times, might tempt innocent users to try using it :P
09:35 ndevos krishnan_p: fine with me :)
09:35 krishnan_p ndevos, my gripe about gf_ref_alloc is that, it introduces split life time for 'the object' and ref object.
09:35 sakshi joined #gluster-dev
09:36 krishnan_p whereas embedded objects implicitly guarantee same life time for 'the object' and ref object. Does that make sense?
09:37 ndevos krishnan_p: it makes sense, but it does not prevent users from calling GF_MALLOC() to create a struct gf_ref
09:38 krishnan_p ndevos, yep. We don't have that much help from our compilers :(
09:38 ndevos krishnan_p: I'm happy to drop it and only have gf_ref_init(), because that is what seems to be the most simple way anyway
09:38 krishnan_p ndevos, I am just saying we could avoid putting ideas into people's heads :P
09:38 ndevos krishnan_p: yeah, perfectly fine with me :)
09:39 krishnan_p ndevos, don't drop it yet. Let's have this conversation with others too
09:39 krishnan_p ndevos, will be afk for a bit.
09:40 nishanth joined #gluster-dev
09:43 soumya joined #gluster-dev
09:56 atinmu joined #gluster-dev
09:56 anekkunt joined #gluster-dev
09:59 nbalacha joined #gluster-dev
10:02 RaSTar joined #gluster-dev
10:04 RaSTar joined #gluster-dev
10:04 krishnan_p|afk aravindavk, could you review http://review.gluster.org/#/c/10993/ ? (snapshot scheduler)
10:04 RaSTar joined #gluster-dev
10:07 ndevos jiffin: awesome subject of bug 1226807 :)
10:07 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1226807 high, high, ---, ndevos, ASSIGNED , FSAL_GLUSTER: Ganesha caches the ACLs for a directory too long, needs some re-validation or something
10:08 nishanth joined #gluster-dev
10:14 shubhendu joined #gluster-dev
10:17 RaSTar joined #gluster-dev
10:19 aravindavk krishnan_p|afk: ok. reviewing now
10:20 kkeithley1 joined #gluster-dev
10:20 atinmu joined #gluster-dev
10:20 krishnan_p|afk aravindavk, thanks
10:20 anekkunt joined #gluster-dev
10:21 nbalachandran_ joined #gluster-dev
10:23 RaSTar joined #gluster-dev
10:23 kkeithley1 joined #gluster-dev
10:23 atinmu joined #gluster-dev
10:23 anekkunt joined #gluster-dev
10:23 nbalachandran_ joined #gluster-dev
10:24 RaSTar joined #gluster-dev
10:28 RaSTar joined #gluster-dev
10:28 krishnan_p|afk ndevos, check reply to 'grape collection' :-)
10:28 nishanth joined #gluster-dev
10:30 RaSTar joined #gluster-dev
10:30 jiffin ndevos: :)
10:31 RaSTar joined #gluster-dev
10:33 ndevos krishnan_p: thanks! so, you would prefer to see "struct gf_ref" -> "typedef gf_ref_t" and drop the gf_ref_(alloc|free) ?
10:33 anoopcs I still see some warnings from tier.c during compilation with gcc 5.1.1. I don't know about its impact on functionality. Recently similar warnings for changelog resulted in failure to start the volume and http://review.gluster.org/#/c/11004/ fixes the same.
10:34 ndevos and, as a grape farmer, I would not grow sour grapes
10:34 RaSTar joined #gluster-dev
10:35 ndevos anoopcs: send a patch, or check with rafi?
10:38 anoopcs ndevos, Ok.
10:38 RaSTar joined #gluster-dev
10:40 RaSTar joined #gluster-dev
10:41 jobewan_ joined #gluster-dev
10:42 hgowtham_ joined #gluster-dev
10:44 RaSTar joined #gluster-dev
11:10 ndevos jiffin: for http://review.gluster.org/11019 , just check the nfs.log of a system and searcj for hashkey
11:12 krishnan_p ndevos, I hadn't thought about gf_ref_t. What would be the effect of not having it?
11:13 krishnan_p aravindavk, a +2 would help especially when you are the primary reviewer :)
11:14 krishnan_p ndevos, I mean what would be the difference of not exporting the typedef and having struct gf_ref alone instead?
11:19 hagarth1 joined #gluster-dev
11:20 krishnan_p hagarth1,  what shall we do about http://review.gluster.org/#/c/11016/ ? It failed 'smoke'. aravindavk wants this in 3.7.1
11:20 krishnan_p hagarth1, other patches that we were waiting for have been merged.
11:20 hagarth1 krishnan_p: checking
11:21 jiffin ndevos++ for the patch
11:21 glusterbot jiffin: ndevos's karma is now 140
11:21 hagarth1 krishnan_p: is it still pending regression from gluster build system?
11:22 overclk krishnan_p, hagarth_  good to have this patch http://review.gluster.org/#/c/11030/  too..
11:22 pranithk left #gluster-dev
11:27 krishnan_p overclk, aravindavk I will be waiting for both your patches to be merged before I make 3.7.1 release.
11:30 overclk krishnan_p, sure. I'll take care of merging and drop a note when done.
11:38 rafi ndevos: I spoke with anoopcs and pushed a patch, thanks guys for reporting the bug
11:38 rafi ndevos++ anoopcs++
11:38 glusterbot rafi: ndevos's karma is now 141
11:38 glusterbot rafi: anoopcs's karma is now 6
11:39 anoopcs rafi, Thanks.
11:39 msvbhat ndevos: Good afternoon. I was facing some issue with nfs v3 mounting a disperse volume. Free to take look?
11:39 rafi anoopcs: ndevos : please review the patch here http://review.gluster.org/11032
11:39 msvbhat ndevos: soumya took a look and asked to consult you as well
11:40 soumya ndevos, I am guessing if it could be an issue with throttling
11:40 soumya ndevos, not sure how to confirm it though..
11:41 ndevos msvbhat, soumya: I'm trying to leave for lunch, is it very urgent?
11:41 soumya ndevos, I think we could do it later
11:43 msvbhat ndevos: I'm going to be here until (almost) evening in Europe :) So no hurry :P
11:44 krishnan_p soumya, which throttling issue are you referring to?
11:45 soumya rpc requests throttle...we see that nfs server is not responding to a particular client
11:45 soumya and in netstat I see huge recv-Q number
11:45 soumya krishnan_p, ^^^
11:46 soumya krishnan_p, I am guessing it could be the issue as that server is able to respond to other client's requests
11:47 ndevos soumya, msvbhat: maybe I have something like that too (not disperse related) when I run tests/basic/mount-nfs-auth.t in a loop?
11:47 soumya I do not remember the limit atm but the Recv-Q number for that client is around  'tcp   335956      0 '
11:48 ndevos the glusterfs process just seems to stop... when I connect gdb, and disconnect the test will continue - does that help for now?
11:48 ndevos ... and disconnect gdb (or write 'continue' in the gdb-prompt)
11:48 ppai joined #gluster-dev
11:49 soumya ndevos, okay .. shall give a try
11:49 ndevos soumya, msvbhat: how likely is it that the problem you see is related to disperse?
11:49 ndevos soumya, msvbhat: also, do you see ping timeout failures in the nfs.log?
11:49 soumya ndevos, I feel its not related to disperse
11:49 msvbhat ndevos: I don't think it's related to disperse
11:50 soumya ndevos, no logs on the server -side
11:50 msvbhat But just found while running tests for disperse volume
11:50 krishnan_p soumya, it could also be the case if one of the epoll thread is hung.
11:50 krishnan_p soumya, the new MT epoll is actually equivalent to many single epoll models working parallely.
11:51 msvbhat krishnan_p: How do we verify that?
11:51 krishnan_p msvbhat, attach gdb to the nfs server process. See what all the epoll worker threads are doing?
11:52 ndevos krishnan_p: I did briefly check my hanging one, 2 epoll workers are both in epoll_wait()
11:52 soumya krishnan_p, ndevos , same here
11:52 ndevos and there are (and should be) only 2 worker threads
11:52 soumya (gdb) info threads
11:52 soumya 8 Thread 0x7f853c58b700 (LWP 21676)  0x00007f854413df3d in nanosleep () from /lib64/libpthread.so.0
11:52 soumya 7 Thread 0x7f853bb8a700 (LWP 21677)  0x00007f854413e4b5 in sigwait () from /lib64/libpthread.so.0
11:52 soumya 6 Thread 0x7f853b189700 (LWP 21678)  0x00007f854413a98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11:52 soumya 5 Thread 0x7f8538557700 (LWP 21680)  0x00007f8543aa0fd3 in epoll_wait () from /lib64/libc.so.6
11:52 soumya 4 Thread 0x7f852ffff700 (LWP 21717)  0x00007f8543a971b3 in poll () from /lib64/libc.so.6
11:52 soumya 3 Thread 0x7f852ee61700 (LWP 21719)  0x00007f8543aa0fd3 in epoll_wait () from /lib64/libc.so.6
11:52 soumya 2 Thread 0x7f852d448700 (LWP 21742)  0x00007f854413a98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11:52 soumya * 1 Thread 0x7f8545269740 (LWP 21675)  0x00007f854413722d in pthread_join () from /lib64/libpthread.so.0
11:52 soumya (gdb)
11:53 ndevos yeah, and epoll_wait() has -1 as timeout, I wonder if a disconnected socket could prevent epoll_wait() from returning, or something like that
11:55 krishnan_p ndevos, disconnected socket would raise EPOLLER on disconnect (eventually).
11:55 krishnan_p what is t 4 doing 'waiting on poll()'
11:55 ndevos krishnan_p: hmm, and how long can "eventually" be?
11:56 krishnan_p ndevos, actually it shouldn't be too long if we our epoll implementation is 'bug-free' :)
11:57 krishnan_p ndevos, we register our sockets on epoll fd in level-triggered mode. If the socket is disconnected, while we were handling an event from previous batch, we would receive the event corresponding disconnect on subsequent epoll_wait call.
11:58 krishnan_p now i need to ask some questions that were already answered :(
11:58 krishnan_p msvbhat, what were you doing to get into this?
11:58 ndevos poll() in thread 4 likely is the nsm_thread from the nfs server
11:58 soumya (gdb) thread 4
11:58 soumya [Switching to thread 4 (Thread 0x7f852ffff700 (LWP 21717))]#0  0x00007f8543a971b3 in poll () from /lib64/libc.so.6
11:58 soumya (gdb) bt
11:58 soumya #0  0x00007f8543a971b3 in poll () from /lib64/libc.so.6
11:58 soumya #1  0x00007f8543ace020 in svc_run () from /lib64/libc.so.6
11:58 soumya #2  0x00007f8536dfdc64 in nsm_thread (argv=<value optimized out>) at nlmcbk_svc.c:121
11:58 soumya #3  0x00007f85441369d1 in start_thread () from /lib64/libpthread.so.0
11:58 soumya #4  0x00007f8543aa09dd in clone () from /lib64/libc.so.6
11:58 soumya (gdb)
11:58 soumya ndevos, right
11:58 ndevos hah!
11:59 * ndevos drops off and really goes for lunch now
11:59 msvbhat krishnan_p: I was running some scripts which kills bricks in disperse volume while some I/O operations are running.
12:00 msvbhat krishnan_p: I had scheduled this on Friday night. And saw today morning that it's hung
12:00 krishnan_p msvbhat, could you confirm if what you are facing looks similar to this -
12:00 krishnan_p https://bugzilla.redhat.com/show_bug.cgi?id=1224016
12:00 glusterbot Bug 1224016: urgent, unspecified, ---, kparthas, ASSIGNED , NFS: IOZone tests hang, disconnects and hung tasks seen in logs.
12:01 * msvbhat looking
12:02 rafi1 joined #gluster-dev
12:02 ppai joined #gluster-dev
12:02 krishnan_p ndevos, does newer mounts to the same NFS share also see their I/Os hung?
12:03 soumya krishnan_p, yes
12:03 soumya krishnan_p, if from the same client..
12:04 krishnan_p soumya, there is a painful way to analyse this. (can't think of anything else now)
12:04 krishnan_p soumya, we need to iterate through the slot table of epoll, look for all the used slots and inspect the corresponding fd/socket pair.
12:04 krishnan_p soumya, one of two things could happen.
12:06 krishnan_p soumya, there may be no fd/socket for this client observing problems
12:06 krishnan_p soumya, actually I can't think of another possibility.
12:07 krishnan_p soumya, ndevos are you guys suspecting that the fd of the client is being throttled?
12:07 sakshi joined #gluster-dev
12:07 krishnan_p soumya, there must be a easier to way to know that
12:07 krishnan_p soumya, wait...
12:07 soumya krishnan_p, not exactly fd..any request from that client..
12:08 soumya krishnan_p, infact in the pkt trace I see that mount svc had responded to the client, its only NFS service which isn't replying to the client's requests
12:08 krishnan_p soumya, the fd I'm referring to is socket fd, which remains constant for all the FOPs for that client-server session.
12:09 soumya krishnan_p, sorry..you are right
12:09 krishnan_p soumya, and do they use different socket connections?
12:09 soumya krishnan_p, yes
12:09 msvbhat krishnan_p: They don't look similar to me
12:13 * msvbhat brb
12:20 kanagaraj joined #gluster-dev
12:52 rafi joined #gluster-dev
12:53 soumya msvbhat, ndevos, KP has taken a look at the setup...after debugging for a while he thinks it may be related to throttle..
12:54 soumya and in addition it doesn't have http://review.gluster.org/#/c/10887/  fix yet..
12:55 soumya msvbhat, he has suggested to raise a bug with a core for the nfs process and all the logs (sos-report) before upgrading to the latest nightly with the fix
13:09 msvbhat soumya: Ah, The version was built on Friday evening IST.
13:09 soumya msvbhat, right
13:09 msvbhat Oh , Tha patch was merged yesterday
13:10 soumya yupp
13:10 soumya so before re-checking with the latest nightly, could you a file a bug with all the logs
13:10 soumya and a core of the process
13:10 msvbhat soumya: Okay, Will file a bug \
13:10 msvbhat core?
13:10 soumya and probably CC all of us including Shyam
13:11 msvbhat There was a crash? I didn't see the crash.
13:11 msvbhat soumya: ^^
13:11 msvbhat soumya: Or you mean the gdb traces?
13:12 soumya msvbhat, there isn't any crash..
13:12 soumya msvbhat, yes
13:12 soumya like stack
13:12 soumya *gstack
13:12 msvbhat soumya: Gotcha :) Thanks
13:13 ndevos msvbhat: try the "gcore" script, it comes with gdb
13:13 shyam joined #gluster-dev
13:14 ndevos soumya: actually, I am running my tests from an updated git/master... I do have that patch already
13:15 soumya ndevos, ahh...KP suspects two code paths socket_throttle (which has his fix) and rpcsvc_request_outstanding which he is not sure of atm
13:16 soumya hence he has asked to have a bug
13:16 ndevos soumya: right, in the nfs.log I can see WRITE fops (to the brick) getting timed-out
13:17 msvbhat ndevos: Ah, Thanks. That's rather simple :)
13:17 * msvbhat refers to gcore
13:17 ndevos msvbhat: gcore is great!
13:17 soumya ndevos, okay..
13:18 soumya ndevos, also do u know if "umount -ld" force unmount actually disconnects socket?
13:18 soumya ndevos, we see that tcp connection still shows in ESTABLISHED state
13:19 ndevos soumya: "umount -l" is almost never what you need - the kernel will then only really do the umount when nothing uses the mountpoint anymore
13:19 * ndevos doesnt know what the -d option does
13:20 soumya ndevos, okay that explains that..even after that umount , I still see pkts from client to which server hasn't responding
13:20 ndevos huh?! "When the unmounted device was a loop device..." I'm sure NFS isnt a loop device :)
13:20 soumya :)
13:21 ndevos soumya: it would be more interesting to see if the nfs-server is able to send packets to the brick, have you checked that?
13:21 soumya ndevos, ah no
13:21 soumya we have taken tcpdump from the nfs-client side but not on server-side
13:21 msvbhat soumya: ndevos: Does capturing the tcmpdump on the server helps?
13:22 ndevos soumya: I think that the NFS-client is just put on hold, the nfs-server is trying to get an answer from the brick, and that gNFS<->brick communication is the problem
13:23 ndevos msvbhat: yes, I always capture on the nfs-server, something like: tcpdump -i any -s 0 -w /var/tmp/nfs-hang.pcap tcp and not port 22
13:23 soumya ndevos, but it does serve other clients' requests
13:25 ndevos soumya: well, that is nice.... I have no idea how the sending of RPCs from a GlusterFS-client to the bricks are scheduled, maybe that is expected?
13:25 soumya ndevos, maybe ..thats when I had suspected if its related to throttle on that connection
13:25 soumya *that particular connection
13:26 soumya ndevos, though I am not much sure of its internals
13:27 ndevos soumya: hmm, me neither...
13:28 krishnan_p joined #gluster-dev
13:29 soumya ndevos, lets wait for krishnan_p's / shyam's analysis of that bug
13:29 aravindavk joined #gluster-dev
13:30 ndevos soumya: yes, and at the mean time, I'm running more tests with mount-nfs-auth.t on a single brick, that seems to work better too
13:30 shyam soumya: Which bug please...
13:30 * ndevos isnt sure how that is related, but since moving from replica 3 to one brick, the test did not fail yet
13:31 soumya shyam, msvbhat is filing it right now
13:31 shyam soumya: k :)
13:31 soumya shyam, :) he has seen it while running a test on dispersed volume
13:31 krishnan_p shyam, hello
13:32 shyam krishnan_p: Yo!
13:32 * shyam Hmmm... should I be reading the channel logs... (doing so anyway)
13:32 soumya ndevos, :) ... btw I have doubts regarding your patch.. probably will ping tmrw
13:32 soumya ndevos, have to leave now ..
13:33 krishnan_p shyam, it doesn't look like idx = -1 issue. And this build doesn't have the fix for socket_throttle.
13:33 ndevos soumya: which patch? but yes, tell me tomorrow :)
13:33 hagarth joined #gluster-dev
13:34 soumya krishnan_p, quick update .. ndevos has been running a test (which has similar issue) on the latest upstream branch which has your fix
13:34 krishnan_p soumya, thanks.
13:34 krishnan_p shyam, so there you have it
13:34 soumya ndevos, hashkey patch..yeah will ping tmrw..I need to re-look at it  :)
13:35 ndevos soumya: sure, have a good evening!
13:35 rafi joined #gluster-dev
13:35 soumya cya..  :)
13:35 krishnan_p shyam, i have asked msvbhat to attach the gcore of the running gluster-nfs process. Hope he remembers to do that
13:36 shyam krishnan_p: ok, is the build which this core is generated from avbl. as well?
13:37 krishnan_p shyam, msvbhat (or the bug he raises) should have all the answers :)
13:37 msvbhat https://bugzilla.redhat.com/show_bug.cgi?id=1226941
13:37 glusterbot Bug 1226941: medium, unspecified, ---, bugs, NEW , nfs mount hung and gnfs server unresponsive to one of the clients
13:37 msvbhat krishnan_p: ndevos: shyam : I will attach all the sosreports and gcore files
13:37 krishnan_p shyam, the corresponding fd doesn't receive disconnect from TCP either :(
13:38 * shyam still reading older chat logs ;)
13:38 msvbhat Phew! gcore file is ~350MB :(
13:38 krishnan_p shyam, last bit of useful info, the fd that was throttled and never 'un-throttled' was 23 and its slot was 12.
13:38 krishnan_p shyam, ereg[0][12]
13:38 shyam msvbhat: If you could get the logs as well it would be good (just to cross reference stuff)
13:39 shyam krishnan_p: cool that helps
13:39 rafi joined #gluster-dev
13:40 krishnan_p msvbhat, that could be due to the large no. of outstanding req and the iobufs belonging to them :(
13:40 ira joined #gluster-dev
13:40 shyam msvbhat: Can I get access to the machine where this core is present
13:40 ndevos msvbhat: maybe you are lucky an xz or gzip compresses it really well?
13:40 shyam Would make my life a little easier
13:40 ndevos hehe
13:41 msvbhat shyam: Sure... Will PM you the setup details
13:41 shyam msvbhat: k, ty
13:42 krishnan_p ndevos, were the requests DRC'd ?
13:44 rafi1 joined #gluster-dev
13:44 krishnan_p shyam, over to you wrt nfs throttle for the time being. Look forward to continue with it tomorrow if it comes to that :)
13:44 shyam krishnan_p: kk
13:45 krishnan_p shyam, thanks.
13:47 ndevos krishnan_p: DRC is off by defaults
13:47 ndevos -s
13:47 nkhare joined #gluster-dev
13:47 ndevos so, I doubt that DRC has anything to do with it
13:47 krishnan_p ndevos, thanks. just checking :)
13:48 ndevos sure :)
13:49 krishnan_p ndevos, it helps to know everything. which is why i like core better!
13:49 krishnan_p ndevos, its like a fossil. tells you a lot more than you were looking for :-)
13:50 ndevos krishnan_p: oh, yes, and cores are so much fun!
13:51 * ndevos hates those problems that happen *so* briefly you can hardly put your finger on it
13:52 krishnan_p ndevos, we have the fossils! we can zero it, with time
13:57 johnmark joined #gluster-dev
13:57 dlambrig1 joined #gluster-dev
14:07 pousley joined #gluster-dev
14:08 wushudoin joined #gluster-dev
14:10 krishnan_p ndevos, what is the ML for glusterfs package maintainers?
14:12 * ndevos checks
14:13 ndevos krishnan_p: http://www.gluster.org/mailman/listinfo sais it is packaging@gluster.org
14:13 krishnan_p ndevos, got it from a mail you sent to -devel earlier :) thanks
14:14 ndevos krishnan_p: and you are the 1st to send an email thare, wohoo!
14:14 krishnan_p ndevos, have I already?
14:14 krishnan_p ndevos, could you add me to that list. I just subscribed :)
14:14 ndevos krishnan_p: you dont have to be subscribed to send emails to the list, I think
14:15 ndevos krishnan_p: also, I think anyone can join that list, for all I know, you can subscribe yourself...
14:16 ndevos krishnan_p: I can subscribe you, if you like... what address?
14:20 kkeithley joined #gluster-dev
14:21 krishnan_p joined #gluster-dev
14:26 rafi joined #gluster-dev
14:44 nishanth joined #gluster-dev
15:28 wushudoin| joined #gluster-dev
15:33 wushudoin| joined #gluster-dev
16:04 krishnan_p joined #gluster-dev
16:15 rafi joined #gluster-dev
16:31 shubhendu joined #gluster-dev
16:48 _Bryan_ joined #gluster-dev
17:01 gem joined #gluster-dev
17:03 gem joined #gluster-dev
17:04 Gaurav_ joined #gluster-dev
17:10 dlambrig1 left #gluster-dev
17:47 dlambrig joined #gluster-dev
17:59 kkeithley_bat ndevos: after userspace-rcu was pushed to stable, 3.7.x still doesn't build on el5 because uuid-devel in RHEL is not available to Koji.  If it's in CentOS then CentOS Storage SIG may be the only place for EL5 users to get glusterfs-3.7.x
18:09 lpabon joined #gluster-dev
18:15 spandit joined #gluster-dev
18:40 jiffin joined #gluster-dev
18:51 jiffin joined #gluster-dev
19:11 kkeithley_bat ndevos: after userspace-rcu was pushed to stable, 3.7.x still doesn't build on el5 because uuid-devel in RHEL is not available to Koji.  If it's in CentOS then CentOS Storage SIG may be the only place for EL5 users to get glusterfs-3.7.x
19:11 kkeithley_bat in fact, there's not even libuuid
19:54 ndevos kkeithley_bat: you need e2fsprogs-devel for libuuid on el5
19:55 ndevos kkeithley_bat: http://review.gluster.org/#/c/10803/7/glusterfs.spec.in addresses all of it
20:13 kkeithley_bat ndevos: see my email
20:35 ndevos kkeithley_bat: see mine :)
20:38 kkeithley_bat hah, right. I'm functioning, if you can call it that, on about three hours of sleep after my flight had mechanical problems and had to return to Boston. I didn't arrive here until 04:00.
20:39 kkeithley_bat anyway, we don't get a el5 build until we get this into the tree. (unless we patch....)
20:45 ndevos kkeithley_bat: I think el5 can wait a little longer, no need to rush it ;-)
20:46 kkeithley_bat I'm certainly not in any rush
20:47 kkeithley_bat and not interested adding patches to the rpm builds
20:58 ndevos wohoo, agreement!
21:00 kkeithley_bat it wasn't that hard
21:06 badone_ joined #gluster-dev
21:08 ndevos I didnt give up easily either
21:10 dlambrig left #gluster-dev
21:49 jbautista- joined #gluster-dev
21:55 jbautista- joined #gluster-dev
23:05 shyam joined #gluster-dev
23:16 JoeJulian I swear I'm going to add a bot to the user mailing list that googles for people.
23:43 badone_ joined #gluster-dev
23:47 badone__ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary