Camelia, the Perl 6 bug

IRC log for #gluster-dev, 2013-05-06

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:15 itisravi joined #gluster-dev
01:56 nickw joined #gluster-dev
02:48 itisravi joined #gluster-dev
02:51 lalatenduM joined #gluster-dev
02:56 vshankar joined #gluster-dev
03:51 bharata joined #gluster-dev
04:00 sgowda joined #gluster-dev
05:02 bulde joined #gluster-dev
05:07 mohankumar joined #gluster-dev
05:07 bala1 joined #gluster-dev
05:09 aravindavk joined #gluster-dev
05:14 lalatenduM joined #gluster-dev
05:16 hagarth joined #gluster-dev
05:18 hagarth JoeJulian: #gluster has turned invite onyl
05:19 hagarth s/onyl/only
05:31 bulde1 joined #gluster-dev
05:34 rastar joined #gluster-dev
05:41 bala1 joined #gluster-dev
05:49 bala1 joined #gluster-dev
05:55 nickw joined #gluster-dev
05:58 bala1 joined #gluster-dev
06:05 rgustafs joined #gluster-dev
07:03 bulde joined #gluster-dev
07:09 hagarth joined #gluster-dev
08:27 spider_fingers joined #gluster-dev
08:27 spider_fingers hi
08:27 spider_fingers i wonder why i can't join #gluster channel anymore
08:28 spider_fingers it says i need an invitation ;0
08:29 hagarth spider_fingers: glusterbot likes locking us out of #gluster at times :)
08:30 spider_fingers hagarth: hmm it seems it's the first time it did that to me
08:30 hagarth spider_fingers: no , this has been the third time over the last 3-4 months, IIRC.
08:31 spider_fingers so what am i supposed to do? wait?
08:39 mohankumar__ joined #gluster-dev
08:46 ndevos spider_fingers: yeah, wait until one of the channel ops opens up the channel again...
08:46 spider_fingers thx
08:49 lalatenduM joined #gluster-dev
08:50 hagarth joined #gluster-dev
08:56 bharata joined #gluster-dev
09:03 rgustafs joined #gluster-dev
09:09 rastar joined #gluster-dev
09:15 edward1 joined #gluster-dev
09:20 sgowda joined #gluster-dev
09:28 shubhendu joined #gluster-dev
09:30 kanagaraj joined #gluster-dev
09:34 bharata joined #gluster-dev
09:45 mohankumar__ joined #gluster-dev
09:46 sahina joined #gluster-dev
09:48 sgowda joined #gluster-dev
09:59 shubhendu joined #gluster-dev
10:03 bharata joined #gluster-dev
10:03 sgowda joined #gluster-dev
10:24 shubhendu joined #gluster-dev
10:24 rastar joined #gluster-dev
10:34 sgowda joined #gluster-dev
10:37 deepakcs joined #gluster-dev
10:49 rastar joined #gluster-dev
10:58 yinyin joined #gluster-dev
11:07 spider_fingers left #gluster-dev
11:08 hagarth joined #gluster-dev
11:16 rgustafs joined #gluster-dev
11:19 an joined #gluster-dev
11:47 nickw joined #gluster-dev
11:50 nickw Hello. I hate to bother you guys here but I'd like to know why #gluster couldn't be joined from yesterday?
11:51 nickw the channel says '#gluster Cannot join channel (+i) - you must be invited'
11:55 hagarth nickw: waiting on JoeJulian to set it right.
11:55 nickw hagarth: good to hear that. thank you
11:56 rastar joined #gluster-dev
12:06 shubhendu joined #gluster-dev
12:34 rastar joined #gluster-dev
12:59 mohankumar__ joined #gluster-dev
13:20 bulde joined #gluster-dev
13:33 shubhendu joined #gluster-dev
13:52 jclift_ joined #gluster-dev
13:52 jclift_ That's weird.  #gluster is saying it's "invite only" atm
13:53 ndevos jclift_: yeah, JoeJulian seems to be the one that is supposed to fix that...
13:54 jclift_ Oops ;)
13:54 awheeler_ jclift_: Sent you an invite
13:54 * jclift_ looks
13:54 * ndevos tries that too: 15:54 -!- #gluster You're not a channel operator
13:54 awheeler_ oh, lol, figured it wouldn't work
13:55 ndevos @channelstats
13:55 glusterbot ndevos: On #gluster-dev there have been 17111 messages, containing 427178 characters, 73595 words, 432 smileys, and 47 frowns; 168 of those messages were ACTIONs. There have been 10047 joins, 144 parts, 9897 quits, 0 kicks, 0 mode changes, and 0 topic changes. There are currently 36 users and the channel has peaked at 49 users.
13:57 bulde1 joined #gluster-dev
14:00 wushudoin joined #gluster-dev
14:13 Supermathie So I HATE to beg, but PLEASE does anyone have time to look at my 'reply submission failed' issue? http://pastie.org/7803022
14:24 nixpanic joined #gluster-dev
14:24 nixpanic joined #gluster-dev
14:30 bulde joined #gluster-dev
14:44 lpabon joined #gluster-dev
15:06 jbrooks joined #gluster-dev
15:07 rastar joined #gluster-dev
15:09 jclift_ awheeler_: yeah, haven't seen invite anywhere.
15:13 ndevos jclift_: semiosis fixed it :)
15:13 jclift_ Cool. :)
15:14 jclift_ semiosis++
15:14 semiosis :)
15:15 Supermathie Does gluster/nfs use Linux's rpc libraries/kernel module for receiving requests & enqueueing replies?
15:15 nickw cool, semiosis+1
15:15 ndevos Supermathie: no, glusterfs does everything in userspace
15:24 * jclift_ noticed the other day that the NFS capabilities seem to be in their own translator
15:24 ndevos yeah, neat, isnt it?
15:24 jclift_ Yeah, haven't looked through that code tho. :)
15:35 Supermathie Speaking of, if anybody wants to critique this diagram and inform me what I've screwed up, it'd be appreciated: http://www.websequencediagrams.com/fi​les/render?link=Iy1dl46ejLv0p2srrCEH :D
16:10 lalatenduM joined #gluster-dev
17:12 __Bryan__ joined #gluster-dev
17:39 Supermathie I'm starting to suspect that some part of the RPC subsystem is unable to handle large read/write requests (multiple packets bundled into a larger call)
17:42 __Bryan__ joined #gluster-dev
18:59 hagarth joined #gluster-dev
19:45 kkeithley hagarth_,avati_: there was some discussion before the BLR dev summit about packaging libgfapi in its own rpm (along with a matching -devel rpm). johnmark put it on the agenda but we never discussed it.
19:45 kkeithley What are your thoughts? Separate RPM?
19:50 hagarth kkeithley: am inclined to do it that way, I think a2 has some thoughts around that.
20:04 a2 kkeithley, yes..a  glusterfs-api and glusterfs-api-devel will be a useful "split" i think.. a lot of changes going into gfapi (which is still new) would be good to be upgradable independent of core glusterfs rpm
20:06 a2 Supermathie, network IO is handled in rpc/* and libglusterfs/event*
20:06 kkeithley Yes, that's what I thought I remembered you saying—  johnmark seemed determined that it needed to be discussed. Anyway....
20:09 Supermathie a2: I'm now fairly sure that it's something to do with gluster's handling of a whole bunch of RPC requests coming in on a single packet.
20:12 a2 Supermathie, what packet? glusterfs only understands streams
20:12 a2 (tcp)
20:13 a2 the reply submission failed is typically when the client (or someone else) has disconnected the tcp connection to the server
20:13 a2 [2013-05-05 01:31:16.259641] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: writev failed (Connection reset by peer) <-- see this.. the "reply submission failed" further on is just a cascaded effect of this log entry
20:14 a2 probably the client disconnected the rpc because of a timeout?
20:18 Supermathie a2: Nope, the tcp session is still established, see the bug I posted.
20:18 Supermathie a2: It's actually responding to *some* RPC calls in the same tcp stream, just not ones with >1 RPC call in a single packet. Looks like the RPC handling has the problem.
20:23 a2 Supermathie, are you sure there were no TWO tcp connections established for the RPC? lot of NFS clients do that
20:23 a2 and just one of them disconnected?
20:25 Supermathie a2: Yes, I'm sure. That 'connection reset by peer' message REALLY looks more like the nfs server is disconnecting from some other part of gluster
20:27 Supermathie a2: OK, now not so sure. But the client will reconnect, reissue the exact same RPCs and still get ignored.
20:29 Supermathie a2: https://www.dropbox.com/s/nfqxhxi​8xtnw7fp/gluster-rpc-failure.pcap
20:29 Supermathie There you go, a complete connection.
20:34 a2 the 'connection reset by peer' is surely not nfs<-->gluster component
20:35 JoeJulian a2: http://ur1.ca/dpx4h is a trace of a client that is connecting over and over again (several times/second) to glusterd... Where does this sequence start? I'll try reading through it and see what I can find.
20:36 a2 JoeJulian, "[2013-05-06 13:33:34.646716] D [client-handshake.c:1717:client_query_portmap_cbk] 1-wpkg-client-11: failed to get the port number for remote subvolume"
20:37 a2 JoeJulian, is the brick alive? gluster volume status shows anything?
20:37 JoeJulian Yep, brick's alive and if I restart the client it'll connect and stop spamming.
20:39 JoeJulian I thought it might be "connect () called on transport already connected"
20:39 a2 JoeJulian, what version is this?
20:41 JoeJulian glusterfs-fuse-3.3.1-14.el5
20:47 a2 Supermathie, strange that there aren't any rpc replies in that pcap
20:47 a2 hmph, RST initiated by server
20:48 a2 JoeJulian, how did you get into that situation?
20:52 JoeJulian a2: no idea... discovered the log was growing fast and that was why.
20:54 Supermathie a2: Yeah, exactly. But it's responding to other, shorter, SSDs.
20:54 Supermathie s/SSDs/RPCs/
20:54 Supermathie (freudian slip :D )
20:54 a2 Supermathie, but why isn't pcap showing the other RPC req/replies?
20:55 Supermathie a2: that was only a single connection
20:57 JoeJulian Looks like it all started right after I set server.allow-insecure on (for some unrelated testing).
20:58 JoeJulian No, that wouldn't have triggered a graph change on the client...
21:01 a2 JoeJulian, hmm, is this issue only with old/already-mounted clients? or does a fresh client mount also have this issue?
21:03 JoeJulian No, a fresh client mount is fine. I had 3 volumes that were doing this. umount ; mount and they're fine now. I left one, though, in case I could diagnose something useful.
21:20 JoeJulian a2: Uninitialized value maybe? The wireshark capture shows the GFS ID = 0xbabe
21:24 JoeJulian Oh! It's asking for the port for a brick that's been replaced.
21:27 JoeJulian I did a "replace-brick wpkg ewcs7:/var/spool/glusterfs/d_wpkg ewcs7:/data/glusterfs/wpkg/d/brick commit force" and that client's still asking for ewcs7:/var/spool/glusterfs/d_wpkg in PORTBYBRICK
21:27 JoeJulian a2
21:45 Supermathie a2: any idea? I think the error messages in the logs are a symptom, not the root cause. gluster gets into this 'failed' state way before those messages show in the logs.
22:09 johnmark kkeithley: a2: hagarth_: yeah, let's do that. what about an SDK split, where we throw in glupy as well, and just make a dev tools package
22:09 johnmark hagarth_: have you arrived yet?
22:09 Supermathie a2: It kind of sounds like *this* is happening: http://review.gluster.org/#/c/6​71/2/xlators/nfs/lib/src/rpcsvc.c
22:10 Supermathie oops - that's the patch which no longer applies to 3.3.1, but the original linked issue is similar.
22:10 Supermathie Question 2: WHY is the transmission buffer getting full? That seems to be my root+1 cause
23:34 yinyin joined #gluster-dev
23:34 yinyin_ joined #gluster-dev
23:35 yinyin joined #gluster-dev
23:36 yinyin- joined #gluster-dev
23:44 yinyin joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary