Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2015-07-06

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:28 kdhananjay joined #gluster-dev
02:42 shaunm_ joined #gluster-dev
03:42 nishanth joined #gluster-dev
03:43 atinm joined #gluster-dev
03:58 gem joined #gluster-dev
04:01 itisravi joined #gluster-dev
04:15 sakshi joined #gluster-dev
04:21 shubhendu joined #gluster-dev
04:26 nbalacha joined #gluster-dev
04:27 kshlm joined #gluster-dev
04:42 nbalacha joined #gluster-dev
04:45 ppai joined #gluster-dev
04:49 kanagaraj joined #gluster-dev
04:49 krishnan_p joined #gluster-dev
04:53 spandit joined #gluster-dev
04:55 ndarshan joined #gluster-dev
05:00 nkhare joined #gluster-dev
05:00 atalur joined #gluster-dev
05:04 kotreshhr joined #gluster-dev
05:05 deepakcs joined #gluster-dev
05:05 pppp joined #gluster-dev
05:09 vmallika joined #gluster-dev
05:10 Guest24908 joined #gluster-dev
05:19 shubhendu joined #gluster-dev
05:23 vimal joined #gluster-dev
05:30 rafi joined #gluster-dev
05:33 Bhaskarakiran joined #gluster-dev
05:36 hgowtham joined #gluster-dev
05:39 Manikandan joined #gluster-dev
05:42 ashiq joined #gluster-dev
05:55 anmol joined #gluster-dev
05:56 Saravana_ joined #gluster-dev
05:56 jiffin joined #gluster-dev
05:57 rjoseph joined #gluster-dev
05:59 hgowtham joined #gluster-dev
06:00 anekkunt joined #gluster-dev
06:02 G_Garg joined #gluster-dev
06:04 raghu joined #gluster-dev
06:05 Bhaskarakiran joined #gluster-dev
06:05 overclk joined #gluster-dev
06:07 hagarth joined #gluster-dev
06:11 kdhananjay joined #gluster-dev
06:23 atalur joined #gluster-dev
06:26 josferna joined #gluster-dev
06:27 spalai joined #gluster-dev
06:35 kdhananjay joined #gluster-dev
06:35 Manikandan ashiq, thanks
06:35 Manikandan ashiq++
06:35 glusterbot Manikandan: ashiq's karma is now 7
06:50 nishanth joined #gluster-dev
06:58 hchiramm gem++
06:58 glusterbot hchiramm: gem's karma is now 16
06:58 hchiramm atinm, do u know the uuid fix have been ported to release 3.7
06:59 atinm hchiramm, which fix are you talking about? the one which I sent few days back?
06:59 atinm hchiramm, thats still in review in mainline and I've not ported it to 3.7
06:59 hchiramm ok..
07:00 hchiramm atinm++
07:00 glusterbot hchiramm: atinm's karma is now 14
07:02 Manikandan anoopcs, thanks
07:02 Manikandan anoopcs++
07:02 glusterbot Manikandan: anoopcs's karma is now 13
07:16 shubhendu joined #gluster-dev
07:22 ndevos hi hchiramm, it seems that not all linked pages have been migrated to readthedocs?
07:23 ndevos hchiramm: or, maybe it's just an incorrect link...
07:23 ndevos the links on http://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/HA%20for%20Ganesha/ that is
07:23 hchiramm ndevos, checking
07:24 hchiramm yeah, could be an incorrect entry
07:24 hchiramm will fix it
07:24 ndevos hchiramm++ thank you
07:24 glusterbot ndevos: hchiramm's karma is now 49
07:25 hchiramm yw
07:29 ndevos oh, at one time I signed up to take an exam, and it seems to be planned for tomorrow: Red Hat Certificate of Expertise in Hybrid Cloud Storage EX236K
07:30 ndevos I wonder if I need to practise things on Red Hat Storage 2.x...
07:34 Manikandan joined #gluster-dev
07:37 ndevos hmm, when I pass that exam, it's the 5th certification I have, meaning that I'll be an RHCA after that
07:37 * ndevos becomes nervous
07:41 vimal joined #gluster-dev
07:41 gem anoopcs++
07:41 glusterbot gem: anoopcs's karma is now 14
07:45 hchiramm ndevos, all the best :)
07:45 ppai joined #gluster-dev
07:48 ndevos thanks hchiramm
07:49 vimal joined #gluster-dev
07:52 nbalacha joined #gluster-dev
07:54 Manikandan joined #gluster-dev
08:00 ndevos hi kshlm: that compile warning with
08:00 ndevos GF_REPLACE_OP_START only happens on release-3.6, did you test that?
08:03 gem joined #gluster-dev
08:12 pranithk joined #gluster-dev
08:15 xavih pranithk: Are you there? I'm reviewing the http://review.gluster.org/11531/
08:16 pranithk xavih: Actually code changes in ec-common.c are not needed... I thought inode->good should never have stale bricks but my understanding is wrong.
08:16 pranithk xavih: :-)
08:16 pranithk xavih: I can remove the changes in ec-common.c and the fix should still work
08:16 xavih pranithk: that was one of my concerns :)
08:17 xavih pranithk: but I think we can do that in a more generic way that will work with any read-like fop sent as part of another write-like fop
08:17 xavih pranithk: what do you think about adding fop->mask &= fop->parent->mask in ec_child_select() when fop->parent != NULL ?
08:19 xavih pranithk: since a successful xattrop will update the mask of a fop before issuing any subfop (like a readv in a writev), we can use that mask to filter bad bricks in subfops
08:20 gem joined #gluster-dev
08:21 pranithk xavih: In the morning I was thinking the same :-)
08:21 pranithk xavih: But I wasn't sure why it wasn't done that way. I will do that and test it again
08:22 pranithk xavih: We don't have any fops where the extra internal fops are performed on superset than the actual fop at the moment?
08:23 pranithk xavih: I will check that once.
08:23 xavih pranithk: this is bad for locks (they need to be sent to good and bad bricks), but since locks are executed before xattrop, it won't have any effect
08:23 xavih pranithk: there were many in self-heal, but currently we don't have any other I think
08:24 xavih pranithk: I'm not sure why I didn't do that on the first place. Probably I missed it... :(
08:24 pranithk xavih: cool :-)
08:25 pranithk xavih: Could you review http://review.gluster.org/11546 while I redo this patch
08:26 xavih pranithk: np, but remember to add me as a reviewer. It will be easier for me to follow reviews and do it sooner...
08:26 pranithk xavih: I just sent it... :-)
08:26 pranithk xavih: I will
08:27 xavih pranithk: ok :)
08:27 pranithk xavih: wait
08:28 pranithk xavih: the fix we thought won't work
08:28 pranithk xavih: unlock's mask should not be touched
08:28 pranithk xavih: If fop fails on some of the bricks where the fop failed then unlock won't be sent there...
08:28 xavih pranithk: oh, true that was why I didn't do that. I've just remembered... :(
08:28 pranithk xavih: great!!
08:28 pranithk xavih: so I will just remove changes in ec-common.c
08:29 pranithk xavih: then check if there are any other fops which do similar thing
08:29 pranithk xavih: ok?
08:29 xavih pranithk: I would like a more generic solution...
08:29 pranithk xavih: ah!
08:30 pranithk xavih: wait
08:30 pranithk xavih: We will do this mask thing only when the mask is -1?
08:30 soumya joined #gluster-dev
08:30 pranithk xavih: unlock will never come with -1
08:31 pranithk xavih: but that is brittle
08:31 xavih pranithk: why mask won't be -1 for unlock ?
08:32 pranithk xavih: because we send unlock on lock->mask which already has bricks that are locked... oh you are saying it can come from fops above :-/
08:32 pranithk xavih: yeah you are right, that won't work either :-(
08:33 xavih pranithk: what if we reset fop->parent->mask to -1 on ec_unlock_lock() ?
08:35 pranithk xavih: I was thinking we will add compund xlator which will do lock+xattrop+heal/tail read in single network fop. When that comes in, this will give a problem. But for now we can do this I suppose.
08:41 kshlm ndevos, I
08:41 kshlm ndevos, I've tested that on netbsd7 and freebsd10, but couldn't reproduce it.
08:42 kshlm ndevos, I also want to test on netbsd6, but that will have to wait a little while as I have other stuff to look at.
08:42 kshlm Based on my first two tests, I'm leaning towards the problem being with the change in question or with the git cloning being done on the slaves.
08:51 pranithk xavih: if (fop->parent && !ec_must_wind(fop)) fop->mask &= fop->parent->mask;
08:51 pranithk xavih: I think that should be good?
08:52 xavih pranithk: yes, I think it's very good :)
08:52 pranithk xavih: great! patch coming up :-)
08:55 ndevos kshlm: oh, okay, thanks for checking
09:04 soumya_ joined #gluster-dev
09:14 gem joined #gluster-dev
09:25 kdhananjay joined #gluster-dev
09:36 vimal joined #gluster-dev
09:42 kotreshhr1 joined #gluster-dev
09:43 spandit joined #gluster-dev
09:50 kaushal_ joined #gluster-dev
09:51 pranithk xavih: It is working fine :-). Just resent it
09:56 xavih pranithk: It seems ok :)
09:58 anekkunt joined #gluster-dev
09:59 G_Garg joined #gluster-dev
09:59 gem joined #gluster-dev
10:04 spandit joined #gluster-dev
10:04 atinm joined #gluster-dev
10:05 shubhendu joined #gluster-dev
10:16 kotreshhr joined #gluster-dev
10:20 soumya_ joined #gluster-dev
10:36 pranithk xavih: Bhaskar just showed me a crash where the mount crashed because of there was a calloc for 34TB in iobuf.... Unfortunately it crashed a bit later ... brb
10:36 kotreshhr1 joined #gluster-dev
10:39 kotreshhr joined #gluster-dev
10:41 atinm joined #gluster-dev
10:42 Manikandan joined #gluster-dev
10:42 Manikandan_ joined #gluster-dev
10:46 soumya_ joined #gluster-dev
10:46 shubhendu joined #gluster-dev
10:50 gem_ joined #gluster-dev
10:51 spalai joined #gluster-dev
10:53 atinm rastar_afk, there?
10:54 atinm rastar_afk, could you take a look ! http://review.gluster.org/#/c/11271/ ?
10:55 atinm ]
10:55 ndevos hi atinm! send out the reminder for the bug truage meeting today?
10:57 atinm ndevos, one day in advance?
10:57 atinm ndevos, today is Monday right?
10:58 ndevos atinm: yes, indeed, one day in advance to remind people on the other side of the world :)
10:58 atinm ndevos, sure then :)
10:58 ndevos atinm: thanks!
10:59 atinm ndevos, yw :)
10:59 ndevos atinm: it probably also helps others to prepare a little more in advance, a one-hour notice is always rather short if you're not prepared
11:01 pranithk xavih: Could you give +2 again for http://review.gluster.org/11506 ? I had to rebase it because port allocation failures are giving problems in the regression runs. Fix went in after I sent this patch, so rebased it now.
11:05 xavih pranithk: done
11:05 vmallika joined #gluster-dev
11:06 pranithk xavih: Bhaskar just showed me a crash where the mount crashed because of there was a calloc for 34TB in iobuf.... Unfortunately it crashed a bit later ...
11:06 pranithk xavih: I think we have a bug in ec...
11:06 pranithk xavih: I am taking a look at all iobuf_get calls...
11:10 ndevos what, 34TB?! is that the one that first got assigned to me for gluster-nfs?
11:10 xavih pranithk: I think it's very unlikely an iobuf_get() could request 34TB. It always depends on the size received to a write or returned by a read...
11:11 xavih pranithk: what was the test doing ?
11:11 pranithk xavih: I saw the log :-) it was 34TB
11:12 pranithk xavih: He was just doing his normal tests, compilation/read/write/ etc even he doesn't know how to re-create it
11:12 ndevos pranithk: some negative value casted to an unsigned?
11:12 pranithk ndevos: that is what I am thinking :-)
11:14 * ndevos goes out for lunch, will be back later
11:20 kshlm joined #gluster-dev
11:22 pranithk xavih: please do the honors :-) http://review.gluster.com/11546
11:25 xavih pranithk: merged :)
11:25 pranithk xavih: thanks :-)
11:31 G_Garg joined #gluster-dev
11:33 kotreshhr joined #gluster-dev
11:35 spalai joined #gluster-dev
11:35 atalur joined #gluster-dev
11:35 anekkunt joined #gluster-dev
11:37 pranithk xavih: iobuf_get calls seem okay...
11:38 xavih pranithk: that's what I thought. I don't see how it can reach 34TB
11:38 xavih pranithk: do you have the exact size in bytes ?
11:39 pranithk xavih: [2015-07-06 07:45:23.832283] A [MSGID: 0] [mem-pool.c:120:__gf_calloc] : no memory available for size (34917844374315) [call stack follows]
11:39 pranithk /usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x7f07d7bed826]
11:39 pranithk /usr/lib64/libglusterfs.so.0(_gf_msg_nomem+0x370)[0x7f07d7bee530]
11:39 pranithk /usr/lib64/libglusterfs.so.0(__gf_calloc+0x139)[0x7f07d7c24959]
11:39 pranithk /usr/lib64/libglusterfs.so.0(iobuf_get_from_stdalloc+0x6e)[0x7f07d7c2633e]
11:39 pranithk /usr/lib64/libglusterfs.so.0(iobuf_get2+0xa3)[0x7f07d7c278e3]
11:39 anmol joined #gluster-dev
11:41 pranithk xavih: It is very small number compared to -1 in uint64_t
11:41 xavih pranithk: yes
11:46 spalai1 joined #gluster-dev
12:05 rajeshj joined #gluster-dev
12:11 dlambrig_ joined #gluster-dev
12:14 Manikandan joined #gluster-dev
12:18 jrm16020 joined #gluster-dev
12:20 ppai joined #gluster-dev
12:21 jrm16020 joined #gluster-dev
12:25 shubhendu joined #gluster-dev
12:27 kotreshhr joined #gluster-dev
12:28 itisravi_ joined #gluster-dev
12:29 soumya_ joined #gluster-dev
12:42 dlambrig_ joined #gluster-dev
12:44 pppp joined #gluster-dev
12:52 pranithk xavih: https://bugzilla.redhat.com/show_bug.cgi?id=1240284 is the bug for this... it crashed somewhere else...
12:52 glusterbot Bug 1240284: unspecified, unspecified, ---, bugs, NEW , Disperse volume: NFS crashed
12:52 pranithk xavih: May be it is a simple double unwind
12:56 hchiramm gem++ Thanks !!!
12:56 glusterbot hchiramm: gem's karma is now 17
12:57 mribeirodantas joined #gluster-dev
13:09 lpabon joined #gluster-dev
13:28 overclk left #gluster-dev
13:43 jobewan joined #gluster-dev
13:49 shyam joined #gluster-dev
13:49 Manikandan joined #gluster-dev
13:51 bfoster joined #gluster-dev
13:53 kotreshhr left #gluster-dev
13:53 ashiq Manikandan++ thanks :)
13:53 glusterbot ashiq: Manikandan's karma is now 11
14:03 pranithk xavih: are you there? I have a question about ec_loc_update
14:03 dlambrig_ joined #gluster-dev
14:04 pranithk xavih: why does it assume that loc->inode will be different than new inode that is passed?
14:06 xavih pranithk: It doesn't assume that it will be different, but if it's different, it's changed
14:07 pranithk xavih: How can it be different?
14:08 pranithk xavih: I am wondering... if it is, we are screwed :-(. Because inode-ctx is also gone!
14:09 xavih pranithk: this is useful when loc->inode is NULL, or if a lookup or any other fop changes the inode (at the time I wrote that, loc management was not very clear. I still have doubts about what can happen and what not)
14:10 pranithk xavih: Gah! I hate dht now :-(
14:10 pranithk xavih: You know what happens with dht?
14:10 pranithk xavih: It sends setattr + setxattr in lookup with fresh inode...
14:10 pranithk xavih: i.e. an inode that is not linked...
14:11 xavih pranithk: I'm not sure what's the problem exactly, but I¡ve had a lot of problems managing loc_t objects, that's why there's so checks and functions to manipulate them...
14:11 pranithk xavih: hmm... okay
14:13 xavih pranithk: unless you are absolutely sure, please don't remove anything. If you think something is not possible, we could add an assert and closely see if some test triggers it before removing any code
14:13 pranithk xavih: definitely.
14:13 pranithk xavih: Have you seen https://bugzilla.redhat.com/show_bug.cgi?id=1240284#c1?
14:13 glusterbot Bug 1240284: unspecified, unspecified, ---, bugs, NEW , Disperse volume: NFS crashed
14:13 pranithk xavih: remove '?' at the end
14:14 pranithk xavih: that is what is happening :-(
14:14 xavih pranithk: yes, many failures after the allocation error
14:14 xavih pranithk: is there anything before the memory error ?
14:15 xavih pranithk: anything interesting I mean :P
14:15 pranithk xavih: nothing...
14:15 nkhare joined #gluster-dev
14:15 xavih pranithk: the memory error is the first thing that happens ?
14:15 pranithk xavih: [2015-07-06 07:45:23.802987] I [MSGID: 109036] [dht-common.c:7106:dht_log_new_layout_for_dir_selfheal] 0-vol3-dht: Setting layout of <gfid:daa695d7-3a29-4c44-b3ee-502e12fa1571>/dir.1/dir.10/dir.635 with [Subvol_name: vol3-disperse-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol3-disperse-1, Err: 22 , Start: 0 , Stop: 0 , Hash: 0 ],
14:16 pranithk xavih: It has lot of messages like above, then memory error, then all those other errors and then a crash :-(
14:16 pranithk xavih: By the time memory error happened, something funny must have already happened
14:17 pranithk xavih: since it kept saying new_lay_out etc, I am assuming dht could have done something... but I am not sure.
14:21 xavih pranithk: if you multiply the alloc size by 4, it resembles a lot to a pointer: 0x7F07C6D74CAC
14:21 wushudoin joined #gluster-dev
14:22 xavih pranithk: an unaligned one, though (on a 64-bit architecture)
14:22 pranithk xavih: interesting
14:23 xavih pranithk: there's a place where iobuf_get2() is called after having divided the size by ec->fragments. Is it possible that Bhaskar did made some test with a 4+x volume ?
14:24 pranithk xavih: I just checked, it is 8+3 disperse
14:26 xavih pranithk: maybe it's only a coincidence...
14:26 pranithk xavih: another way to think about this would be to think of double stack-unwind which
14:27 pranithk xavih: lead to this
14:27 pranithk xavih: One more thing, if you look at the backtrace it crashed at data_unref...
14:30 xavih pranithk: could be. That could also explain the failed assertions on ec_check_complete(). But not sure how/if that can happen...
14:30 lpabon joined #gluster-dev
14:31 ira joined #gluster-dev
14:33 pranithk xavih: Daily bhaskar gives me new puzzle to solve :-)
14:33 pranithk xavih: This one is tough :-D
14:33 xavih pranithk: hehe
14:35 pranithk xavih: I wonder if there is a bug where fop->ref count becomes zero then back to one and again to zero...
14:35 pranithk xavih: leading to double free...
14:35 pranithk xavih: All of these are speculations at this point
14:37 pranithk xavih: it looks clean actually :-(
14:38 xavih pranithk: could you check if there's something at 0x7F07C6D74CAC in the core dump ? if it's really a pointer, maybe it gives some clue...
14:39 pranithk xavih: yes, give me a minute
14:42 spalai joined #gluster-dev
14:42 pranithk xavih: what exactly would you like me to find out? I rarely had to look at core with just addresses...
14:43 xavih pranithk: basically if it's a valid addres, and if it is, what does it contain
14:45 xavih pranithk: for example x/128xb 0x7F07C6D74CAC
14:45 pranithk xavih: it is valid address
14:45 pranithk xavih: I did x /64 though
14:46 pranithk xavih: I sent you pm with the output
14:46 xavih pranithk: I've seen it :)
14:47 pranithk xavih: cool sir!
14:48 lpabon_ joined #gluster-dev
14:49 pranithk xavih: it seems like some structure
14:49 pranithk xavih: with pointers
14:50 xavih pranithk: I think it's part of a mem_header struct
14:51 pranithk xavih: ah!
14:51 pranithk xavih: then we will know type :-)
14:51 xavih pranithk: there's the cafebabe magic number
14:51 pranithk xavih: yes :-)
14:55 shubhendu joined #gluster-dev
14:58 shyam joined #gluster-dev
15:02 dlambrig_ joined #gluster-dev
15:08 xavih pranithk: btw, and maybe unrelated with this specific problem, there are unaligned pointers. This is very dangerous with respect atomic operations because x86 processors do not guarantee that atomic operations are really atomic if the value crosses cache line boundaries...
15:08 pranithk xavih: Hmm...
15:09 xavih pranithk: recently have been added atomic operations for refcounting
15:09 pranithk xavih: the structure we are seeing is pointing to libglusterfs memory types...
15:09 pranithk xavih: yes
15:09 xavih pranithk: we shouldn't work with unaligned addresses...
15:10 pranithk xavih: hmm...
15:10 pranithk xavih: (gdb) p $3->rec[12]
15:10 pranithk $20 = {typestr = 0x7f07d7c6b5d3 "gf_common_mt_fd_ctx", size = 44544, max_size = 96048, num_allocs = 32, total_allocs = 20010, max_num_allocs = 69, lock = 1}
15:10 pranithk xavih: it seems to print correct info...
15:10 pranithk xavih: (gdb) p $3->rec[167]
15:10 pranithk $30 = {typestr = 0x7f07c97891b2 "gf_nfs_mt_inode_ctx", size = 495360, max_size = 495360, num_allocs = 20640, total_allocs = 20654, max_num_allocs = 20640, lock = 1}
15:11 pranithk xavih: the memory address is from nfs
15:13 pranithk xavih: now what do we do?
15:14 xavih pranithk: I'm not sure. Can you send me the core dump ?
15:15 pranithk xavih: It won't be useful for you without the machine... because the code is slightly different from upstream...
15:15 hagarth joined #gluster-dev
15:16 xavih pranithk: oh
15:19 lpabon_ joined #gluster-dev
15:20 dlambrig_ joined #gluster-dev
15:23 pranithk xavih: Damn it, it is very late. I better catch a bus home... cya
15:24 pranithk xavih: I will try and ask if it is recreatable for Bhaskar again...
15:24 pranithk xavih: cya
15:33 shyam joined #gluster-dev
15:35 dlambrig_ joined #gluster-dev
15:37 spalai joined #gluster-dev
15:37 shyam joined #gluster-dev
15:38 shubhendu joined #gluster-dev
15:46 shyam joined #gluster-dev
15:52 Saravana_ joined #gluster-dev
16:36 wushudoin joined #gluster-dev
16:40 shyam joined #gluster-dev
17:26 gem_ joined #gluster-dev
17:37 wushudoin| joined #gluster-dev
17:38 rafi joined #gluster-dev
17:43 wushudoin| joined #gluster-dev
17:44 G_Garg joined #gluster-dev
17:46 jbautista- joined #gluster-dev
17:59 spalai left #gluster-dev
18:04 jbautista- joined #gluster-dev
18:56 wushudoin| joined #gluster-dev
19:02 wushudoin| joined #gluster-dev
19:22 jiffin joined #gluster-dev
19:34 spalai joined #gluster-dev
19:51 jiffin joined #gluster-dev
20:17 jbautista- joined #gluster-dev
20:23 jbautista- joined #gluster-dev
20:31 dlambrig_ joined #gluster-dev
21:04 dlambrig_ joined #gluster-dev
21:40 dlambrig_ joined #gluster-dev
21:52 dlambrig_ joined #gluster-dev
23:12 dlambrig_ joined #gluster-dev
23:40 dlambrig_ joined #gluster-dev
23:51 dlambrig_ joined #gluster-dev
23:53 pranithk joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary