Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2017-05-26

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:26 pranithk1 joined #gluster-dev
01:48 ilbot3 joined #gluster-dev
01:48 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
02:02 susant joined #gluster-dev
02:52 mchangir joined #gluster-dev
03:28 ppai joined #gluster-dev
03:30 riyas joined #gluster-dev
03:35 nbalacha joined #gluster-dev
03:54 aravindavk joined #gluster-dev
03:58 itisravi joined #gluster-dev
04:29 skumar joined #gluster-dev
04:35 PotatoGim Hi! Does anyone know 'GF_RDMA_DEVICE_COUNT' is 8?
04:37 rafi joined #gluster-dev
04:45 gyadav joined #gluster-dev
04:45 gyadav__ joined #gluster-dev
04:46 ankitr joined #gluster-dev
04:57 Shu6h3ndu joined #gluster-dev
05:08 rafi joined #gluster-dev
05:13 hgowtham joined #gluster-dev
05:20 PotatoGim know "why" ~blah~
05:30 ndarshan joined #gluster-dev
05:40 karthik_us joined #gluster-dev
05:40 sanoj joined #gluster-dev
05:50 Saravanakmr joined #gluster-dev
05:54 apandey joined #gluster-dev
05:57 sona joined #gluster-dev
06:01 jiffin joined #gluster-dev
06:03 jiffin ndevos, kkeithley: can u please look into the https://review.gluster.org/#/c/17339/
06:03 Humble joined #gluster-dev
06:05 rafi joined #gluster-dev
06:06 hgowtham joined #gluster-dev
06:10 kdhananjay joined #gluster-dev
06:14 ashiq joined #gluster-dev
06:15 pranithk1 joined #gluster-dev
06:15 rafi1 joined #gluster-dev
06:17 apandey_ joined #gluster-dev
06:31 susant joined #gluster-dev
06:42 ankitr joined #gluster-dev
06:47 poornima_ joined #gluster-dev
06:47 Shu6h3ndu joined #gluster-dev
06:48 ankitr joined #gluster-dev
06:49 rastar joined #gluster-dev
06:51 ndarshan joined #gluster-dev
06:56 itisravi joined #gluster-dev
07:09 ankitr joined #gluster-dev
07:10 ankitr_ joined #gluster-dev
07:14 skumar joined #gluster-dev
07:15 PotatoGim Could anyone let me know why iobuf_arena.arena_size is equals to "(iobuf_pool->arena_size / page_size) * page_size"?
07:23 pranithk1 PotatoGim: Seems like some kind of rounding off.
07:24 pranithk1 kdhananjay: poornima_: do you guys know why? ^^
07:24 kdhananjay PotatoGim: where did you find this line of code? could you point me to the fucntion name?
07:27 PotatoGim yep, In libglusterfs/src/iobuf.h:88-90, some comments for iobuf_arena exist.
07:35 skumar joined #gluster-dev
07:38 riyas joined #gluster-dev
07:49 pranithk1 PotatoGim: It seems like the rounding off code is deleted in http://review.gluster.com/543, so I am not sure if that comment is valid anymore. I may have to ask another guy but he is having lunch now. So may be sometime later
07:51 PotatoGim pranithk1, kdhananjay: Thanks for your help! :)
07:57 aravindavk joined #gluster-dev
08:01 Shu6h3ndu joined #gluster-dev
08:40 kdhananjay PotatoGim: right. line 190 in __iobuf_arena_alloc() no longer follows that equation. So the C comment is dead code anyway. :)
08:44 PotatoGim kdhananjay: Aha. May I have remove the comment? :)
08:45 PotatoGim Or replace?
08:45 kdhananjay PotatoGim: sure, go ahead :)
08:45 PotatoGim kdhananjay: Thanks a lot! :)
08:48 ndarshan joined #gluster-dev
08:49 poornima_ joined #gluster-dev
08:49 skumar joined #gluster-dev
09:08 pranithk1 left #gluster-dev
09:19 poornima_ joined #gluster-dev
09:33 hgowtham joined #gluster-dev
09:37 skumar joined #gluster-dev
09:41 ankitr joined #gluster-dev
10:06 rafi1 joined #gluster-dev
10:57 poornima_ joined #gluster-dev
11:02 ndarshan joined #gluster-dev
11:30 xavih apandey_: are you around ?
11:32 apandey_ xavih: Yes..
11:33 xavih apandey_: is it a good time to talk about the issues you told me ? I've been busy till now...
11:34 apandey_ xavih: Yes, we can discuss those issues...
11:35 apandey_ xavih: There are two issues. One of data corruption and other is deadlock in ec_lk code
11:36 xavih apandey_: yes
11:37 apandey_ xavih: Have you tried running data corruption test?
11:39 xavih apandey_: for the ec_lk problem, maybe the easiest solution would be to not do all the locking logic (parallel non-bnlocking lock followed by sequential blocking lock if first attempt fails) since we have already acquired an inodelk. In this situation, lk request should compete with anyone else
11:39 itisravi joined #gluster-dev
11:39 xavih apandey_: not sure about backward compatibility though...
11:39 xavih no, I haven't had time to test anything
11:40 xavih apandey_: do you know if the corruption is introduced by the patch or it already existed ?
11:41 apandey_ xavih: No xavi , corruption was already existing...This patch has just exposed it..
11:41 apandey_ xavih: We have also checked the previous versions and found it there too..
11:44 apandey_ xavih: So you mean to say that whenever a ec_lk request comes we take inodelk which is a full lock, and any subsequent ec_lk from any client should be completed. We will not release inodelk until we get unlock request. Is this what you want to say?
11:48 xavih apandey_: lk requests should be treated as writing fops (i.e. only one of them can be active from a single client at any time)
11:48 apandey_ xavih: hmmmm..
11:48 xavih apandey_: however I think that it was done the way it's now to allow backward compatibility with older clients that do not use inodelk before lk
11:53 apandey_ xavih: let's keep backward compatibility aside. Suppose a lock request comes from application to ec_lk, it will take inodelk and then will not unlock until we get unlock request form application. The way write works is that it  takes inodelk , write dat and releases it..
11:54 xavih apandey_: no, inodelk is only taken during the life time of the lk requets (ignoring eager-locking)
11:56 ankitr joined #gluster-dev
11:56 rafi joined #gluster-dev
11:57 xavih apandey_: inodelk is taken, lk is sent in parallel and non-blocking to all bricks, if all bricks agree inodelk is released and lk returns successfully. If lk from bricks fail, an unlock of the lk is sent and inodelk released
11:57 xavih apandey_: however this doesn't support backward compatibility
11:57 apandey_ xavih: OK, Now I understand...
12:01 apandey_ xavih: I need time to think about this solution and the cases. So I will clarify it later if I have doubts
12:01 apandey_ xavih: About data corruption...
12:02 hgowtham joined #gluster-dev
12:03 apandey_ xavih: I could only figure out that the 512 bytes of on the boundary of heal and IO are getting corrupted. I could not find the ocde which could be responsible for that...Checked all the offsets and size and it was all fine..
12:04 apandey_ xavih: At least, this is what I found using logs and code browsing.
12:04 xavih apandey_: so the last 512 bytes of the file at the moment of starting the heal are the damaged ones ?
12:05 xavih apandey_: or the first 512 bytes written after the heal has started ?
12:07 xavih apandey_: ec does some reads when the write is not aligned to 512 bytes
12:07 apandey_ xavih:  last 512 bytes starting the file. let's say heal find size = X  so it will trim according to X on a brick which is suppose Y. so Y-512 to Y ...these are the bytes getting corrupted..
12:08 apandey_ xavih: This is on 2+1 volume..
12:08 apandey_ xavih: Yeh, that is what I am suspecting and I am sure there is something ...But code looks fine...and offset and size comes in logs are correct..
12:08 kkeithley @later tell shyam I'm getting "./rfc.sh: line 97: [: =: unary operator expected" in the release-3.11 rfc.sh
12:08 glusterbot kkeithley: The operation succeeded.
12:09 apandey_ xavih: I am only thinking a race case which I am not able to visualize.
12:10 xavih apandey_: but all this info (size X) is always read after having locked the inode, right ?
12:10 apandey_ xavih: exactly :)
12:10 xavih apandey_: and the trim is done before releasing the inode lock used to get the size, right ?
12:10 apandey_ xavih: trimming and marking of heal, everything happens under lock
12:10 apandey_ xavih: right
12:11 xavih apandey_: could it be caused by an xattrop still being processed while the inode lock is released ?
12:11 xavih apandey_: oh, one thing...
12:11 apandey_ xavih: could be..
12:12 xavih apandey_: is the X size being used later during the heal process ?
12:12 apandey_ xavih: yeh. I think so..
12:12 xavih apandey_: how is handled the last write in the heal process ?
12:13 xavih apandey_: if it only reads up to X, the remaining bytes of the 512 bytes block will be cleared
12:13 xavih apandey_: do you have one of those damaged 512 bytes block ?
12:13 apandey_ xavih: No no. That is not happening..
12:14 apandey_ xavih: not now, But I can come up with that. Do you think it will be NULL?
12:14 atinm joined #gluster-dev
12:14 apandey_ xavih: Because that was not the case..
12:15 xavih apandey_: not sure. It's one possibility
12:15 xavih apandey_: but seeing what's wrong compared to a healthy block, maybe we can deduce the cause
12:16 apandey_ xavih: let me see if I still have those files..
12:17 apandey_ xavih: I have sent the diff between two in a mail to you ..
12:18 apandey_ xavih: I think I have messed up the files of which I have taken diff..
12:18 xavih apandey_: true... let me see
12:20 xavih apandey_: it seems random. However it's an encoded fragment...
12:21 apandey_ xavih: Right, we can not figure out anything except it is different...
12:21 apandey_ xavih: I also had a doubt that  "could it be caused by an xattrop still being processed while the inode lock is released ?"
12:22 apandey_ xavih: But the size which is shown in logs does not suggest that..
12:24 apandey_ xavih: So we have a point where we trimmed a brick to heal, marked it as healing. write from continous IO arrived, it should write on all but could not write the data but fop passed.
12:24 PotatoGim Hi. Sorry to cut off... Is Reply feature in review.gluster.com working? When I click that, "Server error" happend...
12:29 gem joined #gluster-dev
12:30 xavih apandey_: the problem is that heal is using an independe inode lock, so healing and writes cannot happen at the same time
12:30 apandey_ xavih: Yes..
12:30 xavih apandey_: the regular write will need to acquire the inode lock again, which in turn will also check the current file size
12:31 xavih apandey_: so the regular write should work as expected, exactly as if another client was writing to the file at the same time
12:31 xavih apandey_: and as long as I know, this works
12:31 xavih apandey_: *as far as*
12:32 xavih apandey_: so I think the problem needs to be in some data acquired during the heal that is reused when it might not be valid
12:33 apandey_ xavih: Yes Xavi, Heal and write is happening completely independent and it looks very clean.
12:33 apandey_ xavih: data acquired?
12:33 apandey_ xavih: It could be...
12:34 apandey_ xavih: But for every block it is reading data fresh
12:34 xavih apandey_: I've just seen something...
12:35 apandey_ xavih: ok..
12:38 apandey_ xavih: In ec_rebuild_data, we send healing in a block of 128k. so  if the for loop sends blocks to heal which goes beyond "size", will that have any diff?
12:40 xavih apandey_: it shouldn't. If a write doesn't fill the whole block, zeros are appended, so reading the full block, even after the real size of the file, will return zeros
12:40 apandey_ xavih: okk..
12:46 xavih apandey_: let's see if this is possible...
12:47 xavih apandey_: when a brick is bring healed, the file is truncated, so basically we have it full of zeros
12:47 apandey_ xavih: right.
12:47 xavih apandey_: when another client tries to write the file, it will see that the brick is damaged and it's being healed
12:48 xavih apandey_: in this case the client *will* send the write to that brick, right ?
12:48 apandey_ xavih: right
12:48 xavih apandey_: even if it's being healed
12:48 xavih apandey_: so, what happens if the write is partial. I mean that the offset is not aligned to the stripe size.
12:48 xavih apandey_: in this case the write fop will send a read as a subfop
12:49 xavih apandey_: can this read be processed by the damaged brick ?
12:49 apandey_ xavih: :)
12:49 xavih apandey_: the bad brick would return zeros
12:49 apandey_ xavih: No, It is not happening.
12:49 xavih apandey_: and combining it with the other data will return garbage
12:51 apandey_ xavih: In ec_child_select, we are taking care of that in first few lines
12:51 xavih apandey_: yes, I've just seen it...
12:52 apandey_ xavih: In heal path also I thought that it could be reading from bad bricks only. But even that is fine...they are using heal->bad and heal->good  clearly..
12:56 gem_ joined #gluster-dev
12:57 xavih apandey_: I'm not seeing anything...
12:57 apandey_ xavih: okk.
12:58 xavih apandey_: I'll try to execute some tests to see if I find something else...
12:58 xavih apandey_: I'll tell you something on monday
12:58 apandey_ xavih: No prob, in case you have any idea let us know. In case you want to reproduce this issue. just run test case provided in patch ..
12:59 xavih apandey_: thanks
12:59 apandey_ xavih: You might have to rerun it in a loop..coz it does not faile everytime
12:59 xavih apandey_: how many times does it fail ?
12:59 apandey_ xavih: 6 out of 10 times...
12:59 xavih apandey_: that a lot of times...
13:00 xavih apandey_: ok, I'll test it
13:00 apandey_ xavih: yeh..
13:00 apandey_ xavih: Sure, will see you on Monday..
13:00 apandey_ xavih: I will also think about lock solution you proposed...
13:01 xavih apandey_: well, it's not a solution because it will have problems with older versions...
13:01 xavih apandey_: have to go. I'll tell you something on monday
13:01 apandey_ xavih: sure..
13:01 apandey_ xavih: Bye and have a happy weekend..
13:02 xavih apandey_: have a good weekend
13:23 susant joined #gluster-dev
13:24 jstrunk joined #gluster-dev
14:00 nbalacha joined #gluster-dev
14:01 shyam joined #gluster-dev
14:08 jiffin joined #gluster-dev
14:29 jiffin joined #gluster-dev
14:32 jiffin joined #gluster-dev
14:38 sanoj joined #gluster-dev
15:04 shyam joined #gluster-dev
15:05 wushudoin joined #gluster-dev
15:05 wushudoin joined #gluster-dev
15:12 gyadav_ joined #gluster-dev
15:12 gyadav joined #gluster-dev
15:26 shyam joined #gluster-dev
15:27 susant joined #gluster-dev
15:28 nbalacha joined #gluster-dev
15:36 Shu6h3ndu joined #gluster-dev
15:45 mchangir obnox, CVE-2017-7494 !
15:46 mchangir obnox, just read hacker news :)
16:00 susant joined #gluster-dev
16:20 jstrunk joined #gluster-dev
16:28 Gambit15 joined #gluster-dev
16:34 gyadav_ joined #gluster-dev
16:34 nbalacha joined #gluster-dev
16:35 gyadav joined #gluster-dev
17:03 nbalacha joined #gluster-dev
17:10 rafi1 joined #gluster-dev
17:17 susant joined #gluster-dev
17:23 rastar joined #gluster-dev
17:38 nbalacha joined #gluster-dev
17:49 shyam joined #gluster-dev
17:51 susant left #gluster-dev
18:17 riyas joined #gluster-dev
18:19 shyam joined #gluster-dev
19:59 shyam joined #gluster-dev
20:00 rastar joined #gluster-dev
20:11 gyadav joined #gluster-dev
20:11 gyadav_ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary