Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2015-07-03

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 badone_ joined #gluster-dev
00:42 pranithk joined #gluster-dev
01:20 kdhananjay joined #gluster-dev
01:31 pranithk joined #gluster-dev
01:47 ilbot3 joined #gluster-dev
01:47 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
02:12 RedW joined #gluster-dev
03:27 ppai joined #gluster-dev
03:35 josferna joined #gluster-dev
03:38 josferna joined #gluster-dev
03:51 atinm joined #gluster-dev
03:54 sakshi joined #gluster-dev
03:57 atinmu joined #gluster-dev
04:00 itisravi joined #gluster-dev
04:07 shubhendu joined #gluster-dev
04:17 nbalacha joined #gluster-dev
04:19 pppp joined #gluster-dev
04:28 hagarth joined #gluster-dev
04:34 RedW joined #gluster-dev
04:39 gem joined #gluster-dev
04:42 anmolb joined #gluster-dev
04:47 ashishpandey joined #gluster-dev
04:48 nkhare joined #gluster-dev
04:54 atinmu nbalacha, pm
04:55 atinmu nbalacha, are you aware that gf_msg_callingfn doesn't log the callers like its counterpart gf_log_callingfn ?
04:55 nbalacha atinmu, no
04:56 nbalacha atinmu, let me take a look
04:56 hchiramm atinmu++
04:56 glusterbot hchiramm: atinmu's karma is now 21
04:56 vimal joined #gluster-dev
04:58 atinmu nbalacha, [2015-07-03 04:25:49.454263] E [glusterd-op-sm.c:249:glusterd_get_txn_opinfo] (--> 0-management: Unable to get       transaction opinfo for transaction ID : 4ad5f5bb-8a83-4c06-9883-c3ed572ad77d [Permission denied]
04:58 glusterbot atinmu: ('s karma is now -7
04:58 atinmu nbalacha, this is a gf_msg_callingfn call
04:59 atinmu nbalacha, And now I am worried that we are passing errno in few places which might give a wrong impression to the admin, we need to be selective and careful about errno
05:00 atinmu nbalacha, I feel "Permission denied" in the above log can confuse admins
05:01 nbalacha atinmu, agreed
05:01 nbalacha atinmu, what is the correct errno for that?
05:04 atinmu nbalacha, In this case we should pass it 0
05:05 spandit joined #gluster-dev
05:05 nbalacha gf_msg_callingfn (this->name, GF_LOG_ERROR, errno,
05:05 nbalacha GD_MSG_DICT_GET_FAILED,
05:05 nbalacha "Unable to get transaction opinfo "
05:05 nbalacha "for transaction ID : %s",
05:05 nbalacha uuid_utoa (*txn_id));
05:05 nbalacha atinmu, looks like errno should not have been passed here
05:08 ndarshan joined #gluster-dev
05:10 ppai joined #gluster-dev
05:14 deepakcs joined #gluster-dev
05:20 hgowtham joined #gluster-dev
05:23 Manikandan joined #gluster-dev
05:24 G_Garg joined #gluster-dev
05:28 ashiq joined #gluster-dev
05:29 atinmu nbalacha, yeah, it should have been 0
05:30 atinmu nbalacha, and its not logging the callers as well
05:30 nbalacha atinmu, will take a look at that
05:31 atinmu nbalacha, thanks you
05:36 rjoseph joined #gluster-dev
05:40 rafi joined #gluster-dev
05:42 anekkunt joined #gluster-dev
05:42 atalur joined #gluster-dev
05:43 Bhaskarakiran joined #gluster-dev
05:45 asengupt joined #gluster-dev
05:47 pranithk joined #gluster-dev
05:52 kdhananjay joined #gluster-dev
05:53 anrao joined #gluster-dev
05:54 jiffin joined #gluster-dev
05:57 tigert joined #gluster-dev
05:59 hchiramm tigert, Good Morning
05:59 anrao joined #gluster-dev
06:01 ashiq joined #gluster-dev
06:05 raghu joined #gluster-dev
06:05 krishnan_p joined #gluster-dev
06:09 hchiramm tigert, how can I navigate to second page of planet.gluster.org ?
06:10 kotreshhr joined #gluster-dev
06:10 tigert hchiramm: morning
06:11 tigert hchiramm: I am actually on summer vacation now, but I think we need to ask garrett
06:11 hchiramm oh..ok :) .. looks like he is not available here..
06:11 tigert hchiramm: I am not sure actually if planet keeps a log of the feeds it pulls
06:11 tigert yeah no worries
06:11 hchiramm tigert, njoy ur vacation..
06:12 tigert he is on #osas on the internal irc at least
06:12 tigert thanks :)
06:12 hchiramm thanks tigert==
06:12 hchiramm tigert++
06:12 glusterbot hchiramm: tigert's karma is now 12
06:12 hchiramm tigert, I will check with him
06:12 tigert but I guess the planet does not keep track of the feed forever
06:12 tigert its more useful to see and follow what is going on right now
06:12 hchiramm hmmm.. thats sad .. Isnt it
06:13 tigert sort of yeah, but to do a canonical log of the whole planet would be interesting thought
06:13 tigert needs to be thought through, maybe it could be done
06:13 hchiramm k.. any way  I will cross check with garret
06:13 tigert but if you can reach garrett you can see what he thinks, he wrote the implementation that is in planet gluster
06:13 tigert cool
06:14 * tigert will be back beginning of august
06:14 hchiramm sure.. :)
06:14 hchiramm tigert, cya
06:14 vmallika joined #gluster-dev
06:16 rgustafs joined #gluster-dev
06:16 kdhananjay1 joined #gluster-dev
06:24 spalai joined #gluster-dev
06:28 vimal joined #gluster-dev
06:29 atalur_ joined #gluster-dev
06:39 schandra joined #gluster-dev
06:40 nbalacha atinmu, is there a BZ for the calling function not printing the caller?
06:41 atinmu nbalacha, I don't think so
06:41 nbalacha atinmu, ok, I will file one
06:41 atinmu nbalacha, great :)
06:48 gem joined #gluster-dev
06:51 Manikandan joined #gluster-dev
07:08 atinmu nbalacha, thanks for filing the bug
07:08 nbalacha atinmu, no problem
07:08 atinmu nbalacha, should we consider it as a blocker?
07:08 atinmu nbalacha, as its a regression IMO
07:08 nbalacha atinmu, I would - it hinders debugging
07:09 atinmu nbalacha, exactly
07:13 atalur_ joined #gluster-dev
07:13 gem joined #gluster-dev
07:14 atinmu krishnan_p, mind to take a look @ http://review.gluster.org/#/c/11271/ ? its merge ready
07:14 atinmu krishnan_p, so as http://review.gluster.org/#/c/10838/
07:23 gem joined #gluster-dev
07:23 pranithk xavih: Bhaskar gave the logs and file which was corrupted. I will go for lunch in a bit. I will also be working on this bug now... let me know how I can help you...
07:54 kdhananjay joined #gluster-dev
08:17 kdhananjay joined #gluster-dev
08:28 anmolb joined #gluster-dev
08:29 shubhendu joined #gluster-dev
08:30 ndarshan joined #gluster-dev
08:41 atalur joined #gluster-dev
08:46 anmolb joined #gluster-dev
08:47 shubhendu joined #gluster-dev
09:01 gem joined #gluster-dev
09:13 xavih pranithk: are you here ?
09:30 Saravana_ joined #gluster-dev
09:31 pranithk xavih: hey!
09:31 pranithk xavih: I just found invalid bound bug in ec_iov_copy_to() need confirmation if my code reading is correct or not...
09:31 pranithk xavih: sorry man! was deep in code :-)
09:32 xavih pranithk: that's what I was going to analyze... write calls and partial reads seem ok
09:33 pranithk xavih: It shouldn't cause much problems I hope
09:33 xavih pranithk: there's a single write where the contents seem to have been corrupted, so it must be a copy error, a memory corruption or an encoding error...
09:34 pranithk xavih: Anyways, this in ec_iov_copy_to is doing offset -= vector[i].iov_len. 'i' could be greater then count..
09:34 xavih pranithk: what's the problem you have seen ?
09:35 xavih pranithk: no, it's not possible, 'i' will always be < count
09:35 xavih pranithk: this is only executed if offset < vector[i].iov_len
09:36 xavih pranithk: sorry if that condition is not satisfied
09:36 xavih pranithk: I mean it's executed if offset >= vector[i].iov_len
09:37 pranithk xavih: sorry, wrong code reading :-)
09:37 xavih pranithk: and at this point i is < count
09:37 pranithk xavih: break is for outer while :-(
09:37 pranithk xavih: so we are good
09:41 xavih pranithk: do you have the latest corrupted file sent by Bhaskarakiran ?
09:42 pranithk xavih: I think I have the mail...
09:42 xavih pranithk: there's a block of 0x1800 bytes written at 0x18000 in a single write operation, but even it's contents are corrupt
09:42 pranithk xavih: I have the mail with 4 attachements...
09:43 xavih pranithk: and all corruptions are very similar, but at (apparently) random offsets
09:43 anmolb joined #gluster-dev
09:43 pranithk xavih: Hmm...
09:44 pranithk xavih: so we need to find how the buffers are corrupted...
09:44 xavih pranithk: yes :(
09:44 pranithk xavih: I am going through writev in detail. Will ask you doubts if I don't understand something...
09:44 xavih pranithk: in theory the contents of this buffer are only touched by ec_iov_copy_to and the encoding function
09:45 xavih pranithk: ok
09:45 pranithk xavih: Good I am at the correct point then :-)
09:46 xavih pranithk: another interesting thing: some corruptions seems to happen at multiples of 256 bytes
09:48 pranithk xavih: hmm... interesting...
09:54 aravindavk joined #gluster-dev
10:09 nbalacha joined #gluster-dev
10:09 gem joined #gluster-dev
10:21 gem_ joined #gluster-dev
10:23 shubhendu joined #gluster-dev
10:23 gem_ joined #gluster-dev
10:24 shubhendu joined #gluster-dev
10:24 xavih pranithk: nfs uses anonymous fd, right ?
10:27 jiffin1 joined #gluster-dev
10:27 rjoseph joined #gluster-dev
10:32 Manikandan joined #gluster-dev
10:33 pppp joined #gluster-dev
10:35 rajeshj joined #gluster-dev
10:41 kotreshhr joined #gluster-dev
10:42 aravindavk joined #gluster-dev
10:45 atalur joined #gluster-dev
10:48 pranithk xavih: yes
10:48 xavih pranithk: I'm seing fd with ctx->flags = O_APPEND in the logs
10:49 xavih pranithk: this means that open() has been called
10:49 pranithk xavih: I think yesterday's test Bhaskarakiran did was from fuse mount?
10:49 pranithk Bhaskarakiran: ^^
10:49 Bhaskarakiran pranithk, yes fuse mount
10:50 xavih Bhaskarakiran: the logs you have sent are from a single run ?
10:50 Bhaskarakiran xavih, yes
10:51 xavih pranithk: then I think that there are some writes in parallel...
10:51 xavih pranithk: could we repeat the tests completely disabling self-heal (daemon and client side) ?
10:51 xavih Bhaskarakiran: ^
10:53 atinmu joined #gluster-dev
10:58 atinmu nbalacha, hi, I was testing your patch 11521
10:58 kotreshhr joined #gluster-dev
10:59 atinmu nbalacha, I don't see gf_msg_callingfn logging anything there
10:59 atinmu nbalacha, did you test it?
10:59 jiffin joined #gluster-dev
10:59 atinmu nbalacha, code looks ok to me though
11:00 xavih Bhaskarakiran, pranithk: I'm seeing a lot of operations in the log that do not match what the test does in theory...
11:00 xavih Bhaskarakiran, pranithk: There are appends starting at offset zero with small sizes (45 bytes each)
11:01 rajeshj joined #gluster-dev
11:03 xavih Bhaskarakiran, pranith: After a while there are mixed appends of size 34 and size 1666
11:03 xavih Bhaskarakiran, pranith: does the test do something like that ?
11:03 Bhaskarakiran xavih, this is pranith, write-behind can merge some of the writes?
11:04 Bhaskarakiran xavih, but write should never be 34...
11:04 xavih Bhaskarakiran: yes, it could, but it doesn't explain what I'm seeing
11:04 Bhaskarakiran xavih, I think the minimum size of write should be 45
11:05 xavih Bhaskarakiran: I see appends of 45 bytes until the file size is 9245
11:05 Bhaskarakiran xavih, You saw a write with size 34?
11:05 Bhaskarakiran xavih, The log which prints this information is before write starts is it?
11:05 xavih Bhaskarakiran: after that there is another sequence starting at offset 0 of 1666 followed by 34
11:05 gem joined #gluster-dev
11:06 xavih Bhaskarakiran: oh, that should be the '-------...' that separates the contents of each cat
11:06 glusterbot xavih: '-----'s karma is now -1
11:07 atinmu nbalacha, I was wrong, it works, thanks
11:09 atalur joined #gluster-dev
11:14 xavih Bhaskarakiran: I see 9 writes of 45 bytes, 91 writes of 46 bytes and 100 writes of 47 bytes starting at offset 0 of one file (this seems to be the md5sums redirected to an empty file)
11:14 xavih Bhaskarakiran: then I see 250 pairs of writes of 1666 and 34 bytes starting also at offset 0 (this seems to be the cat + echo)
11:15 xavih Bhaskarakiran: then I see the same sequence of 45, 46 and 47 bytes 3 times starting at the end of the previous writes
11:15 gem joined #gluster-dev
11:16 xavih Bhaskarakiran: the it starts again at offset 0, but this time the pairs of writes of 1666 and 34 is repeated 500 times
11:18 xavih Bhaskarakiran: Then 45, 46 and 47 bytes (this time with 400 of 47 bytes)
11:18 xavih Bhaskarakiran: is that what the test does ?
11:21 pranithk xavih: the test runs <md5sum> <space> <testfile.$i>
11:22 pranithk xavih: $i ranges from 1 to 100
11:22 pranithk xavih: if $i is single digit then len is 45, 2 digits->46, 3 digits 47
11:22 dlambrig joined #gluster-dev
11:22 pranithk xavih: test adds*
11:22 xavih pranithk: that's not what I'm seeing in the logs...
11:23 xavih pranithk: yes, that's ok, but I see this sequence of 45, 46 and 47 starting at offset 0
11:24 xavih pranithk: then I see 250 cat's + echo's and the sequence of 45, 46, 47 (200 times, not 100)
11:24 nbalacha atinmu, sorry was away
11:24 nbalacha atinmu, yes I tested it
11:24 atinmu nbalacha, no problem
11:24 xavih pranithk: then it seems to start again with 500 cat's + echo's and the sequence of 45, 46 and 47 (but this time 500 times)
11:24 atinmu nbalacha, i tested it now and it works
11:24 atinmu nbalacha, I've acked it
11:24 xavih pranithk: if that's not what the test is doing, we have a problem...
11:25 nbalacha atinmu, ah ok, I just saw your last msg
11:25 xavih pranithk: what I cannot say is if all writes are to the same file or not, or if the file is deleted or truncated between iterations
11:26 aravindavk joined #gluster-dev
11:30 pranithk Bhaskarakiran: ^^
11:31 pranithk xavih: Bhaskarakiran is not in his cube
11:31 pranithk xavih: let us wait for him...
11:31 xavih pranithk: I think I've found something...
11:31 xavih pranithk: there's a write that I think should start after the previous one, but it starts at offset 47 (which is the size of the previous write)
11:35 xavih pranithk: at time 16:22:21.069842 there's the last append from one run. Then there are 6 minutes where the file is not touched (I assume it's creating other files)
11:36 xavih pranithk: at 16:28:23.029259 there is another append. This starts at offset 47. It should be 0 if it's the first write to a file or at the offset of the last write (1076224) if it continues appending
11:40 xavih pranithk: sorry, the offset is 874892
11:42 pranithk xavih: interesting...
11:42 pranithk xavih: I am sorry I am not following this as fast as you... I am still learning things about how it works completely...
11:43 pranithk xavih: Give me an hour or two more and I should be up to date...
11:43 pranithk xavih: I am in readv_rebuild at the moment...
11:43 xavih pranithk: don't worry :)
11:44 Manikandan joined #gluster-dev
11:44 pranithk xavih: in readv_rebuild, what does variable max represent?
11:45 pranithk xavih: I think it is used to comeup with size read...
11:45 xavih pranithk: it's the size that maximum size that have been read
11:45 pranithk xavih: yeah...
11:46 xavih pranithk: first it's initialized to the theoretical size that should have been read (a multiple of the chunk size)
11:46 xavih pranithk: then it's updated with the size of the file (if it's smaller)
11:47 pranithk xavih: yeah...
11:50 pranithk xavih: in ec_writev_merge_head
11:50 pranithk xavih: when can op_ret be less than size? if it is lesser aren't we in trouble?
11:53 pranithk xavih: ah! it is for writes with holes...
11:53 xavih pranithk: we are working with chunks of fixed size and bricks store full fragments, however when a readv is made on the last chunk, readv can return less than the chunk size if the file is smaller
11:54 xavih pranithk: even in this case we need to fill the remaining of the chunk with 0's
11:54 pranithk xavih: I understand that for seek + write we are filling head with zeros. But I didn't understand your previous statement... wait, thinking
11:54 xavih pranithk: I suspect that the problem could be in truncate
11:55 pranithk xavih: Whenever I saw bricks, they always had file size multiples of ec->fragment_size
11:55 ppai joined #gluster-dev
11:55 pranithk xavih: for appending writes, truncate should never come?
11:55 pranithk xavih: You mean write which does truncates?
11:55 xavih pranithk: yes, but readv takes real size of the file into account
11:55 pppp joined #gluster-dev
11:55 xavih pranithk: I think the problem happens when the file is emptied between tests
11:56 pranithk xavih: I don't think he was writing to same file...
11:56 xavih pranithk: It seems there are multiple tests being executed
11:56 pranithk xavih: but let's ask him
11:56 pranithk xavih: oh yeah!, he always does multiple tests :-)
11:56 xavih pranithk: that would be interesting to know
11:57 spalai1 joined #gluster-dev
11:57 pranithk xavih: there is a bug in ec_writev_merge_head
11:57 pranithk xavih: what if size is less than base>
11:57 pranithk xavih: oh no
11:57 pranithk xavih: sorry!
11:58 pranithk xavih: base is already min of size and op_ret, we are safe there as well
11:58 vmallika joined #gluster-dev
12:01 xavih pranithk: I think I've found a bug but I'm not sure if it's the source of the bug Bhaskarakiran found
12:02 pranithk xavih: oh, what is it?
12:02 xavih pranithk: if EC_STATE_PREPARE_ANSWER of truncate, the size of the file is updated using ec_set_inode_size()
12:02 xavih pranithk: this is not correct because ec_set_inode_size only updates the post_size
12:03 xavih pranithk: suppose we have an inode with pre_size = 100 and post_size = 100
12:03 xavih pranithk: now we do a write of 47 bytes: post_size = 147
12:03 xavih pranithk: now we truncate to 0: post_size = 0
12:04 xavih pranithk: when trusted.ec.size is updated, an xattrop with post_size - pre_size will be sent. In this case 0 - 100 = -100
12:05 pranithk xavih: Hmm...
12:05 xavih pranithk: well, that's not a bug if the xattr on the file was 100... which should...
12:06 pranithk xavih: yeah
12:07 xavih pranithk: maybe a flush coming in the middle... I'll continue searching...
12:07 rajeshj joined #gluster-dev
12:07 pranithk xavih: sure sir!
12:10 jiffin1 joined #gluster-dev
12:15 Susant left #gluster-dev
12:16 atinmu joined #gluster-dev
12:17 xavih pranithk, Bhaskarakiran: it would be great to have the full script of the test. Maybe the problem is in an apparently unrelated thing the script is doing instead of the append itself
12:22 kotreshhr joined #gluster-dev
12:22 shubhendu joined #gluster-dev
12:24 itisravi joined #gluster-dev
12:26 kotreshhr left #gluster-dev
12:40 xavih pranithk: I think I've found a possible cause of hung
12:47 pranithk xavih: oh!
12:47 pranithk xavih: what is that?
12:48 xavih pranithk: the problem is that it should happen more often. Suppose the last fop sent to a locked inode is flush
12:48 pranithk xavih: okay...
12:48 xavih pranithk: in ec_unlock() it will be queued for a delayed unlock
12:49 pranithk xavih: yes...
12:49 xavih pranithk: ec_unlock_timer_del() will be called, which will call ec_unlock_now()
12:50 xavih pranithk: here ec_update_info() will be called
12:50 xavih pranithk: oh, I think I've just seen why it works, because flush will always return false for ec_update_info()...
12:51 pranithk xavih: yes :-)
12:52 xavih pranithk: then there's another error, but less common
12:52 pranithk xavih: oh!
12:52 xavih pranithk: in ec_update_size_version(), if something fails, ec_unlock_lock() should be called if fop is not flush, fsync or fsyncdir
12:53 xavih pranithk: otherwise the lock won't be released
12:54 xavih pranithk: still nothing about the other bug, though :(
12:55 pranithk xavih: Yes... at the moment this is not hit that frequently... do you mind adding this to gluster-ec page we have so that we don't forget?
12:55 xavih pranithk: sure
13:04 xavih pranithk: I must go. I'll continue later
13:04 xavih pranithk: bye
13:05 pranithk xavih: yes sir, cya
14:13 pranithk xavih: Are you there?
14:32 pranithk left #gluster-dev
14:49 kdhananjay joined #gluster-dev
14:56 soumya joined #gluster-dev
14:59 shubhendu joined #gluster-dev
16:08 gem joined #gluster-dev
16:17 mribeirodantas joined #gluster-dev
16:46 soumya joined #gluster-dev
16:57 G_Garg joined #gluster-dev
17:01 pranithk joined #gluster-dev
17:01 pranithk xavih: Are you there?
17:15 pppp joined #gluster-dev
17:18 pranithk xavih: sent you the mail with updates...
18:21 dlambrig_ joined #gluster-dev
18:46 dlambrig_ joined #gluster-dev
20:37 _nixpanic joined #gluster-dev
20:37 _nixpanic joined #gluster-dev
21:34 hagarth joined #gluster-dev
23:33 tdasilva joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary