Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2015-06-30

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:33 jrm16020 joined #gluster-dev
00:35 vmallika joined #gluster-dev
00:58 suliba joined #gluster-dev
01:06 mribeirodantas joined #gluster-dev
03:14 overclk joined #gluster-dev
03:22 kshlm joined #gluster-dev
03:26 kdhananjay joined #gluster-dev
03:27 kdhananjay joined #gluster-dev
03:33 shubhendu joined #gluster-dev
03:42 overclk joined #gluster-dev
04:01 itisravi joined #gluster-dev
04:05 krishnan_p joined #gluster-dev
04:17 atinm joined #gluster-dev
04:25 hchiramm_home joined #gluster-dev
04:32 kshlm joined #gluster-dev
04:44 jiffin joined #gluster-dev
04:51 sakshi joined #gluster-dev
04:53 raghu joined #gluster-dev
04:54 gem joined #gluster-dev
04:56 anekkunt joined #gluster-dev
05:00 atinm anekkunt, ping
05:00 glusterbot atinm: Please don't naked ping. http://blogs.gnome.org/mark​mc/2014/02/20/naked-pings/
05:01 anekkunt atinm, pong
05:05 atinm anekkunt, this is regarding http://review.gluster.org/#/c/11120
05:05 atinm anekkunt, did you rebase your changes on top of Anuradha's replace brick changes?
05:05 atinm anekkunt, I believe replace brick is now sync-op
05:07 anekkunt atinm, No ..
05:08 anekkunt atinm, but I dint get any merge conflict
05:08 vimal joined #gluster-dev
05:09 atinm anekkunt, revisit your changes
05:09 atinm anekkunt, its not about merge conflicts
05:09 atinm anekkunt, what I am saying is rb_use_rsp_dict might need some change
05:10 vikumar joined #gluster-dev
05:10 atinm anekkunt, please confirm and get back
05:10 anekkunt atinm, ok ... I  will  do rebase
05:10 atinm anekkunt, its not only auto rebase
05:10 atinm anekkunt, you need to check the flow as well
05:10 anekkunt atinm,  ya ... i got it
05:15 spandit joined #gluster-dev
05:17 vmallika joined #gluster-dev
05:19 ndarshan joined #gluster-dev
05:21 ashish joined #gluster-dev
05:21 hagarth joined #gluster-dev
05:21 anekkunt atinm,  replace brick in op_sm is dead code now  in my patch
05:22 anekkunt atinm, Thanks for pointing , I will resend the patch  with changes
05:22 anekkunt atinm++
05:22 glusterbot anekkunt: atinm's karma is now 13
05:22 hgowtham joined #gluster-dev
05:23 raghu joined #gluster-dev
05:23 ashiq joined #gluster-dev
05:24 Manikandan joined #gluster-dev
05:30 rafi joined #gluster-dev
05:31 Bhaskarakiran joined #gluster-dev
05:34 ashiq joined #gluster-dev
05:51 deepakcs joined #gluster-dev
05:52 pppp joined #gluster-dev
05:57 soumya joined #gluster-dev
06:02 Manikandan joined #gluster-dev
06:04 atalur joined #gluster-dev
06:05 hagarth msvbhat: ping, around?
06:07 Gaurav__ joined #gluster-dev
06:12 nbalacha joined #gluster-dev
06:23 rgustafs joined #gluster-dev
06:26 krishnan_p joined #gluster-dev
06:35 spalai joined #gluster-dev
06:35 pranithk joined #gluster-dev
06:35 msvbhat hagarth: pong
06:37 hagarth msvbhat: are you familiar with how failback works with geo-replication?
06:39 msvbhat hagarth: We will have to manually establish the geo-rep session from slave to master
06:40 nkhare joined #gluster-dev
06:40 msvbhat hagarth: I think I know... I might be missing any newest developement
06:41 hagarth msvbhat: do we just provide a single node on the master for establishing the failback?
06:41 hagarth msvbhat: if the single node were to fail, does the failover/failback bail out?
06:42 msvbhat hagarth: Single node? No this is complete cluster failover
06:42 msvbhat I mean when entire master cluster is down
06:43 msvbhat hagarth: When a single node is down, afr/selfheal should take care of it right?
06:43 hagarth msvbhat: I refer to the node specified in the URI for performing a failover/failback
06:44 Bhaskarakiran joined #gluster-dev
06:45 msvbhat hagarth: AFAIK, Once the session has been established, the node specified in URI doesn't have any special significance.
06:46 hagarth msvbhat: ok, I will check that out with a quick test
06:46 aravindavk joined #gluster-dev
06:46 msvbhat hagarth: Okay, But the command should still have the same URI
06:46 hagarth msvbhat: ok
06:48 soumya joined #gluster-dev
07:00 josferna joined #gluster-dev
07:00 lkoranda_ joined #gluster-dev
07:01 raghu joined #gluster-dev
07:05 pranithk xavih: Appending writes corrupting the file is still happening as per Bhaskarakiran
07:07 xavih pranithk: really ? :(
07:08 xavih pranithk: can you send me the script he is using to reproduce this problem ?
07:08 pranithk Bhaskarakiran: ^^
07:10 Bhaskarakiran xavih, on nfs  mount its easily reproducible.. just doing appends to the file is corrupting it.. there's no script as such
07:10 lkoranda joined #gluster-dev
07:11 xavih Bhaskarakiran: simply doing "echo test >>file" ?
07:12 xavih Bhaskarakiran: many echo's in parallel or just one done multiple times ?
07:12 Bhaskarakiran xavih, yah.. first crerated 100 files with random data and then a file with 100 iterations of /etc/passwd content and did md5sum of 100 files and appended to the file
07:13 kotreshhr joined #gluster-dev
07:13 pranithk xavih: I think it is a for loop, for i in {1..100}; do md5sum testfile.$i >> md5sum; done
07:14 pranithk Bhaskarakiran: ^^ right?
07:14 Bhaskarakiran pranithk, yes
07:14 xavih pranithk, Bhaskarakiran: without doing anything else in parallel ?
07:15 pranithk xavih: The first file creation is something like for i in {1..100}; do dd if=/dev/urandom of=testfile.$i bs=64k count=$i & done; wait; done
07:16 pranithk Bhaskarakiran: I am not sure about anything else in parallel...?
07:16 Bhaskarakiran pranithk, no other io on parallel
07:16 Bhaskarakiran pranithk, after the dd create a file with for i in `seq 1 100`; do cat /etc/passwd >> file ; done
07:17 xavih pranithk, Bhaskarakiran: I'll try to identify the problem, but it seems very weird...
07:17 Bhaskarakiran pranithk, xavih and then for i in `seq 1 100`; do md5sum testfile.$i >> file
07:17 xavih pranithk, Bhaskarakiran: I'll tell you my findings
07:17 Bhaskarakiran xavih, okay
07:17 spalai left #gluster-dev
07:17 xavih Bhaskarakiran: btw, the volume was a simple 8+4 ?
07:18 Bhaskarakiran xavih, yes
07:18 xavih Bhaskarakiran: without distribute, right ?
07:18 pranithk xavih: I am going to send patches for ls -l reporting bad file size and possibly the OOM killers on mount because of too many heals at once today. Second one is a bit of a stretch...
07:18 xavih pranithk: ok :) I'll review them
07:19 Bhaskarakiran xavih, yes there's no distribute
07:21 pranithk Bhaskarakiran: Do you think it is possible to recreate this one without perf xlators? Just want to make sure it is ec issue...
07:22 Bhaskarakiran pranithk, let me know how i can try to recreate
07:22 pranithk Bhaskarakiran: gluster volume set <volname> performance.write-behind off; gluster volume set <volname> performance.quick-read off
07:22 pranithk Bhaskarakiran: same steps you did
07:22 Bhaskarakiran pranithk, let me check with the options set
07:22 pranithk Bhaskarakiran: actually lets disable all perf xlators
07:22 pranithk Bhaskarakiran: wait
07:23 pranithk Bhaskarakiran: https://github.com/pranithk/dotfiles/b​lob/master/scripts/disable-perf-xl.sh
07:23 pranithk Bhaskarakiran: use that one
07:24 Bhaskarakiran pranithk, okay
07:26 nishanth joined #gluster-dev
07:48 xavih Bhaskarakiran: can you send me some of the corrupted files ?
07:54 pranithk xavih: This is lunch time... He will come back after a while...
07:55 xavih pranithk: ok, no problem :)
07:56 xavih pranithk: btw, do you know if the GF_FOP_IPC patch needs to be merged ? there has been some discussion for doing this in a more generic way, but I'm not sure if this will be done later and we should merge the patch now
07:56 pranithk xavih: I have access to output of less... will that be good?
07:57 pranithk xavih: I thought she is going to send a different way by changing it in central place which will be used by all cluster xlators
07:57 pranithk xavih: no no, soumya is going to send it in generic way. We spoke about it yesterday
07:57 xavih pranithk: ok
07:58 soumya joined #gluster-dev
07:58 xavih pranithk: output of less won't correctly show offsets of the changes. I need to know where the corruption has happened
07:59 xavih pranithk: I'll wait until Bhaskarakiran returns, no problem :)
08:02 lkoranda_ joined #gluster-dev
08:10 hchiramm raghu, 3.6.4beta2 packages are avilable @ http://download.gluster.org/pub/glust​er/glusterfs/qa-releases/3.6.4beta2/
08:12 lkoranda joined #gluster-dev
08:25 ndevos pppp: could you chime in with your use-case on "gluster volume restart"? http://review.gluster.org/11455
08:32 Bhaskarakiran xavih, one min i am checking...
08:34 kdhananjay joined #gluster-dev
08:35 rjoseph joined #gluster-dev
08:35 pranithk xavih: For ec_inode_good... we can believe its output only when ctx->have_info is set to true right?
08:37 anekkunt atinm,  I have resent the patch ,  please can you review  this patch  http://review.gluster.org/#/c/11120/
08:40 xavih pranithk: in theory data returned from ec_inode_good should always be valid. ctx contents should not be updated with non-trusted data
08:42 pranithk xavih: for an inode that is created newly, if first call is ec_inode_good, then the inode-ctx is created just then which will give all-zero structure with bad as zero which means everything will be good.
08:42 pranithk xavih: I think that function should be changed to do inode_ctx_get instead...
08:44 xavih pranithk: I don't see exactly what do you mean...
08:45 josferna joined #gluster-dev
08:45 pranithk xavih: ec_inode_get will allocate new ctx structure right?
08:45 pranithk xavih: which is filled with all zeros?
08:46 xavih pranithk: yes
08:46 xavih pranithk: if we don't have info about the inode, we should consider all bricks as valid
08:46 pranithk xavih: why?
08:47 pranithk xavih: if we don't know then we should act like we don't know, right? I mean it has 3 states, good, bad, i-don't-know
08:47 xavih pranithk: ec_inode_good is used to filter out bricks known to be in an invalid state
08:47 xavih pranithk: if we don't have any information about an inode, why we do need to filter out some bricks ?
08:47 pranithk xavih: ah! okay.
08:48 xavih pranithk: we cannot have three states when selecting bricks to be queried
08:48 pranithk xavih: I wanted to re-use that function in figuring out if the brick we got response is a good brick for that inode...
08:49 xavih pranithk: the logic used is right the contrary. We first check which bricks are considered ok and only send the request to them.
08:49 xavih pranithk: only a failure in the response could add a new brick to the bad list
08:50 pranithk xavih: Hmm... how should we find if the brick we queried has correct info for that inode? in readdirp?
08:51 pranithk xavih: I am thinking this way
08:51 pranithk xavih: we do inode_ctx_get(), if it gives a valid ctx and if the idx is not in ctx->bad then go ahead...
08:52 xavih pranithk: in readdirp, when offset is 0, we send a lock and a lookup on the directory. This will update the bad mask if necessary
08:52 xavih pranithk: after that, each following readdirp will only send the request to the selected brick
08:53 pranithk xavih: Not about the directory which we are doing readdirp. I am talking about the info of the entries received in readdirp
08:53 pranithk xavih: If we have file 'a' in directory '/'
08:53 xavih pranithk: oh, I see
08:53 pranithk xavih: how do we know that brick-0 has correct info of 'a' :-)
08:55 Bhaskarakiran xavih, sorry the file is missing ...
08:56 xavih pranithk: I understand now... if we don't have ctx info for that inode or it's considered bad, we should clear the entry->inode, right ?
08:56 pranithk xavih: yes sir
08:56 xavih Bhaskarakiran: if you are able to reproduce the problem, could you send me the file when if happens again ?
08:56 Bhaskarakiran xavih, ya sure
08:57 xavih Bhaskarakiran: thanks :)
08:57 spalai joined #gluster-dev
08:58 xavih pranithk: I think you are right. We should call inode_ctx_get() and, if present, check the bad mask
08:59 pranithk xavih: I am not going to re-use ec_inode_good
08:59 xavih pranithk: yes, we cannot use it as it's written now.
08:59 pranithk xavih: It's purpose seems to be different...
09:01 xavih pranithk: yes. One is used to select bricks to send a request, and the new one is used to check if an inode is valid
09:01 xavih pranithk: they are very similar, but need different checks
09:03 Bhaskarakiran xavih, got the file... to which email i should send it ??
09:05 kaushal_ joined #gluster-dev
09:05 pranithk xavih: But we still can't believe what is present in ctx->bad alone, because of the same reasons, may be it is yet to be updated with good/bad, but the ctx structure could have just been allocated...
09:06 ndevos hey atinm: you'll be hosting the bug triage meeting, right? I will only be able to partially attend
09:07 xavih pranithk: yes, you are right. I think using have_info is not enough safe
09:08 xavih pranithk: maybe we should add a new have_bad or something like that
09:09 xavih pranithk: wait. If the ctx exists, it means that someone has already accessed
09:10 pranithk xavih: which can happen for lookup also...
09:10 xavih pranithk: we could trust it. There will always be a window where data could be invalid if an operation that is taking place on that inode detects an error but has not updated the ctx yet
09:10 pranithk xavih: it is created on ec_child_select
09:11 pranithk xavih: yeah...
09:11 pranithk xavih: Shall we live that race?
09:12 pranithk xavih: I think it is a small change to fix that race...
09:12 pranithk xavih: we just set have_bad to true in ec_update_bad()
09:13 xavih pranithk: we could do that
09:13 pranithk xavih: A brick can go bad because of other mounts, but this mount can still think one of the bricks is not bad.
09:14 pranithk xavih: This is the final kind of race... I hope :-)
09:14 xavih pranithk: I think there are two kinds of discrepancies: one is good and the other is bad
09:15 xavih pranithk: the good is when another mount or even another process on the same mount modifies a file
09:15 pranithk xavih: ...
09:15 xavih pranithk: in this case it's irrelevant if we return the new or the old information
09:16 pranithk xavih: sorry, new or old info of the inode-ctx?
09:16 xavih pranithk: never mind, I was thinking in another thing... :-/
09:17 pranithk xavih: Hmm... The most conservative way to fix it is to consider ctx->bad when ctx->have_info is set. But that is too conservative.
09:17 pranithk xavih: Correct way to fix it is to have generation number as well...
09:17 xavih pranithk: I have to go. I'll return in 2 or 3 hours...
09:17 pranithk xavih: sure
09:18 xavih pranithk: I'll think about this, but I think this kind of problems do not have a way to fix. There will always be a window where the inode could get corrupted and we won't know
09:19 pranithk xavih: okay for now I will code what we decided about 'have_bad'
09:19 xavih pranithk: specially in readdirp
09:19 xavih pranithk: ok
09:20 atinm joined #gluster-dev
09:23 Gaurav__ joined #gluster-dev
09:52 anrao joined #gluster-dev
10:09 soumya pranithk|afk, ping
10:09 glusterbot soumya: Please don't naked ping. http://blogs.gnome.org/mark​mc/2014/02/20/naked-pings/
10:24 pranithk soumya: yes?
10:34 ira joined #gluster-dev
10:35 Gaurav__ joined #gluster-dev
10:46 atinm joined #gluster-dev
10:59 rgustafs joined #gluster-dev
11:07 kkeithley1 joined #gluster-dev
11:09 dlambrig_ joined #gluster-dev
11:23 rjoseph|afk joined #gluster-dev
11:30 rafi1 joined #gluster-dev
11:32 ndevos atinm, rafi1: bug triage meeting? I'm rather occupied with things and will not be able to host it
11:33 rafi joined #gluster-dev
11:36 atinm ndevos, I will not be attending today, rafi has agreed to host it
11:36 ndevos atinm: okay, thanks!
11:36 ndevos rafi++ thank you too :)
11:36 glusterbot ndevos: rafi's karma is now 15
11:36 rafi atinm: I already sent Reminder :)
11:37 atinm rafi, I know :)
11:37 gem_ joined #gluster-dev
11:40 anekkunt joined #gluster-dev
11:44 soumya_ joined #gluster-dev
11:58 gem_ joined #gluster-dev
11:58 rafi REMINDER: Gluster Community Bug Triage meeting starting in another 1 minutes in #gluster-meeting
11:59 rafi joined #gluster-dev
12:05 firemanxbr joined #gluster-dev
12:09 kotreshhr left #gluster-dev
12:12 pranithk xavih: The solution we discussed is not working as expected because bad is updated even by inodelk, so if inodelk succeeds on all the bricks then bad becomes zero
12:17 anekkunt joined #gluster-dev
12:23 pranithk xavih: I am thinking of disabling readdirp :-/
12:23 pranithk xavih: ping me once you are back
12:26 itisravi joined #gluster-dev
12:43 anekkunt joined #gluster-dev
12:47 mribeirodantas joined #gluster-dev
12:49 rafi ndevos++ kkeithley_++ saravana++
12:49 glusterbot rafi: ndevos's karma is now 170
12:49 glusterbot rafi: kkeithley_'s karma is now 3
12:49 glusterbot rafi: saravana's karma is now 1
12:52 xavih pranithk: ctx->bad was thought to avoid failed bricks on new fops, but not to exclude answers. Probably it doesn't fit very well in this case.
12:53 ndevos rafi++ thanks!
12:53 glusterbot ndevos: rafi's karma is now 16
12:53 xavih pranithk: anyway, clearing the inode if ctx is not defined seems quite good. Of course there could be cases in which this could fail, but it's also currently failing for afr. I think it's better this than nothing
13:04 jrm16020 joined #gluster-dev
13:06 pranithk xavih: Okay here is the test case.
13:06 pranithk xavih: While a 2GB file is being healed we do ls -l on the mount point. It sometimes gives 0 size
13:06 pranithk xavih: I don't see any easy way to fix it :-(
13:07 pranithk xavih: I though ctx->bad will aid in fixing this but it is updated at so many places...
13:07 pranithk xavih: thought*
13:08 xavih pranithk: a file being healed should have the bits corresponding to the damaged bricks set in ctx->bad
13:08 anrao joined #gluster-dev
13:08 xavih pranithk: it's only cleared after having healed the file
13:10 pranithk xavih: but ctx->bad is changed for inodelk as well...
13:10 pranithk xavih: so if inodelk succeeds on all bricks then ctx->bad will be zero
13:11 xavih pranithk: all fops only update ctx->bad to set bits, never for clearing them
13:11 xavih pranithk: no, only bricks that fail are updated
13:11 xavih pranithk: if no brick fails, ctx->bad is not updated
13:12 pranithk xavih: Okay, when writev is going on it does the following ops: inodelk, xattrop, fop, xattrop, inodelk
13:12 pranithk xavih: ah!
13:13 pranithk xavih: But I saw ctx->bad to be zero...
13:13 pranithk xavih: wait
13:16 pranithk xavih: I see that it is always updating bad...
13:17 xavih pranithk: inodelk overwrite ctx->bad even if no brick has failed ?
13:17 xavih pranithk: that shouldn't happen
13:17 pranithk xavih: wait wait, I see that bad is touched only when fop->parent is NULL
13:17 pranithk xavih: I need to debug this more
13:17 dlambrig_ left #gluster-dev
13:22 pppp joined #gluster-dev
13:26 soumya_ joined #gluster-dev
14:07 shyam joined #gluster-dev
14:17 aravindavk joined #gluster-dev
14:21 pousley joined #gluster-dev
14:34 spalai joined #gluster-dev
14:34 dlambrig1 joined #gluster-dev
14:44 bfoster joined #gluster-dev
14:51 JustinCl1ft left #gluster-dev
15:13 soumya_ joined #gluster-dev
15:24 kanagaraj joined #gluster-dev
15:30 pranithk joined #gluster-dev
15:35 spalai left #gluster-dev
15:44 pranithk xavih: I think I am done with heal throttling... will send the patch in a while...
15:44 pranithk xavih: testing it
16:01 anrao joined #gluster-dev
16:26 gem_ joined #gluster-dev
16:30 vmallika joined #gluster-dev
16:31 hchiramm_home joined #gluster-dev
16:34 josferna joined #gluster-dev
17:27 hagarth joined #gluster-dev
17:28 Gaurav__ joined #gluster-dev
17:29 hagarth ndevos: this might be of interest to you - http://www.ris.be/ris-belgiu​m-bouwt-cloudstorage-nodes/
17:31 * ndevos reads
17:31 ndevos hagarth: cool, maybe I should visit them>
17:31 ndevos ?
17:32 hagarth ndevos: I think so! can you reach out to them?
17:32 ndevos hagarth: sure, should be easy enough
17:34 ndevos hagarth: are you still in the US?
17:34 hagarth ndevos: yes, will be back in BLR on 6th
17:34 ndevos hagarth: ah, ok, I hope you're enjoying your time there :)
17:35 ndevos hagarth: also, what about the meeting tomorrow? any plans to host it, or should we find someone else again?
17:35 hagarth ndevos: thanks, quite a bit of work and yes it is good fun :)
17:35 hagarth ndevos: somebody else please, don't think I can do a 5:00 am meeting ;)
17:36 ndevos hagarth: oh, not in westford?
17:36 hagarth ndevos: no, am in CA right now
17:37 ndevos hagarth: right, must be better than MA ;-)
17:38 hagarth ndevos: MA had a milder summer ;)
17:40 ndevos hagarth: oh, yes, probably - it's getting 30+ degrees Celcius here the next few days, thats quite hot too
17:52 jobewan joined #gluster-dev
18:10 firemanxbr joined #gluster-dev
18:15 hagarth left #gluster-dev
19:20 shyam joined #gluster-dev
20:04 dlambrig_ joined #gluster-dev
20:28 RedW joined #gluster-dev
21:33 jrm16020 joined #gluster-dev
21:53 badone joined #gluster-dev
22:23 eljrax joined #gluster-dev
22:49 hagarth joined #gluster-dev
23:01 mribeirodantas joined #gluster-dev
23:31 badone_ joined #gluster-dev
23:50 badone__ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary