Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2013-12-30

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:40 davidbie_ joined #gluster-dev
03:10 kshlm joined #gluster-dev
03:45 itisravi joined #gluster-dev
03:48 kanagaraj joined #gluster-dev
03:48 shyam joined #gluster-dev
04:08 ndarshan joined #gluster-dev
04:25 bala joined #gluster-dev
04:40 ppai joined #gluster-dev
04:48 ajha joined #gluster-dev
04:52 ababu joined #gluster-dev
05:32 hagarth joined #gluster-dev
06:22 lalatenduM joined #gluster-dev
07:02 mohankumar__ joined #gluster-dev
07:59 shyam joined #gluster-dev
08:32 pk joined #gluster-dev
08:32 ndarshan joined #gluster-dev
09:07 xavih pk: hi. Sorry for not having answered before. I've had a few holidays.
09:20 pk xavih: realized :-)
09:21 pk xavih: You worked on something related to erasure coding didn't you?
09:24 xavih pk: yes, I'm still working on it
09:25 xavih pk: I'm implementing a disperse translator that uses erasure codes
09:25 pk xavih: Sorry for not paying attention earlier, could you explain the goal of that translator. What is it trying to achieve?
09:28 xavih pk: at a very high level it can be seen as something similar to a RAID-5 or RAID-6 volume for gluster
09:29 xavih pk: it tries to offer a configurable level of fault tolerance while reducing the wasted space
09:30 pk xavih: I am guessing that it is a cluster translator...
09:31 xavih pk: for example, in replicated volumes half of the space is 'wasted'. With disperse you can achive the same level of fault tolerance with 33% or less of waster space
09:31 xavih pk: yes, it is
09:31 pk xavih: cool
09:31 pk xavih: Is there any document or something?
09:31 pk xavih: which erasure coding library are you using to implement this?
09:33 xavih pk: I'm using Rabin's IDA (Information Dispersal Algorithm). It's quite simple but fast enough.
09:33 xavih pk: I'm not using any library. I implemented it using a custom optimization that we worked on
09:34 pk xavih: Lets say some one wants to contribute what is the procedure?
09:34 xavih pk: it can deliver more than 1GB/s of encoded data per core
09:35 xavih pk: I'm not sure. I've been working on this for quite a lot. It's on gluster forge, but it's not complete yet
09:36 pk xavih: If it is completed, there isn't much to contribute, is there? :-)
09:36 xavih pk: this is the first open source project I work on...
09:36 xavih pk: yeah, I know :p
09:37 xavih pk: there have been some problems I needed to solve to make it work at a reasonable speed
09:37 xavih pk: this has involved the creation of 2 additional translators and a library
09:38 xavih pk: the current version on gluster forge still works, but it is quite slow
09:38 pk xavih: Yeah, I am seeing them now
09:38 pk xavih: got it
09:38 xavih pk: the original version had 2 translators: ida and heal
09:38 xavih pk: they still work (though with some bugs), but are slow
09:39 xavih pk: now I've written the gfsys library and dfc translator to try to solve some of the problems I found
09:39 pk xavih: What is your plan on getting them merged upstream? In how many days?
09:40 xavih pk: these two modules seem to work reasonably well and now I'm about to finish the ida modifications to use the new functionalities
09:41 xavih pk: I'm not sure... I supose that when the modules are stable enough, Vijay or some other core developer will make a decision...
09:41 xavih pk: I'm not sure if it's good to have it merged while it still doesn't work...
09:42 xavih pk: also, I'm sure there will be some concerns about the gfsys library...
09:42 pk xavih: Hmm....
09:42 pk xavih: If I understand it properly, the initial version works, but slow (probably has some bugs).
09:43 pk xavih: I want to contribute to this project.
09:44 pk xavih: Mainly to understand about the technology, I am not bad in glusterfs side of things but absolute newbie about erasure-coding, so it will take a while for me to start sending patches.
09:45 pk xavih: I think as a first cut I will try to understand the design and send a design-review mail in gluster-devel. If you already have any such document, you will definitely be saving both of us some time
09:46 xavih pk: well the theory is quite simple. The implementation is also relatively easy, however self-healing process is not implemented very well and some reads need to be done locked
09:46 xavih pk: these are the main reasons to implement the modifications
09:46 pk xavih: yeah, I wonder why inode/entry locks are not used...
09:47 xavih pk: yes, they were used, but they were slow
09:47 pk xavih: Hmm...
09:48 xavih pk: I could have optimized it using some techiniques similar to eager locking, but I though that the implementation was very complex
09:48 pk xavih: hmm...
09:49 xavih pk: The problem with erasure codes is that generic writes needs a RMW cycle
09:50 xavih pk: this adds a lot of latency to many requests
09:50 pk xavih: Hmm....
09:50 pk xavih: It wouldn't be right for me to give any suggestions without any understanding of what the xlator is doing. Let me do one thing. By Jan15th, I am planning to do first round of code-walk through...
09:51 pk xavih: May be then I will be in a position to understand the problems ....
09:52 xavih pk: Any feedback will be welcome :)
09:52 pk xavih: I want this functionality to make it upstream as soon as possible.
09:52 pk xavih: When do you come online generally?
09:53 hagarth pk, xavih: should we host a gerrit project for disperse and let forge be a mirror of that project?
09:54 xavih pk: I'm on GMT+1. I generally start working at 8:00 till 17:30 (GMT)
09:55 pk xavih: I am from India, so GMT+5:30
09:55 xavih hagarth: wouldn't this slow incorporation of changes I'm currently working on ?
09:55 xavih hagarth: I'm really doing a lot of changes right now...
09:56 hagarth xavih: you will be the admin for that project
09:56 xavih hagarth: ida's implementation will be very different
09:56 hagarth xavih: having it on gerrit might help people like pk to understand the nature of changes better
09:56 hagarth xavih: ok..
09:57 xavih hagarth, pk: what I think would be good is to take a look at the gfsys library and dfc translators
09:57 pk xavih: Sure
09:58 xavih hagarth, pk: they are the core of the way I think ida would work better and some ideas could be adapted to other existing translators
09:58 hagarth xavih: ok
09:58 pk xavih: I have a meeting now. cya guys
09:58 xavih hagarth, pk: I'm sure the implementation of gfsys wouldn't like everyone because it's heavily based on macros
09:59 hagarth xavih: ok..
10:00 xavih hagarth, pk: the gfsys library offers some generic and (I hope) very efficient interface for making light weight asynchronous calls inside any translator
10:01 xavih hagarth, pk: it also provides some easy to use macros for error handling, log generation and even a simple RCU implementation similiar to linux kernel
10:03 xavih hagarth, pk: the new implementation of ida I have been working on is heavily based on this library and DFC, so it would be very important to decide if it is good enough or not
10:05 mohankumar__ joined #gluster-dev
10:33 hagarth xavih: will take a look and get back to you
11:10 bala joined #gluster-dev
11:48 shyam joined #gluster-dev
11:56 pk left #gluster-dev
12:05 tg2_ joined #gluster-dev
12:07 kanagaraj_ joined #gluster-dev
12:42 bala joined #gluster-dev
13:03 bala joined #gluster-dev
14:08 bala joined #gluster-dev
14:55 davidbierce joined #gluster-dev
16:20 xavih joined #gluster-dev
17:01 hagarth joined #gluster-dev
17:38 mohankumar__ joined #gluster-dev
18:05 jobewan joined #gluster-dev
18:15 lightman joined #gluster-dev
18:25 popohlon joined #gluster-dev
21:55 jobewan joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary