Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2015-11-20

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 EinstCrazy joined #gluster-dev
00:14 hgichon joined #gluster-dev
00:24 nishanth joined #gluster-dev
01:01 EinstCrazy joined #gluster-dev
01:05 zhangjn joined #gluster-dev
01:25 shyam joined #gluster-dev
01:41 dlambrig_ joined #gluster-dev
02:22 gem_ joined #gluster-dev
02:42 dlambrig_ joined #gluster-dev
02:50 kdhananjay joined #gluster-dev
02:59 dlambrig_ joined #gluster-dev
03:07 overclk joined #gluster-dev
03:23 hchiramm_home joined #gluster-dev
03:38 atinm joined #gluster-dev
03:38 sakshi joined #gluster-dev
04:01 nbalacha joined #gluster-dev
04:04 spalai joined #gluster-dev
04:04 kanagaraj joined #gluster-dev
04:06 gem joined #gluster-dev
04:16 shubhendu joined #gluster-dev
04:19 mohan__ joined #gluster-dev
04:24 spalai1 joined #gluster-dev
04:29 zhangjn joined #gluster-dev
04:46 aspandey joined #gluster-dev
04:51 overclk joined #gluster-dev
04:53 pppp joined #gluster-dev
04:56 jiffin joined #gluster-dev
05:03 ppai joined #gluster-dev
05:04 kotreshhr joined #gluster-dev
05:07 Manikandan joined #gluster-dev
05:07 spalai joined #gluster-dev
05:08 hgowtham joined #gluster-dev
05:09 skoduri joined #gluster-dev
05:11 zhangjn joined #gluster-dev
05:12 hgowtham_ joined #gluster-dev
05:18 zhangjn joined #gluster-dev
05:20 ashiq joined #gluster-dev
05:23 josferna joined #gluster-dev
05:25 kotreshhr joined #gluster-dev
05:27 vimal joined #gluster-dev
05:27 mjrosenb joined #gluster-dev
05:28 kotreshhr left #gluster-dev
05:40 rafi joined #gluster-dev
05:42 vmallika joined #gluster-dev
05:51 asengupt joined #gluster-dev
05:57 spalai left #gluster-dev
05:59 josferna joined #gluster-dev
06:09 mchangir|wfh joined #gluster-dev
06:09 kdhananjay anyone getting '500 Internal server error' when trying to open gerrit?
06:11 mchangir|wfh nbalacha, how is xlators/mgmt/glusterd/src/.libs/glusterd.so   supposed to be used?
06:11 nbalacha mchangir|wfh, one of the glusterd folks would know
06:12 Saravana_ joined #gluster-dev
06:12 mchangir|wfh nbalacha, and who would be those?
06:12 nbalacha mchangir|wfh, try atinm
06:13 mchangir|wfh ok, thanks
06:14 atinm mchangir|wfh, can you little bit more specific on the question?
06:14 kshlm joined #gluster-dev
06:14 anekkunt joined #gluster-dev
06:15 atinm mchangir|wfh, glusterd binary is a link to glusterfsd binary, so when we boot up glusterd glusterfsd.c:main() is invoked and then it loads glusterd.so
06:15 atinm mchangir|wfh, does this answer your question
06:16 mchangir|wfh atinm, thanks
06:16 atinm mchangir|wfh, yw!
06:17 Humble joined #gluster-dev
06:18 * kdhananjay notices kshlm is here.
06:18 kdhananjay (11:39:38  IST) kdhananjay: anyone getting '500 Internal server error' when trying to open gerrit?
06:18 kdhananjay kshlm: Hoping you could help.
06:19 kshlm kdhananjay, Sure. But with what?
06:19 kdhananjay kshlm: trying to open r.g.o. Keeps failing with '500 Internal server error'
06:20 kshlm kdhananjay, I'll check it out.
06:24 overclk joined #gluster-dev
06:25 kshlm r.g.o is out-of-space.
06:25 Gaurav__ joined #gluster-dev
06:25 kshlm I'll free up some space.
06:29 raghu joined #gluster-dev
06:34 spalai joined #gluster-dev
06:36 hchiramm joined #gluster-dev
06:39 atalur joined #gluster-dev
06:44 kshlm kdhananjay, r.g.o is backup. Please check now.
06:44 kdhananjay kshlm: Indeed. Thanks a lot. kshlm++
06:44 glusterbot kdhananjay: kshlm's karma is now 43
06:47 pranithk joined #gluster-dev
06:52 spalai joined #gluster-dev
07:03 overclk joined #gluster-dev
07:20 anekkunt joined #gluster-dev
07:27 overclk ggarg, do you plan to backport patch #10231 to 3.7?
07:27 ggarg overclk, yes, i am going to backport it
07:28 overclk ggarg: nice, thanks!
08:04 zoldar pranithk: Hi. The locking issue seems to be gone now.
08:08 pranithk zoldar: phew!
08:08 pranithk zoldar: ok going for lunch cya.
08:08 pranithk zoldar: Thanks for the help
08:18 zhangjn joined #gluster-dev
08:36 zhangjn joined #gluster-dev
09:10 aravindavk joined #gluster-dev
09:13 kshlm Has anyone hit issues compiling master on EL7? Build fails for me with https://gist.github.com/kshlm/e5d0a7279d98f81cf14d . (fails with a warning about FORTIFY_SOURCE).
09:14 kshlm I'm using the jenkins build script to build. https://github.com/gluster/glusterfs-pat​ch-acceptance-tests/blob/master/build.sh
09:14 kshlm The same script builds without issues on F23.
09:16 atinm kshlm, could it be related to inline usage??
09:17 kshlm atinm, Nope. That had different warnings/errors/
09:19 anoopcs kshlm, I have seen this warning in debug mode on f23.
09:20 anoopcs kshlm, But not treated as error though.
09:21 hchiramm joined #gluster-dev
09:21 Humble joined #gluster-dev
09:25 kshlm anoopcs, This is still a warning, but the build script I gave has `-Werror` which causes the warning to be treated as an error.
09:25 hchiramm_ joined #gluster-dev
09:25 anoopcs kshlm, Yes. you are right.
09:25 kshlm The thing is I didn't see this warning on F23.
09:25 anoopcs kshlm, Really?
09:25 anoopcs kshlm, You need to configure in debug mode. Then you will see
09:26 kshlm Yes. I have the same source code, and using the same build script with the same options on both places. If there was a warning on F23, I would have known.
09:26 kshlm The build script builds in debug mode.
09:27 rastar anoopcs: kshlm it is the opposite
09:27 anoopcs kshlm, I don't know. I could still see the warning on f23 in debug mode.
09:28 anoopcs rastar, meaning?
09:28 rastar FORTIFY source requires optimization level of at least 2
09:28 rastar -g -O2 or -g -O3
09:28 rastar when you use --enable-debug then we set -g -O0
09:28 anoopcs rastar, Yes and in debug mode its 0
09:28 Humble joined #gluster-dev
09:28 rastar then conflicts with FORTIFY_SOURCE
09:28 kshlm rastar, I know that.
09:29 anoopcs rastar, That's what I mentioned above. In debug mode I see the warning.
09:29 rastar so on EL7 the default C/CPP flags are enabling FORTIFY_SOURCE
09:29 kshlm The problem here is that the same code behaves differently about FORTIFY_SOURCE on F23 and EL7.
09:29 rastar yes, default flags might be different or the spec file/make file is different
09:30 kshlm I know that we don't define FORTIFY_SOURCE by default in our build.
09:30 rastar are you using same Makefile?
09:30 kshlm But apparantly it is getting enabled in EL7.
09:30 kshlm rastar, I'm doing a direct source build, with the script I linked to above.
09:30 anoopcs kshlm, autotools version may be different.
09:30 anoopcs kshlm, On F23 and EL7.
09:31 anoopcs autoconf, automake etc.
09:31 zhangjn joined #gluster-dev
09:32 kshlm They are and so are GCC and libc.
09:32 kshlm F23 has newer versions.
09:32 kshlm Generally, stricter rules get enabled with newer versions. But in this case the older versions appear to have a stricter rule enabled.
09:33 ppai joined #gluster-dev
09:33 rastar yes, can you add -D_FORTIFY_SOURCE=0 in configure line of build.sh and try?
09:35 anoopcs rastar, I don't think that will work.
09:38 kshlm Ah, my mistake. I should have seen the same on F23 as well. It turns out glupy wasn't configured and build on F23.
09:38 anoopcs kshlm, :-)
09:38 kshlm F23 has python3 by default, but glupy requires python2.7 (IIRC)
09:39 rastar he he
09:39 rastar kshlm: one question is this all we do for a slave setup https://github.com/gluster/gluster.​org_salt_states/tree/master/jenkins ?
09:41 kshlm rastar, Not just that. We also apply some other states (base, salt.minion, ntp etc).
09:41 kshlm Look at the top.sls file.
09:44 rastar kshlm: thanks!
09:45 rastar kshlm: about glupy, were you able to figure out where it is asking for FORTIFY_SOURCE?
09:47 kshlm rastar, Not yet.
10:15 pranithk|afk xavih: Hey, did you get a chance to implement eager-lock locking api? We are thinking of implementing an xlator which will implement eager-lock as xlator...
10:33 zoldar I have a 3 node setup with one arbiter (3.7.6 at the moment). The volume contains 1-2 big KVM image files. Now every now and then a self-heal operation is underway. There's no loss of connectivity during that time or anything like that. Is that normal? I'm asking because I have a script for nagios monitoring, which treats a presence of files in healed state as critical - I will turn it down to warning if that's nothing unusual.
10:34 pranithk zoldar: how do you check for self-heal operation underway?
10:44 zoldar pranithk: gluster volume heal a-volume info
10:44 zoldar pranithk: the script does it the same way
10:45 zoldar pranithk: it just checks if number of entries is non-zero
10:45 pranithk zoldar: Do you want to share the output when it is non-zero?
10:45 zoldar pranithk: sure, sec
10:47 zoldar pranithk: here's one example http://dpaste.com/16BTWTR
10:48 zoldar pranithk: but usually it looks a bit different - there's a message the the heal is *probably* underway - will try to catch that
10:48 pranithk zoldar: Could you give getfattr -d -m. -e hex <path-to-file-on-brick> output
10:49 zoldar pranithk: from a single node?
10:49 pranithk zoldar: on all
10:50 zoldar pranithk: ok, but this happens just for a couple of seconds - do you want to have this at the moment when heal is happening?
10:51 zoldar or does it make no difference?
10:51 ppai joined #gluster-dev
10:51 pranithk zoldar: you mean after that it doesn't show up as it needs heal?
10:52 zoldar pranithk: yeah, it appears for a short moment
10:53 zoldar pranithk: here's the output anyway http://dpaste.com/2MGJNA0
10:54 pranithk zoldar: arbiter will also have a file. Why is it not there?
10:54 pranithk zoldar: It should have the file and it should be empty
10:54 EinstCrazy joined #gluster-dev
10:54 zoldar hmm seck
10:54 zoldar sec
10:55 zoldar sorry, my stupid assumption, forgot the file itself is actually there
10:55 pranithk zoldar: brb
10:56 zoldar pranithk: updated http://dpaste.com/0RWH1JY
10:56 pranithk zoldar: back, on how many nodes do you run "gluster volume heal <volname> info" ?
10:57 pranithk zoldar: I mean for your nagios
10:58 zoldar pranithk: on one
11:00 pranithk zoldar: I think I am understanding what is happening. How frequently do you see this problem?
11:00 pranithk zoldar: We will need to do one experiment
11:00 zoldar pranithk: every couple of minutes?
11:01 zoldar pranithk: maybe more frequently, not sure
11:01 anekkunt joined #gluster-dev
11:01 pranithk zoldar: Let's do the experiment. "gluster volume set <volname> cluster.self-heal-daemon off"
11:02 pranithk zoldar: This disables self-heal. Tell me if you still observe the problem
11:02 pranithk zoldar: If it still happens then it is a different problem than I thought it is..
11:02 zoldar pranithk: sure. btw here's another instance of the problem: http://dpaste.com/0F4M759
11:03 pranithk zoldar: Hmm... I feel what I am thinking is the problem is mostly it. But the experiment will confirm...
11:04 pranithk zoldar: Will you do the experiment?
11:04 zoldar pranithk: ok, I have changed the settings on one of the volumes
11:05 zoldar I will observe it for some time
11:05 zoldar pranithk: do I have restart anything after disabling the daemon?
11:06 pranithk zoldar: cool. Report to me.
11:06 pranithk zoldar: brb
11:07 zoldar pranithk: ok. anyway, the problem still occurs http://dpaste.com/0F4M759
11:11 nishanth joined #gluster-dev
11:12 pranithk zoldar: :-(. Then not what I thought it is.
11:12 pranithk zoldar: Could you give me the logs of the mount i.e. client machine and all the servers? zip them and attach it to a bug?
11:12 pranithk zoldar: Raise a new bug....
11:13 zoldar pranithk: all machines are clients and servers at the same time
11:13 pranithk zoldar: As per the data you provided me with extended attributes(xattrs from here on) it seems like things are fine
11:14 pranithk zoldar: I just want to find out why this is misleading...
11:14 zoldar pranithk: you need only mount logs?
11:14 pranithk zoldar: just zip /var/log/glusterfs on all the machines and send it to me
11:14 zoldar ok
11:15 pranithk zoldar: By when do you want this resolved? I have few things lined up for my emplyer (Redhat) that I need to deliver. So I need to prioritize. Your inputs will help me.
11:15 pranithk zoldar: Actually I first need to find Root cause. Then I will be able to comment more about this..
11:16 pranithk zoldar: Wait let me see my tasks...
11:17 zoldar pranithk: hah. well I was just about to put first prod stuff there
11:17 zoldar pranithk: but I don't feel in a position to request anything from you :)
11:18 zoldar pranithk: if this doesn't blow me up, I could live with that for some time
11:18 pranithk zoldar: Tell me exactly when you want to put things in production?
11:19 zoldar pranithk: by next week for sure
11:20 pranithk zoldar: ok here is what I will do. I will spend some time tomorrow to figure out if there is something bad happening. I will update the bug with what I found. Based on that we shall take a call?
11:21 pranithk zoldar: I have an important meeting on Wednesday by which I need to complete a feature and read up on Erasure coding algorithms...
11:21 zoldar pranithk: sounds cool, but I wouldn't want to strain you.
11:22 pranithk zoldar: well I wouldn't want to lose a user who takes the time to log bugs and test stuff out :-). So don't worry. I will inform if something comes up and I won't be able to spend time on it
11:22 zoldar pranithk: sure, fantastic, thanks a bunch. I'm preparing those logs
11:22 pranithk zoldar: I will need to leave in a bit. Log the bug and attach the logs to the bz
11:23 zoldar pranithk: will do
11:23 pranithk zoldar: thanks dude
11:25 pranithk xavih: Hey are you online, I see you merging patches?
11:26 xavih pranithk: hi, yes I'm here now
11:26 pranithk xavih: josferna tested both of our patches and confirmed things are fine now :-)
11:26 xavih pranithk: great :)
11:27 pranithk xavih: So we were wondering if you made any progress on that eager-locking library...
11:28 pranithk xavih: remember? Anuradha (atalur), you and I were talking about it a while back?
11:28 xavih pranithk: I haven't done anything else since I returned from holidays :(
11:28 overclk joined #gluster-dev
11:29 pranithk xavih: okay. So couple of developments there are: We were thinking of implementing eager-locking as xlator and use hints to pre-lock in the hopes that a lock will be requested. We want to integrate with compound xlator that I am thinking of doing...
11:30 pranithk xavih: so open fop can be overloaded with inodelk if it is open for writing...
11:30 atalur joined #gluster-dev
11:30 pranithk xavih: same with create fop
11:31 xavih pranithk: who will decide that ? the compound xlator ? or the caller xlator ?
11:32 pranithk xavih: Nothing is finalized yet. But this is what I thought so far...
11:32 xavih pranithk: I'm telling that because ec will need inodelk also for reads ?
11:32 pranithk xavih: we have two xlators between afr and client xlator (similar for ec as well) 1) eager-lock 2) hints
11:33 pranithk xavih: We can configure the xlator by giving options about what all it will compound...
11:33 xavih pranithk: does this means added complexity because of multiple xlators combinations ?
11:33 pranithk xavih: wait I am getting confused. Let me gather my thoughts
11:34 pranithk xavih: hmm... no code will change in ec/afr
11:34 pranithk xavih: At least that is what I am seeing so far...
11:34 xavih pranithk: maybe not in ec/afr, but someone (glusterd ?) will get more complex
11:35 pranithk xavih: Hmm... why? we are just adding two more nodes between afr and client. right?
11:35 pranithk xavih: Just like perf xlators...
11:36 xavih pranithk: you said that some options will be needed for the compound xlator
11:37 pranithk xavih: yes, to configure it based on if it is EC or afr we may want to do things differently...
11:37 xavih pranithk: as I understand it, glusterd will need to know that when using ec, it will need to set some option for compound xlator to tell him that reads also need inodelk
11:37 pranithk xavih: not compound xlator, but hints xlator.
11:38 xavih pranithk: and what about tiering ? if the same volume has afr and ec glusterd will need to have a lot of knowledge about each xlator
11:38 pranithk xavih: That is already solved ;-)
11:38 xavih pranithk: I think this will be bad as things get more complex in the future
11:38 xavih pranithk: how ?
11:39 zoldar pranithk: here's the bug report https://bugzilla.redhat.co​m/show_bug.cgi?id=1283956
11:39 glusterbot Bug 1283956: unspecified, unspecified, ---, bugs, NEW , Self-heal triggered every couple of seconds and a 3-node 1-arbiter setup
11:40 pranithk xavih: tiering has hot and cold tiers. So when it generates volfile, it breaks up the tier volume into individual hot/cold volumes and generates the volfile. So there we can make this change.
11:40 pranithk zoldar: thanks!
11:41 zoldar pranithk: huh, that attachment didn't get added
11:41 zoldar pranithk: readding it
11:41 zoldar pranithk: ok, it's there
11:41 pranithk zoldar: cool
11:42 xavih pranithk: even this way, glusterd needs knowledge about all possible volume configurations and implementation details of each xaltor to decide which options to set for each volume
11:43 pranithk xavih: Hmm... then we will have to add the request to pre-lock as part of xdata in afr/ec
11:44 xavih pranithk: what if ec is rewritten in a way that does not need inodelk for reads ? this will also require a change in glusterd. This kind of dependencies have added a lot of complexity to current glusterd (that's why glusterd 2.0 is being developed)
11:44 pranithk xavih: as part of xdata in open
11:44 xavih pranithk: then ec will need to know about the hints/compound xlator ;)
11:44 pranithk xavih: then it shouldn't pre-lock :-D
11:45 pranithk xavih: Okay, let me tell you the problem...
11:45 pranithk xavih: https://public.pad.fsfe.org/p/afr-small-file-perf
11:46 pranithk xavih: there is scope for ~30% improvement for small file performance if we can detect pre-locks and compound them with fops that must be sent over network
11:46 pranithk xavih: This is profile info of untar
11:47 xavih pranithk: I know the problem and I agree that something must be done. However I don't like the xlator approach because it's not transparent
11:47 itisravi joined #gluster-dev
11:48 xavih pranithk: I did some performance testing some time ago and the communications layer seemed the real problem
11:48 pranithk zoldar: itisravi was looking into the bug
11:48 pranithk xavih: meaning?
11:48 xavih pranithk: if we reduce the network trips, we'll increase performance, but we won't solve the problem
11:48 pranithk xavih: Round trips?
11:48 xavih pranithk: the performance hit was much higher than the network latency
11:48 pranithk xavih: yeah, reducing round trips is what we need..
11:48 itisravi zoldar: so I randomly looked into one of the mount logs and saw disconnect messages like so: www1-client-0: disconnected from data_www1-client-0. Client process will keep trying to connect to glusterd until brick's port is available
11:49 pranithk xavih: didn't understand "pranithk: the performance hit was much higher than the network latency"
11:50 ppai joined #gluster-dev
11:50 xavih pranithk: the request latency was at least one order of magnitude higher than the network latency
11:50 xavih pranithk: I think (but I haven't verified it) that the network communications layer is adding a lot of delays in each request
11:51 pranithk xavih: Are you saying we have bugs in rpc layer?
11:51 pranithk xavih: we *may* have?
11:51 xavih pranithk: if performance is considered a but, then yes, I think we could improve that
11:51 xavih pranithk: s/but/bug/
11:51 pranithk xavih: ah!
11:52 pranithk xavih: Not only reducing number of round trips, we also need to find and fix delays in rpc layer?
11:52 pranithk xavih: Did I understand you correctly?
11:52 xavih pranithk: yes
11:52 xavih pranithk: that's what I think
11:53 xavih pranithk: however I'm not sure we could improve that while keeping rpc protocol...
11:53 pranithk xavih: oh! then?
11:53 xavih pranithk: gluster 4.0...
11:53 spalai joined #gluster-dev
11:53 pranithk xavih: meaning? we will still have rpc right?
11:54 xavih pranithk: that's the problem I see...
11:55 pranithk xavih: What can we replace it with?
11:56 xavih pranithk: there are many options, like google protocol buffers and some others
11:57 pranithk xavih: Hmm... okay. I will talk to Raghavendra who maintains rpc to see if he knows of any issues.
11:57 xavih pranithk: and we could also improve parallel managing of requests (though I don't have much knowledge about current implementation and I'm not sure if it's really efficient already)
11:58 zoldar itisravi, pranithk : ok, so I was wrong suggesting that's not connectivity related - I will double check on firewall setup and network, thanks
11:58 aravindavk joined #gluster-dev
11:59 pranithk xavih: I sent Raghavendra a mail CCing you.
12:00 pranithk xavih: Okay how busy are you next week?
12:00 itisravi zoldar: no problem
12:01 xavih pranithk: I'm working on new version of ec
12:01 pranithk itisravi: if zoldar confirms that there are problems with existing firewall could you close the bug as NOTABUG?
12:01 itisravi pranithk: sure thing.
12:01 pranithk xavih: oh cool. Do you think you can spend some time for discussing about reducing the number of network round trips?
12:02 zhangjn joined #gluster-dev
12:02 pranithk xavih: We want to come up with something we all are happy with. Otherwise we will have two implementations, one for ec and other for afr
12:03 zhangjn joined #gluster-dev
12:03 xavih pranithk: I think I will have some time... let me know when.
12:05 nbalacha joined #gluster-dev
12:09 pranithk kdhananjay: atalur^^
12:09 pranithk xavih: I will need to leave now. I will catch up with kdhananjay, atalur about the timings. Have a nice weekend. Cya
12:09 xavih pranithk: ok. see you later :)
12:21 mchangir|wfh joined #gluster-dev
12:27 jiffin ndevos: are u around??
12:27 ndevos jiffin: yes, I am!
12:38 ira joined #gluster-dev
12:55 Manikandan joined #gluster-dev
12:59 zoldar itisravi: just realised that you were looking at the entries with a timestamp from yesterday - the problem is occuring today and it happens every couple of minutes/seconds
13:00 zoldar no problem as in disconnect happening
13:00 mchangir|wfh joined #gluster-dev
13:09 ggarg joined #gluster-dev
13:16 Manikandan joined #gluster-dev
13:23 RedW joined #gluster-dev
13:31 mchangir_ joined #gluster-dev
13:46 nishanth joined #gluster-dev
13:57 ashiq joined #gluster-dev
14:19 shaunm joined #gluster-dev
14:36 ashiq joined #gluster-dev
14:58 nishanth joined #gluster-dev
14:58 spalai joined #gluster-dev
15:02 rafi joined #gluster-dev
15:04 spalai left #gluster-dev
15:21 lpabon joined #gluster-dev
15:24 lpabon Question: When expanding a volume (as shown in http://www.gluster.org/community/documentati​on/index.php/Gluster_3.1:_Expanding_Volumes ), do the new bricks have to be the same size of the originals?
15:25 lpabon shyam: ^^
15:29 pppp joined #gluster-dev
15:34 ndevos lpabon: not strictly required, and recent versions of dht should take different sizes into account
15:34 lpabon ndevos: cool, can older versions of glusterfs also handle different size bricks in expansion?
15:40 ndevos lpabon: I think it can, but the distribution of the files would not really be even
15:40 lpabon ah i see, ok, thanks
15:41 ndevos lpabon: I do not remember which version introduced this dht feature, shyam or jdarcy might know
15:42 hagarth_ joined #gluster-dev
15:46 rafi joined #gluster-dev
16:00 Manikandan joined #gluster-dev
16:02 jiffin joined #gluster-dev
16:04 jiffin ndevos: are u there??
16:04 ndevos csim: could it be that Gerrit has an issue? I can not "git fetch" or post patches...
16:04 ndevos jiffin: I'm trying to :)
16:05 jiffin ndevos: hold your hands tightly
16:06 ndevos csim: ssh -vv review.gluster.org hangs at "debug1: expecting SSH2_MSG_KEXDH_REPLY"
16:08 csim ndevos: let's do the restart dance...
16:08 csim (I should also learn how to debug java)
16:10 csim ndevos: better now ?
16:10 jiffin ndevos: any update on that issue??
16:12 ndevos csim: my internet is very slow today, but it passes that log message now
16:13 * csim go back on ruby hacking
16:13 ndevos jiffin: I did not have any time to try it out
16:13 hagarth_ ndevos: what are the burning infra issues atm?
16:14 vimal joined #gluster-dev
16:14 hagarth_ ndevos: just trying to get a hang of all known infra problems and work on a plan to resolve them :).
16:14 ndevos csim++ thanks it seems to work again (although *very* slowly for me)
16:14 csim hagarth_: gerrit fail too much, and is rather unmanaged
16:14 glusterbot ndevos: csim's karma is now 17
16:15 csim hagarth_: jenkins should be ok, except "being at iweb" mean broken from time to time
16:15 csim we need to find a way to let people upload to download without at the same time giving root too much
16:16 csim and I think that's the most burning one
16:16 ndevos hagarth_: since a few days, Jenkins does not auto-start test anymore, something with Gerrit broke
16:16 csim ndevos: the security update of jenkins broke the plugin, or something ?
16:17 ndevos csim: I do not know what broke it, kshlm was looking into it
16:18 csim ndevos: that was a rather big jump :/
16:19 ndevos csimL I think he mentioned that the plugin may need an update too
16:31 hagarth_ ndevos: also the gerrit --> github syncing failure right?
16:32 hagarth_ csim: where are we wrt migrating off iweb?
16:33 hagarth_ ndevos: is anybody looking into jenkins not auto-starting tests?
16:34 ndevos hagarth_: yes, gerrit -> github too, I do not think that has been fixed
16:34 ndevos hagarth_: kshlm is aware of the jenkins autostart bit, I think he continues to look into that (see email to maintainers/devel list)
16:34 hagarth_ ndevos: ok..
16:35 hagarth_ my mail management is a mess atm .. hence bringing myself upto speed here :)
16:35 ndevos sure, no problem :)
16:36 hagarth_ ok.. will check the jenkins part with kshlm
16:36 hagarth_ csim: too many gerrit failures? is it related to general slowness or anything else?
16:37 rafi joined #gluster-dev
16:37 ndevos too regular Gerrit failures, I think
16:37 csim hagarth_: I had to restart it twice this week
16:37 csim which is a increase when comapred to the rest of the time
16:38 ndevos csim: got munin to monitor Gerrit?
16:40 csim ndevos: nope, doing so might deploy stuff like ssh hardening, etc, that could break prod
16:41 csim (due to salt pushing all kind of stuff, and not just munin)
16:41 csim but I can surely verify what is pushed and do it on monday
16:41 hagarth_ csim: how about adding more memory to the VM that hosts review.gluster.org?
16:42 hagarth_ I think we have ample memory on the host
16:42 csim hagarth_: I wanted to move it to the RH DC and do that
16:42 hagarth_ csim: if it is going to take longer, we can use this as an interim workaround in iweb
16:44 csim hagarth_: yeah, probem is moving in the DC is blocked on us getting ip, etc
16:45 csim not sure if the iweb host has so much memory left
16:45 csim but I guess killing shell.gluster.com free some space
16:45 hagarth_ csim: the last I checked it did
16:46 csim hagarth_: on engg.gluster.org ?
16:46 hagarth_ csim: yes
16:46 rafi joined #gluster-dev
16:46 csim indeed, with htop and color, we see better
16:47 hagarth_ ndevos: any solutions in mind for the broken gerrit --> github sync?
16:48 ndevos hagarth_: csim and kshlm were looking into a glusterbot account for GitHub, not sure whwre they got with that
16:51 hagarth_ ok, one more email reminder :D
16:52 hagarth_ misc, ndevos: or should we get into a IRC meeting on Monday to sort this over?
16:55 ndevos hagarth_: I'm out on Monday-Wednesday, so maybe Thursday (15:00 CET or later)
16:56 hagarth_ ndevos: I will see what we can do earlier than that. lot of folks will be out in US during the second half of next week owing to Thanksgiving.
16:57 ndevos hagarth_: oh, right, yes
16:57 * ndevos reconnects in the hope his 3g is faster than this aweful wifi
16:58 hagarth_ ndevos: not at home?
16:59 ndevos hagarth_: yes, but there are some issues with several ISPs in amsterdam
16:59 ndevos oh, and this actually is *MUCH* faster
17:00 ndevos I can type, and even before I press "enter" things are on my screen :)
17:00 * ndevos tries to "git fetch" again
17:01 ndevos well, latency still isnt great, but at least IRC works reasonable again
17:01 hagarth_ :)
17:02 Manikandan joined #gluster-dev
17:02 ndevos hagarth_: http://www.gluster.org/pipermail/a​nnounce/2015-November/000051.html
17:03 ndevos a similar email is going through centos-announce@ as we speak
17:06 hagarth_ ndevos: cool, tweeted it out now
17:09 csim hagarth_: I am out on monday
17:10 csim or rather, i am traveling in the train, but I will be working in the morning
17:10 csim ndevos: RH did had issue with Cogent, between our DC in Amstermda and the one in Phoenix
17:11 ndevos csim: yeah, it seems Cogent is rather a big one in amsterdam and connects the rest of the world for several ISPs
17:11 csim ndevos: the whole ISp business and Tier&, etc is a bit funny
17:11 hagarth_ csim: ok, will sync up with kshlm on Monday and reach out to you if anything else is needed.
17:11 csim or depressing
17:12 csim hagarth_: oh, you can always send me email, I will happy to answer or anything
17:13 hagarth_ csim: cool, thanks. will do that.
17:13 ndevos csim: I have no idea how ISPs are working together, but it seems like they have a single-point-of-failure in there
17:15 csim ndevos: could have been a DC who burned, or someone pushing the wrong firmware on the wrong set of routers :)
17:18 hagarth_ ndevos: I also intend setting up a gluster testing status meeting pretty regularly.. maybe will have first of that next week.
17:19 ndevos hagarth_: that would be good, I have access to CentOS' Jenkins and we can run multi-host tests there
17:19 hagarth_ ndevos: cool
17:21 ndevos hagarth_: we should try and run our regression tests there first, see how things go/work and then extend to multi-host and all
17:21 ndevos hagarth_: there should be more than enough hardware to run tests, and it is real hardware too, much faster than VMs :)
17:22 ndevos hagarth_: Arrfab in #centos-devel offered to help with that too, he did some glusterfs update tests before there
17:23 hagarth_ ndevos: let us explore that for sure.
17:24 ndevos hagarth_: yes, I just need one or two helpers for that, I'm running out of time much too quickly
17:25 hagarth_ ndevos: right, how's the kerberos thing coming along?
17:26 * ndevos *cough*
17:27 ndevos hagarth_: http://review.gluster.org/#/c​/12118/2/in_progress/Kerberos.md is the latest doc, it should be pretty complete
17:27 hagarth_ ndevos: ok
17:27 ndevos unfortunately, devs that wanted to help out, are busy with other things :-/
17:27 ndevos ... and so am I :-(
17:28 hagarth_ ndevos: yeah, we need to prioritize this for 3.8. Let us sync up next week or the one after to see how we can make it happen.
17:28 ndevos hagarth_: sure, I might be able to do a quick meeting wednesday afternoon in case you are off on Thursday+Friday
17:29 ndevos eating not turkey
17:29 hagarth_ ndevos: sounds like a plan, can you shoot me an invite whenever it is convenient for you on Wednesday?
17:30 ndevos hagarth_: yes, I'm just checking my flight times, visiting Meghana in Manchester :)
17:30 hagarth_ ndevos: wow, have fun and say hi to her from me :)
17:31 ndevos hagarth_: if you decide t omove to a nice place, I might visit you too :P
17:32 hagarth_ ndevos: wherever i stay always will be a nice place :)
17:32 ndevos lol!
17:36 mchangir_ joined #gluster-dev
17:36 skoduri joined #gluster-dev
17:36 * jiffin steps between ndevos and hagarth_
17:36 hagarth_ jiffin: hi, what's up?
17:37 jiffin hagarth_: good
17:37 * ndevos hands a few bits of internet to jiffin
17:37 jiffin hagarth_: this time winter is much severe in bangalore
17:37 jiffin ndevos: received :)
17:38 hagarth_ jiffin: heard about that
17:38 ndevos jiffin: oh, you got snow?
17:38 jiffin hagarth_: you change boston into a nice place
17:38 hagarth_ with el nino around, I expect a lot of changes in weather patterns too.
17:38 hagarth_ jiffin: it always has been :)
17:40 jiffin ndevos: here winter means just  cold , there is no snow in south india :(
17:40 jiffin hagarth_:  :D
17:42 jiffin ndevos: I need clarification regarding nfs-client <-> nfs-server communication
17:43 ndevos jiffin: ok, but I dont have much time anymore
17:43 ndevos jiffin: in general, the nfs-client <-> nfs-server communication goes over the network!
17:44 jiffin ndevos: hehe , can u please give 10mins from valuable timewallet?
17:44 ndevos jiffin: heh, I'll try, but I need to leave home in ~15 minutes
17:45 jiffin question 1.) when nfs server comes back again , do it send notification to client through *network*??
17:46 ndevos jiffin: NLM/NSM might
17:46 ndevos jiffin: details of the notifications are in "man rpc.statd"
17:46 jiffin ndevos: do our code base contains any information regarding that?
17:46 hagarth_ jiffin: look for sm_notify()
17:46 jiffin ndevos: k
17:47 jiffin hagarth_: thanks
17:48 hagarth_ will be off for a bit now, bbl.
17:48 jiffin ndevos: the paremeters timeo, retrans, hard make client for retry communication right??
17:49 jiffin to server , again *through network*
17:49 jiffin for indefinitely
17:49 ndevos jiffin: yes, "hard" is the default and will retry indefinitely, more details are in "man 5 nfs"
17:50 jiffin i read that , as far i understand both are not happening in the bug
17:50 jiffin from the client logs
17:51 jiffin anyway I will start debugging from sm_notify
17:51 ndevos jiffin: the nfs-client will log something like "server .... is not responding for x seconds, retrying"
17:52 jiffin ndevos: it happens only once in the log last previous 1hr
17:53 jiffin and one more interesting thing is that , nfs-server had come up before that log has printer,
17:53 ndevos jiffin: hmm, maybe the server is responding something, or something else bad happened
17:53 jiffin s/printer/printed/
17:54 ndevos jiffin: well, the connection from the nfs-client might still be "connected", eventhough the nfs-server got restarted (so no handler on the old connection anymore)
17:54 ndevos haI tried to send you an invite, can you check if you got it?
17:55 ndevos oh, thats why tab-completion failed, he's gone
17:55 jiffin ndevos: not yet
17:55 jiffin ndevos: wait
17:56 ndevos jiffin: it is rather tricky to check network connections that the kernel made, they dont show pu in 'netstat' I think
17:57 jiffin ndevos: hmm
17:57 ndevos jiffin: you might be able to see the old connection on the nfs-server side though
17:57 jiffin ndevos: any tool for that??
17:57 _iwc joined #gluster-dev
17:58 jiffin using netstat??
17:58 ndevos jiffin: on the server-side, you can use 'netstat' or 'ss
17:58 ndevos '
17:58 jiffin ndevos: anyway I am taking anymore time from u. Thanks for ur time ndevos++
17:59 glusterbot jiffin: ndevos's karma is now 217
17:59 jiffin s/am/am not/
17:59 jiffin ndevos: say hi to Meghana from ganesha team
17:59 ndevos jiffin: yeah, I will :)
18:00 ndevos jiffin: have a good weekend!
18:00 ndevos I'll probably be able to reply to some emails every now and then, but dont expect quick replies :)
18:00 jiffin and also ask  who is her favorite God :P
18:00 jiffin ndevos: i know that
18:01 jiffin ndevos: wish u luck for the presentation
18:02 jiffin ndevos: enjoy ur good weekend too , cya
18:02 ndevos jiffin: wait, what, I have a presentation?
18:03 * ndevos wonders if he forgot something
18:05 jiffin ndevos: i manipulate dates of T-DOSE incorrectly
18:06 ndevos jiffin: ah, pfew, yes, that should be next weekend :)
18:08 * jiffin is no more holding hands of ndevos , he can leave now and enjoy the rest of the week
18:08 jiffin end
18:08 ndevos cya jiffin!
18:08 jiffin ndevos: bye
18:28 hagarth_ joined #gluster-dev
18:30 Manikandan joined #gluster-dev
19:37 hgowtham_ joined #gluster-dev
21:42 shaunm joined #gluster-dev
22:33 hagarth_ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary