Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2016-07-06

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:48 ilbot3 joined #gluster-dev
01:48 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
02:36 skoduri joined #gluster-dev
04:02 karthik___ joined #gluster-dev
04:08 suliba joined #gluster-dev
04:15 shubhendu joined #gluster-dev
04:25 karthik___ joined #gluster-dev
04:25 mchangir joined #gluster-dev
04:45 penguinRaider joined #gluster-dev
05:08 sakshi joined #gluster-dev
05:15 penguinRaider joined #gluster-dev
05:59 msvbhat joined #gluster-dev
06:34 s-kania joined #gluster-dev
06:36 msvbhat joined #gluster-dev
06:54 penguinRaider joined #gluster-dev
07:29 misc nigelb: FYI, I did managed to make freeipa work on the gluster infra, password is on salt-master, you can create yourself a account
07:30 misc nigelb: but gotta speak in 10 minutes, and didn't access mail since yesterday, gotta send doc next week
07:59 olia joined #gluster-dev
08:09 rraja joined #gluster-dev
08:16 olia Hi, I have a question regarding the options of performance xlator write-behind. The documentation provided online for example here: http://www.gluster.org/community/documentation/index.php/Translators/performance or here: http://www.gluster.org/community/documentation/index.php/Translators_options is not reflected to the options provided in code https://github.com/gluster/glusterfs/blob/master/xlators/performance/write-behind/src/w
08:17 olia Is there any updated documentation I could read? Thank you!
09:20 post-factum olia: aren't xlator options self-documented enough?
09:24 mchangir joined #gluster-dev
09:28 olia Most of them yes, but for example what is the option trickling-writes?
09:29 olia From the code above I found "If trickling writes are enabled, then do not hold back writes if there are no outstanding requests" what is outsanding requests?
09:31 karthik___ joined #gluster-dev
09:41 post-factum those that are not served yet, i assume
09:45 ira joined #gluster-dev
09:47 s-kania joined #gluster-dev
09:52 mchangir joined #gluster-dev
09:55 olia So if I have understood correctly, this option by default is set to on and basically is disabling this translator?
10:03 post-factum i guess, no. this option should have impact on sequential single-streamed workload only
10:05 post-factum unfortunately, i see no major write-behind contributors right now here
10:06 rafi joined #gluster-dev
10:08 pkalever joined #gluster-dev
10:09 olia ok thank you very much for your info! :)
10:11 msvbhat joined #gluster-dev
10:15 olia Generally the documentation for performance xlators is quite confusing because there are different options online and after checking the code realized that they don't always match with the actual ones.
10:16 olia I was wondrering if I could contribute in this as I have gathered them all together
10:20 bfoster joined #gluster-dev
10:33 ndevos olia: contributions for that would be most appreciated!
10:34 ndevos olia: the glusterdocs repository contains something for performance already, maybe you can extend that, or start a new page?
10:34 ndevos https://github.com/gluster/glusterdocs/blob/master/Administrator%20Guide/Performance%20Testing.md
10:35 ndevos if you start a new page, make sure to include it in https://github.com/gluster/glusterdocs/blob/master/Administrator%20Guide/index.md
11:11 pkalever left #gluster-dev
11:35 rraja joined #gluster-dev
11:39 kshlm joined #gluster-dev
11:55 olia ndevos: Maybe a new page could be started for this purpose, because this topic is not so much about testing. Is there any page for the other translators' options?
11:57 olia Also are these pages http://www.gluster.org/community/documentation/index.php/Translators/performance and http://www.gluster.org/community/documentation/index.php/Translators_options still maintained? They should be also changed
12:02 ndevos olia: no, the gluster.org/community/documentation/ is old, it is not maintained anymore, the glusterdocs repository on github should be the current version, and it is rendered on http://gluster.readthedocs.io
12:04 pkalever1 joined #gluster-dev
12:04 ndevos ..c
12:04 ndevos uh... kkeithley? were you hosting todays meeting?
12:07 kshlm Isn't kkeithley around?
12:07 ndevos I dont know...
12:08 kshlm I can host the meeting.
12:08 ndevos hmm, I guess that would be good
12:09 kshlm Okay.
12:09 ndevos kshlm++ thanks!
12:09 glusterbot ndevos: kshlm's karma is now 91
12:09 post-factum kshlm++
12:09 glusterbot post-factum: kshlm's karma is now 92
12:10 kshlm Weekly meeting has started in #gluster-meeting
12:24 pkalever joined #gluster-dev
12:49 pkalever left #gluster-dev
12:50 skoduri joined #gluster-dev
12:53 atinm joined #gluster-dev
13:00 post-factum skoduri: so regarding brick crashes we discussed and debugged before. as you may remember i did not revert the patch that fixed inode leak in order to get another stacktrace just to be sure
13:00 skoduri post-factum, okay
13:00 post-factum skoduri: but got no crash in 30 hours
13:01 post-factum skoduri: and i have a theory
13:01 post-factum skoduri: crashes appeared just in time we deployed gluster monitoring system
13:01 skoduri oh ..
13:02 post-factum skoduri: the monitpring itself was very simple in its idea. get status info with "gluster volume status all --xml" for all volumes, parse it, find out tcp ports and monitor them via zabbix
13:02 post-factum skoduri: looks good, eh?
13:02 skoduri okay..what does it monitor?
13:02 post-factum skoduri: getting the status was done by cron once in 5 minutes
13:02 post-factum skoduri: it monitors bricks tcp port availability
13:02 skoduri okay
13:03 kkeithley kshlm++
13:03 glusterbot kkeithley: kshlm's karma is now 93
13:03 post-factum skoduri: and after we deployed it, bricks started to crash. and some stacktraces were not in fact related to inode unreffing. i saw there something about getting volume status as well
13:04 post-factum skoduri: that led me to idea that now getting volume status is disruptive and buggy, and i want to find out why
13:04 skoduri post-factum, oh do you have that stacktrace referring to volume status?
13:04 post-factum skoduri: that is my fault :(. i threw it away as unuseful
13:05 ndevos post-factum: oh, zabbix, did you ever try out the Gluster templates from https://github.com/htaira/glubix ?
13:05 post-factum skoduri: now i'd like to re-create the setup somewhere not on production system and force it to fail
13:05 skoduri post-factum, oh no issues..so brick process consistently crash when the cron job reading volume status is started
13:05 post-factum skoduri: something like that. and no, we didn't try that templates
13:06 post-factum skoduri: the only thing now, is there some already known issues with getting volume status?
13:06 skoduri post-factum, I do not know of any... atinm , kshlm ^^^ any idea?
13:07 post-factum skoduri: given we have lots of volumes and lots of bricks, i guess, that should be some locking bug. but it is just a guess
13:07 post-factum skoduri: "lots" — 42 bricks × 2 hosts
13:07 kshlm post-factum, How often does volume status run?
13:07 post-factum kshlm: 5 minutes
13:08 post-factum skoduri: 84 bricks in total. and the fun thing is that bricks crashed only on the volume where status was run
13:08 post-factum s/volume/node
13:08 post-factum skoduri: and didn't crash on another node
13:08 kshlm That's not a short enough to cause problems.
13:09 post-factum kshlm: another fact is that after we dropped monitoring, no crashes occur
13:09 kshlm But you do have a large number of bricks.
13:09 kshlm Which volume status command was run?
13:09 kshlm A plain `volume status <name>` shouldn't even reach bricks.
13:09 post-factum kshlm: gluster volume status all --xml
13:10 kshlm post-factum, That is a simple volume status command.
13:10 post-factum kshlm: ok. but additionally, after parsing xml output, zabbix tested all bricks ports
13:10 kshlm That shuoldn't reach the bricks at all.
13:10 post-factum kshlm: could simple tcp handshake fail the whole brick?
13:11 post-factum kshlm: ports testing was run once per 1 minute
13:11 kshlm It possibly could. I've seen simple connection attempts to encrypted ports causing crashes.
13:11 post-factum meh
13:12 post-factum i wish i knew
13:12 skoduri post-factum, but there were crashes in inode_ctx free code path right..that seems high likely to get affected by a simple CLI command
13:12 skoduri or handshake
13:12 post-factum skoduri: so, we have two possible reasons now
13:12 post-factum s/reasons/ways to reproduce/
13:13 kshlm Unencrypted RPC and transport is pretty stable, and I don't believe a simple TCP connection could cause problems.
13:13 post-factum kshlm: that is what i think. first, we removed ports polling, and crashes persisted
13:13 post-factum kshlm: then we removed status from cron, and no crashes
13:13 kshlm Any backtraces?
13:14 skoduri sorry s/high likely/highly unlikely/
13:14 kshlm As I said before, simple volume status should just be glusterd serving the information from what it has.
13:14 post-factum kshlm: i have only backtraces related to inode unreffing
13:15 post-factum oh, wait, and another one
13:15 kshlm Only for `status <> (clients|mempool|inode|fd|..)` glusterd needs to get the information from bricks.
13:15 post-factum kshlm: http://termbin.com/l2jh
13:16 post-factum kshlm: and also http://termbin.com/7nin
13:16 post-factum skoduri: ^^ you haven't seen this one
13:16 skoduri post-factum, its looks similar to older ones
13:16 post-factum aye
13:17 post-factum skoduri: i mean, the second one, with closedir
13:17 atinm well I came in late, so I believe kshlm is the best person now to understand the issue better
13:18 post-factum skoduri: kshlm: also let me check the stacktrace with something related to status in logs
13:18 post-factum i believe it should be there
13:18 skoduri post-factum, okay for me it looks same as http://termbin.com/x0gv , http://termbin.com/66ry ..
13:18 skoduri but anyways the crash is in inode_ctx destroy done as part of inode_table_prune
13:19 kshlm post-factum, I cannot make sense out of those stacktraces. I'm not much of an expert in the glusterfs IO path.
13:20 kshlm I will check to confirm glusterd isn't doing anything new for a volume stauts command.
13:21 post-factum ok, here are another stacktraces from brick log
13:21 post-factum http://termbin.com/3t05
13:21 post-factum http://termbin.com/e5wv
13:22 post-factum http://termbin.com/fkej
13:23 post-factum i thought the last one was about status
13:23 post-factum but it seems it is not
13:24 post-factum /usr/sbin/glusterfsd(glusterfs_handle_brick_status+0x3fa)[0x7f41da0b711a]
13:24 post-factum glusterfs_handle_brick_status ← this?
13:25 kshlm post-factum, That is for volume status
13:26 post-factum kshlm: so, that could be triggered by volume status command
13:27 post-factum kshlm: as i understtod from skoduri explanation, each time brick crashed the code was referring to wrong xlator object
13:27 post-factum skoduri: am i right?
13:29 skoduri post-factum, right..looked like entire inode ctx has garbage values
13:29 skoduri post-factum, but the stack traces which you pasted now look very different..
13:30 kshlm post-factum, Do you the core for the last crash?
13:30 post-factum kshlm: unfortunately, no :(
13:30 skoduri post-factum, you mentioned that you have http://review.gluster.org/#/c/13658/3 patch included as well right..
13:31 post-factum correct, and it worked for long-long time for us
13:31 skoduri post-factum, even with monitoring?
13:31 post-factum that was the part of my memleak-related debugging
13:31 post-factum skoduri: nope, monitoring is new thing :)
13:31 hchiramm_ joined #gluster-dev
13:32 skoduri post-factum, okay..maybe its related ..I see few rpc connection cleanup included as part of that patch..
13:32 skoduri post-factum, but could you try reverting above patch along with http://review.gluster.org/#/c/14739/1 and re-test it?
13:33 post-factum skoduri: that is why i asked you to ping reviewers a week ago or so ;)
13:33 kshlm post-factum, the stacktrace says server_priv_to_dict is being called.
13:33 kshlm That only gets called for volume status clients?
13:33 kshlm s/?/./
13:33 skoduri post-factum, yeah ...I sent mail in gluster-devel ..raghavendra maintainer of rpc code mentioned that he will take a look
13:33 kshlm So something must be running volume status clients
13:34 post-factum skoduri: ok, then the plan is to reproduce the issue first, then revert two patches and see what changed
13:34 skoduri post-factum, I suggest so
13:35 kshlm post-factum, I'm guessing some changes have happened to the server xlators private struct, and the dump function server_priv_to_dict hasn't been updated to reflect the change.
13:35 post-factum kshlm: looks like a bug
13:37 kshlm I'm looking into the code right now, and I don't think it is.
13:40 kshlm server_priv_to_dict iterated over a list of connections to the server xlator and dumps them into a dict.
13:41 kshlm dict_set_str is called only once in server_priv_to_dict, and is used to set the connection identifier (basically the ip) into the dictionary.
13:41 kshlm I don't know when this identifier gets filled.
13:41 kshlm It could be possible that the identifier is only filled if a handshake happens.
13:42 kshlm So, with Zabbix establishing tcp connections to the brick, there could be a connection without an identifier in the list.
13:43 kshlm And brick crashes when trying to add this null id into the dict.
13:43 kshlm 2 things need to be confirmed for this to be true,
13:43 kshlm 1. The identifier is set only after a glusterd rpc handshake happens
13:44 kshlm 2. A volume status clients command was issued, to trigger a glusterd to request clients from bricks.
13:45 post-factum but we didn't issue "status clients" command
13:45 kshlm The reason you see only bricks crash on the node where the command was issued is because glusterd first does any actions locally, and if it fails doesn't bother sending the command out to other nodes.
13:50 post-factum kshlm: so i could create similar volume layout with multiple bricks, perform "nc -zv ip port" over all bricks ports in a loop and call "status volume all" command randomly
13:51 kshlm post-factum, But still, a plain volume status all command doesn't reach bricks.
13:51 kshlm Nothing seems to have changed in the code either.
13:52 kshlm Does your monitoring software run any commands by itself?
13:52 post-factum no, but in the meantime i definitely could run another status command like  "status clients"
13:52 kshlm Someone needs to be running a gluster volume status <volname> clients for the particular path to be hit.
13:53 post-factum so, i will use that one as well
13:53 kshlm Please do.
13:53 post-factum probably, those are different bugs
13:54 kshlm Just to be clear whenever I mentioned 'status clients', I was referring to 'gluster volume status <volname> clients'
13:54 post-factum yep, sure
13:58 post-factum skoduri: kshlm: thanks. once i get (or not get) any results, will let you know
13:58 skoduri sure :)
14:09 shubhendu joined #gluster-dev
14:23 lpabon joined #gluster-dev
14:37 kshlm joined #gluster-dev
14:54 wushudoin joined #gluster-dev
15:06 pkalever joined #gluster-dev
15:30 pkalever left #gluster-dev
16:06 penguinRaider joined #gluster-dev
16:11 jiffin joined #gluster-dev
16:16 jiffin1 joined #gluster-dev
16:30 jiffin1 joined #gluster-dev
16:43 mchangir joined #gluster-dev
17:06 penguinRaider joined #gluster-dev
17:23 hchiramm joined #gluster-dev
18:36 penguinRaider joined #gluster-dev
19:02 pkalever joined #gluster-dev
19:21 rafi joined #gluster-dev
20:11 msvbhat joined #gluster-dev
20:12 pkalever joined #gluster-dev
20:12 pkalever left #gluster-dev
22:14 penguinRaider joined #gluster-dev
22:46 penguinRaider joined #gluster-dev
23:31 hchiramm joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary