Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2016-09-15

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:01 itisravi joined #gluster-dev
01:27 EinstCrazy joined #gluster-dev
01:48 ilbot3 joined #gluster-dev
01:48 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
02:56 nishanth joined #gluster-dev
03:11 nbalacha joined #gluster-dev
03:20 magrawal joined #gluster-dev
03:25 pranithk1 joined #gluster-dev
03:27 baojg joined #gluster-dev
03:37 gem joined #gluster-dev
03:42 kkeithley nigelb: slave33 is sick
03:43 nigelb kkeithley: fixed and back in the pool.
03:45 nigelb kkeithley: do you know what to make of this failure? https://build.gluster.org/job/cage-test/2/console
03:46 nigelb Or is niels better suited?
03:51 mchangir joined #gluster-dev
03:58 atinm joined #gluster-dev
03:59 pranithk atinm: http://review.gluster.org/#/c/15497/ needs to be merged, master patch is already merged. Could you do it once Ravi comes to office? He reviewed it on master
04:00 aspandey joined #gluster-dev
04:01 atinm pranithk, sure, I can do it on afternoon
04:20 itisravi joined #gluster-dev
04:21 itisravi joined #gluster-dev
04:26 Muthu joined #gluster-dev
04:35 ppai joined #gluster-dev
04:46 sanoj joined #gluster-dev
04:46 nishanth joined #gluster-dev
04:50 prasanth joined #gluster-dev
04:50 rafi joined #gluster-dev
04:51 kkeithley nigelb: no, not without digging further.
04:54 prasanth joined #gluster-dev
05:08 ankitraj joined #gluster-dev
05:10 jiffin joined #gluster-dev
05:10 prasanth joined #gluster-dev
05:13 aspandey joined #gluster-dev
05:14 skoduri joined #gluster-dev
05:16 karthik_ joined #gluster-dev
05:23 ndarshan joined #gluster-dev
05:39 Bhaskarakiran joined #gluster-dev
05:49 hgowtham joined #gluster-dev
05:56 spalai joined #gluster-dev
05:58 mchangir joined #gluster-dev
05:58 aravindavk joined #gluster-dev
05:59 k4n0 joined #gluster-dev
06:00 ramky joined #gluster-dev
06:01 kshlm joined #gluster-dev
06:01 ashiq joined #gluster-dev
06:03 poornima joined #gluster-dev
06:09 Saravanakmr joined #gluster-dev
06:17 ndevos nigelb: do you think we should clone bug 1375526 for 3.9, 3.8 and 3.7 too? how much does it affect the infra?
06:17 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1375526 unspecified, unspecified, ---, nigelb, MODIFIED , Kill rpc.statd on Linux machines
06:20 asengupt joined #gluster-dev
06:21 ankitraj joined #gluster-dev
06:22 itisravi joined #gluster-dev
06:23 kdhananjay joined #gluster-dev
06:29 pranithk1 joined #gluster-dev
06:33 itisravi joined #gluster-dev
06:35 Muthu joined #gluster-dev
06:37 aspandey joined #gluster-dev
06:46 karthik_ joined #gluster-dev
06:46 ankit-raj joined #gluster-dev
06:47 aspandey_ joined #gluster-dev
06:48 itisravi_ joined #gluster-dev
06:49 ashiq joined #gluster-dev
06:50 ankitraj joined #gluster-dev
06:51 Saravanakmr joined #gluster-dev
06:53 mchangir joined #gluster-dev
06:57 asengupt joined #gluster-dev
07:01 ashiq_ joined #gluster-dev
07:01 Saravanakmr_ joined #gluster-dev
07:01 ankit-raj joined #gluster-dev
07:04 nigelb ndevos: It's a once-in-a-while thing.
07:06 ndevos nigelb: is that a yes or a no?
07:07 Saravanakmr_ joined #gluster-dev
07:07 ankit-raj joined #gluster-dev
07:08 itisravi_ joined #gluster-dev
07:13 aspandey joined #gluster-dev
07:17 ankitraj joined #gluster-dev
07:18 dlambrig joined #gluster-dev
07:20 aravindavk ping pranithk1
07:20 ashiq_ joined #gluster-dev
07:21 pranithk1 aravindavk: hey
07:21 pranithk1 aravindavk: RC?
07:21 aravindavk pranithk1: yes, this patch is required before RC http://review.gluster.org/#/c/15502/
07:22 aravindavk pranithk1: fixes a conflict error during rpm installation
07:22 aravindavk pranithk1: are we providing rpms or just tagging
07:24 pranithk1 aravindavk: I am not sure. Shall we do it tomorrow night then? I am too sleepy now
07:24 aravindavk pranithk1: sure
07:24 pranithk1 aravindavk: if we tag and run a job it builds rpms. Then for .debs etc we need to send a mail to packaging@gluster.org
07:24 pranithk1 aravindavk: there is a procedure which we used to follow for 3.7.x release tagging
07:25 aravindavk pranithk1: ok
07:25 ndevos pranithk1: dont use the 'build rpms' option, the distributions should provide the rpms in their testign repository
07:25 pranithk1 ndevos: I will ping you tomorrow once we make the tag.
07:26 aspandey_ joined #gluster-dev
07:26 pranithk1 ndevos: to know about the procedure
07:26 ndevos pranithk1: sure
07:26 pranithk1 ndevos: cool
07:27 itisravi joined #gluster-dev
07:28 nigelb ndevos: It's an "I don't know".
07:28 nigelb I've only noticed 2 instances of it and they were spread far apart.
07:29 ndevos nigelb: ok, we'll just keep it like it is now, regression runs with the master branch will (hopefully) happen regulary on all systems
07:36 nigelb ndevos: yeah, that's my hope as well.
07:36 nigelb ndevos: On a different note, I need more help - https://build.gluster.org/job/cage-test/2/console
07:41 ashiq_ joined #gluster-dev
07:41 karthik_ joined #gluster-dev
07:42 Saravanakmr_ joined #gluster-dev
07:43 ndevos nigelb: does the hostname (output of 'hostname --fqdn') match what is in dns? the auth-test does dns-resolving
07:44 ndevos *real* DNS, not only /etc/hosts
07:45 nigelb yes, it does.
07:50 ndevos nigelb: go logs? the link on the bottom of the page is dead
07:50 nigelb yeah, because the logging command needs fixing.
07:50 nigelb where should I look? /var/logs/gluster?
07:51 ndevos nigelb: not sure, the test framework might adjust the location
07:51 ndevos nigelb: if there is /var/log/gluster, the nfs.log would be helpful :)
07:51 * ndevos points to 'nc termbin.com 9999 < /var/log/gluster/nfs.log'
07:53 nigelb Found it.
07:54 nigelb ndevos: http://termbin.com/mb05
07:54 nigelb It was tar'd up.
07:58 nigelb ndevos: hang on.
07:58 nigelb I may have given you the wrong file.
07:59 nigelb ndevos: http://termbin.com/fmq0
08:06 ndevos nigelb: reverse-dns missing?
08:11 misc that's likely
08:14 gvandeweyer hi, we have an issue with a replicate gluster setup. we tried to replace a replicate brick using normal commands, but halfway the new brick host crashed (was a 21Tb brick). Now there is ~15Tb of data on the new brick, with 21T on it's (remaining) replicate. info heal shows a lot of files, but the heal command does not seem to show progress.
08:14 gvandeweyer where can i monitor the heal progress, is there a log file somewhere?
08:15 mchangir joined #gluster-dev
08:21 devyani7 joined #gluster-dev
08:23 nigelb ndevos: ah.
08:23 nigelb misc: how much trouble is rDNS?
08:24 misc nigelb: sending a patch to IT for giving me access
08:24 misc cause I couldn't do the modifcation right now
08:25 nigelb no worries.
08:25 misc if that's urgent, they are in nerf gun range when I am in the office
08:25 nigelb nah, it's not urgent.
08:28 misc I guess it will take 1 or 2 days to get the access, then I will push the fix
08:32 aravindavk joined #gluster-dev
08:37 devyani7 joined #gluster-dev
08:37 nigelb ndevos: hrm, we aren't running into this, are we - https://access.redhat.com/solutions/126183
08:47 ndevos nigelb: well, we dont rnu into that because we use '-o nolock', just make sure no system nfs-services are enabled/running
08:47 nigelb aha, ok
08:47 ndevos nigelb: the tests really require rDNS as well, otherwise it can not match the connecting IP-address to the hostname for checking access permissions
08:48 nigelb okay, we'll get that fixed in the next few days.
08:48 nigelb and then try again.
09:17 ndarshan joined #gluster-dev
09:25 sanoj xavih: regarding https://bugzilla.redhat.co​m/show_bug.cgi?id=1339167 / 1224180
09:25 glusterbot Bug 1339167: high, unspecified, ---, rhs-bugs, NEW , Getting EIO error for the first few files when limit exceeds in disperse volume when we do writes from multiple clients
09:26 ndevos sanoj: that is an RHGS bug, is thre a gluster community version too?
09:26 sanoj 1224180 is visible to community
09:27 sanoj not sure abt 1339167 ?
09:28 xavih sanoj: tell me
09:30 sanoj For a partial write that was not acked to the application (lets say all bricks failed after doing it paritally)
09:30 sanoj whats the expected behaviour
09:31 aravindavk joined #gluster-dev
09:31 sanoj In AFR case we sync the bricks based on dirty flag, In EC(6,4) with 3 node failure we could potentially be returning different data on subsequent reads due to different choices of bricks for reconstruction
09:33 sanoj Should EC also track dirty regions and resync (at least so that subsequent reads return same data, even if it were not old/ new)
09:35 gem joined #gluster-dev
09:39 hgowtham ndevos++
09:39 glusterbot hgowtham: ndevos's karma is now 312
09:39 ndarshan joined #gluster-dev
09:41 Manikandan joined #gluster-dev
09:52 rastar joined #gluster-dev
09:53 atinm joined #gluster-dev
09:55 mchangir ndevos, do we need http://review.gluster.org/15469 for release-3.8 and release-3.9 branches ?
09:59 xavih sanoj: sorry, I was away. EC tracks these inconsistencies using the trusted.ec.version xattr. In this particular case, the fop that caused the problem will return I/O error, and following fops will also fail.
10:00 sanoj If the brick failure was transient? don't we allow subsequent reads to the disk
10:01 xavih sanoj: EC cannot return any data in that situation. It's the same than a 6 disks RAID6 with 3 failed disks. The entire RAID volume is lost. In this case only one file is lost
10:01 xavih sanoj: if no updates are made while the brick was offline, when it comes up again, it will have matching versions with the other bricks, so the data will be recovered
10:02 xavih sanoj: but if updates have been made, the data it contains is outdated and unusable
10:04 xavih sanoj: this will be minimezed when syndrome-based decoding is implemented since it could determine which bricks have consistent data and use them for decoding in a finer way (instead of all or nothing, it could correctly decode small blocks of the file independently)
10:04 sanoj xavih: If we have not updated version, dO we have a case where we reconstruct corrupted data
10:06 xavih sanoj: version is not updated until all bricks have answered, and it's only updated at all if there aren't at least k success responses
10:07 xavih sanoj: if version update itself is the operation that fails in 3 out of 6 bricks, then the same case occurs and the data becomes unreadable
10:13 sanoj xavih: do we have some persistent marker indicating our intent to write to a location.
10:23 xavih sanoj: we have a dirty flag to do so. Currently it's updated lazily, but there's a patch that will be merged soon that will update that flag before any actual write
10:24 xavih sanoj: anyway this flag is not enough for detecting inconsistencies, since it's used basically to decide if self-heal needs to repair a file
10:24 sanoj xavih: should we not have a flag as a preop to write fop
10:25 xavih sanoj: yes, the dirty flag will be used
10:26 sanoj xavih: ok thanks
10:26 xavih sanoj: yw :)
10:32 shyam joined #gluster-dev
10:33 rafi joined #gluster-dev
10:45 ndarshan joined #gluster-dev
10:47 atinm joined #gluster-dev
10:51 nishanth joined #gluster-dev
10:56 ira joined #gluster-dev
11:00 mchangir joined #gluster-dev
11:06 aravindavk joined #gluster-dev
11:18 baojg joined #gluster-dev
11:20 philiph joined #gluster-dev
11:22 philiph I have a split-brain problem on my glusterFS volume after doing a replace-brick operation. The ‘migration’ seemed to go fine for a while but stopped with errors on the new gluster peer stating ‘[2016-09-10 04:38:31.647934] E [MSGID: 108008] [afr-transaction.c:2131:af​r_write_txn_refresh_done] 0-glusterhome-replicate-0: Failing SETXATTR on gfid abe32a8c-4875-418a-9480-9b71a7176d4e: split-brain observed. [Input/output error]’
11:23 philiph The brick is 23T in size and filled for 21T. The new brick only has ~15T of this data, at that point it stopped filling the replicate brick
11:24 philiph How can I try to fix this? Thx!
11:25 gem joined #gluster-dev
11:30 karthik_ joined #gluster-dev
11:30 poornima joined #gluster-dev
11:30 rjoseph|afk joined #gluster-dev
11:30 jiffin joined #gluster-dev
11:30 asengupt joined #gluster-dev
11:30 rafi joined #gluster-dev
11:30 itisravi joined #gluster-dev
11:30 aravindavk joined #gluster-dev
11:30 kdhananjay joined #gluster-dev
11:31 lalatenduM joined #gluster-dev
11:31 skoduri joined #gluster-dev
11:31 pkalever joined #gluster-dev
11:31 kshlm joined #gluster-dev
11:31 ppai joined #gluster-dev
11:31 Saravanakmr_ joined #gluster-dev
11:31 hgowtham joined #gluster-dev
11:32 atinm joined #gluster-dev
11:32 sac joined #gluster-dev
11:32 ashiq_ joined #gluster-dev
11:32 aspandey_ joined #gluster-dev
11:32 spalai joined #gluster-dev
11:34 jiffin1 joined #gluster-dev
11:34 nishanth joined #gluster-dev
11:35 ndarshan joined #gluster-dev
11:36 Manikandan_ joined #gluster-dev
11:40 rastar joined #gluster-dev
11:44 rafi1 joined #gluster-dev
11:53 ira joined #gluster-dev
11:56 mchangir joined #gluster-dev
12:01 atinm tests/bugs/fuse/bug-858215.t is failing multiple times in centos regression
12:01 hagarth joined #gluster-dev
12:02 ndevos mchangir: all bugfixes that apply to older releases should get backported :)
12:02 ndevos mchangir: and that includes release-3.7
12:05 jiffin1 joined #gluster-dev
12:08 kdhananjay joined #gluster-dev
12:13 ndevos nigelb: bug 1376430 is for you :)
12:13 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1376430 unspecified, unspecified, ---, bugs, NEW , All builds on nbslave7g.cloud.gluster.org get aborted
12:14 nigelb oh fun
12:14 nigelb It's been a while since I dove into netbsd
12:14 nigelb I was just being grateful for it's stability.
12:15 nigelb yep, there's a bunch of stuck mount processes.
12:15 ndevos yes, I think it has been pretty stable, this is just a runaway slave
12:16 nigelb just needs a reboot.
12:16 nigelb and I've kicked it off.
12:16 nigelb I need to monitor stuck mount processes across nodes and have it alerting me.
12:16 ndevos kkeithley, jiffin, skoduri: reminder for http://review.gluster.org/14701 :D
12:18 jiffin ndevos: noted
12:19 ndevos also, ganesha.nfsd is eating 20% cpu while doing nothing :-/
12:19 ndevos that might be the upcall thread?
12:20 jiffin ndevos: disable upcall and check
12:21 ndevos jiffin: yeah, will do so later, running some tests to see if I can manage to get a OOM
12:21 philiph I have a split-brain problem on my glusterFS volume after doing a replace-brick operation. The ‘migration’ seemed to go fine for a while but stopped with errors on the new gluster peer stating ‘[2016-09-10 04:38:31.647934] E [MSGID: 108008] [afr-transaction.c:2131:af​r_write_txn_refresh_done] 0-glusterhome-replicate-0: Failing SETXATTR on gfid abe32a8c-4875-418a-9480-9b71a7176d4e: split-brain observed. [Input/output error]’. An
12:21 philiph has an idea how to fix this?
12:21 nigelb kkeithley: have you played around with leaksanitizer/addresssanitizer?
12:21 ndevos I'm running the 2.3.3 version from the stirage-sig/testing repo, it would be nice to see it released :)
12:22 kkeithley nigelb: no, was not aware of it
12:23 ndevos philiph: I guess http://gluster.readthedocs.io/en/l​atest/Troubleshooting/split-brain/ should help you out?
12:23 nigelb http://clang.llvm.org/docs/LeakSanitizer.html / http://llvm.org/releases/3.9.0/tool​s/clang/docs/AddressSanitizer.html
12:23 nigelb I was looking into it when I was looking at clang analyzer.
12:24 kkeithley oh, the clang sanitize stuff.  I did play with it a bit.  Don't really remember any specifics. I'd have to look at it again
12:24 kkeithley fsanitize
12:25 nigelb yeah.
12:25 nigelb Just wondering if it's worth spinning a few builds with it and running regression tests against that build.
12:25 aravindavk joined #gluster-dev
12:25 nigelb something to think about for the future.
12:25 kkeithley yup
12:27 kkeithley yeah, there's a ton of stuff we could run posix tests against an fsanitized build, or against gluster w/ valgrind, once we have someone with cycles to do it
12:27 karthikus joined #gluster-dev
12:27 nigelb Getting the automation bit, I can handle.
12:27 nigelb I need someone to run the other end of things. Deal with results and follow up bugs.
12:27 kkeithley yup, that's the part I'm thinking of
12:28 nigelb strfmt_errors have been green for a while.
12:28 kkeithley the automation part
12:28 nigelb I'm going to turn on voting today.
12:28 kkeithley ;-) yeah
12:30 philiph ndevos: thanks I’ll try to list the problematic files. BTW, when I do the command gluster volume heal <VOLUME> info I get a list of files and gfid’s, but no information behind it (like ‘is in split-brain’). Is that normal?
12:32 kotreshhr joined #gluster-dev
12:36 mchangir joined #gluster-dev
12:38 ndevos philiph: I'm not sure about the output of the command, but I do know that the command can list files that are actively used
12:42 shyam joined #gluster-dev
12:43 ndevos skoduri: it seems that upcall with the cache invalidation does not ++ the rpc-xid, wireshark thinks they are all resends
12:43 ankitraj joined #gluster-dev
12:44 ndevos skoduri: just noting it here, maybe you can file a bug for it?
12:44 * ndevos needs to go downstairs to meet others for dinner
12:44 ankitraj joined #gluster-dev
12:45 ndevos skoduri: https://devos.fedorapeople.org/tmp/upcalls.pcap.gz is a tcpdump with that behaviour
12:46 skoduri ndevos, okay will look at it
12:46 ndevos apply filter 'glusterfs.cbk.proc == 5' and see the different gfids in those packets, definitely should have a rpc.xid++
12:46 glusterbot ndevos: rpc.xid's karma is now 1
12:46 ndevos hah
12:46 ndevos skoduri++ :D
12:46 glusterbot ndevos: skoduri's karma is now 37
12:46 skoduri :)
12:48 devyani7 joined #gluster-dev
12:58 kdhananjay joined #gluster-dev
13:01 mchangir joined #gluster-dev
13:03 hagarth joined #gluster-dev
13:04 nbalacha joined #gluster-dev
13:13 shyam joined #gluster-dev
13:21 Muthu joined #gluster-dev
13:31 ashiq_ joined #gluster-dev
13:36 mchangir joined #gluster-dev
13:37 rraja joined #gluster-dev
13:43 Saravanakmr joined #gluster-dev
13:48 EinstCrazy joined #gluster-dev
13:53 mchangir joined #gluster-dev
14:01 shyam joined #gluster-dev
14:13 mchangir joined #gluster-dev
14:21 mchangir joined #gluster-dev
14:24 Manikandan joined #gluster-dev
14:34 shyam joined #gluster-dev
14:35 hagarth joined #gluster-dev
14:43 gem joined #gluster-dev
14:48 mchangir joined #gluster-dev
14:58 cholcombe joined #gluster-dev
14:58 spalai joined #gluster-dev
14:59 kotreshhr left #gluster-dev
14:59 spalai left #gluster-dev
15:20 nishanth joined #gluster-dev
15:36 shyam joined #gluster-dev
15:39 kdhananjay joined #gluster-dev
15:42 ashiq_ joined #gluster-dev
15:47 ankitraj joined #gluster-dev
15:48 pranithk1 joined #gluster-dev
15:59 jiffin joined #gluster-dev
16:05 jiffin joined #gluster-dev
16:13 Manikandan joined #gluster-dev
16:39 nbalacha joined #gluster-dev
16:42 rafi joined #gluster-dev
16:43 hagarth joined #gluster-dev
17:08 ankitraj joined #gluster-dev
17:26 gem joined #gluster-dev
17:27 jiffin joined #gluster-dev
17:33 gem joined #gluster-dev
17:38 dlambrig joined #gluster-dev
17:39 ppai joined #gluster-dev
17:43 jiffin joined #gluster-dev
18:22 hagarth joined #gluster-dev
18:23 shyam joined #gluster-dev
18:40 rafi1 joined #gluster-dev
18:47 lpabon joined #gluster-dev
19:05 baojg joined #gluster-dev
19:53 hagarth joined #gluster-dev
21:36 shyam joined #gluster-dev
21:44 roost joined #gluster-dev
21:47 overclk joined #gluster-dev
21:51 uebera|| joined #gluster-dev
21:51 uebera|| joined #gluster-dev
21:51 shyam joined #gluster-dev
21:52 sankarshan_away joined #gluster-dev
21:53 hagarth joined #gluster-dev
21:54 PotatoGim joined #gluster-dev
21:58 semiosis joined #gluster-dev
21:58 semiosis joined #gluster-dev
21:59 anoopcs joined #gluster-dev
22:00 hchiramm joined #gluster-dev
22:05 semiosis joined #gluster-dev
22:44 semiosis joined #gluster-dev
22:44 semiosis joined #gluster-dev
22:47 baojg_ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary