Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2016-04-27

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
03:14 tomboy64 joined #moarvm
03:23 tomboy64 joined #moarvm
05:49 Util joined #moarvm
06:10 domidumont joined #moarvm
06:15 domidumont joined #moarvm
06:52 tomboy64 joined #moarvm
07:05 MadcapJake joined #moarvm
07:15 JimmyZ_ joined #moarvm
07:22 zakharyas joined #moarvm
08:36 domidumont joined #moarvm
08:53 jnthn tomboy64: Don't know anything about that, sorry. :(
08:54 tomboy64 jnthn: i think i figured it out. it was lib64 being set without / what made it a security issue.
08:54 tomboy64 jnthn: i changed that in my patch.
08:57 jnthn ah, ok :)
09:04 jnthn tomboy64: Left some comments on your MoarVM PR. Looks good overall.
09:35 zakharyas joined #moarvm
09:47 leont_ joined #moarvm
10:01 Kooda left #moarvm
10:20 dalek MoarVM/reframe: 8b642e4 | jnthn++ | src/6model/reprs/MVMThread.c:
10:20 dalek MoarVM/reframe: Mark tc->thread_obj of unstarted threads.
10:20 dalek MoarVM/reframe:
10:20 dalek MoarVM/reframe: Otherwise, if a GC happens between thread creation and thread start,
10:20 dalek MoarVM/reframe: we'll end up with a TC that has an outdated ->thread_obj pointer.
10:20 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/8b642e4f19
10:21 dalek MoarVM/reframe: 2a022fb | jnthn++ | src/6model/reprs/MVMThread.c:
10:21 dalek MoarVM/reframe: Use MVM_load with an AO_t, to be on the safe side.
10:21 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/2a022fb580
10:27 jnthn That was by far the most common thing we blew up over in S17 tests
10:28 jnthn Fixing it makes various tests happy, and moves various ones on to their next problem
10:30 nwc10 is that a general bug, or one specific to your changes?
10:30 jnthn nwc10: That one was a general bug
10:31 nwc10 aha. *interesting*. That might be why sometimes tests hang for me under ASAN
10:31 nwc10 time & retesting will tell
10:31 nwc10 I'd not been bothering you about this
10:31 nwc10 (and it was way beyond my ability to diagnose quickly)
10:36 jnthn https://gist.github.com/jnthn/2996cce212da18720a0e4660e71fe5a5 is latest status
10:59 dalek MoarVM/reframe: 0861a76 | jnthn++ | src/core/exceptions.c:
10:59 dalek MoarVM/reframe: Missing rooting of frames in backtrace formation.
10:59 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/0861a7633e
10:59 jnthn That one actually was due to my changes :)
11:01 leont_ joined #moarvm
11:05 brrt joined #moarvm
11:16 TimToady joined #moarvm
11:33 jnthn Gosh, S32-exceptions/misc.t takes an eternity under GC torture
11:33 jnthn But provided it doesn't explode, that'll be all the ones with a bad object inside an MVMHash fixed
11:33 moritz it's a big file
11:34 moritz I guess parsing alone is an ordeal :-)
11:34 jnthn aye
11:34 jnthn It's up to test 370 by now :)
11:35 nwc10 something you just did fixed the nativecall tests
11:35 timotimo GC torture really is extra-bad because we only promote things to the gen2 when they've survived a generation
11:35 jnthn :)
11:35 timotimo so i expect we'll end up doing two GCs back-to-back with almost nothing freed in between quite often
11:35 jnthn Aye, though that's in part why it catches quite a lot of bugs
11:35 timotimo btw, do we have any code in place to make sure that when the nursery didn't shrink after a gc run we don't asplode directly after the GC run?
11:36 jnthn Yes :)
11:36 timotimo good
11:36 jnthn Pretty sure that's been in there since day 1
11:36 timotimo not like we'll ever hit that code path in regular user code :)
11:36 timotimo um, actually ... the string split case could hit that
11:36 jnthn Or, the first day we allocated objects :)
11:36 timotimo when we split a gigantic string into a crapton of chunks
11:37 jnthn For the first while there was no GC, so it was like "4MB should be enough for everyone" :)
11:37 timotimo :D
11:37 timotimo we wouldn't have gotten far in rakudo code with that attitude
11:38 jnthn It did run quite a few of the NQP tests though :)
11:39 jnthn Noting that we "cross-compiled" them
11:40 timotimo not so important any more :)
11:42 jnthn lunch &
11:42 TimToady joined #moarvm
11:48 leont_ joined #moarvm
11:49 dalek MoarVM: 2e059d9 | timotimo++ | src/6model/reprs/P6opaque.c:
11:49 dalek MoarVM: "this type" will now actually mention the type
11:49 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/2e059d922e
11:49 dalek MoarVM: 350a1ba | timotimo++ | src/6model/reprs/P6opaque.c:
11:49 dalek MoarVM: "missing serialize" and "rebless" also mention type now
11:49 timotimo ^- i decided the code's good enough to get merged, as we just had the release
11:49 dalek joined #moarvm
11:50 timotimo This improves a bunch of error messages, especially when they used to
11:50 timotimo say "this type", but not *what* type. On top of that, some additional
11:50 timotimo errors gained a bit more detail, like in P6opaque's compose. And also
11:50 timotimo errors like "this type doesn't support associative operations" (and
11:50 timotimo same for positional) now tell you what operation was attempted.
11:50 timotimo https://github.com/MoarVM/MoarVM/commit/1c11d8bbb3e67e71c58085857a2d8a7d97fc5b61
12:05 jnthn timotimo: Sound like nice error-reporting improvements
12:07 domidumont joined #moarvm
12:10 timotimo i hope people find it useful
12:17 jnthn I suspect so :)
12:18 arnsholt That definitely sounds useful! timotimo++
12:18 timotimo the errors in the compose thing are probably useless unless you write your own object system
12:18 timotimo but i figured i might as well.
12:18 jnthn S32-exceptions/misc.t ran to completion under torture
12:19 moritz \o/
12:19 arnsholt timotimo: Just because it's not likely that "ordinary" users will be doing it doesn't mean that we have to make it as painful as possible =)
12:19 arnsholt I'm generally in favour of having as specific errors as possible
12:19 jnthn So, which ones next...
12:20 domidumont joined #moarvm
12:24 timotimo :)
12:24 timotimo oh, "which ones" wasn't refering to error messages :D
12:25 jnthn no, which things from my list of GC torture failies
12:56 * jnthn hunted down another one
12:59 jnthn Bah, so S07-hyperrace/race.rakudo.moar is now a reliable deadlock rather than a reliable crash :P
13:02 timotimo yeah, race was rather fragile to begin with :|
13:04 dalek MoarVM: 4098736 | timotimo++ | src/6model/reprs/Uninstantiable.c:
13:04 dalek MoarVM: Uninstantiable will now error with debug name
13:04 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/40987366fc
13:04 jnthn hyper.rakudo.moar too
13:05 jnthn It either runs to completion or deadlocks
13:05 timotimo aye :\
13:05 jnthn Whereas before it fairly reliably SEGV'd due to a GC issue
13:07 timotimo when reading a perl6 source file we could totally catch a "invalid character in utf8", re-encode with utf8-c8 and give a bit of context around the place to help people figure out what went wrong
13:08 timotimo someone on reddit reported they put an é (or è?) in a source file and that reliably crapped out, but only later found out that for some unknown reason the editor they used decided to use some non-utf8 encoding even though they always saved as utf8 otherwise
13:09 timotimo so when the user expects they've saved something as utf8 and rakudo complains, they'll be convinced rakudo is bugged
13:09 jnthn S17-procasync/basic.rakudo.moar now seems to reliably fail to crash under torture, so at least I get that one off my list :)
13:09 timotimo whereas when we show the user the wrong bytes in question, they have a much better idea of what went wrong
13:09 timotimo thoughts?
13:09 timotimo oh yay :)
13:09 moritz timotimo: would be awesome
13:10 timotimo i'll think about that more today; right now i'm off towadrs the sauna :3
13:10 arnsholt Yeah, that definitely sounds reasonable (and a good idea!)
13:10 timotimo in addition to that, it'd be swell if we could do a bit more "encoding guessing"
13:10 timotimo like "did you expect one of those characters to appear here?"
13:11 timotimo "[PILE OF POO] - your file is probably cp1234
13:11 timotimo [UNICORN] - your file seems to be iso-1234-99999"
13:11 timotimo "almost every other byte of this file seems to be a null-byte; did you pass a utf-16 or ucs-16 file by accident?"
13:12 timotimo don't forget that 2-byte encodings are (allegedly?) really common in at least one of the CJK areas
13:14 tomboy64 joined #moarvm
13:33 dalek MoarVM/reframe: 7d51049 | jnthn++ | src/core/frame.c:
13:33 dalek MoarVM/reframe: The instrumentation level barrier may allocate.
13:33 dalek MoarVM/reframe:
13:33 dalek MoarVM/reframe: Therefore, root collectables around it.
13:33 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/7d51049110
13:34 arnsholt timotimo: I guess there might be useful code to be snarfed in Python's ftfy module?
13:38 jnthn That one changed the status of 8 things in the list (3 seemingly fixed for good)
13:46 tomboy64 joined #moarvm
13:46 timotimo arnsholt: The goal of ftfy is to take in bad Unicode and output good Unicode, for use in your Unicode-aware code. This is different from taking in non-Unicode and outputting Unicode, which is not a goal of ftfy. It also isn't designed to protect you from having to write Unicode-aware code. ftfy helps those who help themselves.
13:47 dalek MoarVM/reframe: 2698ba7 | jnthn++ | src/core/threads.c:
13:47 dalek MoarVM/reframe: Correct cleanup of temp roots at thread exit.
13:47 dalek MoarVM/reframe:
13:47 dalek MoarVM/reframe: Fixes a crash that could occur when doing GC of an exited thread.
13:47 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/2698ba7c2d
13:47 timotimo the problem that user had on reddit was when we didn't get any unicode out of the encoded blob we read
13:48 jnthn Current state of play: https://gist.github.com/jnthn/2996cce212da18720a0e4660e71fe5a5
13:48 arnsholt timotimo: Oh, that's right of course. I got it confused a bit
13:49 timotimo but it was a good suggestion anyway :)
13:49 timotimo jnthn: nice!
14:02 dalek MoarVM/reframe: c0ffff2 | jnthn++ | src/core/frame.c:
14:02 dalek MoarVM/reframe: Mark frame in unwind data, now it's GC-able.
14:02 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/c0ffff27d1
14:02 jnthn That one related to the frames change.
14:03 timotimo right
14:03 jnthn So far, 2 issues I was hoping to tease out with this torture testing fixed, and 3 others that showed up along the way :)
14:10 zakharyas joined #moarvm
14:23 dalek MoarVM/reframe: 3fce8d7 | jnthn++ | src/gc/debug.h:
14:23 dalek MoarVM/reframe: This debugging macro is better with panic.
14:23 dalek MoarVM/reframe:
14:23 dalek MoarVM/reframe: Since otherwise, the exception can be caught. Been using it the panic
14:23 dalek MoarVM/reframe: way all day, and it's been rather easier.
14:23 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/3fce8d7380
14:23 dalek MoarVM/reframe: 32643d1 | jnthn++ | src/core/exceptions.c:
14:23 dalek MoarVM/reframe: Missing root of exception message in `die`.
14:23 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/32643d1c07
14:25 timotimo oh, that's rather a bad miss :)
14:25 jnthn Yeah, and another one that's not thanks to the frames thing
14:26 jnthn But, a fix is a fix.
14:27 timotimo yup
14:28 jnthn Betting this CUnion one will be nothing to do with frames also...
14:32 timotimo sounds likely
14:39 nine_ jnthn++ # hard to imagine that rakudo even ran with so many bugs
14:40 timotimo haha
14:40 jnthn nine_: It's a matter of probability, really
14:40 jnthn These *could* all come up in user's programs and absolutely want fixing, but each rely on GC happening in a very specific place.
14:41 jnthn And when you've 4MB of allocation between each collection, along with the fact that many of these places aren't highly common code-paths, it's not so surprising
14:41 timotimo with many user programs finishing before ~20 GC runs in total, it's hard to hit something so perfectly
14:42 jnthn Right. Of course, it sucks to be the unlucky user who ends up with the SEGV or weird internal error.
14:42 timotimo yes. very.
14:43 jnthn And it sucks to be us faced with such a bug report, because details like the number of env vars they had could be enough to hide it
14:43 timotimo we really ought to be able to build some kind of instrumentation (at the C level) that traverses callframes whenever an allocation call is made and checks to see if variables have been rooted or not
14:44 nine_ Like the NaticeCall temp root bug a year ago that took months just to reproduce with an attached gdb
14:45 jnthn The CUnion one could also explain a little flappiness we saw in the nativecall union tests a while ago
14:47 dalek MoarVM/reframe: 6733e92 | jnthn++ | src/6model/reprs/CUnion.c:
14:47 dalek MoarVM/reframe: Fix CUnion layout computation GC issues.
14:47 dalek MoarVM/reframe:
14:47 dalek MoarVM/reframe: An unfortunately-timed GC in the middle of that could end up leaving
14:47 dalek MoarVM/reframe: dangling pointers. Just allocate this layout-related bits in gen2,
14:47 dalek MoarVM/reframe: as they'll probably live for a while anyway.
14:47 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/6733e927ce
14:51 jnthn https://gist.github.com/jnthn/2996cce212da18720a0e4660e71fe5a5 updated, for those following along :)
14:53 moritz jnthn: did you ever do a similar torture run in master?
14:55 jnthn moritz: Quite a while ago
14:55 jnthn moritz: It kinda wants automating.
14:55 ChanServ joined #moarvm
14:55 jnthn moritz: But it ties up my quad-core box for the at least half a day
14:56 jnthn So not like we could run it on travis or something
15:00 moritz jnthn: understandable :-)
15:02 jnthn Gee, S15-nfg/many-threads.t is pretty upset...
15:02 tomboy64 grml
15:02 * masak .oO( no freading good )
15:02 tomboy64 somehow i managed to break rakudo compilation
15:03 tomboy64 and i'm really struggling to understand how
15:04 tomboy64 https://bpaste.net/show/639b63cc1986
15:04 jnthn I don't think it's anything to do with NFG
15:04 jnthn Given it's a bogus ->outer pointer that it's tripping over
15:25 jnthn Darn, this one is proving a good bit trickier to find too...
16:00 timotimo tomboy64: you shouldn't expect rakudo-jvm to build and run properly today
16:00 jnthn I thought it built fine but had a bunch of test fails?
16:01 tomboy64 timotimo: the very same package built and ran fine a couple of days ago
16:01 timotimo we didn't have a 2016.04 a few days ago; was it perhaps the 2016.01 release?
16:02 timotimo i'll be AFK for a bit
16:03 tomboy64 http://rakudo.org/downloads/2016.04/rakudo-2016.04.tar.gz
16:03 tomboy64 but true, i might have tried the 2016.03 one
16:25 domidumont joined #moarvm
16:25 zakharyas joined #moarvm
16:27 jnthn Urgh, think I'll leave this one for today and come back to it when less tired from all the other fixes. :)
16:40 mst NOT ALLOWED
16:40 * mst injects jnthn with amphetamines
17:33 nine_ .oO(ETOOMANYFIXES?)
17:59 colomon joined #moarvm
18:16 tomboy64 joined #moarvm
18:28 Ven joined #moarvm
19:46 colomon joined #moarvm
20:14 dalek MoarVM/reframe: e52d1bb | jnthn++ | src/gc/roots.c:
20:14 dalek MoarVM/reframe: Mark tc->thread_entry_frame.
20:14 dalek MoarVM/reframe:
20:14 dalek MoarVM/reframe: Now that frames are collectable, and so can move, this pointer must be
20:14 dalek MoarVM/reframe: marked so it is kept up to date. Otherwise, it points to junk that may
20:14 dalek MoarVM/reframe: one day align with the address of an actually executing frame, causing
20:14 dalek MoarVM/reframe: a worker thread to silently exit and typically resulting in a deadlock
20:14 dalek MoarVM/reframe: because another thread is blocked on its work.
20:14 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/e52d1bbf8b
20:14 jnthn That took some darn finding.
20:15 [Coke] jnthn++
20:16 timotimo wow, god damn it
20:16 timotimo that sounds *nasty*
20:16 timotimo that'd be hyperrace right there?
20:17 jnthn Well, a bunch of tests went from "exploding" to "usually deadlock"
20:17 jnthn Those were two of them
20:19 timotimo well, did it go from deadlock to works fine with that last commit? is what i meant
20:19 jnthn timotimo: yes
20:19 timotimo awesome!
20:19 jnthn Well, as fine as it works on master
20:19 timotimo :)
20:19 timotimo yeah
20:20 jnthn I was a little suspicious something in the changes I'd done was causing new deadlocks to appear
20:21 jnthn It seemed like an odd new failure mode.
20:21 jnthn Especially when I'd expect regular nursery collects to make them less likely rather than more likely.
20:21 timotimo :D
20:22 jnthn Along the way, I found that there's probably a bug in handling of the "in a gen2 inter-gen set", and that Promise is vulnerable to a spurious wake-up
20:22 timotimo urgh
20:23 jnthn (Got a local patch for the first and know how to fix the second)
20:23 timotimo i have to read up on what spurious wakeups are and what causes them and such
20:30 jnthn Looks like the deadlocking S17-procasync/no-runaway-file-limit.t doesn't any more either. Alas, it does exhibit a different problem.
20:31 timotimo that just means it's still worth something :)
20:31 jnthn aye
20:31 timotimo https://twitter.com/loltimo/status/725413721921753089  -  did you see my little shiny? :) :)
20:32 jnthn ooh, nice
20:32 jnthn timotimo++
20:32 timotimo \o/
20:32 timotimo and jnthn++ for utf8-c8
20:38 colomon joined #moarvm
20:55 colomon joined #moarvm
20:58 dalek MoarVM/reframe: d956008 | jnthn++ | src/gc/roots.c:
20:58 dalek MoarVM/reframe: Fix clearing of "in inter-gen set" flag.
20:58 dalek MoarVM/reframe:
20:58 dalek MoarVM/reframe: Previously, we toggled it. However, multiple threads can have the
20:58 dalek MoarVM/reframe: same collectable in their inter-gen set, and may end up considering
20:58 dalek MoarVM/reframe: it. All the flag does is avoid duplicate additions, so it's safe to
20:58 dalek MoarVM/reframe: clear it when it could be set. However, before this fix we might have
20:58 dalek MoarVM/reframe: ended up with it set when it should not have been, which would be
20:58 dalek MoarVM/reframe: bad.
20:58 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/d956008c5d
20:59 timotimo oh lord
21:05 jnthn https://gist.github.com/jnthn/2996cce212da18720a0e4660e71fe5a5 # final update for today :)
21:06 timotimo great progress!
21:06 ilmari jnthn: wouldn't gen2roots[i]->flags &= ~MVM_CF_IN_GEN2_ROOT_LIST; be sufficient?
21:06 ilmari instead of the & and ^=
21:08 jnthn ilmari: Hm, yeah, that'd be another way to write it, and get rid of a branch.
21:13 dalek MoarVM/reframe: 42654cc | jnthn++ | src/gc/roots.c:
21:13 dalek MoarVM/reframe: Simplification thanks to ilmari++.
21:13 dalek MoarVM/reframe: review: https://github.com/MoarVM/MoarVM/commit/42654ccd07
21:14 jnthn OK, really enough for today :)
21:15 timotimo in that case, i wish you a good rest :)
21:16 jnthn Thanks! 'night all
21:43 leont_ joined #moarvm
21:44 colomon joined #moarvm
21:53 colomon joined #moarvm
22:33 leont_ joined #moarvm
23:07 geekosaur joined #moarvm
23:39 leont_ joined #moarvm
23:56 synopsebot6 joined #moarvm

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary