Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-08-03

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 MasterDuke i reset to 68031489deb3ba456990f4f88233664d2e079a61, since the next one touched that section of frame.c
00:07 MasterDuke now instead of a segv i get `Cannot invoke this object (REPR: P6opaque; NQPMu)`
00:07 MasterDuke `at gen/moar/stage2/QRegex.nqp:2324  (/home/dan/Source/perl6/install/share/nqp/lib/QRegex.moarvm:parse)`
00:07 MasterDuke #1  0x00007ffff77d5748 in MVM_6model_invoke_default (tc=<optimized out>, invokee=<optimized out>, callsite=<optimized out>, args=<optimized out>) at src/6model/6model.c:451
00:08 MasterDuke #2  0x00007ffff7773ca5 in MVM_interp_run (tc=tc@entry=0x555555758950, initial_invoke=0x0, invoke_data=0x7ffff786b64d) at src/core/interp.c:950
00:09 MasterDuke which is actually the same line of invoke_o as causes the segv at HEAD
00:35 http_GK1wmSU joined #moarvm
00:36 http_GK1wmSU left #moarvm
00:58 yoleaux joined #moarvm
01:52 ilbot3 joined #moarvm
01:52 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
02:53 harrisi jnthn: that is interesting. I hadn't actually thought about that, but that could make things slightly easier. it really then just goes back to translating the semantics, which is awfully annoying.
02:53 harrisi jnthn: but also the most interesting part, I think. :)
03:28 camelia joined #moarvm
03:44 Geth ¦ MoarVM: MasterDuke17++ created pull request #623: JIT pow_I
03:44 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/pull/623
05:59 brrt joined #moarvm
06:09 camelia joined #moarvm
06:17 brrt ohai #moarvm
06:17 yoleaux 2 Aug 2017 19:45Z <nwc10> brrt: ASAN finds even-moar-jit very exciting: http://paste.scsys.co.uk/564761
06:17 yoleaux 2 Aug 2017 20:28Z <nwc10> brrt: first bad commit is 2dbb62f9cade2, "Add sp_decont op" - as the only non comment change in that is to add a template, I guess it reveals a latent bug
06:17 brrt nwc10, i guess so too
06:17 brrt fortunately, i'm sure jit-bisect will tell me
06:26 Geth ¦ MoarVM/even-moar-jit: 30 commits pushed by (Timo Paulssen)++, (Samantha McVey)++, (Jonathan Worthington)++, (Bart Wiegmans)++
06:26 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/compare/1faa30f1e8...b72a544f98
06:27 brrt if you keep on updating NQP_VERSION in this rate, then i'm going to have to merge every other day
06:29 brrt also, ASAN is decidely not as informative on os x, which is really frustrating
06:30 brrt anyway, let's see what jit-bisect says
06:36 brrt joined #moarvm
06:41 brrt http://openbsd-archive.7691.n7.nabble.com/mprotect-W-X-violation-and-JDK-td312186.html
06:42 brrt or, "I'm wrong, but I'm famous, so who cares"
06:42 brrt fuuuuuuuu
06:46 brrt anyway. what if we try setting it to PROT_NONE before setting it to PROT_EXEC
06:48 brrt luajit does the exact same thing we do...
06:50 nwc10 I did wonder if the kernel keeps any state on the region. hence whether one can side step all this "you can't get there from here" by simply going via a boring state
06:51 brrt well, apparently you can sidestep the issue by writing to a file, then mmapping that file as executable
06:51 brrt so the kernel isn't that clever
07:13 brrt joined #moarvm
07:34 brrt CORE.setting with ASAN under bisect is on the slowest bisect i'v seen
07:52 zakharyas joined #moarvm
08:02 nwc10 brrt: what does jit-bisect do?
08:19 brrt are you coming to my talk at TPC, i'll explain all about it :-P
08:20 brrt jit-bisect runs a command and bisects on frame number and subsequently on basic block number to pinpoint the broken piece of code
08:21 brrt so the two envionment variables MVM_JIT_EXPR_LAST_FRAME and MVM_JIT_EXPR_LAST_BB act as 'brakes' on the expr JIT so that it doesn't do anything beyond those
08:22 brrt with MVM_SPESH_BLOCKING I ensure that - in a single threaded application - the order of spesh and JIT compilation is deterministic
08:22 brrt it occured to me that we could do the same thing with spesh
09:02 jnthn moarning o/
09:02 brrt moarning
09:02 brrt jnthn, a spesh-bisector, that seems like a decent idea, wouldn't it?
09:02 jnthn grr, hot weather = crappy sleep
09:02 jnthn brrt: Yeah, it'd save some hand work
09:02 brrt (the null-case for the bisect being to not optimize)
09:04 jnthn Though the one I hunted down last night woulda been tricky, the application just started spitting out wrong results at some point rather than crashing
09:04 zostay joined #moarvm
09:07 jnthn Uff, some lag on this connection :S
09:14 brrt you can do it with any reproducible test case, just have a script that exits nonzero on the wrong ouptput
09:14 brrt but yeah, crashes are definitively easier
09:15 jnthn Aww
09:15 jnthn A stress test of my current invoke branch blows up just after CORE.setting build
09:15 jnthn oh, in loading CORE.setting even
09:16 jnthn "Unknown string encoding 'utf8'"
09:16 jnthn huh
09:17 jnthn Sadly, wasn't the one I fixed last night
09:18 jnthn aww, also it seems it's a mis-compile of something
09:18 brrt ouch
09:18 brrt metacircular systems do have their disadvantages...
09:19 jnthn Yeah...
09:19 jnthn Well, otoh it means we test our stuff against at least one significant size application :P
09:20 brrt that is very true
09:20 jnthn My hope is that I can provoke the crash in tsets
09:20 jnthn make test doesn't; time for make spectest
09:21 jnthn ah, good, seems like some spectests will
09:24 jnthn oh wow, SEGV
09:30 jnthn Curious, it's a crash inside of eliminate_dead in graph building
09:30 jnthn But it's when we're making a graph from an existing spesh candidate for the purpose of inlining
09:30 jnthn It's a bit surprising such a graph would have dead BBs
09:35 jnthn Huh, and yeah, we get a spesh fixup error
09:44 nwc10 brrt: yes, was planning to go to your talk at the TPC formarlly known as YAPC::Europe
09:49 jnthn ah, oops
09:49 jnthn Turns out that we do an inline
09:49 jnthn But we do it inside of a basic block that dies
09:50 jnthn (Due to other optimizations)
09:52 robertle joined #moarvm
09:52 jnthn This isn't new, but we have gotten much better at inlining
09:52 jnthn Or much more capable of inlining
09:52 jnthn So I guess it stayed well hidden
09:55 Geth ¦ MoarVM/spesh-invoke: 5 commits pushed by (Jonathan Worthington)++
09:55 Geth ¦ MoarVM/spesh-invoke: c5a2cae21b | Use callsite info to insert guards before prepargs
09:55 Geth ¦ MoarVM/spesh-invoke: e97010cd6d | Make sure to mark decont for guard used.
09:55 Geth ¦ MoarVM/spesh-invoke: a33c6b516a | Guards need STable, not type object.
09:55 Geth ¦ MoarVM/spesh-invoke: 1f1f27e8a6 | Use type tuple to find spesh candidate if present.
09:55 Geth ¦ MoarVM/spesh-invoke: 388a769b46 | Enable type tuple use in multi-dispatch.
09:55 Geth ¦ MoarVM/spesh-invoke: review: https://github.com/MoarVM/MoarVM/compare/e7f7a18e2e...388a769b46
09:56 jnthn That was just a rebase on master fwiw
09:56 Geth ¦ MoarVM/spesh-invoke: 079dc9e962 | (Jonathan Worthington)++ | 2 files
09:56 Geth ¦ MoarVM/spesh-invoke: Avoid walking of end of BBs linked list.
09:56 Geth ¦ MoarVM/spesh-invoke:
09:56 Geth ¦ MoarVM/spesh-invoke: Could happen when the last block was unreachable.
09:56 Geth ¦ MoarVM/spesh-invoke: review: https://github.com/MoarVM/MoarVM/commit/079dc9e962
10:06 brrt nwc10: awesome :-)
11:06 dogbert17 jnthn: if you're bored I have a gist which might be of interest: https://gist.github.com/dogbert17/04de6ec9fbb925469d378c04a0339c03
11:08 jnthn I'm quite un-bored (though a bit frustrated I'm bug-hunting in spesh rather than putting more opts in...)
11:11 dogbert17 more opts ++, was merely wondering if this is something 'new'
11:11 jnthn I don't think so
11:11 jnthn It looks like a data race
11:12 jnthn On SC lookup
11:12 dogbert17 I had to run the script many times before the problem showed up in ASAN
11:13 dogbert17 perhaps I should do some callgrinding instead
11:28 dogbert17 a week ago I made a gist containing callgrind results for a small program showing results for the then current camelia and the merged spesh-worker branch
11:29 dogbert17 I have now added a new run to the gist, this it's the current spesh-invoke branch. https://gist.github.com/dogbert17/6bb3eeb5fc07d809adf03ac7ab6f31a7
11:29 dogbert17 *this time it's
11:33 jnthn huh, dunno what's going on with that
11:33 jnthn It should be resolving the multi candidates again now
11:33 jnthn lunch &
11:45 markmont joined #moarvm
11:53 markmont brrt: "what if we try setting it to PROT_NONE before setting it to PROT_EXEC" -- I don't think so.  It looks like the Linux kernel won't allow PROT_EXEC unless the mapping is backed by a file, see https://github.com/torvalds/linux/blob/master/security/selinux/hooks.c#L3610
11:54 markmont brrt: "you can sidestep the issue by writing to a file, then mmapping that file as executable" -- yes, but this isn't sidestepping, it is the recommended practice.
11:55 brrt i disagree, that's still very much sidestepping
11:55 brrt if that's what selinux does, then selinux is doing it wrong
11:55 markmont brrt: The reasoning is that if you have the same file mapped twice, once as writable and once as executable, then code executing in the region won't be able to find the writable address.
11:56 brrt .. come again?
11:56 brrt i don't parse
11:56 brrt … okay
11:56 brrt so
11:56 brrt no
11:56 markmont brrt: I don't like this "solution" either; I'm just reporting what I found.  Many people in the Mozilla bug agree with us.  But, currently, SELinux with deny_execmem and NetBSD 8.0_BETA won't allow anything else.
11:57 brrt okay, suppose i have a mmap'ed file, and once i've mmapped it writeable, and once i've mmapped it execable
11:57 brrt then i have a bit of writeable-executable memory, do i not?
11:58 markmont brrt: A bunch of projects, including Chrome V8, are just sticking with mmap()/mprotect() and saying that you have to have deny_execmem off or mark the NetBSD executable as wx-okay.
11:58 brrt and just saying 'well ASLR will make sure that an exploit will not be able to use that fact', then that's just nuts
11:59 brrt what about luajit? i heard theo de raadt thought luajit was doing it right, and luajit is doing just what we do
11:59 markmont Like I said, that's what people in the Mozilla bug said, and I think they are right.  But Firefox's nanojit went ahead with the temporary file mapping.
12:00 markmont I haven't looked at luajit yet, but I have looked at OpenBSD 6.1 and it allows mmap()/mprotect().
12:00 brrt i see
12:00 brrt o.O
12:00 brrt okay, at this point it seems legit just to push back to selinux
12:00 brrt and/or netbsd
12:00 brrt wrong things are just wrong
12:01 markmont That's what Chrome V8 is doing, as well as, I think, Java.
12:01 brrt okay
12:02 brrt can't say i understand the full implications of that linux bit of code
12:03 brrt anyway… i f you *do* want to implement a temporary-file-mmapping-solution, i'd be okay with that, but it'd be very nice if we could make it conditionally-compileable
12:03 brrt so that people that aren't on 'secured' systems don't have their.. .security compromised
12:04 brrt along with a configure flag that says —with-jit-security-theater
12:04 markmont Well, if the file is opened with O_TMPFILE, it never is linked into the filesystem tree and hence "shouldn't" be able to be compromised.
12:04 brrt hmmm
12:05 brrt i'm not even going to ask why we have a distinction between MAP_ANON and O_TMPLFILE
12:05 brrt :-)
12:05 markmont But that assumes that the underlying filesystem supports O_TMPFILE, which many do not.  The Mozilla nanojit code (and the libfficode based upon it) try O_TMPFILE first but then fall back to a regular open() followed by unlink()
12:06 brrt okay, that makes me chuckle
12:09 markmont On the Linux side, I don't see this as a big deal.  RedHat and Fedora currently ship with deny_execmem turned off by default and people who want to turn it on can either not run programs which do JIT or they can write a local policy to allow it just for the program(s) they want to whitelist.
12:09 * jnthn back
12:10 markmont On the NetBSD side, mprotect() will fail unless the moar binary gets installed with a special flag that permits it.  People on NetBSD systems should know this command and running it should be no big deal.
12:10 lizmat jnthn: will it continue to make sense to bind attributes to lexicals for hot loops (like a push-all) ?
12:11 jnthn lizmat: Yes
12:11 timotimo if you have an information disclosure vulnerability, which seems somewhat common as far as exploits go i suppose, then you can find the writable chunk that goes with an executable chunk very easily
12:11 lizmat jnthn: check
12:11 jnthn *sigh* We've really piled up the workarounds in spesh in the past, for lack of time to do things righter
12:14 jnthn What started out as an inline bug turned out to be in part 'cus we didn't really delete basic blocks from an inline that we then decided was unreachable anyway (workaround 1). Fixing that revealed that we weren't so aggressive in deleting basic blocks (workaround 2). Fixing that reveals that we don't delete blocks that have exception handler annotations in them (workaround 3)
12:16 lizmat the very definition of technical debt, I guess  :-(
12:16 jnthn Yeah
12:17 jnthn It's good to clear these things up and we'll get better/smaller code as a result
12:18 jnthn Just means I'll end up spending a good chunk of the day on this instead of the invocation opts I expected
12:19 lizmat sometimes you have to take a step back before can take 3 steps forward  :-)
12:19 jnthn True
12:19 jnthn And the problems are tricky
12:51 [Coke] jnthn++
13:22 Geth ¦ MoarVM/spesh-invoke: 34b6e590be | (Jonathan Worthington)++ | src/spesh/codegen.c
13:22 Geth ¦ MoarVM/spesh-invoke: Validate range of branch target in spesh code-gen.
13:22 Geth ¦ MoarVM/spesh-invoke: review: https://github.com/MoarVM/MoarVM/commit/34b6e590be
13:22 Geth ¦ MoarVM/spesh-invoke: 4c919f0905 | (Jonathan Worthington)++ | 3 files
13:22 Geth ¦ MoarVM/spesh-invoke: Anchor inlinee handlers to inliner entry.
13:22 Geth ¦ MoarVM/spesh-invoke:
13:22 Geth ¦ MoarVM/spesh-invoke: So that they will be considered reachable and not eliminated once we
13:22 Geth ¦ MoarVM/spesh-invoke: start to apply more aggressive dead basic block removal.
13:22 Geth ¦ MoarVM/spesh-invoke: review: https://github.com/MoarVM/MoarVM/commit/4c919f0905
13:23 jnthn Those are two in theory independent parts of a larger local set of patches, fwiw.
13:35 jnthn Phew, think I'm about there
13:41 AlexDaniel joined #moarvm
13:44 Geth ¦ MoarVM/spesh-invoke: 309da4c8dd | (Jonathan Worthington)++ | 6 files
13:44 Geth ¦ MoarVM/spesh-invoke: Fix and expand dead basic block removal.
13:44 Geth ¦ MoarVM/spesh-invoke:
13:44 Geth ¦ MoarVM/spesh-invoke: The specialization graph builder does dead basic block removal before
13:44 Geth ¦ MoarVM/spesh-invoke: it computes the dominance tree, to avoid problems in the calculation.
13:44 Geth ¦ MoarVM/spesh-invoke: In some cases, however, it was finding and removing basic blocks in
13:44 Geth ¦ MoarVM/spesh-invoke: spesh graphs formed from previously optimized code. This was because
13:44 Geth ¦ MoarVM/spesh-invoke: the basic block removal was deliberately incomplete, to work around a
13:44 Geth ¦ MoarVM/spesh-invoke: <…commit message has 8 more lines…>
13:44 Geth ¦ MoarVM/spesh-invoke: review: https://github.com/MoarVM/MoarVM/commit/309da4c8dd
13:51 Geth ¦ MoarVM/master: 9 commits pushed by (Jonathan Worthington)++
13:51 Geth ¦ MoarVM/master: c5a2cae21b | Use callsite info to insert guards before prepargs
13:51 Geth ¦ MoarVM/master: e97010cd6d | Make sure to mark decont for guard used.
13:51 Geth ¦ MoarVM/master: a33c6b516a | Guards need STable, not type object.
13:51 Geth ¦ MoarVM/master: 1f1f27e8a6 | Use type tuple to find spesh candidate if present.
13:51 Geth ¦ MoarVM/master: 388a769b46 | Enable type tuple use in multi-dispatch.
13:51 Geth ¦ MoarVM/master: 079dc9e962 | Avoid walking of end of BBs linked list.
13:51 Geth ¦ MoarVM/master: 34b6e590be | Validate range of branch target in spesh code-gen.
13:51 Geth ¦ MoarVM/master: 4c919f0905 | Anchor inlinee handlers to inliner entry.
13:51 Geth ¦ MoarVM/master: 309da4c8dd | Fix and expand dead basic block removal.
13:51 Geth ¦ MoarVM/master: review: https://github.com/MoarVM/MoarVM/compare/3f33a8419b...309da4c8dd
13:51 jnthn There we go, merged.
13:52 jnthn The main thing we're missing now is the whole rw container checking/guarding that multi-dispatch also wants, which is why I think we miss out on some of them still
13:52 jnthn Also there's a few other fixes needed to the frame simulation done wiht logs
13:59 lizmat time to bump NQP and Rakudo ?
14:00 jnthn Mebbe, though I've got more improvements coming
14:00 jnthn Those were in a branch because for a while they made things worse rather than better. :)
14:02 lizmat but sooner the better wrt to testing, no
14:02 lizmat or would that create too much noise atm ?
14:03 Geth ¦ MoarVM: 67f92011c0 | (Jonathan Worthington)++ | src/core/args.c
14:03 Geth ¦ MoarVM: Toss duplicate use of named arg checking.
14:03 Geth ¦ MoarVM:
14:03 Geth ¦ MoarVM: There's no way to hit this from either NQP or Perl 6, it's totally
14:03 Geth ¦ MoarVM: harmless if it happens anyway, but having to support it gets in the
14:03 Geth ¦ MoarVM: way of a simpler, cheaper, handling of named arg flags in the event of
14:03 Geth ¦ MoarVM: a deopt, which will let us remove something that frustrates inlining.
14:03 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/67f92011c0
14:04 jnthn lizmat: Can bump a bit later on today :)
14:04 jnthn Well, can now if you want, just may as well get evening/overnight testing of all my changes :)
14:05 lizmat I'll wait
14:05 lizmat it's not like I don't have other things on my mind atm  :-(
14:06 jnthn Impending TPC?
14:07 lizmat yeah  :-(
14:51 brrt joined #moarvm
15:06 brrt joined #moarvm
15:16 brrt jnthn; is sp_decont actually invokish, as far as you know?
15:18 jnthn brrt: Yeah, it's just decont without logging
15:18 jnthn So they'd JIT exactly the same
15:22 jnthn Currently working on argument spesh fwiw
15:22 jnthn Trying to elimiante the need to track what args are used at all in a specialized frame
15:23 jnthn Which has taken me to specializing named slurpy args into the code to build the hash
15:23 brrt okay (a noninvokish variant would be cool, but probably too much to ask :-))
15:23 jnthn "It depends" :)
15:24 jnthn Ideally we can spesh noninvokish deconts we know enough about into something better than decont :)
15:26 brrt aye
15:26 brrt ideally, we can specialize them in the JIT further, for those were it isn't practical to do in the interpreter :-)
15:32 timotimo i had started a tiny bit on devirtualizing decont for the guard tree
15:33 timotimo but it turns out we have to follow two pointers in a row for rakudo scalars
15:33 timotimo wasn't sure if it'd make sense making all tree nodes bigger to support that, and also wasn't sure how to store this in a sufficiently generic way into the container spec or repr or what
15:37 AlexDaniel joined #moarvm
15:46 jnthn yay, got the slurpy named hash thing working
15:47 jnthn On the callee side it now just turns into ops to build the hash
16:29 robertle_ joined #moarvm
16:29 timotimo cool, so a nqp::create and a few bindkeys with static strings for keys
16:33 Geth ¦ MoarVM: baaf8b6dfe | (Jonathan Worthington)++ | 16 files
16:33 Geth ¦ MoarVM: More sophisticated named arg handling in spesh.
16:33 Geth ¦ MoarVM:
16:33 Geth ¦ MoarVM: Previously, we would keep track of them using the bit field used by
16:33 Geth ¦ MoarVM: unspecialized named argument handling. Since in an inline the use of
16:33 Geth ¦ MoarVM: that could conflict, the op to do the tracking was `:noinline`. Thus
16:33 Geth ¦ MoarVM: an attempt was made to not include them if we had no slurpy args and
16:33 Geth ¦ MoarVM: no deopt points. However, recent spesh changes have introduced more
16:33 Geth ¦ MoarVM: <…commit message has 10 more lines…>
16:33 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/baaf8b6dfe
16:35 lizmat further lowering++  :-)
16:37 jnthn No regressions from that, or so my stressing of builds and spectest tell me
16:37 Geth ¦ MoarVM: 0df98ee0e7 | (Jonathan Worthington)++ | 8 files
16:37 Geth ¦ MoarVM: Remove unused sp_namedarg_used op.
16:37 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/0df98ee0e7
16:40 jnthn And yes, it does nail the inline that I mentioned in the commit message now :)
16:45 jnthn Alright, enough for now I think
17:13 timotimo fantastic
17:31 zakharyas joined #moarvm
20:10 vendethiel joined #moarvm
20:17 AlexDaniel joined #moarvm
21:05 brrt joined #moarvm
21:07 AlexDaniel joined #moarvm
21:08 AlexDaniel joined #moarvm
21:19 AlexDani` joined #moarvm
22:35 Geth ¦ MoarVM: af52fca88b | MasterDuke17++ | src/jit/graph.c
22:35 Geth ¦ MoarVM: JIT pow_I
22:35 Geth ¦ MoarVM:
22:35 Geth ¦ MoarVM: It's slightly faster to call this from the JIT.
22:35 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/af52fca88b
22:35 Geth ¦ MoarVM: f1bfca0317 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/jit/graph.c
22:35 Geth ¦ MoarVM: Merge pull request #623 from MasterDuke17/jit_pow_I
22:35 Geth ¦ MoarVM:
22:35 Geth ¦ MoarVM: JIT pow_I
22:35 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/f1bfca0317
22:37 MasterDuke jnthn: did you see my comments last night about trying to track down the --profile stuff? https://irclog.perlgeek.de/moarvm/2017-08-02#i_14960192 and a little after
22:38 MasterDuke i'm not sure that i found anything helpful though
22:39 jnthn Yeah, I fear the problem is happening "further back" though and that's just the upshot of it
22:40 MasterDuke that's kind of what i suspected, but lost the energy last night to keep going farther back
23:01 markmont joined #moarvm
23:02 Geth ¦ MoarVM: c8eb5c9ce7 | (Jonathan Worthington)++ | src/io/asyncsocket.c
23:02 Geth ¦ MoarVM: Make it possible to cancel an async socket reader.
23:02 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/c8eb5c9ce7

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary