Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-08-27

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 dogbert11 the odd thing is that I don't think the SEGV shows up if the NFG check is off
00:03 samcv ok i think i may have fixed it. well. i think you might be right they are separate
00:04 samcv well. or maybe not completely. anyway there was issues
00:04 dogbert11 two different problems?
00:05 samcv well maybe differet problems. that just got exposed by the checks
00:06 dogbert11 aha, and you've fixed one?
00:07 samcv well i fixed that NULL problem in re_nfg that was one. and the NULL in the NFG_debug and uh
00:07 samcv so i am fixing things hah
00:08 dogbert11 and the KMP fix as a bonus :)
00:08 samcv and that
00:08 samcv you should try that sometime the clang option i said
00:08 samcv super cool.
00:08 samcv not as slow as valgrind
00:08 samcv though it does kill the program when it finds a problem. but it gives a lot of info
00:09 samcv and there's a problem that the result of concatenation was getting freed during concatenation
00:09 samcv and i'm gonna push that shortly
00:09 dogbert11 cool
00:11 Geth ¦ MoarVM: 3ca3020705 | (Samantha McVey)++ | src/strings/ops.c
00:11 Geth ¦ MoarVM: Fix GC problem during concatenation, MVMROOT `result`
00:11 Geth ¦ MoarVM:
00:11 Geth ¦ MoarVM: Also move the debug checks to inside of the MVMROOT.
00:11 Geth ¦ MoarVM:
00:11 Geth ¦ MoarVM: In MVM_string_replace(), use MVMROOT instead of manually pushing and
00:11 Geth ¦ MoarVM: popping to root.
00:11 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/3ca3020705
00:11 samcv ok now with the clang instrumentation i'm getting the UTF problem :)
00:11 samcv since it aborts and prints out great info if it encounters stuff
00:12 samcv but there are memory problems with utf8
00:13 samcv dogbert11, https://gist.github.com/samcv/8b9288c9b7189a63fa03fbbeb79d7e9c
00:14 samcv at the bottom
00:23 dogbert11 the spectest errors seems to have disappeared with your latest fix :)
00:25 Geth ¦ MoarVM: a22f98dbac | (Samantha McVey)++ | src/strings/utf8.c
00:25 Geth ¦ MoarVM: MVM_string_utf8_decodestream: free the buffer on malformed UTF-8
00:25 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/a22f98dbac
00:25 samcv dogbert11, found three missing MVM_free's in MVM_string_utf8_decodestream
00:26 samcv and i must go now, be back in like 5 or 6 hours
00:26 dogbert11 nice, will be asleep when you get back
00:27 samcv dogbert11, did you try compiling with clang and -fsanitize=address in cflags and ldflags?
00:27 samcv if not try it :)
00:28 samcv they also have a data races detector and many others
00:28 samcv see here for info https://clang.llvm.org/docs/UsersManual.html#controlling-code-generation
00:32 dogbert11 will check it out, don't even think I have clang installed
00:33 dogbert11 Error, MoarVM cannot get Unicode codepoint property for synthetic codepoint -11
00:33 dogbert11 in block <unit> at t/spec/S15-nfg/concatenation.t line 38
00:34 samcv i see that too
00:35 samcv wow
00:35 samcv ok MVMROOT of the result caused it
00:35 samcv but why
00:35 samcv that's... odd
00:35 samcv hm
00:35 samcv well feel free to revert that
00:36 samcv if moarvm needs to get bumped or something. g2g. will check it out when i get back though
00:36 * dogbert11 drops to bed since it's 0230 AM
01:05 Geth ¦ MoarVM: 4eaba7a04b | (Samantha McVey)++ | src/strings/ops.c
01:05 Geth ¦ MoarVM: Revert "Fix GC problem during concatenation, MVMROOT `result`"
01:05 Geth ¦ MoarVM:
01:05 Geth ¦ MoarVM: This reverts commit 3ca30207055d7f8ca28071ef20ffcfb88cd82241.
01:05 Geth ¦ MoarVM:
01:05 Geth ¦ MoarVM: Caused some failing tests in roast.
01:05 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/4eaba7a04b
01:15 Geth ¦ MoarVM: cd41322ec3 | (Samantha McVey)++ | src/strings/ops.c
01:15 Geth ¦ MoarVM: Cautiously initialize more memory pointers as NULL
01:15 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/cd41322ec3
01:22 greppable6 joined #moarvm
01:22 releasable6 joined #moarvm
01:22 quotable6 joined #moarvm
01:22 bisectable6 joined #moarvm
01:22 committable6 joined #moarvm
01:22 coverable6 joined #moarvm
01:22 bloatable6 joined #moarvm
01:22 evalable6 joined #moarvm
01:22 benchable6 joined #moarvm
01:22 unicodable6 joined #moarvm
01:22 statisfiable6 joined #moarvm
01:22 nativecallable6 joined #moarvm
01:52 ilbot3 joined #moarvm
01:52 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
02:24 ilbot3 joined #moarvm
02:24 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
08:32 lizmat_ joined #moarvm
08:42 samcv .tell dogbert17 check out how the address sanitizer for clang works: https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm very interesting how it works. even though this is how it works, it runs way faster than valgrind probably since doesn't have to run inside and have that overhead
08:42 yoleaux samcv: I'll pass your message to dogbert17.
08:46 timotimo samcv: valgrind has the added benefit that you don't have to recompile. which is especially good if you don't have access to the source code of parts or the whole program
08:46 samcv yeah. though the clang one detected problems valgrind never did
08:46 samcv and detected problems accessing stuff on the stack as well
08:47 samcv but yeah. you don't always have the ability to want to quickly recompile
08:48 samcv this is their comparison https://github.com/google/sanitizers/wiki/AddressSanitizerComparisonOfMemoryTools
08:49 samcv optionally you can have t do stack-use-after-return. that's neat
08:49 samcv in case you return an object on the others stack and then you shouldn't access it
09:14 dogbert11 joined #moarvm
09:22 dogbert11 samcv, the revert made the number of failing spectests go up significantly
09:31 timotimo time to move this work off to a branch so we can flatten eventual reverts and such away when we merge?
09:31 timotimo oh
09:31 timotimo it only crashes on travis master m(
10:58 timotimo things we should have: dup and maybe dup2 for IO::Handle. also, creating IO::Handle.from-fileno or .from-native-descriptor
12:07 dogbert11 joined #moarvm
12:16 nine I can now JIT native calls with one uncontainerized int64 or CPointer argument :)
12:20 timotimo nice
12:21 timotimo that's already pretty useful
12:21 timotimo when you get two, that's a big step into useful territory
12:22 nine I think deconting is more important even
12:22 timotimo true
12:22 nine At least if I ever want to merge this thing ;)
12:23 nine I'm just not sure in which layer exactly to do that.
12:24 timotimo if we emit the decont in spesh we can turn it into simpler operations in some cases
12:24 timotimo like known type and such
12:26 nine Yeah, I guess it's best (and probably easiest, too) to decont in the generated routine body when binding params to locals. Right now I access the args buffer of the callsite directly.
12:26 nine So a bit of work to change that, but probably not really difficult.
12:42 Geth ¦ MoarVM/jit_nativecall: ac4d23f9f3 | (Stefan Seifert)++ | 3 files
12:42 Geth ¦ MoarVM/jit_nativecall: JIT native calls with an unconted int64 or CPointer argument
12:42 Geth ¦ MoarVM/jit_nativecall:
12:42 Geth ¦ MoarVM/jit_nativecall: Doesn't handle containers yet as it neither checks nor deconts.
12:42 Geth ¦ MoarVM/jit_nativecall: review: https://github.com/MoarVM/MoarVM/commit/ac4d23f9f3
12:46 AlexDaniel joined #moarvm
12:47 committable6 joined #moarvm
12:47 benchable6 joined #moarvm
12:47 evalable6 joined #moarvm
12:47 quotable6 joined #moarvm
12:47 unicodable6 joined #moarvm
12:47 bloatable6 joined #moarvm
12:47 bisectable6 joined #moarvm
12:47 coverable6 joined #moarvm
12:47 greppable6 joined #moarvm
12:47 releasable6 joined #moarvm
12:47 statisfiable6 joined #moarvm
13:03 lizmat joined #moarvm
13:52 greppable6 joined #moarvm
14:08 nine So what's the correct way to get a deconted param into a local?
14:09 * lizmat has no idea
14:10 timotimo regular param op and a decont after that i'd say
14:12 nine Like this? QAST::Var.new( :name($lowered_name), :scope<local>, :decl<param>), QAST::Op.new( :op<bind>, QAST::Var.new(:scope<local>, :name($lowered_name)), QAST::Op.new(:op<decont>, QAST::Var.new(:scope<local>, :name($lowered_name))),)
14:18 timotimo i think so
14:19 nine Yes, looks good :) Now I only hav eto change the JITed code to access locals instead of params
14:19 timotimo cool
14:20 nine Would be easier though if I weren't as tired
14:22 samcv dogbert11, are you sure they went up?
14:23 samcv i just woke up i can run again but i remember running them again
14:43 dogbert11 hello samcv :)
14:44 dogbert11 ok, I'll check out master and try again ...
14:48 dogbert11 so I'm at the S04 tests now and there's been complaints from 20 files already
14:48 dogbert11 ofc they could all be errors uncovered by the NFG checker
14:49 samcv i have that off right now
14:49 samcv so it must be the NFG checker
14:49 samcv i think it's probably good if we set all pointers to start as NULL. and the compiler will always optimize it out or otherwise the performance difference is basically none
14:50 dogbert11 sounds like a plan, btw have you had your tea yet?
14:50 samcv though it should get compiled out if there's no way for it to be used, if it's okay i will start commiting more pointers to be initialized as NULL
14:50 samcv no!
14:50 dogbert11 uh oh
14:50 samcv yep
14:52 dogbert11 FWIW I rambled a bit about the Malformed UTF8 problem over in the dev channel
14:52 samcv i get full spectest pass
14:52 dogbert11 without the checker so do I
14:53 timotimo i know how to make this faster! i'll build a binary format!
14:53 timotimo -- 10 hours later --
14:56 samcv dogbert11, but you say that with my reversion more tests fail (with the NFG checker) than before?
14:58 nine /win/win 10
14:58 samcv for some reason if i ./Configure.pl --asan then i can't run it. does it include its own version of asan?
14:59 samcv cause if i put -fsanitize=address manually it works fine
15:02 samcv well i'm going to try running spectest with asan and nfg check on and see if it catches anything
15:03 dogbert11 let me try without the revert
15:03 * dogbert11 rebuilds
15:05 dogbert11 if I run asan with gcc I have to use that option
15:05 samcv yeah. probably since we must include the library. but clang has its own never version and otherwise i get a mismatch
15:06 samcv ugh why do i get failing tests with `make spectest` but not running them by themselves
15:06 dogbert11 am at S06 now, only one fail so far ...
15:06 nine timotimo: looks to me like spesh will quite happily throw my deconts away as it doesn't know that the nativecallinvokejit op will access the locals. Any idea how to avoid that?
15:07 timotimo oh
15:07 timotimo yeah that's a problem
15:07 dogbert11 samcv: the synthetics errors we saw yesterday, could those have been legit?
15:08 samcv uh corruption
15:08 samcv idk if that's legit or nonlegit?
15:08 samcv but probably something being freed when it shouldn't
15:09 dogbert11 only five fails now
15:09 timotimo nine: short of inventing an op that does nothing except touch a variable ... :\
15:09 timotimo i mean, we could put the args into the nativecallinvoke op's signature, but we'd have to have a whole bunch of these ops
15:09 samcv oh well i guess maybe it could be legit idk hard to tell. but really. it seems odd...
15:10 samcv i'm getting lots of failures but some of them i would run the test manually and pass fine
15:10 dogbert11 samcv, now t/spec/S15-nfg/concatenation.t fails with error, 'MoarVM cannot get Unicode codepoint property for synthetic codepoint -11'
15:10 samcv yep
15:11 dogbert11 do you think that's due to corruption or could it in fact be legit?
15:12 samcv oh it could be legit idk... it was super confusing to me yesterday
15:12 dogbert11 let's take another look, sec
15:12 samcv yeah i made several changes in that one commit. i'm going to make one at a time
15:13 nine timotimo: or teach spesh about nativecallinvokejit touching locals?
15:14 timotimo okay, but which does it touch?
15:14 timotimo you can teach facts.c about that op
15:14 timotimo but you need to find the right register and version for this to work
15:15 nine timotimo: well version 2 of all locals in that frame
15:15 timotimo if we're willing to rely on this ...
15:16 dogbert11 samcv: here's gdb output https://gist.github.com/dogbert17/b6a5ed3456e4301a67d87cef9868f116
15:17 timotimo so ... you know how i made that binary format to make moar-heapanalyzer faster?
15:17 timotimo it's slower by a factor of like 6
15:17 nine For the first implementation of this I'd say yes.
15:19 timotimo ok, so in facts.c you can, when you stumble upon a nativecallinvokejit, bump up the usages of all locals of version 2, i.e. for loop from 0 to num-locals from somewhere in the spesh graph
15:19 dogbert11 this is the failing test: is-deeply ('a' x 2 ~ "\c[COMBINING ACUTE ACCENT]" x 3).ords, (97, 225, 769, 769), "Concat works with combining repeated characters";
15:20 nine timotimo: yep, that's pretty clear. But not where the facts for the locals are actually
15:21 nine There's num_locals in MVMSpeshGraph, but I can't find any list representing locals
15:21 timotimo you go through the fact functions
15:21 timotimo oh, you pass them a MVMSpeshOperand
15:21 timotimo no, actually, i think you pass register and version separately
15:22 timotimo so no problems at all there
15:22 samcv dogbert11, that fails without it reverted?
15:23 samcv yeah i'm getting that too
15:23 dogbert11 let me check that
15:24 dogbert11 with the revert in place it does not fail
15:26 timotimo 4:46 to get a summary of the snapshot in my new version; 42s in the old version (with compatibility changes that might make it slower)
15:27 nine timotimo: like case MVM_OP_nativecallinvokejit: for (i = 0; i < g->num_locals; i++) { g->facts[i][2]->usages++ } break;?
15:30 timotimo yeah, try that
15:31 nine or rather [i][1] as I guess we start counting at 0? :)
15:32 timotimo could very well be, or maybe we only output it like that for human readability?
15:33 nine No change so far...
15:34 nine LOL setting MVM_SPESH_LOG "fixes" the issue
15:34 timotimo somehow the C recompile got messed up perhaps?
15:35 AlexDaniel joined #moarvm
15:35 nine No...for some reason it just starts to spesh later and thus never. Upping the eiiteration count helps
15:38 timotimo :buffer on IO::Handle won't work for :bin?
15:38 timotimo i.e. it's still doing the exact reads as syscalls that i'm calling from my hll code
15:41 nine According to the spesh log, the usages are still all 0
15:47 samcv ok i found the problem dogbert11
15:47 dogbert11 wee
15:47 dogbert11 what was it?
15:48 samcv gonna run spectest. but
15:49 samcv it was running NFG_CHECK_CONCAT when it shouldn't have. i.e. when it was going to return re_nfg anyway because the concatted string wouldn't have been correct
15:49 samcv though still dunno why it would say something about a negative grapheme but. that was the cause
15:49 samcv gonna run spectest to make sure
15:49 dogbert11 I'll wait :)
15:51 samcv ugh i wish asan wouldn't cause spectest's to fail for no reason then i run them again and it's fine. they show that they didn't run any tests, so i'm not sure what happened. how can i run tests but show full stderr?
15:53 dogbert11 I would like to know that as well
15:54 samcv ugh getting errors
15:54 samcv well from asan
15:54 samcv on make m-spectest
15:54 samcv well starting it
15:55 samcv yeah this is the same issue i had before and why i added `result` to the MVMROOT. let me add that again
15:56 samcv ok cool. fixed it now by adding that :)
15:58 nine timotimo: ha! ase MVM_OP_nativecallinvokejit: for (i = 0; i < g->num_locals; i++) g->facts[i][g->fact_counts[i] - 1].usages++; break;
15:59 samcv dogbert11, this is my fix though https://github.com/MoarVM/MoarVM/commit/08e7874bdd59f92220565d44eaa3a697db7a9de5
16:00 timotimo CurrentThreadScheduler is a godsend
16:00 timotimo why are you going to the facts array directly?
16:00 timotimo oh you were doing that before
16:00 nine timotimo: I dare say this is actually even pretty safe. Worst case it prevents some instruction elimination.
16:00 timotimo why didn't i yell at you
16:01 nine timotimo: tweak_block_handler_usage does it, too
16:01 timotimo hm.
16:01 timotimo naughty naughty
16:03 nine So containers are now handled which gives us full access to CPointer arguments. Unboxing however isn't yet and the deconting code seems to have broken even the workaround (passing a native int)
16:08 samcv ugh --verbosity doesn't seem to work? set it to 10 and get only progress bars. ]set to 1 and i get only progress bar
16:08 timotimo 2:46m now instead of the 442s
16:08 samcv set to 0 and it is the same as it is normally
16:09 dogbert11 perhaps it's not fully implemented
16:10 samcv Zoffix, any help?
16:10 samcv zoffix always knows secrets with roast
16:11 samcv idk if it's skipping these tests because they are taking a long time to run? i get "no subtests run", if i run manually it takes a while for it to start outputting stuff at all. so maybe it's just killing it when it doesn't get output for a period?
16:11 samcv (harness5 here)
16:13 dogbert11 are you running with asan?
16:13 dogbert11 also, harness6 is broken so we have no alternatives atm :)
16:13 samcv yeah with asan
16:14 timotimo someone want to implement buffered binary reading? :)
16:14 dogbert11 I sometimes forget that mem requirements double when usin that switch
16:14 dogbert11 *using
16:14 timotimo maybe i should ask in #perl6, as this is a hll task
16:14 samcv i think it's slower too cause i have it set to `detect_stack_use_after_return=1 ` as well
16:14 samcv i think that slows it down
16:14 dogbert11 no swapping?
16:35 samcv how do i fix "Unhandled exception: Missing or wrong version of dependency "
16:35 samcv i mean when running my system's perl6 inside the rakudo git folder
16:36 samcv i want it not to fail like that
16:36 samcv is there some env var?
16:36 samcv also i hate this error :P
16:49 lizmat joined #moarvm
16:55 dogbert11 samcv: have you tried rebuilding rakudo?
16:56 dogbert11 sry, didn't read properly
16:56 Zoffix samcv: help with what?
16:56 samcv oh. getting make spectest to be verbose
16:57 samcv i want to get all stderr messages to my terminal
16:57 samcv i haven't been able to do it by setting --verbosity but i may be doing it wrong or setting it to the wrong number
16:58 Zoffix samcv: no idea. If you use t/fudgeandrun file1 file2 it'll be verbose. But I don't know an easy way to run all tests verbosely
16:58 nine Ok, integer arguments are just not there yet due to missing unboxing support, but I do support an arbitrary number of pointer arguments now :)
16:59 Zoffix samcv: did you try --verbosity=1 ?
16:59 samcv i think i did. will try again though
16:59 Zoffix samcv: looking at available values here: https://metacpan.org/pod/TAP::Harness#Class-Methods
17:00 timotimo yay
17:00 timotimo new heapanalyzer down to taking "only" 1:16m vs old 38s, but the memory usage is down to 450mb from 1.8gb
17:01 Zoffix \o/
17:05 Zoffix GitHub-- rendering POD instead of showing the script: https://github.com/rakudo/rakudo/blob/nom/t/harness5
17:05 Zoffix samcv: ^ if you edit that file and change `merge => 1` to `merge => 0` and run the following, then you get STDERR on terminal: /home/zoffix/perl5/perlbrew/perls/perl-5.26.0/bin/perl t/harness5 --verbosity=1 --fudge --moar --keep-exit-code --tests-from-file=t/spectest.data
17:05 AlexDaniel xD
17:06 samcv oh nice!
17:06 Zoffix No idea how to make it just normal verbose output with all the things
17:06 * Zoffix goes back lurking
17:06 timotimo whoops, double-freeing all the stuff
17:07 timotimo turns out just nativecasting to Pointer will not mark the Pointer for "will not be freed"
17:08 timotimo how do i even nativecast
17:25 samcv hmm it seems when re_nfg is ran, if the string is utf8-c8, it comes out with a different number of graphemes afterward
17:26 timotimo does it misunderstand the special graphemes we make for our c8 thingies?
17:26 samcv idk i assumed it would
17:27 timotimo me too
17:27 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); $u.=subst("a","b"); say $u
17:27 camelia rakudo-moar 95a70c: OUTPUT: «g????xF8o????xD9????xD2b?»
17:27 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); $u.=subst("a","b"); say $u.chars
17:27 camelia rakudo-moar 95a70c: OUTPUT: «9?»
17:27 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.chars; $u.=subst("a","b"); say $u.chars
17:27 camelia rakudo-moar 95a70c: OUTPUT: «6?9?»
17:27 samcv uh
17:28 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.chars; $u.=subst("a","a"); say $u.chars
17:28 camelia rakudo-moar 95a70c: OUTPUT: «6?9?»
17:28 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.codes; $u.=subst("a","a"); say $u.codes
17:28 camelia rakudo-moar 95a70c: OUTPUT: «15?15?»
17:28 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.ords; $u.=subst("a","a"); say $u.ords
17:28 camelia rakudo-moar 95a70c: OUTPUT: «(103 1114109 120 70 56 111 1114109 120 68 57 1114109 120 68 50 97)?(103 1114109 120 70 56 111 1114109 120 68 57 1114109 120 68 50 97)?»
17:28 timotimo what's with these ords
17:28 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.comb; $u.=subst("a","a"); say $u.comb
17:28 camelia rakudo-moar 95a70c: OUTPUT: «(g ????xF8 o ????xD9 ????xD2 a)?(g ????xF8 o ????xD9 ???? x D 2 a)?»
17:28 samcv well it's utf8-c8
17:28 timotimo but still
17:29 samcv m: my @ints = 103, 248, 111, 217, 210, 97; my $b = Buf.new(@ints); my Str $u=$b.decode("utf8-c8"); say $u.comb.join(', '); $u.=subst("a","a"); say $u.comb.join(', ')
17:29 camelia rakudo-moar 95a70c: OUTPUT: «g, ????xF8, o, ????xD9, ????xD2, a?g, ????xF8, o, ????xD9, ????, x, D, 2, a?»
17:29 samcv so yeah it's losing the utf8-c8 graphemes
17:30 samcv poop
17:30 samcv timotimo, maybe if it loses the graphemes, it can still write as utf-c8 properly?
17:30 benchable6 joined #moarvm
17:30 samcv but still you colud then replace the letters that make it up like replace all D's with something and ruin the utf8-c8
17:31 samcv ah it iterates by codepoint
17:32 samcv poop poop
17:40 samcv well i opened a moarvm issue
17:52 timotimo heapanalyzer rewrite at 1:10 vs 42s
18:04 timotimo 1 minute!
18:11 timotimo 35 seconds!
18:14 timotimo 25 seconds!
18:14 timotimo (but i have a bunch of nqp ops in there)
18:23 Ven`` joined #moarvm
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: f28600e499 | (Timo Paulssen)++ | src/profiler/heapsnapshot.c
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: introduce a binary format for heapsnapshot
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format:
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: it starts with a version identifier to differentiate between
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: the old format and this one. all records except the string heap
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: are fixed sizes and the file ends in a table outlining the
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: sizes of the individual segments.
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format:
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: each segment starts in a little identifier to make sure the
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: snapshot parser doesn't read bogus data, and before a set
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: of entries there's an entry of the entry size.
18:28 Geth ¦ MoarVM/heapsnapshot_binary_format: review: https://github.com/MoarVM/MoarVM/commit/f28600e499
18:28 timotimo this was my idea of fun for the day ...
18:29 timotimo i should have probably written into the file that snapshots end up a noticable bit bigger
18:29 timotimo i'll amend
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: 7960ecc26e | (Timo Paulssen)++ | src/profiler/heapsnapshot.c
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: introduce a binary format for heapsnapshot
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format:
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: it starts with a version identifier to differentiate between
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: the old format and this one. all records except the string heap
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: are fixed sizes and the file ends in a table outlining the
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: sizes of the individual segments.
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format:
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: <…commit message has 7 more lines…>
18:29 Geth ¦ MoarVM/heapsnapshot_binary_format: review: https://github.com/MoarVM/MoarVM/commit/7960ecc26e
19:07 timotimo not sure fi i can get it faster than the 20 seconds i have now
19:08 timotimo maybe if i split the collectables in half as well, not only the references
19:08 timotimo this code will want a bit of cleanup for clarity when i'm done making it fast
19:21 timotimo 17 seconds!
19:33 timotimo pushed all changes to the heapanalyzer
19:38 AlexDaniel joined #moarvm
19:42 timotimo it looks like the amount of ram claim is wrong, or at least i threw out the benefit by parallelizing harder
19:54 samcv so this lets us dump the memory heaps or what
20:00 timotimo it's a step in the right direction
20:00 samcv or it analyzes them? give me a rundown?
20:00 samcv nice
20:01 timotimo it's two parts
20:01 timotimo one is the new format that moar can spit out
20:01 timotimo it can get by completely without MVM objects and hll code
20:01 timotimo the other part is support for this format in the heap analyzer
20:01 timotimo it's around 2x faster, but uses more memory
20:01 timotimo that ought to let us work with stuff a bit faster
20:02 timotimo one other thing that would be nice would be to maybe parallelize the breadth-first search
20:03 timotimo does that make sense?
20:04 timotimo the heapsnapshot_binary_format branch will just unconditionally write to /tmp/heapsnapshot_new_format whenever a heapsnapshot is finished
20:05 timotimo AlexDaniel: was it you who was interested in faster heap snapshot loading?
20:06 timotimo i know it was a whateverable developer :D
20:11 samcv nice
20:11 timotimo next on the feature request list is being able to get all-paths to nodes
20:12 samcv looks like we have a heap-buffer-overflow
20:13 samcv addresssanitizer is angry
20:13 samcv 244 bytes to the left of the closest area
20:14 samcv looks like is_stack_frame() is checking a stack frame's struct value, but that's actually not inside any mapped memory
20:14 samcv timotimo, MVM_gc_allocate_nursery() is that run on start and not like, afterward?
20:14 samcv cause that's a calling function in the chain
20:17 timotimo now i'm wondering if i should always generate an allpaths or only sometimes
20:18 timotimo the "all paths" for type objects can get hilariously big
20:22 timotimo i could either run a bfs and note down all predecessors and just stop adding more after reaching 10 or something, or i do a new DFS each time an all-paths is asked for
20:25 timotimo i'll try DFS in a gather/take and let the user stop it whenever they want to
21:19 samcv wow so far the address saniiser hasn't puked when running roast. yay
21:20 timotimo cool
21:20 timotimo i'm not being very good at the whole "build a dfs" thing
21:22 samcv so this is in perl6 then, not C?
21:22 timotimo it is
21:23 samcv it hung somewhere but it ran 228 tests and all pass
21:28 timotimo okay the dfs for a single object is currently 83k steps away from the root and is still going
21:29 timotimo it's nice that i print " " x $length-of-that-path before each debug output
21:30 samcv heh
21:31 timotimo oops
21:31 timotimo i was looking at the size of the queue
21:33 timotimo it looks like there's a load of gc pauses every now and then but it shouldn't be creating many objects that live long
21:37 timotimo i'll go to bed i think
21:43 * lizmat too
21:44 lizmat good night,!
21:44 * Zoffix 3
21:44 Zoffix Night
21:54 zakharyas joined #moarvm
21:58 jnthn .
21:58 yoleaux 26 Aug 2017 16:31Z <japhb> jnthn: website typos: First HTTP flip page: 'SITINT', last HTTP flip page: 'requset', first Tooling flip page: 'delcaring'.
21:58 yoleaux 14:35Z <AlexDaniel> jnthn: FWIW, remember I asked how to process null-separate output? Turns out there's an identical problem when trying to process \n-separated output. See https://github.com/perl6/doc/issues/1472 and https://rt.perl.org/Ticket/Display.html?id=131923
22:02 committable6 joined #moarvm
22:12 dogbert11 joined #moarvm
23:22 Geth ¦ MoarVM: samcv++ created pull request #665: Fix GC issues and misc. * Make sure MVMROOT'd pointer is initialized as NULL   * Can cause issues if GC runs while the pointer is uninitialized.
23:22 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/pull/665
23:23 samcv ok yay. if travis says it's good then all issues should be good
23:24 samcv memory issues are the least fun part of programming in C :P
23:25 samcv but now AddrSanitiser is clean of any errors of improper accesses on the stack or allocated space etc
23:25 samcv err roast that is
23:42 Geth ¦ MoarVM/master: 6 commits pushed by (Samantha McVey)++
23:42 Geth ¦ MoarVM/master: 7dac233e0e | Initialize more pointers to null cautiously
23:42 Geth ¦ MoarVM/master: efb125acb0 | Only run NFG_CHECK on concat result that *should* be normalized
23:42 Geth ¦ MoarVM/master: 65a4f83f21 | NFG_check, make sure we don't lookup props of synthetics
23:42 Geth ¦ MoarVM/master: ac9faf2b26 | Revise some comments in MVM_string_is_control_full()
23:42 Geth ¦ MoarVM/master: 24e06f8a28 | Make sure MVMROOT'd pointer is initialized as NULL
23:42 Geth ¦ MoarVM/master: 45af3c3df2 | Merge pull request #665 from samcv/concat-nfg-check-fix
23:42 Geth ¦ MoarVM/master: review: https://github.com/MoarVM/MoarVM/compare/cd41322ec3...45af3c3df2
23:44 samcv jnthn, i've been thinking of how we can fix codepoint iterators when renormalizing and retain the UTF8-c8 graphemes. not sure if we should change the codepoint iterator to either have an option you can set to allow it to hand back synthetics (but only utf8c8 synthetics) so they pass them along. and most of our things will assume they're control because they're less than 20

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary