Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-08-06

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 timotimo i had suspected it in line_coverage.c
00:00 timotimo wonder if the output files would get trashed by multiple threads running code simultaneously
00:03 timotimo i could actually re-use telemetry for this and make coverage logging and telemetry mutually exclusive
00:04 MasterDuke seems reasonable
00:05 timotimo the data delivery mechanism of telemeh only permits a little bit of data for every "packet"
00:07 MasterDuke enough for filepath + linenum?
00:11 timotimo nope
00:12 timotimo the biggest is long long + int + const char *
00:12 timotimo and also the thread id
00:13 MasterDuke hm, create some sort of hash of filepaths and just pass keys/indexes?
00:13 timotimo hmm, are the strings in the annotation perhaps null terminated ...
00:14 MasterDuke afk for a bit
00:27 timotimo hm, if i serialize instrumentation barriers so that i am guaranteed exclusive access to a datastructure to register loc info IDs ...
00:49 timotimo ooh, /me learns about the quota boost
01:01 jnthn sleep &
01:04 MasterDuke i feel like there's some joke to be made about upgrading my Perl 6 interpreter with nitrous
01:11 MasterDuke timotimo: fyi, `perl6 -e 'my $a; for ^100_000 { $a = $_.comb.sort.unique }; say $a'` took 4s normally, 8s with MVM_COVERAGE_LOG=c0.log and MVM_COVERAGE_CONTROL=2
01:13 MasterDuke interestingly, when i misspelled .unique as .uniq, with those env variables set i got `MoarVM oops: Spesh: failed to fix up inline 1` instead of `No such method 'uniq' for invocant of type 'Seq'. Did you mean 'unique'?   in block <unit> at -e line 1`
01:52 ilbot3 joined #moarvm
01:52 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
04:33 channing1 joined #moarvm
04:41 channing1 left #moarvm
06:24 buggable joined #moarvm
07:50 Geth joined #moarvm
08:08 Ven joined #moarvm
08:10 Geth ¦ MoarVM: 074145ce5f | (Jan-Olof Hendig)++ | src/io/asyncsocket.c
08:10 Geth ¦ MoarVM: Remove unused variable
08:10 Geth ¦ MoarVM:
08:10 Geth ¦ MoarVM: No longer used after commit 2b0739d8
08:10 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/074145ce5f
08:10 Geth ¦ MoarVM: 1be02e0363 | (Jan-Olof Hendig)++ | src/core/fixedsizealloc.c
08:10 Geth ¦ MoarVM: Remove another unused variable
08:10 Geth ¦ MoarVM:
08:10 Geth ¦ MoarVM: Possible copy/paste error
08:10 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/1be02e0363
08:10 Geth ¦ MoarVM: 715447df4c | niner++ (committed using GitHub Web editor) | 2 files
08:10 Geth ¦ MoarVM: Merge pull request #628 from dogbert17/remove-unused-vars
08:10 Geth ¦ MoarVM:
08:10 Geth ¦ MoarVM: Remove unused vars
08:10 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/715447df4c
09:30 dogbert11 joined #moarvm
09:58 Ven joined #moarvm
10:08 AlexDaniel joined #moarvm
10:42 Ven_ joined #moarvm
11:46 dogbert11 just tried to build the 'even-moar-jit' branch but the configuration step failed with:
11:46 dogbert11 Generating src/gen/config.h ............................ FAIL
11:46 dogbert11 unknown configuration key 'jit_arch'
11:47 dogbert11 what could be the reson for this?
11:47 dogbert11 is it because I'm on 32-bit?
11:59 Ven joined #moarvm
11:59 MasterDuke dogbert11: hm, maybe Configure.pl around line 403 needs another branch in that if statement
12:31 dogbert11 MasterDuke: I was tricked, it did complain with the message 'JIT isn't supported on platforms with 4 byte pointers.'. However this message is hidden in the config output which leads me to wonder why the config process isn't stopped at that point.
12:32 MasterDuke yeah, that's odd
12:33 dogbert11 it just prints the message and continues only to fail completely a short time later with the 'jit_arch' message
12:35 MasterDuke hm, so we lose 32bit support completely with that branch? or is there a way to tell it to not build the jit?
12:36 dogbert11 I don't believe that the jit has ever worked on 32 bit for some unspecified reason
12:37 jnthn I suspect the configure problem is just an accident rather than anything deep
12:37 MasterDuke yeah, but you can build the current moar without it
12:37 jnthn From the error, it seems like it's trying to find a jit-related bit of configuration and include it in a non-JIT build
12:37 http_GK1wmSU joined #moarvm
12:38 jnthn dogbert11: Reason is that x64 machine code and x86 machine code are quite different things to produce. :)
12:39 dogbert11 ah, simple as that :)
12:39 jnthn And so producing the x86 code as well is a substantial amount of work, though the expr JIT will make it possible to do so with less effort
12:40 MasterDuke fyi, on a semi-related note, i just built the even-moar-jit branch, built nqp, passed nqp's tests, built rakudo, and then passed its test and spectest
12:40 Geth ¦ MoarVM: f3fc2029db | MasterDuke17++ | src/strings/ascii.c
12:40 Geth ¦ MoarVM: Del unused var in MVM_string_ascii_encode_substr
12:40 Geth ¦ MoarVM:
12:40 Geth ¦ MoarVM: The variable was just a cast of an argument and only used once, just
12:40 Geth ¦ MoarVM: cast the argument directly.
12:40 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/f3fc2029db
12:40 Geth ¦ MoarVM: a9c421b1a3 | MasterDuke17++ | src/strings/latin1.c
12:40 Geth ¦ MoarVM: Del unused var in MVM_string_latin1_encode_substr
12:40 Geth ¦ MoarVM:
12:40 Geth ¦ MoarVM: The variable was just a cast of an argument and only used once, just
12:40 Geth ¦ MoarVM: cast the argument directly.
12:40 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/a9c421b1a3
12:40 Geth ¦ MoarVM: cbb1faa1bb | (Jonathan Worthington)++ (committed using GitHub Web editor) | 2 files
12:40 Geth ¦ MoarVM: Merge pull request #619 from MasterDuke17/remove_unnecessary_variable_in_MVM_string__encode_substr
12:40 Geth ¦ MoarVM:
12:40 Geth ¦ MoarVM: Remove unnecessary variable in MVM_string_(ascii|latin1)_encode_substr
12:40 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/cbb1faa1bb
12:40 dogbert11 should the new jit branch lead to any performance increases?
12:40 http_GK1wmSU left #moarvm
12:41 Geth ¦ MoarVM: 123dbdb929 | (David Warring)++ | src/6model/serialization.c
12:41 Geth ¦ MoarVM: attempt at a better error message for RT#126341
12:41 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/123dbdb929
12:41 Geth ¦ MoarVM: 8752f9b0fe | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/6model/serialization.c
12:41 Geth ¦ MoarVM: Merge pull request #617 from dwarring/rt126341
12:41 synopsebot6 Link:  https://rt.perl.org/rt3/Public/Bug/Display.html?id=126341
12:41 Geth ¦ MoarVM:
12:41 Geth ¦ MoarVM: attempt at a better error message for RT#126341
12:41 synopsebot6 Link:  https://rt.perl.org/rt3/Public/Bug/Display.html?id=126341
12:41 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/8752f9b0fe
12:41 MasterDuke my spectest time was roughly in line with before, but i don't have exact numbers to compare
12:42 dogbert11 jnthn++ new blog post
12:43 dogbert11 could it be said that Perl6 have three different optimizers, i.e. the jit, spesh and the --optimize switch in the perl6 executable?
12:51 timotimo hm, i suppose so
12:51 timotimo though the current jit doesn't optimize much
12:51 timotimo the new jit is an optimizer compared to the current jit :P
12:51 timotimo but will get more stuff as time passes
12:52 dogbert11 very cool, I guess I'll have to switch to a 64 bit vm sooner or later
12:52 timotimo but yeah, the --optimize flag in the perl6 executable is completely distinct
12:52 timotimo i'd recommend that :)
12:52 dogbert11 how does that work, i.e. what kind of optimizations does it do?
12:53 timotimo it's written in nqp and operates on the QAST representation of the code
12:53 timotimo you can get a look at what it does by comparing --target=ast and --target=optimize
12:54 dogbert11 so that optimizer works on a higher language level than both spesh and the jit?
12:54 timotimo i'd say the most important thing it does is turn lexical variables into local variables because only blocks without lexical variables can be directly inlined into their lexical outers
12:54 timotimo that's right, but it also doesn't have any run-time information
12:55 dogbert11 so it does its job at program startup/parsing?
12:55 timotimo during compilation, yes, including precompilation of modules
12:56 dogbert11 interesting indeed
12:57 timotimo another thing is that the MAST generated from the QAST is sometimes quite ugly
12:57 timotimo spesh can help there, too
12:58 timotimo like an abundance of decont and hllize instructions that don't actually do anything in 99.9% of code execution
12:58 dogbert11 hmm, slightly confusing terminology though, ASA, QAST, MAST and CFG's in SSA form
12:59 dogbert11 ASA -> AST
13:10 timotimo well, QAST is just "what comes after PAST" and PAST was Parrot AST
13:10 lizmat joined #moarvm
13:11 timotimo MAST is a moarvm-specific format that's no longer tree-shaped and very close to moarvm bytecode
13:23 dogbert11 thx for the explanation timotimo
13:25 timotimo no prob
13:26 timotimo i was going to point you at the mastcompiler c file in moarvm and say "that's why this file is so tiny and simple"
13:26 timotimo but it's still pretty big :P
13:27 AlexDaniel joined #moarvm
14:03 timotimo moar is doing something strange
14:03 timotimo opening /proc/self/stat multiple times per second
14:06 timotimo checking if it should do a full gc, huh
14:07 timotimo can't be too slow i suppose; it doesn't show up notably in the perf output
15:12 zakharyas joined #moarvm
15:42 AlexDaniel MasterDuke: hey, so what's the current status of the leak?
15:45 pharv joined #moarvm
15:46 timotimo hm, so, heaptrack, right? can we get it to show more than just the caller of the allocator function itself?
16:08 Ven joined #moarvm
16:51 MasterDuke timotimo: would the flame graph be more useful? http://i.imgur.com/dJXfLVF.png
16:52 timotimo yeah that looks interesting
16:52 timotimo does "allocations" mean "how often was any allocator function called"?
16:53 timotimo is there one that tallies up the size?
16:53 MasterDuke http://i.imgur.com/MobPJRy.png
16:53 timotimo ah dang
16:54 MasterDuke ?
16:54 timotimo whatever function ends up triggering the gc will get the nursery re-allocation counted towards it
16:54 Ven_ joined #moarvm
16:55 timotimo that won't do at all
16:55 timotimo however, we do see that the "little" buffers we allocate for libuv to fill make up a big chunk of what we allocate in total
16:56 MasterDuke http://i.imgur.com/J20JZv5.png by peak allocation
16:56 MasterDuke do you have heaptrack installed?
16:58 timotimo i do
16:58 timotimo i think?
16:58 timotimo hm, okay, the peak of stuff allocated there isn't terribly bad
16:59 MasterDuke i could put my heaptrack recording on the whateverable server if you want to play around with viewing it locally
17:03 timotimo interesting
17:03 timotimo hm but now
17:03 timotimo this is leaking, right?
17:03 timotimo i.e. increasing its memory
17:04 MasterDuke yeah
17:04 timotimo does any part of the flame graph get noticably bigger or does it all just sort of grow together?
17:04 MasterDuke http://i.imgur.com/CnqmYw8.png
17:08 MasterDuke timotimo: don't quite understand the question
17:16 timotimo like do all parts grow equally when you increase the number of iterations?
17:29 Voldenet joined #moarvm
17:29 Voldenet joined #moarvm
17:35 timotimo can you hide individual slices from that mountain graph?
17:36 timotimo everything above the orange is really difficult to make out because it just goes up and down so much
17:43 MasterDuke you can zoom in and hover over the sections and it'll tell you the function they're for. hard to show that in a screenshot though
17:43 timotimo right
17:43 timotimo might have to actually use the heap snapshot profile for this task
17:44 MasterDuke but it was causing immediate segvs
17:45 timotimo wow, damn
17:45 MasterDuke fyi, i just put the heaptrack recording on the whateverable server
17:45 MasterDuke if you want it
17:46 timotimo interesting, it segfaults inside spesh
17:46 timotimo without spesh the heap profiler can run
17:46 jnthn Huh, that's a bit odd...
17:47 timotimo when gc-ing sim frames
17:47 MasterDuke maybe related, but did you notice my comments from late last night ?
17:47 MasterDuke interestingly, when i misspelled .unique as .uniq, with those env variables set i got `MoarVM oops: Spesh: failed to fix up inline 1` instead of `No such method 'uniq' for invocant of type 'Seq'. Did you mean 'unique'?   in block <unit> at -e line 1`
17:47 timotimo i only looked at it a microscopical bit
17:47 timotimo so don't know what actually is broken here
17:47 jnthn The spesh worker thread runs just as a normal thread, so...
17:47 jnthn MasterDuke: Can you file that as a MoarVM issue? I'd like to look at that during the week.
17:48 MasterDuke jnthn: the heap profiler thing? or the "failed to fix up inline"?
17:49 jnthn MasterDuke: The inline one
17:49 MasterDuke k
17:49 jnthn I changed a bunch of inline-related things last week so we can actually deleted dead inlines.
17:49 jnthn *delete
17:49 jnthn It should mark what it removes
17:49 jnthn And then not try to fix them up
17:50 jnthn And I remembering writing the code to make it handle nested inlines
17:50 jnthn So not sure what'd be going on. Will look in the next days.
17:52 MasterDuke https://github.com/MoarVM/MoarVM/issues/629
17:52 jnthn Thanks!
17:53 * jnthn figures he should try and rest some ahead of throwing himself into another week of MoarVM hackery :)
17:54 MasterDuke also, jnthn++ for spesh blog post. very understandable
17:59 jnthn Glad it made some sense; always have to be careful writing such things after having been dug into the problem space for weeks :)
18:07 dogbert11 MasterDuke: I believe I misunderstood your memory problems yesterday, the problem is not memory leaks but rather memory usage. Is that correct?
18:08 lizmat joined #moarvm
18:09 MasterDuke i'd say both
18:10 dogbert11 running your gist from yesterday (with four iterations rather than 20), leaks are negligible but allocated memory is soething else ...
18:11 dogbert11 MoarVM version 2017.07-333-g8752f9b0: total heap usage: 748,062 allocs, 433,024 frees, 1,337,211,640 bytes allocated
18:12 dogbert11 MoarVM version 2017.07: total heap usage: 643,447 allocs, 374,708 frees, 417,122,198 bytes allocated
18:12 MasterDuke yeah, i think there's some of both, but the large allocations is more noticeable
18:14 MasterDuke hm, wonder if i should do some manual bisecting with valgrind on the whateverable server
18:23 AlexDaniel bisecting bisectable again? :)
18:25 MasterDuke turtles all the way down!
18:28 dogbert11 hmm, the gisted code breaks --profile
18:30 * dogbert11 tries another program
18:30 japhb .oO( "Bisecting the bisectables but they don't know / Debugging the buggables but in the end you know / You turn away, what can I say?" )
18:31 MasterDuke dogbert11: yeah, pretty much anything with run/Proc::Async/etc breaks profiling
18:39 Ven joined #moarvm
19:06 Voldenet_ joined #moarvm
19:11 leego joined #moarvm
19:13 MasterDuke looks like it's https://github.com/rakudo/rakudo/commit/0564891
19:15 MasterDuke valgrind says 2,037,358,560 bytes allocated after that commit, 378,997,214 bytes allocated before that commit
19:15 MasterDuke for 5 iterations in my test script
19:18 ggoebel joined #moarvm
19:18 timotimo joined #moarvm
19:18 tbrowder joined #moarvm
19:24 AlexDaniel :o
19:47 brrt joined #moarvm
19:49 brrt .tell jnthn: yeah, i need to rethink how i do the jit-compiler-disabling
19:49 yoleaux brrt: What kind of a name is "jnthn:"?!
19:49 brrt .tell jnthn i need to rethink how to disable the jt compiler at build time, now that it is so much larger than before
19:49 yoleaux brrt: I'll pass your message to jnthn.
19:51 dogbert11 MasterDuke++
19:51 dogbert11 brrt: https://irclog.perlgeek.de/moarvm/2017-08-06#i_14974983
19:51 brrt i saw it. travis ci also saw it.
19:51 brrt :-)
19:52 dogbert11 yay
19:52 brrt root issue: we're trying to compile a bunch of stuff that doesn't need to be compiled
19:53 brrt i'm wondering how i can make it not do that
19:53 brrt well, i know a way..
19:53 dogbert11 sounds promising
19:54 brrt it sounds like it would be a hack; i could wrap the entire thing with an ifdef
19:54 brrt which would work, but i'd rather not
19:55 brrt has to be a better way
19:55 brrt anyway, i'm off again :-)
20:32 zakharyas joined #moarvm
20:47 MasterDuke looks like it's https://github.com/MoarVM/MoarVM/commit/5c67d732b17a432e4ef3766f181c3c46eb9b9058
20:48 timotimo but that should only cause churn, i.e. things being malloced and soon freed again
20:51 MasterDuke well this is just based on valgrind's "total heap usage" line
20:51 MasterDuke the last "allocated" number
20:51 MasterDuke before that commit it's ~300,00,00, after it's ~2,000,000,000
20:52 timotimo that's "total allocated" but not "in use at end" right?
20:53 MasterDuke right, that's number is pretty much the same
20:53 timotimo yeah, so it's just useless information :)
20:55 MasterDuke hm. what would be more useful?
20:55 timotimo hm
20:55 timotimo figuring out what kind of object there is more of after each iteration
20:56 timotimo but if you want you can probably revert that commit to get a less inflated number there
20:56 MasterDuke think it'll revert cleanly?
20:59 MasterDuke how would i get a count of objects if i can't use the heap profiler?
21:01 timotimo does it also crash if you turn spesh off?
21:03 MasterDuke not immediately...
21:08 MasterDuke ah, got a snapshot with spesh disabled
21:09 MasterDuke let see, guess i should get one at 2017.07 to compare
21:13 MasterDuke ugh, 2017.07 gave `MoarVM panic: Memory allocation failed; could not allocate 72085436 bytes`
21:13 * MasterDuke needs moar ram!
21:16 ilmari[m] joined #moarvm
21:20 zakharyas joined #moarvm
21:29 MasterDuke had to reduce the number of iterations in my script to 3
21:30 MasterDuke but now i have a heap profile, with 43 snapshots
21:30 MasterDuke which one(s) should i look at?
21:36 timotimo oof
21:37 timotimo we don't really have a way to cross-reference what happens in the program with the heap snapshots, except for looking really closely
21:37 timotimo one of the reasons i'd like to offer an op that lets you put a piece of data into telemeh
21:40 MasterDuke would one at random in the middle be somewhat representative?
21:40 timotimo i suppose, but i'd rather go near the end
21:41 MasterDuke hm, 43 snapshots at 2017.07, only 34 at head
21:42 timotimo mhm
21:42 timotimo i forgot what exactly it was that jnthn did to often cause us fewer gc runs
21:42 MasterDuke compare 3/4ths of the way through of each?
21:42 timotimo mhm
21:42 MasterDuke say #35 and #28?
21:43 MasterDuke ugh, and don't have to ram to open both at once...
21:44 timotimo oops :(
21:44 timotimo on my laptop i have as much zram as i have actual ram
21:44 timotimo ever since i destroyed the slotted-in ram (or whatever else) i'm suffering under low ram all the time
21:44 MasterDuke i don't have a swap be default
21:44 timotimo oh
21:44 timotimo i have 2x zram compared to normal ram
21:44 timotimo it's actually already as full as the ram totally (i.e. including the compressed parts)
21:46 MasterDuke top objects by size? anything else useful?
21:49 * MasterDuke doesn't have enough ram to have two heap analyzers open on any one computer, but just realized he's using two computers...
21:53 timotimo maybe also top objects by count and top stables by size and count
21:53 timotimo and maybe even top frames by size
21:55 MasterDuke 2017.07 https://gist.github.com/MasterDuke17/2cad493ec5088231df84e196688f031f
21:57 MasterDuke HEAD https://gist.github.com/MasterDuke17/f02204befc98b8f2d5aea66a8948cc42
21:59 pharv joined #moarvm
22:00 timotimo that's a whole lotta buf
22:04 MasterDuke but more at 2017.07 than HEAD
22:05 timotimo huh
22:13 Voldenet joined #moarvm
22:53 MasterDuke joined #moarvm
23:03 Ven joined #moarvm
23:19 pharv joined #moarvm
23:24 MasterDuke jnthn: does any of the the above point anything out to you? something else timotimo and/or i should try?
23:24 jnthn MasterDuke: Not really; I'll see if I can figure it out a bit more tomorrow though.
23:24 jnthn Heading to sleep shortly
23:26 MasterDuke cool, later...
23:31 timotimo i feel like sleep, too
23:31 timotimo have you tried the realloc trick for the bufs that we allocate for libuv btw?
23:34 MasterDuke no, didn't figure out what exactly was getting realloced to what value
23:34 MasterDuke but can try if you can give some more detailed instructions
23:36 timotimo oh, sure
23:36 timotimo what file was that in again ...
23:36 MasterDuke procops.c, somewhere around line 531 i believe
23:38 timotimo right, so buf->base is the pointer that's eligible for reallocing, buf->len is the actual size that it was last allocated with, and nread is how much is actually used inside the buffer
23:38 timotimo since we don't expect the buffer to be changed afterwards, we can probably just realloc it to the exact number of items read
23:38 MasterDuke jnthn had some comment about how buf->len is supposed to be correct (or something like that)
23:39 timotimo if we decide to, for example, round up to the next number aligned to, like, 32 bytes, or whatever, we'd have a different ssize vs elems
23:40 MasterDuke https://irclog.perlgeek.de/moarvm/2017-08-05#i_14973403
23:41 MasterDuke so i shouldn't just change what res_buf->body.ssize gets assigned, i need to do a realloc?
23:42 timotimo ssize will hold what you realloc to, which may be different from the number of elements used
23:43 MasterDuke so realloc, then set ssize to whatever value i used for the realloc?
23:44 MasterDuke btw, i assume the new value should be nread*(sizeof an element), but what is that size?
23:47 MasterDuke oh, tc->instance->str_consts.buf_type perhaps?
23:47 timotimo no, that's all just strings
23:48 timotimo we use 8 bits for all of these
23:49 timotimo that's also why the .i8 slot is selected for assignment
23:49 timotimo (even though it makes no difference)
23:49 timotimo (all platforms we develop for have the same size of pointer for all kinds of things pointed at)
23:49 MasterDuke ah, so just nread*1 ?
23:52 timotimo i believe so
23:55 MasterDuke `error: assignment of member ‘base’ in read-only object`
23:59 MasterDuke 1519408maxresident)k before, 1303076maxresident)k after
23:59 timotimo um, huh?
23:59 MasterDuke res_buf->body.slots.i8 = MVM_realloc(buf->base, nread); res_buf->body.ssize    = nread;
23:59 timotimo oh, huh

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary