Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-04-05

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 timotimo i have a workload that spends about 90% of run time in collapse_strands
00:04 samcv wow it prints way too much with llvm 5.0 if i do by function. everywhere everything is inlined which may not be the best for the overall summary
00:04 samcv hmm timotimo
00:55 samcv timotimo, is that workload just normal use of JSON::Fast?
00:55 timotimo hm
00:56 timotimo well, it takes a ~420mb file of strings separated with null bytes
00:56 timotimo then it splits that into an array
00:56 timotimo that array gets serialized with to-json, then immediately deserialized with from-json
00:56 timotimo and then i join it with \0 to spurt it back into a different file
00:56 timotimo to see if it successfully roundtrips
00:58 timotimo anyway, it's bedtime for me now, i'd say
01:01 samcv night night timotimo :)
01:28 MasterDuke joined #moarvm
01:48 ilbot3 joined #moarvm
01:48 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
02:15 quotable6 joined #moarvm
02:50 agentzh joined #moarvm
03:24 samcv .seen brrt
03:24 yoleaux2 I saw brrt 4 Apr 2017 09:16Z in #moarvm: <brrt> the jit logging needs to be cleaned up anyway
03:47 agentzh joined #moarvm
04:37 spebern joined #moarvm
05:22 agentzh_ joined #moarvm
05:28 spebern left #moarvm
05:39 geekosaur joined #moarvm
05:45 domidumont joined #moarvm
05:48 domidumont joined #moarvm
05:48 domidumont joined #moarvm
06:06 brrt joined #moarvm
06:12 Geth ¦ MoarVM: 33af46191a | (Samantha McVey)++ | src/strings/normalize.c
06:12 Geth ¦ MoarVM: Remove two funct. in normalize.c that have been superseded
06:12 Geth ¦ MoarVM:
06:12 Geth ¦ MoarVM: is_spacing_mark and is_regional_indicator have been superceeded by the
06:12 Geth ¦ MoarVM: switches for the Grapheme_Cluster_Break property in should_break() which is
06:12 Geth ¦ MoarVM: also in normalize.c
06:12 Geth ¦ MoarVM:
06:12 Geth ¦ MoarVM: Was re-reminded about their non-use while examining our new coverage reports
06:12 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/33af46191a
06:12 samcv hello brrt
06:12 brrt hi samcv
06:12 samcv have you seen the new even fancier coverage
06:13 brrt not yet, i think
06:13 samcv https://cry.nu/coverage/ for nqp and https://cry.nu/coverage/roast for roast
06:13 samcv and you can link to line numbers in the files as well
06:14 samcv and has a link at the top to 'jump to the first uncovered line' in a file
06:14 samcv link line numbers like github
06:16 domidumont joined #moarvm
06:24 Geth ¦ MoarVM/even-moar-jit: fd0de679cc | (Bart Wiegmans)++ | docs/jit/plan.org
06:24 Geth ¦ MoarVM/even-moar-jit: Document plan for 'let' to 'do' conversion
06:24 Geth ¦ MoarVM/even-moar-jit:
06:24 Geth ¦ MoarVM/even-moar-jit: This way, we can remove the 'internal rooting' mechanism and avoid a
06:24 Geth ¦ MoarVM/even-moar-jit: number of potential errors, as well as ensure that the template code
06:24 Geth ¦ MoarVM/even-moar-jit: as written is compiled as the developer should expect it, i.e. in
06:24 Geth ¦ MoarVM/even-moar-jit: linear, greedy order.
06:24 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/fd0de679cc
06:38 domidumont joined #moarvm
06:44 domidumont joined #moarvm
06:48 brrt joined #moarvm
06:56 agentzh joined #moarvm
07:44 samcv brrt, let me know if you want me to profile roast and/or nqp test suite with the even-moar-jit branch
07:47 brrt hmmm
07:48 brrt that is actually a pretty good idea
07:48 samcv yeah?
07:48 brrt i think so, yes
07:48 samcv ok roast or nqp test suite. which one should i do
07:48 brrt nqp test suite will do fine
07:48 samcv ok cool
07:48 brrt it's pretty limited for now, after all
07:49 samcv oo can you merge master back in on it first tho
07:49 samcv cause it doesn't have the new ./Configure.pl option
07:56 samcv hmm got a merge problem. i'll just cherry pick it temporarly. will be faster
07:58 brrt merge problem?
07:58 brrt i can merge it again, yes
07:59 brrt the problem is that some of the Configure.pl options are WRONG
07:59 brrt especially the ones regarding dynasm and lua
07:59 brrt can't risk that, we need our own lua
08:02 zakharyas joined #moarvm
08:02 samcv ok started generating it, will be done in 5 mins or so
08:03 brrt cool
08:04 samcv uh oh. nah. you should merge
08:04 samcv heh. it's missing indexic_s op too
08:04 samcv and nqp is unhappy
08:04 samcv well. it can't compile :P
08:06 brrt alright, had no idea you were going so fast :-P
09:29 brrt re: rust evangelism, the one thing that made me chuckle was 'in the future when overflow checking is going to be cheap'
09:31 samcv when will that be lol
09:31 samcv or is that the joke
09:34 brrt okay, trivia question
09:34 brrt do you know why you do get a 'divide by zero' error and not an integer overflow error?
09:35 brrt (well, you would, on some languages, but in C for example)
09:36 samcv you mean compilier error?
09:36 brrt no, rutnime
09:36 brrt (anyway, lunch &, will be back in a bit and leave you in suspsense)
09:36 samcv kk
09:38 samcv but i mean it doesn't make much sense to divide by zero anyway and i doubt it'd be compatible with the division algorithms they have
10:07 brrt i have forgotten how the binary division algorithm typically works
10:08 brrt anyway, the reason is that a divide-by-zero raises a (hardware) interrupt, whereas integer overflow sets a bit in the flags register
10:22 brrt so the one automatically draws you away from normal execution; the second you have to check-and-branch
10:22 brrt and you can do this, but it doubles the size of your numeric code
10:23 brrt rather than add %rax, %rdi; you have add %rax, %rsi; jo label;' everywhere
10:23 brrt and if you want resumeability, you have to translate that into a call
11:53 AlexDaniel joined #moarvm
12:30 synopsebot6 joined #moarvm
12:57 AlexDaniel joined #moarvm
13:11 domidumont joined #moarvm
15:08 brrt joined #moarvm
15:31 Geth ¦ MoarVM/even-moar-jit: 364e0d2ba1 | (Bart Wiegmans)++ | 7 files
15:31 Geth ¦ MoarVM/even-moar-jit: Extract expr op reading into module
15:31 Geth ¦ MoarVM/even-moar-jit:
15:31 Geth ¦ MoarVM/even-moar-jit: To make reading expr ops simpler and consistent, extract the expr
15:31 Geth ¦ MoarVM/even-moar-jit: ops into their own file, and create a module for reading it.
15:31 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/364e0d2ba1
16:33 agentzh joined #moarvm
16:49 agentzh joined #moarvm
17:00 agentzh joined #moarvm
17:12 agentzh joined #moarvm
17:29 domidumont joined #moarvm
18:31 AlexDaniel joined #moarvm
19:08 praisethemoon joined #moarvm
19:35 samcv good * everyone
19:47 lizmat samcv  o/
19:48 lizmat jnthn: what would you think of having a hash-like equivalent to IterationBuffer such as IterationHash (with the same REPR as nqp::hash, but without the auto-HLLizing everywhere)
19:49 lizmat s/IterationHash/IterationSer/
19:49 lizmat *Set
19:49 lizmat grrrr  ;)
19:49 timotimo HashBuffer? LLHash?
19:50 lizmat the thing is that it's very hard to make hotpath subs that take an nqp::hash as a parameter
19:51 lizmat case in point: Set.new-from-pairs and Iterable (|) Iterable
19:51 lizmat I would like to apply the logic of new-from-pairs in the same nqp::hash
19:52 lizmat but if I put this in a sub, I need to de-HLLize everything again and again
19:53 samcv nqp::hash's get HLLized?
19:54 samcv lizmat, does that have any relation to what happens when i assign a nqp::hash to a %sigil
19:54 samcv m: my %hash = nqp::hash; %hash<foo> = 'bar'; say %hash
19:54 camelia rakudo-moar b62d1a: OUTPUT: «===SORRY!=== Error while compiling <tmp>␤Could not find nqp::hash, did you forget 'use nqp;' ?␤at <tmp>:1␤------> my %hash = nqp::hash⏏; %hash<foo> = 'bar'; say %hash␤    expecting any of:␤        argument list␤»
19:54 samcv m: use nqp; my %hash = nqp::hash; %hash<foo> = 'bar'; say %hash
19:54 camelia rakudo-moar b62d1a: OUTPUT: «{foo => bar}␤»
19:55 samcv also what does HLL stand for ;) i have not dived into that section of the codebase yet
19:55 lizmat samcv: the (empty) nqp::hash gets HighLevelLanguaged, is empty, so doesn't assign anything to the %hash
19:56 samcv it does though
19:56 lizmat m: use nqp; my $h := nqp::hash("a",42); sub a(\h) { say h.WHAT }; a $h
19:56 camelia rakudo-moar b62d1a: OUTPUT: «(Hash)␤»
19:56 samcv it assigns something to the %hash
19:56 lizmat m: use nqp; my %h = nqp::hash; dd %h
19:56 camelia rakudo-moar b62d1a: OUTPUT: «Hash %h = {}␤»
19:57 lizmat samcv: don't think so
19:57 samcv m: use nqp; my %hash = nqp::hash; say nqp::existskey(%hash, 'blah')
19:57 camelia rakudo-moar b62d1a: OUTPUT: «0␤»
19:57 samcv m: use nqp; my %hash; say nqp::existskey(%hash, 'blah')
19:57 camelia rakudo-moar b62d1a: OUTPUT: «(signal SEGV)»
19:57 samcv see
19:57 samcv it does *something* at least
19:57 lizmat what does something?
19:58 samcv i can't do nqp::existskey on a %sigil that is unpopulated or else it segV's
19:58 samcv but if i assign nqp::hash to the %sigil thing, then it doesn't segv and nqp::existskey works
19:59 lizmat m: my %h; dd nqp::getattr(%h,Map,'$!storage')
19:59 camelia rakudo-moar b62d1a: OUTPUT: «===SORRY!=== Error while compiling <tmp>␤Could not find nqp::getattr, did you forget 'use nqp;' ?␤at <tmp>:1␤------>  %h; dd nqp::getattr(%h,Map,'$!storage')⏏<EOL>␤»
19:59 lizmat m: use nqp; my %h; dd nqp::getattr(%h,Map,'$!storage')
19:59 camelia rakudo-moar b62d1a: OUTPUT: «Mu $!storage = Mu␤»
19:59 lizmat m: use nqp; my %h = (); dd nqp::getattr(%h,Map,'$!storage')
19:59 camelia rakudo-moar b62d1a: OUTPUT: «{}␤»
19:59 lizmat if you don't try to assign to a Hash, it will not allocate its internal nqp::hash
20:00 samcv m: use nqp; my %hash= (); say nqp::existskey(%hash, 'blah')
20:00 camelia rakudo-moar b62d1a: OUTPUT: «0␤»
20:00 lizmat m: use nqp::existskey(My,'foo')
20:00 camelia rakudo-moar b62d1a: OUTPUT: «===SORRY!===␤Could not find nqp::existskey at line 1 in:␤    /home/camelia/.perl6␤    /home/camelia/rakudo-m-inst-2/share/perl6/site␤    /home/camelia/rakudo-m-inst-2/share/perl6/vendor␤    /home/camelia/rakudo-m-inst-2/share/perl6␤    CompUn…»
20:00 lizmat m: use nqp; nqp::existskey(My,'foo')
20:00 camelia rakudo-moar b62d1a: OUTPUT: «===SORRY!=== Error while compiling <tmp>␤Undeclared name:␤    My used at line 1␤␤»
20:00 lizmat m: use nqp; nqp::existskey(Mu,'foo')
20:00 camelia rakudo-moar b62d1a: OUTPUT: «This type (Mu) does not support associative operations␤  in block <unit> at <tmp> line 1␤␤»
20:00 samcv kk. so it's not specific to assigning to nqp::hash. just that it has to have a hash allocated to it
20:00 lizmat m: use nqp; nqp::existskey(nqp::null,'foo')
20:00 camelia rakudo-moar b62d1a: OUTPUT: «This representation (Null) does not support associative access (for type VMNull)␤  in block <unit> at <tmp> line 1␤␤»
20:00 samcv and the hash that gets allocated isn't the one i assign to it
20:01 lizmat m: use nqp; my %h = nqp::hash  # nqp::hash becomes {}, which becomes (), which causes $!storage to be allocated, but still empty
20:01 camelia rakudo-moar b62d1a: ( no output )
20:02 lizmat samcv: ^^  what happens if you say %h = nqp::hash()
20:02 samcv kk
20:03 samcv yeah. so something happens but not directly
20:03 lizmat m: use nqp; my $h := nqp::hash; nqp::bindattr(my %hash,Map,'$!storage',$h); nqp::bindkey($h,"a",42); dd %hash
20:03 camelia rakudo-moar b62d1a: OUTPUT: «Hash %hash = {:a(42)}␤»
20:04 lizmat samcv: have a low level hash be part of a HLL Hash
20:04 lizmat *having
20:19 samcv hmm MVM_STRING_GRAPHEME_ASCII appears to be unused in MoarVM
20:19 samcv MVM_STRING_GRAPHEME_8 is used though
20:19 samcv not sure why it exists
20:20 samcv since it is never used, and its processing is the same as MVM_STRING_GRAPHEME_8
20:20 samcv and nothing ever sets strings as MVM_STRING_GRAPHEME_ASCII
20:25 MasterDuke i think when you create small string literals from the command line they're *_ASCII
20:25 samcv oh i guess they are sorta different. Grapheme_8 can have synthetics down to -128 or something
20:25 samcv nah they are MVM_STRING_GRAPHEME_8
20:29 samcv MVMGraphemeASCII and MVMGrapheme8 are both MVMint8, one just doesn't have negative numbers
20:29 samcv or any synthetics
20:29 MasterDuke m: use nqp; my @array; say nqp::existspos(@array, 2)
20:29 camelia rakudo-moar b62d1a: OUTPUT: «This type (Array) does not support elems␤  in block <unit> at <tmp> line 1␤␤»
20:29 samcv so it can still fit things into 8 bits in case there are synthetics + no characters above 127
20:29 MasterDuke that's a nicer error than a SEGV
20:30 samcv ya
21:08 quotable6 joined #moarvm
21:08 unicodable6 joined #moarvm
21:08 evalable6 joined #moarvm
21:08 bisectable6 joined #moarvm
21:08 bloatable6 joined #moarvm
21:08 committable6 joined #moarvm
21:08 benchable6 joined #moarvm
21:08 statisfiable6 joined #moarvm
21:20 samcv timotimo, can you show me the workload that uses all the time in collapse_strands please
21:23 jnthn lizmat: If you want it for internal use, just declare it in Rakudo::Internals. :) If you want to expose it, then it'll need more discussion/thought.
21:23 jnthn lizmat: Should be as easy as class Rakudo::Internals::YourNameHere is repr('VMHash') { }
21:24 timotimo samcv: you have access to the whateverable server, right?
21:24 lizmat yeah, it's already in core atm
21:24 samcv yeah
21:25 lizmat jnthn: but I could still move it to Rakudo::Internals
21:25 samcv jnthn, how do you feel about MVM_string_graphs_nocheck because in many places we check twice
21:25 samcv MVM_string_graphs does a check and MVM_string_check_arg does a check
21:25 timotimo /home/bisectable/git/whateverable/irc/cache.tar.gz
21:25 timotimo grab that file and extract it
21:25 timotimo that's the 420 megs file that i'm using
21:26 samcv ok what's the test job
21:26 lizmat jnthn: I called it IterationSet
21:26 samcv i will do that in a bit
21:26 timotimo then check out the latest JSON::Fast, and do like spurt("otherCache", from-json(to-json(slurp("cache").split("\0"))).join("\0"))
21:27 timotimo that's already run for 13 hours on my desktop
21:27 timotimo so you may want to cut the size of "cache" down by like 10x
21:27 timotimo or 100x
21:27 samcv ok. it's just a long text file? or something?
21:27 samcv are there any cp's above 127?
21:28 timotimo it's every line of irc text from some log, but instead of newlines it has \0
21:28 samcv if not, i might know what's slowing it down
21:28 timotimo it most probably does
21:28 samcv if there's only ascii chars in it
21:28 samcv ok
21:28 samcv when we colapse strands and it fits in 8 bits, after collapsing the strand, we then convert taht 32bit string into an 8bit string
21:28 Zoffix timotimo: so JSON::Fast now uses 1/10th of the memory; did it get faster too? By how much?
21:29 timotimo Zoffix: it only got much much faster for your problematic string
21:29 timotimo it got slower for other strings
21:29 Zoffix :(
21:29 lizmat jnthn: testing whether s/IterationSet/Rakudo::Internals::IterationSet/ creates an issue
21:29 jnthn lizmat: What are you actually using it for? But yeah, stick it in internals...I guess i's not in public API
21:29 timotimo i don't have a good set of long json strings to test this on
21:29 jnthn I mean, IterationBuffer plays a role in custom iterator implementations
21:30 Zoffix Well, speed was never an issue for me with it.
21:30 jnthn samcv: Hm, and the C compiler ain't smart enough to avoid 'em? I guess we can
21:30 jnthn (Add one
21:30 samcv idk if it is smart enough to avoid them. probably not because often we do other things afterward
21:30 samcv call functions etc.
21:31 jnthn Yeah, true
21:31 samcv okay cool
21:31 jnthn Anyway, no objections if there's some kind of measurable improvement from it :)
21:31 samcv what if there is non :P
21:31 jnthn callgrind is probably going to be able to tell you best :)
21:31 samcv *none
21:31 samcv maybe it's some very small amount...
21:32 jnthn Sure; callgrind will tell you in CPU instructions, though :-)
21:32 lizmat jnthn: no objections against IterationSet, or do you want it to be R:I:IterationSet ?
21:32 jnthn lizmat: The latter, plesae
21:32 samcv timotimo, copying that file now
21:32 lizmat (the latter is more typing)
21:32 lizmat ok
21:32 jnthn Yes, but exposing stuff we don't know we want to keep is less than good :)
21:33 lizmat ok
21:33 jnthn Or want to call different or whatever
21:33 lizmat fair enough, running spectest now
21:33 jnthn I mean, I'm a bit uneasy with the name IterationSet, but if it's internal we don't need to worry )
21:33 jnthn *:)
21:33 timotimo samcv: looking forward to your findings :)
21:33 timotimo samcv: i didn't touch the whole topic at all after starting that
21:33 lizmat jnthn: and it does make 5% difference on Set.new(42)
21:33 timotimo i thought i could first figure out how long exactly it runs for
21:34 jnthn lizmat: Nice :)
21:34 samcv never finishes?
21:34 samcv how big is the file?
21:34 samcv 400MB or whatever? that's pretty big
21:34 timotimo anyway, yeah, 95% time spent in collapse_strands or functions called by collapse_strands
21:34 lizmat jnthn: with one frame less and one object created less
21:34 timotimo 420 blaze it megabytes
21:34 samcv k i will fix your problem timotimo (maybe)
21:34 lizmat jnthn: for each Set.new
21:35 timotimo whoa
21:36 jnthn Sounds good :)
21:36 timotimo fwiw, all i need this for is to check if json::fast roundtrips things properly
21:39 timotimo the memory usage pattern has changed, it's now peaking up to 5.5gb every few sconds, in between going down to 2.5gb
21:48 Zoffix Is that still 1/10th of previous usage?
21:48 * Zoffix just told people to upgrade JSON::Fast to get awesomification
21:51 timotimo i hope it's not buggy
21:51 timotimo i didn't try this workload with the previous JSON::Fast
21:51 timotimo i can imagine it'd take a bit more memory
22:05 samcv ok timotimo callgrinding
22:06 timotimo how small did you make the file?
22:07 samcv probably not small enough idk
22:10 timotimo i really ought to have a corpus of json files that i can benchmark against all the revisions i've made
22:10 samcv gonna run it with shorter file
22:11 samcv how long for  from-json(to-json(slurp("scache").split("\0"))).join("\0")) btw
22:11 samcv how long will i have to be waiting
22:11 samcv for 120MB file
22:13 timotimo i don't know if the time usage will decrease linearly, quadratically, or exponentially
22:13 timotimo if it's linear, you'll have to wait a minimum of 3 hours
22:14 samcv that is too much
22:16 timotimo well, yeah
22:16 timotimo that's why i suggested going down to 1% rather than 10%
22:16 samcv but why are we splitting and joining by "\0"
22:16 samcv just for fun?
22:16 samcv or is that what we're testing not the json stuff
22:17 timotimo see if it roundtrips perfectly
22:21 timotimo basically, someone suggested i use this file as the ultimate test for my escaping and parsing of strings
22:21 timotimo so i said "yeah why not"
22:21 timotimo and it turned out that it takes rather a long time
22:22 samcv ok i made it 1/100 of the number of characters
22:22 samcv *graphemes that is
22:23 samcv uhm and it's 2.5MB
22:23 samcv so hopefully it doesn't take like hours
22:25 timotimo right
22:25 timotimo let's hope so
22:28 MasterDuke 2mb took 32s, 186728maxresident
22:30 samcv not in cachegrind :S
22:30 timotimo oh, yeah
22:30 timotimo that makes another 2x to 10x difference?
22:30 samcv sometimes feels like it's 1000x XD
22:31 timotimo only when you're watching it :D
22:35 MasterDuke with just one data point for each way, making that change is posited in collapse_strands took 0.1s longer
22:37 samcv gotta be slower than 10x. it's still going after 15 mins
22:37 samcv so at least 32x
22:41 samcv what good are these 16 threads now T_T
22:42 samcv err 8 threads
22:44 timotimo i'm bout to reboot my 'puter, so the time measurement is going to be capped here
22:44 timotimo at 14h10m
22:46 samcv i feel like a nap
22:46 samcv and when i get back it will be done :P
22:47 timotimo i hope it will :)
22:59 samcv oh. no. i accidently didn't use the module
22:59 samcv oh well. runnig again. good thing i haven't started my nap
23:31 agentzh joined #moarvm

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary