Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-01-04

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:43 samcv ok i think i have it working where we can use a faster hex2decimal number thing
01:43 samcv because what it does to get the decomposition characters, it reads a string with hex numbers delimited by spaces
01:44 samcv and strtol has to get the number, then store back as a string in the original pointer. so i wrote one where we just keep it as a char ** instead of char * and just move the char** pointer down a bit
01:44 samcv have to profile now
01:44 samcv but should take trivial processing power
01:45 samcv and when it sees a space (on non ascii) it returns what it's seen so far, and sets the pointer to one after this non hexchar
01:45 samcv and passes the NFD-round trip spectest :)
01:50 samcv hmm i'm still seeing calls to strtol
01:59 samcv ah it was the atoi function. now uses 12% less cpu :)
01:59 samcv down from 14.5% to 2.28
02:01 samcv from 6.5seconds slurping this 200MB file to 5.7
02:48 ilbot3 joined #moarvm
02:48 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
02:59 samcv yeah
02:59 timotimo so this is utf8, i wonder how utf8-c8 compares when the content is 100% valid utf8
03:00 samcv well. the same
03:00 timotimo you'd certainly hope so
03:00 samcv since they both call it when composing graphemes
03:00 timotimo i meant in general, though
03:00 samcv ah
03:00 samcv you mean between utf8-c8 and normal utf8?
03:00 timotimo yes
03:01 timotimo and latin1 is probably a whole lot faster because it just YOLOs all those bytes
03:02 samcv weird. https://www.google.com/trends/explore?date=today%2012-m&q=perl%206,perl
03:03 samcv oh can more easily see trend if you look since 2004
03:03 samcv https://www.google.com/trends/explore?date=all&q=perl%206,perl
03:03 timotimo oof
03:15 mst samcv: blah, doesn't look any better if I correct 'perl 6' to 'perl6' either
03:33 samcv yeah
03:34 samcv https://www.google.com/trends/explore?q=perl%206
03:34 samcv obviously we just need to release 6.d :P
03:34 timotimo samcv: would you be interested in doing the 200MB file with utf8 vs utf8-c8?
03:34 samcv how do I use utf8-c8?
03:34 samcv from nqp/perl6. whichever
03:34 samcv specify the encoding or something?
03:34 timotimo just :enc<utf8-c8>
03:34 samcv (in perl 6) kkk
03:34 samcv col
03:35 samcv * kk *cool
03:37 samcv utf8 5.7s, utf8-c8 7.8 (both with patch applied)
03:37 timotimo interesting, so it's a noticable impact
03:37 samcv yeah
03:41 samcv with the patch and plain utf-8 getting 40% (exclusive) cpu usage by MVM_string_utf8_decodestream and 17.68% by the codepoint normalization function (inclusive)
03:43 samcv but we have 10% happening in MVM_unicode_codepoint_get_property_cstr
03:43 samcv so i can shave another 10% off the cpu
03:44 timotimo nice
03:44 samcv off the 14% (12% speedup using faster atoi + 2% faster by inlining). and the 2% time going to uh
03:44 samcv the fast_atoi should go away too as i make def's and check the property value as integer
05:02 samcv timotimo, maybe we can find some library that uses SIMD to do utf8-decode
05:02 samcv that would be really fast
05:02 samcv i think arm has SIMD now too?
05:06 samcv apparently even SSE4.2 has a string comparison function. interesting
05:47 synopsebot6 joined #moarvm
05:49 synopsebot6 joined #moarvm
05:50 synopsebot6 joined #moarvm
05:54 synopsebot6 joined #moarvm
05:55 synopsebot6 joined #moarvm
06:00 synopsebot6 joined #moarvm
06:12 mst joined #moarvm
07:15 domidumont joined #moarvm
07:20 domidumont joined #moarvm
07:40 FROGGS joined #moarvm
08:13 nine samcv: couldn't you combine strlen and fast_atoi into a single function? A fast_max_3_digit_atoi of sorts?
08:13 samcv for what?
08:13 samcv for ccc?
08:14 samcv why do we check the length of the string anyway. do you know?
08:14 samcv i think it would return like `ccc3` or `ccc51` but those are all more than 3 characters
08:15 samcv er wait. no it returns just the numbers
08:16 samcv well i guess it returns like. numbers or whatever, but they're stored as strings. will have to roll that into my ucd2c.pl that i generate the brackets with
08:16 samcv that generates integer properties only with no string value
08:18 samcv yeah i will make it integer instead of using the string thing soonish. would be nice to change it for now just to get that performance improvement
08:19 samcv i also wrote a much faster hex string function, so it can check decompositions faster, instead of having to parse the first hex in the string "DD38 EAT" then copy the rest of the string to the same pointer location
08:20 samcv but that will not have as much impect unless we're having to decompose a lot. the ccc is checked for any codepoint giong through MVM_unicode_normalizer_process_codepoint_full
08:20 dalek MoarVM: d0e297f | samcv++ | src/strings/normalize.c:
08:20 dalek MoarVM: Use much faster atoi function. 14% less CPU use when slurping a Unicode file
08:20 dalek MoarVM:
08:20 dalek MoarVM: This should greatly speed up any calls made to ccc which is used
08:20 dalek MoarVM: extensively in MVM_unicode_normalizer_process_codepoint_full and
08:20 dalek MoarVM: canonical_composition.
08:20 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d0e297fd43
08:20 dalek MoarVM: 36f3385 | niner++ | src/strings/normalize.c:
08:20 dalek MoarVM: Merge pull request #483 from samcv/fast_atoi
08:20 samcv oh also in canonical_sort and canonical_composition is used extensively too
08:20 dalek MoarVM:
08:20 dalek MoarVM: Use much faster atoi function. 14% less CPU use when slurping a Unicode file
08:20 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/36f3385fa8
08:21 samcv and RE the ccc function, I was thinking about rewriting it to rely less on ccc, maybe even throw it out completely, so we only have to look up grapheme_cluster_break except in rare cases
08:22 samcv thanks for the merge :)
08:22 nine samcv: just picking the smallest of nits really. I'd appreciate having operators surrounded by spaces: value * 10 as especially in C my brain immediately interprets an asterisk running into a name as pointer.
08:22 samcv ah
08:23 samcv yeah that doesn't look that nice
08:27 brrt joined #moarvm
08:30 brrt hey #moarvm
08:30 samcv hey brrt :)
08:30 brrt i have a christmas present, and it's either late or early depending on your background
08:31 dalek MoarVM/even-moar-jit: ab07774 | brrt++ | / (3 files):
08:31 dalek MoarVM/even-moar-jit: linear_scan can now works for NQP compilation
08:31 dalek MoarVM/even-moar-jit:
08:31 dalek MoarVM/even-moar-jit: Fixed the seemingly last few bugs that prevented linear_scan from
08:31 dalek MoarVM/even-moar-jit: working correctly. NQP now compiles and works, and I can start
08:31 dalek MoarVM/even-moar-jit: implementing spilling.
08:31 dalek MoarVM/even-moar-jit:
08:31 dalek MoarVM/even-moar-jit: NB: NQP commit a6c267fcd11cc3c69ec579345c8db959de78af1c, because it's
08:31 dalek MoarVM/even-moar-jit: not fully up to date with master.
08:31 dalek MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/ab077741ec
08:32 brrt this means that as far as I know (and pending a merge with master) the new linear scan algorithms is feature-equivalent with the old inline register allocator
08:35 zakharyas joined #moarvm
08:40 nine Yeah!
08:41 nine brrt: so how close are we to the point where a horde of coders can chime in and add JIT ops?
08:41 brrt actually, considerably closer
08:42 brrt one thing that should theorethically be handled now, is conditional constructs
08:42 brrt e.g. (if (foo?) x y) resolve to either x and y
08:43 brrt if you don't yet see how that is a register allocation problem, well, i didn't either
08:44 brrt but the first priority is to have spilling implemented
08:44 brrt it's a simple algorithm, really
08:44 brrt (the spilling)
08:44 brrt but not triggered very often
08:45 brrt one of the things that might be a bit tricky is handling conflicting register requirements
09:02 dalek Heuristic branch merge: pushed 95 commits to MoarVM/even-moar-jit by bdw
09:08 brrt joined #moarvm
09:18 brrt joined #moarvm
09:30 nine Ok, I'll bite. How are conditionals a register allocation problem especially?
09:31 brrt they're not, but if you maintain your IR in SSA form the register allocator needs to have PHI resolution built in :-P
09:32 brrt so, in my case, handling conditionals was a register allocator problem
09:33 brrt because at the end of the conditional, both value paths must store their value in the same location
09:35 samcv .tell jnthn so i found the code for qt's ascii and utf-8 decoder https://code.woboq.org/qt5/qtbase/src/corelib/codecs/qutfcodec.cpp.html
09:35 samcv damn yoleux is not here
09:36 brrt joined #moarvm
09:43 brrt joined #moarvm
09:51 brrt there ought to be yoleaux2?
09:51 brrt hmmm
09:51 brrt isn't
09:53 samcv :(
09:57 jnthn Lazy bots.
09:57 jnthn morning o/
09:58 samcv morning jnthn
09:58 samcv oh i should probably close like a bunch of unicode tickets i opened
09:58 samcv since i fixed them
09:59 nwc10 good UGT, #moarvm
10:03 brrt good *, nwc10, jnthn
10:08 samcv good *, every*!
10:08 jnthn :)
10:08 * jnthn tries to catch up on backlog here
10:11 jnthn samcv++ # speedup
10:11 jnthn brrt++ # linear allocator \o/
10:13 jnthn re "we first put them into codepoints, then into graphemes" from #perl6-dev - since the normalizer is a streaming thing we often only ever have one codepoint sat in the intermediate buffer.
10:13 jnthn (Which is why there's a fast-path for that case, iirc)
10:13 samcv yep
10:14 samcv was that to me.
10:14 * samcv tries to check what i said
10:14 samcv ah ok i see
10:16 jnthn fwiw, I'm fine with us using SIMD ops and so forth when available for faster utf-8 decoding provided the detection of when we have them is robust
10:17 arnsholt samcv: I'm a fan of GH's "closes #XYZ" handling =)
10:17 samcv yeah that's nice
10:17 arnsholt Saves that additional step (and saves you from forgetting it)
10:17 samcv oh jnthn ok so this hex function i made can replace the strtol we have https://github.com/samcv/MoarVM/commit/5045037de2a25f4b5181c08823a8ab1bc1b26907
10:18 samcv though the commit nine already merged accouts for most of the calls to strtol (called by atoi)
10:18 jnthn Yeah, we ccc a lot more than we decmop :)
10:18 samcv i think i said it here already. instead of using strtol to put what remains of the string back in the original strings position
10:18 jnthn *decomp
10:18 samcv yeah :)
10:19 samcv we use char** and set a further down pointer location and use that. this implementation works, haven't like cleaned it up obviously but wanted to know if it seems fine to do it that way
10:19 jnthn [0 ... 255] = -1,
10:19 * jnthn wonders how portable between compilers that syntax is
10:20 jnthn It looks too nice for MSVC :P
10:21 jnthn But yeah, overall seems reasonable.
10:21 jnthn // bit aligned access into this table is considerably ...
10:21 jnthn Did you mean word aligned?
10:22 jnthn Or 4-byte aligned
10:22 samcv i did not write that part of it ha
10:22 jnthn :P
10:22 jnthn hexdec2 wants marking static too so we don't leak it
10:22 samcv it seemed ambiguous to me too
10:22 jnthn I think it's a thinko
10:23 jnthn It's probably true if you replace "bit" with the appropraite unit
10:23 jnthn Well, may be true
10:23 samcv technically isn't everything bit alligned?
10:23 samcv hahaha
10:23 jnthn Yes! :D
10:23 samcv (one would hope)
10:24 jnthn I mean, certainly reading, say, a 16-bit or 32-bit value when it's not aligned to such a boundary will be slower
10:24 jnthn About reading bytes, not so sure
10:25 samcv also the -1 also easily makes it break for unknown characters as well jnthn
10:25 samcv like when it sees a space it breaks and returns it up to that point. than it can resume after the space on next invocation
10:26 jnthn Yes, the -1 is a nice trick :)
10:26 jnthn Gotta tend to something else, back soon
10:41 brrt joined #moarvm
11:00 samcv not sure why this breaks a few NFKC tests https://github.com/samcv/MoarVM/commit/36b350550e3b58f74adf2e4c86047ef81cb1ea91
11:05 samcv the codepoints that fail both respond with Y/N or 1/0, i checked so hmm
11:13 Dunearhp joined #moarvm
11:13 Ven joined #moarvm
11:25 brrt joined #moarvm
11:29 Ven joined #moarvm
14:37 yoleaux2 joined #moarvm
15:10 lizmat_ joined #moarvm
15:13 Dunearhp_ joined #moarvm
16:02 dogbert17 joined #moarvm
16:17 japhb jnthn: After fighting with some ecosystem problems last night (JSON::Tiny was failing lots of tests), I was able to confirm your callsame fixes for the golfed version indeed fixed the original as well -- THANK YOU again!
16:28 jnthn Hurrah :)
16:28 jnthn Warnings gone too?
16:34 synopsebot6 joined #moarvm
16:45 japhb jnthn: I didn't run it a pile of times to be sure, but I didn't see any in the limited time I had this morning.
16:49 jnthn OK, sounds good. It felt likely to me that they had the same root cause.
16:50 jnthn (And I never saw them again when testing the fix also)
17:02 domidumont joined #moarvm
17:08 Ven joined #moarvm
17:28 Ven joined #moarvm
17:47 Ven joined #moarvm
18:07 Ven joined #moarvm
18:27 Ven joined #moarvm
18:28 timotimo what the hell is [0 ... 255] = -1 syntax, i have never heard of that before
18:31 notviki m: d @ = -1
18:31 camelia rakudo-moar 40d7de: OUTPUT«===SORRY!=== Error while compiling <tmp>␤Undeclared routine:␤    d used at line 1␤␤»
18:31 notviki m: dd @ = -1
18:31 camelia rakudo-moar 40d7de: OUTPUT«Array @ = [-1]␤»
18:32 timotimo i'm talking about C syntax, though
18:32 timotimo +static const long hextable[] = {
18:32 timotimo +   [0 ... 255] = -1, // bit aligned access into this table is considerably
18:32 notviki aw :(
18:36 geekosaur go home C, you're drunk
18:37 dalek MoarVM: c03a35f | jnthn++ | src/profiler/heapsnapshot.c:
18:37 dalek MoarVM: Fix typo.
18:37 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/c03a35fa4f
18:37 dalek MoarVM: d2fb3c3 | jnthn++ | src/profiler/heapsnapshot.c:
18:37 dalek MoarVM: Fix heap snapshot crash on eventloop thread.
18:37 dalek MoarVM:
18:37 dalek MoarVM: It has no cur_frame, since it's not purposed to execute code.
18:37 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d2fb3c3886
18:38 timotimo so ... is that actually going to work in MSVC, too?
18:46 jnthn I dunno, but I'm not betting on it :)
18:50 geekosaur joined #moarvm
19:08 Ven joined #moarvm
19:23 Ven joined #moarvm
19:34 diakopter joined #moarvm
19:36 masak_ joined #moarvm
19:36 domidumont joined #moarvm
19:36 btyler joined #moarvm
19:36 samcv joined #moarvm
19:36 samcv joined #moarvm
19:44 Ven joined #moarvm
19:59 jnthn *sigh* So I wonder what exactly I was thinking when I introduced the getrefref ops
20:00 jnthn *getregref
20:00 jnthn They mean the lifetime of ->work isn't promised to be over at frame exit
20:01 lizmat it's not too late to change them?
20:01 jnthn They're used for QAST localref support
20:01 jnthn Rakudo doesn't even mention it
20:01 jnthn NQP only tests and implements it but doesn't sue it
20:01 jnthn *use it
20:01 jnthn The memory leak I just found is rooted in this.
20:01 jnthn But also needing to support it is an utter pain
20:01 jnthn For all backends
20:02 jnthn I guess my line of thought may have been "it'd be nice to lower more lexicals to locals"
20:02 jnthn But...it'll be horrible
20:03 timotimo that's a thing i hadn't considered when i last looked at that :S
20:03 Ven joined #moarvm
20:03 timotimo so ... we can totally still throw that out
20:03 jnthn Indeed.
20:04 jnthn I can't think of any other reson we can't just kill ->work always upon frame LEAVE
20:04 * jnthn should think about dinner and some rest from the keyboard
20:04 timotimo right, that makes sense
20:04 jnthn Will have a look further tomorrow
20:05 jnthn Guess they went in during the intense year of 2016
20:05 jnthn *2015
20:05 timotimo so, we were leaking stuff because frames were being kept alive and their -> work kept big things alive?
20:05 jnthn Yup
20:05 timotimo and we could just throw out -> work much earlier
20:05 jnthn And also because anything with a ->work stays in inter-gen
20:05 timotimo got it
20:05 jnthn 'cus it assumes we're still in it
20:05 jnthn So yeah
20:05 jnthn moar-ha was right
20:06 timotimo a-ha!
20:06 jnthn Righty, food... :)
20:06 jnthn My wife just came home so I can't forget to eat dinner for any longer. :)
20:06 nwc10 I was wondering how you'd managed to get away with it for so long
20:07 timotimo have a good one!
20:07 timotimo i got up at like 7pm today :S
20:10 notviki what's inter-gen?
20:11 timotimo you know how we have minor and major GC collections?
20:11 notviki sorta, yeah
20:11 timotimo OK
20:11 timotimo so imagine we create an object (in the nursery) and add it to some big list (that already lives in the old generation)
20:12 notviki ok
20:12 timotimo and every other pointer in the nursery that points to this object no longer lives, for example we've already left the routine we created that object in or something
20:12 notviki ok
20:12 timotimo now if we hit a minor collection, and we only follow stuff in the nursery
20:13 timotimo we'll not reach that object, and thus consider it dead
20:13 timotimo but that's wrong, as it is actually kept alive by the object in the old generation
20:13 timotimo and also: the object in the old generation has a pointer to that object which needs to be updated
20:13 timotimo that's where the inter-generational roots come in
20:14 timotimo whenever an object that lives in the old generation gets something assigned to it that lives in the nursery, that pointer (that's inside the old-gen object) gets added to the set of "inter-generational roots"
20:14 notviki Thanks.
20:14 notviki So all the "root" stuff I've seen in commits is about GC?
20:15 timotimo yup
20:15 notviki Thanks.
20:15 timotimo the most difficult part about this roots stuff is when C code comes in
20:15 timotimo because then there's pointers to GC-managed objects on the C stack, and whenever GC happens the underlying objects can change their position in memory
20:16 timotimo if the pointers on the C stack aren't "rooted", the GC won't know to update those pointers
20:16 timotimo (and also things might be considered dead if only the C stack was holding on to them)
20:16 timotimo when we have nqp or p6 code that works with objects, we're unable to forget to root stuff, because the whole stack has roots set up for it automatically
20:17 timotimo other GC things for C languages tend to scan the stack for "pointers that could point at GC-managed objects", but that has to be "conservative", i.e. it may consider some things pointers that aren't actually pointers; for that reason those GCs aren't allowed to be moving GCs, because they could accidentally change values on the stack that aren't actually pointers
20:19 * notviki wonders how long it'd take to learn all of this stuff :(
20:19 timotimo hah
20:20 timotimo there's a book called "The GC Handbook" or something, that's considered to be The Definitive Book On GCs
20:22 notviki I'll give it a read then
20:23 Ven joined #moarvm
20:25 timotimo i haven't read it yet
20:25 timotimo if i understand correctly, it covers every imaginable technique, whereas moar only uses a few (because many are mutually exclusive)
20:26 timotimo did you see "the secret life of garbage collectors"?
20:26 notviki Do you have a CS degree?
20:26 timotimo i do not
20:26 notviki That gives me hope :}
20:26 notviki (to ever learn this stuff, I mean)
20:26 * geekosaur doesn't either, and has a reasonable grasp of practical GC
20:27 timotimo i did start a CS degree, though
20:27 geekosaur mostly obtained operationally (i.e. by diving expeditions in various garbage collectors, and reading discussions thereof)
20:27 geekosaur that said, I, uh, learn somewhat unconventionally :)
20:53 Ven joined #moarvm
21:23 Ven joined #moarvm
21:36 Ven_ joined #moarvm
21:41 dalek MoarVM: 592e536 | samcv++ | Configure.pl:
21:41 dalek MoarVM: Use /usr/bin/env perl for ./Configure.pl
21:41 dalek MoarVM:
21:41 dalek MoarVM: We already do this for NQP and Rakudo now, and will make it be more
21:41 dalek MoarVM: compatible on Unixish systems. Windows systems are unaffected by this
21:41 dalek MoarVM: change.
21:41 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/592e536204
21:41 dalek MoarVM: 7361f46 | niner++ | Configure.pl:
21:41 dalek MoarVM: Merge pull request #484 from samcv/#!
21:41 dalek MoarVM:
21:41 dalek MoarVM: Use /usr/bin/env perl for ./Configure.pl
21:41 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/7361f46836
22:51 dalek MoarVM: 40ee0e8 | samcv++ | tools/ucd2c.pl:
22:51 dalek MoarVM: Generate Decomposition_Type Unicode prop. #define's in ucd2c.pl
22:51 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/40ee0e8eb1
22:51 dalek MoarVM: d119127 | samcv++ | / (4 files):
22:51 dalek MoarVM: Decompose_Type, Unicode: use int lookup instead of str for better perf.
22:51 dalek MoarVM:
22:51 dalek MoarVM: Also generate Canonical_Combining_Class #define's in ucd2c.pl.
22:51 dalek MoarVM: This will reduce our need to compare strings.
22:51 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d119127ece
22:51 dalek MoarVM: 69e2a24 | lizmat++ | / (4 files):
22:51 dalek MoarVM: Merge pull request #485 from samcv/get_property_int_a
22:51 dalek MoarVM:
23:14 jnthn samcv: That commit has commented out code left-behind: https://github.com/MoarVM/MoarVM/commit/d119127ece#diff-60b7c4ce4f16e0e46451e0610dbc21b3R285
23:14 jnthn lizmat: Pity you didn't catch that in code review. If merging MoarVM PRs, please read them very carefully. :-)
23:14 lizmat jnthn: ok, I figured that samcv could be working on this while you were aslpee
23:14 lizmat *asleep
23:15 lizmat will refrain from doing these types of MoarVM PR's in the future
23:16 timotimo asloop*
23:17 japhb .oO( Sloop, there it is ... )
23:17 jnthn In this case it's harmless in terms of effects on users, but in general the cost of hunting bugs that slip into Moar tends to be quite high compared to in NQP/Rakudo.
23:18 jnthn Including my own bugs. I sometimes wonder if we'd move slower, but smoother if we PR'd/reviewed everything non-trivial here.
23:19 jnthn (And then waste less time bug-hunting, so actually move at least as fast anyway in the long run, but with less annoyance downstream.)
23:24 dalek MoarVM: 1994c6e | jnthn++ | src/strings/normalize.c:
23:24 dalek MoarVM: Remove left-behind comment.
23:24 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/1994c6ee25
23:52 jnthn righty, sleep...hopefully...
23:52 jnthn o/

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary