Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-10-03

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 evalable6 joined #moarvm
01:05 evalable6 joined #moarvm
01:09 MasterDuke timotimo: doh, thanks. getting farther in stage parse now
01:10 samcv MasterDuke, can i see your patch so far?
01:12 MasterDuke samcv: https://github.com/MasterDuke17/MoarVM/tree/use_fsa_for_string_storage
01:13 Geth ¦ MoarVM: c5c389101d | (Samantha McVey)++ | .appveyor.yml
01:13 Geth ¦ MoarVM: Appveyor: diable VS2017 builds
01:13 Geth ¦ MoarVM:
01:13 Geth ¦ MoarVM: We can't reenable them until we know where SetEnv.cmd is, since
01:13 Geth ¦ MoarVM: nmake isn't in the path without running it.
01:13 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/c5c389101d
01:56 ilbot3 joined #moarvm
01:56 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
01:58 samcv yeah i did
01:58 samcv i didn't know it was that low, the binsize
01:58 samcv what is the size of it?
01:58 samcv i see MVM_FSA_BINS is 96. that can't be 96 bytes can it?
02:05 samcv i mean if it's really that small, then why don't we just use alloca in MVM_string_join?
02:05 samcv since we don't need it to persist
02:07 MasterDuke i don't remember the exact value, but a quick fprintf shows a byte size of 772 equals a bin of 96
02:07 samcv so you can only allocate up to 772 bytes before it just mallocs it then?
02:08 MasterDuke up to something about 772, don't know the exact value
02:10 MasterDuke 760 i think
02:12 geekosaur samcv, you might want to be careful with alloca anyway/. although on currently supported platforms up to 3k is probably safe
02:13 geekosaur (4k is asking for trouble if it causes multiple pages of stack segment to be allocated, and you need to be aware of anything else that also uses or allocates from stack)
02:14 samcv geekosaur, so you think i should lower it to 3000?
02:14 samcv i have it set to 4096 right now. i'm fine with lowering it
02:14 MasterDuke .tell timotimo down to only 12 errors in stage parse, all exactly the same (gist updated with an example)
02:14 yoleaux MasterDuke: I'll pass your message to timotimo.
02:15 geekosaur if you occasionally see segfaults in the code that uses it, you're running into the problem
02:15 samcv but 760 bytes we could just alloca join onto the stack since that's not very much and it'd be faster than FSA, since we don't need to keep it around
02:16 geekosaur (basically, whether something ounts as a stack allocation or a bad pointer is determined by an OS heuristic that can guess wrong, so allocating too much on stack can cause the next use of the stack (function calls, another allocation, etc.) to be mistaken for a bad pointer
02:16 MasterDuke samcv: for `pieces` in MVM_string_join?
02:16 samcv yeah
02:17 MasterDuke sure, give it a shot
02:17 geekosaur and this is determined by stack pages, so the only safe use is if it allocates exactly one more page to stack. page size is 4096 bytes on supported platforms
02:17 geekosaur basically,, you can try it, if you see occasional segfaults then back the size down and see if they go away
02:18 samcv well i haven't see any segfaults on this, or freebsd or on alpine with musl
02:18 samcv but i can back it down from 4096
02:19 zakharyas joined #moarvm
05:25 Geth ¦ MoarVM: 474ab7cdd1 | (Samantha McVey)++ | src/strings/ops.c
05:25 Geth ¦ MoarVM: In KMP index use malloc not FSA. Set max stack alloc to 3000
05:25 Geth ¦ MoarVM:
05:25 Geth ¦ MoarVM: Reduce stack alloc from 4096 to 3000, as recommended by
05:25 Geth ¦ MoarVM: geekosaur++. Use malloc instead of FSA because FSA would just
05:25 Geth ¦ MoarVM: malloc anyway since it's larger than the max FSA amount.
05:25 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/474ab7cdd1
05:35 samcv MasterDuke, if i bump MVM_FSA_BINS by 2 we don't overflow it during core setting compilation, well before it said 6.2GB max allocated
05:35 samcv after it showed 5.1
05:41 samcv increased it by 4x and it gets 1.5GB of mallocs
05:46 samcv seems to shave 2s off core setting compilation
06:21 domidumont joined #moarvm
06:24 domidumont joined #moarvm
06:31 domidumont joined #moarvm
06:53 brrt joined #moarvm
08:12 leont joined #moarvm
08:29 robertle_ joined #moarvm
09:13 jnthn samcv: What does it do to total memory use, though? Also, of memory use of perl6 -e ''
09:19 dogbert17 to get everyone started: http://dilbert.com/strip/2017-10-02
09:19 brrt joined #moarvm
09:19 nwc10 nice punchline in the 3rd panel
09:24 ilbot3 joined #moarvm
09:24 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
09:24 dogbert17 so the gains will be realized later
09:29 ilbot3 joined #moarvm
09:29 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
09:56 brrt joined #moarvm
10:10 lizmat joined #moarvm
10:34 brrt jnthn: thanks for reviewing!
10:54 timotimo jnthn: we could have a point where we allocate fewer buckets in the same page, for the very big pages
10:54 yoleaux 02:14Z <MasterDuke> timotimo: down to only 12 errors in stage parse, all exactly the same (gist updated with an example)
10:55 jnthn timotimo: We could, yeah, I pondered that before. It's probably sensible.
10:55 jnthn brrt: Welcome :-) Thanks for writing us a new JIT ;)
10:55 timotimo oh, the review is done? awesome!
10:55 timotimo brrt++
10:56 jnthn Yeah, provided brrt is happy for it to do so, it can go in :)
10:56 timotimo i really like the sound of that
10:57 * lizmat is looking forward to the final grant report  :-)
10:57 jnthn I semi-pondered "its the Star release this month" but...given how many far more risky things landed in Rakudo this month, I'd say expr JIT is some way down the risks list :)
10:59 timotimo yeah
10:59 timotimo i'll look into bigger bins getting smaller :)
10:59 jnthn I suspect if I work on anything else in Rakudo ahead of this month's release, it'll be hyper/race, which are so broken at the moment anyway I almost can't make it worse :)
10:59 timotimo i hear ya :|
11:00 jnthn But yeah, overall let's try and be a bit cautious about what we put in over the week leading up to the next bunch of releases
11:00 jnthn The Star ones do get wider use
11:01 timotimo we still have one and a half weeks or so, right?
11:02 jnthn Yeah, indeed
11:03 timotimo could even give the fsa logic to make the first page for a given bin smaller than all the rest
11:03 timotimo to better handle cases like "bin 91 gets five items ever, but 90 and 92 get hundreds"
11:04 timotimo we have setup_bin and add_page, so the distinction is already there in code
11:04 timotimo oh, no, not quite
11:04 timotimo oh, no, it does add a page
11:05 timotimo a very first page
11:18 timotimo i made it limit page sizes to 32kbyte, which means page 95 has 42 elements
11:18 timotimo and i might make the size limit for the very first page half or quarter that
11:19 jnthn OK
11:19 jnthn Let's try that :)
11:19 timotimo let's see, sam wanted to bump the count by 4 ... or really 4x?
11:20 timotimo oh wow
11:21 timotimo for perl6 -e i now get a good view of which sizes get how many pages allocated
11:21 domidumont joined #moarvm
11:21 timotimo a whole lot of 'em only get the initial page, even though i quartered its size
11:23 timotimo hm, the maxresident difference is rather small it seems
11:23 timotimo hm, it looks like i might actually be using more memory
11:24 timotimo nope, there was some other difference in my measurements
11:25 timotimo only like 80k on -e ''
11:27 timotimo the difference is more pronounced for -e 'say "hi"'
11:27 jnthn Save 80k?
11:27 nwc10 at a guess, is that because -e '' doesn't even allocate some bucket sizes?
11:27 jnthn Yeah, probably that
11:27 timotimo m: say 75236 / 75458
11:27 camelia rakudo-moar e2f8a5: OUTPUT: «0.997058␤»
11:27 timotimo m: say 75236 - 75458
11:27 jnthn ooh, lunch time :)
11:27 camelia rakudo-moar e2f8a5: OUTPUT: «-222␤»
11:27 timotimo about 200k saved in that scenario
11:29 timotimo let's do a real measurement: core setting compilation
11:43 AlexDaniel joined #moarvm
11:45 AlexDaniel squashable6: status
11:45 squashable6 AlexDaniel, ⚠🍕 Next SQUASHathon in 2 days and ≈22 hours (2017-10-07 UTC-12⌁UTC+14). See https://github.com/rakudo/rakudo/wiki/Monthly-Bug-Squash-Day
11:50 zakharyas joined #moarvm
11:50 timotimo looks like it actually gets worse from my changes
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: 15ba542eea | (Timo Paulssen)++ | src/core/fixedsizealloc.c
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: limit FSA pages to 32k (8k for very first page)
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes:
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: helps perl6 -e '' and -e 'say "hi"' a lot, but seems
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: to actually increase memory usage in a core setting
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: compilation.
11:52 Geth ¦ MoarVM/fsa_tune_page_sizes: review: https://github.com/MoarVM/MoarVM/commit/15ba542eea
11:54 timotimo though of course the results will be different still with more stuff using the fsa
11:54 lizmat jnthn timotimo: could you give me a sanity check wrt to BUILDALL?
11:54 lizmat if a class has an empty BUILDPLAN (like a mixin that doesn't add any attributes)
11:55 lizmat doesn't that imply I don't need to generate a BUILDALL for that class, as its first parent will have the correct BUILDALL already generated ?
11:55 lizmat the invocant signature might not be 100% correct, but still valid anyway
11:57 timotimo hm, that does sound sensible
11:58 timotimo maybe i want something a little bit faster than the core setting compilation for measuring this :|
12:03 jnthn lizmat: If it has no attributes, then yeah, that sounds reasonable
12:04 lizmat that would save quite a few generated methods, as its quite common to only mixin methods, and not attributes
12:04 timotimo good point
12:10 AlexDaniel joined #moarvm
12:17 timotimo 1342402 is the average before
12:17 timotimo 1341298 is the average after
12:18 timotimo "1338332".."1343264"
12:18 timotimo that's the minmax of "after"
12:18 lizmat 1100 bytes difference ?
12:19 timotimo "1339700".."1343508"
12:19 timotimo that's the minmax of before
12:19 timotimo that's kbytes
12:19 lizmat ah, so ~1MB difference
12:20 lizmat or a bit more...
12:20 timotimo m: say 1338332 - 1343264; say 1339700 - 1343508
12:20 camelia rakudo-moar 0dd6af: OUTPUT: «-4932␤-3808␤»
12:21 timotimo that's how noisy the measurement is
12:22 timotimo the measurements for "with my patch" are further apart
12:22 timotimo i'm not sure if it makes sense to use these measurements, given the amount of noise is ~4x the difference
12:27 Geth ¦ MoarVM/even-moar-jit: dc3d40fb22 | (Bart Wiegmans)++ | 4 files
12:27 Geth ¦ MoarVM/even-moar-jit: Improve the expression JIT documentation
12:27 Geth ¦ MoarVM/even-moar-jit:
12:27 Geth ¦ MoarVM/even-moar-jit: Add a document describing its most important components (expression
12:27 Geth ¦ MoarVM/even-moar-jit: template processor / tree builder, tiler, and register allocator).
12:27 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/dc3d40fb22
12:27 Geth ¦ MoarVM/even-moar-jit: 6de7455e9a | (Bart Wiegmans)++ | 2 files
12:27 Geth ¦ MoarVM/even-moar-jit: More documentation fixes
12:27 Geth ¦ MoarVM/even-moar-jit:
12:27 Geth ¦ MoarVM/even-moar-jit: Some of the things in tiles.md were no longer true
12:27 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/6de7455e9a
12:28 Geth ¦ MoarVM/even-moar-jit: 7f7ce9ca40 | (Bart Wiegmans)++ | 3 files
12:28 Geth ¦ MoarVM/even-moar-jit: ^cu_string - is lazy-loaded so use wrapper
12:28 Geth ¦ MoarVM/even-moar-jit:
12:28 Geth ¦ MoarVM/even-moar-jit: The direct access of MVMCompUnit->body.strings was a legacy from
12:28 Geth ¦ MoarVM/even-moar-jit: simpler days when compunit strings were loaded eagerly. As they're now
12:28 Geth ¦ MoarVM/even-moar-jit: using lazy loading, that isn't really valid anymore.
12:28 Geth ¦ MoarVM/even-moar-jit:
12:28 Geth ¦ MoarVM/even-moar-jit: Possible future development would be to force eager loading during JIT
12:28 Geth ¦ MoarVM/even-moar-jit: compilation and/or upgrading to second-generation memory.
12:28 brrt oh, oops, we're seeing a segv
12:28 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/7f7ce9ca40
12:28 AlexDaniel joined #moarvm
12:31 Geth ¦ MoarVM/even-moar-jit: 33270f003d | (Bart Wiegmans)++ | src/jit/macro.expr
12:31 Geth ¦ MoarVM/even-moar-jit: MVM_cu_string - second argument is *cu
12:31 Geth ¦ MoarVM/even-moar-jit:
12:31 Geth ¦ MoarVM/even-moar-jit: Not idx, oops
12:31 Geth ¦ MoarVM/even-moar-jit: review: https://github.com/MoarVM/MoarVM/commit/33270f003d
12:40 timotimo i wonder what the main source of nondeterminism in core setting compilation is, the one that causes memory usage to vary so drastically
12:41 timotimo could it just be spesh?
12:45 brrt tbh i don't find spesh to be very nondeterministic
12:46 timotimo core setting compilation doesn't start any other threads, so it also won't do the logs it gets in different orders every time
12:50 lizmat random hash order ?
12:50 timotimo hm, how often do we iterate over hashes in compilation i wonder
12:51 lizmat I have no idea :-)
12:55 brrt i expect quite often
12:59 timotimo but we don't randomize hashes yet, do we?
13:03 lizmat timotimo: not sure
14:26 zakharyas joined #moarvm
14:43 AlexDaniel joined #moarvm
14:56 * timotimo is idly playing around with systemtap
14:57 timotimo oh geez
14:57 timotimo tried to record stack traces for every fsa_alloc hit
14:57 timotimo it's filling up my disk good
14:58 timotimo i can't stop it o_O
14:59 timotimo now it did stop and the resulting file is apparently b0rken
15:02 * timotimo frees up some disk space ...
15:06 brrt joined #moarvm
15:28 brrt i'm investigating three spectest failures
15:29 brrt t/spec/S29-os/system.rakudo.moar test 35
15:29 brrt t/spec/S28-named-variables/init-instant.t
15:29 brrt t/spec/S17-supply/watch-path.t
15:30 brrt they are individually succesfull
15:30 brrt so, any objections to me pushing the merge button? :-)
15:31 Zoffix \o/ \o/ \o/
15:31 jnthn I've seen the first happen for a while
15:31 jnthn Not every time, but now and then
15:31 jnthn Others I dunno about
15:31 jnthn But...yeah, let's merge it
15:31 timotimo yeeeeaaahhhh
15:31 jnthn Sounds like timing or other issues if they work outside of harness
15:33 timotimo a sizable portion of allocations is from the nfa (in perl6 -e 'say "hi"')
15:34 timotimo almost a quarter
15:34 timotimo but that's only "how many times is the allocator called", it ignores the size of each allocation
15:34 timotimo and i think it skipped some events because I/O was too slow
15:34 jnthn I already reduced its allocations by a good bit before, I think
15:35 jnthn iirc the remaining one is the result
15:35 jnthn Which we need to hand back, and might live on for a while
15:36 timotimo yeah, it's not bad
15:36 timotimo the code, i mean
15:36 timotimo just an observation
15:36 timotimo another big chunk is from hash entries
15:38 brrt thank you everybody for your incredible patience
15:38 timotimo not sure i can get much sensible information out of this any more
15:39 timotimo but it was good to refresh my memory of how the systemtap portion of perf works
15:40 brrt also, geth is dead, maybe
15:40 jnthn Aww, no commit report
15:40 jnthn brrt++ though
15:40 jnthn Already built it here :)
15:40 brrt cool :-)
15:40 * brrt hoping for the best
15:42 [Coke] did it merge to master? (do I need to build rakudo with nqpmaster and moarmaster?)
15:43 jnthn [Coke]: To MoarVM master, yes. No version bumps as yet
15:44 [Coke] building on win & mac..
15:49 [Coke] (src\profiler\heapsnapshot.c(823): warning C4293: '<<': shift count negative or too big, undefined behavior)
15:50 Zoffix wow
15:50 Zoffix brrt++
15:50 Zoffix \o/
15:51 brrt i'll take a look, although that is not part of the branch :-)
15:52 [Coke] brrt: that was not meant for you specifically. ;)
15:54 brrt [Coke] what platform are you on?
15:54 brrt 32 bit by any chance?
15:54 brrt seems like your long is not long enough :-)
15:54 brrt anyway, rather than 1l, probably better to write UINT64_C(1)
15:54 brrt which is i think stdint defined
15:54 [Coke] brrt: that is on a win64 (I think) win 10 vm.
15:55 brrt hmm, that's somewhat surprising
15:55 [Coke] 64-bit, x64
15:55 brrt i think i may have seen that on the JIT as well on windows
15:56 [Coke] using ms vs Community 2017
15:56 brrt anyway, can you pls check if this helps: https://gist.github.com/bdw/cb8ecce419ec0a01db1fab2e883b5dc9
15:56 brrt i have a VM as well, i just don't feel like starting it up :-)
15:56 zakharyas joined #moarvm
15:56 [Coke] brrt: let me finish the as-is build first, then will re-try that.
15:57 brrt also, i will *not* be online for the coming afternoon / night, so, if any problems arise, i can't respond; if trouble is severe, you know where the revert button is
15:57 brrt i don't expect it very much, but just so you know :-)
15:58 [Coke] spectest clean on mac.
15:58 brrt \o/
15:59 [Coke] nativecall cpp tests still failing on win 10 `nmake test`, no change there. kicking of win10 spectest.
16:01 [Coke] (as I recall, we have a lot of win failures atm so I'm not sure this will show anything. :|
16:02 [Coke] if `perl6 -V | grep -i jit` has moar::jit_arch set to a value, does that mean I have a jit?
16:06 [Coke] (or is there a better way to tell?)
16:06 bartolin it would be great if someone could take a look at https://github.com/MoarVM/MoarVM/pull/714 . currently opening a socket to a remote host is broken on freebsd.
16:10 jnthn hm, thought i already reviewed/merged that one...
16:10 Geth ¦ MoarVM: d04c8dccbc | usev6++ | src/io/syncsocket.c
16:10 Geth ¦ MoarVM: Fix getaddrinfo failing with EAI_HINTS on FreeBSD
16:10 Geth ¦ MoarVM:
16:10 Geth ¦ MoarVM: This fixes spectest failures (e.g. in S32-io/IO-Socket-INET.t) on FreeBSD.
16:10 Geth ¦ MoarVM:
16:10 Geth ¦ MoarVM: According to the man page of getaddrinfo() '[a]ll other elements of the
16:10 Geth ¦ MoarVM: addrinfo structure passed via hints must be zero or the null pointer'
16:10 Geth ¦ MoarVM: (similiar wording on Linux). This requirement is actually enforced on
16:10 Geth ¦ MoarVM: FreeBSD:
16:10 Geth ¦ MoarVM:
16:10 Geth ¦ MoarVM:   https://github.com/freebsd/freebsd/blob/89c48e12204df05d7db296d3ae5d2fd002b4b510/lib/libc/net/getaddrinfo.c#L429
16:10 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d04c8dccbc
16:10 Geth ¦ MoarVM: 3da7cce276 | (Jonathan Worthington)++ (committed using GitHub Web editor) | src/io/syncsocket.c
16:10 Geth ¦ MoarVM: Merge pull request #714 from usev6/getaddrinfo_hints
16:10 Geth ¦ MoarVM:
16:10 Geth ¦ MoarVM: Fix getaddrinfo failing with EAI_HINTS on FreeBSD
16:10 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/3da7cce276
16:11 Geth ¦ MoarVM: 23c16b3031 | (Patrick Sebastian Zimmermann)++ | 2 files
16:11 Geth ¦ MoarVM: Probe for gcc -Werror=* support. This allows building MoarVM on older GCCs.
16:11 Geth ¦ MoarVM:
16:11 Geth ¦ MoarVM: The -Werror=* probe has to run before setting the cc/ldmiscflags. The
16:11 Geth ¦ MoarVM: compiler_usability check only makes sense before the -Werror=* probe. Thus
16:11 Geth ¦ MoarVM: that one is also moved earlier.
16:11 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/23c16b3031
16:11 Geth ¦ MoarVM: 48f5efaaf9 | (Jonathan Worthington)++ (committed using GitHub Web editor) | 2 files
16:11 Geth ¦ MoarVM: Merge pull request #696 from patzim/master
16:11 Geth ¦ MoarVM:
16:11 Geth ¦ MoarVM: Probe for gcc -Werror=* support
16:11 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/48f5efaaf9
16:12 bartolin thanks anyway :-)
16:13 Geth ¦ MoarVM/master: 9 commits pushed by Mario++, M++, (Jonathan Worthington)++
16:13 Geth ¦ MoarVM/master: 44e3c12053 | getsockname / end of non-void function
16:13 Geth ¦ MoarVM/master: 36b17b01b1 | throw_error with no_return
16:13 Geth ¦ MoarVM/master: d478fa456d | no need for the ignore-no-return-pragma
16:13 Geth ¦ MoarVM/master: 8dcbde02f2 | #include <ws2tcpip.h>
16:13 Geth ¦ MoarVM/master: bb15348138 | typo
16:13 Geth ¦ MoarVM/master: 24eeb3683c | per request, throw_error removed from header file.
16:13 Geth ¦ MoarVM/master: aae496920f | pre request
16:13 Geth ¦ MoarVM/master: 35424eb7d9 | throw_error removed from header
16:13 Geth ¦ MoarVM/master: a4fef0bd36 | Merge pull request #692 from duke-m/patch-2
16:13 Geth ¦ MoarVM/master: review: https://github.com/MoarVM/MoarVM/compare/48f5efaaf9...a4fef0bd36
16:42 [Coke] looks like win64 spectest just hung with 3 remaining tests.
16:45 travis-ci joined #moarvm
16:45 travis-ci MoarVM build canceled. Jonathan Worthington 'Merge pull request #696 from patzim/master
16:45 travis-ci https://travis-ci.org/MoarVM/MoarVM/builds/282804503 https://github.com/MoarVM/MoarVM/compare/3da7cce276df...48f5efaaf92a
16:45 travis-ci left #moarvm
16:55 lizmat joined #moarvm
17:20 domidumont joined #moarvm
17:47 * Zoffix stresstests and version bumps
17:50 Zoffix "2017.09.1-553-ga4fef0b" that's quite a high number :D
18:03 travis-ci joined #moarvm
18:03 travis-ci MoarVM build passed. Jonathan Worthington 'Merge pull request #692 from duke-m/patch-2
18:03 travis-ci https://travis-ci.org/MoarVM/MoarVM/builds/282805138 https://github.com/MoarVM/MoarVM/compare/48f5efaaf92a...a4fef0bd36cc
18:03 travis-ci left #moarvm
18:20 AlexDaniel joined #moarvm
19:09 brrt joined #moarvm
19:21 zakharyas joined #moarvm
19:30 samcv jnthn, 4x is max 65.5MB of memory usage. compared to 56.7MB
19:34 timotimo samcv: is that actually going from ~90 bins to ~360?
19:34 samcv yeah
19:34 samcv 96*4
19:34 samcv and with 4x we only have 1.2GB alloced with FSA instead of 6.2GB
19:35 timotimo wow
19:35 samcv for core setting compilation. doing 2x, we have 5.1GB malloced in FSA
19:35 samcv sorry i should say malloced, not alloced
19:35 jnthn samcv: Remind what you're tweaking this for? :)
19:35 samcv also core setting is 2s faster
19:35 samcv reducing the mallocs
19:35 samcv that the FSA has to do
19:36 samcv it seems to make  many GB more than is really needed
19:36 samcv optimally
19:37 jnthn Yeah, but I thought you had a particular use case that 4x covers?
19:37 samcv i was recording the setting compilation
19:37 jnthn Ah, OK
19:38 jnthn I thought it was something about a KMP table fitting in or something
19:38 samcv from 6.2GB malloced to 1.2GB
19:38 samcv nah
19:39 jnthn ah, OK
19:39 jnthn But we gain 10MB extra memory use for doing nothing?
19:39 timotimo did i already push my tuning branch for the fsa page size?
19:39 timotimo it would probably want a minimum items per page, too
19:40 timotimo otherwise it'll go down to 0 items per page, or 1
19:42 timotimo also, perhaps we should make bigger steps after a given page size
19:45 timotimo item size i mean
19:48 jnthn Yeah, possibly that also
19:49 samcv jnthn, yep we gain 10MB extra memory doing nothing
19:50 timotimo samcv: check the fsa_tune_page_sizes branch
19:50 timotimo give it a minimum "effective_item_count"
19:50 timotimo then let's see if it helps at all
19:50 samcv how do i get a branch i don't have
19:50 timotimo it should be enough to, after "git fetch", just git checkout fsa_tune_page_sizes
19:51 timotimo it ought to set it up to track origin/that_branch_name for you
19:51 samcv that does not work
19:51 samcv oh i got it now
19:51 jnthn samcv: OK, then further tweakery needed to try and have our cake and eat it, IMO
19:51 jnthn I suspect something along the lines of what timotimo has looked at may do it
19:52 samcv and timotimo's branch has 52.8MB peak. so down 4mb
19:52 samcv will try the setting now
19:52 timotimo i tried measuring the core setting
19:53 timotimo and the difference between two runs was enormous
19:53 samcv you mean two runs of the same branch?
19:54 timotimo yes
19:54 samcv interesting...
19:55 timotimo so you might have to be extra careful in your measurements, too
19:57 samcv so i got 1.2GB peak. which is similar to having the bin 4x
19:57 samcv so that seems good
19:57 samcv err wait. no it's 1.6GB
19:57 samcv i was looking at the peak. 1.6G is the amount allocated by the FSA in total
19:57 timotimo you're using heaptrack for this?
19:59 samcv err no. the amonut allocated by malloc
19:59 samcv my bad
19:59 samcv but that's much closer to the 1.2Gb i got :)
20:00 samcv yeah
20:00 samcv i'm looking at the total amount that was malloc'd and comparing it. though i should also compare the peak usage as well
20:04 leont_ joined #moarvm
20:40 lizmat joined #moarvm
20:46 timotimo i wonder if we'd benefit from a "pre-size this hash for n elements" operation that we can call in the deserialization code
20:49 MasterDuke timotimo: did you try with MVM_SPESH_BLOCKING=1? did that reduce the variability?
20:49 timotimo i did not
21:03 MasterDuke oh, and how were you measuring?
21:03 timotimo just "time" on the commandline
21:05 MasterDuke nice and simple (and doesn't slow the build!)
21:31 samcv i'm going to case the switches for 8bit and ascii strings being joined (flat)
21:32 samcv tests show 2.4x speed improvement joining very long 8bit strings
21:32 samcv (that are flat)
21:33 lizmat m: use experimental :collation; $*COLLATION.set(:!tertiary); dd "a" coll "A"  # samcv: is that correct ?
21:33 camelia rakudo-moar 98fae3: OUTPUT: «Order::More␤»
21:33 zakharyas joined #moarvm
21:33 samcv ah. quaternary needs to be disabled
21:33 samcv that breaks ties by codepoint
21:34 samcv m: use experimental :collation; $*COLLATION.set(:!tertiary, :!quaternary); dd "a" coll "A"
21:34 camelia rakudo-moar 98fae3: OUTPUT: «Order::More␤»
21:34 samcv err maybe i spelled it wrong
21:34 lizmat yeah  :-)
21:34 samcv m: use experimental :collation; $*COLLATION.set(:!tertiary, :!quaternary); dd "a" coll "A"
21:34 camelia rakudo-moar 98fae3: OUTPUT: «Order::More␤»
21:34 samcv m: use experimental :collation; $*COLLATION.set(:!tertiary, :!quaternary); say $*COLLATION
21:34 camelia rakudo-moar 98fae3: OUTPUT: «collation-level => 5, Country => International, Language => None, primary => 1, secondary => 1, tertiary => 0, quaternary => 0␤»
21:35 samcv 5
21:35 samcv that sounds right
21:35 lizmat 5?
21:35 samcv hm
21:35 lizmat but More should be Same, right?
21:36 samcv yeah 5 is right. but More should be same when quaternary is removed
21:36 samcv looks like a bug. will look at it in about an hour when i get back
21:36 lizmat shall I rakudobug it ?
21:36 samcv yeah
21:36 lizmat oki
21:40 lizmat samcv: RT #132216
21:40 synopsebot RT#132216 [new]: https://rt.perl.org/Ticket/Display.html?id=132216 'a' coll 'A" not Same but More
21:59 Geth ¦ MoarVM: d5db8486bb | (Timo Paulssen)++ | src/spesh/stats.c
21:59 Geth ¦ MoarVM: skip stats for frames beyond spesh max bytecode size
21:59 Geth ¦ MoarVM:
21:59 Geth ¦ MoarVM: an example file with a gigantic mainline - a huge hash
21:59 Geth ¦ MoarVM: literal - spent more than 95% of its time inside by_offset
21:59 Geth ¦ MoarVM: for the benefit of a frame that was going to be ignored by
21:59 Geth ¦ MoarVM: the planner anyway. This makes it as fast as running without
21:59 Geth ¦ MoarVM: spesh.
21:59 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d5db8486bb
22:00 jnthn Hm, though that happens on another thread...
22:00 jnthn Nice catch, though :-)
22:00 timotimo it does
22:01 jnthn Though I wonder
22:01 timotimo but the regular threads waits for spesh to reach its gc sync point
22:01 jnthn Ah
22:01 jnthn Maybe we should not log such huge frames at all..hmm
22:01 timotimo that'd be a check inside the instructions that do logging
22:01 jnthn Though we'd have to look at the bytecode size on frame entry when we're logging
22:01 timotimo yes
22:01 jnthn No, I was thinking of doing it in MVM_frame_invoke
22:01 jnthn If we don't give it an entry record or correlation ID
22:02 jnthn Then it won't log anything for it
22:02 jnthn And given we'll never spesh it, that's fine
22:02 timotimo oh, that would prevent all ops from logging with a check that's already in place anyway
22:02 jnthn Yeah
22:02 jnthn So then we'd not even write into the log
22:02 timotimo yes, that'd increase spesh efficiency, too
22:03 jnthn aye
22:03 jnthn May help some of the giant test files too :)
22:03 timotimo it could very well!
22:03 jnthn Was the giant file with a hash in actually constructing it in the module mainline?
22:04 jnthn If it's all constant data, sticking a BEGIN in front of it would mean we make it at compile time and serialize it, which'd be faster still :)
22:05 timotimo yup, it's just "our %systems = ( ... )"
22:05 timotimo but you're right
22:05 timotimo i hadn't thought of that
22:06 timotimo a bit shameful that we use about a gig maxrss to compile this beast
22:06 timotimo i'll put "constant" in front, that should have the same effect
22:06 jnthn Yeah
22:07 jnthn Then it should load faster still
22:07 timotimo i'll have numbers in a mo'
22:08 timotimo ah, snap
22:08 timotimo it then has to be %( ) rather than just ( )
22:10 timotimo wow, yeah
22:10 timotimo that's much, much faster
22:11 timotimo from about 0.88 down to about 0.26
22:16 timotimo now i'm checking if the association id thing works
22:17 timotimo yeah, it has the speed increase effect
22:17 timotimo i'll run "make test" and "make spectest" this time, though
22:17 MasterDuke association id?
22:18 timotimo yeah, in order to do spesh logging we give every frame an ID
22:18 timotimo if a frame has no ID, no logging can take place
22:18 MasterDuke ah
22:20 timotimo that's not quite accurate
22:21 timotimo it also interplays with the simulated stack and such
22:21 timotimo i.e. spesh recreates what it thinks the callstack looked like when the log was created, so we don't have to do too complicated computations when creating the individual log entries
22:25 MasterDuke and we can just skip all that if the frame size is too big?
22:25 timotimo aye
22:25 timotimo jnthn: you think this can have bad effects if a huge frame comes between two frames that get speshed?
22:27 Geth ¦ MoarVM: d0646fafb9 | (Timo Paulssen)++ | 2 files
22:27 Geth ¦ MoarVM: don't even generate log entries for huge frames
22:27 Geth ¦ MoarVM:
22:27 Geth ¦ MoarVM: not giving a frame a correlation ID prevents any logging
22:27 Geth ¦ MoarVM: from taking place, the logs are not filled with useless
22:27 Geth ¦ MoarVM: data, fewer runs for the spesh worker to do.
22:27 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/d0646fafb9
22:28 timotimo wouldn't have been able to come up with this as fast if not for MasterDuke being my rubber duck :)
22:28 jnthn timotimo: It'll cope
22:28 jnthn The worst that'd happen is it infers something wrong from the stats and sticks a guard in that always fails, but that would be quite unlikely
22:29 MasterDuke "those who can't do, duck"?
22:29 timotimo haha
22:30 timotimo hm, lock-async seems to be hanging, but i think it also hangs for others
22:30 timotimo nope, it finished
22:30 timotimo just took a while
22:32 jnthn That one's odd; on my office machine it always completes fast. In my VM at home it often does, then occasinally goes super slow.
22:32 jnthn Always completes eventually though
22:40 bloatable6 joined #moarvm
22:45 buggable joined #moarvm
22:57 arnsholt joined #moarvm

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary