Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2016-11-18

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:14 xtt joined #moarvm
02:25 dalek MoarVM/smaller_string_storage: ec76e75 | timotimo++ | src/strings/latin1.c:
02:25 dalek MoarVM/smaller_string_storage: latin1 strings may be stored in 8bits sometimes
02:25 dalek MoarVM/smaller_string_storage:
02:25 dalek MoarVM/smaller_string_storage: if their graphemes are strictly ASCII and the synthetics
02:25 dalek MoarVM/smaller_string_storage: don't go out of range.
02:25 dalek MoarVM/smaller_string_storage: review: https://github.com/MoarVM/MoarVM/commit/ec76e75f84
02:25 dalek MoarVM/smaller_string_storage: 4fd3220 | timotimo++ | src/strings/utf8.c:
02:25 dalek MoarVM/smaller_string_storage: some utf8-encoded strings also fit into 8bits.
02:25 dalek MoarVM/smaller_string_storage: review: https://github.com/MoarVM/MoarVM/commit/4fd3220daa
02:27 timotimo looking into what the best way is to make indexingoptimized also able to have 8bit storage
02:27 timotimo (because if you can't sleep, might as well try to be productive)
02:48 ilbot3 joined #moarvm
02:48 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
03:14 timotimo Unhandled exception: Error encoding UTF-8 string: could not encode codepoint 1918989427
03:14 timotimo uh oh, what have i done
03:18 geekosaur forgotten to handle your 8-bits somewhere
03:19 geekosaur that's either "rats" or "star"
03:19 timotimo oh, you were kind enough to check the value as a char :)
03:19 timotimo that's cool
03:20 geekosaur well, did it to hex and I know most of the 20-7e from long experience :)
03:21 geekosaur (i.e. staring at way too many hex dumps :p )
03:21 timotimo ah
03:21 timotimo i forgot to turn the storage type into the other one when i do my conversion
03:22 timotimo i made a short function to do it so i don't have to retype the conversion over and over
03:22 timotimo and that function ended up buggy %)
03:24 timotimo i get through a rakudo build now :)
03:33 timotimo hm. with perl6 -e '' i get that function run 24 times
03:34 timotimo for the strings start, parse, ast, mast, mbc, moar ... over and over
03:36 timotimo that's from NQPHLL's my $compiler := HLL::Compiler.new()
03:37 timotimo the way it sets up its stages is by splitting a string by space
03:39 timotimo what i don't get is why it won't give me the "compression" output for cmdoptions ... perhaps nqp's compiler actually has constant folding for that?!
04:16 dalek MoarVM/smaller_string_storage: 79a161f | timotimo++ | src/strings/utf8.c:
04:16 dalek MoarVM/smaller_string_storage: kick out debug output for utf8 encoding
04:16 dalek MoarVM/smaller_string_storage: review: https://github.com/MoarVM/MoarVM/commit/79a161f9d7
04:16 dalek MoarVM/smaller_string_storage: ddf58d7 | timotimo++ | src/strings/ops.c:
04:16 dalek MoarVM/smaller_string_storage: teach a few str functions to compress results into 8bit
04:16 dalek MoarVM/smaller_string_storage:
04:16 dalek MoarVM/smaller_string_storage: collapse_strands, substring of complicated stranded strings,
04:16 dalek MoarVM/smaller_string_storage: string_indexing_optimized, string_escape, string_flip,
04:16 dalek MoarVM/smaller_string_storage: string_chr.
04:16 dalek MoarVM/smaller_string_storage: review: https://github.com/MoarVM/MoarVM/commit/ddf58d7fc4
04:27 timotimo since it's spectest clean, i'm thinking of merging the branch
04:57 timotimo jesus, sharing the string memory buffer with the mmapped compunit is a noticable memory save
04:58 timotimo like half a megabyte for perl6 -e ''
05:05 mst joined #moarvm
05:27 dalek MoarVM/share_string_mem_with_compunit: 7d31a88 | timotimo++ | src/ (4 files):
05:27 dalek MoarVM/share_string_mem_with_compunit: share some string's memory with compunit memory
05:27 dalek MoarVM/share_string_mem_with_compunit:
05:27 dalek MoarVM/share_string_mem_with_compunit: if the string itself is already latin1, and it doesn't contain
05:27 dalek MoarVM/share_string_mem_with_compunit: any crlf, then the memory will be 1:1 the same between what we
05:27 dalek MoarVM/share_string_mem_with_compunit: decoded and what's in the compunit string heap blob.
05:27 dalek MoarVM/share_string_mem_with_compunit:
05:27 dalek MoarVM/share_string_mem_with_compunit: we then make sure to un-share memory when the compunit dies.
05:27 dalek MoarVM/share_string_mem_with_compunit: review: https://github.com/MoarVM/MoarVM/commit/7d31a884f8
05:28 timotimo this doesn't seem quite as safe as the other parts in the branch, but it can probably go into master, too. perhaps after the release?
05:28 timotimo could be helpful to have an ASAN-enabled spec test run for this
05:28 * timotimo tries to catch up on sleep missed tonight
05:30 xtt joined #moarvm
05:34 timotimo 11407 strings got into the latin1 memory sharing thing for 104794 bytes in total, where the average size was 9.18 bytes
05:35 timotimo 3 6 and 12 are the 25%, 50% and 75% percentiles
06:24 domidumont joined #moarvm
06:57 domidumont joined #moarvm
07:01 domidumont joined #moarvm
07:02 nwc10 timotimo: SEGV http://paste.scsys.co.uk/539798
07:04 domidumont joined #moarvm
07:15 domidumont joined #moarvm
08:35 zakharyas joined #moarvm
08:46 mtj_ joined #moarvm
09:28 FROGGS joined #moarvm
10:47 brrt joined #moarvm
12:46 timotimo cool, thanks, nwc10
12:47 timotimo i can imagine what went wrong there, heh.
12:50 timotimo not sure why it didn't asploded on my end, though
12:53 timotimo once we have weak references implemented, strings that haven't been needed from the string heap can also be cleaned up
12:53 timotimo i'm imagining since we usually have a compilation stage and a run stage in our programs, that could be helpful
12:56 dalek MoarVM/share_string_mem_with_compunit: 530f3cf | timotimo++ | src/6model/reprs/MVMCompUnit.c:
12:56 dalek MoarVM/share_string_mem_with_compunit: prevent access to un-deserialized strings in gc_free
12:56 dalek MoarVM/share_string_mem_with_compunit: review: https://github.com/MoarVM/MoarVM/commit/530f3cfdf5
13:29 timotimo spec tests are clean with that btw
13:42 lizmat still, it feels wise to merge this after the release
13:43 timotimo sure, no problem
13:43 lizmat many things aren't covered yet by spectests  (as I've found out the hard way)
13:50 lizmat that's why I'm not going to commit anything important until after the release  :-)
15:04 ggoebel joined #moarvm
15:07 [Coke] this is why I don't like our single threaded nom-as-mainline-dev-and-release-branch strategy. Do wish we were taking more advantage of branching.
15:08 * jnthn has had stuff in branches and merged it post-release before
15:08 timotimo right, we do that all the time
15:08 timotimo especially me, who is known for committing dangerous, broken, and plain low-grade code :s
15:08 jnthn I see it happen more in MoarVM than Rakudo, to be fair.
15:09 jnthn timotimo: You're smart enough to do it in a branch most of the time though :P :P
15:09 timotimo right
15:09 jnthn I still kinda like the automated delivery to release branch idea.
15:10 timotimo yeah, we ought to build that at some point
15:10 jnthn That is, we have a CD-style thing that every so often picks up the mainline development branch, stresstests it, tests a bunch of modules on it, and if all is well fast-forwards a release-candidate branch to match it.
15:11 jnthn And releases are cut from the latter.
15:16 lizmat fwiw, my experiences with branches isn't that good
15:16 lizmat not from a technical point of view, but from a code review / community point of view  :-(
15:17 timotimo right, people end up not looking at it for months ...
15:17 timotimo the person who initiated the branch gets discouraged
15:18 jnthn lizmat: Doesn't stop you committing stuff you're working on in a branch just so you can keep working "as normal" on things whlie a release happens, and merging it after.
15:18 lizmat jnthn: git stash is my friend  :-)
15:19 timotimo mine, too
15:19 jnthn Same, though I use it for different workflows than branches :)
15:19 timotimo right
15:19 timotimo i often use it to transplant changes i haven't committed yet across branches
15:20 timotimo because just checking stuff out, or "git pull" or something might complain
15:20 timotimo and especially git rebase. git rebase really doesn't want unstaged changes in your working copy
15:20 jnthn So, Rakudo release is this weekend?
15:21 lizmat I think that's the plan, afaik
15:21 jnthn OK, then I'll finish up my working week by doing the MoarVM release, so I don't have to worry about it (or block the Rakudo release) during the weekend. :)
15:21 lizmat jnthn++
15:35 jnthn Wow, I'd missed that 96d74e0c80bc6a happened
15:35 jnthn timotimo++
15:41 dalek MoarVM: 1e2d7b4 | jnthn++ | docs/ChangeLog:
15:41 dalek MoarVM: Changelog entries for 2016.11.
15:41 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/1e2d7b42bd
15:42 dalek MoarVM: 21193c5 | jnthn++ | VERSION:
15:42 dalek MoarVM: Bump VERSION.
15:42 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/21193c5250
15:47 timotimo oh, that
15:47 timotimo yeah, that was cool :)
15:48 dalek MoarVM: 81fd0ac | jnthn++ | docs/ChangeLog:
15:48 dalek MoarVM: Missed a Changelog entry.
15:48 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/81fd0ac47a
15:48 jnthn timotimo: What does it actually mean, though? Can valgrind catch overruns *within* FSA-allocated things?
15:49 jnthn Or better put: can it catch overruns of memory allocated using the FSA?
15:49 timotimo yes
15:49 timotimo i've added support for a redzone
15:49 timotimo that is exactly that
15:49 jnthn Nice
15:50 timotimo one thing we're missing is a long queue for freeing stuff to more agressively trigger use-after-free
15:50 timotimo i.e. keep 10k allocated bits around before actually starting to free stuff
15:50 jnthn I normally flipped the FSA into the debug mode (just uses malloc and records the size for verification)
15:50 jnthn To use valgrind
15:50 jnthn :)
15:51 jnthn Well, when I was bug-hutning and suspected such overruns, that is :)
15:51 timotimo makes sense
15:51 timotimo but when you're trying to change things in the FSA, basically turning the FSA off isn't the way to go ;)
15:52 jnthn Indeed
15:52 jnthn I was more thinking debugging things that (mis-)use the FSA
15:53 jnthn OK, anybody got any reason for me not to pull the release trigger?
15:53 jnthn Feel free to glance the Changelog for mistakes...
15:53 timotimo right
15:54 * timotimo looks over the changelog
15:56 timotimo the stuff in it seems good. haven't looked if anything's missing
15:56 timotimo you generally don't miss anything because of the way you build the changelog, so that's not something i worry about
15:56 jnthn I did it using git log as usual, but this time there were a ton of libatomicops commits imported
15:56 jnthn Condensed those to one entry :)
15:57 timotimo right
15:59 jnthn Full build of the release = 18s
15:59 jnthn In my VM with 4 cores allocated to it
15:59 jnthn Not bad :)
15:59 timotimo \o/
15:59 timotimo clearly that's not clang
16:00 jnthn gcc :)
16:02 jnthn http://www.moarvm.org/releases/MoarVM-2016.11.tar.gz
16:03 moritz jnthn++
16:04 timotimo \o/
16:04 timotimo should i recklessly merge my two branches, since they are both spectest-clean? ;)
16:05 jnthn I dunno, are they good? :P
16:05 timotimo i think they are good
16:05 timotimo good for memory usage :)
16:05 moritz then go for it
16:06 timotimo yay!
16:14 travis-ci joined #moarvm
16:14 travis-ci MoarVM build failed. Jonathan Worthington 'Changelog entries for 2016.11.'
16:14 travis-ci https://travis-ci.org/MoarVM/MoarVM/builds/177055968 https://github.com/MoarVM/MoarVM/compare/1ba1dd217766...1e2d7b42bd17
16:14 travis-ci left #moarvm
16:16 [Coke] ohnoes.
16:17 timotimo oh no, the change log blew it all up!
17:16 FROGGS joined #moarvm
17:16 FROGGS o/
17:20 timotimo o/
17:25 domidumont joined #moarvm
18:11 jnthn FROGGS: Don't suppose you've got a list of the platforms you've done MoarVM portability improvements for this month? Thought it'd be nice to put in the release blurb on the homepage for this month... :)
18:12 FROGGS jnthn: do you have open office?
18:12 FROGGS (or libre office)
18:14 jnthn Surely one of them
18:14 jnthn Libre
18:16 FROGGS http://froggs.de/perl6/platform_matrix.html
18:16 FROGGS pass means nqp and rakudo pass their 'make test'
18:16 FROGGS ffi means that libffi is supported on that platform
18:17 FROGGS grey means, there is no such os/arch combination
18:17 FROGGS the qemu column means that there is a qemu system emulator available
18:18 FROGGS jnthn: does that help?
18:18 [Coke] this page is in maltese!
18:19 FROGGS [Coke]: the thing I just posted? O.o
18:19 * psch guesses chrome offering to translate
18:20 geekosaur chrome here didn't see a problem
18:20 FROGGS jnthn: I can give you a list of operating systems that needed help too
18:20 FROGGS but one can read that in the commits too
18:21 [Coke] psch: ayup
18:21 [Coke] FROGGS: ayup
18:22 japhb FROGGS: Is it your intent that any os/arch that has *both* qemu and ffi support should be supportable by MoarVM?
18:25 FROGGS japhb: qemu support doesnt matter, but makes it easier for me
18:25 FROGGS japhb: every os that is supported by libuv, libffi, libatomic_ops etc and which has a modern enough arch with enough ram is a valid target
18:26 FROGGS and I try to get rakudo there if I can
18:30 FROGGS example: m68k is most likely not a desirable target
18:30 japhb FROGGS: A laudable goal ... do you have support lists for libuv and libatomic_ops as well?  Or are they just "try them and see if they compile and pass tests"?
18:32 jnthn FROGGS: Yes, thanks :)
18:33 jnthn What does "ffi" in a column mean? That ffi support is the blocker?
18:34 lizmat jnthn: is there a reason why nqp::split is generating a list_o rather than a list_s ?
18:35 timotimo maybe it means we have to use libffi instead of dyncall?
18:35 jnthn lizmat: To save another loop, I guess
18:35 lizmat another loop ?
18:36 timotimo did y'all know our index and rindex ops don't actually do boyer-moore?
18:36 jnthn lizmat: In most cases we need to hand back boxed strings; it'd take another iteration to do that.
18:37 lizmat ok
18:38 jnthn I'm not sure whether the Perl 6 split actually uses nqp::split though?
18:38 jnthn Since it can potentially work lazily
18:38 timotimo right, probably uses index instead
18:40 lizmat split( Str(Cool) ) uses nqp::split
18:41 lizmat and returns a fully reified list
18:41 jnthn Ah, OK
18:41 jnthn That's...probably often faster
18:41 jnthn m: say 'omg'.split(/m/).WHAT
18:41 camelia rakudo-moar b5aa3c: OUTPUT«(List)␤»
18:41 jnthn m: say 'omg'.split('m').WHAT
18:41 lizmat well, if the string is 2G, then probably not
18:41 camelia rakudo-moar b5aa3c: OUTPUT«(List)␤»
18:41 jnthn Good, it's consistent
18:42 jnthn True :)
18:42 * jnthn is a tiny bit surprised that we didn't return a Seq there
18:42 jnthn m: say 'omg'.comb(/m/).WHAT
18:42 camelia rakudo-moar b5aa3c: OUTPUT«(List)␤»
18:42 jnthn Ah, consistent with that also :)
18:43 jnthn (if the string is 2G) depends how many pieces it breaks up in to... :)
18:44 jnthn bbi30
19:03 FROGGS jnthn: ffi means that libffi is available according to the libffi.org page
19:03 FROGGS jnthn: though, that does not mean that the other deps, namely libuv, are available
19:11 FROGGS jnthn: I either fixed compilation or fixed issues in tests for linux on: arm, aarch64(arm64), mips, mips64, ppc, ppc64, s390x, x32 and for solaris@x86_64, as well as making moarvm and nqp build on aix@ppc
19:13 FROGGS japhb: there is https://github.com/libuv/libuv/issues/983
19:16 FROGGS japhb: and libatomic_ops runs almost everywhere I think, that includes hpux, true64, irix, ...
19:49 FROGGS jnthn: ahh, btw, the aix changes are still in a branch
19:50 domidumont joined #moarvm
20:27 dalek moarvm.org: c87d9cc | jnthn++ | / (2 files):
20:27 dalek moarvm.org: Update website for 2016.11 release.
20:27 dalek moarvm.org: review: https://github.com/MoarVM/moarvm.org/commit/c87d9cc45a
20:28 jnthn Our 35th release.
20:29 jnthn Time flies...
20:38 FROGGS wow
21:31 FROGGS looks like I can get nqp on aix test clean quite easily
21:31 FROGGS the weird test failures I had yesterday were about a too recent libuv
21:31 jnthn Nice!
21:32 FROGGS it only fails two tests, and these two were about inf/neginf... I hacked something up quickly so that moar builds
21:32 FROGGS and I just need to fix that
21:47 FROGGS Stage parse      : MoarVM panic: Memory allocation failed; could not allocate 16384 bytes
21:47 FROGGS gmake: *** [CORE.setting.moarvm] Error 1
21:47 FROGGS :~(
22:59 MrJones joined #moarvm
23:00 MrJones hi
23:00 MrJones can moarvm be used as a static library out of a C program which then launches it on some runtime-provided bytecode blob?

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary