Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2014-11-30

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 timotimo also, a tiny bit of compression would do wonders to our .moarvm files, since we decode that part anyway
00:03 timotimo like, a quite big chunk of the serialized blob is just AAA due to 64bit numbers that are 0
00:03 timotimo or "small"
00:24 jnthn We shouldn't be storing serialization stuff in .moarvm any more
00:24 jnthn Uh
00:24 jnthn As in, as strings we should not
00:24 jnthn Should look into that
00:24 jnthn I wish people would remember that if you compress, you can't mmap.
00:25 jnthn As soon as your 2 processes in, you're a 50% reduction ahead.
00:26 jnthn And s/your/you're/, but grammar is optional after drinking an eviltwin. :
00:26 jnthn *:)
00:28 jnthn As for recording where code was when a GC occurred - probably not useful. You just need to modify a tiny thing to make that happen at a different point. Or another thread can affect it. Maybe over a program than runs for a very long time you can get something interesting statistically, but the existing allocation profile already gives you data like that.
00:29 jnthn Put another way, GC is amortized over all allocations, so which one triggers it is not really so interesting.
01:13 timotimo OK, fair enough
01:14 timotimo my point about the base64 thing is: we can't mmap it anyway because it's base64 and we turn it into regular data right away
01:14 timotimo and going from base64 to regular binary data will get us from 4/3 down to 1
01:25 timotimo and since we are already base64-decoding before we read stuff from the serialized blob, we could just as well replace base64 with some compression scheme
01:44 Util joined #moarvm
02:45 JimmyZ joined #moarvm
02:46 JimmyZ_ timotimo: I think jnthn++ meant compressing all the .moarvm like jar does blocks mmap
02:47 JimmyZ_ and you meant compressing only the string part
02:55 timotimo yes, only the serialisation blobs
02:57 JimmyZ_ timotimo: + 1 to it :)
02:59 timotimo what kind of compression library can we use without adding a dependency that our users may not have our want?
02:59 timotimo I think RLE is not good enough
03:00 JimmyZ_ pack ?
03:01 JimmyZ_ as just write strings as binary string?
03:04 timotimo well yeah
03:04 timotimo we would put the serialised blob somewhere where binary data can live
03:05 timotimo but on top of that, having small numbers represented with fewer bytes may be worth something
03:05 JimmyZ_ that is, out of .moarvm files?
03:05 timotimo maybe just encoding every 64 bit chunk as a varint
03:05 timotimo no, inside these files
03:06 JimmyZ_ jnthn++ said 'We shouldn't be storing serialization stuff in .moarvm any more'
03:06 timotimo like, the bytecode section already contains binary data
03:06 JimmyZ_ I couldn't follow he
03:06 timotimo he corrected himself
03:06 timotimo he meant we should not use the string heap
03:07 JimmyZ_ so why?
03:07 timotimo so why what?
03:08 JimmyZ_ why we shouldn't store strings in .moarvm?
03:08 timotimo no no no
03:09 timotimo we should not store serialised objects in the strings
03:09 timotimo but strings will stay where they are
03:09 JimmyZ_ oh
03:09 timotimo :)
03:10 JimmyZ_ that is, replace base64 code with binary data in Strings heap?
03:11 timotimo I mean there is nothing wrong with storing the serialised blob as "packed" data
03:11 JimmyZ_ ok
03:11 timotimo but there are so many null bytes
03:11 timotimo we should store the serialised blob somewhere in the moarvm file that is not the string heap
03:12 timotimo make a new section for serialised blobs
03:12 JimmyZ_ fair enough
03:12 timotimo I am very tired and should go to bed soon
03:12 JimmyZ_ and still use base64 ?
03:12 JimmyZ_ good night
03:13 timotimo maybe you want to build a simple compression by using write varint
03:13 timotimo no, get rid of base64
06:02 dalek MoarVM: a533a69 | jimmy++ | src/ (2 files):
06:02 dalek MoarVM: Small fixes.
06:02 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/a533a69648
06:45 dalek MoarVM: 228e07e | jimmy++ | / (5 files):
06:45 dalek MoarVM: rename nodes_moar.h to nodes.h, since there is no other nodes_*.h files.
06:45 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/228e07e191
07:46 JimmyZ_ joined #moarvm
08:32 dalek MoarVM: 8fa0c7d | TimToady++ | src/6model/reprs/NFA. (2 files):
08:32 dalek MoarVM: detect useless SUBRULE edges
08:32 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/8fa0c7d9e3
09:11 JimmyZ_ joined #moarvm
09:13 lizmat joined #moarvm
09:22 woolfy joined #moarvm
09:32 JimmyZ_ joined #moarvm
09:36 JimmyZ_ joined #moarvm
10:11 brrt joined #moarvm
10:40 brrt left #moarvm
11:51 JimmyZ_ joined #moarvm
12:32 JimmyZ_ timotimo: about base64, I didn't find how to replace it with varint :(
12:36 timotimo OK, maybe i'll have a look
12:36 timotimo the problem is also that we can't just put the data back into the string heap
12:36 timotimo or at least we shouldn't
12:36 kjs_ joined #moarvm
12:36 timotimo that means we need to have another way to give the serialized blob to the deserialize instruction
12:37 JimmyZ_ why we can't?
12:38 jnthn But...we already *do* have that way
12:38 timotimo can the string heap store all-binary data without trouble?
12:38 jnthn That's why I can't figure out why you're seeing base64 strings.
12:38 jnthn The string heap stores strings in utf-8
12:39 jnthn The seriealized data has its own binary section of the bytecode file
12:39 timotimo oh?
12:39 timotimo i must be speaking of something else, then
12:39 jnthn And has for ages
12:39 jnthn Well, we may have had some regression at some point that makes it do the base-64 thing...
12:39 timotimo check_and_dissect_input does a base64 decode on its input
12:40 timotimo and one of the instructions in a moarvm --dump of a module reads 00803   deserialize  loc_1_str, loc_12_obj, loc_6_obj, loc_10_obj, loc_13_obj
12:40 timotimo where loc_1_str is the base64 encoded blob
12:40 JimmyZ_ Yeah, I saw it, I was even not sure what base64_encode encodes..
12:42 JimmyZ_ oh, it encodes serialized things
12:43 JimmyZ_ not strings
12:44 timotimo yes
12:44 timotimo that's what i was trying to explain %)
12:44 timotimo i was saying we store a serialized blob on the string heap
12:45 jnthn Yes, in all .moarvm files, or just a handful?
12:46 timotimo i shall see.
12:47 jnthn OK. I gotta head to the station fairly soon, but will be online on the train. :)
12:47 timotimo cairo has it
12:48 timotimo HTTP::Server::Async has a null string that it passes to deserialize
12:49 timotimo same with ::Request
12:49 timotimo could be about repossession conflicts, actually
12:50 timotimo SDL2::Raw has a base64 blob
12:51 timotimo zavolaj has a null string, too
12:54 colomon joined #moarvm
13:53 ggoebel111111114 joined #moarvm
14:53 zakharyas joined #moarvm
15:03 kjs_ joined #moarvm
15:42 vendethiel joined #moarvm
15:51 kjs_ joined #moarvm
16:25 woolfy left #moarvm
16:30 zakharyas joined #moarvm
17:52 FROGGS__ joined #moarvm
18:45 zakharyas joined #moarvm
20:39 FROGGS__ joined #moarvm
20:47 vendethiel- joined #moarvm

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary