Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2016-04-07

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:44 colomon joined #moarvm
01:08 [Coke] joined #moarvm
01:34 colomon joined #moarvm
02:30 MadcapJake joined #moarvm
06:51 domidumont joined #moarvm
07:18 zakharyas joined #moarvm
09:48 timotimo jnthn: are we interested in .moarvm files that segfault or hang moar when it tries to load it?
09:49 jnthn timotimo: Yeah, that shouldn't really happen
09:49 jnthn We go to quite some effort not to read past the ends of buffers when picking apart bytecode files
09:49 timotimo well, i'm running the American Fuzzy Lop to corrupt some moarvm files :)
09:50 jnthn Any hangs/SEGVs so far? :)
09:50 timotimo 330 unique crashes, 218 unique hangs
09:50 timotimo but i think i might be doing it wrong
09:51 timotimo i've fed it the nqp stage0 as a starting point
09:51 jnthn hah
09:51 timotimo but there's interdependencies
09:52 timotimo oh, neat
09:52 timotimo timo@hack:~/fuzzing/fuzzing-input$ ~/fuzzing/install/bin/moar ~/fuzzing/fuzz-state/crashes/id:000329,sig:11,src:000006,op:flip1,pos:64117
09:52 timotimo Segmentation fault
09:53 timotimo at the moment it seems to mostly be finding new hangs, which means it's only able to do like 30 tests per second
09:53 timotimo it used to do 10x that in the midle, and 20x that at the beginning
09:54 timotimo i have the pretty graphs :)
09:55 timotimo http://t.h8.lv/afl/
09:55 timotimo there they are
09:57 timotimo if i read the status screen correctly, it's still processing the 6th input file
09:57 timotimo which means we're looking at easily a year of fuzzing time :P
09:58 timotimo i'm considering re-running the fuzzing but giving it --dump on the commandline
09:59 timotimo that'd concentrate the fuzzing effort on actually loading one bytecode file and if i guess correctly it'd skip trying to load dependencies
09:59 timotimo ah. yes, it would
10:10 arnsholt When I discussed this with jnthn a while back, it seemed to me generating really small bytecode files would probably be useful as well
10:10 timotimo yeah
10:10 arnsholt Sadly I never got around to actually implementing that
10:14 timotimo i chopped the dumper down to size so it only forces deserialization of most frames and actualization of some strings
10:16 timotimo with the fuzzing run that doesn't just --dump, i expect many crashes would be located around the place where it starts to run bytecode
10:16 timotimo i wonder if i should keep that run going at all?
10:50 timotimo http://hack.p6c.org/~timo/afl1/  -  running a full moar *.moarvm
10:50 timotimo http://hack.p6c.org/~timo/afl2/  -  running moar --dump *.moarvm, but with most of the bytecodedump code cut out
11:46 mtj__ joined #moarvm
11:46 mtj__ joined #moarvm
12:37 zakharyas joined #moarvm
13:13 diakopter timotimo: that's really neat
13:14 diakopter timotimo: feel free to email me some of those crashes/hangs
13:53 timotimo diakopter: http://hack.p6c.org/~timo/crashes_hangs.tar.gz  -  those are the ones we have so far for --dump
14:19 diakopter timotimo: did you try afl-tmin
15:04 domidumont joined #moarvm
15:27 timotimo no
15:37 timotimo did you try out any of those crashing cases yet?
15:39 synopsebot6 joined #moarvm
16:23 diakopter m: EVAL q[] while 1
16:23 camelia rakudo-moar 61d231: OUTPUT«Memory allocation failed; could not allocate 385792 bytes␤»
16:24 diakopter timotimo: well, no, I was hoping you could run afl-tmin to try to minimize the cases
16:25 timotimo well, i can try
16:26 timotimo i have to run it for each individual result file?
16:26 diakopter no idea
16:26 diakopter I just saw it mentioned in the readme in the .tar.gz
16:28 timotimo it seems like that stuff works fine
16:28 timotimo File size reduced by : 1.78% (to 11276 bytes)
16:28 timotimo Characters simplified : 99.45%
16:28 timotimo the first file is now littered with "0" ascii bytes :)
16:28 diakopter lol
16:29 timotimo http://hack.p6c.org/~timo/simple_crash.moarvm
16:29 timotimo take a look yourself
16:29 diakopter r: http://hack.p6c.org/~timo/simple_crash.moarvm
16:29 camelia rakudo-jvm 5eaa15: OUTPUT«cannot connect to eval server: Connection refused␤»
16:29 camelia ..rakudo-moar 61d231: OUTPUT«===SORRY!=== Error while compiling /tmp/tmpfile␤Confused␤at /tmp/tmpfile:1␤------> http:⏏//hack.p6c.org/~timo/simple_crash.moarvm␤    expecting any of:␤        colon pair␤»
16:29 timotimo haha, as if.
16:29 diakopter :D
16:30 timotimo Program received signal SIGSEGV, Segmentation fault.
16:30 timotimo MVM_bytecode_unpack (tc=tc@entry=0x6037c0, cu=0x661cb0) at src/core/bytecode.c:878
16:30 timotimo 878         MVM_ASSIGN_REF(tc, &(cu->common.header), cu_body->hll_name,
16:30 timotimo i'll be AFK for a bit.
16:31 timotimo fwiw, the "--dump only" hasn't discovered a new path in 1 hour and 18 minutes
16:31 timotimo but it's only at the 5th input file, which it has finished 90% of apparently
16:31 timotimo so we'll see what happens once it reaches the 6th
16:31 diakopter how many input files are there
16:31 timotimo 10 or 9
16:32 diakopter billion?
16:32 timotimo files
16:32 diakopter billion exabytes?
16:32 timotimo that's the initial files, i just copied over the stage0 from nqp
16:32 diakopter hehehe
16:32 timotimo http://hack.p6c.org/~timo/  -  the "total paths"/"pending paths" will tell you how many files it's got in its queue
16:33 timotimo i suspect the "full run" has stumbled upon the interpreter and is now joyously exploring our instruction set
16:47 diakopter lolz
17:00 domidumont joined #moarvm
17:06 timotimo damn it.
17:06 timotimo it's now still in the 5th file, but in a different strategy
17:06 timotimo "arith 8/8"
17:07 diakopter hehe
17:07 timotimo it'll take 24M execs and it does about 600 execs per second
17:07 diakopter ETOOMANY
17:08 timotimo it's already reached 2% !!
17:08 timotimo it seems like it found a new path a quarter hour ago
17:08 timotimo so that's nice
17:09 timotimo but it'll still take forever until it noms up the remaining ~600 paths
17:10 timotimo and all in all it's at 0.77% - that number goes down every time a new path gets discovered, though
17:11 timotimo i should probably have used afl-cmin first to reduce the incoming files a bit, so it doesn't have to mess with all of the files
17:12 timotimo [+] Narrowed down to 9 files, saved in 'minimized_corpus'.
17:12 timotimo gee, thank you!
17:13 diakopter ur a corpse
17:14 timotimo you're too kind
17:14 diakopter XD
17:17 timotimo i'm not patient enough for fuzzing :|
17:18 timotimo it's really not something you should sit in front of and attentively watch, that's for sure
17:48 leont joined #moarvm
18:16 FROGGS joined #moarvm
18:39 timotimo i'm now minimizing all those crashes i've found so far for the "only --dump" case
18:55 diakopter :D
19:02 timotimo the files fall into two categories so far: one where the space saving is like 2%, the other where the space saving is like 25%
19:08 timotimo *** Error in `../../install-changed-dump/bin/moar': malloc(): memory corruption: 0x00007f3c41b60610 ***
19:08 timotimo we have a .moarvm file that gives us that
19:08 timotimo at least i think so?
19:08 diakopter splendid
19:21 zakharyas joined #moarvm
19:25 timotimo i'm quite worried that the speed of the second run might take a gigantic dip like the first run did when it reaches file number 6
19:33 TimToady joined #moarvm
20:05 timotimo "auto extras"
20:05 timotimo i have no idea what that is or does
20:29 domidumont joined #moarvm
20:39 timotimo diakopter: want the minimized crash files?
20:41 timotimo diakopter: http://hack.p6c.org/~timo/minimized_dumponly_crashes.tar.gz
21:12 diakopter timotimo: cool, thanks :)
22:17 dalek MoarVM: a5cf746 | timotimo++ | src/core/bytecode.c:
22:17 dalek MoarVM: bytecode_unpack: bail out if HLL name string index is invalid
22:17 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/a5cf7466de
22:19 jnthn timotimo: Nice catch :)
22:19 timotimo the first 10 crashes afl found were because of this
22:20 timotimo actually
22:20 timotimo all the crashes it found were this ...
22:20 timotimo all 16 crashes
22:22 timotimo but there's 20 unique crashes by now. i can bring the tarball up to date soon
22:25 timotimo oh, interesting, a segfault in reand_int32 called by finish_frame
22:25 timotimo of course with our lazy deserialization, exceptions about corrupt bytecode files will now fly at random moments in time :)
22:25 timotimo fun!!
22:27 jnthn :)
22:29 timotimo missed a few ensure_can_reads that i now littered across the function
22:32 timotimo hum. where do i reach the rs i need to pass to ensure_can_read when i'm in finish_deserialize?
22:32 timotimo i thought the StaticFrame or maybe the CompUnit has a pointer to the ReaderState, but it doesn't seem to be the case
22:33 timotimo hm, maybe we just store the full size in some place
22:34 timotimo is data_size the right number here?
22:34 timotimo no, serialized_size is the right one i bet
22:40 timotimo hm. when we stumble upon that problem, we can't "cleanup_all" like we usually do
22:40 timotimo cleaning up the whole compunit is probably not so easy when we're potentially in the middle of code in there?
22:55 timotimo now to figure out the reentrant mutex auto-unlocking stuff
22:56 timotimo oh, i can just have it as an argument. for this single-purpose helper function it should be fine
23:00 timotimo Unhandled exception: Could not finish deserializing frame: Read past end
23:00 timotimo [Inferior 1 (process 1511) exited with code 01]
23:06 dalek MoarVM: 9c2368b | timotimo++ | src/core/bytecode.c:
23:06 dalek MoarVM: spread some range checks in MVM_bytecode_finish_frame
23:06 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/9c2368bd53
23:06 dalek MoarVM: bab36ec | timotimo++ | src/core/bytecode.c:
23:06 dalek MoarVM: finish_frame: don't try to set flags beyond num_lexicals
23:06 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/bab36ec541
23:06 timotimo well, damn. now i went ahead and b0rked it for "real" code :(
23:09 timotimo Unhandled exception: Could not finish deserializing frame: Read -1292547922 past end
23:09 timotimo holy moly.
23:11 timotimo this is a bit less obvious than i had hoped. i'll revert and branchify
23:16 dalek MoarVM: e42d949 | timotimo++ | src/core/bytecode.c:
23:16 dalek MoarVM: Revert "spread some range checks in MVM_bytecode_finish_frame"
23:16 dalek MoarVM:
23:16 dalek MoarVM: This reverts commit 9c2368bd53bcbb17ed4a53af9c55b2ad07265ce0.
23:16 dalek MoarVM:
23:16 dalek MoarVM: That commit got the limit calculation wrong and made legit files
23:16 dalek MoarVM: unreadable.
23:16 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/e42d9499ae
23:17 travis-ci joined #moarvm
23:17 travis-ci MoarVM build failed. Timo Paulssen 'finish_frame: don't try to set flags beyond num_lexicals'
23:17 travis-ci https://travis-ci.org/MoarVM/MoarVM/builds/121568422 https://github.com/MoarVM/MoarVM/compare/a5cf7466ded2...bab36ec54124
23:17 travis-ci left #moarvm
23:31 timotimo travis is yet to report it, but my revert fixed the build again

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary