Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2017-04-06

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:07 samcv dropped my laptop and the screen went all blocked up and making bad noises
01:07 samcv maybe dropping it tapped the ram? idk
01:07 samcv turned on fine once i turned it off
01:07 samcv timotimo, with 1/100 the size i didn't see collapse strands btw
01:09 timotimo interesting!
01:14 vendethiel joined #moarvm
01:48 ilbot3 joined #moarvm
01:48 Topic for #moarvm is now https://github.com/moarvm/moarvm | IRC logs at  http://irclog.perlgeek.de/moarvm/today
01:52 MasterDuke and --profile shows the most expensive routine (by exclusive time) is `infix:<~> (SETTING::src/core/Str.pm:2801)`, at 30%, next most is 5.8%
01:54 MasterDuke that's the Str:D multi
02:01 samcv yeah i bet it is
02:01 samcv that thig always sucks
02:01 samcv MasterDuke, what are you using to measure? and how big is your file?
02:03 MasterDuke perf record and --profile, of `use JSON::Fast; spurt("otherCache", from-json(to-json(slurp("cache").split("\0"))).join("\0"))`, with the first 2.5Mb of the file timotimo was using
02:05 samcv argh complaining i don't have enough permissio
02:05 samcv it's set to -1
02:09 samcv at 502017, unexpected \ inside list of things in an array
02:21 samcv whatever i just made a coverage report of it
02:21 samcv will check that function
02:22 samcv ok i see
02:22 samcv interesting
02:23 samcv uploading now
02:25 samcv MasterDuke, this must be G as in giga https://cry.nu/coverage/json-1/libmoar/coverage/home/samantha/git/MoarVM/src/strings/ops.c.html#L77
02:26 samcv but the whole function itself runs 4.5k times
02:26 samcv for loop is pretty hot
02:26 samcv but looks like turn_32bit_into_8bit_unchecked is never hit. so that's not causing the issue
02:27 samcv though it does still hit that function 25.8k times
02:29 MasterDuke this http://i.imgur.com/xg5qvIk.png is heaptrack's overview
02:29 samcv if (g < -127 || g > 127) probably (hopefully? gets optimized to not check sign and just check binary hopefully. since that's all that's needed
02:31 MasterDuke oh, oops, all that measuring and profiling was with my modified version
02:31 samcv modified how
02:32 MasterDuke https://gist.github.com/MasterDuke17/73b3709e16c807a7dcda174e528a64e6
02:34 samcv but
02:34 samcv why would you change it like that. won't it possibly break
02:34 samcv oh i see i guess you already collapse it
02:34 samcv then check there. any change in time?
02:35 MasterDuke i just time it once before and once after. 0.1s slower after my change (out of 31s)
02:41 samcv ok i've made it faster
02:41 agentzh joined #moarvm
02:42 samcv let me calculate how much. but this is only measuring differences between a single change i made, and i made a couple other changes that may have a minor effect idk
02:44 samcv average of 5 times made it go from 22.188 to 23.43 seconds
02:44 samcv so 1.2s shorter
02:45 samcv 5.4% faster
02:50 Geth ¦ MoarVM: 7beffd390c | (Samantha McVey)++ | src/strings/ops.c
02:50 Geth ¦ MoarVM: collapse_strands 5.4% speed boost under some workloads
02:50 Geth ¦ MoarVM:
02:50 Geth ¦ MoarVM: This loop gets ran a huge number of times under collapsing strands
02:50 Geth ¦ MoarVM: workloads. If we're already set can_use_8bit, don't bother checking if the
02:50 Geth ¦ MoarVM: following graphemes are < -127 and > 127
02:50 samcv pushing that change MasterDuke
02:50 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/7beffd390c
02:51 samcv i'm thinking of making that MVMint8 a plain old int, so it'll use the fastest type on whichever cpu it's ran on. it's bool so doesn't matter the size of it
02:59 MasterDuke you mean 23.43 to 22.188?
03:00 samcv yes
03:01 samcv i'm making MVM_string_graphs_nocheck too, so that should help make string things faster cause that gets called a ton of times
03:08 samcv uhm and got 21.408s with some more of my changes
03:15 vendethiel joined #moarvm
03:19 samcv i think i found a bug
03:20 samcv https://github.com/MoarVM/MoarVM/blob/master/src/strings/ops.c#L157-L165 here. if it's an ascii string, we should access blob_ascii
03:20 samcv which is what is used in every other places
03:23 samcv can can compare them with memcmp but we can't point to the same place
03:24 samcv i don't think we run that command anywhere on ascii strings since they're hardly used (or something)
03:25 samcv imo it would be easier to deal with if they both used blob_8 and we still have the string be of type ASCII, but they are both stored in blob_8's
03:26 samcv then we won't have to have branching when we're really just comparing two things that are subsets of each other, and can be memcmp'd fine
05:51 brrt joined #moarvm
05:53 domidumont joined #moarvm
05:59 domidumont joined #moarvm
06:00 domidumont joined #moarvm
06:36 geekosaur joined #moarvm
06:44 brrt joined #moarvm
07:14 brrt merge pushed
07:15 brrt the 'has-dynasm' thingy has been removed, because, a): we have our own dynasm fork, and it's not going to be the one that's installed, b): dynasm is a *built-time-dependency*, damnit, and it doesn't make sense to use some other guys' version
07:36 domidumont joined #moarvm
07:41 zakharyas joined #moarvm
09:05 domidumont joined #moarvm
10:43 vendethiel- joined #moarvm
11:17 timotimo it occurs to me that collapse_strands uses the grapheme iterator in all cases. we could probably special-case one or two different storage types there
11:26 MasterDuke timotimo: ah, interesting idea
11:27 timotimo i'm doing a run on a quarter (elementwise) of the whole cache
11:28 MasterDuke timotimo: did you notice samcv's possible bug find in MVM_string_substrings_equal_nocheck? it does look odd
11:29 timotimo typedef MVMint8  MVMGraphemeASCII;
11:29 timotimo typedef MVMint8  MVMGrapheme8;       /* Future use */
11:36 timotimo i imagine if we split it the compiler will merge it again until we change one of these typedefs
12:00 timotimo so since reducing it down to like a tenth makes it complete in mere minutes, i was hoping a quarter would give me a nice hour or so run time
12:01 timotimo like, just a tiny bit up the hockey stick
13:27 spebern joined #moarvm
14:10 nine_ joined #moarvm
14:13 SmokeMachine joined #moarvm
14:33 spebern joined #moarvm
15:06 timotimo almost 4 hours now
15:08 brrt joined #moarvm
15:09 brrt good * #moarvm
15:47 brrt joined #moarvm
16:39 domidumont joined #moarvm
17:02 samcv good *
17:03 samcv where do you get future use timotimo
17:03 samcv <timotimo> typedef MVMint8  MVMGrapheme8;       /* Future use */
17:03 samcv it is used currently though
17:04 timotimo that's just part of the code :)
17:04 timotimo i think i was Future Man who got to Use that
17:04 geekosaur ^
17:04 samcv heh
17:05 samcv well it's used currently :P
17:05 geekosaur couple weeks(?) ago timotimo implemented that optimization for strings restricted to the ISO8859 subset
17:05 geekosaur so yes, the future is now(tm)
17:05 timotimo the future is so last week
17:06 samcv was longer than couple weeks ago
17:08 samcv also timotimo what are your thoughts of the ascii thing. they're both stored int int8's and that places looks like a bug right? in MVM_string_substrings_equal_nocheck. nowhere else do i see an ascii type string have blob_8
17:08 samcv but i wouldn't be against making ascii strings just use blob_8 anyway or something
17:09 timotimo we should decide that blob_8 is supposed to be signed, just like ascii is. and then we can throw it out :)
17:09 samcv they'd still be 8 bit strings and still be distinguishable, but would make comparison of ascii and others less prone to errors
17:09 samcv they are both signed
17:09 samcv 8 bit numbers
17:10 timotimo yeah, it's important to have signedness here
17:10 timotimo for our synthetics
17:10 samcv yeah
17:10 timotimo also, ascii is - of course - only defined up to 127
17:11 samcv 12. what.
17:11 timotimo if you try to store latin-1 in "ascii", we'll not be in agreement
17:12 samcv we store latin-1 in blob_8
17:12 geekosaur enh, question is whether synthetics or latin-1 make more sense there. I'm not sure there are that many useful cases for synthetics in that range
17:13 timotimo we do? :o
17:13 samcv yes timotimo
17:13 samcv they're just numbers...
17:13 timotimo but but but but
17:13 timotimo signed!
17:13 samcv so now we want another blob format for no reason XD
17:13 samcv blob_latin1 XD
17:13 TimToady well, even ASCII can have CRLF in it...
17:14 timotimo yeah, and we turn that into a negative number
17:14 samcv yep
17:15 samcv would be easier if ascii, latin-1 and 8bit all stored in blob_8. we don't have a Latin-1 type of string btw
17:15 samcv just ascii, grapheme 8, grapheme 32, and strand
17:16 samcv can still have the string be of type ascii, but no need to confuse things and make comparing 8 bit strings more work than it needs to be
17:16 samcv programming errors etc
17:21 samcv also i kind of think CRLF should be a constant synthetic grapheme. that could save some having to check what crlf is in the trie
17:22 Geth ¦ MoarVM: affec75b9c | (Samantha McVey)++ | 4 files
17:22 Geth ¦ MoarVM: Add MVM_string_graphs_nocheck funct, use it places we prev. already check
17:22 Geth ¦ MoarVM:
17:22 Geth ¦ MoarVM: There are many places where we check arguments with MVM_string_check_arg, and then
17:22 Geth ¦ MoarVM: will later on call MVM_string_graphs. This is redundant because MVM_string_graphs
17:22 Geth ¦ MoarVM: runs the same checks every time it runs that MVM_string_check_arg has already done.
17:22 Geth ¦ MoarVM:
17:22 Geth ¦ MoarVM: Shows a minor, but measurable speed increase.
17:22 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/affec75b9c
17:29 samcv is this supposed to be called MVN_unicode_normalizer_form MVN? and not MVM?
18:17 AlexDaniel joined #moarvm
18:22 samcv hmm. i. want to steal some code
18:23 samcv Knuth-Morris-Pratt search, memmem (searching for memory within memory, so we can use it on whatever size grapheme we want)
18:23 samcv https://lists.gnu.org/archive/html/bug-gnulib/2008-01/msg00031.html
19:02 zakharyas joined #moarvm
19:08 * TimToady wondered whether CRLF should always be -1
19:10 timotimo we can easily make that happen inside moarvm
19:10 timotimo by just creating -1 as soon as moarvm is started
19:29 samcv yeah what timotimo said
19:44 dalek joined #moarvm
19:44 SourceBaby joined #moarvm
19:50 samcv ugh i can't get `memmem` working
19:51 samcv i thought it would return a pointer to the location in the haystack the substring starts at. which is what the manpage says
19:52 samcv but it returns numbers that are inconsistently different than haystack
19:52 samcv i thought if i did `haystack - memmemresult` i would get the size_t from the start of the haystack to the found result and it should be consistent. but it isn't
19:52 samcv and i'm very confused
20:00 samcv whatever it's pointing to seems to hold the same data every time i run it. but it isn't the right memory region...
20:01 samcv timotimo, help https://gist.github.com/60469bf4ea8db41db3f21d95c2f8a957
20:01 samcv man page if anybody needs it... http://man7.org/linux/man-pages/man3/memmem.3.html#top_of_page argh.
20:02 samcv if we can get it to work we can get kruth-morris-pratt optimized string search tho
20:04 samcv then include the source for mac/windows since it's not a standard c lib function
20:22 timotimo hmm
20:24 timotimo samcv: seems correct to me
20:24 timotimo you're telling it to look for "a" and it finds it right at the beginning
20:24 timotimo when you make needlelen 2 instead of 1, it'll find it a bit later
20:24 timotimo haystack[4195968] needle[4195976] found[4195972]
20:24 timotimo i.e. 4 bytes in
20:25 samcv ok that's not what i get
20:25 samcv i get something totally not that
20:25 samcv haystack[94595572664408] needle[94595572664416] found[3212937304]
20:25 timotimo did you #define _GNU_SOURCE?
20:25 timotimo it doesn't compile otherwise
20:25 timotimo 32bit machine perchance?
20:26 samcv it compiles for me. without that. it's 64bit
20:26 timotimo the char output is b0rked on my end
20:26 samcv and adding #define _GNU_SOURCE does not help me
20:26 samcv ah
20:26 samcv that's okay though. as long as it's giving the right answer...
20:27 samcv i just get random different found values....
20:27 samcv that are like much smaller than anything else
20:27 samcv and it's not even consistent when i do haystack - found
20:27 timotimo oh
20:27 timotimo you're storing the result of memmem into an int
20:27 timotimo int isn't defined to be able to store a pointer
20:28 samcv yes
20:28 timotimo that's probably why you're getting such a bogus result. and i have no idea why i'm getting a correct one. perhaps it's storing these things in a memory location much closer to 0 so that it fits into 32bit?
20:28 samcv it still does not work XD that's what i tried FIRST
20:28 samcv if i set it void * or char * i get even CRAZIER ranging values
20:28 samcv haystack[94913660565544] needle[94913660565552] found[18446744072887842856]
20:28 samcv X\
20:30 samcv T_T
20:30 samcv ok now it's working.
20:30 samcv idk what options my in editor C compiler uses. but it works fine compiling myself
20:31 samcv weird.......
20:31 samcv will have to look into that
20:33 samcv ok well at least i have a working example. will try and make it work in mvm now
20:35 timotimo sorry, i was just out on the balcony to see an iridium flare
20:35 timotimo but isn't memmem using byte-granularity?
20:35 timotimo so we'd have to continue searching until we hit a properly aligned one, right?
20:36 samcv ok i think i fixed it
20:37 samcv in mvm
20:37 samcv oh you mean if by some magic the byte is found between bytes?
20:38 samcv err for grapheme32's i guess. but yeah i think it does by byte
20:38 samcv i don't think that will happen. but we will have to have a check to make sure that does not occur
20:39 samcv however unlikely it may be. it is possible
20:39 timotimo mhm
20:39 timotimo hm, say ...
20:39 samcv (maybe possible)
20:39 timotimo if it does a fancy algorithm, it may discover by itself that it can skip almost all alignments, except of course the proper one
20:39 samcv but until we know for sure we need to check
20:39 samcv well it does do fancy
20:39 samcv Knuth-Morris-Pratt
20:39 samcv which is why i want it
20:40 timotimo how much fancier than boyer-moore is this?
20:41 samcv maybe less? idk
20:41 timotimo it's not mentioned once in the wikipedia article
20:41 samcv i do know it's a lot faster than what we have now
20:41 timotimo right
20:41 timotimo it's sad we hardly spend any time searching for a needle in a haystack when compiling the core setting
20:41 samcv i've heard of it before. it's different than booyer moore. not quite as complex i think
20:43 samcv gonna run spectest now
20:46 samcv yay spectest pass
20:47 samcv ok. now. so can a 32 bit number exist in an array of 32 bit numbers, not at a normal offset
20:48 samcv i am not sure how to prove or disprove that
20:49 samcv well easiest would be to try to construct one synthetically i guess
20:51 samcv nice timotimo. it's 2.2x faster :O
20:51 samcv doing 'a' x 100000 ~~ /b/;
20:51 samcv worst case
20:52 samcv that is huge
21:03 timotimo nice
21:04 timotimo it's easy for such a 32bit number to exist inside two 32bit numbers
21:04 timotimo how many f is 32bit again %)
21:05 timotimo two is 8bit, so 8 is 32bit
21:05 timotimo 0000ffff, ffff0000 has ffffffff in it at a non-aligned offset
21:05 samcv yeah. wondering what the probability of that is
21:07 timotimo hm
21:07 timotimo imagine you have a number that has its lowest byte full
21:07 timotimo m: say 0x100.uniname
21:07 camelia rakudo-moar 605e9e: OUTPUT: «LATIN CAPITAL LETTER A WITH MACRON␤»
21:08 timotimo okay, so imagine we have two nullbytes and then this, and we're looking for \1
21:08 timotimo m: say ("A".ord +< 8).uniname
21:08 camelia rakudo-moar 605e9e: OUTPUT: «<CJK Ideograph Extension A>␤»
21:08 timotimo hmm
21:35 samcv ok i need a break. bbs
21:39 samcv i'm getting very frustrated trying to use the memmem function standalone.
21:40 samcv i try and rename it and import it. and then it just says it can't find it
21:40 samcv timotimo, let me know if you come up with some way i could test the bug and check when it triggers. would be nice to write a test for it
21:51 samcv timotimo, https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm
21:54 timotimo that's the page i looked at
21:54 samcv timotimo, maybe you can at least write some code to detect if it's not on the boundary. or help me
21:54 samcv https://github.com/samcv/MoarVM/commit/88994712163518f8e6dbd1f0dd90671eea0af3aa here's the commit
21:55 samcv it works fine. but for the 32bit graphemes has that rare bug with overlap
21:55 samcv probably
21:56 samcv so the pointer subtraction works as expected since i store the result of memmem in a MVMGrapheme32*
21:56 samcv so uh. do i cast as char * or what and then subtract them? not 100% sure
21:56 samcv to find out if it's not divisible cleanly by 32
21:57 samcv (cast both memmemrtrn32 and haystack->body.storage.blob_32 as char * i mean)
22:26 agentzh_ joined #moarvm
22:57 timotimo okay so i quartered the workload and it's still going after 11h30m
22:57 samcv well it compiled fine on mac on travis
22:58 samcv bsd has a non optimized memmem thing. not sure how it compares in speed to what we previously did
22:59 timotimo maybe it's better to cast to intptr_t, but char* should be fine
22:59 timotimo the perf.data file is 11 gigs big %)
23:00 samcv nice
23:00 samcv amazing
23:00 timotimo just waiting for it to finish writing it out
23:00 timotimo oh, i think it just finished
23:01 timotimo i wonder how long it'll take to perf report that :D
23:02 timotimo [ perf record: Woken up 43617 times to write data ]
23:02 timotimo [ perf record: Captured and wrote 10904.887 MB perf.data (156696959 samples) ]
23:33 samcv timotimo, i think there must be a bug
23:34 samcv at 502017, unexpected \ inside list of things in an array
23:34 samcv in sub parse-array
23:34 samcv in JSON::Fast
23:38 samcv can you compress perf data
23:46 samcv also working on making that loop even tighter. i love now i can run roast and have it finish in 3.5mins
23:58 Geth ¦ MoarVM: samcv++ created pull request #573: Have two part loop in collapse strands to make loop tighter when possible
23:58 Geth ¦ MoarVM: review: https://github.com/MoarVM/MoarVM/pull/573

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary