Perl 6 - the future is here, just unevenly distributed

IRC log for #moarvm, 2014-07-25

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:10 FROGGS__ joined #moarvm
02:53 btyler joined #moarvm
04:21 TimToady http://rosettacode.org/wiki/Category:Perl_6 seems to be leaking at a rate of about 75Mb per minute
04:22 TimToady (I changed the ^6 to ^7 to try to find the 7th number, but it's already used 4 gigs of ram in about 50 minutes of runtime
04:22 TimToady oops
04:23 lue TimToady: did you mean a specific problem? Or are you referring to the rate at which problems are solved for Perl 6? :)
04:23 TimToady http://rosettacode.org/wiki/Find_palindromic_numbers_in_both_binary_and_ternary_bases#Perl_6
04:27 TimToady the resident set is also going up to the same vicinity, which might indicate the GC is revisiting most of the pages
04:27 TimToady or maybe it just means I have too much memory :)
04:28 ventica_desktop moar backend?
04:29 TimToady yup
04:30 TimToady it's actually paging other stuff out, so I think it's probably all being visited
04:32 TimToady starting to swap out X windows, so I'm killing it after an hour
04:33 TimToady er, 75MB per minute, not Mb :)
04:34 * lue realizes he should probably get to know the more sysadminy parts of his system sometime :)
04:34 TimToady so about 1¼ megs per second
04:40 TimToady actually, in the early stages, it goes up by about 3 megs a second
04:40 TimToady that's without a gather/take too
04:48 woolfy joined #moarvm
04:48 TimToady okay, loop { .base(3) } does not leak
04:49 TimToady .base(3) for 1..* leaks about 6 megs a second
04:49 lizmat joined #moarvm
04:51 TimToady 42 for 1..*;  # leaks 7 megs a second
04:52 TimToady something's obviously not throwing away the used numbers, or something like that
04:53 lizmat # ore like 9 megs / sec for me
04:53 lizmat *more
04:54 TimToady does not leak on jvm
04:54 lizmat for 1..* { } also leaks
04:54 TimToady though jvm starts with 3.5gig :)
04:55 TimToady yes, the original loop was in that form, and all the stuff inside just served to hide the leak :)
04:56 lizmat FWIW, I seem the same kind of leakage on parrot
04:57 TimToady maybe jvm just hides it by allocating a gig of GC to start with...
04:57 lizmat I wouldn't be surprise
04:57 lizmat d
05:01 lue Right now I'm using 17.9% percent of my ram on ^7, and climbing (I don't know which of the VIRT, RES, or SHR memory counts are relevant here)
05:02 * lizmat is exhausted and will probably go to bed soon
05:07 TimToady VIRT mostly
05:08 TimToady RES is how much of VIRT is actually resident and presumably active, at least under load
05:08 lue 897M is what htop's telling me right now
05:08 TimToady at 17.9% it might not care
05:09 TimToady took me 50 minutes or so to get up to 4 gig
05:10 lue 4GiB is how much RAM I have at all :P
05:10 TimToady then I'd suggest you not run it quite that far :)
05:10 TimToady it won't find the 7th in 4 gigs
05:11 * TimToady is currently running the jvm version to see if it leaks before finding the 7th
05:11 lue yeah, it's never a fun time when the kernel steps in to kill things.
05:12 TimToady my jvm's VIRT is nailed to 3482060 for most of the last hour, and hasn't budged
05:27 TimToady RES on the jvm is creeping up at about 2 meg a minute; dunno if that means anything
06:36 ventica ok... so looking for guidance on Async IO testing/bugs/code-clean up (this was suggested as a good place to begin contributing)... are there bug reports I should go read?
06:36 ventica Async Socket* IO
06:55 zakharyas joined #moarvm
06:57 brrt joined #moarvm
07:04 ventica joined #moarvm
07:06 ventica_desktop joined #moarvm
07:16 brrt \o
07:25 sergot o/
07:26 Ven joined #moarvm
07:33 * brrt figured out that it was much simpler to install the mingw32-w64 toolchain than to install a windows vm
07:36 brrt and it is, but the answers are confusing
07:39 FROGGS joined #moarvm
07:53 brrt it seems mingw64 compiles the full 8 bytes of stack space for every argument, even though they are 32 bits wide
07:58 moritz alignment :-)
08:00 brrt (obviously, posix isn't quite so friendly as it does take the argument size into account)
08:03 ventica_desktop which windows vm?
08:14 brrt any, i guess?
08:14 brrt i can download a trial version of windows 8 or 7
08:14 brrt and install this on a vm, but it's just work, you know
08:14 brrt and i'm lazy :-)
08:17 ventica_desktop :P
08:19 brrt it seems i do the win64 ABI completely wrong
08:22 brrt or, mostly wrong
08:22 moritz that explains the blowups :-)
08:23 brrt well, yes
08:23 brrt the rules are a bit more arcane than i had imagined
08:27 jnthn heh, so it wasn't a bastion of sanity over the posix one, after all? :)
08:27 jnthn I use the MSVC compiler, fwiw. I think you can install Visual Studio Express for C++ to get the toolchain.
08:28 * ventica_desktop shudders at the thought of MSVC
08:29 * jnthn wonders what particular aspect of it causes the shudders
08:30 FROGGS yes, the express does work well, I use that too
08:30 FROGGS you just have to register at some point me thinks
08:31 ventica_desktop jnthn: I guess it's more the whole dll hell thing when you get to linking... the compiler itself generates tight code but compatibility in the windows world is such a headache
08:32 ventica_desktop or maybe interoperability is a better word than compatibility
08:33 * moritz finds the whole PIC / non-PIC approach of DLLs interesting
08:33 jnthn Perhaps that's why with .Net they went more with the "just copy all the things" approach :)
08:33 FROGGS I love how you can compile something on your system, just zip it and ship, and it will work on any other windows system...
08:33 ventica_desktop lol
08:34 FROGGS try to do that on linux, I've got a debian 3 (!!) here for such things
08:34 jnthn ventica_desktop: Did you manage to dig into the async code at all?
08:35 ventica_desktop I looked through the asyncsocket.c (sp?) file, but it's pretty incomprehensible... watched a prez on libuv
08:35 brrt calling conventions are never sane
08:35 * brrt is now downloading windows 8.1 trial
08:35 ventica_desktop incomprehensible *to me* (ramp and all that)
08:36 ventica_desktop jnthn: After running the tests on my box and they passed, my question was if there's a specific place I need to be looking?
08:37 jnthn Well, the async socket ones only seem to fail under load... :( Typical of race-y issues, I guess.
08:38 ventica_desktop who wrote it?
08:38 ventica_desktop the async socket code
08:38 jnthn Me
08:38 brrt bbi2h
08:39 jnthn Doesn't mean the race is obvious to me, though, otherwise I'd have fixed it by now. :)
08:40 ventica_desktop what are the kinds of I/O that asyncsocket.c is responsible for? sockets can connect lots of kinds of things and i can see that there are other files responsible for different kinds of I/O...
08:40 ventica_desktop under /src/io that is
08:40 jnthn At the moment, asyncsocket.c is just about providing TCP sockets (client and server)
08:40 jnthn Should really teach it UDP also.
08:41 ventica_desktop ok
08:41 ventica_desktop and i see the tests are exercising  the localhost has the code ever been tested on remote connections?
08:42 jnthn Well, when I first wrote it I had it doing non-local web requests at lest.
08:42 jnthn *least
08:42 ventica_desktop ok, cool
08:42 jnthn (And that worked out fine.)
08:43 jnthn errand; back in 10 mins
08:55 * jnthn back
10:47 Ven joined #moarvm
11:44 brrt joined #moarvm
11:49 brrt i've looked at the actual calling behaviour of win64, and conclude that the problem is more subtle than it appeared
11:49 brrt also, more generic
11:49 brrt to be specific, when i call a function and push parameters on the stack, i don't 'unpush' them, i.e. the stack top keeps on growing
11:49 brrt the function epilogue doesn't take this into account
11:50 jnthn Oh...
11:50 jnthn Well, glad you're figuring it out :)
11:51 moritz so, it's not a stack? :-)
11:51 brrt it is a stack
11:52 brrt it's... complicated
11:52 brrt it's a good subject for a blog post, too
11:53 FROGGS brrt: I'm already leaning back with a coffee in my hand :o)
11:53 brrt know what, i'll just blog about it
11:53 brrt :-)
11:54 FROGGS :o)
11:54 FROGGS brrt++
12:36 * brrt bbiab
12:53 brrt joined #moarvm
13:02 jnap joined #moarvm
13:47 jnap joined #moarvm
14:04 brrt joined #moarvm
14:19 jnap joined #moarvm
14:49 lizmat joined #moarvm
15:06 TimToady jnthn: opinions on our memory leak we noticed last night?
15:08 jnthn TimToady: If it's happening in Parrot *and* MoarVM, my first suspicion would fall on something being held onto for too long somewhere in, perhaps, MapIter
15:10 TimToady but isn't that a case that you optimize away from MapIter?
15:12 jnthn 1..1000000 is
15:12 jnthn 1..* isn't
15:13 jnthn It actually uses a native int, so it's bounds-checking the upper bound to fit within that at present.
15:13 btyler joined #moarvm
15:16 brrt shameless blog plug: http://brrt-to-the-future.blogspot.nl/2014/07/bugs-updates-and-abis.html
15:16 brrt also, apparantly i'm going on a short holiday for the next 4 days, so i'll be afk until tuesday late
15:16 brrt (starting tomorrow :-))
15:22 jnthn brrt++ # blog post
15:22 jnthn Do you expect to get any progress on the Windows JIT explosion today, before leaving?
15:23 [Coke] brrt++
15:23 jnthn On extops - I'm planning to take a look at them this evening to see if I can make things a little easier
15:30 brrt ok, nice :-)
15:30 * TimToady read abis as 'alibis'
15:30 brrt i hope so, but i'm not sure, it's a big change (wrt windows jit explosion)
15:31 brrt what the heart is full of
15:31 brrt also, wrt to extops, i planned to change the 'invokish' field into something like 'jittitude' or 'jittivity' and have it have flags for invokish, throwish, branchish
15:32 brrt or may_invoke, may_throw, may_branch if you like non-whimsical names
15:37 woolfy1 joined #moarvm
15:38 woolfy joined #moarvm
15:41 * TimToady has no problem with whimsy, as long as it's also understandable
15:53 jnthn I don't think we have any that branch, fwiw
15:56 brrt i thought so
15:56 brrt that's good, too :-)
15:57 * brrt dinner &
16:07 jnap joined #moarvm
16:19 brrt joined #moarvm
16:23 moritz brrt++ § blog
16:23 brrt :-) thanks
16:24 timotimo brrt++ $ blog
16:25 brrt oh, timotimo, i forgot to note your contributions, as well as a small guide on how to contribute
16:25 timotimo i contributed?
16:25 timotimo i already forgot :P
16:25 timotimo ah, you mean the simple ops i hooked up to the respective c functions?
16:26 brrt yes, these :-)
16:26 brrt it's fun to see that i'm not the only one who can manipulate the code sucessfully
16:27 timotimo well, that's only the part of the code where manipulation doesn't include changing the dynasm code
16:27 timotimo because i haven't done any x86 assembly in my life so far and i was hoping i could skip that experience for a long, long time
16:29 brrt why? it's fun :-)
16:29 timotimo you mean °FUN°?
16:30 timotimo the paragraph about type objects being asked for attributes seems like it doesn't explain what actually got returned instead of null ...
16:30 TimToady degree quotes?
16:31 brrt good question, i'd have to look at the respective at_key implementations
16:31 timotimo m: say q°FUN°
16:31 camelia rakudo-moar 189d1c: OUTPUT«FUN␤»
16:31 timotimo ^- yays!
16:32 timotimo m: say q×TATTERED×
16:32 camelia rakudo-moar 189d1c: OUTPUT«TATTERED␤»
16:32 timotimo m: say q+FINE+
16:32 camelia rakudo-moar 189d1c: OUTPUT«FINE␤»
16:32 timotimo m: say q*SUPERIOR*
16:32 camelia rakudo-moar 189d1c: OUTPUT«SUPERIOR␤»
16:32 TimToady just don't try :
16:32 timotimo m: say q≡EXCEPTIONAL≡
16:32 camelia rakudo-moar 189d1c: OUTPUT«EXCEPTIONAL␤»
16:32 timotimo m: say q°MASTERWORK°
16:32 camelia rakudo-moar 189d1c: OUTPUT«MASTERWORK␤»
16:33 timotimo all of those work ♥
16:33 timotimo m: say q♥Dwarf Fortress♥
16:33 camelia rakudo-moar 189d1c: OUTPUT«Dwarf Fortress␤»
16:33 TimToady some of the future is unevenly distributed to the past
16:34 brrt much wow
16:35 TimToady in fact, those would work in Perl 5 too
16:35 timotimo now it'd be swell if there was a way to have these delimitors mix in a role when they are used
16:35 TimToady (given appropriate uses at the top)
16:35 timotimo do we have any mechanism specced to do that? i think we really ought to
16:36 TimToady well, we used to do things like treat qx'' differently from qx"" in P5, but gave that up as a bad idea
16:36 TimToady and I think if you really want such a language tweak, you don't want the q
16:36 timotimo yes, i like the q vs qq more
16:37 TimToady just write your own quote slang regex
16:37 timotimo but how do i show dwarf fortress fans how awesome perl 6 is for being able to incorporate their memes into the language?
16:39 raiph joined #moarvm
16:58 dalek MoarVM/moar-jit: 9b3c483 | (Bart Wiegmans)++ | src/jit/emit_ (3 files):
16:58 dalek MoarVM/moar-jit: Pre-allocate stack space
16:58 dalek MoarVM/moar-jit:
16:58 dalek MoarVM/moar-jit: This is the first step in fixing the ABI for windows
16:58 dalek MoarVM/moar-jit: compatibility. The frame was 'walking upwards' with every
16:58 dalek MoarVM/moar-jit: C call that used any stack space; as a result gibberish
16:58 dalek MoarVM/moar-jit: was inserted in the place of the callee-save registers.
16:58 dalek MoarVM/moar-jit: review: https://github.com/MoarVM/MoarVM/commit/9b3c483fb4
17:10 ggoebel111116 joined #moarvm
17:43 brrt joined #moarvm
17:44 brrt jnthn, timotimo: ping?
17:47 brrt i'm wondering if it's an acceptable tradeoff to move all call arguments into a scratch register before putting them in their final place
17:47 jnthn brrrt: pong
17:47 brrt it makes dealing with the different call conventions greatly simpler
17:47 jnthn Well, I'll choose working over broken any day ;)
17:47 brrt no longer will i need the addarg and addarg_f macro
17:48 brrt but it is twice as slow for simple cases
17:48 jnthn Will it make a big performance impact?
17:48 jnthn Hmm
17:48 brrt probably not
17:48 brrt i don't think you'll feel register-to-register moves
17:48 brrt (which is worst case what will happen for the most of these)
17:48 jnthn *nod*
17:48 jnthn Well, simple, working code first
17:48 brrt ok
17:48 jnthn We can always optimize it later
17:49 brrt very well, i
17:49 brrt 'll get to it
18:15 Ven joined #moarvm
18:28 btyler joined #moarvm
18:41 brrt gcc-mingw32-w64 makes weeeird code
18:48 ggoebel111116 joined #moarvm
18:48 vendethiel- joined #moarvm
18:59 brrt also, it seems that win64 puts stack arguments in right-to-left order, as well
19:01 jnthn oh...
19:02 brrt which is good, because that's what posix does as well
19:03 brrt i.e. leftmost order is topmost (or actually, bottommost) on stack
19:04 rurban BTW: beware of icc, this is backwards
19:05 brrt if icc doesn't abide by an ABI then it is my professional opinoin that this is ICC's problem :-)
19:06 rurban Well, in the rare case if someones wants to use it, you should know. But I forgot the details. icc generates faster code, so...
19:07 brrt its actually a bit strange so few people use ICC, no?
19:07 brrt but then again, if faster code = broken code (as seen from POV from developers)
19:25 raiph joined #moarvm
20:03 woolfy left #moarvm
20:05 raiph joined #moarvm
20:12 timotimo brrt, assembly - or machine code - is actually more like a declarative language
20:12 timotimo these register switches are probably going to disappear while running
20:17 woolfy joined #moarvm
20:18 woolfy left #moarvm
20:42 brrt joined #moarvm
21:04 brrt register switches?
21:04 brrt timotimo?
21:05 timotimo shuffling data between registers
21:05 brrt oh, right
21:05 brrt i see :-)
21:05 brrt very well
21:05 timotimo you know how values can overtake each other inside the pipeline and stuff like that? :)
21:06 brrt well, i think that's something i don't want to know too much about
21:06 timotimo :D
21:10 TimToady another leak: :3('1201212') while 1
21:12 TimToady (I suspect, but let me run it a while to see if it stabilizes)
21:12 TimToady it does stabilize in parrot at about half a gig
21:14 TimToady what's the incantation to limit heap memory?
21:23 dalek MoarVM/moar-jit: 3d979c6 | (Bart Wiegmans)++ | src/jit/ (5 files):
21:23 dalek MoarVM/moar-jit: Fix stack argument passing.
21:23 dalek MoarVM/moar-jit:
21:23 dalek MoarVM/moar-jit: This should mean JIT on win64 lives again. NB that
21:23 dalek MoarVM/moar-jit: I cannot find it documented what the proper size
21:23 dalek MoarVM/moar-jit: for stack arguments should be, but I assume it is
21:23 dalek MoarVM/moar-jit: 8 bytes for all arguments.
21:23 dalek MoarVM/moar-jit: review: https://github.com/MoarVM/MoarVM/commit/3d979c6d56
21:23 brrt jnthn: please review ^^, see if it works now :-)
21:23 * brrt afk
21:23 brrt left #moarvm
21:43 jnthn TimToady: Limit heap memory at what level?
21:44 FROGGS when we're leaking we are leaking in C land I suppose
21:53 FROGGS do we need to free any of these? https://github.com/MoarVM/MoarVM/blob/master/src/math/bigintops.c#L861
21:54 jnthn store_bigint_result stores them inside an Int or so, iirc
21:55 FROGGS hmmm, almost thought so
22:00 jnthn brrt: Sadly, no luck; we SEGV much more now, and my debugger is incredibly confused about the stack, which looks rather corrupted.
22:03 jnthn brrt: 9b3c483 seems to be to blame, as if I try that one, it will explode in just the same way
22:03 jnthn (and it failed differently before that one)
22:07 FROGGS here is a valgrind output of :3("111") while... http://paste.scsys.co.uk/409598
22:07 FROGGS (cannot create gists for some reason since a few days)
22:08 FROGGS ==12728== 432,954 (432,064 direct, 890 indirect) bytes in 6,751 blocks are definitely lost in loss record 1,718 of 1,763
22:08 FROGGS ==12728==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
22:08 FROGGS ==12728==    by 0x4C2C74F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
22:08 FROGGS ==12728==    by 0x4FA43DB: initialize_ssa_var_info (graph.c:810)
22:11 jnthn Hmm
22:11 jnthn But that wouldn't explain a leak over time; it'll only spesh stuff once...
22:12 FROGGS hopefully :P
22:13 jnthn Well, think I can fix that one easy enough :)
22:14 FROGGS I'm dumping all reachables now too
22:15 FROGGS ==12997== 4,646,152 bytes in 112,982 blocks are still reachable in loss record 1,768 of 1,772
22:15 FROGGS ==12997==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
22:15 FROGGS ==12997==    by 0x4F8590C: deserialize (MVMArray.c:1098)
22:16 timotimo does valgrind cope with out "just crash" exit style?
22:16 FROGGS timotimo: no, that would surprise me :o)
22:17 FROGGS would make sense to do the "slow shutdown" for a valgrind run...
22:17 jnthn Well, insead of doing 1000 iterations of that loop, do 100,000
22:17 jnthn It should amplify whatever it is that's leaking
22:17 timotimo i'm not sure we can even do the slow shutdown without crashing there, too :P
22:17 timotimo hasn't been active and probably not tried for a long time
22:17 FROGGS that fits the iteration count:
22:17 FROGGS ==12997== 929,566 bytes in 999 blocks are still reachable in loss record 1,742 of 1,772
22:17 FROGGS ==12997==    at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
22:17 FROGGS ==12997==    by 0x4F67A74: MVM_validate_static_frame (validation.c:588)
22:18 FROGGS jnthn: okay, but when I do that, I'll go to bed :o)
22:18 FROGGS because slow valgrind is slow
22:19 jnthn oh
22:19 jnthn Well, maybe 10,000 is enough :)
22:21 dalek MoarVM: b2e3884 | jnthn++ | src/spesh/graph.c:
22:21 dalek MoarVM: Fix a number of SSA/dominance memory leaks.
22:21 dalek MoarVM:
22:21 dalek MoarVM: Found by FROGGS++ using Valgrind.
22:21 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/b2e388414f
22:22 jnthn That should clear up a number of things
22:22 FROGGS nice!
22:23 timotimo .o( also, less memory use :P )
22:24 FROGGS okay, valgrind is running now
22:26 Ven joined #moarvm
22:28 ventica joined #moarvm
22:28 dalek MoarVM: 895bcc8 | jnthn++ | src/spesh/inline.c:
22:28 dalek MoarVM: Fix memory leak in inlining.
22:28 dalek MoarVM:
22:28 dalek MoarVM: Found by FROGGS++ using Valgrind.
22:28 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/895bcc89e6
22:33 jnthn That's all the spesh ones at least :)
23:19 ggoebel111117 joined #moarvm
23:22 [Coke]_ joined #moarvm
23:41 dalek MoarVM: 6753df5 | jnthn++ | src/core/interp.c:
23:41 dalek MoarVM: Detect frame change in extop call.
23:41 dalek MoarVM:
23:41 dalek MoarVM: This means we'll be able to remove the nasty cur_op fiddling in the
23:41 dalek MoarVM: Rakudo ext-ops that invoke.
23:41 dalek MoarVM: review: https://github.com/MoarVM/MoarVM/commit/6753df5d76
23:44 timotimo could i emit a method call in spesh without allocating new registers?
23:47 jnthn Don't think so
23:48 jnthn But the new register thing is really near the top of my spesh todo list now.
23:49 timotimo good :)
23:49 timotimo i forgot what the other thing was that i wasn't able to do so far due to that :\
23:58 timotimo i wonder what'll happen next before register stuff happens :)

| Channels | #moarvm index | Today | | Search | Google Search | Plain-Text | summary