Perl 6 - the future is here, just unevenly distributed

IRC log for #p6dev, 2016-04-06

| Channels | #p6dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:48 ilbot3 joined #p6dev
01:48 Topic for #p6dev is now Perl 6 language and compiler development
02:07 _z left #p6dev
03:21 geekosaur joined #p6dev
03:37 vendethiel joined #p6dev
07:25 masak good morning, #perl6
07:26 masak b2gills: if we had a .prime-factorize, should it return a Bag? :)
07:26 masak m: say bag("b", "a", "b", "b", "a").list
07:26 camelia rakudo-moar c8ec5a: OUTPUT«(a => 2 b => 3)␤»
07:26 masak hm
07:28 masak oh, this is #p6dev... :)
07:51 RabidGravy joined #p6dev
08:40 btyler joined #p6dev
09:01 RabidGravy "Stage mbc        : Memory allocation failed; could not allocate 6386688 bytes
09:01 RabidGravy Makefile:434: recipe for target 'CORE.setting.moarvm' failed
09:01 RabidGravy make: *** [CORE.setting.moarvm] Error 1"
09:01 RabidGravy Boo
09:02 RabidGravy it worked a month ago
09:02 psch ...building rakudo?
09:04 brrt joined #p6dev
09:04 RabidGravy yeah, albeit on a Raspberry Pi
09:05 psch ah, okay.  guess moar went up in build time memory demand
09:06 * RabidGravy stops all the things and tries again
09:14 jnthn MoarVM itself has mostly come down in memory use. But only a bit, and probably not enough to make up for growth in CORE.setting :)
09:16 RabidGravy this is a Pi 2 so the rev 1 is going to be utterly stuffed
09:30 RabidGravy stopping samba and icecast freed enough memory
11:42 brrt joined #p6dev
11:51 stmuk hmmm pi
13:26 dalek roast: 2a715e5 | lizmat++ | S (2 files):
13:26 dalek roast: Adapt tests to changed Hash.gist behaviour
13:26 dalek roast:
13:26 dalek roast: As implemented with 6a2ff75a61fdd49b55
13:26 dalek roast: review: https://github.com/perl6/roast/commit/2a715e572e
13:32 dalek roast/6.c-errata: ca4aca8 | lizmat++ | S (2 files):
13:32 dalek roast/6.c-errata: Adapt tests to changed Hash.gist behaviour
13:32 dalek roast/6.c-errata:
13:32 dalek roast/6.c-errata: As implemented with 6a2ff75a61fdd49b55
13:32 dalek roast/6.c-errata: review: https://github.com/perl6/roast/commit/ca4aca88d8
13:32 lizmat [Coke]: ^^^  These tests should maybe be marked todo ???
13:33 lizmat as they will now fail on vanilla 6.c / 2015.12
13:34 skids joined #p6dev
13:34 jnthn lizmat: No, 2015.12 supports 6.c, not 6.c + errata :)
13:34 jnthn lizmat: And we don't truly version errata.
13:34 jnthn It's meant to be unconsequential enough that it doesn't matter
13:35 lizmat ok, so I've done the right thing ?
13:35 jnthn Yeah, looks like :)
13:35 lizmat ok  *phew*  :-)
13:35 jnthn errata is "what we wish 6.c had been if only we'd had a time machine" :)
13:35 lizmat fwiw, I think these tests have improved with this change
13:35 lizmat as more of the structure is visible in the test string
13:38 * skids wonders what the performance hit of allowing Iterator.is-lazy to also return an Int for "I'm lazy but I know I'll produce this many more values" would likely be.
13:39 jnthn I worry more about the API complexity cost.
13:41 skids Well, any suggestions how to make the API actually handle those cases efficiently without adding complexity?
13:42 jnthn What does it handle inefficiently now?
13:42 skids Things like List.combinations(Range).elems need to iterate al combinations.
13:42 jnthn Or rather, why is how many values something marked lazy *can* produce interesting?
13:43 jnthn Isn't there a count-only that's used for that?
13:43 lizmat skids: but it doesn't, it the iterator supplies a "count-only" method
13:43 lizmat *if
13:43 skids count-only iterates.
13:43 skids Or he deafult does.
13:43 lizmat skids: it doesn't have to if there's an alternative
13:43 skids More importantly it exhausts.
13:43 lizmat yes, because the default has no choice
13:45 jnthn Indeed, but you can implement an iterator that does something smarter by implementing count-only
13:45 skids Well, it's docced to exhaust.
13:45 skids Which I consider a contract.
13:45 jnthn What is?
13:45 skids count-only
13:45 lizmat skids: e.g., the iterator for Str.comb() has: method count-only() { nqp::p6box_i($!pos = $!chars) }
13:46 lizmat this both exhausts (because it sets $!pos) and returns the number of chars immediately
13:46 lizmat skids: so, yes, it should exhaust, but that can be as easy as (re-)setting an attribute
13:48 skids I always took it that count-only was added so that .elems would not be used for the side effect of exhausting.
13:48 brrt joined #p6dev
13:48 timotimo we've made this design so that users will have to explicitly ask for values from an iterator to be kept
13:49 skids Right but values != pre-known number of values.
13:51 jnthn .elems should ensure any side-effects are performed.
13:51 jnthn As should .count-only
13:51 lizmat jnthn: but caching is not a side-effect, right >
13:51 lizmat ?
13:52 jnthn That doesn't mean you have to produce any values that are side-effect-free in order to answer count-only though
13:52 jnthn lizmat: Caching in what sense?
13:53 skids In that case what I'm looking for is to add the ability to know how many elems you have prior to performing side-effects, when possible, so I guess it would have to be an entirely new API addition.
13:53 lizmat if you numerify an iterator, it will cache the values produced by the iterator
13:53 lizmat sorry, Seq
13:54 skids ...which can be a lot of cache.
13:54 lizmat if you call .elems on a Seq, it won't cache
13:55 jnthn Right, .elems exhausts...
13:55 lizmat indeed, and it doesn't cache
13:56 jnthn So that @foo.grep(/.../).elems doesn't have to build a result list, for example
13:56 b2gills Perhaps there could be a role that states "this iterator knows how many values it produces, and requires reimplementing .count-only()" ( I have already tried this and I think it exposes a bug in List.Int(), or a bug in my own brain )
13:56 lizmat jnthn: indeed
13:56 jnthn b2gills: Why do we need a role, you just implement the method in your iterator, like all the "I know better than the default" methods to...
13:57 jnthn *do
13:57 lizmat m: my $a = <a b c d a>.grep(* eq "a"); say $a.elems; for @$a { .say }
13:57 camelia rakudo-moar c8ec5a: OUTPUT«2␤This Seq has already been iterated, and its values consumed␤(you might solve this by adding .cache on usages of the Seq, or␤by assigning the Seq into an array)␤  in block <unit> at /tmp/3gF0giOmjX line 1␤␤»
13:57 b2gills The role is to push more information to the compiler
13:57 lizmat m: my $a = <a b c d a>.grep(* eq "a"); say +$a; for @$a { .say }
13:57 camelia rakudo-moar c8ec5a: OUTPUT«2␤a␤a␤»
13:57 jnthn b2gills: But...what would it do with it?
13:58 lizmat b2gills: but you can already override all of the Iterator methods ???
13:58 skids Another thing I considered was a LimitedIterator where you predeclare the number of elements, and it chops if you try to produce more, or pads with Nils if you do not produce enough, and whether that would be useful for infix:<Z>  This could double for my purposes in the case where the limit matches reality.
13:59 lizmat skids: sounds like a useful role to have in Rakudo::Internals
13:59 jnthn Sounds more like a module to me.
13:59 b2gills There was some code that I was looking into that would work a little better if it had a way of knowing that method was re-implemented, but again it may not be the best way of changing it
14:00 jnthn No, introspection-based solutions are generally not a good way to go
14:00 b2gills I was thinking of using it for multi-method dispatch
14:00 skids Well, a module doesn't prevent combinations.elems from taking huge swaths of time/cache.
14:01 jnthn I still don't see why it has to *now*
14:02 jnthn Why can't it hand back an iterator that implements count-only?
14:02 skids It doesn't in the general case but the range case needs to build a map, and the map hardcodes it's Seq/Iterator, so there's no way to inject that without a full iterator re-implementation in that one case.
14:03 skids (map in my PR, GATHER currently)
14:03 skids It's the problem of having to do everything from scratch just to add a behavior tweak.
14:04 jnthn You can grab the iterator underlying a Seq, no?
14:05 jnthn So if you got the iterator back from map, you can just mix in a role that adds the count-only?
14:05 skids Sure and do what a run-time mixin?
14:05 jnthn Should work out fine
14:06 jnthn In theory if it's your map you can even get away with "does" to do it in place and return the very same wrapping Seq
14:06 skids I was under the impression that would be frowned on in core as hard to optimize.
14:06 jnthn Well, yes and no.
14:06 jnthn If you do mixins on a hot-path, it's a problem, sure
14:07 skids Well, we don;t know what the user is going to do with it in this case.
14:07 jnthn But it doesn't seem to me calls to combinations is the hot path, it's producing the combinations...
14:07 jnthn And there's no added cost when iterating the combinations
14:08 skids I was just thinking beyond this to more general cases, how to make that pre-knowlege useful in a composeable fashion.
14:09 jnthn Well, the other option that avoids mixins is just an iterator that delegates to the other, but is given a closure to run for count-only
14:10 timotimo composition instead of ... the other thing
14:10 skids Right and that begs the question of how smart we are going to get collapsing Iterator stacks in the optimizer.
14:11 skids Because if the answer is "very smart" then just wrapping it makes sense
14:11 moritz .oO( ASAP = As Smart As Possible :-)
14:11 timotimo hm, how good is the "handles" trait? isn't it very good?
14:11 timotimo maybe we have to use the compiler services to make it the best it can be
14:12 moritz it's good enough that I used it on occasion
14:12 timotimo not sure how it's currently implemented
14:12 timotimo moritz: i meant in terms of raw overhead
14:12 jnthn Well, iterator chains are just method calls from one to the next
14:12 timotimo like, do we generate a little closure that makes sure the right thing gets called on the right object?
14:12 moritz timotimo: probably uses the same closure thingies that accessors used to
14:12 timotimo or do we somehow have something smarter than that?
14:12 timotimo OK, so a target for the compilation services
14:13 skids Still, it leaves me wondering whether the fact that the Seq/Iterator can non-exaustively tell you its elems would be useful to allow the querying of in the future.
14:14 skids Or some less wordy version of that sentence.
14:14 lizmat skids: but it can ?
14:14 skids How?
14:15 lizmat just create a "count-only" method that does what it is supposed to do without iterating
14:15 timotimo if the "count-only" method doesn't exhaust, that's fine
14:15 timotimo but how is the code that uses the iterator supposed to know it can do that?
14:15 skids RIght but how does an external user of the Seq know that that is the case.
14:16 lizmat the external user can never know this, not without looking inside
14:16 skids (before accidentally exhausting or caching)
14:16 skids They could if we had API for it.
14:16 timotimo .start-prevent-exhaustion, .end-prevent-exhaustion
14:17 lizmat this feels like a lot of overhead for the generic iterator case  :-(
14:17 timotimo mhm
14:17 skids Or.. just have is-lazy return an Int in that case and then if you get a Bool you know, "nope"
14:19 lizmat I still don't see how this would generically work
14:19 lizmat away for 30 mins or so
14:22 skids Code that thinks it can do something useful with a pre-known .elems (like plan a hyper batch configuration, prealloc, whatnot) could 'given .is-lazy { when Int { do something useful } Bool { if $_ { do lazy stuff } else { do other stuff } }'
16:02 sortiz joined #p6dev
16:07 vendethiel joined #p6dev
16:36 lizmat fwiw, thinking about it more, I think I like the idea of an iterator being able to tell a client how many values it will produce without actually producing them
16:36 lizmat using "is-lazy" for this, feels hacky to me
16:37 lizmat I'm more thinking along the line of having a method "will-produce"
16:37 lizmat that would be implemented in the Iterator role as "method will-produce() { Nil }"
16:41 lizmat jnthn: ^^^ thoughts ?
16:59 Woodi but what it means "without producing" ? already produced&available without more work   or  total that can be produced from this point ?
17:26 jnthn lizmat: That name's certainly a bit off. May also want to consider ranges as well as an Int
17:27 jnthn lizmat: So grep for example could return ^$underlying-iterator's-prediction
17:27 jnthn lizmat: It may have some kind of use in batching for parallelism also
17:28 jnthn predict-elems or so might be closer name wise but probably still not right
17:32 lizmat Woodi jnthn : I envisioned this method to be called before any values have been produced
17:32 lizmat hence the "will-"
17:36 jnthn Yeah, I guess it puns badly with .produce
17:36 jnthn And doesn't really indicate a unit
17:36 jnthn It's too late for it now, though it's a slight pity count-only wasn't called elems-only...
17:37 lizmat but elems-only could be construed as needing to produce elems only
17:38 lizmat hence I named it "count-only"
17:39 jnthn But we do only need the number of elems? :)
17:40 lizmat iterators are really about a number of values, so to me that's self-explanatory
17:42 lizmat but I can also see value (pun intended) in a method that indicates: we're going to produce at least this many more values now
17:42 lizmat e.g. the IO::Handle.words Seq which reads in chunks, could perceivably do all the work on that chunk and return that number
17:44 jnthn I thought this was for asking for a prediction *before* iterating?
17:44 jnthn Asking how many things you've got "in the buffer" is the wrong way around
17:45 jnthn The design we have is pretty clearly about pushing requirements down the chain
17:45 jnthn Thus push-at-least
17:46 b2gills I think there should be some two way communication ( .is-lazy() for example )
17:46 * lizmat tries to wrap her head around racing an IO::Handle.words Seq
17:48 b2gills It might even be useful if a sequence knows it is sorted so that a .sort() ( that is with no arguments ) doesn't need to do any work. (as an example)
17:48 jnthn b2gills: Yes, for for static knowledge about an iterator, not dynamic.
17:49 jnthn b2gills: That sounds like one of those ideas that sounds great in theory but almost never crops up in practice...
17:49 jnthn lizmat: You'd want to race the operation you do on it, most probably
17:49 jnthn $foo.words.hyper.map(...) means that the parallelism is applying to the map
17:50 b2gills I didn't say it was a good example, just that it is sometimes a good idea for knowledge to pass both ways in a chain
17:50 lizmat but it would be nice if the hyper could takes chunks of words that sorta match chunks read from disk ?
17:51 jnthn lizmat: The thing implemenitng the hyper-ing can happily use push-at-least (if it isn't already)
17:51 lizmat .oO( I guess I should take a closer look at HyperSeq first )
17:51 jnthn HyperSeq, .hyper, and .race are on my list of things I really need to look at also :)
17:52 jnthn What made it in was a reasonably sensible interface backed by proof of concept code :)
17:57 b2gills jnthn: That reminds me I meant to ask your opinion on this set of hyper/race map/grep tests https://github.com/perl6/roast/commit/5016fe1ef278972cc650c0aa27f025bfd657938f
17:58 lizmat m: use NativeCall; sub system (Str) is native {}; say system 'cowsay "what security?"'
17:58 camelia rakudo-moar c8ec5a: OUTPUT«sh: cowsay: command not found␤(Mu)␤»
17:58 jnthn The sleep doesn't inspire me much :)
17:58 b2gills I went for simple so it does rely on timing a little bit. I could have written it so that it is purely deterministic but that would rely on more features working the same
17:58 lizmat moritz: perhaps we should uninstall NativeCall on camelia ?
17:59 b2gills jnthn: to be fair it sometimes runs correctly without the sleeps
18:00 jnthn lizmat: It's been that way for ages and I don't think anyone's abused it...I'm more inclined to hope people will continue not to suck and make it so we can continue to try NativeCall things out ;)
18:00 jnthn b2gills: Then I'd remove them. Tests that depend on sleeping tend to be flappy
18:01 timotimo nobody needs to go through system via nativecall when you already can use all of nqp::
18:02 jnthn b2gills: I don't think the sleeps are needed at all
18:02 jnthn b2gills: They look fine without them
18:03 b2gills I tested it many times, it does sometimes end up out of order
18:03 jnthn Yeah, but is that a test bug or an impl bug? :-)
18:03 jnthn Could easily be that the test is ok
18:03 b2gills I think it is purely that Rakudo+MoarVM isn't entirely deterministic
18:04 jnthn It doesn't have to be deterministic, it just has to give the correct results :)
18:04 b2gills I did have a version that used a start block, but it was very messy
18:04 jnthn Which for .hyper should be ordered
18:04 jnthn Doing it with promises seems sensible enough
18:04 b2gills I wanted a test that showed implementors that it should be out of order if .race was called
18:05 jnthn The hyper ones are OK
18:05 jnthn But the .race ones are bogus
18:05 b2gills So I tried to make it deterministic with non-deterministic features
18:05 jnthn They over-specify it
18:06 jnthn And, knowing a bit about how it's implemented, you'll not get those to pass reliably
18:06 jnthn And that's not a bug.
18:06 b2gills its like testing a random number generator
18:06 jnthn Pretty much
18:06 jnthn I'm not sure there's any value in trying to test that race doesn't preserve order
18:07 jnthn Just that it gives the right values in some order
18:07 b2gills It was meant more as an indicator to future implementations that it shouldn't even try to keep the order
18:08 b2gills perhaps just a sort afterwards ( and remove the sleeps )
18:08 jnthn Yeah, for .race I'd sort afterwards
18:09 jnthn Typically the way these things are implemented is a "master" thread sending batches of work off to workers
18:10 jnthn And so it's very easy for the results to come back in a difference sequences than the work was completed.
18:10 b2gills well part of the point of the tests are that they are supposed to be implementation agnostic so it may be a good thing I don't know the details
18:12 jnthn Sure, but the .race ones assume too much of the details :)
18:12 b2gills If that's it I will start work on a PR
18:14 jnthn Sounds good
18:14 jnthn Thanks for adding some tests... :)
18:24 sortiz joined #p6dev
18:46 b2gills done https://github.com/perl6/roast/pull/111 ( I so want to test that the first value isn't a 1, and the last isn't a 5 in the .race() tests )
18:51 jnthn yeah but it legit could be sometimes :)
19:18 * jnthn off to rest for today, but for those not following #moarvm also: the EVAL memory leak issue (and no doubt numerous other leaks) are fixed in HEAD, and I've done some GC tuning. Expcet to try another tuning patch tomorrow, then will bump HEAD.
19:19 jnthn And full write-up in my next post :)
19:30 dalek roast: 0eda7e0 | (Brad Gilbert)++ | S07-hyperrace/ (2 files):
19:30 dalek roast: Remove timing related stuff from hyper/race tests
19:30 dalek roast: review: https://github.com/perl6/roast/commit/0eda7e0b46
19:30 dalek roast: 25649a8 | moritz++ | S07-hyperrace/ (2 files):
19:30 dalek roast: Merge pull request #111 from b2gills/reversed_hyper_race_2
19:30 dalek roast:
19:30 dalek roast: Remove timing related stuff from hyper/race tests
19:30 dalek roast: review: https://github.com/perl6/roast/commit/25649a82c2
19:32 dalek rakudo/nom: 2a04e13 | skids++ | src/core/List.pm:
19:32 dalek rakudo/nom: Combinations fixes
19:32 dalek rakudo/nom:
19:32 dalek rakudo/nom:  Fix sub combinations to return () not ((),) for absurd $k
19:32 dalek rakudo/nom:  Fix List.combinations(Range) to use right values with careted Range endpoints
19:32 dalek rakudo/nom:  Fix List.combinations(Range) to return () on an iterably-empty range
19:32 dalek rakudo/nom:  Ensure only one empty list appears in result, only when a k==0 is involved
19:32 dalek rakudo/nom:  Use a map/map instead of gather/take, no discernable performance impact
19:32 dalek rakudo/nom:  ...someone may wish to weigh in on which will optimize better in future.
19:32 dalek rakudo/nom:  ...this may help a plan to avoid pulling during .elems on the resulting Seq
19:32 dalek rakudo/nom: review: https://github.com/rakudo/rakudo/commit/2a04e135ab
19:32 dalek rakudo/nom: 0287290 | moritz++ | src/core/List.pm:
19:33 skids moritz++
19:35 dalek roast: ab1da4f | (Christopher Bottoms)++ | S32-str/lines.t:
19:35 dalek roast: Added test for RT #126270
19:35 dalek roast: review: https://github.com/perl6/roast/commit/ab1da4f924
19:38 dalek roast: 38fdf08 | usev6++ | S04-statements/return.t:
19:38 dalek roast: Use 'run' to test for returning outside of routine
19:38 dalek roast:
19:38 dalek roast: We can't test for 'Attempt to return outside of any Routine'
19:38 dalek roast: with our toolchain from Test.pm or Test::Util.
19:38 dalek roast: review: https://github.com/perl6/roast/commit/38fdf0819b
19:38 dalek roast: 4832f2a | moritz++ | S04-statements/return.t:
19:39 dalek roast: Merge remote-tracking branch 'origin/return_outside_routine'
19:39 dalek roast: review: https://github.com/perl6/roast/commit/4832f2a30b
19:44 dalek rakudo/nom: ec52cce | skids++ | src/core/Any-iterable-methods.pm:
19:44 dalek rakudo/nom: Fix several issues with List.squish
19:44 dalek rakudo/nom:
19:44 dalek rakudo/nom:   Only evaluate :&as and :&with the minimum number of times
19:44 dalek rakudo/nom:   Prevent IterationEnd from leaking into :&as and :&with calls
19:44 dalek rakudo/nom:   Use intuitive ordering of :&with args for non-commutative :with
19:44 dalek rakudo/nom:   Always pass :&with consecutive elements of the original list
19:44 dalek rakudo/nom:   Once committed to doing a push-all, no need to mutate anymore
19:44 dalek rakudo/nom:   Might possibly fix r-j RT#126527 -- needs someone to test
19:44 dalek rakudo/nom:   Will commit roast tests for these behaviors after merge
19:44 dalek rakudo/nom: review: https://github.com/rakudo/rakudo/commit/ec52cce383
19:44 dalek rakudo/nom: d17e3a1 | skids++ | src/core/Supply.pm:
19:56 dalek roast: 0f70bbf | moritz++ | S32-str/lines.t:
19:56 dalek roast: Fix test plan
19:56 dalek roast: review: https://github.com/perl6/roast/commit/0f70bbf650
19:58 psch skids++
19:58 psch i'm gonna check ec52cce against r-j
20:31 travis-ci joined #p6dev
20:31 travis-ci Rakudo build passed. Moritz Lenz 'Merge pull request #732 from skids/combos
20:31 travis-ci https://travis-ci.org/rakudo/rakudo/builds/121249048 https://github.com/rakudo/rakudo/compare/c8ec5aca8bb4...0287290dd114
20:31 travis-ci left #p6dev
20:34 colomon joined #p6dev
20:42 brrt joined #p6dev
21:11 hankache joined #p6dev
21:16 travis-ci joined #p6dev
21:16 travis-ci Rakudo build passed. Moritz Lenz 'Merge pull request #685 from skids/squish
21:16 travis-ci https://travis-ci.org/rakudo/rakudo/builds/121251835 https://github.com/rakudo/rakudo/compare/0287290dd114...61d231c5d139
21:16 travis-ci left #p6dev
21:29 dalek roast: 0b3d846 | lizmat++ | S04-statements/return.t:
21:29 dalek roast: Unfudge now passing test
21:29 dalek roast: review: https://github.com/perl6/roast/commit/0b3d846a5c
21:33 colomon joined #p6dev
22:16 skids joined #p6dev

| Channels | #p6dev index | Today | | Search | Google Search | Plain-Text | summary