Perl 6 - the future is here, just unevenly distributed

IRC log for #perl6-toolchain, 2016-01-07

| Channels | #perl6-toolchain index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
18:13 ilbot3 joined #perl6-toolchain
18:13 Topic for #perl6-toolchain is now Discussion area for the Perl 6 toolchain | https://github.com/perl6/toolchain-bikeshed
18:13 nine The rest has either already been fixed and just awaits confirmation by the reporter or are straight implementation bugs.
18:27 lizmat joined #perl6-toolchain
18:29 nine is the irclog already on?
18:29 flussence nope
18:30 * lizmat is listening :-)
18:30 flussence moritz runs the logbot, so we'll have to wait until he's got a free moment
18:31 flussence (AIUI, might be a while)
18:31 mst if anything important happens, I can supply
18:33 nine Just to get that thought on record: while system wide precompilation is really hard, what's easy is to provide precompiled versions of leaf modules system wide, i.e. modules that have no dependencies.
18:34 lizmat nine: this is about packaging ?
18:34 nine More or less. Packaging makes the issue much more visible. But even without packaging we currently don't use the system precomp files.
18:35 nine That's why I think the next step should actually be to disable precomp on installation as it's pretty useless as it is right now.
18:35 mst I think we should kill system precomp for the moment
18:35 nine However that does not mean that I want to rip out the code completely.
18:35 mst focus on making sure the ~/.perl6 version is solid
18:35 mst then re-introduce system precomp more thoughtfully
18:35 mst with a view to packaging
18:36 mst somebody should look at how distros do stuff like link optimisation
18:36 nine I think we should later go back and add it back in step by step maybe beginning for example with those leaf modules.
18:36 mst which is a post-install step that cares about transitive dependencies
18:36 lucasb joined #perl6-toolchain
18:36 mst that might give us some insight into what we can do in the same vein
18:37 nine lucasb, alpha123: I'd need your github ids for https://github.com/perl6/toolchain-bikeshed
18:38 mst we must remember to drag leont in too, since while he's been doing testing, I full expect this place to end up being the intersection of installation, testing and packaging
18:38 mst since that's basically what's happend with perl5
18:38 lucasb oh hai, nine! thanks for the invite, but I'll probably not be editing that repo. I just came here out of curiosity :)
18:39 mst lucasb: I'm sorry, did you confuse us with people who were offering you a choice?
18:39 mst github id please, you can always just not use the commit bit :)
18:39 lizmat nine: for my understanding: would that mean we would need to re-install modules after each setting compile ?
18:40 lizmat because I think it's *very* important we keep at least that
18:41 mst re-install?
18:42 flussence s/install/compile/?
18:42 mst lizmat: ?
18:42 nine lizmat: no, never re-install
18:42 alpha123 nine: yeah I'm just here because it's interesting, I'm probably not that useful
18:42 lizmat nine: *phew*
18:42 nine lizmat: it's all about re-compilation
18:43 mst alpha123: I've heard that many times before. github id anyway please, even if you only ever use it once it was still 100% better than not
18:45 lizmat feels to me that the only way to fix precomp with packaging, is to allow relative paths ?
18:45 nine Btw. I hate that I didn't rip out the version directories in ~/.perl6. Because they mean that you have to re-install modules installed to your $HOME after upgrading rakudo.
18:45 alpha123 mst: https://github.com/alpha123
18:45 mst nine: we can turn that ibnto one command in an installer
18:46 mst generally, speaking as a local::lib author, I hate that I didn't put version directories in there by default
18:46 mst though that was mostly an XS module thing
18:46 nine lizmat: precomp can only work with leaf packages or when the precompilation happens in a post-install script. Otherwise we'd end up changing packaged files which is a no-go
18:46 mst I think post-install script may be very viable though
18:46 mst anything else would require packaging N*M*O*P*Q versions or something similarly insane
18:47 nine mst: true, it's totally safe to just rename the version directory to the newer version... at least right now
18:47 mst nine: yeah, and later if it ever isn't, we'll be glad to've had the choice
18:47 mst so ... I think solve it via tooling is correct
18:47 mst could even have a rakudobrew step that automatically shifts stuff for you
18:48 lizmat I can't help but thinking we would need to make precomp faster
18:48 lizmat s/faster/better/
18:48 lizmat I mean, we're talking about composability all of the time
18:48 nine Btw. if anyone has trouble following what I say, please just ask. I've been thinking about this stuff for weeks on straight. Some things may be very obvious to me and totally nonsensical to everyone else.
18:48 lizmat yet, precomp files are anything but composable
18:48 mst if precomp didn't require all transitive deps
18:48 mst right
18:48 nine even if you're just here out of curiosity! ^^^^
18:49 mst right, confused newbies are a *great* tool for forcing people to clarify their thinking
18:49 lizmat so before we try to fix the problem at this level
18:49 lizmat I wonder what is making it so hard to have composable precomp
18:49 nine Well our precomp is static linking with dynamic loading of the linked code. A combination that's difficult to handle.
18:49 lucasb idk much, but the only opinion I have about this is to *not* nest the site and vendor directories inside the perl/core one. I think it would be cleaner with they existed at the same level. I already told nine about this and he was kindly to acknowledge the situation. But I understand changing this now would inflict having to keep telling new users to "nuke everything".
18:50 lizmat well, to me that just feels like another layer of punisihing core devs
18:50 lizmat to an extent, I feel like I'm my boss now, many years ago
18:51 mst I don't think we have to tell new users to nuke everything
18:51 nine But having pure static linking with embedded code wouldn't be that much better. Because then installing a new version of some dependency would still not make the reverse-dependencies use the new code, confusing the users.
18:51 lizmat when I told my boss that what he wanted, wasn't possible
18:51 mst I think we'll just need to figure out how to upgrade things under them without them noticing
18:51 lizmat guess what, in the end it turned out to be possible after all
18:51 lizmat even though I was convinced it couldn't be done
18:51 nine mst: yes, we just need to detect the crewup in the install script and move some directories. Manageable.
18:52 nine lizmat: been there, done that :)
18:52 mst composability of precomp files is above my pay grade right now
18:52 mst but no matter what, I think 'fall back to .perl6, then start again carefully' is going to be a great option
18:52 mst means we're less likely to trap ourselves so badly
18:52 nine And it means that we have loads of time.
18:53 mst and I'd rather look at cleaning up the non-precomp CUR stuff first and making that a bit more flexible and making sure that can be packaged
18:53 mst and then we weave precomp into it after that
18:53 nine lucasb: please add the moving of the perl6 repo to https://github.com/perl6/toolchain-bikeshed/blob/master/README.md as todo
18:54 lucasb nine: can I just open a issue instead? issues are better, they can be commented on.
18:54 flussence my 2¢ is that it shouldn't put anything in "system" dirs by default, that ought to require some sort of `perl6 --I'm-doing-something-fancy` invocation
18:55 flussence that'd neatly skip having to put a bunch of mkdir juggling right in the compunit code, because IMO that sort of stuff belongs in makefiles
18:56 mst the compunit code's mkdir juggling is hilariously wrong and awful, but I've not had a chance to go through and work out what it's supposed to be yet
18:58 flussence and since perl6 is being positioned as a compiler, there's nothing wrong with giving it a million command line switches for this sort of stuff. Every other compiler does it
19:07 nine mst: the mkdir juggling was just one of those shortcuts that made it into released code. It's really make install that should create the system repos and it does. Except for the precomp directory in each repo I think. I missed that.
19:07 nine But the mkdir code also helps with ~/.perl6 where make install won't do anything
19:07 mst I think "first .install() into a CUR creates the directories" is likely best
19:07 lizmat nine: is @*INC still a thing, or not ?
19:08 nine lizmat: nope, that's gone
19:09 lizmat ok, because src/core/CompUnit/RepositoryRegistry.pm seems to still contain references to it
19:10 lizmat as does ModuleLoader.nqp
19:10 nine oh in method files
19:10 lizmat I will get out the chansaw
19:10 nine CompUnit::RepositoryRegistry::files can go alltogether.
19:13 lizmat nine: will make it so
19:16 lizmat nine: so what's up with @INC in method setup-repositories() ?
19:17 nine That's just a local variable with a recognizable name
19:17 lizmat nine: ah, I see... it's only used internally
19:18 nine just for confusing the reader ;)
19:22 tadzik joined #perl6-toolchain
19:22 flussence yayit'stadzik!
19:23 tadzik \o/
19:23 tadzik I'll lurk here :)
19:24 flussence yeah yeah, that's what I said about #moarvm... :)
19:25 tadzik :P
19:26 mst tadzik: but you're already the author of a key toolchain component accidentally
19:27 mst several of the things I have in mind are going to lean on rakudobrew ;)
19:30 tadzik yeah, panda also became the thing for a while :P
19:35 mst I think we'll need to pop a couple more components off the yak stack before I start having ideas that'll lean on panda
19:35 mst so depending on how uppity shadowcat's customers are you may yet get a fair stay of exeuction there ;)
19:38 tadzik :o
19:38 tadzik aw, shucks, I was planning to transfer all my stuff to community org and retire :D
19:39 Zoffix joined #perl6-toolchain
19:40 ugexe joined #perl6-toolchain
20:03 diakopter joined #perl6-toolchain
20:06 diakopter such solace
20:08 [Coke] If tadzik wants, we can fork panda and keep the "official" copy in perl6 to avoid pain.
20:08 [Coke] (I'm also happy to leave it where it is)
20:10 mst I trhink transferring stuff to orgs is a good thing in general
20:10 mst I think we can't afford to let tadzik escape just yet either way ;)
20:11 tadzik *blushes*
20:12 tadzik well, almost all I ever did in Perl 6 was dogfooding
20:12 tadzik now I don't really use Perl 6 anymore, the interests have shifted, so I don't really work on my stuff these days
20:12 tadzik (because I don't really need it)
20:12 mst right, so let's get the repos into community orgs
20:13 mst and you can stick around and answer stupid questions while we figure out who's running with the baton next
20:13 tadzik at times when I do use Perl 6 all my stuff just works for me, largely because of all the awesome people who quickfix it :)
20:16 lucasb tadzik++  # for all the cool stuff he did
20:16 lucasb I'm +1 for moving rakudobrew and panda to perl6 org
20:19 diakopter +
20:21 mst right, then it'll be easier for us to inflict commit bits on people
20:21 FROGGS joined #perl6-toolchain
20:21 FROGGS o7
20:21 tadzik oh hai
20:23 diakopter nine: lol
20:24 FROGGS can I ask my panda/star related questions too?
20:24 nine absolutely
20:25 FROGGS cool
20:25 FROGGS so, I bootstrap panda in star, and it just goes off to test and install panda
20:25 FROGGS without testing and installing its deps
20:25 mst I was expecting we'd cover CompUnitRepo, Precomp, testing, installers, and packagers in here
20:25 FROGGS maybe...
20:25 mst basically the whole spectrum
20:25 FROGGS maybe I have to delete the statefile...
20:25 * mst needs to summon leont
20:25 tadzik :3
20:25 * flussence puts on Fake Gentoo Maintainer hat
20:26 nine FROGGS: oh yes, panda's statefile may interfere
20:26 mst amusingly, when panda bootstraps itself, it doesn't record its own deps in its statefile
20:26 FROGGS ohh, that looks better!
20:26 mst that took me a moment of confusion when trying to test 'panda --installed list'
20:26 tadzik that's a bug
20:27 nine By now I would really, really like to get rid of it :) So unless someone brings up an argument soonish, I'm gonna merge the query_repos branch
20:27 mst I was chasing lizmat's rakudobug at the time so didn't care that much
20:28 nine One thing that's keeping me from ripping the statefile out: it carries (literally) one extra bit of information: a flag if something was installed at user request or as dependency.
20:28 nine I don't know if panda is actually doing anything with this information, but it seems usefull.
20:28 tadzik which I put there so you can sensemakingly uninstall stuff at some point
20:28 tadzik with GC, so to say
20:29 tadzik I don't recall Perl 5 ever having that, and it always pissed me off
20:29 nine Yep, I figured.
20:29 nine But I don't know where else to store it.
20:30 mst lack of sensible uninstallation in perl5 is a multifacted clusterfuck
20:34 * mst casts 'summon freebsd port nerd'
20:34 Kassandry joined #perl6-toolchain
20:39 FROGGS btw, this is the panda patch needed for star: https://github.com/rakudo/star/blob/release/panda-bindir.patch
20:39 FROGGS which is possibly not complete
20:40 FROGGS and this is the rakudopatch: https://github.com/rakudo/star/blob/release/rakudo-cur-bindir.patch
20:40 FROGGS so, not much actually
20:40 mst flussence: were you doing an overlay for portage?
20:41 alpha123 mst: huh
20:41 flussence yep
20:42 flussence https://gitlab.com/flussence/perl6-overlay
20:42 mst kassandry was pondering what'd be involved in getting a port going
20:42 alpha123 well we don't even have a perl6 port yet
20:42 alpha123 or at least didnt
20:42 flussence right now I've got it down to a plain Configure.pl && make && make install pattern
20:43 FROGGS hmmm, I think I should "release" an RC0 of star...
20:43 flussence it's an overlay and not distro-upstream because moar & friends do a whole bunch of things gentoo's QA people hate
20:43 flussence (which I've been writing down in the issue tracker there)
20:44 mst YAKS FOR THE YAK STACK
20:44 mst SKULLS FOR THE SKULL THRONE
20:44 diakopter oh no, he's uncorked
20:48 TimToady joined #perl6-toolchain
20:57 nine FROGGS: looks sane at a first glance. I'm not sure I like it very much but can also not point to anything specific
20:58 FROGGS mhm
20:59 nine I think it's just that is pretty much a hack necessitated by the lack of design. And my utter reluctance to add API before we have a solid plan.
20:59 nine But clearly this is usefull and will be for distro packaging, too
21:03 FROGGS and maybe also useful for a repo that hists the modules non-locally
21:03 FROGGS but the bin dir has to be in PATH, whatever you do, so it must be locally
21:03 FROGGS not saying it's not a hack
21:05 mst so https://github.com/rakudo/star/blob/release/rakudo-cur-bindir.patch
21:05 nine Just an observation: there's a 1:1 relation between ~/.perl6 and ~/bin and between /usr/share/perl6/site and /usr/bin
21:05 mst is basically just "pick the directory the shim scripts get generated into" ?
21:05 nine But of course, that's just on UNIX
21:05 nine mst: yes
21:06 mst FROGGS: of course, for non-local modules we may end up needing a smarter shim script, but yes, in theory, but oh gods let's leave that one to later
21:06 mst nine: though, btw, local-but-fetched-on-demand is one of the things I'd been thinking about as an alternative CUR format, with the per-repo format file thing I was talking about
21:07 mst (useful example to anchor the idea off if nothing else)
21:10 TimToady is this channel logged yet?
21:10 mst I believe it's scheduled to be, but isn't yet, but I have logs of everything if we discover we missed something
21:10 mst so "not publically, but nothing will be lost"
21:10 leont joined #perl6-toolchain
21:11 FROGGS nine: if we're not going for the bindir hack, we need another way to specify the directory layout of a repo
21:12 mst I think having the shim dir configurable seems perfectly reasonable
21:12 lizmat nine: I'm looking at streamlining setup-repositories
21:12 FROGGS because currently a repo looks like: bin/, dist/, (panda/), resources/, short/, sources/, repo.lock
21:12 FROGGS and no packager wants to put that in /usr
21:13 TimToady on the shared precomp thing, I always thought, if we really can guarantee immutability of a module and it's compiled results, a cron job that periodically goes off and copies identical unshared modules into a shared repo and deletes the unshared one could work out somehow and remove the objections of wasted space on a multi-user system
21:14 lizmat nine: I think I can get about 15 msecs off of each call, which which would mean something like 60 msecs improvement on loading something like Tex::CSV
21:14 TimToady *its
21:14 FROGGS so most of the repo will be in /usr/lib/.../perl6/... and the runners in /usr/bin
21:14 lizmat nine: so from 235 to 170 msecs
21:14 lizmat nine: do you think this is worth it at this point ?
21:15 lizmat (got most of the code already written btw)
21:15 mst TimToady: I suspect it may be doable via post-install scripts actually at each CUR level
21:15 mst oh, sorry, you're talking about space usage
21:16 mst tell them to get a better filesystem already ;)
21:16 mst (seriously, the world is moving towards single copy fses anyway)
21:16 * leont always imagined packagers creating their own CUR subclass, to make p6 do whatever they wanted to
21:17 mst that wouldn't be terrible but I'd like "what they want it to" to be something I approve of, so making sure it leads them down that path would be preferable ;)
21:18 masak joined #perl6-toolchain
21:25 FROGGS leont: yes, but we've got open gh issues about packagers that want to continue packaging star
21:25 FROGGS leont: they'll learn how to create a nice cur eventually
21:26 thzre joined #perl6-toolchain
21:26 mst also, frankly, vendors all doing things subtly and incompatible differently has been the bane of our fucking lives on the p5 side
21:26 mst I would rather minimise the extent to which they have to do that
21:26 mst and instead give them sufficient knobs to twiddle that they can have it "their way" atop a standard set of features
21:27 FROGGS most packagers (at least the few I know) are not hackers that want to ship their stuff
21:28 FROGGS they use want us to fiddle it right and provide the right knobs
21:29 leont Fair enough
21:29 FROGGS but cur surely leaves options, and I like that
21:30 masak hi everybody. I'm just a Perl 6 user, but I'm very interested in #toolchain stuff, to the extent it can impede my productivity with Perl 6 ;)
21:31 FROGGS and even if a cur exists that goes in the wrong direction, they can fix it, because nothing relies on what the cur actually does internally
21:31 FROGGS ... or filesystem wise
21:31 FROGGS masak: :o)
21:31 mst right, time to go meet $*GF
21:31 mst later all
21:31 FROGGS o/
21:32 * FROGGS .oO( his dynamic girlfriend just callframed... )
21:32 hoelzro o/ masak
21:42 nine lizmat: sure :)
21:42 lizmat okidoki  :-)  almost done
21:43 FROGGS ***ALL*** - just made an RC0 of rakudo star, as of today's codebase: http://froggs.de/perl6/rakudo-star-2016.01-RC0.tar.gz
21:43 FROGGS ***ALL*** - please test it and report bugs to the rakudo/star repository, but keep in mind it is an RC0, so dont update your production servers just now
21:43 FROGGS (just because it is more visible here)
21:44 hoelzro \o/
21:44 hoelzro FROGGS++
21:44 hoelzro I like that ***ALL*** convention
21:44 FROGGS :o)
21:44 FROGGS and it is not a netsplit, even :o)
21:45 nine TimToady: I'm not sure anyone has objections because of wated space (precomp files aren't that large) but because of every user on a system having to wait for the first compilation
21:46 FROGGS hmmm, does this problem exist?
21:46 FROGGS not that many users touch my laptop or my servers actually
21:48 nine It's the one concrete issue that yesterday's discussion brought up IIRC, so it seems to be for at least some people.
21:48 hoelzro nine, TimToady: I'm pretty new to thinking about the whole precomp thing, but if a user is installing a module without admin perms, it seems like it should belong to that user only
21:48 hoelzro admins should be able to create "global" installs of precomp'd module
21:49 nine hoelzro: that's what currently happens
21:49 hoelzro much like site lib and local::libs
21:49 hoelzro nine: the current issue is that there are paths embedded in the precomp files, right?
21:49 nine if the site lib is not writable by the user, panda installs into ~/.perl6
21:49 FROGGS and the precomped product will always be a different one, because of timestamps etc
21:49 hoelzro true
21:49 nine hoelzro: that's one of the issues that make packaging harder
21:49 hoelzro right
21:50 hoelzro I'
21:50 hoelzro ugh
21:50 hoelzro I'm not sure how I feel about ~/.perl6 by default, but that's a whole other story
21:51 nine hoelzro: add it to https://github.com/perl6/toolchain-bikeshed :)
21:51 hoelzro is that a panda policy or a CUR policy?
21:51 hoelzro will do=)
21:51 nine it's panda's policy
21:52 hoelzro alright
21:52 hoelzro which scripting languages (other than Perl 6 and Python) store compilation artifacts on disk?
21:52 nine leont: I've fantasized about a rpm database based CUR, but I'm not sure if it makes sense
21:53 TimToady well, if you use C as a scripting language... :P
21:53 hoelzro TimToady: ;)
21:53 leont I've tought the same (well, about deb though)
21:53 hoelzro I should clarify: languages that compile to a disk artifact without explicitly being asked
21:53 nine rpm does have name, version, vendor and more so at least that part would match
21:53 hoelzro import my_module in Python generates my_module.pyc transparently
21:54 hoelzro another system that's quite different that may be useful to think about is .NET's GAC model
21:54 hoelzro I can't speak to how well it works in practice, but it may provide good examples of what not to do =P
22:01 nine FROGGS: everything about those scripts is a hack. That repos have a bin/ directory was a quick hack to get panda going. Now we get $bin-dir.
22:07 FROGGS nine: no, these scripts are fine, really
22:07 FROGGS nine: or do you think that dists are about modules only?
22:08 nine No, don't mind what I'm saying too much. I should be in bed instead of uttering nonsense :)
22:11 FROGGS then let's continue on another day :o)
22:12 FROGGS gnight
22:14 lizmat gnight FROGGS
22:20 hoelzro night FROGGS
22:37 diakopter .oO( .NET has a precomp thingie (also system-wide and per-user) called ngen... )
22:37 diakopter (native gen)
22:38 flussence ISTR niecza's readme advised running that for something or other
22:39 diakopter but it's opt-in because it strictly disables most of the JIT...  at least, 8 years ago.......

| Channels | #perl6-toolchain index | Today | | Search | Google Search | Plain-Text | summary