Perl 6 - the future is here, just unevenly distributed

IRC log for #askriba, 2017-11-10

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:31 haarg joined #askriba
01:32 haarg joined #askriba
02:56 ilbot2 joined #askriba
02:56 Topic for #askriba is now Where random Perl5-related questions get definitive answers. Conversation log: https://is.gd/askriba_irclog (searchable). Currently no preset business hours due to low demand: just highlight my name and I'll answer ASAP \o/
04:44 karjala_ joined #askriba
05:04 dboehmer joined #askriba
07:59 karjala_ joined #askriba
09:38 ribasushi karjala_: you were looking for me yesterday, what's up?
09:39 karjala_ hi yes
09:39 karjala_ all's good, but I had a question that didn't get answered
09:39 ribasushi sure what's the question?
09:39 karjala_ My problem is depicted here. In short it's that I can't combine '+columns' with union: https://pastebin.com/AvVsvgGP
09:40 ribasushi you need to give "extra_field" a RDBMS-side-visible alias, so that the scoping works out correctly
09:40 ribasushi in other words:
09:40 karjala_ aha
09:40 karjala_ in practice, how's that done?
09:40 ribasushi extra_field => \'1 AS extra_field'
09:41 karjala_ ok!
09:41 ribasushi this way the *outer* SELECT knows what "extra_field" means
09:41 karjala_ oh
09:41 karjala_ thanks!
09:41 karjala_ :)
09:41 ribasushi ( this is what the error said as well btw: Unknown column 'me.extra_field' in 'field list'
10:20 karjala_ I wonder: shouldn't +columns automatically add 'as extra_field' after '1'?
10:21 ribasushi in your case it definitely is prohibited from doing so
10:22 ribasushi \'...' is an explicit contract of "this will be passed to the RDBMS as-is, with no processing"
10:22 ribasushi in the general case of '+columns' => { 'dbic_side name' => 'sql_side name' } => ideally it would be nice if it did
10:23 ribasushi but because it wasn't done at the start too many things will break by the addition of an explicit 'AS dbic_side_name' that wasn't there before
10:23 ribasushi so that's also a no-go
15:54 karjala_ $ITEMS->create({name => "karjala"})->id === "3", not 3
15:55 karjala_ when JSON-encoding it, i mean
15:55 karjala_ whereas $ITEMS->find(3)->id === 3
15:55 karjala_ shall I file a bug report about this?
15:59 karjala_ ok maybe not true the above... (checking)
16:23 karjala_ yes, it is true (I confirmed)
16:25 karjala_ Will you be using github's issue tracker for your module
16:25 karjala_ ?
16:25 karjala_ I prefer it, it's hard to post bugs on RT
16:30 ribasushi karjala_: which DBIC version?
16:30 karjala_ I'll check but I believe the latest
16:30 karjala_ how do I check the version inside /local ?
16:31 ribasushi usually just running `perl -M<modname>\ 999` is sufficient
16:32 ribasushi provided you have your env setup so that `perl ...` just works
16:32 karjala_ 0.082840
16:32 karjala_ oh just a sec
16:33 karjala_ DBIx::Class version 999 required--this is only version 0.082840.
16:34 ribasushi ok let me see locally, this should have been fixed...
16:34 ribasushi regarding bugtrackers - there is *nothing* easier to use than RT
16:34 ribasushi https://twitter.com/ribasushi/status/628169595283750912
16:34 ribasushi karjala_: ^^
16:36 karjala_ But I have to remember all this
16:36 karjala_ ok
16:36 karjala_ plus I have to publicise my email address
16:36 karjala_ if I'm at work it's not a very good idea to publish my work email address though
16:37 ribasushi gmail accounts are free ;)
16:38 ribasushi anyway - to answer your question: no, none of my github repos tend to have issues enabled
16:40 ribasushi karjala_: regarding your question: I recall correctly, base DBIC does not mess with the integeres: https://paste.debian.net/plain/995039
16:40 karjala_ ok
16:40 ribasushi karjala_: it *might* be a bug in DBIC core, but it is more likely a bad interaction with some plugin
16:40 karjala_ ok!
16:40 ribasushi karjala_: before filing a bug - you need to figure out what exactly is causing it ( check by removing various components )
16:41 karjala_ I'll do a vanilla setup and try adding the plugins one by one
16:41 ribasushi make sure to check the vanilla setup first too
16:41 karjala_ yes
16:41 karjala_ but
16:41 ribasushi warn encode_json({ $your_object->get_columns }) <--- is sufficient
16:41 karjala_ I think you tested the wrong thing
16:42 ribasushi how so?
16:42 karjala_ That debian.net link you pasted above, doesn't mention create, only next
16:42 karjala_ next works fine, I know. It's create that messes the id up
16:42 karjala_ on my computer
16:42 ribasushi sure let's try to create
16:42 karjala_ like this:
16:42 karjala_ $CLASS->create({...})->id
16:43 karjala_ where {...} doesn't contain id
16:43 karjala_ id is autoincrement
16:43 ribasushi ~/devel/dbic$ perl -It/lib -Ilib -MDBICTest -MDevel::Peek -MJSON -e 'my $artist = DBICTest->init_schema->resultset("Artist")->create({ name => "foo" }); Dump [ $artist->id, $artist->name ]; warn encode_json({ $artist->get_columns })' <--- same
16:43 ribasushi ( if you checkout a dbic repo - you can run the exact oneliner as above )
16:44 karjala_ ok thasnks
16:45 ribasushi I have to run, will read later &
16:46 karjala_ k thx
18:51 karjala_ joined #askriba
20:25 aeruder joined #askriba
20:25 frew ribasushi: meet andy
20:25 frew aeruder: meet riba
20:25 aeruder hello hello
20:25 ribasushi o/
20:25 frew andy uses a weird emacs thing
20:26 frew ribasushi toured museums with my wife and I on my honeymoon
20:26 * aeruder is back on neovim for now :)
20:26 frew end of intros! andy is working on some patches for DBIC mostly to fix outstanding issues, iirc
20:26 ribasushi frew: hahahahahaha this is probably the best bit of trivia you get to tell people isn't it?
20:26 aeruder i'm equally dissatisfied with all text editors
20:26 frew ribasushi: when introducing you, yes ;)
20:27 ribasushi aeruder: happy to help with whatever questions you may have
20:27 frew so andy was/is willing to do footwork to do the big upgrade at the same time as fixing this bug
20:27 aeruder yea, i've been running our testsuite against dbix-class master
20:27 frew andy: I've shared both of your patches with riba, the former I think is pretty clear, the newer one is sorta unclear to me and riba had thoughts on how it could be improved
20:28 ribasushi aeruder: let's link stuff anew, so that the conversation doesn't go sideways
20:28 aeruder yep
20:28 frew aeruder: I would personally share anya nd aoll details, including the profiles, openly if it's not too much of a hassle.
20:28 frew https://gist.github.com/frioux/ad2ddb06bd834dd034820f0944d2a864
20:28 ribasushi assume I haven't spoken to the frew ;)
20:28 frew ++
20:28 frew I have to go to lunch, looking forward to reading backlog
20:29 aeruder yea, so re: the inflated column patch, i resubmitted to GH as a pull request against the maintenance branch
20:29 ribasushi aeruder: regarding sharing profiles - this channel is permanently publicly logged, so might not be best idea to leave perm. links here
20:29 aeruder yep
20:29 aeruder noticed this already :)
20:29 aeruder i need to figure out the best place to put the new tests in the new testsuite
20:30 aeruder so i'll resubmit that one eventually
20:30 aeruder so, i ran a profiling run of one of our tests against our current DBIx::Class (which is unfortunately 0.082820 not 40)
20:31 aeruder and then against master. I've already disabled all the sanity checking stuff (although did enable it in one of our CI tests)
20:32 aeruder profiling time-wise, the old dbix = 62s, new dbix = 130s
20:32 ribasushi not great at all
20:33 ribasushi aeruder: so according to frew's gist the compactions are a massive time drain
20:33 ribasushi it does stand to reason that it will be the case on large schemas
20:34 ribasushi it never occurred to me to test this
20:34 aeruder https://github.com/dbsrgits/dbix-class/blob/master/lib/DBIx/Class/MethodAttributes.pm#L62
20:34 ribasushi right
20:34 ribasushi however the (+)-part of the patch in https://gist.github.com/frioux/ad2ddb06bd834dd034820f0944d2a864 actually doesn't anything
20:35 ribasushi there is no scenario when $cache->{$code} will exist yet its weakened slot will be undef
20:35 ribasushi you might as well remove the entire thing
20:35 ribasushi now - historically lack of compaction has been a slow-leak problem in the past
20:35 ribasushi this is why I added these without thinking
20:36 aeruder ah, ok, i'm actually very unclear on what causes those things to go away
20:36 ribasushi the way I'd implement the fix would be to have a high start-mark
20:36 ribasushi let's say 1000
20:36 ribasushi and do the dance IFF keys %$cache > $mark
20:36 ribasushi also IFF the size is no more than 10% higher than $current_value_of_mark - bump it up
20:37 ribasushi something along these lines
20:37 aeruder that sounds reasonable
20:37 ribasushi do it for all the spots where I do a compaction ( the idiom is the same and I left the same comment I think ) - should be good to go
20:38 aeruder so, maybe something like, compact, calculate new watermark as new level + 50% ?
20:38 ribasushi *possibly*
20:38 aeruder i can fiddle with it some
20:38 ribasushi intuitively I think allowing the mark to drop *under* the default is wasteful
20:38 ribasushi so compact and re-set it if nothing was compacted
20:39 ribasushi so you don't end up running it over and over with nothing to do
20:39 ribasushi but only grow it up, never down
20:39 aeruder ok, sounds fair
20:39 aeruder i just don't want to get into a place where it is recompacting often because you're sitting at 880/1000 most of the time
20:39 aeruder or whatever
20:40 aeruder but yea, we hit that about 10k times = 46.6s
20:40 ribasushi yeah sorry about that :/
20:40 aeruder yea, no worries, there's a lot of good stuff in new DBIx::Class that would make my life easier
20:40 aeruder so worthwhile effort
20:41 ribasushi but yes: "don't want to get into a place where it is recompacting often because you're sitting at 880/1000" <--- that's exactly what you want to avoid, thus only grow and grow aggressively
20:41 aeruder the new resolve_relationship_condition works way better
20:41 ribasushi what this basically does is prevents the *hash* itself from accumulating more and more keys that do not point at anything anymore
20:42 ribasushi because while these are cheap, they do add up over time
20:42 ribasushi if the actual members of the hash are all there - you already pay the price of keeping them in memory, there's nothing to do
20:42 ribasushi a HoH with 1k members is essentially free
20:42 ribasushi 10k - not so much
20:42 ribasushi hence the compactor
20:43 aeruder ok, makes sense
20:44 aeruder the next thing is basically the same, so same logic could be applied there
20:44 ribasushi precisely
20:44 ribasushi <ribasushi> do it for all the spots where I do a compaction ( the idiom is the same and I left the same comment I think ) - should be good to go
20:44 aeruder https://github.com/dbsrgits/dbix-class/blob/master/lib/DBIx/Class/ResultSource.pm#L201 (btw)
20:45 aeruder a lot of time in Sub::Quote
20:46 ribasushi that's not really feasible to avoid - it's basically trading off compile-time slowdowns for run-time speedups
20:46 ribasushi if a *specific* callsite shows way more than others - maybe things can be scaled back
20:46 aeruder https://github.com/dbsrgits/dbix-class/blob/master/lib/DBIx/Class/_Util.pm#L278 -- almost 11 seconds on that
20:46 ribasushi but if it is "uniform" - not much to do
20:46 ribasushi yeah - this has may second-level callers - which is the most prominent if any?
20:47 aeruder oh whoops, i see
20:48 aeruder https://github.com/dbsrgits/dbix-class/blob/master/lib/DBIx/Class/Relationship/Accessor.pm#L36 - ~5 seconds
20:50 aeruder really, almost all of them are out of there in some form or another, hits about 1500 quote_subs
20:51 ribasushi right - previously this was a simple closure: https://github.com/dbsrgits/dbix-class/blob/6243d42b1ac9b614382e6126ca76a3a42953a7e9/lib/DBIx/Class/Relationship/Accessor.pm#L29-L53
20:51 ribasushi ( this and other spots like that )
20:52 ribasushi with S::Q what we do is static-inline as much info as possible
20:52 ribasushi to gain not much but some speedup at runtime
20:52 ribasushi specifically: https://github.com/dbsrgits/dbix-class/commit/8d73fcd44
20:53 ribasushi ( also this is an old change - it's been there since 082800 - so you *are* running it now anyway )
20:53 aeruder oh really
20:53 aeruder let me see if i can pull this up on the old profiling
20:56 ribasushi aeruder: https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082820/lib/DBIx/Class/Relationship/Accessor.pm#L27-58
20:57 aeruder yea, i wonder if something else is leaking into there, this takes way longer now vs before, and the only difference i really see is the qsub_args
20:57 ribasushi oh!
20:57 ribasushi there is one new thing of course
20:58 aeruder acc_type eq 'single' maybe doesn't happen?
20:58 ribasushi sec
20:58 aeruder http://ruder-babombpi2.nsupdate.info/files/d1f94703f.png
20:58 aeruder old profile
20:59 aeruder http://ruder-babombpi2.nsupdate.info/files/ed3b94a98.png
20:59 aeruder new profile
21:00 ribasushi sec, grabbing a link...
21:00 aeruder hm, kinda bizarre how it isn't showing a count/time on the old profile there
21:00 ribasushi NYTP is... not great
21:01 ribasushi hm... it is hard for me to reason just by code what is taking long
21:01 ribasushi you'll have to rerun NYT with statement-profiling granularity
21:01 ribasushi and look within *S::Q* itself
21:01 aeruder i should 100% admit that i barely know wtf i'm doing :)
21:01 ribasushi because what changed is the arguments we supply
21:01 ribasushi worry not, let me get you stuff...
21:02 ribasushi NYTPROF=subs=1:stmts=1:sigexit=1:clock=2  <--- rerun the profile with that
21:02 ribasushi and click through all the way to S::Q, see what *it* does
21:02 aeruder aha, ok
21:03 aeruder one sec
21:04 aeruder *130 sec ;)
21:04 ribasushi right so... hm...
21:04 ribasushi let me think
21:07 ribasushi aeruder: this is a crappy problem, I am thinking how one would better address it...
21:07 ribasushi it all comes from this:
21:09 ribasushi https://github.com/dbsrgits/dbix-class/commit/09d8fb4a05e#diff-869dae76e57f4950b9b3d1537c5b09d0R28
21:09 ribasushi when perl compiles the sub in question it now has to call back into the attributes machinery, which in turn calls into user-side code again to apply the non-core attributes correctly
21:10 ribasushi this all takes time, and in your case ( many relationships ) the time is visible
21:11 ribasushi there *is* a clean way to fix this, but it is somewhat complex
21:11 ribasushi let me check how is the test coverage on this front...
21:13 aeruder i'm still running the profile, was running it through some of the bootstrap stuff accidentally
21:13 ribasushi aeruder: nah don't worry about it at this point, the reason is clear
21:14 ribasushi in fact you can trivially test this, let me put together a test-patch
21:16 aeruder i gtg for a bit, should be back in < than an hour
21:16 ribasushi aeruder: I will have to leave myself
21:16 ribasushi will type up stuff, read when you're back
21:16 ribasushi if not - Monday
21:17 aeruder ok, sounds good, if i disconnect, i'll check the public log in the topic, my znc setup is in .. not a good place .. so not using it right now
21:18 ribasushi aeruder: https://paste.debian.net/995093/
21:18 ribasushi with this you should see the time go back to where it was
21:18 aeruder ok, i'll check that out
21:18 aeruder obviously all the times i'm quoting are pretty astronomically bad because they're profiler times
21:19 ribasushi the fix would be to do *exactly* that, and then "apply" the attributes by poking the DBIC-internal state within the hashes
21:19 ribasushi so you basically do not let perl do it, but do it yourself, for only the attributes you know
21:20 aeruder aha
21:20 aeruder i think that makes sense
21:20 ribasushi ( of course it is complicated by you needing to ensure you control the entire pipeline, and in general not drop anything on the floor, but it is doable )
21:20 aeruder so basically split out some of the attribute functionality to post-apply it
21:20 ribasushi and I have sufficient amount of tests to catch things
21:20 ribasushi precisely, but has to be done very carefully
21:21 ribasushi I'd be happy to review something if you guys get around to it
21:21 ribasushi or I will do it on my end in a month or two, PAUSE-legalese permitting
21:21 aeruder yea, i'll try to poke at it some this weekend, this is sorta a sideproject after messing around with that other inflation thing
21:21 ribasushi aeruder: thank you so much for looking into this btw, this is great help
21:24 aeruder yea, no problem, learning a lot.  I'm pretty noobish with perl and even more so with the perl "ecosystem", so its a lot of adding print/data::dumper all over the place until i can piece together how it all fits together
21:24 ribasushi ++
21:25 ribasushi it doesn't help that this is one of the darker corners of this codebase, but you are doing great, so keep at it ;)
21:25 aeruder i.e. here's my janky test method http://ruder-babombpi2.nsupdate.info/files/82777f8e7.png
21:26 ribasushi um... that's hardly "janky" - it works doesn't it? :)
21:26 ribasushi like... when I was developing this set of changes
21:26 ribasushi this was my "downstream CI":
21:26 ribasushi https://github.com/dbsrgits/dbix-class/commit/12e7015aa9372
21:26 ribasushi ( the commit message a bit down )
21:28 aeruder lol
21:28 ribasushi aeruder: ok I have to run, talk to you more in the near future &
21:28 aeruder yep, cya

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary