Perl 6 - the future is here, just unevenly distributed

IRC log for #askriba, 2018-01-01

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:47 mohawk joined #askriba
00:48 mohawk hi ribasushi, do you have any thoughts about what i said in #dbix-class about async DBIC?
00:51 mohawk_pts joined #askriba
00:54 mohawk uhhhh, that guy follows me everywhere
00:57 mohawk dboehmer, i just scrolled back a couple of days on the /topic log - do you mind if i ask what was the DBIC idea? it seems to me like DBIC is experiencing some governance friction, so some new thinking (omg maybe even a fork) might end up beneficial for everyone
00:58 mohawk (my concept is to split apart the parts of DBIC that are inherently synchronous ie data transformation, from the parts that aren't)
02:59 ilbot2 joined #askriba
02:59 Topic for #askriba is now Where random Perl5-related questions get definitive answers. Conversation log: https://is.gd/askriba_irclog (searchable). Currently no preset business hours due to low demand: just highlight my name and I'll answer ASAP \o/
04:53 karjala_ joined #askriba
05:04 dboehmer_ joined #askriba
06:04 vanstyn joined #askriba
07:51 ribasushi mohawk: that's a number of questions posing as one
07:52 ribasushi regarding the split of concerns you speak of - this is how DBIC already works, and the gist you already found/linked earlier speaks of how to do it towards the end
07:52 mohawk please say what the questions you see are? :-)
07:53 ribasushi there is a more fundamental problem of api-coherence that is buried in the conversations in the same gist - this part hasn't really been tackled yet ( i.e. everyone asks "how do I do X with DBIC", without having a ~5 line test they want to see work )
07:54 mohawk ah HA
07:54 ribasushi furthermore, when you work async you usually have a floor where switching to the ioloop and back ends up being way more expensive than just plowing ahead "old style"
07:54 mohawk let me dig out my DBIC-caller uber-field-resolver from GraphQL::Plugin::Convert::DBIC
07:55 mohawk here is the current call:
07:55 ribasushi networking stuff doesn't usually fall into this trap, as median response time is 1s+
07:55 ribasushi the usual SQL query however rtts ( with results! ) in 0.1s or less
07:55 mohawk https://metacpan.org/source/ETJ/GraphQL-Plugin-Convert-DBIC-0.04/lib/GraphQL/Plugin/Convert/DBIC.pm#L158-174
07:56 ribasushi everything else is CPU intensive folding of the result which does not benefit from async work
07:56 mohawk 100ms? who's got that much time when they could be serving another request? ;-)
07:56 mohawk i'll c+p the relevant call as currently is:
07:57 ribasushi where I am going with this is: make sure you have a good idea what you are trying to optimize, it doesn't seem you have the various tradeoffs in a line properly
07:57 mohawk sorry, it's been a few weeks since i touched that code! that's for the accessors
07:57 mohawk here's the relevant actual bit
07:57 ribasushi mohawk: slow down a bit before we go into code
07:58 mohawk $dbic_schema_cb->()->resultset($name)->search(+{ map { ("me.$_" => $args->{$_}) } keys %$args } {prefetch => \@subfieldrels})
07:58 ribasushi there is a final component to what you are working on: if you are specifically interested in changes to the DBIX-Class.tar.gz dist-file: I am not able to make such changes at all, due to the "friction" you cited earlier
07:59 mohawk sorry, i took " without having a ~5 line test they want to see work" as you wanting to see some code people wanted to work
07:59 ribasushi if you are asking about general design advice - then I've already given it: I *personally* do not see bottom-up-async working out in the context of DBIC at all, and one would need to sit down and spend some time on the API side without any internals code to start with, before going further
07:59 mohawk so what i would like is for "search" there, to be replaceable with "search_p", that returns a promise of the same thing that "search" would return
07:59 ribasushi mohawk: right - let's see that portion then
08:00 mohawk that's it
08:01 ribasushi mohawk: how is it supposed to interact with an enclosing txn_do? ( in reality 90+% of all dbic code runs in a "roll-backable" thus isolated scope )
08:01 ribasushi does it refuse to work? does it do something else?
08:01 mohawk so instead of eg $result = $blah->search(whatever); transform_result($result);
08:01 mohawk i'd go:
08:02 mohawk first note the  transform_result($result); is implicitly returned, which i'll do explicitly in the next bit:
08:02 mohawk return $blah->search_p(whatever)->then(sub { transform_result($_[0]) })
08:03 mohawk this isn't hypothetical, this kind of thing is absolutely routine in JS-land
08:03 mohawk great question re the txn
08:04 mohawk return $dbh->open_txn_p_i_dont_know_the_correct_method_name(blah)->then(sub { next_step($_[0]) })
08:04 mohawk ...->then(sub { $somevar_tbc->commit_txn })
08:05 mohawk promises are how to write sequential but async code
08:05 ribasushi I understand how async works ( as in have projects that run on a loop ), this is not what I am saying when I say "API"
08:05 mohawk fair enough!
08:05 mohawk can you help me understand what you do mean?
08:05 ribasushi my point is more broadly: if all you were doing is talking to a read-only ( i.e. slave ) db - then there are no questions at all: everything is obvious
08:06 ribasushi DBIC is inherently a read/write framework, with explicitly isolated write-scopes to go with it
08:06 ribasushi you need to come up with a plan how *competing* ( because you just made them such ) reads/writes work within the scope of a single program
08:07 ribasushi what does the dbi-handle pool look like
08:07 ribasushi how much do you need to explicitly signal via the API ( i.e."block this if this other thing is running" )
08:07 ribasushi how much works implicitly
08:07 ribasushi stuff like that - that API, the... UX-ey one
08:08 ribasushi otoh - if all you want to use DBIC for is a sort of Mojo::mysql - then the answer I gave Steve/Joel way back when already contains everything you need
08:08 mohawk how is that currently managed with eg preforked web servers?
08:09 ribasushi mohawk: each process has its own handle, a $dbh is strictly not allowed to be shared between a fork/thread: there is code that checks "is this a native $dbh? no? - reconnect"
08:11 mohawk so effectively that's one handle per active request to the db?
08:11 ribasushi one physical TCP connection, yes
08:12 mohawk ok
08:12 mohawk what's wrong with having an extensible pool of those so each "new" request just gets a new handle?
08:13 ribasushi I am not sure I am following
08:14 mohawk so currently there is one per process?
08:14 mohawk (/thread)
08:14 mohawk is there a really fundamental problem with >1 per process?
08:15 mohawk my above was intended to be a bit hand-wavey, which doesn't help - the management of those >1 per process doesn't matter compared with the code being able to grok that
08:16 ribasushi I was going to say something to that effect - yes
08:16 ribasushi you can of course have as many connections as you want within a process
08:16 ribasushi you can even have ( rudimentary ) can_read()-like functionality on most DBDs
08:17 ribasushi but again - each of these got to be tied into a neat... "bow", that would make sense within DBIC
08:17 ribasushi I urge you to refocus on this statement from earlier:
08:17 ribasushi <ribasushi> otoh - if all you want to use DBIC for is a sort of Mojo::mysql - then the answer I gave Steve/Joel way back when already contains everything you need
08:18 ribasushi making $rs->all  async is extraordinarily simple
08:18 mohawk what i want is the powerful, ResultSet management of query-building and results-parsing usable asynchronously
08:18 mohawk as i mentioned on #dbic
08:19 ribasushi you have the query building part in ->as_query already
08:19 ribasushi you take any $rs, with whatever attrs applied to it, and it will render the exact SQL it "means"
08:19 ribasushi and the inflator is already pluggable as well ( though the next-from-cursor part is a bit clunky )
08:20 ribasushi let me just point you to the exact line, perhaps you didn't read the entire gist
08:20 mohawk you got me, i'm still only a third of the way through the gist :-)
08:21 ribasushi https://gist.github.com/ribasushi/032fbb8b58e2b0d62fae#file-gistfile1-txt-L476-L478
08:21 mohawk people keep asking me questions about other software i've made/broken. something about public holidays...
08:21 mohawk i saw your link in steve nolte's commit commentary, and i know it's line 476. i'm going to go read the rest right now
08:22 ribasushi mohawk: no need for the deeper stuff - the linked portion is *literally* all you need for your specific use-case
08:23 ribasushi here is an example of another custom cursor: https://metacpan.org/source/ARCANEZ/DBIx-Class-Cursor-Cached-1.001004/lib/DBIx/Class/Cursor/Cached.pm#L14-48
08:25 mohawk yes, my slightly-deep grepping of the DBIC source also pointed me at cursor as another async/sync boundary
08:25 mohawk fundamentally what i would like to do is isolate from each other the sync/async parts, so those "boundaries" are clearly separated
08:26 mohawk then the overhead, the sync-async duhcision point, if you will, is very cheap and can co-exist
08:26 ribasushi um... it's strange that you say "another" - this is the only boundary for reads there is
08:26 mohawk well
08:26 ribasushi ( ok there is a boundary-less reader, but it is mostly discouraged, and you won't encounter it much )
08:26 mohawk you say "for reads"
08:26 mohawk naturally i don't just want to read DBs
08:27 ribasushi making writes async is something I'd seriously discourage
08:27 ribasushi there is no "far side" processing on 99.9% of them
08:27 ribasushi so you can't win anything
08:28 mohawk "far side"? no comprendo
08:29 ribasushi the only time async even makes sense is when there is a noticeable gap between data leaving your machine, and the "ok" ( or not) response comes back
08:29 ribasushi this happens on reads quite often on more complex queries ( but not nearly as often as one would think )
08:29 ribasushi but it practically never happens on writes
08:30 mohawk while a partial solution is better than no solution at all
08:30 mohawk read-only-async-but-blocking-writes doesn't feel like any solution at all to me :-(
08:31 ribasushi well - it's all perl, you can certainly come up with a solution if you believe it to be sufficiently a problem
08:31 mohawk for me, the answer to "can it do async" of "but it's quite quick" is not a valid one (well, it is - but it's a "no" masquerading as a "but it doesn't matter")
08:32 ribasushi mohawk: you have to understand the general context: for me an async ORM is a dubious proposition in the first place ( and I am not alone: the author of SQLAlchemy wrote a similar sentiment in a blog quite a while back )
08:33 ribasushi mohawk: while I can help you with answers to "here is how you hook in spot X" and "spot Y does not exist"
08:33 mohawk fair enough
08:33 mohawk and i'm not asking you to make one :-)
08:33 ribasushi you can't really expect me to get excited about the entire thing - I am not excited about this at all
08:33 mohawk not sure what i said that sounded like i asked you to get excited about anything ;-)
08:34 ribasushi you got that vibe ;)
08:34 mohawk what i did do was ask for information and insight, and you've offered lots of that, which is really helpful
08:34 ribasushi anyway - do you have further questions or we are good?
08:34 mohawk sorry if i accidentally inspired you
08:34 mohawk i'll try to cut that out ;-)
08:35 mohawk let me recap what i've got so far
08:35 mohawk for reads, what i'm getting so far is that cursor is the place to target for the sync/async boundary?
08:35 mohawk for writes, you've advised that it's not a great idea to try, which i hear and accept as your thought
08:36 mohawk if i wanted to charge into bad-idea-land, where would i target for writes' boundary?
08:36 mohawk i'm assuming there are a few, like begin_txn, commit, update?
08:37 ribasushi the txn start/end aren't really "writes" - they are a category on their own
08:37 mohawk sure, but i'm including those in the "write" side of life. i fully accept the distinction you're making
08:37 ribasushi you don't want to trap these in any way, as DBIC has a very complex handling for those ( they do not stack in reality, but DBIC makes it appear so by keeping counts of nested transactions )
08:38 ribasushi so what you want there is to check $schema->storage->transaction_depth > 0
08:38 mohawk ok
08:38 ribasushi and if you are in a txn... act accordingly ( which with async... is difficult to define )
08:38 ribasushi for the rest ( insert / delete/ update )
08:38 mohawk async can still be sequential
08:39 ribasushi your boundary... is... not well defined
08:39 mohawk and that means it can still be blocked by/have to wait for things
08:39 mohawk i hear you
08:39 ribasushi due to many of the insert/update involving a read after the write
08:39 ribasushi ( an implicit one )
08:39 mohawk sure
08:39 mohawk but that's just sequence again, which is fine
08:39 ribasushi no
08:40 mohawk no?
08:40 ribasushi it's sequence *within the core code of DBIC*, where you can't easily inject an asyn chain ( it requires an IO loop, even if fake one, for starters )
08:41 ribasushi specifically here:
08:42 ribasushi https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082840/lib/DBIx/Class/Storage/DBI.pm#L1934-2041
08:42 * mohawk looks...
08:44 ribasushi again - focus on what your users are interested in
08:44 ribasushi async isn't useful for it's name
08:44 ribasushi it's useful because it improves resource utilization *in specific cases*
08:44 mohawk up to 1975, it's all synchronous data-transform, which is fine
08:44 mohawk 1976 is obviously a boundary
08:46 ribasushi so is 1934
08:46 mohawk ahh, i read that as all client-side, fair enough
08:46 mohawk noted
08:47 ribasushi also you need to measure *which* part is the expensive one ( which varies per DBD )
08:47 ribasushi $dbh->prepare ?
08:47 ribasushi $dbh->execute ?
08:47 ribasushi $sth->fetch_* ?
08:47 ribasushi ( if you are really drilling down at this level )
08:48 mohawk if it goes to the DB, it's a boundary
08:48 mohawk and sync and async code ought to be isolated from each other
08:48 mohawk i'll tell you a little story
08:49 ribasushi you can certainly find all the spots within DBIC where this happens ( mainly grep for '\b_dbh\b' )
08:49 mohawk i may have mentioned i'm porting graphql to perl
08:49 ribasushi but you are up a rough creek if you want to "async-wrap" *all* of them
08:49 mohawk ha! i was searching for dbh in github, but it returned too many results
08:49 mohawk when i mention the porting thing to JS people, they look at me, baffled - "why?"
08:50 mohawk (subtext - "but isn't perl dead?")
08:50 mohawk my answer now is, because perl has great libraries, especially its ORM
08:51 mohawk i think that DBIC being async (or the inherently-sync parts being usable as such) would yank it forward to 2018
08:51 mohawk that seems like a good thing
08:52 mohawk because gosh knows it's a powerful piece of software, well made with lots and lots of knowledge baked in
08:53 ribasushi I do not share your view in the slightest, but this shouldn't stop you - the code is there, try stuff
08:53 mohawk i think i shall
08:53 mohawk and you've really helped with a lot of insight, as i mentioned before
08:53 mohawk thanks enormously!
08:54 mohawk i hope you'll be open to answering follow-ups in due course? :-)
08:54 ribasushi always happy to help
08:54 mohawk you're a legend
08:54 ribasushi ( though I do not monitor #dbix-class closely, so a highlight here helps )
08:55 mohawk sure, that would be my starting point
08:56 mohawk parenthetically, my experience of doing transformations on bits of code has been that limitations often lead to improvements - at least, limitations of the MOAR PURITY kind
08:56 mohawk eg, whenever i've gone back to something and stopped it doing mutations and instead just return things, the code usually gets simpler, and often shorter too
08:57 mohawk my gut tells me that this isolation exercise has a very decent chance of bringing that kind of benefit
08:57 mohawk time will tell
08:58 ribasushi I must remind you again though - if you are looking to submit actual code for review: I am not currently available for that, you'll have to look elsewhere ( or as you suggested earlier: fork things yourself )
09:11 mohawk i will probably start down the github-fork route
09:11 mohawk and see how things play out
09:16 mohawk first though, i will make DBIC::Schema::Loader grok SQLT schemas as well as direct-DBI
09:16 mohawk just to get my feet wet in dbic-land
10:31 karjala_ joined #askriba

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary