Perl 6 - the future is here, just unevenly distributed

IRC log for #askriba, 2017-08-25

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:10 ribasushi dboehmer: regarding your question in #sqlt: could you open an issue containing an addition to https://github.com/dbsrgits/sql-translator/blob/master/t/sqlite-rename-field.t demonstrating the issue?
02:10 ribasushi it's not entirely clear from your description what the failcase is exactly
02:10 dboehmer hi ribasushi
02:10 dboehmer I am looking into the test file if I understand that
02:12 ribasushi I have to detach, will check back in several hours &
02:12 dboehmer I don't think the test file is testing what I was talking about. may I try to explain it again?
02:15 dboehmer I'll just try to explain:
02:16 dboehmer I use DBIC deploymenthandler for DB migrations. Adding a column with a default value is straight forward: e.g. https://github.com/dboehmer/coocook/blob/master/share/ddl/SQLite/upgrade/2-3/001-auto.sql
02:17 dboehmer but if the column has no default value the SQL is valid but will always fail because the NOT NULL constraint fails for existing values
02:18 dboehmer I'd like to generate migration SQL code that does not just add the column but uses the fallback method with temporary table. the new column can then be inserted with any value.
02:20 dboehmer this only convenience for me as a user of DeployHandler. as SQL Translator's SQL is perfectly valid it's not failing. I'd just love to have like a config option which DeploymentHandler could use
02:25 dboehmer see https://github.com/dboehmer/coocook/blob/master/share/ddl/SQLite/upgrade/3-4/001.sql#L22 for an example of the fallback. all the SQL is autogenerated (as I understand with SQL Translator) -- I only changed it insert the value from column 'id' into the new column (2nd, 'position') and this is much more to type if SQL Translator had not generated the temporary table code
04:04 dboehmer_ joined #askriba
07:36 ashimema joined #askriba
08:36 Relequestual joined #askriba
10:32 ashimema any chance I could quiz you about fork safety in relation to dbic some time riba?
10:33 ashimema I have a feeling I'm overlooking a bunch of stuff I shouldn't
10:56 ribasushi dboehmer_: mmm my DH-fu is a bit rusty
10:57 ribasushi dboehmer_: I understand *what* you want, but I can not visualize the steps you took to arrive at this kind of error with the current code
10:57 ribasushi could you describe a bit better how a DBIC schema evolves to land you in this situation?
10:58 ribasushi i.e upgrading a fixture from $this to $that using @steps
10:59 ribasushi ashimema: sure thing ( though note that what is on CPAN does contain bugs ( unaccounted corner cases ), and even the unshipped stuff in master isn't yet perfected )
10:59 ribasushi but ask away anyway
11:31 * ribasushi has to away again &
11:40 karjala_ joined #askriba
12:24 karjala_ Hi all. Is Time::Out the recommended way to set a timeout on a function?
12:30 ashimema oops.. missed him.
12:31 ashimema I'm about to head off for a week.. so I'll make a note of this question and ask when I get back rather than asking and then disappearing ;)
12:33 ribasushi ashimema: I am here now
12:33 ashimema I basically wanted to know if there's anything special I should be doing if I'm using a dbic schema that's instantiated with 'state' in a mojo app from inside a Mojo::IOLoop::Subprocess
12:33 * ashimema digs out a code example
12:34 ribasushi ashimema: are you asking in terms of "I anticipate this would be a problem, is my concern justified?" or in terms of "I did this, and things broke... what do I do?"
12:35 ashimema erm.. sorta both..
12:35 ashimema I anticipated, and went ahead regardless to try and prove my anticipation
12:35 ribasushi generally everything should just work out of the box
12:36 ashimema now I'm in a state of.. i'm not sure if it's what I anticipate going wrong or another fringe thing
12:36 ribasushi the known ( fixed / unfixed ) bugs lie in a different plane, shouldn't affect what you are planing to do
12:36 ribasushi i.e. to have an instance with ->connection, use it for a bit, fork off a number of things, and expect the instance to be usable in parent and children
12:36 ashimema I sometimes see a 'Subprocess gone away' error.. but I also seem to see more db connections than I would expect so it feels like somethings not getting destroyed and this is my most likely candidate
12:37 ribasushi elaborate on "more db connections than I would expect"
12:37 ashimema yup.. that's precisely what I'm hoping to do/am doing
12:37 ribasushi DBIC does not maintain a conn pool - each fork/thread has a separate new connection
12:38 ashimema hard to elaborate as I'm still digging..
12:38 ashimema yeah.. so I understand that each fork/thread will have it's own connection..
12:39 ashimema it's more that those forked/threaded connections seem to be hanging around beyond the life of the forked/threaded subroutine
12:39 ashimema but it's all 'feeling based' thus far.. so don't dig to hard.. I need to prove that's the case ;)
12:40 ashimema just wanted ot make sure I wasn't going down a dark alley that you were confident I could avoid digging into ;)
12:40 ribasushi <ashimema> it's more that those forked/threaded connections seem to be hanging around beyond the life of the forked/threaded subroutine <---- that isn't possible provided the process/thread has shut down
12:40 ribasushi there will be nothing to keep the connection alive
12:41 ribasushi so either you are mis-calculating how many concurrent processes you actually run
12:41 ashimema do connections get disconnected then upon fork/thread end
12:41 ribasushi or you have a tangential bug in your main dispatcher ( which stays alive for the entire run )
12:41 ribasushi the entire child process shutting down means that the underlying TCP circuit is terminated
12:42 ribasushi so everything is implicitly closed
12:42 ashimema that makes sense
12:42 ashimema thanks ribasushi.. so it does work as I suspected
12:42 ribasushi I still feel I didn't say anything of value, but if it helped you: great :D
12:42 ashimema and my thinking that might be an avenue to explore was worth asking you about before I explored further :)
12:43 ashimema haha.. it's helped solidify a picture in my head :0
12:43 ashimema so now I just need to work on my other bug.. which is Mojo::Pg instead of dbic.. it's one of those bugs that's impossible to replicate when someone's looking at it :(
12:44 ashimema connections going away and Mojo::Pg not detecting it
12:45 ashimema think that's one for jberger upon my return again.. thouhg I fear he's already tiring of my lack of being able to consistently replicate it :(
12:45 ribasushi sec, let me find something thay may be helpful
12:45 ashimema that's promising :)
12:46 ashimema I actually fixed the bug once already but I couldn't work out how to write a test for it.. but unfortunately my fix lead to an entirely new bug which was equally horrible.
12:49 ribasushi this is a test for a similarly hard to replicate condition with mysql
12:49 ribasushi https://github.com/ribasushi/dbix-class/blob/729656c504/t/71mysql.t#L474-L490
12:49 ribasushi on line 489 I instruct the C-level driver to disconnect after a specific timeout
12:49 ribasushi and then I sleep() within the RDBMS itself on line 481
12:50 ribasushi as a result I am able to reliable test proper behavior from DBI onwards
12:50 * ashimema reads
12:51 ribasushi this is just to give you ideas how to jam things at the right level, obviously your test case will look differently
12:52 ribasushi karjala_: your question is super-broad, hard to answer
12:52 ashimema you say 489 drops the connection at the C level..
12:53 ribasushi https://metacpan.org/pod/DBD::mysql#mysql_read_timeout <--- it is a libmysql option, the effect of which happens "underneath" DBI itself, within DBD::mysql.so
12:53 ashimema I 'think' https://github.com/mrenvoize/mojo-pg/blob/master/t/pubsub.t#L102 is attempting to do the same thing
12:53 ashimema ah, I see
12:54 ashimema which is what stumped me.. because when I came to write that test I found it already sitting there.. and always passing
12:54 ribasushi basically allows you to control when the "server went away" will happen, and then build the rest of your test accordingly
12:55 ashimema so I came to the conclusion that pg_terminate_backend was 'nicer' that I needed.
12:56 ribasushi yeah but in your case you want the disconnect to happen *within* the loop, not before it
12:56 ribasushi that's the crucial difference that makes things hard to replicate
12:57 ashimema ah... light bulb moment I think there
12:58 ashimema so actually I could in theory almost copy this test verbatim but move the pg_terminate_backend line to somewhere 'sensible' after the loops started
12:58 ribasushi that's a way to do it yes
12:58 ashimema thanks.. that's really helped.. I hadn't at all grasped that point
12:58 ashimema man it's subtle!
12:59 ribasushi still a little iffy as you don't control precisely when what will be called by the loop
12:59 ashimema mm..
12:59 ribasushi whereas in the test I showed you I know that things will blow up *precisely* when I do the "sleep for longer than the timeout select"
12:59 * ashimema isn't sure how to make it less iffy yet
13:05 ribasushi ashimema: I got to go in a bit, did we cover what we had to cover?
13:06 ashimema I think so
13:06 ashimema thanks riba.. very helpful again
13:07 ribasushi you bet
13:07 ribasushi have a good... whatever it is you are doing for a week ;)
13:07 ashimema :).. I'll be enjoying some french wine for a week :)
13:08 ashimema have a godo week yourself :
13:26 dboehmer_ ribasushi: I think I'm lacking knowledge of SQL Translator and our mutually missing knowledge could be a problem;-)
13:26 ribasushi dboehmer_: this is why I asked you for a set of actual steps that lead to you needing the change
13:27 dboehmer_ I'm writing it down ...
13:27 ribasushi <3
13:29 dboehmer_ I created a schema and called my App::DH->write_ddl. (I actually changed the schema, increased $Schema::VERSION and called write_ddl again but that doesn't matter) Then I changed to schema by adding 1 column to a result class with no default value. Then I called write_ddl and it produced a migration SQL with simple "ALTER TABLE foo ADD COLUMN bar". to mitigate the problem I changed the conditional line SQL translator and reran write_ddl.
13:29 dboehmer_ so it produced the desired output
13:30 dboehmer_ I forgot that I increased $Schema::VERSION before calling write_ddl. I guess that's clear and irrelevant
13:30 ribasushi ALTER TABLE foo ADD COLUMN bar <--- but why is this a problem?
13:31 ribasushi that's a valid alter statement isn't it?
13:32 dboehmer_ It's technically correct but I have to fill in a value for existing rows. It's much work to come up with the fallback solution on my own. I tried to explain that: I think SQL translator works fine but for that usage with DH I'd love to have a config option or similar to force  the fallback style
13:33 ribasushi ok then I guess I am missing does the "fallback style" provide a sensible default for you - where does it take it from?
13:33 dboehmer_ I'd describe the condition like this: if only columns are added but any of the columns has no default value and is not nullable then use the fallback, too
13:33 ribasushi *how does the...
13:33 dboehmer_ I can then insert it into the generated SQL easily by hand. I just need all the boilerplate for copying the other columns back and forth
13:34 ribasushi offffff I see... icky...
13:34 ribasushi so basically you want to reuse SQLTs product as a "template" to edit by hand after the fact
13:34 dboehmer_ my suggestions for the value are 1, NULL or 'TODO' or similar
13:35 dboehmer_ ribasushi: the automatically generated SQLs are fine for some situations but most often I add kind of business logic for making a sane upgrade path
13:36 dboehmer_ e.g. yesterday when I wrote the question I changed that certain items belong to a "project" in my database. not only do I need to add that column but I also added a script which copies the existing items for every existing project. the autogeneration is VERY useful but only a starting point
13:38 ribasushi dboehmer_: let me check something...
13:38 dboehmer_ you might claim that it's wrong to insert any value for the new column but I think it's not worse. "alter table add column" will only work with 0 rows and the fallback style will have the same result for 0 rows. for >=1 rows "add column" will *always* fail. inserting NULL will also fail (actually the very same issue as ADD COLUMN) but is easy to modify
13:42 ribasushi dboehmer_: so I think what you actually want
13:42 ribasushi is an {extra} field of 'initially_populated_from_column'
13:42 ribasushi which is handled just like 'renamed_from' for columns themselves
13:43 ribasushi https://metacpan.org/source/ILMARI/SQL-Translator-0.11021/lib/SQL/Translator/Diff.pm#L390
13:43 ribasushi then the boilerplate can literally generate what you want without any hand editing
13:43 ribasushi and remains usable outside of your particular case as well ( it is a useful feature in general )
13:44 ribasushi probably just 'initially_populated_from' - takes both a scalar ( a column name ) and a scalarref ( a literal default )
13:45 ribasushi dboehmer_: that's the direction I'd go, though of course you need to run this by the new management
13:45 ribasushi ( I will look into incorporating this into SQLT::Boring when the prerequisites for a fork are in place )
13:46 dboehmer_ your solution sounds good although I don't know enough about DH and SQLT to verify it. will DH need to change to make us of that feature?
13:46 ashimema ::Boring ;)
13:47 ribasushi dboehmer_: no, this would be an SQLT-only hange, and then all you'd need to do is add the annotation into the column_info of the new column ( at add_columns time )
13:47 dboehmer_ I also don't know about the status of the CPAN distros. Does that mean you cannot contribute code to SQLT anymore or does it depend on the "new management"'s decision?
13:49 ribasushi dboehmer_: mainly a decision on my part to never again collaborate on/contribute to projects where certain people are involved: life is too short
13:49 dboehmer_ so I'd probably add "initially_populated_from => \1" to my Result::Foo->add_columns(...), generate the DDL and remove that key?
13:50 ribasushi dboehmer_: and creating forks of the things I consider in need of extra work is blocked on certain policies being drafted so that the situation from last October can never repeat
13:50 ribasushi dboehmer_: yes, except it'd be {extra}{initially_populated_from}...
13:50 ribasushi that'd be the workflow ( leaving it in also wouldn't hurt as DBIC won't look at it )
13:52 dboehmer_ nice solution! I don't have much insight into the situation about DBIC & co. but I'm sorry to hear about that hope you get along with your own plans and projects and one day it may be possible to settle this conflict
13:53 dboehmer_ hmm, one more question: if you start a fork do I need to convince the maintainer of DH to make that use your fork?? just thinking
13:54 ribasushi no, I am going to reuse the namespaces, all you will have to do is ensure during your dependency install time you got the proper versions in place
13:55 ribasushi tooling is already either in place or in final staged of being completed to make all of this possible/painless
13:55 dboehmer_ great! thanks for your effort and your time to answer all my questions!
13:56 ribasushi but there will almost certainly be backlash from certain sides, hence me waiting on the "legalese" to be in place so I don't have to write emails for weeks on end again, but can point the unhappy towards a policy and shut down the rest of the debate
13:57 ribasushi dboehmer_: note that the entire "conflict" was carried in the open ( on my insistence ) and there is ample documentation of all the communication that ensued. So if you are curious - it's all there for you to read, though I do not recommend it: it'd be a waste of your time in the end
14:26 ribasushi dboehmer_: aight I got to go - please don't forget to file an issue describing the proto-api, so it doesn't fall off the radar
14:26 ribasushi dboehmer_: feel free to link to the discussion here and/or C/P it
14:26 dboehmer_ where do I file an issue? I don't want to confuse that because of your forking
14:46 ribasushi dboehmer_: either RT or github
14:46 ribasushi dboehmer_: it won't really confuse anything as I intend to keep reading them
14:48 dboehmer_ I feared to post to the wrong project if there were multiple ones. I am writing an issue for https://github.com/dbsrgits/sql-translator/issues
14:49 ribasushi that's the one currently looked at by the rest yes, and fear not: a written issue can always be copied if it is in the wrong spot
15:04 dboehmer_ https://github.com/dbsrgits/sql-translator/issues/94

| Channels | #askriba index | Today | | Search | Google Search | Plain-Text | summary