The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-06-23

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 zivester joined #mojo
00:52 tchaves joined #mojo
01:03 klapperl_ joined #mojo
01:07 aborazmeh joined #mojo
01:46 tholen_ joined #mojo
02:06 diegok I think I've found a bug on Mojo::DOM, it removes the   entities: https://gist.github.com/diegok/2d01cfcc73b98bfb56b880efce92c14e (tested with 7.33, 7.01 & 6.66)
02:23 noganex_ joined #mojo
02:31 brunoramos joined #mojo
02:36 Grinnz all_text returns text, not html
02:37 Grinnz but it should be there in content
02:39 diegok Grinnz: I've found it replace the nbsp for it non-visible (codepoint?), I've commented on my gist -> https://gist.github.com/diegok/2d01cfcc73b98bfb56b880efce92c14e#gistcomment-2130564
02:40 Grinnz in my own tests, the NBSP is there, just as a NBSP character
02:40 Grinnz which is valid
02:40 diegok ^ right!
02:40 Grinnz https://perlbot.pl/p/u5l35s
02:40 diegok me bad :-/
02:56 genio So, it looks like on Windows, I won't be able to use any pre-installed LibUV as they're all built with MSVC and we have to work with mingw
02:58 genio Which means we need libtoolize, aclocal, autoconf, automake.  yet, none of the libtool or autotools Alien modules on CPAN will install.
02:59 genio gah. wrong channel.  sorry about that
04:03 kaare joined #mojo
05:32 dod joined #mojo
05:38 dod joined #mojo
06:05 dod joined #mojo
06:07 inokenty-w joined #mojo
06:38 karjala_ joined #mojo
06:45 AndrewIsh joined #mojo
07:24 salv0 joined #mojo
07:26 trone joined #mojo
07:42 prg joined #mojo
08:00 salv0 joined #mojo
08:42 nic It's a diegok!!!
08:43 diegok nic o/
08:43 nic I was looking through old code at the weekend, your name came up, and I was wondering how you are these days
08:44 diegok nic: fine, too few slipping hours this week tho :-/
08:44 nic ah.  crazy heat?
08:46 diegok no, I mean, yes, but I'm used to it. I'm working to fix something at $work (almost done!)
08:46 nic good luck
08:46 purl You'll need it.
08:46 nic shut it purl
08:46 * purl shuts it
10:21 rshadow joined #mojo
10:31 karjala_ batman, Could Mojo::mysql::Migrations help me with Migrations that need a bit of code executed for every record of a new table?
10:32 karjala_ I've only seen fixed SQL strings in the migrations
10:32 karjala_ Maybe it could be enriched with calling a subroutine? Maybe, I don't know.
11:17 batman karjala_: why not the other way around? write some code that generates a string that you can feed to `from_string()`
11:18 batman oh. i misunderstood
11:18 batman what's the usecase?
11:18 purl hmmm... the usecase is package Foo; use Bar; ....; Bar applies modifiers for all methods of Foo here
11:26 karjala_ batman, I don't have a specific usecase in mind right now. With my previous employer, we very often wrote migration scripts in perl that processed rows though (eg pre-filled a new field with a computed value based on other fields - and it was not possible to do with just SQL)
11:27 batman why can't you just do that after the migration has completed?
11:27 karjala_ I could
11:28 karjala_ It's not as automatic :-)
11:28 karjala_ that's all
11:28 karjala_ ok
11:28 karjala_ sry
11:33 karjala_ By the way I created a mixin in vue.js that makes a component update its view at most once every X ms. (saves a lot of processing power when the model changes very often)
11:33 karjala_ vue rocks
11:33 karjala_ sorry, offtopic
12:05 diegok nic: thanks... I'm fighting against time here... time always win :-/
12:10 dod joined #mojo
12:45 dod joined #mojo
13:08 maschine joined #mojo
13:11 trone joined #mojo
13:11 Grinnz karjala_: the other problem with running perl code during migrations is it could be difficult to keep it all in the same transaction
13:11 Grinnz that's already a problem for mysql anyway though..
13:13 sri mysql migrations will suck no matter what you do
13:19 Pyritic joined #mojo
13:25 karjala_ Could we write a super-simple ORM based on Mojo::Base && Mojo::mysql/Pg ?
13:25 karjala_ Would it be possible?
13:25 karjala_ no, e? ok
13:27 depesz sri: i tried to run the minion bench
13:27 depesz it failed immediately because of missing Minion module. So I tried to install it, but make test fails with: https://share.riseup.net/#YAjQWLLhtyR990di9dThKw
13:27 depesz any idea what I did wrong?
13:28 depesz it looks that I have Mojolicious 6.15 installed
13:28 sri your Mojolicious is too old
13:28 depesz ok. getting newer from cpan
13:29 sri much newer :)
13:29 depesz 7.33, i assume it will be ok
13:29 sri 7.29 is required
13:29 Grinnz karjala_: Mad::Mapper is an attempt at that
13:29 sri 6.15 is 550 lines of Changes away ;p
13:30 batman i wish i could find time to make Mad::Mapper better.
13:30 batman it doesn't feel right :/
13:31 aborazmeh joined #mojo
13:32 sri depesz: no need to install Minion though, just run the benchmark with "perl -Ilib examples/minion_bench.pl" after checkout
13:32 depesz ok, i started it, it printed "Clean start with 10000 jobs", but i see no db connections ?!
13:33 depesz it does show that jobs are being finished. what am i missing ?
13:33 sri depesz: did you edit the postgresql:// connection string?
13:33 depesz never mind. got it.
13:33 depesz was looking at pg process on bad account.
13:33 sri heh
13:33 sri was wondering why there were no angry error messages
13:35 sri btw. i check out the minion repo and run the benchmark with "perl -Ilib examples/minion_bench.pl"
13:35 sri and then edit the SQL queries in the backend module
13:37 depesz with 10k jobs, on my server, it takes its time. so far it's been running for 7 minutes
13:39 nic Monitoring is going to be interested in the current level of locks
13:39 nic At the moment you can ask for a lock with duration 0, which will tell you whether a lock is obtainable
13:48 arcanez joined #mojo
13:49 sri depesz: takes 9 seconds on my laptop
13:49 sri nic: yea, i've been thinking about that
13:49 dantti_laptop joined #mojo
13:52 gregf_ joined #mojo
13:53 sri nic: negative duration is actually used for testing, so i suppose that's kinda supported
13:53 nic heh, I hadn't thought of neg duration  [haven't got to tests yet]
13:55 depesz still hasn't finished ?!
13:55 depesz something is fishy
13:56 sri what stage is it at?
13:56 sri dequeue/finish?
13:56 depesz interestingly, select count(*) from minion_jobs is 10005
13:57 sri that's correct, 5 are inserted always as dependencies
13:57 depesz https://share.riseup.net/#rIH-Xzbgg5gj2eLze3RfyQ
13:57 depesz that's all it did.
13:58 sri what i see is https://gist.github.com/anonymous/5d5b5e7558b15db11b3405e0160068a5
13:58 depesz this is couple of last lines from pg log: https://share.riseup.net/#wH4yuFdAh9vnXnqAmW4KyQ
13:59 sri think i've actually seen the very slow start thing, and the next run you start will be fast
13:59 depesz finished
13:59 purl finished is any language finished?
13:59 sri never been able to replicate it
13:59 depesz https://share.riseup.net/#DlVW7AG3lNatupkLhyA6ew
13:59 depesz running second time
13:59 sri ye, second iteration is already fast
14:00 depesz https://share.riseup.net/#w0AV5Z86CuuPk3blUphu3Q
14:01 sri yea, that's what i would expect
14:01 depesz ok. got some base info. will check the queries and get back to you if I'll have any ideas.
14:01 sri if you could figure out what the slow start is about that would be really awesome too :D
14:01 depesz i think i already know.
14:01 depesz bad stats
14:01 depesz it would probably help to run "analyze" on all the tables after initial population.
14:02 depesz will check it.
14:02 depesz but now i have to do some $WORK :(
14:02 sri interesting, i suppose we could add that to our ->repair method
14:03 sri me too... even though it's so warm i'm melting again :S
14:04 sri actually it's not even that warm... but i guess the high humidity makes it worse
14:05 sri suppose coolo in the south has it worse
14:08 depesz sri: i'm not familiar with minion code, is there any way to run single sql from within minion_bench in some way ?
14:11 sri all the minion queries are abstracted into methods in this class, you could extract them from there https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm#L232-L245
14:11 depesz no, no. i mean - another query. literally: "analyze;"
14:11 sri ah
14:11 depesz i want to verify my idea about slow first run.
14:12 sri $minion->backend->pg->db->query('analyze ....')
14:12 depesz thanks.
14:12 zivester joined #mojo
14:13 sri your idea works btw. :)
14:14 depesz :)
14:14 sri i could get bad results by manually deleting everything minion from the database and then running the benchmark again
14:14 sri drop table minion_jobs; drop table minion_workers; drop table minion_locks; drop table mojo_migrations; drop type minion_state; drop function minion_lock(text, int, int);
14:14 pink_mist <depesz> but now i have to do some $WORK :( <-- surely that should be ₩ork :P
14:14 depesz you can put it in bench script immediately - in real like analyze will happen from autovac daemon, but in here we start immediately after loading data so we get bad results.
14:15 depesz sri: yeah, i'm doing it on db level (dropdb, createdb, run bench)
14:21 PopeFelix joined #mojo
14:31 depesz sri: so, i analyzed query logs from this run on 10k jobs. and it looks ok to me. the most common queries are https://paste.depesz.com/s/Yk and https://paste.depesz.com/s/EsP and they seem to be correctly, and optimally, indexed.
14:35 depesz it is possible that you might get some speedup from create index zz on minion_locks (name, expires);
14:35 depesz it is being used by https://paste.depesz.com/s/y5
14:36 depesz also, given the usage, I would suggest to make minion_locks table unlogged.
14:37 depesz drawback is that its contect will die on pg crash, but i don't think it really matters in case of pg crash.
14:38 sri yea, that's a great idea
14:38 depesz that's all i can find now. looks that it really is in great shape.
14:40 sri \o/
14:40 sri although, i'm a little sad you have no ideas for making ->stats faster
14:42 nic https://upload.wikimedia.org/wikipedia/commons/thumb/d/db/Faster_pussycat_kill_kill_poster_%281%29.jpg/1200px-Faster_pussycat_kill_kill_poster_%281%29.jpg
14:43 depesz moment, checking stats...
14:44 depesz sri: what is the minimal pg version that you want to run on?
14:48 sri depesz: we check for 9.5+
14:48 depesz well, it can be made faster. there are two obvious things that can be "fixed": 1. do not scan minion_jobs many time, and do it just once (currently it's 3 scans if I read correctly. 2. precalculate the stats using triggers
14:48 depesz oh, sweet.
14:48 sri we depend very much on skip locked :)
14:49 depesz sri: ok. give me a moment.
14:49 disputin joined #mojo
14:49 depesz sri: do you have in minion any kind of job pruning after some time ?
14:50 depesz as in: finished, done jobs - get removed from database.
14:50 sri yes, ->repair deletes jobs after a configurable time
14:50 sri usually two days
14:50 depesz ok. cool. so technically the table should never be prohibitively large.
14:50 sri right
14:51 sri currently we aim for 100k - 3m maybe
14:52 depesz you could consider doing something slighly different (will get back to stats in a moment). when job is finished, instead of updating it, delete it, and insert to "done" table, that inherits main jobs table.
14:52 depesz thanks to this main jobs table, with just active queries will be always kept small.
14:53 depesz but for now, let me get some data to look at stats query
15:05 depesz sri: so, with 10k jobs, the improvement is negligible, but it will be better with more jobs.
15:05 depesz your original query: https://paste.depesz.com/s/cY gives: https://explain.depesz.com/s/apjj
15:05 depesz new query: https://paste.depesz.com/s/Z3 - https://explain.depesz.com/s/ANVK
15:06 depesz it has the drawback of having to list stated in the query. i could have extracted it from minion_state type, but it looks like overkill.
15:08 sri interesting
15:08 * sri runs a test with a million jobs
15:08 depesz there are ways to make such query much faster, but it would involve triggers and overhead on all changes to minion_jobs table, which I don't think is something you'd be ok with
15:10 stryx` joined #mojo
15:12 vicash sri: if it helps, for my 3M job run I have to enqueue 1000 jobs on a recurring timer (every 5 sec for now), to avoid slowing down the database while the jobs are being processed simultaneously by 32 workers. One option for me is to separate out the Minion DB into its own database but I haven't tried it yet. the recurring timer approach is a good enough bandaid
15:15 sri vicash: you're doing weird stuff, we won't optimize for that
15:16 vicash fair enough. my case is unique
15:16 mcsnolte joined #mojo
15:17 zapwai joined #mojo
15:21 zapwai joined #mojo
15:21 sri hmm, the ->stats improvement is not as much as i had hoped, but i guess 2/s is better than 1/s :)
15:21 sh14 joined #mojo
15:22 pink_mist that's a 100% improvement :P
15:24 arcanez pink_mist++ # ever the optimist
15:24 depesz sri: consider another approach - calculate the stats in the minion itself, when working on stuff.
15:25 nic vicash: I'm curious why you didn't put those 1000 jobs in a file and make processing that a single Minion job.  You would have 30k files, but that seems a better route to me
15:25 depesz would still need some work if one would run more than one minion master, but it would be possible, and definitely cheaper.
15:27 sri it adds a complexity i'm not sure i would like
15:28 depesz yeah. i can understand it.
15:28 depesz well, you could add triggers (i can write them for you) to keep most of stats updated
15:28 depesz so getting stats would be basically constant time, regardless of how much jobs there are.
15:28 depesz but it's also another layer of complexity, which I'm not sure you want ;)
15:29 Grinnz it means operations could be slower, but stats queries wont slow things down ever
15:30 Grinnz i dislike triggers, but thats cause i'm forced to use mysql mostly where they're horrible...
15:30 depesz well, i'm pg dba, so i love triggers
15:32 sri well, i've applied the optimizations proposed so far https://github.com/kraih/minion/commit/8417e366b83ad14a7be445d388c22fa5130fe542
15:33 sri that stats query is HUUUUUUUGE
15:33 sri not a fan of that... but performance is performance
15:33 depesz put it in a view
15:34 depesz consider using materialized view that gets refreshed every so often - worst case - you have stale stats.
15:34 sri makes adding new stats more annoying though
15:35 jberger I don't think I like a materialized view for stats
15:35 jberger seems to defeat most actual use-case purposes
15:35 depesz well, if you could handle stats returned to you as single row with multiple columns, and not many rows with 2 columns - the query would be smaller, and slighly faster.
15:35 sri yea, not sure where i stand on stale stats
15:35 depesz very slighly
15:36 sri what the query returns is not so important, as long as the method can transform it into the hash we need with Perl code
15:37 depesz sri: well, then I would say that returning it as single row would be better, don't you think?
15:37 sri since row should be totally fine
15:37 depesz moment.
15:37 sri s/since/single/
15:37 depesz sri: do you have ready db with some jobs in it?
15:39 depesz https://paste.depesz.com/s/WW
15:39 depesz i haven't tested it now, but it should be ok (as in: no typos).
15:40 depesz it's shorter, should be slighly faster, and return data as single row with appropriately named columns.
15:40 depesz but, if you have some test data, would love to see explain analyze out of it.
15:40 depesz basically to see if loops for minion_workers is 1 or more.
15:42 sri no, but i'll try it with a new 1 million minion_bench run
15:42 sri it does pass the tests, so that's nice
15:49 depesz sri: could I make small request?
15:49 sri depesz: sure
15:50 depesz I see that you did put me in credits in Minion.pm. Any chance you could change it to "Hubert depesz Lubaczewski" or even just "depesz" ?
15:50 depesz also, thanks for the credit.
15:50 sri hmm, we normally just list real names
15:51 depesz it's up to you. it's just that, at least pg-wise, more people know me as depesz than under my real name.
15:51 sri suppose we could start doing something like Hubert "depesz" Lubaczewski for folks that really want their nick too
15:51 depesz it was deliberate on my part to always use nickname and virtually never real name
15:51 depesz thanks.
15:52 sri well, your real name is on your github profile :)
15:52 sri not sure how to handle people that only want to show their nick and otherwise stay anonymous
15:53 depesz it is listed, sure, i'm not trying to hide it.
15:53 sri that has come up before with CandyAngel, and i ended up mentioning just the nich in Changes and no entry in credits... it's still an open question i'm not sure about
15:53 depesz it's just that I prefer to be known as "depesz" than as "hubert lubaczewski"
15:54 sri we do have "chromatic" in CREDITS, but he even has books published under that name
15:54 arcanez sri: it's a tad harder to find his real name anymore
15:54 arcanez almost like ingy
15:54 arcanez although, that is ingy's real name now isn't it?
15:54 sri ingy legally changed his name
15:55 arcanez I wish I went with jdh or justin or something resemebling my real name instead of 'arcanez'
15:55 sri i was a little sad that ingy didn't join suse from hpe
15:56 nic my real name is martin_1986
15:59 arcanez what is the i for in sri?
15:59 sri just the second letter of my last name
16:00 sri sr doesn't make for a good nick
16:01 Grinnz i'm still gold sr :(
16:04 jberger did we ever get the story of "kraih"?
16:05 PryMar56 joined #mojo
16:05 karjala_ joined #mojo
16:05 sri i needed a backup nick, so i looked for stuff in a dictionary for a german dialect very few people speak ;)
16:06 sri mostly just older people where i'm from can still speak it
16:06 sri means crow
16:06 tianon according to Google, it's "doing" in Hindi, which feels pretty appropriate O:)
16:07 tianon (from https://translate.google.com/#auto/en/kraih)
16:07 sri haha, i like that
16:09 arcanez Joel owns Buttons
16:15 vicash arcanez: Ingy's old name is Brian Ingerson
16:16 sri depesz: yea, i like the small version *a lot* better :) https://github.com/kraih/minion/commit/d3a752b3c0afff80480696ad94fab286f2f4cf6c#diff-6c2d02af894f2ce916d89f7b04f4b54aL185
16:17 sri wonder if there's a non-ugly way to get "$stats->{inactive_workers} -= $stats->{active_workers}" into the query
16:18 Grinnz sometimes it's better to keep stuff out of the query, it has enough on its mind :P
16:18 sri only one i know is to create active_workers in FROM with a second SELECT
16:20 sri anyway
16:20 sri depesz++ # thanks for the review!
16:21 arcanez I haven't seen the RHS syntax before
16:21 arcanez is there any perf difference or just readability
16:22 Grinnz the what?
16:23 arcanez right-hand-side
16:24 Grinnz of what?
16:24 arcanez 09:16 < sri> depesz: yea, i like the small version *a lot* better :) https://github.com/kraih/minion/commit/d3a752b3c0afff80480696ad94fab286f2f4cf6c#diff-6c2d02af894f2ce916d89f7b04f4b54aL185
16:24 Grinnz there's a lot of right hand sides there, i dont know what you mean :P
16:25 arcanez I guess really the filter synta
16:25 arcanez x
16:25 Grinnz ah
16:25 arcanez select count(*) filter (where state = 'inactive') as inactive_jobs
16:25 arcanez had never seen that
16:26 Grinnz in sqlite that would just be spelled count(case state when 'inactive' then 1 else null end)
16:29 Grinnz too bad there isn't a reverse nullif() to make that more concise
16:31 Grinnz nullifnot()
16:31 Grinnz or nullunless() for perliness :)
16:33 depesz sri: it depends on what you define as ugly :)
16:35 depesz sri: but in any way - doing it in perl would be probably simpler/saner.
16:47 sri ¯\_(ツ)_/¯
16:52 salv0 joined #mojo
16:53 zapwai joined #mojo
17:13 sri thinking some more about stats, don't think i want them ever to be stale
17:15 sri if you just want a general overview of what's in the queue, that's fine
17:15 sri but that's not among the most common use cases
17:16 sri when i use stats i want to know exactly what's going on right now, so i can react to it
17:23 dod joined #mojo
17:35 rshadow joined #mojo
17:46 cheako joined #mojo
19:02 gryphon joined #mojo
19:34 sri i still have no idea how to make batch enqueue fast
19:50 rshadow joined #mojo
20:32 dod joined #mojo
20:59 cheako joined #mojo
21:53 cfedde joined #mojo
22:23 CandyAngel I do wonder how some people generate webpages sometimes
22:24 Grinnz either very carefully, or with reckless abandon
22:24 CandyAngel <meta name="description" content="escape " characters, who would do that?">
22:26 salv0 joined #mojo
22:29 CandyAngel (Reasonably,) Mojolicious borks on that, stopping at the " after 'escape'
22:30 CandyAngel Maybe there is a "parse insanity" setting :P
22:30 arcanez CandyAngel: see mst for that.
22:56 disputin joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary