The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-06-18

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:52 aborazmeh joined #mojo
02:29 noganex joined #mojo
03:34 aborazmeh joined #mojo
04:03 irqq joined #mojo
04:46 disputin joined #mojo
06:56 dod joined #mojo
07:14 eseyman joined #mojo
07:18 fuogo joined #mojo
07:30 Petru_ joined #mojo
09:08 trone joined #mojo
09:22 irqq joined #mojo
10:37 sh14 joined #mojo
10:47 sri i guess it really might be best to just get rid of the slow query in ->repair
10:48 sri but i need ideas for changing ->dequeue efficiently
10:48 sri https://github.com/kraih/minion/issues/52#issuecomment-309250061
10:48 sri the parents[] column is an array of ids of other minion_jobs
10:49 karjala_ joined #mojo
10:49 sri so i need a subquery that checks that all referenced rows have the state finished, or are missing
10:49 rshadow joined #mojo
10:49 sri postgres puzzle
11:09 someguy joined #mojo
11:09 someguy What kind of control do I have over connection pooling with UserAgent?
11:10 someguy I'm scraping some stuff and hitting (libev) select: Invalid argument Abort trap: 6, which the tubes suggest to be "you've hit the ulimit and libev hates that"
11:12 someguy I guess I have to queue everything :S
12:07 * sri does have a solution for considering missing parents to be resolved in minion
12:07 sri things get really really fast
12:07 sri ->repair goes from 3/s to 300/s
12:09 sri i mean, missing parent is really an edge case, you'd have to manually delete a job, so i think handling it this way is fine
12:23 dod joined #mojo
12:25 sri https://github.com/kraih/minion/commit/5d069b1f57d01f74b041b65b51c2a38666e230ab
12:26 sri adding 50k jobs is now the slowest part of the benchmark
12:35 Petru_ joined #mojo
12:55 sri TIL you never add a LIMIT 1 to an EXISTS subquery in postgres, because it disables certain optimizations in the query planner
13:20 irqq joined #mojo
13:30 jberger Nice
13:44 sri but something very weird happened, i can't replicate my job_info results from yesterday anymore
13:44 sri i mean, in EXPLAIN ANALYZE the new version is still much much better
13:44 sri but not in the benchmark Oo
14:01 marty joined #mojo
14:03 sri this is sooo weird
14:03 sri https://github.com/kraih/minion/issues/52#issuecomment-309279559
14:03 sri maybe i'm overlooking something, but the EXPLAIN ANALYZE results don't reflect reality
14:04 sri the first one should be slower in the benchmark too, but it ends up being faster
14:04 sri 200/s vs 300/s
14:20 karjala_ joined #mojo
14:30 * sri shrugs
15:08 aborazmeh joined #mojo
15:11 stryx` joined #mojo
15:13 sri ok, it appears that using the gin index is 30% slower with around 10k jobs, but an order of magnitude faster at 100k jobs
15:28 sri ok, test run with a million jobs in the queue
15:29 sri ->repair is steady at 262/s
15:29 sri so that's a non-issue now
15:29 sri ->dequeue/->finish is a total non-issue at 2111/s
15:30 sri ->job_info at 29/s is kinda ok
15:30 sri just ->stats at 1.7/s is a bit low
15:41 PryMar56 joined #mojo
15:50 * sri closes https://github.com/kraih/minion/issues/52
16:21 mattp_ sri: is the stats query the one in your linked comment?
16:28 nic https://metacpan.org/release/NICZERO/Minion-Backend-Storable-6.051 supports Minion latest (v6.06)
16:31 sri mattp_: no, that is the stats query https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm#L184-L197
16:31 sri nic++
16:33 mattp_ 1.7/s isnt great, but i cause why are you calling it more often than that :)
16:34 mattp_ s/cause/guess/
16:34 sri because an admin ui might want to live update stats
16:38 sri if anyone feels like trying to make ->stats faster, to get a test environment just run this benchmark with a million here https://github.com/kraih/minion/blob/master/examples/minion_bench.pl#L6
16:38 sri afterwards you have the same queue state i used
16:38 sri then just explain analyze away on the psql shell
17:02 genioweb joined #mojo
17:22 dod joined #mojo
18:08 kes joined #mojo
18:10 kes joined #mojo
19:00 dod joined #mojo
19:11 nic Is there a way to increase my chances of my perl5 project on github being flagged 'perl' rather than 'perl6'?
19:12 sri see .gitattributes in our repo
19:12 nic ah, forgot about that.  thanks
20:22 CandyAngel Just in case anyone is curious, I started doing the server side to my Mojo-powered project :D
20:23 CandyAngel I also discovered that mentally tracing through the Mojolicious source gets instantly ruined if someone asks if you want pie before you complete it..
20:26 sri that's a known bug
20:28 Grinnz where's the Mojo::Pie branch?
20:33 dod joined #mojo
20:42 CandyAngel Ohhh that's nifty
20:46 schelcj joined #mojo
21:00 irqq_ joined #mojo
21:49 sri think i'll release rate limiting for minion tomorrow.... so please review https://github.com/kraih/minion/compare/v6.06...master#diff-b30d9d6fe78ad2d38daed2446ba66d92R401
22:17 sri hmm, now that i think about it, perhaps Mojo::Util should have a function that turns values like "1 minute" into seconds
22:17 sri you know, for creating cookies, locks with timeouts and the like
22:18 sri lots of places where that could be used
22:56 Grinnz Mojo::Duration?

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary