The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-06-17

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 sri well, i've committed it for now https://github.com/kraih/minion/commit/78de449787a7926a45021a960a617f9dd448427a
00:06 sri guess the method description needs something to explain the bucket nature
00:06 sri also for how to optimize the stored procedure
00:07 sri it's pretty small already, but i bet it could be smaller
00:08 sri works very nicely for rate limits
00:09 sri oh right, indexes might need optimization too
00:10 sri gonna have to run some tests with lots of locks
00:12 sri it's fun, you can have exclusive locks with ->lock('foo', 300)/->unlock('foo'), a concurrency limit of 5 with ->lock('bar', 300, {limit => 5})/->unlock('bar'), and an actual rate limit like 100 requests per minute with ->lock('foo', 60, {limit => 100}), just don't unlock for the real rate limit
02:19 preaction Minion locks sound fun! I'm about to implement a map-reduce job in Minion for my Perl Conference talk
02:19 karjala_ joined #mojo
02:30 noganex_ joined #mojo
02:35 aborazmeh joined #mojo
03:19 nicomen this explained the leaky bucket algorithm so that even I got it: http://www.geeksforgeeks.org/leaky-bucket-algorithm/ - seemed to contain many others too
03:33 stryx` joined #mojo
03:42 mtj joined #mojo
03:58 disputin joined #mojo
04:04 dboehmer_ joined #mojo
06:05 marty joined #mojo
06:45 Petru_ joined #mojo
06:52 dod joined #mojo
07:08 dod joined #mojo
07:09 dod joined #mojo
07:13 dod joined #mojo
07:22 Craftsmanship joined #mojo
07:49 karjala_ A good variation of leaky bucket algo is the token bucket algo (which i use in my projects)
08:09 trone joined #mojo
08:16 genioweb joined #mojo
08:53 stryx` joined #mojo
09:33 Petru_ joined #mojo
10:11 tchaves joined #mojo
10:18 tchaves joined #mojo
10:24 sh14 joined #mojo
10:28 rshadow joined #mojo
10:43 tchaves joined #mojo
11:26 Petru_ joined #mojo
11:43 sri hmm, i guess my lock function is not quite correct
11:44 sri i think there needs to be an exclusive lock in there https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm#L926-L937
11:45 sri for the select followed by insert
12:13 Craftsmanship I guess slurp/spurt aren't async, is there a more asyncy way to write things (post params and the body of things requested with ->ua) to files?
12:16 sri file i/o can't be async, you'd have to use a separate process/thread
14:07 tchaves joined #mojo
14:08 gryphon joined #mojo
14:37 tchaves joined #mojo
15:02 disputin1 joined #mojo
15:07 bobkare joined #mojo
15:24 dod joined #mojo
15:34 PryMar56 joined #mojo
15:38 stryx` joined #mojo
15:40 firnsy joined #mojo
15:41 anparker joined #mojo
15:51 tchaves joined #mojo
16:34 marty joined #mojo
16:35 sri ok, i think this is a pretty reasonable implementation for minion locks, i'd appreciate reviews https://github.com/kraih/minion/compare/v6.06...master
16:40 jberger how horrible is it to use the <base> tag
16:40 jberger feels pretty horrible
16:40 jberger but I'm back to this again: https://groups.google.com/forum/#!topic/mojolicious/22PKGB4gMjU
16:43 sri guess i'll call the feature just rate limiting
16:44 jberger sri: looks fine to me
16:45 sri in sidekiq that's an enterprise feature btw. https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting
16:49 sri suppose the Minion::lock method could take an extra option to handle the sleep or die after a number of seconds
16:49 sri $minion->lock('some_web_service', 60, {limit => 5, wait => 30})
16:50 sri well, that can still be added later on
16:51 sri for now we have many options for using it from $job->retry to sleep 1
16:52 sri think my first use at work will be $job->finish('Last job is still running') unless $job->minion->lock('unique_job', 3600)
16:52 tchaves joined #mojo
16:53 sri and a crontab entry to call .../script/myapp minion job -e 'some_unique_job'
16:54 sri btw. i'm also not 100% on the terminology of lock/unlock
16:54 sri something that makes it sound more related to tokens and rate limiting might be good
16:55 itaipu joined #mojo
16:56 jberger acquire/free?
16:56 jberger lock and unlock probably work
17:08 sri lock/unlock seem clearer
17:11 sri suppose unique jobs are supported too now
17:11 sri guess that could go into the feature list too ;p
17:12 sri since it's just a rate limit of 1
17:12 sri or are there use cases for unique jobs that wouldn't work that way?
17:14 sri i suppose if you wanted to enqueue unique jobs with a very high frequency it would kinda suck
17:15 sri since all those would run for a moment
17:22 marty joined #mojo
17:48 sri jberger: i don't mind <base>
17:49 sri so
17:49 sri everyone please review https://github.com/kraih/minion/compare/v6.06...master#diff-b30d9d6fe78ad2d38daed2446ba66d92R401
17:49 jberger I think I'm going to make the links relative again, conditionally add a base tag and have the base default to /
17:49 jberger then somehow during export let the base be whatever I need for github pages
17:50 jberger I don't love it, but it doesn't break the user I made the change for and lets me proceed
17:56 disputin joined #mojo
17:59 Petru_ joined #mojo
18:02 trone joined #mojo
18:06 itaipu joined #mojo
18:08 disputin joined #mojo
18:12 tchaves joined #mojo
18:15 marty joined #mojo
18:17 jberger every time I'm going to be giving a presentation, I spend more time working on my presentation layer than my presentation
18:17 jberger :s
18:18 disputin joined #mojo
18:27 marty joined #mojo
18:32 rshadow joined #mojo
18:34 tchaves joined #mojo
19:06 Petru_ joined #mojo
19:16 tchaves joined #mojo
19:29 sri maybe stop doing that
19:50 Craftsmanship joined #mojo
20:17 gabiruh joined #mojo
20:19 gabiruh joined #mojo
20:26 itaipu joined #mojo
20:46 sri can't say i like how much the minion backend api has grown... but all those methods are so valuable :S https://github.com/kraih/minion/blob/master/lib/Minion/Backend.pm#L47-L65
20:56 genioweb joined #mojo
21:11 cheako joined #mojo
22:02 sri hmm, i guess the gin index on job parents is coming back
22:03 sri it makes job_info *A LOT* faster
22:03 sri like going from 80/s to 4890/s
22:04 sri even for a small queue with 50k jobs on a super fast ssd
22:39 sri looking into ->repair
22:39 sri it spends 99% of the time in one SQL statement
22:40 sri https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm#L152-L159
22:40 sri that is our biggest bottleneck in minion
22:47 * sri posts data https://github.com/kraih/minion/issues/52#issuecomment-309244792
22:58 Lee_ joined #mojo
22:58 leejo left #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary