The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-12-11

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:28 kiwiroy joined #mojo
00:37 itaipu joined #mojo
00:59 kiwiroy joined #mojo
01:10 kiwiroy joined #mojo
02:26 mohawk this promise-using bulk mirror script was able to make about 100 requests per sec, using chaining promises to only process 10 at a time
02:27 mohawk i'll write a blog post about it now
02:27 mohawk jberger, want to put such a beast on your advent doodah?
02:32 jberger I'd certainly consider it
02:32 jberger can I see the code?
02:32 mohawk sure!
02:33 mohawk jberger, https://github.com/mohawk2/perl-sudoku/blob/master/bulkget
02:34 mohawk works right if there are no requests to be made, if there are less than the max-concurrent to start, and obviously if more than that
02:35 mohawk i'm thinking about making it a Mojolicious::Command as well
02:35 mohawk any opinions on that?
02:35 jberger but it then waits until all max-concurrent are done before starting another batch?
02:35 mohawk no
02:35 mohawk see how makepromise calls itself? it returns a promise
02:36 mohawk i'm calling that chaining, because that's what i gather is the usual term
02:36 mohawk (calls itself in the ->then, i mean)
02:36 mohawk and there's a shared pool of work still to be done, @suffixes
02:37 jberger ah yes there it is
02:37 mohawk massive kudos to sri for how the promise stuff just quietly starts up the ioloop and finishes it after it's done, with ->wait
02:38 jberger and because it is created from within another promise it is therefore attached to the current on?
02:38 jberger *one?
02:38 mohawk no, that's the great bit
02:38 mohawk it's returned by it
02:38 jberger but ->all knows how to wait on it
02:38 mohawk that's the idiomatic way to achieve synchronous-style programming - return another promise in your "callback"
02:38 tchaves joined #mojo
02:39 mohawk this will be another section of the post: "all at same time (->all) or chained?"
02:40 jberger anyway, I'd love a promises post and this seems to be an interesting case
02:40 mohawk ok, i'm going to write it on gist i think
02:40 jberger certainly different than how I've done a sliding window of urls to be fetched
02:40 jberger (using delays)
02:40 mohawk are you happy with me also putting it on blogs.perl.org or do you want exclusivity? ;-)
02:40 mohawk i think it would be great if you'd write this using delays for comparison?
02:40 jberger I don't mind it being cross posted, but at least let me be first/simultaneous
02:41 mohawk i'll put it on b.p.o a day later or so
02:41 mohawk so i'll put it in a gist for starters
02:41 jberger tbh I think this is a better case for delays, I can show you my similar code
02:41 mohawk that would be awesome
02:41 jberger https://gist.github.com/jberger/5153008
02:42 jberger everything up until package main is the thing that drives the queue
02:57 mohawk i was thinking, "gosh dang that's fast"
02:57 jberger oh I've had that for a long time
02:57 mohawk i feel less bad given you'd already written it ;-)
02:57 jberger I've almost released it to cpan a few times
02:57 mohawk since '13 or so? or '15? i saw you comment on an SO answer
02:57 jberger then I don't want to be responsible for people who'd turn it directly into an (evil) spider
02:57 mohawk ha
02:57 jberger looks like March of 13 from the SO post
02:57 mohawk this thing took me about 15 mins to write, starting from the promises concurrent bit in the cookbook
02:57 jberger it does have the modular ability to add more onto the queue
02:57 mohawk yours?
02:57 jberger yours would probably add a few lines to be able to do that, but I suspect it would still be more succinct
02:57 mohawk i don't think making a M::Command out of it is opening pandora's box :-)
02:57 mohawk i suspect so
02:57 jberger spiders need rate limiting and robots.txt handling at a minimum
02:57 mohawk quite
02:57 jberger without that I wasn't comfortable putting it on cpan
02:57 jberger but to each their own
02:57 jberger obviously it wasn't too much to not put the code out there in the public, just not to "bless" its presence on cpan
02:57 mohawk ah, i see yours is actually spidering, so that's a functional difference
02:57 jberger the spider all comes after main
02:57 jberger the fact that the queue is accessible to the processing is what makes that possible
02:57 mohawk seen
02:58 ilbot2 joined #mojo
02:58 Topic for #mojo is now 🍩 nom nom | http://mojolicious.org | http://irclog.mojolicious.org | http://code-of-conduct.mojolicious.org
03:03 mohawk done
03:40 mohawk jberger, is $delay->end still the right way?
03:40 mohawk it's saying "Mojo::Reactor::Poll: I/O watcher failed: Can't locate object method "end" via package "Mojo::IOLoop::Delay""
03:44 mohawk ok, so the pod now says $end = $delay->begin; and do $end->() in the callback
03:44 mohawk so i'm doing that
03:46 jberger blog post a little early tonight https://twitter.com/joelaberger/status/940064998541242373
03:46 jberger heh, yeah, ->end has been gone for a long time
03:48 jberger actually my gist already fixed that
03:48 mohawk ha
03:48 jberger are you working from the SO post
03:48 mohawk well, got it working now
03:48 jberger ?
03:48 jberger cool
03:48 mohawk i was, but only just
03:49 mohawk jberger, for your perusal: https://github.com/mohawk2/perl-sudoku/blob/master/bulkget-delay
03:51 jberger it still handles queue growth
03:51 jberger if you were going to remove that
03:51 mohawk there is no way for the queue to grow
03:51 mohawk i left that idiom in because it didn't bulk it up at all
03:52 mohawk yours is ~10 lines longer in the 10-20 lines that differ, by my squinting
03:52 jberger yeah but the code accounts for it still exists
03:52 mohawk i don't see it? it's using the $idle to limit concurrency, nothing else?
03:52 jberger the while loop in the callback
03:52 jberger that fills the workers back up
03:52 mohawk yes, it does
03:53 mohawk how else would one limit how many are going?
03:53 jberger your only does that once, not on each iteratoin
03:53 jberger iteration
03:53 jberger nm, not a big deal, just isn't totally symmetric
03:53 mohawk right, because that's the only way i can think of with promises
03:54 mohawk if you can think of a way that's closer but still idiomatic, i'm all ears :-)
03:54 jberger ah, see, delays win again \o/
03:54 jberger (kidding, mostly)
03:54 mohawk like i said, to get that facility with promises, just make it promise-wait for queue-filling ;-)
03:54 mohawk ha ha
03:54 jberger its fine, leave it as it is
03:55 mohawk i was intending to ;-)
03:55 mohawk i'll gist this up into a blog post now
03:55 jberger the one change I would recommend would be to move the wait out of the method
03:55 jberger and nix the unless, since we know the state
03:55 mohawk i'll try that now
03:56 jberger (useless use of "unless loop is running" is a bit of a pet peeve of mine :-P)
03:56 mohawk changed
03:57 jberger also, why not use Mojo::Base -strict; rather than the three pragma imports?
03:57 mohawk i'll change both
03:58 mohawk done
03:58 jberger actually once the getsuffixes and handle_result were added to yours they really don't look too different at all
03:58 mohawk indeed
03:59 mohawk handle_result was specifically to de-boilerplate the differences :-)
03:59 jberger certainly
04:00 mohawk title: Mojolicious Advent Calendar entry: You Promised To Call!
04:00 jberger hahaha excellent
04:02 jberger the only things I'd recommend you include in the text are the newness of mojo's promises (this is one of the first times it is being proclaimed loudly)
04:02 jberger "mojo's new built-in promises"
04:02 jberger or some such
04:02 mohawk "A new feature of [Mojolicious](https://metacpan.org/release/Mojolicious), as of 7.49, is the implementation of Promises/A+."
04:02 mohawk waaaay ahead of you ;-)
04:02 jberger and also mention being promises/A+ compliant
04:02 mohawk that's all i've written so far :-)
04:02 jberger perfect
04:03 jberger great and/or terrible minds
04:03 mohawk fools seldom differ, they say
04:03 * jberger bows
04:04 jberger I'm quite happy with that Content Generator post
04:04 jberger I really wonder what my total word count is so far
04:04 * jberger is afraid to look
04:04 mohawk ahhh, go on
04:04 jberger actually I'm not sure how to get a meaningful count out
04:05 jberger so I'll leave well enough alone and go to bed
04:05 mohawk ha ha!
04:05 jberger (always sad when the Europeans are still cranking away at stuff when I go to bed)
04:05 mohawk night
04:05 mohawk ha ha
04:05 mohawk not all the europeans
04:06 mohawk and ze britisch are always entertained at being referred to as european
04:10 jberger given my position on this blue marble, you are much closer to Europe than I am :-P
04:10 jberger blah blah pond blah blah
04:15 mohawk sure
04:15 mohawk ze britisch are guarding it
04:16 mohawk but they don't really "identify" as european
04:16 mohawk even the pro-EU ones
04:50 kaare joined #mojo
04:57 kaare joined #mojo
04:57 mohawk jberger, 1040 words (including code, according to `wc -w`): https://gist.github.com/mohawk2/3bf6ea5047f1133ca9a9d47e271d6db5
06:13 inokenty-w joined #mojo
06:28 tyldis Don't think any European identify as such
06:35 mohawk outside of the brussels bubble, at least
06:36 mohawk and even they are all secretly at least somewhat pursuing their national interest
06:36 mohawk anyhooo
06:38 tyldis hehe, back on topic
06:44 mohawk jberger, for comparison, including code, your day-9 post was 1740 words
06:44 mohawk if you'd have used promises it would have been about 1000 words as well #trolling
06:55 itaipu joined #mojo
06:59 Vandal joined #mojo
06:59 dod joined #mojo
07:12 mohawk jberger, i've now commandified bulkget
07:13 mohawk one use for it might be for performance-testing one's own app, and/or extracting information from it
07:16 mohawk jberger, also it might furnish material for another blog post on making a Command? in any case: https://github.com/mohawk2/Mojolicious-Command-bulkget
07:21 Lee joined #mojo
08:06 omega joined #mojo
08:07 omega joined #mojo
08:08 omega joined #mojo
08:11 jkp joined #mojo
08:23 trone joined #mojo
08:41 batman sri: a bit late, but i was +1 because i agree with the description in the task
09:11 ashimema joined #mojo
09:11 kd joined #mojo
09:11 kd good evening
09:11 kd I finally wrote a mojolicious::lite app :)
09:12 CandyAngel kd: Yay :)
09:23 geospeck joined #mojo
09:44 karjala_ joined #mojo
10:10 itaipu joined #mojo
10:19 Kharec joined #mojo
10:36 tchaves joined #mojo
10:40 kes joined #mojo
10:51 sri hmm, looks like i just found another minion bug at $work :S
10:52 sri suppose $minion->guard should actually watch the time and not unlock after the lock expired
11:09 good_news_everyon joined #mojo
11:09 good_news_everyon [minion] kraih pushed 1 new commit to master: https://git.io/vb0sq
11:09 good_news_everyon minion/master 737e6ca Sebastian Riedel: fix guard method in Minion not to release already expired locks
11:09 good_news_everyon left #mojo
11:09 sri quick fix though
11:37 good_news_everyon joined #mojo
11:37 good_news_everyon [minion] kraih tagged v8.07 at e8b5609: https://git.io/vb0nJ
11:37 good_news_everyon left #mojo
11:40 gregf_ joined #mojo
12:53 dod joined #mojo
13:16 cascardo_ joined #mojo
13:45 sri mohawk/jberger: you really think an intro post for promises should even mention delays?
13:45 sri personally, i'd introduce them completely separate
13:46 sri it's a complicated topic, why make it more complicated for no good reason
13:48 dotan_convos joined #mojo
13:50 sri mohawk: btw. you can also do my $fh = path($suffixesfile)->open('<');
13:51 sri mohawk: be aware that only very few mojo users know what delays do
13:51 sri they were never a mainstream success
13:53 sri also i think "return unless..." is bad style
13:54 sri same for the implicit return of a promise
14:00 CandyAngel Bad style in general or just in regards to promises?
14:01 sri always
14:01 sri damian agrees on that too now btw.
14:01 sri (so PBP is outdated there)
14:02 sri context should never be ambiguous
14:05 CandyAngel Do you mean returning without an argument..?
14:06 CandyAngel return undef unless vs return unless
14:06 sri return undef vs return ()
14:07 CandyAngel Okay. Was super confused then because Mojo has a bunch of 'return unless' and 'return undef unless' in it
14:07 sri bare return is only used when a function doesn't have a specced return value
14:07 sri if a return value is to be expected there is always return undef
14:07 * CandyAngel nods
14:08 CandyAngel Makes sense, thankies
14:09 jberger sri you are probably right about introducing them separately
14:10 jberger I wasnt really considering that as I was talking with mohawk last night
14:11 sri if you want to introduce both two days could be cool, solving the same problem with both
14:12 sri since mohawk chose a problem that works fir both
14:14 jberger and yeah, now finally reading over his first draft there, I'd certainly say it focuses way too much on delays
14:15 jberger whether I love it or not, promises are the future and delays were not adopted to the level I would have wanted
14:15 sri you definitely want to introduce the basic functionality of promises first imo
14:15 sri then/catch/finally
14:15 jberger trying to teach promises as if the reader is intimately familiar with delays is the wrong stance
14:15 jberger agreed
14:16 sri if you can put that into a calendar post i'll prolly steal it for the core docs :D
14:16 CandyAngel Is there anything else you want to replace in Mojo? Seems every time I get to using something, something happens to it (deprecated, replaced etc.)
14:16 jberger I myself would like several different promise examples, not saying that mohawk needs to do that, but if not him then a second post with other promise examples doing other things
14:16 jberger CandyAngel: I don't want to replace delays, I love them
14:16 jberger but they have always proved hard to teach
14:16 sri CandyAngel: no
14:17 CandyAngel I am the Harbinger of Dooooom
14:17 sri promises have been a topic here for years, we bet on delays (flow control helpers), but sadly it turned out that promises won, so we had to catch up
14:17 bwf joined #mojo
14:18 jberger I can also say (I think this is safe) that if Delays ever get kicked out of the Mojo core, it would be spun out onto cpan
14:18 jberger mostly because I have too much code that uses them :-P
14:23 sri one remaining risk for breakage is http/2, but i don't think we'll get to work on that anytime soon
14:28 dod joined #mojo
14:29 gryphon joined #mojo
14:34 gizmomathboy joined #mojo
14:34 marty joined #mojo
14:39 Pyritic joined #mojo
14:56 Pyritic joined #mojo
15:08 will joined #mojo
15:17 geospeck joined #mojo
15:23 perlpilot joined #mojo
15:33 rcz joined #mojo
15:43 cosimo joined #mojo
15:52 ChmEarl joined #mojo
15:55 sh14 joined #mojo
16:08 marty joined #mojo
16:09 gryphon joined #mojo
16:09 geospeck joined #mojo
16:38 mohawk sri, i will make a "this is what promises are (then/catch/finally)" post first
16:39 mohawk with the bulkget doodah as a followup?
16:39 mohawk jberger, thoughts?
16:39 jberger sounds great
16:39 jberger sorry to have moved the goalposts on you there
16:40 jacoby joined #mojo
16:40 mohawk that's not what's happened :-)
16:41 mohawk i'll rejig what we already have (including some of the intro) into that, then use the other half for the compare/contrast
16:41 mohawk jberger, do you also fancy the idea of the Command-making post?
16:41 gryphon joined #mojo
16:42 jberger I already have a post about commands
16:42 jberger a section in the article about forming it into a command would ok though
16:42 jberger but I don't think it would need another full article
16:42 jberger https://mojolicious.io/blog/2017/12/06/day-6-adding-your-own-commands/
16:55 jamesaxl joined #mojo
16:56 mohawk jberger, cool
16:57 mohawk i have actually read all of them so far, but i didn't re-check before asking :-)
17:10 Grinnz sri: i dont know if it would be a problem, but basing guard on steady_time would make it potentially differ from what the database considers "expired" if the clock changed
17:13 sri Grinnz: yes, but it would be so close that it doesn't really matter
17:14 sri worst case, you end up with with a lock running a few ms lomger than it has to
17:15 sri as opposed to a job going longer than expected removing a lock from another job
17:15 sri causing all sorts of bad side effects
17:16 Grinnz couldn't it go the other way? remove the lock after something else deleted it because it expired?
17:16 sri ?
17:16 disputin joined #mojo
17:17 Grinnz say the clock jumps ahead enough that the database considers the lock expired, so another attempt at locking cleans it up, then the guard is cleared and hasn't reached its timeout yet so unlocks it again
17:18 Grinnz even more complicated when the database server can have a different time set :s
17:18 sri how could we possibly deal with database time jumps?
17:20 sri unique identifiers for locks would be hard to add now
17:20 sri but don't let me stop you from making your point
17:20 sri i'm curious where your argument was leading
17:21 Grinnz not trying to make a point, just trying to think of how it could be more robust :/
17:23 Grinnz without unique identifiers there's no way to base it on the database time... the only other option would be to somehow keep expired locks from being cleaned up until the guard was done, and i dont know if theres any way to do that other than keep them around forever
17:25 Grinnz there is an id on the locks table... could the guard somehow get and store that?
17:27 sri i mean, sure, there is always a way
17:28 sri things get pretty ugly then though
17:29 karjala_ joined #mojo
17:30 sri if you want to redesign locks then that would have to be a consistent new api and replace the old one in a 9.0 release
17:42 Seth joined #mojo
17:43 Seth1 joined #mojo
17:53 disputin joined #mojo
17:58 karjala_ joined #mojo
18:01 tchaves joined #mojo
18:22 Pyritic joined #mojo
18:25 disputin joined #mojo
18:30 berov joined #mojo
18:36 tchaves joined #mojo
18:46 trone joined #mojo
18:49 Pyritic joined #mojo
18:56 Kharec_ joined #mojo
19:08 Pyritic joined #mojo
19:18 eseyman joined #mojo
19:20 disputin joined #mojo
19:28 disputin joined #mojo
19:47 purl joined #mojo
19:56 dod joined #mojo
20:23 Kharec joined #mojo
20:25 disputin joined #mojo
20:26 maschine joined #mojo
20:56 disputin joined #mojo
20:57 mattp joined #mojo
21:46 orev joined #mojo
22:35 sri TIL that it's best practice not to activate ntp on your database server
22:37 Grinnz we didn't 5 years ago, and thats why our application and database were 5 minutes out of sync with each other which caused more issues :|
22:38 Grinnz (i don't think it was intentionally disabled, though)
22:39 pink_mist so you mean you did?
22:39 Grinnz yes, they run ntpd now
22:40 sri batman: i remember you being opposed to adding a time_in_words function to mojo because everybody deals with it client side using moment.js and friends anyway
22:40 sri batman: what about time difference between server and client?
22:41 Grinnz that shouldn't matter, you'd be dealing in a number of seconds either way
22:41 sri i recently had jobs in the minion ui that were added "in 10 minutes"
22:41 itaipu joined #mojo
22:41 Grinnz ah i see, to make the whole thing relative to server time
22:42 sri yes
22:42 sri i have epoch time from the server
22:42 sri and moment.js uses the current client side epoch time to calculate the difference
22:43 jberger I never did understand why ntp was discouraged
22:43 jberger I recently had a similar problem
22:43 jberger and yeah, I had to enable it
22:43 Grinnz for applications that dont deal with time adjustments well, i imagine
22:44 mohawk aka "kicking the can down the road until i can get out of this dump"?
22:46 Grinnz was just wondering how redis deals with this in expires... https://redis.io/commands/expire#expires-and-persistence
22:46 Grinnz "not well, because it needs to work over periods of time when redis isn't running"
22:47 sri time jumps seem to be a doomsday scenario for all databases
22:47 jberger so do people really just let their databases walk out of time sync?
22:48 sri i imagine many just don't have data that depends on time very much
22:48 sri expiring data is kind of niche
22:49 jberger maybe, but displaying when a job was entered isn't very niche :-P
22:49 Grinnz at least with redis, expiration is usually not required to be precise anyway
22:49 sri sidekiq really stores a crazy amount of data for locks https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting
22:53 Grinnz having to deal with durations persistently as opposed to in a consistently running process (that can use a monotonic clock) is what makes everything difficult
23:26 cfedde___ joined #mojo
23:30 cfedde___ joined #mojo
23:33 good_news_everyon joined #mojo
23:33 good_news_everyon [minion] kraih pushed 1 new commit to master: https://git.io/vbEHO
23:33 good_news_everyon minion/master 6d766d3 Sebastian Riedel: test lock management
23:33 good_news_everyon left #mojo
23:34 cfedde joined #mojo
23:35 CandyAngel Has anyone implemented anything where an app adds detected abuse to block at firewall or anything like that?
23:36 good_news_everyon joined #mojo
23:36 good_news_everyon [minion] kraih pushed 1 new commit to master: https://git.io/vbEHa
23:36 good_news_everyon minion/master 16e7276 Sebastian Riedel: more consistent tests
23:36 good_news_everyon left #mojo
23:37 Grinnz CandyAngel: fail2ban
23:37 purl fail2ban is slightly too effective. Corion accidently set it off by trying password auth from dromedary
23:37 Grinnz lol
23:37 Grinnz CandyAngel: if you mean in perl, then dunno
23:37 Grinnz but you could easily add a fail2ban jail that watches an access log of your perl app
23:38 Grinnz (i usually just use the nginx one)
23:38 CandyAngel Ah okies, thanks. Will bear that in mind :)
23:40 disputin joined #mojo
23:44 aborazmeh joined #mojo
23:48 Seth joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary