The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2016-07-09

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 marty joined #mojo
00:19 marty joined #mojo
00:29 dvinciguerra_ joined #mojo
00:30 lluad_ joined #mojo
01:20 marty joined #mojo
01:47 absolut_todd joined #mojo
02:21 marty joined #mojo
02:52 noganex joined #mojo
03:17 lluad joined #mojo
03:23 marty joined #mojo
03:29 kid51 joined #mojo
03:57 kaare joined #mojo
04:02 jberger Vacation phase 2 has begun! o/ from the dry side of the island
04:24 marty joined #mojo
04:48 seatek joined #mojo
05:24 marty joined #mojo
05:44 bobkare joined #mojo
06:25 marty joined #mojo
07:26 marty joined #mojo
07:28 Vandal joined #mojo
08:21 marty joined #mojo
08:51 berov joined #mojo
09:39 odc joined #mojo
10:28 punter joined #mojo
11:09 cuechan joined #mojo
11:23 * sri yawns
11:49 kid51 joined #mojo
12:16 vicash left #mojo
13:29 gtodd left #mojo
13:35 nic I'm choosing to understand that as the side of the island having no whisky
13:38 sri hahahahahaha http://i.imgur.com/navpuPi.gifv
13:52 lluad joined #mojo
13:56 kaare joined #mojo
14:02 marcus sri: accurate :)
14:12 tchaves joined #mojo
14:34 tchaves joined #mojo
15:02 cuechan joined #mojo
15:39 PryMar56 joined #mojo
15:42 cuechan joined #mojo
15:45 cuechan_ joined #mojo
15:49 dod joined #mojo
16:07 punter joined #mojo
16:09 dod joined #mojo
16:10 dod joined #mojo
16:44 noganex joined #mojo
17:05 dod joined #mojo
17:12 mishanti1 So i hear Mr Robot being mentioned several places. It is that good? I feel most shows that incorporate tech is massively cringe-inducing. :-/
17:12 jberger nic: if so those beers last night were a problem
17:13 sri mishanti1: yes, most of the tech stuff so far has been plausible
17:15 mishanti1 sri: Might have to give it a try one of these years. Thanks. :)
17:17 sri mishanti1: one of the tech advisors even apologized on twitter for using an invalid ip address in one scene :) https://twitter.com/samesmail/status/608868894267330560
17:19 sri no wait, that's the show creator
17:21 mishanti1 sri: Hah. Awesome.
17:23 sri in one of the first episodes there was a gnome vs kde conversation
17:27 mishanti1 Did someone mention Xmonad? :p
17:29 trone joined #mojo
17:30 asarch joined #mojo
17:43 coolo sri: that was really painful to me
17:48 sri accurate though ;p
17:49 sri hmm, now dhl has sent my laptop from the netherlands to east germany, basically flying over why i am :S
17:50 sri s/why/where/
17:50 coolo sri: the problem is that kde vs gnome is basically the story of my life - watching it in prime time (even if it was just amazon prime time :) was really shoking :)
17:51 sri but delivery is now scheduled for monday, so that's good at least
17:59 jberger marcus and batman both make the top 200 CPAN directory sizes list!
17:59 jberger https://gist.github.com/xdg/7691458b8230226dbd854add540df747
17:59 jberger Clean up, my brothers!
18:00 sri haha, was about to say, that's not a good thing ;p
18:01 xdg fyi: not my analysis -- this is just a sort of the indices/du-k.gz file
18:02 sri on the other hand, cleaning your cpan dirs could also be considered a screwup
18:03 sri i kinda regret starting to delete regularly
18:03 xdg everything is still on backpan, though
18:03 sri but without checksums
18:03 xdg yeah. :-(
18:04 jberger Could backpan be improved?
18:04 sri not that checksums mean much
18:04 sri but if they did...
18:04 sri cpan security is still pretty broken
18:04 jberger Yeah
18:05 jberger We at least have a full https path now
18:05 xdg https://github.com/andk/pause/issues/224 filed for the next QAH
18:05 sri just glad we have "cpanm -M https://cpan.metacpan.org"
18:05 jberger But signatures still seems to be an afterthought
18:05 jberger xdg++
18:06 sri heh
18:06 sri xdg++
18:06 jberger If I'm lucky enough to be asked back to qah next year i could work on that
18:06 xdg (well volunteered!)
18:07 jberger I know i was at the bottom of the list this year, its ok
18:07 xdg once you show up, it improves the odds for the next year. :-)
18:08 jberger I find myself really hoping to be invited, that was a special experience
18:09 sri who makes the list?
18:09 sri can we mobilize the mojo army to get you bumped on the list? :)
18:10 jberger I don't know if they're is a defined process but it certainly starts with maintainers of toolchain level modules
18:10 jberger My involvement with Test2 (even as much as it was) pushed me over to the gray side of the line
18:11 jberger Turned out to be handy as the metacpan guys have started to use minion
18:13 sri so many cool things that could still be done with minion
18:13 jberger Which reminds me, I still need to redo the internals of Test::Mojo::Role::Phantom now that Test2 is live (which was the original reason die my involvement)
18:13 sri been thinking about broadcasting commands to workers
18:13 jberger Argh, s/die/for/
18:14 jberger sri: That combined with a web frontend would be really powerful
18:14 sri yea
18:14 sri "kill job 123", "increase worker pool by 1"
18:15 jberger I have so many things i want to do
18:15 jberger I kinda want to open sieve the queue I just wrote (very different goals)
18:16 sri at least minion is a rock solid foundation now
18:16 * jberger gets real keyboard
18:17 sri hacking on job queues is really fun
18:17 sri could do it all day :)
18:17 sri if only there was a bigger market for perl enterprise job queues
18:19 jberger ok now hopefully with fewer swipe typos
18:24 jberger yeah minion is nice, and it is still going to be my primary go-to
18:24 jberger mine is for when you need sequential queues and check-locks-before-dequeue locking
18:26 sri what is a sequential queue?
18:27 jberger for each (named) queue, jobs are FIFO
18:27 jberger FIFO is probably the better name
18:27 sri same in minion
18:28 jberger but only one at a time, which minion doesn't do (maybe that's where the word sequential comes from)
18:28 sri you mean only one active job per named queue at a time?
18:29 jberger yes
18:29 sri that's a weird requirement ;p
18:29 sri think that's normally handled with locks
18:29 jberger mostly for hardware provisioning
18:30 sri in all job queues i know at least
18:30 jberger its such a common requirement there that it basically needed to be baked in at the low level
18:30 jberger each server we manage has its own queue
18:31 jberger and you would never want someone provisioning an os at the same time as someone else reboots it
18:31 jberger but all the jobs for that server should go in order, one at a time
18:31 sri yea, definitely a weird design
18:31 sri (named queue per server)
18:32 sri totally would have gone with locks
18:32 kid51 Does the minion you're referring to appear on this disambiguation page? https://en.wikipedia.org/wiki/Minion
18:32 jberger there are also lots of levels of locking
18:32 jberger this is basically just one such level (implemented basically the same way
18:32 sri kid51: http://mojolicious.org/perldoc#SPIN-OFFS
18:32 jberger ) # ocd
18:33 sri jberger: i would have done it this way https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting#concurrent
18:34 sri would have loved a nice shared lock primitive in minion
18:35 jberger that would be nice
18:36 jberger not sure if it could be done on the database side (for performance) in all backends though
18:38 sri don't think performance is a huge concern
18:38 jberger anyway, I'm still happy that job dependencies came out of my $work stuff into minion, I'm pretty proud of that
18:38 sri latency maybe, but we are already relying on eventual consistency a lot
18:39 sri i mean, worst case would be for ->repair to invalidate locks
18:39 jberger I could go back to my old $work branch (when I was still working as an enhanced backend for minion with pg) and see if the locking could be taken
18:40 sri did you research the best shared lock implementation for postgres?
18:40 sri kinda curious now :)
18:41 jberger I have no idea if it was a "best" of any kind
18:41 jberger I just had a table that was checked before dequeue
18:41 sri oh
18:41 jberger that's where things went to stored procs pretty quickly
18:41 sri implementation i had in mind wouldn't run before dequeue
18:41 jberger part of why I gave back job deps when I did is that I didn't need stored proces
18:42 jberger s/es$/s/
18:42 sri it would be called from inside the task closure
18:42 jberger that doesn't seem to scale well if you have a lot of locked jobs / pending jobs
18:42 jberger (which in our case we will likely have)
18:43 jberger (which is where the sequential queues notions came from)
18:44 sri you migth be overestimating the problem
18:44 jberger always possible
18:45 sri if you have a lot of jobs blocking each other active at the same time, it seems like those should have used dependencies
18:46 sri since they would have to be enqueued at the same time
18:46 sri otherwise, insert order should already take care of it
18:47 jberger I don't know if this will be TL;DR level, but it is mostly just taking the Pg backend (as it existed before job dependencies obviously) and adding things into the dequeue method (which is now basically a server-side function)
18:47 jberger https://gist.github.com/jberger/3cb978b1a376b87f5f3bff827c13f96d
18:48 jberger this is really the dequeue https://gist.github.com/jberger/3cb978b1a376b87f5f3bff827c13f96d#file-pgextra-pm-L760-L838
18:49 sri i think the problem is generally called "rate limit"
18:49 punter joined #mojo
18:49 sri which in your case is a rate limit of 1 per server
18:52 jberger the non-minion version of the code that this became now has rate limiting too, we actually need it to pause between jobs because embedded hardware often falls over if you hit it too often
18:52 jberger seriously, embedded hardware really sucks
18:55 jberger I don't know if this is a good thing or a bad thing, but with the simplicity of Mojo::Pg's migrations I'm not scared of stored procedures anymore
18:56 jberger because they can exist in version control and be deployed simply
18:57 sri interesting, celery has a different strategy
18:57 Vitrifur joined #mojo
18:57 sri they have a rate limit per named queue, per worker
18:58 sri like "1 per minute", and each worker just stops requesting jobs for a certain named queue
18:58 sri kue and sidekiq go with the shared lock implementation i mentioned above
18:59 sri (only planned for kue so far though)
18:59 sri lots of 3rd party modules with token bucket implementations
19:33 asarch joined #mojo
19:41 jberger so a really strange thing here
19:41 jberger I expected better internet on the dry side since infrastructure must be easier to do over here
19:41 jberger and yet I think it is the opposite
19:42 jberger because it is dry here they have satellite internet whereas on the wet side satellite wouldn't cut it so everything is cable internet
19:42 jberger the connection here seems spotty, there is was amazing
19:43 marcus jberger: where are you?
19:44 jberger kauai
19:45 jberger I was in wainiha, now I'm in Kekaha
19:47 marcus neat
19:48 marcus I just installed yet another Autodesk product on my computer. I think they have a secret agenda to take over our family
19:49 jberger heh, so from here the beach faces south and I was thinking, I wonder how far it is in that direction before you hit any land
19:49 jberger almost 7k miles
19:49 jberger antarctica
19:49 marcus A friend of mine is going there in November
19:49 marcus he's summiting a mountain then walking to the south pole on skis afterwards
19:50 jberger hahaha
19:50 jberger seriously?
19:50 marcus yes. You could say he has a somewhat worse 40-year crisis than average :)
19:51 dod joined #mojo
19:51 jberger at first I thought you meant Kauai, then you said "south pole" and I assumed you were joking, then I realized ...
19:51 marcus also he was lucky enough to inherit a boatload of money from a swedish aunt
19:51 marcus it's not very cheap to do that.
19:51 jberger ummm, give some to me, I'm sure i'll use it better
19:51 marcus I think we'd all use it better: )
19:52 jberger like we found some undeveloped land near Haena beach that is going for $2.6M
19:52 jberger done
19:53 jberger https://www.google.com/maps/place/5-7858+Kuhio+Hwy,+Kilauea,+HI+96754/@22.220484,-159.5654622,221m/data=!3m2!1e3!4b1!4m5!3m4!1s0x7c06fa197896f7fd:0xd3ae7c4ca678e95c!8m2!3d22.220484!4d-159.564915
19:53 jberger between the two houses
19:54 berov joined #mojo
19:54 marcus :D
20:15 tchaves joined #mojo
20:27 bpmedley https://github.com/kraih/minion/compare/master...brianmed:worker_addendum <-- sri, jberger what do y'all think of this approach for adjusting the watched queues and max jobs performed on the fly?
20:30 sri bpmedley: disgree strongly
20:31 bpmedley Interesting.  Is there anything in the patch worth saving?  May I ask your approach?
20:31 sri i had a pubsub solution in mind, with arbitrary commands to be broadcastable to all workers
20:33 sri anyway, the basic implementation is not really the problem
20:33 sri making everything testable is
20:37 bpmedley Cool, thanks for perusing the code..
20:38 sri don't put too much effort into it, i have a very specific design already in mind
20:39 bpmedley Sure, that's what I just gathered.. I learned a few things, tho.. :)
20:39 sri your solution is also rather inefficient
20:39 bpmedley The table gets a select every iteration of the _work loop.  Is that what you're talking about?
20:40 sri like, you're not reusing register_worker
20:40 sri commands are hardcoded as table fields
20:40 sri and job command flags
20:42 sri anyway, testability first
20:43 bpmedley Sure, it's rough around the edges in several places.  Given that your desired outcome is pubsub vs a DB lookup, I'd say it's not worth perusing much anymore.. :-P
20:43 sri when i say pubsub, i mean the pattern, not the Mojo::Pg feature
20:44 bpmedley I think I see your point about register_worker.  Thanks for pointing that out.
20:44 sri i would maybe still use a query, but i'd add an inbox array column to the worker table and check it in register_worker
20:44 sri much more efficient
20:45 bpmedley Hrrm, in a long running system with dynamic workers, then could the inbox array grow rather large?
20:45 sri depending on whichever turns out to be the most efficient solution after testing
20:45 bpmedley By dynamic I mean a worker that is short lived.
20:46 sri now i need to get food
20:53 tchaves joined #mojo
21:14 vicash joined #mojo
21:48 punter joined #mojo
21:48 punter joined #mojo
21:54 tchaves joined #mojo
22:33 Mikey joined #mojo
22:42 seatek joined #mojo
23:51 trobotham joined #mojo
23:52 bpmedley https://github.com/kraih/minion/compare/master...brianmed:worker_addendum <-- sri, thoughts on a minion_msg table?  Tests are included.. :)

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary