| Time |
Nick |
Message |
| 00:07 |
|
marty joined #mojo |
| 00:19 |
|
marty joined #mojo |
| 00:29 |
|
dvinciguerra_ joined #mojo |
| 00:30 |
|
lluad_ joined #mojo |
| 01:20 |
|
marty joined #mojo |
| 01:47 |
|
absolut_todd joined #mojo |
| 02:21 |
|
marty joined #mojo |
| 02:52 |
|
noganex joined #mojo |
| 03:17 |
|
lluad joined #mojo |
| 03:23 |
|
marty joined #mojo |
| 03:29 |
|
kid51 joined #mojo |
| 03:57 |
|
kaare joined #mojo |
| 04:02 |
jberger |
Vacation phase 2 has begun! o/ from the dry side of the island |
| 04:24 |
|
marty joined #mojo |
| 04:48 |
|
seatek joined #mojo |
| 05:24 |
|
marty joined #mojo |
| 05:44 |
|
bobkare joined #mojo |
| 06:25 |
|
marty joined #mojo |
| 07:26 |
|
marty joined #mojo |
| 07:28 |
|
Vandal joined #mojo |
| 08:21 |
|
marty joined #mojo |
| 08:51 |
|
berov joined #mojo |
| 09:39 |
|
odc joined #mojo |
| 10:28 |
|
punter joined #mojo |
| 11:09 |
|
cuechan joined #mojo |
| 11:23 |
* sri |
yawns |
| 11:49 |
|
kid51 joined #mojo |
| 12:16 |
|
vicash left #mojo |
| 13:29 |
|
gtodd left #mojo |
| 13:35 |
nic |
I'm choosing to understand that as the side of the island having no whisky |
| 13:38 |
sri |
hahahahahaha http://i.imgur.com/navpuPi.gifv |
| 13:52 |
|
lluad joined #mojo |
| 13:56 |
|
kaare joined #mojo |
| 14:02 |
marcus |
sri: accurate :) |
| 14:12 |
|
tchaves joined #mojo |
| 14:34 |
|
tchaves joined #mojo |
| 15:02 |
|
cuechan joined #mojo |
| 15:39 |
|
PryMar56 joined #mojo |
| 15:42 |
|
cuechan joined #mojo |
| 15:45 |
|
cuechan_ joined #mojo |
| 15:49 |
|
dod joined #mojo |
| 16:07 |
|
punter joined #mojo |
| 16:09 |
|
dod joined #mojo |
| 16:10 |
|
dod joined #mojo |
| 16:44 |
|
noganex joined #mojo |
| 17:05 |
|
dod joined #mojo |
| 17:12 |
mishanti1 |
So i hear Mr Robot being mentioned several places. It is that good? I feel most shows that incorporate tech is massively cringe-inducing. :-/ |
| 17:12 |
jberger |
nic: if so those beers last night were a problem |
| 17:13 |
sri |
mishanti1: yes, most of the tech stuff so far has been plausible |
| 17:15 |
mishanti1 |
sri: Might have to give it a try one of these years. Thanks. :) |
| 17:17 |
sri |
mishanti1: one of the tech advisors even apologized on twitter for using an invalid ip address in one scene :) https://twitter.com/samesmail/status/608868894267330560 |
| 17:19 |
sri |
no wait, that's the show creator |
| 17:21 |
mishanti1 |
sri: Hah. Awesome. |
| 17:23 |
sri |
in one of the first episodes there was a gnome vs kde conversation |
| 17:27 |
mishanti1 |
Did someone mention Xmonad? :p |
| 17:29 |
|
trone joined #mojo |
| 17:30 |
|
asarch joined #mojo |
| 17:43 |
coolo |
sri: that was really painful to me |
| 17:48 |
sri |
accurate though ;p |
| 17:49 |
sri |
hmm, now dhl has sent my laptop from the netherlands to east germany, basically flying over why i am :S |
| 17:50 |
sri |
s/why/where/ |
| 17:50 |
coolo |
sri: the problem is that kde vs gnome is basically the story of my life - watching it in prime time (even if it was just amazon prime time :) was really shoking :) |
| 17:51 |
sri |
but delivery is now scheduled for monday, so that's good at least |
| 17:59 |
jberger |
marcus and batman both make the top 200 CPAN directory sizes list! |
| 17:59 |
jberger |
https://gist.github.com/xdg/7691458b8230226dbd854add540df747 |
| 17:59 |
jberger |
Clean up, my brothers! |
| 18:00 |
sri |
haha, was about to say, that's not a good thing ;p |
| 18:01 |
xdg |
fyi: not my analysis -- this is just a sort of the indices/du-k.gz file |
| 18:02 |
sri |
on the other hand, cleaning your cpan dirs could also be considered a screwup |
| 18:03 |
sri |
i kinda regret starting to delete regularly |
| 18:03 |
xdg |
everything is still on backpan, though |
| 18:03 |
sri |
but without checksums |
| 18:03 |
xdg |
yeah. :-( |
| 18:04 |
jberger |
Could backpan be improved? |
| 18:04 |
sri |
not that checksums mean much |
| 18:04 |
sri |
but if they did... |
| 18:04 |
sri |
cpan security is still pretty broken |
| 18:04 |
jberger |
Yeah |
| 18:05 |
jberger |
We at least have a full https path now |
| 18:05 |
xdg |
https://github.com/andk/pause/issues/224 filed for the next QAH |
| 18:05 |
sri |
just glad we have "cpanm -M https://cpan.metacpan.org" |
| 18:05 |
jberger |
But signatures still seems to be an afterthought |
| 18:05 |
jberger |
xdg++ |
| 18:06 |
sri |
heh |
| 18:06 |
sri |
xdg++ |
| 18:06 |
jberger |
If I'm lucky enough to be asked back to qah next year i could work on that |
| 18:06 |
xdg |
(well volunteered!) |
| 18:07 |
jberger |
I know i was at the bottom of the list this year, its ok |
| 18:07 |
xdg |
once you show up, it improves the odds for the next year. :-) |
| 18:08 |
jberger |
I find myself really hoping to be invited, that was a special experience |
| 18:09 |
sri |
who makes the list? |
| 18:09 |
sri |
can we mobilize the mojo army to get you bumped on the list? :) |
| 18:10 |
jberger |
I don't know if they're is a defined process but it certainly starts with maintainers of toolchain level modules |
| 18:10 |
jberger |
My involvement with Test2 (even as much as it was) pushed me over to the gray side of the line |
| 18:11 |
jberger |
Turned out to be handy as the metacpan guys have started to use minion |
| 18:13 |
sri |
so many cool things that could still be done with minion |
| 18:13 |
jberger |
Which reminds me, I still need to redo the internals of Test::Mojo::Role::Phantom now that Test2 is live (which was the original reason die my involvement) |
| 18:13 |
sri |
been thinking about broadcasting commands to workers |
| 18:13 |
jberger |
Argh, s/die/for/ |
| 18:14 |
jberger |
sri: That combined with a web frontend would be really powerful |
| 18:14 |
sri |
yea |
| 18:14 |
sri |
"kill job 123", "increase worker pool by 1" |
| 18:15 |
jberger |
I have so many things i want to do |
| 18:15 |
jberger |
I kinda want to open sieve the queue I just wrote (very different goals) |
| 18:16 |
sri |
at least minion is a rock solid foundation now |
| 18:16 |
* jberger |
gets real keyboard |
| 18:17 |
sri |
hacking on job queues is really fun |
| 18:17 |
sri |
could do it all day :) |
| 18:17 |
sri |
if only there was a bigger market for perl enterprise job queues |
| 18:19 |
jberger |
ok now hopefully with fewer swipe typos |
| 18:24 |
jberger |
yeah minion is nice, and it is still going to be my primary go-to |
| 18:24 |
jberger |
mine is for when you need sequential queues and check-locks-before-dequeue locking |
| 18:26 |
sri |
what is a sequential queue? |
| 18:27 |
jberger |
for each (named) queue, jobs are FIFO |
| 18:27 |
jberger |
FIFO is probably the better name |
| 18:27 |
sri |
same in minion |
| 18:28 |
jberger |
but only one at a time, which minion doesn't do (maybe that's where the word sequential comes from) |
| 18:28 |
sri |
you mean only one active job per named queue at a time? |
| 18:29 |
jberger |
yes |
| 18:29 |
sri |
that's a weird requirement ;p |
| 18:29 |
sri |
think that's normally handled with locks |
| 18:29 |
jberger |
mostly for hardware provisioning |
| 18:30 |
sri |
in all job queues i know at least |
| 18:30 |
jberger |
its such a common requirement there that it basically needed to be baked in at the low level |
| 18:30 |
jberger |
each server we manage has its own queue |
| 18:31 |
jberger |
and you would never want someone provisioning an os at the same time as someone else reboots it |
| 18:31 |
jberger |
but all the jobs for that server should go in order, one at a time |
| 18:31 |
sri |
yea, definitely a weird design |
| 18:31 |
sri |
(named queue per server) |
| 18:32 |
sri |
totally would have gone with locks |
| 18:32 |
kid51 |
Does the minion you're referring to appear on this disambiguation page? https://en.wikipedia.org/wiki/Minion |
| 18:32 |
jberger |
there are also lots of levels of locking |
| 18:32 |
jberger |
this is basically just one such level (implemented basically the same way |
| 18:32 |
sri |
kid51: http://mojolicious.org/perldoc#SPIN-OFFS |
| 18:32 |
jberger |
) # ocd |
| 18:33 |
sri |
jberger: i would have done it this way https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting#concurrent |
| 18:34 |
sri |
would have loved a nice shared lock primitive in minion |
| 18:35 |
jberger |
that would be nice |
| 18:36 |
jberger |
not sure if it could be done on the database side (for performance) in all backends though |
| 18:38 |
sri |
don't think performance is a huge concern |
| 18:38 |
jberger |
anyway, I'm still happy that job dependencies came out of my $work stuff into minion, I'm pretty proud of that |
| 18:38 |
sri |
latency maybe, but we are already relying on eventual consistency a lot |
| 18:39 |
sri |
i mean, worst case would be for ->repair to invalidate locks |
| 18:39 |
jberger |
I could go back to my old $work branch (when I was still working as an enhanced backend for minion with pg) and see if the locking could be taken |
| 18:40 |
sri |
did you research the best shared lock implementation for postgres? |
| 18:40 |
sri |
kinda curious now :) |
| 18:41 |
jberger |
I have no idea if it was a "best" of any kind |
| 18:41 |
jberger |
I just had a table that was checked before dequeue |
| 18:41 |
sri |
oh |
| 18:41 |
jberger |
that's where things went to stored procs pretty quickly |
| 18:41 |
sri |
implementation i had in mind wouldn't run before dequeue |
| 18:41 |
jberger |
part of why I gave back job deps when I did is that I didn't need stored proces |
| 18:42 |
jberger |
s/es$/s/ |
| 18:42 |
sri |
it would be called from inside the task closure |
| 18:42 |
jberger |
that doesn't seem to scale well if you have a lot of locked jobs / pending jobs |
| 18:42 |
jberger |
(which in our case we will likely have) |
| 18:43 |
jberger |
(which is where the sequential queues notions came from) |
| 18:44 |
sri |
you migth be overestimating the problem |
| 18:44 |
jberger |
always possible |
| 18:45 |
sri |
if you have a lot of jobs blocking each other active at the same time, it seems like those should have used dependencies |
| 18:46 |
sri |
since they would have to be enqueued at the same time |
| 18:46 |
sri |
otherwise, insert order should already take care of it |
| 18:47 |
jberger |
I don't know if this will be TL;DR level, but it is mostly just taking the Pg backend (as it existed before job dependencies obviously) and adding things into the dequeue method (which is now basically a server-side function) |
| 18:47 |
jberger |
https://gist.github.com/jberger/3cb978b1a376b87f5f3bff827c13f96d |
| 18:48 |
jberger |
this is really the dequeue https://gist.github.com/jberger/3cb978b1a376b87f5f3bff827c13f96d#file-pgextra-pm-L760-L838 |
| 18:49 |
sri |
i think the problem is generally called "rate limit" |
| 18:49 |
|
punter joined #mojo |
| 18:49 |
sri |
which in your case is a rate limit of 1 per server |
| 18:52 |
jberger |
the non-minion version of the code that this became now has rate limiting too, we actually need it to pause between jobs because embedded hardware often falls over if you hit it too often |
| 18:52 |
jberger |
seriously, embedded hardware really sucks |
| 18:55 |
jberger |
I don't know if this is a good thing or a bad thing, but with the simplicity of Mojo::Pg's migrations I'm not scared of stored procedures anymore |
| 18:56 |
jberger |
because they can exist in version control and be deployed simply |
| 18:57 |
sri |
interesting, celery has a different strategy |
| 18:57 |
|
Vitrifur joined #mojo |
| 18:57 |
sri |
they have a rate limit per named queue, per worker |
| 18:58 |
sri |
like "1 per minute", and each worker just stops requesting jobs for a certain named queue |
| 18:58 |
sri |
kue and sidekiq go with the shared lock implementation i mentioned above |
| 18:59 |
sri |
(only planned for kue so far though) |
| 18:59 |
sri |
lots of 3rd party modules with token bucket implementations |
| 19:33 |
|
asarch joined #mojo |
| 19:41 |
jberger |
so a really strange thing here |
| 19:41 |
jberger |
I expected better internet on the dry side since infrastructure must be easier to do over here |
| 19:41 |
jberger |
and yet I think it is the opposite |
| 19:42 |
jberger |
because it is dry here they have satellite internet whereas on the wet side satellite wouldn't cut it so everything is cable internet |
| 19:42 |
jberger |
the connection here seems spotty, there is was amazing |
| 19:43 |
marcus |
jberger: where are you? |
| 19:44 |
jberger |
kauai |
| 19:45 |
jberger |
I was in wainiha, now I'm in Kekaha |
| 19:47 |
marcus |
neat |
| 19:48 |
marcus |
I just installed yet another Autodesk product on my computer. I think they have a secret agenda to take over our family |
| 19:49 |
jberger |
heh, so from here the beach faces south and I was thinking, I wonder how far it is in that direction before you hit any land |
| 19:49 |
jberger |
almost 7k miles |
| 19:49 |
jberger |
antarctica |
| 19:49 |
marcus |
A friend of mine is going there in November |
| 19:49 |
marcus |
he's summiting a mountain then walking to the south pole on skis afterwards |
| 19:50 |
jberger |
hahaha |
| 19:50 |
jberger |
seriously? |
| 19:50 |
marcus |
yes. You could say he has a somewhat worse 40-year crisis than average :) |
| 19:51 |
|
dod joined #mojo |
| 19:51 |
jberger |
at first I thought you meant Kauai, then you said "south pole" and I assumed you were joking, then I realized ... |
| 19:51 |
marcus |
also he was lucky enough to inherit a boatload of money from a swedish aunt |
| 19:51 |
marcus |
it's not very cheap to do that. |
| 19:51 |
jberger |
ummm, give some to me, I'm sure i'll use it better |
| 19:51 |
marcus |
I think we'd all use it better: ) |
| 19:52 |
jberger |
like we found some undeveloped land near Haena beach that is going for $2.6M |
| 19:52 |
jberger |
done |
| 19:53 |
jberger |
https://www.google.com/maps/place/5-7858+Kuhio+Hwy,+Kilauea,+HI+96754/@22.220484,-159.5654622,221m/data=!3m2!1e3!4b1!4m5!3m4!1s0x7c06fa197896f7fd:0xd3ae7c4ca678e95c!8m2!3d22.220484!4d-159.564915 |
| 19:53 |
jberger |
between the two houses |
| 19:54 |
|
berov joined #mojo |
| 19:54 |
marcus |
:D |
| 20:15 |
|
tchaves joined #mojo |
| 20:27 |
bpmedley |
https://github.com/kraih/minion/compare/master...brianmed:worker_addendum <-- sri, jberger what do y'all think of this approach for adjusting the watched queues and max jobs performed on the fly? |
| 20:30 |
sri |
bpmedley: disgree strongly |
| 20:31 |
bpmedley |
Interesting. Is there anything in the patch worth saving? May I ask your approach? |
| 20:31 |
sri |
i had a pubsub solution in mind, with arbitrary commands to be broadcastable to all workers |
| 20:33 |
sri |
anyway, the basic implementation is not really the problem |
| 20:33 |
sri |
making everything testable is |
| 20:37 |
bpmedley |
Cool, thanks for perusing the code.. |
| 20:38 |
sri |
don't put too much effort into it, i have a very specific design already in mind |
| 20:39 |
bpmedley |
Sure, that's what I just gathered.. I learned a few things, tho.. :) |
| 20:39 |
sri |
your solution is also rather inefficient |
| 20:39 |
bpmedley |
The table gets a select every iteration of the _work loop. Is that what you're talking about? |
| 20:40 |
sri |
like, you're not reusing register_worker |
| 20:40 |
sri |
commands are hardcoded as table fields |
| 20:40 |
sri |
and job command flags |
| 20:42 |
sri |
anyway, testability first |
| 20:43 |
bpmedley |
Sure, it's rough around the edges in several places. Given that your desired outcome is pubsub vs a DB lookup, I'd say it's not worth perusing much anymore.. :-P |
| 20:43 |
sri |
when i say pubsub, i mean the pattern, not the Mojo::Pg feature |
| 20:44 |
bpmedley |
I think I see your point about register_worker. Thanks for pointing that out. |
| 20:44 |
sri |
i would maybe still use a query, but i'd add an inbox array column to the worker table and check it in register_worker |
| 20:44 |
sri |
much more efficient |
| 20:45 |
bpmedley |
Hrrm, in a long running system with dynamic workers, then could the inbox array grow rather large? |
| 20:45 |
sri |
depending on whichever turns out to be the most efficient solution after testing |
| 20:45 |
bpmedley |
By dynamic I mean a worker that is short lived. |
| 20:46 |
sri |
now i need to get food |
| 20:53 |
|
tchaves joined #mojo |
| 21:14 |
|
vicash joined #mojo |
| 21:48 |
|
punter joined #mojo |
| 21:48 |
|
punter joined #mojo |
| 21:54 |
|
tchaves joined #mojo |
| 22:33 |
|
Mikey joined #mojo |
| 22:42 |
|
seatek joined #mojo |
| 23:51 |
|
trobotham joined #mojo |
| 23:52 |
bpmedley |
https://github.com/kraih/minion/compare/master...brianmed:worker_addendum <-- sri, thoughts on a minion_msg table? Tests are included.. :) |
