The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-07-05

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:20 tchaves joined #mojo
01:22 jnbek joined #mojo
01:29 schelcj joined #mojo
02:17 noganex joined #mojo
02:32 gizmomathboy joined #mojo
02:40 schelcj joined #mojo
03:13 VVelox joined #mojo
04:23 maschine joined #mojo
04:35 inokenty-w joined #mojo
05:27 jnbek joined #mojo
06:59 AndrewIsh joined #mojo
07:05 leont_ joined #mojo
07:23 dod joined #mojo
07:25 Vandal joined #mojo
07:31 karjala_ joined #mojo
07:37 trone joined #mojo
07:45 prg joined #mojo
08:31 tyldis Need design input. I got a list of tasks to be executed a given timestamps (well, an ->emit at given time). Needs to be accurate to within a second.
08:33 tyldis The list of tasks is in a sqlite DB is to poll the DB and ->emit if the timestamp matches. This isn't very efficient and is kinda prone to executing multiple ->emit unless I also introduce a state variable.
08:34 CandyAngel tyldis: You can add a job to Minion through cron
08:34 tyldis CandyAngel: Appreciate the idea, trying to make the app self contained, though.
08:36 karjala_ CandyAngel, wont' the minion job that cron will add, be executed immediately? So the timing will have minute-accuracy, not second
08:37 sri really too bad minion workers started from within the app are not a thing yet, even though we have the technology
08:37 sri (having the technology meaning that we only need to support platforms that have pipes and real fork)
08:41 rshadow joined #mojo
08:41 CandyAngel karjala_: You can delay a job added to Minion, so you could really add it at any time before it needs to be run with the appropriate delay
08:42 CandyAngel There isn't an assurance it will run at exactly that time though
08:42 CandyAngel Like, if the workers are busy it could be late
08:43 CandyAngel Also, I think workers dequeue a job a second, and that is hard-coded..
08:43 CandyAngel Or they dequeue X jobs a second, I don't remember
08:44 karjala_ systemd can schedule jobs with second accuracy also
08:44 karjala_ with .timer units
08:44 CandyAngel You could start a worker that uses a queue named after the time and it will just sit and wait for that job though :P
08:46 CandyAngel I know about the 1 second hard-coded delay because I wanted Minion to do 1 job at a time but without a second delay between each one..
08:47 CandyAngel (I basically wanted the equivalent of --random-wait from wget)
08:47 karjala_ this 1 second hard-coded delay places a hard-coded limit on how many jobs/second the whole system can do
08:48 karjala_ a theoretical limit
08:48 purl a theoretical limit is 100 hours
08:48 CandyAngel Uhuh, it would be workers*jobs/second
08:49 karjala_ jobs/second can't exceed 1, though
08:49 karjala_ can it?
08:49 purl NO!  IT CAN'T!
08:49 karjala_ see?
08:49 purl O_O
08:49 CandyAngel Yeah
08:49 CandyAngel Each worker will dequeue X jobs (defaults to 4)
08:49 karjala_ a
08:49 CandyAngel But it will wait a second between each dequeue
08:49 CandyAngel So if it does those 4 jobs in 10ms, it'll idle for 990ms
08:49 karjala_ why does it wait?
08:50 karjala_ is there a reason?
08:52 CandyAngel I presume it is to stop them just chewing CPU looking for jobs
08:52 CandyAngel https://github.com/kraih/minion/blob/master/lib/Minion/Command/minion/worker.pm#L74
08:52 CandyAngel I hope sri's quietness isn't him head-desking because I'm misunderstood how it works and everything I'm saying is wrong :P
08:52 CandyAngel I've misunderstood*
08:54 karjala_ But couldn't the workers get efficiently notified of new jobs through pub/sub?
08:54 karjala_ on those backends that support it
09:32 stryx` joined #mojo
09:44 CandyAngel Oh yeah, Minion doesn't work on (native) Windows.. was going to make a simple test to double check
09:47 cosimo joined #mojo
09:56 sri karjala_: minion already uses pub/sub with the Pg backend
09:56 karjala_ but it also adds a one-second delay between dequeings?
09:57 sri no
10:00 CandyAngel If you have 60 jobs queued and start a worker with '-j 1', it takes (at least) 60 seconds to clear the queue
10:01 CandyAngel That's what karjala_ is referring to
10:01 sri well, if he has a better idea patches welcome
10:08 CandyAngel I'm up for submitting a patch to make the wait a setting. karjala_, interested?
10:10 karjala_ Well, I don't know the point is of sleep(1) there... Surely there must be a reason (otherwise sri wouldn't have put it there), and the user (who will choose the optional setting) will have to know the reprecussins
10:10 karjala_ ions
10:10 karjala_ I don't know them, at least
10:10 karjala_ So I dont' know whether such a patch would be a good thing
10:11 karjala_ CandyAngel, do you understand the reasoning behind sleeping there? (since, in pg at least, pub/sub is used)
10:11 CandyAngel No, I only know that it prevented me using Minion for a certain task (running jobs one-at-a-time, but as fast as possible)
10:11 karjala_ Maybe we should ask sri
10:12 sri it doesn't want more jobs, blocking on the pub/sub channel to wait for more jobs would be silly
10:12 sri it would be nothing else than a sleep
10:12 sri makes no sense at all
10:13 sri if there was no sleep 1 it would just be looping very fast burning cpu an actual job could have used
10:15 karjala_ So there's no way of waiting for pub/sub without consuming lots of CPU?
10:15 sri what are you talking about?
10:15 sri what would it wait for on the pub/sub channel?
10:16 karjala_ a "there's a new job" message (I can't say I udnerstood 100% what you wrote above, so I'm asking further)
10:16 sri pub/sub is for newly enqueued jobs to be announced, that has abolutely no value for a worker that's at full capacity
10:16 sri it only does sleep 1 if it is already at full capacity
10:17 CandyAngel If a worker dequeues a job that finishes in 10ms, it will sit and do nothing for a second before finding a new job to do
10:17 sri and you can optimize that how?
10:18 sri my invitation for patches stands, what i'm saying here is that "just doing something with the pub/sub channel" is no solution
10:18 sri it doesn't work
10:18 purl It's a Y2K error!  Panic!  Sue!
10:18 CandyAngel I think that is just confusing matters
10:18 karjala_ I'll look at it from home
10:19 sri please, prove me wrong
10:19 sri i'd be delighted to be wrong
10:20 karjala_ :-)
10:20 CandyAngel As I understand it, the worker dequeues a job, forks to handle it, then checks if it has reached the job limit
10:20 CandyAngel If it has, it sleeps for 1 second
10:20 karjala_ Do patch submitters get mentioned in the Changelog?
10:21 sri and credits section
10:21 CandyAngel But if the fork finishes in 0.1 seconds, the worker is not doing anything for 0.9 seconds
10:22 sri CandyAngel: i think we all get it
10:22 CandyAngel Ah okies
10:23 sri it's just that there is no solution, no other sensible syscall to block on
10:23 CandyAngel Hm? Why can't it use usleep?
10:24 CandyAngel Or sleep from Time::HiRes
10:25 sri what does that change?
10:26 sri 1s is a good value for keeping cpu usage low on current cpus
10:26 CandyAngel You can sleep for fractional seconds when the job limit is reached, so it will check if jobs are done more often
10:28 sri you might be able to make it perform well using SIGCHLD
10:28 sri please don't argue about better sleep values
10:28 sri just make it better
10:30 karjala_ SIGCHLD! Excellent idea
10:31 karjala_ no?
10:31 sri if it works on all the common unix versions and WSL, sure
10:36 sri CandyAngel: making it use usleep and the value configurable would be a last resort, if nothing else works
10:37 sri but i'm sure there are better solutions
10:37 CandyAngel Mhm, I'm making a patch/PR for that implementation
10:38 sri my first thought was changing the api a bit and blocking on waitpid, but that's still a rather shitty solution...
10:38 karjala_ sri: could this entry of the Config module tell us whether the system supports SIGCHLD? https://metacpan.org/source/XSAWYERX/perl-5.26.0/Porting/Glossary#L2901
10:39 karjala_ oh
10:39 karjala_ sorry
10:40 sri do we have to know? could it perhaps be implemented in a way that it does nothing on platforms where it isn't supported?
10:40 karjala_ yes we dont have to know
10:42 karjala_ Minion could even check itself manually at the beginning whether SIGCHLD does anything
10:42 karjala_ by testing itself
11:04 tyldis karjala_: The thing I'm tryinf to replace used crontab combined with sleep to achive sub-minute accuracy
11:04 CandyAngel Okay, I have made my changes and running a worker with no wait interval (0) results in 0 CPU while there are no jobs to do..
11:06 sri with no jobs it would block on pub/sub
11:06 CandyAngel Is there pub/sub with Storable backend?
11:06 sri the sleep is only used when it is maxed out with jobs
11:07 sri don't use the storable backend
11:07 sri only Pg really counts for minion core features
11:10 CandyAngel Ah okie
11:20 CandyAngel If this gets fixed, I get to change one of my systems back to using Minion ^_^
11:32 tchaves joined #mojo
11:47 sri CandyAngel: oh, i thought you were making a patch with SIGCHLD
11:48 CandyAngel I would have a go, but I don't have WSL to test it on
11:50 sri SIGCHLD works on WSL
11:52 sri considering this in the release notes "Corrected timing race with SIGCHLD and wait() when using ptrace" i'd assume it even works pretty well https://msdn.microsoft.com/en-us/commandline/wsl/release_notes
12:10 tchaves joined #mojo
12:14 tardisx joined #mojo
12:28 sri btw. i've been wondering how people here handle initializing staging environments for their mojo apps?
12:49 gizmomathboy joined #mojo
12:53 karjala_ Is that like an environment you create quite often, based on the staging branch of your git repo, where you run all the tests you've written for the project?
12:54 aborazmeh joined #mojo
12:54 sri guess i should have said dev/staging environment
12:54 nic sri: Surely Staging should be the same as Production
12:56 sri i've started adding a staging/ directory to my apps with start.pl/stop.pl scripts to create and destroy a smaller version of the live system for testing/development
12:56 bjakubski joined #mojo
12:56 nic ah, different meaning of the word 'staging'
12:57 sri yea, i'm conflating dev and staging
12:57 nic perhaps what I'd call UAT
12:57 nic or QA
12:59 sri i'm also interested in setups that have more stages
13:00 sri if you're setting up dev and staging environments separately
13:00 sri or even add tests and qa stages
13:01 sri i've been trying to make scripts that create a downsized version of the production system, without really aiming for a specific stage
13:03 tchaves joined #mojo
13:22 gryphon joined #mojo
13:23 Pyritic joined #mojo
13:25 marty joined #mojo
13:54 Pyritic joined #mojo
14:36 Pyritic joined #mojo
14:45 stryx` joined #mojo
14:48 jberger we just have different config files
14:48 jberger myapp.<<area>>.conf
14:49 jberger then we have a different vm for each area
14:53 zivester joined #mojo
15:24 sri jberger: and how do you fill your database?
15:25 jberger I make an application command that runs migrate on all our different mojo migrations modules and then I have a few config files with the few fixtures we need
15:26 PryMar56 joined #mojo
15:45 karjala_ joined #mojo
15:49 Pyritic joined #mojo
16:02 jabberwok joined #mojo
16:22 rshadow joined #mojo
16:36 stephan48 can i detect in startup() if i am run in a webserver or a worker?
16:40 sri no
16:43 bwf joined #mojo
16:43 sri intersting, i've been wondering if using next::method instead of SUPER:: would have a performance impact in mojolicious
16:44 sri my mojo app went from 15k rps down to 13.8k rps
16:44 sri quite noticeable
16:56 dod joined #mojo
17:03 genio that's a pretty big slowdown. o.O
17:53 Pyritic joined #mojo
17:59 rshadow joined #mojo
18:24 trone joined #mojo
18:41 mattp_ Is it possible to send a scalar directly to a response handle without copying?
18:42 mattp_ Ie serving large blobs, it looks like write and write chunk take copies of input
19:04 preaction mattp_: iirc you can use Mojo::Asset::Memory, but newer perls also have copy-on-write for scalars
19:33 tchaves joined #mojo
19:37 brunoramos joined #mojo
20:18 tchaves joined #mojo
20:21 tchaves joined #mojo
20:24 FROGGS joined #mojo
20:34 schelcj joined #mojo
20:54 schelcj joined #mojo
20:59 sri since nobody else wanted to teach minion about SIGCHLD... https://github.com/kraih/minion/commit/84d43e3ba3e2442c7e4bc3a769d24b000da4b8d0
21:00 sri that's all you need, the signal interrupts the sleep
21:01 sri simple test case MOJO_LOG_LEVEL=debug perl -Ilib -Mojo -E 'plugin Minion => {Pg => "postgres://tester:testing\@/test"}; app->minion->add_task(foo => sub {}); app->start' minion worker -j 1
21:01 sri with a queue filled with foo jobs (like after a minion_bench run)
21:02 sri CandyAngel: ^
21:10 Grinnz just used a template layout for the first time... did similar in template toolkit but dunno why i haven't in mojo :P
21:11 Grinnz i don't do html templates for work so it hasnt really been a problem i guess
21:11 * nic <3 layouts
21:14 maschine joined #mojo
21:15 maschine trying to use plugin->Config with Mojolicious::Lite, where I have my Controller/Model split into their own modules
21:16 maschine my $config = app->plugin(Config => {file => 'myapp.conf'}); - how do I make this accessible inside my modules without stating it in them?
21:16 maschine or is that just the easy way to do it?  (I don't like repeating code, as I'm sure most others agree).
21:17 sri controllers have access through $self->app
21:17 Grinnz in a lite app you just do: plugin Config => {file => 'myapp.conf'};
21:17 sri models get the value passed during instantiation
21:17 Grinnz that will populate app->config
21:17 sri (manually, by you)
21:18 * sri has tons of MyApp::Model::Foo->new(pg => $app->pg, bar => $config->{bar})
21:19 sri it's good abstraction
21:19 maschine thanks - the examples for full Mojo were pretty clear, but I haven't quite got there yet. :)
21:39 jabberwok joined #mojo
21:40 Grinnz jberger: https://www.usatoday.com/story/news/nation/2017/07/05/july-4th-weekend-shootings-chicago/452585001/# :(
21:40 thayner joined #mojo
21:40 jberger a sad reality here in Chicago
21:41 jberger actually shootings track with temperature pretty closely too
21:41 preaction i'd kill for a/c
21:41 jberger just a reminder though, the violence is mostly confined to a few neighborhoods
22:53 arcanez what's the best way to translate a Mojo::Pg::Results to { foo => [ 'bar', 'baz' 'quux' ] }? (my query is "SELECT foo FROM x")
22:53 arcanez some variant of hashes/arrays
22:55 Grinnz as in, an array of the values from each row?
22:55 arcanez yeah
22:55 arcanez right now, I'm returning ->hashes and doing
22:55 arcanez my @results = map { $_->{host} } @$hosts;
22:55 arcanez where $hosts is ->query->hashes
22:56 Grinnz ->hashes->map(sub { $_->{host} }) maybe
22:57 Grinnz that would give you a Mojo::Collection of the host values, which is basically an arrayref
22:57 arcanez that works
22:58 arcanez for most things, I can get away with ->hashes, and at time, ->to_array (just rendering it as json)
22:59 Grinnz i much prefer the ->hashes and Mojo::Collection interface over the few methods DBI gives you
23:00 arcanez does Mojo provide any helpers to pluralize/singularize/
23:01 Grinnz ... words?
23:03 arcanez yes
23:04 Grinnz no... that would require a dictionary
23:04 jabberwok joined #mojo
23:04 Grinnz look in the Lingua::EN:: namespace for things to do this
23:05 Grinnz https://metacpan.org/pod/Lingua::EN::Inflexion
23:05 Grinnz that one looks neat
23:06 arcanez good ole conway
23:11 Grinnz there's also https://metacpan.org/pod/Lingua::EN::Inflect::Number which is much lighter weight, but seems to be not quite correct in some instances
23:12 Grinnz and Inflexion seems to be a reworked version of that anyway
23:26 schelcj joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary