The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-10-11

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:18 rba joined #mojo
00:28 itaipu joined #mojo
00:32 rba_ joined #mojo
02:36 noganex_ joined #mojo
02:47 ilbot2 joined #mojo
02:47 Topic for #mojo is now 🍩 nom nom | http://mojolicious.org | http://irclog.mojolicious.org | http://code-of-conduct.mojolicious.org
02:47 dotan_convos joined #mojo
02:49 jabberwok joined #mojo
02:49 anparker joined #mojo
02:49 haarg joined #mojo
02:49 meredith joined #mojo
04:04 dboehmer_ joined #mojo
04:23 itaipu_ joined #mojo
05:21 inokenty-w joined #mojo
05:44 mkrull joined #mojo
05:55 dod joined #mojo
05:59 dod joined #mojo
06:05 dod joined #mojo
06:20 mkrull joined #mojo
06:28 mkrull joined #mojo
06:46 Vandal joined #mojo
07:07 aborazmeh joined #mojo
07:13 AndrewIsh joined #mojo
07:18 karjala_ joined #mojo
07:19 petru joined #mojo
07:28 trone joined #mojo
07:39 petru joined #mojo
09:27 petru joined #mojo
09:40 irqq joined #mojo
09:47 rshadow joined #mojo
09:49 mr_boo joined #mojo
10:03 rba joined #mojo
10:30 leont_ joined #mojo
10:32 tchaves joined #mojo
10:36 marcus I also liked seveneves
10:37 marcus sri: Silo triology is good but very dark. Not similar to RPO imo
10:37 dustinm joined #mojo
10:46 leont_ left #mojo
11:12 margeas joined #mojo
11:15 margeas hi! I'm trying to find something inside mojo user agent which map this functionality https://metacpan.org/pod/LWP::UserAgent#credentials, any hints ?
11:16 pink_mist http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Basic-authentication
11:35 salva0 joined #mojo
13:02 ChmEarl joined #mojo
13:12 tcohen joined #mojo
13:25 gizmomathboy joined #mojo
13:43 rba joined #mojo
13:51 sri genio: btw. i watched the gifted yesterday, so far it's pretty good
13:52 sri marcus: i like dark sci-fi
13:52 margeas pink_mist: thanks!
13:53 sri genio: they definitely have some world building to do and the plot is still a bit weak, but fun to watch so far
13:55 margeas the problem is that I need to transact a lot of singular URL endpoints which need basic HTTP auth...the LWP approach is convenient as it does so automatically...
13:58 margeas maybe I can work this around by using Mojo::Headers directly?
14:00 margeas I mean something to force auth headers on all the requests...: $ua->on(start => sub {$tx->req->headers->header()}) ?
14:00 rshadow joined #mojo
14:01 sri $tx->req->url can be modified too
14:01 sri $tx->req->url->userinfo('foo:bar')
14:01 margeas perfect! nice!
14:18 petru joined #mojo
14:23 genio sri: I got half-way through the first episode last night (had to quit and help Mrs. genio with some things). I want to restart it and watch it all the way through tonight.  I didn't hate what I saw so far, though.
14:23 genio I knew in 5 minutes that Inhumans was going to be terrible. I didn't get that same feeling with The Gifted
14:32 maschine joined #mojo
14:56 kes joined #mojo
15:02 mrEriksson Hi folks! I'm trying to figure out a way to get an event in the Worker when a job is finished. I haven't been able to find a way to do this, am I missing something or can't it be done?
15:02 mrEriksson With Minion that is, of course
15:04 CandyAngel mrEriksson: Do you mean the "finished" or "failed" events?
15:04 gryphon joined #mojo
15:04 CandyAngel http://mojolicious.org/perldoc/Minion/Job#finished
15:11 mrEriksson CandyAngel: Kind of, but from what I understand, those are emitted in the forked child process, not the parent
15:11 mrEriksson Or am I wrong? If so, then I'm all good with finished and failed events :)
15:14 jberger mrEriksson: you might look at Minion::Notifier, it uses a message bus backend to get notifications from events in other processes
15:15 CandyAngel Oh, you said "get an event in the Worker", which would be the forked process!
15:17 jberger CandyAngel: good point, mrEriksson I'm not sure why you would need an event in the worker
15:17 CandyAngel You could always make Minion do the notifying too, for successful jobs. Just make a notify task and set the actual task as its parent
15:18 mrEriksson jberger: That seems to be exacly what I'm looking for actually, and I understand you have some experience from this plugin too :-)
15:18 jberger I do indeed
15:20 mrEriksson CandyAngel: Last I talked to the people here, I referred to the process that performed the job as a 'worker', but understood things as the 'worker' being the top process
15:21 mrEriksson CandyAngel: But I want it to be executed in the parent process, not in a forked child
15:21 jberger mrEriksson: yes but how would you subscribe to an event in that process?
15:22 jberger you have no control over that process
15:23 mrEriksson jberger: Well, my tasks tie up resources in the host they are running on (physical devices), and the availability of those varies, so I would like to do an assesment of the amount of resources available after each job has finished and change the number of concurrent jobs based on that. Does that make sense?
15:23 sh14 joined #mojo
15:23 mrEriksson jberger: Yeah, that is kind of what I expected, but thought I would ask anyways. And Minion::Notifier seems to be a solution to this?
15:23 jberger I've never used it that way
15:24 jberger I've used it from a web server context to notify web clients (usually over websocket) that the job is complete
15:24 jberger the minion worker process (apart from your own task) is intended to be a "black box"
15:25 mrEriksson I understand
15:25 CandyAngel Maybe using the locks might be of use?
15:25 karjala_ joined #mojo
15:26 CandyAngel And have a recurring watchdog task which changes the lock limit somewhere?
15:26 mrEriksson Not really
15:26 jberger must the reassessment occur after each job?
15:26 mrEriksson Locks are used to lock resources and make sure multiple jobs doesn't use it at the same time (which works great)
15:27 CandyAngel It can be used like that, but it isn't the only way
15:27 mrEriksson But I want to avoid situations where a task is executed on a host with no resources available
15:28 mrEriksson jberger: Well,, Must is a strong word. You're thinking something like a recurring event in ioloop?
15:28 CandyAngel You can pause workers
15:28 mrEriksson jberger: Wait, no, it has to be done after each job is executed and before the worker starts to look for new jobs to do
15:29 mrEriksson CandyAngel: Pause workers how?
15:29 TheAthlete joined #mojo
15:29 jberger mrEriksson: I'm not sure a job queue is really what you need here
15:29 CandyAngel You set the number of jobs a worker will do to 0
15:30 jberger checking the availability of resources between each run is really micromanaging of the queue IMO
15:30 jberger that's not really its strong suit
15:30 mrEriksson CandyAngel: Ok, go on, what do I do after that?
15:31 CandyAngel Se it back to 1 (or whatever) when you want it to start work again?
15:31 CandyAngel Set it*
15:31 jberger https://metacpan.org/pod/Minion::Command::minion::worker#jobs
15:32 jberger if you don't need it to be between each job, you could just cron a script to check your resources and use that call to change the number of concurrent jobs
15:32 mrEriksson jberger: Perhaps, but what it does right now works really good, but now I have to add support for jobs that consumes multiple resources, not just one.
15:32 mrEriksson jberger: Yes, looked in to that too, but then there might be a race condition
15:32 jberger well that sounds like locks
15:32 jberger as CandyAngel says
15:34 mrEriksson CandyAngel: That is kind of what I want ot do, but then I want to know when a job has terminated so that I can update the number of paralell jobs
15:34 CandyAngel I actually use both in a similar-ish situation
15:34 CandyAngel My forum crawler gets throttled if it requests too fast, so when the response takes a while, it will pause the workers for a while
15:35 mrEriksson jberger: Locks is what I use today, but the problem is that if all resources are locked already, and the worker still accepts another job, that job will sit there and idle for hours, waiting for it's turn, while there might be free resources on another host
15:35 CandyAngel And I use locks to throttle the request rate
15:35 CandyAngel mrEriksson: Sounds like you need to check the resources before the job starts
15:35 CandyAngel And then just make it retry if there aren't any
15:35 CandyAngel I mean
15:36 CandyAngel When the job starts
15:36 mrEriksson CandyAngel: Yes, that would be the best
15:36 jberger CandyAngel++
15:36 CandyAngel And you can even pause the worker if it does, so the retry won't happen on the same host
15:36 mrEriksson Best would be to do this after the job has been dequeued but before it is performed, so that it can be pushed back if there are no resources
15:37 mrEriksson But I haven't found a way to do this
15:37 mrEriksson Other than to actually fail the job and have the queue do a retry
15:37 CandyAngel Mhm, that's what I mean
15:38 mrEriksson But then I will have failed jobs, that aren't really failed
15:38 CandyAngel You could reschedule the job with the same arguments
15:38 mrEriksson Create a new job you mean?
15:38 CandyAngel Yeah, and finish the original (maybe with a note saying it was rescheduled)
15:38 mrEriksson Then I would loose my jobid
15:39 CandyAngel Add it as a parent to the new job
15:39 mrEriksson Which breaks everything sadly
15:40 mrEriksson If I fail a job, could I perhaps change the state to 'inactive' afterwards? That way, I wouldn't have failed jobs that aren't failed
15:40 CandyAngel $app->enqueue(taskname, parents => [$original_id])
15:41 mrEriksson CandyAngel: The process that enqueued the task monitores it's progress over time using the jobid
15:41 CandyAngel Oh
15:41 CandyAngel Just call "retry" on it then
15:42 CandyAngel http://mojolicious.org/perldoc/Minion/Job#retry
15:42 CandyAngel That makes it inactive, but not failed
15:43 CandyAngel You could even make "resource-retry" ones a higher priority so they will be chosen byanother worker first
15:43 mrEriksson Oh, that might actually work, hadn't seen that one before. I could just retry the job and then update the resources available. Might actually work.
15:43 CandyAngel $job->retry(priority => $job->info->{priority}++);
15:44 mrEriksson But if it is first in line and then I retry it, wouldn't it be very close to first in line afterwards anyways?
15:45 CandyAngel I'd imagine so, but I'm not sure
15:45 mrEriksson I'll give it a try
15:45 mrEriksson Thank's guys for letting me bounce ideas, this was very helpful!
15:46 CandyAngel Happy to help! ^_^
15:48 mrEriksson Only sad part now is that I'm extremely excited to try this out, but I won't have time for it before Friday :-)
15:48 CandyAngel Can totally relate to that :)
16:25 disputin joined #mojo
16:29 sri mrEriksson: i'm open to adding a reap event to Minion::Job, as the opposite of the spawn event
16:32 sri (even though locks+retry are a good solution for your specific problem)
16:33 sri btw. i'm really liking the vim extension for vscode
16:34 sri much higher wuality than the vim extension for atom
16:34 sri s/w/q/
16:35 rba_ joined #mojo
16:39 cheako joined #mojo
16:40 cheako Hello, how can I force mojo to return an HTTP/1.0 response?
16:41 cheako What's the best way to pick port numbers on Debian?
16:41 cheako I need 7 or 8
16:46 rba joined #mojo
16:47 cheako I've gtg.
16:49 mrEriksson sri: Would that be an event that would be triggered in the parent process?
16:49 sri yes, just like spawn
16:50 bwf cheako: http://mojolicious.org/perldoc/Mojo/Message#version
16:57 mrEriksson sri: Sounds good, but, question, how would I be able to subscribe to spawn or reap on a Job? I mean, looking at the source code, it seems that it is dequeued and then performed right after, where would I be able to do ->on(spawn) on that object?
16:57 sri from the dequeue event of Minion::Worker
16:59 sri and you can access that from the worker event of Minion :)
17:00 sri app->minion->on(worker => sub { pop->on(dequeue => sub { pop->on(reap => sub { ... }) }) })
17:05 sri adding the event is a fairly simple patch to Minion::Job, needs to be emitted by perform and is_finished, just testing requires a little creativity
17:08 rshadow joined #mojo
17:17 dod joined #mojo
17:25 trone joined #mojo
17:38 mrEriksson sri: Just of curiosity, since you fork child processes to do the work, how will you be able to emit event in parent when child is done?
17:59 cheako joined #mojo
18:08 karjala_ joined #mojo
18:22 irqq joined #mojo
19:23 karjala_ joined #mojo
19:26 sri mrEriksson: normal unix process management with ipc
19:29 CandyAngel That reminds me, I was going to see about making a forkless worker thing..
19:32 cheako joined #mojo
19:55 gryphon joined #mojo
20:19 Janos joined #mojo
20:28 maschine joined #mojo
20:41 jnbek joined #mojo
21:15 gryphon joined #mojo
21:20 sri mrEriksson: added https://github.com/kraih/minion/commit/d91c8cfdfc1e077425de0ffc042980dcf3b24164
21:20 sri since i can think of quite a few use cases
21:32 gizmomathboy joined #mojo
21:37 Grinnz "after the process performing it has died." also when it exits normally, right?
21:59 sri same wording as Mojo::Server::Prefork
22:10 dod joined #mojo
22:19 cheako joined #mojo
22:45 mattp joined #mojo
22:48 mattp joined #mojo
23:09 dod joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary