The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-06-22

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 Lee_ joined #mojo
00:04 Lee_ left #mojo
00:04 tchaves joined #mojo
00:37 Lee_ joined #mojo
00:38 Lee_ left #mojo
01:04 Lee_ joined #mojo
01:09 Lee_ left #mojo
01:16 aborazmeh joined #mojo
01:28 PryMar56 joined #mojo
01:55 brunoramos joined #mojo
02:24 noganex joined #mojo
02:58 mcsnolte joined #mojo
03:24 abra_ joined #mojo
03:27 stryx` joined #mojo
03:27 vicash I am enqueuing over 100K jobs in Minion for about 28-32 workers. However, it seems that Minion::Worker forks for every job, despite each job being independent. Can this be avoided ? Can't the Minion::Worker just get a job, execute it and get the next job rather than forking all the time. Forking is expensive !
03:34 Grinnz it's not really, but preforking is cheaper when you're talking thousands a second
03:35 Grinnz i dunno if minion is really built for that kind of rate
03:57 preaction forking is expensive on which platform?
03:59 vicash preaction: for jobs that take about 0.1-0.2s and having to fork for each is a bit much, don't you think ?
04:00 preaction it depends on what the forking is meant to achieve
04:00 preaction i mean, i'm shortly going to finish a project that will do something similar, so i'm sure i'll have a different opinion then
04:02 vicash if each worker can only process 1 self-contained job at a time, and if each worker is allocated N jobs, then N jobs processed one after the other without forking N times will be faster than N forks and 1 job processed by each fork.
04:03 preaction there's way too much wiggle room in there for that to be in any way universally-applicable
04:04 preaction also, workers can process N jobs at a time
04:07 pink_mist vicash: forking is extremely cheap .. are you actually encountering issues because of forking? 0_o
04:08 vicash pink_mist: not encountering issues. but i kinda expected that if I gave Minion::Worker  a "-j 10" option, it would have 10 workers running continuously processing jobs one after the other. however, what i noticed over 24 hours was that the PID kept changing on all the workers every second.
04:09 preaction it has up to 10 forked children running continuously
04:09 pink_mist vicash: well as I said, forking is extremely cheap, so if you're not facing issues because of it, don't worry about it
04:10 preaction if you are having issues, show the profiler output
04:10 pink_mist imagining problems where there are none helps nobody
04:10 vicash ok
04:32 stryx` joined #mojo
05:51 ashimema joined #mojo
05:56 inokenty-w joined #mojo
05:59 disputin joined #mojo
06:44 AndrewIsh joined #mojo
07:19 trone joined #mojo
07:23 prg joined #mojo
07:25 karjala_ joined #mojo
07:25 karjala_ I forgot how to get the homedir of a Mojolicious app.
07:25 karjala_ it's not $app->home
07:25 sri vicash: Minion has the perfect design for what it does
07:25 karjala_ anyone?
07:25 purl Somewhere, someplace, in some universe, somebody uses whatever you just asked about.
07:26 sri karjala_: yes it is
07:26 karjala_ o, sorry!
07:26 sri vicash: but the API is rich enough to build whatever kind of worker you want
07:26 karjala_ just couldn't see it in Mojolicious.pm pod
07:27 sri you could even make your own kind of worker and run it side by side with normal workers, and then just have each watch different named queues
07:28 sri but yea, fork is dirt cheap, if that's a bottleneck for you, what the hell are you even using background jobs for tasks that are so fast? Oo
07:51 karjala_ joined #mojo
07:53 ysg joined #mojo
07:57 karjala_ batman, hi
07:58 karjala_ batman, I have a problem with Mojolicious::Plugin::CGI. Even though I've set plenv local 5.16.3 in my cgi-bin directory, when the scripts there get executed, the perl version is 5.26.0 (just like the morbo that is running)
07:59 karjala_ When I execute the cgi-bin scripts manually, from the shell prompt, though, then $^V == 5.16.3
07:59 karjala_ Why doesn't 5.16.3 get picked up from Mojolicious::Plugin::CGI?
08:03 karjala_ oh, it's ok: $ENV{PLENV_VERSION} got inherited in the script. I can easily fix that in Plugin's configuration!
08:03 karjala_ thanks
08:20 karjala_ hm... it's not as simple to set up the perl version as it sounds
08:20 karjala_ You see the old CGI website ran on an older perl, and I'd like to preserve that. The new pages should run on Mojolicious with a new perl
08:20 karjala_ So I'm trying to do that setup
08:30 batman karjala_: you can also pass in your own "env" hash if you want to set a different path or something
08:31 batman example: `my %env = {%ENV}; delete $env{PLENV_VERSION}; plugin CGI => {env => \%env};`
08:32 batman or just change the hashbang in the cgi script
08:39 karjala_ I tried it, but it's not enough. Many ENV vars have already changed, for example PATH=/home/karjala/.plenv/bin/something/5.26.0/
08:39 karjala_ I would need to change PATH also, to make sure the right perl is executed
08:39 karjala_ it's a difficult problem
08:40 karjala_ I'm hoping for a cleaner solution
08:41 karjala_ Executing morbo under plenv did all that
08:41 nic vicash: It sounds like you would benefit from changing the granularity of your tasks
08:41 karjala_ If I could somehow execute plenv shell 5.16.3, right before executing the cgi script...
08:41 nic If you do plan changes, it would be great if you can capture metrics pre- and post-
08:42 karjala_ Maybe: system("source plenv shell 5.16.3") will do?
08:42 karjala_ in the before hook
08:42 nic karjala_ Is it possible plenv is getting in the way?  Have you considered using build-perl instead?
08:43 karjala_ I haven't. I might however if I fail to do the job with plenv.
08:44 nic the plenv setup you're mentioning can be achieved manually of course, as the Caped Crusader explained above
08:47 karjala_ It would require modification of a number of env vars, including PATH
08:47 karjala_ How do I execute source from inside system("source") ?
08:47 karjala_ system(...) ?
08:49 rshadow joined #mojo
08:49 karjala_ ok
08:55 karjala_ plenv shell is returning me an error
08:55 karjala_ that the shell command (which is documented) does not exist
08:55 karjala_ I'll complain higheruip
08:58 karjala_ oh ok it was my fault (documented also) thanks
08:58 karjala_ :-)
09:49 sri weather sucks
09:49 purl Something failed in connecting to the NOAA web server. Try again later.
09:50 sri first a heat wave, and now storm warnings here :S
09:51 geheimnis` joined #mojo
09:57 rshadow joined #mojo
10:10 dod joined #mojo
10:22 nic sri: There is an invocation scenario not currently tested in Minion.
10:22 nic i'm wondering if you're interested in a PR similar to https://github.com/niczero/minion-backend-storable/blob/master/test/40-commands.t
10:23 nic It turns out that "myapp.pl minion job" will invoke list_jobs with params set as undef
10:24 nic one of my backends wasn't handling that, hence the addition of the test
10:24 karjala_ Success!! batman, nic: https://pastebin.com/FjFgHFNJ
10:24 karjala_ this is how i succeeded
10:24 nic cool
10:28 nic sri: Also https://github.com/niczero/minion-backend-storable/blob/master/test/10-file.t#L650-L665
10:29 nic ...for checking what happens when repairing a job that has a missing parent
10:35 sri nic: i would want those tests to be clean like the mojo core command tests
10:35 sri also, your handling of missing parents doesn't match minion 7
10:36 nic heh, yes, I forgot that would be an issue
10:36 nic right, I'll save the ideas for after I've got my head around v7
10:37 nic (this weekend)
10:38 * sri hopes Minion::Backend::SQLite will get updated for Minion 7
10:49 ashimema can anyone point me to the doc around how mojo add's default headers?
10:49 ashimema I'm getting a bit confused as to where a 'Last-Modified' header is coming from
10:50 sri not documented i think
10:50 ashimema ah.. ok..
10:51 ashimema any hints on where to start looking?
10:51 ashimema code wise?
10:51 sri static
10:51 ashimema thanks
10:53 ashimema next question then would be how does mojo identify something as static?
10:53 * ashimema throws together a paste of his controller
11:00 nic ashimema: If it can't be found in dynamic, look in static, and if not there not_found
11:00 ashimema https://gist.github.com/mrenvoize/8dc2e925f0c6cc1811dbd37501309887
11:01 ashimema tried to create a minimal(ish) gist of the relevant route in my app
11:01 ashimema sorry.. just read nic's comment..
11:02 ashimema ok.. so I don't fully understand dynamic vs static then I feel
11:03 ashimema I bet I'm doing something really obviously wrong to those in the know
11:05 nic ashimema: I missed where you've described what the problem is
11:06 ashimema ok.. good point.. looks like I asked what I thought the answer was.. not the actual question..
11:06 ashimema in short, I have a route with dynamic content (reasonably rarely changing, but changing none the less)..
11:07 ashimema but right now every time I hit it, the same content is coming back
11:07 ashimema once I started digging, it looked like it was getting cached somewhere..
11:07 ashimema I've traced it to the controller action never getting called (by sticking some warnings into the action code)
11:08 ashimema then i spotted that the headers has some pretty ancient 'Last-Modified' dates (along with other headers I'd not added myself) in the response.
11:08 ashimema so.. the question is 2 fold..
11:08 ashimema why am I not hitting the action code in the controller
11:09 nic Are you allowing the content to be cached for up to 1 day?
11:09 ashimema and where did the 'Last-Modified' come from.. as it's way older than the most recent change to the content of the response
11:09 ashimema https://rl-dev-1.rebus.ptfsadmin.uk0.bigv.io/app/layout/csl-templates/templates/book.html is the resource in question if it's remotely helpful
11:10 * nic is a sucker for click-bate
11:10 nic s/bate/bait/
11:10 ashimema yes, I was allowing it to be cached for up to a day (for logged in users, if I understand 'private' correctly)
11:11 nic erm, no
11:11 ashimema haha.. yeah.. sorry for the clickbait.. it's an angular template.. nothing especialyl clever looking ;)
11:11 nic 'private' is whether public caches are allowed to hold onto it -- it doesn't affect the browser of the end user
11:12 ashimema mm.. re-reading that.. yeah
11:12 sri ETag: "023404c9717436329da0a12d662c0492-gzip"
11:12 sri mojolicious can't generate that
11:12 ashimema my bad.. been ages since I added that line
11:12 ashimema and everything worked until recently (and still works on all but my dev server)
11:13 ashimema hmm.. I have no idea where the ETag is coming from either.. I presumed the same place as the Last-Modified
11:13 * ashimema is well and truly confused
11:14 ashimema this is sitting behind apache
11:16 * ashimema tries to remove apache from the equation to rule it out
11:21 ashimema ok, removing apache.. I still get the same issues..
11:22 ashimema even still get an ETag.. though it's lost the `-gzip` postfix?
11:23 nic how about building from the bottom up?  Start with a lite app running with daemon or morbo
11:28 sri umm... there was an apache in front of it and you didn't even mention that?
11:28 sri argh
11:28 ashimema sorry.. I forgot there was an apache in front :(.. though I was hitting the morbo gateway directly..
11:28 ashimema but removing it doesn't seem to make much difference at all
11:29 rshadow joined #mojo
11:29 ashimema though it does make the request quicker to respond
11:30 ashimema but the Last-Modified tag remains.. as does the ETag but with -gzip removed
11:32 Sebbe joined #mojo
11:58 ashimema so, am I right in thinking that the router will skip the controller if it thinks it already has an up to date 'static' response somewhere?
11:59 * ashimema wonders how that persists across app restarts
12:29 rshadow joined #mojo
12:36 tchaves joined #mojo
12:53 trippeh joined #mojo
12:55 ashimema yup.. still baffled :(
13:10 maschine joined #mojo
13:26 ashimema seems it's they way I'm routing
13:27 * ashimema breaks out the router documentation again
13:29 Pyritic joined #mojo
13:48 tchaves joined #mojo
13:55 mcsnolte joined #mojo
14:19 Pyritic joined #mojo
14:26 bwf joined #mojo
14:31 Dandre1 joined #mojo
14:37 Intern42 joined #mojo
14:46 disputin joined #mojo
15:02 tchaves joined #mojo
15:08 coolo joined #mojo
15:13 tchaves joined #mojo
15:39 PopeF0 joined #mojo
15:39 depesz joined #mojo
15:41 depesz hi. most likely trivial question, but how can I get to request data from template? I'd like to reach req from within template?
15:41 depesz I know I can copy some data in controller to stash, or whatever, but it would help me if I could get to $req directly
15:47 sri https://github.com/kraih/mojo/blob/master/lib/Mojolicious/resources/templates/mojo/debug.html.ep#L235
15:49 sri depesz: while you're here, we would very much appreciate a review of the postgres minion backend, and perhaps some suggestions for optimizing the queries :) https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm
15:50 depesz sri: thanks a lot. as for review - i can definitely do it, though not at the moment. will add myself a reminder to do it, and will most likely get to it tomorrow.
15:51 sri \o/
15:52 sri our biggest bottleneck btw. https://github.com/kraih/minion/blob/master/lib/Minion/Backend/Pg.pm#L185-L196
15:53 CW joined #mojo
16:01 depesz sri: any chance you have some copy of db where this shows problems?
16:01 depesz as in: dump of db that I could play agains with these queries
16:01 depesz I don't use minion myself (maybe i should, but I haven't yet), and playing with pure sql would be easier for me, but i need some realistic-ish data to be able to optimize it sensibly
16:01 sri we don't actually have problems, just looking for ways to optimize :)
16:02 sri i like to use this script and change the enqueue line to a million https://github.com/kraih/minion/blob/master/examples/minion_bench.pl#L6
16:02 sri it leaves behind a good sample
16:02 depesz cool. will play with that
16:08 stryx` joined #mojo
16:17 PryMar56 joined #mojo
16:22 Dandre1 left #mojo
16:59 dod joined #mojo
17:06 * sri preorders a DLC for the first time :o
17:07 sri doesn't even have a release date yet, but i don't care!
17:07 sri HZD is just that good
17:11 zen it sure is
17:11 depesz HZD == ?
17:12 haarg horizon zero dawn
17:12 purl i heard horizon zero dawn was one of the best games i've ever played :o
17:13 sri great original story and hunting robot dinosaurs with bow and arrow :D
17:14 sri which is actually as challenging as it sounds
17:19 zen if you dont shoot aimed at certain parts it's going to be tough :)
17:20 sri zen: even then... those Rockbreakers were tough!
17:20 zen true, I didnt like the glinthawks
17:21 zen *try to aim at certain part*
17:21 zen bird flies a little to the lef
17:22 sri the flying robots didn't bother me much once i had a rope caster
17:22 zen damn, totally forgot about that :)
17:22 sri lol
17:23 zen did you know once you follow a path on a mount you dont have to steer?
17:24 zen just tap X every now and then, the mount will continue following the path
17:24 sri but yea, i love that every new type of robot you encounter is a new puzzle to figure out the best strategy to kill them
17:24 sri zen: Oo
17:24 sri i had no idea
17:39 zivester joined #mojo
18:10 rshadow joined #mojo
18:17 tchaves joined #mojo
18:26 tchaves joined #mojo
18:31 dod joined #mojo
18:39 trone joined #mojo
18:39 Intern42 joined #mojo
19:31 tchaves joined #mojo
20:49 zapwai joined #mojo
21:38 cheako joined #mojo
22:27 tchaves joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary