The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-03-14

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:55 hafizh joined #mojo
01:06 aborazmeh joined #mojo
01:20 stryx` joined #mojo
01:27 Xyem Ah, the best kind of bug. The ones that happen "randomly".
01:28 Xyem My log in form sometimes ends up with a request of '/action/:/login', rather than '/action/login'.
01:36 marty joined #mojo
01:57 robinsmidsrod joined #mojo
02:20 marty joined #mojo
02:21 Janos joined #mojo
02:29 Grinnz timing bugs!
02:46 veryrusty joined #mojo
03:19 jberger ok I didn't get quite through it all
03:19 jberger but I'm down to 2 dists on cpan that use Mojo::Util::slurp|spurt
03:19 jberger and I finally deprecated the two dists that I've been meaning to
03:23 jberger http://grep.cpan.me/?q=Mojo%3A%3AUtil.%2A%28slurp%7Cspurt%29
03:23 jberger I have one false-positive in there, the Changes file from a dist that I fix pops up
03:24 jberger (whenever it reindexes cpan at least)
03:39 noganex joined #mojo
04:04 veryrusty joined #mojo
04:11 asarch joined #mojo
04:28 inokenty-w joined #mojo
04:37 veryrusty joined #mojo
05:04 dboehmer joined #mojo
05:13 polettix joined #mojo
05:47 faraco joined #mojo
06:35 veryrusty joined #mojo
06:58 dod joined #mojo
07:00 faraco joined #mojo
07:05 dod joined #mojo
08:19 Xyem Grinnz: It doesn't seem to be a timing error. I've narrowed down when it does/doesn't happen. Hopefully I can resolve it tonight.
08:20 trone joined #mojo
08:21 AndrewIsh joined #mojo
08:21 Xyem Something is adding a spurious ':' to the REQUEST_URI, and I've seen Mojo itself emit it (as a 'Location: : login' header).
08:30 rshadow joined #mojo
08:33 Bloke joined #mojo
08:34 Bloke Hi guys, I'm wondering if there's a way for a plugin to suspend the auto-render, I have a delay in around_dispatch and I'd like to not auto-render until the responses happen, but if I disapble it outright silly things happen
08:41 veryrusty joined #mojo
09:02 veryrusty joined #mojo
09:07 Paddy joined #mojo
09:07 Paddy Hi guys
09:08 Paddy If I wanna run mojo on SSL, can I just add this line to .pl? app->config(hypnotoad => { listen => [ 'https://localhost:1234?cert=/my.crt&key=/my.key&ca=/my_cabundle.pem' ] , heartbeat_timeout => 60, inactivity_timeout => 40, proxy => 1 } );
09:09 sri so, are there any objections to the Test::Mojo and config plugin changes?
09:10 sri i kinda need that at work and want to release it :)
09:11 sri https://github.com/kraih/mojo/compare/v7.28...master
09:24 batman sri: I really like that you can pass arguments to new() now.
09:24 rshadow joined #mojo
09:25 bjakubski so with config_override I will no longer need to load config plugin conditionally in my app (as tests have their own local config)? seems good
09:25 batman I'm not sure about "config_override". how about "disable_config_plugin"? too long?
09:26 sri jberger will object to that
09:26 sri it's not specific to our config plugin
09:26 batman ok.
09:27 batman but passing arguments to new() is so fantastic... been wanting that for so long :)
09:28 sri i don't mind chaning the name of the key
09:28 sri just has to be generic
09:29 sri so nobody feels like "oh, that looks private, i better not use that in my new magical config system"
09:30 batman right.
09:31 batman i'm more concerned about Test::Mojo->new(...). Is it too much typing if it was Test::Mojo->new("MyApp", {config => {....}}); ?
09:32 batman would be nice to override other attributes as well, and not just the config.
09:37 sri i don't know about that
09:38 sri you can just make those conditional based on config settings
09:38 sri what other attributes do you really want to override?
09:40 sri config is the essential building block for bootstrapping a test environment with startup, everything else can basically depend on that system
09:44 batman that's a valid point. i guess startup() could act based on config variables
09:44 batman i was thinking if startup() did something like $app->some_attribute->do_stuff, where you want some_attribute to be a mocked object
09:46 sri i mean, if there's a real need we realize in the future, the API has enough room for additions
09:46 sri only the hashref argument is taken for config overrides
09:47 sri Test::Mojo->new("MyApp", config => {....}, moniker => "lalala");
09:47 sri that will be an option if necessary
09:47 sri but i don't think it will
09:47 batman ok, then it's fine. just didn't see how the api would have room for further extensions
09:50 batman +1 from me then :)
10:27 sri :)
10:29 irqq joined #mojo
10:47 Phil21 joined #mojo
10:48 schelcj joined #mojo
10:50 sri allright, last chance to complain, after lunch i'll release it
10:52 sri (also about the config_override name)
10:56 itaipu joined #mojo
11:11 bobkare I like it! I have so many ugly hacks to override config for tests
11:14 veryrusty joined #mojo
11:33 itaipu_ joined #mojo
11:42 gregf_ joined #mojo
12:09 tchaves joined #mojo
12:49 itaipu joined #mojo
12:51 marty joined #mojo
12:52 stryx` joined #mojo
12:53 marty joined #mojo
13:11 tchaves joined #mojo
13:14 aborazmeh joined #mojo
13:22 sri unique jobs are a really tricky problem, especially now that minion has job dependencies
13:24 sri we just made a job group like job1 -> job2, job2, job2, job2 -> job3
13:24 itaipu joined #mojo
13:24 sri and that whole group is supposed to be unique based on a shared value
13:25 sri (job2 runs parallel to process chunks of data)
13:28 VVelox joined #mojo
13:28 sri so, what i ended up doing is set a job1_lock_$unique_value in memcached/redis in job1 that gets removed in job3
13:28 sri and any new job1 checks for the lock and bails if it is already there
14:06 Janos joined #mojo
14:07 itaipu joined #mojo
14:13 batman Lee: isn't it easier to just watch less files? https://github.com/kraih/mojo/pull/1070
14:17 Lee no
14:17 Lee because then i may as well just stop/start the server manually
14:18 Lee this isn't an issue *for me* but others don't have as powerful dev machines
14:19 tchaves joined #mojo
14:20 sri there would have to be real issues for me to consider making such an ugly implementation detail public
14:22 batman Lee: how many files are "many files" ? 100? 1000? 100000?
14:22 coolo sri: http://paste.opensuse.org/37734268 - it seems job -s is not optimized :)
14:23 Lee batman: it's not a lot in this case, around 600, but enough to sit on the CPU of a vm when the sleep is 1s
14:30 batman Ok
14:31 Lee FWIW i tested on my vm and saw a signigicant CPU drop with 5s rather than 1s
14:31 Lee so having it being configurable with the env var is useful i think
14:31 Lee implementation may be ugly, but hey some are
14:32 sri if it was just the comand line flag and worded more neutral it might have had a chance
14:33 sri but introducing the env var as public set the bar very high for a use case
14:33 gryphon joined #mojo
14:33 sri coolo: it's at least supposed to be reasomably well optimized ;p
14:34 itaipu_ joined #mojo
14:34 sri coolo: if you find indexes to make it better let me know, i'll add them to the minion schema
14:36 Lee sri: you're reading too much into the wording (assuming "neutral language" is a reference to the Motivation section of the PR)
14:37 sri Lee: no, i meant something like "-s <seconds>   Try not to spend more time than this without looking for file changes"
14:37 sri being unspecific about what actually happens
14:38 sri some of us are still hoping to optimize morbo somehow
14:42 tchaves joined #mojo
14:53 Pyritic joined #mojo
15:00 marty joined #mojo
15:02 marty joined #mojo
15:06 jberger so this talk uses Mojolicious as a demo for using Perl on App Engine: https://cloudnext.withgoogle.com/schedule#target=you-can-run-that-on-app-engine-89abaf38-fce4-451c-b0e3-7726013300df
15:06 itaipu joined #mojo
15:06 jberger start at 26:00
15:06 jberger the quote I like "I think its Perl 5"
15:06 jberger sigh
15:07 * Lee has tweaked the PR
15:11 jberger "let's try a more Earth-bound situation"
15:11 jberger argh
15:16 itaipu joined #mojo
15:31 lluad joined #mojo
15:36 sri looks like we have uncovered a minion problem at work
15:37 sri we have like 350k jobs with lots of dependencies... and suddenly ->repair takes over 20 minutes
15:37 sri https://metacpan.org/source/SRI/Minion-6.02/lib/Minion/Backend/Pg.pm#L147
15:38 sri there's already a gin index on it https://metacpan.org/source/SRI/Minion-6.02/lib/Minion/Backend/Pg.pm#L854
15:38 sri but it looks like that's not enough
15:39 sri looks like it might have been updating for like 45 minutes now
15:39 sri :(
15:40 zivester joined #mojo
15:42 sri jberger: i think you came up with that one ;p
15:42 jberger parents or the index?
15:43 sri the update query
15:43 sri the cardinality stuff
15:43 jberger probably
15:44 sri almost looks like it can't use the index
15:44 sri my explain analyze won't finish
15:44 jberger I was just going to ask, wow, that's ... odd
15:45 jberger how about just explain
15:45 sri yea, it does a seq scan
15:46 coolo 49 minutes and counting :(
15:46 sri https://gist.github.com/anonymous/00ff1b7629599607f16efb2021e5c13c
15:46 coolo 300K rows is a lot, but it makes you wonder
15:47 coolo sri: I don't think I've seen a cost 20601590869.34 before :)
15:47 sri 300k subqueries
15:48 sri why is it not using the gin index
15:51 sri wow, not even SET enable_seqscan  = off; makes it use the gin index
15:52 PryMar56 joined #mojo
15:53 coolo is now a good moment to mention that GRU doesn't have this problem? :)
15:54 sri it has other problems though ;p
15:56 itaipu joined #mojo
15:57 coolo sri: so stop the worker, hot patch the repair out of the rpm and start it again - and wait for the job number to go down to < 100?
15:57 sri if you want a quick short term workaround
15:57 * coolo has no faith this will finish
15:58 sri i mean, it will be much harder for me to find a solution if the data vanishes ;p
16:00 coolo you're implying I had faith in your postgresql vodoo? :)
16:00 sri here it gets even more crazy
16:00 sri "explain select count(*) from minion_jobs where '{23}' <@ parents;"
16:00 sri that actually uses the index
16:01 sri the cardinality crap must be tripping it up maybe
16:02 itaipu_ joined #mojo
16:03 khfeng joined #mojo
16:26 gizmomathboy joined #mojo
16:27 sri lol
16:27 sri yea, nobody even realized that the whole query is bullshit
16:28 sri of course it's not optimized
16:29 sri omg is that stupid when you think about it
16:29 sri jberger: WHAT THE HELL WERE WE THINKING!
16:29 pink_mist lol
16:30 * sri double checks just to make sure
16:30 CHYC Just OOI, what did the array give you that a join table didn't?
16:31 sri this is what the query plan should have looked like https://gist.github.com/anonymous/0941a6da15bcb4ef1abecfcbf70654b8
16:31 Janos joined #mojo
16:33 sri coolo: will have a fix soon
16:33 sri i can only guess how the postgres query planner works of course
16:34 sri but we pull in j.parents from the outer scope, there is no point where it even could check the gin index
16:35 sri there is no query actually running on the parents field
16:35 sri it's just a comparison on .parents from the outer query, against an array we make
16:36 sri it makes no sense!
16:36 gryphon joined #mojo
16:36 sri what should have happened is a simple "select count(*) from minion_jobs where id = any (j.parents)"
16:36 sri which can just use the primary key index
16:37 sri coolo: and that's how we ended up with 300k * 300k sequential scans ;p
16:38 sri that i actually added a gin index on the field is ridiculous
16:39 sri OMG
16:39 sri https://github.com/kraih/minion/commit/a42fae8a19439f1095c5b55bcca18f5b638a1ee3#diff-6c2d02af894f2ce916d89f7b04f4b54aL150
16:39 sri i broke it
16:39 sri it was correct before
16:41 * jberger is vindicated ;-P
16:41 sri i prolly benchmarked job_info which got much better with the index
16:41 jberger actually I would have (and still have) no idea
16:41 sri and just shoved the code into all the other methods
16:42 sri without testing
16:42 disputin joined #mojo
16:43 sri the delete might axtually work
16:44 sri i'm going to test very carefully now before i release the fix ;p
16:55 sri ok, that commit looks like total bullshit after benchmarking on real world data
16:55 sri not even job_info is good
17:01 ashimema joined #mojo
17:04 sri https://github.com/kraih/minion/commit/64cde429bd32bfbebe625fba0c580a92e79f6003
17:04 sri basically a rollback of the previous changes
17:05 sri just bad code
17:05 sri no idea what happened that day
17:05 sri coolo: that will make repair finish in under 10 seconds i bet
17:07 sri i'll upload a release later, together with mojolicious.... so others can review and make sure i didn't screw up again :)
17:08 prg btw, is there a way for repair not to clean up old jobs?
17:08 coolo wow, we're still not done? :)
17:08 sri coolo: just kill it
17:08 coolo kill what?
17:08 purl kill is probably send a signal to a process or process group or at language.perl.com/ppt/src/kill/ or Keep It Light, Loser
17:08 sri put it out of it's misery
17:09 sri coolo: you were not referring to your worker still repairing?
17:09 coolo sri: I did. but killing postgresql thread or minion
17:09 jberger minion
17:09 coolo I decided to do the former and that killed the later :)
17:10 jberger err, hmmm, I don't know
17:10 sri ah, minion
17:10 coolo Cannot write to ‘64cde429bd32bfbebe625fba0c580a92e79f6003.patch’ (Success).
17:10 coolo I love errno :)
17:11 coolo sri: I hotpatched the Pg.pm and it now runs jobs
17:12 coolo "enqueued_jobs" => 380889,
17:12 sri yay
17:12 sri definitely one of my better screwups
17:13 coolo to say it with Mrs. Spears: Oh! you shouldn't have!
17:13 robert joined #mojo
17:13 asarch joined #mojo
17:15 coolo (for the younger folks: https://youtu.be/CduA0TULnow?t=185)
17:15 rwh21 joined #mojo
17:25 coolo sri: these self-duplicating jobs are really troublesome - we're up to 424T. we might create skynet at the end of the night
17:25 sri who says that wasn't my plan from the beginning?
17:27 sri https://s-media-cache-ak0.pinimg.com/originals/51/25/82/51258263c15b323b45a3762db8c82382.jpg
17:28 genio that's a lot of minions!
17:34 jberger Tesla is an interesting unit of jobs, also 424 Tesla would be a strong enough magnet to tear your building apart, so you might want to take cover
17:35 jberger </science-joke>
17:36 genio dang physicists
17:41 sri that's going on my bucket list... tear apart a building with a giant magnet
17:41 preaction bulldozer would be easier
17:50 jberger sri: you are a robot super-villain after all
17:56 vicash jberger: in Breaking Bad, Walter white destroys evidence with a giant set of magnets in a truck
17:58 genio I don't remember that episode
18:05 tyldis Oh, action. Seems I'm not the only one needing a whiskey tonight.
18:10 vicash genio: http://breakingbad.wikia.com/wiki/Live_Free_or_Die
18:20 rshadow joined #mojo
18:20 kes joined #mojo
18:23 coolo sri: watch X-Men (the first) to know how it's done :)
18:49 tchaves joined #mojo
18:49 pink_mist genio: https://www.youtube.com/watch?v=gzCXowhks80
19:19 itaipu joined #mojo
19:41 irqq joined #mojo
20:03 itaipu_ joined #mojo
20:48 irqq_ joined #mojo
20:53 stryx` joined #mojo
21:02 genio oh yea.  I remember that now
21:32 polettix joined #mojo
21:32 cfedde joined #mojo
21:36 Bloke Would a local $c->stash('mojo.rendered' => 1) be enough to resume the auto-render behaviour after I drop into a delay?
21:38 * Bloke would appreciate a nudge toward a plugin that asyncs in an around_dispatch hook without breaking ->render_later
21:41 jberger Bloke: the 'mojo.' stash variables are private
21:48 Bloke there's an accessor though
21:48 Bloke so, like ...
21:49 Bloke that variable is for controlling the auto-render behaviour, and I don't want to interrupt it just because my plugin is a jerk
21:49 jberger http://mojolicious.org/perldoc/Mojolicious/Controller#stash
21:50 Bloke I do understand the idea of a stash.
21:50 jberger that's where it is documented as private
21:50 Bloke is it the "Note that all stash values with a mojo.* prefix are reserved for internal use."?
21:50 jberger yes
21:50 Bloke "sub render_later { shift->stash('mojo.rendered' => 1) } …"
21:51 Bloke where it's documented as "the thing that fixes this guys issue"
21:51 jberger I'm not sure what you are trying to do, and I can't really dig in now, I need to leave for something, but we wouldn't bless that as a solution
21:51 jberger that's already what render_later does
21:52 Bloke is there a way to un-do that thoughJ?
21:53 jberger even if you could it wouldn't help
21:53 jberger once autorendering is checked that's it, it isn't checked again
21:53 Bloke package ...::Plugin::AutoFetchStuff; sub register {   $app->hook(around_action => sub {  ... ->ua->get( ... sub { $c->stash( ... $tx->res->json ) ; $next->() } ) } ) }
21:54 Bloke Aah.
21:54 Bloke then I should ->render after my next?
22:42 asarch joined #mojo
23:03 asarch joined #mojo
23:12 veryrusty joined #mojo
23:12 ashimema joined #mojo
23:17 jberger if I recall the around_action hook correctly, the autorender will come during the call to $next->()
23:18 jberger so if that's the case you shouldn't need to deal with
23:19 jberger ... deal with autorendering
23:26 good_news_everyon joined #mojo
23:26 good_news_everyon [mojo] kraih tagged v7.29 at cfd9aa1: https://git.io/vyMYa
23:26 good_news_everyon left #mojo
23:28 good_news_everyon joined #mojo
23:28 good_news_everyon [mojo] kraih pushed 1 new commit to master: https://git.io/vyMYM
23:28 good_news_everyon mojo/master c937f06 Sebastian Riedel: bump version
23:28 good_news_everyon left #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary