The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-07-06

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 gryphon joined #mojo
01:17 dabudabu joined #mojo
01:41 aborazmeh joined #mojo
02:16 noganex_ joined #mojo
02:26 karjala_ joined #mojo
02:41 schelcj joined #mojo
04:40 zivester joined #mojo
05:08 inokenty-w joined #mojo
06:41 AndrewIsh joined #mojo
07:00 dod joined #mojo
07:08 dod joined #mojo
07:24 Vandal joined #mojo
07:44 karjala_ joined #mojo
08:13 CandyAngel sri: I'm glad the solution was a simple one :)
08:16 trone joined #mojo
08:18 CHYC arcanez: 'query("select array_agg(foo) as foo from x")->hash' will be quite a bit faster than doing it in perl
08:26 karjala_ CandyAngel, did adding 2 lines in Minion really solve the problem?
08:26 karjala_ $SIG{CHLD} = sub {}; was all that was needed?
08:27 CandyAngel Seems so, it works much better now when I tested it. Don't know if it leaves zombies around though..
08:28 sri technically, it didn't even require adding a single line
08:29 CandyAngel I went to bed as soon as I got in from work last night because of illness/lack of sleep, which is why I hadn't looked at it myself >.<
08:29 sri https://github.com/kraih/minion/compare/7511ce57bb2bd50499bb5d670f09ca494654a580...b39eaa9db4f165c3dfe81db6ea61d4306d1a4e92
08:30 sri minion 7.02 is already on cpan, try it
08:30 karjala_ well done
08:30 karjala_ (i like my steak well done)
08:38 prg joined #mojo
08:44 CandyAngel Well, I can't get it to leave any zombie processes behind, so it seems to work fine :)
09:04 rshadow joined #mojo
09:23 sri SIGCHLD does not prevent waitpid
11:36 tchaves joined #mojo
12:00 jabberwok joined #mojo
12:32 vicash joined #mojo
13:16 CandyAngel I didn't realise you actually called waitpid anywhere. I thought it was set to default so they got autoreaped
13:17 CandyAngel I don't use signals a lot :P
13:19 CandyAngel I can't use it at work anyway, which is super annoying because it would *really* nice to do some things through Minion
13:19 CandyAngel Because of getting stats and stuff
13:19 sri i don't think there is such a thing as autoreap
13:20 sri SIGCHLD default is to just ignore the signal
13:21 CandyAngel "As for reaping, children of a parent whose SIGCHLD handler is explicitely set to IGNORE will be reaped automatically by the system."
13:21 CandyAngel "$SIG{CHLD} = 'DEFAULT'; causes your process to treat SIGCHLD signals as it would had you not messed with $SIG{CHLD} or equivalent. According to kill(1), a process on my system ignores SIGCHLD by default"
13:22 ribasushi sri: CandyAngel is correct, `perldoc perlipc` has more verbiage to same effect
13:22 CandyAngel I took that as meaning DEFAULT == IGNORE == autoreap
13:22 ribasushi it's a posix-ey thing
13:22 sri now i'm curious how that doesn't mess up waitpid
13:23 ribasushi CandyAngel: default != ignore
13:24 ribasushi it's a subtle difference: from perls pov they are the same, from libc's pov they are not
13:24 CandyAngel Ah
13:24 ribasushi ( if I remember my stuff right, it's been ~4 years since I dug there )
13:24 sri ah, and IGNORE does mess up waitpid
13:24 CandyAngel Well, this is why I made a PR with a solution I knew would work, rather than the best one where I didn't know what I was doing :)
13:24 ribasushi sri: well... it results in -1
13:24 sri that makes sense
13:40 Pyritic joined #mojo
13:53 MojoBeginner joined #mojo
13:56 gryphon joined #mojo
13:57 MojoBeginner joined #mojo
13:57 tchaves joined #mojo
14:07 tchaves joined #mojo
14:26 sri hmm, i remember someone complaining about not being able to use minion because fork was too slow, wonder if that was just the sleep 1
14:29 CandyAngel That sounds possible..
14:59 mpapec joined #mojo
15:07 ashimema possibly a daft question..
15:07 ashimema I'm slowly migrating my app towards Mojo::Pg from DBIx::Class..
15:08 ashimema any tips/hints on how I might share the connection so I'm not connecting to the db twice?
15:08 ashimema (or rather, twice per process?)
15:09 zivester joined #mojo
15:09 preaction Mojo::Pg connects lots of times, so i hope your database lets you connect more than twice
15:09 ashimema haha, indeed it does
15:10 ashimema but nto an infinite number of times ;)
15:10 ashimema and doubling connections in one go felt 'nasty'
15:10 preaction Mojo::Pg does not connect an infinite number of times
15:10 sri preaction: for most uses that's not actually the case
15:10 ashimema i suppose I didn't mean 'share connection' but more 'share connection management'
15:10 sri if it can it will reuse the same connection for as long as possible
15:10 ashimema if that's a 'think'
15:11 preaction you can pass, to DBIx::Class, a built $dbh, but i suspect that will behave strangely if shared with Mojo::Pg and used to do async stuff
15:11 ashimema which is also what dbic does unless I'm misunderstanding
15:11 purl okay, ashimema.
15:12 ashimema wonder if ribasushi might have any hints/clues on this crazy question ;)
15:12 Lee joined #mojo
15:13 sri reuse $pg->db->dbh and don't do anything async
15:14 sri or stateful (->begin/->commit)
15:15 ashimema hmm
15:15 jberger ashimema: I wouldn't try too hard, just let DBIC have its own connections
15:15 ashimema hadn't anticipated that cross purposes issue with sharing a connection
15:15 ashimema I'll just let them have their own connections
15:15 sri it's actually also a problem with Mojo::Pg and Minion, i'm open for suggestions for handling shared connections easier
15:15 ashimema thanks chaps.. saved me falling down the well ;)
15:16 jberger sri: how is it a problem for minion?
15:16 ashimema oh really.. didn't know there was an issue with Mojo::Pg and Minion?
15:16 sri jberger: minion has its own connection
15:16 sri or rather, its own Mojo::Pg instance
15:17 ashimema my understanding is that postgres doesn't like very large numbers of connections
15:17 sri and you can't just pass a Mojo::Pg instance to Minion... because of migrations
15:18 sri so if you use Mojo::Pg and Minion in your app you have at least two connections per worker if both are used
15:18 * ashimema realises he has dbic, minion, mojo::pg all existing in startup now
15:18 sri that's 3 connections per worker then
15:18 ashimema so this would be adding another mojo::pg
15:18 sri it's totally fixable in Minion somehow, i just don't know how
15:18 ashimema no wonder I start to run out of connections in Pg.. ;)
15:19 sri actually, i think that happens in a work project too... where we have minion jobs running out of postgres connections sometimes
15:20 ashimema 3 connections per worker.. with the current app set to a large number of workers accepting single connections.. and many apps running in parallel per customer using the software
15:20 * sri waits for coolo to complain about it too
15:20 jberger ah, gotcha
15:21 ashimema hmm.. just checking my math here..
15:22 jberger I have started to construct my own migrations instances actually ...
15:22 jberger I've gotten a little paranoid
15:22 ashimema if I have hypnotoad set to 10 workers, 1 client, and I have one minion worker running with -j 1 and one minion worker running with -j 4 per 'instance' of my app
15:23 sri ah, the real problem is auto_migrate
15:23 jberger which I never use for production apps
15:23 jberger (I do use for demo apps)
15:24 ashimema am I right in thinking that would be 10 * 3 + 4 * 1 + 1* 1 = 35 connections per instance on average
15:24 ashimema or am I misunderstanding how it all works
15:25 jberger at least, yes
15:25 jberger minion might use more if the job itself uses the database
15:25 sri we need to fix the Mojo::Pg/Minion problem
15:25 jberger oh, actually
15:25 jberger no the -j 4 only uses 1 for the manager
15:25 ashimema ah.. all my minion tasks so far also use the db
15:25 ashimema ah, I see..
15:25 leont_ joined #mojo
15:25 sri so Minion->new(Pg => $pg) works
15:25 jberger but if each job does then yes
15:25 ashimema the -j 4 is number of jobs that manager can accept..
15:26 ashimema misremembered that one
15:26 sri there is no need to spam the channel with your calculations
15:26 sri i think we're all aware that it's too many connections
15:26 ashimema sorry.. I was asking for clarification.. won't spam any further
15:27 jberger I wouldn't call that spam myself, but lets move on to what to do about it
15:28 sri ii think Mojo::Pg->new(Mojo::Pg->new) needs to work somehow to make it share the same connection cache
15:29 jberger this comes back to my concerns of a while ago about the schema names
15:30 sri ?
15:30 jberger sorry, search_path
15:30 sri oh
15:30 jberger since new connections are built with a search_path from the object, I'm not sure how you could do that
15:30 sri yes, we've designed ourselves into a corner it seems
15:31 jberger (and its why I'm SUPER careful about not initializing the Mojo::Pg object too early)
15:31 jberger (and not allowing future developers to accidentally do so)
15:34 sri not even a custom migration Minion::Backend::Pg would work
15:34 sri since we've sold $minion->backend->pg->auto_migrate(0) as a feature
15:35 sri *+in
15:35 sri it's either huge breaking changes or two connections forever
15:37 PryMar56 joined #mojo
15:38 CandyAngel Go beyond the impossible. Don't believe in yourself. Believe in the me that believes in you!
15:40 ashimema lol
15:40 ashimema sorry to open that can of worms guys..
15:41 ashimema can I ask for one more clarification on the minion jobs connections above..
15:41 ashimema if a job requires a connection, does that connection get closed once the job completes.. or is there somthing I can call manually to close it.. or are they somehow shared so I don't need to worry about closing it..
15:42 ashimema feels like I'm 'leaking connections'
15:42 pink_mist CandyAngel++ # break the unbreakable
15:42 ashimema again.. i might be totally misunderstanding :(
15:42 CandyAngel I think pink_mist knows what series I'm watching through!
15:42 pink_mist definitely :P
15:44 jberger the connections should be cleaned up
15:44 jberger I would HOP
15:44 jberger HOPE
15:45 jberger sri: I'm not even sure what I would prefer in its place though
15:46 jberger you can't share connections that don't have the same search path
15:46 jberger I think this is the api we are lead to given the nature of the connections themselves
15:46 ashimema you'd almost need to pool them on search_path
15:47 jberger ashimema: but you can already do that by passing the $pg object around
15:47 ashimema or ensure you know the connection in questions current search path and prepend any query to that connection with a set search path call if and when required
15:47 * ashimema doesn't understand this stuff well enough so will 'stfu now
15:49 jberger but that will affect later usage of that connection
15:49 jberger so you'd be safer to then destroy the connection rather than returning it to the poll
15:49 jberger *pool, thus defeating the pool
15:50 ashimema fun
15:50 ashimema yup.. you clever people are much more likely to work out a solution than I
15:51 karjala_ joined #mojo
15:51 sri i think the assumption needs to be that you don't change the search_path of $pg2 when you do $pg2 = Mojo::Pg->new($pg1)
15:52 sri search_path is already part of the connection settings
15:52 sri ?search_path=...
15:52 sri that means only migrations would run for both
16:09 geheimnis` joined #mojo
16:09 dod joined #mojo
16:11 sh14 joined #mojo
16:13 sri i think i have a solution
16:14 sri it's not pretty
16:14 sri Mojo::Pg would get a parent attribute, which overrides all the connection settings
16:15 mib_f2tqm2 joined #mojo
16:20 brunoramos joined #mojo
16:25 sri it's getting less ugly now
16:32 sri https://gist.github.com/anonymous/41b27e69e27114aa42ff5dd196464f21
16:32 sri it seems to work reasonably well
16:35 sri search_path is just a connection setting, like all the others
16:35 sri and ignored by Mojo::Pg instances that don't create connections
16:36 sri migrations are  tied to the Mojo::Pg object, and run when it gets its first connection, from whatever source
16:38 sri actually there's a logic bug now ;p
16:43 sri that's more like it https://gist.github.com/anonymous/59af8a0a64f8e89e59bb1dc3714ca68e
16:43 sri phew... designed out of the corner again
16:55 arcanez CHYC: mmm, good all (array_agg), but I ended up needing multiple columns and to represent them in json anyway, so ->hashes worked fine
16:57 leont_ joined #mojo
17:07 jberger sri: looks ok at a glance
17:07 sri and i guess i found another connection leak
17:07 sri auto migrations seem to leak another connection
17:07 jberger (heh, right as I said it)
17:07 jberger what was the logic bug?
17:08 sri don't remember, there were multiple
17:08 sri i also had to make auto migrations cascade
17:08 sri so they run on all layers for the first connection
17:10 ashimema apreciate you guys digging straight into this.. impressed as every by the reactivity
17:10 ashimema and care you guys give this stuff
17:12 jberger its breaking my brain a bit, because I'm trying to work on the security stuff in Mojolicious::Plugin::OpenAPI
17:16 sri well, it's actually a problem at work too
17:17 sri this will actually go into my weekly report for work related upstream contributions
17:18 ashimema :D
17:18 sri which is a thing at suse, yay
17:34 pink_mist sri: next make a case for suse wanting http/2 support in mojo so you can legitimately work on that too :P
17:34 sri haha, that's a tough one... considering i don't even believe in http/2 myself
17:35 jberger sri: you've gone that far?
17:35 jberger I know you weren't that excited for it
17:35 sri remember when google was pushing so hard for http/2?
17:36 sri now most of their traffic goes over QUIC
17:36 sri we'll get new protocols all the time now
17:46 ptolemarch joined #mojo
17:56 sri https://github.com/kraih/mojo-pg/commit/0c600420b40f2a16e362960a4a4c842f3df3d7e0
18:05 jberger ugh
18:05 jberger and yet, will QUIC become a standard?
18:06 sri google is already pushing it in the IETF
18:06 sri gonna be HTTP/3 :D
18:07 sri (i've not actually seen anyone push for that yet... but you know it's gonna come up)
18:13 sri and the Minion part https://github.com/kraih/minion/commit/ffdd43a811f769da690c32bc11e13717e7657f9d
18:15 sri yea, makes minion test apps actually easier
18:44 jberger man I wish big companies wouldn't toy with the standards bodies like that
18:45 jberger if you aren't going to stick to the standard you just rammed through, then don't ram it through!
18:55 sri btw. please test the Mojo::Pg and Minion changes
18:55 sri i'll release them soon
18:56 * sri is not afraid to break stuff, if something goes wrong i'll release a 5.0 tomorrow if necessary :p
19:07 abracadaniel joined #mojo
19:08 arcanez 5.0: https://www.youtube.com/watch?v=gMbxOUgPDsI
19:36 maschine to make $config accessible in my models, I just made a helper:  helper config => sub { $config };
19:36 jberger there is already a config helper
19:36 maschine then in my models, I can just use $self->config->{foo}.  Nothing wrong with that right?
19:36 jberger or is it just on the app?
19:36 jberger either way, its right there
19:37 jberger you don't have the helper keyword in your models, do you?
19:37 maschine it wasn't clear to me, sri said yesterday you had to do it manually for models, but it worked automatically for controllers?
19:37 maschine no
19:37 maschine I'm probably making it more complicated than neccesary
19:37 sri you don't use helpers in models
19:37 jberger the quick and dirty way is to pass your application instance into your model on instantiation
19:37 jberger never your controller
19:38 sri jberger: don't teach that garbage!
19:38 jberger but even passing the application in isn't preferable
19:38 jberger like sri says
19:38 jberger I've been working on patterns that I don't hate for this and the closest I have is ...
19:38 maschine what is wrong with using a helper in a model?
19:38 jberger https://metacpan.org/pod/Mojo::TypeModel
19:38 jberger maschine: how do you access it?
19:39 maschine has 'config'; (the name of my helper)
19:39 jberger that's an attribute
19:39 sri models should be independent of your application, so you could use them in a non-web context too
19:40 maschine but if I'm never going to?
19:40 jberger think of it as a guiding principle
19:40 sri then it's still bad design
19:41 jberger by building it with that in mind, even if you don't need to, it will help you build it more robustly
19:41 sri there are other positive side effects, like making tests a lot easier
19:42 jberger Mojo::TypeModel is basically a factory pattern which wraps copying specific data into your "types"
19:43 maschine ok thanks, I'll study that one a bit.
19:44 Craftsmanship joined #mojo
19:45 sri fwiw. i don't understand the typemodel thing
19:46 Craftsmanship How do you test logging, it seems like Mojo::Test just swallows them.
19:47 maschine I actually got the idea of using a helper in the model from this example:
19:47 maschine https://github.com/kraih/mojo-pg/blob/master/examples/blog/lib/Blog/Model/Posts.pm
19:48 maschine at least, that's how I understood it.
19:48 preaction Craftsmanship: $t->app->log->history is what i do. i also set max_history_size if i need to
19:48 jberger maschine: those aren't helpers
19:48 sri maschine: there are definitely no helpers there
19:48 preaction then i make sure everything logs through Mojo::Log via things like Log::Any::Adapter::MojoLog
19:48 arcanez is there a good way to have all endpoints render pretty json (via JSON::XS) if an HTTP param is passed and just use ->render(json => { .. }) for non-pretty?
19:48 jberger Craftsmanship: alternatively, you can add a log event which you can attach to in your test
19:49 maschine I guess I don't understand "has 'pg';
19:49 jberger maschine: that's an attribute
19:49 sri https://github.com/kraih/mojo-pg/blob/master/examples/blog/lib/Blog.pm#L17
19:50 jberger helpers are method that have some sugar on them to make them available on the controllers, the application and as functions in the templates
19:50 jberger attributes are lazy data attached to a class instance
19:50 jberger with accessors
19:51 jberger arcanez: I'm not sure of one, but in theory it shouldn't be hard to create a renderer that does that
19:52 jberger s/renderer/handler/
19:53 jberger then again, I don't use pretty printers for JSON responses anymore
19:53 jberger on the command line I have jq and in the browser I have a plugin
19:54 arcanez is there a way to see if an HTTP param exists? so I could say ?pretty or ?p instead of ?pretty=1 / ?p=1
19:56 arcanez jberger: it'd be nice if I could control end users :)
19:59 jberger why are end-users reading JSON directly?
20:00 jberger arcanez: anyway, I wrote this gist not too long ago
20:00 jberger https://gist.github.com/jberger/687fff38763e70b6712975f6aea8779c
20:01 jberger it does even more than just give you a new handler, it also will take over the json handler if you ask it to
20:06 trone joined #mojo
20:10 arcanez I wonder if I can detect browser (hooman) and pretty from tehre
20:10 arcanez I do believe jq from the command line is more betterer
20:17 Pyritic joined #mojo
20:52 dantti_laptop joined #mojo
21:01 rshadow joined #mojo
21:32 gryphon joined #mojo
21:38 maschine joined #mojo
21:38 sri btw. the connection leak from automatic migrations is confirmed
21:38 sri ouch
21:59 sri ok, all released
23:15 jberger I don't understand how automatic migrations leaked
23:17 jberger did it dequeue inside dequeue and somehow cause that to go pear shaped? I really don't see it

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary