Camelia, the Perl 6 bug

IRC log for #mojo, 2013-05-31

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 sivoais joined #mojo
00:49 shmuel joined #mojo
01:05 whitebook joined #mojo
01:07 inokenty joined #mojo
01:28 good_news_everyone joined #mojo
01:28 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/hOOleg
01:28 good_news_everyone mojo/master af152b8 Sebastian Riedel: made Mojo::IOLoop acceptors are little more versatile
01:28 good_news_everyone left #mojo
01:28 sri hmm
01:28 d4rkie joined #mojo
01:28 sri i wonder if the default max_message_size should be increased a little from 5mb
01:34 egopro joined #mojo
01:36 sri maybe something like 25mb
01:49 fildon__ joined #mojo
01:57 Meiermann joined #mojo
01:59 xxtjaxx joined #mojo
02:02 asarch joined #mojo
02:06 good_news_everyone joined #mojo
02:06 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/nE51KQ
02:06 good_news_everyone mojo/master 554f222 Sebastian Riedel: small optimizations
02:06 good_news_everyone left #mojo
02:18 good_news_everyone joined #mojo
02:18 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/7Nm3zA
02:18 good_news_everyone mojo/master 45aa2e4 Sebastian Riedel: increased default max_message_size from 5MB to 10MB in Mojo::Message
02:18 good_news_everyone left #mojo
02:18 sri or just a more conservative 10mb
02:23 good_news_everyone joined #mojo
02:23 good_news_everyone [mojo] kraih tagged v4.09 at 92acc54: http://git.io/zt59iw
02:23 good_news_everyone left #mojo
02:24 sri down to 8891 lines of code btw :D
02:36 moltar joined #mojo
02:40 whitebook joined #mojo
03:06 fildon_ joined #mojo
03:07 jberger wow 4.09 already
03:08 sri yea, i need to take it a little slower again...
03:08 asarch joined #mojo
03:08 jberger sri is really excited about Mojo 5.0
03:09 sri nobody reports bugs during the weekend though, so that will be easy :)
03:10 * sri still wonders if 10mb is the right default
03:11 sri it's security vs convenience
03:11 jberger I assumed you had some app you were working on that needed 9mb messages and releasing 4.09 was easier than changing each app :-)
03:11 sri guilty ;p
03:12 jberger your framework, you get perks \m/
03:13 sri i would revert it if i was sure 5mb was a better default for the majority though
03:14 jberger I don't think there's much in it
03:14 * sri can think of quite a few exploits
03:14 jberger that attack 10mb but not 5?
03:15 sri mostly memory exhaustion attacks, those get at least easier with 10mb
03:16 sri the limit is also applied before gzip
03:16 jberger if you have a zombie net army that change doesn't matter much
03:16 sri sure
03:17 sri with 10mb i wanted to get a little closer to the 16mb mongodb limit for documents
03:18 jberger so why not just do it
03:18 jberger did they profile that number somehow?
03:18 sri small steps, lets see what explodes first :)
03:18 sri the number is completely arbitrary
03:19 * jberger plants a cherry-bomb in Mojo::DOM::CSS :-P
03:19 sri i'm actually scared of increasing that number too much
03:20 jberger understandable
03:20 sri gzipped json document or urlencoded forms can cost a lot
03:23 sri we stream everything over 256kb into temp files, but once you go res->json or res->param(...) everything gets loaded into memory and parsed
03:23 jberger I wonder how big some of these client-side javascript app framework's apps can get?
03:23 sri s/res/req/
03:23 jberger oh yeah, I can see that
03:25 sri for example, the single page version of the HTML5.1 spec is 6mb gzipped
03:25 sri if you parse it with Mojo::DOM you get to about 140mb
03:25 jberger the dom representation must be pretty big then
03:25 jberger yeah
03:26 sri about 122k tags
03:26 jberger what about memory mapping?
03:27 jberger if File::Map is available could we use that for large dom/json/etc
03:27 jberger dom especially I would think
03:27 sri perl -Mojo -E 'say g("www.w3.org/html/wg/drafts/html/master/​single-page.html")->dom->find("*")->size'
03:27 sri actually... try it yourself :)
03:28 jberger is this going to fry my netbook? :-P
03:28 jberger message size exceeded
03:28 sri 12s on my macbook air
03:28 * jberger hasn't upgraded yet
03:29 jberger its still doing something even though the error popped
03:30 sri upgrade to 4.09 first... 5mb limit :)
03:30 jberger but why does the process keep going?
03:30 jberger once the message popped, shouldn't it stop?
03:30 sri it's parsing the partial, you're not checking for errors
03:30 jberger oh it gets the first 5mb?
03:30 sri ye
03:31 jberger well I'm upgrading now
03:31 sri takes more time to repair the html though
03:31 * sri is still fascinated by how much Mojo::DOM can actually repair
03:33 jberger hmmm, cpanm wasn't downloading correctly
03:33 jberger maybe a bad mirror
03:33 jberger going now
03:36 jberger 52s real
03:36 jberger on the netbook
03:36 sri not too bad
03:37 mrphilov1 joined #mojo
03:39 jberger running it on my bigger box via ssh
03:40 jberger 17s real
03:40 jberger that box was top-of-the-line for 2007
03:40 jberger still holds its own
03:41 preflex_ joined #mojo
03:57 jberger I wonder if I used Mojo with Ember, would I have static html files and template the javascript?
03:58 * jberger needs to learn more about these client-side frameworks
03:58 jberger I know preaction knows some of this
03:59 jberger these things make so much sense for rich websocket apps
04:00 sri wow, jamadam has backported mojolicious 4 :o https://github.com/jamadam/mojo-legacy
04:03 ynonp joined #mojo
04:05 jnbek joined #mojo
04:23 ynonp joined #mojo
04:31 d4rkie joined #mojo
04:37 D4RK-PH0ENiX joined #mojo
04:45 preaction jberger, for the most bit, you'd get your data via the socket / the REST API, and then yeah, the static js/html fills it all in
05:08 d4rkie joined #mojo
05:11 D4RK-PH0ENiX joined #mojo
05:20 Meiermann joined #mojo
05:22 ynonp joined #mojo
05:24 d4rkie joined #mojo
05:49 Britzel joined #mojo
06:00 suy joined #mojo
06:09 dod joined #mojo
06:40 dod joined #mojo
06:45 ynonp joined #mojo
06:47 yakudza joined #mojo
06:54 denisboyun joined #mojo
07:08 kwa sri, apologies about not commenting on the patch. I read after you posted it that it broke embedded apps so thought you might not want to apply it. I like it though. It's the same as the hack I implemented, so happy about that. :)
07:15 Vandal joined #mojo
07:19 dod joined #mojo
07:34 dod joined #mojo
07:52 jzawodn joined #mojo
08:07 nicolaas joined #mojo
08:07 dpetrov_ joined #mojo
08:14 cosmincx joined #mojo
08:47 fhelmber_ joined #mojo
09:13 mkrull joined #mojo
09:23 depesz joined #mojo
09:24 depesz hi. i can't seem to find a way to get to request params from template. do i have to pass the req object to stash manually?
09:24 depesz or is there some automatic, given, way?
09:29 sri kwa: it does not "break" embedded apps, they work just the same
09:33 kwa sri: Ah, I must of misunderstood you then. Were you saying embedded apps already had the format detected?
09:33 kwa (and available to respond_to already)
09:35 kwa Regardless, I like it. *upvotes*
09:36 yakudza depesz, http://mojolicio.us/perldoc/​Mojolicious/Controller#param
09:36 yakudza depesz, http://mojolicio.us/perldoc/Mojoli​cious/Plugin/DefaultHelpers#param
09:37 depesz yakudza: first one i saw, but it is not obvious from there that I can use it in templates too.
09:37 depesz the second one helps, thanks.
09:37 depesz on the other hand - id didn't work because I'm stupid.
09:38 depesz s/id/it/
09:38 depesz trying to reach to param from template, that I did ->redirect() to, doesn't make much sense.
09:40 maxhq joined #mojo
09:45 nic redirect all the params
09:46 mrphilov joined #mojo
09:50 mkrull joined #mojo
09:51 ladnaV joined #mojo
10:00 sri kwa: nope, respond_to with format does currently and will still not work in the future after the patch
10:00 sri anyway, don't think the patch has much of a chance anymore, considering none of the core devs have voiced an opinion
10:01 kwa sri: I'll have to wait until they write a RESTful API then. :)
10:03 kwa jberger: up vote the respond_to format patch. It makes sense! :p
10:12 sri it's just the same for hooks too, respond_to with format won't start working there either
10:12 xxtjaxx joined #mojo
10:14 * sri wonders if POST parameter support in respond_to is actually a security issue
10:14 sri maybe it should be limited to GET parameters by default
10:14 cosmincx joined #mojo
10:15 denisboyun joined #mojo
10:23 kwa sri: Oo not hooks either? From a RESTful point of view, respond_to working in bridges (and hooks now you've mentioned it) seems like it should be the de facto. Guess it's subjective.
10:24 kwa May I ask the security issue with POST params in respond_to?
10:24 kwa P.s. When do you sleep, sri? :/
10:27 sri memory exhaustion attack with carefully crafted requests
10:28 sri with the 10mb default limit i can make respond_to requests grow to 1gb
10:39 sri http://pastie.org/7988160 # the fix
10:39 sri jberger, marcus, tempire, crab: not sure if we consider this a security issue
10:43 sri another possible fix would be a parameter limit in Mojo::Parameters i suppose
10:43 sri like 100 or so
10:43 * sri shrugs
10:44 sri or maybe i'm overthinking it again and the user is responsible for validating all requests
10:47 sri Mojo::Content::MultiPart would also need a max_parts limit
10:50 kitt_vl joined #mojo
10:54 nic It would be nice to have overrideable defaults for those limits
10:57 Adura Doesn't PHP do 1000, rather than 100?
11:05 Adura Not that I know how expensive populating post/get vars are in PHP...
11:10 denisboyun_ joined #mojo
11:30 whitebook joined #mojo
11:31 Vandal joined #mojo
11:40 mire joined #mojo
11:40 ryozi joined #mojo
11:44 ilyuhan joined #mojo
12:03 rihegher joined #mojo
12:06 denisboyun_ joined #mojo
12:10 libsysguy joined #mojo
12:11 mkrull joined #mojo
12:20 jberger I guess I dislike things that are special cases
12:20 jberger I am less for the patch if its only for GET requests
12:20 rihegher left #mojo
12:21 jberger and how much of the system doesn't work in embedded apps?
12:24 moltar joined #mojo
12:25 moltar joined #mojo
12:38 mattp joined #mojo
12:45 libsysguy wow... https://jira.mongodb.org/browse/PYTHON-532
13:00 kwa jberger: I understand that not everyone will be writing a RESTful API, so it is a special case. But if someone is explicity calling respond_to in a bridge, hook, under etc. it just makes sense to me to respond with the correct format. As sri pointed out, however, it's not quite DWIM.
13:01 kwa It might be good if I could turn on something which makes respond_to 'strict'. But I guess rather than that I might as well just apply the hack.
13:11 denisboyun_ joined #mojo
13:16 ladnaV joined #mojo
13:21 Vandal joined #mojo
13:27 mkrull libsysguy: somebody was in a mood after having a great night of debugging mongo :P
13:27 libsysguy it certainly looks that way
13:29 libsysguy they are going to change is name to Ragin' Mike
13:44 kej joined #mojo
13:45 btyler joined #mojo
13:59 btyler joined #mojo
14:20 ynonp joined #mojo
14:46 nic https://jira.mongodb.org/browse/PYTHON-532
14:51 tempire why would post support in respond_to be a security issue?
15:01 libsysguy I'm a little confused by the under documentation in the router
15:01 libsysguy if I am under something do I still have to give a full path to the next path
15:02 sri tempire: like i said, memory exhaustion attack
15:02 libsysguy for example, if I have a get for /admin and have that assigned to an under variable, can I say $var->get('test') and end up with a url path like /admin/test
15:02 sri jberger_: only for GET requests?
15:02 tempire oooooooh
15:03 tempire hmm
15:03 sri respond_to implicitly grabs POST parameters, in turn triggering them getting parsed
15:03 sri if you call ->param() you're supposed to know what you can trigger
15:04 sri suppose we need a demonstration
15:05 sri perl -Mojo -E 'a(sub { shift->respond_to(any => {text => "ohoh!"}) })->start' daemon
15:05 sri perl -Mojo -E 'say p("127.0.0.1:3000" => {"Content-Type" => "application/x-www-form-urlencoded"} => "a=b&" x 1572864)->body for 1 .. 1000'
15:05 sri run those and watch the memory
15:06 sri or
15:06 sri perl -Mojo -E 'say p("127.0.0.1:3000" => {"Content-Type" => "application/x-www-form-urlencoded"} => "a=b&" x 3145728)->body for 1 .. 1000'
15:07 sri if you have 4.09
15:07 sri it's not a very sophisticated attack, but you get the idea
15:07 nic libsysguy: try it -- "./script/myapp routes"
15:08 denisboyun joined #mojo
15:09 libsysguy I get: /admin, then under it I get +/test
15:09 sri the point of respond_to not using POST parameters only means it won't automatically trigger parmeter parsing... if you call ->param() without validating the request first you still lost
15:09 libsysguy I'm not sure what the + means
15:12 libsysguy jberger_: what you said yesterday about semicolons apparently needed to marinate overnight in my head so it got through.  Today i realized it was causing one of my errors :-/
15:12 sri btw. i'm not sure any perl project has a protection mechanism for that kind of attack
15:12 sri but that doesn't mean we shouldn't think about it
15:14 nic libsysguy: The '+' line is also indented; means it's added to the thing above that is relatively outdented
15:14 libsysguy so the full path for the indented thing should be /admin/test
15:14 libsysguy which is what I wanted
15:15 libsysguy but TIL how to get routes…I was doing mojo routes which was not very helpful :p
15:24 dod joined #mojo
15:31 tagg joined #mojo
15:32 sri thing is, even if you allow only 1000 parameters, it's just as effective
15:32 sri perl -Mojo -E 'say p("127.0.0.1:3000" => {"Content-Type" => "application/x-www-form-urlencoded"} => join("&", map {"a=" . $_ x 100000} 0 .. 1000))->body for 1 .. 1000'
15:32 sri so that's no solution
15:37 tagg joined #mojo
15:38 d4rkie joined #mojo
15:42 tempire limiting parameters is definitely not the way
15:42 * sri nods
15:43 sri that's why my first thought was to limit implicit parameter parsing
15:43 ilyuhan joined #mojo
15:44 sri $self->param() is like $self->req->json or $self->req->dom, a concious decision to parse the request, *you* take responsibility
15:44 sri the respond_to case doesn't feel the same
15:46 delias joined #mojo
15:47 tagg joined #mojo
15:53 tempire max memory size doesn't cover this?
15:54 sri nope, that only covers storing the raw content
15:55 sri it's also on a very different layer
15:56 sri it would actually be pretty easy to make it apply with a few exceptions... but i' not sure you really want that
15:57 sri die "Can't parse files!" if $self->content->asset->is_file;
15:57 sri result however would be that respond_to for example dies whenever you have a request with more than 256kb
16:04 r0b3rt joined #mojo
16:10 sh4 joined #mojo
16:12 SmokeMachine joined #mojo
16:12 asarch joined #mojo
16:15 sri or we could just reject the content if it has to be stored in temp files
16:16 sri my $content = $asset->is_file ? '' : $asset->slurp;
16:16 basiliscos joined #mojo
16:17 sri how far do we want to go protecting people from shooting themselves in the foot?
16:18 sri what do you want ->json to do if the message content exceeded max_memory_size and got stored in a temp file?
16:22 tagg joined #mojo
16:26 tempire it's a good point to raise that using ->param at all without considering the consequences puts you at risk for an attack
16:27 tempire regarding json, max_memory_size, I think a warning is fine
16:27 tempire though I wonder if a strict mode would be good
16:28 sri what kind of warning?
16:28 sri what kind of strict mode? this spans a dozen layers...
16:33 sri like a POD warning? "Also note that to parse parameters message content needs to be loaded into memory, so you need to make sure it is not excessively large."
16:34 bluescreen joined #mojo
16:36 sri http://pastie.org/7989598 # that's a lot of warnings
16:42 tagg joined #mojo
16:46 kwa sri, what would be a sane limit for request content limit?
16:47 sri kwa: what is a request content limit?
16:47 kwa As I'm writing a public facing API, so after reading your coments would like to enforce a limit.
16:47 kwa For POST data.
16:47 sri i don't know
16:47 kwa "Also note that to parse parameters request content needs to be loaded into memory"
16:48 kwa I wondered if during your testing you found a goldilocks zone. :P
16:53 tagg joined #mojo
16:53 ron-slc joined #mojo
16:53 bluescreen_ joined #mojo
16:55 ajmrch joined #mojo
16:56 good_news_everyone joined #mojo
16:56 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/Lim9pw
16:56 good_news_everyone mojo/master 04ceeba Sebastian Riedel: mention which methods need to load message content into memory to be able to parse it
16:56 good_news_everyone left #mojo
16:56 sri allright, no longer my problem :)
17:03 rem_lex|pivo joined #mojo
17:03 tagg joined #mojo
17:08 tagg joined #mojo
17:13 mkrull joined #mojo
17:13 tagg joined #mojo
17:14 mkrull joined #mojo
17:28 tagg joined #mojo
17:42 libsysguy sri thank you for implementing proper url parameter handling in Mojo::URL
17:42 ron-slc joined #mojo
18:04 btyler joined #mojo
18:12 whitebook joined #mojo
18:15 good_news_everyone joined #mojo
18:15 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/ZnpMyA
18:15 good_news_everyone mojo/master 5065000 Sebastian Riedel: documentation tweaks
18:15 good_news_everyone left #mojo
18:27 rem_lex| joined #mojo
18:43 sri oh, it's not just respond_to, the taghelpers also use ->param() to pre-fill elements
18:52 ron-slc joined #mojo
19:13 doby joined #mojo
19:22 asarch__ joined #mojo
19:23 bluescreen joined #mojo
19:49 mugenken joined #mojo
20:00 basiliscos joined #mojo
20:10 marty mojolicious routing is soooo groovy
20:17 sh4|2 joined #mojo
20:19 sh4|2 joined #mojo
20:22 radixo joined #mojo
20:22 radixo Hello guys.. I am new with Perl and mojolicio.us
20:22 radixo can you help me with deploy?
20:23 radixo how do I setup the Hypnotoad ?
20:24 marty radixo:  Have you found the guides?
20:24 radixo marty: Yes..
20:25 radixo but I don't know how to use app->config(hypnotoad => {listen => ['http://*:80']});
20:25 radixo I am not using Mojolicious::Lite
20:28 marty In the startup section of your app you'll put something like...  $self->config(hypnotoad => {listen => ['http://*:80']});;
20:29 marty You can also use Mojolicious::Plugin::Config or Mojolicious::Plugin::JSONConfig to keep things clean.
20:33 maxhq joined #mojo
20:42 sri kwa: hopefully the "goldilocks zone" is the current 10mb
20:43 sri supporting 1k concurrent connections per process makes it rather tricky though
20:44 sri small numbers quickly add up
20:48 SmokeMachine joined #mojo
20:58 sri hmmm
20:58 sri looking through the parameter code... does anyone know what the hell x-application-urlencoded is?
21:03 sri haha, google only shows mojolicious hits... i must have been drunk
21:04 coff joined #mojo
21:07 good_news_everyone joined #mojo
21:07 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/txQm-g
21:07 good_news_everyone mojo/master 87e1711 Sebastian Riedel: small optimizations
21:07 good_news_everyone left #mojo
21:17 coff joined #mojo
21:19 BeDa joined #mojo
21:32 basiliscos joined #mojo
21:37 sri kwa: btw. the only way to make respond_to work with hooks would be something like $self->stash->{format} = $self->req->url->path->ext
21:37 sri the router runs *after* most hooks
21:37 sri so you could only guess based in the URL
21:38 sri s/i/o/
22:07 coff joined #mojo
22:13 root joined #mojo
22:16 al802 joined #mojo
22:22 al802 Hi Guys, I'm using ApacheBench trying to max out one of my servers and at best I can get is 25% CPU usage, the server has 16 cores (32 threads)
22:25 al802 I'm also getting SSL handshake failed (5) errors (lots of them)
22:28 al802 Any ideas how I can scale to use all the CPU?
22:29 al802 even using a 1000 workers does not help
22:30 good_news_everyone joined #mojo
22:30 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/bnM_Zw
22:30 good_news_everyone mojo/master 52253ce Sebastian Riedel: improved performance of multipart/form-data parser a little
22:30 good_news_everyone left #mojo
22:30 sri haha, exactly 8888 lines of code now \o/
22:50 good_news_everyone joined #mojo
22:50 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/w2dn5w
22:50 good_news_everyone mojo/master 5cf6108 Sebastian Riedel: check filename first
22:50 good_news_everyone left #mojo
23:20 chansen 88 isn't a good number IMO

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary