The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2014-02-02

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 * sri wonders if a Mojo::DOM::all would make sense, to get all Mojo::DOM and Mojo::DOM::Node objects in the whole tree
00:05 sri $dom->all->grep(sub { $_->node eq 'comment' })->remove;
00:06 sri maybe ->all_contents
00:16 jb360 joined #mojo
00:34 sri my @all = ($dom->contents->each, $dom->find('*')->contents->flatten->each)
00:34 sri doesn't look as nice
00:50 bpmedley I wonder if Mojo::DOM::all would be the better name?
00:50 sri all_contents would be in line with all_text
00:51 bpmedley I see
00:51 sri normally i would like all better too, but consistency is nice
00:51 bpmedley Understood.  I'm still learning, um, all the framework sub names.. ;)
00:52 sri also still on the edge about its usefulness
00:52 bpmedley It looks much better than find("*").
00:52 geck find = $() ?
00:52 bpmedley Easier to type at the shell.
00:53 bpmedley For -Mojo onelines.
00:53 sri yea
00:53 sri also kinda nice for everyday html sanitizing tasks
00:53 geck sri: what's the use caes for Mojo::DOM?
00:54 bpmedley geck: Do you do much web scraping?
00:54 geck yes.. but not inside of mojo apps
00:54 geck which is why I'm a bit confused, it seems like a fine module
00:54 geck just not sure why it's part of mojolicious and not in it's own distribution
00:54 bpmedley geck: http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Web_scraping <-- CSS selectors!
00:55 sri original use case was testing mojo apps, but it has grown up since then
00:55 geck gotcha
00:55 geck I'll give it a shot next time I have a scraping a project, it looks really nice
00:55 geck just seems out of place
00:55 geck (part of the testing framework makes sense)
00:55 bpmedley It also works really well for a JSON request in a router action.
00:56 bpmedley The UserAgent, that is.
00:56 geck ya.. I may be refactoring some of my stuff to use that bit rather than HTTP::Tiny
00:56 geck since I'm already dependent on mojo
00:57 sri geck: it is used in many places within the framework now http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Command_line
00:57 Mikey hey i'm wondering if you guys have ever run into the message "Failed to inflate a frame" when sending large messages across websockets to chrome
00:58 bpmedley Oh sweet.  Use "mojo" instead of curl.. :)
00:58 sri basically all components in the framework are heavily intertwined
00:59 geck sri: it definitely does a good job of ensuring I go "you only need Mojo"
00:59 geck well usually mojo + some DB or memcache layer
00:59 geck one day I'll get mojo into our product at $work...
00:59 sri yea, you pretty much always need a model layer too... which is why i always call mojo a gateway drug for cpan :)
01:00 geck *nod*
01:00 Mikey also does MOJO_MAX_WEBSOCKET_SIZE limit frame size or the size of whatever's passed to send()
01:01 sri Mikey: sounds like a possible compression bug
01:01 Mikey on whose side?  it works fine in firefox.. makes me think it's in chrome
01:01 sri inflate implies that chrome failed to decompress a message
01:01 sri firefox has no compression i believe
01:02 Mikey is there a way to have the mojo side disable compression on websockets?
01:03 sri it would be more helpful if you tried tracking it down, maybe give me a test case
01:03 Mikey *nod*
01:04 Mikey i owe this project some elbow grease
01:04 Mikey i'll figure this one out
01:04 sri does it require multiple roundtrips?
01:05 sri here's a simple self contained app https://github.com/kraih/mojo/blob/master/examples/websocket.pl
01:05 sri if you can replicate with such an app i can track it down
01:05 Mikey well it emits one frame after i call send() .. what i pass to send is 470838 bytes, the frame it emits is 2595 bytes
01:05 sri *+it
01:05 Mikey okay i'll set it up
01:07 Mikey im doing a bunch of client side crypto stuff when the frame is received, i've sent large messages before.. this makes me think im messing with the state of something i shouldn't be messing with while the decompression is happening
01:07 Mikey but thats far fetched.  lemme see if i can duplicate just sending the message over the wire
01:08 sri i have a feeling i know where things go wrong, but i need a test case to track it
01:08 sri websocket compression has this quirk where it removes a few bytes from the end of a compressed message
01:08 sri i might be doing that wrong, since i don't know the deflate format in detail
01:09 Mikey gimme like 15 and i'll put something up
01:25 Mikey hmm im getting "Event "read" failed: Wide character in Compress::Raw::Zlib::Deflate::deflate input parameter at /usr/local/academica/sys/perl/perl-5.18.0/lib/site_perl/5.18.0/Mojo/Transaction/WebSocket.pm line 98, <DATA> line 48."
01:25 Mikey but i donno if it's my weirdness or what.. also i haven't updated mojolicious in a while i should do that, too
01:26 sri wait wait wait
01:26 sri which mojolicious version are you using?
01:26 Mikey production is 4.59
01:26 Mikey that test is using newest
01:26 Mikey 4.73.. just updated
01:27 Mikey before i was getting a ton of wide character spam about EV::Mojo::Reactor
01:27 Mikey Mojo::Reactor::EV: Write failed: Wide character in syswrite at /usr/local/academica/sys/perl/perl-5.18.0/lib/5.18.0/x86_64-linux-thread-multi-ld/IO/Handle.pm line 483, <DATA> line 48.
01:27 sri that's something you're causing
01:28 Mikey im not surprised
01:28 Mikey i wonder if binary data leaked in there without being properly encoded
01:33 sri opposite, wide characters are not binary
01:34 Mikey there is a single unicode character in the text, and i've sent it across before
01:34 Mikey does compression only kick in if messages are over a certain size?
01:35 sri no
01:35 Mikey yay i got it working in my test!!
01:35 Mikey it only gives the "Failed to inflate a frame" now
01:36 Mikey do you have a preferred method of getting my test?
01:37 Mikey http://pastebin.com/KfSZG7yb
01:38 sri paste is fine
01:38 sri OMG!
01:38 * sri falls off chair
01:39 Mikey hahaha i just took the message and plopped it in there
01:39 Mikey it works!
01:40 sri i figured you would generate some test data
01:40 Mikey there's no test data like real data
01:40 Mikey its just a conversation some coworkers and i had, plus the content is encrypted
01:40 Mikey though our names are in there
01:40 sri but i can confirm the problem
01:41 sri will take a look in a bit
01:41 Mikey good.  i also tested it with chrome canary
01:41 Mikey let me know if i can do anything
01:41 Mikey thank you so much, man
01:48 bpmedley Mikey: What version of chrome and what os?
01:50 Mikey tested on 34.0.1817.4 canary, macos
01:50 Mikey and 32.whatever macos and windows
01:50 Mikey exact version numbers coming
01:51 Mikey 32.0.1700.102 mac
01:51 bpmedley document.body.innerHTML += JSON.parse(event.data).render_as;  <-- I changed the code to that and get: "Testing WebSockets: aom-generic".
01:51 bpmedley Where are you seeing the error?
01:52 Mikey the javascript console in chrome
01:52 Mikey it works fine in firefox
01:52 bpmedley Version 32.0.1700.102  <-- My version; I'm not seeing any error.
01:52 Mikey im getting
01:52 Mikey WebSocket connection to 'ws://academica-dev.wayne.edu:4040/test' failed: Failed to inflate a frame
01:53 bpmedley Hrmm
01:53 Mikey sri was able to duplicate on his machine
01:53 Mikey it's weird it's broken for me in all versions of chrome, but firefox works exactly as you say
01:53 Mikey Testing WebSockets: aom-generic
01:54 sri websocket compression is an extension only chrome supports
01:54 sri for firefox it's simply not negotiated
01:54 bpmedley It works for me in morbo and fails in daemon.
01:55 sri anyway, to disable it you can just remove the Sec-WebSocket-Extensions response header
01:55 sri then it won't be negotiated
01:55 sri as a temp workaround
01:56 sri mojolicious adds it before your action
01:56 bpmedley My morbo was using 5.16.3 and my daemon was using 5.18.1.. Mikey what version of perl are you on?
01:58 sri 5.18.2 here
01:59 sri tested on 5.10.1, doesn't change a thing
02:02 sri oh, actual workaround is a little more verbose
02:02 sri $self->tx->compressed(0)->res->headers->remove('Sec-WebSocket-Extensions');
02:02 sri works afterwards here
02:02 bpmedley It's ActiveState 5.16.3
02:03 bpmedley And self-compiled 5.18.1
02:03 Mikey my test is in 5.18.0 btw
02:03 Mikey i'll disable compression for now
02:03 Mikey thanks, sri
02:03 sri bpmedley: prolly some outdated mojolicious without compression
02:03 bpmedley I upgraded to latest.. seems to work in morbo - I could have a mistake somewhere.
02:09 bpmedley There are two versions installed: one in "user" and one in "site".  I was using old in "site".  New one in "user" has the problem for 5.16.3.  Gaw - sorry.
02:10 sri ooooh, ->all_contents would have another neat side effect... elements would be in the right order, ->find('*')->contents mixes them up wildly
02:10 Mikey sweet that workaround works perfectly, sri thank you again
02:10 sri thanks for the test case
02:11 Mikey np.  im trying to get the university to let me open source what i'm working on, i really think it'd be a nice showcase for mojo.  we've been working on it for 2 years now.
02:11 Mikey and we've pretty seamlessly moved from 2.x -> 3.x to 4.x
02:13 Mikey the most amazing recent feature is client to client encrypted messaging.  pretty much PGP re done up with DJB'd NaCL crypto library
02:13 bpmedley Mikey: What are you working on?
02:14 Mikey it's an academic portal suite..
02:14 Mikey called Academica
02:14 bpmedley Cool stuff
02:14 Mikey we're finally going live with it at Wayne State University this fall
02:14 Mikey we make extensive use of both Mojolicious and Backbone.js
02:15 Mikey we've bound them together in novel ways, with Mojo handling the routing + kicking off backbone views
02:15 Mikey at first mojo and backbone seemed to be playing tug of war with routing
02:16 Mikey it became a bit schitzo for a second there.
02:16 bpmedley :)
02:16 damaya Are there any good examples of RESTful APIs writtin in Mojo?
02:16 damaya s/writtin/written/
02:16 Mikey but the academic portal is built around a realtime message bus, that scales up horizontally with the help of ZeroMQ
02:16 damaya I see on this one that a route is added for each CRUD operation
02:16 bpmedley damaya: What are you trying to do?  You can just return JSON from a route.
02:16 damaya FastNotes is what I am looking at.
02:17 bpmedley Mikey: How many concurrent users can you support?
02:17 Mikey well right now, our production environment is two amazon m1 mediums, and two m1 extra larges
02:18 Mikey our hardware budget has quite a bit of room for expansion though
02:18 Mikey we're barely in pilot/private beta mode
02:18 bpmedley That's cool.  How many do you think it can scale to?
02:19 Mikey well it has to scale to 40k, which I think it can, but we'll find out
02:19 damaya bpmedley: I think I need to wrap my mind around what I am doing first. I have a full app, so what I was doing was sub vacation { ... my $action = $self->param('action'); } and then the call would be /vacation?action=create&user_id=...
02:20 bpmedley damaya: What do you want to return from the route?
02:20 Mikey personally i feel like i could scale it to a half a million concurrent users without having to rethink things
02:20 Mikey half a million websockets
02:20 Mikey open
02:20 damaya I guess after looking at it and hearing about RESTful APIs, I figured maybe my way of doing this is a bit outdated. My API was going to be basically that, and then it returns a JSON response.
02:20 bpmedley Mikey: Nice!
02:20 Mikey the database increasingly will become the bottleneck
02:21 sri hmmm... all_contents also seems about 3 times faster than ->find('*')->contents :o
02:21 bpmedley damaya: Ok, perhaps a small Mojolicious::Lite app that just retruns some JSON?
02:21 sri might be worth it
02:21 Mikey and mojo/zeromq scale horizontally so well im very confident i will be able to meet most every need.. i was able to simulate a message delivery using just my weak single core/two thread app servers .. i delivered a message to 5000 active websocket recipients in 4 seconds
02:22 damaya The API is to be used as part of my full app... There are many things one can do in the full app, and I wanted everything to do be done through an API, so I created lib/API.pm and I am defining API functions there.
02:22 bpmedley damaya: Can you show us a small Mojolicious::Lite app that demonstrates the type of api you want?
02:22 damaya Sure, I can work on that.
02:23 bpmedley I wonder if you just need a different way to look at your problem and/or "talk" about it out loud.
02:23 Mikey the one thing thats gonna hold me back / make me reconsider going with AWS is their lack of multicast support.. im going to saturate the NIC throughput with TCP-based zeromq pubsub
02:24 Mikey the more app servers i add, the worse my throughput problem will become.. but i have cloud hosting plan-b and plan-c
02:24 damaya Thanks bpmedley.
02:24 damaya And yeah, that is the thing, I am stuck at this point where I am unable to really think about what I am trying to do anymore.
02:25 bpmedley Ok, perhaps put up the Lite app and we can yell and scream?
02:26 klapperl_ joined #mojo
02:28 damaya hehe, sounds good... I need to work on it for a bit, so I can better define what I am trying to do.
02:29 good_news_everyone joined #mojo
02:29 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/ZKG4Hw
02:29 good_news_everyone mojo/master 30e520a Sebastian Riedel: added all_contents method to Mojo::DOM
02:29 good_news_everyone left #mojo
02:30 Mikey when writing APIs i always find it helps to "use" them first, play around with how you'd want it to work, and then build something that gives you the interface you dreamed up
02:30 sri perl -Mojo -E 'say g("reddit.com")->dom->all_contents->grep(sub { $_->node eq "comment" })'
02:33 bpmedley IE6sux .. ;)   Those oneliners look much better.
02:44 sri one feature i kinda miss from jquery is being able to return to previous result sets
02:47 ka2u joined #mojo
02:47 d4rkie joined #mojo
02:52 sri lol, i can actually make that work!
02:56 sri haha, it's actually terrible... but here's a patch for future reference... http://pastie.org/8690220
03:07 d4rkie joined #mojo
03:11 sri ok, this might be bad... i've been unable to track down the compression problem
03:12 sri this is the problem when you're trying to support features that have no finished spec yet :/
03:13 sri might have to rip the whole thing out again until things stabilize
03:30 damaya Is there any way to grab mojolicious parameters as a hashref? I.e., ?action=view&user_id=5
03:33 sri jberger, marcus, tempire, crab: opinions would be very welcome, i'm out of ideas for how to fix websocket compression, should i just rip it out for now?
03:33 damaya my $params      = Mojo::Parameters->new($self->param()); is totally wrong, and with $self->param i'm not really sure how to grab values without specifying the exact param.
03:33 sri seems to fail for certain messages, might be size related
03:34 tempire I haven't looked at that at all
03:34 sri tempire: you don't have to, it's more about proper process
03:34 sri basically, websockets fail badly with chrome atm
03:35 tempire well that sounds fun
03:35 tempire even without compression?
03:35 sri nope
03:35 sri only compression
03:35 sri basically, compression fails randomly
03:36 sri and it might be a bug in chrome for all we know
03:36 sri the spec is only a draft...
03:36 tempire sounds like a huge time sucker
03:36 tempire not a fan of those
03:36 sri i was a bit foolish not calling it experimental
03:36 tempire hmm
03:37 sri think i'll remove it and anyone who disagrees can fix it and i'll readd it right away
03:37 damaya Ah, interesting... $self->req->params->to_hash;
03:37 damaya sexy
03:37 tempire Having it just not work mysteriously would be the devil
03:38 tempire removing it without it being labeled experimental is unfortunate...but it's better than a mystery feature.
03:38 tempire especially since everyone who cares about websocket compression is probably using chrome
03:38 Mikey did google at least document their implementation?
03:39 sri http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-17
03:39 sri it's a badly written draft
03:40 sri the sorry state of Compress::Raw::Zlib sure doesn't help either :(
03:42 Mikey im 100% for ripping it out because it fixes my problem, and i've already disabled it anyway.  that said, mobile devices are going to be using this and i'd like to get this working right with compression
03:42 Mikey so i will take a stab at fixing it
03:50 Mikey you just chop off the last 4 bytes with substr?
04:01 jack joined #mojo
04:04 howitdo joined #mojo
04:05 dvinciguerra joined #mojo
04:14 sri Mikey: not quite, you also have to make sure there's an empty deflate block at the end
04:14 sri but that's not the problem, i checked
04:15 good_news_everyone joined #mojo
04:15 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/o7GaXw
04:15 good_news_everyone mojo/master e98254f Sebastian Riedel: removed support for permessage-deflate WebSocket compression, since there have been too many problems with Chrome
04:15 good_news_everyone left #mojo
04:17 sri https://chromium.googlesource.com/chromium/blink/+/master/Source/modules/websockets/WebSocketPerMessageDeflate.cpp # that's the file where "failed to inflate a frame" is from
04:17 Mikey yeah thats where i found the error
04:17 sri doesn't tell me much though
04:17 Mikey thats like the only place on the web it exists
04:18 Mikey i was gonna dig into their inflator code but then i thought, hey, it might be mojo's fault.. and i know perl better than i know cpp, and chrome is infinitely more vast than mojo.. little pile first.
04:18 ka2u joined #mojo
04:19 sri it's really not much code
04:19 Mikey well i can't even get Compress::Raw::Zlib to roundtrip that code so im gonna get an understanding of that first
04:19 sri my first guess was windowbits mismatch... but tried all options without luck
04:19 sri oh
04:21 sri well, i'm going to make a new release anyway, then we can sort it out without a hurry
04:21 Mikey *nod*
04:21 damaya joined #mojo
04:21 Mikey i can pull my workaround out of the repo and upgrade mojo everywhere
04:21 sri the context takeover stuff is wrong anyway
04:21 Mikey this makes me happy.
04:21 sri (doesn't matter here though)
04:24 sri i expect one of two outcomes... a) i messed up the Compress::Raw::Zlib configuration or b) another Compress::Raw::Zlib bug
04:24 Mikey that sounds about right.. once i get it round tripping i'll work with chrome
04:26 ka2u joined #mojo
04:26 sri oh, no need to hurry because of mobile devices btw., so far only desktop chrome supports it
04:27 Mikey ahh hmm.. this might be moot, well if i can't solve it in the next hour or so i'll quit for now.
04:27 Mikey i just wanna contribute something :)
04:28 sri oh, don't mean to discourage you :o
04:28 sri yay Mikey \o/
04:28 Mikey haha thanks :)
04:29 good_news_everyone joined #mojo
04:29 good_news_everyone [mojo] kraih tagged v4.74 at 1f75ed5: http://git.io/Jk6D-A
04:29 good_news_everyone left #mojo
04:30 sri ok, now at least nobody else stumbles over this
04:30 cpan_mojo Mojolicious 4.74 by Sebastian Riedel - http://metacpan.org/release/SRI/Mojolicious-4.74
04:31 sri heh
04:31 * sri pats cpan_mojo
04:34 bpmedley I think it's because the DEFLATED $out ends with 0x0000FFFF.  Therefore an empty DEFLATE block with no compression needs to be padded to $out.
04:38 sri i'm no expert, but as far as i understand zlib, the Z_SYNC_FLUSH should take care of it
04:38 Mikey If the parameter flush is set to Z_SYNC_FLUSH, all pending output is flushed to the output buffer and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. (In particular avail_in is zero after the call if enough output space has been provided before the call.) Flushing may degrade compression for some compression algorithms and so it should be used only when necessary. This completes the current deflate
04:39 Mikey but there might be "more" data.. i think we might have filled up a buffer or something, we might have to hit this many times
04:39 Mikey 474648 bytes compressing down to 8975 bytes feels "too" good
04:45 sri hmm... it's true, i can't even get it to roundtrip
04:45 sri inflate returns data error
04:45 Mikey yeah that's what im getting, too
04:46 Mikey and im noticing this stupid library isn't returning any of the numeric answers i expect, it's returning undef which == Z_OK but only because undef == 0
04:47 * Mikey likes when C bindings are as close to the library they're exposing as possible.
04:47 sri yea, it's silly
04:47 Mikey cos that way you get the benefit of all the underlying library's documentation
04:47 sri most return values are actually untested
04:48 sri https://rt.cpan.org/Public/Bug/Display.html?id=92521
04:48 sri i just reported that a few days ago
04:49 Mikey at least they replied
04:49 sri also funny, if i change the message to "$message = 'a' x 474648;" it works
04:49 Mikey good thing i didn't just make some test data eh ;)
04:50 sri oh, i actually never got a notification for that reply :o
04:51 Mikey tweaking memlevel to 9 makes the compressed output substantially larger
04:51 Mikey 18607 bytes.. up from 8971 .. still doesn't round trip though.
04:54 sri $message = join '', map { int rand(9) }  1 .. 474648;
04:54 sri that seems to fail too
04:54 Mikey enough entropy to not compress small enough is my guess
04:54 sri ye
04:58 sri http://pastie.org/8690386 # a small test case
04:58 sri in case anyone else wants to join the fun ;p
04:59 Mikey i got it working with Compress::Zlib
04:59 Mikey 474648
04:59 Mikey 135947
04:59 Mikey stream end
04:59 Mikey 474648
04:59 Mikey round tripping anyway
04:59 sri do tell
04:59 Mikey sec
05:00 Mikey http://pastie.org/8690388
05:01 Mikey ripped out my huge json structure and replaced with your random bytes
05:01 sri wait what
05:01 purl there's more!
05:01 Mikey lemme put it into mojo and see if it works
05:02 sri hmm... no Z_SYNC_FLUSH though
05:02 Mikey that should work just fine i think
05:02 Mikey about to test
05:04 sri the limit is also quite important
05:04 purl okay, sri.
05:04 sri otherwise the server can get zip bombed
05:08 davido_ If only it were raptors... http://www.thingiverse.com/errorik
05:10 Mikey well i can say using Compress::Zlib at least works in chrome
05:11 Mikey how does that limit work?
05:11 Mikey ahh i see its an output limit
05:15 Mikey hmm Compress::Zlib just wraps Compress::Raw::Zlib... maybe i just need to use it how its using it
05:16 Mikey sri: i think i know what's happening
05:16 Mikey flush() appends more to $out.... so deflate() is setting the value of $out, and then you're wiping out everything that was there with the flush()
05:17 Mikey clobbered
05:17 Mikey testing my theory
05:17 Mikey yep ;){
05:17 Mikey $deflate->deflate(\$frame->[5], my $out);
05:17 Mikey $deflate->flush(my $rest, Z_SYNC_FLUSH);
05:17 Mikey $out .= $rest;
05:18 Mikey that works.
05:18 Mikey not even worth a pull request.
05:19 Mikey thats without using Compress::Zlib, btw.. still using Raw
05:20 bpmedley Compress::Raw::Zlib::Deflate->new(WindowBits => -15, MemLevel => 8, AppendOutput => 1);  <-- Another option
05:21 Mikey yeah that's probably the better option
05:21 Mikey saves yourself a variable
05:22 Mikey yep that works, too
05:23 Mikey now the question is.. why did it always work on your setup, bpmedley?
05:23 bpmedley I had a mistake.  There were two versions of Mojolicious installed and I was using an old one.
05:25 sri oh noes!
05:25 sri Mikey++
05:25 Mikey yay i contributed!!! (to a feature that got yanked out)
05:26 Mikey well, i will enjoy the superbowl extra tomorrow.  this has been fun.
05:26 sri hmmm.... i said i would readd it immediately if someone fixed it :S
05:27 Mikey well then!  i contributed!!! to a feature that will be readded immediately
05:27 Mikey haha feelsgoodman
05:27 sri you are mgregoro?
05:45 Mikey yes thats me
05:46 Mikey on cpan MGREGORO, too
05:53 sri ok, if i bring it back i'll make a few small changes, like disabling compression in the user agent by default
05:53 sri a little more defensive
05:53 Mikey and i will keep my eye on it obviously, i'm using it in production, and will have many different types of users using it over the next few months
05:54 Mikey like i was saying before 30-40k concurrent websockets come this fall
05:54 Mikey with that in mind it's probably a good thing that i dipped my toe into the internals before then
05:55 * sri wonders if Compress::Raw::Zlib is the right module to use
05:55 sri only picked it because the documentation made me want to poke my eyes out the least
05:56 Mikey i think it's probably the best perl module that exists today for this problem
05:59 Mikey can go even lower level and just use IO::Compress::Deflate
06:03 Mikey i hate flimsy object oriented wrappers around procedural c libraries.  it looks like none of that suite lets you get away from that
06:03 sri an actual unit test would have been nice ;p
06:03 Mikey ahh sorry haha
06:03 sri wonder if it happens with smaller messages too
06:04 sri something that actually roundtrips
06:04 Mikey it seemed to work fine with smaller messages.. the overhead for encrypted messages is pretty large, it increases them by about 200%, and we had a really long conversation last night in encrypted messages
06:05 Mikey so, it was about 3x as large as anything i've ever passed through a websocket
06:16 ka2u joined #mojo
06:34 good_news_everyone joined #mojo
06:34 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/KX9zqg
06:34 good_news_everyone mojo/master aeee9e8 Sebastian Riedel: fixed and readded support for permessage-deflate WebSocket compression
06:34 good_news_everyone left #mojo
06:34 sri Mikey: please test
06:35 sri on the client side it's now opt-in, that means we can skip the context takeover mess for now
06:35 sri this way the implementation got really simple
06:36 sri tried to keep everything a bit isolated, in case we have to remove again
06:38 good_news_everyone joined #mojo
06:38 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/tKJYLg
06:38 good_news_everyone mojo/master 5ebab41 Sebastian Riedel: better compression example
06:38 good_news_everyone left #mojo
06:48 sri looks like i have a working test case
06:48 sri but it depends on random data :S
06:51 good_news_everyone joined #mojo
06:51 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/NQkcvw
06:51 good_news_everyone mojo/master eb127ea Sebastian Riedel: test compression edge case
06:51 good_news_everyone left #mojo
06:52 sri it won't fail randomly, but might pass when it shouldn't if stars align
06:53 good_news_everyone joined #mojo
06:53 good_news_everyone [mojo] kraih tagged v4.75 at 80e9265: http://git.io/QgvxQg
06:53 good_news_everyone left #mojo
06:54 sri phew
06:55 cpan_mojo Mojolicious 4.75 by Sebastian Riedel - http://metacpan.org/release/SRI/Mojolicious-4.75
06:59 * sri crashes
06:59 damaya joined #mojo
07:08 damaya If I have the directory structure lib/API.pm and lib/API/Blah.pm, how exactly do I call things that are in Blah? I tried: $r->get('/api/vacation/retrieve')->to( namespace => 'Calm::API', action => 'vacation#retrieve' ); but for some reason that does not work.
07:09 damaya I just get "Page not found yet" and no errors in my terminal (running Morbo)
07:11 damaya Oh, interesting... I guess if I call it with /api/vacation/retrieve it works fine, but not with /api/vacation/retrieve?param=blah
07:12 mikegrb hah, was just pulling up some code to compare
07:13 damaya I don't get with RESTful apis how to allow parameters to go anywhere... Would I actually have to set up a route '/api/vacation/retrieve/#user_id/#user_email... etc?
07:14 damaya What about optional params? I think I may just write this API my old school way (with params) rather than trying to figure out this RESTful stuff :)
07:14 damaya mikegrb: Thanks :)
07:14 damaya I appreciate anyone trying to help me
07:15 mikegrb yeah you would set routes similiar to those as well
07:16 mikegrb for example I Have:
07:16 mikegrb $r->get('/notices/')->to( 'PublicNotices#index', active_tab => 4 );
07:16 mikegrb $r->get('/notices/:page')->to( 'PublicNotices#index', active_tab => 4 );
07:17 damaya Interesting... What if there are multiple params (e.g., 10), and they are optional, and you don't want to enforce any order? That's the part I am having a hard time understanding.
07:17 mikegrb yeah
07:17 mikegrb then it gets tricky :p
07:17 damaya I've looked at some of the examples, and most are similar to what you have. Unfortunately, I haven't found on yet that has a bunch of params.
07:17 damaya Heh
07:17 mikegrb I'm no mojo expert either
07:18 damaya http://en.wikipedia.org/w/api.php?action=query&amp;prop=revisions&amp;rvprop=content&amp;format=xml&amp;titles=Main%20Page
07:18 damaya Like that, for example. I don't get how to do that with these routes :D
07:18 mikegrb I actually popped over to this window tonight to ask a questsion myself but just missed sri
07:18 damaya Ah, well if I tried to help I'd probably just mislead ya.
07:18 damaya heh
07:19 mikegrb I think generally what you want is to put the fundamental non optional stuff in the path part of the address, operation, entity id, etc
07:20 mikegrb and all the optional arguments as url params or so
07:20 mikegrb then you don't have to worry about order there
07:21 mikegrb normally you are using a post for entity creation where you have a lot of args
07:22 mikegrb a restful url for the wikipedia example might look like http://en.wikipedia.org/api/revisions/Main%20Page.xml
07:22 damaya Ah, that's helpful
07:22 damaya I've been reading about REST all night
07:23 mikegrb it's slightly different way to think about constructing urls
07:24 damaya It's more than that though, right? I mean, a GET query returns something to identify the result, and then you can do methods (PUT, DELETE, etc) on what was returned rather easily?
07:24 mikegrb though I guess back in the day we didn't think about constructing urls at all, the url was just for the cgi script itself
07:24 damaya That is what I am seeming to understand of it.
07:24 mikegrb yes that's correct
07:24 damaya So rather than having the URL inform the backend of what to do, the URL is actually performing the action (somewhat)...
07:25 damaya I think I get it, just need to wrap my mind around implementing it :D
07:25 damaya Your example of the Wikipedia URL is actually really helpful.
07:25 damaya Thanks a lot mikegrb!
07:25 mikegrb instead of instead of http://some.api.com/urmom.cgi?action=delete&amp;type=page&amp;title=foo you do an HTTP DELETE request against http://some.api.com/page/foo
07:33 Vandal joined #mojo
07:48 LiveLover joined #mojo
07:57 rem_lex joined #mojo
08:06 jberger joined #mojo
08:22 jberger sri++ Mikey++
08:28 cpan_mojo MojoX-JSON-RPC 0.06 by Henry Tang - http://metacpan.org/release/HENRYYKT/MojoX-JSON-RPC-0.06
09:16 cpan_mojo Test-WWW-Mechanize-Mojo v0.0.16 by Shlomi Fish - http://metacpan.org/release/SHLOMIF/Test-WWW-Mechanize-Mojo-v0.0.16
09:27 jack joined #mojo
09:30 zivester joined #mojo
10:02 basiliscos joined #mojo
10:03 trone joined #mojo
10:16 denis_boyun joined #mojo
10:29 KindTwo joined #mojo
10:34 plenum joined #mojo
10:47 abra joined #mojo
10:59 arpadszasz_ joined #mojo
11:27 sh4 joined #mojo
11:30 denis_boyun joined #mojo
12:06 bowtie joined #mojo
12:11 odc joined #mojo
12:19 zM joined #mojo
13:03 denis_boyun joined #mojo
13:04 dvinciguerra joined #mojo
13:18 bowtie_ joined #mojo
13:52 punter joined #mojo
14:27 ryozi joined #mojo
14:31 denis_boyun joined #mojo
14:37 mire__ joined #mojo
14:51 d4rkie joined #mojo
14:54 gryphon joined #mojo
15:02 denis_boyun_ joined #mojo
15:07 d4rkie joined #mojo
15:09 ka2u joined #mojo
15:14 KindOne joined #mojo
15:27 * sri was a bit worried about all_contents, but it's really worth it
15:28 sri in case you missed it over the whole permessage-deflate thing :) https://github.com/kraih/mojo/commit/30e520aeff3421abbc04c0fbb376999fee3a0b46
15:35 sri perl -Mojo -E 'my $dom = g("www.reddit.com")->dom; $dom->all_contents->grep(sub { $_->node eq "text" && $_->content =~ /^\s+$/ && $_->remove }); say $dom'
15:36 sri fun stuff, this one scrubs all useless whitespace
15:38 nicomen +1
15:38 purl 1
15:45 nicomen http://streaming.fosdem.org/ <-- wasn't there a perl track there?
16:49 denis_boyun joined #mojo
17:05 ka2u joined #mojo
17:19 KindOne joined #mojo
17:22 zackiv31 joined #mojo
17:27 jberger__ joined #mojo
17:45 jberger joined #mojo
17:47 sri hmmm
17:47 sri $dom->all_contents('comment')->remove;
17:47 sri $dom->all_contents('comment', 'text')->remove;
17:50 sri might be going a bit too far now :)
17:57 btyler joined #mojo
18:04 Mikey yay!  4.75 in production, took out my compression workaround, everything's working great!
18:05 sri \o/
18:05 sri is compression worth it in your app?
18:05 Mikey now an afternoon of cooking chili and fixing nasty bugs
18:05 Mikey yes.  definitely visibly faster.
18:06 Mikey chrome always stomped firefox performance wise and now i know why
18:06 sri mmmmmm... chilli... *drools*
18:06 Mikey with infinite scrolling we pull more messages from the websocket, and on chrome it's lightspeed
18:06 Mikey on firefox you can see it loading one message at a time
18:06 Mikey there are obviously other factors but i definitely could tell the difference
18:07 sri interesting, i was expecting it to be less visible, considering that you said your data was encrypted, which shouldn't be very compressible
18:07 mgrimes joined #mojo
18:07 Mikey it's all base64 encoded
18:08 Mikey verry compressible.
18:08 sri ooh :)
18:08 Mikey that message i gave you was "encrypted" and it shrunk down well over 50%
18:08 Mikey and most messages in the system are not encrypted at all
18:10 dod joined #mojo
18:18 dod joined #mojo
18:26 zivester joined #mojo
18:27 good_news_everyone joined #mojo
18:27 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/ylEnhQ
18:27 good_news_everyone mojo/master b43ada2 Sebastian Riedel: mention how to activate permessage-deflate compression in user agent
18:27 good_news_everyone left #mojo
18:34 nicomen Mikey: are you saying that the reading of websocket data itself is much slower in FF than Chrome?
18:39 Mikey i haven't timed just that, i'm saying that on the whole chrome was noticeably slower with compression off than with compression on (anecdotally)
18:39 Mikey and i'm also saying that on the whole chrome handles my js-heavy web app much better than FF does.
18:40 Mikey some of this is obviously my fault!  however, we're supposed to be living in the post-browser-compatibility age.
18:45 nicomen hm, ok
18:50 jack_ joined #mojo
19:22 ka2u joined #mojo
19:43 good_news_everyone joined #mojo
19:43 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/ZBoTAw
19:43 good_news_everyone mojo/master 70f4911 Sebastian Riedel: show how to disable gzip compression
19:43 good_news_everyone left #mojo
19:49 mojobot23379 joined #mojo
20:03 jberger joined #mojo
20:15 good_news_everyone joined #mojo
20:15 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/gUBGyQ
20:15 good_news_everyone mojo/master 1d5d6b9 Sebastian Riedel: more realistic user agent examples
20:15 good_news_everyone left #mojo
20:26 bpmedley damaya: Did you get your api interface working?
20:28 sri neat, the signatures patch is still growing https://github.com/Perl/perl5/commit/152e657b639920a69db529cce93d5b0cbb7dec55#diff-9e7cc5aeaa35c709063fd1031d94555bR405
20:44 lukep joined #mojo
20:55 mire__ joined #mojo
20:57 maxhq joined #mojo
21:00 maxhq joined #mojo
21:06 zM joined #mojo
21:14 sri hmm... $dom->all_contents('comment') is not even an extra line of code
21:17 sri http://pastie.org/8692338
21:43 damaya bpmedley, not really, still reading on REST. This is what the API looks like now: http://pastebin.com/fTzyPADi
21:44 damaya I realize that my way of doing it is a bit 'old school', so trying to make this use RESTful routes. Still trying to figure out multiple params, optional params, stuff like that though.
21:46 damaya I'm having fun figuring this stuff out though :)
21:58 punter joined #mojo
22:25 thowe joined #mojo
22:29 damaya joined #mojo
22:31 thowe Hi
22:34 jberger damaya++ # "I'm having fun figuring this stuff out though :)"
22:37 damaya :)
23:34 good_news_everyone joined #mojo
23:34 good_news_everyone [mojo] kraih pushed 1 new commit to master: http://git.io/MxrUrA
23:34 good_news_everyone mojo/master 438ec4a Sebastian Riedel: speed up collecting of nodes
23:34 good_news_everyone left #mojo
23:40 btyler joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary