| Time |
Nick |
Message |
| 00:03 |
* sri |
wonders if a Mojo::DOM::all would make sense, to get all Mojo::DOM and Mojo::DOM::Node objects in the whole tree |
| 00:05 |
sri |
$dom->all->grep(sub { $_->node eq 'comment' })->remove; |
| 00:06 |
sri |
maybe ->all_contents |
| 00:16 |
|
jb360 joined #mojo |
| 00:34 |
sri |
my @all = ($dom->contents->each, $dom->find('*')->contents->flatten->each) |
| 00:34 |
sri |
doesn't look as nice |
| 00:50 |
bpmedley |
I wonder if Mojo::DOM::all would be the better name? |
| 00:50 |
sri |
all_contents would be in line with all_text |
| 00:51 |
bpmedley |
I see |
| 00:51 |
sri |
normally i would like all better too, but consistency is nice |
| 00:51 |
bpmedley |
Understood. I'm still learning, um, all the framework sub names.. ;) |
| 00:52 |
sri |
also still on the edge about its usefulness |
| 00:52 |
bpmedley |
It looks much better than find("*"). |
| 00:52 |
geck |
find = $() ? |
| 00:52 |
bpmedley |
Easier to type at the shell. |
| 00:53 |
bpmedley |
For -Mojo onelines. |
| 00:53 |
sri |
yea |
| 00:53 |
sri |
also kinda nice for everyday html sanitizing tasks |
| 00:53 |
geck |
sri: what's the use caes for Mojo::DOM? |
| 00:54 |
bpmedley |
geck: Do you do much web scraping? |
| 00:54 |
geck |
yes.. but not inside of mojo apps |
| 00:54 |
geck |
which is why I'm a bit confused, it seems like a fine module |
| 00:54 |
geck |
just not sure why it's part of mojolicious and not in it's own distribution |
| 00:54 |
bpmedley |
geck: http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Web_scraping <-- CSS selectors! |
| 00:55 |
sri |
original use case was testing mojo apps, but it has grown up since then |
| 00:55 |
geck |
gotcha |
| 00:55 |
geck |
I'll give it a shot next time I have a scraping a project, it looks really nice |
| 00:55 |
geck |
just seems out of place |
| 00:55 |
geck |
(part of the testing framework makes sense) |
| 00:55 |
bpmedley |
It also works really well for a JSON request in a router action. |
| 00:56 |
bpmedley |
The UserAgent, that is. |
| 00:56 |
geck |
ya.. I may be refactoring some of my stuff to use that bit rather than HTTP::Tiny |
| 00:56 |
geck |
since I'm already dependent on mojo |
| 00:57 |
sri |
geck: it is used in many places within the framework now http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Command_line |
| 00:57 |
Mikey |
hey i'm wondering if you guys have ever run into the message "Failed to inflate a frame" when sending large messages across websockets to chrome |
| 00:58 |
bpmedley |
Oh sweet. Use "mojo" instead of curl.. :) |
| 00:58 |
sri |
basically all components in the framework are heavily intertwined |
| 00:59 |
geck |
sri: it definitely does a good job of ensuring I go "you only need Mojo" |
| 00:59 |
geck |
well usually mojo + some DB or memcache layer |
| 00:59 |
geck |
one day I'll get mojo into our product at $work... |
| 00:59 |
sri |
yea, you pretty much always need a model layer too... which is why i always call mojo a gateway drug for cpan :) |
| 01:00 |
geck |
*nod* |
| 01:00 |
Mikey |
also does MOJO_MAX_WEBSOCKET_SIZE limit frame size or the size of whatever's passed to send() |
| 01:01 |
sri |
Mikey: sounds like a possible compression bug |
| 01:01 |
Mikey |
on whose side? it works fine in firefox.. makes me think it's in chrome |
| 01:01 |
sri |
inflate implies that chrome failed to decompress a message |
| 01:01 |
sri |
firefox has no compression i believe |
| 01:02 |
Mikey |
is there a way to have the mojo side disable compression on websockets? |
| 01:03 |
sri |
it would be more helpful if you tried tracking it down, maybe give me a test case |
| 01:03 |
Mikey |
*nod* |
| 01:04 |
Mikey |
i owe this project some elbow grease |
| 01:04 |
Mikey |
i'll figure this one out |
| 01:04 |
sri |
does it require multiple roundtrips? |
| 01:05 |
sri |
here's a simple self contained app https://github.com/kraih/mojo/blob/master/examples/websocket.pl |
| 01:05 |
sri |
if you can replicate with such an app i can track it down |
| 01:05 |
Mikey |
well it emits one frame after i call send() .. what i pass to send is 470838 bytes, the frame it emits is 2595 bytes |
| 01:05 |
sri |
*+it |
| 01:05 |
Mikey |
okay i'll set it up |
| 01:07 |
Mikey |
im doing a bunch of client side crypto stuff when the frame is received, i've sent large messages before.. this makes me think im messing with the state of something i shouldn't be messing with while the decompression is happening |
| 01:07 |
Mikey |
but thats far fetched. lemme see if i can duplicate just sending the message over the wire |
| 01:08 |
sri |
i have a feeling i know where things go wrong, but i need a test case to track it |
| 01:08 |
sri |
websocket compression has this quirk where it removes a few bytes from the end of a compressed message |
| 01:08 |
sri |
i might be doing that wrong, since i don't know the deflate format in detail |
| 01:09 |
Mikey |
gimme like 15 and i'll put something up |
| 01:25 |
Mikey |
hmm im getting "Event "read" failed: Wide character in Compress::Raw::Zlib::Deflate::deflate input parameter at /usr/local/academica/sys/perl/perl-5.18.0/lib/site_perl/5.18.0/Mojo/Transaction/WebSocket.pm line 98, <DATA> line 48." |
| 01:25 |
Mikey |
but i donno if it's my weirdness or what.. also i haven't updated mojolicious in a while i should do that, too |
| 01:26 |
sri |
wait wait wait |
| 01:26 |
sri |
which mojolicious version are you using? |
| 01:26 |
Mikey |
production is 4.59 |
| 01:26 |
Mikey |
that test is using newest |
| 01:26 |
Mikey |
4.73.. just updated |
| 01:27 |
Mikey |
before i was getting a ton of wide character spam about EV::Mojo::Reactor |
| 01:27 |
Mikey |
Mojo::Reactor::EV: Write failed: Wide character in syswrite at /usr/local/academica/sys/perl/perl-5.18.0/lib/5.18.0/x86_64-linux-thread-multi-ld/IO/Handle.pm line 483, <DATA> line 48. |
| 01:27 |
sri |
that's something you're causing |
| 01:28 |
Mikey |
im not surprised |
| 01:28 |
Mikey |
i wonder if binary data leaked in there without being properly encoded |
| 01:33 |
sri |
opposite, wide characters are not binary |
| 01:34 |
Mikey |
there is a single unicode character in the text, and i've sent it across before |
| 01:34 |
Mikey |
does compression only kick in if messages are over a certain size? |
| 01:35 |
sri |
no |
| 01:35 |
Mikey |
yay i got it working in my test!! |
| 01:35 |
Mikey |
it only gives the "Failed to inflate a frame" now |
| 01:36 |
Mikey |
do you have a preferred method of getting my test? |
| 01:37 |
Mikey |
http://pastebin.com/KfSZG7yb |
| 01:38 |
sri |
paste is fine |
| 01:38 |
sri |
OMG! |
| 01:38 |
* sri |
falls off chair |
| 01:39 |
Mikey |
hahaha i just took the message and plopped it in there |
| 01:39 |
Mikey |
it works! |
| 01:40 |
sri |
i figured you would generate some test data |
| 01:40 |
Mikey |
there's no test data like real data |
| 01:40 |
Mikey |
its just a conversation some coworkers and i had, plus the content is encrypted |
| 01:40 |
Mikey |
though our names are in there |
| 01:40 |
sri |
but i can confirm the problem |
| 01:41 |
sri |
will take a look in a bit |
| 01:41 |
Mikey |
good. i also tested it with chrome canary |
| 01:41 |
Mikey |
let me know if i can do anything |
| 01:41 |
Mikey |
thank you so much, man |
| 01:48 |
bpmedley |
Mikey: What version of chrome and what os? |
| 01:50 |
Mikey |
tested on 34.0.1817.4 canary, macos |
| 01:50 |
Mikey |
and 32.whatever macos and windows |
| 01:50 |
Mikey |
exact version numbers coming |
| 01:51 |
Mikey |
32.0.1700.102 mac |
| 01:51 |
bpmedley |
document.body.innerHTML += JSON.parse(event.data).render_as; <-- I changed the code to that and get: "Testing WebSockets: aom-generic". |
| 01:51 |
bpmedley |
Where are you seeing the error? |
| 01:52 |
Mikey |
the javascript console in chrome |
| 01:52 |
Mikey |
it works fine in firefox |
| 01:52 |
bpmedley |
Version 32.0.1700.102 <-- My version; I'm not seeing any error. |
| 01:52 |
Mikey |
im getting |
| 01:52 |
Mikey |
WebSocket connection to 'ws://academica-dev.wayne.edu:4040/test' failed: Failed to inflate a frame |
| 01:53 |
bpmedley |
Hrmm |
| 01:53 |
Mikey |
sri was able to duplicate on his machine |
| 01:53 |
Mikey |
it's weird it's broken for me in all versions of chrome, but firefox works exactly as you say |
| 01:53 |
Mikey |
Testing WebSockets: aom-generic |
| 01:54 |
sri |
websocket compression is an extension only chrome supports |
| 01:54 |
sri |
for firefox it's simply not negotiated |
| 01:54 |
bpmedley |
It works for me in morbo and fails in daemon. |
| 01:55 |
sri |
anyway, to disable it you can just remove the Sec-WebSocket-Extensions response header |
| 01:55 |
sri |
then it won't be negotiated |
| 01:55 |
sri |
as a temp workaround |
| 01:56 |
sri |
mojolicious adds it before your action |
| 01:56 |
bpmedley |
My morbo was using 5.16.3 and my daemon was using 5.18.1.. Mikey what version of perl are you on? |
| 01:58 |
sri |
5.18.2 here |
| 01:59 |
sri |
tested on 5.10.1, doesn't change a thing |
| 02:02 |
sri |
oh, actual workaround is a little more verbose |
| 02:02 |
sri |
$self->tx->compressed(0)->res->headers->remove('Sec-WebSocket-Extensions'); |
| 02:02 |
sri |
works afterwards here |
| 02:02 |
bpmedley |
It's ActiveState 5.16.3 |
| 02:03 |
bpmedley |
And self-compiled 5.18.1 |
| 02:03 |
Mikey |
my test is in 5.18.0 btw |
| 02:03 |
Mikey |
i'll disable compression for now |
| 02:03 |
Mikey |
thanks, sri |
| 02:03 |
sri |
bpmedley: prolly some outdated mojolicious without compression |
| 02:03 |
bpmedley |
I upgraded to latest.. seems to work in morbo - I could have a mistake somewhere. |
| 02:09 |
bpmedley |
There are two versions installed: one in "user" and one in "site". I was using old in "site". New one in "user" has the problem for 5.16.3. Gaw - sorry. |
| 02:10 |
sri |
ooooh, ->all_contents would have another neat side effect... elements would be in the right order, ->find('*')->contents mixes them up wildly |
| 02:10 |
Mikey |
sweet that workaround works perfectly, sri thank you again |
| 02:10 |
sri |
thanks for the test case |
| 02:11 |
Mikey |
np. im trying to get the university to let me open source what i'm working on, i really think it'd be a nice showcase for mojo. we've been working on it for 2 years now. |
| 02:11 |
Mikey |
and we've pretty seamlessly moved from 2.x -> 3.x to 4.x |
| 02:13 |
Mikey |
the most amazing recent feature is client to client encrypted messaging. pretty much PGP re done up with DJB'd NaCL crypto library |
| 02:13 |
bpmedley |
Mikey: What are you working on? |
| 02:14 |
Mikey |
it's an academic portal suite.. |
| 02:14 |
Mikey |
called Academica |
| 02:14 |
bpmedley |
Cool stuff |
| 02:14 |
Mikey |
we're finally going live with it at Wayne State University this fall |
| 02:14 |
Mikey |
we make extensive use of both Mojolicious and Backbone.js |
| 02:15 |
Mikey |
we've bound them together in novel ways, with Mojo handling the routing + kicking off backbone views |
| 02:15 |
Mikey |
at first mojo and backbone seemed to be playing tug of war with routing |
| 02:16 |
Mikey |
it became a bit schitzo for a second there. |
| 02:16 |
bpmedley |
:) |
| 02:16 |
damaya |
Are there any good examples of RESTful APIs writtin in Mojo? |
| 02:16 |
damaya |
s/writtin/written/ |
| 02:16 |
Mikey |
but the academic portal is built around a realtime message bus, that scales up horizontally with the help of ZeroMQ |
| 02:16 |
damaya |
I see on this one that a route is added for each CRUD operation |
| 02:16 |
bpmedley |
damaya: What are you trying to do? You can just return JSON from a route. |
| 02:16 |
damaya |
FastNotes is what I am looking at. |
| 02:17 |
bpmedley |
Mikey: How many concurrent users can you support? |
| 02:17 |
Mikey |
well right now, our production environment is two amazon m1 mediums, and two m1 extra larges |
| 02:18 |
Mikey |
our hardware budget has quite a bit of room for expansion though |
| 02:18 |
Mikey |
we're barely in pilot/private beta mode |
| 02:18 |
bpmedley |
That's cool. How many do you think it can scale to? |
| 02:19 |
Mikey |
well it has to scale to 40k, which I think it can, but we'll find out |
| 02:19 |
damaya |
bpmedley: I think I need to wrap my mind around what I am doing first. I have a full app, so what I was doing was sub vacation { ... my $action = $self->param('action'); } and then the call would be /vacation?action=create&user_id=... |
| 02:20 |
bpmedley |
damaya: What do you want to return from the route? |
| 02:20 |
Mikey |
personally i feel like i could scale it to a half a million concurrent users without having to rethink things |
| 02:20 |
Mikey |
half a million websockets |
| 02:20 |
Mikey |
open |
| 02:20 |
damaya |
I guess after looking at it and hearing about RESTful APIs, I figured maybe my way of doing this is a bit outdated. My API was going to be basically that, and then it returns a JSON response. |
| 02:20 |
bpmedley |
Mikey: Nice! |
| 02:20 |
Mikey |
the database increasingly will become the bottleneck |
| 02:21 |
sri |
hmmm... all_contents also seems about 3 times faster than ->find('*')->contents :o |
| 02:21 |
bpmedley |
damaya: Ok, perhaps a small Mojolicious::Lite app that just retruns some JSON? |
| 02:21 |
sri |
might be worth it |
| 02:21 |
Mikey |
and mojo/zeromq scale horizontally so well im very confident i will be able to meet most every need.. i was able to simulate a message delivery using just my weak single core/two thread app servers .. i delivered a message to 5000 active websocket recipients in 4 seconds |
| 02:22 |
damaya |
The API is to be used as part of my full app... There are many things one can do in the full app, and I wanted everything to do be done through an API, so I created lib/API.pm and I am defining API functions there. |
| 02:22 |
bpmedley |
damaya: Can you show us a small Mojolicious::Lite app that demonstrates the type of api you want? |
| 02:22 |
damaya |
Sure, I can work on that. |
| 02:23 |
bpmedley |
I wonder if you just need a different way to look at your problem and/or "talk" about it out loud. |
| 02:23 |
Mikey |
the one thing thats gonna hold me back / make me reconsider going with AWS is their lack of multicast support.. im going to saturate the NIC throughput with TCP-based zeromq pubsub |
| 02:24 |
Mikey |
the more app servers i add, the worse my throughput problem will become.. but i have cloud hosting plan-b and plan-c |
| 02:24 |
damaya |
Thanks bpmedley. |
| 02:24 |
damaya |
And yeah, that is the thing, I am stuck at this point where I am unable to really think about what I am trying to do anymore. |
| 02:25 |
bpmedley |
Ok, perhaps put up the Lite app and we can yell and scream? |
| 02:26 |
|
klapperl_ joined #mojo |
| 02:28 |
damaya |
hehe, sounds good... I need to work on it for a bit, so I can better define what I am trying to do. |
| 02:29 |
|
good_news_everyone joined #mojo |
| 02:29 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/ZKG4Hw |
| 02:29 |
good_news_everyone |
mojo/master 30e520a Sebastian Riedel: added all_contents method to Mojo::DOM |
| 02:29 |
|
good_news_everyone left #mojo |
| 02:30 |
Mikey |
when writing APIs i always find it helps to "use" them first, play around with how you'd want it to work, and then build something that gives you the interface you dreamed up |
| 02:30 |
sri |
perl -Mojo -E 'say g("reddit.com")->dom->all_contents->grep(sub { $_->node eq "comment" })' |
| 02:33 |
bpmedley |
IE6sux .. ;) Those oneliners look much better. |
| 02:44 |
sri |
one feature i kinda miss from jquery is being able to return to previous result sets |
| 02:47 |
|
ka2u joined #mojo |
| 02:47 |
|
d4rkie joined #mojo |
| 02:52 |
sri |
lol, i can actually make that work! |
| 02:56 |
sri |
haha, it's actually terrible... but here's a patch for future reference... http://pastie.org/8690220 |
| 03:07 |
|
d4rkie joined #mojo |
| 03:11 |
sri |
ok, this might be bad... i've been unable to track down the compression problem |
| 03:12 |
sri |
this is the problem when you're trying to support features that have no finished spec yet :/ |
| 03:13 |
sri |
might have to rip the whole thing out again until things stabilize |
| 03:30 |
damaya |
Is there any way to grab mojolicious parameters as a hashref? I.e., ?action=view&user_id=5 |
| 03:33 |
sri |
jberger, marcus, tempire, crab: opinions would be very welcome, i'm out of ideas for how to fix websocket compression, should i just rip it out for now? |
| 03:33 |
damaya |
my $params = Mojo::Parameters->new($self->param()); is totally wrong, and with $self->param i'm not really sure how to grab values without specifying the exact param. |
| 03:33 |
sri |
seems to fail for certain messages, might be size related |
| 03:34 |
tempire |
I haven't looked at that at all |
| 03:34 |
sri |
tempire: you don't have to, it's more about proper process |
| 03:34 |
sri |
basically, websockets fail badly with chrome atm |
| 03:35 |
tempire |
well that sounds fun |
| 03:35 |
tempire |
even without compression? |
| 03:35 |
sri |
nope |
| 03:35 |
sri |
only compression |
| 03:35 |
sri |
basically, compression fails randomly |
| 03:36 |
sri |
and it might be a bug in chrome for all we know |
| 03:36 |
sri |
the spec is only a draft... |
| 03:36 |
tempire |
sounds like a huge time sucker |
| 03:36 |
tempire |
not a fan of those |
| 03:36 |
sri |
i was a bit foolish not calling it experimental |
| 03:36 |
tempire |
hmm |
| 03:37 |
sri |
think i'll remove it and anyone who disagrees can fix it and i'll readd it right away |
| 03:37 |
damaya |
Ah, interesting... $self->req->params->to_hash; |
| 03:37 |
damaya |
sexy |
| 03:37 |
tempire |
Having it just not work mysteriously would be the devil |
| 03:38 |
tempire |
removing it without it being labeled experimental is unfortunate...but it's better than a mystery feature. |
| 03:38 |
tempire |
especially since everyone who cares about websocket compression is probably using chrome |
| 03:38 |
Mikey |
did google at least document their implementation? |
| 03:39 |
sri |
http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-17 |
| 03:39 |
sri |
it's a badly written draft |
| 03:40 |
sri |
the sorry state of Compress::Raw::Zlib sure doesn't help either :( |
| 03:42 |
Mikey |
im 100% for ripping it out because it fixes my problem, and i've already disabled it anyway. that said, mobile devices are going to be using this and i'd like to get this working right with compression |
| 03:42 |
Mikey |
so i will take a stab at fixing it |
| 03:50 |
Mikey |
you just chop off the last 4 bytes with substr? |
| 04:01 |
|
jack joined #mojo |
| 04:04 |
|
howitdo joined #mojo |
| 04:05 |
|
dvinciguerra joined #mojo |
| 04:14 |
sri |
Mikey: not quite, you also have to make sure there's an empty deflate block at the end |
| 04:14 |
sri |
but that's not the problem, i checked |
| 04:15 |
|
good_news_everyone joined #mojo |
| 04:15 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/o7GaXw |
| 04:15 |
good_news_everyone |
mojo/master e98254f Sebastian Riedel: removed support for permessage-deflate WebSocket compression, since there have been too many problems with Chrome |
| 04:15 |
|
good_news_everyone left #mojo |
| 04:17 |
sri |
https://chromium.googlesource.com/chromium/blink/+/master/Source/modules/websockets/WebSocketPerMessageDeflate.cpp # that's the file where "failed to inflate a frame" is from |
| 04:17 |
Mikey |
yeah thats where i found the error |
| 04:17 |
sri |
doesn't tell me much though |
| 04:17 |
Mikey |
thats like the only place on the web it exists |
| 04:18 |
Mikey |
i was gonna dig into their inflator code but then i thought, hey, it might be mojo's fault.. and i know perl better than i know cpp, and chrome is infinitely more vast than mojo.. little pile first. |
| 04:18 |
|
ka2u joined #mojo |
| 04:19 |
sri |
it's really not much code |
| 04:19 |
Mikey |
well i can't even get Compress::Raw::Zlib to roundtrip that code so im gonna get an understanding of that first |
| 04:19 |
sri |
my first guess was windowbits mismatch... but tried all options without luck |
| 04:19 |
sri |
oh |
| 04:21 |
sri |
well, i'm going to make a new release anyway, then we can sort it out without a hurry |
| 04:21 |
Mikey |
*nod* |
| 04:21 |
|
damaya joined #mojo |
| 04:21 |
Mikey |
i can pull my workaround out of the repo and upgrade mojo everywhere |
| 04:21 |
sri |
the context takeover stuff is wrong anyway |
| 04:21 |
Mikey |
this makes me happy. |
| 04:21 |
sri |
(doesn't matter here though) |
| 04:24 |
sri |
i expect one of two outcomes... a) i messed up the Compress::Raw::Zlib configuration or b) another Compress::Raw::Zlib bug |
| 04:24 |
Mikey |
that sounds about right.. once i get it round tripping i'll work with chrome |
| 04:26 |
|
ka2u joined #mojo |
| 04:26 |
sri |
oh, no need to hurry because of mobile devices btw., so far only desktop chrome supports it |
| 04:27 |
Mikey |
ahh hmm.. this might be moot, well if i can't solve it in the next hour or so i'll quit for now. |
| 04:27 |
Mikey |
i just wanna contribute something :) |
| 04:28 |
sri |
oh, don't mean to discourage you :o |
| 04:28 |
sri |
yay Mikey \o/ |
| 04:28 |
Mikey |
haha thanks :) |
| 04:29 |
|
good_news_everyone joined #mojo |
| 04:29 |
good_news_everyone |
[mojo] kraih tagged v4.74 at 1f75ed5: http://git.io/Jk6D-A |
| 04:29 |
|
good_news_everyone left #mojo |
| 04:30 |
sri |
ok, now at least nobody else stumbles over this |
| 04:30 |
cpan_mojo |
Mojolicious 4.74 by Sebastian Riedel - http://metacpan.org/release/SRI/Mojolicious-4.74 |
| 04:31 |
sri |
heh |
| 04:31 |
* sri |
pats cpan_mojo |
| 04:34 |
bpmedley |
I think it's because the DEFLATED $out ends with 0x0000FFFF. Therefore an empty DEFLATE block with no compression needs to be padded to $out. |
| 04:38 |
sri |
i'm no expert, but as far as i understand zlib, the Z_SYNC_FLUSH should take care of it |
| 04:38 |
Mikey |
If the parameter flush is set to Z_SYNC_FLUSH, all pending output is flushed to the output buffer and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. (In particular avail_in is zero after the call if enough output space has been provided before the call.) Flushing may degrade compression for some compression algorithms and so it should be used only when necessary. This completes the current deflate |
| 04:39 |
Mikey |
but there might be "more" data.. i think we might have filled up a buffer or something, we might have to hit this many times |
| 04:39 |
Mikey |
474648 bytes compressing down to 8975 bytes feels "too" good |
| 04:45 |
sri |
hmm... it's true, i can't even get it to roundtrip |
| 04:45 |
sri |
inflate returns data error |
| 04:45 |
Mikey |
yeah that's what im getting, too |
| 04:46 |
Mikey |
and im noticing this stupid library isn't returning any of the numeric answers i expect, it's returning undef which == Z_OK but only because undef == 0 |
| 04:47 |
* Mikey |
likes when C bindings are as close to the library they're exposing as possible. |
| 04:47 |
sri |
yea, it's silly |
| 04:47 |
Mikey |
cos that way you get the benefit of all the underlying library's documentation |
| 04:47 |
sri |
most return values are actually untested |
| 04:48 |
sri |
https://rt.cpan.org/Public/Bug/Display.html?id=92521 |
| 04:48 |
sri |
i just reported that a few days ago |
| 04:49 |
Mikey |
at least they replied |
| 04:49 |
sri |
also funny, if i change the message to "$message = 'a' x 474648;" it works |
| 04:49 |
Mikey |
good thing i didn't just make some test data eh ;) |
| 04:50 |
sri |
oh, i actually never got a notification for that reply :o |
| 04:51 |
Mikey |
tweaking memlevel to 9 makes the compressed output substantially larger |
| 04:51 |
Mikey |
18607 bytes.. up from 8971 .. still doesn't round trip though. |
| 04:54 |
sri |
$message = join '', map { int rand(9) } 1 .. 474648; |
| 04:54 |
sri |
that seems to fail too |
| 04:54 |
Mikey |
enough entropy to not compress small enough is my guess |
| 04:54 |
sri |
ye |
| 04:58 |
sri |
http://pastie.org/8690386 # a small test case |
| 04:58 |
sri |
in case anyone else wants to join the fun ;p |
| 04:59 |
Mikey |
i got it working with Compress::Zlib |
| 04:59 |
Mikey |
474648 |
| 04:59 |
Mikey |
135947 |
| 04:59 |
Mikey |
stream end |
| 04:59 |
Mikey |
474648 |
| 04:59 |
Mikey |
round tripping anyway |
| 04:59 |
sri |
do tell |
| 04:59 |
Mikey |
sec |
| 05:00 |
Mikey |
http://pastie.org/8690388 |
| 05:01 |
Mikey |
ripped out my huge json structure and replaced with your random bytes |
| 05:01 |
sri |
wait what |
| 05:01 |
purl |
there's more! |
| 05:01 |
Mikey |
lemme put it into mojo and see if it works |
| 05:02 |
sri |
hmm... no Z_SYNC_FLUSH though |
| 05:02 |
Mikey |
that should work just fine i think |
| 05:02 |
Mikey |
about to test |
| 05:04 |
sri |
the limit is also quite important |
| 05:04 |
purl |
okay, sri. |
| 05:04 |
sri |
otherwise the server can get zip bombed |
| 05:08 |
davido_ |
If only it were raptors... http://www.thingiverse.com/errorik |
| 05:10 |
Mikey |
well i can say using Compress::Zlib at least works in chrome |
| 05:11 |
Mikey |
how does that limit work? |
| 05:11 |
Mikey |
ahh i see its an output limit |
| 05:15 |
Mikey |
hmm Compress::Zlib just wraps Compress::Raw::Zlib... maybe i just need to use it how its using it |
| 05:16 |
Mikey |
sri: i think i know what's happening |
| 05:16 |
Mikey |
flush() appends more to $out.... so deflate() is setting the value of $out, and then you're wiping out everything that was there with the flush() |
| 05:17 |
Mikey |
clobbered |
| 05:17 |
Mikey |
testing my theory |
| 05:17 |
Mikey |
yep ;){ |
| 05:17 |
Mikey |
$deflate->deflate(\$frame->[5], my $out); |
| 05:17 |
Mikey |
$deflate->flush(my $rest, Z_SYNC_FLUSH); |
| 05:17 |
Mikey |
$out .= $rest; |
| 05:18 |
Mikey |
that works. |
| 05:18 |
Mikey |
not even worth a pull request. |
| 05:19 |
Mikey |
thats without using Compress::Zlib, btw.. still using Raw |
| 05:20 |
bpmedley |
Compress::Raw::Zlib::Deflate->new(WindowBits => -15, MemLevel => 8, AppendOutput => 1); <-- Another option |
| 05:21 |
Mikey |
yeah that's probably the better option |
| 05:21 |
Mikey |
saves yourself a variable |
| 05:22 |
Mikey |
yep that works, too |
| 05:23 |
Mikey |
now the question is.. why did it always work on your setup, bpmedley? |
| 05:23 |
bpmedley |
I had a mistake. There were two versions of Mojolicious installed and I was using an old one. |
| 05:25 |
sri |
oh noes! |
| 05:25 |
sri |
Mikey++ |
| 05:25 |
Mikey |
yay i contributed!!! (to a feature that got yanked out) |
| 05:26 |
Mikey |
well, i will enjoy the superbowl extra tomorrow. this has been fun. |
| 05:26 |
sri |
hmmm.... i said i would readd it immediately if someone fixed it :S |
| 05:27 |
Mikey |
well then! i contributed!!! to a feature that will be readded immediately |
| 05:27 |
Mikey |
haha feelsgoodman |
| 05:27 |
sri |
you are mgregoro? |
| 05:45 |
Mikey |
yes thats me |
| 05:46 |
Mikey |
on cpan MGREGORO, too |
| 05:53 |
sri |
ok, if i bring it back i'll make a few small changes, like disabling compression in the user agent by default |
| 05:53 |
sri |
a little more defensive |
| 05:53 |
Mikey |
and i will keep my eye on it obviously, i'm using it in production, and will have many different types of users using it over the next few months |
| 05:54 |
Mikey |
like i was saying before 30-40k concurrent websockets come this fall |
| 05:54 |
Mikey |
with that in mind it's probably a good thing that i dipped my toe into the internals before then |
| 05:55 |
* sri |
wonders if Compress::Raw::Zlib is the right module to use |
| 05:55 |
sri |
only picked it because the documentation made me want to poke my eyes out the least |
| 05:56 |
Mikey |
i think it's probably the best perl module that exists today for this problem |
| 05:59 |
Mikey |
can go even lower level and just use IO::Compress::Deflate |
| 06:03 |
Mikey |
i hate flimsy object oriented wrappers around procedural c libraries. it looks like none of that suite lets you get away from that |
| 06:03 |
sri |
an actual unit test would have been nice ;p |
| 06:03 |
Mikey |
ahh sorry haha |
| 06:03 |
sri |
wonder if it happens with smaller messages too |
| 06:04 |
sri |
something that actually roundtrips |
| 06:04 |
Mikey |
it seemed to work fine with smaller messages.. the overhead for encrypted messages is pretty large, it increases them by about 200%, and we had a really long conversation last night in encrypted messages |
| 06:05 |
Mikey |
so, it was about 3x as large as anything i've ever passed through a websocket |
| 06:16 |
|
ka2u joined #mojo |
| 06:34 |
|
good_news_everyone joined #mojo |
| 06:34 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/KX9zqg |
| 06:34 |
good_news_everyone |
mojo/master aeee9e8 Sebastian Riedel: fixed and readded support for permessage-deflate WebSocket compression |
| 06:34 |
|
good_news_everyone left #mojo |
| 06:34 |
sri |
Mikey: please test |
| 06:35 |
sri |
on the client side it's now opt-in, that means we can skip the context takeover mess for now |
| 06:35 |
sri |
this way the implementation got really simple |
| 06:36 |
sri |
tried to keep everything a bit isolated, in case we have to remove again |
| 06:38 |
|
good_news_everyone joined #mojo |
| 06:38 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/tKJYLg |
| 06:38 |
good_news_everyone |
mojo/master 5ebab41 Sebastian Riedel: better compression example |
| 06:38 |
|
good_news_everyone left #mojo |
| 06:48 |
sri |
looks like i have a working test case |
| 06:48 |
sri |
but it depends on random data :S |
| 06:51 |
|
good_news_everyone joined #mojo |
| 06:51 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/NQkcvw |
| 06:51 |
good_news_everyone |
mojo/master eb127ea Sebastian Riedel: test compression edge case |
| 06:51 |
|
good_news_everyone left #mojo |
| 06:52 |
sri |
it won't fail randomly, but might pass when it shouldn't if stars align |
| 06:53 |
|
good_news_everyone joined #mojo |
| 06:53 |
good_news_everyone |
[mojo] kraih tagged v4.75 at 80e9265: http://git.io/QgvxQg |
| 06:53 |
|
good_news_everyone left #mojo |
| 06:54 |
sri |
phew |
| 06:55 |
cpan_mojo |
Mojolicious 4.75 by Sebastian Riedel - http://metacpan.org/release/SRI/Mojolicious-4.75 |
| 06:59 |
* sri |
crashes |
| 06:59 |
|
damaya joined #mojo |
| 07:08 |
damaya |
If I have the directory structure lib/API.pm and lib/API/Blah.pm, how exactly do I call things that are in Blah? I tried: $r->get('/api/vacation/retrieve')->to( namespace => 'Calm::API', action => 'vacation#retrieve' ); but for some reason that does not work. |
| 07:09 |
damaya |
I just get "Page not found yet" and no errors in my terminal (running Morbo) |
| 07:11 |
damaya |
Oh, interesting... I guess if I call it with /api/vacation/retrieve it works fine, but not with /api/vacation/retrieve?param=blah |
| 07:12 |
mikegrb |
hah, was just pulling up some code to compare |
| 07:13 |
damaya |
I don't get with RESTful apis how to allow parameters to go anywhere... Would I actually have to set up a route '/api/vacation/retrieve/#user_id/#user_email... etc? |
| 07:14 |
damaya |
What about optional params? I think I may just write this API my old school way (with params) rather than trying to figure out this RESTful stuff :) |
| 07:14 |
damaya |
mikegrb: Thanks :) |
| 07:14 |
damaya |
I appreciate anyone trying to help me |
| 07:15 |
mikegrb |
yeah you would set routes similiar to those as well |
| 07:16 |
mikegrb |
for example I Have: |
| 07:16 |
mikegrb |
$r->get('/notices/')->to( 'PublicNotices#index', active_tab => 4 ); |
| 07:16 |
mikegrb |
$r->get('/notices/:page')->to( 'PublicNotices#index', active_tab => 4 ); |
| 07:17 |
damaya |
Interesting... What if there are multiple params (e.g., 10), and they are optional, and you don't want to enforce any order? That's the part I am having a hard time understanding. |
| 07:17 |
mikegrb |
yeah |
| 07:17 |
mikegrb |
then it gets tricky :p |
| 07:17 |
damaya |
I've looked at some of the examples, and most are similar to what you have. Unfortunately, I haven't found on yet that has a bunch of params. |
| 07:17 |
damaya |
Heh |
| 07:17 |
mikegrb |
I'm no mojo expert either |
| 07:18 |
damaya |
http://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=Main%20Page |
| 07:18 |
damaya |
Like that, for example. I don't get how to do that with these routes :D |
| 07:18 |
mikegrb |
I actually popped over to this window tonight to ask a questsion myself but just missed sri |
| 07:18 |
damaya |
Ah, well if I tried to help I'd probably just mislead ya. |
| 07:18 |
damaya |
heh |
| 07:19 |
mikegrb |
I think generally what you want is to put the fundamental non optional stuff in the path part of the address, operation, entity id, etc |
| 07:20 |
mikegrb |
and all the optional arguments as url params or so |
| 07:20 |
mikegrb |
then you don't have to worry about order there |
| 07:21 |
mikegrb |
normally you are using a post for entity creation where you have a lot of args |
| 07:22 |
mikegrb |
a restful url for the wikipedia example might look like http://en.wikipedia.org/api/revisions/Main%20Page.xml |
| 07:22 |
damaya |
Ah, that's helpful |
| 07:22 |
damaya |
I've been reading about REST all night |
| 07:23 |
mikegrb |
it's slightly different way to think about constructing urls |
| 07:24 |
damaya |
It's more than that though, right? I mean, a GET query returns something to identify the result, and then you can do methods (PUT, DELETE, etc) on what was returned rather easily? |
| 07:24 |
mikegrb |
though I guess back in the day we didn't think about constructing urls at all, the url was just for the cgi script itself |
| 07:24 |
damaya |
That is what I am seeming to understand of it. |
| 07:24 |
mikegrb |
yes that's correct |
| 07:24 |
damaya |
So rather than having the URL inform the backend of what to do, the URL is actually performing the action (somewhat)... |
| 07:25 |
damaya |
I think I get it, just need to wrap my mind around implementing it :D |
| 07:25 |
damaya |
Your example of the Wikipedia URL is actually really helpful. |
| 07:25 |
damaya |
Thanks a lot mikegrb! |
| 07:25 |
mikegrb |
instead of instead of http://some.api.com/urmom.cgi?action=delete&type=page&title=foo you do an HTTP DELETE request against http://some.api.com/page/foo |
| 07:33 |
|
Vandal joined #mojo |
| 07:48 |
|
LiveLover joined #mojo |
| 07:57 |
|
rem_lex joined #mojo |
| 08:06 |
|
jberger joined #mojo |
| 08:22 |
jberger |
sri++ Mikey++ |
| 08:28 |
cpan_mojo |
MojoX-JSON-RPC 0.06 by Henry Tang - http://metacpan.org/release/HENRYYKT/MojoX-JSON-RPC-0.06 |
| 09:16 |
cpan_mojo |
Test-WWW-Mechanize-Mojo v0.0.16 by Shlomi Fish - http://metacpan.org/release/SHLOMIF/Test-WWW-Mechanize-Mojo-v0.0.16 |
| 09:27 |
|
jack joined #mojo |
| 09:30 |
|
zivester joined #mojo |
| 10:02 |
|
basiliscos joined #mojo |
| 10:03 |
|
trone joined #mojo |
| 10:16 |
|
denis_boyun joined #mojo |
| 10:29 |
|
KindTwo joined #mojo |
| 10:34 |
|
plenum joined #mojo |
| 10:47 |
|
abra joined #mojo |
| 10:59 |
|
arpadszasz_ joined #mojo |
| 11:27 |
|
sh4 joined #mojo |
| 11:30 |
|
denis_boyun joined #mojo |
| 12:06 |
|
bowtie joined #mojo |
| 12:11 |
|
odc joined #mojo |
| 12:19 |
|
zM joined #mojo |
| 13:03 |
|
denis_boyun joined #mojo |
| 13:04 |
|
dvinciguerra joined #mojo |
| 13:18 |
|
bowtie_ joined #mojo |
| 13:52 |
|
punter joined #mojo |
| 14:27 |
|
ryozi joined #mojo |
| 14:31 |
|
denis_boyun joined #mojo |
| 14:37 |
|
mire__ joined #mojo |
| 14:51 |
|
d4rkie joined #mojo |
| 14:54 |
|
gryphon joined #mojo |
| 15:02 |
|
denis_boyun_ joined #mojo |
| 15:07 |
|
d4rkie joined #mojo |
| 15:09 |
|
ka2u joined #mojo |
| 15:14 |
|
KindOne joined #mojo |
| 15:27 |
* sri |
was a bit worried about all_contents, but it's really worth it |
| 15:28 |
sri |
in case you missed it over the whole permessage-deflate thing :) https://github.com/kraih/mojo/commit/30e520aeff3421abbc04c0fbb376999fee3a0b46 |
| 15:35 |
sri |
perl -Mojo -E 'my $dom = g("www.reddit.com")->dom; $dom->all_contents->grep(sub { $_->node eq "text" && $_->content =~ /^\s+$/ && $_->remove }); say $dom' |
| 15:36 |
sri |
fun stuff, this one scrubs all useless whitespace |
| 15:38 |
nicomen |
+1 |
| 15:38 |
purl |
1 |
| 15:45 |
nicomen |
http://streaming.fosdem.org/ <-- wasn't there a perl track there? |
| 16:49 |
|
denis_boyun joined #mojo |
| 17:05 |
|
ka2u joined #mojo |
| 17:19 |
|
KindOne joined #mojo |
| 17:22 |
|
zackiv31 joined #mojo |
| 17:27 |
|
jberger__ joined #mojo |
| 17:45 |
|
jberger joined #mojo |
| 17:47 |
sri |
hmmm |
| 17:47 |
sri |
$dom->all_contents('comment')->remove; |
| 17:47 |
sri |
$dom->all_contents('comment', 'text')->remove; |
| 17:50 |
sri |
might be going a bit too far now :) |
| 17:57 |
|
btyler joined #mojo |
| 18:04 |
Mikey |
yay! 4.75 in production, took out my compression workaround, everything's working great! |
| 18:05 |
sri |
\o/ |
| 18:05 |
sri |
is compression worth it in your app? |
| 18:05 |
Mikey |
now an afternoon of cooking chili and fixing nasty bugs |
| 18:05 |
Mikey |
yes. definitely visibly faster. |
| 18:06 |
Mikey |
chrome always stomped firefox performance wise and now i know why |
| 18:06 |
sri |
mmmmmm... chilli... *drools* |
| 18:06 |
Mikey |
with infinite scrolling we pull more messages from the websocket, and on chrome it's lightspeed |
| 18:06 |
Mikey |
on firefox you can see it loading one message at a time |
| 18:06 |
Mikey |
there are obviously other factors but i definitely could tell the difference |
| 18:07 |
sri |
interesting, i was expecting it to be less visible, considering that you said your data was encrypted, which shouldn't be very compressible |
| 18:07 |
|
mgrimes joined #mojo |
| 18:07 |
Mikey |
it's all base64 encoded |
| 18:08 |
Mikey |
verry compressible. |
| 18:08 |
sri |
ooh :) |
| 18:08 |
Mikey |
that message i gave you was "encrypted" and it shrunk down well over 50% |
| 18:08 |
Mikey |
and most messages in the system are not encrypted at all |
| 18:10 |
|
dod joined #mojo |
| 18:18 |
|
dod joined #mojo |
| 18:26 |
|
zivester joined #mojo |
| 18:27 |
|
good_news_everyone joined #mojo |
| 18:27 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/ylEnhQ |
| 18:27 |
good_news_everyone |
mojo/master b43ada2 Sebastian Riedel: mention how to activate permessage-deflate compression in user agent |
| 18:27 |
|
good_news_everyone left #mojo |
| 18:34 |
nicomen |
Mikey: are you saying that the reading of websocket data itself is much slower in FF than Chrome? |
| 18:39 |
Mikey |
i haven't timed just that, i'm saying that on the whole chrome was noticeably slower with compression off than with compression on (anecdotally) |
| 18:39 |
Mikey |
and i'm also saying that on the whole chrome handles my js-heavy web app much better than FF does. |
| 18:40 |
Mikey |
some of this is obviously my fault! however, we're supposed to be living in the post-browser-compatibility age. |
| 18:45 |
nicomen |
hm, ok |
| 18:50 |
|
jack_ joined #mojo |
| 19:22 |
|
ka2u joined #mojo |
| 19:43 |
|
good_news_everyone joined #mojo |
| 19:43 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/ZBoTAw |
| 19:43 |
good_news_everyone |
mojo/master 70f4911 Sebastian Riedel: show how to disable gzip compression |
| 19:43 |
|
good_news_everyone left #mojo |
| 19:49 |
|
mojobot23379 joined #mojo |
| 20:03 |
|
jberger joined #mojo |
| 20:15 |
|
good_news_everyone joined #mojo |
| 20:15 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/gUBGyQ |
| 20:15 |
good_news_everyone |
mojo/master 1d5d6b9 Sebastian Riedel: more realistic user agent examples |
| 20:15 |
|
good_news_everyone left #mojo |
| 20:26 |
bpmedley |
damaya: Did you get your api interface working? |
| 20:28 |
sri |
neat, the signatures patch is still growing https://github.com/Perl/perl5/commit/152e657b639920a69db529cce93d5b0cbb7dec55#diff-9e7cc5aeaa35c709063fd1031d94555bR405 |
| 20:44 |
|
lukep joined #mojo |
| 20:55 |
|
mire__ joined #mojo |
| 20:57 |
|
maxhq joined #mojo |
| 21:00 |
|
maxhq joined #mojo |
| 21:06 |
|
zM joined #mojo |
| 21:14 |
sri |
hmm... $dom->all_contents('comment') is not even an extra line of code |
| 21:17 |
sri |
http://pastie.org/8692338 |
| 21:43 |
damaya |
bpmedley, not really, still reading on REST. This is what the API looks like now: http://pastebin.com/fTzyPADi |
| 21:44 |
damaya |
I realize that my way of doing it is a bit 'old school', so trying to make this use RESTful routes. Still trying to figure out multiple params, optional params, stuff like that though. |
| 21:46 |
damaya |
I'm having fun figuring this stuff out though :) |
| 21:58 |
|
punter joined #mojo |
| 22:25 |
|
thowe joined #mojo |
| 22:29 |
|
damaya joined #mojo |
| 22:31 |
thowe |
Hi |
| 22:34 |
jberger |
damaya++ # "I'm having fun figuring this stuff out though :)" |
| 22:37 |
damaya |
:) |
| 23:34 |
|
good_news_everyone joined #mojo |
| 23:34 |
good_news_everyone |
[mojo] kraih pushed 1 new commit to master: http://git.io/MxrUrA |
| 23:34 |
good_news_everyone |
mojo/master 438ec4a Sebastian Riedel: speed up collecting of nodes |
| 23:34 |
|
good_news_everyone left #mojo |
| 23:40 |
|
btyler joined #mojo |
