The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2014-11-01

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 tempire oh hey
00:03 tempire I completely forgot you were in chicago
00:03 tempire I've been here for a week
00:04 * jberger shakes tempire
00:04 jberger where at?
00:04 tempire lakeview
00:04 jberger oh so actually Chicago then
00:04 jberger found any trixies?
00:05 jberger how long are you here?
00:05 tempire I don't know what that means. But I'm going to say yes.
00:05 tempire I've been here since last Thursday, leaving tomorrow afternoon
00:05 jberger oh the lunches we could have had!
00:06 jberger I work downtown
00:06 jberger I live out in the burbs
00:07 jberger tempire: did you like the ice/snow/wind today?
00:07 tempire ridiculous
00:07 tempire especially since on monday, it was a little too warm for jeans
00:08 jberger And from Lakeview maybe you saw the lake today too?
00:08 tempire It was hailing for a bit
00:08 jberger http://breakingnews.suntimes.com/transportation/waves-force-closure-of-bike-path-slow-going-on-lake-shore-drive/
00:08 jberger That's Chicago weather for you
00:09 tempire crazy
00:14 jberger tempire: work or play?
00:14 ignacio_ joined #mojo
00:14 jberger I'm sure you found plenty of dancing on the north side either way
00:15 tempire I don't differentiate between the two
00:18 jberger guide to Chicago:
00:18 jberger fun = north side
00:18 jberger get shot =  south and west sides
00:18 jberger be wet = "east side"
00:19 jberger And then there's the suburbs
00:22 sujithm joined #mojo
00:24 jberger might be calling it a night at 155
00:34 KCL_ joined #mojo
00:35 jberger a few more 160
00:35 jberger turning out the lights
00:37 d4rkie joined #mojo
00:44 ua joined #mojo
01:30 D4RK-PH0ENiX joined #mojo
01:32 hoppie joined #mojo
01:37 tbushell joined #mojo
01:57 sri and now the channel is all dark :(
02:26 KCL joined #mojo
02:35 klapperl joined #mojo
02:41 tempire woooOOOOoooooo
02:56 saki scare very much
03:01 * sri trips and falls down the stairs
03:11 noganex joined #mojo
03:20 d4rkie joined #mojo
03:29 tempire so scare. much ghost.
03:29 mst sri: you have been saved from the terrible secret of space
03:31 preaction we are here to protect you
03:51 jberger BOO!
04:16 basic6_ joined #mojo
04:34 doublelel joined #mojo
04:45 irq joined #mojo
05:33 Eke- joined #mojo
05:37 rem_lex|pivo joined #mojo
06:03 preaction someone is building a mongodb-like API on top of postgres, so a mongo driver could be used for postgres: https://github.com/torodb/torodb
06:05 sujithm joined #mojo
07:33 jamesaxl joined #mojo
07:44 sh4 joined #mojo
08:18 Vandal joined #mojo
08:21 sh4|2 joined #mojo
08:33 dod joined #mojo
09:25 basiliscos joined #mojo
09:48 vytas joined #mojo
10:24 * marcus is trying to cartonify iusethis.
10:24 marcus SO.MANY.DEPS
10:28 neyasov joined #mojo
12:02 marty_ joined #mojo
12:05 xevz joined #mojo
12:12 d4rkie joined #mojo
12:26 bjoernfan joined #mojo
12:32 d4rkie joined #mojo
12:32 D4RK-PH0ENiX joined #mojo
12:38 neyasov joined #mojo
12:54 rawler joined #mojo
12:55 xevz joined #mojo
12:55 jberger marcus: I read that as "marcus is trying to cartoonify iusethis"
12:55 jberger for the record I support both endevors
13:23 basiliscos joined #mojo
13:39 jamesaxl joined #mojo
13:43 marmez joined #mojo
13:56 marty joined #mojo
14:18 woz joined #mojo
14:22 stryx` joined #mojo
14:26 KCL_ joined #mojo
14:43 * batman is working on new ui for convos. fun-fun-fun :)
14:43 batman http://home.thorsen.pm/private/raw/notverysecret/convos/1-convos-black-sidebar.png
14:43 batman http://home.thorsen.pm/private/raw/notverysecret/convos/2-convos-black-sidebar-goto-anything.png
14:43 batman http://home.thorsen.pm/private/raw/notverysecret/convos/3-convos-black-sidebar-widescreen.png
14:45 batman Ooops! not in #convos :(
14:45 batman sorry
14:57 jamesaxl joined #mojo
15:01 marcus joined #mojo
15:26 sri i guess a pirate cloud would work :) http://i.imgur.com/VhTMsek.png
15:26 sri doesn't even need the 8-bit treatment
15:29 hoppie joined #mojo
15:30 sri just not sure where to put the text http://i.imgur.com/sTm3p6f.png
15:32 hoppie Hi. Getting a lot of "Premature connection close" errors. I got a tip from jberger, "... guessing you are garbage collecting your transaction objects too early. It's a common nonblocking bug." http://pastie.org/9689667
15:33 hoppie I am doing a write up showcasing LWP, AnyEvent and Mojolicious. In addition showcasing MCE + (all three, LWP, AnyEvent, Mojolicious).
15:34 sri http://mojolicio.us/perldoc/Mojolicious/Guides/FAQ#What-does-Premature-connection-close-mean
15:34 hoppie Thank you sri.
15:34 marcus sri: maybe under the skull?
15:34 marcus under the bones even
15:36 hoppie but not getting that from AnyEvent and LWP; these are passing.
15:36 mst hoppie: please showcase Mojolicious combined with POE and/or IO::Async instead
15:36 mst hoppie: AnyEvent considered hostile to CPAN
15:37 hoppie I know... yes...  I am doing some demos demonstrating MCE + various modules
15:37 hoppie I knew I couldn't just demo AnyEvent alone.
15:37 mst sure. but after the AnyEvent author started adding die() calls to his code to stop you using modules he didn't like
15:37 mst I don't like AnyEvent being demo'ed at all :)
15:38 mst I prefer things that actually work with CPAN
15:38 good_news_everyon joined #mojo
15:38 good_news_everyon [mojo] kraih pushed 1 new commit to master: http://git.io/zfIGfw
15:38 good_news_everyon mojo/master 0aaaae7 Sebastian Riedel: the user agent can close the connection too
15:38 good_news_everyon left #mojo
15:38 sri the faq answer was a bit too specific
15:39 mst and if I'm going to be stuck on a desert island *without* cpan, I'd probably ask for my luxury to be Mojolicious because frankly sri's code is more pleasant to work with
15:39 sri oops... i'm a doofus
15:41 good_news_everyon joined #mojo
15:41 good_news_everyon [mojo] kraih pushed 1 new commit to master: http://git.io/DUbUig
15:41 good_news_everyon mojo/master fba99a9 Sebastian Riedel: if the user agent closed the connection there would be a different error message
15:41 good_news_everyon left #mojo
15:42 hoppie The MCE + Mojo::IODelay passes more often though. http://pastie.org/9689681   getting less "Premature connection close" errors -- often zero
15:44 hoppie I have a list of URLs (291 entries).  Not sure about pasting that out there. Randomly picked.
15:45 sri umm, is your box atually capable of handling 291 concurrent sockets?
15:46 sri once you go past 100 things need to be tuned
15:46 marty joined #mojo
15:46 hoppie Yes, I think so. Will check. Running on CentOS 7. btw, this is passing with LWP and AnyEvent.  The MCE code runs all simultaneously (each worker 40 connections).
15:47 jberger sri: I love the design!
15:48 jberger can absolutely work as is
15:48 hoppie Leaving out LWP -- that is serialized. But AnyEvent is running. Wanted so badly for the Mojo example to work -- because I love Mojo -- really like it.
15:48 jberger I'd you want to play up the pirate, you might add an eye patch to the cloud/head
15:52 hoppie Ok, I tried with a list containing just 80 entries. Still failing -- many Premature connection close errors.
15:52 hoppie humm. This seems like a good test case -- the list of 80 entires.  Another list of 80 entries (different) is passing.
15:56 hoppie The list (80 entires) currently failing with http://pastie.org/9689667 (has many premature connection close errors) is passing with http://pastie.org/9689681. Was expecting for the code without MCE to work as well.
15:57 sri jberger/marcus/tempire: i think above will work :) http://i.imgur.com/ivoOspm.png
15:57 sri jberger: still needs a lot of polishing!
15:57 jberger yeah, that looks good, adding an eyepatch would be too much
15:57 * jberger WANTS
15:58 sri hmm, cloud with an eyepatch, i can work with that!
16:02 hoppie jberger: I sent an email containing the list of 80 URLs.
16:03 hoppie The test case is reproducible with that list of URLs.
16:11 tbushell left #mojo
16:11 hoppie btw, the output should only have 2 errors (intentionally), not the "Premature connection close" errors. I will remove the 2 bad URLs and see if the premature errors goes away.
16:14 hoppie same thing. I have run out of options.
16:15 mst hoppie: have you tried stracing it?
16:16 mst that should show you the C-level syscalls on the sockets
16:16 mst which might provide enlightenment
16:16 mst that'd probably be my next step of debugging for something like this
16:17 hoppie ok, will capture the strace output. first will try to minimize the list further more.
16:19 hoppie I'm doing this to pass knowledge to a person who created slides for parallelizing network related. He had left out Mojolicious. Currently, at this time in my life, not doing any Perl at the moment. Simply want to showcase Mojo to someone.
16:19 hoppie Doing this for the good -- :)
16:19 mst well, the strace part is straight up unix debugging
16:21 hoppie I came across this presentation the other day. http://www.openfusion.net/talks/pwp/   I felt the need inside to try out Mojolicious and pass on the knowledge to this fellow.
16:22 hoppie :)
16:22 jberger when I start to get tons of requests, I use this pattern: https://gist.github.com/jberger/5153008
16:22 jberger not sure if its great, but it at least lets me limit things
16:22 jberger I'm adapting it for your example
16:23 jberger it seems that there are lots of Can't connect: Name or service not known
16:23 jberger which I'm guessing the the resolver, which is blocking
16:24 hoppie Interesting. Should only have two errors (2 bad URLs intentionally).
16:24 jberger so it makes sense why MCE would be beneficial, forking off workers to do the resolving is essentially how non-blocking resolvers work
16:24 jberger I'm still using the first list
16:26 hoppie yes, lots of errors from the first list (mojou does poorly with that one). Will run again to give you the intentional failure count. Some are met to fail. This is to test the Module actually.
16:27 hoppie The 2 bad URLs if from the 2nd list (80 entries) sent via email.
16:28 jberger right, but remember that IOLoops are single threaded, so if you do something blocking (or many things blocking) you step closer and closer to timing things out
16:28 jberger http://pastie.org/9689740
16:29 hoppie the first list (191 entries) has 7 errors, should not be more.
16:30 hoppie trying the pastie code.
16:32 hoppie thank you jberger. Had no idea the trouble I was getting myself in when trying to inspire the instructor who left out Mojolicious in his presentation. :)
16:33 jberger its not a fruitless persuit, but firing off everything at once can lead to strange behavior sometimes
16:33 woz joined #mojo
16:34 mst hoppie: if people had any idea the trouble they were getting into
16:34 mst hoppie: a lot less cool code would get written :D
16:34 hoppie :)
16:34 jberger I have shown off that pattern a few times and dotan even mentioned it in his mojoconf talk
16:35 hoppie jberger, it is passing for the 80 list. Will run now with the 191 list.
16:35 jberger should I finally put it onto cpan?
16:35 jberger the original pastie shows its power, you can easily add urls to the queue, make it easy to build a spider
16:35 hoppie Use of uninitialized value in hash element at /usr/local/share/perl5/Mojo/Reactor/Poll.pm line 46, <> line 191.
16:35 basic6 joined #mojo
16:35 jberger sorry original gist
16:36 jberger hoppie: yeah, actually I got that once too
16:36 hoppie Hurray... has 6 errors (<= 7).  Passing.
16:36 hoppie for the 191 list
16:37 jberger so now if you rerun, and make "workers" like 200 you should get the same original result
16:37 jberger breaking
16:37 hoppie Will create one MCE + your code.  Will obviously study it too to see.
16:39 hoppie Thank you for your time. I really appreciate it. The last thing I wanted to do is give the wrong code snippet to the author of the presentation.
16:39 jberger hoppie: there is nothing "wrong" with your original snippet, its just that firing off that many connections sometime is unusal
16:40 jberger especially when some of them don't resolve
16:41 jberger what I call "workers" is really connections
16:41 hoppie Oh, not unusual at all. There are plenty of use case for apps like this. Will take a look at Mojo::SNMP.
16:41 jberger or "simultaneous requests"
16:42 jberger not unusual, but might give unusual results by virtue of pure scale
16:42 denis_boyun joined #mojo
16:42 hoppie At my previous job, we had a server sustaining 24 for CPU load collecting metrics from 20,000+ devices (a cable company).
16:42 hoppie Oops, not 20,000 instead 20 million+
16:43 jberger right, but you don't have a single thread maintaining those connections
16:43 batman hoppie: what about Mojo::SNMP? (i made that)
16:43 hoppie no, one could do this with MCE + Mojo::SNMP
16:43 jberger sure, forking makes this easier
16:43 sri if you know what you're doing maintaining 20k connections in mojolicious is no problem at all
16:44 hoppie not tested.  At the time we did it with MCE + AnyEvent::SNMP.   I think I ran into the same problem when trying Mojolicious before.
16:44 jberger sri: while that is true, part of his test includes sites that don't resolve and quite clearly they block while trying
16:45 jberger that causes other requests to timeout
16:45 hoppie The chunking nature of MCE is beneficial (100 workers, chunk_size set to 300; 30k simultaneous).
16:45 jberger hoppie: indeed
16:45 sri jberger: right, that's where the "know what you're doing" part comes into play
16:45 jberger :)
16:46 hoppie :) but shouldn't it Mojo handle that gracefully behind the scene with the initial example posted on pastie.
16:46 sri AnyEvent uses a shitty non-blocking pure-DNS implementation, that helps with timeouts, but gets you in trouble with mdns and the like
16:46 hoppie got it.
16:46 hoppie It seems like a good test case though.
16:47 mst and IO::Async has a more complicated approach that I believe works, but is more more complexity than Mojo core wants to take
16:47 sri i disagree actually
16:47 mst but, realistically, concurrency isn't magic
16:47 mst if you aren't restricting how many things are happening at once
16:47 mst then the framework shouldn't save you, because that's just going to make you crash much more horribly later when the design has already calcified
16:48 sri you don't crawl the web with user agents
16:48 mst whipuptitude is good, but not at the expense of manipulexity
16:48 sri normally you would have a dedicated resolver process/cluster
16:53 sri mst: i'm not convinced that the IO::Async solution works portably
16:54 sri jberger has been working on reliable background processes for ages and there's still problems
16:54 hoppie jberger and friends; if including the example by jberger with Mojolicious, there's a warning.
16:54 hoppie Use of uninitialized value in hash element at /usr/local/share/perl5/Mojo/Reactor/Poll.pm line 46, <> line 191.
16:56 sri in fact, i remember mentioning that i would add background process support to Mojo::IOLoop if someone can come up with a truly portable solution :)
16:57 jberger hoppie: IO::Poll must be reporting a ready handle that isn't
16:57 sri hoppie: mojo version?
16:58 hoppie Mojolicious-5.56.tar.gz
16:58 jberger hoppie: can you narrow it down to a specific url in the list?
16:58 jberger I got that error one time too
16:59 hoppie sure, will do.
17:01 hoppie Looking back, I got the warning twice against the list.191.
17:03 bpmedley Are the lists available in a pastie?
17:04 hoppie no, not sure if a good idea to post (although randomly picked URLs). anything on pastie is searchable.
17:05 hoppie It is hard to narrow down the particular URL. I got 3 warnings this time.
17:06 jberger unfortunately, I'm about out of time atm
17:06 hoppie thanks again
17:06 jberger glad to help
17:08 hoppie bpmedley, I can email the 2 list it if you want
17:09 jberger sri: for the record, the last problem with portable background processes is a longstanding perl bug, of which the author of the original code has passed away and no one else seems to understand
17:09 jberger :(
17:09 bpmedley Gaw, what’s the bug?
17:10 jberger I tried to send an email to p5p offering a $100 reward for fixing it, but google groups ate the email and I haven't sat down to rewrite
17:10 jberger bpmedley: https://github.com/jberger/Mojo-IOLoop-ForkCall/issues/5
17:11 jberger the "see also" isn't relevant, I have removed all the code that tripped that particular beast
17:11 jberger but it is worth knowing
17:11 jberger *Note also
17:14 hoppie Not able to reproduce the warning with just a handful of URLs going through the entire list. However, able to reproduce the warning with a list containing 50 entries.
17:15 hoppie well, take it back. hard to reproduce this.
17:15 hoppie needs the list containing 191 entries.
17:16 hoppie jberger has the list. I will be glad to email this list if someone wants it.
17:16 bpmedley hoppie: I sent a /msg asking for the list
17:16 inokenty left #mojo
17:18 hoppie The warning is seen when running http://pastie.org/9689740 against the 191 list.
17:19 sri mst: there you go, shitty windows fork makes the IO::Async implementation not a general purpose solution
17:20 mst sri: I'm fairly sure he did something with ithreads for that, or at least was working on it
17:20 jberger I just need the darn thing not to abort() !
17:21 jberger preaction had some thoughts
17:21 sri i had high hopes for this one... but portability problems :( https://metacpan.org/release/Net-DNS-Native
17:22 bpmedley sri: Did they fix the pthreads issue with that one?
17:22 sri mst: if Leonerd really had a good solution, i'm sure he would have released it as a standalone module ;p
17:22 sri bpmedley: no
17:27 hoppie Wondering if I should create a mini-mce using 1 data channel (not 8) and strip everything else out, just have something tiny. The IPC in MCE is cross-platform including windows. btw, defaults to threads on windows.
17:33 hoppie Took time off recently from work. Creating MCE for Ruby. MCE 1.6 for Perl maybe a rewrite. In the meantime, focusing on MCE for Ruby. It has channel locking working on all the environments so to get away from file locking. Works similar to Semaphore locking, but does not require posix semaphore.
17:35 hoppie folks, not seeing the warning from jberger's snippet. Changed 40 to 2:  has workers => 2;
17:36 jberger hoppie:
17:36 jberger ooops
17:36 jberger hoppie: the other workaround would be to install EV
17:37 jberger then you would use the other (higher performance) backend and you probably don't get that error
17:37 jberger maybe you get a different one from EV?
17:37 hoppie will install EV. Will also retest the initial example.
17:38 hoppie do I have to add a line or does Mojo look for EV and if present uses it.
17:39 hoppie When writing the MCE + Mojo, will have workers => 40 and chunk_size set to the same value.
17:40 sri jberger: BAD ADVICE!
17:40 jberger ?
17:40 sri jberger: EV defaults to select()
17:40 jberger ewwwww
17:40 hoppie :(
17:41 sri http://mojolicio.us/perldoc/Mojo/IOLoop#DESCRIPTION
17:41 jberger I thought EV was supposed to be "high performance", I guess you need to set all the epoll flags etc
17:41 jberger hmmmmm, I need to do that for FWBM
17:42 * jberger adds to TODO list
17:42 jberger too bad EV doesn't just DTRT
17:43 sri anyway, i've made an issue https://github.com/kraih/mojo/issues/700
17:43 sri anyone is welcome to work on it
17:44 jberger if/when forkcall becomes truely portable, it wouldn't be too hard to make core, just need to inline IO::Pipely
17:44 Flying_Squirrel joined #mojo
17:44 jberger which basically just makes a socket pair on systems that cant make pipe pairs
17:47 jberger sri: wow, that was a lot of lost code! it predates my involvement, I don't recall any of that
17:47 hoppie zero warnings; after installing libev and the Perl EV module.
17:47 sri it wasn't particularly good code either
17:50 jberger Mojolicious::Guides::Cheetsheet!
17:56 hoppie got it. LIBEV_FLAGS=4 perl jberger.pl < in6 > o
17:57 hoppie Learn a lot of things today.
17:57 jberger \o/
17:57 jberger epoll on linux, kqueue on mac, weep bitterly on windows
17:58 hoppie in6 is the same as urls.txt in the zip sent in the email
17:58 hoppie :)
18:00 jberger doesn't it seem like EV should be able to pick the best available poller for your system?
18:00 hoppie If ever I'm asked to poll millions of devices via SNMP, will do so using MCE + Mojo::SNMP. It's kind of cool actually to witness such power.
18:01 jberger batman++
18:04 jberger ok, really this time, I'm out for a while
18:04 jberger o/
18:04 d4rkie joined #mojo
18:06 D4RK-PH0ENiX joined #mojo
18:17 sri can you imagine a pool of hypnotoad workers where every worker process has a separate process for resolving? :)
18:17 mst I can imagine many more horrifying things than that
18:21 marty_ joined #mojo
18:31 Adura joined #mojo
19:17 neyasov joined #mojo
19:18 neyasov joined #mojo
19:20 amon joined #mojo
19:24 jamesaxl joined #mojo
19:29 punter joined #mojo
19:33 woz joined #mojo
19:34 denis_boyun joined #mojo
19:46 basiliscos joined #mojo
20:21 * sri somehow has a feeling that issue #700 is pointless... and that nobody will ever solve the problem
20:22 sri maybe it's because we've already been waiting for it since 2011 ;p
20:23 denis_boyun_ joined #mojo
20:25 hoppie joined #mojo
20:26 hoppie Finally, have a better understanding of how Mojolicious works. Phew ... :)
20:28 hoppie after additional testing, 40 is too much in jberger's code snippet.  http://pastie.org/9689740   So, changed it to 32.
20:31 sri hoppie: do you actually have an example for a dns lookup that takes a long time?
20:32 * mst notes that timeouts often do, even if successes are fast
20:35 hoppie Here, the MCE version: http://pastie.org/9690197
20:35 hoppie totally parallel.
20:35 hoppie little code
20:36 sri "max_connectinos"
20:36 hoppie MCE->say(...) is same as say "text", but output is serialized to the main process so that output is not garbled.
20:36 hoppie for this demo no need to have persistent connections.
20:36 hoppie Perhaps did not understand the documentation on that
20:38 mst hoppie: you've misspelled it max_connectinos
20:38 mst hoppie: I think you meant max_connections
20:38 hoppie oops -- last minute thing -- I had removed it and then put it back in.
20:38 hoppie yes
20:38 mst that's what sri was pointing out
20:39 hoppie oh
20:39 hoppie thanks.
20:39 mst http://trout.me.uk/facepaw.jpg :D
20:39 hoppie Well, happy to report that MCE + Mojo is a happy combination.
20:40 hoppie MCE compliments Mojo quite nicely.
20:40 axlrose joined #mojo
20:40 hoppie Did not know that if one goes about 32 ~ 40 for depth, things starts to fail. The chunking nature in MCE is beneficial.
20:41 hoppie s/about/above/
20:42 sri don't generalize
20:42 hoppie ok
20:43 sri it depends on many variables
20:43 sri scaling is hard
20:43 hoppie yes. of course.
20:43 hoppie chunking is helpful though.
20:44 hoppie What Joel did is another way of chunking so to speak -- max depth 40
20:44 sri the fact that you've not actually found the problem keeps me sceptical
20:44 hoppie ?
20:44 sri you don't know why you ran into scaling problems
20:44 mst reducing concurrency being a fix doesn't explain why concurrency needed to be reduced
20:44 hoppie I took Joe's example and changed workers => 40 to 191 and it failed similarly.
20:45 mst yes. the point is that you've treated the symptoms, but we still don't know the root cause
20:45 sri you're throwing stuff at the wall to see what sticks ;p
20:45 hoppie yes, correct -- did not expect to have to set to a low number.
20:45 mst sri: which is a good approach for testing underwear for re-wearability
20:46 mst "if it sticks ... don't"
20:46 hoppie there is something. this is good test case.
20:46 sri a test case is something others can replicate
20:47 sri i can't replicate what you did
20:47 hoppie yes, take joel's code and change workers => 191 against the list. Shouldn't that work with workers => 100?
20:47 sri what list?
20:48 hoppie what about Joel's code? Oh yeah, you need to right list to reproduce the issue.
20:48 hoppie I can email the list of 191 URLs.
20:48 sri put it on a paste site with the code
20:48 hoppie Do not feel right to post that on pastie as that is searchable.
20:49 sri then use a private gist
20:49 bpmedley http://pastie.org/9690220  <— Is this even remotely correct for using Net::DNS and Mojo::IOLoop together?
20:49 sri bpmedley: why would you want to do that?
20:50 bpmedley Could you use a perl resolver for async lookups?
20:50 sri no
20:50 bpmedley May I ask why?
20:51 sri bpmedley: https://github.com/kraih/mojo/issues/700
20:54 hoppie sri: sent email to your cpan email address
20:55 hoppie egrep -c "::" output afterwards, should not be more than 6
20:55 hoppie Joel's code:  http://pastie.org/9689740  (change workers => 40 to a higher value).
20:59 hoppie yeah, would be nice if it could handle 100 or 300 even better. My SNMP poller was configured with 300 for chunk_size. That was using AnyEvent::SNMP. One day will do similar using Mojo::SNMP.
20:59 ningu joined #mojo
21:00 sri hoppie: it just exits withouts doing anything
21:00 ningu sri: ping. I have a couple questions/bug reports for Mojo::DOM, I thought it might be easier to tell you than report on github this time :)
21:00 hoppie script.pl < url.list > out
21:01 sri hoppie: nothing
21:01 hoppie or joelscript.pl < url.list > out
21:01 ningu sri: so in the latest Mojolicious, $foo->find() generates: Mojo::Collection::AUTOLOAD is DEPRECATED in favor of Mojo::Collection::pluck
21:01 ningu so I guess I'll just do 2>/dev/null for all my scripts, but it's annoying. :P
21:02 hoppie sri: http://pastie.org/9689740
21:02 ningu also, for $foo->find('bar')->text -- this works well when something is found, but when nothing is found, the result is $VAR1 = bless( [], 'Mojo::Collection' );
21:03 sri hoppie: yes, it does nothing
21:03 ningu shouldn't the result be undef or an empty string?
21:03 sri ningu: no time
21:03 ningu sri: so github then?
21:03 sri ningu: or try finding someone else to help you, i doubt you found bugs
21:04 ningu these are recent changes
21:04 mst ningu: ->find is actually calling ->pluck('find')
21:04 hoppie sri: joel's script is working for me. he is reading from stdin.  my @urls = <>; chomp @urls;
21:04 mst ningu: yes, because the AUTOLOAD was a one liner only sugar thing, and then people misused it horribly
21:04 mst ningu: you'll want pluck/map instead, and everything will work
21:04 ningu mst: ok, but I'm not intentionally using AUTOLOAD at all. I'm just calling ->find.
21:05 mst ningu: there is no 'just'
21:05 ningu it seems to be the module's own methods should not raise deprecation warnings
21:05 ningu they should be removed or fixed
21:05 ningu to me
21:05 mst it's NOT THE MODULE'S METHOD
21:05 mst that's the entire point
21:05 ningu Mojo::DOM is part of Mojolicious.
21:05 sri hahaha
21:05 mst Mojo::Collection has no ->find method
21:05 mst you're calling ->find on a Mojo::Collection and relying on AUTOLOAD falling it through to Mojo::DOM
21:06 mst if it was a method on the thing you actually called it on, it wouldn't call the fucking autoload, would it now?
21:06 ningu ok, I guess that makes sense.
21:06 mst don't engage in magical thinking :P
21:07 mst the fact you called a method on the wrong object and it happened to apparently work ... is exactly the sort of confusing shit that resulted in the deprecation of the AUTOLOAD :)
21:07 sri hoppie: i have confirmed that the url list is loaded correctly, it is the delay stuff that doesn't work
21:07 sri hoppie: are you absolutely sure you're using a recent version of mojolicious?
21:07 mst ningu: make sense now?
21:07 hoppie ok sri. so the test is reproducible.
21:07 hoppie yes
21:08 hoppie let me provide the Perl output
21:08 ningu mst: ok. fair enough. but in most DOM libraries I've used, the root $dom object has exactly the same methods/properties as the found collections (e.g., jquery) so this was not at all obvious.
21:08 ningu it seems obvious only after understanding the internals of the module.
21:08 mst I haven't understood the internals of anything
21:08 mst I just used 'ref()' and read the docs
21:08 mst and it isn't about the root thing
21:09 mst you did something that returned a collection *of* Mojo::DOM objects, I think
21:09 ningu yes, I must have.
21:09 mst and *that* behaviour makes perfect sense
21:09 mst a collection of document fragments isn't a document fragment
21:10 mst this is true of anything that follows the W3C HTML DOM standard
21:10 mst and true of everything else I've used as well
21:10 hoppie sri: perl -MMojo::IOLoop -e 'print $Mojo::IOLoop::VERSION, "\n"',  where is the VERSION located
21:10 bpmedley sri: I assume the situation with issue 700 is the getaddrinfo call blocks?
21:10 mst ningu: basically: the autoload made confused code work, and therefore your mental model became confused - and if it's not clear, I'm blaming the AUTOLOAD for this
21:11 mst hence why it's now deprecated :)
21:11 mst (well, "the people with votes also blamed the AUTOLOAD, hence ...")
21:11 hoppie I installed using Mojolicious-5.56.tar.gz
21:12 hoppie checking if the lastest
21:12 hoppie I ran perl Makefile.PL
21:12 hoppie etc...
21:12 hoppie 5.56 seams like the latest on metacpan.org
21:13 hoppie s/seams/seems/
21:13 sri haha
21:13 ningu mst: turns out my ->find's were fine because they were always on Mojo::DOM elements. but I often have done things like: $foo->find('bar')->text. which now should be ->pluck('text') I guess.
21:14 sri hoppie: see, this is why i don't want emails with shitty formatted files... now i got an extra newline in the file and the script can't handle that
21:15 mst ningu: right
21:15 hoppie sri: ok... :(
21:15 sri it is working now though
21:16 hoppie change has workers => 40 to 191
21:16 ningu hmm... so ->text actually returns a Mojo::Collection too, which stringifies to its text.
21:16 ningu or rather, ->pluck('text') does.
21:16 hoppie you may then see the many "Premature connection close" errors
21:17 mst right, because ->pluck is a map {}
21:17 mst you probably wanted $foo->find('bar')->pluck('text')->join('')
21:17 mst which is equivalent to
21:17 mst $foo->find('bar')->map(sub { $_->text })->join('')
21:18 ningu mst: I like the magical stringify behavior :P
21:18 ningu but yeah.
21:18 mst ningu: the magical stringify behaviour is also scheduled for death
21:18 sri mst: not anymore i'm afraid
21:18 mst sri: oh?
21:19 mst you got outvoted by the peanut gallery?
21:19 sri https://github.com/kraih/mojo/issues/684#issuecomment-59544773
21:19 mst ningu: ok. the magical stringify behaviour should still be scheduled for death, and probably will be again once we figure out how to do it right
21:21 ningu hmm
21:21 ningu I realized that a lot of these things would/could be solved by ->findOne which would be sugar for ->find()->first
21:21 ningu or I could just use ->find()->first
21:22 woz joined #mojo
21:22 sri you mean ->at()
21:22 ningu oh :)
21:23 sri which is actually a highly optimized ->find() that stops right after finding the first result... soo much better than ->find()->first
21:23 ningu ah, but one disadvantage (so to speak) which is you have to check for undef before calling ->text :P
21:25 sri hoppie: of course i have not a single premature connection close
21:25 sri lots of connect timeouts, but that's it
21:28 sri hoppie: anyway, i can confirm that it is the resolver timing out a few times
21:29 ningu I guess this isn't so bad:     $_ = $_ ? $_->text : '' for ($wc, $dm, $tel, $eng, $hin);
21:29 hoppie sri: ok. thanks. in the meantime know of workaround and to set a low chunk_size.
21:34 sri but honestly, i don't believe this is a good test case at all... it's so unrealistic
21:35 sri who would perform 200 concurrent requests to different hosts?
21:39 hoppie sri, perhaps not this, but something like that is done using SNMP.
21:40 ningu 200 is a bit much I guess but not totally insane
21:40 sri hoppie: we have no SNMP in core
21:40 hoppie There are companies which reach out to 20 million
21:40 hoppie devices
21:41 sri our concern is HTTP and WebSocket for now
21:41 hoppie this was just a demonstration, no problem.
21:41 ningu well for websockets if you want to broadcast...?
21:41 ningu anyway I dunno what the original problem was
21:41 hoppie you are correct, not something one would do for HTTP
21:42 sri hoppie: also, why would your DNS lookups time out in the SNMP example?
21:42 sri hoppie: connect is still totally non-blocking, it's only the getaddrinfo() call that blocks
21:43 hoppie Was not expecting that for the HTTP example actually.
21:44 sri https://github.com/kraih/mojo/issues/700#issuecomment-61384863
21:44 hoppie I'm not working at the company anymore. The other day came across a presentation on the web. http://www.openfusion.net/talks/pwp/  Mojo is not mentioned there.
21:44 hoppie So, I wanted to try Mojo using his examples. That's how I got to here.
21:45 hoppie Afterwards, was going to send the author Mojo examples performing the same.
21:45 hoppie It was nothing more than good nature to spread the news on Mojolicious. That's all.
21:46 hoppie I searched the web after releasing MCE 1.519 and that's how I found the presentation. I felt bad that Mojo was not mentioned.
21:48 * sri chains hoppie to the channel
21:48 sri once you're in you're in
21:49 preaction that presentation should use $cv->begin and $cv->end instead of $cv->send if $done
21:49 sri no escaping now! }:-)
21:49 hoppie The author tries to do some things with LWP and AnyEvent, and then the same with MCE. Basically, something inside of me see to inspire the fellow man to Mojo.
21:49 hoppie s/see/said/
21:50 hoppie :)
21:50 jamesaxl joined #mojo
21:50 sri sad thing is, it looks like the AnyEvent::DNS approach causes no problems whatsoever... because the author hides all criticism :S
21:50 sri but we sadly know better by now
21:51 hoppie sri: yes, I have sent him an email and updated code samples using LWP, AnyEvent, also Mojo.
21:51 sri i mean mlehmann, the AnyEvent author
21:52 hoppie got it.
21:52 hoppie The only thing was that something wasn't right with the Mojo snippet -- and that's how I came here.  All that -- long hours -- wanted nothing more for the Mojo samples to work.
21:53 hoppie Now, I understand.
21:53 hoppie The reason is that I see this person as a big seed spreading news, giving presentation -- and somehow Mojo is not mentioned.
21:53 dvinciguerra joined #mojo
21:54 hoppie All I could do was to inspire to the best of my abilities.
21:55 hoppie Some good news, he wrote back to me and said he was going to create or update his slides. Hoping that he will add Mojo in there as well.
21:56 sri perhaps a note about the blocking resolver in the docs would be in order
21:57 sri no idea what to write though
21:57 sri and where
22:00 bpmedley sri: Would you consider using something like Net::DNS in a similar way that Mojo::IOLoop::Resolver used to be implemented?
22:01 sri bpmedley: did you get why the old resolver was a bad idea?
22:01 bpmedley No, I’m clueless in that respect.
22:01 sri "Operating system specific extensions such as mDNS and hosts files need to "just work"."
22:01 sri that's what i wrote in #700
22:02 sri and that's the actual problem
22:02 sri getaddrinfo() is not just DNS, far from it
22:02 bpmedley I see.  That makes sense.
22:03 Adura joined #mojo
22:04 sri node.js started out only using DNS too, and it went horribly wrong
22:04 sri now they have a thread pool calling getaddrinfo()
22:10 hesperaux_ joined #mojo
22:12 sri browsers these days do the same thing btw.
22:12 sri thread pool
22:13 bpmedley Interesting, I never considered that.  That’s why sometimes DNS timeouts cause other stuff to “hang” in chrome and whatnot.
22:14 sri it shouldn't these days
22:14 sri they push getaddrinfo() call onto a thread pool specifically not to block anything else
22:15 sri although, it changes constantly in chrome... they are quite unhappy and experiment all the time
22:17 inokenty joined #mojo
22:19 bpmedley Sorry for my confusion, is that one thread for each call to getaddrinfo to parallize them?
22:20 sri yes
22:20 hesperaux joined #mojo
22:23 hesperaux_ joined #mojo
22:27 hesperaux joined #mojo
22:38 sri "Connections are established non-blocking, but name resolution needs to be performed blocking, so you should limit concurrency if resolver timeouts are a possibility."
22:38 sri does that sound like something worth adding to Mojo::IOLoop?
22:38 mst yes
22:40 good_news_everyon joined #mojo
22:40 good_news_everyon [mojo] kraih pushed 1 new commit to master: http://git.io/CyachQ
22:40 good_news_everyon mojo/master c3290fa Sebastian Riedel: mention the blocking resolver
22:40 good_news_everyon left #mojo
22:42 hesperaux joined #mojo
22:44 sri i still kinda wish someone would magically make this module work with non-threaded perls https://metacpan.org/pod/Net::DNS::Native
22:53 bpmedley Is the issue with that the pthread data structures don’t get initialized properly because perl is not linked with pthreads?
22:58 bpmedley Also, just out of curiosity, is the issue with processing 200 URLs concurrently that every URL has to complete its request before the entire operation can finish?  And that just takes time.
22:59 bpmedley Especially consider that DNS lookups block.
22:59 bpmedley *considering
23:01 nicomen how about an XS module that uses getaddrinfo_a()  ?
23:02 bpmedley nicomen: I believe it also needs pthreads.
23:02 nicomen oh, you even made one?
23:02 bpmedley https://metacpan.org/pod/Net::DNS_A
23:02 nicomen yeah looking at it ;)
23:07 nicomen how do you see that it needs pthreads working?
23:08 zivester joined #mojo
23:10 nicomen ah the source code of it itself...
23:10 bpmedley ldd /lib64/libanl.so.1
23:11 woz joined #mojo
23:14 nicomen hehe http://discuss.joelonsoftware.com/default.asp?joel.3.475011.13
23:15 hoppie sri: when running the SNMP poller to 20 million devices, IP addresses were used, not the FQDN.
23:16 sri hoppie: yea, that would be completely non-blocking in mojolicious
23:17 sri btw. cloud with eyepatch doesn't work, it would need more facial features
23:17 sri http://i.imgur.com/1vyjrAG.png # but i've fixed the dimensions
23:28 jberger sri++ give me one now!
23:29 nicomen bpmedley: curl uses this, have you looked at that? http://c-ares.haxx.se/
23:32 bpmedley nicomen: I had no knowledge of that library.
23:32 sri jberger: shush! they are for mojoconf
23:32 sri \o\
23:32 jberger AAAAAAHHHH
23:32 sri /o/
23:33 jberger -o/
23:33 jberger \o-
23:33 sri nicomen: wasn't that just dns?
23:36 nicomen sri: like not files?
23:36 nicomen or wins shit?
23:36 nicomen files = /etc/resolv.conf et al
23:49 jberger I'm finding some real gems looking for a non-blocking resolver :o
23:49 jberger http://www.monkey.org/~provos/libdnsres/
23:49 jberger " In fact, it is an extremely ugly hack to make the BSD resolver library non-blocking and thread-safe without changing the API too much.  Although libdnsres has not been tested heavily, it should compile on BSD systems and Linux. The library comes with a simple test program that can be use to verify if it works correctly. Good luck!"
23:49 * jberger runs away
23:51 suy joined #mojo
23:52 jberger http://25thandclement.com/~william/projects/dns.c.html
23:53 jberger "todo  Write documentation."
23:54 sri if you want to make a new wrapper for a library, pick libuv ;p
23:54 sri it already has a non-blocking resolver
23:54 jberger https://metacpan.org/pod/POE::Component::Resolver < just uses POE::Wheel, which is basically the same thing as ForkCall
23:57 sri a well done libuv wrapper would solve so many problems for us :S

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary