The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2015-12-11

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 CandyAngel sri: *pokes*
00:03 CandyAngel sri: Thank you for listening to and discussing me ideas and suggestions and such :)
00:44 meshl joined #mojo
00:45 sri CandyAngel: buy more t-shirts from the mojolicious store! :)
01:02 sue joined #mojo
01:06 henq joined #mojo
01:11 Zoffix joined #mojo
01:36 voldemortensen joined #mojo
02:45 kaare joined #mojo
03:31 miller joined #mojo
03:58 noganex joined #mojo
04:38 irqq joined #mojo
04:45 inokenty-w joined #mojo
06:21 melo joined #mojo
06:30 jwang joined #mojo
07:05 melo joined #mojo
07:14 Vandal joined #mojo
07:43 sugar joined #mojo
08:07 trone joined #mojo
08:09 DFox joined #mojo
08:25 AndrewIsh joined #mojo
08:45 eseyman joined #mojo
09:17 irqq joined #mojo
09:27 vanHoesel1 joined #mojo
09:29 pierrick joined #mojo
10:04 meshl joined #mojo
10:10 bpmedley joined #mojo
10:19 flamey joined #mojo
10:21 flamey hi guys, noob here.. trying to run small mojo lite script.
10:22 flamey if I run hypnotoad -f ./demo.pl , i works fine, I can access the site http://ip:8080/u/blah
10:23 flamey but if I just run hypnotoad ./demo.pl , I get "something when very wrong" throwingup unicorn screen
10:23 flamey in log file I get:
10:24 flamey [Fri Dec 11 05:14:28 2015] [error] Error while executing command: get: Server returned error message Can't connect to localhost:8910 Connection refused at /usr/local/share/perl/5.18.2/LWP/Protocol/http.pm line 47.  instead of data at /usr/local/share/perl/5.18.2/Selenium/Remote/Driver.pm line 310.
10:24 bpmedley flamey : How many lines is the script?  Is it pastable?
10:25 nic http://paste.scsys.co.uk/mojo
10:25 nic flamey: Does your app connect back to itself?
10:27 flamey http://mibpaste.com/UUNJWH
10:27 flamey it's not the whole script, but probably enough to see where the problem could be
10:28 bpmedley flamey : Is the Mechanize in the mojo lite script, or are they separate files?
10:28 shadowpaste "flamey" at 217.168.150.38 pasted "problem with hypnotoad running in background" (37 lines) at http://paste.scsys.co.uk/502629
10:29 nic flamey: Where does 8910 come into the mix?
10:29 flamey there's not other script
10:29 flamey nic, I have no idea where this comes from -- it's not in my script
10:30 flamey there are no other use-s or require-s
10:30 nic flamey: Have you established you need Mechanize rather than Mojo::Useragent?
10:31 flamey and, like I said, if I run it in the foreground, everything works fine. also if I run ./demo.pl daemon
10:31 flamey i need WWW::Mechanize::PhantomJS to run javascript on the page I get
10:32 nic I believe there's phantomjs for mojo (tho I haven't tried it yet)
10:44 henq joined #mojo
10:47 flamey I don't understand how to use it yet :( looks like it's just some wrapping to testing
10:48 flamey but it's strange that my script works in foreground mode, but fails in background mode
10:56 berov joined #mojo
10:57 sue joined #mojo
11:45 melo joined #mojo
11:49 asarch joined #mojo
12:06 neilhwatson joined #mojo
12:53 kaare joined #mojo
13:01 mib_65q359 joined #mojo
13:05 berov joined #mojo
13:05 dod joined #mojo
13:06 berov joined #mojo
13:09 henq_ joined #mojo
13:44 mgrimes joined #mojo
13:57 orev joined #mojo
14:23 gryphon joined #mojo
14:59 Onigiri left #mojo
15:01 sue joined #mojo
15:19 sh4 joined #mojo
15:57 kivilahtio joined #mojo
16:00 hummeleBop joined #mojo
16:20 disputin joined #mojo
16:24 PopeFelix joined #mojo
16:51 fiyorin joined #mojo
16:54 miller joined #mojo
17:01 disputin joined #mojo
17:05 marty joined #mojo
17:32 voldemortensen joined #mojo
17:34 dod joined #mojo
18:33 jjohn left #mojo
18:59 dod joined #mojo
19:00 kafkaesque joined #mojo
19:02 kafkaesque I have 1,000 proxies and I'd like to crawl 1,000 URLs concurrently, each thread (or useragent) using one of the proxies. The queue is shared between all threads, and as it's crawling each thread is pushing/shifting from the queue. I'd like each thread to hit a URL, then wait 15 seconds before moving on.
19:03 kafkaesque So, I've been working on this using Mojo::UA, and having some issues. I stumbled across this Gist for URLQueue.pl by jberger (https://gist.githubusercontent.com/jberger/5153008/raw/b45a405c961f6958752e6310f1a668f8e7b5d038/URLQueue.pl), and I'm trying to modify it to fit my needs. I am wondering, if I want to do this, do I just set concurrency to 1,000 and before it does a $ua->get, set the proxy?
19:04 kafkaesque From what I can gather, with this URLQueue.pl script, there's no way to configure 1,000 useragents each with their own proxy.
19:05 sue joined #mojo
19:05 preaction you just create new $ua objects, that should work fine
19:05 preaction you won't hit 1,000 inflight requests though, i suspect
19:08 kafkaesque preaction: I think what I'm having a hard time understanding is this: It seems that as the queue grows, _refresh will be called multiple times, thus it's not really possible to create a persistent UA object in refresh. I was thinking about creating 1,000 in start, then passing to refresh? Not sure where to do it is the problem :)
19:10 jabberwok joined #mojo
19:10 preaction ua->get is only called in one place. you could create it right there
19:11 kafkaesque Awesome, I was thinking about doing it right above $ua->get, just thought that because it's shifting the url queue that it'd be called millions of times, not a thousand
19:12 jberger kafkaesque: you'll want to be sure to have Net::DNS::Native
19:12 kafkaesque Thank you kindly preaction.
19:13 kafkaesque jberger: I installed it. Seemed to fix a minor issue for me.
19:13 kafkaesque jberger: It actually worked without it, but I had a few issues with it, and how awesome that the very person I was talking about happens to be here... I didn't even think to check :)
19:13 jberger you were timing out somtimes?
19:14 kafkaesque jberger: Yes, exactly. But it was intermittent, not every time.
19:14 * jberger blushes
19:14 jberger and yeah, the DNS request is blocking on the system, with tons of requests that can be enough to timeout the request
19:14 jberger with NDN the DNS requests aren't blocking
19:15 jberger I didn't write URLQueue with the idea of multiple proxies but you should be able to adapt it
19:22 kafkaesque It seems with some minor modifications, using minion on the backend, this could be made distributed.
19:37 jberger this script is older than minion :P
19:50 sri damn, $150k just to add websockets to django :O https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
19:50 sri maybe i should be doing more python
19:56 meshl joined #mojo
19:57 sri and here i am dealing with angry emails from people demanding free support because they are doing me a favor using mojolicious :S
20:01 CandyAngel Don't they know they can just come here and get free support.. and tshirts?
20:15 kafkaesque Is there any dependencies for using proxies? I have IO::Socket::SSL, but getting
20:16 kafkaesque I hate when I hit enter
20:16 CandyAngel Not as bad as when it hits you back
20:16 kafkaesque 'Proxy connection failed.' but just realized my other script works perfectly fine, so not sure what's ging on here.
20:21 disputin joined #mojo
20:23 disputin joined #mojo
20:23 PryMar56 joined #mojo
21:00 jberger http://cube-drone.com/comics/c/pipes
21:00 pink_mist jberger: several people became interested in Mojo::Phantom when I recommended it to someone earlier today! =)
21:01 jberger pink_mist: \o/
21:01 jberger I haven't had time to give it much love lately
21:01 jberger that was going to be a core part of our testing at $old_job
21:01 jberger but then $old_job went away
21:01 pink_mist dang
21:02 jberger $new_job needs more minion things :-P
21:02 jberger wonder if anyone noticed :-P
21:02 pink_mist haha, I recommended that too to someone else a couple of hourse earlier than the Phantom recommendation :P
21:02 pink_mist *hours
21:11 henq joined #mojo
21:17 jberger pink_mist: did you see https://metacpan.org/pod/Minion::Notifier ?
21:18 pink_mist ohh, sounds nice =)
21:18 jberger still very early, but I'm working it into $work_codebase in the next few days
21:18 lluad That looks interesting.
21:28 punter joined #mojo
23:05 flamey joined #mojo
23:44 meshl joined #mojo
23:47 sue joined #mojo
23:52 harleypig Ok. I know I'm doing something stupid, or missing something so obvious it's just waiting for me to see it so it can smack me. I'm trying to add logging to my controllers and models. In my lib/MyApp.pm module, in the startup method, I can do $app->log->debug( 'msg' ). But in my controllers and models do $self->log->debug( 'msg' ) gives me  'Can't locate
23:52 harleypig object method "log" via package "MyApp::Controller::MyController", even if I use Mojo::Log in the module. What am I missing?
23:53 Grinnz $self in the controllers is a controller object, not an app object; $self->app->log->...
23:55 harleypig Oh, for Pete's sake ... thank you. I knew it was something stupid simple.

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary