Perl 6 - the future is here, just unevenly distributed

IRC log for #metacpan, 2015-01-29

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:54 vroom joined #metacpan
01:37 jnbek joined #metacpan
01:52 punter joined #metacpan
02:01 vroom1 joined #metacpan
02:06 vroom joined #metacpan
02:14 klapperl joined #metacpan
02:26 cjm joined #metacpan
03:23 vroom joined #metacpan
04:30 vroom joined #metacpan
05:32 vroom joined #metacpan
05:43 vroom1 joined #metacpan
05:45 vroom2 joined #metacpan
05:53 ChoiKyuSang joined #metacpan
07:16 reneeb joined #metacpan
07:49 reneeb joined #metacpan
08:19 neilb joined #metacpan
08:20 oiami joined #metacpan
09:09 neilb joined #metacpan
10:18 punter joined #metacpan
11:31 alnewkirk joined #metacpan
11:49 reneeb joined #metacpan
12:39 punter joined #metacpan
12:40 punter joined #metacpan
12:42 vroom joined #metacpan
12:46 vroom left #metacpan
13:08 vroom joined #metacpan
13:08 vroom left #metacpan
13:23 andreeap joined #metacpan
13:30 vroom joined #metacpan
13:30 vroom left #metacpan
13:56 metacpan joined #metacpan
13:56 metacpan [metacpan-web] rwstauner pushed 3 new commits to master: http://git.io/Fw4K
13:56 metacpan metacpan-web/master 8663e72 Randy Stauner: Do string test before creating object...
13:56 metacpan metacpan-web/master fad7683 Randy Stauner: Invert and simplify "if we found a match" logic...
13:56 metacpan metacpan-web/master 76873d8 Randy Stauner: Also handle ".../tree/master" github repo urls...
13:56 metacpan left #metacpan
14:04 neilb rwstauner++
14:04 rwstauner ;-)
14:09 ranguard rwstauner++
14:26 vroom joined #metacpan
14:47 oalders rwstauner++
17:19 andreeap I used Devel::NYTProf to see why the indexer takes so long to complete on more than 200 tarballs and this is the result https://2914fd829145ea6075bb74984e2a7f2d4ddb89d9-www.googledrive.com/host/0B6-VBFJvhL8AN0s1ejRDdDZQdjQ/.
17:19 andreeap how do you guys think I could handle this so I can get a better time?
17:40 chmrr joined #metacpan
17:43 oalders anybody have an idea why it's spending so much time in waitpid()?
17:43 oalders ^^
17:44 rafl that's your program waiting for external programs to finish
17:45 neilb joined #metacpan
18:00 rwstauner right... the indexer forks to process multiple files at a time, and waits for one to finish before forking again
18:00 rafl might wanna run the profiler in just those subprocesses then
18:26 oalders andreeap ^^
18:27 andreeap I will run the profiler for subprocesses then
18:27 rafl see also: https://metacpan.org/pod/Devel::NYTProf#RUN-TIME-CONTROL-OF-PROFILING and https://metacpan.org/pod/distribution/Devel-NYTProf/bin/nytprofmerge
18:28 oalders nice. did not know that
18:32 andreeap I used nytprofmerge and got totally different results
18:39 oalders andreeap: can you share the results?
18:40 andreeap yes, but there are almost 900 files and they are still uploading
18:40 andreeap when it's ready, I will share the link
18:57 andreeap this is for 23 tarballs only(I will increase the number but it will take a lot more time) https://9a519cdb574952328a0cfd7308ede4cdd5e9abf5-www.googledrive.com/host/0B6-VBFJvhL8AaTBXeExhOG05bnc/
19:13 oalders you could try running it without Data::Printer, since it seems to be spending a lot of time there
19:16 andreeap I will remove it and try again
20:29 garu andreeap: did it work?
20:31 andreeap yes, except the DDP that I removed, the results are pretty much the same
20:38 andreeap these are the results https://602dbc767f2057f6a53f2bff271d8b8c452f9bda-www.googledrive.com/host/0B6-VBFJvhL8AcFF0aEZCWFdnOWs/ but at this point I don't know what I could change to improve the time
20:52 rafl looks like you're spending most of your time unpacking tarballs using Archive::Tar and waiting on the network to receive replies from elasticsearch
20:52 rafl Archive::Tar is, iirc, a pure per implementation of the tar standard. i bet it's a lot slower than the various tar implementations in c
20:53 rafl maybe you could give those a try?
20:54 rafl also, do you actually have to wait for responses from elasticsearch? is there perhaps a way that you can continue doing useful work while elasticsearch is still doing its thing? maybe you can batch some requests together and reduce the total number of requests you're making?
20:55 andy_j joined #metacpan
20:56 andy_j hi, I apparently have a metacpan account linked to my github account
20:56 andy_j but I cant seem to link my shiny new PAUSE ID
20:56 andy_j pause only has a secret email configured
20:57 haarg Archive::Tar::Wrapper should be api compatible with Archive::Tar but uses the tar binary to do the work
20:57 grantm andy_j: how shiny and new is your PAUSE ID?
20:57 andreeap I will try to change the way it unpacks the archives and see how much it improves and for the second one, I have to do a bit of research in what I can and can't do because I am not sure 100%
20:57 andy_j grantm: a couple of weeks
20:58 andy_j hm test email to my @cpan.org email isn't arriving on my server either
20:58 grantm that should be fine then
20:59 andy_j I think pause may be delaying the change I made to the address setup
21:00 andy_j i'll try again later.  thanks!
21:00 rafl andreeap: cool. let us know if you run into any problems! :)
21:00 andreeap I will, thanks for all your help :)
23:56 chansen joined #metacpan

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary