Camelia, the Perl 6 bug

IRC log for #metacpan, 2013-08-08

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:46 shmuel joined #metacpan
01:54 klapperl_ joined #metacpan
02:11 shmuel_ joined #metacpan
03:38 preflex_ joined #metacpan
06:10 daxim joined #metacpan
06:12 dpetrov_ joined #metacpan
07:05 shmuel joined #metacpan
09:37 tianon joined #metacpan
12:58 * talexb is trying to understand why a local API access is giving a 404 result.
13:56 oalders talexb: can you gist the request and response?
13:56 talexb Sure .. but what's gist? Is that the nopaste that you use?
14:00 talexb https://gist.github.com/ta​lexb/95153b422598cd499509
14:00 dipsy [ Testing API -- getting 404 for my local API access. ]
14:10 oalders talexb: try "cpan_v1" for the index name
14:12 talexb oalders, Looks like the same error ..
14:12 talexb (Added the new result to the gist.
14:16 oalders talexb: yeah, that was the wrong advice :)
14:16 talexb LOL
14:16 oalders talexb: have you created your mappings locally yet?
14:16 talexb I'm just trying to get a feel for the API .. digging through the API tests doesn't reveal much.
14:17 talexb oalders, I've cloned the entire CPAN .. not sure about creating local mappings.
14:17 oalders talexb: first off, add "trace_calls => \*STDOUT," to the args when you create the ES object
14:18 oalders then, on this page: https://github.com/CPAN-API/metacpan-developer
14:18 dipsy [ CPAN-API/metacpan-developer · GitHub ]
14:18 oalders look for this section "Now, you are ready to run through the API loader commands."
14:19 talexb OK, yes, I think I did do those commands.
14:19 oalders you'll want to do the mapping command and at least "bin/metacpan author" i
14:20 talexb OK -- running now.
14:20 talexb Good news was that it started to index 10808 authors ..
14:20 oalders heh
14:20 oalders it won't take long
14:20 talexb Bad news is that there was a shard failure ..
14:21 talexb No active shards. I've added that to the gist.
14:22 oalders this is after you deleted the mapping?
14:23 talexb Oops. No. I'll delete then do the authors again. Sorry.
14:24 oalders it's kind of like creating the tables in RDBMS terms
14:26 talexb Right.
14:27 talexb Got 'Timed out while waiting for socket to become ready for reading (599)' ..
14:29 talexb I think I'll take the VM down, refresh everything and try again.
14:33 oalders in my case, i give it about 4 GB of RAM if i'm messing with the api
14:35 talexb Wow, 4G?
14:35 talexb OK, the VM's up again -- I'll give this a try, and if it doesn't go, I'll up the RAM.
14:38 talexb Yep, same error. Halting and increasing RAM.
14:53 oalders 4G is probably more than i need, but i figured i'd give it enough that it won't complain
14:54 talexb Hmm .. still failing. The repos are up to date, let me just check that I have the latest modules installed.
14:57 talexb Yup .. latest there.
14:57 talexb OK -- taking a break. Back In A Bit.
15:01 oalders talexb: when you're back, on the README i pointed you at earlier there are instructions on how to increase the RAM for ES
15:02 bowtie_ joined #metacpan
15:53 dsteinbrunner joined #metacpan
16:49 talexb oalders, Thanks, I have increased /etc/puppet/modules/elasti​csearch/manifests/init.pp from 1024 to 2048 ..
16:57 oalders talexb: so, hopefully after you run puppet you'll have success :)
16:57 talexb :/ Same error.
16:57 talexb Yes, I ran puppet ..
16:57 talexb It took 33 odd seconds.
16:58 talexb I'm wondering if trying to handle the entire CPAN as a mistake on my part.
16:58 talexb I got the error on doing the mapping --delete part ..
17:00 talexb I didn't restart the ElasticSearch service after running puppet .. maybe I should do that ..
17:01 oalders yeah, try restarting ES
17:01 talexb Running now.
17:01 oalders you don't need to index the entire CPAN
17:01 oalders but having it all is helpful anyway
17:03 talexb Well. That's interesting.
17:03 talexb It failed again, but this time it looks like it got much further.
17:05 talexb https://gist.github.com/ta​lexb/95153b422598cd499509
17:05 dipsy [ Testing API -- getting 404 for my local API access. ]
17:11 oalders talexb: what does "curl 127.0.0.1:9200" give you?
17:16 talexb Added it to gist .. looks OK .. name is Vibraxas?
17:17 oalders talexb: how about "curl 127.0.0.1:9200/cpan_v1/author/_mapping"
17:18 talexb Wow. A while bunch of stuff. Added to gist.
17:22 oalders so, you have mappings
17:22 oalders and if you try "bin/metacpan author" now?
17:24 talexb It's looking good ..
17:24 talexb ACK! :/
17:25 talexb Hmm .. This time, 500 / Internal Server Error.
17:25 talexb Something about shard_failures.
17:26 talexb It's added to the gist.
17:38 oalders talexb: do you see anything helpful in opt/elasticsearch-0.20.2/logs
17:38 oalders er /opt/elasticsearch-0.20.2/logs
17:40 talexb Wow .. yeah, lots of errors.
17:40 oalders :)
17:41 talexb Added the last 50 or so lines to the gist.
17:48 oalders what are the permissions on /var/elasticsearch/metacpan_v1/nodes/​0/indices/cpan_v1/2/index/write.lock ?
17:49 talexb The fle doesn't exist.
17:50 oalders so maybe work your way up the folder chain and check the first thing that does exist?
17:50 talexb There are lots of files in that directory -- all timestamped July 30.
17:50 talexb And those files are all root/root
17:50 cooper joined #metacpan
17:52 talexb OK -- /var/elasticsearch/metacpan_v1/nodes/0/indices is root/root, but the other two files in /var/elasticsearch/metacpan_v1/nodes/0/ are owned by es/es -- node.lock and _state directory.
17:52 oalders AFAIK everything in that folder should be owned by es/es
17:52 oalders so maybe chown -R?
17:52 talexb OK -- should I shown -R indicies and try again?
17:52 talexb :)
17:53 oalders yeah :)
17:53 oalders tree -u /var/elasticsearch/metacpan_v1
17:53 oalders that should only show es as the owner
17:53 talexb Yup.
17:54 talexb OK, let me try a maping delete .. Oooh .. CPU has gone to the rails.
17:54 talexb Heh .. looks like elasticsearch is busy workin' on something.
17:55 oalders you'll be an expert on this soon enough
17:55 talexb OK, CPU is back t normal.
17:55 talexb I hope so! :)
17:55 talexb I have to go back through the scrollback and Take Good Notes (sorry, inside joke from university).
17:56 talexb Cool .. mapping --delete Just Worked.
17:56 oalders ah, excellent
17:56 talexb doing author now, since it should be fast.
17:57 talexb OK, this is looking better.
17:57 talexb CPU is active but not heavy.
17:59 talexb oalders, Thanks, author completed in 1:45.
17:59 oalders \o/
18:00 talexb :)
18:00 oalders talexb: now try that script you were testing hours ago :)
18:00 talexb I will as soon as I take some notes .. it's clear in my mind now, but won't be in four weeks time.
18:02 oalders :D
18:05 talexb OK -- the index of cpan_v1 is no good?
18:06 oalders cpan_v1 is only if you're hitting port 9200
18:06 oalders i didn't read your code closely enough initially
18:06 oalders port 5000 is the Catalyst wrapper around port 9200
18:07 talexb I'm trying to hit 5000 .. should I be using 9200 instead?
18:07 oalders so using 5000 is the most portable way to do it
18:07 talexb OK.
18:07 oalders but the api has to be up and running
18:07 oalders that's the only caveat
18:29 talexb oalders, Basically what I'm trying to do is replicate API calls (searches) that access api.metacpan.org by replacing that URL with localhost:5000.
18:29 talexb I'm expecting it to work, but I am still getting 404. I've moved on to 1-fetch-single-author-es.pl, which just fetches mst's details.
18:31 dsteinbrunner question from #perl-help
18:31 dsteinbrunner https://metacpan.org/module/JLLO​YD/NetSuite-1.03/lib/NetSuite.pm
18:31 dipsy [ NetSuite - A perl-based interface to the NetSuite SuiteTalk (Web Services) API - metacpan.org ]
18:31 dsteinbrunner this shows but is not on the authors page
18:32 dsteinbrunner and is not available from search.cpan.org
18:32 dsteinbrunner is this a matter of metacpan not clearing pages for modules that have been pulled?
18:33 dsteinbrunner hmmm
18:33 dsteinbrunner ok, the back pan thing was brought up
18:33 dsteinbrunner nevermind
18:35 dsteinbrunner but this makes me think.. if an author has a back pan module it would be nice to be able to toggle the author page to show them
18:38 * talexb coffee break
18:40 haarg the author page only shows indexed (like PAUSE) dists
18:40 haarg anything else is very undiscoverable currently, which would be nice to improve somehow
18:41 dsteinbrunner well if back pan is supported at the module level it seems like the author page should support it to in part due to consistency
18:47 haarg it's not quite that simple
18:48 haarg because what you are really asking for is something like "show unindexed dists that don't have an indexed version, but only if it's the latest unindexed version"
18:49 dsteinbrunner yeah, something like that ;-P
18:49 dsteinbrunner I didn't suggest how complex is would be...
18:50 haarg it's also of very limited use
18:52 dsteinbrunner that could be argued if you felt argumentative
18:52 haarg could also be argued that if an author deleted a dist they don't want it showing up on their author page anymore
18:53 dsteinbrunner they likely don't want it showing up at all
18:53 dsteinbrunner if they feel that way
19:02 oalders talexb: let me know when you're back :)
19:03 oalders did 1-fetch-single-author-es.pl work for you?
19:03 oalders dsteinbrunner: making the backpan releases more discoverable is something we'd accept a patch for
19:05 dsteinbrunner yeah, I figured
19:07 oalders it's just a low priority thing
19:08 oalders but really maybe just a dropdown search box on the author page that would auto-suggest dist names based on everything the author has ever released would do the trick
21:39 talexb oalders, I'm back .. stopped to finish painting the deck. Hot work.
21:43 oalders :)
21:44 talexb No, 1-fetch-single-author-es.pl didn't work for me.
21:44 talexb Or rather, it didn't when I ran it against my local server at localhost:5000.
21:44 talexb It worked fine against api.metacpan.org
21:45 talexb status_msg was 'Not Found'.
21:46 talexb I also did a tail -f on the metacpan_v1.log file, and nothing's appeared since you helped me get things going.
21:46 talexb I guess I would have expected to see a little activity there.
21:52 oalders talexb: did you add "trace_calls => \*STDOUT," to the args when you create the ES object ?
21:57 talexb Yes, but not much came out ..
21:58 talexb Puzzling.
21:58 talexb https://gist.github.com/ta​lexb/95153b422598cd499509 updated ..
21:58 dipsy [ Testing API -- getting 404 for my local API access. ]
22:03 oalders yeah, there's something weird about that.
22:04 talexb Thanks so much for helping me out with this ..
22:04 talexb I have to get going shortly to the Damian talk (biking down to UofT) ..
22:04 oalders nice! i won't make it tonight again.  :(
22:04 oalders talexb: curl localhost:9200/cpan_v1/author/OALDERS?pretty
22:04 oalders what does that give you?
22:05 talexb And I'm visiting my parents in Waterloo tomorrow, but I'll come back and see what I can do on this.
22:05 talexb Oh, that seems to work ..
22:05 oalders i get similar errors on my VM, so i'll look into this and get back to you
22:05 talexb OK -- MSTROUT also works.
22:06 oalders 9200 works fine for me
22:06 talexb And I'm very glad to hear you're seeing the same error.
22:06 oalders :)
22:06 talexb (Phew!)
22:06 oalders so, for the script you could change port to 9200 and the index to cpan_v1
22:06 oalders and it should work
22:06 oalders we could maybe make that configurable in the examples via $ENV or something
22:07 oalders talexb: curl localhost:5000/author/MSTROUT
22:09 talexb That works.
22:09 oalders i think the /v0/ rewrite happens in nginx
22:09 talexb OK, I'll try that (scribbling quickly).
22:09 oalders so, hitting port 5000 directly bypasses that
22:12 oalders so, 9200 and cpan_v1 are the way to go for now
22:12 talexb Yup -- that works.
22:12 talexb Thanks a ton for this .. gotta get onto my bike. See you tonight at the talk?
22:12 oalders Not tonight, but I'll be at the meeting later this month.
22:13 oalders enjoy!
22:13 talexb Cool!
22:13 * talexb skittles off.
22:34 thaljef joined #metacpan
23:22 kentnl joined #metacpan

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary