Camelia, the Perl 6 bug

IRC log for #metacpan, 2012-03-14

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:23 shibayu36 joined #metacpan
00:48 shibayu36 joined #metacpan
01:02 Levi joined #metacpan
01:04 theory joined #metacpan
01:16 scott joined #metacpan
02:23 hanekomu_ joined #metacpan
02:35 klapperl joined #metacpan
03:17 scott joined #metacpan
03:51 theory_ joined #metacpan
06:05 scott_ joined #metacpan
07:07 dpetrov_ joined #metacpan
07:19 virtualsue joined #metacpan
08:50 clintongormley joined #metacpan
10:48 shibayu36 joined #metacpan
11:08 daxim joined #metacpan
11:17 daxim http://stackoverflow.com/questions/96​96717/how-can-we-identify-all-the-mod​ules-belongs-to-a-given-distribution
11:17 dipsy urgh. long url. Try http://tinyurl.com/78zgksh
11:17 dipsy [ perl - How can we identify all the modules belongs to a given distribution - Stack Overflow ]
11:21 dpetrov_ daxim: https://api.metacpan.org/rel​ease/SHLOMIF/XML-LibXML-1.95
11:22 dpetrov_ or https://metacpan.org/module/MetaCPAN::API::Release
11:22 dipsy [ MetaCPAN::API::Release - Distribution and releases information for MetaCPAN::API - metacpan.org ]
11:24 daxim thanks, answered.
11:25 dpetrov_ I think that should be the answer :D
11:26 daxim hold on, where is the json is the "provides" field?
11:26 daxim it should say  XML::LibXML XML::LibXML::AttributeHash XML::LibXML::Boolean XML::LibXML::Common XML::LibXML::Devel etc.
11:27 Mithaldu joined #metacpan
11:27 dpetrov_ so probably wrong answer :)
11:27 dpetrov_ I thought he was looking for the dependencies
11:27 dpetrov_ ignore me
11:28 daxim try to come up with the right answer
11:29 ribasushi daxim: also I think dpetrov != http://search.cpan.org/~dpetrov/
11:29 dipsy [ Denis Petrov - search.cpan.org ]
11:30 ribasushi dpetrov_: or am I wrong? :)
11:31 dpetrov_ nope :P
11:31 dpetrov_ dpetrov != dpetrov
11:31 dpetrov_ :D
11:32 BinGOs as dynamic_config: 1 for XML-LibXML, the 'dependencies' listed are worthless anyways
11:32 BinGOs kthnx.
11:43 rafl oh!
11:43 rafl dpetrov_: as you're a contributor to metacpan and the other guy on cpan probably isn't, i'm gonna need your email address
11:43 dpetrov_ sent
11:44 dpetrov_ so I guess Denis Petrov is "dp"
12:01 dpetrov_ daxim: actually reading the question again
12:01 rafl thanks
12:01 dpetrov_ "we need a way to find the distribution name for a given module name"
12:02 dpetrov_ probably what he need is http://api.metacpan.org/module/​Catalyst::Exception::Interface
12:02 rafl http://api.metacpan.org/module/XML::LibXML->{distribution}
12:02 rafl just so
12:02 dpetrov_ and get the distribution
12:03 rafl but note that it's "the distribution module $foo is currently indexed in"
12:03 rafl other dists might have a module of the same name
14:24 shibayu36 joined #metacpan
14:51 dpetrov_ Perlbal::Manual::ReverseProxy
14:51 dpetrov_ seems to be missing
14:51 doherty joined #metacpan
14:54 rafl everything's missing right now - my bad. recovering as we speak
14:55 ilmari joined #metacpan
14:55 ilmari did metacpan just lose its index?
14:55 Mithaldu rafl's working on it
14:56 rafl it didn't lose data - just ES having problems spinning up after i had to restart it. on it
14:56 dams hi
14:57 doherty /topic WE KNOW, now let us work.
14:58 rafl actually, /topic all back to normal
14:58 doherty ^5
14:59 rwstauner not according to the explorer
15:00 haarg yeah still lots of things broken on the site
15:02 rafl ElasticSearch is still recovering. give it a few more minutes
15:02 rafl but also feel free to tell us what you feel is broken still
15:03 rwstauner explorer returns a 404 for the /release/Moose example
15:03 * rafl confirms that
15:04 haarg https://metacpan.org/release/Catalyst-Runtime same thing here
15:04 rafl http://explorer.metacpan.org/?url​=%2Frelease%2FDOY%2FMoose-2.0001 works though
15:04 dipsy [ MetaCPANExplorer ]
15:04 rafl some shards are still coming up
15:04 rwstauner they're still sleepy
15:05 doherty poor guys, making them wake up so early in the morning -_-
15:06 rwstauner yeah, really
15:06 rwstauner did they also wake up at 3am to a noise coming from beneath the shower?
15:12 rafl and we're fully back
15:16 dams thanks guys
15:25 erwan joined #metacpan
15:27 erwan Hi there, I have several questions about metacpan and how to use the web API
15:28 doherty left #metacpan
15:32 erwan I search a way to nicely list every distinct release.distributions name. I wonder if I could use something like : http://api.metacpan.org/_search?q=release.distr​ibution:D*&size=100&fields=distribution (and if I could use an additionnal parameter to only have distinct distribution, it would be perfect).
15:32 erwan I'm a real newbie with metacpan and this API, so any clue will be greatly appreciated :)
15:32 dipsy urgh. long url. Try http://tinyurl.com/7p9vxuc
15:34 erwan oups sorry !
15:38 erwan with this http://tinyurl.com/7p9vxuc , I don't list every distributions where the name starts by a "D",  I don't really understand why :(
15:42 rafl erwan: should be something like { query  => { match_all => {} }, size => 9999999, fields => [qw(distribution)], } on the /release endpoint
15:46 erwan oky let me try it
15:47 dams hm, using elastic search, I can get the "name", "version", "distribution" fields, but what field should I use to get the module name ?
15:47 dams like, Foo::Bar ? I tried "module" but that's not it
15:48 dams my original need is to get the list of all modules and their version that are provided by perl
15:48 dams ( perl the distribution)
15:48 dams this list basically https://metacpan.org/release/perl
15:48 dipsy [ perl-5.14.2 - The Perl 5 language interpreter - metacpan.org ]
15:49 clintongormley rafl: you DON'T want to use sizes like that
15:49 rwstauner rafl: mo has mentioned not specifying huge sizes
15:49 rwstauner rafl: clintongormley has mentioned not specifying huge sizes
15:49 clintongormley :D
15:50 erwan I try with a low value for the size :)
15:50 rwstauner :-)
15:52 mstratman joined #metacpan
15:54 rafl clintongormley, rwstauner: mo is also the person to help me construct that query in 2011-07 :-)
15:54 * rafl is innocent
15:54 clintongormley and wildcard queries perform really badly
15:54 clintongormley need to use a field that has edge ngrams enabled
15:54 rwstauner yeah i know, when i was told about size was when i copied a query from the explorer :-)
15:55 erwan ah ? so it's better to match_all instead of use wildcard
15:56 clintongormley match_all will match all docs
15:56 clintongormley ALL
15:56 clintongormley not ones beginning with D :)
15:56 rafl i guess so. you can always trim the data down locally. that might very well be faster and less stressful on our resources, sufficiently fast network access provided
15:59 erwan I wonder to list all docs but I just want to be nice with the infra :p
16:23 erwan I think I will use an URL like this: http://tinyurl.com/7fq7xl6
16:23 erwan then I will sort uniq distribution name in my script...
16:30 theory joined #metacpan
16:30 Mithaldu joined #metacpan
16:52 erwan sorry I have another dumb question: with this kind of url http://tinyurl.com/7s3756q is it possible to "page" the results (one url for the first 1000 results then 1001 to 2000 and so on) ?
16:55 clintongormley erwan: you don't want to do that either
16:55 clintongormley at least not with a normal search
16:55 clintongormley you can do a scrolled 'scan' search
16:55 clintongormley easiest way to do that is to use ElasticSearch.pm and do $es->scrolled_search(search_type=>'scan')
16:57 erwan crap, actually I have java behind this is why I try to use normal search (and then parse the JSON result)
16:58 clintongormley erwan: this isn't for sidekick is it?
16:58 erwan nop !
16:59 clintongormley why are you wanting to pull all of this data?
16:59 clintongormley it may be better to query ES for the relevant data instead
17:00 clintongormley otherwise it is a bit like saying "i want to download google in alphabetical order"
17:02 erwan query ES ?
17:02 clintongormley ElasticSearch, by which I mean query the API
17:02 clintongormley as opposed to downloading all of the data
17:03 erwan oky sorry :)
17:04 clintongormley that's why i'm asking what it is for
17:07 erwan Actually I work for Antelink and we create Antepedia: www.antepedia.com, we will be proud to add the content from the CPAN (we already have a CPAN mirror), and we would like to use the metacpan api to qualify this content... don't worry I don't want to stress your infra :)
17:09 erwan metacpan got every data we would like to use but we search a way to know every distinct release.distribution name and this is why I'm here :)
17:10 erwan if you want I could explain reasoning behind
17:11 clintongormley i've had a look at the site
17:11 clintongormley but this means that you're going to be spidering regularly
17:11 clintongormley currently metacpan only has one machine behind it
17:12 erwan I know
17:12 clintongormley so i'm a bit nervous about allowing large scale spidering
17:12 erwan I know :( and I'm here to see how to do this without disturbing your service
17:12 clintongormley yeah, appreciate it :)
17:13 erwan (may be we could imagine to do mirror or something else ?)
17:13 clintongormley erwan: i'd suggest: write up what data you'd like to get, how often, etc, and post it as an issue to github
17:13 clintongormley when somebody who knows the api better is around, they may be able to come up with a better solution
17:16 erwan oky I will open a ticket about it !
17:17 erwan but we like our work and if you need a backup mirror, I think we could help you and do some work together
17:18 erwan moreover we could request the api only every 10 seconds or 1 minute if the frequency is too high
17:23 clintongormley erwan: what may be easier is to use the cpan mirror functionality http://www.cpan.org/SITES.html
17:24 dipsy [ CPAN Sites - www.cpan.org ]
17:24 clintongormley that way you could just rsync to get the latest data
17:24 clintongormley but i'm not that familiar with the infrastructure, which is why i suggested opening an issue with all the details instead
17:29 erwan I already have the data from the cpan (and I have a weekly rsync with a french mirror). Now I need info to qualify the cpan content (data, license, repository, downloadurl, etc...) with metacpan data.
17:31 erwan I will open a github issue about my request but like I said, we would be proud if could help you :)
17:31 oalders erwan: sounds good :)
17:40 Mithaldu joined #metacpan
18:05 erwan https://github.com/CPAN-AP​I/metacpan-web/issues/505 I opened the issue :)
18:05 dipsy [ Issue #505: Using Metacpan on a large scale · CPAN-API/metacpan-web · GitHub ]
18:10 erwan left #metacpan
19:58 virtualsue joined #metacpan
21:15 theory joined #metacpan
21:32 hanekomu joined #metacpan
22:24 chansen joined #metacpan
22:55 shibayu36 joined #metacpan

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary