| Time |
S |
Nick |
Message |
| 00:23 |
|
|
shibayu36 joined #metacpan |
| 00:48 |
|
|
shibayu36 joined #metacpan |
| 01:02 |
|
|
Levi joined #metacpan |
| 01:04 |
|
|
theory joined #metacpan |
| 01:16 |
|
|
scott joined #metacpan |
| 02:23 |
|
|
hanekomu_ joined #metacpan |
| 02:35 |
|
|
klapperl joined #metacpan |
| 03:17 |
|
|
scott joined #metacpan |
| 03:51 |
|
|
theory_ joined #metacpan |
| 06:05 |
|
|
scott_ joined #metacpan |
| 07:07 |
|
|
dpetrov_ joined #metacpan |
| 07:19 |
|
|
virtualsue joined #metacpan |
| 08:50 |
|
|
clintongormley joined #metacpan |
| 10:48 |
|
|
shibayu36 joined #metacpan |
| 11:08 |
|
|
daxim joined #metacpan |
| 11:17 |
|
daxim |
http://stackoverflow.com/quest[…]iven-distribution |
| 11:17 |
|
dipsy |
urgh. long url. Try http://tinyurl.com/78zgksh |
| 11:17 |
|
dipsy |
[ perl - How can we identify all the modules belongs to a given distribution - Stack Overflow ] |
| 11:21 |
|
dpetrov_ |
daxim: https://api.metacpan.org/relea[…]F/XML-LibXML-1.95 |
| 11:22 |
|
dpetrov_ |
or https://metacpan.org/module/Me[…]PAN::API::Release |
| 11:22 |
|
dipsy |
[ MetaCPAN::API::Release - Distribution and releases information for MetaCPAN::API - metacpan.org ] |
| 11:24 |
|
daxim |
thanks, answered. |
| 11:25 |
|
dpetrov_ |
I think that should be the answer :D |
| 11:26 |
|
daxim |
hold on, where is the json is the "provides" field? |
| 11:26 |
|
daxim |
it should say XML::LibXML XML::LibXML::AttributeHash XML::LibXML::Boolean XML::LibXML::Common XML::LibXML::Devel etc. |
| 11:27 |
|
|
Mithaldu joined #metacpan |
| 11:27 |
|
dpetrov_ |
so probably wrong answer :) |
| 11:27 |
|
dpetrov_ |
I thought he was looking for the dependencies |
| 11:27 |
|
dpetrov_ |
ignore me |
| 11:28 |
|
daxim |
try to come up with the right answer |
| 11:29 |
|
ribasushi |
daxim: also I think dpetrov != http://search.cpan.org/~dpetrov/ |
| 11:29 |
|
dipsy |
[ Denis Petrov - search.cpan.org ] |
| 11:30 |
|
ribasushi |
dpetrov_: or am I wrong? :) |
| 11:31 |
|
dpetrov_ |
nope :P |
| 11:31 |
|
dpetrov_ |
dpetrov != dpetrov |
| 11:31 |
|
dpetrov_ |
:D |
| 11:32 |
|
BinGOs |
as dynamic_config: 1 for XML-LibXML, the 'dependencies' listed are worthless anyways |
| 11:32 |
|
BinGOs |
kthnx. |
| 11:43 |
|
rafl |
oh! |
| 11:43 |
|
rafl |
dpetrov_: as you're a contributor to metacpan and the other guy on cpan probably isn't, i'm gonna need your email address |
| 11:43 |
|
dpetrov_ |
sent |
| 11:44 |
|
dpetrov_ |
so I guess Denis Petrov is "dp" |
| 12:01 |
|
dpetrov_ |
daxim: actually reading the question again |
| 12:01 |
|
rafl |
thanks |
| 12:01 |
|
dpetrov_ |
"we need a way to find the distribution name for a given module name" |
| 12:02 |
|
dpetrov_ |
probably what he need is http://api.metacpan.org/module[…]eption::Interface |
| 12:02 |
|
rafl |
http://api.metacpan.org/module/XML::LibXML->{distribution} |
| 12:02 |
|
rafl |
just so |
| 12:02 |
|
dpetrov_ |
and get the distribution |
| 12:03 |
|
rafl |
but note that it's "the distribution module $foo is currently indexed in" |
| 12:03 |
|
rafl |
other dists might have a module of the same name |
| 14:24 |
|
|
shibayu36 joined #metacpan |
| 14:51 |
|
dpetrov_ |
Perlbal::Manual::ReverseProxy |
| 14:51 |
|
dpetrov_ |
seems to be missing |
| 14:51 |
|
|
doherty joined #metacpan |
| 14:54 |
|
rafl |
everything's missing right now - my bad. recovering as we speak |
| 14:55 |
|
|
ilmari joined #metacpan |
| 14:55 |
|
ilmari |
did metacpan just lose its index? |
| 14:55 |
|
Mithaldu |
rafl's working on it |
| 14:56 |
|
rafl |
it didn't lose data - just ES having problems spinning up after i had to restart it. on it |
| 14:56 |
|
dams |
hi |
| 14:57 |
|
doherty |
/topic WE KNOW, now let us work. |
| 14:58 |
|
rafl |
actually, /topic all back to normal |
| 14:58 |
|
doherty |
^5 |
| 14:59 |
|
rwstauner |
not according to the explorer |
| 15:00 |
|
haarg |
yeah still lots of things broken on the site |
| 15:02 |
|
rafl |
ElasticSearch is still recovering. give it a few more minutes |
| 15:02 |
|
rafl |
but also feel free to tell us what you feel is broken still |
| 15:03 |
|
rwstauner |
explorer returns a 404 for the /release/Moose example |
| 15:03 |
|
* rafl |
confirms that |
| 15:04 |
|
haarg |
https://metacpan.org/release/Catalyst-Runtime same thing here |
| 15:04 |
|
rafl |
http://explorer.metacpan.org/?[…]OY%2FMoose-2.0001 works though |
| 15:04 |
|
dipsy |
[ MetaCPANExplorer ] |
| 15:04 |
|
rafl |
some shards are still coming up |
| 15:04 |
|
rwstauner |
they're still sleepy |
| 15:05 |
|
doherty |
poor guys, making them wake up so early in the morning -_- |
| 15:06 |
|
rwstauner |
yeah, really |
| 15:06 |
|
rwstauner |
did they also wake up at 3am to a noise coming from beneath the shower? |
| 15:12 |
|
rafl |
and we're fully back |
| 15:16 |
|
dams |
thanks guys |
| 15:25 |
|
|
erwan joined #metacpan |
| 15:27 |
|
erwan |
Hi there, I have several questions about metacpan and how to use the web API |
| 15:28 |
|
|
doherty left #metacpan |
| 15:32 |
|
erwan |
I search a way to nicely list every distinct release.distributions name. I wonder if I could use something like : http://api.metacpan.org/_searc[…]elds=distribution (and if I could use an additionnal parameter to only have distinct distribution, it would be perfect). |
| 15:32 |
|
erwan |
I'm a real newbie with metacpan and this API, so any clue will be greatly appreciated :) |
| 15:32 |
|
dipsy |
urgh. long url. Try http://tinyurl.com/7p9vxuc |
| 15:34 |
|
erwan |
oups sorry ! |
| 15:38 |
|
erwan |
with this http://tinyurl.com/7p9vxuc , I don't list every distributions where the name starts by a "D", I don't really understand why :( |
| 15:42 |
|
rafl |
erwan: should be something like { query => { match_all => {} }, size => 9999999, fields => [qw(distribution)], } on the /release endpoint |
| 15:46 |
|
erwan |
oky let me try it |
| 15:47 |
|
dams |
hm, using elastic search, I can get the "name", "version", "distribution" fields, but what field should I use to get the module name ? |
| 15:47 |
|
dams |
like, Foo::Bar ? I tried "module" but that's not it |
| 15:48 |
|
dams |
my original need is to get the list of all modules and their version that are provided by perl |
| 15:48 |
|
dams |
( perl the distribution) |
| 15:48 |
|
dams |
this list basically https://metacpan.org/release/perl |
| 15:48 |
|
dipsy |
[ perl-5.14.2 - The Perl 5 language interpreter - metacpan.org ] |
| 15:49 |
|
clintongormley |
rafl: you DON'T want to use sizes like that |
| 15:49 |
|
rwstauner |
rafl: mo has mentioned not specifying huge sizes |
| 15:49 |
|
rwstauner |
rafl: clintongormley has mentioned not specifying huge sizes |
| 15:49 |
|
clintongormley |
:D |
| 15:50 |
|
erwan |
I try with a low value for the size :) |
| 15:50 |
|
rwstauner |
:-) |
| 15:52 |
|
|
mstratman joined #metacpan |
| 15:54 |
|
rafl |
clintongormley, rwstauner: mo is also the person to help me construct that query in 2011-07 :-) |
| 15:54 |
|
* rafl |
is innocent |
| 15:54 |
|
clintongormley |
and wildcard queries perform really badly |
| 15:54 |
|
clintongormley |
need to use a field that has edge ngrams enabled |
| 15:54 |
|
rwstauner |
yeah i know, when i was told about size was when i copied a query from the explorer :-) |
| 15:55 |
|
erwan |
ah ? so it's better to match_all instead of use wildcard |
| 15:56 |
|
clintongormley |
match_all will match all docs |
| 15:56 |
|
clintongormley |
ALL |
| 15:56 |
|
clintongormley |
not ones beginning with D :) |
| 15:56 |
|
rafl |
i guess so. you can always trim the data down locally. that might very well be faster and less stressful on our resources, sufficiently fast network access provided |
| 15:59 |
|
erwan |
I wonder to list all docs but I just want to be nice with the infra :p |
| 16:23 |
|
erwan |
I think I will use an URL like this: http://tinyurl.com/7fq7xl6 |
| 16:23 |
|
erwan |
then I will sort uniq distribution name in my script... |
| 16:30 |
|
|
theory joined #metacpan |
| 16:30 |
|
|
Mithaldu joined #metacpan |
| 16:52 |
|
erwan |
sorry I have another dumb question: with this kind of url http://tinyurl.com/7s3756q is it possible to "page" the results (one url for the first 1000 results then 1001 to 2000 and so on) ? |
| 16:55 |
|
clintongormley |
erwan: you don't want to do that either |
| 16:55 |
|
clintongormley |
at least not with a normal search |
| 16:55 |
|
clintongormley |
you can do a scrolled 'scan' search |
| 16:55 |
|
clintongormley |
easiest way to do that is to use ElasticSearch.pm and do $es->scrolled_search(search_type=>'scan') |
| 16:57 |
|
erwan |
crap, actually I have java behind this is why I try to use normal search (and then parse the JSON result) |
| 16:58 |
|
clintongormley |
erwan: this isn't for sidekick is it? |
| 16:58 |
|
erwan |
nop ! |
| 16:59 |
|
clintongormley |
why are you wanting to pull all of this data? |
| 16:59 |
|
clintongormley |
it may be better to query ES for the relevant data instead |
| 17:00 |
|
clintongormley |
otherwise it is a bit like saying "i want to download google in alphabetical order" |
| 17:02 |
|
erwan |
query ES ? |
| 17:02 |
|
clintongormley |
ElasticSearch, by which I mean query the API |
| 17:02 |
|
clintongormley |
as opposed to downloading all of the data |
| 17:03 |
|
erwan |
oky sorry :) |
| 17:04 |
|
clintongormley |
that's why i'm asking what it is for |
| 17:07 |
|
erwan |
Actually I work for Antelink and we create Antepedia: www.antepedia.com, we will be proud to add the content from the CPAN (we already have a CPAN mirror), and we would like to use the metacpan api to qualify this content... don't worry I don't want to stress your infra :) |
| 17:09 |
|
erwan |
metacpan got every data we would like to use but we search a way to know every distinct release.distribution name and this is why I'm here :) |
| 17:10 |
|
erwan |
if you want I could explain reasoning behind |
| 17:11 |
|
clintongormley |
i've had a look at the site |
| 17:11 |
|
clintongormley |
but this means that you're going to be spidering regularly |
| 17:11 |
|
clintongormley |
currently metacpan only has one machine behind it |
| 17:12 |
|
erwan |
I know |
| 17:12 |
|
clintongormley |
so i'm a bit nervous about allowing large scale spidering |
| 17:12 |
|
erwan |
I know :( and I'm here to see how to do this without disturbing your service |
| 17:12 |
|
clintongormley |
yeah, appreciate it :) |
| 17:13 |
|
erwan |
(may be we could imagine to do mirror or something else ?) |
| 17:13 |
|
clintongormley |
erwan: i'd suggest: write up what data you'd like to get, how often, etc, and post it as an issue to github |
| 17:13 |
|
clintongormley |
when somebody who knows the api better is around, they may be able to come up with a better solution |
| 17:16 |
|
erwan |
oky I will open a ticket about it ! |
| 17:17 |
|
erwan |
but we like our work and if you need a backup mirror, I think we could help you and do some work together |
| 17:18 |
|
erwan |
moreover we could request the api only every 10 seconds or 1 minute if the frequency is too high |
| 17:23 |
|
clintongormley |
erwan: what may be easier is to use the cpan mirror functionality http://www.cpan.org/SITES.html |
| 17:24 |
|
dipsy |
[ CPAN Sites - www.cpan.org ] |
| 17:24 |
|
clintongormley |
that way you could just rsync to get the latest data |
| 17:24 |
|
clintongormley |
but i'm not that familiar with the infrastructure, which is why i suggested opening an issue with all the details instead |
| 17:29 |
|
erwan |
I already have the data from the cpan (and I have a weekly rsync with a french mirror). Now I need info to qualify the cpan content (data, license, repository, downloadurl, etc...) with metacpan data. |
| 17:31 |
|
erwan |
I will open a github issue about my request but like I said, we would be proud if could help you :) |
| 17:31 |
|
oalders |
erwan: sounds good :) |
| 17:40 |
|
|
Mithaldu joined #metacpan |
| 18:05 |
|
erwan |
https://github.com/CPAN-API/me[…]an-web/issues/505 I opened the issue :) |
| 18:05 |
|
dipsy |
[ Issue #505: Using Metacpan on a large scale · CPAN-API/metacpan-web · GitHub ] |
| 18:10 |
|
|
erwan left #metacpan |
| 19:58 |
|
|
virtualsue joined #metacpan |
| 21:15 |
|
|
theory joined #metacpan |
| 21:32 |
|
|
hanekomu joined #metacpan |
| 22:24 |
|
|
chansen joined #metacpan |
| 22:55 |
|
|
shibayu36 joined #metacpan |
