Perl 6 - the future is here, just unevenly distributed

IRC log for #metacpan, 2016-11-29

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 karjala Is there any reason to create new MetaCPAN::Client objects every now and then? Am I ok if I just create one and use it forever on my webserver?
00:10 Guest69 joined #metacpan
01:22 karjala I think I found a huge bug:
01:22 karjala I included the line "package Algorithm::Diff;" inside my lib/XML/MyXML.pm module on CPAN
01:23 karjala just to test something
01:23 karjala and now MetaCPAN says that Algorithm::Diff inside XML::MyXML is authorized!!!
01:23 karjala my $module = $mcpan->module('XML::MyXML');
01:23 karjala p $module->module;
01:24 karjala It should *not* be authorized, because this module (Algorithm::Diff) has existed since forever, by someone else
01:26 karjala alternatively, check here: http://api.metacpan.org/v0/module/XML::MyXML
01:32 karjala https://github.com/metacpan/metacpan-api/issues/588
01:38 haarg karjala: https://pause.perl.org/pause/authenquery?pause99_peek_perms_by=me&pause99_peek_perms_query=Algorithm::Diff&pause99_peek_perms_sub=1
01:38 haarg you have comaint
01:39 karjala since when?
01:40 karjala I never spoke to Algorithm::Diff guy
01:40 karjala ever
01:40 karjala should I have comaint without anyone having given it to me?
01:40 haarg possibly a pause bug
01:40 haarg it has no first-come
01:41 haarg only modulelist
01:42 karjala Will it get fixed?
01:42 karjala My site relies on this not happening.
01:42 karjala this complex query that was shown to us this evening
01:43 haarg metacpan is handling everything correctly
01:43 karjala it returns XML::MyXML as well, if you search for Algorithm::Diff
01:44 karjala who can we notify to have the PAUSE bug fixed?
01:49 haarg https://github.com/andk/pause/
01:50 haarg karjala: if you need to distinguish them, sort for the largest version
01:50 haarg and for equal versions, the latest release
01:55 karjala how does high version filter out mistakes/evildoings?
01:56 karjala anyone can upload a high version
01:58 haarg you hit an edge case regarding pause permissions, with the modulelist permission
01:58 haarg metacpan is reflecting what pause did correctly.  if your version number was greater, then your module would have been indexed for Algorithm::Diff.
01:59 haarg your dist didn't get indexed for Algorithm::Diff because your version number is lower.
02:00 haarg this is obviously something that should be fixed in pause, but metacpan is reflecting reality correctly, and if you limit to the largest version, falling back to the latest release, you will be accurately following what pause does.
05:09 metacpan joined #metacpan
05:09 metacpan [metacpan-web] tsibley pushed 1 new commit to tsibley/metahack-sponsors: https://git.io/v1tpw
05:09 metacpan metacpan-web/tsibley/metahack-sponsors b6dda3a Thomas Sibley: Add alt text for Perl Careers logo per Neil's request
05:09 metacpan left #metacpan
07:27 nakiro joined #metacpan
07:35 nakiro joined #metacpan
07:39 karjala excellent haarg, thanks!
07:44 karjala I think I should use some of the bignum modules for sorting versions, though
07:45 karjala because versions can have many decimal digits (which I shouldn't miss)
08:11 karjala My query with MetaCPAN::Client returns an iterator of 810 search results. It takes a very long time to iterate through all of them. Maybe the size of the query search results is too small and MetaCPAN.org is hit often?
08:14 karjala ok,not very long time, it took 6 seconds.
08:14 karjala on my home connection
08:34 neilb joined #metacpan
08:54 metacpan joined #metacpan
08:54 metacpan [metacpan-web] ranguard pushed 1 new commit to master: https://git.io/v1qqc
08:54 metacpan metacpan-web/master b6ec537 Leo Lapworth: Merge pull request #1821 from metacpan/tsibley/metahack-sponsors...
08:54 metacpan left #metacpan
09:04 Relequestual joined #metacpan
09:05 Relequestual joined #metacpan
09:21 neilb_ joined #metacpan
09:34 osfabibisi joined #metacpan
12:27 Tempesta joined #metacpan
14:00 osfabibisi joined #metacpan
14:02 pombreda joined #metacpan
14:04 pombreda Howdy :) what would be the gentle and polite way to collect/mirror all the CPAN packages metadata?
14:08 ranguard pombreda: when you say 'all' - do you mean backpan, or just cpan?
14:08 ranguard and when you say 'metadata' what info does that include?
14:15 pombreda ranguard: well I am building a package catalog for https://github.com/nexB/scancode-toolkit/
14:16 pombreda ranguard: I want eventually all of the modules available and release history in there
14:17 * ilmari wonders how https://libraries.io/cpan does it
14:18 pombreda and about metadata, I mean the essentials ... you can get a vague idea of what that would be by looking at the fields listed in there
14:18 pombreda https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/models.py#L477
14:18 pombreda ilmari: using the API
14:18 * pombreda wants to make sure I use an OK way for this :)
14:19 ranguard pombreda: do you run Elasticsearch yourself :) ?
14:19 pombreda ranguard: nope, but I could easily, if there were a way to rsync the indexes or get a JSON dummp of it
14:20 pombreda I am aware of the API and the scroll API, but I want to avoid being a pain and overload your servers
14:20 ranguard mickey: ^^ ?
14:21 mickey how often are you going to query us?
14:21 ranguard pombreda: it's a very big index - but I was just curious
14:22 pombreda mickey: once I get the base, I would just need increments, may be once a day at most?
14:22 mickey should be fine
14:23 mickey you can also get the base in time ranges... to make it in smaller chunks
14:23 ranguard pombreda: we have quite a bit of capacity... at least currently :)
14:23 mickey you'll probably have to ... because timeouts :)
14:23 pombreda mickey: so I think this mostly an issue for getting the base first.
14:25 pombreda ok, so to get the base, I can use the scroll API then, using small chunks based on time ranges?
14:25 mickey that should work
14:26 pombreda mickey, ranguard: thanks! What I can do is ping you to have a look at the code once I have a working prototype so you can check quick if that sounds reasonable
14:27 mickey pombreda: sure
14:27 * pombreda apologize as this will not be in perl.... though I would be quite happy to reuse existing perl code if there is some
14:27 pombreda ... that would do already such thing
14:27 mickey you mean like MetaCPAN::Client ? :)
14:27 ranguard pombreda: Metacpan::Client and https://github.com/metacpan/metacpan-examples are useful to look at
14:28 mickey but that's not a problem ... your issue is more about the ES query
14:28 pombreda yep
14:28 mickey just make sure not to scroll-bomb us
14:29 ranguard pombreda: please also add your details here: https://github.com/metacpan/metacpan-api/wiki/fastapi-Consumers  ( name / url for app / contact details / useragent if relevant)
14:29 pombreda I will for sure!
14:29 mickey i hope that was a reply for ranguard and not for my last comment :)
14:31 pombreda :D
14:32 pombreda that was indeed
14:32 mickey :)
14:32 * pombreda goes into his basement to manufacture scroll-bombs now ... whatever these are ;)
14:32 ilmari pombreda: they're like pipe bombs, but made of parchment
14:33 ilmari or papyrus
14:33 pombreda :D
14:33 mickey pombreda: i mean don't create too many scrollers at once... they have lifetime you set on creation, so you can make sure to wait it before created the next bunch
14:34 pombreda ilmari, actually they are made with cpanyrus
14:34 mickey otherwise - yay! free stress test
14:35 pombreda mickey: ;) agreed, in all cases I run only one request on a site at once, and never more than one per second (or less depending on robots.txt)
14:35 pombreda s/at once/at a time
14:37 mickey cool, i think the default lifetime is 1m
14:37 mickey but you can adjust it
14:38 dolmen joined #metacpan
15:29 metacpan joined #metacpan
15:29 metacpan [metacpan-client] mickeyn pushed 2 new commits to master: https://git.io/v1qxZ
15:29 metacpan metacpan-client/master 85113da Mickey Nasriachi: use Ref::Util for ref checks
15:29 metacpan metacpan-client/master 6791362 Mickey Nasriachi: documentation fix (download_url location)
15:29 metacpan left #metacpan
15:43 oalders Do we know why this is broken? https://travis-ci.org/metacpan/metacpan-web/jobs/179697915#L341
15:45 oalders ranguard: are we equipped to set up a staging box to test jberger's search changes?
15:46 oalders seems like something we should have moving forward if we're going to fiddle with search
15:46 ranguard oalders: don't know why broken - maybe a haarg or mickey question?
15:47 ranguard I have 'staging' box on my todo list, but want that to have a full set of the data on it as well
15:48 oalders could we have a staging "lite" to start with that just points at at fastapi?
15:48 oalders ie just the front end
15:48 * ranguard nods
15:48 ranguard I'll see what I can do
15:48 oalders ranguard++
15:49 haarg fc9054241698a5ef413b89b1bee2d352e39966e7 changed the file being checked and the expected result
15:50 haarg but i'm wondering if that expected result was based on the previous code where documentation wasn't being populated
15:50 haarg i reindexed all the perl releases to fix that, which is most likely why the test is now failing
15:54 oalders haarg: should i just revert the change to the regex?
15:56 haarg oalders: yeah
15:56 jberger once that is all in and working, then we can look at adding query parameters for the boosts
15:56 oalders haarg: ok. will do that in a bit
15:57 haarg the old title it was searching for is correct
16:28 metacpan joined #metacpan
16:28 metacpan [metacpan-web] ranguard created stage from jberger/web_search_endpoint (+0 new commits): https://git.io/v1mkp
16:28 metacpan left #metacpan
16:30 metacpan joined #metacpan
16:30 metacpan [metacpan-web] ranguard deleted stage at d29d96e: https://git.io/v1mIq
16:30 metacpan left #metacpan
16:31 metacpan joined #metacpan
16:31 metacpan [metacpan-web] ranguard created stage from jberger/web_search_endpoint (+0 new commits): https://git.io/v1mkp
16:31 metacpan left #metacpan
16:43 ranguard heh, we've hit Fastly's soft limit of 10 services...
16:44 * ranguard emails their support for more :)
16:46 dolmen joined #metacpan
16:47 metacpan joined #metacpan
16:47 metacpan [metacpan-web] oalders created oalders/fix-perl-test (+1 new commit): https://git.io/v1mqA
16:47 metacpan metacpan-web/oalders/fix-perl-test 1feea29 Olaf Alders: Partially revert change made in fc9054241.
16:47 metacpan left #metacpan
16:57 oalders can I get a review/merge of https://github.com/metacpan/metacpan-web/pull/1824 ?
17:00 ranguard done
17:00 metacpan joined #metacpan
17:00 metacpan [metacpan-web] ranguard pushed 1 new commit to master: https://git.io/v1mOY
17:00 metacpan metacpan-web/master 6176d35 Leo Lapworth: Merge pull request #1824 from metacpan/oalders/fix-perl-test...
17:00 metacpan left #metacpan
17:00 metacpan joined #metacpan
17:00 metacpan [metacpan-web] ranguard deleted oalders/fix-perl-test at 1feea29: https://git.io/v1mOO
17:00 metacpan left #metacpan
17:01 oalders ranguard++
17:04 ranguard oalders: https://web-stage.metacpan.org/
17:04 oalders ranguard++
17:04 ranguard jberger: https://web-stage.metacpan.org/ - that's running the 'stage' branch of mc-web (which I pushed from your branch)
17:05 ranguard this is backed by bm-mc-03 - with puppet runs on the hour (if you push some different code to stage)
17:06 ranguard it's a seperate fastly service, so you'll need to purge manually ( use https://manage.fastly.com )
17:07 ranguard service is called 'staging env for testing' :)
17:08 ranguard n.b I had some old local branches with 'stage' in the name, so needed to prune and delete otherwise git wasn't happy
17:11 Grinnz search does not seem to be responding
17:13 jberger https://fastapi.metacpan.org/search/web?q=WWW::Mechanize isn't responding either
17:13 ranguard hmm, load shot up on lw 3
17:14 oalders puppet run?
17:15 ranguard we seem to be back
17:15 ranguard I've not done anything, just seen the load coming down
18:41 neilb joined #metacpan
19:32 karjala To check if a distro exists today (and not only in backpan) I search the modules type for distribution=$distro, status=latest. Should I also add authorized=true in the criteria?
19:33 karjala and indexed?
19:50 karjala I added the criterion anyway
20:47 pombreda joined #metacpan
21:18 ranguard trs: ^^ where is that doc you wrote about all the flags ?
21:21 ranguard I can't see it in https://github.com/metacpan/metacpan-api/tree/master/docs - or am I missing something?
21:26 haarg https://github.com/metacpan/metacpan-api/blob/tsibley/document-indexer-flags/docs/indexing.md
21:28 ranguard that feels like we should get it merged :)
21:29 ranguard trs: we should merge this great pull request: https://github.com/metacpan/metacpan-api/pull/590 ;)
22:01 trs :)  I'll take a look again this evening.  I had some more scribbles to jot down in something more coherent, but it doesn't need to block the whole thing.
22:14 metacpan joined #metacpan
22:14 metacpan [metacpan-api] haarg deleted mickey/fix_cron_scripts_index at 54612e3: https://git.io/v1YTD
22:14 metacpan left #metacpan
22:14 metacpan joined #metacpan
22:14 metacpan [metacpan-api] haarg deleted mickey/fix_author_script at d57ab1b: https://git.io/v1YTy
22:14 metacpan left #metacpan
22:48 neilb_ joined #metacpan
22:55 karjala thanks
22:56 karjala I have a new problem. I'm trying to do a case-insensitive search. I fail. Can someone please help me transform this query into a case-insensitive one, so that it returns a result? (The real module name is XML::MyXML) http://pastebin.com/Nb1UAHWP
22:57 karjala I thought "match" was c.i.
22:58 karjala as opposed to "term"
22:58 karjala but I was wrong
23:55 alh Does metacpan have something in the UI to show who has access to a particular module space?
23:55 alh (release perms)

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary