Perl 6 - the future is here, just unevenly distributed

IRC log for #metacpan, 2016-05-20

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:46 bpmedley joined #metacpan
02:14 oalders ether: i think that problem is hard to solve in one query with the v0 api.  i'm not sure there's a bug on the cpanm side other than it could also be checking 02packages or making a second query, i guess
02:41 ether the v1 API isn't up yet, is it?
03:05 Khisanth joined #metacpan
04:18 vanstyn_ joined #metacpan
06:04 oiami joined #metacpan
07:06 neilb joined #metacpan
07:59 neilb joined #metacpan
08:05 Relequestual joined #metacpan
08:19 neilb joined #metacpan
09:45 ranguard So V1 cluster has some issue, mickey restarted all the ES's which isn't usually a good idea
09:45 ranguard curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' is useful
09:45 ranguard restarting 1 node is ok, but should then wait for the status to be green again before restarting another one
09:46 ranguard we're going to give ES some time to try sort itself out, but we might have hit split brain, not sure
09:47 ranguard localhost:9200/_plugin/kopf/ ( if you open an ssh tunnel ) or http://www.elastichq.org/ (again over ssh tunnel) are useful tools
09:47 * ranguard is writing this here for the log mostly
09:48 * mickey thinks it should go into the maintenance manual that ranguard is writing :)
09:49 ranguard mickey: it's already written: https://www.elastic.co/guide/en/elasticsearch/guide/2.x/administration.html :)
09:50 ranguard https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_rolling_restarts.html
09:50 mickey oh yeah... but we need the metacpan troubleshooting summary... you don't want me to start reading THAT while stuff is broken
09:51 mickey k, biab
09:51 * ranguard has never seen this sort of broken before I'm afraid :)
09:52 ranguard oh, does look like it's fixing itself
09:54 ranguard https://gist.github.com/ranguard/74f9fc695cedbb1a8124c750f1f7f256 for reference
10:03 mickey green again (all shards initialized)
10:03 mickey still getting 500s on queries though
10:11 ranguard Could not retrieve a list of active servers:
10:12 mickey yeah /_cluster is not returning anything
10:12 mickey specifically /_cluster/nodes
10:12 ranguard that's because that's OLD es
10:13 mickey that's the error that keeps filling the log
10:18 ranguard we had this because of a duff snapshot a while ago
10:19 ranguard wierd the error talks about /home/metacpan/carton/metacpan-api/lib/perl5/ElasticSearch/Transport/HTTPTiny.pm - but that doesn't exist
10:21 ranguard 'ElasticSearch' shouldn't be being used
10:22 ranguard oh, this is watcher
10:24 ranguard which hasn't been restarted since Mar30
10:28 ranguard ok, seems alright now - but I'll need to do some puppet cleanup (I'll get it to restart each time the api code is updated)
10:49 mickey nice catch :)
14:45 jnbek joined #metacpan
14:46 oalders mickey: what was the reason for restarting ES on all the boxes?
14:47 mickey there errors
14:47 mickey they were there first :)
14:48 mickey it seems like ES nodes suddently lost communication
14:49 mickey just got 500s for queries and internally the log was full of "ElasticSearch::Error::NoServers" errors
14:58 ranguard mickey: but it wasn't ES, it was the old watcher code still running
14:58 ranguard oalders: ^^
14:59 mickey ranguard: you found that later :)
15:00 mickey TIL: kopf
15:00 mickey nice tool
15:00 oalders was just wondering what set the chain of events in motion :)
15:01 mickey oh, that was me seeing the errors and attempting to "fix it"
15:01 oalders :)
15:01 oalders so, once ranguard tweaks the watcher to restart on updates, that'll be helpful
15:01 mickey TIL 2: don't restart all ES servers at once :)
15:02 oalders heh
15:02 oalders anything else we should be restarting?  rrr-client could need a restart if some of our modules are updated
15:08 _dolmen_ joined #metacpan
15:12 castaway joined #metacpan
15:33 metacpan joined #metacpan
15:33 metacpan [cpan-api] mickeyn created mickey/fix_documentation_mapping (+1 new commit): https://git.io/vrztD
15:33 metacpan cpan-api/mickey/fix_documentation_mapping c6d204d Mickey Nasriachi: Fix 'documentation' mapping...
15:33 metacpan left #metacpan
15:41 mickey joined #metacpan
15:46 metacpan joined #metacpan
15:46 metacpan [metacpan-web] mickeyn created mickey/single_valued_arrayref_to_scalar (+1 new commit): https://git.io/vrzmQ
15:46 metacpan metacpan-web/mickey/single_valued_arrayref_to_scalar 7394ce6 Mickey Nasriachi: Moved Elasticsearch::Adapter (single_valued_arrayref_to_scalar)...
15:46 metacpan left #metacpan
16:08 neilb joined #metacpan
16:20 neilb joined #metacpan
17:00 neilb joined #metacpan
17:51 metacpan joined #metacpan
17:51 metacpan [cpan-api] mickeyn force-pushed mickey/fix_documentation_mapping from c6d204d to 3350fe3: https://git.io/vrzVT
17:51 metacpan cpan-api/mickey/fix_documentation_mapping 3350fe3 Mickey Nasriachi: Fix 'documentation' mapping...
17:51 metacpan left #metacpan
18:01 metacpan joined #metacpan
18:01 metacpan [cpan-api] oalders pushed 1 new commit to mi/es2: https://git.io/vrzrC
18:01 metacpan cpan-api/mi/es2 3a5c168 Olaf Alders: Merge pull request #484 from CPAN-API/mickey/fix_documentation_mapping...
18:01 metacpan left #metacpan
18:08 metacpan joined #metacpan
18:08 metacpan [metacpan-web] oalders pushed 1 new commit to mi/es2: https://git.io/vrzog
18:08 metacpan metacpan-web/mi/es2 0910337 Olaf Alders: Merge pull request #1696 from CPAN-API/mickey/single_valued_arrayref_to_scalar...
18:08 metacpan left #metacpan
18:08 metacpan joined #metacpan
18:08 metacpan [metacpan-web] oalders deleted mickey/single_valued_arrayref_to_scalar at 7394ce6: https://git.io/vrzo2
18:08 metacpan left #metacpan
18:10 ranguard mickey: so watcher seems to fall over on lw-mc-03 /etc/init.d/metacpan-watcher start
18:11 ranguard /var/log/starman/metacpan-api/metacpan.log gets an entry, but after that hte process is not running
18:11 oalders ranguard: could we set the watcher up on one of the other boxes?
18:11 oalders and move it to use the queue?
18:12 oalders obviously doesn't fix the issue on lw3
18:12 ranguard oalders: it wasn't working before afaikt
18:13 ranguard it shouldn't have been running on all the lw boxes (I'm fixing that)
18:13 oalders :)
18:13 ranguard using the queue is for you/mickey to work out how to best spend your time :)
18:14 oalders indeed. i think the watcher is a good candidate for that
18:27 ranguard mickey: Not a HASH reference at /home/metacpan/carton/metacpan-api/lib/perl5/Search/Elasticsearch/Role/Client/Direct.pm line 15.
18:28 ranguard if on lw-mc-03 you run: /home/metacpan/bin/metacpan-api-carton-exec /home/metacpan/metacpan-api/bin/metacpan watcher
18:41 mickey ranguard: checking
19:08 mickey ranguard: it looks like the code using bulk is not up to date ... i'll fix it
19:14 ranguard mickey++
19:21 metacpan joined #metacpan
19:21 metacpan [cpan-api] mickeyn created mickey/test_watcher (+1 new commit): https://git.io/vrzHb
19:21 metacpan cpan-api/mickey/test_watcher 6eab0ae Mickey Nasriachi: updating bulk code for watcher script
19:21 metacpan left #metacpan
19:25 metacpan joined #metacpan
19:25 metacpan [cpan-api] mickeyn force-pushed mickey/test_watcher from 6eab0ae to 72ed180: https://git.io/vrzQM
19:25 metacpan cpan-api/mickey/test_watcher 72ed180 Mickey Nasriachi: updating bulk code for watcher script
19:25 metacpan left #metacpan
19:29 metacpan joined #metacpan
19:29 metacpan [cpan-api] mickeyn force-pushed mickey/test_watcher from 72ed180 to a4ae567: https://git.io/vrzQM
19:29 metacpan cpan-api/mickey/test_watcher a4ae567 Mickey Nasriachi: updating bulk code for watcher script
19:29 metacpan left #metacpan
19:35 _dolmen_ joined #metacpan
19:43 metacpan joined #metacpan
19:43 metacpan [cpan-api] mickeyn force-pushed mickey/test_watcher from a4ae567 to 9965142: https://git.io/vrzQM
19:43 metacpan cpan-api/mickey/test_watcher 9965142 Mickey Nasriachi: updating bulk code for watcher script
19:43 metacpan left #metacpan
19:45 mickey ranguard: https://github.com/CPAN-API/cpan-api/pull/485
19:46 mickey tested on lw-mc-03 ;)
19:58 ranguard mickey: thanks
19:59 mickey np, do you want to merge so I can deploy before starting the re-index?
20:04 metacpan joined #metacpan
20:04 metacpan [cpan-api] ranguard pushed 2 new commits to mi/es2: https://git.io/vrzN1
20:04 metacpan cpan-api/mi/es2 9965142 Mickey Nasriachi: updating bulk code for watcher script
20:04 metacpan cpan-api/mi/es2 72a38e9 Leo Lapworth: Merge pull request #485 from CPAN-API/mickey/test_watcher...
20:04 metacpan left #metacpan
20:04 ranguard give me a mo, just sorting out puppet as well
20:05 mickey sure, waiting for your go
20:19 ranguard mickey: go go go :)
20:19 ranguard I've run puppet on all boxes, so code is deployed
20:19 mickey cool
20:19 * mickey is rebooting the world
20:19 ranguard watcher is running on lw-03 - but haven't got it automatically restarting when you run puppet
22:29 _dolmen_ joined #metacpan
22:38 neilb_ joined #metacpan

| Channels | #metacpan index | Today | | Search | Google Search | Plain-Text | summary