Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-08-16

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:13 pmidford2 joined #opentreeoflife
01:47 ilbot3 joined #opentreeoflife
01:47 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
02:31 jimallman joined #opentreeoflife
07:08 ilbot3 joined #opentreeoflife
07:08 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
10:50 jimallman joined #opentreeoflife
14:12 jimallman joined #opentreeoflife
15:17 pmidford2 Morning Jim.  I'm working on installing the scripts on the production machine.  No problems so far.
15:17 pmidford2 <jimallman>
15:17 jimallman sounds good. just reviewing my pages now...
15:18 jimallman pmidford2: just curious, why do we gather dev stats? it seems more useful at this point to copy production stats to the dev server and test against those.
15:19 pmidford2 The main reason for gathering dev stats was because we didn't thing we were ready to install stuff on production.
15:20 jimallman ok. in that case, i’m tempted to copy stats from production => dev once they’re ready (are they ready now?)
15:20 pmidford2 I'm trying to remember why we gather on production and then push the files back to production.
15:21 jimallman i’m having trouble parsing that sentence
15:21 pmidford2 The history script for both production and dev is done.  I'll put them up on dev
15:22 jimallman them = the resulting stats?
15:22 pmidford2 Yes
15:22 jimallman sounds good, thanks
15:24 pmidford2 Sorry for the confusion.  I need to think about this some more.
15:25 jimallman ok. if it helps, i’m interested in seeing the production stats on the dev About page, so we can see whether dates and counts are accurate (vs. the “history” of dev, which is probably fairly random)
15:29 pmidford2 Ok, I have put the dev history on devhistory.json and the production history on production.json.  I think the existing dev log probably has some production history aways back (I may have pointed the harvester script at the wrong phylesystem).
15:31 jimallman i’m familiar with ott.json, phylesystem.json, synthesis.json. what do you mean by “history”?
15:31 pmidford2 To clarify, the files I just uploaded were from the phylesystem repos I found from the curator page on Friday afternoon.
15:32 pmidford2 Ok, the devhistory.json is the phylesystem.json I generated from the dev repo, the production.json is the phylesystem.json I generated from the production repo.
15:33 jimallman gotcha. thanks! ok to rename these for testing purposes?
15:34 pmidford2 Sure, that should be fine (not going to lose anything, since they are from Friday and yesterday).
16:59 jimallman pmidford2: i’m ready to try moving the *history.json files. i thought they’d be on production tree.opentreeoflife.org (ot14), but i don’t see them there. where should i be looking?
17:00 pmidford2 sorry, they're on ot10, where I've been running the crontabs.
17:00 jimallman ah, ok. thanks
18:04 jimallman pmidford2: where are the most current synthesis stats? any chance we’ll be able to set the ‘OTT version’ for each release? (i can these edits manually, if need be, but i’ll need a bit more information)
18:05 pmidford2 Yes, we certainly can set 'OTT version'  I don't know if I have that change running at the moment - let me check
18:07 * jimallman is reviewing our prior converation on IRC:  http://irclog.perlgeek.de/opentreeoflife/2015-07-30#i_10979480
18:12 pmidford2 Ok, the version thats running on ot10 did not have the 'OTT version' change.  I have updated that doing a test run.  If it works there will be a version of the synthesis stats with the 'OTT version'
18:13 jimallman thanks!
18:14 jimallman that’ll remove the last lingering “warning” about our stats:
18:14 jimallman (seen in red here):  https://devtree.opentreeoflife.org/about/progress
18:15 pmidford2 Except that it will only work going forward - we don't have any way to regenerate history for synthesis
18:17 jimallman gotcha. we talked (see IRC link above) about adding ‘OTT version’ manually for old synthesis releases, and jar286 helpfully provided the values for historic releases.
18:17 jimallman i can make those changes manually once your run is complete.
18:17 pmidford2 got it
18:17 jimallman i’m also planning to clobber lots of the unwanted releases, but it would help to know which dates (just a few) are the “official” releases.
18:36 pmidford2 Ok, a synthesis report with the last entry including the OTT version should in the dev server web2py area, also in ot10://opentree.statistics/devstats/synthesis.json
18:37 pmidford2 I'm a little worried that the git merge for all of this will be a mess.
18:58 pmidford2 I'm going to run a couple of errands.
18:59 pmidford2 left #opentreeoflife
20:21 pmidford2 joined #opentreeoflife
20:42 jimallman hi pmidford2 (hope you had a good run!)
20:43 jimallman do you know which of the (many) releases in synthesis.json is an “official” synthesis release, or how I could determine this?
20:47 jimallman jar286 describes these as v1, v2, etc. in previous conversation, but i don’t know where these are documented: http://irclog.perlgeek.de/opentreeoflife/2015-07-30#i_10979487
20:52 pmidford2 Hmm....
20:56 pmidford2 Unfortunately, the stats job just grabbed what was available - snapshots driven by crontab. I don't recall how synthesis releases were announced, if there was any formal announcement.
21:04 jimallman ok, thanks. i’ll look around and check with jar286 if i come up empty.
21:05 jimallman ah, this page describes v1, v2, and v3. i’ll try to find these records in the stats and remove the rest: http://files.opentreeoflife.org/trees/
21:08 pmidford2 Sounds good.
21:12 jimallman pmidford2: argh, all the official relases were in 2014. but all the records in synthesis.json are for 2015! thoughts?
21:18 pmidford2 No, there might be something back to November, but it looks like we would need back to September, and I hadn't started working on stats back then
21:19 jimallman can stats be reconstructed after the fact?
21:19 jimallman (i’m guessing not...)
21:20 pmidford2 They are based on API calls; unless there is something we could restore the (neo4j?) database from, I don't know how we could reconstruct.
21:21 jimallman gotcha. hm, i suppose we could add “ghost” entries for these in synthesis.json
21:21 pmidford2 Sounds like the best of a poor set of options
21:21 jimallman perfect!
21:22 jimallman :D
21:22 pmidford2 :D
21:26 jimallman so the latest (2015-08-16T18Z) synthesis entry, marked with “OTT_version”: “ott2.9draft8”… is this an important/official release (v4, perhaps)? or does your script just mark the last record by date?
21:27 jimallman and then i’ll leave you alone, i promise :)
21:28 pmidford2 No, just what the API returned at that time.  I think this was against the devapi.  Let me try the production API and see what it returns.
21:31 jimallman gotcha. that makes sense.
21:32 jimallman glad we’re using the API now. the old chat said these values were going to be hard-coded in your script, but of course this is better.
21:37 pmidford2 Right, I wouldn't see much value in hardcoding that in the script.
21:49 jimallman it sounds like these stats should be gathered manually (when someone had declared an official release), since we’re still getting daily updates added.
21:49 pmidford2 That's easy to fix, reset the crontab to once/month
21:50 pmidford2 the string I get from the production api is "ott2.8draft5"
21:50 * jimallman nods
21:51 pmidford2 but changing the crontab doesn't make it trigger on release.  The script is easy enough to call though.
21:52 jimallman but it’s not even monthly, right? the historic releases are v1 (April 2014), v2 (Sept 2014), v3 (May 2014)  — incidentally, i’m guessing that last date is wrong and it’s v3 (May 2015)
21:52 jimallman sorry, just read your last comment.
21:52 jimallman yes, a manual trigger seems to be the right answer, along with passing in a release version identifier (unless this can also be captured programmatically)
21:53 pmidford2 Agree - the handwritten date is wrong, look at the Data of construction timestamp
21:54 pmidford2 Or just release and trigger the script as part of cleanup once the api is updated.
21:54 pmidford2 s/Data/date
22:07 jimallman i’ve submitted a PR with this correction: https://github.com/OpenTreeOfLife/files.opentreeoflife.org/pull/2
22:15 pmidford2 sounds good

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary