Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-07-30

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:42 jar286 joined #opentreeoflife
10:25 scrollback1 joined #opentreeoflife
11:39 jar286 joined #opentreeoflife
13:03 jar286 joined #opentreeoflife
14:12 jar286 joined #opentreeoflife
15:17 kcranstn joined #opentreeoflife
16:24 jar286 joined #opentreeoflife
17:01 jar286 joined #opentreeoflife
17:24 pmidford2 joined #opentreeoflife
18:02 jar286 PR review …
18:02 jar286 I’m working on https://github.com/OpenTreeOfLife/germinator/pull/20 right now
18:03 jimallman yep, and as usual: https://github.com/pulls?user=OpenTreeOfLife
18:03 jar286 so nothing new this week.
18:04 jimallman stats is almost ready, i think. i’d like to incorporate pmidford’s latest tweaks to the stats JSON.
18:08 pmidford2 Let me know how that goes
18:08 jimallman OK, here are the loose ends: https://github.com/OpenTreeOfLife/opentree/issues/666#issuecomment-124135349
18:11 pmidford2 Thanks - so nothing new yet (read the comment when you posted it)
18:16 jimallman i haven’t made any changes  here yet… any progress on point #2 (clarifying OTT version used for each synth release)?
18:17 jar286 OTT 2.8 for synth v2 and v3.
18:17 jar286 I’ll look up v1, just a minute…
18:17 jimallman ah, thanks. i was hoping to get this as part of the routine stats in ‘synthesis.json’
18:18 jar286 July 23, 2014…
18:18 jimallman if we’re going to do these manually, I suppose I can add ‘synthesis_releases’ to the (already manual) ‘ott.json’
18:19 jar286 could have been 2.6 or 2.7, but my memory is that stephen never picked up 2.7, and stuck with 2.6 for the synthesis
18:19 jar286 yes, that sounds right
18:25 jar286 ok, https://github.com/OpenTreeOfLife/germinator/pull/20  is merged & closed
18:25 pmidford2 I could add these to synthesis.json if there is an api call that would tell me (or just hardcode it for you if that's the best we have).
18:25 pmidford2 Saw your gh post.
18:25 jimallman i seem to recall we’d have a hard time connecting these dots programmatically.
18:25 jar286 there is no retrospective api call for that information…
18:26 jar286 in future, maybe, but for the past ones it has to be manual…  I have a feeling that information is available from current api but could be wrong… isn’t there something that gives general information about the synthetic tree (or GoL)?
18:27 jimallman sounds familiar. this is worth a closer look,.
18:30 jimallman this API call works (returns a property “taxonomy_version" with a value like "ott2.8draft5"), but only for the current draft tree: curl -X POST https://devapi.opentreeoflife.org/v2/tree_of_life/about
18:31 jimallman same information for the current graph (will this always match the above?):  curl -X POST https://devapi.opentreeoflife.org/v2/graph/about
18:34 jar286 yes, will always match
18:34 jimallman so a new graph implies a draft tree as well?
18:35 jimallman i’m concerned if someone’s looking at a “fresh” graph but a “stale” draft tree.. but maybe these can never co-exist in a deployed system?
18:36 jimallman pmidford2: if hard-coding these OTT versions in synthesis.json makes sense, that’s OK by me. it does suggest we put that file under version control, if only to preserve these versions. so far i’ve treated stats as ephemeral, except for the manually-edited ‘ott.json’ which is in git.
18:37 jimallman if synthesis.json is otherwise automatically generated, i’m inclined to put all the manual effort into a single file (ott.json)
18:37 pmidford2 We could just put the python that generated them under vc, right?
18:37 jimallman ! ah, i see.
18:38 jimallman yes, that works for me. and my code is happy to use any values that pop up in synthesis.json (as property ’OTT version’)
18:38 pmidford2 got it
18:39 jimallman in the event that we’re slow to update that script, my code will make the current (dumb) assumption that the a synthesis release used the most recent taxonomy version. which is not terrible.
18:40 jimallman (assuming that the ‘OTT version’ property is omitted if there’s no clear mapped value for a synth. release)
18:41 pmidford2 makes sense
18:42 jimallman thanks! this should Just Work once your change is made (and the script is run, i guess). if you can’t regenerate past synth statistics to add this property, i guess it’ll require manual changes this once.
18:45 pmidford2 right, I don't think I've done anything historical with synthesis (just phylesystem)
18:58 jimallman pmidford2: regarding point #3 in my comment, am i correct that you recently added something to the statistics? i’m checking germinator for likely recent changes but not finding any.
18:59 pmidford2 I'm not sure how recent the changes were
18:59 jimallman i’m seeing the “latest” commits back in January.
18:59 pmidford2 That's actually about right.
19:00 pmidford2 jimallman I wasn't sure when you had looked at this
19:00 jimallman hm, i believe my stats logic is already using that output. will confirm.
19:01 jimallman i thought there was a more recently added property, but i can’t recall if i read that in an email, or chat, or it came to me in a dream. i’ll drop it if i don’t find something soon.
19:09 jimallman found it! pmidford2, i’m looking at an email from you on July 9, subject is “[OpenTree-software] Re: Statistics update”
19:11 jimallman it describes ‘nominated_study_unmapped_OTU_count’ in the phylesystem results. but i don’t see this name in the current germinator code. should i be looking somewhere else?
19:11 pmidford2 No, that was just the documentation in the gh thread - updated it to match the wiki - sorry for the confusion
19:11 jimallman i did find one instance on github, in peyotl:  https://github.com/OpenTreeOfLife/peyotl/blob/86e77b6e532c914fffd4df6e7dd6060dadcad579/examples/phylesystem/ott_status_report.py#L29
19:11 jimallman ah, gotcha
19:11 jimallman i guess this comes from peyotl then
19:12 pmidford2 Right - this is a TBD for me as well
19:12 pmidford2 Except that jar doesn't think counts of unmapped OTUs are necessarily that useful
19:13 pmidford2 I think that's why I didn't go off and implement it
19:16 jimallman agreed. i gather this is not quite the same as “OTUs in phylesystem, but not in synthesis”, so i won’t use it on the Progress page.
19:17 pmidford2 I wouldn't interpret those as the same thing either
19:18 jimallman i currently calculate this from other synthesis stats, subtracting synthesis.total_OTU_count from phylesystem.unique_OTU_count
19:18 jimallman so i’ll remove Step 3 entirely from my issue above.. Progress!
19:29 pmidford2 updated the repository statistics wiki page to reflect that we're not implementing this field
19:31 jimallman pmidford2: it’s not implemented? i see ‘nominated_study_unmapped_OTU_count’ everywhere in these stats:  https://devtree.opentreeoflife.org/static/statistics/phylesystem.json
19:32 pmidford2 Ah, must have gotten the wrong field.
19:32 jimallman like i said, it’s defined in peyotl:  https://github.com/search?q=nominated_study_unmapped_OTU_count&type=Code&utf8=%E2%9C%93
19:38 jimallman …assuming this is an input to your synthesis stats. it’s in the peyotl ‘examples’ folder, so probably not.
19:43 pmidford2 OK, that field is there, but it has been all along.  So I guess step 3 is still possible, but isn't necessary.
19:47 pmidford2 I could go ahead and remove it from the reports
20:01 jimallman i’d leave it, if there’s any chance it would be useful to someone else.
20:01 pmidford2 ok
20:12 jar286 pmidford2, ot10 is a sui generis server in that it handles files.opentreeoflife.org, which is halfway outside the deployment, and the cron job, which should only exist on a single server (we don’t have dev & production cron jobs, or replication, right?)
20:12 jar286 so I don’t mind the cron job being manually set up, although the setup should be described in a wiki page or other doc
20:12 jar286 s/the deployment/the deployment system/
20:13 pmidford2 ok, makes sense.  I'll find or create an appropriate wiki page then?  Or document some other way?
20:14 jar286 where is the stats setup documented?… I’ll look, I would expect an opentree wiki page, or germinator
20:15 pmidford2 <jar286> There was going to be a production cron job at one point, but we never put the hooks on the production machine.
20:15 pmidford2 On germinator
20:15 pmidford2 e.g., https://github.com/OpenTreeOfLife/germinator/wiki/Overview-of-repository-statistics
20:16 jar286 yep.  I would put it there
20:18 jar286 jimallman, are webapp requests to neo4j made using https ?
20:19 jar286 (just off the top of your head. if you don’t know I can check myself)
20:20 jimallman it’s a quick check… one sec
20:22 jimallman they currently “follow suit” based on the user’s connection. so in production we’re all-HTTPS, but in a local development server it could be either HTTP or HTTPS.
20:22 jimallman but generally, yes.
20:24 jimallman would a ‘crontab.example’ be useful, maybe in the root of the germinator repo?
20:25 jimallman ah, now i see the examples now in the wiki page. never mind.
20:26 jar286 tnx
20:30 jimallman so i guess the revised (monthly) crontab line would be something like “0 0 1 * * /home/opentree/statistics/phylesystem_stats.sh …”
20:31 jimallman or “@monthly      /home/opentree/statistics/phylesystem_stats.sh …”, i guess
20:31 jar286 maybe we should do them weekly, and the webapp could ignore the ones in between… ?  redundancy, just in case something goes wrong
20:31 jar286 and we should be backing up the stats  (I’m backing up various other things, could add to list)
20:32 jimallman yes, i can filter out first-of-month (or closest previous) from weekly stuff
20:35 jimallman so cron might be “0 0 1,8,15,22 * * /home/opentree/statistics/phylesystem_stats.sh …”
20:35 jimallman ie, capture stats on the first, eigth, fifteenth, and twenty-second of the month.
20:41 jar286 2,3,5,7,11,13,17,19,23,29,31   just kidding
20:47 jimallman lemme just whip up a little fibonnaci sequence in bash…
22:18 kcranstn joined #opentreeoflife
22:24 kcranstn joined #opentreeoflife
22:35 kcranstn joined #opentreeoflife
23:26 kcranstn joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary