Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-05-29

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:45 jar286 joined #opentreeoflife
01:45 jar286_ joined #opentreeoflife
01:46 jar286 left #opentreeoflife
12:02 josephwb joined #opentreeoflife
12:03 josephwb Mornin' ilbot3!
12:18 jar286 joined #opentreeoflife
12:20 josephwb joined #opentreeoflife
12:33 josephwb joined #opentreeoflife
13:18 kcranstn joined #opentreeoflife
13:22 josephwb joined #opentreeoflife
15:27 kcranstn jimallman?
15:27 kcranstn and josephwb, too
15:27 kcranstn Q about input trees
15:28 kcranstn checking suppl.  table 1 (the list of input trees) and that list does not match what at https://tree.opentreeoflife.org/about/references
15:28 kcranstn or $ curl -X POST http://api.opentreeoflife.org/v2/tree_of_life/about -H "content-type:application/json"  -d '{"study_list”:true}’
15:29 kcranstn unless the API returns an unsorted list
15:29 kcranstn and also list here : https://docs.google.com/spreadsheets/d/11T7O2Yre9SB3l1viokGlUsO3LVh3wgjf8_FEQHieXy0/edit#gid=947887154
15:29 kcranstn different again
15:30 kcranstn hello?
15:30 josephwb the problem is that treemachine disregards "trivial" subproblems
15:31 josephwb if a tree spans subproblems (not conflicting with taxonomy), it is not recorded
15:31 kcranstn what does that mean for the list of trees and tree ranks?
15:31 josephwb sorry; be back later; fixing computer
16:12 jar286 joined #opentreeoflife
16:14 * jimallman is back now, looking at the reference list
16:26 jar286 the API call lists 480 trees
16:26 jar286 curl -X POST http://api.opentreeoflife.org/v2/tree_of_life/about -H "content-type:application/json"  -d '{"study_list":true}' | grep tree_id | wc
16:27 kcranstn the results of that call have changed since the last pnas submissin
16:27 kcranstn submission
16:27 jar286 on ot10, I can see a list of preprocessed trees in ~/files.opentreeoflife.org/preprocessed/v3.0/trees
16:27 jar286 and there are 485 of them …
16:28 jar286 so joseph said something about treemachine disregarding useless trees, maybe that’s the difference between 485 and 480
16:29 jar286 the other QC check would be to make sure every tree cam from a study listed in any study list (e.g. the google doc)
16:29 jar286 s/cam/came/
16:30 kcranstn the ranking from the API call different from the list we submitted last time (to PNAS)
16:31 jar286 there are 477 studies listed in the google doc, which sounds about right
16:31 kcranstn that list had 476 trees
16:31 kcranstn but ranks completely different
16:32 jar286 I got 477, not 476… 488 rows minus 1 header row…
16:32 jar286 no matter…
16:32 jar286 s/488/477/
16:33 kcranstn I am comparing to the supplemental table 1 that we submitted with the previous pnas paper
16:33 kcranstn let me upload that as a google do
16:33 kcranstn c
16:35 jimallman kcranstn: the references list in the webapp is constructed using different API calls (tracing this now)…
16:35 jimallman main this one: http://api.opentreeoflife.org/treemachine/v1/getSynthesisSourceList
16:35 jar286 checked ‘Metadata for trees’ revision history, it’s from January
16:35 kcranstn yes, I created that from the API call after we got the revisions back
16:36 jimallman …and organizes these tree ids using study ids from this API call:  http://api.opentreeoflife.org/oti/v1/findAllStudies
16:37 josephwb in the previous submission there were 485. 1 was a duplicate
16:37 kcranstn https://docs.google.com/spreadsheets/d/1E44ijiNTfw0oZ4RhBtuUniGW8NygbL1DwWzjfhhkraw/edit#gid=1193745559
16:37 josephwb so we now have 484
16:38 kcranstn then the table we submitted last time was wrong? Contains 476 trees
16:38 josephwb i can fix that, but still working on my broken computer
16:38 josephwb ah, i know: is it listing only unique *studies*? some contribute 2 trees
16:39 jar286 the documentation for getSynthesisSourceList says nothing about sort order
16:39 kcranstn https://tree.opentreeoflife.org/about/references
16:39 kcranstn does
16:39 jar286 yes, there are 485 (or 480) trees, but only 477 studies
16:40 josephwb ok
16:40 jar286 or 476
16:40 josephwb 1 tree is a duplicate
16:40 kcranstn where is the ranking that you used for treemachine?
16:40 josephwb i will get it to you when i can get to my computer
16:40 josephwb hopefully soon
16:40 kcranstn not online anywhere?
16:41 josephwb probably?
16:41 josephwb yes
16:41 josephwb just not sure where at the moment
16:41 kcranstn sound like the references page shouldn’t show trees and priority ranked?
16:41 kcranstn trees *as* priority ranked
16:44 jimallman hm, the references page purports to show a priority-ranked (by default) list of studies. if it’s actually *trees* that have priority, and a study has two differently-ranked trees, this could go wrong.
16:44 josephwb trees get ranks, not studies
16:44 kcranstn but we aren’t even sure that the API returns sorted studies, right?
16:45 josephwb hmm
16:45 kcranstn the sorting https://docs.google.com/spreadsheets/d/1E44ijiNTfw0oZ4RhBtuUniGW8NygbL1DwWzjfhhkraw/edit#gid=1193745559
16:45 josephwb what call are you using?
16:45 kcranstn is completely different than http://api.opentreeoflife.org/v2/tree_of_life/about
16:45 kcranstn and therefore also https://tree.opentreeoflife.org/about/references
16:45 jimallman i believe the references page uses the order provided by oti’s findAllStudies method, which is supposed to return these in rank order
16:45 kcranstn ah
16:46 kcranstn but ranking on the references page does not match what we submitted to PNAS last time
16:46 jar286 doesn’t make sense that OTI would even know what the rank order is.
16:46 josephwb yeah
16:46 josephwb oti is not involved
16:47 jar286 getSynthesisSourceList  just uses the order from the  sourcenames  property of the ‘meta node’
16:47 kcranstn I was hoping to update Table 1 with the metadata from https://docs.google.com/spreadsheets/d/11T7O2Yre9SB3l1viokGlUsO3LVh3wgjf8_FEQHieXy0/edit#gid=947887154
16:47 josephwb i wouldn't trust that ordering
16:47 kcranstn that = ?
16:48 josephwb getSynthesisSourceList
16:48 kcranstn (sorry about your broken computer, BTW)
16:48 josephwb the creation of metadata nodes does not follow rank ordering
16:48 josephwb determined by subproblem loading
16:49 josephwb i don't believe the rank ording is stored anywhere in the graph
16:49 jar286 I’m creating an issue for the references  page
16:49 kcranstn thanks
16:50 josephwb ok, i can log in remotely. getting the list
16:52 jar286 https://github.com/OpenTreeOfLife/opentree/issues/646
16:52 kcranstn we also need to figure out how to expose this list through the APIs (and the UI)
16:52 kcranstn or, more specifically, expose the tree ranks
16:54 jimallman if we want “sort by tree rank” in the References page, i can create per-tree entries with repeated studies where needed.
16:55 kcranstn but it doesn’t seem that we have a service that returns tree ranks
16:55 josephwb i think that is correct
16:55 jimallman perhaps /v2/tree_of_life/about? checking this now...
16:55 kcranstn no, I already checked that
16:56 jimallman drat
16:56 josephwb it is not in the graph
16:56 josephwb ranks are on a subproblem basis, so no single list
16:56 josephwb ok, where should i send this list of 484 trees?
16:56 kcranstn to me
16:57 josephwb ok
16:57 kcranstn *crossing fingers that ranks have not changed*
16:57 josephwb sent
16:57 josephwb huh?
16:58 kcranstn from last submissin
16:58 kcranstn submission
16:58 josephwb there is 1 fewer (duplicate) tree
16:59 kcranstn whew! matches https://docs.google.com/spreadsheets/d/1E44ijiNTfw0oZ4RhBtuUniGW8NygbL1DwWzjfhhkraw/edit#gid=1193745559
16:59 kcranstn or, does at first glance
16:59 kcranstn looking closer
16:59 josephwb it is longer
17:00 kcranstn why?
17:00 josephwb i think the spreadsheet does not have more than 1 tree per study
17:00 kcranstn rows 20-21
17:01 josephwb hmm, then i don't know why the spreadsheet is wrong
17:01 josephwb the list i sent you is correct
17:02 josephwb some are trivial, so not stored in the graph or reported by treemachine services
17:02 * jimallman is adding my notes to jar286’s issue: https://github.com/OpenTreeOfLife/opentree/issues/646
17:03 kcranstn ok, this might not get re-submitted today if I have to update that table
17:03 josephwb it won't take long
17:04 kcranstn also want to add the metadata from the other spreadsheet
17:04 jar286 is it just a question of reordering the verbose table so it matches the tree order you just obtained?
17:04 josephwb some are missing
17:04 jar286 hmm, the tree id ought to go in that table too
17:04 josephwb 476 vs. 484
17:05 kcranstn I can’t keep track when we refer to “that table
17:05 kcranstn "
17:05 josephwb maybe we are talking about different things. which table?
17:05 kcranstn too many tables
17:05 jar286 right.
17:06 jar286 how about: Table W (for Want) has one row per tree, in rank order, with the detailed metadata from the ‘Metadata for Trees’ google spreadsheet
17:06 jar286 that would be the goal, right?
17:07 kcranstn Table W has the following columns: Rank, StudyID, TreeID, CitationFocal cladeData typeInference methodTree typeNotesCurator link
17:07 jar286 exactly.
17:08 josephwb i will paste in the missing entries into the spreadsheet
17:08 jimallman should i revise the References page to remove sort options, so as not to make promises we can’t keep?
17:08 jar286 sounds like a script will be needed to synthesize it from two sources (probably doable in excel, but that’s beyond my skill with excel)
17:08 kcranstn yes, please jimallman
17:08 jar286 yes
17:09 jar286 unless maybe Metadata for Trees is already in priority order ??
17:09 jar286 that would make life easier
17:09 kcranstn no
17:09 kcranstn it isn't
17:09 jar286 darn.
17:09 kcranstn because it came from an API call
17:10 jar286 ok, so this would be one of those 15-line python programs that reads two csv files and emits a third
17:10 jar286 (maybe 25 line)
17:11 jar286 josephwb, is it ever the case that study A has a tree whose rank is in between the ranks of two trees from study B?
17:12 josephwb could be
17:12 jar286 kcranstn, can you think of any (quicker) alternative to writing a script? I can’t
17:13 josephwb there are only 8 missing
17:13 kcranstn probably not (and doing this manually is likely to introduce errors)
17:13 kcranstn I will upload file from josephwb
17:13 jar286 8 - that’s exactly the number I got (see above 484-476).  good
17:14 josephwb pg_1366 is duplicated in your list
17:14 jar286 I need to get lunch. I know you’re capable of writing the script, but if you want me to help I can after lunch
17:14 josephwb it is easy enough
17:16 jar286 ok, I’ll be back soon
17:17 josephwb everything is fine until tree 461; script seems pointless
17:18 kcranstn ok
17:19 kcranstn so you are adding the 8 missing trees, all of the treeids and the ranks to “Inputs to synthesis v3.0”?
17:20 josephwb where is that? not the spreadsheet?
17:20 kcranstn https://docs.google.com/spreadsheets/d/11T7O2Yre9SB3l1viokGlUsO3LVh3wgjf8_FEQHieXy0/edit#gid=947887154
17:20 kcranstn the one wiht the metadata
17:20 josephwb i can do that. is it in the same order?
17:20 kcranstn no
17:20 josephwb crap
17:20 kcranstn there are no ranks
17:21 josephwb alright, i will do it on the condition that i move up 2 spots in the author order ;)
17:21 kcranstn ha
17:31 josephwb trees are in. where does the citation come from?
17:31 kcranstn API call or peyotl
17:31 kcranstn or manual cut-and-paste
17:32 josephwb i'll do the latter since there are so few
17:32 jimallman kcranstn: simplified (alpha-sorted) references list is working on devtree:  https://devtree.opentreeoflife.org/about/references
17:32 kcranstn thanks
17:32 jimallman if this looks good, i’ll make a simple PR
17:32 kcranstn looks good
17:34 josephwb fin
17:35 josephwb now onto the table. or do you want to do that?
17:35 kcranstn sure, on the table is good
17:36 josephwb ?
17:36 josephwb me or you?
17:36 kcranstn seems easier for you to paste in than email to me so I can paste in
17:36 josephwb need to reorder/match things, right?
17:36 kcranstn I assume so
17:37 josephwb this will take a script
17:37 kcranstn :)
17:37 josephwb why 2 spreadsheets?
17:38 kcranstn the one titled “Supplemental Table 1 (previous submission)” is what we submitted last time
17:38 kcranstn the one titled “Inputs to synthesis v3.0” is the new version and contains metadata about trees
17:39 josephwb why did i update the other one?
17:39 kcranstn I don’t knwo
17:39 josephwb arg
17:40 kcranstn conversation above at 1:20 seems to indicate we were talking about the same doc
17:41 josephwb too many tables
17:41 josephwb anywho, i got this.
17:41 kcranstn ok!
17:41 josephwb must. submit. paper.
17:41 kcranstn indeed
17:41 kcranstn I am getting the other docs uploaded
17:41 josephwb sweet
17:41 jar286 joined #opentreeoflife
17:41 jimallman kcranstn: thanks so much for wrangling this.
17:42 josephwb some cells in that sreadsheet are empty
17:42 kcranstn I know
17:55 kcranstn better most metadata than no metadata
17:58 kcranstn jimallman - https://percy.io/
17:59 jimallman neat!
18:34 josephwb joined #opentreeoflife
18:44 josephwb data are in the table. needs formatting
18:45 kcranstn I don’t see any changes
18:45 josephwb i reverted to try to format things in paste
18:45 josephwb see them now
18:45 josephwb ?
18:46 kcranstn ah, yes
18:46 kcranstn 484 is the correct number of trees?
18:46 josephwb yes
18:46 kcranstn awesome, thanks!
18:46 josephwb i think soemthing might have messed up in paste. just a sec
18:47 josephwb oh no, it is ok
18:48 josephwb wait, one more paste attempt
18:48 josephwb ok
18:48 kcranstn I think we should delete the Tree type column, because it wasn’t well-defined
18:49 kcranstn I suspect that many of the ‘species trees’ are supermatrix
18:49 josephwb yeah
18:49 josephwb pitch it
18:49 kcranstn poof!
18:49 josephwb notes are crappy too
18:49 kcranstn taht’s fine
18:49 kcranstn I might try and fill in a few more of the metadata fields
18:49 josephwb ok
18:50 josephwb not sure why the last few curator links are not hyperlinks
18:53 kcranstn it’s going to get saved as csv, so no worries about that
18:53 kcranstn thanks for your help!
18:53 josephwb ok, good
18:54 josephwb i need to go do other stuff now. you go submit.
19:30 kcranstn holy crap, the pnas submission process became much more onerous than the last time
20:07 kcranstn and different formats for resubmission than initial submissino
20:07 kcranstn submission
20:07 kcranstn ack

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary