Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-10-10

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:31 towodo joined #opentreeoflife
01:20 towodo joined #opentreeoflife
06:43 mtholder joined #opentreeoflife
07:36 jimallman joined #opentreeoflife
11:45 josephwb joined #opentreeoflife
12:12 josephwb you there jimallman?
12:23 towodo joined #opentreeoflife
12:47 josephwb you there towodo?
12:47 towodo sort of
12:47 josephwb i tried pushing a db to ot19
12:47 josephwb not sure if it worked
12:48 josephwb bunch of errors
12:48 towodo out of space?
12:48 josephwb no, neo4j errors
12:48 josephwb just a sec
12:49 josephwb just warnings, maybe
12:49 josephwb maybe it worked
12:50 towodo you followed the deploy/README.md instructions?
12:50 josephwb yes
12:51 josephwb just a little worried, since changes were made to the dev*.config files, but not the ot19.config
12:51 towodo I doubt that matters, but might be worth looking at the diffs
12:52 josephwb it would be nice if the web tree viewer could display the "about" information
12:52 josephwb just to make sure i am looking at the new tree, and not the old one
12:52 towodo I think so too. file an issue
12:53 josephwb will do
12:53 josephwb i thin kthings are fine; will look for parts of the tree that should differ between the old and new trees
12:53 josephwb thanks
12:54 towodo just looked at the deployed-systems diff and none matters for ot19
12:54 josephwb great
12:54 towodo back in a bit
12:55 mtholder joined #opentreeoflife
14:18 jimallman joined #opentreeoflife
14:45 towodo joined #opentreeoflife
14:57 josephwb you there jimallman?
14:57 jimallman yes, hi!
14:57 josephwb i am running into a problem.
14:58 josephwb nearly every nexson i am downloading is dying in treemachine. when I copy-and-paste a replacement tree newick in the curator, things work again.
14:58 josephwb i am stumped
14:59 jimallman by “replacement tree newick”, do you mean you’re clobbering the only tree in a study and replacing it with fresh imported tree from newick?
14:59 josephwb i get the newick from the curator, reformat it slightly (e.g. replace spaces with underscores)
14:59 josephwb sorry, let me bck up
14:59 jimallman thanks
14:59 josephwb 1) study doesn't work
14:59 josephwb 2) go to curator, download newick
14:59 jimallman in treemachine...
14:59 josephwb 3) reupload newick
14:59 jimallman doesn’t work in treemachine, right?
15:00 josephwb yes, in treemachine. something with "names"
15:00 josephwb 4) things work
15:00 jimallman are you mapping the OTUs in original or replacement tree?
15:00 josephwb these particular trees have (arbitrary) branch lengths. i remove these as well, so that could be a problem
15:00 jimallman if not, the new tree might be a no-op in treemachine
15:01 josephwb i am doing NO remapping at all
15:01 josephwb (probably should...)
15:01 jimallman scary, you might be undoing someone’s mapping work
15:02 josephwb i don't know what you mean by "no-op". should be exact same name
15:02 josephwb http://tree.opentreeoflife.org/curator/study/edit/ot_201/?tab=otu-mapping
15:02 josephwb this is one where i did not delete the offending tree
15:02 jimallman maybe i don’t understand how nexson feeds treemachine. i would have assumed that mapped OTUs (those with a discoverable ottid) are treat better, and that unmapped OTUs might even be ignored.
15:02 josephwb i do not see new names
15:03 josephwb yes: unmapped taxa are ignored.
15:03 * jimallman is reviewing the trees now in curation app
15:03 jimallman different rooting (ignoring this for now)
15:03 josephwb i *think* the nexson it gets might have some invalid field, so that no taxa have valid ids
15:04 josephwb same rooting
15:04 josephwb i did not change the newick in that way
15:04 jimallman my bad, i was distracted by branch lengths in the old tree
15:04 josephwb yes
15:04 josephwb got rid of those (arbitrary)
15:05 jimallman nice! the previous mapped OTUs “stay mapped” in the new tree.
15:05 josephwb right
15:05 josephwb the names should match perfectly
15:05 jimallman (this might be a happy consequence of keeping the old tree around, at least until the import is complete. i was afraid the old OTUs would be cleared otherwise.)
15:06 josephwb let me check another study
15:06 josephwb this is one where i deleted the "bad" tree: http://tree.opentreeoflife.org/curator/study/edit/ot_203/?tab=otu-mapping
15:07 josephwb names look fine
15:07 jimallman ok, so it’s nothing obvious. i’d start by diffing the old and new Nexson files (feel free to email them, or we can try it on GitHub if you’ve saved these changes in the curation app)
15:07 josephwb ok, i have a copy of one of those ready to go
15:07 jimallman re: ot_203, did you delete the old before or after importing the new one?
15:07 josephwb no, only delete things when i know that the new one works
15:08 jimallman gotcha, thanks
15:08 josephwb so, after
15:08 jimallman i have a hunch that “before” would result in losing the “extra” OTUs , so try this first without saving.
15:08 jimallman onward, to the diff-mobile!
15:09 jimallman starting from here: https://github.com/OpenTreeOfLife/phylesystem-1/commits/master/study/ot_01/ot_201/ot_201.json
15:09 josephwb hmm. "extra" OTUs? i don't see those
15:10 jimallman when we import a tree, i think they would be seen as such during the server-side merge …
15:10 jimallman if the old tree was deleted in the curation app.
15:10 jimallman the server does some cleanup and consolidation, and i thought removing “unused” OTUs was part of that. i could be wrong!
15:13 jimallman ok, the basic diff from your last edit + save (URL above) looks sensible… i’m pulling the latest nexson into vim, to compare tree1 and tree2...
15:15 jimallman differences:
15:15 jimallman tree2 has a stored MRCAName, MRCAOttId (shouldn’t matter to treemachine)…
15:15 josephwb ok, i sent the "nexsa" through email as well
15:16 jimallman both trees have ingroup clade, different node IDs but that’s probably right...
15:17 josephwb node IDs?
15:18 jimallman one thing jumps out at me, and it’s not good.
15:18 josephwb oh, phuck…
15:18 jimallman it looks like tree1 has a (badly) malformed annotation
15:18 josephwb ah!
15:19 josephwb yes, that must be it
15:19 josephwb all of these "bad" studies were uploaded by the same person
15:19 josephwb what is the annotation?
15:19 jimallman hm, now it looks mostly ok…
15:19 jimallman look for the ^ot:messages property
15:20 jimallman should be just one message, with @code ‘SUPPORTING_FILE_INFO’
15:20 jimallman apologies, this actually looks OK. my vimdiff display was garbled for some reason
15:21 jimallman so on to branch lengths…
15:21 jimallman (as a sidenote, the bogus “file info” in tree2 is because you pasted the newick instead of uploading a file. this will be fixed! in the next push to production)
15:22 jimallman (@filename and @url, i mean)
15:22 jimallman simplest explanation: is it possible that treemachine doesn’t expect an edge with a @length property?
15:23 josephwb hmm.
15:23 josephwb i need to check what we use for branch lengths
15:23 josephwb but, yes, we read them in and store them for sure
15:24 josephwb no, we read in "length" (looking at an older study that I know works)
15:25 josephwb i mean, "length" should not be the culprit
15:25 jimallman problem might be length vs @length, then? i was using Badgerfish convention here
15:26 jimallman what’s the older study?
15:26 josephwb sorry, we read in "@length"
15:26 josephwb NOT "length"
15:26 josephwb older study = pg_420
15:27 jimallman ok, good to know @length is working.
15:27 jimallman can you tell me more about the treemachine error msg?
15:28 josephwb error messages are suppressed the way I am calling it, but it dies with one taxon
15:28 josephwb i believe it prunes "bad" taxa until 1 is left, then it dies
15:29 josephwb basically, an empty tree, because no valid taxa
15:29 josephwb i will run it differently and let you know the specifics; will take a bit to set up (working from home)
15:30 jimallman sorry for the hassle, but i’m stumped at this point..
15:33 josephwb hmm, seems *no* valid taxa are coming through
15:33 josephwb i see what you mean by different node IDs
15:34 jimallman that’s to be expected (node ids are unique within the study)
15:35 josephwb right, as they should be
15:35 jimallman but that’s why i’m pleased to see that OTU mappings were carried over.. not sure how Mark is pulling that off. maybe exact match on original node labels?
15:36 josephwb this might take a bit to track down what is wrong
15:36 josephwb feels like an incorrect format problem to me.
15:37 jimallman wrong nexson version, perhaps? (but then just using a new tree shouldn’t fix things)
15:37 josephwb right
15:37 jimallman what constitutes a “valid taxon”? clear mapping of OTU to OTT?
15:37 jimallman mapping of node to otu to ott, i guess
15:37 josephwb and i have a nexson with the old and new trees in it. works with new tree, not with old. same file.
15:37 jimallman ok, clear enough
15:38 jimallman hm, i see that old tree is marked as “preferred” in the curation app, but new is not
15:38 jimallman “preferred” is my clumsy way of saying “use this tree in synthesis”.
15:38 josephwb only for that one. i changed all others. wanted to have an example for you
15:39 jimallman ok.
15:39 josephwb not sure why ot:branchLengthTimeUnit "Myr" is set if branch lengths are arbitrary, but that is something else...
15:40 josephwb *don't trust curators. ever.*
15:41 jimallman that (Myr) might be a dumb default? can’t recall
15:41 josephwb i don't see a default
15:41 josephwb but maybe
15:41 josephwb something dumb, anyway ;-)
15:43 jimallman hrm. the otus in this study look ok. and as you say, they’re working in the new tree but not the old (in same file)
15:44 josephwb i think NexsonReader may be choking on something. it comes out of there with no tree structure
15:44 josephwb NexsonReader extracts the tree from the nexson
15:44 josephwb in case you didn't know
15:45 josephwb we haven't touched that code in forever
15:46 josephwb is there a mix of nexson version stuff going on? maybe a renamed variable? i am stumped why it doesn't go through
15:47 josephwb BAD: Ingested tree number of external nodes 1
15:48 josephwb GOOD: Ingested tree number of external nodes 5
15:48 josephwb so definitely not being read in correctly
15:49 josephwb i got to run for a bit, but I will figure out what NexsonReader doesn't like
15:49 josephwb i *do* have the workaround, but it is a hassle for a large number of studies
15:51 jimallman hrm. ok, i’ll be here. we should definitely come up with something better.
15:54 josephwb i see in NexsonReader that we temporarily turned off warnings involving "nexson messages". if they are not properly terminated, maybe the cause? this was your first guess, right?
15:58 jimallman yes, but i think i was mistaken (the message looked well-formed after i refreshed my diff viewer)
15:59 jimallman https://github.com/OpenTreeOfLife/treemachine/blob/0087bfbdc012aa7982391929fd769f8e6000f72d/src/main/java/jade/tree/NexsonReader.java#L178-L185
15:59 jimallman this is interesting (link above)… it reads as if NexsonReader wants ^ot:inGroupClade to have a ‘$’ property with the value inside
16:00 jimallman but it’s simpler than that.. in both old and new trees.
16:00 jimallman remind me: are the new (re-imported) trees really working in treemachine? or are they coming up “empty”?
16:01 josephwb they are working
16:01 jimallman so “BAD: Ingested tree number of external nodes 1” is the old tree.. ok
16:02 josephwb yes
16:02 josephwb those are treemachine messages
16:02 jimallman that helps me orient myself in the code (near the link above)
16:02 jimallman https://github.com/OpenTreeOfLife/treemachine/blob/0087bfbdc012aa7982391929fd769f8e6000f72d/src/main/java/jade/tree/NexsonReader.java#L312
16:02 josephwb right
16:03 josephwb ok, really have to go now. you don't need to waste time with the treemachine junk.
16:04 jimallman it’s ok, i could use a little more exposure here.. i’ll just backtrack and let you know if anything jumps out.
16:04 jimallman ttyl
16:55 towodo joined #opentreeoflife
16:59 mtholder joined #opentreeoflife
17:55 pmidford2 joined #opentreeoflife
18:24 towodo jimallman, I looked over your conv with joseph
18:25 jimallman does anything catch your eye?
18:25 towodo IIUC, treemachine was able to process a nexson just fine in early September, but it’s now choking on what ought to be practically the same nexson.  yes?
18:25 towodo the nexsons aren’t different because phylesystem changed
18:26 towodo so did peyotl change?  they are dynamically translated from the newer format to the older one
18:26 jimallman hm, good thought. i think it does some kind of internal translation..
18:26 jimallman it would be good to find a study that definitely hasn’t changed since last synthesis, and test that one.
18:27 towodo actually I now see joseph is talking about very new studies.
18:27 towodo ot_201, ot_203
18:28 jimallman yes, signs point to invalid nexson coming from the curation app.. but so far i can’t find anything different about the “old” vs. “new” trees (those joseph has re-imported form newick)
18:28 towodo ot_201 was uploaded on Aug 18
18:28 towodo but not clear whether it was in our early sept synthesis
18:30 jimallman isn’t this archived somewhere? (i would hope so)
18:30 * jimallman recalls a semi-recent conversation on this question.. writing synth properties out to a release/version page
18:31 towodo good point.  we’d have to look at old versions of gcmdr. probably most knowledge about the previous tree resides on disks in the smith lab
18:32 jimallman and just a quick test of feeding the latest (Aug 18) version to treemachine. pass or fail would be interesting in any case
18:33 towodo the latest is the one from today
18:33 jimallman sorry, i mean the Aug 18 version of ot_201
18:34 jimallman just to see if we’re building a different kind of nexson, or treemachine is reading it differently
18:35 towodo i guess we should wait for joseph to return. too many questions
18:36 jimallman agreed
18:50 mtholder joined #opentreeoflife
19:23 josephwb joined #opentreeoflife
20:15 josephwb joined #opentreeoflife
20:41 josephwb joined #opentreeoflife
20:56 josephwb joined #opentreeoflife
21:29 josephwb joined #opentreeoflife
21:30 josephwb jimallman: i haven't been able to find the problem.
21:30 josephwb i'm baffled
21:30 josephwb drewout the trees from the nexson; seem fine. but *something* is different!
21:30 towodo I read the conversation and couldn’t figure out what the problem was exactly
21:30 jimallman hrm. are you certain we’re finding the ingroup, despite not using a proper ‘$’ property?
21:30 josephwb yes, 100%
21:31 jimallman drat.
21:31 josephwb off to a cross country meet; will return to this later.
22:34 josephwb joined #opentreeoflife
23:19 josephwb joined #opentreeoflife
23:56 josephwb joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary