Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-08-13

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:29 josephwb joined #opentreeoflife
01:08 josephwb joined #opentreeoflife
02:43 jimallman For the record, I tracked down the trove4j dependency in treemachine (only) and noticed that the trove .jar (which should have been in ot10:~/.m2/repository/net/sf/trove4j/) was missing. I clobbered the treemachine .jar file there, which forced a recompile, and this time we got the trove .jar properly. Weird.
03:15 josephwb jimallman: is this something we (treemachine people) need to fix?
03:16 jimallman i don’t think so, the pom files look right. i’m guessing the download of the GNU trove pom (or its .jar) fails intermittently.
03:16 jimallman i don’t recall seeing any error messages about this, but it’s possible they snuck by.
03:17 jimallman it’s a sneaky error, and won’t be fixed by more deployments until something changes in the repo, or someone manually clobbers the treemachine .jar file.
03:18 jimallman (towodo has already added this hack to the taxomachine deployment stuff, the old .jar gets clobbered every time.)
03:19 josephwb sneaky errors == the most fun errors
03:19 josephwb good to know
03:20 josephwb i basically assume all errors are my fault at this point, and am elated when it is proven otherwise :-P
03:20 jimallman yep. i assume there’s a subtle error message when a maven dependency fails to load. it would be good to find it and make it a bit more splashy…
03:21 jimallman nah, this system breaks itself :)
03:23 jimallman (i’ve had that gun-shy feeling before. a certain amount of paranoia is healthy, but it’s important not to lose your nerve. keep doing stuff and breaking things! but, you know, not every day.)
03:23 jimallman now i’m tightening up the two MRCA tests, should have them on devtree shortly.
03:26 kcranstn joined #opentreeoflife
03:30 jimallman josephwb: will the MRCA result from taxonomy include a mrca_unique_name if one is available?
03:31 josephwb no, i turned that off, as it didn't return anything from any of my tests
03:31 josephwb should i turn it back on?
03:31 jimallman hm
03:31 josephwb not sure what it is
03:31 josephwb gives mrca_name
03:31 jimallman yes, if you don’t mind. this should provide valuable clarification when the same name appears for different taxa
03:32 josephwb ok, will do
03:32 jimallman (for example, try Drosophila in the synth-tree viewer’s taxon search)
03:32 josephwb ah, i see
03:33 josephwb should always test drosophila
03:33 josephwb * doh *
03:33 jimallman agreed! i think this field is only used when it’s different from the primary name
03:33 jimallman again with the sneaky
03:33 jimallman anyway, this is working well so far..
03:34 jimallman for now, i’m using the all-purpose getMRCA service and the treeSource switch. i assume this is the same underlying code as the source-specific methods..
03:35 josephwb yerp
03:37 jimallman ah, found the code. neat!
03:38 jimallman hm, it does look like the mrca_unique_name will need to be added in both getMRCA and getTaxonomyMRCAForNodes
03:38 jimallman https://github.com/OpenTreeOfLife/treemachine/blob/4fd14da2e88fd903762679dd4697c20ccb269c5e/src/main/java/opentree/plugins/GoLS.java#L265
03:39 jimallman https://github.com/OpenTreeOfLife/treemachine/blob/4fd14da2e88fd903762679dd4697c20ccb269c5e/src/main/java/opentree/plugins/GoLS.java#L173
03:39 josephwb ok. my bad.
03:40 jimallman no biggie, but someday we should refactor these methods to de-dupe the code as much as possible.
03:40 josephwb of course, yes
03:40 jimallman if you’re OK with calling the new field ‘mrca_unique_name’, i’ll go ahead and push the client-side stuff so you can see it.
03:41 josephwb fine with me
03:45 josephwb hold on, it still returns "nearest_taxon_mrca_unique_name"
03:45 josephwb is that not it?
03:45 josephwb i am checking earlier versions...
03:46 jimallman that’s one of the fields for a synth-tree MRCA result
03:46 jimallman i noticed you’re using different field names for taxonomic MRCA
03:46 josephwb yes, sorry...
03:47 josephwb for taxonomy results, "nearest" didn't make sense, so i pitched it
03:47 jimallman no problem there.
03:47 josephwb i.e. taxonomy is an exact hit, not nearest
03:47 josephwb so, change name?
03:47 jimallman but the switch from underscore to camel-case threw me for a minute
03:47 josephwb so: i broke things.
03:48 josephwb * shit *
03:48 josephwb there goes my elation
03:48 jimallman no biggie, but let’s stick to the lowercase and underscore convention used in the synth-tree MRCA
03:48 josephwb what would you like?
03:48 josephwb ok
03:48 jimallman so just drop the ‘nearest_taxon_’ prefix for those names (good call there)
03:49 jimallman mrca_name, mrca_unique_name, mrca_ott_id
03:49 josephwb *changing back *
03:50 josephwb getDraftTreeMRCAForNodes will return the original values:
03:50 josephwb "mrca_node_id
03:50 josephwb nearest_taxon_mrca_name
03:50 josephwb nearest_taxon_mrca_unique_name
03:50 josephwb nearest_taxon_mrca_rank
03:51 josephwb nearest_taxon_mrca_ott_id
03:51 josephwb nearest_taxon_mrca_node_id
03:51 josephwb found_nodes
03:51 josephwb changing getTaxonomyMRCAForNodes to underscore flavour
03:52 josephwb found_nodes
03:52 josephwb mrca_node_id
03:52 josephwb mrca_name
03:52 josephwb mrca_rank
03:52 josephwb mrca_ott_id
03:52 jimallman schweet
03:52 josephwb to be clear: "nearest" doesn';t make sense when dealing with taxonomy, but does when against the synth tree
03:53 josephwb or, should everything just be named the same?!?
03:53 jimallman yes, that’s my understanding (nearest = nearest OT taxon to an arbitrary MRCA node in the synth tree)
03:54 jimallman sorry, by “yes” i mean I like the shorter/simpler names for taxonomy MRCA
03:54 josephwb ok, good
03:55 josephwb so, when doing getMRCA, you will get back different things depending on treeSource
03:55 josephwb yes?
03:55 josephwb or i can change it
03:55 josephwb * i don't know what i am doing *
03:55 jimallman i’m expecting different field names, no problem there (code is already in place)
03:55 josephwb ok
03:56 jimallman different depending on treeSource, as you said
03:56 jimallman https://github.com/OpenTreeOfLife/opentree/blob/07ced0b184a9ff58d831a15781e60e0ee81351de/curator/static/js/study-editor.js#L5430-L5436
03:57 josephwb * updating no *
03:57 josephwb now
03:59 jimallman cool, pushing the latest curation code to devtree
03:59 josephwb just pushed the updated code myself
03:59 jimallman hm, looks like we’ve lost the plugin build again
04:00 jimallman see the failure message here:  http://devtree.opentreeoflife.org/curator/study/edit/pg_2584/?tab=trees&tree=tree5990
04:00 josephwb i know that is not my fault (or, think so)
04:01 josephwb hmm, can't access that link
04:03 josephwb there now, but don't see error
04:03 jimallman yeah, i can’t curl or ping to either api.opentreeoflife.org or devapi.
04:03 jimallman DNS problem? weird
04:04 josephwb if i try either, i get "Primates"
04:04 josephwb so, good?
04:04 jimallman that’s what i’ve been seeing as well. i think it’s probably right.
04:04 josephwb maybe have 2 result fields. not clear which is displayed
04:05 jimallman (it’s definitely calling with different treeSource and getting different field names in each case)
04:05 josephwb yay
04:05 josephwb (?)
04:05 jimallman this was mainly to test the results from both searches. i’ve assumed we would choose just one for the real UI.
04:05 jimallman do you think there’s value in testing both ways?
04:05 * jimallman assumes the DNS outage is a personal problem..
04:06 josephwb ok. i thought it would be 2 results. if they don't agree, curator sees so, and can determine whether 1) mapping problem, or 2) incongruence with synth tree (i.e. taxonomy is bad)
04:07 josephwb i will try to find a study where i know the taxonomy and synth mrcas are different (on devtree)
04:09 josephwb having a single result show is fine if it clear
04:09 jimallman it’s good to know they’re both useful. i think if a curator understands the difference between teh two searches, they’ll have a pretty good grasp of how taxonomy and synth are related.
04:11 josephwb unfortunately, most of the test studies i know are from prod, not dev
04:12 josephwb so it may take a bit to find one where the mrcas differ
04:14 jimallman drat. there’s a lot of overlap, of course, except for the most recent stuff.
04:14 jimallman i’m tweaking the curation UI to include both tests, and separate results for each.
04:14 jimallman but i can
04:14 jimallman but i can’t test while DNS here is borked. i think it’s a sign to take a break.  :)
04:15 josephwb agreed
04:16 josephwb i'll have to get back to this tomorrow
04:16 josephwb i think things work, but my tests were limited
04:17 jimallman kewl, talk to you tomorrow
12:12 josephwb joined #opentreeoflife
13:09 kcranstn joined #opentreeoflife
13:54 josephwb joined #opentreeoflife
14:00 jimallman josephwb: fyi, the new UI (with both MRCA tests) is available now on devtree
14:11 kcranstn joined #opentreeoflife
14:12 josephwb hey jimallman, that's great
14:12 josephwb are things working, ya think?
14:13 josephwb i can't imagine how it might fail
14:13 jimallman yes, so far so good.
14:13 josephwb since the methods only query the treemachine DB, there cannot be any out-of-bounds taxa
14:13 josephwb e.g. new taxa
14:13 josephwb ok, good
14:13 jimallman (and so far i get consistent results from both MRCA tests. it’ll be interesting when they differ)
14:13 josephwb i will put some trees i know that differ in mrca-flavours up on devtree and test them there
14:14 jimallman thanks!
14:14 josephwb i will also have a new synth tree DB later today that we can put up
14:14 josephwb don't know how / don't want to break things
14:15 josephwb it contains all plants + fungi + 99% animals + some important microbe studies
14:15 josephwb still trying to get the microbe studies to play nice
14:15 jimallman i can push that up to devapi (assuming it’s compatible with the OT taxonomy db that’s there already)
14:15 josephwb but enough of an advance over the previous/current tree
14:16 josephwb yes, devapi would be the place
14:16 josephwb will be many hours from now, so i'll ping you then
14:16 jimallman ok
14:18 josephwb man, irc is so much more efficient than emails! 8-)
14:22 josephwb jimallman: i noticed on devtree that it is reporting duplicate DOIs, when they are not duplicates
14:22 josephwb eg. pg_1130, pg_2829
14:22 jimallman yes, i have a ticket open for Cody on this problem.
14:22 josephwb very similar DOIs (just year different)
14:23 josephwb for those 2, anyway
14:23 jimallman i haven’t figured out the pattern yet, but generally similar DOIs (say, before the last decimal place) will show up as dupes
14:23 josephwb weird, seems like a simple check
14:23 josephwb it is oti?
14:24 jimallman yes, it’s in oti. and it’s supposed to be a super-simple test for exact match, so i don’t know what’s up
14:24 jimallman https://github.com/OpenTreeOfLife/oti/issues/19#issuecomment-51729104
14:26 josephwb so weird...
14:28 josephwb ok, so mrca *can* fail. getting our old friend TaxonNotFoundException
14:29 josephwb i *think* it is taxa mapped from the new ott not being in the past (old-ott) tree
14:29 josephwb i should be skipping those. should these be reported back? probably, yes?
14:31 josephwb did i mention it was "bulletproof "earlier? i "nerf-proof" ^o)
14:31 josephwb i meant "nerf-proof"
14:31 jimallman hm, but the ottids should be backward-compatible… not sure i understand the old vs. new here
14:32 josephwb a taxon can be mapped to an ottID in the curator from ott2.8draft5. that ottID may not be in ott2.6, which the tree is built upon.
14:32 josephwb i.e. a "new" taxon
14:32 jimallman tree = synth-tree, you mean
14:32 jimallman ?
14:32 josephwb yes
14:33 josephwb the synthtree DB is built on ott2.6
14:33 josephwb or, did stephen put up something newer?
14:33 josephwb i am assuming it is the tree from the past submission
14:33 jimallman gotcha. weird that these would be out of sync, but i guess it’ll happen sometimes.
14:34 josephwb anyway, the mrca test should be able to handle such things
14:34 jimallman i guess i’d swallow that exception and skip to the next tip
14:34 josephwb yes. i had that, but the problem seemed to go away, so i dropepd it
14:34 josephwb poor testing
14:35 jimallman not a biggie unless the whole “neighborhood” isn’t in the old synth-tree. in that case there might be no (or just one) tip that’s found.
14:35 josephwb exactly
14:35 jimallman i only complain (on the client side) if no tips are mapped. i guess i should be checking for at least two :)
14:36 josephwb the TM needs a minimum of 1
14:36 josephwb i could switch it to 2
14:36 jimallman things will go more smoothly if we try to keep these in sync on production. basically, as soon as we have a new taxonomy, we should use it to build a new synthetic tree.
14:36 jimallman lemme check my code, maybe i was more sensible than i recall..
14:36 josephwb easier said than done, but yes, i agree
14:39 jimallman sure enough, i was just checking for 1+ mapped tips. changed to 2+, pushed to master
14:42 jimallman josephwb: i don’t see mrca_unique_name here (not pushed to GitHub?): https://github.com/OpenTreeOfLife/treemachine/blob/master/src/main/java/opentree/plugins/GoLS.java#L173
14:42 jimallman i think i can add this, based on what we’re doing in the synth-tree test: https://github.com/OpenTreeOfLife/treemachine/blob/master/src/main/java/opentree/plugins/GoLS.java#L123
14:43 josephwb huh, it was there last night. checking now.
14:45 josephwb oh, it is there for one, but not the other
14:45 josephwb adding now
14:46 josephwb pushed
14:58 jimallman cool, thanks
15:09 towodo joined #opentreeoflife
16:07 josephwb jimallman: all MRCA services should now properly handle unmatched Ids (e.g. typos, or "new" ids)
16:07 jimallman hooray! thanks.
18:46 jimallman josephwb: do you know of any taxa that will show up on devtree as “orphaned” in the synth-tree viewer? in other words, taxa that are included in synthesis but not connected to anything else.. we previously go this result for one of the taxa named Drosophila, but now it’s placed in the larger tree:  http://devtree.opentreeoflife.org/opentree/argus/ottol@34907/Drosophila-genus-in-Drosophiliti-
18:52 josephwb hi jimallman
18:53 josephwb that should be *any* non-monophyletic higher taxon
18:53 josephwb * looking for an example *
18:54 josephwb wait, is that tree *only* taxonomy?
18:55 josephwb if so (it looks like it) then there should be no orphans
18:55 josephwb yeah, i don't see any synth relationships, only taxonopmy
19:02 jimallman whoa, did not expect that
19:03 jimallman but that explains the difference wrt Drosophila
19:13 josephwb jimallman: is this possibly a tree stephen put up?
19:14 josephwb might only have fungi + plants
19:14 josephwb * looking there *
19:14 josephwb nope, only taxonomy there too
19:20 josephwb that would explain why the 2 different mrca tests give the same answer
19:26 jimallman once again, we snatch defeat from the jaws of victory
19:26 * jimallman actually believes the code is fine as-is..
19:27 josephwb wee
19:27 josephwb except, we need a synth tree in the db to test against
19:28 josephwb i have the previous one (from submission)
19:29 josephwb i can put it somewhere
19:29 josephwb it is what i have been testing against on my local instance
19:29 josephwb jimallman ^
19:29 jimallman i can push that to devapi, but i wonder if someone’s testing anything with the tax-only db..
19:30 jimallman if so, they’re being sneaky about it
19:30 josephwb who could have put it there? i don't know how
19:30 josephwb must be a commit log
19:31 jimallman yes, i’ll go fish
19:31 josephwb must have been relatively recent
19:32 josephwb prod DB contains a tree
19:32 josephwb i.e. not just taxonomy
19:34 jimallman two entries in ot10:~/log/messages:
19:34 jimallman messages:Wed Jul 23 14:44:38 BST 2014 smitty ot10  Installed treemachine neo4j database
19:34 jimallman messages:Wed Jul 23 18:54:30 BST 2014 smitty ot10  Installed treemachine neo4j database
19:34 jimallman i take it smitty is stephen..?
19:35 josephwb yerp
19:35 jimallman that’s quite a while back (~3 weeks)
19:36 josephwb hmm. but after the move to prod, so i wouldn't have been looking there
19:37 jimallman this was apparently (ot10/devapi) based on this email msg: “where do i put an updated tree of life for romina to check in the browser?”
19:37 josephwb oh
19:37 jimallman so it was intended to have more than just taxonomy.. maybe this was replaced since then, and it’s just not showing up in the logs?
19:37 josephwb stephen was vetting the fungal trees
19:38 jimallman just to avoid confusion: you’re finding a taxonomy-only tree on devapi, not production. correct?
19:38 josephwb yes
19:38 jimallman thanks
19:38 josephwb to be clear: i only saw taxonomy relationships
19:39 josephwb if stephen did a synthesis using only fungal trees, synth relationships would only be in fungi, and taxonomy relationships elsewhere (like birds, where i look)
19:39 josephwb but i am not seeing synth rels in fungi either
19:39 josephwb so, prolly taxonomy only
19:40 jimallman funky. the log says he installed a treemachine db twice on that day, 4 hrs apart
19:40 josephwb regardless, not generally useful
19:40 jimallman yeah, let’s replace it with a recent, sensible db
19:41 josephwb you can copy the one over from prod
19:41 josephwb the one i was making crashed, so it will be hours from now until it completes
19:42 josephwb the one from prod will be the older ott v, but shouldn't be a problem
19:43 jimallman ok, that will make a nice test of your fault-tolerant MRCA code too
19:43 jimallman (testing new taxa against old synth-tree, i mean)
19:46 josephwb right
19:51 * jimallman is doing a lateral pull (scp) of treemachine db from ot18 to ot10, about 15 min to go…
19:52 josephwb *is holding breath for the duration*
20:00 jimallman bad news, josephwb. 10 mins to go..
20:00 josephwb :-$
20:22 jimallman ok, let’s try the new treemachine db
20:23 jimallman hm, Drosophila link from above is still part of a large (tax-only?) tree:  http://devtree.opentreeoflife.org/opentree/argus/ottol@34907/Drosophila-genus-in-Drosophiliti-
20:25 jimallman There are many synthesis sources listed in the Bib References page:  http://devtree.opentreeoflife.org/about/references
20:26 josephwb i don't see any synth relationships
20:27 josephwb i don't understand
20:28 josephwb looks ok on prod, but not on dev
20:28 josephwb prod: http://tree.opentreeoflife.org/opentree/argus/ottol@81461/Aves
20:28 josephwb dev: http://devtree.opentreeoflife.org/opentree/argus/ottol@81461/Aves
20:31 jimallman try this one: http://devtree.opentreeoflife.org/opentree/argus/ottol@33893/Botryosphaeria
20:31 jimallman there are definitely phylo-based edges here. start from the About > Bib References page and try links to focal clades there.
20:32 josephwb ok, found some
20:33 jimallman score! i also found an orphan (oddly, the focal clade of the first listed Bib Ref)
20:33 jimallman http://devtree.opentreeoflife.org/opentree/argus/ottol@737448/Agaricus
20:33 josephwb but what is going on? copied from prod, but clearly not the same
20:34 jimallman hm, i see your point.
20:34 josephwb those are fungi only
20:34 josephwb by checking references
20:34 jimallman maybe someone installed the production db from another location?
20:35 josephwb :/
20:35 jimallman also, note that it’s possible the Bib Refs page is not in sync, if it’s coming from oti (probably needs to be re-indexed)
20:35 josephwb anyway, that was what we expected before (i.e. fungi-only)
21:30 josephwb hey jimallman. i am getting different taxon mapping results on prod vs. dev for the same taxon / context
21:30 jimallman !
21:30 josephwb it looks like dev is doing a better job at finding synonyms?
21:30 jimallman different taxonomy versions, perhaps different behavior regarding synonyms/etc.
21:31 josephwb are they calling different taxomachines?
21:31 jimallman we really do need to be able to fetch flags for any taxon, to see what’s up
21:31 jimallman yes, taxomachine instances on api and devapi, respectively
21:31 josephwb ok
21:32 jimallman same for treemachine, oti, and phylesystem-api (and of course different NexSON docstore/repo in each case)
21:32 josephwb right

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary