Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-09-05

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:28 towodo joined #opentreeoflife
00:41 codiferous joined #opentreeoflife
11:57 codiferous joined #opentreeoflife
12:33 towodo joined #opentreeoflife
13:03 towodo jimallman, I’ve merged all open pull requests for opentree.  how can we now get master and Cavia in sync?  or are they already?  gh says Cavia is 18 commits ahead, 14 behind
13:19 jimallman towodo: i’ll take a look. i believe there were some merge conflicts along the way, which can muddy up the history.
13:20 towodo maybe a merge of Cavia to master is in order
13:20 jimallman i had proposed that we just make a new Danio branch from master, if there are no other big changes on master
13:21 towodo I was going to put continuous deployment into practice, and not create any production branch at all
13:22 towodo that’s why I was looking to make sure the master branches were squeaky clean
13:24 jimallman i see what you mean. in that case, yeah, Cavia to master might make sense. i see that we do have good stuff in Cavia now that we’d want on master (like the latest deployment-fixes):
13:24 jimallman https://github.com/OpenTreeOfLife/opentree/compare/Cavia?expand=1
13:24 towodo ahh… ok
13:25 codiferous joined #opentreeoflife
13:25 jimallman Cavia also has the somewhat-improved OTU mapping UI (allows choosing from a list of fuzzy matches), which is surprisingly not on master. but i’d say that’s an improvement.
13:28 jimallman after reviewing all the diffs in the link above, i think it’s all improvements. merging Cavia to master is a Good Thing.
13:29 jimallman it does look like there may be conflicts (GH says “we can’t automatically merge these branches”), but we can pick through those.
13:29 jimallman i’ve created a PR for clarity: https://github.com/OpenTreeOfLife/opentree/pull/420
13:30 * jimallman is off to grab some coffee, brb
13:43 jimallman to be clear, this may not present conflicts per se, but its overlapping commits will probably just need a manual (two- or three-step) merge on the command line. this is usually no problem.
14:03 towodo right, i’ve done those before
14:08 josephwb imallman: i added a comment to issue#407: By "always on", do you mean it will start attempting to map names before a context is selected?
14:26 jimallman josephwb: see reply there:   https://github.com/OpenTreeOfLife/opentree/issues/407#issuecomment-54630961
14:26 codiferous joined #opentreeoflife
14:35 towodo josephwb, I’m reviewing commits to treemachine master since last production deployment, which was Jul 24 (commit 8c97)
14:35 towodo is treemachine master deployed to development now?
14:35 towodo (I guess I could check the log, but it’s easier to ask you)
14:36 josephwb um, i'm not sure.
14:36 josephwb i think so?
14:36 towodo I’ll check the log
14:38 towodo log says last checkout was 3e84 on Aug 20. that doesn’t include latest two commits
14:38 josephwb one is the turning off of branch lengths that you requested
14:39 josephwb the other is basically testing functions
14:39 towodo those will have to get onto ot10 before I’m comfortable moving them to production
14:39 josephwb basically prototyping new services
14:40 josephwb i am working on a different branch at the moment
14:41 josephwb i don't know if that is relevant to pushing to ot10
14:41 josephwb maybe not
14:41 towodo not. I’m talking about getting the master branch onto ot10, sinc the master branch is what’s going to be production
14:42 josephwb right
14:42 josephwb would you like me to do that?
14:42 towodo sure, that would be great
14:42 josephwb ok, i'll do that in a bit
14:45 josephwb this look correct?:
14:45 josephwb ./push.sh -c ../../deployed-systems/mls/ot10.config treemachine
14:45 josephwb refreshed everything
14:45 josephwb oops, wait
14:45 josephwb that is wrong
14:46 towodo no, not mls
14:46 josephwb yes
14:46 josephwb i mean, yes: you are correct
14:46 josephwb edited wrong thing from .bash_history
14:46 towodo devapi
14:47 towodo = ot10
14:47 josephwb ./push.sh -c ../../deployed-systems/development/ot10.config treemachine
14:48 josephwb i belive that one is correct
14:48 josephwb believe
14:49 josephwb can i get a thumbs-up?\
14:50 josephwb *paranoid about breaking things*
14:50 towodo hang on
14:50 towodo yes, that looks right.
14:50 josephwb thanks. going.
14:51 josephwb and done.
15:08 towodo ok, how do we know it works? … browser works, my API tests pass…
15:24 towodo jimallman, the latest opentree commit to devtree was August 23… it’s hard for me to review the cavia pull request if I can’t try out the changes
15:28 * jimallman is catching up on the conversation...
15:29 jimallman towodo: a few options here: (1) we can push Cavia to devtree, perhaps after merging master to Cavia...
15:29 jimallman (2) i can wrangle the Cavia-to-master pull request and push updated master for testing...
15:30 jimallman (3) i can just push the latest master, though i doubt it has much that’s really new
15:30 towodo let’s follow the new process - I will be the one to merge to master.
15:30 towodo how about treating Cavia as a feature branch.
15:30 towodo first step would be to merge master back into Cavia to make sure the reverse merge will go smoothly (I think you’ve done this)
15:31 towodo then, deploy Cavia to devtree
15:31 towodo then, I can do the review & merge
15:32 jimallman agreed, but note that we haven’t merged master to Cavia recently. should be a Good Thing, but we should take care that it doesn’t revert any desired changes.
15:32 jimallman (i think this is low risk, but not impossible due to a few “overlapping” commits)
15:33 jimallman we can check for this by keeping the current PR (Cavia-to-master) changes open in a browser window, and comparing to a fresh listing after the merge from master-to-Cavia
15:35 towodo what about merging one commit at a time? would that help?
15:39 jimallman i think the manual (command line) procedure suggested on the PR page will do a careful job, and won’t make things worse. a single before-and-after comparison should tell the tale with a minimum of fuss.
15:39 jimallman (and this is probably mostly paranoia on my part)
15:40 towodo ok, that’s fine then, whatever works.
15:41 jimallman so i’ve got the current page for this URL as the “before” view: https://github.com/OpenTreeOfLife/opentree/pull/420/files
15:41 jimallman once you do the manual dance and push back to master, we’ll hit this page again for the “after”
15:43 towodo wait, did you merge master to cavia already?
15:44 jimallman no
15:44 towodo wouldn’t that be the first thing to do?
15:44 jimallman sorry, i misspoke above. should have said: once you merge from master to Cavia, we’ll hit this page again for the “after”
15:45 jimallman towodo: ^ sorry for confusion
15:45 towodo umm… I gues I could to this, but as a matter of roles, you’re the one preparing the Cavia feature branch and I’m the reviewer. I might get confused if I get involved in preparing Cavia for review
15:45 jimallman ah, ok. will do.
15:46 towodo I’ve captured the ‘files’ page/tab in a browser window…
15:47 jimallman same here, merging master to Cavia now...
15:47 towodo after Cavia is ready, and deployed, I’m happy to do the merge of Cavia into master
15:47 jimallman sounds good. i’m slogging through conflicts (no surprise), but this should make your final merge easy.
15:52 * towodo stepping away from IRC for 1/2 hour or so.
15:57 jimallman towodo: updated Cavia (after merge from master) is ready for review. i’m comparing now…
16:10 jimallman towodo: fyi, i’ve reviewed the merge from master-to-Cavia, and everything looks great (no desired changes were lost, and merge back to master should be a snap). i’ll leave it to you to push Cavia to devtree for testing…
16:12 jimallman ah, i see by your suggested guidelines that i should push+test and document. makes sense!
16:13 towodo I suppose the division of labor is up for negotiation.
16:28 jimallman no, this way makes sense. i’m only concerned about the possible lag time between developer’s “smoke test” and the reviewer’s test, since the branches on dev* servers might change in the meantime.
16:28 jimallman as suggested, i’m writing up some suggested tests in the PR comments
16:48 jimallman simple test plan here: https://github.com/OpenTreeOfLife/opentree/pull/420#issuecomment-54650960
16:52 towodo wouldn’t it be better if tests were accumulated in the repo, rather than relegated to comments where they’ll get lost?
16:57 jimallman i hadn’t thought about them being cumulative. sort of a CHANGELOG approach.
16:59 jimallman maybe a TESTS.md file in the root of each repo? with timestamped additions at the top?
17:02 towodo something like that, yes.
17:05 jimallman https://github.com/OpenTreeOfLife/opentree/blob/e56f31dfd10f70c80bd071bf999c72568fa7226e/TESTS.md
17:08 towodo yes, that makes sense to me
17:09 jimallman OK, noted and linked from the PR comment (as an update).
17:09 towodo files.opentreeoflife.org is deployed on ot10 so can be ‘tested’ by having someone navigate to it in the browser (I assume it always gets deployed immediately when a change is made, since it gets copied out from the local copy of the repo and not github, so it’s a post facto test, but that doesn’t mean it shouldn’t be reviewed)
17:33 jimallman joined #opentreeoflife
17:34 jimallman joined #opentreeoflife
17:42 jimallman towodo: updated TESTS.md to reflect your last message. (I didn’t realize we push files.opentreeoflife.org changes to production for testing.)
17:43 jimallman https://github.com/OpenTreeOfLife/opentree/blob/Cavia/TESTS.md
17:43 towodo well, it only gets pushed when you do do push.sh -c … files.  I have been doing that immediately on making any change since it’s a low-risk operation (just static web content) and that’s the only way to test it at present.  but it still should get review at some point
17:44 * jimallman nods
17:45 towodo the synonym processing is very odd.
17:47 towodo e.g. if you ask for fuzzy matching for Sarcophaga nanula, then Helicobia aurescens is shows as ‘synonym for Sarcophaga parvula’
17:48 towodo but it’s the other way around. H. a. is a synonym for S. p., not the way the UI shows it
17:48 towodo (in taxonomy, synonymy is not commutative)
17:48 towodo jimallman ^
17:49 towodo wait a minute, did i get that backward
17:49 towodo sorry, it seems to be right as is…
17:50 towodo but it’s still very weird.  why show the synonym at all? the choice list should show the taxon it’s a synonym for, not the synonym
17:50 jimallman ok, let me (or Cody) know if you see “reverse” synonymy anywhere..
17:50 towodo I don’t think H. a. should show up as a choice at all.  S. p. should be offered as a choice
17:50 jimallman this is sooo not my strong suit. perhaps we’re getting right name in the data, and i’m showing the wrong one.. lemme see..
17:51 towodo you certainly have the right name, it shows up in the mouseover and as the taxon label after selection
17:52 towodo I checked the taxonomy and indeed the final label (after selection) is the correct one.  it’s only when displayed as a choice that you see the synonym
17:52 jimallman hm, i see what you mean. i can change this pretty easily…
17:54 towodo good, i think we need it before releasing
17:55 jimallman i’m a little cautious though. the returned match for “Helicobia aurescens” includes these properties:
17:55 jimallman matched_name: "Sarcophaga parvula"
17:55 jimallman matched_node_id: 1775164
17:55 jimallman ot:ottId: 4371052
17:55 jimallman ot:ottTaxonName: "Helicobia aurescens"
17:56 jimallman i’m concerned that we’re saving mismatched values for ottId and ottTaxonName (unless these resolve to the same ottId, which would make sense to me)
17:56 towodo ok, you’re right, it’s H. a. is not the synonym.
17:56 towodo S. p. is a synonym for H. a., I just looked at the taxonomy
17:56 towodo grep "Helicobia aurescens" tax/2.8/*
17:56 towodo tax/2.8/synonyms.tsv:Opsophyto lahillei |       4371052 |               |       Opsophyto lahillei (synonym for Helicobia aurescens)    |
17:56 towodo tax/2.8/synonyms.tsv:Sarcophaga parvula |       4371052 |               |       Sarcophaga parvula (synonym for Helicobia aurescens)    |
17:56 towodo tax/2.8/taxonomy.tsv:4371052    |       795583  |       Helicobia aurescens     |       species |       gbif:1483533    |               |
17:57 towodo in which case  I revert to my previous complaint, which is that the mouseover is incorrect
17:58 towodo the mouseover says H. a. is a synonym of S. p., but it’s not
17:58 jimallman gotcha. so mouseover might say “matched on synonym Sarcophaga parvula”
17:58 jimallman (this seems to connect the dots in a sensible way)
17:59 towodo yes, something to that effect.
18:00 towodo I say go for it.
18:02 codiferous just fyi, looks like i'm going to need to do another rebuild of the taxomachine db
18:02 codiferous contexts currently aren't being used because of a bug that prevents storing a property used to identify which context to use
18:02 * jimallman is pushing this change to devtree (Cavia)
18:03 jimallman towodo: it’s ready for review (thanks for catching this!)
18:05 towodo much better
18:05 towodo codiferous, I’ve already copied the taxomachine db to the staging machine and started the build… this doesn’t sound critical
18:11 codiferous not having contexts?
18:13 towodo compared to the curation interface problem, it’s not critical.  but maybe you can talk me into it, how long will it take to fix & test?
18:14 towodo I was suggesting I can deploy what I have, then we can update next week. alternatively, put current deployment on hold until new taxo db is ready
18:14 codiferous well the fix is nearly done. testing should be straightforward, either it works or it doesn't. i'm testing against asterales right now
18:15 codiferous next week is fine
18:15 codiferous well, fine with me
18:15 towodo building the full db takes a while, yes?
18:15 codiferous it will make otu mapping more painful
18:15 codiferous it takes a couple of hours
18:17 towodo but otu mapping is already painful in that way and no one has complained…
18:19 codiferous no one is using it
18:19 codiferous except joseph, who has complained
18:20 codiferous i don't think delaying until next week will be a problem
18:21 codiferous more time for testing anyway
18:21 josephwb <complain>
18:22 josephwb i haven't been able to upload a tree for weeks
18:23 towodo why?
18:23 josephwb well, mapping, anyway
18:23 josephwb first, the fuzzy matching problem
18:23 josephwb then, the failure to map when using a context
18:23 towodo I was only aware of the first, and that’s what I was driving to fix today (or tomorrow)
18:24 josephwb the context bug has been in the curator for a whi;e
18:24 towodo so do you want the first tomorrow and the second at some unspecified later date, or do you want both mid next week?
18:24 josephwb the 2nd being the context search?
18:24 towodo yes
18:25 josephwb i guess together
18:25 codiferous to my knowledge, all the bugs that actually prevent mapping should be in the latest commit to master
18:25 josephwb i don't trust mapping large trees without a context
18:26 codiferous this current bug just prevents the use of any context other than "all life"
18:26 josephwb paranoid about exact matches to wrong parts of the taxonomy
18:26 towodo you mean these are new bugs, that weren’t there before?
18:26 towodo i.e. in the current production system?
18:26 josephwb these have been there for a few weeks
18:27 josephwb the failure of using specific contexts
18:27 josephwb cody fixed it
18:27 josephwb it just hasn't been deployed
18:27 codiferous the one i'm talking about right now has likely been there longer than that
18:27 codiferous the other ones (which actually prevented mapping) have been fixed in the latest commit to master
18:27 towodo I don’t understand “all the bugs… should be in the latest commit to master” - that means they haven’t been fixed in master yet, right?
18:28 josephwb i think they are fixed there
18:28 josephwb fixed before the PR flow
18:28 josephwb right?
18:29 josephwb maybe we are talking about different bugs
18:30 towodo latest taxomachine commit to master was Aug 14
18:30 codiferous all of them except this no-contexts-but-all-life bug have been *fixed* in the latest commit to master
18:31 josephwb but the context thing was fixed too, no?
18:32 josephwb it was fixed before i left for NESCent
18:32 towodo oops, not aug 14, aug 22. latest commit to master = 2bffd3 on Aug 22 “Adding consistency updates back to master “
18:32 codiferous this is a different context bug... not sure when it was introduced
18:32 josephwb ok
18:33 jimallman fwiw, now that the UI allows choosing between multiple fuzzy matches, it would be annoying (but not a show-stopper) if taxomachine still returns multiple identical mappings. This was the underlying problem in one bug, where contexts other than ‘All life’ would fail to map.
18:33 towodo I like the method name ‘TNRSHit’ …
18:33 towodo josephwb, did you review the OTU mapping interface on dev?
18:34 josephwb how new is this?
18:34 towodo good question. the past week or two? let me check the list of closed pull requests
18:34 josephwb last i checked, i believe it worked as desired
18:35 josephwb let me check my go-to study
18:36 josephwb ok, so it seems to work, including using a context
18:36 towodo but cody said that using a context was broken ?!
18:36 codiferous towodo, yeah, the TNRS classes are pretty heavily overwrought. early exercise in java language feature exploration. been meaning to simplify it, but it works so i haven't bothered yet
18:36 codiferous it could be that the bug was introduced during the changes to fix other bugs
18:37 josephwb context: yes, unless it is only using "life" (i.e. ignoring what I specify)
18:37 towodo wondering what other profanity is lurking in your method names
18:37 josephwb i will try a bogus context
18:38 josephwb nope, looks good here
18:38 * jimallman reaches for his filthy-dictionary scanner…
18:38 codiferous :p
18:38 josephwb i.e. did not find the names when i specified the wrong context
18:38 jimallman interesting! josephwb, are you testing this on devtree, or production?
18:38 codiferous hm. the bug may only manifest when inferring a context
18:39 towodo well, codiferous & josephwb, please advise.  I have a new server with software installed and  treemachine and taxomachine dbs. I can continue on with oti setup, or I can pause and wait for a new taxomachine db.
18:39 towodo are you sitting in the same suite of offices? you can confer if you like…
18:41 codiferous we say don't wait
18:41 towodo ok. proceeding.
18:42 codiferous jimallman, the curator should probably make use of the context inference service rather than defaulting to all life
18:43 jimallman and how would i do that? pass an empty context by default?
18:44 towodo jimallman, can we pick a time to do the final prod. install? I think I can do all needed steps, but want you on call just in case.
18:44 jimallman keep in mind that we currently pass each OTU label in isolation.. will inference work that way, or would it be better to pass several / all labels at once?
18:44 jimallman towodo: sure, i can probably do this at your convenience. oh, i may have a thing on Sat afternoon, let me check in with someone.
18:45 codiferous you would need to pass all the labels to infer the context for the study
18:45 towodo I’m thinking Saturday morning maybe, or maybe after 5 today
18:46 towodo you could pass a random selection to infer the context  (thinking about latency if the study is huge)
18:46 codiferous the infer context method is pretty fast
18:46 jimallman codiferous: thought so.. hm, in that case we’d need to switch to true “bulk mapping” by passing all the marked OTU labels at once. how long would it take to get a response for ~20 labels, if fuzzy matching were active?
18:46 towodo is it robust to undiscovered homonyms?
18:46 codiferous infer context doesn't use fuzzy matching
18:47 codiferous it is a preliminary step before attempting to map names
18:47 josephwb i think we should *make* the curator choose the context (veen if it is "life:)
18:47 jimallman oh, i guess you’re saying we do a quick call to infer context, set the context drop-down accordingly, then do normal mapping one-by-one
18:47 codiferous yes
18:47 codiferous it will only use non-ambiguous (i.e. not homonym or synonym) names to identify a context
18:47 jimallman interesting! need to square this with josephwb’s idea of a (forced) explicit choice.
18:48 codiferous the inferred context should be robust
18:48 towodo but it could fail if Foo bar is a plant in the study unknown to OTT, and OTT does know about an animal named Foo bar
18:48 jimallman (this is all a bit of a tangent, sorry towodo) maybe we infer context when the OTU Mapping tab loads, preset the context widget with this value, and then the curator can override if they so choose
18:48 josephwb what if the tree crosses contexts, but exact matches are only within one?
18:49 josephwb ok
18:49 josephwb as long as the context is reported to the curator
18:49 codiferous it's possible, but unlikely. in that case, the curator should be aware
18:49 towodo curator can always override… so it doesn’t matter if it’s 100% correct
18:49 josephwb right, so the curator needs to know which context is being used
18:50 codiferous yes, it should be obvious. no sense is being sneaky about context inference. i think jim's idea of setting the widget to the inferred context and letting the curator override if they choose
18:50 josephwb i still like part of the curator workflow to include explicitly choosing a context
18:50 josephwb but if this works without that step, fine
18:51 codiferous could still ask them to select a context if the inference returns all life
18:51 codiferous or if there are a large number of ambiguous names (these are returned by the inference service)
18:52 josephwb ok, maybe i am biased from working on the earlier system, where my first step was *always* to choose the context before doing anything esle
18:52 josephwb if this is robust and fast, it should be easier
18:53 josephwb i guess i just feel we could save the curator from doing some work
18:54 josephwb curator, being the app (not the person)
18:54 josephwb e.g. if i have a bird tree, get right to it
18:55 jimallman it sounds like a quick AJAX call, which we can do on the initial load of the curation app (or even ahead of time, like we do for git history and other metadata)
18:55 josephwb *i realize my use of "curator" is not consistent*
18:55 jimallman it’s almost like it’s a terrible name for an app (my fault)
18:55 josephwb so, for a tree of 3000 tips this is still fast?
18:56 codiferous what do curators use?
18:56 jimallman thumbs, probably
18:56 jimallman a discerning palate
18:57 josephwb anyway, i am not really arguing for or against forcing the curator (person) to explicitly choose a context in the curator (app)
18:59 jimallman i think we’re all in favor of automatic (pretty good) context inference, bolstered by manual (awesome) context if the user disagrees.
18:59 jimallman s/bolstered/overridden
19:07 josephwb yeah, i'm on board, just relaying my past workflow
19:08 codiferous for 5000 names it takes about 1 second on my local machine
19:08 codiferous curl wont accept argument lists much longer than that
19:09 codiferous it will be slower over the interwebs of course, and i'm testing against asterales so its a small db, likely a bit slower on larger ones
19:09 codiferous but worth a try
19:09 jimallman i would expect gathering all originalLabel values from the tree will also take < 1 sec on the client, even for very large trees. would it be better to send the first ~1000 names found, or a scattered subset from throughout the tree?
19:09 jimallman (and i suppose we’d rather send pre-mapped names than originalLabel, if both are available)
19:10 codiferous 3.5 secs against devapi
19:11 codiferous it would be better to send more names
19:11 codiferous up to speed bottlenecks. jonathan's concerns about missing important exemplars become more likely to occur if names are pruned
19:12 codiferous e.g. a 5000 tip study of group X rooted with 5 exemplars of group Y
19:12 jimallman whup, you lost me there.
19:13 jimallman group X? exemplars? are these the explicitly chosen (by the curation user) examplars in nexson?
19:13 jimallman he left! does that mean i win?
19:14 josephwb yes
19:14 codiferous joined #opentreeoflife
19:15 * jimallman sings the Brave Sir Robin song (drat, he’s back)
19:16 jimallman codiferous: i was just trying to parse your msg about group X, and exemplars from group Y…
19:16 codiferous just names
19:16 jimallman are these the explicitly chosen (by the curation user) examplars in nexson?
19:17 codiferous no, they would be chosen by the original producer of the tree
19:17 codiferous say group X is land plants, and group Y is green algae
19:17 * jimallman nods...
19:18 codiferous a 5000 tip of land plants, with 5 green algae as outgroups. you have to include at least one alga in the context inference of you'll get "land plants" as your context, and none of the alga will produce matches during otu mapping
19:18 josephwb i thought i said that above
19:18 jimallman ah, i see the potential problem anyway. how important is it that we map the outgroup OTUs?
19:19 josephwb that depends
19:19 josephwb there are outgroups, and then there are OUTgroups
19:19 josephwb i think we want everything mapped, right?
19:20 codiferous could also be a problem with paraphyletic grades consisting of small groups leading up to a large one
19:20 codiferous yes, i don't see why we would not want to map something if we can
19:20 codiferous in any event, use all names if at all possible
19:21 jimallman as a practical matter, i can certainly make sure to include some outgroup names when checking for inferred context. it still seems like we’ll need to bundle up a subset if we have a limit of 5000 names
19:22 josephwb i guess this is the sort of instance that a user-specified context would be helpful, if this is going to be a burden/bottleneck
19:22 codiferous 5000 is just the (approximate) limit for curl based on the length of the argument string
19:22 towodo jimallman, I’m ready for curator app downtime, whenever it’s convenient for you.  I would need to send email to opentreeoflife ggroup saying when downtime will be
19:22 jimallman codiferous: right, so we should just pick a safe number and test it
19:22 towodo step 19 at https://docs.google.com/document/d/1JU3kXu7zQHG0ZriwkUIbZuR9QhTitykRKeV8L1491IU/edit
19:23 codiferous we may just want to turn off the auto-inference procedure for very large trees
19:23 codiferous because, for instance, what if the outgroup is wrong because the tree hasn't been rooted properly yet?
19:24 jimallman towodo, it’s easy to set this up. i’ll just need some text (explanation) to display. for example, here’s the most recent message: "Study creation and editing are disabled while we upgrade to the latest code and features. Please pardon the inconvenience. We expect to be back online for editing studies later this evening (Thursday, July 10)."
19:25 towodo that’s fine except for the date
19:25 jimallman codiferous: interesting question, but there are lots of ways a tree can be improved (incl. ongoing mapping decisions). maybe the inferrred-mapping can be something explicitly triggered by the user (like our MRCA test)?
19:25 towodo do you want to do it today at 5?  get it over with?
19:25 jimallman towodo: yes, that sounds great (assuming the db we want will be ready)
19:26 towodo no, we’re not waiting for a db.
19:26 codiferous jimallman, that might be useful
19:26 jimallman ok, so just change to today’s date…
19:26 towodo the only thing left to do is capture phylesystem-1 and index it.
19:27 codiferous in any event, it should not be a concern for trees with few enough tips that all the names can be sent to the inference procedure
19:27 codiferous seems like 5000 is a minimum number for that, it may well be much larger
19:30 jimallman ok. just give the word. it’ll take just a few seconds (and apache restart) to suspend curation
19:30 towodo jimallman, amazingly, github tells me the cavia branch can be automatically merged.  doing so now.
19:30 towodo no, not now
19:30 towodo at 5pm
19:30 jimallman right, not yet!
19:31 towodo need to give the appearance of orderliness
19:31 jimallman yes, the merge from master-to-Cavia flushed out all the conflicts, so it should be smooth sailing back to master
19:31 jimallman towodo: ok to suspend curation now?
19:31 josephwb jimallman: when is the curator (app) due to go down?
19:31 towodo 5pm
19:31 josephwb oh, there it is
19:32 towodo suspend curation at 5pm please
19:32 josephwb great
19:32 jimallman got it. ok to say “later this evening” for predicted resumption of services?
19:32 josephwb i think someone recently remapped a taxon which is breaking synthesis
19:32 codiferous trivial question: do we care about reporting the name of the parent taxon for tnrs results?
19:33 towodo I’ve installed the old synthetic tree, so current synthesis efforts not relevant for today’s update
19:33 josephwb right, but i am working on the paper
19:34 josephwb so, yeah, separate things
19:34 josephwb i.e. i am not trying to get you a tree today to put anywhere
19:35 josephwb codiferous: i don't know the answer to your question
19:36 josephwb might not be useful / couldn't hurt
19:36 josephwb maybe send out to list (including hackathoners)
19:36 josephwb arlin et al. might have a note or two
19:37 codiferous ok, thanks
19:37 codiferous does anybody immediately know an example of a homonym with a unique name in ott?
19:37 josephwb Drosophila
19:37 josephwb right?
19:37 josephwb aren't there a few of those?
19:38 codiferous yep
19:38 codiferous thanks
19:44 jimallman codiferous: josephwb: i’ve added an “enhancement” issue for the inferred-context test: https://github.com/OpenTreeOfLife/opentree/issues/421
19:44 jimallman feel free to add comments and flesh this out
19:44 jimallman (and note my concern that OTUs may come from multiple trees, raising additional questions…)
19:45 josephwb i don't think that is a concern
19:45 josephwb only the names are sent, not the tree structure, right?
19:45 josephwb names will be the same in either case
19:45 josephwb BUT inferred MRCA can be wrong
19:45 josephwb (i think)
19:45 josephwb (probably)
19:46 jimallman it just muddies the determination of ingroup, outgroup, any other settings that might differ per tree
19:47 josephwb right
19:47 josephwb that could be real, or dirty data
19:47 jimallman so i’ll need to come up with a quick, pretty-good sampling strategy for the weirdest case (several trees, some-but-not-all marked for synthesis, with overlapping OTUs)… unless i’m mistaken, and we maintain separate OTUs for each tree.
19:48 josephwb it seems like inferred MRCAs are tree-specific, right
19:48 jimallman yes, absolutely
19:48 josephwb i thought os
19:49 jimallman (current implementation, at least)
19:49 josephwb good, it should be
19:50 jimallman i believe (will need to confirm) that we include only OTUs from *preferred* trees (marked for synthesis) in OTU mapping. in that case, i suppose only those trees “matter” in terms of ingroup/outgroup testing. i’m tempted to use a more brute-force strategy, like “sample every nth OTU label from this list, where n gives us a subset under 5000 labels”
19:51 jimallman it seems to me this would capture ingroup and outgroup labels nicely, or at least make it very unlikely we’d miss a big chunk of the tree. but it might miss an important outlier (curator-person would have to catch this and modify context)
19:52 codiferous outgroups are likely to always be very small chunks
19:52 jimallman even then, this would probably lead to fast and accurate mapping of most names, and they have tools to clean up the outlier(s)
19:52 codiferous easily missed with subsampling
19:52 jimallman hmmm
19:52 codiferous i would recommend not subsampling
19:53 codiferous just don't do auto-inference if the tree is too big, make the user select a context
19:53 codiferous much lower bar with likely better results
19:53 codiferous just my suggestion
19:53 jimallman ah, i see. but that’s really a count of the total OTU collection (from all preferred trees), not of a single tree.
19:54 jimallman OK by me, I can disable the Infer Context button if we have lots
19:54 codiferous will context inference be automatic?
19:54 codiferous that would be nice
19:54 jimallman if the big idea makes sense, we can play with these ideas on a feature branch.
19:54 codiferous yep
19:55 josephwb right, that would be slick. just make sure the curator is made clearly aware what has happened
19:56 josephwb "Oh, I see you are uploading a tree of Angiosperms! Cool!"
19:56 jimallman automatic would be a “best guess” based on the initial set of OTUs (when we load the study into curation app): some mapped, some not. and the trees already selected for synthesis. probably worth it, possibly even the right answer! =)
19:56 codiferous likely the right answer in a lot of cases. treemachine was using this feature for a year or so with good results
19:56 jimallman then manual, with different results based on recent user actions (and possibly disabled, if selecting additional trees puts us over the 5000 limit)
19:57 codiferous i would try 10000 for dev, easily reduced if its too big
19:58 jimallman ok.. if this is a hard limit (based on HTTP or some other protocol), i can probably also measure the size of the payload before sending it.
19:58 codiferous it should scale linearly, so likely 5-6 secs for 10000 with a fast connection
19:58 codiferous don't know what neo4j's data payload limit is, but i think it's very large
19:59 jimallman hey, we post complete updated nexson! so i don’t imagine the list of labels (obv a subset of nexson) would choke this.
19:59 jimallman ah, so it’s a (possible) neo4j limit
20:00 codiferous possible, but i don't think we'll hit that with just taxon names
20:01 jimallman on further reading: HTTP doesn’t impose a limit, but server-side receivers often do (php, asp.net, etc.). so this might also be a “tunable” setting in neo4j
20:41 josephwb hey jimallman: so the curator (app) will be down at 17:00. will be up again later today?
20:43 jimallman later this evening, that’s my understanding.
20:43 josephwb ok, great
20:44 jimallman i was hoping towodo would confirm that, maybe he’s out there…
20:45 towodo judge for yourself. at 17:00 I’m going to start with step 19 at https://docs.google.com/document/d/1JU3kXu7zQHG0ZriwkUIbZuR9QhTitykRKeV8L1491IU/edit
20:45 josephwb and the context angle will be working then?
20:45 towodo the time consuming part is oti indexing, but that’s less than an hour IIRC
20:46 towodo so could be back up at 6pm, pretty sure by 7pm, but I’m paranoid and always expect something to go wrong, so who knows
20:46 josephwb ok, i will keep that in mind
20:46 josephwb thanks
20:47 jimallman right, the the message “later this evening” sounds good. i’ll be around to help if things get weird or you just want more hands.
20:59 jimallman towodo, just FYI - curation is suspended on production
20:59 towodo ok thanks
21:06 josephwb is it?
21:06 josephwb doesn't seem like it
21:07 josephwb i had a study open
21:07 josephwb i won't add any changes
21:07 josephwb left #opentreeoflife
21:07 towodo nothing has changed yet, except that any changes made to the phylesystem won’t be reflected in the new version of the system
21:09 towodo the reason curation is ‘shut down’ is because any changes made could be lost
21:19 jimallman fyi - this feature blocks new requests to the create- and edit-study URLs. it won’t shut down an open curation session, and it shouldn’t stop them from saving changes. it’s mostly intended to stop new sessions before they start.
21:54 towodo oti indexing done (39 minutes 8 seconds)
21:58 towodo browser works (ot14)
21:59 jimallman nice!
22:00 towodo curator doesn’t work… maybe an auth problem?
22:00 towodo the setup is temporary, i haven’t touched dns yet
22:01 towodo but still, i would have expected http://ot14.opentreeoflife.org/curator/study/view/pg_2788/?tab=history  to work
22:01 towodo oh… maybe it does work… but the main page for that study is empty too
22:02 towodo no, can’t view any study metadata
22:02 towodo shall I just forge ahead with the DNS change, or do you want to dive into this a bit?
22:02 jimallman i’ll chase this, just a minute
22:04 jimallman towodo: it looks like the study fetch is failing when it tries to include duplicate study IDs. is oti up and running? if not, that would explain it.
22:04 * jimallman is trying oti now…
22:05 towodo oti indexing just finished. that requires that oti be up and running. also Mark’s curl tests passed
22:05 jimallman hrm..
22:06 towodo when it says ‘loading study’ is that going to phylesystem-api?
22:06 towodo because curl http://ot20.opentreeoflife.org/phylesystem/v1/study/pg_2893  works fine
22:09 jimallman yes, it’s the expanded response from phylesystem-api (includes commit history, duplicate study IDs, etc)
22:10 jimallman i’m getting an error from the oti api.opentreeoflife.org : “Request to do a query where neither exact nor fulltext indexes are to be searched. This is illegal--at least one type of indexed must be searched."
22:10 jimallman this suggests it’s out of date, before we got a fix for this bug.
22:10 towodo but it shouldn’t be talking to api.  it should be configured to talk to ot20
22:10 towodo is it possible there’s a hardwired domain name somewhere?
22:11 towodo or a place that wouldn’t get overwritten during a push.sh ?
22:11 towodo grepping
22:12 jimallman here’s a CURL call that mimics what phylesystem-api is doing here: curl http://api.opentreeoflife.org/oti/v1/singlePropertySearchForStudies -X POST -H "Content-type: application/json" --data '{"property":"ot:studyId", "value":"pg_2893", "exact":false }'
22:12 towodo grep -r "api.opentreeoflife.org" curator   turns up nothing
22:12 jimallman the domain name is probably in a config somewhere… just a sec
22:13 towodo if it were in a config then the grep would have found it, yes?
22:13 jimallman what server-config file(s) are you using for this? in deployed-systems, i mean
22:13 towodo I’ll push them out now.
22:17 towodo ok, on github now (a merge was required, I’m going to double check contents)
22:17 jimallman my mistake on the oti hostname, i made an unwarranted assumption...
22:18 jimallman checking phylesystem-api/private/config on ot20 now...
22:18 jimallman oti_base_url = http://ot20.opentreeoflife.org/oti
22:19 jimallman hm, should probably be oti/v1… (checking against devapi)
22:19 jimallman confirmed. must be a glitch in the deployment scripts.
22:19 towodo ah, hadn’t expected that.  shouldn’t oti_base_url be initialized via OPENTREE_API_BASE_URL in the api config?
22:22 jimallman yes, and the default used in push.sh is outdated.
22:23 jimallman it’s safer to include all *_BASE_URL values in all server-config scripts
22:24 towodo grumble. https://en.wikipedia.org/wiki/Principle_of_least_privilege
22:26 towodo can’t we just change the one line [ "x$OTI_BASE_URL" != x ] || OTI_BASE_URL=http://$OPENTREE_NEO4J_HOST/oti
22:26 towodo and be good?
22:26 jimallman if you mean that this is no concern of the phylesystem API, we should talk about that. the ability to retrieve metadata and commit history is largely a performance issue, so we don’t need to do a cascade of AJAX fetches in the curation app, but i suppose the alternative is to do one big fetch with all this stuff directly from the curation webapp.
22:27 towodo no, that’s not what I meant, I was responding to “it’s safer to include all *_BASE_URL values in all server-config scripts”
22:27 jimallman we could set the default OTI_BASE_URL to add /v1 at the end, but it’s a fluke if $OPENTREE_NEO4J_HOST is set properly. (i made a ticket about this yesterday)
22:28 towodo why not deal with OTI_BASE_URL the same as all the other base_urls?
22:29 towodo oh… hmm… trying to figure out what you mean about /v1 …
22:29 jimallman that’s what i’m suggesting by “include all *_BASE_URL values”. i think these default values are dangerous, since they assume a sensible value for OPENTREE_NEO4J_HOST (which is deprecated in our recent config files)
22:29 towodo agreed. i thought you were talking about something else
22:30 towodo but let’s remove the defaulting later (make an issue perhaps), i just want this sucker to work
22:30 jimallman here’s the relevant ticket, by the way: https://github.com/OpenTreeOfLife/opentree/issues/417
22:31 jimallman agreed. for now, just add /v1 to the default value, and i think we’re golden (i can do this now)
22:31 towodo fixing api.config
22:33 jimallman ok, that’ll do it
22:33 codiferous joined #opentreeoflife
22:34 jimallman this should work in api.config (just to make sure we’re on the same page):
22:34 jimallman OTI_BASE_URL=//${OPENTREE_TAG}.opentreeoflife.org/oti/v1
22:36 towodo yep, that’s sort of what I did.
22:37 towodo OPENTREE_API_BASE_URL  is supposed to have http: on the front? That’s inconsistent with the others… but that’s the way it is, will leave it alone
22:38 jimallman good catch! these should be scheme-relative URLs (but probably no harm done in this case)
22:39 towodo funny, redeployed ‘api’ component to ot20 and no change in curator behavior. maybe I screwed up
22:39 jimallman hrm
22:40 jimallman ot20’s phylesystem-api/private/config looks right to me… i’ll the curl call (copied from curation app)
22:42 jimallman weird. i think the missing “http:” protocol is the problem here!
22:43 jimallman here’s the study fetch as CURL: curl 'http://ot20.opentreeoflife.org/phylesystem/v1/study/pg_2788?output_nexml2json=1.0.0&amp;auth_token=ANONYMOUS'
22:43 jimallman the error “unknown url type” is because it’s not getting the protocol. i should be prepending this in web2py…
22:44 towodo oops. let me look
22:44 jimallman (we do this in several other places). i can do a quick fix, or we can prepend the http: scheme to this in a config file
22:47 towodo I just copied from tree.config … oh I see… I deleted the http:// and I need to put it back
22:47 jimallman yes, that should fix things for now.
22:47 jimallman here’s the standard fix for server-side AJAX:  https://github.com/OpenTreeOfLife/opentree/blob/2d2468a124942f4ba4eadcf2b5b3d42055a130d2/webapp/controllers/about.py#L57-L60
22:48 jimallman this will prepend these URLs with http:  if no scheme is found, or leave it alone (so no harm done)
22:50 jimallman i’ll need to add this snippet to the master branch of phylesystem-api (I’ll make a tiny PR for this)
22:50 towodo i don’t like DWIMmish things like that
22:51 jimallman DWIM=do what i mean?
22:51 towodo the more magic is used, the hard the code is to understand
22:51 towodo yes
22:51 towodo s/hard/harder/
22:52 towodo ahh… much better now
22:52 towodo can go on to next step
22:53 jimallman i’ve documented the scheme-relative expectation in the sample config file, fwiw. i don’t know how else to handle this, since the API base URLs are used for both client- and server-side AJAX calls. (the former will use the current scheme in the browser, the latter needs to have a scheme prepended)
22:54 towodo I see… server side calls are always http: ? because no session state available at those points?
22:55 jimallman yes, and (iirc) we’re never passing auth information… i should be 100% sure, and i’m not.
22:56 towodo ok, redirecting api. to ot20. , ready in 5 minutes
23:03 travis-ci joined #opentreeoflife
23:03 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#591 (fix-incomplete-oti-base-url - a834595 : Jim Allman): The build passed.
23:03 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/commit/a834595c0f87
23:03 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/34544367
23:03 travis-ci left #opentreeoflife
23:03 towodo Fail:  http://tree.opentreeoflife.org/curator/study/view/pg_2893
23:04 towodo Fail: http://tree.opentreeoflife.org/curator/study/view/ot_122  “loading study data…”
23:05 * jimallman is checking this now...
23:05 * towodo would love to be able to look over shoulder…
23:05 jimallman failed once for me, but now it’s working (on refresh)
23:05 jimallman both URLs are working, i should say
23:06 towodo ot_122 still failing for me.  i guess it could be a timeout issue. will look at ot20 activity
23:06 jimallman towodo: sadly, i didn’t have a debug bar open on the failed request so i’m not sure what failed there.
23:07 towodo no activity.  the ot_122 problem is reproducible for me
23:07 jimallman here’s a CURL version of the critical call, please try this from command line: curl 'http://api.opentreeoflife.org/phylesystem/v1/study/ot_122?output_nexml2json=1.0.0&amp;auth_token=ANONYMOUS'
23:08 towodo ok, but I’m logged in
23:08 jimallman hm, i’m not… let me try that.
23:09 jimallman still works for me. what browser are you using?
23:09 towodo ff
23:09 jimallman do you have debug tools, with a Net or Network tab?
23:10 jimallman i’m in FF now, retracing your steps
23:10 jimallman logged in and looking at ot_122
23:10 towodo recreated problem, network tab visible… what am I looking for?
23:10 jimallman assuming you don’t have Firebug (3rd-party dev tools), try this in “vanilla” FF
23:11 jimallman ah, ok. if Network tab is empty, reload the page and check again.
23:12 towodo no, it’s not empty, it shows two GETs
23:12 towodo html and json
23:12 jimallman once you have Network traffic, we’re looking for this request GET ot_122?output….
23:12 towodo got it… try it from curl?
23:12 jimallman (probably be the last request in the tab) for now, just click the URL and you should see right-hand details panel
23:13 jimallman check the Response tab
23:13 towodo yes… I see a nexson thing
23:13 jimallman mine shows a sort of property listing from the JSON response, starting with commentHTML: “<p></p>”
23:14 towodo yes
23:14 jimallman hm, if you can see that, the curation app must be choking on the NexSON…
23:14 towodo i’ll try a different browser
23:14 jimallman let’s check the Console (main tab of dev tools)
23:15 towodo TypeError: viewModel.duplicateStudyIDs is not a function
23:15 towodo no problem in Safari
23:15 jimallman interesting! maybe some kind of race condition? or outdated (cached) JS in firefox…
23:15 towodo oh yeah, need to log in
23:16 jimallman try this in the FF console:  type “viewModel.duplicateStudyIDs”
23:16 towodo still no problem in safari
23:16 towodo trying chrome
23:17 towodo no problem there.
23:17 jimallman right. and i don’t think login is relevant. i suspect it’s either a race condition that only happens in FF (though it’s working for me), or ...
23:17 jimallman possibly stale/cached JS in your FF browser only.
23:18 towodo hmm… will try clearing cache, that makes perfect sense
23:18 towodo that was it.
23:18 towodo should have thought of that
23:19 jimallman great! Safari is the notorious browser for super-sticky caching.. maybe they’ve improved that.
23:19 towodo do you have a cache clearing reflex ? why didn’t this happen to you?
23:20 jimallman it Just Worked for me, but i’m normally in Chrome (maybe less sticky?)  i hate even having to suggest clearing cache to users… maybe an enhancement for later (append timestamps to JS urls, or somesuch)
23:20 towodo I’m going to declare victory now… if there are problems we’ll hear about them
23:21 jimallman agreed.
23:21 jimallman if it becomes a repeat problem, i’ll add an enhancement ticket…
23:21 towodo thanks for your help.  i’ll do PRs for my changes now
23:22 jimallman ok, and i’ve created one for the URL fix in phylesystem-api: https://github.com/OpenTreeOfLife/phylesystem-api/pull/105
23:22 jimallman shall i update the default API base URLs in push.sh, to add /v1 in each case?
23:24 towodo I suppose so. not urgent. it might be better if there were no defaults, and presence of a definition was checked at the points where it was needed.
23:25 towodo because there is no more OPENTREE_NEO4J_HOST (for example)
23:25 jimallman agreed. the old system sort of assumed that all APIs are likely on a single server (and that $OPENTREE_NEO4J_HOST was defined), both of which are questionable assumptions
23:26 jimallman I see, so an individual component could check just before using one of these variables, and complain if it’s not defined. nice.
23:26 jimallman i’ll add this to the related ticket: https://github.com/OpenTreeOfLife/opentree/issues/417
23:26 towodo is there a way to initiate a pull request from the command line?…
23:27 codiferous joined #opentreeoflife
23:28 towodo I was very tempted to commit to master, but resisted, so you are now assigned a pull request for opentree repo
23:30 jimallman re: PR from the command line, that’s a great question! i think of pull requests as a GitHub-ism, but they’re actually a feature in git… i think email was the vehicle of choice, vs. the web UI. now i’m curious...
23:31 towodo I also assigned you a PR for deployed-systems.
23:31 towodo Do you get email when you get assigned to an issue?
23:32 jimallman ah yes, hadn’t been checking. got an email for the PR, and another for the assignment. but both for #423 (opentree), nothing for deployed-systems..
23:33 jimallman oops. just updated email inbox and there it is!
23:33 jimallman (2 more emails)
23:34 jimallman if these can wait, i’m taking a break. i’m starving.
23:35 towodo of course they can wait, we’re done for the night (that’s what ‘declare victory’ meant)
23:50 codiferous joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary