Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-08-22

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:06 codiferous joined #opentreeoflife
12:30 codiferous joined #opentreeoflife
13:27 towodo joined #opentreeoflife
13:40 codiferous jimallman: sorry about the bugs on devapi tnrs. i think they are fixed now. let me know if you find more
13:40 codiferous there is a kludge in place that should keep them from resurfacing while i do another db rebuild to try and eliminate the source of the problem
13:55 josephwb mapping looks to be working on devtree
14:09 josephwb jimallman: new stuff going to production soon?
14:09 josephwb TNRS stuff
14:26 josephwb you there jimallman?
14:41 jimallman josephwb: hi, catching up now
14:42 jimallman i’m working on a simple UI for choosing among mapping alternatives. not sure if it’ll be ready soon enough to wait for it.
14:43 josephwb left #opentreeoflife
15:17 josephwb joined #opentreeoflife
15:19 josephwb jimallman: i missed your response, as I had to shut down *everything* in order to get MLS (an alternative method to treemachine) to work
15:19 jimallman no biggie, hereit is again:
15:19 jimallman i’m working on a simple UI for choosing among mapping alternatives. not sure if it’ll be ready soon enough to wait for it.
15:19 jimallman we’re sorting out the cleanest path to production for several recent changes...
15:19 josephwb but is the exact matching up on production?
15:20 josephwb ok
15:39 jimallman josephwb: production is currently running the (very) conservative OTU mapping, with exact matching. but it still has the bug that causes mapping to fail in a too-specific search context (for example, mapping from ‘All life’ works, but ‘Insects’ fails).
15:44 josephwb okay, thanks for the update
15:44 josephwb cody fixed that problem.
15:44 josephwb it works nicely on dev
15:45 towodo joined #opentreeoflife
15:51 codiferous joined #opentreeoflife
16:06 jimallman codiferous: i’m implementing a multiple-mappings UI patterned on your mockup, so far so good
16:06 jimallman https://github.com/OpenTreeOfLife/opentree/issues/407
16:06 jimallman (screenshot above)
16:15 josephwb looks nice
16:16 josephwb but we are going to get the simple exact matching up on production before this fuzzier matching is complete, yes?
16:17 josephwb jimallman ^
16:18 jimallman i’ll see if i can cherry-pick Cody’s fix for the search-context bug.
16:28 jimallman Here’s a ticket and a report of a temporary fix (not sure if this is ready for prime time, asking Cody here): https://github.com/OpenTreeOfLife/taxomachine/issues/62
18:23 towodo joined #opentreeoflife
19:43 codiferous joined #opentreeoflife
19:44 codiferous jimallman, you there?
19:44 jimallman codiferous: hi there
19:51 codiferous hi jim
19:52 codiferous i was just doing a little testing on devapi
19:52 codiferous i get a 404 from github when trying to log in
19:52 codiferous https://github.com/login/oauth/authorize?state=1bb6d75360bd16f7015605ddb7700e18&redirect_uri=devtree.opentreeoflife.org%2Fwebapp%2Fuser%2Flogin&response_type=code&client_id=ID_NOT_PROVIDED&scope=public_repo
19:53 jimallman ok, will try to reproduce here.
19:54 jimallman hm, i can login to both the synth-tree viewer and curation tool. which one gave you trouble?
19:55 jimallman duh, i see it in the URL
19:55 jimallman codiferous: any chance you’re playing games with /etc/hosts? mapping devtree to another box?
19:56 codiferous not intentionally
19:56 codiferous haven't done anything with /etc/hosts
19:56 jimallman ping devtree.opentreeoflife.org will tell the tale
19:56 jimallman for me, this shows ot16 as the real hostname, and 54.190.10.24 as the IP
19:57 * jimallman is trying to think how we could be on the same devtree, but only one has the GitHub client ID
19:58 codiferous joined #opentreeoflife
19:58 codiferous also get the same effect on my desktop
19:58 codiferous has been working fine previously
19:58 * jimallman is trying to think how we could be on the same devtree, but only one has the GitHub client ID
19:59 josephwb i just logged in
19:59 codiferous ah!
19:59 codiferous that was it. i was using devapi not devtree
19:59 codiferous devtree works
20:00 codiferous interesting, so we have read-only curator instances running on servers other than tree and devtree?
20:00 josephwb jimallman: can i close this: https://github.com/OpenTreeOfLife/treemachine/issues/111
20:00 josephwb seems we have this working with getMRCA
20:01 jimallman re: read-only curator instance… no, that was my bad. i accidentally pushed opentree to the devapi server
20:01 towodo there could easily be a redundant webapp/curator running on ot10 (devapi).  maybe we should shut it down
20:01 jimallman and vice versa (taxomachine to the devtree server). we don’t have a clean un-install tool yet, so i just left them. i didn’t think they could do any harm, but i see there’s a edge case…
20:02 jimallman towodo, what’s the best way to clean these up? i suppose removing the symlinks in web2py/applications is the cleanest. we can’t just stop web2py, because the phylesystem-api instances is running there too
20:03 towodo rm  /web2py/applications/curator  or something like that
20:03 towodo yes, just remove the symlinks
20:03 jimallman yep, will do
20:03 jimallman how about taxomachine on ot16 (devtree)? i did that one, too. maybe we need a component whitelist in the server-config files.
20:05 codiferous joined #opentreeoflife
20:06 jimallman josephwb: yes, the test cases in #111 work now on devtree, but not on production. i’ll close it but make a note of that.
20:06 josephwb i got impatient and closed it myself :-D
20:07 jimallman done and done!
20:07 jimallman josephwb: codiferous: if we have a sensible UI for choosing from among mapping options, do you think we should also use it for “bulk” mapping? it slows things down, but mainly for the OTUs without easy choices. and it won’t auto-select anything, just show choices to the curator.
20:08 jimallman asking because i’m getting close to done.. probably later this afternoon.
20:08 josephwb can we do an exact pass first?
20:09 jimallman yes, it will just force the curator to be very precise. (still waiting for which commit(s) to cherry-pick from codiferous)
20:10 josephwb maybe try this out on devtree, get some empirical folks to use it.
20:10 josephwb like Dail.
20:10 josephwb he works with the messiest data
20:11 codiferous you could always try an exact query first, then only execute a fuzzy query if there are x | 2 > x > 0 hits
20:11 codiferous that would be fast and reliable
20:12 codiferous and would not require the user to use a specified order of operations
20:12 codiferous guess i could have said x != 1 ;p
20:14 codiferous and regarding commits, everything is up on devtree now. it will require a database update to push to production. i have tested against various curl calls from previous bug reports, but it would be good for all of us to do a bit more testing before pushing to production
20:15 codiferous waiting to submit a pull request pending that
20:16 * jimallman has disabled the offending apps on wrong servers
20:16 josephwb jimallman: you don't use the "getDraftTreeID" service, do you?
20:17 josephwb looking at codiferous issue here: https://github.com/OpenTreeOfLife/treemachine/issues/103
20:17 jimallman yes, i believe we use this in the synth-tree viewer, to get initial starting point
20:18 josephwb really? it is pretty new.
20:18 josephwb glad it might be useful, tho.
20:19 josephwb looking at the treemachine services. most have ottId and nodeId as long. a few as string.
20:19 jimallman confirmed, we use this (have for quite some time, actually) to get the draft tree ID and starting node ID
20:20 codiferous right. we should try to get the services' treatments of ids consistent before the hackathon
20:20 josephwb yes, i am looking at that right now.
20:21 jimallman OK by me. But i didn’t see a final resolution of the disagreement in that issue. towodo, are you OK with longs instead of strings for neo4j node IDs?  https://github.com/OpenTreeOfLife/treemachine/issues/103
20:21 josephwb need to make sure any changes are complemented by jimallman
20:21 codiferous joseph, we we put make a branch on treemachine to update all the services to using the proper datatypes, etc. and then pass that off to jim to make the required changes in the web clients?
20:22 josephwb sounds good.
20:22 jimallman that seems like a clean approach, thanks
20:22 towodo longs for neo4j node ids, sure.  not for ott ids
20:23 towodo in fact I’m not keen on occurrence of ‘ott’ in method names. what if someone wants to build a system using a different taxonomy?
20:23 codiferous which method names?
20:24 codiferous if someone wants to use a different taxonomy, then there will need to be heavy duty refactoring of treemachine and taxomachine
20:24 codiferous and ot-base
20:24 codiferous well, maybe ot-base
20:24 towodo no method names in the documented API I guess, but I thought I heard some under discussion
20:25 jimallman maybe use “taxid” instead? this change would touch a *lot* of code, but it should be easy enough to find and (carefuly) replace.
20:25 josephwb services with nodeid as string: getSourceTree, getSyntheticTree.
20:25 codiferous switching to strings will touch a lot of code as well
20:25 codiferous taxomachine uses longs for all ottids
20:25 towodo I wouldn’t rock the boat at this point. There are cases where ottId is appropriate, like in the parameters and attributes, since other kinds of ids could be added
20:26 josephwb treemachine does for most too
20:26 towodo oh well. smasher has to use strings because some of the input taxonomies use non-integer ids
20:26 codiferous so can we treat ott ids as longs then, and leave open the option for other ids to be other datatypes?
20:27 towodo I suppose.  If the assumption is already that deep may as well be up front about it
20:28 codiferous ok
20:28 jimallman fwiw, it does seem like a move to more free-form taxonomic ids is going to need lots of refactoring + testing anyway. i don’t think we’re making it much harder for the Tomorrow People by standing pat now.
20:28 josephwb jimallman: do you use: getSourceTree, getSyntheticTree
20:29 towodo smasher is perfectly capable of taking any taxonomy and replacing its ids with integers
20:30 jimallman josephwb: i see getSourceTree in the argus code, but that might be dead code… lemme check for getSyntheticTree
20:30 josephwb ok
20:32 jimallman yes, definitely using getSyntheticTree to move from node to node in argus (synth-tree view)
20:32 josephwb okay.
20:33 josephwb that is one change we would need to be aware of.
20:33 josephwb i.e. string -> long
20:34 jimallman josephwb: looks like we’re not current using getSourceTree, but we’ll likely revive this when we start showing source trees in argus.
20:34 josephwb ok\
20:34 codiferous yes, and any exposed services may be used by hackathon participants
20:34 josephwb but we are in agreement to convert all of these id args to longs?
20:35 codiferous towodo has approved
20:35 josephwb ok.
20:35 josephwb i can do this.
20:36 towodo no database changes, right? just the plugin(s)?
20:36 josephwb yes
20:36 towodo and can it be done upward compatibly?  or does it have to be done in sync with client change?
20:37 codiferous it needs to be synchronized with client change
20:37 codiferous can't have two arguments with same name
20:38 codiferous joseph could make the changes on a branch and then issue a pull request to jim so he can make corresponding changes before merging
20:38 towodo that sounds good to me
20:38 towodo change to documentation required?
20:39 towodo I mean to https://github.com/OpenTreeOfLife/opentree/wiki/Open-Tree-of-Life-APIs
20:41 josephwb added to the issue
20:41 codiferous yep
20:45 towodo I’m concerned about other downstream effects, e.g. any hackathon participants who might have started programming against the API, or Lyndon’s script (I doubt that it cares, but would be nice to be sure)
20:47 codiferous currently, i think the biggest effects will be on our clients
20:47 jimallman hmm. we have the API versioning, but it seems a bit drastic to bump to v2 over this.
20:48 codiferous if we wait until closer to the hackathon, the effects will likely be much more widespread
20:48 jimallman can you recognize/coerce any errant strings to longs?
20:48 jimallman (actually, i thought this was already happening)
20:49 codiferous the service will fail if the wrong datatype is supplied for any argument
20:49 towodo that’s out of the control of the plugin - automagic on neo4j’s part
20:49 jimallman based on JSON types, i gues
20:49 codiferous yes
20:49 codiferous only some types are acceptable
20:49 towodo it all goes through Java’s type system
20:50 * jimallman nods
20:50 jimallman well, the hackateers should all be on the mailing list. should be easy to update their API clients, they just need to be notified.
20:50 codiferous yes, i agree
20:51 codiferous and the consistency will be helpful for those of them learning the api
20:54 josephwb +1
20:54 josephwb but this can wait until after the name mapping is fixed, yes?
21:00 towodo I’d like to see a minimal production deployment to fix name mapping, yes
21:00 josephwb i hope people are not contributing trees now
21:00 towodo jimallman is waiting to hear from codiferous re which commits should be in it
21:00 josephwb or over the past week
21:01 jimallman josephwb: if it’s that bad, we have the means to suspend logins and curation. sorry i didn’t mention this before.
21:01 josephwb i don't know if anyone other than me was trying to use it
21:01 jimallman i’ll check for recent activity in the docstore on GitHub
21:02 jimallman https://github.com/OpenTreeOfLife/phylesystem-1/commits
21:02 towodo https://github.com/OpenTreeOfLife/phylesystem-1/commits/master  Chris Owen has been working away
21:02 codiferous all the changes are on devtree
21:02 codiferous we can push anytime. we should probably do some testing first
21:03 codiferous pushing to production will require a db update
21:03 towodo but that’s the point. we were trying to push the names change to production without doing a db update
21:03 towodo is that impossible?
21:04 codiferous essentially, yes
21:04 jimallman i was hoping we might cherry-pick just the fix described here: https://github.com/OpenTreeOfLife/taxomachine/issues/62
21:04 josephwb i fear chris is an "accept-all" sort of person...
21:04 codiferous there had already been a db update before that temporary fix
21:04 jimallman i see, gotcha
21:05 josephwb jimallman: stephen and i have a treemachine DB built using "alternative methods" would like to put it on the dev server.
21:05 josephwb still a treemachine DB
21:05 josephwb so, everything should work.
21:05 josephwb just want feedback from others.
21:06 towodo are we ready for a synthetic tree update to production?  if we’re going to have downtime and the long update process, may as well go whole hog
21:06 jimallman so no other ripples, it should just import normally?
21:06 josephwb yes
21:06 josephwb reference links won't work
21:07 towodo ‘reference links’? won’t work in what?
21:07 codiferous @towodo, yes. hence my requests to do some otu-mapping testing to make sure what we push is solid
21:08 codiferous sorry, that was confusing. yes, long update process. re:synthetic tree update, i don't know
21:08 josephwb links to sources won't work,. because the sources are subsetted supertrees
21:08 josephwb i.e. no distinct reference
21:08 jimallman …because this db was built with “alternative methods”?
21:08 josephwb but that doesn't matter. just want people to look at it.
21:09 josephwb an alternative supertree method (required by reviewers)
21:09 jimallman neat
21:09 josephwb no, not.
21:09 josephwb it is shitty
21:09 jimallman i see
21:10 towodo I don’t care much what’s on devtree/devapi, that’s for your experimentation.  The question is whether there’s a synthetic tree using non-alternative methods that’s ready to push out to production. This is what the hackathoners would see
21:10 josephwb i have a much newer version. it is on devtree right now (I believe)
21:11 towodo if you want another server for the alternate method tree that can be arranged.
21:11 josephwb devtree is fine
21:13 towodo So it sounds like we’re moving toward a production release with new curation features, new taxomachine db, and new treemachine db, all based on what’s now on dev.  Then after that’s done, clobber dev with alternative tree.
21:14 josephwb i have a slightly newer synth DB. should i upload that?
21:14 josephwb or just copy from dev to prod?
21:15 towodo do *not* copy to production (you can’t anyhow because you don’t have the key).  put the right thing on dev so it can be checked.  then I’ll copy to prod from there.
21:16 josephwb not me, you
21:16 josephwb yes
21:16 towodo I have no way to decide whether the slightly newer one is better, so I leave that up to you.  The install probably won’t happen until next Monday at the earliest
21:17 towodo I have to set aside four hours, warn users, and so on. Probably Tuesday
21:18 jimallman i’d love to squeeze in the latest mapping tool (choose among options) if we can
21:18 jimallman i can have it up for testing on devtree tonight, if we think Gail and others can take a look.
21:19 towodo Dail?
21:19 jimallman Dail, sorry
21:19 josephwb i just need exacting matching on production so i can fix some trees
21:21 jimallman josephwb: you might make some headway by just sticking to the ‘All life’ context, since that won’t trigger the duplicate-mapping bug
21:21 josephwb i am afraid of homonyms :-(
21:22 josephwb Does this look right for copying to dev?
21:22 josephwb git pull the opentree repository
21:22 josephwb tar -C life_synth.db -czf treemachine.db.tgz .
21:22 josephwb mv treemachine.db.tgz opentree/deploy/treemachine.db.tgz
21:22 josephwb cd opentree/deploy/
21:22 josephwb bash push.sh -c ../../deployed-systems/development/ot10.config pushdb treemachine.db.tgz treemachine
21:22 jimallman the code currently on production should “fail safe”, right? it will just block some mappings that would have worked
21:24 * jimallman hasn’t done this procedure, comparing it to towodo’s notes...
21:25 josephwb so, I have 2 DBs: 1) new synth, 2) alternative method
21:25 towodo you need to refresh the deployed-systems repo
21:25 josephwb want people to see the alt method results
21:25 josephwb so, push it first?
21:25 towodo hey wait.
21:26 towodo I thought we were going to push out a new non-alternate synth tree, scrutinize it, then push to production.
21:26 josephwb yes
21:26 towodo then either push the alt tree to a different server, or wait until after the release
21:27 towodo so you’re trying to push the new non-alt tree?
21:28 towodo if that’s what you’re doing then the commands above are about right
21:28 josephwb can you push the DB on dev right now to production?
21:29 josephwb then i can put the alt DB on dev
21:29 towodo no!  the whole process takes about 4 hours
21:29 josephwb i can wait 4 hours
21:30 josephwb we want people to scrutinize the alt results
21:30 josephwb ASAP
21:30 jimallman sounds like a second server might make sense, then
21:30 jimallman main devtree for testing the pending production goodies, and another just for alt-tree
21:32 towodo I will have 4 hours on Monday or Tuesday. Tuesday is better since that gives more of a chance to test Jim’s work
21:32 towodo I can reserve a server and you can do whatever you like with it
21:33 towodo but I need to go now, will do it tonight or tomorrow
21:34 josephwb i can push the most recent treemachine synth DB now
21:34 josephwb ?
21:36 towodo no opinion.  if it’s better than what’s there, and is non-alt, then why not.
21:36 jimallman sounds like we can import the non-alt db to devtree, yes
21:36 josephwb ok, doing
21:38 towodo bye, gotta run
21:38 jimallman i can help with the db import
21:41 josephwb it is going
21:42 josephwb bash push.sh -c ../../deployed-systems/development/ot10.config pushdb Life_v2.8draft5_mapcompat_13August2014.db.tgz treemachine
21:42 jimallman ok, give a shout when it’s up (or if you’d rather do all the steps, just let me know)
21:43 josephwb erg. i am leaving.
21:43 josephwb *should have screened it*
21:44 josephwb does that use rsync?
21:44 jimallman no problem, i’ll check for done-ness
21:44 josephwb thanks
21:45 josephwb i can log in later to see what is operating
21:46 jimallman ok, and i’ll send email either way
21:46 josephwb thanks
21:47 josephwb left #opentreeoflife
22:15 codiferous it looks as though an oti rebuild might be required to correct some matching problems
22:43 codiferous nevermind. think i've got a fix
23:03 * jimallman nods
23:05 codiferous i think its fixed. just pushed to devapi
23:15 towodo joined #opentreeoflife
23:17 codiferous towodo: can i close this issue: https://github.com/OpenTreeOfLife/taxomachine/issues/53
23:18 towodo I think so
23:19 codiferous excellent

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary