Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-06-13

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:29 kcranstn joined #opentreeoflife
00:57 josephwb joined #opentreeoflife
01:00 mtholder joined #opentreeoflife
01:17 kcranstn joined #opentreeoflife
03:21 josephwb joined #opentreeoflife
11:01 josephwb joined #opentreeoflife
11:25 towodo joined #opentreeoflife
11:37 josephwb joined #opentreeoflife
11:57 josephwb joined #opentreeoflife
12:34 josephwb joined #opentreeoflife
12:37 josephwb towodo: you haven't refreshed treemachine, have you?
12:38 josephwb if you let me know how to do it, I can
12:38 towodo you can log in to ot10 and look at log/messages to find out when the last git pull on any repo happened
12:38 towodo if that's what you mean.
12:39 towodo the database was copied from ot9
12:43 josephwb um, i don't don't know how to log into otX
12:53 josephwb i'll talk to cody and stephen
12:58 mtholder joined #opentreeoflife
13:26 blackrim joined #opentreeoflife
13:30 kcranstn joined #opentreeoflife
13:36 josephwb joined #opentreeoflife
13:37 blackrim OK, so just read all the discussion about the prefixes. What do I need to do
13:42 blackrim ok, no one on so just responded on the issue - i think, if i understand correctly, this is a gcmdr issue not treemachine
13:50 towodo blackrim, from what i understand, i agree with you.
13:51 towodo if treemachine ever parses then "123_456" syntax then it will need to be modified.
13:51 towodo but mainly it's a gcmdr thing.
13:53 blackrim ok
14:15 kcranstn joined #opentreeoflife
14:16 kcranstn hey @blackrim - why the rush to get to synthesis v 2.0?
14:17 blackrim because we are hoping to have it before the paper is published (hopefully before the revisions would be due -- and if we are accepted we are looking at probably 2 weeks at most for the revision)
14:17 kcranstn ah, I didn’t know that was what you are thinking
14:17 blackrim yeah, would love to get more of the fungal and microbe things better
14:18 kcranstn @jimallman - what is the status of the ssl / oauth issues? Are we ready to unleash the app on the curators?
14:25 mtholder joined #opentreeoflife
14:29 mtholder joined #opentreeoflife
14:32 jimallman kcranstn: i think i’ve got the mixed-content warnings under control, but most of my testing is in the tree-view.
14:32 jimallman testing in the curator now. for now, the path of least resistance is to stay in HTTP
14:33 jimallman i do not expect to make much trouble (server restarts, etc) today, though we might need to make some apache tweaks to address problems with CORS + redirects.
14:37 blackrim josephwb: tried to clean up gcmdr a bit more
14:37 josephwb me too
14:37 josephwb deleted some history for binary/nexson files
14:37 blackrim i pushed, so you might want to pull to see if we are overlapping on that
14:37 blackrim sounds god
14:37 josephwb not sure what it accomplished
14:37 blackrim good
14:38 blackrim i was going to add the bit for interacting with the api. going to need to download all the studies first and then do the synth.
14:40 josephwb ok
14:41 blackrim with the files the way they are now, we are going to have to set the python path to run things in the files_for_submission folder, just fyi
14:42 josephwb ok
14:51 mtholder blackrim see email. you can certainly roll your own interactions with the api if you want, but (to a large extent) that is why I was writing peyotl
14:52 mtholder the docs aren't great (but they will be next week as I prep for my talk at ievobio ;-)
14:52 blackrim no problem. emailed back, but jonathan is right we can just use the api
14:53 mtholder there are also wrappers for calling the api in peyotl. It is that get_from arg ('api' instead of 'local', though you can't loop over all studies if it is 'api')
14:54 mtholder FWIW, on the "synthesis right now" issue: I think that it is *far* more important that we have a sane workflow for building the synthesis and browsing the tree than it is to make sure that the tree people see when the pub comes out is not the tree from the paper
14:56 mtholder on the gcmdr repo size: deleting files won't have much of an affect on cloning times, right? (given the way git works)
14:56 blackrim sorry, not sure I get what you mean by "the tree people see when the pub comes out is not the tree from the paper"
14:57 mtholder It sounds like you really wanted to replace the synth tree very soon
14:57 mtholder (I'm not saying we trash the old one)
14:57 blackrim yeah, i want to get the fungi and microbes better (if possible) and basically turn the crank once more before publication
14:58 blackrim will be good to have it go once more to show that we actually can update as well
15:00 blackrim i am not sure i understand which you mean is more important that is all
15:00 mtholder perhaps I'm misunderstanding you.
15:00 mtholder my concern:
15:01 mtholder it seemed like earlier in the week we were having issues with the new curation tool
15:01 mtholder and a feeling of "we need it working by friday"
15:01 mtholder or we'll use the old way.
15:01 mtholder rereading your email of this morning, I think I was worried about nothing...
15:02 mtholder I do feel some urgency, but mainly an urgency for the next round of synth to not use a lot of stop-gaps in the workflow.
15:03 mtholder AFAIK, the new curation tool is working well enough for us to say: let's edit all studies here.
15:03 mtholder (and not in pg)
15:03 mtholder but josephwb pointed out some useability issues (with lots of otus, for example).
15:04 mtholder so I don't want that to be a game changer wrt what tool we use for curation. I'd rather put a high priority on fixing them, rather than go back to editing in pg.
15:04 mtholder that is my 2 cents
15:05 mtholder now I'll go back to breaking the phylesystem-api
15:05 blackrim all sounds good. i am good with using the new application (though I think that there needs to be some redirecting or something from pg for a bit until pg can take in the nexsons, etc. to clear up confusion for the users)
15:06 blackrim for the stopgaps, hopefully the only one is really where we get the nexsons and switching to the curl call and api will fix that (doing that now). if there are others, let me know and I can deal with that
15:07 jimallman mtholder: i did some digging into the CORS+redirect problems last night...
15:07 jimallman are you working on this as well?
15:07 mtholder well...
15:08 mtholder i'm being a bit stupid/crazy
15:08 mtholder i'm looking at how tough it would be to use pyramids rather than web2py
15:08 mtholder there is a sledgehammer approach
15:08 mtholder but I'm not committed to that.
15:09 mtholder did you find a solution to CORS in web2py?
15:10 mtholder (pyramids just for phylesytem-api, not any of the other pieces)
15:16 josephwb can anyone tell me how to connect to the otX machines?
15:16 mtholder jimallman has been rendered mute by my stupidity...
15:16 josephwb is there documentation somewhere?
15:16 mtholder first you need the private ssh-key.
15:16 mtholder can you get that from blackrim?
15:16 mtholder or cody?
15:17 blackrim you mean the pem
15:17 mtholder yeah
15:17 jimallman sorry, trying to follow emails and IRC and… catching up now
15:17 blackrim I can send that to joseph
15:18 mtholder then set up your .ssh/config with the info in https://github.com/OpenTreeOfLife/deployed-systems/blob/master/opentree-ssh-config.txt
15:18 mtholder I just groupings like:
15:18 mtholder Host ot11
15:18 mtholder Hostname ot11.opentreeoflife.org
15:18 mtholder User opentree
15:18 mtholder IdentityFile /home/mholder/.ssh/opentree.pem
15:19 mtholder in that config (jim taught me about the User and IdentityFile bits)
15:19 mtholder after that I can "ssh ot10" and be logged in as the opentree user
15:19 mtholder or ot#
15:20 jimallman or login directly as admin using “ssh admin@ot10”
15:20 jimallman https://gist.github.com/jimallman/0a9f7cec6f29429e2d53
15:21 jimallman josephwb: see link above for convenient snippet of .ssh/config
15:22 jimallman mtholder: regarding CORS+redirect, it’s important that the initial “preflight” call (http verb = OPTIONS)  returns a 2xx response code, not a 302
15:24 mtholder Ahh. thanks
15:24 mtholder that is in the v1 func in the api
15:24 mtholder which is why it is not set for those routes outside of that.
15:25 mtholder (those where we get the v1 part of the url via apache proxy tricks)
15:27 mtholder I'm not sure how easy it will be to fix in our current configuration, though.
15:36 jimallman are any of these calls being routed through apache?
15:36 jimallman (redirected before they get to web2py, i mean)
15:37 jimallman mtholder: ^ sorry, i’m still bouncing between tasks here
15:37 towodo blackrim, there's been no hard progress on microbes (re: "love to get more of the fungal and microbe things better") other than the addition of a small number of "unclassified X" clades
15:40 blackrim towodo: you mean in the taxonomy?
15:40 josephwb jimallman: for the synthetic tree viewer, is it possible to "remember" the setting of "hide properties" when traversing the tree? i hide them, click one node up/down, and the properties come back.
15:40 blackrim i mean just in some of the major lineages that I know there are trees for
15:40 towodo yes, in the taxonomy.
15:40 blackrim yea, wasn't expecting a big change in taxonomy but I think there are some other bits
15:41 jimallman josephwb: this is a known issue (pending request), but there’s no super-easy way to accomplish this yet.
15:41 jimallman the argus view does remember some useful bits, like zoom-level, so i should be able to build on that. haven’t been in that code for awhile…
15:43 jimallman josephwb: please feel free to add ideas to either of these issues:
15:43 jimallman https://github.com/OpenTreeOfLife/opentree/issues/184
15:43 jimallman https://github.com/OpenTreeOfLife/opentree/issues/182
15:43 josephwb ok, no biggie, just curious
15:46 towodo mtholder, we / you are trying to get rid of the redirects in the apache config, right?
15:46 towodo you're renaming the web2py 'api' application to 'phylesystem', right?
15:46 towodo and updating its clients to match?...
15:47 mtholder yes. and yes.
15:47 towodo and leaving api behind as a redirect, with no expectations it will work well?… anyhow that's what I was thinking
15:48 mtholder the v1 stuff is part of the problem, too.
15:48 towodo but that works in api/ … how is phylesystem different?
15:48 mtholder study_list -> v1/study_list needs a redirect.
15:48 mtholder that never was namespaced in api
15:49 towodo but it's the same kind of problem… rename api/study_list to phylesystem/v1/study_list … why would that be hard?
15:49 mtholder moving it inside the def v1(): in web2py does not work
15:49 towodo ouch. ouch.
15:50 kcranstn joined #opentreeoflife
15:50 towodo and you've already explored routes, and explained it to me (now I forget but it doesn't matter).
15:50 towodo so you've concluded web2py is the weak link. … more ouch.
15:50 mtholder it is the fact that we're using the @request.restful() for magic
15:51 mtholder well. i'm not sure.
15:51 mtholder i'm probably overreacting.
15:51 mtholder i just need a few hours to get mad at pyramids and then I'll simmer down and fix the web2py version.
15:51 towodo I never trusted it… but that means much, I'm paranoid about everything
15:51 mtholder (or I'll get it all working in pyramids)
15:52 mtholder web2py's strengths are the DAL and its templating language, AFAIK
15:52 mtholder the phylesystem api uses neither.
15:52 towodo s/means much/ doesn't mean much/
15:55 jimallman mtholder: i’ve almost got my head above water here.. can probably help to sort out web2py weirdness, but i have a Tree Illustrator call in a few minutes (back ~1pm)
15:56 mtholder no real rush on that stuff. thanks, though. jimallman
16:01 jimallman ah, good to know
16:09 mtholder talking to snacktavish. think we figured out how to deal with this in web2py.
16:09 mtholder damn.
16:09 mtholder i really wanted to abandon it and have a good reason that I could point to.
16:09 mtholder does "capriciousness" count as a good reason?
16:15 snacktavish joined #opentreeoflife
16:18 jimallman s/capriciousness/spite, perhaps?
16:19 blackrim branch2sha will always be in the api gathered studies right?
16:19 mtholder s/capriciousness/righteous indignation/
16:20 * jimallman ponders what it is about web2py that inspires such passion
16:20 mtholder blackrim: yes (but probably only of use to the curator app).
16:20 blackrim I have been recording the sha, but not i need to do the branch and the sha? that is going to make the tree id in treemachine quite long but that is fine
16:20 mtholder just the sha is enough
16:20 mtholder the branches are ephemeral.
16:21 mtholder don't get me started jimallman
16:21 blackrim ok ,and i can always get it from that key
16:21 mtholder 'sha' is the best key for that.
16:21 mtholder branch2sha is for indicating other branches (and their shas) that have not been merged to master.
16:22 mtholder that would let the curator app alert the curators that there are some different versions, and someone needs to merge.
16:22 blackrim sounds good. using sha
16:22 mtholder I think that for synth the rule should be "it must be on master"
16:29 mtholder I think that I'll finish up the move to pyramids (because I've started and I'm not that far from done). I'll put it on a branch with no plan to deploy it. If we take that route (sorry for the pun), it'll be easier to resurrect.
16:29 mtholder then, jimallman, you'll see the source of my passion.
16:30 kcranstn pyramids?
16:31 kcranstn ah, is this the successor to pylons?
16:31 jimallman i think it’s Pyramid (singular)
16:31 mtholder yeah.
16:32 towodo what you build to escape mortality.
16:32 mtholder or yeahs if i feel like adding an s
16:33 jimallman It’s a great project, actually. The Pyramid team is responsible for lots of early python web frameworks, and pyramid embodies lots of Lessons Learned (and regrets) from those projects.
16:35 jimallman …and it has the most “metal” schwag of any FOSS project i know of: http://www.pylonsproject.org/static/images/pyramid-tee-banner.png
16:35 mtholder does look like an iron maiden cover.
16:36 jimallman you can’t get away with something that juvenile unless the technology is really good
16:37 blackrim so these nexsons have different format than treemachine is use to
16:37 blackrim are they the same if you go from "data" key?
16:38 mtholder output_nexml2json=0.0.0 in your query
16:38 blackrim where do i do this?
16:39 mtholder the GET of the study. https://github.com/OpenTreeOfLife/phylesystem-api/blob/master/docs/README.md
16:39 mtholder https://github.com/OpenTreeOfLife/phylesystem-api/blob/master/docs/README.md#get-arguments specifically
16:40 mtholder or (if you want to do it client side) there is scripts/nexson/nexson_nexml.py in peyotl
16:42 blackrim well, i can just take from data (i think) if that is the same as how phylografter output them from there. i would rather use defaults so that other users have an easier time
16:42 mtholder ?
16:44 blackrim if the nexson format from the data key using the default curl call is the same as the format from phylografter export, then I would rather just do the defaults - testing now
16:45 josephwb towodo: stephen got me direction on how to connect to ot10
16:45 josephwb i am in the treemachine directory now.
16:45 towodo ok, good. i would have had to hunt for instructions. (not that it's complicated)
16:45 towodo no… I mean ~/log/messages
16:45 josephwb do i need to do anything before pulling from github?
16:46 josephwb e.g. stop neo4j?
16:46 blackrim I can skip over these, but is this ok Exception in thread "main" java.lang.RuntimeException: missing property value for name: {"message":[],"@xsi:type":"nex:LiteralMeta","@property":"ot:messages"}
16:47 mtholder in general, josephwb: don't pull on that machine (unless towodo says to)
16:47 mtholder we use the opentree/deploy/ scripts
16:47 mtholder (but you can pull and mess around to debug issues, of course)
16:47 josephwb how do i do that?
16:47 towodo mtholder, you think we should create another server for new tree testing?…
16:47 josephwb yes, just testing/debugging now
16:48 blackrim ok, getting a bunch of comments out on load of the nexsons -- just want to make sure they are ok, i will send in an email
16:48 mtholder treemachine load?
16:49 mtholder i'll wait for the email
16:49 towodo I had in mind that smith lab would use ot10 for testing.  but you're right there is a curator/taxomachine interaction
16:49 mtholder (or the movie)
16:49 blackrim yeah, using nexson reader from jonathan
16:49 towodo wow that was a long time ago
16:49 mtholder ot:messages are skippable
16:50 mtholder (mainly things the curator app might show)
16:51 josephwb blackrim: is it saying something like "I don't know what that is"?
16:52 josephwb or "Warning: dealing with unknown property"?
16:52 blackrim mostly it is having a hard time getting some taxon ids. i am investigating a bit before sending an email. there are also a bunch of "missing property value for name"
16:53 josephwb or something else?
16:53 blackrim getting pgloadind     OTT ID missing name "null" nexsonid "node1139235"
16:54 blackrim I am going to turn off the TNRS and just leave the pruning and will investigate while that goes on
16:55 jimallman blackrim: most likely that message (missing property “mesage”) is due to a recent change in Nexson, from “messages” : {“message”: [ <msg>, <msg>, … ] }   to simply  â€œmessages”: [ <msg>, <msg>, … ]
16:56 jimallman the former is badgerfish, the latter is our preferred (simplified) structure
16:56 josephwb yerp
17:03 josephwb mtholder: so, to update treemachine on ot10 I just run push.sh from opentree/deploy?
17:03 blackrim something is up with the node ids on reading "pgloadind   postpruning newick tree "((((((((,,,,,,,),(,,,,),,,,,,),(,,,,,,),(,)),,((,,),(,,),),((,),(,,,,(,,),,(,,),)),,,,(,(,,,,,),(,,,,,,),),,),,),,(,,,,)),(((,,,),,(,),,(,,,,,,,,,,(,,,),(,,,))),((,,,,,,,),),((,,,),(,,,,),(,),(,,),,,,,))),(,,(,)),(,,,),(,,),,,(,),(,,,(,,,,)),,)""
17:04 jimallman josephwb: if your filesystem layout is typical (‘deployed-systems’ and ‘opentree’ repos side by side), it looks like this:
17:04 jimallman opentree/deploy (master u=) $ ./push.sh -c ../../deployed-systems/development/ot11.config
17:04 josephwb yikes
17:04 mtholder plus a component name
17:04 jimallman s/ot11/ot10
17:04 mtholder treemachine perhaps? checking...
17:05 josephwb and git pull directly on ot10 is not cool?
17:05 mtholder for testing deploy problems that is OK.
17:05 mtholder but the deploy script does the necessary neo4j dance.
17:06 josephwb gotcha
17:06 jimallman here are default components pushed in ot10.config: OPENTREE_COMPONENTS='api oti treemachine taxomachine'
17:06 mtholder and if we don't use the deploy system, it gets out of synce.
17:06 josephwb good to know
17:06 josephwb so, i don't need to change any of the settings in ot10.config?
17:07 mtholder not if it was just treemachine code that changed.
17:07 mtholder those control how the apps talk to one another.
17:07 mtholder they only change when we do things like "lets make the curator on ot11 use the api on ot10"
17:08 josephwb ok, trying it… :-#
17:08 mtholder you may need to do a pull on the opentree repo to make sure deploy scripts are up to date
17:09 mtholder (the local copies from your machine will be scp'ed to the server)
17:10 jimallman josephwb: and please do a pull on deployed-systems as well (lots of recent changes there)
17:10 josephwb i am pulling everything!
17:10 jimallman PULL ALL THE THINGS
17:11 jimallman (and checkout ‘master’)
17:13 josephwb well, that didn't work
17:13 josephwb wait
17:14 josephwb think i know what is up
17:16 blackrim there is no longer a @label with each of the otus, this is the issue. seems like we will need to do ottTaxonName instead for tree node labels
17:17 josephwb things are happening!
17:17 josephwb 8-)
17:18 josephwb thanks jimallman and mtholder
17:18 jimallman blackrim: right, we’ve sort of deprecated @label as being too.. squishy
17:18 blackrim no problem. just updating things
17:18 jimallman i believe the notion was that we might restore these (from other properties) when converting form Nexson to NEXML…?
17:19 jimallman josephwb: glad to help. it’s fun when it works, innit?
17:19 josephwb oof. "Repository treemachine is unchanged since last time"
17:19 josephwb wha?!?
17:19 blackrim we used these for labeling nodes in the tree because they will usually be valid for newick (because they came in that way)
17:21 jimallman hm, less fun when it doesn’t work.
17:21 josephwb jimallman: what am i doing wrong? treemachine *has* been updated.
17:21 jimallman let’s see if you’re on the right branch… pushing from ot10.config, right?
17:22 jimallman default branch is ‘master’, and i don’t see any other specified in the config file
17:22 jimallman (that would be something like “opentree_branch treemachine some_other_branch”)
17:22 josephwb right
17:23 jimallman are your expected changes definitely on github? https://github.com/OpenTreeOfLife/treemachine/commits/master
17:24 mtholder joined #opentreeoflife
17:25 * jimallman is reviewing the deployment logs on ot10, in /home/opentree/log/messages
17:26 jimallman i don’t see any sign of you installing treemachine.. just peyotl
17:26 jimallman josephwb: ^
17:26 josephwb i'll check the config
17:27 jimallman OPENTREE_COMPONENTS should list the ones installed by default. if you added a component name to your call, it will override this
17:27 josephwb i did not
17:27 josephwb didn't edit anything
17:27 jimallman just to repeat, here’s my typical push:
17:27 jimallman opentree/deploy (master u=) $ ./push.sh -c ../../deployed-systems/development/ot11.config
17:27 jimallman drat, should be ot10!
17:27 josephwb i copied and pasted that (for ot10)
17:28 jimallman you might try specifically installing just treemachine, to see what happens:
17:28 jimallman opentree/deploy (master u=) $ ./push.sh -c ../../deployed-systems/development/ot10.config treemachine
17:29 mtholder blackrim: the idea is that (given the ambiguity of whether someone wants the originalLabel or the ottId or the ottTaxonName) we'd ask the user to be explicit about which field should be the @label.
17:29 mtholder (when we convert to newick, for instance)
17:29 jimallman josephwb: i’m tailing the log on ot10, to see what happens…
17:29 mtholder the default for -> nexml translation is ot:originalLabel.
17:31 josephwb hmm, no go
17:31 jimallman anything interesting in console output? maybe you’re missing a required secure file in ~/.ssh/opentree
17:32 josephwb perl warning: Setting locale failed.
17:32 mtholder context for that? I didn't know that we had any perl in the pipeline
17:32 jimallman hm, not sure. i’m guessing that’s not the show-stopper, though.
17:33 jimallman here’s a bit on what we expect to push from your local ~/.ssh/opentree
17:33 jimallman https://github.com/OpenTreeOfLife/deployed-systems/blob/master/README.md
17:33 jimallman more direct link: https://github.com/OpenTreeOfLife/deployed-systems/blob/master/README.md#sensitive-information
17:33 mtholder I don't think that the neo4j deploys need much in the way of special ssh keys, though...
17:34 jimallman true, just looking for clues. josephwb , mind if i try the same push from here? just to isolate the problem…
17:34 josephwb please do
17:35 josephwb ./push.sh -c ../../deployed-systems/development/ot10.config treemachine
17:35 jimallman OK, trying now with default components…
17:36 jimallman hm, the deployment log just shows peyotl…
17:36 jimallman (as for josephwb
17:37 jimallman sure enough, i get the same report: “Repository treemachine is unchanged since last time”
17:37 josephwb OPENTREE_COMPONENTS does not list peyotl
17:37 blackrim ok, i think i have the nexson reader in treemachine and the gcmdr working correctly with the api and updated formats
17:37 mtholder it is a dependency
17:37 jimallman right, but i assume this is a dependency of one of the named components
17:38 mtholder (peyotl is installed with phylesystem_api)
17:38 jimallman (there’s not a one-to-one relationship between repos and components)
17:38 josephwb ok
17:38 mtholder towodo and I were having a lot of issues with maven not thinking that things had changed, but can you tell if you are even getting to the point of invoking maven?
17:39 josephwb blackrim: sweet
17:39 jimallman i’m going spelunking in the ot10 repos, to see what SHA its treemachine is on
17:40 towodo i think the maven trouble was in taxomachine only.
17:41 jimallman ot10 treemachine is sitting on SHA 580706d4a4ef374adeb53756238c19da57ed5124, which matches the commit list on github (assuming we missed blackrim’s commit a few minutes ago):
17:41 towodo there are two taxomachine jars. the first gets built. then there's a check to see if the second needs to be built. it does, but the check says it doesn't. so the first jar gets updated, and no attempt is made to update the second.
17:41 jimallman https://github.com/OpenTreeOfLife/treemachine/commits/master
17:41 towodo 3 days ago. that sounds right.
17:42 josephwb yup
17:43 jimallman hm, it seems like we have some recent changere working already (optional ‘treeID’ arg for getSyntheticTree)
17:44 jimallman curl -X POST 'http://ot10.opentreeoflife.org/treemachine/ext/GoLS/graphdb/getSyntheticTree'   -H 'Content-Type: application/json'   --data-binary '{"format":"arguson","maxDepth":"3","subtreeNodeID":"414022"}'
17:44 jimallman (that’s a working curl call to ot10, w/ no treeID)
17:44 towodo and i'm not sure maven is the culprit.  the problem is in the change cody made, to add a non-plugin version of taxomachine, without a corresponding change in dependency management. (not that he could have anticipated what a problem this would be.)
17:44 towodo cool
17:45 josephwb huh. i was trying:
17:45 josephwb curl -X POST http://api.opentreeoflife.org/treemachine/v1/getSyntheticTree -H "content-type:application/json" -d '{"format":"newick", "maxDepth":"3", "subtreeNodeID":"3534540"}'
17:45 jimallman i think ot10 is ‘devapi’, and ot12 is ‘api’ (not sure)
17:45 josephwb ARG!
17:45 josephwb thanks jim
17:45 josephwb so, it does work?!?
17:46 josephwb i had just copied the curl example from the documentation.
17:46 towodo I think that's what we agreed in VA… devapi = development version of api… and api is in production because of hackathon
17:46 jimallman works for me, using devapi
17:47 jimallman and i’m relieved to see that “format”:”arguson” is working.. it just returned newick in my first curl test
17:47 josephwb me too
17:48 jimallman hm, i think i had misspelled it ‘tformat’. my bad.
17:48 josephwb missing trailing semicolon, tho
17:48 josephwb in newick, i mean
17:48 josephwb i think i can find this.
17:49 josephwb and then update!
17:49 josephwb *all by myself*
17:50 towodo I've been doing updates with  ./push.sh {path to deployed-systems repo}/development/ot10.config treemachine
17:50 mtholder -c in there
17:50 towodo right, -c
17:53 kcranstn joined #opentreeoflife
17:55 josephwb blackrim: it seems treemachine's getNewick never appends a ';'. instead, we manually add this on in many places in the code.
17:55 josephwb rather than just add another fix, should I change getNewick itself?
17:55 josephwb or will that break things?
17:58 towodo let it break things.  our test system will find all the places that need to be update as a consequence
17:58 towodo :)
18:00 blackrim yeah, getNewick can't do it because it is a recursive call (unless there is a different one) so it would need to be the thing that calls get newick that adds it
18:02 josephwb yeah, i seem to recall something like that. thanks.
18:04 jimallman mtholder: is that -c before the component name, you mean?
18:07 mtholder no before the config file
18:08 josephwb done. boom.
18:08 josephwb thanks again guys
18:08 jimallman mtholder: doh! of course
18:13 josephwb snacktavish: did you see this? seems like it could be useful: http://bit.ly/N1I6Ly
18:15 blackrim so study id is not prepended with the pg_ or whatever?
18:15 snacktavish That's what I get for IRC lurking!
18:16 mtholder from where, blackrim
18:16 josephwb ;-)
18:17 blackrim in the nexsons
18:17 blackrim so the name may be pg_2818 but that isn't the studyid, the study id would be 2818 and pg_ is not in the nexson anywhere
18:17 jimallman snacktavish: please don’t leave! we’re never gonna let you down
18:17 jimallman never gonna tell a lie and hurt you
18:17 snacktavish :)
18:18 snacktavish Never gonna web2py and hurt me?
18:18 mtholder are you looking on api.opentree or devapi?
18:18 mtholder blackrim^
18:19 blackrim just looking at the nexson i get from returning a single study http://api.opentreeoflife.org/phylesystem/v1/study/pg_719
18:19 mtholder before this week the pg_ was not in the studyId field, I fixed that on the "real" repo (on ot10). which study are you...
18:19 mtholder api is old stuff. devapi is what the curator is dealing with
18:19 blackrim pg_2818 is what i am looking at
18:20 blackrim arg, github wiki api needs updating cause that is where i got it from
18:20 mtholder http://ot10.opentreeoflife.org/api/v1/study/2818
18:20 mtholder well... that is the public (frozen one)
18:20 blackrim so i don't add pg to the call there?
18:20 mtholder http://ot10.opentreeoflife.org/api/v1/study/pg_2818
18:20 mtholder both work
18:21 mtholder it is an alias. I'd use the prefix if i were you.
18:21 mtholder I'll add a note to the wiki that we have a bleeding edge devapi...
18:22 blackrim ok, thanks
18:29 travis-ci joined #opentreeoflife
18:29 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#558 (master - e13b31f : Mark T. Holder): The build passed.
18:29 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/compare/a5247beed518...e13b31f26de4
18:29 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/27516528
18:29 travis-ci left #opentreeoflife
18:30 mtholder thanks travis-ci so nice to know that my edit of the README did not break the build.
18:31 jimallman REMINDER: you can see all the service base-URLs by clicking “Show/hide server details” in the footer of the curation app (and the tree viewer).
18:32 jimallman i’m planning (right now!) to update ot11.config to use the proper domain names (devapi.opentreeoflife.org versus ot10, etc) to make this more transparent.
18:32 jimallman classic travis!
18:34 mtholder sounds good to me (though towodo was talking about another dev machine for treemachine - would the ot# clearer if we have multiple dev machines?)
18:37 jimallman mtholder: good question. maybe we can use ‘devapi’ and ‘devtree’ for the “blessed” dev instances, and ot# for the more bleeding-edge or special-purpose machines..?
18:37 mtholder that sounds very reasonable
18:39 blackrim mtholder: ok, that fixed things on my end (changing servers)
18:40 blackrim maybe we need a service that returns what servers should be used for what
18:40 blackrim that way before running a synth, i can be sure to pull from the right thing
18:40 mtholder we have the deployed-systems repo we could note things there.
18:41 blackrim seems like it should be an api call though right?
18:41 mtholder GET to GitHub ;-)
18:42 mtholder we could edit a file in a static dir in a web2py instance after deploying. then that URL would be up to date
18:42 towodo jimallman, re server details, would be nice to see what {*_domain} are bound to, or to have the values substituted
18:47 jimallman it’s there, in the “domains” section above
18:47 jimallman towodo: but yes, i could substitute these in place…
19:02 towodo hmm...
19:03 kcranstn joined #opentreeoflife
19:30 mtholder pyramid port is done but untested (https://github.com/mtholder/pyraphyletic for the curious). back to real work...
19:42 jimallman josephwb: i’ve bumped up the pagination for OTU Mapping, from 20 to 1000 at a time.
19:42 jimallman http://devtree.opentreeoflife.org/curator/study/edit/2853
19:42 jimallman that should make bulk mapping much less of a chore, while not bogging down the browser with tens of thousands
19:51 snacktavish joined #opentreeoflife
19:55 towodo great. (Idly wishing the OTU mapping dialog had hyperlinks to taxa in either taxonomy or synthetic tree…)
19:55 towodo I wonder in what order the OTUs are presented.  I guess same order as in the nexson file...
19:56 towodo that order could be meaningless and perhaps there's a more useful sort order
19:56 jimallman see the list filter. options are unmapped OTUs first, mapped OTUs first, fwd and reverse alphabetic sort.
19:56 josephwb jimallman: i'll try it out. thanks.
19:56 mtholder not sure about that (the same order as the nexson file bit). I think that they are sorted by id (which is not visible or intuitive)
19:57 towodo no, i mean within each of those categories
19:57 mtholder sorry I'm late
19:57 jimallman (which are orthogonal choices, i guess, but i was aiming for simplest UI)
19:57 jimallman right, i’m not sure of the “natural” order w/ mapped or unmapped first
19:57 mtholder late with my comments.
19:57 jimallman i could impose a secondary alpha-sort, if this is annoying/confusing
19:58 towodo like, it could be alphabetical, or it could be a traversal of the tree(s)… the latter would be complicated in presence of multiple trees… just thinking aloud
19:58 jimallman hmmm
19:58 mtholder in nexson 1.2.1 we lose the order as they occur in a NEXUS file.
19:58 towodo or taxomachine could assign sequence numbers in the taxonomy, and we could sort by those
19:58 towodo sequence according to a pre- or postorder tree traversal
19:59 mtholder the id's (because they are generated automatically) often have the same sort order as the original taxa list.
19:59 towodo that's a lot of work for a feature no one has ever requested. so don't listen
19:59 mtholder I think sorting by some score of certainty of mapping would be useful.
20:00 mtholder but TNRS returns only a boolean exact field now.
20:01 towodo what happens with closely related homonyms, like cousins or siblings?
20:01 jimallman unrelated, just FYI: i’ve updated the “visible” service URLs (in ot11.config), so the client is using devapi instead of ot10. i suspect devs may trace this traffic to learn the ropes, so this will encourage them to use the right/stable URLs for API calls.
20:01 towodo I guess now that there's a documented api i could just try it out and see...
20:02 towodo sounds good
20:04 kcranstn joined #opentreeoflife
20:08 josephwb jimallman: notice any slowdown?
20:08 josephwb i just mapped 1000 taxa (well, most)
20:09 jimallman i’m not seeing performance changes here.. how about you?
20:11 josephwb seems not as speedy as before, but hard to judge. and of course i didn't time anything, because why would i do something useful like that?
20:12 mtholder did you save? I didn't see the commit
20:12 josephwb not yet. i have several more pages to go through
20:12 josephwb and a lot of clicking checkmarks
20:13 josephwb or "approve all"s
20:13 mtholder you can save at any time (just to make sure you don't lose much work if we have any problems on the save)
20:13 jimallman eek, i just updated webapps… hope that doesn’t screw you up josephwb
20:14 josephwb hmm, hung on "Approve all"
20:14 jimallman JS console message?
20:14 josephwb now: Warning: Unresponsive script
20:14 jimallman ah, OK.
20:14 mtholder how dare they call us unresponsive. we're all over irc
20:14 jimallman basic javascript timeout. try to say Continue and it should finish up.
20:15 jimallman maybe i should back that off to 100 otus at a time. or take a closer look at the JS, i’m sure it’s not optimized for speed.
20:15 josephwb trying
20:15 josephwb stopped again
20:15 josephwb yeah, 100 might be better
20:16 jimallman hm, i wonder if it refreshes the list every time a single otu-mapping is accepted. that might be negligible with 20 items, but painful with 1000
20:16 josephwb ok, keeps stopping
20:16 jimallman keep the faith!
20:16 josephwb my clicker is getting tired
20:16 mtholder switch fingers
20:16 josephwb 8o|
20:17 jimallman http://1.bp.blogspot.com/_lpAazdH2SuA/TPKko6KdJKI/AAAAAAAAAJ4/_pMIlxouFPo/s200/dipping-bird.jpg
20:17 jimallman http://stream1.gifsoup.com/webroot/animatedgifs1/1158223_o.gif
20:18 * jimallman is actually working on a fix for this, not just wasting time...
20:18 mtholder can you pipe "yes" to your browser?
20:18 josephwb man, that episode is good
20:19 josephwb sort of. i checked "Don't ask me again"
20:19 jimallman now go make a sandwich..
20:19 josephwb something is going on, i think
20:19 josephwb sudo make me a sandwich
20:19 jimallman :)
20:20 josephwb this seems like a good test study
20:21 jimallman agreed, which on is it (ID)?
20:21 jimallman i’ve been using 2853, with less than 1000 otus (wimpy!)
20:21 josephwb ot_8
20:21 josephwb ?
20:22 josephwb i can't look at things while it is doing whatever it is doing
20:23 jimallman omg you’re blind!?
20:24 josephwb Q: how are studies indexed when enetered through the curator?
20:24 mtholder read the LICENSE: no warranty...
20:24 josephwb r we starting at 1?
20:24 mtholder wait you can't read anymore...
20:24 josephwb i.e. not trying to keep with PG indexing, right?
20:24 mtholder what do you mean?
20:24 mtholder the study list is sorted by date
20:24 josephwb i mean, the Hackett study is 420
20:25 josephwb in PG
20:25 mtholder there are aliases of 450 to pg_450
20:25 josephwb the latest study in PG is something like 2600
20:25 josephwb a new study in the curator won't be 2601, right?
20:25 jimallman oti re-indexing is triggered by saving (creating, deleting) the study… on push to github repo
20:25 mtholder no
20:25 mtholder it would be ot_# (10 or so)
20:25 josephwb thought so
20:26 jimallman oh, i see what you mean.. yeah, study ids are incrementing for a given “namespace” (prefix)
20:26 mtholder we can't coordinate # with pg. so we prefix
20:26 josephwb sounds good
20:26 josephwb pg_1-2600
20:26 josephwb ot_1-a_gazillion
20:27 mtholder right.
20:27 mtholder we'd better use longs for that.
20:27 josephwb unsigned longs
20:28 mtholder is chuckles out loud a thing?
20:28 jimallman that would constitute project success, i think
20:28 mtholder I'll set up a cron job right now...
20:33 josephwb script finished
20:33 josephwb study saved
20:33 mtholder https://github.com/OpenTreeOfLife/phylesystem-1/commit/ed318962521a4fd91479854e03a750b87f680ff5
20:33 jimallman whew.. sorry about that. i think i see the problem. my JS function mapOTUToTaxon does a lot of UI refresh, so that just happened 1000 times…
20:34 jimallman derp
20:34 kcranstn first real commit to phylesystem from curator!
20:34 kcranstn buy that man a beer
20:34 josephwb boom
20:34 mtholder which man
20:34 mtholder ?
20:34 mtholder oh josephwb. agreed
20:35 josephwb this is the Davis-Page avian supertree
20:35 kcranstn josephwb - he was a patient first tester
20:35 jimallman indeed
20:35 kcranstn although I am sure I will buy mark at least one beer at Evolution
20:35 mtholder woohoo
20:35 josephwb *probably* should not have started with a 6k taxon study...
20:35 kcranstn living on the edge
20:35 mtholder i'm sure I'll drink at least 1 beer at evolution
20:35 jimallman go big or go home
20:36 kcranstn I have faith in mtholder
20:36 * jimallman was encouraging brave curation, not binge drinking
20:36 josephwb jimallman: working on page #2. i do notice refreshing of UI
20:37 josephwb kcranstn: actually have a beer, watching the world cup.
20:37 josephwb in the Smith lab, of course.
20:37 kcranstn did some of that over lunch
20:37 josephwb its how we roll.
20:39 kcranstn ok folks, heading home
20:39 towodo bye
20:42 josephwb getting ~0.1 seconds / name
20:42 josephwb 1000 taxa. should be 1000 seconds, right?
20:43 josephwb 100 (typo)
20:45 josephwb took exactly 11.5 minutes
20:48 mtholder OK. I think that I have a fix to the v1 issue (production branch of web2py version of the phylesystem-api).
20:48 mtholder I'm going to redeploy to ot12
20:49 travis-ci joined #opentreeoflife
20:49 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#559 (fixing-v1 - 1fd56bf : Mark T. Holder): The build passed.
20:49 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/commit/1fd56bfcf81f
20:49 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/27527923
20:49 travis-ci left #opentreeoflife
20:49 travis-ci joined #opentreeoflife
20:49 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#560 (production - 1fd56bf : Mark T. Holder): The build passed.
20:49 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/compare/1392ca66defa...1fd56bfcf81f
20:49 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/27527962
20:49 travis-ci left #opentreeoflife
20:49 josephwb wish i was going to evolution this year
20:49 josephwb this talk looks great: http://tiny.cc/bquehx
20:51 josephwb same session as snacktavish
20:52 towodo twisted
20:53 mtholder so, towodo, jimallman: do we want the branches on ot12 to be "master"? The services there are supposed to be a bit more stable than that, right?
20:55 mtholder I'm not aware of bugs on master (of phylesystem-api), so I'll go ahead...
20:57 towodo hmm. ot12 is api producton. do we need two production branches?
20:59 towodo where api includes phylesystem, oti, taxo tree.
20:59 mtholder It is the only machine with some sort of production api/oti.
20:59 mtholder so we could deploy production branch of api/oti there and master branch of tree/taxomachine
21:00 towodo right. we'd only need another branch if we needed different tree/taxo between ot9 and ot12
21:00 mtholder But I think I need to use the master branch of opentree when doing deployments to ot12, right?
21:00 towodo I don't see why
21:00 mtholder is the proxy pass stuff in the production deployment?
21:01 mtholder i mean the apache proxy for v1 in urls
21:04 mtholder yeah. it looks like it is. I'll deploy the production branch of api to ot12 with the production branch of opentree/deploy...
21:05 jimallman josephwb: help is on the way… i’m streamlining the JS now.
21:06 jimallman re: 11.5 minutes… i’m guessing the actual mapping was fast, right? just the “bulk acceptance” was slow?
21:06 josephwb no, that is the mapping time
21:06 josephwb mapping + ui refresh, amybe?
21:07 josephwb maybe?
21:07 jimallman amybe so
21:08 jimallman i’ll look at the mapping code, as well. 1000 may have been over-reaching, but let’s keep trying
21:09 josephwb sounds good
21:09 mtholder there is a taxomachine call that will take several names, right?
21:09 mtholder we could use that rather that the autoboxcomplete one
21:10 jimallman hmmm, never got around to that
21:10 jimallman when you say “several”, is there a practical limit?
21:10 josephwb cody would know that
21:10 mtholder 1000 might crash it. we'll have to ask cody
21:10 josephwb first
21:11 mtholder second!
21:11 mtholder but even batches of 50 might make a big difference
21:12 josephwb prolly
21:15 jimallman yeah, it’s probably a balancing act between absolute vs. perceived performance.
21:15 jimallman we need to do single-otu mapping in any case
21:20 kcranstn joined #opentreeoflife
21:25 jimallman josephwb: i’ve done some performance improvements for bulk OTU (un)mapping, basically the low-hanging fruit
21:25 jimallman this should radically speed up approving or clearing hundreds of OTUs. i also dropped the page size from 1000 to 500 otus, since there are still some annoyances.
21:25 jimallman mapping time is mostly unchanged, but a smidge faster
21:26 jimallman drat! missed him
21:27 snacktavish I am done lurking for the day, so no further rickrolling is necessary!
21:28 jimallman standing down, then
21:49 kcranstn joined #opentreeoflife
21:59 mtholder towodo and jimallman: check it out with all of the UI flair that you expect from me: http://phylo.bio.ku.edu/status/status.html
21:59 mtholder thanks for the ajax help jim.
21:59 jimallman sweet! looking good (functionally speaking)
22:00 mtholder the v1 redirection inside web2py seems to have fixed things. I took out the apache redirection on ot12
22:00 mtholder the "(NaN)" is supposed to be a time delta. fwiw.
22:00 towodo monday's topic is testing , so this is timely
22:01 mtholder I'm headed home. bye
22:01 towodo good stuf.. bye
22:08 josephwb joined #opentreeoflife
22:27 mtholder joined #opentreeoflife
23:17 travis-ci joined #opentreeoflife
23:17 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#562 (master-fixing-v1 - 48544ce : Mark T. Holder): The build passed.
23:17 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/compare/6e06bdc7c660^...48544ce3df3c
23:17 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/27538296
23:17 travis-ci left #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary