Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-08-14

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:05 jar286 joined #opentreeoflife
01:47 ilbot3 joined #opentreeoflife
01:47 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
11:55 jar286 joined #opentreeoflife
12:14 kcranstn joined #opentreeoflife
14:18 kcranstn joined #opentreeoflife
15:52 kcranstn what is the status of the Download Newick string link on every node? clicking on it at the root seems to do nothing except spin...
15:56 kcranstn and now I can’t load tree.opentreeoflife.org at all
16:00 kcranstn ok, loading now
16:13 jimallman kcranstn: afaik this feature is working, but (as you’ve seen) it might be very slow for deep nodes.
16:14 kcranstn we should probably remove it from nodes where it doens’t work (or is so slow as to not appear to work
16:15 jimallman it looks like something else is going on. i’m trying to fetch this tiny Newick and it’s also too slow: http://devtree.opentreeoflife.org/opentree/opentree3.0@2383448/Candidate-division-WS3
16:15 kcranstn same here for https://tree.opentreeoflife.org/opentree/opentree3.0@3859542/Panthera-tigris--Panthera-gombaszoegensis--
16:20 jimallman here’s the web2py action, which in turn calls the appropriate treemachine service (getDraftTreeForottId, getDraftTreeForNodeID): https://github.com/OpenTreeOfLife/opentree/blob/51a96f3885c5eecc3a14547d6c664a0e714d49b8/webapp/controllers/default.py#L72
16:20 jimallman i’ll boil this down to some typical curl calls we can use to troubleshoot the slowdown
16:20 kcranstn cool
16:22 kcranstn strange that this is not using the call described in the API doc: https://github.com/OpenTreeOfLife/opentree/wiki/Open-Tree-of-Life-APIs#subtree_tree
16:23 jimallman perhaps it should. i doubt anyone has touched the download-Newick feature in a long time.
16:25 kcranstn curl -X POST http://api.opentreeoflife.org/v2/tree_of_life/subtree -H "content-type:application/json" -d '{"ott_id":563159}' works
16:25 kcranstn but not sure that method is limited to a reasonable number of nodes
16:25 kcranstn josephwb - do you know?
16:26 jimallman here’s what we call (downloading Newick) for the tiny subtree ‘Candidate-division-Ws3’:
16:26 jimallman curl -X POST -H 'Content-Type: application/json' https://devapi.opentreeoflife.org/treemachine/v1/getDraftTreeForNodeID -d '{"nodeID":"2383448", "maxepth": 4}'
16:27 jimallman it’s a single node, should return instantly. it seems my request was waiting in line after another long-running request (probably ‘cellular organisms’ or similar)
16:28 kcranstn we should be using the publicized APIs
16:28 kcranstn in the spirit of Eat Thine Own Dog Food
16:28 jimallman to download Newick for ‘cellular organisms’ (i just called this, it’s still chugging)…
16:28 jimallman curl -X POST -H 'Content-Type: application/json' https://devapi.opentreeoflife.org/treemachine/v1/getDraftTreeForNodeID -d '{"nodeID":"1", "maxepth": 4}'
16:29 jimallman agreed on using v2 APIs, but we’re still using some features that are v1-only (can’t recall which services offhand)
16:32 kcranstn when we load the tree, do we know the number of descendents?
16:33 jimallman if you mean the argus representation, i don’t believe so. nor do we get a leaf countl.
16:35 kcranstn ok, so there is no way to know when that call is safe
16:35 jimallman correct. regarding the maximum depth, the v2 method ‘subtree’ instead uses a maximum number of tips (currently 25000) and returns an error if the subtree is larger than that:
16:35 jimallman https://github.com/OpenTreeOfLife/treemachine/blob/2214f2ae435991f968c5ad555a78f42247a7b895/src/main/java/opentree/plugins/tree_of_life.java#L401-L404
16:36 kcranstn ah!
16:36 kcranstn that’s good to know
16:36 kcranstn we should switch
16:36 kcranstn I can do that
16:36 kcranstn sound ok?
16:37 jimallman sounds good. this will touch the private/config file as well, which can be prickly when specifying v2 methods
16:37 kcranstn can we remove the link from the page for the root node and instead put a link to the download page?
16:38 kcranstn is there a way to know when a user is on the root?
16:38 kcranstn since that is the landing page and I fear many people will hit that ‘download tree” link and then hang / get an error
16:38 jimallman kind of, but it might be better to bake these instructions into the error message
16:39 kcranstn I don’t want to provide a link on the landing page that is going to generate an error 100% of the time
16:39 jimallman understandable. perhaps we can measure the length of the path-to-root, and catch it that way.
16:40 jimallman tip count in argus would be ideal… let me double-check, to see if that’s been added.
16:40 kcranstn perhaps in the (very) short term we should change that link to point to the download page
16:40 jimallman yes, as a stopgap.
16:40 jimallman i suspect that its node id will always be 1 (josephwb, is that correct?)
16:49 kcranstn tree_of_life/subtree returns newick
16:49 kcranstn that won’t work, right?
16:52 kcranstn oh, it will because that link specifically says newick
16:52 kcranstn doh
16:54 jimallman TGIF
17:02 mtholder joined #opentreeoflife
17:22 josephwb i'm here
17:22 josephwb curl calls say devapi is not available
17:23 josephwb is neo4j just not running?
17:23 josephwb jimallman ^
17:23 josephwb although it was yesterday...
17:24 kcranstn it was down for a short bit yesterday
17:27 josephwb i am not getting a response from the production api
17:29 kcranstn https://tree.opentreeoflife.org/opentree/argus/opentree3.0@1
17:29 kcranstn loading for me
17:29 josephwb curl -X POST http://api.opentreeoflife.org/treemachine/ext/graph/graphdb/about
17:29 josephwb browser isn't working either
17:30 josephwb browser is trying
17:31 josephwb yeah, i can't get anything to work
17:34 kcranstn browser is loading for me, albeit slowly
17:36 josephwb subtree service works on my local machine (master branch)
17:37 josephwb api responding to curl now. devapi "unavailable"
17:39 josephwb subtree service works on api too, but slooooooow
17:39 kcranstn I am not sure how to troubleshoot this
17:48 * jimallman is back from making a sandwich, reading now...
17:48 josephwb seems fast now
17:48 josephwb still no devapi
17:48 kcranstn the server is up
17:48 kcranstn but calls not working
17:48 kcranstn devapi, that is
17:49 jimallman same, i get a 503 (service unavailable) error when calling treemachine services
17:50 josephwb just restarted neo4j and it works (on devapi)
17:50 josephwb so, what is the issue?
17:51 jimallman it seems like some of these subtree queries are bogging down treemachine badly
17:51 kcranstn not sure why neo4j was down
17:51 kcranstn I am removing that link as we speak, er, type
17:51 jimallman down, or hung?
17:51 kcranstn dunno
17:52 josephwb ok, ping me if you need me. things seem to work phine for me
17:52 kcranstn will do
17:53 jimallman hm, devapi is very snappy now, even for some queries (eg, Archaea) that were failling with just ‘ERROR’ as the response.
17:54 josephwb jimallman you saw that i restarted neo4j, yes?
17:54 josephwb yes, very fast
17:54 jimallman yes, just noticing that we seem to have a creeping slowdown
17:54 josephwb hmm
17:55 kcranstn is that true on dev and production?
17:55 jimallman not sure about production. it’s possible that our dev* instances are provisioned differently.. jar286 will know.
17:56 kcranstn jar286 is on vacation without internet access
17:56 jimallman yes, and yet he’s here in (irc) spirit
17:56 kcranstn I’ve also seen his spirit lurking on google docs ;)
17:57 jimallman if it’s pressing, i can investigate the AWS settings or ssh in and gather some stats on RAM, etc.
17:57 kcranstn devapi is ashby, which is not AWS
17:57 kcranstn it’s at MIT
17:57 jimallman kcranstn: by “removing that link as we speak”, do you mean the download-Newick links in synth-tree viewer?
17:57 kcranstn yes
17:58 jimallman ah, true about devapi/ashby
17:58 kcranstn then we can discuss a fix
18:11 mtholder joined #opentreeoflife
18:19 jimallman josephwb: i’m seeing some weird output from the treemachine on dev, basically missing support information (sourceToMetaMap[ ] values of “null”, instead of the expected object). i’m not seeing this on production (all’s well). are you taking shortcuts in dev treemachine?
18:45 kcranstn just did a dev redeploy but got an error because of local changes to requirements.txt
18:46 kcranstn so the git fetch failed
18:46 kcranstn or git pull
19:05 mtholder kcranstn, what command are you giving to push.sh
19:05 mtholder ?
19:06 mtholder (I don't think that I've run it locally on a server)
19:06 kcranstn ./push.sh -c ../../deployed-systems/development/devtree.config
19:06 kcranstn running locally on my machine
19:07 kcranstn and my deployed-systems is up to date
19:07 mtholder and you have the "opentree" on the end of that push.sh command?
19:09 kcranstn using instructions here: https://github.com/OpenTreeOfLife/opentree/wiki/Suggested-workflow-using-'development'-branch-in-opentree-repo
19:09 kcranstn which have worked for me previously (no ‘opentree’ at the end of that command)
19:10 kcranstn if I log onto ot16, and git status the opentree repo, I see a modified requirements.txt
19:10 mtholder that happens during deployment.
19:10 mtholder see jim's 2nd comment on https://github.com/OpenTreeOfLife/opentree/issues/695#issuecomment-131207799
19:11 mtholder but the deploy script is supposed to call a git_refresh command that gets the repo to a clean state.
19:16 mtholder you might want to add a "set -x" to push.sh and run again to get a verbose listing of what it is doing... It is interesting to me that push.sh ever worked w/o a component listed.
19:17 kcranstn add set -x to the script itself or to the call?
19:18 mtholder I'd try it in my shell, and if that does not give a step by step of the script, then I'd put it in the script (but not commit that change).
19:24 kcranstn well, that’s a lot of output ;)
19:27 josephwb jimallman i'll take a look at that
19:28 pmidford2 joined #opentreeoflife
19:29 pmidford2 Question for jar286 or jimallan - want to be sure I have the right urls for the production and dev repositories before I regenerate history files
19:29 pmidford2 Sorry jimallman
19:31 pmidford2 Is https://github.com/OpenTreeOfLife/phylesystem-1.git the production or dev repository?
19:32 mtholder kcranstn, I'd be looking to see if the git operation was happening outside of the git_refresh function. I don't think that it is supposed to happen outside of that call.
19:34 kcranstn that *** git failure *** warning is coming from the git_refresh function
19:39 mtholder hmm. I guess that "this will need some tweaking..." comment in functions.sh is accurate.
19:41 kcranstn the modification of requirements is supposed to happen after the git_refresh of the opentree repo
19:41 kcranstn but the file is already modified on devtree
19:41 kcranstn so running push.sh is failing
19:42 mtholder but how is is supposed to get reset from a previous deployment?
19:42 mtholder have we had that mod of requirements.txt in the deployment for a while?
19:42 kcranstn https://github.com/OpenTreeOfLife/opentree/commits/17b39d1745beafe16f54d14f8bcc68af6d2a4558/deploy/setup/install-web2py-apps.sh
19:43 jimallman mtholder, the git failure is a known (and long-standing) problem. lately i’ve been using ssh and clobbering unsaved changes on the server with ‘git reset —hard’
19:43 mtholder Ah, I didn't realize that you were doing that. should've read the issue more carefully...
19:44 mtholder I guess that is your answer, kcranstn
19:44 kcranstn if the git fails, the whole script should stop. No point going through all of the deployment if the repo changes aren’t there
19:44 kcranstn the errors happens right at the top and then gets scrolled up with many more output lines
19:45 mtholder agreed. it looks like it just echos and moves on, which is not cool.
19:48 kcranstn I can’t even remember what I was doing when I went down this rabbit hole...
19:50 jimallman kcranstn: shall i reset the repo? were you trying to push to dev or production?
19:50 kcranstn dev
19:50 kcranstn testing that change to the newick download link
19:51 jimallman gotcha. i’ll add the incantation to the issue comments above
19:52 jimallman (it’s reset now, should be able to push to devtree)
19:53 kcranstn thanks
19:53 kcranstn https://github.com/OpenTreeOfLife/opentree/issues/702
19:54 jimallman pmidford2: sorry for my slow reply. you can check for the active phylesystem in the footer of the curation app
19:55 jimallman click ‘Show/hide server details’ and look for the phylesystem shard(s) list at the top.
19:55 pmidford2 jimallman - thanks
19:55 jimallman current production is phylesystem-1
19:55 kcranstn jimallman - push.sh worked but not using more recent commits of development branch
19:55 jimallman and current dev uses phylesystem-0
19:56 jimallman kcranstn: ? meaning it fails when pushing from latest development?
19:57 kcranstn no warnings or failures in script but I don’t see the changes reflected in browser
19:57 kcranstn https://devtree.opentreeoflife.org/opentree/argus/opentree3.0@1
19:58 kcranstn https://github.com/OpenTreeOfLife/opentree/blob/6d45918fdefeacbed026a307efb5a3e86cf00397/webapp/static/js/treeview.js#L1128
19:58 jimallman hm, taking a look now…
19:59 jimallman works for me (have you reset your /etc/hosts file?)
20:00 kcranstn refresh page (argh)
20:00 kcranstn ok, all good now
20:00 jimallman :D
20:01 kcranstn good thing happy hour starts in 30 minutes
20:01 jimallman Do you think we need to specify Newick in the link? or is that sort of obvious?
20:01 jimallman oh, the download page spells it out
20:04 kcranstn damn - I forked that freakin branch off dev, not master
20:04 kcranstn arrrrrrrrgh
20:06 kcranstn off to rebase
20:06 kcranstn or something
20:09 jimallman yep, that’s bad
20:09 jimallman if it’s a single commit, the fix is easy
20:09 kcranstn two commits
20:09 jimallman still pretty easy. start over from master (git checkout master, git pull)
20:10 jimallman then create the new branch (git branch my-new-thing), and cherry-pick the commits you want
20:10 jimallman git cherry-pick SHA1..SHA2
20:11 jimallman note: SHA1 should be the older commit
20:13 jimallman you can get the commit SHAs using git log
20:16 jimallman looks like this would be ‘git cherry-pick 5f77a2e..d18449d’
20:16 jimallman kcranstn: ^
20:16 jimallman i believe this will work with mini-SHAs above. if not, try this instead:
20:17 jimallman git cherry-pick 5f77a2ebb9803433037e592090c0df2425493524..d18449d159188cbf5efaeffddfd2c8716803f778
20:19 kcranstn could not apply 2nd commit
20:19 kcranstn sigh
20:19 kcranstn that one was minor
20:19 kcranstn just redoing manually
20:20 * jimallman nods
21:17 kcranstn joined #opentreeoflife
22:53 mtholder joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary