Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2013-12-06

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
02:46 jimallman joined #otol
13:15 towodo joined #otol
15:20 jimallman joined #otol
16:28 towodo joined #otol
16:58 towodo joined #otol
17:26 towodo joined #otol
19:16 dukeleto joined #otol
19:55 dukeleto towodo: ping
19:55 towodo ping.
19:56 dukeleto towodo: i would like to merge my "local" branch soon and want to coordinate with you about changes needed in deployment
19:57 dukeleto towodo: https://trello.com/c/nj3kgqHm/210-merge-local-branch-of-api-opentreeoflife-org-and-teach-deployment-script-about-necessary-settings
19:58 towodo ok
19:58 towodo there is also a private key for logging in to the ec2 hosts, we can manage the other key similarly
19:59 towodo have you looked at the script?
19:59 dukeleto towodo: i have been reading through all the deploy scripts
19:59 towodo I hope it doesn't look complicated
20:00 dukeleto towodo: trying to decide where the GIT_SSH and PKEY env vars should be set
20:00 towodo in the config file?
20:01 towodo can put the keys in separate files, or directly in the config file.  currently the ssh private key is in its own file
20:02 towodo if you need it in an env var, well, a command can do it after the config file is sourced
20:02 towodo I've never made something like this, making it up as I go along
20:02 dukeleto towodo:  :)
20:03 dukeleto towodo: the GIT_SSH and PKEY env vars need to be present in the process environment that web2py runs in
20:03 towodo and the values are the keys themselves?
20:04 dukeleto towodo: no, they are the locations of the files
20:04 towodo oh, so that's easy.
20:04 towodo there's a choice
20:04 dukeleto towodo: GIT_SSH=/../api.opentreeoflife.org/bin/git.sh
20:05 towodo either you say GIT_SSH=… in the config file, with adefault setting in the surrounding code , … (see opentree.pem for pattern)
20:05 towodo or else invent new config variable with name parallel to the others, OPENTREE_GIT_SSH= ...
20:05 towodo and then do assignments to move them over
20:06 towodo I am uncertain how to name these config variables… lower case might be better… but the momentum is for all caps
20:06 towodo neo4j in its config files use  org.neo4j.foo.bar.baz=12345  which seems excessive
20:07 towodo I don't think it matters a whole lot… we can always upgrade compatibly later
20:07 towodo if we change our minds
20:07 dukeleto towodo: i don't believe i currently have access to the ec2 instance or opentree.pem
20:08 dukeleto towodo: i have not been actively involved in the deployment process until now
20:08 towodo ah… that can be arranged
20:08 towodo there is a server free to experiment with
20:08 towodo currently all have the same private key
20:08 dukeleto towodo: this is the git.sh file, fyi: https://github.com/OpenTreeOfLife/api.opentreeoflife.org/blob/master/bin/git.sh
20:09 towodo do you have an account on norbert?  (the server formerly known as 'dev.opentreeoflife.org')
20:10 towodo hmm… I don't see any reason to ever use the default keyfile for scripted setup
20:11 dukeleto towodo: i agree. That pass through is mostly for the case of when GIT_SSH is defined but PKEY is not. Do you want me to remove that?
20:12 dukeleto towodo: i believe i do have an account on norbert
20:13 towodo I would just do ssh -i… like other parts of the script do … but it's really up to you
20:14 towodo if you have a PGP key I can send you the .pem in email, otherwise I can put it in your home dir on norbert (abusing my root privs)
20:14 dukeleto towodo: the git.sh is required because you can't provide an -i to the "git" command-line
20:15 towodo oh I see, I didn't get that. fine, proceed
20:15 dukeleto towodo: in my home dir is fine
20:15 dukeleto towodo: what is the full hostname of norbert?
20:28 dukeleto towodo: also, i don't see the deploy script doing anything with the private/config of the api repo, to set various config params. Has that been a manual process?
20:28 towodo norbert.csail.mit.edu, maybe I didn't set you up?  or you can use files.opentreeoflife.org
20:29 towodo hang on I would have to look.
20:30 dukeleto towodo: just figured out that files.opentreeoflife.org points to norbert and just logged in
20:31 towodo haven't copied the file yet
20:32 towodo ok, look for opentree.pem in your home dir
20:34 towodo I haven't a clue about private/config … that may be a TBD. Jim did the api setup in the script
20:35 dukeleto towodo: i see opentree.pem
20:35 towodo host name: ...
20:35 towodo ec2-54-202-160-175.us-west-2.compute.amazonaws.com
20:35 towodo also known informally as 'ot1'
20:36 towodo it's an ubuntu server (set up before we switched to debian)
20:38 dukeleto towodo: i see that pull-changes-and-restart-opentree.sh is what actually stops and starts web2py on norbert
20:38 towodo no
20:38 towodo don't look at norbert, that was only for key transfer
20:38 towodo you'll debug on AWS
20:39 towodo norbert is just a random server at MIT left over from another project
20:39 dukeleto towodo: ok
20:39 towodo you want to run push.sh on your own computer, and set up config file that points to the ec2 host whose DNS I gave you above
20:40 jimallman hi guys. i saw the mention of private/config above, not sure of the context... can i help?
20:41 dukeleto jimallman: wondering how private/config in the API repo gets populated on deployment
20:41 dukeleto jimallman: it is what has the github api oauth details as well as some new config keys for finding OTI and where the treenexus git repo is located
20:42 dukeleto towodo: ok, i just successfully logged into ot1
20:42 jimallman in the past, these were done manually.. but i believe towodo's server-side script now makes some changes,
20:42 jimallman ...lemme see...
20:42 towodo ok. but you generally don't need to log into ot1
20:42 towodo run push.sh locally and *it* logs into ot1
20:45 jimallman here's an example (should work with current push.sh)
20:45 jimallman https://gist.github.com/jimallman/43d58216eef9122628aa
20:46 towodo I recommend using a config file (new feature), see documentation
20:46 towodo but that command works too
20:47 towodo oh… you need the admin user to be 'ubuntu' for ot1
20:47 towodo -u
20:47 jimallman oops, my bad. yeah, time for server-specific config files!  ;-)
20:48 jimallman yeah, that was the wrong server IP too.. adjust accordingly.
20:48 jimallman hostname, i mean.
20:49 dukeleto towodo: can i use the default neo4j host in my testing? Will that mess up anybody elses workflow?
20:49 * jimallman will stop trying to help now...
20:50 towodo it won't mess anyone up.
20:50 dukeleto jimallman: no worries. Your gist was useful
20:51 towodo I think that would be ot2 = ec2-54-212-192-235.us-west-2.compute.amazonaws.com
20:51 jimallman thanks. jonathan's script was a head game for me at first.. it does a lot!
20:51 jimallman and definitely trumps my old 'pull-changes-and-restart' script
20:51 towodo I looked and it doesn't seem to set up private/config for the api
20:51 dukeleto yes, i am trying grasp everything that is going on in there
20:54 towodo you should be able to run push.sh - try it to get a feel for how it works
20:55 dukeleto towodo: running it now and it is doing various things :)
20:56 dukeleto towodo: i am still not understanding something. What actually stops and starts things like web2py after the deploy script runs?
20:56 towodo there's no web2py process.
20:56 towodo actually there is, sorry.  it's started by the apache/wsgi magic
20:57 towodo all taken care of by apache
20:57 towodo apache gets restarted by push.sh I think (privileged operation)
20:59 dukeleto push.sh got pretty far along, but errored here:
20:59 dukeleto attempting to recompile treemachine plugins
20:59 dukeleto Error: JAVA_HOME is not defined correctly.
20:59 dukeleto We cannot execute /usr/lib/jvm/java-6-openjdk-amd64/bin/java
20:59 dukeleto probably a difference of ubuntu vs. debian
20:59 towodo !!   there shouldn't be any java-6, I worked hard to get that right.  ouch
21:00 towodo your opentree repo is up to date?
21:00 dukeleto towodo: as-admin line 100 has export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
21:01 towodo JAVA_HOME is also set in functions.sh
21:01 towodo It printed a line "JAVA_HOME is …", what did that say?
21:02 dukeleto Fri Dec  6 20:56:52 UTC 2013 Installing treemachine and taxomachine
21:03 dukeleto JAVA_HOME is /usr/lib/jvm/java-6-openjdk-amd64
21:05 towodo this doesn't make any sense.
21:06 towodo JAVA_HOME is not set by default
21:07 towodo hmm… yes probably an ubuntu/debian difference
21:07 towodo java comes preinstalled on ubuntu I think… and it's java 6
21:07 towodo installing java 7 doesn't override the setting in /etc/alternatives
21:08 towodo so we'll have to kludge it
21:08 towodo can you edit functions.sh?  add "if false; then … else javalink=/usr/lib/jvm/java-7-openjdk-amd64; fi"
21:09 towodo where … = that series of four javalink= commands at the top
21:09 towodo that's what it used to do, before I tried to make it more robust to version changes
21:10 towodo so the earlier version of functions.sh worked on ot1 and o2… and both versions work on ot3 and ot4
21:10 towodo sorry about that
21:12 dukeleto towodo: i completely understand
21:13 dukeleto towodo: that kludge worked and now it is indexing nexsons and doing neo4j stuff
21:13 towodo it's not supposed to index nexsons. that's supposed to be a separate operation. i'll have to check cody's work
21:14 dukeleto which throws   "exception" : "NoClassDefFoundError",
21:14 dukeleto towodo: it still restarted apache though
21:15 towodo shouldn't have, we're doing set -e
21:16 towodo must be java not returning non-0 exit status on error
21:16 towodo I don't see how that code is being invoked
21:17 towodo the script sets up the neo4j instances even if they're not going to be used.
21:17 towodo that ought to be controllable through the config file. i haven't gotten around to it
21:18 dukeleto towodo: the indexing is done by a python script and I that script always returns successful
21:19 towodo I don't see it
21:19 dukeleto towodo: https://github.com/OpenTreeOfLife/oti/blob/master/index_current_repo.py
21:19 towodo in the version i see it's under an 'if false'
21:20 mtholder joined #otol
21:20 towodo funny that it's referencing 7474… oti is on 7478
21:21 dukeleto towodo: https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/install-neo4j-apps.sh#L100
21:21 dukeleto towodo: says "if true"
21:21 towodo ok i have the wrong version
21:21 towodo need to pull
21:22 towodo I say change true to false and email cody
21:25 dukeleto towodo: since there is no web2py process to inject an env var into, perhaps GIT_SSH and PKEY should be set in the .bashrc of the opentree user?
21:26 dukeleto towodo: i see a call to "source /home/opentree/setup/activate" in the current opentree .bashrc which sets some environment variables
21:27 towodo let me think
21:27 dukeleto towodo: but on second thought, the "wsgi magic" doesn't run the web2py code as the opentree user
21:28 towodo it doesn't?...
21:28 towodo it does. see apache-config
21:28 dukeleto towodo: i am not sure, actually. I need to read the apache configs again
21:29 towodo i don't know whether .bashrc gets run.
21:29 towodo that's why the activate is kludged into wsgihandler.py
21:29 towodo as well as the .bashrc
21:29 towodo it could be redundant i suppose, not sure
21:30 towodo hmm… i think apache may fire up python directly, no .bashrc
21:31 towodo I don't understand how that works (see fragment.tmp), just did voodoo I got off the web
21:35 towodo looks like any env that's needed is set up in python code… venv/bin/activate_this.py … I think I'll leave it to you to figure out…
21:43 dukeleto towodo: do you want me to turn off oti indexing only on my branch or on the master branch?
21:44 towodo well, it doesn't work, so i'd say master, if you plan to check in to master…
21:44 towodo would you like me to email cody? (could create an issue but that seems heavyweight with things in such flux)
21:46 dukeleto towodo: sure, if you would email cody, that would be great. You are the czar of deployment :)
21:46 dukeleto towodo: i just created this issue about the indexing script needing to return a useful exit code: https://github.com/OpenTreeOfLife/oti/issues/3
21:50 towodo email sent
21:51 dukeleto towodo: just saw, thanks
21:54 towodo dukeleto: going home now. ttyl
21:54 dukeleto towodo: have a good weekend!
22:41 travis-ci joined #otol
22:41 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#210 (local - 99944ad : Jonathan "Duke" Leto): The build passed.
22:41 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/162499d3ea6d...99944ad8f0c7
22:41 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/15065911
22:41 travis-ci left #otol

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary