Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-09-26

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:15 towodo joined #opentreeoflife
01:04 josephwb joined #opentreeoflife
03:32 kcranstn joined #opentreeoflife
07:46 mtholder joined #opentreeoflife
11:19 josephwb joined #opentreeoflife
11:48 josephwb joined #opentreeoflife
12:57 kcranstn joined #opentreeoflife
13:04 josephwb joined #opentreeoflife
13:14 towodo joined #opentreeoflife
13:36 josephwb towodo: i cannot recreate your asterales problems: https://github.com/OpenTreeOfLife/treemachine/issues/138#issuecomment-56910506
13:37 towodo what did you do to try?
13:37 josephwb link ^
13:37 josephwb i did the 2 curl calls you tried
13:38 josephwb i mean, i built it locally, and it works locally
13:39 towodo My treemachine was built using a different version of the taxonomy, but I don’t see how that could do it
13:39 josephwb no, that should be fine
13:39 josephwb is the server properties for http://aster.opentreeoflife.org set right?
13:39 towodo what do you mean?
13:39 josephwb *is checking something*
13:40 josephwb the neo4j server properties. you need to tell it where the DB is
13:40 josephwb in: conf/neo4j-server.properties
13:40 towodo I used the deployment system. The db goes in the usual place, in graph.db, which is the default.
13:41 towodo I don’t understand where "taxonomy_version" : "ottv2.6" comes from in your output. makes no sense.
13:41 josephwb ok, i just put an invalid DB path in my conf/neo4j-server.properties, and i get the same error as you
13:42 towodo checking
13:42 josephwb "makes no sense"?
13:42 towodo that’s not the version of the aster taxonomy.  there is no version file for the aster taxonomy
13:43 towodo to install db I did ./push.sh -c ../../deployed-systems/production/api.config install-db downloads/treemachine.db.tgz treemachine
13:43 josephwb it is set in the file run_asterales_example.py
13:43 josephwb line 33
13:44 towodo that doesn’t mean it makes sense… if you need a version you could use the write date of the taxonomy.tsv file… but this doesn’t matter
13:44 josephwb it is a required argument
13:44 josephwb when you initialize the DB
13:45 josephwb before it was hardcoded as "taxonomy", which wasn't useful
13:45 kcranstn isn’t the aster taxonomy a subset of OTT? and therefore derives from the same version?
13:45 towodo sorry ignore that push.sh above
13:45 josephwb kcranstn: yes
13:45 towodo no, it’s not a subset, it’s independently synthesized
13:45 kcranstn why?
13:46 josephwb oh
13:46 towodo it’s and end to end test, needs to exercise as much of the system as possible
13:46 josephwb well, then put something else in the "ottv" argument
13:46 kcranstn so it runs smasher. I see.
13:46 towodo it uses the same ott ids, but the taxonomy could be different
13:46 towodo yes it has its own smasher script
13:47 josephwb ottv="test_taxonomy"
13:47 kcranstn where is the code for asterales system?
13:47 towodo in reference-taxonomy/smasher/t/
13:48 josephwb there is a function to read in an OTT distribution, which grabs the version from the "version.txt" file
13:48 kcranstn no, all of the documentation for installing the asterales system
13:48 towodo germinator repo wiki
13:48 towodo ok, josephwb, I’ll assume I didn’t set up the graph.db properly
13:49 josephwb should be an easy fix then
13:50 towodo graph.db is 85M
13:50 towodo org.neo4j.server.database.location=data/graph.db
13:50 josephwb maybe put the full path?
13:51 josephwb mine:
13:51 towodo no, that’s what it always is. has worked fine since time immemorial
13:51 josephwb org.neo4j.server.database.location=/Users/josephwb/Projects/OToL/gcmdr/example/asterales_synth.db
13:51 josephwb even on the asterales server?
13:51 towodo it’s the default value from the neo4j distribution
13:51 towodo yes, the deployment system *always* puts the db in data/graph.db
13:51 josephwb do you have multiple neo4j versions on that machine?
13:52 towodo yes, three, just like on ot10 and ot20
13:52 towodo hang on. no, not multiple versions.
13:52 josephwb hmm.
13:52 josephwb it seems clear that it is not finding the DB
13:52 towodo ot17 is practically a clone of ot10. set up in exactly the same way
13:52 towodo the only difference is that it’s asterales instead of ott
13:54 josephwb i don't know what to say then
13:55 towodo trying push-db again
13:56 towodo no change.
13:56 towodo will try re-creating the treemachine db.
13:56 josephwb i can send you mine if it would help.
13:57 towodo no, the point is that I need a process that’s reproducible
13:57 josephwb so, the db is called "asterales_synth.db"
13:57 josephwb do you copy it and change the name?
13:57 towodo doing python run_asterales_example.py now
13:57 josephwb i.e. it is not "graph.db"
13:58 towodo remember that the db always gets renamed to ‘.’ if one follows the deployment instructions
13:58 towodo which perhaps I didn't
13:58 josephwb ok, i thought you were building it on the server
13:59 josephwb *i think i understand the workflow now*
13:59 towodo no, I’m trying to replicate the usual dev/prod process to the extent possible, since my purpose is to set up a staging procedure
13:59 josephwb yes, okay
13:59 josephwb then, beats me.
14:00 towodo reading documentation…
14:00 josephwb when you query the DB locally, does it work?
14:01 towodo no
14:01 josephwb it doesn't work?
14:01 josephwb it works for me and cody
14:01 josephwb we even use different versions of neo4j
14:02 josephwb does it produce a synthesis newick tree?
14:02 towodo yes
14:02 towodo I wonder if the error messages can be improved.
14:02 josephwb almost certainly
14:03 josephwb wait, which error messages?
14:03 josephwb do you get any when it is synthesizing?
14:03 josephwb or you mean service error messages?
14:03 towodo when I did those POSTs it didn’t occur to me that a missing db might be the problem.
14:03 josephwb i agree, if it cannot find the DB, it should say so, instead of saying "no synthesis tree"
14:04 towodo I guess the deployment system could check for this.
14:04 josephwb i can add that check, but it is a very odd when
14:04 josephwb odd one
14:04 towodo ok, that was the problem.  I skipped the db deployment step.  need to check my own instructions.
14:05 josephwb i only put checking for the absence of a synthetic tree in as a local test, not meant to be useful
14:05 josephwb e.g. querying a DB that has not yet undergone synthesis
14:06 towodo I’m doing this a lot - failing to find my own bugs - and I’m sorry to bug you all
14:07 towodo http://ot17.opentreeoflife.org/opentree/argus/otol.draft.22@28
14:08 josephwb wee!
14:08 josephwb glad you got it working
14:08 kcranstn yay!
14:08 kcranstn towodo - I am happy to try and replicate this and update docs
14:10 towodo yes that might be a good idea… you could try it on a smaller server this time, I think it should work. or on a vagrant vm if you’re adventurous
14:11 towodo I’m looking at my shell transcript and instructions and can’t figure out what I did wrong…
14:11 kcranstn how much space / memory required?
14:11 towodo 4G disk being used now
14:11 josephwb towodo: do you want error message added to all services if the DB cannot be found?
14:12 towodo not sure… I am thinking maybe doing it in the deployment system and not bothering with plugin checks
14:12 josephwb ack. wait.
14:12 josephwb if you specify an invalid DB, it *creates* it!
14:13 towodo kcranstn, ‘top’ says 385M memory used
14:13 towodo asterales is pretty small
14:13 towodo plus 1.7G swap … not sure where that comes from
14:14 towodo I guess the oti index
14:14 kcranstn interesting...
14:14 kcranstn might try and install on iplant atmosphere
14:15 towodo that would be cool.  if it’s a different os then dependencies setup (as-admin.sh) might need tweaking
14:16 kcranstn what os are you using now?
14:16 towodo debian 7.x (7.5 I think)
14:16 towodo ubuntu would work just as well
14:16 kcranstn that is an option (ubuntu)
14:17 towodo josephwb, yes, I know, neo4j is trying to be helpful.  that’s why the check should be done in deployment
14:18 kcranstn their “tiny” is 4GB memory and 30 GB disk. That should work, right?
14:18 towodo I think so, yes
14:19 kcranstn I should probably not start this now. It will eat up my whole day, and I don’t have a day to give today ;)
14:24 kcranstn ok, I got sucked in. Helps having an iplant sysadmin close by
14:26 towodo checking the db setup instructions now.
14:27 towodo ahh… found the bug… and extraneous space
14:27 towodo in my own instructions
14:28 josephwb joined #opentreeoflife
14:31 towodo s/and/an/
14:45 kcranstn is this really the right order of commands?
14:45 kcranstn git clone git@github.com:OpenTreeOfLife/reference-taxonomy.git
14:45 kcranstn export TARDIR=$PWD
14:45 kcranstn (cd reference-taxonomy; make aster-tarball)
14:45 kcranstn TARDIR = the dir *above* reference-taxonomy?
14:46 kcranstn and why are some of the commands in parentheses?
14:47 towodo back
14:47 towodo parens make the ‘cd’ scoped
14:48 kcranstn Q about where I should execute export TARDIR=$PWD?
14:48 towodo the instructions put files in the same directory as the repo… I guess that’s not very clean
14:48 kcranstn so, what is the correct order of commands?
14:49 towodo as documented, I believe
14:49 towodo I guess there could be more explanation…
14:49 kcranstn I will be adding some :)
14:49 kcranstn if you followed those explcitly, you would be cloning repos inside other repos
14:50 towodo really? I don’t think so… let me check
14:50 towodo I see one bug, a popd without a matching pushd
14:51 kcranstn mkdir: invalid option -- 'c'
14:51 towodo but other that ‘cd gcmdr’ all the cds are in parens and therefore temporary to the one command
14:51 kcranstn Try `mkdir --help' for more information.
14:51 kcranstn make: *** [aster-tarball] Error 1
14:51 kcranstn I didn’t get that
14:51 towodo the ‘cd gcmdr’ ought to be ‘pushd gcmdr’
14:51 kcranstn i.e. I didn’t understand the convention of “all the cds are in parens and therefore temporary to the one command”
14:51 towodo it’s a standard shell idiom… an insider joke as it were
14:52 kcranstn let’s not put those in our documentation
14:52 kcranstn ok, now getting make errors when building the asterales taxonomy
14:52 towodo ‘those’ = ?
14:52 kcranstn insider jokes
14:53 towodo but (cd … ; …) is perfectly good I think.  if explained it would be educational. I do it all the time
14:53 towodo otherwise you need a second cd to get back out… error prone
14:54 towodo do you think the .tgz files should be put someplace else? like in the respective repos?
14:54 kcranstn ok, there seems to be a problem with $TARDIR
14:55 kcranstn I have it set to /home/cranston
14:55 towodo I’m thinking we should discourage people from making their own asterales taxonomy… requires ncbi and gbif downloads, sort of a big deal
14:56 kcranstn ah, but I see that the Makefile overrides that export command
14:56 towodo darnit.
14:56 kcranstn and sets it to ./tarballs
14:57 kcranstn which doesn’t exist
14:57 kcranstn and the mkdir is failing
14:57 towodo right. I think there’s a gnu make thing that allows defaulting
14:57 towodo ?= or something like that
14:57 towodo checking manual
14:58 kcranstn what is your intention about where that tarballs directory should be?
14:58 towodo There is another assignment operator for variables, ‘?=’. This is called a conditional variable assignment operator, because it only has an effect if the variable is not yet defined.
14:59 towodo I have no intention really.  originally it was the files.opentreeoflife.org/ott/ directory
14:59 kcranstn I don’t understand why I am getting this “mkdir: invalid option -- ‘c’” error
15:00 kcranstn I only see mkdir -p
15:00 towodo if you like you can change Makefile TARBALLS=x to TARBALLS?=x
15:01 kcranstn still (cd t/tax && \
15:01 kcranstn mkdir -p /home/cranston $ \
15:01 kcranstn tar -czvf /home/cranston/aster.tgz.tmp aster && \
15:01 kcranstn mv /home/cranston/aster.tgz.tmp /home/cranston/aster.tgz )
15:01 kcranstn mkdir: invalid option -- 'c'
15:01 kcranstn Try `mkdir --help' for more information.
15:01 kcranstn make: *** [aster-tarball] Error 1
15:01 kcranstn I don’t see a -c option, though
15:01 towodo neither do I...
15:03 towodo I have some uncommitted changes to the reference-taxonomy repo. pushing them out
15:05 kcranstn hmm… if I run that set of commands manually, all is well
15:05 kcranstn odd
15:05 kcranstn and run-tests.sh is failing
15:06 towodo I didn’t know there was a run-tests.sh
15:06 towodo doing ‘make-test’ on my mac in the ‘fix-add’ branch
15:07 towodo works fine
15:07 towodo you are running on a mac?
15:07 kcranstn unbuntu
15:08 towodo oops, I’ll change the TARBALLS as well in that branch
15:08 kcranstn ok, moving on. Have aster.tgz
15:10 towodo you may want to make a new aster.tgz, the fixes on the branch are important
15:11 towodo you kind of surprised me here. when I did the ot17 setup I did find a few things I needed to fix, fixed them but didn’t push
15:11 towodo so pushing now
15:13 kcranstn and treemachine database build fails...
15:14 kcranstn should I wait
15:14 towodo https://github.com/OpenTreeOfLife/reference-taxonomy/pull/101
15:14 towodo right.
15:15 towodo would you like to merge or should I? - Destry rides again
15:17 mtholder joined #opentreeoflife
15:18 kcranstn I merged
15:18 kcranstn what is the error with that popd command in the treemachine setup?
15:18 kcranstn almost there...
15:18 towodo the preceding cd should have been a pushd.
15:19 towodo are you editing the instructions? if not i’ll fix that
15:19 kcranstn no - keeping notes separately. Please do edit
15:19 josephwb joined #opentreeoflife
15:19 kcranstn make aster_tarball now passes with that last PR
15:20 kcranstn let me know when I should refresh
15:21 towodo ok now
15:22 josephwb joined #opentreeoflife
15:23 kcranstn cranston@vm64-124:~/gcmdr$ popd
15:23 kcranstn -bash: popd: directory stack empty
15:24 kcranstn ah, crap.
15:24 kcranstn just saw the missing pushd
15:24 kcranstn but already past that
15:25 towodo cd .. does the same thing
15:27 kcranstn you learn something new every day. I never use pushd / popd
15:27 josephwb joined #opentreeoflife
15:28 towodo this whole thing could be scripted…  that should be done in conjunction with the vagrant scripting
15:34 kcranstn ok, when I get to “Choose or provision a machine for deployment” the documentation becomes fuzzy. Not sure what to do next
15:35 kcranstn e.g. “Edit the config file”
15:35 kcranstn which is where?
15:38 towodo looking
15:39 towodo will rework that paragraph
15:39 kcranstn you are right about scripting the whole thing, though
15:40 jimallman re: vagrant scripting… last night i went to an intro + Q&A on Ansible (local TriPython meeting). I knew it was good stuff, but it has some great features that might make it a good fit for Open Tree (agentless, encrypted data into repos, cross-distro service mgt, tons of existing playbooks and modules).
15:41 kcranstn hey, jimallman!
15:41 jimallman howdy
15:41 jimallman heading your way soon…
15:41 kcranstn oh, are you coming over?
15:41 kcranstn excellent
15:42 kcranstn I am still at home…
15:42 jimallman oh, remote is OK too.
15:42 kcranstn no, planning to go in
15:42 kcranstn it would be good to see you in person!
15:43 jimallman yes! if you have a little time afterward, i’d like to touch base on FCDB and general stuff.
15:43 kcranstn I have a phone call at 2, but that should be short, and I am free afterwards
15:44 towodo ok, kc, try that paragraph again
15:44 towodo oops I need to change ot17.confiig to my.config
15:44 jimallman thanks! i love loitering at NESCent.
15:46 towodo ok, it was only used in one place (the alias).
15:47 kcranstn OPENTREE_API_HOST=aster.opentreeoflife.org ?
15:47 towodo no, it should be the name of whatever machine you’re installing to
15:47 kcranstn ok
15:47 towodo something.iplant.something
15:47 kcranstn so probably the same as OPENTREE_HOST
15:47 kcranstn ?
15:48 kcranstn since everything is on one machine/
15:48 kcranstn ?
15:48 towodo yes
15:49 kcranstn I have opentree.pem but not opentreeapi-gh.pem
15:50 josephwb joined #opentreeoflife
15:50 towodo it should be possible to finish the install without it.  if not that’s a bug.
15:50 kcranstn ok
15:51 towodo you can copy opentreeapi-gh.pem from ot17 or ot10. I think it’s called ~/.ssh/opentree which is a terrible name but we haven’t gotten around to fixing it yet
15:52 kcranstn where should these be executed?
15:52 kcranstn https://github.com/OpenTreeOfLife/germinator/wiki/Setting-up-the-asterales-system#deploy-treemachine-and-taxomachine-neo4j-databases
15:52 kcranstn oops
15:52 kcranstn yes, those
15:52 kcranstn in cd opentree/deploy/ ?
15:52 towodo yes, working directory has to be opentree/deploy/   (lazy programming)
15:52 kcranstn ok
15:54 jar_ joined #opentreeoflife
15:54 kcranstn deploy is asking for an admin password
15:54 kcranstn cranston@vm64-124:~/opentree/deploy$ deploy
15:54 kcranstn host=128.196.64.124, admin=admin, pem=/home/cranston/.ssh/opentree/opentree.pem, controller=cranston, command=
15:54 kcranstn Syncing
15:54 kcranstn admin@128.196.64.124's password:
15:54 kcranstn Connection closed by 128.196.64.124
15:54 kcranstn lost connection
15:54 towodo ahh… that should be in the instructions.  everything is done with ssh public key authentication
15:55 towodo so you need to use the private key for that host, instead of opentree.pem
15:55 towodo did you get ssh keys when you were given that machine?
15:55 kcranstn pem=/home/cranston/.ssh/id_rsa ?
15:56 towodo how did you log in to the machine - password or key?
15:56 kcranstn no, I don’t have ssh keys for the hose
15:56 kcranstn password
15:56 kcranstn but I have put my key on the host
15:58 towodo it needs to be in authorized_keys for the ubuntu account (or admin account). …
15:58 towodo what account have you been using for all this setup?
15:58 kcranstn my user
15:59 towodo did they give you admin access?
15:59 towodo or sudo?
15:59 towodo because the assumption is you’ll be able to do sudo on some account (e.g. admin or ubuntu)
15:59 kcranstn I have sudo access
16:01 kcranstn and my public key is in authorized_kays
16:01 kcranstn keys
16:01 towodo ok. so just add OPENTREE_ADMIN=kc to the config file, where kc is your use name  (checking this)
16:02 towodo yes, that should do it (this is documented in sample.config)
16:02 kcranstn but not in https://github.com/OpenTreeOfLife/deployed-systems/blob/master/asterales/ot17.config
16:02 kcranstn which is the sample file linked from the instructions
16:02 towodo you made your own copy of that config file, right? and you edited the host name? that copy.
16:03 josephwb joined #opentreeoflife
16:03 towodo sample.config is also linked
16:03 kcranstn yes, edited a copy of that file
16:03 kcranstn should pem=/home/cranston/.ssh/opentree/opentree.pem
16:04 kcranstn OPENTREE_IDENTITY=~/.ssh/opentree/opentree.pem
16:04 towodo yes, that should do it
16:05 kcranstn still not working
16:05 towodo oh. wrong identity. you’re using your own identity.
16:05 towodo should be OPENTREE_IDENTITY=~/.ssh/id_rsa
16:05 towodo I think
16:06 towodo you can say ‘deploy echo x’ as a test
16:06 kcranstn that gives me host=128.196.64.124, admin=cranston, pem=/home/cranston/.ssh/id_rsa, controller=cranston, command=echo
16:06 kcranstn Syncing
16:07 towodo looks good
16:07 kcranstn but when I deploy, it asks for my password and then fails
16:07 kcranstn cranston@128.196.64.124's password:
16:07 kcranstn as-admin.sh                                                          100% 6294     6.2KB/s   00:00
16:07 kcranstn cranston@128.196.64.124's password:
16:07 kcranstn sudo: no tty present and no askpass program specified
16:07 kcranstn Sorry, try again.
16:07 kcranstn sudo: no tty present and no askpass program specified
16:07 kcranstn Sorry, try again.
16:07 kcranstn sudo: no tty present and no askpass program specified
16:07 kcranstn Sorry, try again.
16:07 kcranstn sudo: 3 incorrect password attempts
16:07 towodo oh… sudo is insisting on reading a password… that’s not needed on AWS
16:07 towodo I’m not sure how to get around this.
16:08 mtholder PreferredAuthentications in ~/.ssh/config can help with this sort of thing (i haven't been following this thread...)
16:09 towodo I think you can manually run the as-admin.sh script (on the server, without ‘deploy’). then after that it shouldn’t need to prompt
16:09 towodo i.e. it’s a one-time setup
16:09 towodo i have to leave the house now to go to MIT so will be offline
16:10 kcranstn what is that deploy command doing?
16:10 kcranstn ok, later
16:10 towodo the one with no args?  it’s trying to install all the software
16:10 towodo I can check IRC from my phone…
16:10 kcranstn it is hard to debug, because I can’t tell what happens when I type deploy
16:10 kcranstn and which deploy gives nothing
16:11 towodo deploy is an alias.
16:12 towodo you created it while following the setup instructions.
16:12 towodo it invokes push.sh with some arguments
16:12 kcranstn oops
16:13 kcranstn push.sh resets a bunch of the variables from the config file
16:40 kcranstn joined #opentreeoflife
17:24 josephwb joined #opentreeoflife
17:51 mtholder joined #opentreeoflife
18:03 towodo joined #opentreeoflife
18:04 towodo kcranstn, push.sh sources the config file, so it doesn’t matter
18:29 kcranstn joined #opentreeoflife
18:31 kcranstn jimallman is at NESCent, so he and I might take a stab at figuring this out
18:31 kcranstn this = installation of asterales system on iplant atmosphere instance
18:34 towodo I think you’re almost done. you just need to run as-admin.sh, after which ‘deploy’ should work
18:34 kcranstn do I need to run that as sudo?
18:34 towodo no, it does the sudos itself
18:35 towodo and it can be run as any user that can do sudo
18:35 kcranstn ok
18:35 kcranstn trying that
18:35 towodo it should prompt for password once, then proceed
18:40 kcranstn ad-admin assumes that you are in your home dire
18:40 kcranstn directory
18:40 kcranstn and I see that a copy of as-admin is now in my home dir...
18:41 towodo yes, the deploy script copies as-admin.sh to home dir of the admin account, so that it can run it there
18:41 kcranstn running the script in my home directory works. In opentree/deploy does not
18:42 towodo umm… remember this is designed so that you run deploy (push.sh) from machine A, which then tweaks machine B
18:42 towodo you are doing the unexpected by using the same machine for both roles. I never tried that
18:42 towodo or even thought that anyone might try it
18:42 kcranstn where is that documented?
18:43 josephwb joined #opentreeoflife
18:43 towodo let me look
18:43 kcranstn ah, I was unclear that all of the instructions were to be done on my local machine
18:43 towodo well, it’s implicit in the asterales setup instruction, where you don’t even get to ‘provision a machine’ until pretty far down in the process
18:44 kcranstn “implicit” is a dangerous word in documentation
18:44 kcranstn I just demonstrated that
18:44 towodo in other words it never told you to do this stuff on the target machine.  but it didn’t tell you not to
18:44 towodo yes, it should probably tell you to set up a staging area for the whole process, or something
18:45 towodo I can’t think of any deep reason why using the same machine is a bad idea
18:47 kcranstn ok, I am going to start over
18:47 kcranstn on my local
18:47 towodo oh no!
18:47 towodo you are almost there…
18:48 kcranstn no, still getting the same errors. And the whole point of this was to make sure that the documentation was sufficient
18:48 kcranstn so, better to update the documentation with “Do This Locally” than try to solve this strange edge case
18:48 towodo which ‘same errors’?  they will need to be debugged regardless
18:49 kcranstn cranston@vm64-124:~/opentree/deploy$ deploy
18:49 kcranstn host=128.196.64.124, admin=cranston, pem=/home/cranston/.ssh/id_rsa, controller=cranston, command=
18:49 kcranstn Syncing
18:49 kcranstn as-admin.sh                                                 100% 6294     6.2KB/s   00:00
18:49 guest|8913 joined #opentreeoflife
18:49 kcranstn sudo: no tty present and no askpass program specified
18:49 kcranstn Sorry, try again.
18:49 kcranstn sudo: no tty present and no askpass program specified
18:49 kcranstn Sorry, try again.
18:49 kcranstn sudo: no tty present and no askpass program specified
18:49 kcranstn Sorry, try again.
18:49 kcranstn sudo: 3 incorrect password attempts
18:49 kcranstn I have already run as-admin.sh
18:49 kcranstn successfully
18:50 towodo then it’s a bug.
18:50 towodo I read through as-admin.sh to see if it would need to do a ‘sudo’ the second time around and concluded no. maybe I was wrong
18:52 guest|8913 left #opentreeoflife
18:52 towodo oh, I didn’t read all of it
18:52 kcranstn push_bot_identity and push_phylesystem_api
18:52 towodo this is a bug that I need to fix before you can finish
18:53 kcranstn ok
18:54 jimallman_Kiwi joined #opentreeoflife
18:59 jimallman_Kiwi joined #opentreeoflife
18:59 jimallman_Kiwi test
19:01 jimallman_Kiwi joined #opentreeoflife
19:15 towodo kcranstn, https://github.com/OpenTreeOfLife/opentree/pull/453
19:31 towodo kcranstn, wrote some introductory remarks
19:31 towodo (I think you’re in a meeting or something)
19:33 kcranstn nope. Meeting done
19:33 towodo I think it’s ready for another try
19:33 kcranstn what do I need to update?
19:35 towodo opentree repo I guess… or you could start going through the instructions from the top if you want to switch to controlling from your local machine
19:36 towodo everything is idempotent
19:36 towodo (except the ‘git clone’s )
19:37 kcranstn I started from the top on my own machine, and I am still getting the repeated password attempts and sudo error
19:38 kcranstn what should the OPENTREE_IDENTITY be?
19:38 towodo you will still need to run as-admin.sh manually (outside deploy) and enter the password.  but it won’t be needed after the first time
19:39 kcranstn this time, it didn’t copy as-admin to my home directory. Do I need to do that manually?
19:39 towodo OPENTREE_IDENTITY setting is documented now
19:40 towodo you run as-admin.sh on the target computer
19:40 towodo I did’t document that, will do so now
19:43 towodo refresh https://github.com/OpenTreeOfLife/germinator/wiki/Setting-up-the-asterales-system#deploy-software-webapps-neo4j-plugins-etc-and-phylesystem-clone
19:45 kcranstn when I deploy echo x, I get “host=128.196.64.207, admin=cranston, pem=/Users/karen/.ssh/id_rsa, controller=karen, command=echo”
19:45 kcranstn which seems right
19:45 kcranstn but then it wants a password for opentreee@128.196.64.207
19:46 kcranstn (as-admin.sh run successfully on target)
19:46 towodo oh no.  I bet sshd is misconfigured on your target
19:46 towodo or maybe not… I don’t quite understand…
19:47 towodo when does it want the password? when you do simply ‘deploy’?
19:48 towodo 128.196.64.207 is the target I presume
19:51 kcranstn yes
19:51 kcranstn when I deploy, yes
19:51 kcranstn noting that there is no opentree user on that target
19:52 towodo really?? as-admin.sh creates one
19:52 kcranstn ah, I see
19:52 kcranstn I didn’t know that
19:52 kcranstn what is the password for that user?
19:52 towodo no password. its authorized_keys is copied from the authorized_keys for the admin user
19:52 kcranstn interesting
19:53 towodo so same private key should work for both accounts
19:53 towodo no passwords involved.
19:53 towodo if the as-admin.sh script ran to completion on the target, there should be an opentree account there
19:53 kcranstn there is an opentree account, but I cannot log into it
19:55 towodo have you been typing in a password to get into the admin account?
19:55 kcranstn no
19:56 towodo can you log in to the admin account, do ‘sudo bash’, and poke around a bit for me?
19:56 kcranstn sure
19:56 towodo want to see if ~opentree/.ssh/authorized_keys is set up
19:56 kcranstn root!
19:58 towodo so what does ls -al ~opentree/.ssh/ say?
19:58 kcranstn diff cranston/.ssh/authorized_keys opentree/.ssh/authorized_keys
19:58 kcranstn no diffs
19:58 kcranstn root@vm64-207:/home# ls -al opentree/.ssh
19:58 towodo the file protection modes are important, for the .ssh directory
19:58 kcranstn total 12
19:58 kcranstn drwx--S--- 2 opentree opentree 4096 Sep 26 12:57 .
19:58 kcranstn drwxrwsr-x 6 opentree opentree 4096 Sep 26 12:42 ..
19:58 kcranstn -rw-r--r-- 1 opentree opentree  406 Sep 26 12:33 authorized_keys
19:59 towodo hmm, same as on AWS
19:59 towodo what about ls -al ~/.ssh   (for comparison)
19:59 towodo if you’re wiling
19:59 kcranstn drwx------  3 cranston iplant-everyone  4096 Sep 26 12:33 .
19:59 kcranstn drwxr-xr-x 21 cranston iplant-everyone  4096 Sep 26 12:57 ..
19:59 kcranstn -rw-r--r--  1 cranston iplant-everyone   406 Sep 26 12:33 authorized_keys
20:00 towodo bleh.
20:00 kcranstn group issue?
20:01 towodo next thing would be for you to email me /etc/sshd_config,  and/or to try ssh -v opentree@target.  sorry this is so awful but I see no other way
20:02 towodo maybe sshd is configured in some funny way on those machines.
20:02 towodo like, maybe it locks out all users other than you.
20:03 towodo or disables public key authentication for everyone besides you.
20:04 kcranstn the guy I would ask is idle on gchat
20:05 towodo I could also redo the deployment system so that it can optionally work with only one user account.
20:05 towodo that might not take long, it’s not very big, I would just search for all occurences of ‘opentree’ and turn them into a variable
20:06 towodo actually it’s already like this.  I thought ahead
20:07 towodo you can edit push.sh to set OT_USER to whatever you like (e.g. the admin account)
20:07 towodo the resulting setup isn’t as secure but it should work
20:07 towodo I’ll make it settable in the config file
20:11 kcranstn OMG I think it is working
20:11 kcranstn after changing OT_USER
20:12 kcranstn but darren is looking into the permissions issue
20:12 kcranstn we should be able to create new users
20:13 towodo https://github.com/OpenTreeOfLife/opentree/compare/opentree-user?expand=1
20:14 towodo what do you mean?  multiple logins on the target machine?
20:15 towodo multiple users using the same target machine?… we’d have to coordinate use of port 80
20:15 kcranstn on these atmosphere instances, when we create the opentree user, it has to be in group “users” in order to ssh
20:15 towodo oh. you mean the script should work as designed on your iplant machine. forgive my jabber
20:15 towodo hmm…
20:15 towodo that would have shown up in the sshd_config
20:16 kcranstn AllowGroups users core-services root
20:16 towodo that’s going to be different for every cloud (so to speak)
20:17 kcranstn so, we just need to be able to be flexible
20:17 towodo not sure what the instructions should say.  “Make sure that it’s possible to ssh in as the opentree user” … ?
20:18 kcranstn Make sure that it is possible to ssh in as the OT_USER
20:18 towodo I can never remember the command for adding a user to a group.  adduser maybe, or useradd, or groupadd, or addgroup
20:19 kcranstn usermod -a -G users opentree
20:19 towodo yeah, that one
20:19 kcranstn deployment blew up partway through, I think due to mismatch of usernames
20:19 kcranstn ./setup/install-api.sh: line 134: cd: /home/opentree/repo/phylesystem-api/bin: No such file or directory
20:20 kcranstn going to try again with opentree user (now that group set properly)
20:21 towodo you can merge the PR if you want and put OPENTREE_USER= in your config
20:22 towodo oh.  I see, I didn’t get all occurrences of ~opentree.
20:23 kcranstn ok, redeploying
20:23 kcranstn after this works, I need a beer
20:23 kcranstn iplant is switching over to ansible, so there is another excuse
20:24 kcranstn for using ansible, not for beer
20:24 towodo yes I expect the deployment system to be replaced.  much is salvageable regardless of platform, so not a loss
20:29 kcranstn Cloning into 'jade'...
20:29 kcranstn Error: JAVA_HOME is not defined correctly.
20:29 kcranstn We cannot execute /usr/lib/jvm/java-7-openjdk-amd64/bin/java
20:29 kcranstn got farther this time without failing...
20:29 kcranstn JAVA_HOME is /usr/lib/jvm/java-7-openjdk-amd64
20:30 kcranstn which does not exist
20:31 kcranstn I think I shall go cry for a while
20:31 towodo ah. I think that’s another shortcut I took. There are lots of AWS assumptions built in.
20:32 towodo not your fault, you are just pushing the envelope.
20:32 kcranstn I did hypothesize that this would suck up my whole day
20:32 towodo java setup is always a nightmare.  it’s very hard to do it in a general way, so I took a shortcut, and you are paying the price now
20:32 kcranstn you owe me a beer ;)
20:33 towodo neo4j needs for JAVA_HOME to be set.  there’s no version-independent way to figure out what JAVA_HOME should be.
20:33 towodo I probably owe you several at this point
20:33 kcranstn :)
20:34 towodo now I have to think.  we can either try to force java 7, or I can read up on how to set JAVA_HOME
20:34 towodo forcing is not desirable since we’ll be moving to java 8 at some point
20:36 towodo http://www.wikihow.com/Set-Java-Home   does not answer the question
20:37 towodo I think I will do a case statement
20:38 kcranstn it seems that we need to move a bunch of tests up front so that we don’t waste time on deployment steps just to get a failure at the end
20:39 towodo setting JAVA_HOME many have to be a manual step.  I don’t see how to do it generally
20:39 towodo could go in the config file…
20:40 towodo this is sort of funny http://archive.oreilly.com/pub/post/mac_os_x_java_wheres_my_java_h.html
20:42 towodo so you will need to figure it out for your machine.
20:42 towodo ls -l `which java`   and so on
20:43 kcranstn is this on my local or the target?
20:43 towodo umm… depends… what is the command that’s failing?
20:43 kcranstn deploy
20:43 kcranstn ;)
20:44 kcranstn the error message shows up on my local shell, but I am not sure whether the error is happening locally or on the server
20:45 towodo that’s the target then.  but now I’m wondering why exactly it’s needed.  oh, I think it’s for the neo4j *server* (command line doesn’t need, so command line works on os x without setting java_home)
20:45 towodo target
20:45 kcranstn and it appears that the target should be /usr/lib/jvm/java-7-openjdk-amd64/bin/javaws
20:45 kcranstn not /usr/lib/jvm/java-7-openjdk-amd64/bin/java
20:46 towodo hmm.
20:51 towodo (interrupted, back in a bit)
21:09 towodo I have to do my tdwg registration. last day i think
21:10 towodo no, it’s the 29th.
21:12 kcranstn we are running into some strange issues where the JAVA_HOME is getting updated to the AWS location no matter what we try
21:16 kcranstn jimallman thinks it might be pulling fresh copy from github
21:16 towodo right. I just found out why. it’s set in .bashrc
21:16 towodo I’m looking at this ‘activate’ script
21:16 towodo this is a pretty horrible piece of code
21:17 kcranstn boo
21:17 towodo I don’t think JAVA_HOME needs to be set in .bashrc.  I’m not convinced it needs to be set at all
21:17 kcranstn config
21:18 kcranstn ok, it’s Friday 5:30 and I am officially giving up for the day
21:18 towodo I’ll try to have it in better shape on monday & will continue testing on ot17
21:25 towodo ok, I just compiled and installed oti with no JAVA_HOME set, so I think I will just flush it
21:25 jimallman_Kiwi joined #opentreeoflife
23:17 josephwb joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary