Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-07-08

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:42 towodo joined #opentreeoflife
01:33 towodo joined #opentreeoflife
01:40 jimallman joined #opentreeoflife
01:54 jimallman towodo: i have no strong feelings on the 3-letter taxa. To me, they seem cryptic, but if we assign them alphabetically it does give a quick sense of precedence. I think the most likely problem is wondering, “Let’s see, was that the ‘Isa’ release, or the ‘Iso’ that came just after?”
01:54 jimallman For that reason, I’d advocate using just one A, B, C, D, … and repeating the alphabet with fresh names, a la hurricane naming.
01:54 towodo jimallman, I’m trying out recognizable short named taxa.  E.g. Atta and Bos are genera many biologists know. And stepping through them alphabetically (like hurricane names).
01:55 jimallman i like it! memorable beats short IMHO
01:55 towodo as  you suggest, great minds think alike
01:56 towodo already renamed ‘production’ in my copy of the deployed-systems repo.
01:56 towodo ot14 is there, can be initialized now.  ot15 tomorrow
02:40 jimallman towodo: ok, i’ll pull the latest and take a look, once it’s available
02:40 towodo not done, will push tomorrow a.m..
02:57 towodo joined #opentreeoflife
03:09 jimallman joined #opentreeoflife
05:02 jimallman joined #opentreeoflife
05:20 jimallman joined #opentreeoflife
10:01 towodo joined #opentreeoflife
12:47 jimallman joined #opentreeoflife
13:44 towodo jimallman, hi
13:44 jimallman towodo: good morning!
13:45 jimallman #opentreeoflife is getting sleepy again.. time for fabulous prizes!
13:45 towodo good morning.  i think it’s about ready to go.  there is just some funniness with neo4j dbs, after everything else is set up.
13:46 towodo I have awarded two… not yet time for the next batch
13:46 towodo burning candle at both ends?
13:47 towodo anyhow see https://github.com/OpenTreeOfLife/deployed-systems/compare/Bos?expand=1
13:47 jimallman ah, i’m looking at the Bos branch now (just taking a peek)...
13:47 jimallman neat!
13:49 towodo both servers are up and running and have their dns entries
13:50 jimallman i see there’s going to be a bit of a dance when we “advance” the production + dev domains from one system to the next… esp. the docstore and data. we might even want to disable / redirect these briefly, just to avoid accidental fetches (or commits) to/from the wrong docstore during changeover..
13:51 jimallman re: the candle, i’m juggling work with watching over our HVAC installation today. fun!
13:51 jimallman but getting stuff done. i’m working on an old ticket for right-click support on nodes in argus..
13:54 jimallman re: the “dance”, we’ll also need to tweak *_GITHUB_CLIENT_ID and *_GITHUB_REDIRECT_URI, since these are domain-specific.
13:55 jimallman (but that’s already required)
13:55 jimallman towodo: ^
13:56 towodo right.  that would make a good comment in the config file.
13:56 jimallman is “splashopentreeoflife.org” a typo? or just a funky domain name?
13:57 towodo I wouldn’t worry about right click too much.  Moving to this new version is more of a pain point.
13:57 jimallman (in opentree-ssh-config.txt)
13:57 towodo missing .
13:57 jimallman gotcha
13:57 towodo just changed it to ot13  (splash page server)
13:57 towodo (it’s a free tier server, tiny)
13:58 jimallman i have the Bos branch in my local repo. shall i add comments as outlined above?
13:59 towodo oh sure, and you can s/splashopen/ot13.open/ while you’re at it
14:00 towodo if you could help with the server setup that would be great. I’ve got 3 other major things to do this week (before breakfast)
14:00 towodo I’m thinking about testing
14:00 towodo going to put a stub in the otb repo.
14:09 jimallman sure, i’m at your disposal! just give me some direction.
14:20 towodo jimallman, my first thought is to record all tests that you do, informally.
14:20 towodo a la https://github.com/OpenTreeOfLife/otb/blob/master/integration/README.md
14:21 towodo they can go in that file
14:21 jimallman OK, will do. i’m updating the deployed-systems README (and other files) in Bos now, to explain the new system.
14:22 towodo excellent. second, do push.sh for the two new servers, in either order  ot14 & ot15
14:22 towodo oh. before that, make sure the phylesystem choice is reasonable (should not point to production phylesystem yet, during testing)
14:23 towodo so that was step 1.5
14:23 towodo step 3. copy the neo4j databases over from other servers (ot10 and ot9).  for this purpose I set up a second ssh key, which probably sort of silly
14:24 towodo I called it ‘lateral’ and can give it to you
14:24 towodo this is a bit fussy.  would be nice to automate but the first step of *that* would be to write down what one does
14:25 towodo so, add lateral public key to ot9 and ot10 authorized_keys, …
14:25 jimallman agreed. the end result should probably be a script or three…
14:25 towodo and add lateral private key to ot15
14:25 towodo I mean, set its id_rsa to the lateral private key
14:26 towodo s/its/ot15’s/
14:26 jimallman is lateral on a server already? i can just scp to grab it..
14:26 towodo hmm.  maybe on norbert, maybe not. i’ll check.  or do you have a PGP key?
14:26 jimallman hmm, i think i do around here somewhere… it’s been a long time since i’ve used it    :-/
14:27 towodo oh, i just got a new computer and haven’t installed gpg on it yet, so scratch that
14:27 jimallman norbert == ashby? i forget..
14:27 towodo no
14:28 towodo norbert is the big server we were using last fall.  ashby is a revival of an older smaller machine
14:28 towodo they’re both getting on in years
14:29 towodo oh yeah.  the lateral public key is on… ot12 I think
14:29 towodo oops.  publick key is on ot12
14:30 towodo oops. private on ot12
14:30 towodo public on ot9 I think
14:31 towodo that was the last lateral transfer I did.
14:31 towodo or you could copy the .db.tgz files locally and follow the instructions.  just an optimization
14:31 towodo there’s a new push.sh command for the lateral case.
14:32 towodo i haven’t automated the new oti setup yet - you re-use the taxomachine graphdb to initialize oti
14:32 towodo and we have to get the right databases for each component, see yesterday’s spreadsheet
14:36 towodo After machines are set up, test.  Fix bugs.  When tests (other than those needing credentials) pass, create branches.  Then shut down production.  Then update configuration to point to right phylesystem, right api keys, etc. Change DNS entries. Restart production. Test again, this time trying things that couldn’t be tried without real DNS entries. Fix those. Something like that.
14:36 towodo we should probably wait until after 5pm to shut down to production.
14:45 * jimallman is catching up now..
14:46 towodo I think what I meant was, test as much as possible without certs, then do dns switch, then test again
14:46 towodo the risk should be low since it’s all been tested in development setup.
14:47 jimallman re: new push.sh command for the lateral, i’m not finding it by that name.. what’s the command/component name for this?
14:47 towodo devapi will need to be blocked too, so we can switch its phylesystem
14:47 towodo hmm… install-db i think, let me check
14:48 towodo install_db.  it assumes the .db.tgz file has already been put in the downloads/ directory.
14:48 jimallman got it, thanks
15:01 towodo jimallman, i’m going to be in transit for a bit, back on probably by 12:30
15:01 jimallman OK, ttyl
15:34 josephwb joined #opentreeoflife
15:34 josephwb you there, jimallman?
15:34 jimallman josephwb: hi!
15:34 josephwb for this study: http://devtree.opentreeoflife.org/curator/study/view/ot_41
15:34 jimallman note that my recent fixes to OTU mapping aren’t visible on devtree. shal i push them now?
15:35 josephwb no tree is displayed
15:35 josephwb and i don't see a treeid when i click on it
15:35 josephwb most taxa are mapped
15:35 jimallman when i click the tree name, the URL changes to show its ID: http://devtree.opentreeoflife.org/curator/study/view/ot_41/?tab=trees&tree=tree1
15:35 jimallman but yes, something’s blocking the tree display… checking now...
15:36 josephwb so, the id is 1?
15:36 josephwb wasn't sure if that was a default
15:36 josephwb i.e. first in a list
15:36 jimallman yes, ‘tree1’ is probably the default for first/only tree.
15:36 josephwb ok, good to know
15:36 josephwb thanks
15:37 jimallman i believe we serialize these when importing trees, hopefully the same when importing whole studies :)
15:37 jimallman regarding the broken tree view, it looks like we’re missing OTUs (!) in this tree..
15:38 josephwb you mean they are not mapped?
15:38 josephwb when trees come from treebase, do the tree ids correspond to the treebase ids? looks like it: http://devtree.opentreeoflife.org/curator/study/view/ot_23/?tab=trees&tree=Tr64856
15:39 jimallman ‘node5’ thinks it’s mapped to ‘otu47’, but that OTU doesn’t exist. mtholder is chasing a known bug along these lines.
15:39 josephwb i guess i just didn't know how tree ids were set. with phylografter, all tree ids were unique
15:39 josephwb ok, i will come back to this later then.
15:40 jimallman in the new setup, tree ids are unique within a study, but not across them.
15:40 josephwb right, that is what i wasn't getting
15:41 josephwb are people on board with that?
15:41 josephwb with (potentially most) trees all having the same id?
15:42 jimallman as far as i know, this is accepted, since our docstore contains each tree within a study, and the API assumes we know both
15:42 jimallman i’ve added this study+tree to the GitHub issue on this missing-tree bug: https://github.com/OpenTreeOfLife/opentree/issues/349
15:42 josephwb i saw that
15:43 josephwb i don't want to start a discussion, but the PG id system allowed you to grab a tree (by its id) without necessarily knowing what the study id is. would that be useful?
15:55 jimallman i suppose, but when would you have just the one piece of information? iirc, we discussed using GUIDs long ago but went in another direction. i have no strong feelings about this, other than to note that it would take some work to retroactively assign unique tree ids to the current docstore.
15:55 jimallman another direction = serial ids by element type, eg, ‘tree2’, ‘otu499’
16:04 josephwb i don't have strong feelings on this
16:38 lcoghill joined #opentreeoflife
16:48 lcoghill jimallman: are you there by chance?
16:48 jimallman hi! just thinking about lunch… what’s up?
16:49 lcoghill quick question. i mised a good part of the chat yesterday in call thanks to our great wifi here.
16:50 lcoghill from what I could tell, the request was for a scrip that would calc otus, unique otus across all studies and just the studies in synth?
16:50 lcoghill wow, that was a lot of typos. i need to slow down
16:50 jimallman n prob. yes, the general idea is, let’s trigger a nightly script to gather key metrics for Open Tree...
16:51 lcoghill ok. thought so. I put a python script together that calculates total otus, unique otus across all studies, and total otus and unique otus across just the studies in synth.
16:51 jimallman presumably a cron job. some of the metrics will be gathered by particular services like oti or synthesis machinery, others would be handled by this nightly sweep of the Nexson docstore.
16:52 jimallman ah, that sounds great!
16:52 lcoghill yep, the script I wrote just uses the API calls.
16:52 lcoghill right now the results are just dumped to a csv file. is that ok for you to parse out, or is something else better?
16:52 jimallman perfect. ideally, these will also serve as examples for people building their own tools for bulk analysis.
16:53 jimallman regarding output to CSV.. i guess that’s OK, i can certainly wrangle that with python to build a web page showing key metrics.
16:54 jimallman will your CSV format allow tracking these values over time? it’s not an immediate need, but these metrics become much more meaningful as trends emerge.
16:54 jimallman lcoghill: ^
16:54 lcoghill ok, well I can store it in whatever is more ideal. That was just the fastest off the top of my head. I have it set now to save each entry to a new line with a date attached
16:56 lcoghill jimallman: thanks for the input. I'll stop bugging you and let you grab lunch.
16:59 jimallman thanks, lyndon. this sounds great!
17:13 towodo joined #opentreeoflife
17:18 josephwb joined #opentreeoflife
17:21 towodo lcoghill, script sounds great.  maybe do json instead of csv ?  can you send me a sample?  how long does it take to run?
18:21 mtholder joined #opentreeoflife
18:47 towodo joined #opentreeoflife
19:09 josephwb joined #opentreeoflife
19:41 towodo jimallman, how are you doing?
20:07 jimallman i’ve gathered the lateral keys, made my list, moving through the steps now (about to push)
20:12 jimallman towodo: at what point should we point ‘production’ branches to the HEAD of ‘master’?
20:13 jimallman (for now, i’m just planning to push master to all components in Bos)
20:17 towodo actually I’m wondering if we should just use branch names like Atta, Bos, etc.
20:18 towodo we can always delete them later but I’m not sure why we would need to use the same branch name for all versions
20:18 towodo jimallman  ^
20:19 jimallman hm! i see what you mean. so we’d gradually accumulate branches instead of subfolders.
20:20 jimallman i think we’d actually have a better sense of history this way, since GitHub would show the most recently changed branches first: https://github.com/OpenTreeOfLife/deployed-systems/branches
20:20 jimallman in that case, would we retain the original folder names ‘development’ ‘production’, etc?
20:21 josephwb joined #opentreeoflife
20:21 * jimallman is thinking through the dance… how does it change?
20:22 jimallman towodo: ^ it does seem we’d have a more consistent migration process if we move to a named branch, and expect to find the usual folders inside (development, production, etc). once the branch is specified, everything would flow nicely.
20:23 towodo no, I was saying just call is ‘Bos’ instead of ‘production’
20:23 towodo then on the next release, call it ‘Cavia’ instead of ‘production’
20:23 towodo so branch name = directory name
20:23 jimallman it = the branch? or a subfolder? or the final deployed system?
20:24 jimallman and/or, i guess
21:03 jimallman towodo: ^ sorry for my confusion here.. what is “it”?
21:03 towodo it = the directory containing the config files for the production system.
21:05 towodo there’s a deployed-systems/Bos/ and a Bos branch of deployed-systems. and no ‘production’ directory or branch.
21:05 towodo just thinking about it.
21:05 jimallman gotcha. i assumed as much from the way you set up the Bos branch
21:05 towodo I didn’t make a branch yet… just a dir
21:07 jimallman ? that’s where I’ve been working: https://github.com/OpenTreeOfLife/deployed-systems/tree/Bos
21:07 towodo oh right sorry. i *did*. forgot.
21:10 jimallman it’s a hall of mirrors. one reason i’m intrigued by branches like Atta and Bos, with subfolders named by convention (development, production).. though I see that this blurs the meaning of Bos from “one working system, with domains TBD” to “a co-existing set of working systems, including production and development”. ugh
21:10 towodo I can’t imagine why you’d want a Bos development branch.  development doesn’t need to be versioned in that way.
21:11 towodo I mean directory, not branch
21:11 towodo the whole point is to make staging smooth
21:11 towodo you don’t stage development.
21:11 jimallman i see your point
21:12 towodo if it doesn’t work it’s not going to be that hard to change.
21:12 jimallman so once Bos is approved and “blessed” as the new production, does branch Bos become branch ‘master’? or do we abandon the notion of a primary/default branch in this repo?
21:13 towodo no, master = development.
21:13 jimallman mm, ok
21:13 towodo Bos is a production branch
21:13 jimallman gotcha
21:13 jimallman like a tagged release
21:13 towodo we have lots of production branches, only one development.
21:13 towodo similarly for directories in deployed-systems.
21:14 towodo in fact, we don’t really need the branches, for the deployed-systems repo.
21:14 towodo but we do need them for the other repos.
21:15 towodo no reason not to merge Bos to master now, I think, or as soon as deployment happens. the only incompatibility is I renamed the production dir to Atta, but nobody should care about that
21:15 towodo merge in the deployed-systems repo that is.
21:15 towodo in other words: we have Atta, Bos, etc. dirs on master in deployed-systems, and Atta, Bos, etc. branches in all other repos
21:21 towodo jimallman, does that make sense?
21:22 jimallman yes, and i agree that branches are an unnecessary complication for deployed-systems. i’d rather the folders accumulate as you said. it leaves a record of past development, like so:
21:22 jimallman http://i-edu.org/wp-content/uploads/spiral-shell.jpg
21:25 josephwb joined #opentreeoflife
21:25 jimallman regarding the named branches in other repos, should we be considering tags (tagged releases) instead? these are basically frozen branches in git, and we can keep the Atta, Bos, etc names.
21:26 jimallman nope, scratch that. tags are brittle and very hard to change if we need to: https://www.kernel.org/pub/software/scm/git/docs/git-tag.html#_on_re_tagging
21:27 jimallman i think you’re right, named branches (Bos, etc) have the right balance of history and flexibility
21:27 towodo branches are just tags that you can change…
21:28 * jimallman nods
21:29 jimallman i thought tags had more appropriate connotations re: history, but their “stickiness” is a liability. i fear someone coming to the code later and being confused by all these mysteriously-named branches. i’ve already had some push-back from other developers for the sprawl of feature branches.
21:34 jimallman towodo: Meanwhile… I'm debating (with myself) the best time to update the ‘production’ references to point to latest commit in master, in all repos. Ah, but now it these would be ‘Bos’ branches instead of ‘production’. (i see how this all starts to pay off in deployment.)
21:35 jimallman i think i should create these branches now, so we have stability during the push+test.
21:37 towodo no, i would test first. that lets you fix problems first.
21:37 towodo i’m hoping the transition period is only about a day
21:37 towodo so, test, freeze, test, promote, test
21:38 towodo that’s why we need automated tests…
21:38 towodo freeze = create Bos branches
21:39 towodo do you want to try to go to production tonight? otherwise it should wait until tomorrow night  (assuming you work after 5, which is what i’ve observed)
21:45 towodo jimallman ^
21:45 jimallman i’ll keep working at it tonight and see how the testing goes.
21:46 towodo you saw my testing ‘script’? do you think you’ll want to / be able to extend it?
21:46 jimallman regarding creation of ‘Bos’ branches: my thinking was, it’s a more controlled situation if i create them and merge fixes from master (as needed), rather than risk having someone else’s changes on master cause trouble. but i’ll try using master for now.
21:47 jimallman if you mean the integration README, i already have: https://github.com/OpenTreeOfLife/otb/blob/master/integration/README.md
21:47 towodo ok, i just thought it would be a hassle. but that’s fine if you’re will to keep track of it all.
21:48 towodo we’re trying to get to low-overhead release, so anything you can figure out to make the process easier the third time would be excellent.
21:48 jimallman agreed. i’m keeping notes along the way.
21:49 towodo I’ll update the deployment instructions with the new OTI setup.
22:41 josephwb joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary