Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-01-23

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:18 kcranstn joined #opentreeoflife
00:31 jimallman joined #opentreeoflife
05:02 mtholder joined #opentreeoflife
05:36 mtholder left #opentreeoflife
05:54 jimallman joined #opentreeoflife
13:18 kcranstn joined #opentreeoflife
13:58 jar286 joined #opentreeoflife
14:10 jar286 joined #opentreeoflife
14:18 jar286 joined #opentreeoflife
14:26 blackrim joined #opentreeoflife
14:34 kcranstn joined #opentreeoflife
14:35 kcranstn blackrim decides to hang out with the geeks today?
14:36 blackrim haha, at least decided to hang with *these* geeks
14:45 blackrim brb
14:46 blackrim joined #opentreeoflife
14:54 blackrim joined #opentreeoflife
15:14 blackrim joined #opentreeoflife
15:46 jar286 joined #opentreeoflife
16:11 jar286 jimallman, you there?
16:11 jimallman yes, hi
16:11 jar286 hi. i wanted to see if we could set a *regular* time for production upgrades
16:11 jar286 weekly
16:11 jimallman sure!
16:12 jimallman what about immediately after PR review?
16:12 jar286 here are my thoughts. (1) during conventional work hourse (2) at beginning or end of day (3) soon after PR review
16:12 jar286 yes.
16:12 jar286 well, I don’t want to interrupt curators any more than necessary
16:12 jimallman hm, right. i’m pretty flexible on work hours.
16:13 jar286 so maybe 4 instead of 3. but that’s awkward. could do 3 or 4.
16:13 jar286 or 1st thing friday morning.
16:14 jimallman early mornings are more of a stretch for me (i’ve become a night owl lately), but it does sound better for curators.
16:14 jar286 well I don’t want to stretch you.
16:14 jar286 how about 4 on thursdays?
16:15 jimallman nice for me, but maybe a bummer for curators. you have a way to monitor current activity, yes?
16:15 jar286 all I do is run tail on the apache log. not very sophisticated, and not very good, since we have all these bot visitors
16:16 jar286 maybe grep curator would help, I don’t know
16:16 jimallman we have the ‘maintenance’ page as well, and the ability to block new curation. maybe we could set those up before PR review..
16:16 jar286 or after, yes.
16:16 jimallman beforehand would give us an hour to clear out lingering curators.
16:16 jar286 i want to announce a time to the opentreeoflife list, at the same time I announce today.
16:16 jar286 ok.
16:17 jar286 maybe 1/2 hour. I can say: Weekly maintenance time is 3:30 to 5 on Thursdays.
16:17 jimallman that sounds like a safe window, yes.
16:18 jar286 ok. will put on calendar and announce
16:18 jimallman oh, i may have misunderstood.
16:19 jimallman i was thinking 2pm: put up the maintenance warning (no new curation, please) and start PR review. 3pm: start deployment to production.
16:19 jimallman but 3-3:30 is also a pretty good window to clear out. it just leaves us waiting around for a bit.
16:21 jar286 no, I mean have people clear out between 3:30 and 4:00, then do the maintenance at 4:00.
16:21 jar286 we can work on other things between 3 and 4.
16:22 jimallman i see. sounds good!
16:22 jar286 putting it at the end of the day give people bigger uninterrupted periods of time.
16:22 * jimallman nods
16:24 jimallman i see the new milestone (Inspector Lee!). this looks good to me. some of these will require broader conversations, of course.
16:26 jimallman and i’ll pursue the final Columbus Day issue today. i was waiting to see what came of the email conversation with hyanwong.
16:26 jimallman oops, not the same issue.
16:27 jimallman (link to EOL vs. link to TreeBASE/Dryad)
16:27 * jimallman is stepping away for coffee
16:44 * jimallman is back
17:00 kcranstn one of these weeks, I want to do the re-deployment
17:01 kcranstn but not today
17:01 jimallman cross-training! yes, let’s do that.
17:01 jimallman but not today
17:01 kcranstn or the next three weeks ;)
17:01 jimallman :D
17:13 kcranstn joined #opentreeoflife
17:20 jar286 joined #opentreeoflife
17:59 jar286 joined #opentreeoflife
18:20 jar286 jimallman, I got another 1/2 hour later today, can we talk at 2 instead of 1:30?
18:21 jimallman sure! works for me
18:22 kcranstn joined #opentreeoflife
18:57 pmidford2 joined #opentreeoflife
18:57 jar286 jimallman, here now
18:58 * jar286 thinking about how to find hackers-at-work page documentation
18:58 kcranstn ?
18:58 jar286 there’s a page saying maintenance in progress, please come back later
18:59 jar286 https://github.com/OpenTreeOfLife/opentree/tree/master/deploy#notifying-users-of-scheduled-downtime
19:00 jar286 hmm, requires me to edit a file on devtree
19:00 jar286 emacs is installed, we’re in luck...
19:02 jimallman here now
19:02 jimallman yes, i’ve tried to document the site maintenance how-to in a couple of places…
19:02 jar286 done!
19:02 jimallman great, thanks
19:02 jar286 very nice
19:03 jar286 ok, now to production deployment, yes? or should we try it all on dev?
19:03 jar286 I guess production
19:04 jimallman hm, no harm in a test on devtree, but we’ve got some known issues with phylesystem-api there now
19:04 jimallman (branches out of sync, under discussion w/ mtholder)
19:04 jar286 let’s go straight to production
19:04 jimallman sounds good
19:04 jar286 first, I’ll update devapi using push.sh …
19:04 jimallman devapi? or api?
19:04 jar286 api
19:05 jar286 no, first I’ll refresh my git repos
19:05 jimallman fyi, i don’t see any site-maintenance message on production
19:05 jimallman did you mean to enable that?
19:06 jimallman (went to curation app, found a study, logged in to edit… no message so far). web2py restart required?
19:06 jar286 ot14?
19:07 jimallman yep, tree.opentreeoflife.org = ot14
19:07 jar286 well, a restart would sort of defeat the purpose - doesn’t that kill sessions?
19:07 jimallman can’t recall at the moment. i think not, actually.
19:07 jar286 or is session state in the file system?
19:07 jar286 hang on, I need to do a pit stop, back in 3 minutes
19:07 jimallman i think so, or in web2py db
19:08 jimallman fyi, sessions are in filesystem, for ex: opentree/curator/sessions
19:10 jar286 back
19:10 jimallman see above for session info.
19:10 jar286 right.  will restart web2py
19:11 jar286 I guess it’s just a matter of doing an apache restart…
19:11 jimallman yes
19:12 jimallman ssh admin@ot14 sudo apache2ctl graceful
19:12 jimallman or something like that
19:12 jimallman ok, looking good now (thanks)
19:12 jar286 humph.
19:12 jar286 not working for me
19:13 jar286 edited config and stoped & started apache
19:13 jimallman message should appear if you start to edit a study
19:13 jar286 trying again, this time with a ‘2'
19:13 jimallman works for me. clicked a study in the list (initial read-only view), then the message when i try to edit
19:13 jar286 got it.  moving on...
19:14 jar286 ./push.sh -c ../../deployed-systems/production/api.config
19:14 jar286 right?
19:14 jimallman sounds good, yes
19:15 jar286 I’ve refreshed the opentree and deployed-systems repos on my machine
19:15 jimallman good deal. looks like all production configs are using the correct values for release (vs testing)
19:17 jimallman and the sneaky OTI_BASE_URL looks right to me (scheme-relative, same as in development/devtree.config)
19:18 jar286 doing  ../../germinator/test.sh api.opentreeoflife.org  now
19:18 jar286 most fail...
19:18 jimallman !
19:18 jar286 maybe I need to say /v2 or something
19:19 jimallman i haven’t run these tests, looking there now...
19:19 jar286 maybe those tests never worked.
19:20 jimallman looking at the TESTING.md there:  https://github.com/OpenTreeOfLife/germinator/blob/master/TESTING.md
19:20 jimallman here’s mtholder’s web tool: http://phylo.bio.ku.edu/status/status.html
19:20 jimallman half failing, half working
19:20 jar286 right, I never documented test.sh.  probably should ignore it
19:21 jar286 Mark’s tests have already run and passed
19:21 jar286 his peyotl tests that is
19:21 jar286 I’l make an issue about that
19:21 jar286 I mean about test.sh
19:21 jar286 ok, moving on…
19:22 jar286 ./push.sh -c ../../deployed-systems/production/tree.config    …
19:23 jar286 Whoops! The call to get the tree around a node did not work out the way we were hoping it would. Show details
19:23 jimallman hm, maybe i’m jumping the gun, but i try to see the tree view and get a missing-plugin error from api.opentreeoflife.org
19:23 jimallman missing GoLS plugin(!)
19:24 jar286 that’s what test.sh was saying, in effect
19:24 jar286 let me see what’s been going on with treemachine…
19:24 jimallman yes, mark’s web test page as well
19:24 jar286 we can always make a branch off an earlier commit if we need to
19:25 jimallman yes. api.config looks sensible (using master branch for treemachine and everything else)
19:26 jar286 previously we were running treemachine 567e1eeca0905824e18dd2daad01e4466346866e
19:26 jar286 there have been 4 commits since
19:29 jar286 I don’t see anything that could have broken it.
19:29 jimallman same here… trying to remember where plugins are “registered” in neo4j
19:29 jar286 will check neo4j log
19:29 jar286 it’s in some turd file in the .jar
19:30 jimallman other plugins look OK, assuming that’s all of them:  curl http://api.opentreeoflife.org/treemachine/ext
19:32 jimallman searching treemachine repo for “GetSASJsons” shows similar results as for “GoLS”
19:32 jimallman perhaps the GoLS plugin failed to complle?
19:33 jar286 gobs of useless debug output in the log file
19:33 jar286 will check, hang on
19:33 jar286 BUILD SUCCESS
19:34 jimallman drat, i was looking at lots on the wrong box (ot14)..
19:35 jar286 Branch master set up to track remote branch master from origin.
19:35 jar286 Warning: you are leaving 11 commits behind, not connected to
19:35 jar286 any of your branches:
19:35 jimallman that’s the unattached head message, probably a red herring
19:36 jar286 at b262f5
19:36 jar286 i.e. master head is b262f5, which is latest commit on github
19:36 jimallman ok, i see the same in git log
19:37 jar286 so any messages about herrings are detached heads
19:37 jar286 :)
19:37 jimallman fish heads
19:37 jimallman perhaps a restart of treemachine (neo4j)?
19:38 jar286 Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
19:38 jar286 complaints about overlapping classes, but I think that’s normal
19:39 jimallman i’d say check the apache config, but we’re seeing other plugins...
19:40 jar286 will restart neo4j but I don’t think that’s it
19:40 jar286 so test.sh was right after alll…
19:41 jar286 failing...
19:42 jar286 will revert to 567e1e, just for fun
19:43 jar286 recompiling…
19:43 jimallman yes, worth a try
19:44 jar286 not much else we can do
19:44 jimallman i tried most of the treemachine method URLs defined in opentree/private/config, and they all want the GoLS plugin
19:45 jar286 Successes: 10 Failures: 0
19:45 jar286 whew!!
19:46 jar286 I don’t see how those 4 commits could break it
19:46 jar286 working old version is the ‘december’ branch
19:46 jar286 git WINS!
19:46 jimallman same here. i’ll review more closely for any reference to plugins or GoLS
19:46 jimallman meanwhile, googling on “missing neo4j plugin” with no joy so far
19:47 jar286 everything seems to be working now… of course haven’t tested exhastively…
19:48 jimallman right, and we’ve lost recent commits, right?
19:48 jimallman but yes, the tree view is back :)
19:48 jar286 no, only lost those 4 treemachine commits
19:49 jar286 submitting treemachine issue
19:52 jimallman just taking a moment to stare at the 4 recent commits…
19:52 jar286 https://github.com/OpenTreeOfLife/treemachine/issues/152
19:53 jar286 This is why we have PR review
19:53 jimallman yep, added a note there about missing GoLS plugin
19:54 jimallman nothing obviously relevant in the diffs of last 4 commits :-/
19:54 jar286 I need to run off.  The maintenance message got overwritten, so we’re back in motion now.
19:54 jar286 If you would do some more sanity checking I’d be most grateful
19:55 jimallman absolutely. i’ll dig up our old migration checklist and walk through it.
19:55 jar286 thanks, ttyl
19:56 jar286 joined #opentreeoflife
20:05 josephwb i have no idea what happened with GoLS
20:06 jimallman yeah, it’s funky. it just disappears in the newest version
20:06 jimallman i found a StackOverflow answer about adding plugins, that says you need to completely stop neo4j before adding one.
20:07 jimallman but in this case, it’s not even a new plugin..? here’s the link in case it rings a bell: http://stackoverflow.com/a/18283563
20:07 josephwb none of the changes look relevant.
20:08 josephwb is this just a failure to compile?
20:09 jimallman nope, jonathan said there were no compilation errors. but i wonder if the build skipped over it for some reason.
20:09 jimallman i keep seeing references to `META-INF/services`. is this in git?
20:10 josephwb compiles for me here.
20:10 jimallman i only see a reference to it in pom.serverplugins.xml:  https://github.com/OpenTreeOfLife/treemachine/search?q=META-INF
20:10 jimallman curious!
20:11 josephwb i don't know what META-INF/services is.
20:12 josephwb did you try any of the intervening commits? Would be nice to narrow things down.
20:18 jimallman i have not, sorry.
20:19 jimallman jar was driving the bus, so i’m not sure i can reproduce his steps exactly. i’m working through our manual checklist to see if everything else is OK.
20:28 josephwb I can compile and run the plugin locally.
20:29 jimallman thanks, good to know. maybe we can take another swing at the latest treemachine when jar is back at his desk.
20:33 josephwb jimallman: i did get some warnings during compiling.
20:33 jimallman anything surprising?
20:33 josephwb "overlapping classes"
20:33 jimallman ah, that’s so common we don’t freak anymore
20:33 jimallman :-/
20:36 josephwb i haven't refreshed the other repos. maybe there is some conflict?!?
20:36 josephwb [trying now]
20:37 jimallman good thought, should be master branch for all repos
20:40 josephwb hmm, that wasn't it.
20:41 josephwb jimallman: is the older TM code being used at the moment?
20:42 jimallman yes, 4 commits back
20:42 jimallman is that a minor irritation, or a big problem?
20:42 josephwb both prod and dev?
20:42 jimallman prod for sure, not sure about dev (will check now)
20:42 josephwb doesn't bother me; just trying to help track this down.
20:42 josephwb both work for me.
20:42 josephwb curl -X POST http://api.opentreeoflife.org/treemachine/ext/graph/graphdb/about
20:42 josephwb curl -X POST http://devapi.opentreeoflife.org/treemachine/ext/graph/graphdb/about
20:43 jimallman curl http://api.opentreeoflife.org/treemachine/ext
20:43 jimallman that’s the simple plugin listing
20:44 josephwb oh, wait. jonathan was talking about tree_of_life, not GoLS
20:44 jimallman in the latest version, there were a few missing plugins (now that i’m looking more closely)
20:44 jimallman not just GoLS
20:44 josephwb will check that now
20:45 jimallman this is what i saw on api.opentreeoflife.org with the very latest code
20:45 jimallman curl http://api.opentreeoflife.org/treemachine/ext
20:45 jimallman {
20:45 jimallman "CypherPlugin" : "http://api.opentreeoflife.org:7474/db/data/ext/CypherPlugin",
20:45 jimallman "GremlinPlugin" : "http://api.opentreeoflife.org:7474/db/data/ext/GremlinPlugin",
20:45 jimallman "GetSASJsons" : "http://api.opentreeoflife.org:7474/db/data/ext/GetSASJsons"
20:45 jimallman }
20:45 jimallman just those three plugins
20:46 josephwb weird.
20:47 josephwb "CypherPlugin" : "http://localhost:7474/db/data/ext/CypherPlugin",
20:47 josephwb "graph" : "http://localhost:7474/db/data/ext/graph",
20:47 josephwb "GremlinPlugin" : "http://localhost:7474/db/data/ext/GremlinPlugin",
20:47 josephwb "GetSASJsons" : "http://localhost:7474/db/data/ext/GetSASJsons",
20:47 josephwb "PhylografterUpdater" : "http://localhost:7474/db/data/ext/PhylografterUpdater",
20:47 josephwb "GoLS" : "http://localhost:7474/db/data/ext/GoLS",
20:47 josephwb "tree_of_life" : "http://localhost:7474/db/data/ext/tree_of_life"
20:48 josephwb apologies for the messiness
20:48 jimallman (I added the above to the issue)
20:48 josephwb this does not appear to be a TM problem.
20:48 josephwb did something change with deploy?
20:50 jimallman hm, lemme see. i thought perhaps a change in the apache config.. but that wouldn’t clobber plugins in this way.
20:55 josephwb do you know if jonathan tried compiling more than once?
20:56 josephwb jimallman^
20:56 jimallman i thought so, but he might have just checked the logs and found no errors.. reviewing now
20:59 josephwb well, crap, i have no idea what is going wrong.
20:59 jimallman josephwb: it looks like jonathan compiled once with new code (no errors reported, but plugins missing), and again with older code (success)
20:59 jimallman i can’t help wondering if we tried the new code again, and recompiling, what would happen.
20:59 josephwb shall i try doing this myself on dev?
20:59 jimallman good idea. would be useful if we can reproduce there
21:00 josephwb ok, trying to remember exactly how to do that now.
21:00 jimallman (push all master branches to devapi, i mean)
21:00 josephwb yup
21:00 jimallman pull repo deployed-systems, and make sure all branches are set to master
21:01 josephwb yeah, i think i've got it
21:01 jimallman (in development/devapi.config)
21:01 jimallman then in ../opentree/deploy, do
21:01 jimallman ./push.sh -c ../../deployed-systems/development/devapi.config
21:01 josephwb shall i just do treemachine, or everything?
21:02 jimallman hm, let’s try just treemachine first
21:02 josephwb why is the push script and config files in different repos?
21:03 jimallman different concerns (how to vs. system state) i suppose
21:04 josephwb problem with OPENTREEAPI_OAUTH_TOKEN
21:04 jimallman oh, probably a good idea to ‘git pull’ in the opentree repo, to make sure you’re using the latest deployment scripts
21:05 josephwb lots of errors
21:05 jimallman missing on your local filesystem, perhaps? i can do it from here instead...
21:05 josephwb yes
21:05 josephwb seemed to build everything
21:06 josephwb and it works
21:06 josephwb curl -X POST http://devapi.opentreeoflife.org/treemachine/ext/graph/graphdb/about
21:07 jimallman drat, i just started a fresh push.. just a sec
21:08 josephwb sorry; i wasn't clear about my "yes" above.
21:08 josephwb i got this error:
21:08 josephwb Cannot find OPENTREEAPI_OAUTH_TOKEN file /home/josephwb/.ssh/opentree/OPENTREEAPI_OAUTH_TOKEN
21:08 josephwb i don't think i ever had that
21:08 jimallman curl http://devapi.opentreeoflife.org/treemachine/ext
21:09 jimallman this is showing all plugins (durnit)
21:09 jimallman i’ll try again with all devapi components, to see if something else is breaking things
21:09 jimallman we’ll need to put OPENTREEAPI_OAUTH_TOKEN in a safe place where you can dowload it
21:09 josephwb thanks
21:09 josephwb is this newish?
21:10 josephwb i have opentree.pem
21:10 jimallman it’s been around for awhile, iirc
21:12 josephwb i think we should change the name of this issue...
21:12 jimallman ok, i pushed all components to devapi, and we still show the full plugin list (ie, it’s working). weird.
21:13 josephwb i guess i will close the issue.
21:13 josephwb cool?
21:13 jimallman let’s keep it open til we sort things out.
21:13 josephwb alrighty
21:13 jimallman something’s obviously goofy on production.
21:14 josephwb a fluke?
21:14 jimallman again, i’m tempted to push master there and see if it Just Works.  :0
21:14 josephwb are you going to try again?
21:14 josephwb yes
21:14 josephwb do that
21:14 jimallman ok, first i’ll make sure i can get back to the current state. :)
21:14 josephwb i blame ilbot3.
21:20 jimallman damnit, it worked.
21:21 josephwb good.
21:21 jimallman i pushed just treemachine (master branch) and all plugins are reporting
21:21 josephwb sweet
21:22 josephwb maybe jonathan did not have everything in sync?
21:23 jimallman possible, i guess, but he was pretty careful. and he pulled a fresh deployed-systems, so it should have pointed to treemachine > master unless he has local modifications (unlikely)
21:23 josephwb that was a lot of running around for mothing.
21:23 jimallman yes, gotta love the false alarms.
21:23 jimallman i suspect some failure by Java to detect and recompile stale stuff, or by neo4j to recognize all plugins..
21:24 jimallman either that or we missed a build error message somewhere, something bad enough to block several plugins
21:24 jimallman hm, i’ll check the neo4j logs for treemachine. maybe there’s something hidden there..
21:27 josephwb germinator/test.sh api.opentreeoflife.org
21:27 josephwb Successes: 10 Failures: 0
21:28 blackrim joined #opentreeoflife
21:30 jimallman yeah, ot20: /home/opentree/neo4j-treemachine/data/log/console.log shows all plugins loading most of the time, but two restarts(?) where only three “core” plugins loaded.
21:30 jimallman no indication as to why, which makes me think either the compiled output for these plugins didn’t make it into place, or some kind of plugin registration didn’t occur normally.
21:36 jimallman i’ve updated the issue for jonathan, going to grab a snack...
22:29 kcranstn joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary