Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-06-24

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:01 kcranstn joined #opentreeoflife
01:27 5EXABI0RC joined #opentreeoflife
01:36 5EXABI0SQ joined #opentreeoflife
01:38 scrollback3 joined #opentreeoflife
02:13 scrollback joined #opentreeoflife
04:27 scrollback4 joined #opentreeoflife
06:31 scrollback joined #opentreeoflife
06:55 scrollback joined #opentreeoflife
12:19 jar286 joined #opentreeoflife
12:36 kcranstn joined #opentreeoflife
12:59 kcranstn joined #opentreeoflife
13:07 mtholder joined #opentreeoflife
13:08 mtholder jimallman, jar286 is https://tree.opentreeoflife.org/ working for you?
13:08 mtholder time curl 'https://tree.opentreeoflife.org/' is taking about 2 minutes for me.
13:10 jar286 probably waiting for api.etc
13:10 jar286 host is not busy
13:11 jar286 java 195% CPU on api.
13:11 jar286 treemachine is the culprit
13:12 mtholder yeah. I was surprised that would cause a slow cURL call.
13:13 jar286 still busy
13:13 jar286 nothing on neo4j console log
13:15 jar286 oddly, I don’t see any http log file
13:16 jar286 most recent one is from two days ago
13:16 jar286 this suggests neo4j is broken
13:17 jar286 I’m confident that if I just restarted treemachine everything would work again
13:17 jar286 but want to understand the failure a bit better
13:18 kcranstn joined #opentreeoflife
13:18 mtholder yeah. it would be nice to have logs that would let us diagnose this sort of thing.
13:18 kcranstn thanks, jar286
13:19 jar286 console.log is 853M, mostly useless (submitted issue)
13:21 kcranstn not much justification on that issue
13:21 kcranstn (i.e. not clear why this is important)
13:21 jar286 typed carriage return without it realizing that meant submit
13:22 kcranstn ha
13:22 jar286 there’s a second log file, this one in graph.db/ instead of log/
13:22 kcranstn I am going to be offline for a few hours this morning
13:23 jar286 most recent message is from yesterday and looks routine, but hard to tell
13:23 jar286 ok
13:24 jar286 time to set up nagios
13:24 jar286 (soon)
13:28 jar286 the log files are not being helpful.  all we know is that treemachine is running flat out for a very long time without generating any output.
13:28 jar286 I’m going to  stop & restart treemachine  (and move the old logs out of the way in between)
13:29 kcranstn https://mail.google.com/mail/u/0/#inbox/14e25bf0d34cb535
13:29 kcranstn oops
13:29 kcranstn https://github.com/OpenTreeOfLife/opentree/issues/658
13:29 kcranstn mark doesn’t just want a response page, but a fancy timer… ;)
13:30 mtholder indeed.
13:31 mtholder 'twould be cool if we had a "services status page" that we could point people to, when these things happen.
13:31 kcranstn agreed
13:31 jar286 open nagios status page
13:35 jar286 restarting neo4j treemachine
13:36 jar286 [parsererror] SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
13:36 jar286 "message" : "No such ServerPlugin: \"tree_of_life\"",
13:37 jar286 stopping treemachine
13:37 jar286 starting treemachine
13:37 jar286 still   "message" : "No such ServerPlugin: \"tree_of_life\"",
13:38 jar286 attempting plugin reinstall, hang on…
13:39 jar286 ./push.sh -c ../../deployed-systems/production/api.config treemachine
13:40 jar286 much better now.
13:40 jar286 weird.  reinstalling the plugin shouldn’t have made any difference, but it did.
13:41 jar286 tree.opentreeoflife.org is back in business
13:52 mtholder joined #opentreeoflife
13:52 mtholder thanks
14:53 mtholder joined #opentreeoflife
15:37 mtholder joined #opentreeoflife
16:01 mtholder joined #opentreeoflife
16:47 kcranstn joined #opentreeoflife
16:54 mtholder joined #opentreeoflife
16:58 kcranstn https://plus.google.com/hangouts/_/calendar/OGI4ZTA3aGVvZDM2OGloNGl1YjloYnNtcGdAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ.1ofaiee416e4mcq3bp7kkkd0mk?authuser=0
17:00 jar286 joined #opentreeoflife
17:01 kcranstn jar286 - you joining the hangout?
18:03 jar286 kcranstn, my memory was that you were going to set up nagios monitoring (https://trello.com/c/49dLpVfA).  if you want me to take this over let me know.
18:04 kcranstn I was going to set that up on nescent servers, but that isn’t going to happen now
18:04 kcranstn what are our hosting options for nagios?
18:10 jar286 EC2: existing instance or new instance; Creative Commons: ashby (ot10), varela, norbert;  whatever Mark or Stephen might have (although those would be behind perimeter security, so not easy to modify)
18:10 jar286 norbert is already running nagios, I think
18:11 jar286 oh yeah, there’s also mumble.net, which ran nagios at one point
18:11 jar286 and just got an os upgrade, so fresh and clean
18:21 kcranstn cool
18:21 kcranstn if norbert is already running nagios, does that make the most sense?
18:25 jar286 well… not sure… I’ve been wanting to do a fresh os install on norbert, the one it’s running is from 2007 and hasn’t had security updates
18:26 jar286 varela would probably be better, or a new EC2 instance (not sure how are budget looks?)
18:26 kcranstn what is varela?
18:27 jar286 ashby, varela, and norbert are all left over from science commons. physically they reside in a data center at MIT near my office
18:28 jar286 varela has 40G, 2 SSD disks, and one 2T spinning disk
18:28 jar286 40G RAM that is
18:28 jar286 nice machine, I use it for taxonomy work
18:29 kcranstn ok, sounds good to me
18:29 jar286 running wheezy, so not totally up to date, but I can fix that
18:29 jar286 ok, I’ll install it & do some basic config and give you an account
18:29 kcranstn thanks
18:33 kcranstn did you see my message on the other channel?
18:35 jar286 yes, tried to reply and it said you weren’t online.  Will close channel, re-open, and try again
18:35 jar286 not connecting, going to gchat
20:10 kcranstn joined #opentreeoflife
23:08 jar286 joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary