Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-09-12

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:27 josephwb joined #opentreeoflife
02:02 josephwb pull request sent
02:03 josephwb i imagine there will be discussion on the 1) names and 2) number of things returned
02:13 josephwb joined #opentreeoflife
02:45 towodo josephwb, that server is just sitting there mostly unused (I use it for taxonomy development). just say the word
02:46 towodo it’s aging a bit (2010?) so there could be glitches with software installs, but it’s a fine machine
02:49 josephwb thanks towodo
02:50 josephwb i am not getting into my machine today
02:50 josephwb i fear there is some serious hardware failure
02:51 josephwb not experienced with linux diagnostics
02:51 josephwb oh, you're not here…
02:57 josephwb hey jimallman
02:57 josephwb how are fossils treating you?
03:06 jimallman hi josephwb!
03:06 josephwb hiya
03:06 jimallman it’s going pretty well. i’ve just closed the last known bug. woot!
03:07 josephwb sweet
03:07 josephwb last *known* tho…
03:07 josephwb maybe people have not been looking hard enough?
03:09 jimallman that is entirely possible (likely)
03:09 josephwb hah
03:10 josephwb i'm sure you are on top of it all
03:30 * jimallman is bracing for the unknown unknowns…
03:35 josephwb1 joined #opentreeoflife
04:36 josephwb joined #opentreeoflife
04:39 josephwb joined #opentreeoflife
06:05 travis-ci joined #opentreeoflife
06:05 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#607 (alert-clogged-push - 1ffd535 : Mark T. Holder): The build passed.
06:05 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/compare/bf01f63c3bc1...1ffd53552a39
06:05 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/35089865
06:05 travis-ci left #opentreeoflife
09:33 mtholder joined #opentreeoflife
11:10 mtholder joined #opentreeoflife
12:16 kcranstn joined #opentreeoflife
12:23 josephwb joined #opentreeoflife
12:40 towodo joined #opentreeoflife
12:52 kcranstn mtholder - there was some discussion recently about the supporting files (over IRC, I think?)
12:54 kcranstn http://irclog.perlgeek.de/opentreeoflife/2014-09-04
12:54 josephwb towodo: i get that you want the path to the root, but how do you want this?
12:54 josephwb a string?
12:54 josephwb a list?
12:55 josephwb other?
12:55 josephwb in treemachine we print something like:
12:56 josephwb Stellula_calliope->Stellula->Trochilidae->Apodiformes etc.
13:00 mtholder thanks kcranstn
13:04 towodo a list. josephwb, I was thinking about this, and wondering if maybe you want additional information beyond just the node id. i wonder if it’s too soon to be implementing this feature - it wasn’t part of the API we talked about last week and it seems to need additional design work. is there a reason you’re hot to work on this now
13:04 towodo ?
13:06 josephwb towodo i was just perusing the issues
13:06 josephwb it is an easy task
13:07 towodo i say assign it to me and move on
13:07 josephwb i am happy to do it
13:07 josephwb are you familiar with neo4j path traversals?
13:08 kcranstn mtholder - link to your ievobio peyotl slides?
13:11 towodo no
13:11 towodo there is not much point in implementing something that turns out not to be what anyone wants
13:13 towodo the logically coherent thing is to return a list of node ids, but then the first thing a client will want to do is get information about each node
13:13 towodo since we don’t know what information will be wanted (e.g. by the curator app) it seems premature to implement the service
13:14 towodo I think there should be a little circuit of design proposal, review, discussion, then implementation
13:14 kcranstn I am worried about setting ourselves up for people hammering the API by trying to do traversals
13:16 josephwb kcranstn: i think some things can be baked into the graph, negating the need for traversals. this is once such instance.
13:17 josephwb just so you know, a number of the services do indeed perform traversals
13:17 kcranstn using v2 api methods?
13:17 josephwb e.g. induced_subtree
13:17 josephwb ]yes
13:18 josephwb in both treemachine and taxomachine
13:18 josephwb but they are super fast
13:19 kcranstn I am thinking about that query on the google doc, where the Andrea didn’t want to download a subtree, she wanted to repeatedly query the APIs for ancestor / descendant relationships
13:21 josephwb graph/node_info doesn't currently return this information, but it could
13:22 towodo seems like node_info should at least return the id of the parent node.
13:23 towodo there’s no way to prevent people from doing stupid things with the API other than technically (request throttling)
13:24 towodo I don’t really want to request API keys but it may become necessary to enable communication with the requestor
13:24 kcranstn we can discuss at hackathon
13:30 kcranstn towodo - you online on skype?
13:31 josephwb towodo: node_info is fairly general: you can query nodes not in the synthetic tree. when you say report the parent node, you mean the parent node in the synthetic tree (if present)? or taxonomic parent node? or source tree parent node? etc.
13:31 kcranstn Skype call starting now
13:31 towodo my skype window is open, yes
13:31 towodo no activity in it though
14:20 kcranstn mtholder - link to your ievobio peyotl slides?
14:26 josephwb cody's documentation generator is awesome!
14:26 kcranstn indeed!
14:27 mtholder http://phylo.bio.ku.edu/slides/ot.html
14:27 kcranstn thanks, mtholder!
14:27 mtholder kein Problem
14:27 kcranstn are those in github, too?
14:28 mtholder generated from running https://github.com/OpenTreeOfLife/phylesystem-api/blob/master/docs/build-presentations.sh
14:28 mtholder (some dependencies on that build...)
14:30 mtholder oops, kcranston. those are my slides from the meeting with the ad board. Hold on...
14:30 mtholder kcranstn ^
14:31 mtholder http://phylo.bio.ku.edu/ievobio/index.html has links to my talk and EJ's
14:31 kcranstn thanks!
14:36 guest|46004 joined #opentreeoflife
14:37 kcranstn1 joined #opentreeoflife
14:37 kcranstn1 testing from scrollback
14:37 kcranstn1 left #opentreeoflife
14:43 tavooca_ joined #opentreeoflife
14:43 tavooca_ hi
14:49 towodo I’ll be in transit for the next hour, then will work on new taxomachine db install on production
15:12 kcranstn hey tavooca_ - who’s this?
15:14 travis-ci joined #opentreeoflife
15:14 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#609 (study-file - f3668c8 : Mark T. Holder): The build passed.
15:14 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/commit/f3668c8f8f19
15:14 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/35128811
15:14 travis-ci left #opentreeoflife
15:22 travis-ci joined #opentreeoflife
15:22 travis-ci [travis-ci] OpenTreeOfLife/phylesystem-api#610 (study-file - fb4266b : Mark T. Holder): The build passed.
15:22 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem-api/compare/f3668c8f8f19...fb4266b158b7
15:22 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem-api/builds/35129707
15:22 travis-ci left #opentreeoflife
15:29 jimallman kcranstn: i’m inclined to move forward on the proposed UI changes for OTU mapping. thoughts?
15:29 jimallman https://github.com/OpenTreeOfLife/opentree/issues/407
15:30 jimallman i think we’re in general accord now, that the user will control starting / pausing OTU mapping (vs. “always on”)
15:30 kcranstn sound good
15:30 kcranstn sounds
15:55 mtholder joined #opentreeoflife
16:25 towodo joined #opentreeoflife
16:32 tavooca_ q
16:33 towodo taxomachine database copy has been in progress for 1:40, not done yet
16:33 kcranstn yikes
16:36 towodo I can’t proceed with the prod update until it’s done.  looks like it’s done 5.5G out of 6.5G
16:36 kcranstn why is it so big?
16:36 towodo I don’t know. seems neo4j is extremely space-intensive
16:37 towodo that’s 6.5G compressed tarball… after deployment it’s much bigger
16:47 towodo for comparison, OTT (the only content in taxomachine) is a 119M tarball… factor of 50.  some of that is index, but still rather extreme.
17:08 kcranstn joined #opentreeoflife
17:13 jimallman kcranstn: we had some prior discussion about neo4j db sizes: http://irclog.perlgeek.de/opentreeoflife/2014-07-21#i_9058556
17:14 jimallman they do not seem overly concerned with sprawl, and so far i haven’t found good tips on compacting or recovering space in a neo4j db
17:20 josephwb will there be room on ot10 for the treemachine DB (whenever it gets built)?
17:35 towodo joined #opentreeoflife
17:36 towodo I don’t know, log in and check. we can delete ~/download/treemachine.tar.tgz if necessary (since you guys have a copy), but the amount of space needed beyond that will be as given by du -s -m neo4j-treemachine
17:37 towodo jimallman, you there? i’m ready to do the prod update
17:42 jimallman hi, here now
17:44 jimallman towodo: ^
17:47 towodo_ joined #opentreeoflife
17:47 towodo_ can you hear me now?
17:47 jimallman towodo_: load and clear
17:47 towodo_ ok.
17:49 towodo so I have a new taxomachine db uploaded, and all the master branches are ready to go, so in principle this should be easy
17:50 towodo so can we put up the notice on production saying there’s work in progress?
17:50 towodo jimallman ^
17:50 mtholder joined #opentreeoflife
17:51 PEM joined #opentreeoflife
17:51 jimallman sure, will hand-wave this with “back later today”…
17:51 towodo yep
17:52 towodo back in a minute, then will start taxomachine db
17:55 * jimallman has blocked curation with this message: “Study creation and editing are disabled while we upgrade to the latest features and APIs. Please pardon the inconvenience. We expect to be back online for editing studies later today (Friday, Sept 12).”
17:57 kcranstn joined #opentreeoflife
17:59 towodo josephwb, if there’s not enough space we can set up a new dev server
18:00 josephwb it will be a while (a day). synthesis isn't even started. waiting for someone to look at my machine
18:00 towodo although I’ve just learned it takes 2 hours to copy the taxomachine db tarball from MIT to AWS
18:00 towodo you don’t want to use my server?
18:00 josephwb maybe
18:00 josephwb should know anytime now
18:01 josephwb due to show up
18:01 towodo that means it will take 4 hours to copy the treemachine db.  so setting up a new server will take at least 7 or 8 hours
18:02 towodo not sure how much time i’ll have over the weekend
18:02 josephwb last time i copied a DB it was surprisingly fast
18:02 josephwb to dev
18:02 towodo copying to ot10 (MIT) is fast because MIT has a firehose of an internet connection
18:02 josephwb ok
18:02 towodo it’s going to AWS that’s slow.
18:02 josephwb ok
18:02 towodo I can look into premium net speed but I think it’s $$$
18:02 josephwb yes, copying to dev took minutes
18:03 josephwb disconcertingly fast (I thought it didn't work)
18:03 towodo I could run out and buy a 1T disk to put on ot10, that wouldn’t be too hard
18:04 kcranstn note that AWS has a service where you can mail them a disk and they will upload
18:04 kcranstn ;)
18:04 josephwb sweet!
18:05 towodo I really want a new synthetic tree, so want to do what I can to help
18:08 towodo if we want the new disk on ot10 before I leave for AA I need to do that tomorrow, so we need a decision on that today
18:11 towodo kcranstn, really doing ‘continuous deployment’ in the current architecture is going to mean frequent downtime
18:11 towodo I will need to rethink how deployment works in order to shorten downtime on updates
18:11 kcranstn ok
18:12 towodo ok, i have updated the web apps and treemachine on production, proceeding to oti and phylesystem now
18:23 towodo ok, jimallman, mtholder, I’m getting
18:23 towodo Calling "http://127.0.0.1:7478/db/data/ext/studies/graphdb/index_studies" with data="{'urls': ['http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1968.json']}"
18:23 towodo Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1968.json
18:23 towodo 404 Client Error: Not Found
18:24 towodo looks like I need to change the indexing script to use a different URL
18:24 jimallman hm, this URL works from here, but yes it seems fishy
18:25 towodo oh you’re right.  why shouldn’t this work then?
18:25 jimallman this URL meaning “http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1968.json”, which (from an external request) changes to “http://api.opentreeoflife.org/phylesystem/default/v1/study/pg_1968.json”
18:25 jimallman so maybe it’s about the domain name called (modified by apache config)?
18:26 towodo i updated phylesystem-api …
18:26 jimallman in any case, let’s look at where these (v1) URLs are defined
18:26 towodo but if the URL works… why doesn’t the indexing script work?…
18:27 jimallman right. i was thinking “because it was called from localhost”, but that doesn’t really make sense
18:27 towodo I didn’t redeploy oti plugin.  will try that
18:28 towodo no change.
18:31 jimallman ah, i’m reading submit_indexing_request in index_current_repo.py...
18:31 towodo wget of that URL on ot20 works
18:31 jimallman it’s the *primary* URL that’s failing: http://127.0.0.1:7478/db/data/ext/studies/graphdb/index_studies
18:31 mtholder sorry. just checkec back in on this (forgot to log out).  Why is there the v1/../default/v1 instead of just v1   ?
18:32 jimallman that’s returning the 404, i mean
18:32 mtholder does there need to be an oti in there?
18:32 towodo oh.
18:32 mtholder nm
18:32 towodo ../default is annoying but a red herring
18:32 jimallman mtholder: this is weird construction based on “API base URLs”, shouldn’t be a problem (and can probably go away as we move to more sensible v2)
18:33 jimallman i assume we’ll need to test the :7478 URLs from devapi (localhost)… heading there now
18:35 towodo I sent email to Cody telling him we’re debugging this
18:36 towodo we can just try one of these calls on ot10, they’re harmless
18:36 jimallman i’ll bet this would actually work (from your *local* oti/ directory): ./index_current_repo.py http://devapi.opentreeoflife.org/oti http://devapi.opentreeoflife.org/api
18:37 towodo I’d rather not launch that right now… takes 40 minutes to run
18:37 towodo can we try just one of the calls? curl using data from the error message?
18:37 jimallman right. i just tried wget http://127.0.0.1:7478/db/data/ext/studies/graphdb/index_studies, and it did respond
18:37 jimallman (not a 404, anyway)
18:38 mtholder gotta sign off for a bit...
18:38 towodo ok.
18:38 towodo I’ll try it on ot20
18:38 towodo Not found.
18:40 jimallman curiouser…
18:41 towodo I’m going to update oti plugin on dev
18:42 towodo no i won’t… git tells me that dev, github, and prod are all at the same commit
18:42 towodo doesn’t make sense, prod and dev are running the same code
18:43 * jimallman is looking even closer at the call from submit_indexing_request… checking the apache lots to see what’s actually being called.
18:44 * jimallman smacks head.. these calls aren’t going to apache, but neo4j
18:46 towodo oti neo4j is not listing ‘studies’ as a plugin
18:48 towodo oti neo4j on dev does list ‘studies’ as a plugin
18:50 towodo assuming it’s a deployment bug.  recompiling oti on ot20
18:52 kcranstn joined #opentreeoflife
18:53 jimallman towodo: ah, my call ‘wget http://127.0.0.1:7478/db/data/ext/studies/graphdb/index_studies' above was on ot10 (devapi) and not not10
18:53 towodo on ot20, the plugin was written on sep 5, but on ot10, it was written sep 11
18:53 jimallman hmmm. but from the same commit?
18:54 towodo no.  ot10 is current (sep 11 / master).  ot20 was out of date.  somehow didn’t get rebuilt just now
18:55 towodo ok… the plugin is there…
18:56 jimallman yes, and my wget test is working now
18:56 towodo retrying index.
18:57 towodo so I’ve got some weird bug in deployment for oti
18:57 towodo I guess it’s time to switch to ansible
18:58 towodo working now.
18:59 jimallman victory!  (weeps softly)
19:03 towodo ah. i see the bug.
19:07 towodo basically the same as the taxomachine plugin build bug. it refreshed the repo without recompiling the plugin
19:07 towodo if it were all done with ‘make’ it would work
19:10 jimallman noted
19:12 towodo josephwb, if there’s a chance you’ll have a new tree ready before next friday, i’d like to go ahead and buy that disk. is there?
19:21 scrollback joined #opentreeoflife
19:26 josephwb yes
19:26 josephwb hopefully will have a tree by tomorrow
19:29 kcranstn what’s the mailing address for smithlab?
19:42 towodo ok, update is done.
19:42 towodo v2 api should be on line at api.opentreeoflife.org
19:42 jimallman shall i restore curation?
19:43 towodo just going to do one or two tests first
19:44 towodo ok, seems fine
19:44 towodo sure, restore curation, and i’ll send email
19:47 jimallman done (curation is back online)
19:48 towodo thanks.  some day you need to teach me how to do that myself
19:48 kcranstn me too
19:59 jimallman i put some notes for this online… just a sec...
20:02 jimallman https://github.com/OpenTreeOfLife/opentree/tree/master/deploy#notifying-users-of-scheduled-downtime
20:03 jimallman this also describes how to edit a maintenance page (in the github-pages site) where we can divert all traffic
20:04 jimallman i can add more concrete, step-by-step instructions on where to find these files, how to restart apache properly, etc.
20:16 towodo i’m thinking maybe more deployment scripts.  push.sh -c … suspend-curator
20:16 towodo commands that is (that invoke scripts)
20:19 mtholder jimallman, towodo : is it OK for me to deploy some branches of phylesystem-api (study-file branch) and peyotl (supp-file branch) to ot10 ?
20:19 towodo yes, that’s what ot10 is for
20:21 jimallman towodo: deployment command sounds sensible. not sure where we should define the message text.
20:21 towodo yes, that’s a problem. could be a file name argument passed on the command line
20:21 jimallman it seems weird for a server-config file.. or specify a separate text-file on the command (sort of like naming a .tgz file for database commands)
20:22 jimallman ok, sounds like a plan. i’ll make a ticket for this...
20:26 jimallman https://github.com/OpenTreeOfLife/opentree/issues/435
20:28 kcranstn what’s the quirk with neo4j and post vs get?
20:28 kcranstn when I look at the neo4j docs, I see lots of GET examples http://docs.neo4j.org/chunked/stable/rest-api.html
20:32 towodo you can do get to get the metadata, but not to invoke a plugin method
20:32 kcranstn ah, ok (Q via twitter)
20:33 mtholder GET on a plugin method gives you the help for the method, I believe.
20:33 towodo http://docs.neo4j.org/chunked/milestone/server-plugins.html
20:34 kcranstn thanks
20:50 gaurav_enwp joined #opentreeoflife
20:51 gaurav_enwp left #opentreeoflife
21:05 josephwb joined #opentreeoflife
21:55 josephwb joined #opentreeoflife
23:06 ilbot3 joined #opentreeoflife
23:38 scrollback1 joined #opentreeoflife
23:38 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
23:41 ilbot3 joined #opentreeoflife
23:42 scrollback1 joined #opentreeoflife
23:42 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
23:59 scrollback1 joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary