Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2014-02-05

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:21 mtholder joined #opentreeoflife
00:21 mtholder joined #opentreeoflife
00:29 kcranstn joined #opentreeoflife
00:41 blackrim joined #opentreeoflife
00:42 blackrim josephwb: you watching ken ham?
00:42 kcranstn I am!
00:42 kcranstn my head explodes
00:42 blackrim hehe, fun stuff
00:43 kcranstn at nescent. with beer
00:43 blackrim nice! i am writing a talk. you are having more fun.
00:43 kcranstn I am also writing a talk
00:43 blackrim after this talk, i might have to renounce open tree. it should be open orchard
00:44 blackrim evolution = "get rid of spare cats — get rid of spare kids"
00:45 kcranstn it's all historical science, anyway
00:45 blackrim bill nye better represent
00:50 kcranstn he's pretty awesome
01:04 kcranstn "traditional fish sex"
01:04 blackrim yeah, we all know about that
01:05 blackrim good old fashioned, American, traditional, fish sex
01:20 dukeleto joined #opentreeoflife
01:27 jimallman kcranstn: i'm doing minor bug fixes, and i'd like to (carefully) push and test on dev this evening. do you mind?
01:27 kcranstn that's fine
01:27 jimallman i'll test whatever i touch!
01:41 blackrim http://s1.hubimg.com/u/4679516_f520.jpg
01:52 josephwb i have the computer hooked up to the tv, so couldn't comment.
01:53 josephwb "traditional fish sex" was definitely a highlight. nothing "kinky".
01:53 josephwb wish bill would not argue bible facts. stick to science
01:53 mtholder joined #opentreeoflife
01:54 blackrim yeah, not worth arguing about the bible in this forum (there are others that could do that better)
01:54 josephwb eugenie scott would have done better.
01:54 blackrim yeah, not a celeb though and they (answersingenesis) wanted a celeb
01:55 josephwb word
01:56 kcranstn joined #opentreeoflife
01:57 josephwb bill could have done a lot better on simple things.
01:58 josephwb didn't mention genetics.
01:58 josephwb could have torn the dog argument apart.
01:58 josephwb and what about the Theobald paper? come on, bill!
01:59 blackrim yeah, i think the genetics are lacking in both
02:00 blackrim i really enjoyed the orchard bit. is there a beast model for that?
02:00 josephwb hah!
02:01 josephwb i wish bill had spent more time on the predictive ability of science. despite being "historical science", evolution makes predictions.
02:01 josephwb had the one example, but didn't make the srgument very well
02:02 blackrim yeah, it is a good point to make (though HAM suggests the bible predicts — but nothing more apparently because he is only talking about things that we know from "observable science")
02:10 josephwb blackrim: are you doing the entire synthesis, or just plants?
02:10 josephwb i have metazoa going; should be done any minute.
02:10 josephwb my taxonomy db was corrupted somehow
02:10 blackrim i was doing just plants. there are some issues but they aren't going to the base of life. i will need to do some digging
02:12 josephwb i think, through some of the vetting, i had more than one synthetic tree in the db, which was messing up extracting "the" tree
02:12 blackrim i see
02:13 josephwb i'll just go ahead and start the whole thing when metazoa completes, with the hope that karen will have something for thursday
02:14 blackrim sounds good
02:14 josephwb kcranstn: do you want a newick of the taxonomy tree?
02:15 blackrim i think she will want the database on dev server
02:16 kcranstn I'd like something more up-to-date on the dev server
02:16 josephwb ok
02:16 kcranstn thanks!
02:37 jimallman kcranstn: as i suspected, the reason we can't load most studies in the curation dashboard is that oti (and thus the study list) points to the complete phylesystem, while the API is set to load and save to phylesystem_test. i'm going to try reconciling them... which repo (phylesystem or phylesystem_test) would you prefer?
02:38 jimallman i'm not sure if oti *can* point to phylesystem_test, investigating now...
02:40 kcranstn I think we want to point to phylesystem
02:41 jimallman ok. i do see a config argument for the push scripts (OPENTREE_STORE) that should do the trick...
02:47 ilbot3 joined #opentreeoflife
02:47 Topic for #opentreeoflife is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife | http://irclog.perlgeek.de/opentreeoflife/today
02:53 jimallman kcranstn: OK, i'm able to edit all(?) studies now, which is nice.
02:53 jimallman now i'm working on the other study-list glitches we discussed...
02:53 kcranstn yay, thanks
02:53 kcranstn I am writing my talk...
03:52 kcranstn joined #opentreeoflife
04:00 Guest___ joined #opentreeoflife
04:06 Guest_ joined #opentreeoflife
04:37 kcranstn joined #opentreeoflife
04:48 kcranstn joined #opentreeoflife
04:48 mtholder joined #opentreeoflife
04:49 mtholder Hi, jimallman: I just pushed a refresh of studies from phylografter. Still the old syntax, but now with the extra taxon name info in the meta elements.
04:49 jimallman cool! can't wait to see this..
04:50 jimallman i was just cleaning up the study list, actually, so this is well timed.
04:50 jimallman when should i commence re-indexing the oti?
04:50 mtholder it was a very large commit.
04:51 jimallman mtholder: i'll bet! so we're ready to re-index (oti)?
04:51 mtholder I'm not sure what the oti procedure is.
04:51 jimallman there's a command in the push (deployment) script, doing it now...
04:51 mtholder I'm afraid that I have not done that before (I'm still hazy on the deployment protocol).
04:57 jimallman i think i've got it. the current setup is a bit tricky, with three different server-config files and different services on different machines. instead of a single push script that orchestrates all machines, we run a uniform script on each machine, with options to use only some of the installed stuff.
04:59 jimallman ...re-indexing now. a few studies are throwing 500 errors (1150, 1155, 1162...) but it might just be due to latency
05:00 jimallman 1257, 1332, 1384...
05:00 jimallman 1482. i don't suppose you see a pattern there..
05:00 mtholder I don't see anything obvious
05:01 jimallman too much console chatter, i'll need to tee to a file to capture these.
05:05 mtholder I'm seeing some errors in those files, but they are the type of errors that are present in a lot of phylografter exports (e.g. non leaf nodes marked with ot:isLeaf value of "true"), so I'm not sure why this would be specific to those studies...
05:11 jimallman mtholder: my hunch is it's not the files, but a bottleneck in the indexing process. it seems to happen in bursts. i'll try tee'ing results to a file and see if the same studies have problems, or if it's random.
05:12 jimallman that indexing run finished with a fair number of errors, but I'm getting decent results in the study list:
05:12 jimallman http://dev.opentreeoflife.org/curator
05:12 jimallman ...so hopefully a failed indexing attempt just keeps the last-known-good data for a study.
05:12 mtholder cool.
05:13 mtholder i'll try to get familiar with the deploy system and the indexing so I can see the console messages...
05:13 jimallman the real meat of it is just a python script, lemme see...
05:13 mtholder i have a snow day tomorrow, so i might get some work done...
05:14 jimallman :D
05:14 jimallman https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/index-doc-store.sh
05:14 jimallman this is run directly by the push script, and it in turn calls index_current_repo.py...
05:15 mtholder do you do all of this on the ot# machines, or do you run these locally on your dev machine as well?
05:16 jimallman standalone use of index_curent_repo.py described here: https://github.com/OpenTreeOfLife/oti/blob/ac360f9e89d4344c8762f5a7f9ad291f60f02059/README.md
05:17 jimallman i generally run web2py and its apps on my local, and use services from ot# machines.
05:17 mtholder great. thanks.
05:22 jimallman mtholder: interesting! it does seem to be the same studies failing (though it looks like #1384 succeeded this time).
05:24 mtholder I have not looked at the oti code-base at all, but I'll try to check it out tomorrow.
05:24 jimallman OK. FYI - it seems oti pulls studies through the OTOL API, and at a pretty good clip. it looks like the API is sometimes return 500 errors (most of the cases noted above), but then it'll do a burst of 400 Bad Request errors.
05:24 jimallman i suspect that might be traffic problems, but it eventually recovers and starts indexing normally again.
05:25 jimallman and on that note, i'm done for the night. i'm going to leave all this stuff alone so Karen will have no surprises, but i'll be here working on my local setup tomorrow.
05:26 jimallman SADLY, i'm not seeing the taxon name in the study list (or JSON list form oti)...
05:26 mtholder that makes sense. I hope to push the new syntax for nexsons, but that will be a branch on phylesystem and should not affect anything that is deployed...
05:27 mtholder "that makes sense" was in relation to previous msg
05:27 jimallman joys of irc
05:35 kcranstn joined #opentreeoflife
07:24 jimallman joined #opentreeoflife
08:40 kcranstn joined #opentreeoflife
09:42 kcranstn joined #opentreeoflife
11:54 josephwb joined #opentreeoflife
13:03 josephwb joined #opentreeoflife
13:05 kcranstn joined #opentreeoflife
13:55 josephwb joined #opentreeoflife
13:57 kcranstn joined #opentreeoflife
14:09 josephwb joined #opentreeoflife
14:15 josephwb joined #opentreeoflife
14:19 sckott joined #opentreeoflife
14:56 mtholder joined #opentreeoflife
15:34 jimallman joined #opentreeoflife
15:40 kcranstn jimallman - thanks for getting the dashboard up. Looks great!
15:41 kcranstn how do we delete the "please delete" study?
15:41 jimallman Sorry about taxon names -- hopefully we'll have those soon.
15:41 kcranstn np
15:42 jimallman there's a 'Delete Study' button in the curation tool (i didn't want to make it TOO easy from the main list)
15:43 jimallman (ok, this is embarrassing. i don't think that button works yet.)
15:44 jimallman let me see if I can do it via the API (in curl)
15:45 kcranstn :)
15:50 jimallman here's the incantation in curl: curl -X GET 'http://ashby.csail.mit.edu/api/v1/study/1019?auth_token=26b5a59d2cbc921bdfe04ec0e9f6cc05c879a761'
15:50 * jimallman is fat-fingered today
15:50 jimallman curl -X DELETE 'http://ashby.csail.mit.edu/api/v1/study/1019?auth_token=26b5a59d2cbc921bdfe04ec0e9f6cc05c879a761'
15:51 jimallman it seems this works, but only in a personal WIP branch? needs to merge back to master to change this list.
15:54 jimallman i'm attempting a manual merge (via pull request on Github), but need to resolve coflicts
15:54 jimallman conflicts, which is odd.
15:58 jimallman OK, did a manual delete from Github, re-indexing oti now...
16:03 jimallman gone from the list! and bugs noted
16:04 mtholder jimallman: it looks like phylografter is emitting ot:ottTaxonName in the node objects, not the otu objects. I'll submit an issue (assuming that is not right), but thought I'd mention it in case you wanted to use that data in its current (presumably wrong and temporary) position
16:05 travis-ci joined #opentreeoflife
16:05 travis-ci [travis-ci] OpenTreeOfLife/phylesystem#103 (jimallman_study_1019 - a385a88 : Jim Allman): The build passed.
16:05 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem/commit/a385a88a90ce
16:05 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem/builds/18279339
16:05 travis-ci left #opentreeoflife
16:12 travis-ci joined #opentreeoflife
16:12 travis-ci [travis-ci] OpenTreeOfLife/phylesystem#104 (master - e206d23 : Jim Allman): The build passed.
16:12 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem/compare/af7f2b6a0787...e206d2351d7d
16:12 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem/builds/18279867
16:12 travis-ci left #opentreeoflife
16:15 jimallman mtholder: i'm not sure tjat
16:15 jimallman gah! i'm not sure that's wrong, actually... lemme see..
16:16 jimallman that might be OK
16:17 jimallman my main interest was in a new metatag on studies, to:focalCladeOTTTaxonName
16:20 mtholder I'm seetin that in the study meta, at least for some studies (230, for example...)
16:20 mtholder warning: there is an ESC (\u001b) in an original label for that study...
16:32 jimallman whoa, i'm seeing some seriously fubar quoting in these Nexson files... scroll through some of the results here:
16:32 jimallman https://github.com/OpenTreeOfLife/phylesystem/search?q=ot%3AfocalCladeOTTTaxonName&ref=cmdform
16:32 jimallman https://github.com/OpenTreeOfLife/phylesystem/search?q=ot%3AfocalCladeOTTTaxonName&ref=cmdform
16:33 jimallman mtholder: ^^ lots of encoded '"' mixed in with normal ones...
16:34 mtholder you don't like that? it's kind of quirky
16:35 jimallman lol
16:35 jimallman it's got moxie, i'll grant you that
16:36 mtholder keeps you on your toes. or your "toes" or you "toes" or hell even your "toes"
16:36 mtholder the files don't look like that in the phylesystem repo AFAIK
16:37 mtholder must be from oti or in the wrapping of the oti results for calls that ricochet off the api layer
16:38 jimallman but the link is to a Github search results page...
16:38 jimallman maybe i'm seeing a glitch in their rendering code.. :-/
16:39 jimallman yes, looks like that was a false alarm. here's the first actual study, note that to:studyPublicationReference is intact:
16:39 jimallman https://github.com/OpenTreeOfLife/phylesystem/blob/244291bb2ffcafd57445f5cda807ec4a55ccec43/study/1803/1803.json
16:39 jimallman my bad
16:40 mtholder ahh, i was interpreting the "search" as an oti call. didn't even click that I was seeing that on github.. more coffee needed here..
16:41 kcranstn it's early in kansas
16:42 mtholder not really, but thanks for that. Feels early to me. Snow day => sleep in.
16:43 mtholder and I have shoveled a lot of what appears to be very cold cocaine. I have not investigated it closely. could be this "snow" that everyone is talking about.
16:51 mtholder jimallman what do you think of one more NexSON tweak (I know I said "finalized NexSON?" in an email header)... Instead nodes having "^ot:isLeaf": true or "^ot:isLeaf": false properties, we just emit the tree with a "node" array "leaf" array
16:52 mtholder "node" array anad a "leaf" array
16:52 mtholder anad = and
16:52 mtholder dammit
16:52 jimallman whoa
16:52 mtholder is that a good whoa?
16:53 jimallman fwiw, that's harder to wrangle on the client side. it means checking for the existing of a value in a (potentially huge) array, which is slow as heck in JS
16:53 jimallman i can adapt if it's a big difference in file size,
16:54 mtholder I meant the same data structures, just breaking the nodes into two arrays instead of one big node array.
16:54 mtholder If it is harder on the client side, we should not do it. I thought it might be easier.
16:54 jimallman by adapt, i mean making a JS object with all values or IDs as properties, then doing a quick test of "if targetID in quickLookup..."
16:55 jimallman pooh, so a node in "leaf" would not also be in "node"... hm
16:55 jimallman s/pooh/oooooh
16:55 mtholder yup. just a cheesy way to use the name to give you the isLeaf bit of info.
16:56 jimallman again, it adds a few kinks to client-side logic, but not bad if it makes for much smaller files.
16:56 jimallman what's the proper name for a non-leaf node?
16:56 mtholder "internal node" is what I usually say
16:57 jimallman yes, or "parent" i suppose could work (but implies context)
16:57 mtholder "ancestor" is better than parent for the reason that you mention.
16:57 travis-ci joined #opentreeoflife
16:57 travis-ci [travis-ci] OpenTreeOfLife/phylesystem#105 (hbf - 43dc419 : Mark T. Holder): The build passed.
16:57 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem/compare/44df616c365e...43dc4195b643
16:57 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem/builds/18283139
16:57 travis-ci left #opentreeoflife
16:57 mtholder but some trees have ancestral taxa in them
16:58 jimallman mind if i take a quick look at my code?
16:58 mtholder no problem. We really don't have to do this if it will complicate things. just a thought...
17:01 jimallman just looking for wrinkles.. re-rooting a tree, for one, would mean shuffling a lot of nodes between these lists..
17:02 jimallman which could complicate later diffs, though our smart-diff tool would presumably handle that.
17:02 jimallman come to think of it, i should make sure I'm updating isLeaf properties when I'm re-rooting.. not sure it's happening
17:03 mtholder isleaf should not change on rerooting
17:03 mtholder it is only true for the nodes that have no descendants, and we never root on a tip, only on internal nodes.
17:03 jimallman ah, good point. unless someone chooses a leaf and makes it the root..?
17:04 jimallman (i realize that's not a sensible thing to do)
17:04 mtholder btw, phylografter is emitting isLeaf=true for lots of internals I'll submit an issue
17:04 jimallman hm, should i block choosing a leaf as root in the UI? or is it possible someone imports a jumbled tree with arbitrary rooting and needs to do this to make things right?
17:05 mtholder rooting on a leaf is logically possible, but we never assume that we know the ancestor and have it as a leaf in analyses. So I dont' think we'd see it
17:05 mtholder most treeviz programs "bend" the egde leading to a tip, if you reroot at a tip.
17:05 mtholder that means adding a new internal node and a new edge.
17:07 jimallman interesting!
17:08 jimallman my first reaction to splitting leaves from internal nodes (two lists) is that it forces my code to look in both, but in practice it seems I'm usually interested in one group or the other, so it might speed things up here...
17:09 mtholder it would speed things up the tiniest bit for the validator because some most checks apply to one or the other.  but that speed is trivial, and should not factor into our decision.
17:14 jimallman the situation client-side looks about the same: most checks are type-specific, and there would be some speed improvements, maybe noticeable on slow machines.
17:14 jimallman the code changes are scattered around, but not too hard to isolate.. maybe a few hours work with testing.
17:15 jimallman my only other concern would be further drift away from the expected(?) NeXML structure, for someone trying to grok this later.
17:15 mtholder yeah. that is a valid point.
17:15 jimallman once we start down this page, it's tempting to convert trees into a nested hierarchy instead of nodes + edges, etc.
17:15 jimallman s/page/road
17:16 mtholder or an array of pairs of node + edge-to-parent
17:16 jimallman again.. whoa
17:17 mtholder that is how I usually store things for fast access. hierarchies can be cumbersome - the appealing recursive access can be really inefficient.
17:17 jimallman true that
17:17 jimallman i suppose it becomes a question of purpose: what is the point of Nexson? if it's about interchange, and moving NeXML to a more widely supported format, i'd stay close.
17:18 jimallman if it's about making a study easily manipulated with Javascript, while being round-trippable to NexSON, then any mutation that supports easier/faster manipulation is a win.
17:18 mtholder Yeah. I think that close is good. when we were writing nexml, I wish that we had used node and leaf elements.
17:19 mtholder any client of nexml that wants to ignore the distinction can easily ignore the difference and concatenate the lists internally.
17:19 jimallman but see above about easy manipulation. i feel like we made that call by allowing JS primitives in the Nexson
17:19 jimallman re: concatenation, i suppose they'd concatenate while adding ^isLeaf markers to the leaf nodes.
17:20 mtholder in pure nexml there is not isLeaf, the client just has to detect that.
17:21 * jimallman nods
17:21 jimallman i need to test large studies (e.g., 438) on more slow machines, and see if curation is painfully slow. that might push me off the fence.
17:23 mtholder I'll look at the size difference
17:42 travis-ci joined #opentreeoflife
17:42 travis-ci [travis-ci] OpenTreeOfLife/phylesystem#106 (master - e809ff8 : Mark T. Holder): The build passed.
17:42 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/phylesystem/compare/e206d2351d7d...e809ff8f38d0
17:42 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/phylesystem/builds/18286208
17:42 travis-ci left #opentreeoflife
17:44 mtholder the size difference is pretty trivial for our files (which have few trees). 438 is 7.2% smaller. 639 (which has multiple trees) is 8.2% smaller.
17:47 jimallman hmm. maybe not worth the extra work, at least not now.
17:48 jimallman but i guess putting off changes to Nexson would just create confusion later..
17:49 mtholder yeah. we should probably decide and stick with it.  but maybe I should write an email to the larger group...
17:53 mtholder putting the edge-to-parent inside the descendant edge would shave off another 8.5% from study 438. it would be a bigger change to processing code, but potentially snappier because you'd only have to look up the parent of an edge (the child node would be the containing object).
17:53 mtholder inside the descendant *node*
17:55 kcranstn +1 for moving decisions to the mailing list (possibly with nexml-discuss copied)
17:56 mtholder we'd still roundtrip to NeXML. I don't mind adding that list, but certainly not thinking of these changes directly affecting them. though I agree that there are several interested parties on that list...
17:57 mtholder I'm going to shovel more (I'm old and I have to take it in tiny stages), then I'll write a message to opentree and nexml lists.
17:58 kcranstn took me a minute to realize that you meant *literal* shovelling
17:58 mtholder god. I wish I could just: while test -f snow ; do shovel ; done
17:59 jimallman yeah, programming can make other things in life seem really tedious... and hard to undo.
18:02 * mtholder nods, then sighs, then heads into the bitter cold
18:58 josephwb joined #opentreeoflife
19:00 scrollback joined #opentreeoflife
19:12 sckott joined #opentreeoflife
19:22 jimallman sckott: hi!
19:22 sckott Hi!
20:01 sckott jimallman - possible to test out the API yet? only if I install locally, or is there a public address?
20:01 jimallman yes, there's at least one public API server.. i'll post some curl examples
20:04 jimallman here's a simple fetch of a study, using curl:
20:04 jimallman curl 'http://ashby.csail.mit.edu/api/v1/study/2659?auth_token=ANONYMOUS'
20:05 jimallman read-only operation, so anonymous is OK
20:05 jimallman sckott, API docs are here: https://github.com/OpenTreeOfLife/api.opentreeoflife.org/tree/master/docs
20:06 jimallman (of course you'll need to replace the host:port and get your own auth token for most operations)
20:07 kcranstn noting that these docs are all describing access to the data store of input phylogenies, not the synthetic tree
20:07 sckott Awesome
20:07 jimallman we're currently serving the OTOL API from ashby.csail.mit.edu, on port 80
20:08 jimallman kcranstn: good point!
20:10 sckott ok, thanks jimallman
20:13 mtholder joined #opentreeoflife
21:00 sckott Awesome, tried out the curl call, works great
21:35 mtholder joined #opentreeoflife
22:10 kcranstn jimallman - getting all sorts of "Received CTCP 'DCC CHAT chat 1194485288 1024' (to kcranstn) from jimallman" messages
22:10 jimallman kcranstn: sorry, i tried to start a chat a few hours ago..? i guess it keeps knocking.
22:11 jimallman just wanted to compare notes on future availability, FEEDS, etc.
22:11 jimallman (esp. if you had a chance to speak with Hilmar)
22:12 kcranstn no - in a hackathon all this week and rather swamped
22:12 jimallman no problem.. i'll follow up with him directly.
22:12 mtholder joined #opentreeoflife
22:22 mtholder jimallman re the speed of processing on the email thread: but you still have to walk through the entire node array (to get the node ids) before you can start piecing together the tree, right? With preorder, one could start reading the edge array and reading the nodes in a lazy manner. Doesn't matter if your going to build the whole tree, of course, but if you are just building a slice of it.
22:23 mtholder point taken though: it is not a plus for us now.
22:24 jimallman mtholder: i currently walk all nodes up front and stash their node IDs into a quick-lookup object (each id becomes a property, and node becomes its value). after that it's very quick to fetch a node by id, or test to see if it's in a tree, etc.
23:06 jimallman left #opentreeoflife
23:13 kcranstn joined #opentreeoflife
23:36 kcranstn joined #opentreeoflife
23:44 mtholder joined #opentreeoflife
23:46 josephwb joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary