Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2013-10-29

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
04:53 travis-ci joined #otol
04:53 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#97 (local_git - 5a67d2f : Jonathan "Duke" Leto): The build is still failing.
04:53 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/6364b30ed458...5a67d2f8cb74
04:53 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/13191759
04:53 travis-ci left #otol
15:21 jimallman joined #otol
18:16 dukeleto joined #otol
18:18 jimallman dukeleto: hi! i'm around this afternoon if you want to set up a chat
18:19 jimallman oh, just found your email.. responding now.
18:26 dukeleto jimallman: hola
18:27 dukeleto jimallman: i am hitting some odd thing with web2py
18:27 dukeleto jimallman: at first, it was only on Travis CI
18:27 dukeleto jimallman: absolutely no output from web2py. Not an error about the port not being available, nothing
18:28 dukeleto jimallman: then a similar thing happened to me on the dev server
18:28 jimallman hrm, this is on your local server? or where?
18:28 dukeleto jimallman: on norbert
18:28 dukeleto jimallman: aka dev.opentreeoflife.org
18:28 jimallman lt's running for me.. odd: http://dev.opentreeoflife.org/opentree/argus/otol.draft.22@1
18:29 jimallman (double-checked my /etc/hosts, i'm pretty sure i'm looking in the right place)
18:29 dukeleto jimallman: yes, i am talking about trying to start a web2py instance to access my API repo
18:30 dukeleto jimallman: but I guess it should just be symlinked in the applications/ directory?
18:30 jimallman yes, it should be able to run in the same web2py instance as other apps
18:30 dukeleto /dsk/leto/git/opentree/api.opentreeoflife.org
18:30 jimallman unless we're separating it to isolate long-running processes, etc
18:30 dukeleto jimallman: would you mind symlinking it? I don't want to break things and I am still new on this server
18:31 dukeleto jimallman: the symlink "api" is what I have been using
18:31 jimallman sure! i'll need to clear some of the old clutter... and possibly rename some stuff, since i've been calling the app by its full name (api.opentreeoflfe.org), which was dumb..
18:32 dukeleto jimallman: yeah, that leads to really long URLs :)
18:33 jimallman no wait, i was mistaken. the app name is sensibly 'api', but i left the directory name unchanged as 'api.opentreeoflife.org'.... so not bad, actualy
18:34 jimallman i'm going to move the existing 'api' subdirectory to a safe place on the server, just in case it's picked up important data.... then replace it with just one symlink like the others...
18:36 dukeleto jimallman: sounds good
18:37 jimallman dukeleto: OK, that's done. i'll restart the server...
18:38 dukeleto jimallman: http://dev.opentreeoflife.org/api/default/api/study/10.json gives me a 500, which is something
18:38 dukeleto jimallman: it probably can't find the datastore git repo
18:38 jimallman FYI - backup of old app dir (mix of symlinks and static) is at /raid/not_backed_up/opentree/api.BACKUP
18:38 jimallman for sure any private/config has been moved...
18:38 dukeleto jimallman: thank you! _ilbot will store that path for future searching :)
18:39 jimallman :)  nothing in private/ ... i should restart the server and watch the fireworks.
18:39 dukeleto jimallman: it was using ../treenexus to find the git datastore, which is no longer true, methinks
18:39 dukeleto jimallman: is there any way to see the actual error on dev?
18:39 dukeleto jimallman: instead of just a pretty 500?
18:40 jimallman yes, they're stored as static files.. just a sec..
18:40 dukeleto jimallman: i noticed that as something unique to web2py. Each error is it's own file, which is actually convenient for debugging
18:40 dukeleto jimallman: but is also a new workflow to get used to
18:41 jimallman right, kind of sensible since they're monster tracebacks..
18:41 jimallman i don't see an errors/ dir in the api/ space... nothing new in curator/errors
18:42 dukeleto jimallman: http://dev.opentreeoflife.org/api/default/api/study/10.json
18:42 dukeleto jimallman: try hitting that URL
18:42 jimallman right...
18:43 dukeleto jimallman: i am still learning about web2py logging. Is seems unfamiliar
18:43 jimallman it's funky.. i'll try tailing the main logs...
18:44 dukeleto jimallman: is dev.opentreeoflife.org running on web2py 2.4.4 ?
18:44 dukeleto jimallman: i was wondering if I was running into the "no output bug" because I was testing against 2.7.4
18:45 jimallman hrm, not sure. here's how to tail stdout on the running web2py:
18:45 jimallman sudo -u neo4j tail -f /raid/not_backed_up/opentree/web2py-2.4.4/nohup.out
18:46 dukeleto jimallman: thanks
18:47 dukeleto SocketTimeout: Socket timed out before request.
18:47 dukeleto seems odd
18:47 jimallman let's get more context... sudo -u neo4j tail -f /raid/not_backed_up/opentree/web2py-2.4.4/nohup.out -n 100
18:48 jimallman it's looking for a static json file in local  avatol_nexsons... is that expected?
18:49 dukeleto jimallman: no, i don't think so
18:49 jimallman lots of other files in the neighborhood... but not that one:
18:49 jimallman ls /raid/not_backed_up/opentree/avatol_nexsons/ingest/12*
18:50 jimallman dukeleto: i still haven't restarted web2py ... (looking up the steps to kill+restart) ... just a sec
18:51 dukeleto jimallman: yeah, because that error doesn't look like it is from my code
18:53 dukeleto jimallman: ah, it is a permission error
18:53 jimallman (restart is complete)
18:53 dukeleto jimallman: it is trying to create a "models" directory
18:53 dukeleto jimallman: because it is running as neo4j
18:54 jimallman i see. yes, all this stuff should be owned by group 'neo4j'
18:54 jimallman lemme see..
18:55 jimallman hm, it looks to me like everything in 'api' is OK (owned by jim:neo4j)
18:56 dukeleto jimallman: i see
18:56 dukeleto jimallman: i thought you made a symlink. That is a slighly out of date copy
18:56 dukeleto jimallman: do you mind if I update it and use it for development?
18:56 jimallman (i just created a 'models' folder from the command-line.. just to see)
18:57 dukeleto jimallman: it is currently checkouted out to one of your branches right now
18:57 jimallman dukeleto: hm, let me take a quick look and see if changes are useful..
18:57 dukeleto jimallman: now it is attempting to create "databases"
18:58 dukeleto jimallman: all this web2py magical behavior makes things very hard to test and integrate
18:58 dukeleto jimallman: OSError: [Errno 13] Permission denied: '/raid/not_backed_up/opentree/web2py-2.4.4/applications/api/databases'
18:58 dukeleto jimallman: from gluon/admin.py
18:58 jimallman ..and no doubt failling for the same reason (whatever that is)... odd, i'm almost certain the effective user is 'neo4j' and they should have permission here. let me try from the cmd-line
18:58 dukeleto jimallman: the default umask may need tweaking
18:58 jimallman FYI - the local api is running my 'cross-domain-api-support' branch, which would be good to merge to master if you haven't already
18:59 jimallman good call, i'm unable to mkdir here as neo4j:
18:59 jimallman sudo -u neo4j mkdir test
19:02 jimallman dukeleto: there's nothing to commit in the local 'api', so feel free to checkout master and pull changes. but you might want to merge cross-domain-api-support first:
19:02 jimallman https://github.com/OpenTreeOfLife/api.opentreeoflife.org/pull/10
19:03 dukeleto jimallman: looking now
19:03 jimallman i'm not facile wtih umask, so feel free to change it. otherwise, i'll need to do some research first..
19:03 jimallman dukeleto: ^
19:03 dukeleto jimallman: yes, umask can be tricky
19:04 dukeleto jimallman: looks like the default is 022
19:04 dukeleto jimallman: when I type "umask" on norbert
19:04 jimallman i just set g+w on the api/ directory...
19:05 jimallman chmod g+w .
19:05 dukeleto jimallman: that means it takes away write permission from world and the group
19:05 dukeleto jimallman: or that :)
19:05 dukeleto jimallman: sorry I missed this pull request. It doesn't apply cleanly any more, but I can make it work
19:05 jimallman drat. let me know if i can help clean up.
19:06 dukeleto jimallman: reading your code, making sure I understand it all
19:06 dukeleto jimallman: does it mean that a request can POST JSON with auth_token="deadbeef" to authenticate?
19:08 jimallman yes. this should be pulled from the user_auth table, so the actual user (human) doesn't need to even know their token
19:09 dukeleto jimallman: so, once I get past these permission issues, I should see a valid auth.* object from the API web2py app?
19:10 dukeleto jimallman: currently I have to fake it when I was testing the API app by itself on my local machine
19:11 jimallman ah, i'm not sure about that. i know that the curator has a modified auth.user setup to include a field for the OAuth token from Github, and we pass that to the OTOL API. I assume that the API will pass the token directly to the Github API on each call.
19:11 jimallman i should review, it's possible that i'm not passing the token to OTOL API yet..
19:12 jimallman but yeah, that's the idea. the OTOL API might not need any concept of users, sessions, etc.
19:12 jimallman just a middleman to the Github APIs
19:12 jimallman dukeleto: ^
19:13 dukeleto jimallman: that sounds nice
19:13 jimallman if needed, we could share auth.user tables from the curator app, but I don't think we want them that closely coupled, right?
19:13 travis-ci joined #otol
19:13 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#98 (local_git - 60c455c : Jonathan "Duke" Leto): The build is still failing.
19:13 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/5a67d2f8cb74...60c455ccf5eb
19:13 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/13225795
19:13 travis-ci left #otol
19:14 dukeleto jimallman: from what i see, it takes 3 or 4 requests to the Github API to create a commit, because you have to create blob and tree object too
19:14 dukeleto jimallman: it is 3 if you are doing it on an existing branch, another if you need to create a branch
19:15 jimallman that sounds likely, yeah. maybe there's a way to bundle them..?
19:15 dukeleto jimallman: just wanted to let you know. It has a lot of moving parts and I am prototyping a quick "simplest possible thing that could work" which commits to a local git repo and pushes that
19:16 jimallman Interesting! so for now it's being done as a single user..?
19:16 dukeleto jimallman: i don't think it is feasible to get all that API interaction working and tested by Thursday, which is my goal for write support
19:16 dukeleto jimallman: i have a "local_git" branch
19:16 jimallman understood. i think it's a reasonable first step, if we can see the goods in Github.
19:17 dukeleto jimallman: yes, using the Github API is definitely the goal, but I know people have been waiting on API write support for a while
19:17 jimallman yes, and the ability to reload and see changes will be sweet.
19:17 dukeleto jimallman: and yes, there are many more moving parts than I thought, to get it working properly
19:17 jimallman you might need to coordinate with mtholder... does this change his workflow?
19:18 dukeleto jimallman: he instructed me to basically concentrate on the simplest possible thing that could work, so I think I am following his instructions
19:18 jimallman kewl
19:18 dukeleto jimallman: but yes, I will be coordinating with him about this
19:20 Topic for #otol is now Open Tree Of Life | opentreeoflife.org | github.com/opentreeoflife
19:20 jimallman dukeleto: FYI, i'm currently wrestling with UI + design for the study creation path... if i can get it to a reasonable state to demo client-side dancing, i can probably help with the Github API dance.
19:20 dukeleto jimallman: awesome! I think some pair programming would go over well in the near future :)
19:24 jimallman i hope the changes in 'cross-domain-api-support' make sense. it's mostly tweaks to support JSON-P, a new generic view for same, and tweaks to handle large JSON responses. Plus the added auth-token info for later..
19:25 dukeleto jimallman: yes, it looks very reasonable
19:26 dukeleto jimallman: am going to attempt to apply https://github.com/OpenTreeOfLife/api.opentreeoflife.org/pull/10.patch and fix the conflicts
19:26 jimallman ok, standing by here..
19:27 dukeleto meh, doesn't apply. Going to try and merge the branch, which is what I should have done first...
19:28 dukeleto jimallman: much better. Only one conflicted file
19:28 dukeleto oy the joy of git...
19:29 jimallman :)
19:29 dukeleto jimallman: should I be using the @auth.requires_login decorator on the POST method?
19:30 dukeleto jimallman: i added that recently. when you send the PR, it wasn't there
19:30 dukeleto jimallman: it would not allow the oauth_token in the json workflow, though
19:30 jimallman dukeleto: yes, i would think (in the final scenario) we'll need auth.user to have the required OAuth token
19:31 jimallman "it would not allow..." ? i don't follow you here.
19:33 dukeleto jimallman: if we are allowing a request to put its auth info directly into the JSON, the code would still want the user to be logged in via the web2py app
19:33 dukeleto jimallman: that is what we want long term, so that is fine
19:34 dukeleto jimallman: for testing purposes (like hitting the API with curl), I was toying with the idea of that being the sole authentication mechanism
19:35 dukeleto jimallman: so that I am not dealing with Janrain/Github oauth while trying to get an MVP out the door
19:35 dukeleto jimallman: it will sort itself out
19:35 dukeleto jimallman: i am about to push the merged branch
19:35 jimallman dukeleto: i think we're on the same page here. in the curator app, i've modified the login+auth machinery to work *via* Github login, and to capture and store the oauth token in the process.
19:35 dukeleto jimallman: and would appreciate a code review
19:35 jimallman for now, i'm tempted to still include the login requirement just to show the UX and task flow.
19:35 dukeleto jimallman: that sounds really cool
19:37 * jimallman is looking at the merge now...
19:42 travis-ci joined #otol
19:42 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#99 (master - 36e82de : Jonathan "Duke" Leto): The build is still failing.
19:42 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/6dfd93735f54...36e82dec989b
19:42 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/13227174
19:42 travis-ci left #otol
19:43 dukeleto i will try to get Travis CI to run web2py 2.4.4 instead of 2.7.4 and see if that fixes the issue
19:43 dukeleto if you look at the output from Travis, there is absolutely no output from python web2py.py ....
19:44 jimallman hmm, i see what you mean.
19:46 jimallman actually, maybe i don't. where would this normally appear (web2py output)?
19:47 jimallman i see lots of chatter in the previous travis report:
19:47 jimallman https://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/13225795
19:48 dukeleto jimallman: click on the arrow next to line 62
19:48 dukeleto jimallman: it shows nothing
19:48 dukeleto jimallman: and also the next line starts at 63, which also tells us that
19:49 jimallman i see.. i didn't get the format here, but now i see (clicking next to line 21 "unfolds" the pip chatter, etc)
19:49 dukeleto jimallman: yes, the travis UI takes some getting used to...
19:49 dukeleto jimallman: i just pushed a commit that attempts to use web2py 2.4.4 instead
19:49 dukeleto jimallman: so we will have that datapoint soo
19:49 dukeleto soon
19:49 jimallman i'll poke around the web2py docs, maybe there's something in the way we're calling web2py.py that is shunting output to a file..?
19:50 dukeleto jimallman: it never starts listening on a port
19:50 dukeleto jimallman: and using python -mtrace shows it in an infinite loop
19:50 jimallman dukeleto:    --debug=0 might be the culprit
19:50 jimallman or hiding the errors, anyway
19:51 dukeleto jimallman: that actually shows all debug output, according to the docs, iirc
19:51 dukeleto jimallman: take a look at /dsk/leto/git/opentree/web2py/*.log
19:51 jimallman so it does! weird.
19:51 dukeleto jimallman: those are large files that show the python -mtrace output
19:51 dukeleto jimallman: it just goes on and on. I had to kill both of them
19:51 jimallman cool... so we should be seeing something more like this..?
19:52 jimallman in Travis, i mean?
19:52 dukeleto jimallman: i have not enabled a travis run with python -mtrace, but that could be soon
19:52 jimallman dukeleto: oh, i thought you meant that was the output with debug=0
19:53 dukeleto jimallman: no, web2py debug levels don't do so much
19:53 travis-ci joined #otol
19:53 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#100 (master - 8bc38d6 : Jonathan "Duke" Leto): The build is still failing.
19:53 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/36e82dec989b...8bc38d667874
19:53 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/13227764
19:53 travis-ci left #otol
19:54 dukeleto jimallman: https://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/jobs/13227766
19:55 dukeleto jimallman: if you look at the bottom of the output, I do a "ps auxw|grep python" and it shows the web2py process
19:55 dukeleto jimallman: but web2py.log has no output
19:55 dukeleto jimallman: really odd
19:55 jimallman yes, very weird
19:55 dukeleto jimallman: i have meetings between 1:30pm-3:30pm, fyi
19:55 dukeleto jimallman: i spent a bit too much time debugging that oddness
19:55 dukeleto jimallman: which is why I have switched over to doing development on dev.opentreeoflife.org
19:56 jimallman i'll be glad to chase this symptom, if we can reproduce on dev server. is travis running on a machine i can reach via ssh?
19:56 jimallman (I'm guessing not)
19:58 dukeleto jimallman: not really. It creates and throws away a VM for each test run
19:58 jimallman figures..
19:59 dukeleto quite an odd issue. But I must focus on MVP write support, so I am putting travis down for a bit...
20:00 jimallman yep, must ... not ... get ... distracted..
20:00 dukeleto jimallman: indeed :)
20:00 * jimallman has a tendency to chase rabbits
20:00 dukeleto jimallman: still seeing permissions issues on http://dev.opentreeoflife.org/api/default/api/study/10.json
20:00 dukeleto jimallman: me too!
20:00 jimallman taking a look now (bad perms)
20:01 dukeleto OSError: [Errno 13] Permission denied: '/raid/not_backed_up/opentree/web2py-2.4.4/applications/api/models'
20:01 dukeleto is the first
20:01 dukeleto the directory exists
20:02 dukeleto jimallman: it needs group write support
20:02 jimallman yes, and then again on 'databases' (which also already exists)
20:02 dukeleto jimallman: perhaps you need the -R (recursive) flag?
20:02 jimallman ah, i tweaked the api/ directory, but not those.. indeed,  -R it is.
20:04 jimallman .../applications/api> sudo -u neo4j chmod -R g+w .
20:04 jimallman done
20:06 jimallman nothing new in the log ( /raid/not_backed_up/opentree/web2py-2.4.4/nohup.out). ... but i'm still getting a 500 page
20:07 jimallman dukeleto: i sense that this is not the live log.. :-/
20:07 dukeleto jimallman: interesting
20:07 dukeleto jimallman: well, if there weren't so many bugs, life would be boring...
20:08 jimallman that's the spirit!
20:08 dukeleto jimallman: is there a way to get rid of the "pretty 500" error page and actually show error output?
20:08 dukeleto jimallman: it *is* dev.opentreeoflife.org
20:09 jimallman i agree, but we'd need to relax apache config and have jonathan restart it.  for now, let's look for new trouble tickets in api/errors...
20:09 jimallman hooray! lots of errors there.
20:11 jimallman in the latest, i see a socket timeout, and a reference (in error details) to this URLL:
20:11 jimallman https://groups.google.com/forum/#!topic/web2py/P_Gw0JxWzCs
20:11 jimallman (a discussion in the web2py watering hole)
20:13 jimallman dukeleto: it's not exactly illuminating. we're already running python2.7, right?
20:18 jimallman NO, scratch that. the "SSL bug caused closure" was a read herring, just some code that was near the raised SocketTimeout exception. sorry about that.
23:38 jimallman FYI - found and fixed the problems with study fetch.. it was failing due to errors in the JSON template (fixing this, and JSONP template, now)

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary