Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2013-12-30

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
17:30 jimallman joined #otol
17:30 ChanServ left #otol
17:37 dukeleto joined #otol
17:37 dukeleto jimallman: hola
17:39 jimallman hi! i didn't want to chatter away in the Github issue until we have a lock on the problem.
17:39 dukeleto jimallman: good call
17:40 dukeleto jimallman: this is an odd issue
17:40 jimallman agreed. is it possible some kind of request is bubbling up from locket and being intercepted by the restful wrapper?
17:40 dukeleto jimallman: has writing ever worked before?
17:41 jimallman definitely, we have a few studies with many successful writes..
17:41 dukeleto jimallman: or is this a new issue?
17:41 dukeleto jimallman: ok, good to know
17:41 jimallman but it's been a while since i can confirm a good one.
17:41 jimallman so i'm not sure when we went off  the rails..
17:41 jimallman we could check some of the WIP branches to see the latest mod date...
17:42 jimallman i'll check this now, but we need to ignore some of the most recent edits, since i was doing manual tweaking (outside of the web UI) to test Badgerfish compatibility
17:42 dukeleto jimallman: looks like 12 days ago on jimallman_study_9
17:43 jimallman yep
17:46 jimallman dukeleto: hm, here's something.. the traceback points to API's controller/default.py, line 300. This is the 'OPTIONS' function, which is a CORS check. Does this need to be inside v1?
17:47 dukeleto jimallman: no, I don't think so
17:48 jimallman oh wait, it's the REST response for the OPTIONS verb in HTTP, so I think it does belong here.
17:48 jimallman and it's def a RESTful response
17:50 jimallman maybe it shouldn't do the raise HTTP(200) ? googling for known issues...
17:51 jimallman by the way, that code has been there for a long time now..
17:53 jimallman same issue being discussed here? https://groups.google.com/forum/#!topic/web2py/IqJxYLR6b1g
17:53 dukeleto jimallman: yes, that hasn't changed in quite a while
17:55 dukeleto jimallman: acquire_lock() requires write permissions to the treenexus repo, to modify the lockfile
17:56 dukeleto jimallman: perhaps that has something to do with it?
17:56 jimallman that makes sense... let's see if user opentree has it
17:59 dukeleto jimallman: i see that all the things in repo/* are only write-able by the opentree user, but not the group
18:00 jimallman i thought the app (ultimately, apache) runs as user opentree...? maybe not..
18:01 jimallman should the lock file be written alongside the JSON file (e.g., study/10/10.json)?
18:02 dukeleto jimallman: no, the lock file is .git/API_WRITE_LOCK
18:02 jimallman ah, OK
18:03 jimallman in the API repo, or tree nexus?
18:03 dukeleto jimallman: treenexus
18:03 dukeleto jimallman: it should get created if it doesn't exist, but I just deleted it a few minutes ago and it is not being created
18:04 dukeleto jimallman: also, i get the same error when hitting the API on ot3 from my local machine, too. fwiw
18:04 jimallman good to know
18:04 jimallman here are .git/ permissions on ot3: drwxr-sr-x    8 opentree opentree  4096 Dec 30 17:56 .git/
18:06 jimallman (what's the 's' in group permissions?)
18:07 jimallman ah, set-groupID (should run with the effective permissions of the assigned group?) i sort-of get this..
18:08 dukeleto jimallman: http://stackoverflow.com/questions/7828065/what-is-the-usage-of-the-setgid-bit-on-a-server
18:09 jimallman thanks
18:10 dukeleto jimallman: i knew what it meant for an executable, but not a dir :)
18:10 jimallman fwiw, i just created a file 'TESTFILE' in .git, as user opentree
18:10 jimallman just did $ touch TESTFILE
18:10 jimallman i suppose the question is, who's the effective user at the moment of acquiring a lock?
18:11 * jimallman has removed TESFILE
18:11 jimallman TESTFILE
18:14 dukeleto jimallman: i am going to modify the controller on ot3 to test something out
18:14 shalvl joined #otol
18:14 jimallman kewl
18:16 * jimallman is going for coffee, back in a few
18:18 dukeleto shalvl: hola
18:23 travis-ci joined #otol
18:23 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#236 (master - ffb0092 : Jonathan "Duke" Leto): The build passed.
18:23 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/5ef127465b5c...ffb009241c1b
18:23 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/16152377
18:23 travis-ci left #otol
18:25 kameneva joined #otol
18:27 dukeleto jimallman: i am seeing a 400 when trying to write to the API on ot7, too
18:29 dukeleto jimallman: that error about restful is from Dec 20th
18:31 dukeleto jimallman: only 500's generate those error files, iirc
18:31 dukeleto jimallman: it would be really nice to get back the actual json error response instead of that useless html...
18:31 jimallman agreed
18:32 jimallman i've generally resorted to adding print() statements in web2py methods, to dump something coherent to the error.log
18:34 jimallman Nooo, sorry about the old error!
18:34 jimallman i should report the date in those *-last-error.sh scripts
18:38 jimallman are these the effective users for apache? https://gist.github.com/jimallman/31fac242b75d1fa71e20
18:39 jimallman if so, we should check groups for 'www-data' at least
18:53 dukeleto jimallman: this seems relevant: https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/apache-config#L50
18:53 dukeleto jimallman: but i am wondering what changed recently to make this break
18:55 jimallman ah, good. so that's probably the web2py user..
18:56 jimallman looks like that hasn't changed once mid-November.
18:59 jimallman i've added partial output to the gist above: https://gist.github.com/jimallman/31fac242b75d1fa71e20
18:59 jimallman ...showing open tree as the effective web2py user (as i understand it)
19:09 jimallman dukeleto: i suppose at this point, we could try using a breakpoint debugger, or maybe drive the github operations locally in python?
19:10 dukeleto jimallman: removing this line https://github.com/OpenTreeOfLife/opentree/blob/master/SITE.routes.py#L38 allows the API JSON errors to propagate
19:10 dukeleto jimallman: i am testing this on ot7
19:10 jimallman thanks, i've been meaning to try that.
19:10 jimallman (removing the routes_onerror patterns)
19:11 dukeleto jimallman: do you want to try that on ot3 and see what error comes back for you?
19:11 jimallman we should probably disable all of these during development
19:11 jimallman yes, will do this on ot3
19:11 jimallman (did this require an apache restart to work?)
19:11 dukeleto jimallman: yes
19:11 dukeleto jimallman: this is the error i get: https://gist.github.com/leto/839334db94e167855dbf
19:12 jimallman ! cool, sounds like they just want email/userid for github
19:13 jimallman via env variables or .git/config
19:13 dukeleto jimallman: are you getting the same error?
19:14 jimallman sorry, was googling your "*** Please tell me who you are." message... will try now
19:17 jimallman ah, slightly different (appears in AJAX response, see my comment here): https://gist.github.com/leto/839334db94e167855dbf
19:18 dukeleto jimallman: are you mixing together treenexus_test and treenexus ?
19:18 jimallman i was just thinking that.. :-/
19:18 jimallman checking config files now...
19:20 dukeleto jimallman: and deleting a study is a different codepath that creating/modifying (it doesn't do a pull)
19:21 jimallman agreed, but this was an attempted save (possibly a typo in the message?)
19:21 jimallman here's the repo specified in api.opentreeoflife.org/private/config:
19:21 jimallman repo_path   = /home/opentree/repo/treenexus_test
19:21 jimallman ... so who's asking for treenexus?!
19:23 jimallman fyi - i see the "Could not push deletion" message in both DELETE() and do_commit()
19:23 jimallman ...guessing the second one is a typo
19:24 dukeleto jimallman: what does .git/config look like?
19:24 jimallman in repo/treenexus, or repo/api.opentreeoflife.org?
19:24 jimallman dumb question... checking treenexus
19:25 dukeleto jimallman: in treenexus_test
19:25 dukeleto jimallman: and thanks for catching that bug with the error message
19:25 towodo joined #otol
19:25 jimallman BINGO! see gist: https://gist.github.com/jimallman/408275f3f68386aefd7a
19:26 jimallman sorry, hold the phone.. that was treenexus/.git/config
19:26 towodo dukeleto, r u there?
19:26 dukeleto towodo: yes
19:26 towodo wondering if you could help split a github repo.
19:26 jimallman https://gist.github.com/jimallman/3388422d04179105e530
19:27 towodo i want to move the smasher/ directory out of opentree and into its own repo on github
19:27 jimallman (remotes here look right to me)
19:27 towodo google gives me many suggestions, thought i'd check with you first tho
19:28 towodo yes, remotes ought to be involved. i can create the new repo on github, empty; manipulate local copy of old repo; then push subset to new using an appropriate remote.  i think.
19:28 dukeleto towodo: git filter-branch is most likely what you want
19:29 towodo ok…i saw an example https://help.github.com/articles/splitting-a-subpath-out-into-a-new-repository
19:29 towodo followed by a 'git remote' command? and then a push?
19:30 dukeleto towodo: pretty much
19:31 dukeleto towodo: do you have branches other than master?
19:31 towodo not now, nothing i care about
19:34 dukeleto towodo: ok, then it is pretty straight-forward
19:34 towodo can i use 'fork' on github, and then trim out the stuff i don't want using filter-branch?...
19:34 dukeleto towodo: no, you don't want to fork
19:34 towodo ok...
19:35 dukeleto towodo: so, you have an opentree git directory locally
19:35 dukeleto towodo: make sure it is up to date
19:35 dukeleto towodo: then "cp -R opentree smasher"
19:35 dukeleto towodo: cd smasher
19:35 dukeleto towodo: git remote rm origin
19:35 dukeleto towodo: git remote add origin git@github.com:OpenTreeOfLife/smasher.git # or whatever you called it
19:36 * towodo mulls
19:36 dukeleto towodo: git filter-branch --prune-empty --subdirectory-filter smasher master
19:36 dukeleto towodo: git push -u origin master
19:37 dukeleto towodo: that should create a smasher repo with history intact
19:37 dukeleto towodo: a "git gc" in the smasher.git is not necessary, but will make things a bunch faster by getting rid of all the old opentree commits in the git database
19:37 dukeleto towodo: then you can go into the original opentree repo and do a "git rm -rf smasher" and commit that to remove smasher from opentree.git
19:38 dukeleto jimallman: back to our debugging
19:38 dukeleto jimallman: where are you at?
19:39 dukeleto jimallman: "Permission to OpenTreeOfLife/treenexus_test.git denied to OpenTreeOfLife/treenexus" is really odd to me
19:39 towodo tnx… mulling, there's something i'm not quite getting
19:39 jimallman dukeleto: i still haven't found the errant 'treenexus' reference.. here's the private/config in repo/treenexus_text, and it looks right to me... https://gist.github.com/jimallman/408275f3f68386aefd7a
19:40 jimallman it sure looks like the ot3 API app is using the wrong local repo....
19:40 dukeleto jimallman: that gist is the .git/config
19:40 jimallman right, sorry. that's what i meant.
19:41 dukeleto jimallman: what does your deployment config look like?
19:41 jimallman hmmm, let's see
19:42 dukeleto towodo: what is not clear?
19:42 jimallman https://gist.github.com/jimallman/bb12a0601ccc5f1a30da
19:43 jimallman (again, looks right to me)
19:43 dukeleto jimallman: we set the originssh here in deployment: https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/install-api.sh#L55
19:43 travis-ci joined #otol
19:43 travis-ci [travis-ci] OpenTreeOfLife/api.opentreeoflife.org#237 (master - d1964c3 : Jonathan "Duke" Leto): The build passed.
19:43 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/api.opentreeoflife.org/compare/ffb009241c1b...d1964c35cf4e
19:43 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/api.opentreeoflife.org/builds/16155680
19:43 travis-ci left #otol
19:44 towodo wondering why i can do 'git push' from a clone of A to a different repo B. there is no connection between A and B. just seems weird
19:45 towodo what you said is clear, it just doesn't feel right
19:45 towodo to me
19:45 towodo not saying it's wrong
19:47 dukeleto towodo: it is the distributed nature of Git. You can push and pull to as many remotes as you want.
19:48 dukeleto towodo: filter-branch changes history and changing origin essentially makes it a new repository with a new place to send changes
19:48 dukeleto towodo: git even allows you to push multiple histories into one repo (i.e. Git allows you to have as many root commits as you want) but that is rarely done
19:49 towodo but there is no common ancestry. what does this do to merges (if there were similarly named files in B, which there aren't in this case)? does it act as if the common ancestor is an empty file at the beginning of time?...
19:52 dukeleto towodo: not sure exactly how to answer you question, but filter-branch changes history and only chooses commits that touched the specified directory
19:52 dukeleto towodo: common ancestry is not required
19:55 dukeleto jimallman: does repo_path in private/config match the repo in .git/config set to originssh ?
19:57 jimallman dukeleto: i think so, confirming now...
19:58 jimallman api's private/config says:
19:58 jimallman repo_path   = /home/opentree/repo/treenexus_test
19:59 jimallman treenexus_test/.git/config says:
19:59 jimallman [remote "originssh"]
19:59 jimallman url = git@github.com:OpenTreeOfLife/treenexus_test.git
19:59 jimallman at a glance, this looks right to me.
20:02 dukeleto jimallman: do you get the same kind of error if you attempt to do something to a different study?
20:03 jimallman good question, i'll check... meanwhile, take a look at fetch_study:
20:03 jimallman repo_path = 'OpenTreeOfLife/treenexus'  # TODO: pull from config file?
20:03 jimallman (in module github_client.py)
20:03 dukeleto towodo: https://github.com/OpenTreeOfLife/opentree/pull/166
20:04 jimallman same error for study #9, will try one more...
20:05 jimallman dukeleto: same error for studies #9 and #11 (which I believe I've never edited before)
20:06 dukeleto jimallman: github_client.fetch_study isn't used. The only method in there that is used is .compare()
20:06 towodo oops
20:07 jimallman OK. i'm just looking for hard-coded values... but did you say this works on ot7? or shows same error?
20:07 towodo dukeleto, shouldn't the same config be done for all repos?
20:08 travis-ci joined #otol
20:08 travis-ci [travis-ci] OpenTreeOfLife/treenexus#88 (leto_study_1003 - d952ddd : Duke Leto): The build passed.
20:08 travis-ci [travis-ci] Change view : https://github.com/OpenTreeOfLife/treenexus/commit/d952ddd2dffc
20:08 travis-ci [travis-ci] Build details : http://travis-ci.org/OpenTreeOfLife/treenexus/builds/16156389
20:08 travis-ci left #otol
20:08 towodo and why is this done in as-admin…? I mean, it's OK, but couldn't it be done in the unprivileged file init-whatever.sh?
20:08 dukeleto towodo: it is only really needed for dealing with treenexus, but the --global means that it is the default for all operations as the opentree user
20:08 towodo ah.
20:09 dukeleto towodo: it is for when we pull a WIP branch before writing to it and git has to do a merge, which means it creates a commit
20:09 towodo so i would expect it to be done in install-api.sh
20:10 towodo or in functions.sh  (which perhaps should be renamed)
20:11 dukeleto towodo: i just put it where i saw various other sudo commands
20:11 towodo but if it runs as the opentree user it doesn't need a sudo, if it's in an install-xx file
20:11 towodo trying to minimize use of sudo
20:12 dukeleto towodo: ok, i will move it to install-api.sh
20:12 towodo tnx
20:13 towodo darnit, i accidentally flushed my irc transcript
20:13 towodo is it archived?
20:13 dukeleto towodo: yes
20:13 dukeleto towodo: http://irclog.perlgeek.de/otol/2013-12-30
20:14 towodo oops. attempting to set chat topic
20:15 towodo this irc isn't the same as w3c, there are no helpful robots
20:16 dukeleto towodo: now you should be able
20:16 dukeleto towodo: and I updated the pull request to put the git config stuff in install-api.sh
20:16 Topic for #otol is now http://irclog.perlgeek.de/otol/YYYY-MM-DD
20:18 Topic for #otol is now Open Tree Of Life opentreeoflife.org http://irclog.perlgeek.de/otol/today
20:18 dukeleto towodo: :)
20:18 dukeleto towodo: /today always points to the right day
20:18 towodo dukeleto: thanks. going to try the repo split now
20:20 dukeleto towodo: https://github.com/OpenTreeOfLife/opentree/pull/167
20:20 dukeleto towodo: sorry for burying you in pull requests
20:20 towodo no problem
20:21 Topic for #otol is now #otol Open Tree Of Life | http://opentreeoflife.org/ | http://github.com/opentreeoflife | http://irclog.perlgeek.de/otol/today
20:22 towodo cool
20:22 jimallman i've set this topic a few times, but it eventually goes away.. not sure why.
20:23 dukeleto jimallman: when nobody is in there room, it default to nothing
20:23 dukeleto jimallman: occasionally _ilbot loses a connection when nobody is here
20:23 jimallman that's hilarious... ChanServ is falling down on the job.
20:25 jimallman oh, i'm thinking of another channel. we don't run chanserv here.. it will also keep the topic
20:29 jimallman dukeleto: looks like you can lock the topic with '/msg ChanServ SET #otol KEEPTOPIC ON'
20:30 jimallman (i don't have permission to do this)
20:33 towodo dukeleto: "Cannot rewrite branch(es) with a dirty working directory."  - going to retry with remote manipulation after the filter-branch
20:34 towodo dukeleto: ng. getting error from filter-branch.   will do a fresh clone of the repo this time, maybe my local copy was corrupt in a way not surfaced by 'git status'
20:36 towodo dukeleto: ng. "Cannot rewrite branch(es) with a dirty working directory."  Will email you transcript
20:38 towodo dukeleto: maybe it's because I'm using cp -a instead of cp -R   ??????? that doesn't make any sense tho.
20:38 towodo email sent
20:38 jimallman known issue? some workarounds here:
20:38 jimallman http://stackoverflow.com/questions/2030477/git-error-cannot-rewrite-branches-with-a-dirty-working-directory
20:38 jimallman towodo: that was for you
20:40 towodo that's bonkers, how could a fresh clone contain uncommitted changes? will try it though.
20:40 towodo 'git status' even says 'working directory clean' - a contradiction
20:42 towodo ng after 'git stash' (which says no local changes to save).  will ask google
20:42 jimallman towodo: see the StackOverflow thread. there's a bug in pre-1.7.7.1 git that shows this msg.
20:42 towodo or rather duckduckgo
20:43 jimallman just try 'git update-index -q --ignore-submodules --refresh' then filter-branch again
20:43 jimallman this forces an update of the local index, apparently
20:43 jimallman towodo: ^
20:43 towodo hm.  i'm running 1.7.4.4
20:44 dukeleto towodo: maybe just do a new clone to get the latest repo with a pristine dir
20:44 towodo that's what I did, still no good. but i tried the update-index and now it seems happy
20:44 dukeleto jimallman: I told chanserv to keep the topic. Good call!
20:45 jimallman sweet
20:46 towodo aside: at w3c transcripts are captured by a bot (rrsagent), but it doesn't keep /command lines e.g. /me xyz
20:46 towodo so you could make all your off-color, private remarks in /me commands and posterity would never know
20:47 towodo the sadness and glory of open.
20:47 jimallman indeed, that's a nice feature
20:47 towodo s/know/know./
20:48 towodo testing
20:48 towodo s/g/g x/
20:48 towodo weird, s// doesn't work
20:48 towodo oh yeah, that was done by a bot at w3c and only affected the log
20:48 jimallman not sure what you're expecting to see. ah, ok.
20:56 jimallman dukeleto: i think this is the real problem... we have a mismatched key (not repo)
20:56 jimallman https://help.github.com/articles/error-permission-to-user-repo-denied-to-user-other-repo
20:57 jimallman is there a special .pem file i should be using to reach treenexus_test?
21:01 jimallman dukeleto: when you have a moment, please share the Founder privileges for ChanServ, like so:
21:02 jimallman '/msg ChanServ FLAGS #otol jimallman +F'
21:02 jimallman it would probably be wise to do the same for jar (towodo)
21:04 * jimallman has a quick errand, back in a bit
21:11 * dukeleto shared founder privs with jimallman and towodo
21:12 dukeleto jimallman: good find. The deploy key attached to treenexus.git is not attached to treenexus_test.git
21:14 dukeleto jimallman: evidently a particular deploy key can only be used on one repo
21:27 jimallman dukeleto: the implied solution is to attach the key to a user account, versus a repo. not sure if there *is* a proper 'opentree' user account...

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary