Camelia, the Perl 6 bug

IRC log for #darcs, 2012-12-10

| Channels | #darcs index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:35 mizu_no_oto joined #darcs
00:53 mizu_no_oto joined #darcs
02:57 intripoon_ joined #darcs
03:13 gbeshers joined #darcs
04:04 rdesfo joined #darcs
04:30 schlaftier joined #darcs
06:54 gbeshers joined #darcs
09:51 intripoon joined #darcs
10:05 kmels joined #darcs
10:27 idnaria joined #darcs
10:30 dleverton_ joined #darcs
10:44 raichoo joined #darcs
11:00 owst joined #darcs
11:24 kmels_ joined #darcs
13:32 mizu_no_oto joined #darcs
14:13 mizu_no_oto joined #darcs
15:06 mizu_no_oto joined #darcs
15:08 kmels_ joined #darcs
15:13 donri joined #darcs
15:29 gbeshers joined #darcs
15:45 markstos joined #darcs
16:14 owst joined #darcs
17:20 gbeshers joined #darcs
17:23 schlaftier joined #darcs
17:29 markstos I was running a "darcs amend" to add a single line to an existing patch which I just created.  After waiting some minutes for it to complete, I've restarted the command with "--debug-verbose" to see what's going on.
17:30 markstos I can see now it's very slowly going through a process of copying lots of patches:
17:30 markstos I'm doing copyFileUsingCache on patches/0000001146-5b27780993a3e23da6abf37​ec276eb932c13433135052716cb4d4faa40c9e8b0
17:31 markstos It's not clear to me why it would need to do that just to mike a single line amendment to an existing, just-created patch.
17:33 markstos I tried again after running "darcs optimize -disable-patch-index", but that didn't stop all the copying from happening.
17:33 markstos This is with darcs 2.9.6. I'll try again with darcs 2.8.x and see if it's a regression.
17:38 * markstos sighs. Now I can't re-enable patch index without re-compiling darcs, because after I got "patch index" working, I must have done a minor upgrade to darcs and forgot to include the "RTS" flags. so I'll be re-compiling darcs again to add that flag back...
17:40 markstos The "darcs amend" slowness due to using copyFileUsingCache is not a recent regression. Darcs 2.8.0 has the same behavior.
17:40 markstos darcs amend is usually fairly fast. I'm not sure what has changed.
17:49 markstos I created a bug report for the slow amend-record behavior: http://bugs.darcs.net/issue2273
17:49 markstos I gave up after > 10 minutes of waiting. I'll just record a second patch, or unrecord/re-record.
17:51 markstos For comparison, I used --debug-verbose when recording a new patch instead.
17:52 markstos I see it also makes lots of calls to "copyFileUsingCache". I see it's copying patches created back in 2004 at the beginning of the project. I have no idea why those patches are related to creating a new single line patch.
17:52 markstos Oops, now "record" failed as well with a stack overflow due to the RTS issue.
17:53 * markstos sighs. I wish higher RTS values were enabled by default.
17:55 raichoo joined #darcs
18:05 mizu_no_oto joined #darcs
18:06 markstos I tried adding "+RTS -K100M -RTS " to my default options in _darcs/prefs/defaults, but that didn't work:  darcs: Bad default option: command 'ALL' has no option '+RTS'.
18:07 markstos Is there an environment variable I can use to set this instead: "+RTS -K100M -RTS " ?
18:09 markstos I found related docs which I think will answer that for me: http://www.haskell.org/ghc/docs/7.6.1/​html/users_guide/runtime-control.html
18:15 markstos I'm trying to bake-in higher defaults at compile time, but it's not working:  cabal install -frts --with-rtsopts="-K100M" .
18:15 markstos How do I pass the --with-rtsopts flag to cabal install?
18:15 dcoutts markstos: you want to pass them to ghc, not cabal itself
18:16 dcoutts so you have to use a passthrough
18:16 dcoutts ie ask cabal to pass extra flags to ghc
18:16 markstos dcoutts: Thanks. However, I don't see such an option documented in "cabal install --help". (Searching for "ghc" in that output).
18:17 markstos I also find inconsistency in the GHC docs about "--with-rtsopts" (2 leading dashes) vs "-with-rtsopts" (1 dash)
18:17 dcoutts markstos: it is in the --help, but it doesn't mention ghc, since it works for any program
18:17 dcoutts --with-PROG=PATH               give the path to PROG
18:17 dcoutts --PROG-options=OPTS            give extra options to PROG
18:17 dcoutts --PROG-option=OPT              give an extra option to PROG (no need to
18:17 dcoutts quote options containing spaces)
18:18 Igloo You should find it if you search for ghc, though
18:18 Igloo "The flags --with-PROG and --PROG-option(s) can be used with the following programs: [...] ghc [...]"
18:18 dcoutts yes, that too
18:19 markstos Got it. Thanks. Trying now.
18:20 markstos So I confirmed the single-leading-dash is correct for -with-rtsopts. If anyone here has GHC doc commit access, there's a typo here:
18:20 markstos http://www.haskell.org/ghc/docs/7.6.1/​html/users_guide/runtime-control.html where it says "--with-rtsopts"
18:21 markstos Great, I'm also set with my rebuilt darcs. Now it's see if it helps.
18:21 markstos For reference, here's what I used:  cabal install -frts --ghc-options='-with-rtsopts="-K100M"' .
18:24 markstos Hmm,,, I see I'm also getting "copyFileUsingCache" slowness when attempting to unrecord a single one-line change. I fear deeper problems. I what happened that's causing these various slow-downs for me.
18:26 markstos I need a --debug--verbose-verbose flag.
18:28 markstos Could have a lots of changes in "whatnew" be problematic? I see changes active in ~35 files currently.
18:32 markstos Anyone have ideas on why copyFileUsingCache is being used for unrecord and amend, why it might be slow, or what could cause it to "hang" ?
19:12 markstos I'm going to try a couple of things which seem good to do anyway: 1. Free up more system memory, and see if that helps. 2. Clean up my repo to minimize "whatsnew" changes, and see if that helps.
19:18 markstos free'ing a couple of gigs of memory didn't help.
19:20 Heffalump is there anything in _darcs/patches/pending?
19:21 * markstos checks
19:22 markstos No, just empty curly braces. ( I've temporary put all my "whatsnew" changes into a temporary patch, if that matters. )
19:22 Heffalump how big is the patch you're trying to amend?
19:23 Heffalump also, how big is _darcs/hashed_inventory
19:23 * markstos checks
19:24 markstos hashed_inventory is 312K
19:24 Heffalump I'm stabbing in the dark here, it's not really obvious to me why there should be a problem
19:24 markstos The patch I'm trying to add one-line amendment to itself modifies just one file-- the same file,  with "-43 +22".
19:25 markstos This is why I don't understand why there is so much copying of patches going on.
19:25 markstos Thanks for your help, Heffalump
19:26 markstos grep 'hash:' ../../_darcs/hashed_inventory  | wc -l ... reports "1016" patches
19:28 markstos What's causing all the "copyFileUsingCache" calls? Is it "Adding changes to pending..." or "Adjusting the context of the unrevert changes...
19:28 markstos " or "Removing 1 patches in removeFromUnrevertContext!
19:28 markstos " ... the --debug-verbose output is not clear about which piece is responsible.
19:31 * markstos searches source code for references to copyFileUsingCache
19:32 markstos Looks like there may be multiple caches that it could be copying from, and the debugging doesn't say which it is.
19:32 markstos My ~/.darcs/cache dir is 1.3G. :)
19:33 markstos I'm going to try moving ~/.darcs/cache dir aside and see what happens.
19:35 markstos That appeared to have no effect.
19:39 markstos Looks like there's just a few high level functions involved: withGutsOf,  tentativelyRemovePatches, YesUpdateWorking and finalizeRepositoryChanges
19:42 markstos I'm going to do an experiment by converting a copy of the repo from 'hashed' format to 'darcs-2' and see if the problem persists.
19:44 amgarchIn9 joined #darcs
19:46 markstos Some patch -index feedback for "darcs convert" performance: I'm running darcs convert --debug-verbose. I can see that's pausing periodically to create a patch index. Would it faster to create the patch index just once at the end?
19:47 markstos Curiously, "darcs convert" also makes many calls to "copyFilefromCache", but doesn't hang.
19:48 `nand` Does darcs build on GHC 7.6?
19:50 markstos nand: My understand is that darcs targets the Haskell Platform, which is still at 7.4.2.
19:51 markstos nand: from my own experience, attempting to build with a newer GHC than what is provided by the Platform is problematic.
19:51 `nand` I'm just sticking with the static version that I last compiled on 7.4.2 until darcs gets upgraded to compensate
19:51 `nand` the problem is that it's messing with my dependency chain; since packages are starting to drop support for 7.4
19:52 markstos They are depending on a newer version of GHC than what's provided than the current Haskell Platform release? That's interesting.
19:53 favonia joined #darcs
19:53 markstos It looks like the most time consuming part of a large "darcs convert" is the frequence pauses to create the Patch Index.
20:02 carter_ joined #darcs
20:11 sm_ joined #darcs
20:12 lispy joined #darcs
20:12 Heffalump markstos: converting from hashed to darcs-2 isn't likely to help (except perhaps by coincidence/side-effect)
20:13 Heffalump `nand`: not yet, sorry. Gentoo have some hacks to make it build, but they break unicode
20:13 Heffalump `nand`: I am actively working on it at the moment
20:13 Heffalump s/break unicode/break non-unicode encodings (I think)/ - certainly something encoding related
20:13 markstos Heffalump: Thanks for the feedback.
20:14 Heffalump markstos: the 1016 patches in hashed_inventory _might_ explain things, and if so laying down a tag just before the patch you are amending would help
20:16 markstos Heffalump: "darcs convert" did fix it. The question is then: "why".
20:17 Heffalump markstos: if darcs convert fixed it, I'd expect darcs get to a fresh directory would also fix it
20:17 markstos This is an interesting result, but is not yet a practical solution, we have several interconnected repos to upgrade. I would prefer to not take that detour now, but I *was* sort of looking for something to push us to upgrade.
20:17 Heffalump personally I wouldn't upgrade unless you actually have problems with conflict handling
20:17 markstos Heffalump: and "get" will preserve the repo format by default, right?
20:18 Heffalump yes
20:18 markstos Trying that now.
20:21 Heffalump oh, and if you can easily, please preserve this repo for when my mythical obfuscator comes along :-)
20:21 markstos The stuck-state version, I presume?
20:22 Heffalump yes
20:23 markstos Heffalump: You were right: darcs get also fixed it.  What's really interesting is that there are no calls to "copyFileUsingCache" for patches in the new repo, running the same command.
20:23 markstos There are some calls to copyFileUsingCache in the debug output, but only for inventory files.
20:23 markstos So my guess that darcs appeared to be off track with all those calls before was correct.
20:23 Heffalump right. I'm going to sort out GHC 7.6 encoding issues, then if I still feel motivated I'm going to write the obfuscator
20:24 Heffalump I can't remember if we discussed this before, but just to check before I do it: you'd be comfortable with something that hashed each content line, optionally with a salt you provided?
20:25 markstos So, it obfuscates the content of every line, the name of every file and directory, and the patch names and descriptions as well, well preserving the intent of changes, conflicts, etc?
20:25 Heffalump I was hoping to get away with just the content of every line, but I could do names of files and directories and patch names and descriptions too if necessary. Files and directories is probably the most fiddly, but I haven't thought through the details yet.
20:26 Heffalump oh, err, token replace
20:26 markstos The file contents are the critical part, but the others would be nice for privacy.
20:26 Heffalump ok
20:26 Heffalump but, arrgh, token replace :-)
20:33 markstos Heffalump: You can see before/after examples of the working/de-railed debugging output here now: http://bugs.darcs.net/issue2273
20:33 markstos I wonder if this line in the bad output is "interesting": "Removing 1 patches in removeFromUnrevertContext!"
20:33 markstos Does the exclamation point there convey excitement or concern?
20:34 Heffalump what's the size of hashed_inventory in the newly fetched repo? (the patch count from hash: is the ideal info)
20:35 markstos I'm pretty sure I just found the source of the misery.
20:35 markstos There was a massive set of merges in _darcs/patches/unrevert
20:36 markstos I don't think anyone on my team has unreverted anything for 5+ years. Too bad I can't tell darcs I don't care about that feature, and not to devote any resources to it.
20:37 Heffalump markstos: ahhhh!
20:37 markstos grep merger unrevert | wc -l == 104
20:37 Heffalump we've seen this before, but too rarely for me to ever remember the next time
20:37 markstos So the shorthand solution could have been: rm -f _darcs/patches/unrevert ?
20:37 Heffalump yes :-)
20:38 markstos This also explains why no one else on my team is having problems today.
20:40 markstos I've now confirmed the issue was fixed with: mv _darcs/patches/unrevert _darcs/patches/unrevert.bak.
20:42 Heffalump ok, so preserving the repo isn't interesting any more
20:42 markstos That's good. Saves me a gig of space. :)
20:42 markstos I updated the bug report with the "fix"
20:42 markstos http://bugs.darcs.net/issue2273
20:44 markstos I'm going to get some tea now and then "return my regularly scheduled programming", as they say. Thanks for your help, #darcs.
21:27 mizu_no_oto joined #darcs
21:40 xymox joined #darcs
21:59 mizu_no_oto joined #darcs
22:25 dixie joined #darcs
22:34 kmels_ joined #darcs
23:13 mndrix joined #darcs
23:13 mndrix joined #darcs
23:16 stepcut joined #darcs
23:41 donri sm_: hey are repos on hub backed up in any way?
23:58 gh_ joined #darcs

| Channels | #darcs index | Today | | Search | Google Search | Plain-Text | summary