Camelia, the Perl 6 bug

IRC log for #parrot, 2012-07-01

Parrot | source cross referenced

| Channels | #parrot index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:25 whiteknight pmichaud: I suspect TOTAL_PMCs is the total that have been allocated.
01:25 whiteknight ACTIVE_PMCS is probably the number of PMCs which are currently alive (or alive-ish)
01:26 whiteknight probably
01:26 whiteknight I have to read through the code
01:41 dalek nqp: bec0eba | pmichaud++ | src/HLL/Compiler.pm:
01:41 dalek nqp: Restore a basic dumping capability from Parrot's Data::Dumper.
01:41 dalek nqp: We really should reimplement a dumper in pure NQP, though.
01:41 dalek nqp: review: https://github.com/perl6/nqp/commit/bec0ebaae4
02:10 whiteknight pmichaud: I'll take a look at the code tomorrow to verify.
02:26 wagle joined #parrot
02:28 wagle joined #parrot
03:16 pmichaud aloha: msg whiteknight (ACTIVE_PMCS)  it would be really helpful to have that verified (with the gms gc).  The numbers I'm getting back from some NQP compiles look really.... bizarre
03:16 aloha pmichaud: OK. I'll deliver the message.
03:16 pmichaud thanks, aloha
04:05 alvis_ joined #parrot
04:16 wagle joined #parrot
04:28 wagle joined #parrot
05:00 birdwindupbird joined #parrot
05:06 dalek nqp: 2bd8dcb | pmichaud++ | src/HLL/Compiler.pm:
05:06 dalek nqp: HLL::Compiler --stagestats=3 (or greater) now introduces prompts after each compilation stage.
05:06 dalek nqp: This makes it easier to determine the system resources consumed by each stage.
05:06 dalek nqp: check for system resource usage.
05:06 dalek nqp: review: https://github.com/perl6/nqp/commit/2bd8dcbbf8
05:32 wagle joined #parrot
06:23 zby_home joined #parrot
08:59 lucian joined #parrot
11:54 whiteknight joined #parrot
11:57 Liz joined #parrot
12:06 whiteknight good morning, #parrot
12:08 PacoAir joined #parrot
12:15 whiteknight msg pmichaud: src/gc/gc_gms.c:1339. It appears to be a count of all PMCS which are not in the free list. This includes PMCs currently alive, and PMCs in older generations which are dead but no yet swept
12:15 aloha OK. I'll deliver the message.
12:33 lucian joined #parrot
13:09 dalek Rosella: c5fc925 | Whiteknight++ | / (2 files):
13:09 dalek Rosella: [String] Fix handling of {{ in FormatPrinter. Add a test
13:09 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/c5fc925e82
13:09 dalek Rosella: a5c3335 | Whiteknight++ | / (2 files):
13:09 dalek Rosella: [String] Update FormatPrinter.format_obj to take field-level formats as well, for more fine-grained control
13:09 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/a5c333542a
13:09 dalek Rosella: 7bb8a07 | Whiteknight++ | src/unstable/xml/Visitor.winxed:
13:09 dalek Rosella: [Xml] Add some function-level docs for Xml.Visitor
13:09 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/7bb8a07ef4
13:09 dalek Rosella: 90933d1 | Whiteknight++ | src/string/FormatPrinter.winxed:
13:09 dalek Rosella: [String] Add some docs for FormatPrinter
13:09 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/90933d1d36
13:10 Psyche^ joined #parrot
13:20 dalek Rosella: a38f3bc | Whiteknight++ | / (3 files):
13:20 dalek Rosella: [String] Rename FormatPrinter methods for consistency with other StringFormatters.
13:20 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/a38f3bc953
14:41 kid51 joined #parrot
15:32 fperrad joined #parrot
16:16 dalek Rosella: 21a4351 | Whiteknight++ | src/unstable/ (3 files):
16:16 dalek Rosella: [Xml] Add Tag.add_children and Tag.add_children_from_hash methods, to help with constructing XML documents from native data types
16:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/21a4351c7e
16:16 dalek Rosella: 16949a7 | Whiteknight++ | src/unstable/xml/Document.winxed:
16:16 dalek Rosella: [Xml] Autogenerate an XML header if we don't have one already.
16:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/16949a7711
16:16 dalek Rosella: 63d4926 | Whiteknight++ | src/ (2 files):
16:16 dalek Rosella: [Feed] Add a set_feed_details method to the Atom feed writer. A few other small fixes
16:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/63d4926fef
16:29 lucian joined #parrot
17:13 jashwanth joined #parrot
17:57 zby_home joined #parrot
18:27 birdwindupbird joined #parrot
18:34 dalek m1: c526c01 | kjs++ | src/ (7 files):
18:34 dalek m1: improve type handling in parser and sem checker.
18:34 dalek m1: review: https://github.com/parrot/m1/commit/c526c0147e
18:34 dalek m1: d09d3d2 | kjs++ | src/ (4 files):
18:34 dalek m1: cleanups.
18:34 dalek m1: review: https://github.com/parrot/m1/commit/d09d3d2f59
18:34 dalek m1: 5bd1885 | kjs++ | src/ (3 files):
18:34 dalek m1: don't store ints < 256*255 && >= 0 in consts segment as they're loaded with set_imm.
18:34 dalek m1: review: https://github.com/parrot/m1/commit/5bd1885eb8
18:34 dalek m1: eddba64 | kjs++ | src/gencode.c:
18:34 dalek m1: add code for preloading constants 0 and 1.
18:34 dalek m1: review: https://github.com/parrot/m1/commit/eddba642ae
18:44 kjs joined #parrot
18:51 kjs joined #parrot
18:54 kjs_ joined #parrot
18:55 birdwindupbird joined #parrot
19:32 contingencyplan joined #parrot
19:35 Liz joined #parrot
19:47 kjs joined #parrot
19:54 dalek rakudo/nom: 1edbd37 | moritz++ | / (2 files):
19:54 dalek rakudo/nom: anonymous subset types with "subset :: where ...;"
19:54 dalek rakudo/nom: review: https://github.com/rakudo/rakudo/commit/1edbd3732e
20:06 perlite joined #parrot
20:16 dalek Rosella: 07f8594 | Whiteknight++ | src/unstable/xml/Tag.winxed:
20:16 dalek Rosella: [Xml] Fix Tag.to_xml to be more compact when the tag has zero children or exactly one Text child. Add a Tag.last_child method to mirror the .first_child method
20:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/07f8594195
20:16 dalek Rosella: ecebcab | Whiteknight++ | src/unstable/xml/tag/Collection.winxed:
20:16 dalek Rosella: [Xml] Add methods to Collection for accessing items
20:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/ecebcab552
20:16 dalek Rosella: e0d1fb8 | Whiteknight++ | src/unstable/feed/Writer.winxed:
20:16 dalek Rosella: [Feed] Make sure to add some rel= attributes to link tags in Atom feeds, when necessary
20:16 dalek Rosella: review: https://github.com/Whiteknig​ht/Rosella/commit/e0d1fb86c1
20:27 timo left #parrot
20:29 lucian joined #parrot
20:43 kjs joined #parrot
21:02 birdwindupbird joined #parrot
23:14 davidfetter joined #parrot
23:28 pmichaud There's been some recent discussion on perl6-compiler and other places about Rakudo's build memory requirements, so I'm trying to understand Parrot's memory allocation a bit better. (more)
23:28 pmichaud I have a small test program that allocates and releases PMCs and I get some odd (to me) memory statistics.... https://gist.github.com/3030004
23:29 pmichaud can someone confirm that this is expected behavior for Parrot?
23:34 pmichaud oddly, it appears that no gc runs took place in the execution of that program.  If I increase the number of RPAs created to 10 million (from 1 million), I get some GC runs:  https://gist.github.com/3030034
23:34 pmichaud but by this point Parrot has allocated over 192MB of ram to hold one active PMC.
23:35 pmichaud (if this belongs in a different forum such as an issue ticket or a mailing list, let me know.)
23:36 Coke pmichaud: trying running it under a ulimit?
23:36 Coke since you can't tell the PVM to only use X Mb at runtime, it may never feel any pressure to collect.
23:37 pmichaud it still feels a bit excessive to me for some reason.
23:37 Coke (granted, it should do better than "never free anything" even without this hint.)
23:38 Coke yah, seems crazy.... does it do any better run through the optimizer?
23:38 pmichaud ...optimizer?
23:40 whiteknight ...optimizer?
23:40 pmichaud what option would I give with ulimit to limit the amount of memory available?
23:40 whiteknight no such limit
23:41 whiteknight there's a long-standing open ticket for that issue
23:41 whiteknight it bounces between "can't fix" and "maybe later"
23:42 whiteknight pmichaud: I don't know how the memory threshold is calculated, but I think it's a percentage of available memory
23:42 whiteknight so it's entirely possible that a million entries might not trigger GC on your box
23:42 dukeleto pmichaud: i just commented on your gist
23:43 dukeleto pmichaud: you might try using a PIR function instead of goto+label to see if the stats change dramatically
23:43 whiteknight no, that measures the CallContexts allocated
23:43 whiteknight it would turn the number of pmcs from one million to two million
23:43 dukeleto pmichaud: my hunch is that you are creating many PMCs in the same scope and that may be a situation where our default gc (gms2) does poorly
23:44 dukeleto pmichaud: i understand that you are trying to emulate how the rakudo build works and I also agree that our gc should do better than it is currently performing
23:44 pmichaud dukeleto: I'm trying to get a sense of how much memory the rakudo build is actually taking, and what is eating up all of the memory
23:45 pmichaud so really I'm looking for ways to measure that.
23:45 whiteknight I don't think the scope has anything to do with it
23:45 pmichaud it looks like the interpinfo stats don't really help me much there, though
23:45 whiteknight the GC isn't limited by context
23:45 dukeleto whiteknight: --gc-nursery-size is my default, 2
23:46 dukeleto pmichaud: i would also play around with --gc-nursery-size . The first 2 generations of allocated stuff get collected much less frequently than everythign else, iirc
23:47 dukeleto pmichaud: you also may want to use --gc-dynamic-threshold during the Rakudo build
23:47 whiteknight pmichaud: Is your concern trying to get an accurate (and sensible) reading of memory usage, or to decrease it?
23:47 pmichaud how the heck do I do that?!
23:47 pmichaud whiteknight: both, but in order to do the second I think I have to have the first.
23:48 pmichaud if there's a way I can force gc runs, that might help.
23:48 whiteknight --gc-dynamic-threshold is the threshold at which a GC run is triggered, as a percentage of total available memory, I think
23:48 dukeleto pmichaud: you can force gc runs with a relative ( X mb ) or dynamic (X% of ram) flag, but tuning them is very machine-specific
23:48 pmichaud when we're doing the expensive part of the Rakudo build, it's Rakudo itself that is running.
23:48 pmichaud it's not a Parrot command.
23:48 whiteknight I can't remember the details, I can look it up if you want
23:49 dukeleto pmichaud: you would then have to build a knob in Rakudo such that you can set those values from PIR/C
23:49 pmichaud so how do I set those from PIR/C?
23:50 dukeleto pmichaud: you can definitely set them from C, I am not sure if there is a PIR api for them.
23:50 pmichaud is there a C api?
23:50 dukeleto pmichaud: if you poke into how the --gc-* command line flags are implemented, they just set various flags in the C API, iirc
23:50 pmichaud or would this be an instance of "rakudo poking into parrot's guts"?   ;-)
23:50 dukeleto pmichaud: that code was written by nwellnhof++, i believe
23:51 whiteknight I don't think pbc_to_exe generates a binary that can take that option
23:51 whiteknight we would have to add it
23:52 pmichaud (force gc runs)  I meant "Is there a way to force gc runs from PIR"
23:52 whiteknight yes, there's a sweep opcode
23:52 pmichaud ...checking
23:52 whiteknight sweep 1
23:52 pmichaud okay, that seems to do it.
23:52 whiteknight that opcode *should* trigger GC unconditionally
23:53 pmichaud with sweep 1, I get (one million RPAs):
23:53 pmichaud gc_mark_runs=1
23:53 whiteknight it's not used often, and testing it is a bear, so I wouldn't be surprised to hear it's gone bitrotten
23:53 pmichaud gc_collect_runs=0
23:53 pmichaud total_pmcs=1004190
23:53 pmichaud active_pmcs=1588
23:53 pmichaud total_mem_alloc=64913408
23:53 pmichaud total_mem_used=285875
23:53 whiteknight okay, yes. That seems better
23:53 pmichaud 1588 is about right for active PMCs, I think.
23:53 whiteknight yeah. Our "base" level is far too high
23:54 whiteknight every built-in class is a PMC, the interp, globals, every METHOD, etc
23:54 whiteknight so there's a minimum which is depressingly high
23:56 dukeleto pmichaud: this area of code may help you https://github.com/parrot/parrot/blob​/master/frontend/parrot2/main.c#L473
23:57 pmichaud surely initargs is long gone by the time rakudo is running, though, yes?
23:57 whiteknight pbc_to_exe sets up a simple version of it with default values to start the interp
23:57 pmichaud or is initargs the structure that gms2 looks at for its threshhold information?
23:57 dukeleto pmichaud: the latter, iirc
23:57 whiteknight That structure is used to initialize the interp and gc_private structs, which is what the GC looks at at runtime
23:58 pmichaud yeah; that sounds to me like we'd be poking around in parrot's internals a bit too much.  We're really trying to avoid that where possible -- I don't think we actually do much poking these days.
23:58 pmichaud anyway, I can play with it a bit if need be.
23:59 whiteknight pmichaud: if you have any specific feature requests don't hesitate to ask

| Channels | #parrot index | Today | | Search | Google Search | Plain-Text | summary

Parrot | source cross referenced