Perl 6 - the future is here, just unevenly distributed

IRC log for #marpa, 2014-09-14

| Channels | #marpa index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
03:01 kkrev joined #marpa
03:05 jeffreykegler joined #marpa
03:06 jeffreykegler I've just uploaded Marpa-R2 2.093_000 to CPAN
03:06 jeffreykegler The main features are the new $slg->parse() and $slr->ambiguous() methods, which are best learned about from the new top-level page --
03:07 jeffreykegler I rewrote the Marpa::R2 page is use $slg-parse(), which means that beginners don't have to (initially) worry about grammar vs. recognizer objects.
03:09 kkrev I'm having  a very bizarre problem with perl+marpa using up all memory and getting killed by the out of memory killer. I run my script with the grammar against a bunch of files. Each file individually seems to work fine. But if I run the script in series I get a memory blow up: for i in `ls *.sim | shuf`; do perl my_parse.pl $i; done
03:10 jeffreykegler Because I expect most new users will quickly outgrow $slg->parse, there's also a new doc Marpa::R2::Tutorial2, all about how to move over to the lower level methods.
03:10 kkrev When I run that it will blow up at some point, but then when I manually test the last few files in the list they all work fine.
03:12 jeffreykegler kkrev: multiple recognizers?  multiple perl scripts?  or all one?
03:12 kkrev all one script with one recognizer.
03:13 jeffreykegler So it will be from Marpa's point of view one long parse, even if the input is in many files.
03:13 jeffreykegler Yes, memory could become an issue
03:13 kkrev Um, the parse gets run, then perl exits. Then perl restarts and does another parse.
03:14 jeffreykegler Perl restarts for each file?
03:14 kkrev yes
03:14 jeffreykegler But even so memory blows up in long multi-file runs?
03:14 kkrev Yes. I did say it's bizarre.
03:15 jeffreykegler Well, one fault Marpa does not have is hanging on to memory after its process ends. :-)
03:16 jeffreykegler Out of curiousity, what OS?
03:18 jeffreykegler Cygwin? Linux?
03:20 kkrev fedora
03:21 jeffreykegler So the OS is not one of those known to be insane.
03:22 jeffreykegler If it were my bug, I'd look first at the shell script -- perhaps the memory issue is that thing you have in backticks.
03:23 jeffreykegler I will usually instead pipe the output to a shell "while" loop, one which reads from STDIN, in order to avoid creating huge argument lists in the shell.
03:28 kkrev The stuff in the backticks is really just an ls on a couple hundred files. I was originally using xargs -n1 and getting the same OOM problem. It seems like some of these parses are "randomly" blowing up even though the files are only a megabyte.
03:30 jeffreykegler A command line containing 200 files, if the file names are long, is a big command line, but if you're sure that's not the problem, fine.
03:31 jeffreykegler I started shell programming in 1971, and there was no way the shell would handle a command line like that in those days.
03:32 kkrev I will try to set up a loop that reliably reproduces the problem and maybe post it. Nevermind until then. I was just wondering if there was something obvious. It smells to me like some sort of uninitialized memory bug cascading into an allocation.
03:32 jeffreykegler Generally speaking, any series of Perl scripts, serially run, having a memory blowup should be reproducible by running just one of the scripts.
03:33 jeffreykegler And the idea is that the random memory allocation is OK for certain series of commands, but is a large number for others?
03:33 jeffreykegler Could be?
03:34 jeffreykegler I, and others on this channel, regularly do valgrind on Marpa, by the way, ...
03:34 kkrev Well with these UMR bugs you'll get say an int that is almost always zero but then very occasionally not.
03:35 jeffreykegler as far as we know it is squeaky clean.
03:35 jeffreykegler There are programmers who feel a memory leak is OK, or a minor issue ... I am *not* one of those.
05:13 jdurand joined #marpa
05:14 jdurand Re http://irclog.perlgeek.de/marpa/2014-09-14#i_9354126 - can we have the "shuf" script content and an example of the *.sim filenames and/or content
08:29 lwa joined #marpa
12:14 rns joined #marpa
12:15 rns left #marpa
12:16 lwa joined #marpa
15:04 lwa joined #marpa
17:16 lwa joined #marpa
18:15 jeffreykegler joined #marpa
21:06 ronsavage joined #marpa

| Channels | #marpa index | Today | | Search | Google Search | Plain-Text | summary