Perl 6 - the future is here, just unevenly distributed

IRC log for #marpa, 2014-10-02

| Channels | #marpa index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
03:06 ronsavage joined #marpa
07:27 lwa joined #marpa
07:28 ronsavage joined #marpa
07:37 CQ joined #marpa
07:40 CQ anyone awake? I don't understand how the lexer / parser work logically... I understand the concept, but don't get the order of lexical matching... esp. when which lexemes are considered and why?
07:55 rns joined #marpa
08:01 rns CQ: https://metacpan.org/pod/release/JKEGL/Marpa-R2-2.094000/pod/Scanless/DSL.pod#L0-G1-and-lexemes — L0 is the lexer grammar, its rules are lhs ~ rhs (not G1's ::=); if an L0 lhs is on an rhs of a G1 (parser grammar) rule, such L0 lhs is a lexeme.
08:08 ronsavage CQ: See also http://savage.net.au/Perl-modules/html/marpa.papers/
08:36 CQ thanks, I'll have a look. If I don't solve it in teh next day or two I'll post something in the forums with code.
08:36 rns CQ: good, happy coding.
08:38 CQ how does token matching work? do the tokens have to be completely unambiguous?
08:39 CQ the proble I have is that I have tags (starting with +#@ and ending at a space, easy), comments (starting with // ending at EOL, also not hard), and then normal text, quoted text, and line continuations \\\n
08:39 rns nope, you can have ambiguous tokens with external scanning — https://metacpan.org/pod/release/JKEGL/Marpa-R2-2.095_000/pod/Scanless/R.pod#External-scanning
08:41 CQ the last three are where i don't see how to do it, esp. quotes. I guess I need to look for the first two, and with the quotes if I find one then process tokens until one ends with a quote
08:42 rns re http://irclog.perlgeek.de/marpa/2014-10-02#i_9444263 — you can describe them all in L0, e.g. read after # allow comments in https://metacpan.org/pod/release/JKEGL/Marpa-R2-2.095_000/pod/Scanless/DSL.pod#Synopsis
08:45 rns for normal/quoted text, you can see JSON parser in Marpa test suite — https://github.com/jeffreykegler/Marpa--R2/blob/master/cpan/t/sl_json_ast.t#L274
08:46 rns for quote escaping you can see http://irclog.perlgeek.de/marpa/2014-09-30#i_9432817
09:12 rns left #marpa
09:12 rns joined #marpa
10:50 flaviu1 joined #marpa
13:22 CQ anyone still around? I have an example somewhat doing something, but not the right thing...
13:22 CQ let me know and I can pastebin it
13:23 rns CQ: curious to see it ...
13:24 CQ actually, I think I haev it, just need to see why the data::dumper doesn't output the token names...
13:25 CQ http://paste.debian.net/124157/
13:25 CQ for now I just wanted to get the tokenizing working, that seems to be OK. I still need to add the line continuations with "\$"
13:30 rns add a couple of lines — https://gist.github.com/rns/d9e24130722b98ab0db7 — $Data::Dumper::Deepcopy = 1; for better dumps, lexeme default = action => [ name, value ] latm => 1 to add token names to ast, looks better?
13:35 CQ rns sorry, where does the lexeme default action go?
13:37 rns CQ: can go anywhere, but usually at the grammar start, after :default action. it just say Marpa evaluator produce [ name, value ] arrayref as the lexeme value. Hope I got your question right.
13:41 CQ can you give me just that one line of code? I don't understand if I need to modify my default line, or if to add that line as a new separate line (that doesn't work) Sorry, but I don't understand what you mean
13:41 rns lexeme default = action => [ name, value ] latm => 1
13:42 rns insert after line starting with :default — you can also look in the gist — https://gist.github.com/rns/d9e24130722b98ab0db7#file-with-dumper-pl-L23
13:43 CQ thanks, I thought the "lexeme" was part of your comment, not the code :)
13:45 CQ http://paste.debian.net/124164/ ...here is the current version... could you have a look at teh continuation line? It doesn't seem to grab it, although I would have thought that with the || newline it should find the continuation first
13:47 rns For continuation — can you look in https://gist.github.com/rns/41c605f59a04f39537e0 for an example? I cooked it earlier than I saw you last message. :))
13:48 CQ grr... I think its the single \ in the input...
13:48 CQ just put \\ at the end of that input line and it works
13:49 CQ great, so now I have a complete and correct token stream :)
13:49 CQ this is suppostd to be a text based task manager btw, because todotxt isn't good enough, and the rest doesn't do the tagging and handling of tasks teh way I would like to use it
13:50 rns Will be interesting to see.
13:50 CQ if you're interested in teh concept I can pastebin you a sample task file that I wrote to define the grammar... suggestions welcome
13:52 rns Well, why not. Now you can add semantics as in sl_json.t or just process the ast as in sl_json_ast.t in marpa test suite — https://github.com/jeffreykegler/Marpa--R2/tree/master/cpan/t
13:53 CQ http://paste.debian.net/124166/ here's the tasklist-spec-example-testcase-file ... all in one ;)
13:54 CQ it'll probably turn into something including a preprocessor, that spits out and can rea
13:54 CQ d separate files with all tasks in one project for example, etc. Should be a fun project
13:56 rns Thanks, looking into it ...
13:58 lwa joined #marpa
14:07 rns Well, looks good to my programmer's eye. :)
14:08 rns BTW, do you plan to open-source the code? Re-implementing todo.txt spec in open source also looks like a good creative disruption. :)
14:09 rns Your syntax looks easier on the eye than todo.txt's.
14:14 rns And the grammar serves as a data model, which eases understanding.
14:30 CQ yes, as soon as I have something I have no problem putting it on github etc. or something similar
14:30 CQ problem is if I do it now, I lose the programming experience from figuringit out myself ;)
14:34 rns This I can understand. :)
14:55 CQ rns another question... would you now process the token stream, or try to condense it further? The sequence of nstrings which are the task name would be easier to handle as one token...
14:57 CQ nstring ~ [\w] [^\h\n\r]+
14:57 CQ why doesn't this work?
14:59 rns you need
14:59 rns nstring ~ [\w] nstring_chars
14:59 rns nstring_chars ~ [^\h\n\r]+
15:00 CQ nstring ~ wordstart no_newline
15:00 CQ wordstart ~ [\w]
15:00 CQ no_newline ~ [^\h\n\r]+
15:00 rns You've got the idea.
15:00 CQ just tried this, this breaks the continuation somehow though
15:02 CQ nvm, should be a * instead of +, else it fails on "a "
15:02 rns for continuation you possibly can use a sequence rule with separator => [\]
15:03 rns cont_line ::= line separator => [\]
15:03 rns line ~ [^\]+
15:04 CQ I'll have to think avbout that, I started with a line based approach and failed misearbly ;)
15:05 rns sorry, forget the above rules
15:05 rns cont_line ::= line+ separator => [\\]
15:05 rns line ~ [^\\]+
15:06 jeffreykegler joined #marpa
15:06 rns Well, input is better seen as one big string
15:06 rns you can add [\n]'s as needed
15:07 jeffreykegler I'm coming into this without having seen the context but taking a stab at it ...
15:08 CQ http://paste.debian.net/124180/ current version
15:08 rns CQ is developing a grammar for his text-based todo manager — http://irclog.perlgeek.de/marpa/2014-10-02#i_9444123
15:08 jeffreykegler I would at first glance think of continuations as a special kind of line-internal whitespace with a special lexeme: '\' [\n] (don't know if I have the syntax right)
15:09 jeffreykegler That is, I'd separate whitespace into line-internal and line-terminating, and the continuations would be one of the line-internal lexemes.
15:09 jeffreykegler Does that make any sense in context?
15:09 rns Yes, right on the money.
15:10 rns and that special lexeme can serve as a sequence separator?
15:10 rns per http://irclog.perlgeek.de/marpa/2014-10-02#i_9446164
15:11 jeffreykegler Btw, there's work going on in my apartment, so you'll have to forgive sudden distraction and/or outright sudden disappearance.
15:12 rns No problem, that's actually been the mode of the previous conversation. :)
15:13 jeffreykegler Jean-Damien & I worked out a complex scheme for EMCAScript where newlines are sometimes significant and sometimes not -- someone recently found it and put it on the backlog -- it'd be instructive in this case, I think
15:15 jeffreykegler It's at http://irclog.perlgeek.de/marpa/2014-09-27#i_9422161
15:16 jeffreykegler That is, the cross-references are there.
15:16 jeffreykegler flaviu++ for tracking them down!
15:16 jeffreykegler Personally, doing the lines as a sequence would not be my first choice.
15:17 jeffreykegler I like to keep the sequence rules for simple stuff -- but that's as much a matter of personal taste as anything.
15:27 rns Sure. That automatic semicolon insertion is impressive stuff — still reading to comprehend.
15:30 CQ rns want a last look at the grammar? it seems to work, I get allthe tokens I need... http://paste.debian.net/124186/
15:31 CQ I'll play with the continuations next, and then work on the storage data structures (array of hashes)
15:31 CQ the continuations work, but could probably be simpler
15:32 rns cont_line ::= line+ separator => cont
15:32 rns line ~ [^\\]+
15:32 rns on continuations, my last take is
15:32 rns cont_line ::= line+ separator => cont
15:32 rns line ~ [^\\]+
15:32 rns cont ~ '\' [\n]
15:33 rns but if you are not going to allow \ for escaping the last version would do.
15:34 jeffreykegler If it were a language I were designing I'd allow optional horizontal whitespace between the '\' and the [\n]
15:34 jeffreykegler Thinking that the frustration I save some day might be my own. :-)
15:35 rns :)
15:37 CQ lol
15:37 CQ rns no, no escaping... else that's for tasklist 3.1 for workgroups or some later version ;)
15:38 rns CQ: ok, got it.
15:38 rns re data structures — you can define semantic actions for rules which will, e.g. insert the needed data into the DB once parsed, as in https://metacpan.org/pod/distribution/Marpa-R2/pod/Scanless/DSL.pod
15:38 rns and avoid the need for intermediary storage.
15:40 CQ thanks, I'll look at that in the next few days... need to run now, thanks a lot for the help!!
15:40 rns Good luck.
15:43 CQ actually re DB: I'm not planning on storing in a DB... it should just parse, follow some rules, and dump out to file(s) again. And have a "shell" of sorts to edit tasks (mark done, shift due dates with d+1w for move forward 1 wk, etc.)
15:44 CQ hence the plain array of hashes
15:46 rns I see.
15:49 rns Then you can simply process the ast to build the data structure as in sl_json_ast.t
15:51 rns This must be a bit faster vs. the semantic actions, which use eval().
16:25 sivoais joined #marpa
19:51 qwebirc25568 joined #marpa
19:51 qwebirc25568 test from Marpa::R2 dis page, please ignore. ^)
19:52 rns qwebirc25568
19:52 rns qwebirc25568
19:52 rns qwebirc25568: passed, thank you. :)
19:53 qwebirc25568 left #marpa
19:57 flaviu1 joined #marpa
22:09 flaviu1 joined #marpa
22:13 flaviu1 joined #marpa
22:24 flaviu1 joined #marpa
22:44 flaviu1 joined #marpa

| Channels | #marpa index | Today | | Search | Google Search | Plain-Text | summary