On Oct 22, 2015, at 5:20 PM, Matt Wette <matthew.wette@verizon.net> wrote:


On Oct 18, 2015, at 8:53 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
And I'm grad that you write a new lexer generator (before it I only know
silex), it's great! Would you like to make the generated tokens
compatible with scm-lalr? If so, people may rewrite their lexer module
with your lexer generator, and no need to rewrite the parser. I saw the
token name is string rather than symbol, so I guess it's not compatible
with scm-lalr.

Actually, the lexer-generator uses convention of internally turning certain lexemes, like strings, into symbols like ‘$string, or integers into ‘$fixed.  The argument to the lexer-generator is a “match-table” which says how to map the read items quoted items are identifiers (e.g., “while”) or character sequences (e.g., “+=“) to something the parser wants to see.  For example, if you use the symbol WHILE to denote the source text “while” then you would have an entry (“while” . ‘WHILE) in the match table.   So I think the lexer-generator should be adaptable to other parsers.

I didn’t describe this very well.   I will try again.   The code actually provides a lexical analyzer (aka lexer) generator-generator.   To make a lexer you call make-lexer-generator with a match-table as argument:

 (define gen-lexer (make-lexer-generator match-table))


Then when you pass a generated lexer each time you call the parser: 

  (parse (gen-lexer))

The reason is that the lexer keeps state information (e.g., the beginning-of-line condition).  Now the match table argument indicates how the user wants lexemes, read from the input, to be reported to the parser.   If you want “while” in the input to be reported as ‘WHILE to the parser, then the match table would include an entry ‘(“while” . WHILE).   The generator uses special symbols to represent quoted strings, numbers and comments.  If you want quoted strings returned with the symbol ‘STRING, then the match table would include an entry ‘($string . STRING).  

In many cases I have nyacc "hashify” my parser so that it uses integers instead of symbols.  Here is the match table generated for the hashified matlab parser:

(define mtab

  '(($lone-comm . 1) ($string . 2) ($float . 3) ($fixed . 4) ($ident . 5) (

    ";" . 6) (".'" . 7) ("'" . 8) ("~" . 9) (".^" . 10) (".\\" . 11) ("./" .

    12) (".*" . 13) ("^" . 14) ("\\" . 15) ("/" . 16) ("*" . 17) ("-" . 18) (

    "+" . 19) (">=" . 20) ("<=" . 21) (">" . 22) ("<" . 23) ("~=" . 24) ("=="

    . 25) ("&" . 26) ("|" . 27) (":" . 28) ("case" . 29) ("elseif" . 30) (

    "clear" . 31) ("global" . 32) ("return" . 33) ("otherwise" . 34) ("switch"

    . 35) ("else" . 36) ("if" . 37) ("while" . 38) ("for" . 39) ("," . 40) (

    ")" . 41) ("(" . 42) ("=" . 43) ("]" . 44) ("[" . 45) ("function" . 46) (

    #\newline . 47) ("end" . 48) ($end . 49)))


and here is the match table generated for the non-hashified match table for the same language:

(define mtab

  '(($lone-comm . $lone-comm) ($string . $string) ($float . $float) ($fixed 

    . $fixed) ($ident . $ident) (";" . #{$:;}#) (".'" . $:.') ("'" . $:') ("~"

    . $:~) (".^" . $:.^) (".\\" . $:.\) ("./" . $:./) (".*" . $:.*) ("^" . 

    $:^) ("\\" . $:\) ("/" . $:/) ("*" . $:*) ("-" . $:-) ("+" . $:+) (">=" . 

    $:>=) ("<=" . $:<=) (">" . $:>) ("<" . $:<) ("~=" . $:~=) ("==" . $:==) (

    "&" . $:&) ("|" . $:|) (":" . $::) ("case" . $:case) ("elseif" . $:elseif)

    ("clear" . $:clear) ("global" . $:global) ("return" . $:return) (

    "otherwise" . $:otherwise) ("switch" . $:switch) ("else" . $:else) ("if" 

    . $:if) ("while" . $:while) ("for" . $:for) ("," . $:,) (")" . #{$:\x29;}#

    ) ("(" . #{$:\x28;}#) ("=" . $:=) ("]" . #{$:\x5d;}#) ("[" . #{$:\x5b;}#) 

    ("function" . $:function) (#\newline . #\newline) ("end" . $:end) ($end . 

    $end)))