Nyacc: how to lex comments?

Nyacc: how to lex comments?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

Hi,

in my quest to play parsing, I'm failing when I try to cope with
comments.

Here's an excerpt from my current file

    (define my-grammar
      (lalr-spec
       (start my-file)
       (grammar
        ;; boring grammar details elided
           )))
    
    (define mach (make-lalr-machine aq-grammar))
    (define mtab (lalr-match-table mach))
    (define gen-lexer (make-lexer-generator mtab
                       #:comm-reader (make-comm-reader '(("#" . "\n")))))
    (define raw-parse (make-lalr-parser mach))
    (define (parse) (raw-parse (gen-lexer)))
    
    ;; (gen-lexer)
    (parse)

Comments are shell-like, from "#" to end of line. Grammar
works fine so-far, but when (parse) sees a hash-comment
(even just a one-line input

  # this is a comment

it panics:

    Backtrace:
    In ice-9/boot-9.scm:
      1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
    In unknown file:
               5 (apply-smob/0 #<thunk 557754319740>)
    In ice-9/boot-9.scm:
        718:2  4 (call-with-prompt ("prompt") #<procedure 5577543308e0 …> …)
    In ice-9/eval.scm:
        619:8  3 (_ #(#(#<directory (guile-user) 5577543c1f00>)))
    In ice-9/boot-9.scm:
       2806:4  2 (save-module-excursion #<procedure 5577543b31b0 at ice-…>)
      4351:12  1 (_)
    In nyacc/parse.scm:
         78:4  0 (_ _ #:debug _)
    
    nyacc/parse.scm:78:4: Throw to key `nyacc-error' with args `("~A:~A: parse failed at state ~A, on input ~S" "(unknown)" 1 0 " a comment")'.

I feel I'm doing something stupid. What can I do to understand
what's going on?

Thanks & cheers
-- tomás



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Nyacc: how to lex comments?

[Matt Wette]

On 3/16/20 2:39 PM, tomas@tuxteam.de wrote:
> Hi,
>
> in my quest to play parsing, I'm failing when I try to cope with
> comments.
>
> Here's an excerpt from my current file
>
>      (define my-grammar
>        (lalr-spec
>         (start my-file)
>         (grammar
>          ;; boring grammar details elided
>             )))
>      
>      (define mach (make-lalr-machine aq-grammar))
>      (define mtab (lalr-match-table mach))
>      (define gen-lexer (make-lexer-generator mtab
>                         #:comm-reader (make-comm-reader '(("#" . "\n")))))
>      (define raw-parse (make-lalr-parser mach))
>      (define (parse) (raw-parse (gen-lexer)))
>      
>      ;; (gen-lexer)
>      (parse)
>
> Comments are shell-like, from "#" to end of line. Grammar
> works fine so-far, but when (parse) sees a hash-comment
> (even just a one-line input
>
>    # this is a comment
>
> it panics:
>
>      Backtrace:
>      In ice-9/boot-9.scm:
>        1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>      In unknown file:
>                 5 (apply-smob/0 #<thunk 557754319740>)
>      In ice-9/boot-9.scm:
>          718:2  4 (call-with-prompt ("prompt") #<procedure 5577543308e0 …> …)
>      In ice-9/eval.scm:
>          619:8  3 (_ #(#(#<directory (guile-user) 5577543c1f00>)))
>      In ice-9/boot-9.scm:
>         2806:4  2 (save-module-excursion #<procedure 5577543b31b0 at ice-…>)
>        4351:12  1 (_)
>      In nyacc/parse.scm:
>           78:4  0 (_ _ #:debug _)
>      
>      nyacc/parse.scm:78:4: Throw to key `nyacc-error' with args `("~A:~A: parse failed at state ~A, on input ~S" "(unknown)" 1 0 " a comment")'.
>
> I feel I'm doing something stupid. What can I do to understand
> what's going on?
>
> Thanks & cheers
> -- tomás
>
>
make-comm-reader will not eat newlines at end of comment by default.
If you want the comment # ... \n to include the newline try adding 
#:eat-newline #t.

Matt

Re: Nyacc: how to lex comments?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

On Mon, Mar 16, 2020 at 05:11:59PM -0700, Matt Wette wrote:
> On 3/16/20 2:39 PM, tomas@tuxteam.de wrote:
> >Hi,
> >
> >in my quest to play parsing, I'm failing when I try to cope with
> >comments.
> >

[...]

> make-comm-reader will not eat newlines at end of comment by default.
> If you want the comment # ... \n to include the newline try adding
> #:eat-newline #t.

D'oh, thanks. Also, I noticed I've to set #:comm-skipper in 'gen-lexer',
instead of #:comm-reader.

I didn't quite understand the difference between both. Would I have
to include comments explicitly in the grammar if the lexer uses
#:comment-reader? 

Thanks again
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Nyacc: how to lex comments?

[Matt Wette]

On 3/17/20 12:46 AM, tomas@tuxteam.de wrote:
> On Mon, Mar 16, 2020 at 05:11:59PM -0700, Matt Wette wrote:
>> On 3/16/20 2:39 PM, tomas@tuxteam.de wrote:
>>> Hi,
>>>
>>> in my quest to play parsing, I'm failing when I try to cope with
>>> comments.
>>>
> [...]
>
>> make-comm-reader will not eat newlines at end of comment by default.
>> If you want the comment # ... \n to include the newline try adding
>> #:eat-newline #t.
> D'oh, thanks. Also, I noticed I've to set #:comm-skipper in 'gen-lexer',
> instead of #:comm-reader.
>
> I didn't quite understand the difference between both. Would I have
> to include comments explicitly in the grammar if the lexer uses
> #:comment-reader?
>
> Thanks again
> -- tomás
Yes.  Use comm-reader if your grammar accepts comments.
Use comm-skipper if your grammar does not.

The ffi-helper does read comments in the C grammar and
dumps them in the output files, for example.

Re: Nyacc: how to lex comments?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

On Tue, Mar 17, 2020 at 05:31:32AM -0700, Matt Wette wrote:
> 
> 
> On 3/17/20 12:46 AM, tomas@tuxteam.de wrote:

[...]

> >I didn't quite understand the difference between both. Would I have
> >to include comments explicitly in the grammar if the lexer uses
> >#:comment-reader?
> >
> >Thanks again
> >-- tomás
> Yes.  Use comm-reader if your grammar accepts comments.
> Use comm-skipper if your grammar does not.
> 
> The ffi-helper does read comments in the C grammar and
> dumps them in the output files, for example.

Ah, I see, thanks. Thank you for your invaluable help!

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]