nyacc 0.73.0 released

nyacc 0.73.0 released

[Matt Wette]

NYACC, for Not Yet Another Compiler Compiler!, is set of guile modules for
generating parsers and lexical analyzers.  It also provides sample parsers
and pretty-printers using SXML trees as an intermediate representation.

Version 0.73.0 has been released.  It provides some minor fixes.  I have not been working on this for a few months due to other priorities.  I am hoping to spend some quality time on this project now. This version fixes a few bugs in the C99 parser and pretty printer.

git clone git://git.savannah.nongnu.org/nyacc.git <git://git.savannah.nongnu.org/nyacc.git>

Or

http://download.savannah.gnu.org/releases/nyacc/ <http://download.savannah.gnu.org/releases/nyacc/>

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

Matt Wette writes:

> Version 0.73.0 has been released.  It provides some minor fixes.  I
> have not been working on this for a few months due to other
> priorities.  I am hoping to spend some quality time on this project
> now. This version fixes a few bugs in the C99 parser and pretty
> printer.

Great!  Just yesterday I released the first version of Mes that can load
and run Nyacc and started work on replacing Mes' own minimal C-parsing
LALR frontend with Nyacc's C99 parser.

Nyacc is awesome!  I wrote a proof of concept ANSI C parser (as small as
it can possibly get) and delayed further work on that.  Ludovic advised
me to look into Nyacc; it took me quite a bit of work to mature Mes into
a state were it could run it...and now I have a better Scheme with a(n
almost?) full C99 parser too!

Thank you!
Greetings,
Jan.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Matt Wette]

> On Dec 25, 2016, at 11:29 PM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> 
> Nyacc is awesome!  I wrote a proof of concept ANSI C parser (as small as
> it can possibly get) and delayed further work on that.  Ludovic advised
> me to look into Nyacc; it took me quite a bit of work to mature Mes into
> a state were it could run it...and now I have a better Scheme with a(n
> almost?) full C99 parser too!

I believe C99 parser is complete.  There may be errors, but I think all the elements are there. 

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

Matt Wette writes:

> I believe C99 parser is complete.  There may be errors, but I think
> all the elements are there.

Great!  As it turns out, I was using some GNU extensions (anonymous
unions inside structs.

I have now picked-up my work on Mes's simple C compiler backend, using
Nyacc.  I have already replaced my LALR parser with Nyacc's AST[0].

How far is the Nyacc's C99 preprocessor?  I have two problems with it.

It would be very helpful if it would disregard anything inside a false
conditional directive.  It seems I cannot conditionally comment things
out that Nyacc does not parse, like

--8<---------------cut here---------------start------------->8---
    #if __GNUC__
    void
    _start ()
    {
      puts ("Hello micro-mes!\n");

      ///int r = main (0,0);
      int r;
      asm (
           "push $0\n\t"
           "push $0\n\t"
           "call main\n\t"
           "movl %%eax,%0\n\t"
           : "=r" (r)
           : //no inputs "" (&main)
           );
    #endif // GNUC
--8<---------------cut here---------------end--------------->8---

==>  micro-mes.c:199: parse failed at state 379, on input ":"

or like GCC headers that Nyacc does not parse, like <assert.h>, which is
a bit of a pain.

Also, it seems like it doesn't like it if an #includ'ed file is meant to
go inside a function, like (simplified example)

--8<---------------cut here---------------start------------->8---
    // main.i
    r = 3;
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
    // main.c
    int
    main ()
    {
      int r;
      #include "main.i"
      return r;
    } 
--8<---------------cut here---------------end--------------->8---


==> ./main.i:2: parse failed at state 43, on input "r"

Greetings,
Jan

[0] https://gitlab.com/janneke/mes/tree/wip-mescc

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Matt Wette]

> On Dec 31, 2016, at 6:15 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> 
> Matt Wette writes:
> 
>> I believe C99 parser is complete.  There may be errors, but I think
>> all the elements are there.
> 
> Great!  As it turns out, I was using some GNU extensions (anonymous
> unions inside structs.
> 
> I have now picked-up my work on Mes's simple C compiler backend, using
> Nyacc.  I have already replaced my LALR parser with Nyacc's AST[0].
> 
> How far is the Nyacc's C99 preprocessor?  I have two problems with it.
> 
> It would be very helpful if it would disregard anything inside a false
> conditional directive.  It seems I cannot conditionally comment things
> out that Nyacc does not parse, like
> 
> --8<---------------cut here---------------start------------->8---
>    #if __GNUC__
>    void
>    _start ()
>    {
>      puts ("Hello micro-mes!\n");
> 
>      ///int r = main (0,0);
>      int r;
>      asm (
>           "push $0\n\t"
>           "push $0\n\t"
>           "call main\n\t"
>           "movl %%eax,%0\n\t"
>           : "=r" (r)
>           : //no inputs "" (&main)
>           );
>    #endif // GNUC
> --8<---------------cut here---------------end--------------->8---
> 
> ==>  micro-mes.c:199: parse failed at state 379, on input “:"

I need to document better.  There is an argument to handle this.  I think this may accomplish what you want:

(define (my-xdef? name mode) (if (equal? name “__GNUC__”) #f (env? mode ‘code))

(parse-c99 #:xdef? my-xdef? …)

> headers that Nyacc does not parse, like <assert.h>, which is
> a bit of a pain.

(define my-td-dict ‘((“assert.h”)))

(parse-c99 #:td-dict my-td-dict …)

See std-dict in nyacc/lang/c99/body.scm.  This is added by default, but should probably not be.
td-dict is an a-list of include files with typedefs in those files.


> 
> Also, it seems like it doesn't like it if an #includ'ed file is meant to
> go inside a function, like (simplified example)
> 
> --8<---------------cut here---------------start------------->8---
>    // main.i
>    r = 3;
> --8<---------------cut here---------------end--------------->8---
> 
> --8<---------------cut here---------------start------------->8---
>    // main.c
>    int
>    main ()
>    {
>      int r;
>      #include "main.i"
>      return r;
>    } 
> --8<---------------cut here---------------end--------------->8---
> 
> 
> ==> ./main.i:2: parse failed at state 43, on input “r"

This seems like a bug to me.  I will check it out.

Re: nyacc 0.73.0 released

[Matt Wette]

> On Dec 31, 2016, at 7:57 AM, Matt Wette <matt.wette@gmail.com> wrote:

>   I think this may accomplish what you want:
> 
> (define (my-xdef? name mode) (if (equal? name “__GNUC__”) #f (env? mode ‘code))
> 
> (parse-c99 #:xdef? my-xdef? …)
> 

The idea here is that there are two “modes” for parsing.  "file" mode will pass the CPP statements to the AST and “code” mode will not.  There is a procedure keyword argument #mode to control these.

Matt

Re: nyacc 0.73.0 released

[Matt Wette]

> On Dec 31, 2016, at 6:15 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> 
> Matt Wette writes:
> 
>> I believe C99 parser is complete.  There may be errors, but I think
>> all the elements are there.

> Also, it seems like it doesn't like it if an #includ'ed file is meant to
> go inside a function, like (simplified example)
> 
> --8<---------------cut here---------------start------------->8---
>    // main.i
>    r = 3;
> --8<---------------cut here---------------end--------------->8---
> 
> --8<---------------cut here---------------start------------->8---
>    // main.c
>    int
>    main ()
>    {
>      int r;
>      #include "main.i"
>      return r;
>    } 
> --8<---------------cut here---------------end--------------->8---
> 
> 
> ==> ./main.i:2: parse failed at state 43, on input "r"

Jan,

So this is a bug.  And I am not sure how to proceed yet.  The parser was designed to return a AST with code from includes under a subtree.  This allows a file to be processed with respect to code only in that file.  The way it works is to parse the included file as a new parse.  That only works at the top level.  This breaks for files included inside functions etc.  

How are you processing the file?  Do you want to be able to discriminate between a file and the included files, and only if included files are at the decl level (e.g., #include <stdio.h>) but code inside functions gets included as is (i.e., the AST looks as if the code was in the parent file)?

This will take some sort of parser-lexer hook I think.  I want to think about a clean architecture for all the use cases.

Matt

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

Matt Wette writes:

>     On Dec 31, 2016, at 7:57 AM, Matt Wette <matt.wette@gmail.com> wrote:
>
>       I think this may accomplish what you want:
>    
>     (define (my-xdef? name mode) (if (equal? name “__GNUC__”) #f (env? mode ‘code))
>    
>     (parse-c99 #:xdef? my-xdef? …)
>
> The idea here is that there are two “modes” for parsing.  "file" mode will pass the
> CPP statements to the AST and “code” mode will not.  There is a procedure keyword
> argument #mode to control these.

Ah, I get the idea, but not the suggested code.  I don't have the procedure
`env?'

I'm now using

    (define (gnuc-xdef? name mode)
      (cond ((equal? name "__GNUC__") #t)
            ((equal? name "asm") #f)))

    (define (mescc)
      (parse-c99 #:inc-dirs (string-split (getenv "C_INCLUDE_PATH") #\:)
                 #:cpp-defs '(("__GNUC__" . "0"))
                 #:xdef? gnuc-xdef?))

together with the attached patch (I'm sure is not right).  Something
like this seems necessary for (if,elif,endif) to be expanded.  Setting
eval-flow? always to #t does not work.

It's very handy that #if now works, making a lot of round trips
between gcc and mescc to bootstrap the compiler.

Greetings,
Jan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-nyacc-c99-always-expand-cpp-conditionals.patch --]
[-- Type: text/x-patch, Size: 1527 bytes --]

From c737ebb8fbaeff75e8914a695951be466cf3d695 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sat, 31 Dec 2016 20:35:50 +0100
Subject: [PATCH] nyacc: c99: always expand cpp conditionals.

* module/nyacc/lang/c99/body.scm (gen-c-lexer): Always expand conditionals.
---
 module/nyacc/lang/c99/body.scm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/module/nyacc/lang/c99/body.scm b/module/nyacc/lang/c99/body.scm
index b3e7843..a881849 100644
--- a/module/nyacc/lang/c99/body.scm
+++ b/module/nyacc/lang/c99/body.scm
@@ -319,7 +319,7 @@
 		 (rem-define (cadr stmt)))
 		((if) ;; and ifdef, ifndef
 		 (cpi-push)
-		 (if (eval-flow?)
+		 (if #t ;;(eval-flow?)
 		     (let* ((defs (cpi-defs info))
 			    (rhs (cpp-expand-text (cadr stmt) defs))
 			    ;; rhs = "defined(1)" :(
@@ -333,7 +333,7 @@
 			(else
 			 (set! skip (cons* 'skip-one (car skip) skip)))))))
 		((elif)
-		 (if (eval-flow?)
+		 (if #t ;;(eval-flow?)
 		     (let* ((defs (cpi-defs info))
 			    (rhs (cpp-expand-text (cadr stmt) defs))
 			    (exp (parse-cpp-expr rhs))
@@ -353,7 +353,7 @@
 			 (set! skip (cons* 'skip-one 'skip (cdr skip))))))
 		     (cpi-shift)))
 		((else)
-		 (if (eval-flow?)
+		 (if #t ;; (eval-flow?)
 		     (cond
 		      ((eq? 'skip-look (car skip))
 		       (cpi-shift)
@@ -363,7 +363,7 @@
 		     (cpi-shift)))
 		((endif)
 		 (cpi-pop)
-		 (if (eval-flow?)
+		 (if #t ;; (eval-flow?)
 		     (set! skip (cons 'skip-one (cdr skip)))))
 		((error)
 		 stmt)
-- 
2.10.2


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

Matt Wette writes:

Hi Matt!

>> Also, it seems like it doesn't like it if an #includ'ed file is meant to
>> go inside a function, like (simplified example)

> So this is a bug.  And I am not sure how to proceed yet.  The parser
> was designed to return a AST with code from includes under a subtree.
> This allows a file to be processed with respect to code only in that
> file.  The way it works is to parse the included file as a new parse.
> That only works at the top level.  This breaks for files included
> inside functions etc.

That makes sense.

> How are you processing the file?  Do you want to be able to
> discriminate between a file and the included files, and only if
> included files are at the decl level (e.g., #include <stdio.h>) but
> code inside functions gets included as is (i.e., the AST looks as if
> the code was in the parent file)?

For a user, best is to have error messages that point to the file+line
where the actual code is; so in each case the included file.  Gcc's cpp
always adds # <line> "<file>" markers; it makes no difference where the
file is included.

What I am currently doing, is generate lists of symbol declarations and
definitions by snarfing the original mes.c file.  The declarations are
included at toplevel, which is fine.  The definitions go inside
functions.

So for my specific use case it does not really much either way, although
having correct file:line:message error messages is nice. ;-)

> This will take some sort of parser-lexer hook I think.  I want to
> think about a clean architecture for all the use cases.

Sure.

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

Matt Wette writes:

Hi Matt!

Find small bugfix for GOTO attached.

I'm making some progress with Mescc.  The compiler layout starts to make
some sense now and after getting function calls and parameters in place
and many special cases for call/if/for/when it seems that I finally have
an idea that works for handling tests and expressions.

So, now I'm starting list of tests that uses goto, which i'll need
anyway for the core evaluator.

Greetings,
Jan.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-nyacc-bugfix-unquoto-goto-label.patch --]
[-- Type: text/x-patch, Size: 1401 bytes --]

From 19310561509a9c8babd57e0067d49b10980cf388 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sat, 7 Jan 2017 12:00:59 +0100
Subject: [PATCH] bugfix: unquoto goto label.

* module/nyacc/lang/c99/mach.d/c99act.scm (act-v): Unquote goto label.
---
 module/nyacc/ChangeLog                  | 5 +++++
 module/nyacc/lang/c99/mach.d/c99act.scm | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/module/nyacc/ChangeLog b/module/nyacc/ChangeLog
index 1b08290..549a544 100644
--- a/module/nyacc/ChangeLog
+++ b/module/nyacc/ChangeLog
@@ -1,3 +1,8 @@
+2017-01-07  Jan Nieuwenhuizen  <janneke@gnu.org>
+
+	* module/nyacc/lang/c99/mach.d/c99act.scm (act-v): Unquote goto
+	label.
+
 2017-01-02  Matt Wette  <mwette@alumni.caltech.edu>
 
 	* lang/c99/cppbody.scm (expand-cpp-mref): skip ws between ident
diff --git a/module/nyacc/lang/c99/mach.d/c99act.scm b/module/nyacc/lang/c99/mach.d/c99act.scm
index 10b5ffe..17f3f43 100644
--- a/module/nyacc/lang/c99/mach.d/c99act.scm
+++ b/module/nyacc/lang/c99/mach.d/c99act.scm
@@ -690,7 +690,7 @@
    ;; opt-expression => expression
    (lambda ($1 . $rest) $1)
    ;; jump-statement => "goto" identifier ";"
-   (lambda ($3 $2 $1 . $rest) `(goto $2))
+   (lambda ($3 $2 $1 . $rest) `(goto ,$2))
    ;; jump-statement => "continue" ";"
    (lambda ($2 $1 . $rest) '(continue))
    ;; jump-statement => "break" ";"
-- 
2.10.2


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Matt Wette]

> On Jan 7, 2017, at 3:14 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> 
> Matt Wette writes:
> 
> Hi Matt!
> 
> Find small bugfix for GOTO attached.


> 
> --- a/module/nyacc/lang/c99/mach.d/c99act.scm
> +++ b/module/nyacc/lang/c99/mach.d/c99act.scm
> @@ -690,7 +690,7 @@
>    ;; opt-expression => expression
>    (lambda ($1 . $rest) $1)
>    ;; jump-statement => "goto" identifier ";"
> -   (lambda ($3 $2 $1 . $rest) `(goto $2))
> +   (lambda ($3 $2 $1 . $rest) `(goto ,$2))
>    ;; jump-statement => "continue" ";"
>    (lambda ($2 $1 . $rest) '(continue))
>    ;; jump-statement => "break" ";"
> -- 

Thanks.  The bug is actually in lang/c99/mach.scm.  c99act.scm is output from running make-lalr-{spec,machine} on mach.scm,

    (jump-statement			; S 6.8.6
     ("goto" identifier ";" ($$ `(goto ,$2))). ;; <= was `(goto $2)
     ("continue" ";" ($$ '(continue)))
     ...

Also, I am currently working on the fixes branch to cleanup the C-preprocessor support so that #:mode ‘code works more robustly.

Matt

Re: nyacc 0.73.0 released

[Jan Nieuwenhuizen]

Matt Wette writes:

> Thanks.  The bug is actually in lang/c99/mach.scm.  c99act.scm is output from running make-lalr-{spec,machine} on mach.scm,
>
>     (jump-statement			; S 6.8.6
>      ("goto" identifier ";" ($$ `(goto ,$2))). ;; <= was `(goto $2)
>      ("continue" ";" ($$ '(continue)))
>      ...

Ah!  :-)
Thanks.

> Also, I am currently working on the fixes branch to cleanup the
>C-preprocessor support so that #:mode ‘code works more robustly.

That's great.  In the meantime my expression/test template starts to
work, so mescc can compile all kinds of basic C stuff now.  Next up:
more arithmetic stuff and I'll be looking at globals, typedefs, structs.

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: nyacc 0.73.0 released

[Chaos Eternal]

This small code does not work with example/cxp:

/*Begin C Code*/
#if defined __HAB__
#define __HAB__
#endif
#define A 1

int a = A;
/*end C code*/

cxp output:
$ ./cxp kk2.h
(unknown):1: bad CPP defined
kk2.h:1: CPP error
kk2.h:
#f


On Mon, Dec 26, 2016 at 12:06 PM Matt Wette <matt.wette@gmail.com> wrote:

> NYACC, for Not Yet Another Compiler Compiler!, is set of guile modules for
> generating parsers and lexical analyzers.  It also provides sample parsers
> and pretty-printers using SXML trees as an intermediate representation.
>
> Version 0.73.0 has been released.  It provides some minor fixes.  I have
> not been working on this for a few months due to other priorities.  I am
> hoping to spend some quality time on this project now. This version fixes a
> few bugs in the C99 parser and pretty printer.
>
> git clone git://git.savannah.nongnu.org/nyacc.git <git://
> git.savannah.nongnu.org/nyacc.git>
>
> Or
>
> http://download.savannah.gnu.org/releases/nyacc/ <
> http://download.savannah.gnu.org/releases/nyacc/>
>
>
>

Re: nyacc 0.73.0 released

[Matt Wette]

Thanks.  This still in 0.74.1.  I’ve been working on the CPP a lot the last week.  Will work it. — Matt


> On Jan 10, 2017, at 6:33 PM, Chaos Eternal <chaoseternal@shlug.org> wrote:
> 
> This small code does not work with example/cxp:
> 
> /*Begin C Code*/
> #if defined __HAB__
> #define __HAB__
> #endif
> #define A 1
> 
> int a = A;
> /*end C code*/
> 
> cxp output:
> $ ./cxp kk2.h 
> (unknown):1: bad CPP defined
> kk2.h:1: CPP error
> kk2.h:
> #f
> 
> 
> On Mon, Dec 26, 2016 at 12:06 PM Matt Wette <matt.wette@gmail.com <mailto:matt.wette@gmail.com>> wrote:
> NYACC, for Not Yet Another Compiler Compiler!, is set of guile modules for
> generating parsers and lexical analyzers.  It also provides sample parsers
> and pretty-printers using SXML trees as an intermediate representation.
> 
> Version 0.73.0 has been released.  It provides some minor fixes.  I have not been working on this for a few months due to other priorities.  I am hoping to spend some quality time on this project now. This version fixes a few bugs in the C99 parser and pretty printer.
> 
> git clone git://git.savannah.nongnu.org/nyacc.git <http://git.savannah.nongnu.org/nyacc.git> <git://git.savannah.nongnu.org/nyacc.git <http://git.savannah.nongnu.org/nyacc.git>>
> 
> Or
> 
> http://download.savannah.gnu.org/releases/nyacc/ <http://download.savannah.gnu.org/releases/nyacc/> <http://download.savannah.gnu.org/releases/nyacc/ <http://download.savannah.gnu.org/releases/nyacc/>>
> 
>