ECMAScript support broken?

ECMAScript support broken?

[Kan-Ru Chen]

Hi,

I recently want to start a small project that uses guile as the
JavaScript interpreter, but the current git HEAD has errors regard
object definition.

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,L ecmascript
Happy hacking with ECMAScript!  To switch back, type `,L scheme'.
ecmascript@(guile-user)> var test = {};
While compiling expression:
ERROR: unrecognized tree-il ()
ecmascript@(guile-user)> var test = { bar: 1 };
While compiling expression:
ERROR: unrecognized tree-il ((apply (primitive cons) (const bar) (const 1)))
--8<---------------cut here---------------end--------------->8---


Happy new year!
Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

Re: ECMAScript support broken?

[Noah Lavine]

Hi all,

I looked at this. For the first error, I think the bug is in (language
ecmascript compile-tree-il), where the file around line 369 says:

      ((object . ,args)
           (@impl new-object
                  (map (lambda (x)
                         (pmatch x
                                 ((,prop ,val)
                                  (-> (apply (-> (primitive 'cons))
                                             (-> (const prop))
                                             (comp val e))))
                                 (else
                                  (error "bad prop-val pair" x))))
                       args)))

This produces (@impl new-object ()) if args is '(), but it should just
be (@impl new-object). I tried changing that bit to

      ((object . ,args)
       (if (null? args)
           (@impl new-object)
           (@impl new-object
                  (map (lambda (x)
                         (pmatch x
                                 ((,prop ,val)
                                  (-> (apply (-> (primitive 'cons))
                                             (-> (const prop))
                                             (comp val e))))
                                 (else
                                  (error "bad prop-val pair" x))))
                       args))))

and it now works. (It's terribly hackish - does anyone have a better
solution?) I don't have time to look at the rest right now, but I'll
try to soon unless someone else figures it out (please :-) ).

Noah


On Sun, Jan 2, 2011 at 11:46 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
> Hi,
>
> I recently want to start a small project that uses guile as the
> JavaScript interpreter, but the current git HEAD has errors regard
> object definition.
>
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> ,L ecmascript
> Happy hacking with ECMAScript!  To switch back, type `,L scheme'.
> ecmascript@(guile-user)> var test = {};
> While compiling expression:
> ERROR: unrecognized tree-il ()
> ecmascript@(guile-user)> var test = { bar: 1 };
> While compiling expression:
> ERROR: unrecognized tree-il ((apply (primitive cons) (const bar) (const 1)))
> --8<---------------cut here---------------end--------------->8---
>
>
> Happy new year!
> Kanru
> --
> A badly written book is only a blunder. A bad translation of a good
> book is a crime.
>                -- Gilbert Highet
>
>

Re: ECMAScript support broken?

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2870 bytes --]

Hello again,

Second problem solved as well. The attached patch fixes both issues.

Anything else you notice being wrong with ecmascript? (I mean our
implementation of it, of course ...)

Noah

On Sun, Jan 2, 2011 at 10:19 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> Hi all,
>
> I looked at this. For the first error, I think the bug is in (language
> ecmascript compile-tree-il), where the file around line 369 says:
>
>      ((object . ,args)
>           (@impl new-object
>                  (map (lambda (x)
>                         (pmatch x
>                                 ((,prop ,val)
>                                  (-> (apply (-> (primitive 'cons))
>                                             (-> (const prop))
>                                             (comp val e))))
>                                 (else
>                                  (error "bad prop-val pair" x))))
>                       args)))
>
> This produces (@impl new-object ()) if args is '(), but it should just
> be (@impl new-object). I tried changing that bit to
>
>      ((object . ,args)
>       (if (null? args)
>           (@impl new-object)
>           (@impl new-object
>                  (map (lambda (x)
>                         (pmatch x
>                                 ((,prop ,val)
>                                  (-> (apply (-> (primitive 'cons))
>                                             (-> (const prop))
>                                             (comp val e))))
>                                 (else
>                                  (error "bad prop-val pair" x))))
>                       args))))
>
> and it now works. (It's terribly hackish - does anyone have a better
> solution?) I don't have time to look at the rest right now, but I'll
> try to soon unless someone else figures it out (please :-) ).
>
> Noah
>
>
> On Sun, Jan 2, 2011 at 11:46 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
>> Hi,
>>
>> I recently want to start a small project that uses guile as the
>> JavaScript interpreter, but the current git HEAD has errors regard
>> object definition.
>>
>> --8<---------------cut here---------------start------------->8---
>> scheme@(guile-user)> ,L ecmascript
>> Happy hacking with ECMAScript!  To switch back, type `,L scheme'.
>> ecmascript@(guile-user)> var test = {};
>> While compiling expression:
>> ERROR: unrecognized tree-il ()
>> ecmascript@(guile-user)> var test = { bar: 1 };
>> While compiling expression:
>> ERROR: unrecognized tree-il ((apply (primitive cons) (const bar) (const 1)))
>> --8<---------------cut here---------------end--------------->8---
>>
>>
>> Happy new year!
>> Kanru
>> --
>> A badly written book is only a blunder. A bad translation of a good
>> book is a crime.
>>                -- Gilbert Highet
>>
>>
>

[-- Attachment #2: 0001-Ecmascript-Compiler-Fix.patch --]
[-- Type: application/octet-stream, Size: 1855 bytes --]

From 00323240da703c9364bf9ec0f160c816f9adc0fb Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Mon, 3 Jan 2011 02:22:35 -0500
Subject: [PATCH] Ecmascript Compiler Fix

This fixes two bugs in the ECMAScript compiler that were both fundamentally
due to a lack of splicing in the construction of some tree-il.
---
 module/language/ecmascript/compile-tree-il.scm |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/module/language/ecmascript/compile-tree-il.scm b/module/language/ecmascript/compile-tree-il.scm
index a97e555..a5276ab 100644
--- a/module/language/ecmascript/compile-tree-il.scm
+++ b/module/language/ecmascript/compile-tree-il.scm
@@ -366,16 +366,16 @@
        `(apply ,(@implv new-array)
                ,@(map (lambda (x) (comp x e)) args)))
       ((object . ,args)
-       (@impl new-object
-              (map (lambda (x)
-                     (pmatch x
-                       ((,prop ,val)
-                        (-> (apply (-> (primitive 'cons))
-                                   (-> (const prop))
-                                   (comp val e))))
-                       (else
-                        (error "bad prop-val pair" x))))
-                   args)))
+       `(apply (@ (language ecmascript impl) new-object)
+               ,@(map (lambda (x)
+                         (pmatch x
+                                 ((,prop ,val)
+                                  (-> (apply (-> (primitive 'cons))
+                                             (-> (const prop))
+                                             (comp val e))))
+                                 (else
+                                  (error "bad prop-val pair" x))))
+                       args)))
       ((pref ,obj ,prop)
        (@impl pget
               (comp obj e)
-- 
1.7.3.2

Re: ECMAScript support broken?

[Kan-Ru Chen]

Hi,

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hi all,
>
> I looked at this. For the first error, I think the bug is in (language
> ecmascript compile-tree-il), where the file around line 369 says:
>
>       ((object . ,args)
>            (@impl new-object
>                   (map (lambda (x)
>                          (pmatch x
>                                  ((,prop ,val)
>                                   (-> (apply (-> (primitive 'cons))
>                                              (-> (const prop))
>                                              (comp val e))))
>                                  (else
>                                   (error "bad prop-val pair" x))))
>                        args)))

I don't know the guile internal (yet), but after looking around
(language tree-il) and (language ecmascript base) it seems `new-object'
want a list of pairs instead a list of lists.  I changed the above code
to

       `(apply ,(@implv new-object)
               ,@(map (lambda (x)
                        (pmatch x
                                ((,prop ,val)
                                 (-> (apply (-> (primitive 'cons))
                                            (-> (const prop))
                                            (comp val e))))
                                (else
                                 (error "bad prop-val pair" x))))
                      args)))

and it works.

Hope this helps you to diagnose the problem.

Cheers,
Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

Re: ECMAScript support broken?

[Kan-Ru Chen]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello again,
>
> Second problem solved as well. The attached patch fixes both issues.
>
> Anything else you notice being wrong with ecmascript? (I mean our
> implementation of it, of course ...)

Hi,

I got the same answer (previous mail), I should check mail before send it ;-)

I plan to run some ecmascript test suite as well as the library I want
to use in my project.  Do you know how to run guile as a ecmascript
interpreter from command line?

Cheers,
Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

Re: ECMAScript support broken?

[Noah Lavine]

Hello,

> I got the same answer (previous mail), I should check mail before send it ;-)

I like your use of @implv, though.

> I plan to run some ecmascript test suite as well as the library I want
> to use in my project.  Do you know how to run guile as a ecmascript
> interpreter from command line?

If you mean give guile a '.js' file have it interpret that with
ecmascript, then I think it's not possible right now, although I
suspect that such a feature could be added easily.

If you have a file that you only want to use with Guile, though, you
might be able to put ",L 'ecmascript" as the first line.

Noah

Re: ECMAScript support broken?

[Ludovic Courtès]

Hello Noah & happy new year!

Thanks for the patch!

Can you resend it with a proper ChangeLog-style log and a corresponding
test in ecmascript.test?  That would be great.  :-)

Ludo’.

Re: ECMAScript support broken?

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

Hello,

Here's a new version. I think the commit message is better - please
let me know if this is not right. There's also a test for it.

Noah

On Mon, Jan 3, 2011 at 6:00 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello Noah & happy new year!
>
> Thanks for the patch!
>
> Can you resend it with a proper ChangeLog-style log and a corresponding
> test in ecmascript.test?  That would be great.  :-)
>
> Ludo’.
>
>
>

[-- Attachment #2: 0001-Ecmascript-Compiler-Fix.patch --]
[-- Type: application/octet-stream, Size: 2405 bytes --]

From f57e253826584c6d3d5f8dadfe02869047905299 Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Mon, 3 Jan 2011 02:22:35 -0500
Subject: [PATCH] Ecmascript Compiler Fix

* module/language/ecmascript/compile-tree-il.scm (compile-tree-il):
  generate correct tree-il for construction of new objects.
* test-suite/tests/ecmascript.test: test whether we generate new
  objects correctly.
---
 module/language/ecmascript/compile-tree-il.scm |   20 ++++++++++----------
 test-suite/tests/ecmascript.test               |    1 +
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/module/language/ecmascript/compile-tree-il.scm b/module/language/ecmascript/compile-tree-il.scm
index a97e555..a5276ab 100644
--- a/module/language/ecmascript/compile-tree-il.scm
+++ b/module/language/ecmascript/compile-tree-il.scm
@@ -366,16 +366,16 @@
        `(apply ,(@implv new-array)
                ,@(map (lambda (x) (comp x e)) args)))
       ((object . ,args)
-       (@impl new-object
-              (map (lambda (x)
-                     (pmatch x
-                       ((,prop ,val)
-                        (-> (apply (-> (primitive 'cons))
-                                   (-> (const prop))
-                                   (comp val e))))
-                       (else
-                        (error "bad prop-val pair" x))))
-                   args)))
+       `(apply (@ (language ecmascript impl) new-object)
+               ,@(map (lambda (x)
+                         (pmatch x
+                                 ((,prop ,val)
+                                  (-> (apply (-> (primitive 'cons))
+                                             (-> (const prop))
+                                             (comp val e))))
+                                 (else
+                                  (error "bad prop-val pair" x))))
+                       args)))
       ((pref ,obj ,prop)
        (@impl pget
               (comp obj e)
diff --git a/test-suite/tests/ecmascript.test b/test-suite/tests/ecmascript.test
index 955296d..2d497e8 100644
--- a/test-suite/tests/ecmascript.test
+++ b/test-suite/tests/ecmascript.test
@@ -65,6 +65,7 @@
   (ecompile "true;" #t)
   (ecompile "2 + 2;" 4)
   (ecompile "\"hello\";" "hello")
+  (ecompile "var test = { bar: 1 };")
 
   ;; FIXME: Broken!
   ;; (ecompile "[1,2,3,4].map(function(x) { return x * x; });"
-- 
1.7.3.2

Sputnik test result (was Re: ECMAScript support broken?)

[Kan-Ru Chen]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

Hi,

Noah Lavine <noah.b.lavine@gmail.com> writes:

> If you mean give guile a '.js' file have it interpret that with
> ecmascript, then I think it's not possible right now, although I
> suspect that such a feature could be added easily.

Right, I've cooked a little script to interpret .js files directly.

I ran the sputniktests[1] from google using the attached guile-es-parse
script, which only tests the parser.

  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log

The result is impressive (full log attached):

  === Summary ===
   - Ran 5246 tests
   - Passed 4410 tests (84.1%)
   - Failed 836 tests (15.9%)

Where the failed tests have

   - 245 unicode errors (unicode literal is not supported)
   - 393 rbrace errors  (see below)
   - 39 Math.LN2 errors (see below)
   - 159 remain to sort out

The rbrace errors are from

   function test() {}
   // Syntax error: unexpected token :  in form rbrace

   function foo() { this.bar = function() { return 0; } };
   // Syntax error: unexpected token :  in form rbrace

I also tried to compile the parsed tests, but halted because too many
errors like

   Object.prototype.toString = function () {return "something";};
   // No applicable method for #<<generic> pput (6)> in call (pput
   // #<unbound> toString #<procedure 1e1c438 ()>)

I thought the tests won't run correctly without this.

[1]: https://code.google.com/p/sputniktests/

Cheers,
Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

[-- Attachment #2: guile-es-parse --]
[-- Type: text/plain, Size: 208 bytes --]

#!../meta/guile \
-e main -s
!#

(import (language ecmascript parse))

(define (main args)
  (if (> (length args) 1)
      (let ((es-src (cadr args)))
        (call-with-input-file es-src read-ecmascript))))

[-- Attachment #3: guile-es-parser-test-log.gz --]
[-- Type: application/octet-stream, Size: 54373 bytes --]

Re: ECMAScript support broken?

[Kan-Ru Chen]

Hi,

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello,
>
> Here's a new version. I think the commit message is better - please
> let me know if this is not right. There's also a test for it.

> +       `(apply (@ (language ecmascript impl) new-object)

I think if you use ,(@implv new-object) here it would be more consistent
with the ,(@implv new-array) case just above this code snippet :)

Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

Re: ECMAScript support broken?

[Ludovic Courtès]

Hi,

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Here's a new version. I think the commit message is better - please
> let me know if this is not right. There's also a test for it.

Great, thanks!  Pushed with a change to allow the test to actually pass.

Ludo’.

Re: ECMAScript support broken?

[Noah Lavine]

Yes, I agree.

On Tue, Jan 4, 2011 at 11:52 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
> Hi,
>
> Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> Hello,
>>
>> Here's a new version. I think the commit message is better - please
>> let me know if this is not right. There's also a test for it.
>
>> +       `(apply (@ (language ecmascript impl) new-object)
>
> I think if you use ,(@implv new-object) here it would be more consistent
> with the ,(@implv new-array) case just above this code snippet :)
>
> Kanru
> --
> A badly written book is only a blunder. A bad translation of a good
> book is a crime.
>                -- Gilbert Highet
>

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Hello,

The attached patch should add support for ECMAScript unicode literals.

Noah

On Tue, Jan 4, 2011 at 4:23 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
> Hi,
>
> Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> If you mean give guile a '.js' file have it interpret that with
>> ecmascript, then I think it's not possible right now, although I
>> suspect that such a feature could be added easily.
>
> Right, I've cooked a little script to interpret .js files directly.
>
> I ran the sputniktests[1] from google using the attached guile-es-parse
> script, which only tests the parser.
>
>  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log
>
> The result is impressive (full log attached):
>
>  === Summary ===
>   - Ran 5246 tests
>   - Passed 4410 tests (84.1%)
>   - Failed 836 tests (15.9%)
>
> Where the failed tests have
>
>   - 245 unicode errors (unicode literal is not supported)
>   - 393 rbrace errors  (see below)
>   - 39 Math.LN2 errors (see below)
>   - 159 remain to sort out
>
> The rbrace errors are from
>
>   function test() {}
>   // Syntax error: unexpected token :  in form rbrace
>
>   function foo() { this.bar = function() { return 0; } };
>   // Syntax error: unexpected token :  in form rbrace
>
> I also tried to compile the parsed tests, but halted because too many
> errors like
>
>   Object.prototype.toString = function () {return "something";};
>   // No applicable method for #<<generic> pput (6)> in call (pput
>   // #<unbound> toString #<procedure 1e1c438 ()>)
>
> I thought the tests won't run correctly without this.
>
> [1]: https://code.google.com/p/sputniktests/
>
> Cheers,
> Kanru
> --
> A badly written book is only a blunder. A bad translation of a good
> book is a crime.
>                -- Gilbert Highet
>

[-- Attachment #2: 0001-ECMAScript-Unicode-Literals.patch --]
[-- Type: application/octet-stream, Size: 1179 bytes --]

From c847f6bf15160c6f4b4086354d1bcc56ba606a35 Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Tue, 11 Jan 2011 16:32:17 -0500
Subject: [PATCH] ECMAScript Unicode Literals

Add ECMAScript Unicode literal support

  * module/language/ecmascript/tokenize.scm: add unicode literals
---
 module/language/ecmascript/tokenize.scm |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/module/language/ecmascript/tokenize.scm b/module/language/ecmascript/tokenize.scm
index f721445..4054820 100644
--- a/module/language/ecmascript/tokenize.scm
+++ b/module/language/ecmascript/tokenize.scm
@@ -150,7 +150,11 @@
                 (else
                  (syntax-error "bad hex character escape" loc (string a b))))))
             ((#\u)
-             (syntax-error "unicode not supported" loc #f))
+             (let* ((a (read-char port))
+                    (b (read-char port))
+                    (c (read-char port))
+                    (d (read-char port)))
+               (integer->char (string->number (string a b c d) 16))))
             (else
              c))))
       (let lp ((str (read-until terms port loc)))
-- 
1.7.3.2

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2191 bytes --]

This patch fixes a lot of the "unexpected token: rbrace" errors that
had been messing up the Sputnik test results. The issue was that Guile
didn't allow functions with empty bodies.

Noah

On Tue, Jan 11, 2011 at 4:38 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> Hello,
>
> The attached patch should add support for ECMAScript unicode literals.
>
> Noah
>
> On Tue, Jan 4, 2011 at 4:23 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
>> Hi,
>>
>> Noah Lavine <noah.b.lavine@gmail.com> writes:
>>
>>> If you mean give guile a '.js' file have it interpret that with
>>> ecmascript, then I think it's not possible right now, although I
>>> suspect that such a feature could be added easily.
>>
>> Right, I've cooked a little script to interpret .js files directly.
>>
>> I ran the sputniktests[1] from google using the attached guile-es-parse
>> script, which only tests the parser.
>>
>>  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log
>>
>> The result is impressive (full log attached):
>>
>>  === Summary ===
>>   - Ran 5246 tests
>>   - Passed 4410 tests (84.1%)
>>   - Failed 836 tests (15.9%)
>>
>> Where the failed tests have
>>
>>   - 245 unicode errors (unicode literal is not supported)
>>   - 393 rbrace errors  (see below)
>>   - 39 Math.LN2 errors (see below)
>>   - 159 remain to sort out
>>
>> The rbrace errors are from
>>
>>   function test() {}
>>   // Syntax error: unexpected token :  in form rbrace
>>
>>   function foo() { this.bar = function() { return 0; } };
>>   // Syntax error: unexpected token :  in form rbrace
>>
>> I also tried to compile the parsed tests, but halted because too many
>> errors like
>>
>>   Object.prototype.toString = function () {return "something";};
>>   // No applicable method for #<<generic> pput (6)> in call (pput
>>   // #<unbound> toString #<procedure 1e1c438 ()>)
>>
>> I thought the tests won't run correctly without this.
>>
>> [1]: https://code.google.com/p/sputniktests/
>>
>> Cheers,
>> Kanru
>> --
>> A badly written book is only a blunder. A bad translation of a good
>> book is a crime.
>>                -- Gilbert Highet
>>
>

[-- Attachment #2: 0001-Ecmascript-Syntax-Fix.patch --]
[-- Type: application/octet-stream, Size: 1041 bytes --]

From dc44cb2e40273036578a2cb2af8d20a7b098e038 Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Tue, 11 Jan 2011 17:29:09 -0500
Subject: [PATCH] Ecmascript Syntax Fix

  * module/language/ecmascript/parse.scm: allow empty function bodies.
---
 module/language/ecmascript/parse.scm |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/module/language/ecmascript/parse.scm b/module/language/ecmascript/parse.scm
index b8868a3..3a3f417 100644
--- a/module/language/ecmascript/parse.scm
+++ b/module/language/ecmascript/parse.scm
@@ -85,7 +85,8 @@
                    (SourceElements SourceElement) : (if (and (pair? $1) (eq? (car $1) 'begin))
                                                          `(begin ,@(cdr $1) ,$2)
                                                          `(begin ,$1 ,$2)))
-   (FunctionBody (SourceElements) : $1)
+   (FunctionBody (SourceElements) : $1
+                 () : '(begin))
 
    (Statement (Block) : $1
               (VariableStatement) : $1
-- 
1.7.3.2

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2487 bytes --]

And this fixes an error that came because Unicode 00A0 (no-break
space) is supposed to be considered whitespace in ECMAScript.

Noah

On Tue, Jan 11, 2011 at 5:33 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> This patch fixes a lot of the "unexpected token: rbrace" errors that
> had been messing up the Sputnik test results. The issue was that Guile
> didn't allow functions with empty bodies.
>
> Noah
>
> On Tue, Jan 11, 2011 at 4:38 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>> Hello,
>>
>> The attached patch should add support for ECMAScript unicode literals.
>>
>> Noah
>>
>> On Tue, Jan 4, 2011 at 4:23 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
>>> Hi,
>>>
>>> Noah Lavine <noah.b.lavine@gmail.com> writes:
>>>
>>>> If you mean give guile a '.js' file have it interpret that with
>>>> ecmascript, then I think it's not possible right now, although I
>>>> suspect that such a feature could be added easily.
>>>
>>> Right, I've cooked a little script to interpret .js files directly.
>>>
>>> I ran the sputniktests[1] from google using the attached guile-es-parse
>>> script, which only tests the parser.
>>>
>>>  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log
>>>
>>> The result is impressive (full log attached):
>>>
>>>  === Summary ===
>>>   - Ran 5246 tests
>>>   - Passed 4410 tests (84.1%)
>>>   - Failed 836 tests (15.9%)
>>>
>>> Where the failed tests have
>>>
>>>   - 245 unicode errors (unicode literal is not supported)
>>>   - 393 rbrace errors  (see below)
>>>   - 39 Math.LN2 errors (see below)
>>>   - 159 remain to sort out
>>>
>>> The rbrace errors are from
>>>
>>>   function test() {}
>>>   // Syntax error: unexpected token :  in form rbrace
>>>
>>>   function foo() { this.bar = function() { return 0; } };
>>>   // Syntax error: unexpected token :  in form rbrace
>>>
>>> I also tried to compile the parsed tests, but halted because too many
>>> errors like
>>>
>>>   Object.prototype.toString = function () {return "something";};
>>>   // No applicable method for #<<generic> pput (6)> in call (pput
>>>   // #<unbound> toString #<procedure 1e1c438 ()>)
>>>
>>> I thought the tests won't run correctly without this.
>>>
>>> [1]: https://code.google.com/p/sputniktests/
>>>
>>> Cheers,
>>> Kanru
>>> --
>>> A badly written book is only a blunder. A bad translation of a good
>>> book is a crime.
>>>                -- Gilbert Highet
>>>
>>
>

[-- Attachment #2: 0001-Ecmascript-Syntax.patch --]
[-- Type: application/octet-stream, Size: 945 bytes --]

From ebc1f05c2e29c44eb1d74a5feb56b87e7a625f74 Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Tue, 11 Jan 2011 17:41:21 -0500
Subject: [PATCH] Ecmascript Syntax

  * module/language/ecmascript/tokenize.scm: an unbreakable space
      counts as whitespace.
---
 module/language/ecmascript/tokenize.scm |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/module/language/ecmascript/tokenize.scm b/module/language/ecmascript/tokenize.scm
index 4054820..594518a 100644
--- a/module/language/ecmascript/tokenize.scm
+++ b/module/language/ecmascript/tokenize.scm
@@ -412,7 +412,7 @@
   (let ((c   (peek-char port))
         (loc (port-source-location port)))
     (case c
-      ((#\ht #\vt #\np #\space)         ; whitespace
+      ((#\ht #\vt #\np #\space #\x00A0) ; whitespace
        (read-char port)
        (next-token port div?))
       ((#\newline #\cr)                 ; line break
-- 
1.7.3.2

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2719 bytes --]

This parses decimal numbers with leading zeros correctly.

Noah

On Tue, Jan 11, 2011 at 5:45 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> And this fixes an error that came because Unicode 00A0 (no-break
> space) is supposed to be considered whitespace in ECMAScript.
>
> Noah
>
> On Tue, Jan 11, 2011 at 5:33 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>> This patch fixes a lot of the "unexpected token: rbrace" errors that
>> had been messing up the Sputnik test results. The issue was that Guile
>> didn't allow functions with empty bodies.
>>
>> Noah
>>
>> On Tue, Jan 11, 2011 at 4:38 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>>> Hello,
>>>
>>> The attached patch should add support for ECMAScript unicode literals.
>>>
>>> Noah
>>>
>>> On Tue, Jan 4, 2011 at 4:23 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
>>>> Hi,
>>>>
>>>> Noah Lavine <noah.b.lavine@gmail.com> writes:
>>>>
>>>>> If you mean give guile a '.js' file have it interpret that with
>>>>> ecmascript, then I think it's not possible right now, although I
>>>>> suspect that such a feature could be added easily.
>>>>
>>>> Right, I've cooked a little script to interpret .js files directly.
>>>>
>>>> I ran the sputniktests[1] from google using the attached guile-es-parse
>>>> script, which only tests the parser.
>>>>
>>>>  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log
>>>>
>>>> The result is impressive (full log attached):
>>>>
>>>>  === Summary ===
>>>>   - Ran 5246 tests
>>>>   - Passed 4410 tests (84.1%)
>>>>   - Failed 836 tests (15.9%)
>>>>
>>>> Where the failed tests have
>>>>
>>>>   - 245 unicode errors (unicode literal is not supported)
>>>>   - 393 rbrace errors  (see below)
>>>>   - 39 Math.LN2 errors (see below)
>>>>   - 159 remain to sort out
>>>>
>>>> The rbrace errors are from
>>>>
>>>>   function test() {}
>>>>   // Syntax error: unexpected token :  in form rbrace
>>>>
>>>>   function foo() { this.bar = function() { return 0; } };
>>>>   // Syntax error: unexpected token :  in form rbrace
>>>>
>>>> I also tried to compile the parsed tests, but halted because too many
>>>> errors like
>>>>
>>>>   Object.prototype.toString = function () {return "something";};
>>>>   // No applicable method for #<<generic> pput (6)> in call (pput
>>>>   // #<unbound> toString #<procedure 1e1c438 ()>)
>>>>
>>>> I thought the tests won't run correctly without this.
>>>>
>>>> [1]: https://code.google.com/p/sputniktests/
>>>>
>>>> Cheers,
>>>> Kanru
>>>> --
>>>> A badly written book is only a blunder. A bad translation of a good
>>>> book is a crime.
>>>>                -- Gilbert Highet
>>>>
>>>
>>
>

[-- Attachment #2: 0001-Parse-Decimal-Numbers.patch --]
[-- Type: application/octet-stream, Size: 1047 bytes --]

From 3de859a333795c5122fc51fd1b0d2e71ebfc981d Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Tue, 11 Jan 2011 17:58:34 -0500
Subject: [PATCH] Parse Decimal Numbers

 * module/language/ecmascript/parse.scm: handle numbers with leading
    decimals correctly.
---
 module/language/ecmascript/parse.scm |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/module/language/ecmascript/parse.scm b/module/language/ecmascript/parse.scm
index 3a3f417..be41e4b 100644
--- a/module/language/ecmascript/parse.scm
+++ b/module/language/ecmascript/parse.scm
@@ -197,6 +197,7 @@
                       (StringLiteral) : `(string ,$1)
                       (RegexpLiteral) : `(regexp ,$1)
                       (NumericLiteral) : `(number ,$1)
+                      (dot NumericLiteral) : `(number ,(string->number (string-append "." (number->string $2))))
                       (ArrayLiteral) : $1
                       (ObjectLiteral) : $1
                       (lparen Expression rparen) : $2)
-- 
1.7.3.2

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2994 bytes --]

And this fixes hexadecimal constants. Guile now parses all Sputnik
tests for numeric literals.

Noah

On Tue, Jan 11, 2011 at 5:59 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> This parses decimal numbers with leading zeros correctly.
>
> Noah
>
> On Tue, Jan 11, 2011 at 5:45 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>> And this fixes an error that came because Unicode 00A0 (no-break
>> space) is supposed to be considered whitespace in ECMAScript.
>>
>> Noah
>>
>> On Tue, Jan 11, 2011 at 5:33 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>>> This patch fixes a lot of the "unexpected token: rbrace" errors that
>>> had been messing up the Sputnik test results. The issue was that Guile
>>> didn't allow functions with empty bodies.
>>>
>>> Noah
>>>
>>> On Tue, Jan 11, 2011 at 4:38 PM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
>>>> Hello,
>>>>
>>>> The attached patch should add support for ECMAScript unicode literals.
>>>>
>>>> Noah
>>>>
>>>> On Tue, Jan 4, 2011 at 4:23 AM, Kan-Ru Chen <kanru@kanru.info> wrote:
>>>>> Hi,
>>>>>
>>>>> Noah Lavine <noah.b.lavine@gmail.com> writes:
>>>>>
>>>>>> If you mean give guile a '.js' file have it interpret that with
>>>>>> ecmascript, then I think it's not possible right now, although I
>>>>>> suspect that such a feature could be added easily.
>>>>>
>>>>> Right, I've cooked a little script to interpret .js files directly.
>>>>>
>>>>> I ran the sputniktests[1] from google using the attached guile-es-parse
>>>>> script, which only tests the parser.
>>>>>
>>>>>  python tools/sputnik.py --full-summary --command ./guile-es-parse|tee log
>>>>>
>>>>> The result is impressive (full log attached):
>>>>>
>>>>>  === Summary ===
>>>>>   - Ran 5246 tests
>>>>>   - Passed 4410 tests (84.1%)
>>>>>   - Failed 836 tests (15.9%)
>>>>>
>>>>> Where the failed tests have
>>>>>
>>>>>   - 245 unicode errors (unicode literal is not supported)
>>>>>   - 393 rbrace errors  (see below)
>>>>>   - 39 Math.LN2 errors (see below)
>>>>>   - 159 remain to sort out
>>>>>
>>>>> The rbrace errors are from
>>>>>
>>>>>   function test() {}
>>>>>   // Syntax error: unexpected token :  in form rbrace
>>>>>
>>>>>   function foo() { this.bar = function() { return 0; } };
>>>>>   // Syntax error: unexpected token :  in form rbrace
>>>>>
>>>>> I also tried to compile the parsed tests, but halted because too many
>>>>> errors like
>>>>>
>>>>>   Object.prototype.toString = function () {return "something";};
>>>>>   // No applicable method for #<<generic> pput (6)> in call (pput
>>>>>   // #<unbound> toString #<procedure 1e1c438 ()>)
>>>>>
>>>>> I thought the tests won't run correctly without this.
>>>>>
>>>>> [1]: https://code.google.com/p/sputniktests/
>>>>>
>>>>> Cheers,
>>>>> Kanru
>>>>> --
>>>>> A badly written book is only a blunder. A bad translation of a good
>>>>> book is a crime.
>>>>>                -- Gilbert Highet
>>>>>
>>>>
>>>
>>
>

[-- Attachment #2: 0001-Fix-Hex-Constants.patch --]
[-- Type: application/octet-stream, Size: 940 bytes --]

From b14eadd441a957c25c505c78c7400128ad83a342 Mon Sep 17 00:00:00 2001
From: Noah Lavine <nlavine@haverford.edu>
Date: Tue, 11 Jan 2011 18:03:04 -0500
Subject: [PATCH] Fix Hex Constants

 * module/language/ecmascript/tokenize.scm: hexadecimal constants can
    now use 'X' in addition to 'x'.
---
 module/language/ecmascript/tokenize.scm |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/module/language/ecmascript/tokenize.scm b/module/language/ecmascript/tokenize.scm
index 594518a..4448bf6 100644
--- a/module/language/ecmascript/tokenize.scm
+++ b/module/language/ecmascript/tokenize.scm
@@ -262,7 +262,7 @@
          (c1 (peek-char port)))
     (cond
      ((eof-object? c1) (digit->number c0))
-     ((and (char=? c0 #\0) (char=? c1 #\x))
+     ((and (char=? c0 #\0) (or (char=? c1 #\x) (char=? c1 #\X)))
       (read-char port)
       (let ((c (peek-char port)))
         (if (not (char-hex? c))
-- 
1.7.3.2

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Ludovic Courtès]

Hi,

Thanks for looking into this!

Noah Lavine <noah.b.lavine@gmail.com> writes:

> The attached patch should add support for ECMAScript unicode literals.

I applied this one, along with corresponding test cases.

Can you please resubmit the remaining patches with test case(s) for
each, and each in a thread of its own?

Thanks,
Ludo’.

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Ludovic Courtès]

Hi,

ludo@gnu.org (Ludovic Courtès) writes:

> Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> The attached patch should add support for ECMAScript unicode literals.
>
> I applied this one, along with corresponding test cases.
>
> Can you please resubmit the remaining patches with test case(s) for
> each, and each in a thread of its own?

I finally applied them and wrote the test cases by myself.

Thanks again for the patches, but please do write test cases next time.
:-)

Ludo’.

Re: Sputnik test result (was Re: ECMAScript support broken?)

[Noah Lavine]

Hi,

> I finally applied them and wrote the test cases by myself.

Thanks a lot!

> Thanks again for the patches, but please do write test cases next time.

I will do that. Sorry for the delay - I've been quite busy lately, and
basically all of my Guile mail has been sitting in my inbox waiting to
be handled. (I also plan to, for instance, respond to your email about
the PEG parser.)

Noah

Re: Sputnik test result

[Andy Wingo]

Hi Kan-Ru,

On Tue 04 Jan 2011 10:23, Kan-Ru Chen <kanru@kanru.info> writes:

> I ran the sputniktests[1] from google using the attached guile-es-parse
> script, which only tests the parser.
>
>   === Summary ===
>    - Ran 5246 tests
>    - Passed 4410 tests (84.1%)
>    - Failed 836 tests (15.9%)

Excellent test case!  I just ran the tests again, with the recent
patches, and we have:

=== Summary ===
 - Ran 5246 tests
 - Passed 4655 tests (88.7%)
 - Failed 591 tests (11.3%)

So an improvement, but still quite a ways to go.  I think the ones that
were fixed are the unicode errors that you mentioned:

> Where the failed tests have
>
>    - 245 unicode errors (unicode literal is not supported)
>    - 393 rbrace errors  (see below)
>    - 39 Math.LN2 errors (see below)
>    - 159 remain to sort out

So we are left with the rest of them to sort out.  Then of course to
actually compile them, eh...

Andy
-- 
http://wingolog.org/

Re: Sputnik test result

[Noah Lavine]

Hi all,

When I was looking at this a while ago, it looked like a big issue
(possibly *the* big issue) with Guile's parser is that it doesn't do
automatic semicolon insertion. (In Javascript, semicolons are optional
in certain contexts, when the place where the semicolon would be is
followed by a newline.)

Unfortunately, dealing with semicolon insertion is sort of a pain. I
was really hoping to find a way to get the tokenizer to deal with it,
and possibly there is one. But if there's not, we would have to make
newlines into tokens, and then put them into the grammar at all places
where newlines are allowed, and then use that to make semicolons
optional in the right places.

Noah

On Thu, Jan 27, 2011 at 11:08 AM, Andy Wingo <wingo@pobox.com> wrote:
> Hi Kan-Ru,
>
> On Tue 04 Jan 2011 10:23, Kan-Ru Chen <kanru@kanru.info> writes:
>
>> I ran the sputniktests[1] from google using the attached guile-es-parse
>> script, which only tests the parser.
>>
>>   === Summary ===
>>    - Ran 5246 tests
>>    - Passed 4410 tests (84.1%)
>>    - Failed 836 tests (15.9%)
>
> Excellent test case!  I just ran the tests again, with the recent
> patches, and we have:
>
> === Summary ===
>  - Ran 5246 tests
>  - Passed 4655 tests (88.7%)
>  - Failed 591 tests (11.3%)
>
> So an improvement, but still quite a ways to go.  I think the ones that
> were fixed are the unicode errors that you mentioned:
>
>> Where the failed tests have
>>
>>    - 245 unicode errors (unicode literal is not supported)
>>    - 393 rbrace errors  (see below)
>>    - 39 Math.LN2 errors (see below)
>>    - 159 remain to sort out
>
> So we are left with the rest of them to sort out.  Then of course to
> actually compile them, eh...
>
> Andy
> --
> http://wingolog.org/
>
>

Re: Sputnik test result

[Kan-Ru Chen]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hi all,
>
> When I was looking at this a while ago, it looked like a big issue
> (possibly *the* big issue) with Guile's parser is that it doesn't do
> automatic semicolon insertion. (In Javascript, semicolons are optional
> in certain contexts, when the place where the semicolon would be is
> followed by a newline.)

And the ambiguities around the / operator.  Dealing with the / operator
the tokenizer has to decide whether it's a divide operator or a RegExp
literal, which needs the context from the parser.  This explains the 39
Math.NL errors that I forgot in previous mail.

Kanru