syncase code issue 1.8.8 -> 2.0.11

syncase code issue 1.8.8 -> 2.0.11

[Matt Wette]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

Hi Folks,

Anyone interested in looking at my syntax-case code?   I wrote this several years ago under 1.8.8.  Now moving to 2.0.11: not working :(.

Matt

This uses syntax-case.  It works in 1.8.8, but I get errors in 2.0.11.   I'm guessing I don't understand something about syntax-case in R6RS.  I learned (or mislearned) syntax-case from the papers by Kent Dybvig, not from the R6RS spec.

Here is what I'm trying to do:
@item define-tokenizer name (rex1 tok1) (rex2 tok2) ...
Define tokenizer for which one can build string tokenizers.

Example:
(define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))
(define mt (make-tokiz "abc=def"))
(tokiz-latok mt) -> #\a
(tokiz-laval mt) -> "abc"
(tokiz-match mt #\a)
(tokiz-latok mt) -> #\=

In 1.8.8, I get the above output.   In 2.0.11 I get

Syntax-case macros are now a part of Guile core; importing (ice-9 syncase) is no longer necessary.   <=(no issue here)
ice-9/psyntax.scm:1274:12: In procedure dobody:
ice-9/psyntax.scm:1274:12: Syntax error:
u1.scm:106:18: definition in expression context, where definitions are not allowed, in form
    (define make-tokiz
      (lambda (string)
        (let ((pobj (vector string 0 (string-length string) #\nul "")))
          (tokiz-match pobj #\nul) pobj)))

Here is synopsis of the code (see attached for all of it):

(define-syntax define-tokenizer
  ;; pobj = parse object = #(string ix nd latok laval)
  (lambda (x)
    (syntax-case x ()
      ((_ name (c1 r1) ...)
       (with-syntax ((maker (sc-gen-id (syntax name) "make-" (syntax name)))
                     ...
                     )
	 (syntax
	  (begin
	    (define maker
	      (lambda (string)
		(let ((pobj (vector string 0 (string-length string) #\nul "")))
		  (match pobj #\nul)  ; prime latok/laval in parse-object
		  pobj)))
             .....
           )))))



(define sc-gen-id  ;; (gc-gen-id (syntax name) "make-" (syntax name)) where name=foo -> make-foo 
  (lambda (template-id . args)
    (datum->syntax-object template-id
      (string->symbol
       (apply string-append
	      (map (lambda (x) 
		     (if (string? x) x
			 (symbol->string (syntax-object->datum x))))
		   args))))))



[-- Attachment #2: u1.scm --]
[-- Type: application/octet-stream, Size: 3839 bytes --]

(use-modules (ice-9 regex))
(use-modules (ice-9 format))
(use-modules (ice-9 syncase))

;; @item sc-gen-id template-id . args
;; For use in syntax-case/with-syntax, generates an identifier given
;; template-identifier and pattern from remaining args of symbol or string.
;; Example:
;;   (sc-gen-id (syntax name) (syntax name) "-setter") -> foo-setter
;; where (syntax name) is 'foo
(define sc-gen-id
  (lambda (template-id . args)
    (datum->syntax-object template-id
      (string->symbol
       (apply string-append
	      (map (lambda (x) 
		     (if (string? x) x
			 (symbol->string (syntax-object->datum x))))
		   args))))))


;; @item define-tokenizer name (rex1 tok1) (rex2 tok2) ...
;; Define tokenizer for which one can build string tokenizers.
;; (define t (make-name string)) generates the tokenizer for the string.
;; (name-latok t) returns the lookahead token, a character.
;; (name-laval t) returns the lookahead value, a string.
;; The end-of-string results in latok of #\nul, laval of "".
;; Example:
;; (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))
;; (define mt (make-tokiz "abc=def"))
;; (tokiz-latok mt) -> #\a
;; (tokiz-laval mt) -> "abc"
;; (tokiz-match mt #\a)
;; (tokiz-latok mt) -> #\=
;; (tokiz-match mt #\=)
(define-syntax define-tokenizer
  ;; pobj = parse object = #(string ix nd latok laval)
  (lambda (x)
    (syntax-case x ()
      ((_ name (c1 r1) ...)
       (with-syntax ((maker (sc-gen-id (syntax name) "make-" (syntax name)))
		     (latok (sc-gen-id (syntax name) (syntax name) "-latok"))
		     (laval (sc-gen-id (syntax name) (syntax name) "-laval"))
		     (match (sc-gen-id (syntax name) (syntax name) "-match"))
		     ((def ...)
		      (let f ((ix 1)
			      (cz (syntax (c1 ...)))
			      (rz (syntax (r1 ...))))
			(if (null? cz) '()
			    (cons (list (sc-gen-id
					 (car cz) "pat-" (number->string ix))
					(list
					 make-regexp 
					 (list string-append "^" (car cz))))
				  (f (+ 1 ix) (cdr cz) (cdr rz))))))
		     ((v1 ...)
		      (let f ((ix 1)
			      (cz (syntax (c1 ...)))
			      (rz (syntax (r1 ...))))
			(if (null? cz) '()
			    (cons (sc-gen-id
				   (car cz) "pat-" (number->string ix))
				  (f (+ 1 ix) (cdr cz) (cdr rz)))))))
	 (syntax
	  (begin
	    (define maker
	      (lambda (string)
		(let ((pobj (vector string 0 (string-length string) #\nul "")))
		  (match pobj #\nul)  ; prime latok/laval in parse-object
		  pobj)))
	    (define latok (lambda (pobj) (vector-ref pobj 3)))
	    (define laval (lambda (pobj) (vector-ref pobj 4)))
	    (define match
	      (let (def ...)
		(lambda (pobj tokn)
		  (if (not (eq? tokn (latok pobj))) (error "syntax error"))
		  ;; Shift.
		  (vector-set! pobj 1 (+ (vector-ref pobj 1)
					 (string-length (vector-ref pobj 4))))
		  ;; Find new latok/laval.
		  (let* ((ps (substring	(vector-ref pobj 0)
					(vector-ref pobj 1)
					(vector-ref pobj 2)))
			 (pc (if (eq? (vector-ref pobj 1)
				      (vector-ref pobj 2)) #\nul
				 (string-ref (vector-ref pobj 0)
					     (vector-ref pobj 1))))
			 (ms (lambda (rx) (regexp-exec rx ps)))
			 (set-la (lambda (tok val) ; set and return
				   (vector-set! pobj 3 tok)
				   (vector-set! pobj 4 val)
				   tok)))
		    ;; next lookahead
		    (let iter ((vs (list v1 ...))  (rs (list r1 ...)))
		      (cond
		       ((eq? (vector-ref pobj 1) (vector-ref pobj 2)) ; end
			(set-la #\nul ""))
		       ((null? vs)	; character
			(set-la pc (string pc)))
		       ((ms (car vs)) => ; pattern
			(lambda (m)
			  (set-la (car rs) (match:substring m))))
		       (else
			(iter (cdr vs) (cdr rs))))))))))))))))

(format #t "~a\n" (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a)))
(format #t "~a\n" (define mt (make-tokiz "abc=def")))
(format #t "~a\n" (tokiz-latok mt))
(format #t "~a\n" (tokiz-laval mt))
(format #t "~a\n" (tokiz-match mt #\a))

Re: syncase code issue 1.8.8 -> 2.0.11

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

2014-09-18 3:18 GMT+02:00 Matt Wette <mwette@alumni.caltech.edu>:

> Hi Folks,
>
> Anyone interested in looking at my syntax-case code?   I wrote this
> several years ago under 1.8.8.  Now moving to 2.0.11: not working :(.
>
> Matt
>
>

> [...]
> In 1.8.8, I get the above output.   In 2.0.11 I get
>
> Syntax-case macros are now a part of Guile core; importing (ice-9 syncase)
> is no longer necessary.   <=(no issue here)
> ice-9/psyntax.scm:1274:12: In procedure dobody:
> ice-9/psyntax.scm:1274:12: Syntax error:
> u1.scm:106:18: definition in expression context, where definitions are not
> allowed


I haven't time to look at your code more closely, but the error message is
quite informative here: clearly the usage of your macro introduces a
definition in expression context, which doesn't conform to the Scheme
specification. It seems that guile 1.8 treated definitions like
expressions, but it is no longer the case.

Apparently, your code expands to a series of definitions. Therefore the
invocation

(format #t "~a\n" (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a)))

makes no sense (e.g. because "define-tokenizer" produces no value, that
should be passed to format, and after expansion it contains no expression
that could be evaluated), and neither does

(format #t "~a\n" (define mt (make-tokiz "abc=def")))

I don't know what was your intention in the first case, but you could
rearrange the second line in the following way:

(define mt (make-tokiz "abc=def"))
(format #t "~a\n" mt)

The first line could simply be replaced with the sole definition, i.e.
(define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))

(I am guessing that the invocation from within "format" stems from
debugging)

regards

[-- Attachment #2: Type: text/html, Size: 2707 bytes --]

Re: syncase code issue 1.8.8 -> 2.0.11

[Taylan Ulrich Bayirli/Kammer]

Matt Wette <mwette@alumni.caltech.edu> writes:

> Hi Folks,
>
> Anyone interested in looking at my syntax-case code?  I wrote this
> several years ago under 1.8.8.  Now moving to 2.0.11: not working :(.
>
> Matt

Hi Matt,

I see your code is doing things like

(format #t "~a\n" (define mt (make-tokiz "abc=def")))

The argument to `format' there is necessarily an "expression" in the
grammar of Scheme.  Definitions like (define ...) are not a valid type
of expression in Scheme.

The only places you can use definitions are

- the top-level of a program/library

- the *beginning* of a "code body" like the body of a `lambda', the body
  of a `let', etc. can have a sequence of definitions; the first
  non-definition expression terminates that sequence

- when you have a (begin ...) form in a position where a definition
  would otherwise be allowed, then the body of this begin may also start
  with a series of definitions; again, the first non-definition
  expression terminates this sequence

The following are examples of well-formed code:

(define (foo)
  (define (helper1)
    ...)
  (define (helper2)
    ...)
  (do-something-with (helper1))
  (do-something-with (helper2)))

(let (...)
  (define (helper1)
    ...)
  ...
  (do-something-with (helper1)))

(let (...)
  (begin
    (define (helper1)
      ...)
    (define (helper2)
      ...)
    (do-something-with (helper1)))
  (do-something-with (helper2)))

The following are not well-formed:

(define (foo)
  (do-something)
  (define (helper1)
    ...)
  (do-something-with (helper1)))
;; Because the helper1 definition is in the middle of the foo lambda
;; body, with a preceding non-definition expression.

(let (...)
  (do-something)
  (begin
    (define (helper1)
      ...)
    (do-something-with (helper1)))
  (do-something)
;; Because the whole `begin' is in the middle of a let body, so the
;; begin may have no definitions at all.  The begin is "in an expression
;; context".

Hope that helps.

Taylan

Re: syncase code issue 1.8.8 -> 2.0.11

[Matt Wette]

On Sep 18, 2014, at 3:20 AM, Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> wrote:
> 
> I see your code is doing things like
> 
> (format #t "~a\n" (define mt (make-tokiz "abc=def")))
> 
> The argument to `format' there is necessarily an "expression" in the
> grammar of Scheme.  Definitions like (define ...) are not a valid type
> of expression in Scheme.

yes, just debug that didn't matter - please disregard


> The only places you can use definitions are
> 
> - the top-level of a program/library
> 
> - the *beginning* of a "code body" like the body of a `lambda', the body
>  of a `let', etc. can have a sequence of definitions; the first
>  non-definition expression terminates that sequence
> 
> - when you have a (begin ...) form in a position where a definition
>  would otherwise be allowed, then the body of this begin may also start
>  with a series of definitions; again, the first non-definition
>  expression terminates this sequence

This is syntax-case.   Go check share/guile/2.0/boot-9/psyntax.scm for similar code.

Matt

Re: syncase code issue 1.8.8 -> 2.0.11

[mwette]

Yikes.  I get it now.

When isolating one bug I introduced another.
Removing the define from the argument to format fixed that.
Now I can get back to my debugging ...

Thanks,
Matt

Re: syncase code issue 1.8.8 -> 2.0.11

[Matt Wette]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]


I found the root bug, I think, flagged by this message:
scheme@(guile-user)> (define-tokenizer tokiz ("[a-z]+" #\a))
While compiling expression:
ERROR: build-constant-store: unrecognized object #<procedure make-regexp (_ . _)>

I was putting procedure-objects in the macro when I should have been putting in syntax.

works:
  (list
   (datum->syntax (car cz) 'make-regexp)
   (string-append
    "^" (syntax->datum (car cz)))))

broken (but worked in guile 1.8.8):
   (list
    make-regexp 
    (list string-append "^" (car cz))))


Matt


[-- Attachment #2: Type: text/html, Size: 1296 bytes --]