* syncase code issue 1.8.8 -> 2.0.11
@ 2014-09-18 1:18 Matt Wette
2014-09-18 7:49 ` Panicz Maciej Godek
2014-09-18 10:20 ` Taylan Ulrich Bayirli/Kammer
0 siblings, 2 replies; 6+ messages in thread
From: Matt Wette @ 2014-09-18 1:18 UTC (permalink / raw
To: guile-user
[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]
Hi Folks,
Anyone interested in looking at my syntax-case code? I wrote this several years ago under 1.8.8. Now moving to 2.0.11: not working :(.
Matt
This uses syntax-case. It works in 1.8.8, but I get errors in 2.0.11. I'm guessing I don't understand something about syntax-case in R6RS. I learned (or mislearned) syntax-case from the papers by Kent Dybvig, not from the R6RS spec.
Here is what I'm trying to do:
@item define-tokenizer name (rex1 tok1) (rex2 tok2) ...
Define tokenizer for which one can build string tokenizers.
Example:
(define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))
(define mt (make-tokiz "abc=def"))
(tokiz-latok mt) -> #\a
(tokiz-laval mt) -> "abc"
(tokiz-match mt #\a)
(tokiz-latok mt) -> #\=
In 1.8.8, I get the above output. In 2.0.11 I get
Syntax-case macros are now a part of Guile core; importing (ice-9 syncase) is no longer necessary. <=(no issue here)
ice-9/psyntax.scm:1274:12: In procedure dobody:
ice-9/psyntax.scm:1274:12: Syntax error:
u1.scm:106:18: definition in expression context, where definitions are not allowed, in form
(define make-tokiz
(lambda (string)
(let ((pobj (vector string 0 (string-length string) #\nul "")))
(tokiz-match pobj #\nul) pobj)))
Here is synopsis of the code (see attached for all of it):
(define-syntax define-tokenizer
;; pobj = parse object = #(string ix nd latok laval)
(lambda (x)
(syntax-case x ()
((_ name (c1 r1) ...)
(with-syntax ((maker (sc-gen-id (syntax name) "make-" (syntax name)))
...
)
(syntax
(begin
(define maker
(lambda (string)
(let ((pobj (vector string 0 (string-length string) #\nul "")))
(match pobj #\nul) ; prime latok/laval in parse-object
pobj)))
.....
)))))
(define sc-gen-id ;; (gc-gen-id (syntax name) "make-" (syntax name)) where name=foo -> make-foo
(lambda (template-id . args)
(datum->syntax-object template-id
(string->symbol
(apply string-append
(map (lambda (x)
(if (string? x) x
(symbol->string (syntax-object->datum x))))
args))))))
[-- Attachment #2: u1.scm --]
[-- Type: application/octet-stream, Size: 3839 bytes --]
(use-modules (ice-9 regex))
(use-modules (ice-9 format))
(use-modules (ice-9 syncase))
;; @item sc-gen-id template-id . args
;; For use in syntax-case/with-syntax, generates an identifier given
;; template-identifier and pattern from remaining args of symbol or string.
;; Example:
;; (sc-gen-id (syntax name) (syntax name) "-setter") -> foo-setter
;; where (syntax name) is 'foo
(define sc-gen-id
(lambda (template-id . args)
(datum->syntax-object template-id
(string->symbol
(apply string-append
(map (lambda (x)
(if (string? x) x
(symbol->string (syntax-object->datum x))))
args))))))
;; @item define-tokenizer name (rex1 tok1) (rex2 tok2) ...
;; Define tokenizer for which one can build string tokenizers.
;; (define t (make-name string)) generates the tokenizer for the string.
;; (name-latok t) returns the lookahead token, a character.
;; (name-laval t) returns the lookahead value, a string.
;; The end-of-string results in latok of #\nul, laval of "".
;; Example:
;; (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))
;; (define mt (make-tokiz "abc=def"))
;; (tokiz-latok mt) -> #\a
;; (tokiz-laval mt) -> "abc"
;; (tokiz-match mt #\a)
;; (tokiz-latok mt) -> #\=
;; (tokiz-match mt #\=)
(define-syntax define-tokenizer
;; pobj = parse object = #(string ix nd latok laval)
(lambda (x)
(syntax-case x ()
((_ name (c1 r1) ...)
(with-syntax ((maker (sc-gen-id (syntax name) "make-" (syntax name)))
(latok (sc-gen-id (syntax name) (syntax name) "-latok"))
(laval (sc-gen-id (syntax name) (syntax name) "-laval"))
(match (sc-gen-id (syntax name) (syntax name) "-match"))
((def ...)
(let f ((ix 1)
(cz (syntax (c1 ...)))
(rz (syntax (r1 ...))))
(if (null? cz) '()
(cons (list (sc-gen-id
(car cz) "pat-" (number->string ix))
(list
make-regexp
(list string-append "^" (car cz))))
(f (+ 1 ix) (cdr cz) (cdr rz))))))
((v1 ...)
(let f ((ix 1)
(cz (syntax (c1 ...)))
(rz (syntax (r1 ...))))
(if (null? cz) '()
(cons (sc-gen-id
(car cz) "pat-" (number->string ix))
(f (+ 1 ix) (cdr cz) (cdr rz)))))))
(syntax
(begin
(define maker
(lambda (string)
(let ((pobj (vector string 0 (string-length string) #\nul "")))
(match pobj #\nul) ; prime latok/laval in parse-object
pobj)))
(define latok (lambda (pobj) (vector-ref pobj 3)))
(define laval (lambda (pobj) (vector-ref pobj 4)))
(define match
(let (def ...)
(lambda (pobj tokn)
(if (not (eq? tokn (latok pobj))) (error "syntax error"))
;; Shift.
(vector-set! pobj 1 (+ (vector-ref pobj 1)
(string-length (vector-ref pobj 4))))
;; Find new latok/laval.
(let* ((ps (substring (vector-ref pobj 0)
(vector-ref pobj 1)
(vector-ref pobj 2)))
(pc (if (eq? (vector-ref pobj 1)
(vector-ref pobj 2)) #\nul
(string-ref (vector-ref pobj 0)
(vector-ref pobj 1))))
(ms (lambda (rx) (regexp-exec rx ps)))
(set-la (lambda (tok val) ; set and return
(vector-set! pobj 3 tok)
(vector-set! pobj 4 val)
tok)))
;; next lookahead
(let iter ((vs (list v1 ...)) (rs (list r1 ...)))
(cond
((eq? (vector-ref pobj 1) (vector-ref pobj 2)) ; end
(set-la #\nul ""))
((null? vs) ; character
(set-la pc (string pc)))
((ms (car vs)) => ; pattern
(lambda (m)
(set-la (car rs) (match:substring m))))
(else
(iter (cdr vs) (cdr rs))))))))))))))))
(format #t "~a\n" (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a)))
(format #t "~a\n" (define mt (make-tokiz "abc=def")))
(format #t "~a\n" (tokiz-latok mt))
(format #t "~a\n" (tokiz-laval mt))
(format #t "~a\n" (tokiz-match mt #\a))
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: syncase code issue 1.8.8 -> 2.0.11
2014-09-18 1:18 syncase code issue 1.8.8 -> 2.0.11 Matt Wette
@ 2014-09-18 7:49 ` Panicz Maciej Godek
2014-09-18 10:20 ` Taylan Ulrich Bayirli/Kammer
1 sibling, 0 replies; 6+ messages in thread
From: Panicz Maciej Godek @ 2014-09-18 7:49 UTC (permalink / raw
To: Matt Wette; +Cc: guile-user@gnu.org
[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]
2014-09-18 3:18 GMT+02:00 Matt Wette <mwette@alumni.caltech.edu>:
> Hi Folks,
>
> Anyone interested in looking at my syntax-case code? I wrote this
> several years ago under 1.8.8. Now moving to 2.0.11: not working :(.
>
> Matt
>
>
> [...]
> In 1.8.8, I get the above output. In 2.0.11 I get
>
> Syntax-case macros are now a part of Guile core; importing (ice-9 syncase)
> is no longer necessary. <=(no issue here)
> ice-9/psyntax.scm:1274:12: In procedure dobody:
> ice-9/psyntax.scm:1274:12: Syntax error:
> u1.scm:106:18: definition in expression context, where definitions are not
> allowed
I haven't time to look at your code more closely, but the error message is
quite informative here: clearly the usage of your macro introduces a
definition in expression context, which doesn't conform to the Scheme
specification. It seems that guile 1.8 treated definitions like
expressions, but it is no longer the case.
Apparently, your code expands to a series of definitions. Therefore the
invocation
(format #t "~a\n" (define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a)))
makes no sense (e.g. because "define-tokenizer" produces no value, that
should be passed to format, and after expansion it contains no expression
that could be evaluated), and neither does
(format #t "~a\n" (define mt (make-tokiz "abc=def")))
I don't know what was your intention in the first case, but you could
rearrange the second line in the following way:
(define mt (make-tokiz "abc=def"))
(format #t "~a\n" mt)
The first line could simply be replaced with the sole definition, i.e.
(define-tokenizer tokiz ("[0-9]+" #\1) ("[a-z]+" #\a))
(I am guessing that the invocation from within "format" stems from
debugging)
regards
[-- Attachment #2: Type: text/html, Size: 2707 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: syncase code issue 1.8.8 -> 2.0.11
2014-09-18 1:18 syncase code issue 1.8.8 -> 2.0.11 Matt Wette
2014-09-18 7:49 ` Panicz Maciej Godek
@ 2014-09-18 10:20 ` Taylan Ulrich Bayirli/Kammer
2014-09-18 12:35 ` Matt Wette
1 sibling, 1 reply; 6+ messages in thread
From: Taylan Ulrich Bayirli/Kammer @ 2014-09-18 10:20 UTC (permalink / raw
To: Matt Wette; +Cc: guile-user
Matt Wette <mwette@alumni.caltech.edu> writes:
> Hi Folks,
>
> Anyone interested in looking at my syntax-case code? I wrote this
> several years ago under 1.8.8. Now moving to 2.0.11: not working :(.
>
> Matt
Hi Matt,
I see your code is doing things like
(format #t "~a\n" (define mt (make-tokiz "abc=def")))
The argument to `format' there is necessarily an "expression" in the
grammar of Scheme. Definitions like (define ...) are not a valid type
of expression in Scheme.
The only places you can use definitions are
- the top-level of a program/library
- the *beginning* of a "code body" like the body of a `lambda', the body
of a `let', etc. can have a sequence of definitions; the first
non-definition expression terminates that sequence
- when you have a (begin ...) form in a position where a definition
would otherwise be allowed, then the body of this begin may also start
with a series of definitions; again, the first non-definition
expression terminates this sequence
The following are examples of well-formed code:
(define (foo)
(define (helper1)
...)
(define (helper2)
...)
(do-something-with (helper1))
(do-something-with (helper2)))
(let (...)
(define (helper1)
...)
...
(do-something-with (helper1)))
(let (...)
(begin
(define (helper1)
...)
(define (helper2)
...)
(do-something-with (helper1)))
(do-something-with (helper2)))
The following are not well-formed:
(define (foo)
(do-something)
(define (helper1)
...)
(do-something-with (helper1)))
;; Because the helper1 definition is in the middle of the foo lambda
;; body, with a preceding non-definition expression.
(let (...)
(do-something)
(begin
(define (helper1)
...)
(do-something-with (helper1)))
(do-something)
;; Because the whole `begin' is in the middle of a let body, so the
;; begin may have no definitions at all. The begin is "in an expression
;; context".
Hope that helps.
Taylan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: syncase code issue 1.8.8 -> 2.0.11
2014-09-18 10:20 ` Taylan Ulrich Bayirli/Kammer
@ 2014-09-18 12:35 ` Matt Wette
0 siblings, 0 replies; 6+ messages in thread
From: Matt Wette @ 2014-09-18 12:35 UTC (permalink / raw
To: Taylan Ulrich Bayirli/Kammer; +Cc: guile-user
On Sep 18, 2014, at 3:20 AM, Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> wrote:
>
> I see your code is doing things like
>
> (format #t "~a\n" (define mt (make-tokiz "abc=def")))
>
> The argument to `format' there is necessarily an "expression" in the
> grammar of Scheme. Definitions like (define ...) are not a valid type
> of expression in Scheme.
yes, just debug that didn't matter - please disregard
> The only places you can use definitions are
>
> - the top-level of a program/library
>
> - the *beginning* of a "code body" like the body of a `lambda', the body
> of a `let', etc. can have a sequence of definitions; the first
> non-definition expression terminates that sequence
>
> - when you have a (begin ...) form in a position where a definition
> would otherwise be allowed, then the body of this begin may also start
> with a series of definitions; again, the first non-definition
> expression terminates this sequence
This is syntax-case. Go check share/guile/2.0/boot-9/psyntax.scm for similar code.
Matt
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: syncase code issue 1.8.8 -> 2.0.11
@ 2014-09-18 14:08 mwette
2014-09-19 3:54 ` Matt Wette
0 siblings, 1 reply; 6+ messages in thread
From: mwette @ 2014-09-18 14:08 UTC (permalink / raw
To: guile-user
Yikes. I get it now.
When isolating one bug I introduced another.
Removing the define from the argument to format fixed that.
Now I can get back to my debugging ...
Thanks,
Matt
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: syncase code issue 1.8.8 -> 2.0.11
2014-09-18 14:08 mwette
@ 2014-09-19 3:54 ` Matt Wette
0 siblings, 0 replies; 6+ messages in thread
From: Matt Wette @ 2014-09-19 3:54 UTC (permalink / raw
To: guile-user
[-- Attachment #1: Type: text/plain, Size: 561 bytes --]
I found the root bug, I think, flagged by this message:
scheme@(guile-user)> (define-tokenizer tokiz ("[a-z]+" #\a))
While compiling expression:
ERROR: build-constant-store: unrecognized object #<procedure make-regexp (_ . _)>
I was putting procedure-objects in the macro when I should have been putting in syntax.
works:
(list
(datum->syntax (car cz) 'make-regexp)
(string-append
"^" (syntax->datum (car cz)))))
broken (but worked in guile 1.8.8):
(list
make-regexp
(list string-append "^" (car cz))))
Matt
[-- Attachment #2: Type: text/html, Size: 1296 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-09-19 3:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-09-18 1:18 syncase code issue 1.8.8 -> 2.0.11 Matt Wette
2014-09-18 7:49 ` Panicz Maciej Godek
2014-09-18 10:20 ` Taylan Ulrich Bayirli/Kammer
2014-09-18 12:35 ` Matt Wette
-- strict thread matches above, loose matches on Subject: below --
2014-09-18 14:08 mwette
2014-09-19 3:54 ` Matt Wette
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).