Making apostrophe, backtick, etc. hygienic?

Making apostrophe, backtick, etc. hygienic?

[Taylan Ulrich Bayırlı/Kammer]

This is a bit of a crank idea, but here goes.

Today I wasted some time trying to find the bug in the following piece
of code:

    (define (syntax-car syntax)
      (syntax-case syntax () ((car . cdr) #'car)))

Better error reporting in macro-expansion errors might have made it less
painful, but maybe we can solve the problem itself.

How about making 'foo turn into something like (__quote__ foo), and
similar for `foo, #'foo, etc.?  Where __quote__ is just a synonym to
quote, and the original works too.  Ideal would be a symbol that's not
as noisy (in debug output) but still highly improbable to appear in user
code and be accidentally shadowed.

Maybe it would not be standards-compliant in the strict sense, but I
believe it would be an improvement.

Am I missing any obvious downsides?  Or any subtle ones?

I peeked into Guile's sources to see where one might start implementing
this (read.c, expand.c, psyntax.scm?) but better to ask for some input
since it might be a controversial change, and I'm still a total noob in
Guile's code-base.

Taylan

Re: Making apostrophe, backtick, etc. hygienic?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]

2015-08-30 14:30 GMT+02:00 Taylan Ulrich Bayırlı/Kammer <
taylanbayirli@gmail.com>:

> This is a bit of a crank idea, but here goes.
>
> Today I wasted some time trying to find the bug in the following piece
> of code:
>
>     (define (syntax-car syntax)
>       (syntax-case syntax () ((car . cdr) #'car)))
>
> Better error reporting in macro-expansion errors might have made it less
> painful, but maybe we can solve the problem itself.
>
> How about making 'foo turn into something like (__quote__ foo), and
> similar for `foo, #'foo, etc.?  Where __quote__ is just a synonym to
> quote, and the original works too.  Ideal would be a symbol that's not
> as noisy (in debug output) but still highly improbable to appear in user
> code and be accidentally shadowed.
>
>
You mean that #'x is synonymous to (syntax x), and that's where the problem
stems from?


> Maybe it would not be standards-compliant in the strict sense, but I
> believe it would be an improvement.
>
> Am I missing any obvious downsides?  Or any subtle ones?
>
>
I think that every lisper should know that 'x is synonymous to (quote x),
and in some contexts it might be desirable to bind a new meaning to the
"quote" form (and this is already done by some libraries, notably in the
(ice-9 match) module).

As to "syntax", the use of #'x is much rarer, and the idea that #'x is
(syntax x) is indeed a bit controversial. But this regards the whole
syntax-case macro system, and I think that it would be more valuable to
think how to fix its flaws, rather than change the very fundamentals of the
language.


Best regards,
M.

[-- Attachment #2: Type: text/html, Size: 2282 bytes --]

Re: Making apostrophe, backtick, etc. hygienic?

[Taylan Ulrich Bayırlı/Kammer]

Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> You mean that #'x is synonymous to (syntax x), and that's where the
> problem stems from?

Yup.  I shadow 'syntax', but I don't explicitly shadow "#'".  It gets
shadowed implicitly.  Lexical scoping and hygiene are supposed to let
the programmer forget about such worries.

> I think that every lisper should know that 'x is synonymous to (quote
> x), and in some contexts it might be desirable to bind a new meaning
> to the "quote" form (and this is already done by some libraries,
> notably in the (ice-9 match) module).

One may know, but still forget when sufficiently tired, and/or when
using the word "quote" to mean something conceptually different than the
quoting in lisp.

For instance, some kind of text processing program might give the term
"quote" a specific meaning in the program's problem domain, and once
you're immersed deeply enough in this domain, you might find yourself
naming some function parameter "quote" without giving it a second
thought.  Kind of difficult to explain what I mean, but I know it
happens to me when I'm not careful.  As another example, it also keeps
happening to me that I write code like:

    (syntax-rules ()
      ((_ foo)
       (begin
         ...
         (let ((foo bar))
           ...))))

where I forget that the 'foo' there will not be bound freshly by that
let form, and that despite that I understand how 'syntax-rules' works
very well (externally, not necessarily internally!).  One is just
accustomed to be able to let-bind whatever one wants, and lexical
scoping and hygiene take care of all worries ... except when not. :-)
(In this case it has nothing to do with quote/syntax/etc., just giving
an example of what silly mistakes I can make when not careful.)

(Nowadays I name all my pattern variables <foo> for that reason.  Reads
like BNF too, which is nice.  And I don't see it ever clashing with
record type names in practice.)

> As to "syntax", the use of #'x is much rarer, and the idea that #'x is
> (syntax x) is indeed a bit controversial. But this regards the whole
> syntax-case macro system, and I think that it would be more valuable
> to think how to fix its flaws, rather than change the very
> fundamentals of the language.

Hmm, I'm not sure what flaws of syntax-case you have in mind.  IMO it's
a pretty nice system.  But either way, I don't think making #'foo expand
to (__syntax__ foo), and simply making __syntax__ a synonyms to syntax,
are fundamental changes.  I would have thought it's a rather superficial
change...

> Best regards,
> M.

Kind regards,
Taylan

Re: Making apostrophe, backtick, etc. hygienic?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 5123 bytes --]

2015-08-30 15:16 GMT+02:00 Taylan Ulrich Bayırlı/Kammer <
taylanbayirli@gmail.com>:

> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>
> > You mean that #'x is synonymous to (syntax x), and that's where the
> > problem stems from?
>
> Yup.  I shadow 'syntax', but I don't explicitly shadow "#'".  It gets
> shadowed implicitly.  Lexical scoping and hygiene are supposed to let
> the programmer forget about such worries.
>
> > I think that every lisper should know that 'x is synonymous to (quote
> > x), and in some contexts it might be desirable to bind a new meaning
> > to the "quote" form (and this is already done by some libraries,
> > notably in the (ice-9 match) module).
>
> One may know, but still forget when sufficiently tired, and/or when
> using the word "quote" to mean something conceptually different than the
> quoting in lisp.
>

You could say that about practically any keyword in Scheme, or for that
matter any other language. The advantage of Scheme is that it really allows
you to redefine any keyword you like. Your point is that quote (and
unquote, and quasiquote, and syntax, and unsyntax, and quasisyntax) is a
reader macro, so one might forget that 'x is really (quote x) -- because
that indeed cannot be infered from the source code.


> For instance, some kind of text processing program might give the term
> "quote" a specific meaning in the program's problem domain, and once
> you're immersed deeply enough in this domain, you might find yourself
> naming some function parameter "quote" without giving it a second
> thought.  Kind of difficult to explain what I mean, but I know it
> happens to me when I'm not careful.


You've got the point, but I think that the only reasonable solution would
be to make the compiler issue warning whenever reader macro identifiers are
being shadowed.


>   As another example, it also keeps
> happening to me that I write code like:
>
>     (syntax-rules ()
>       ((_ foo)
>        (begin
>          ...
>          (let ((foo bar))
>            ...))))
>
> where I forget that the 'foo' there will not be bound freshly by that
> let form, and that despite that I understand how 'syntax-rules' works
> very well (externally, not necessarily internally!).  One is just
> accustomed to be able to let-bind whatever one wants, and lexical
> scoping and hygiene take care of all worries ... except when not. :-)
> (In this case it has nothing to do with quote/syntax/etc., just giving
> an example of what silly mistakes I can make when not careful.)
>
>
Well, misspellings happen all the time, but I think that just as their
consequences can usually be avoided in real life by use of sanity,
unit-testing your code is the best way to make sure that it is sane.


> (Nowadays I name all my pattern variables <foo> for that reason.  Reads
> like BNF too, which is nice.  And I don't see it ever clashing with
> record type names in practice.)
>
> > As to "syntax", the use of #'x is much rarer, and the idea that #'x is
> > (syntax x) is indeed a bit controversial. But this regards the whole
> > syntax-case macro system, and I think that it would be more valuable
> > to think how to fix its flaws, rather than change the very
> > fundamentals of the language.
>
> Hmm, I'm not sure what flaws of syntax-case you have in mind.  IMO it's
> a pretty nice system.


syntax-rules are nice, because they allow to comprehend macro
transformations in terms of textual substitutions (as it is the case with
functional programming), but because they expand in normal (rather than
applicative) order, it's difficult to write more complex macros.
The well-known solution is to use CPS macros (which are very difficult to
comprehend) or Oleg Kiselyov's idea to implement the CK abstract machine in
them.
The third way, often the most intuitive, to influence the order of
expansion, is to use syntax-case. However, if you do so, you can no longer
(in general) analyze the macros in terms of textual substitution. And you
need to use some new weird special forms, like the aforementioned "syntax"

But either way, I don't think making #'foo expand
> to (__syntax__ foo), and simply making __syntax__ a synonyms to syntax,
> are fundamental changes.


Putting the issue with "syntax" aside, making 'foo expand to (__quote__
foo) would be surprising to anyone who actually wanted to shadow "quote".
As I mentioned earlier, there are libraries that make use of the fact that
'x is (quote x). Take a look in here, for example:
http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/ice-9/match.upstream.scm;h=ede1d43c9ff8b085cb5709678c4227f5ecaaa8a5;hb=HEAD#l335

(match '(a b)
  (('a 'b) #t)
  (_ #f))

would no longer evaluate to #t, because the ('a 'b) pattern would actually
be read as ((__quote__ a) (__quote__ b)). You'd need to change all
occurences of "quote" with "__quote__" in the match.upstream.scm (and in
every other library that shadows quote for its purpose) in order to make it
work, thus making Guile non-RnRS-compliant.

[-- Attachment #2: Type: text/html, Size: 7097 bytes --]

Re: Making apostrophe, backtick, etc. hygienic?

[Taylan Ulrich Bayırlı/Kammer]

Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> Your point is that quote (and unquote, and quasiquote, and syntax, and
> unsyntax, and quasisyntax) is a reader macro, so one might forget that
> 'x is really (quote x) -- because that indeed cannot be infered from
> the source code.

Yup, exactly.

> You've got the point, but I think that the only reasonable solution
> would be to make the compiler issue warning whenever reader macro
> identifiers are being shadowed.

That's a good idea as well.  It might annoy some users though, when they
really want to shadow 'quote' (or 'syntax').  Dunno.

> Putting the issue with "syntax" aside, making 'foo expand to
> (__quote__ foo) would be surprising to anyone who actually wanted to
> shadow "quote". As I mentioned earlier, there are libraries that make
> use of the fact that 'x is (quote x). Take a look in here, for
> example:
> http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/ice-9/match.upstream.scm;h=ede1d43c9ff8b085cb5709678c4227f5ecaaa8a5;hb=HEAD#l335
>
> (match '(a b)
> (('a 'b) #t)
> (_ #f))
>
> would no longer evaluate to #t, because the ('a 'b) pattern would
> actually be read as ((__quote__ a) (__quote__ b)). You'd need to
> change all occurences of "quote" with "__quote__" in the
> match.upstream.scm (and in every other library that shadows quote for
> its purpose) in order to make it work, thus making Guile
> non-RnRS-compliant.

Hmm, that gets a little complicated, yeah.  Still, in highly RnRS
compliant systems, macros actually match their "literal" inputs by
(hygienic) "bindings" and not the names of identifiers.  I.e., if the
quote and __quote__ identifiers hold the *same binding*, then a macro
that has 'quote' in its literals list will also match '__quote__' for
that literal.  (Magic!)  I seem to remember Guile 2.2 really does this
the pedantically right way, while Guile 2.0 is more lax about it.

This would even help those libraries then.  The user could shadow the
'quote' identifier's binding locally, meaning 'match' is going to reject
it because it's not the binding which 'match' knows, but 'foo will keep
working in 'match' because it will expand to '__quote__' which has not
been shadowed and so still holds the same binding as the 'quote' binding
with which 'match' was defined.

The only RnRS-compliant code that would break is code which itself
shadows 'quote' and expects its shadowing to work with 'foo.  Like:

    (let ((quote -)) '9)  ;=> -9

Dunno if there's any serious Scheme/Guile code out in the wild which
actually relies on this working.

Taylan

Re: Making apostrophe, backtick, etc. hygienic?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 4791 bytes --]

2015-08-30 16:47 GMT+02:00 Taylan Ulrich Bayırlı/Kammer <
taylanbayirli@gmail.com>:

> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>
> > Your point is that quote (and unquote, and quasiquote, and syntax, and
> > unsyntax, and quasisyntax) is a reader macro, so one might forget that
> > 'x is really (quote x) -- because that indeed cannot be infered from
> > the source code.
>
> Yup, exactly.
>
> > You've got the point, but I think that the only reasonable solution
> > would be to make the compiler issue warning whenever reader macro
> > identifiers are being shadowed.
>
> That's a good idea as well.  It might annoy some users though, when they
> really want to shadow 'quote' (or 'syntax').  Dunno.
>

This could actually be solved by some additional means -- for example, one
could have to write some additional statements that would confirm that he's
aware that the reader macros are being shadowed. For instance

(define-syntax match
  (syntax-rules (.... quote ....)
    ....))

(assert (syntax-shadows? match 'quote)) ;; removes the warning

But to be honest, I don't think that this is a real problem. The problem
manifested itself with the "syntax" form, and not the "quote" form, and I
think the reason for that is not just accidental. The "quote" form is more
common and more commonly used, while the "syntax" form is exotic and
surprising -- especially because everyone unfamiliar would read #' as
"hash-quote" rather than "syntax"


> > Putting the issue with "syntax" aside, making 'foo expand to
> > (__quote__ foo) would be surprising to anyone who actually wanted to
> > shadow "quote". As I mentioned earlier, there are libraries that make
> > use of the fact that 'x is (quote x). Take a look in here, for
> > example:
> >
> http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/ice-9/match.upstream.scm;h=ede1d43c9ff8b085cb5709678c4227f5ecaaa8a5;hb=HEAD#l335
> >
> > (match '(a b)
> > (('a 'b) #t)
> > (_ #f))
> >
> > would no longer evaluate to #t, because the ('a 'b) pattern would
> > actually be read as ((__quote__ a) (__quote__ b)). You'd need to
> > change all occurences of "quote" with "__quote__" in the
> > match.upstream.scm (and in every other library that shadows quote for
> > its purpose) in order to make it work, thus making Guile
> > non-RnRS-compliant.
>
> Hmm, that gets a little complicated, yeah.  Still, in highly RnRS
> compliant systems, macros actually match their "literal" inputs by
> (hygienic) "bindings" and not the names of identifiers.  I.e., if the
> quote and __quote__ identifiers hold the *same binding*, then a macro
> that has 'quote' in its literals list will also match '__quote__' for
> that literal.  (Magic!)  I seem to remember Guile 2.2 really does this
> the pedantically right way, while Guile 2.0 is more lax about it.
>
>
I think that this is the case for R6RS or R7RS, but as far as I can tell,
in R5RS it would be problematic.

The only RnRS-compliant code that would break is code which itself
> shadows 'quote' and expects its shadowing to work with 'foo.  Like:
>
>     (let ((quote -)) '9)  ;=> -9
>
> Dunno if there's any serious Scheme/Guile code out in the wild which
> actually relies on this working.
>
>
You'll never know. While it may seem unlikely to fix quote to mean minus,
in mathematical analysis the symbol is often used to mean the derivative of
a unary function, so it is quite possible that someone would wish to write
in some context

(let ((quote deriv))
  (+ (f x) ('f x) (''f x)))

On the other hand, your soultion would work if someone decided to write

(let ('deriv)
 (+ (f x) ('f x) (''f x)))

(which is IMO more elegant)

Nevertheless, I think that even if 'x would map to (__quote__ x), it could
still happen that someone was using the __double_underscore__ convention in
her code (for some reason), and your allegations would apply to this new
situation as well.

As I said earlier, I think that the problem isn't caused by the fact that
'x is (quote x), because it is likely that every lisp programmer reads 'x
as "quote x", but by the fact that there is this weird "syntax" form (and
its family) which has only one application in Scheme, namely -- syntax-case
macros.

You could ask the question on comp.lang.scheme newsgroup, but I think that
the solution you suggest would only introduce unnecessary divergence from
Lisp and Scheme, and for a dubious reason. Furthermore, while it is common
to use these __underscores__ in C, PHP or Python, it is an alien practice
in the Scheme code base. (Instead of "quote x", you'd need to read 'x as
"underscore underscore quote underscore underscore x", which is unhandy and
brain-damaging)

Regards,
M.

[-- Attachment #2: Type: text/html, Size: 6738 bytes --]

Re: Making apostrophe, backtick, etc. hygienic?

[Taylan Ulrich Bayırlı/Kammer]

Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> I think that this is the case for R6RS or R7RS, but as far as I can
> tell, in R5RS it would be problematic.

I think the R5RS is implicitly silent on that by virtue of not having a
library system.  It says that a literal in a pattern will match "if it
has the same lexical binding" and if an extension to R5RS provides a way
to create different identifiers with the "same lexical binding" then
that wouldn't contradict the standard. :-)

I don't think we should obsess over compliance that much anyway though.
The unhygienic 'foo is one of the few very strange things in Scheme IMO.

> You'll never know. While it may seem unlikely to fix quote to mean
> minus, in mathematical analysis the symbol is often used to mean the
> derivative of a unary function, so it is quite possible that someone
> would wish to write in some context
>
> (let ((quote deriv))
>   (+ (f x) ('f x) (''f x)))
>
> On the other hand, your soultion would work if someone decided to
> write
>
> (let ('deriv)
>   (+ (f x) ('f x) (''f x)))
>
> (which is IMO more elegant)

IMO the rarity of such code, plus the easiness of fixing it, make that a
non-issue.

> Nevertheless, I think that even if 'x would map to (__quote__ x), it
> could still happen that someone was using the __double_underscore__
> convention in her code (for some reason), and your allegations would
> apply to this new situation as well.

That's much less likely, and it needn't be underscores either.  E.g. it
could be %%quote instead.  (Maybe we already have a convention for this,
I don't know.)


Anyway, if people don't like the idea for whatever reason then I won't
push it.  I just thought it would be neat to uphold the hygiene a little
farther, and not make it break when a user binds an identifier that's a
plain word like quote or syntax.

Taylan

Re: Making apostrophe, backtick, etc. hygienic?

[Mark H Weaver]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> How about making 'foo turn into something like (__quote__ foo), and
> similar for `foo, #'foo, etc.?

This is part of the standard behavior of 'read', e.g. section 4.3.5 of
the R6RS, and has been for over 50 years.  Lots of existing code assumes
this.  Changing it now is out of the question.

Also, the subject of this thread is misleading, because your proposal
would not make these reader abbreviations hygienic.

> Where __quote__ is just a synonym to quote, and the original works
> too.

Not when some arbitrary code looks at the result of 'read', or inside a
quoted datum like '(nested quote 'x).

     Mark