Re: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

Re: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Andy Wingo]

Hi Ian!

In the future, please include bug-guile@gnu.org in bug reports.  It
creates a ticket so we don't forget.  The system changed a bit recently,
but we hope that it works OK for you.  It's certainly working better for
us.

Anyway:

On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:

> (define-public (void? x) (eq? x (begin)))

This is really not right, if I understand what you are checking for.
These will work:

  (define-public (void? x) (unspecified? x))
  (define-public (void? x) (eq? x *unspecified*))
  (define-public (void? x) (eqv? x *unspecified*))
  (define-public (void? x) (eq? x (if #f #f))

> This works in V1.8, and apparently used to work in 2.0.2 (no errors),
> but in 2.0.3
> (begin) is OK at the repl, (with readline enabled and activated), but
> in any sort of procedure using it causes a diagnostic, reporting
> either at the repl or in .scm files,
>
> xxx: source expression failed to match any pattern in form (begin).

Interesting.  If it changed incompatibly in 2.0.x, that is a Guile bug.
Sorry about that!  We'll fix it.

That said, though, you have a few misunderstandings here.

> 6.13.1 Evaluating a series of expressions
> <snip>
> — syntax: begin expr1 expr2 ...

First of all, this documentation is not quite correct.  Since R5RS there
have been *two* kinds of begins: one that splices and one that
sequences.

I know that sounds complicated, but let me explain.  It's not widely
known, and many Schemers consider it to be something of a bug that they
both share the same name.

Scheme has two contexts: definition context and expression context.
Like, you can only put nested definitions at the top of a function,
before the expressions, and a function has to end with an expression.

`Begin' in definition context "splices" its subforms into its containing
form.  This is to allow macros to expand to multiple definitions.
In definition context, `begin' is just syntactic.

`Begin' in expression context evaluates its subforms in order.  It is
used for its semantic property of sequencing effects.

It is possible for a `begin' in definition context to have no subforms.
But in expression context, you need an expression, and if there are no
expressions to evaluate, the result is a syntactic error.  That is what
you are running into.

>     Guile also allows the expression (begin), a begin with no
> sub-expressions. Such an expression returns the `unspecified' value.

So, I guess we have to fix this in 2.0.x.  This behavior will probably
not be available in 2.2, however, so it's best to change your code now,
at the same time that we fix Guile to have the old behavior (and fix the
docs).

Finally, we are considering having expressions that produce unspecified
values actually produce 0 values.  So, for example, (if #f #f) would be
the same as (values).  Of course it's not something we can do in 2.0,
and we would try to publicise it well, but do keep it in mind.

Regards,

Andy
-- 
http://wingolog.org/

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[David Kastrup]

Ian Hulin <ian@hulin.org.uk> writes:

> Hi Andy,
> On 23/11/11 12:27, Andy Wingo wrote:
>> Hi Ian,
>> 
>> Did it really work in v2.0.2, I wonder?
>> 
> Apparently not, see below for explanation.

This mail addresses two different issues, mixing them in somewhat
surprising ways.  I'll try to split the issues again.

>> On Tue 22 Nov 2011 00:02, Andy Wingo <wingo@pobox.com> writes:
>>> On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:
>>>> (define-public (void? x) (eq? x (begin))) This works in V1.8,
>>>> and apparently used to work in 2.0.2 (no errors),
>>> 
>>> Interesting.  If it changed incompatibly in 2.0.x, that is a
>>> Guile bug. Sorry about that!  We'll fix it.
>> 
>> I cannot reproduce this issue with 2.0.2.  I suspect that this code
>> has been this way since early 1.9 releases, and is a consequence of
>> switching to the new syntax expander.
>> 
> Looks like the change was already in 2.0.2. Another LilyPond developer
> (David Kastrup, dak@gnu.org) noticed the advertised API and decided to
> use it in part of an ongoing clean-up of the Lily parser.  Meanwhile I
> was working on the Guile V2 migration stuff and it broke.

More concretely regarding "the advertised API": the Guilev1.8
documentation does neither mention `*unspecified*', nor `unspecified?'.
On the C side, it documents SCM_UNSPECIFIED.  On the Scheme side, the
only available information is the documentation for begin:

 -- syntax: begin expr1 expr2 ...
     The expression(s) are evaluated in left-to-right order and the
     value of the last expression is returned as the value of the
     `begin'-expression.  This expression type is used when the
     expressions before the last one are evaluated for their side
     effects.

     Guile also allows the expression `(begin)', a `begin' with no
     sub-expressions.  Such an expression returns the `unspecified'
     value.

So it is somewhat disingenuous to laugh off such uses of begin.  At the
very least you should add documentation for *unspecified* and
unspecified? to the documentation of the 1.8 branch.  That way, you at
least can point people to documentation relevant for the migration.

There _are_ a few problems even in Guilev1.8: v1.8 refuses to accept
some forms that should likely be *unspecified*:
guile> (lambda ())

Backtrace:
In standard input:
   1: 0* (lambda ())

standard input:1:1: In procedure memoization in expression (lambda ()):
standard input:1:1: In file "standard input", line 0: Missing expression in (lambda ()).
ABORT: (syntax-error)
guile> ((lambda () (begin)))

Backtrace:
In standard input:
   4: 0* [#<procedure #f ()>]

standard input:4:1: In procedure memoization in expression ((lambda () #)):
standard input:4:1: Missing body expression in ((begin)).
ABORT: (syntax-error)
guile> ((lambda () (begin (begin))))

Backtrace:
In standard input:
   6: 0* [#<procedure #f ()>]

standard input:6:1: In procedure memoization in expression ((lambda () #)):
standard input:6:1: Missing body expression in ((begin (begin))).
ABORT: (syntax-error)
guile> ((lambda () #f (begin)))
guile> 

I have switched to using *unspecified* and unspecified? by now, but it
might be worth ironing out the strange behaviors of (begin) nevertheless
in particular where they conflict with the behavior to be expected from
the documentation (preferably both in v1.8 as well as v2), in addition
to mentioning the more straightforward commands.

So much for that.  The next quote is for a totally different issue, the
availability of local environments and evaluation in them.  Lilypond has
an input syntax of its own, and it allows interspersing Scheme code. $
or # switches to the Scheme interpreter (for one sexp) when in Lilypond
syntax, and #{ ... #} switches to Lilypond inside.

We have constructs like

spacingTweaks =
#(define-music-function (parser location parameters) (list?)
   (_i "Set the system stretch, by reading the 'system-stretch property of
the `parameters' assoc list.")
   #{
     \overrideProperty #"Score.NonMusicalPaperColumn"
     #'line-break-system-details
     #(list (cons 'alignment-extra-space (cdr (assoc 'system-stretch parameters)))
	     (cons 'system-Y-extent (cdr (assoc 'system-Y-extent parameters))))
   #})

As you can see, the music function argument "parameters" is used in
Scheme code embedded in Lilypond code.  Previously, #{ (implemented in
the Scheme reader) captured the enclosed material at read time as a
string for interpretation (using the Scheme reader to skip over any
#sexp or $sexp inside) as well as (procedure-environment (lambda ()
'())), then called the Lilypond parser at execution time, and the parser
executed all embedded Scheme in the procedure-environment captured in
the lexical scope of the define-music-function text.

My _current_ implementation can't capture an execution environment, so I
need to precompile a whole host of closures, and later correlate them
with input source positions.  This is somewhat error-prone since the
full parsing of Lilypond might not agree with the cursory scan of the #{
... #} body.  Since the parser and lexer are context-dependent and can
switch behavior according to variables at execution time, a more
complete parse at compile time is not feasible.

To give an example, the above #{ ... #} nowadays gets translated into

(let* ((clone (ly:parser-clone parser (list (lambda () "Score.NonMusicalPaperColumn") (lambda () (quote line-break-system-details)) (lambda () (list (cons (quote alignment-extra-space) (cdr (assoc (quote system-stretch) parameters))) (cons (quote system-Y-extent) (cdr (assoc (quote system-Y-extent) parameters)))))))) (result (ly:parse-string-expression clone "
     \\overrideProperty #\"Score.NonMusicalPaperColumn\"
     #(quote line-break-system-details)
     #(list (cons (quote alignment-extra-space) (cdr (assoc (quote system-stretch) parameters))) (cons (quote system-Y-extent) (cdr (assoc (quote system-Y-extent) parameters))))
   "))) (if (ly:parser-has-error? clone) (ly:parser-error parser (_ "error in #{ ... #}"))) result)

Namely, all contained # expressions are precompiled into
(list (lambda () "Score.NonMusicalPaperColumn")
      (lambda () (quote line-break-system-details))
      (lambda () (list (cons (quote alignment-extra-space)
			     (cdr (assoc (quote system-stretch)
					 parameters)))
		       (cons (quote system-Y-extent)
			     (cdr (assoc (quote system-Y-extent)
					 parameters))))))

Previously, all that was needed to record instead was

(procedure-environment (lambda () '()))

(again note the useless '() to bypass the issue of complaints about a
missing body) and just use that for feeding local-eval inside of
ly:parse-string-expression.

This results in quite heavy storage requirements and will cause problem
if the full parsing does not use the same # $ starting points as the
preliminary scanning, like when # or $ are inside of Lilypond comments:
they will get turned into lambdas at compile time, but not regarded at
execution time.

I have a patch in the queue for Lilypond that also records the starting
positions in the string, and skips "obvious" constants, only wrapping
symbols and pairs with a car different from 'quote into lambdas.  That
reduces the stored amounts of data and improves reliability, at the cost
of considerable complexity.

> I'm sure David will let me know if I've over-simplified stuff or got
> things wrong here.
>
> However, it boils down to lousy timing for LilyPond finding a use for
> this bit of the Guile API, which you guys had decided to withdraw
> because no-one apparently had a use for it :-{

The use would be in implementing embedded lexical environments not
exclusively written in Scheme.  The thrust of Guilev2 to completely
abandon support of interpreted and nested environments in favor of
getting speedups for compiled applications casts doubt on its usefulness
as an _extension_ language, and that was the principal aim
distinguishing it from compiled Scheme variants like Chicken which, I
would guess, have enough of a headstart compared to Guile that
abandoning Guile's original goals and trying to compete with the
established compilers on their home ground is not likely to increase the
core audience of Guile.

As I said: for this particular application, I have coded a rather
inelegant and resource-grabbing workaround that really is not going to
help performance since the intertwined Lilypond interpreter does not
benefit from precompilation of mostly trivial lambda functions when the
actual procedure-environment is unlikely to ever reference more than
five variables.

So for smooth integration into an interpretative environment, this is a
definite step for the worse.  It was perhaps a reasonably easy decision
to make since the original functionality was heavily underused, likely
also due to abysmally bad documentation that provides not much more than
the motivation to hunt through source code and search the web for other
people who actually managed to make something work.

Please feel free to split this twofold mail into separate parts and
answer in appropriate lists.  I apologize for not having a clue what
belongs where myself.

-- 
David Kastrup

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Andy Wingo]

Hi David,

Thanks for the note, and good to hear from you on the Guile side.  You
are right in pointing out areas that can improve.  As you might imagine,
we are currently more focussed on the current stable 2.0.x and
development 2.2.x series, so we probably won't do very much to the 1.8
documentation, for example.  I understand if you need to keep 1.8
support for a time, but the 2.0 documentation is much better.

I am going to open a new thread for the local-eval issue, if you don't
mind.

On Fri 25 Nov 2011 11:37, David Kastrup <dak@gnu.org> writes:

> (lambda ())
> ((lambda () (begin)))
> ((lambda () (begin (begin))))

FWIW, we are not going to change to support these in any branch of
Guile.  No other Scheme does so, and we mostly take our language design
hints from the Scheme standards.

> I have switched to using *unspecified* and unspecified? by now, but it
> might be worth ironing out the strange behaviors of (begin) nevertheless
> in particular where they conflict with the behavior to be expected from
> the documentation (preferably both in v1.8 as well as v2), in addition
> to mentioning the more straightforward commands.

I'll see what we can do.  I think the decision here is going to depend
on what we do in 2.2 regarding (if #f #f).  In any case, we do need to
improve our documentation.

Andy
-- 
http://wingolog.org/

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Andy Wingo]

Hi Ian,

Did it really work in v2.0.2, I wonder?

On Tue 22 Nov 2011 00:02, Andy Wingo <wingo@pobox.com> writes:
>On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:
>> (define-public (void? x) (eq? x (begin)))
>> This works in V1.8, and apparently used to work in 2.0.2 (no errors),
>
> Interesting.  If it changed incompatibly in 2.0.x, that is a Guile bug.
> Sorry about that!  We'll fix it.

I cannot reproduce this issue with 2.0.2.  I suspect that this code has
been this way since early 1.9 releases, and is a consquence of switching
to the new syntax expander.

That said, this change was not mentioned in NEWS, and the documentation
has not been updated, so perhaps we should do something.  I would prefer
just to leave it as it is, unless you think that adding a compatibility
hack could help you somehow.  What do you think, Ian?

Andy
-- 
http://wingolog.org/

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Ian Hulin]

Hi Andy,
On 23/11/11 12:27, Andy Wingo wrote:
> Hi Ian,
> 
> Did it really work in v2.0.2, I wonder?
> 
Apparently not, see below for explanation.

> On Tue 22 Nov 2011 00:02, Andy Wingo <wingo@pobox.com> writes:
>> On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:
>>> (define-public (void? x) (eq? x (begin))) This works in V1.8,
>>> and apparently used to work in 2.0.2 (no errors),
>> 
>> Interesting.  If it changed incompatibly in 2.0.x, that is a
>> Guile bug. Sorry about that!  We'll fix it.
> 
> I cannot reproduce this issue with 2.0.2.  I suspect that this code
> has been this way since early 1.9 releases, and is a consequence of
> switching to the new syntax expander.
> 
Looks like the change was already in 2.0.2. Another LilyPond developer
(David Kastrup, dak@gnu.org) noticed the advertised API and decided to
use it in part of an ongoing clean-up of the Lily parser.  Meanwhile I
was working on the Guile V2 migration stuff and it broke.
> That said, this change was not mentioned in NEWS, and the
> documentation has not been updated, so perhaps we should do
> something.  I would prefer just to leave it as it is, unless you
> think that adding a compatibility hack could help you somehow.
> What do you think, Ian?
My learned colleague is working round it, but at the very least you
need to deprecate the support for this and document it as such.

Or is there a way of keeping scm_local_eval and procedure-environment
in the API for use only when interpreting?

Here's why I ask the question.

We were wanting to use it in Scheme statements embedded in LilyPond
source code, especially where these needed to pass values back to the
LilyPond parser.  Using scm_local_eval/local-eval solved problems with
lots of kludgy things involving parser look-aheads and suchlike.

We are using #{ and #} to delimit LilyPond code within a Scheme block
which is itself embedded in LilyPond code.
$ is typically used to mark variables to be accessed by the inner
LilyPond which are declared in the intermediate Scheme block.

This is a quote from David's LilyPond issue tracker commentary for the
work-around:

> Lambdaize $ and # in #{ ... #} to make Guile V2 happy.
> 
> Unfortunately Guile V2 was not happy about the implementation of 
> closures around #{ ... #} relying on local-eval and 
> procedure-environment.  This patch abandons this approach and gets 
> back to a scheme similar like the old one, namely compiling the 
> expressions inside of #{ ... #} in advance.

I'm sure David will let me know if I've over-simplified stuff or got
things wrong here.

However, it boils down to lousy timing for LilyPond finding a use for
this bit of the Guile API, which you guys had decided to withdraw
because no-one apparently had a use for it :-{

Cheers,

Ian Hulin

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Andy Wingo]

Greetings,

On Fri 25 Nov 2011 00:49, Ian Hulin <ian@hulin.org.uk> writes:

>>> On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:
>>>> (define-public (void? x) (eq? x (begin)))

>> That said, this change was not mentioned in NEWS, and the
>> documentation has not been updated, so perhaps we should do
>> something.  I would prefer just to leave it as it is, unless you
>> think that adding a compatibility hack could help you somehow.
>> What do you think, Ian?

> My learned colleague is working round it, but at the very least you
> need to deprecate the support for this and document it as such.
>
> Or is there a way of keeping scm_local_eval and procedure-environment
> in the API for use only when interpreting?

Is local-eval related to `(begin)' in expression context?  I am a bit
confused here.

>> Unfortunately Guile V2 was not happy about the implementation of 
>> closures around #{ ... #} relying on local-eval and 
>> procedure-environment.  This patch abandons this approach and gets 
>> back to a scheme similar like the old one, namely compiling the 
>> expressions inside of #{ ... #} in advance.
>
> I'm sure David will let me know if I've over-simplified stuff or got
> things wrong here.
>
> However, it boils down to lousy timing for LilyPond finding a use for
> this bit of the Guile API, which you guys had decided to withdraw
> because no-one apparently had a use for it :-{

Yeah, I'm sorry for the miscommunication here!  Allow me to explain.

Local-eval is a case of "scripting" being in opposition to
general-purpose programming.  For an simple interpreter (as Guile 1.8
was), it's easy to support, but local-eval makes a compiler's job
harder.  No other Scheme besides SCM had it.  By way of comparison, note
that local-eval in JavaScript is now deprecated and removed entirely in
strict mode.

We wanted to remove it because we wanted to make Guile faster.  We were
able to remove it because it was non-standard and not really used.  It
was a tough decision, but I think the right one in this context.

If you really want something like local-eval, you need to implement your
own `eval'.  That sounds like a PITA, but in reality it will be as fast
as Guile's anyway.  I posted one implementation here:

  http://lists.gnu.org/archive/html/guile-user/2011-02/msg00032.html

If you are still curious, search the internet for "fexpr", which is
related to the-environment and local-eval.

Finally, I recommend having another look at the NEWS for Guile 2.0.  It
is a long read, but reading it will probably prevent future
misunderstandings.

Peace,

Andy
-- 
http://wingolog.org/

bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[Andy Wingo]

Hi,

On Wed 23 Nov 2011 07:27, Andy Wingo <wingo@pobox.com> writes:

> On Tue 22 Nov 2011 00:02, Andy Wingo <wingo@pobox.com> writes:
>>On Mon 21 Nov 2011 18:25, Ian Hulin <ian@hulin.org.uk> writes:
>>> (define-public (void? x) (eq? x (begin)))
>>> This works in V1.8, and apparently used to work in 2.0.2 (no errors),
>>
>> Interesting.  If it changed incompatibly in 2.0.x, that is a Guile bug.
>> Sorry about that!  We'll fix it.
>
> I cannot reproduce this issue with 2.0.2.  I suspect that this code has
> been this way since early 1.9 releases, and is a consquence of switching
> to the new syntax expander.
>
> That said, this change was not mentioned in NEWS, and the documentation
> has not been updated, so perhaps we should do something.  I would prefer
> just to leave it as it is, unless you think that adding a compatibility
> hack could help you somehow.  What do you think, Ian?

I have appended the updated documentation to this reply to clarify that
(begin) in expression context is invalid syntax.  I have also added a
backwards-compatibility shim to issue a deprecation warning and
translate it to *unspecified*, in Guile 2.0.  That should fix this bug.

Regards,

Andy


6.13.1 Sequencing and Splicing
------------------------------

As an expression, the `begin' syntax is used to evaluate a sequence of
sub-expressions in order.  Consider the conditional expression below:

     (if (> x 0)
         (begin (display "greater") (newline)))

   If the test is true, we want to display "greater" to the current
output port, then display a newline.  We use `begin' to form a compound
expression out of this sequence of sub-expressions.

 -- syntax: begin expr1 expr2 ...
     The expression(s) are evaluated in left-to-right order and the
     value of the last expression is returned as the value of the
     `begin'-expression.  This expression type is used when the
     expressions before the last one are evaluated for their side
     effects.

   The `begin' syntax has another role in definition context (*note
Internal Definitions::).  A `begin' form in a definition context
"splices" its subforms into its place.  For example, consider the
following procedure:

     (define (make-seal)
       (define-sealant seal open)
       (values seal open))

   Let us assume the existence of a `define-sealant' macro that expands
out to some definitions wrapped in a `begin', like so:

     (define (make-seal)
       (begin
         (define seal-tag
           (list 'seal))
         (define (seal x)
           (cons seal-tag x))
         (define (sealed? x)
           (and (pair? x) (eq? (car x) seal-tag)))
         (define (open x)
           (if (sealed? x)
               (cdr x)
               (error "Expected a sealed value:" x))))
       (values seal open))

   Here, because the `begin' is in definition context, its subforms are
"spliced" into the place of the `begin'.  This allows the definitions
created by the macro to be visible to the following expression, the
`values' form.

   It is a fine point, but splicing and sequencing are different.  It
can make sense to splice zero forms, because it can make sense to have
zero internal definitions before the expressions in a procedure or
lexical binding form.  However it does not make sense to have a
sequence of zero expressions, because in that case it would not be
clear what the value of the sequence would be, because in a sequence of
zero expressions, there can be no last value.  Sequencing zero
expressions is an error.

   It would be more elegant in some ways to eliminate splicing from the
Scheme language, and without macros (*note Macros::), that would be a
good idea.  But it is useful to be able to write macros that expand out
to multiple definitions, as in `define-sealant' above, so Scheme abuses
the `begin' form for these two tasks.


-- 
http://wingolog.org/