bug#52749: G-expressions don't consistently preserve #nil

[Philip McGrath <philip@philipmcgrath.com>]

Hi!

Just as a general disclaimer, I'm a Racketeer and only incidentally a 
Schemer, so I'm not very familiar with the universe of Guile libraries.

On 12/23/21 01:59, Maxime Devos wrote:
 > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
 >> G-expressions currently do not consistently preserve the distinction
 >> between #nil and '(), which causes trouble for programs that rely on
 >> that distinction. In particular, the issue affects programs that use
 >> (guix build json), because that library uses #nil to represent the JSON
 >> value `null', whereas it uses '() to represent an empty JSON array.
 >
 > The constant #nil is only for elisp compatibility and not something
 > supposed to be used in Scheme code that isn't for Scheme/elisp
 > compatibility, so this seems more a bug in (guix build json) to me.

That was not the impression I had gotten from `info "(guile)Nil"`. For 
example, I think someone who wanted to finish the implementation 
described in `info "(guile)ECMAScript"` might want to use #nil for one 
of the false-y ECMAScript values to take advantages of the documented 
efficiencies in its bit-level representation. More concretely, 
guile-json@1 and guile-json@3 use #nil in the same way as (guix build json).

On 12/23/21 12:58, Maxime Devos wrote:
 > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
 >> G-expressions currently do not consistently preserve the distinction
 >> between #nil and '(), which causes trouble for programs that rely on
 >> that distinction. In particular, the issue affects programs that use
 >> (guix build json), because that library uses #nil to represent the JSON
 >> value `null', whereas it uses '() to represent an empty JSON array.
 >>
 >> The following program exposes the error:
 >> [
 >> ;...]
 >>
 >> ; This one fails!
 >> (check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
 >>                 "{\"k\":null}"
 >>                 "gexp: null in object")
 >
 > A simpler test:
 >
 > Compare this:
 >    (cdr (gexp->approximate-sexp #~("stuff" . #nil)))
 >    ; output: #nil --- seems like everything is ok?
 >
 > with:
 >    (gexp->approximate-sexp #~("stuff" . #nil))
 >    ; output: ("stuff") --- where did the #nil go?
 >
 > I think the idea is that, if you construct a list (a b c . #nil)
 > in elisp, and pass it to Scheme, then Scheme should treat it as a
 > Scheme list, so it should be printed as (a b c) when using Scheme's
 > 'write' or 'display'.

Since `write` and `list?` are specified by various Scheme standards, I 
think it is the correct choice for `write` to use a Scheme-compatible 
external representation for values recognized by `list?`, at least by 
default. (Perhaps a parameter could control this behavior?)

I think the behavior of `gexp->approximate-sexp` is at least defensible, 
since its documentation (`info guix "gexp->approximate-sexp"`) warns 
that "some information can be lost".

But I think the implementation of G-expressions faces more stringent 
obligations. I see it as analogous to the implementation of syntax 
objects, a macro expander, or a compiler, in that it should have a 
semantics-preserving representation of arbitrary Guile code, including 
Guile's extensions to Scheme.

(I haven't yet understood at a theoretical level how "strata" and 
"staging" relate to the more familiar concept of "phases", but my 
intuition is that, while the R6RS model of phases wouldn't be enough, it 
seems like would probably to express staging/strata in terms of phases 
with Racket enhancements like the label phase level and arbitrary 
submodule-implemented phases.)

So, I agree that:

On 12/25/21 06:13, Maxime Devos wrote:
> That said, it would be less surprising if the #nil/() distinction is
> preserved by gexp->derivation and friends. This can be done by writing
> our own 'write' procedure. Downside: it might be less efficient than
> Guile's write which is implemented in C. Can be resolved by writing our
> own 'write' procedure in C.

I haven't looked at the implementation at all, but extending `write` 
certainly would be a reasonable option, and, longer-term, it might be 
possible to upstream a patch adding the needed behavior.

A more radical option could be to use a format other than plain-text 
s-expressions for compiled G-expressions. For example, Racket has a 
forward-compatible "fast-load serialization" binary format for the kinds 
of values that can be embedded in compiled code.[0] There are obvious 
disadvantages to a binary format, but advantages include the ability to 
preserve source-location information and to avoid some the quirks that 
come with functions like `write` and `read`, for historical reasons or 
for the convenience of humans writing code directly. The implementation 
is in Racket, so it should be fairly easy to port to Guile, if that were 
wanted.[1] Or maybe there's something related to Guile bytecode that 
would work, or maybe just making a `#nil`-preserving version of `write` 
would be easier.

-Philip

[0]: https://docs.racket-lang.org/reference/fasl.html
[1]: 
https://github.com/racket/racket/blob/master/racket/collects/racket/fasl.rkt