bug#52749: G-expressions don't consistently preserve #nil

bug#52749: G-expressions don't consistently preserve #nil

[Philip McGrath]

G-expressions currently do not consistently preserve the distinction 
between #nil and '(), which causes trouble for programs that rely on 
that distinction. In particular, the issue affects programs that use 
(guix build json), because that library uses #nil to represent the JSON 
value `null', whereas it uses '() to represent an empty JSON array.

The following program exposes the error:

--8<---------------cut here---------------start------------->8---
(use-modules (guix build json)
              (guix gexp)
              (guix monads)
              (guix store)
              (guix derivations)
              (rnrs conditions)
              (ice-9 textual-ports))

(define (check-equal? actual expected message)
   (define who 'check-equal?)
   (unless (equal? actual expected)
     (raise-exception
      (condition
       (make-assertion-violation)
       (make-who-condition 'check-equal?)
       (make-irritants-condition (list actual))
       (make-message-condition
        (format #f "~a: ~a\n  ~a: ~s\n  ~a: ~s\n  ~a: ~s"
                who "test failed"
                "message" message
                "expected" expected
                "actual" actual))))))

(define (sexp->json-string sx)
   (call-with-output-string
     (lambda (out)
       (write-json sx out))))

(define (gexp->json-string gx)
   (run-with-store (open-connection)
     (mlet* %store-monad ((drv (gexp->derivation "example.json"
                                 (with-imported-modules `((guix build json))
                                   #~(begin
                                       (use-modules (guix build json))
                                       (call-with-output-file #$output
                                         (lambda (out)
                                           (write-json #$gx out)))))))
                          (_built (built-derivations (list drv))))
       (return (call-with-input-file (derivation->output-path drv)
                 get-string-all)))))

(check-equal? (sexp->json-string '())
               "[]"
               "sexp: empty array")

(check-equal? (gexp->json-string #~'())
               "[]"
               "gexp: empty array")

(check-equal? (sexp->json-string #nil)
               "null"
               "sexp: null")

(check-equal? (gexp->json-string #~#nil)
               "null"
               "gexp: null")

(check-equal? (sexp->json-string '(@ ("k" . #nil)))
               "{\"k\":null}"
               "sexp: null in object")

;; This one fails!
(check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
               "{\"k\":null}"
               "gexp: null in object")
--8<---------------cut here---------------end--------------->8---

-Philip

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

Hi,

Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
> G-expressions currently do not consistently preserve the distinction 
> between #nil and '(), which causes trouble for programs that rely on 
> that distinction. In particular, the issue affects programs that use 
> (guix build json), because that library uses #nil to represent the JSON 
> value `null', whereas it uses '() to represent an empty JSON array.

The constant #nil is only for elisp compatibility and not something
supposed to be used in Scheme code that isn't for Scheme/elisp
compatibility, so this seems more a bug in (guix build json) to me.

Greetings,
Maxime.

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Maxime Devos schreef op do 23-12-2021 om 06:59 [+0000]:
> Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
> > G-expressions currently do not consistently preserve the distinction 
> > between #nil and '(), which causes trouble for programs that rely on 
> > that distinction. In particular, the issue affects programs that use 
> > (guix build json), because that library uses #nil to represent the JSON 
> > value `null', whereas it uses '() to represent an empty JSON array.
> 
> The constant #nil is only for elisp compatibility and not something
> supposed to be used in Scheme code that isn't for Scheme/elisp
> compatibility, so this seems more a bug in (guix build json) to me.

That said, it would be less surprising if the #nil/() distinction is
preserved by gexp->derivation and friends. This can be done by writing
our own 'write' procedure. Downside: it might be less efficient than
Guile's write which is implemented in C. Can be resolved by writing our
own 'write' procedure in C.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#52749: G-expressions don't consistently preserve #nil

[Philip McGrath]

Hi!

Just as a general disclaimer, I'm a Racketeer and only incidentally a 
Schemer, so I'm not very familiar with the universe of Guile libraries.

On 12/23/21 01:59, Maxime Devos wrote:
 > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
 >> G-expressions currently do not consistently preserve the distinction
 >> between #nil and '(), which causes trouble for programs that rely on
 >> that distinction. In particular, the issue affects programs that use
 >> (guix build json), because that library uses #nil to represent the JSON
 >> value `null', whereas it uses '() to represent an empty JSON array.
 >
 > The constant #nil is only for elisp compatibility and not something
 > supposed to be used in Scheme code that isn't for Scheme/elisp
 > compatibility, so this seems more a bug in (guix build json) to me.

That was not the impression I had gotten from `info "(guile)Nil"`. For 
example, I think someone who wanted to finish the implementation 
described in `info "(guile)ECMAScript"` might want to use #nil for one 
of the false-y ECMAScript values to take advantages of the documented 
efficiencies in its bit-level representation. More concretely, 
guile-json@1 and guile-json@3 use #nil in the same way as (guix build json).

On 12/23/21 12:58, Maxime Devos wrote:
 > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
 >> G-expressions currently do not consistently preserve the distinction
 >> between #nil and '(), which causes trouble for programs that rely on
 >> that distinction. In particular, the issue affects programs that use
 >> (guix build json), because that library uses #nil to represent the JSON
 >> value `null', whereas it uses '() to represent an empty JSON array.
 >>
 >> The following program exposes the error:
 >> [
 >> ;...]
 >>
 >> ; This one fails!
 >> (check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
 >>                 "{\"k\":null}"
 >>                 "gexp: null in object")
 >
 > A simpler test:
 >
 > Compare this:
 >    (cdr (gexp->approximate-sexp #~("stuff" . #nil)))
 >    ; output: #nil --- seems like everything is ok?
 >
 > with:
 >    (gexp->approximate-sexp #~("stuff" . #nil))
 >    ; output: ("stuff") --- where did the #nil go?
 >
 > I think the idea is that, if you construct a list (a b c . #nil)
 > in elisp, and pass it to Scheme, then Scheme should treat it as a
 > Scheme list, so it should be printed as (a b c) when using Scheme's
 > 'write' or 'display'.

Since `write` and `list?` are specified by various Scheme standards, I 
think it is the correct choice for `write` to use a Scheme-compatible 
external representation for values recognized by `list?`, at least by 
default. (Perhaps a parameter could control this behavior?)

I think the behavior of `gexp->approximate-sexp` is at least defensible, 
since its documentation (`info guix "gexp->approximate-sexp"`) warns 
that "some information can be lost".

But I think the implementation of G-expressions faces more stringent 
obligations. I see it as analogous to the implementation of syntax 
objects, a macro expander, or a compiler, in that it should have a 
semantics-preserving representation of arbitrary Guile code, including 
Guile's extensions to Scheme.

(I haven't yet understood at a theoretical level how "strata" and 
"staging" relate to the more familiar concept of "phases", but my 
intuition is that, while the R6RS model of phases wouldn't be enough, it 
seems like would probably to express staging/strata in terms of phases 
with Racket enhancements like the label phase level and arbitrary 
submodule-implemented phases.)

So, I agree that:

On 12/25/21 06:13, Maxime Devos wrote:
> That said, it would be less surprising if the #nil/() distinction is
> preserved by gexp->derivation and friends. This can be done by writing
> our own 'write' procedure. Downside: it might be less efficient than
> Guile's write which is implemented in C. Can be resolved by writing our
> own 'write' procedure in C.

I haven't looked at the implementation at all, but extending `write` 
certainly would be a reasonable option, and, longer-term, it might be 
possible to upstream a patch adding the needed behavior.

A more radical option could be to use a format other than plain-text 
s-expressions for compiled G-expressions. For example, Racket has a 
forward-compatible "fast-load serialization" binary format for the kinds 
of values that can be embedded in compiled code.[0] There are obvious 
disadvantages to a binary format, but advantages include the ability to 
preserve source-location information and to avoid some the quirks that 
come with functions like `write` and `read`, for historical reasons or 
for the convenience of humans writing code directly. The implementation 
is in Racket, so it should be fairly easy to port to Guile, if that were 
wanted.[1] Or maybe there's something related to Guile bytecode that 
would work, or maybe just making a `#nil`-preserving version of `write` 
would be easier.

-Philip

[0]: https://docs.racket-lang.org/reference/fasl.html
[1]: 
https://github.com/racket/racket/blob/master/racket/collects/racket/fasl.rkt

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2695 bytes --]

Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
> Hi!
> 
> Just as a general disclaimer, I'm a Racketeer and only incidentally a 
> Schemer, so I'm not very familiar with the universe of Guile libraries.
> 
> On 12/23/21 01:59, Maxime Devos wrote:
>  > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
>  >> G-expressions currently do not consistently preserve the distinction
>  >> between #nil and '(), which causes trouble for programs that rely on
>  >> that distinction. In particular, the issue affects programs that use
>  >> (guix build json), because that library uses #nil to represent the JSON
>  >> value `null', whereas it uses '() to represent an empty JSON array.
>  >
>  > The constant #nil is only for elisp compatibility and not something
>  > supposed to be used in Scheme code that isn't for Scheme/elisp
>  > compatibility, so this seems more a bug in (guix build json) to me.
> 
> That was not the impression I had gotten from `info "(guile)Nil"`. For 
> example, I think someone who wanted to finish the implementation 
> described in `info "(guile)ECMAScript"` might want to use #nil for one 
> of the false-y ECMAScript values to take advantages of the documented 
> efficiencies in its bit-level representation. More concretely, 
> guile-json@1 and guile-json@3 use #nil in the same way as (guix build json).

There is

‘Guile has chosen to support ‘nil’ as a separate value, distinct from
‘#f’ and ‘'()’.  This allows existing Scheme and Elisp code to maintain
their current semantics.  ‘nil’, which in Elisp would just be written
and read as ‘nil’, in Scheme has the external representation ‘#nil’.’

and

‘This decision to have ‘nil’ as a low-level distinct value facilitates
interoperability between the two languages.  Guile has chosen to have
Scheme deal with ‘nil’ as follows: [...]’

and this is only documented under ‘Emacs Lisp’, though this doesn't
explicitely say it's not supposed to be used elsewhere.

Also, see e.g.
<https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00017.html>.

Anyway, this doesn't really matter here, because:

> So, I agree that:
> 
> On 12/25/21 06:13, Maxime Devos wrote:
> > That said, it would be less surprising if the #nil/() distinction is
> > preserved by gexp->derivation and friends. This can be done by writing
> > our own 'write' procedure. Downside: it might be less efficient than
> > Guile's write which is implemented in C. Can be resolved by writing our
> > own 'write' procedure in C.
> [...]

I'll try to look into other parts of your response later.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
> I think the behavior of `gexp->approximate-sexp` is at least defensible, 
> since its documentation (`info guix "gexp->approximate-sexp"`) warns 
> that "some information can be lost".

But no information is lost in this case?

$ guix repl
> (use-modules (guix gexp))
> (gexp->approximate-sexp #~(a . #nil))
$1 = (a) ; sure, the #nil isn't printed ...
> (cdr $1)
$2 = () ; ... also not printed ...
> (values (eq? (cdr $1) #nil) (eq? (cdr $1) '()))
$3 = #t ; but it's still there!
$4 = #f

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3054 bytes --]

Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
> I haven't looked at the implementation at all, but extending `write` 
> certainly would be a reasonable option, and, longer-term, it might be 
> possible to upstream a patch adding the needed behavior.

Not sure what the API should be (an optional argument #:preserve-nil?
#true? A parameter nil-writing-style? A procedure 'write-preserving-
nil'?), but sure.

> A more radical option could be to use a format other than plain-text 
> s-expressions for compiled G-expressions. For example, Racket has a 
> forward-compatible "fast-load serialization" binary format for the kinds 
> of values that can be embedded in compiled code.[0] There are obvious 
> disadvantages to a binary format, but advantages include the ability to 
> preserve source-location information and to avoid some the quirks that 
> come with functions like `write` and `read`, for historical reasons or 
> for the convenience of humans writing code directly. The implementation 
> is in Racket, so it should be fairly easy to port to Guile, if that were 
> wanted.[1]

The primary purpose of that data format appears to be able to load
S-expressions _fast_, which seems less useful in Guix because every
derivations are only built once (ignoring GC and build failures).
More important in Guix, is being able to _write_ G-exps fast.
Though perhaps fasl can be written fast?

Anyway, this fasl format might be one way to represent preserve
information. jpoiret on #guix was interested in preserving position
information, see <https://logs.guix.gnu.org/guix/2021-11-19.log>.

>  Or maybe there's something related to Guile bytecode that 
> would work, or maybe just making a `#nil`-preserving version of `write` 
> would be easier.

The G-exp machinery doesn't compile things to bytecode, because the
bytecode format changes between Guile versions and for bootstrapping,
old Guiles are used (from the 2.0 series IIRC). Also, this can lead
to world-rebuilds whenever an optimisation in guile is added or
tweaked.

Anyway, I think that in the short term, it would be easiest to
modify (guix build json) to not use #nil (though that would lead
to rebuilding all rust and node stuff because it is used in cargo-
build-system). Longer term, maybe '(guix build json)' could be
eliminated and we could use (json) from 'guile-json' instead,
like documented in the manual ((guix)G-Expressions):

   ‘In the same vein, sometimes you want to import not just pure-Scheme
modules, but also “extensions” such as Guile bindings to C libraries or
other “full-blown” packages.  Say you need the ‘guile-json’ package
available on the build side, here’s how you would do it:

     (use-modules (gnu packages guile))  ;for 'guile-json'

     (with-extensions (list guile-json)
       (gexp->derivation "something-with-json"
                         #~(begin
                             (use-modules (json))
                             ...)))’

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#52749: G-expressions don't consistently preserve #nil

[Maxime Devos]

Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
> G-expressions currently do not consistently preserve the distinction 
> between #nil and '(), which causes trouble for programs that rely on 
> that distinction. In particular, the issue affects programs that use 
> (guix build json), because that library uses #nil to represent the JSON 
> value `null', whereas it uses '() to represent an empty JSON array.
> 
> The following program exposes the error:
> [
> ;...]
> 
> ; This one fails!
> (check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
>                "{\"k\":null}"
>                "gexp: null in object")

A simpler test:

Compare this:
  (cdr (gexp->approximate-sexp #~("stuff" . #nil)))
  ; output: #nil --- seems like everything is ok?

with:
  (gexp->approximate-sexp #~("stuff" . #nil))
  ; output: ("stuff") --- where did the #nil go?

I think the idea is that, if you construct a list (a b c . #nil)
in elisp, and pass it to Scheme, then Scheme should treat it as a
Scheme list, so it should be printed as (a b c) when using Scheme's
'write' or 'display'.

Greetings,
Maxime.