* bug#52749: G-expressions don't consistently preserve #nil @ 2021-12-23 4:25 Philip McGrath 2021-12-23 6:59 ` Maxime Devos 2021-12-23 17:58 ` Maxime Devos 0 siblings, 2 replies; 8+ messages in thread From: Philip McGrath @ 2021-12-23 4:25 UTC (permalink / raw) To: 52749 G-expressions currently do not consistently preserve the distinction between #nil and '(), which causes trouble for programs that rely on that distinction. In particular, the issue affects programs that use (guix build json), because that library uses #nil to represent the JSON value `null', whereas it uses '() to represent an empty JSON array. The following program exposes the error: --8<---------------cut here---------------start------------->8--- (use-modules (guix build json) (guix gexp) (guix monads) (guix store) (guix derivations) (rnrs conditions) (ice-9 textual-ports)) (define (check-equal? actual expected message) (define who 'check-equal?) (unless (equal? actual expected) (raise-exception (condition (make-assertion-violation) (make-who-condition 'check-equal?) (make-irritants-condition (list actual)) (make-message-condition (format #f "~a: ~a\n ~a: ~s\n ~a: ~s\n ~a: ~s" who "test failed" "message" message "expected" expected "actual" actual)))))) (define (sexp->json-string sx) (call-with-output-string (lambda (out) (write-json sx out)))) (define (gexp->json-string gx) (run-with-store (open-connection) (mlet* %store-monad ((drv (gexp->derivation "example.json" (with-imported-modules `((guix build json)) #~(begin (use-modules (guix build json)) (call-with-output-file #$output (lambda (out) (write-json #$gx out))))))) (_built (built-derivations (list drv)))) (return (call-with-input-file (derivation->output-path drv) get-string-all))))) (check-equal? (sexp->json-string '()) "[]" "sexp: empty array") (check-equal? (gexp->json-string #~'()) "[]" "gexp: empty array") (check-equal? (sexp->json-string #nil) "null" "sexp: null") (check-equal? (gexp->json-string #~#nil) "null" "gexp: null") (check-equal? (sexp->json-string '(@ ("k" . #nil))) "{\"k\":null}" "sexp: null in object") ;; This one fails! (check-equal? (gexp->json-string #~'(@ ("k" . #nil))) "{\"k\":null}" "gexp: null in object") --8<---------------cut here---------------end--------------->8--- -Philip ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-23 4:25 bug#52749: G-expressions don't consistently preserve #nil Philip McGrath @ 2021-12-23 6:59 ` Maxime Devos 2021-12-25 11:13 ` Maxime Devos 2021-12-23 17:58 ` Maxime Devos 1 sibling, 1 reply; 8+ messages in thread From: Maxime Devos @ 2021-12-23 6:59 UTC (permalink / raw) To: Philip McGrath, 52749 Hi, Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: > G-expressions currently do not consistently preserve the distinction > between #nil and '(), which causes trouble for programs that rely on > that distinction. In particular, the issue affects programs that use > (guix build json), because that library uses #nil to represent the JSON > value `null', whereas it uses '() to represent an empty JSON array. The constant #nil is only for elisp compatibility and not something supposed to be used in Scheme code that isn't for Scheme/elisp compatibility, so this seems more a bug in (guix build json) to me. Greetings, Maxime. ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-23 6:59 ` Maxime Devos @ 2021-12-25 11:13 ` Maxime Devos 2021-12-27 18:38 ` Philip McGrath 0 siblings, 1 reply; 8+ messages in thread From: Maxime Devos @ 2021-12-25 11:13 UTC (permalink / raw) To: Philip McGrath, 52749 [-- Attachment #1: Type: text/plain, Size: 1045 bytes --] Maxime Devos schreef op do 23-12-2021 om 06:59 [+0000]: > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: > > G-expressions currently do not consistently preserve the distinction > > between #nil and '(), which causes trouble for programs that rely on > > that distinction. In particular, the issue affects programs that use > > (guix build json), because that library uses #nil to represent the JSON > > value `null', whereas it uses '() to represent an empty JSON array. > > The constant #nil is only for elisp compatibility and not something > supposed to be used in Scheme code that isn't for Scheme/elisp > compatibility, so this seems more a bug in (guix build json) to me. That said, it would be less surprising if the #nil/() distinction is preserved by gexp->derivation and friends. This can be done by writing our own 'write' procedure. Downside: it might be less efficient than Guile's write which is implemented in C. Can be resolved by writing our own 'write' procedure in C. Greetings, Maxime. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 260 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-25 11:13 ` Maxime Devos @ 2021-12-27 18:38 ` Philip McGrath 2021-12-27 20:24 ` Maxime Devos ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: Philip McGrath @ 2021-12-27 18:38 UTC (permalink / raw) To: Maxime Devos, 52749 Hi! Just as a general disclaimer, I'm a Racketeer and only incidentally a Schemer, so I'm not very familiar with the universe of Guile libraries. On 12/23/21 01:59, Maxime Devos wrote: > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: >> G-expressions currently do not consistently preserve the distinction >> between #nil and '(), which causes trouble for programs that rely on >> that distinction. In particular, the issue affects programs that use >> (guix build json), because that library uses #nil to represent the JSON >> value `null', whereas it uses '() to represent an empty JSON array. > > The constant #nil is only for elisp compatibility and not something > supposed to be used in Scheme code that isn't for Scheme/elisp > compatibility, so this seems more a bug in (guix build json) to me. That was not the impression I had gotten from `info "(guile)Nil"`. For example, I think someone who wanted to finish the implementation described in `info "(guile)ECMAScript"` might want to use #nil for one of the false-y ECMAScript values to take advantages of the documented efficiencies in its bit-level representation. More concretely, guile-json@1 and guile-json@3 use #nil in the same way as (guix build json). On 12/23/21 12:58, Maxime Devos wrote: > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: >> G-expressions currently do not consistently preserve the distinction >> between #nil and '(), which causes trouble for programs that rely on >> that distinction. In particular, the issue affects programs that use >> (guix build json), because that library uses #nil to represent the JSON >> value `null', whereas it uses '() to represent an empty JSON array. >> >> The following program exposes the error: >> [ >> ;...] >> >> ; This one fails! >> (check-equal? (gexp->json-string #~'(@ ("k" . #nil))) >> "{\"k\":null}" >> "gexp: null in object") > > A simpler test: > > Compare this: > (cdr (gexp->approximate-sexp #~("stuff" . #nil))) > ; output: #nil --- seems like everything is ok? > > with: > (gexp->approximate-sexp #~("stuff" . #nil)) > ; output: ("stuff") --- where did the #nil go? > > I think the idea is that, if you construct a list (a b c . #nil) > in elisp, and pass it to Scheme, then Scheme should treat it as a > Scheme list, so it should be printed as (a b c) when using Scheme's > 'write' or 'display'. Since `write` and `list?` are specified by various Scheme standards, I think it is the correct choice for `write` to use a Scheme-compatible external representation for values recognized by `list?`, at least by default. (Perhaps a parameter could control this behavior?) I think the behavior of `gexp->approximate-sexp` is at least defensible, since its documentation (`info guix "gexp->approximate-sexp"`) warns that "some information can be lost". But I think the implementation of G-expressions faces more stringent obligations. I see it as analogous to the implementation of syntax objects, a macro expander, or a compiler, in that it should have a semantics-preserving representation of arbitrary Guile code, including Guile's extensions to Scheme. (I haven't yet understood at a theoretical level how "strata" and "staging" relate to the more familiar concept of "phases", but my intuition is that, while the R6RS model of phases wouldn't be enough, it seems like would probably to express staging/strata in terms of phases with Racket enhancements like the label phase level and arbitrary submodule-implemented phases.) So, I agree that: On 12/25/21 06:13, Maxime Devos wrote: > That said, it would be less surprising if the #nil/() distinction is > preserved by gexp->derivation and friends. This can be done by writing > our own 'write' procedure. Downside: it might be less efficient than > Guile's write which is implemented in C. Can be resolved by writing our > own 'write' procedure in C. I haven't looked at the implementation at all, but extending `write` certainly would be a reasonable option, and, longer-term, it might be possible to upstream a patch adding the needed behavior. A more radical option could be to use a format other than plain-text s-expressions for compiled G-expressions. For example, Racket has a forward-compatible "fast-load serialization" binary format for the kinds of values that can be embedded in compiled code.[0] There are obvious disadvantages to a binary format, but advantages include the ability to preserve source-location information and to avoid some the quirks that come with functions like `write` and `read`, for historical reasons or for the convenience of humans writing code directly. The implementation is in Racket, so it should be fairly easy to port to Guile, if that were wanted.[1] Or maybe there's something related to Guile bytecode that would work, or maybe just making a `#nil`-preserving version of `write` would be easier. -Philip [0]: https://docs.racket-lang.org/reference/fasl.html [1]: https://github.com/racket/racket/blob/master/racket/collects/racket/fasl.rkt ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-27 18:38 ` Philip McGrath @ 2021-12-27 20:24 ` Maxime Devos 2022-01-03 10:28 ` Maxime Devos 2022-01-03 10:49 ` Maxime Devos 2 siblings, 0 replies; 8+ messages in thread From: Maxime Devos @ 2021-12-27 20:24 UTC (permalink / raw) To: Philip McGrath, 52749 [-- Attachment #1: Type: text/plain, Size: 2695 bytes --] Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]: > Hi! > > Just as a general disclaimer, I'm a Racketeer and only incidentally a > Schemer, so I'm not very familiar with the universe of Guile libraries. > > On 12/23/21 01:59, Maxime Devos wrote: > > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: > >> G-expressions currently do not consistently preserve the distinction > >> between #nil and '(), which causes trouble for programs that rely on > >> that distinction. In particular, the issue affects programs that use > >> (guix build json), because that library uses #nil to represent the JSON > >> value `null', whereas it uses '() to represent an empty JSON array. > > > > The constant #nil is only for elisp compatibility and not something > > supposed to be used in Scheme code that isn't for Scheme/elisp > > compatibility, so this seems more a bug in (guix build json) to me. > > That was not the impression I had gotten from `info "(guile)Nil"`. For > example, I think someone who wanted to finish the implementation > described in `info "(guile)ECMAScript"` might want to use #nil for one > of the false-y ECMAScript values to take advantages of the documented > efficiencies in its bit-level representation. More concretely, > guile-json@1 and guile-json@3 use #nil in the same way as (guix build json). There is ‘Guile has chosen to support ‘nil’ as a separate value, distinct from ‘#f’ and ‘'()’. This allows existing Scheme and Elisp code to maintain their current semantics. ‘nil’, which in Elisp would just be written and read as ‘nil’, in Scheme has the external representation ‘#nil’.’ and ‘This decision to have ‘nil’ as a low-level distinct value facilitates interoperability between the two languages. Guile has chosen to have Scheme deal with ‘nil’ as follows: [...]’ and this is only documented under ‘Emacs Lisp’, though this doesn't explicitely say it's not supposed to be used elsewhere. Also, see e.g. <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00017.html>. Anyway, this doesn't really matter here, because: > So, I agree that: > > On 12/25/21 06:13, Maxime Devos wrote: > > That said, it would be less surprising if the #nil/() distinction is > > preserved by gexp->derivation and friends. This can be done by writing > > our own 'write' procedure. Downside: it might be less efficient than > > Guile's write which is implemented in C. Can be resolved by writing our > > own 'write' procedure in C. > [...] I'll try to look into other parts of your response later. Greetings, Maxime. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 260 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-27 18:38 ` Philip McGrath 2021-12-27 20:24 ` Maxime Devos @ 2022-01-03 10:28 ` Maxime Devos 2022-01-03 10:49 ` Maxime Devos 2 siblings, 0 replies; 8+ messages in thread From: Maxime Devos @ 2022-01-03 10:28 UTC (permalink / raw) To: Philip McGrath, 52749 [-- Attachment #1: Type: text/plain, Size: 586 bytes --] Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]: > I think the behavior of `gexp->approximate-sexp` is at least defensible, > since its documentation (`info guix "gexp->approximate-sexp"`) warns > that "some information can be lost". But no information is lost in this case? $ guix repl > (use-modules (guix gexp)) > (gexp->approximate-sexp #~(a . #nil)) $1 = (a) ; sure, the #nil isn't printed ... > (cdr $1) $2 = () ; ... also not printed ... > (values (eq? (cdr $1) #nil) (eq? (cdr $1) '())) $3 = #t ; but it's still there! $4 = #f Greetings, Maxime [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 260 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-27 18:38 ` Philip McGrath 2021-12-27 20:24 ` Maxime Devos 2022-01-03 10:28 ` Maxime Devos @ 2022-01-03 10:49 ` Maxime Devos 2 siblings, 0 replies; 8+ messages in thread From: Maxime Devos @ 2022-01-03 10:49 UTC (permalink / raw) To: Philip McGrath, 52749 [-- Attachment #1: Type: text/plain, Size: 3054 bytes --] Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]: > I haven't looked at the implementation at all, but extending `write` > certainly would be a reasonable option, and, longer-term, it might be > possible to upstream a patch adding the needed behavior. Not sure what the API should be (an optional argument #:preserve-nil? #true? A parameter nil-writing-style? A procedure 'write-preserving- nil'?), but sure. > A more radical option could be to use a format other than plain-text > s-expressions for compiled G-expressions. For example, Racket has a > forward-compatible "fast-load serialization" binary format for the kinds > of values that can be embedded in compiled code.[0] There are obvious > disadvantages to a binary format, but advantages include the ability to > preserve source-location information and to avoid some the quirks that > come with functions like `write` and `read`, for historical reasons or > for the convenience of humans writing code directly. The implementation > is in Racket, so it should be fairly easy to port to Guile, if that were > wanted.[1] The primary purpose of that data format appears to be able to load S-expressions _fast_, which seems less useful in Guix because every derivations are only built once (ignoring GC and build failures). More important in Guix, is being able to _write_ G-exps fast. Though perhaps fasl can be written fast? Anyway, this fasl format might be one way to represent preserve information. jpoiret on #guix was interested in preserving position information, see <https://logs.guix.gnu.org/guix/2021-11-19.log>. > Or maybe there's something related to Guile bytecode that > would work, or maybe just making a `#nil`-preserving version of `write` > would be easier. The G-exp machinery doesn't compile things to bytecode, because the bytecode format changes between Guile versions and for bootstrapping, old Guiles are used (from the 2.0 series IIRC). Also, this can lead to world-rebuilds whenever an optimisation in guile is added or tweaked. Anyway, I think that in the short term, it would be easiest to modify (guix build json) to not use #nil (though that would lead to rebuilding all rust and node stuff because it is used in cargo- build-system). Longer term, maybe '(guix build json)' could be eliminated and we could use (json) from 'guile-json' instead, like documented in the manual ((guix)G-Expressions): ‘In the same vein, sometimes you want to import not just pure-Scheme modules, but also “extensions” such as Guile bindings to C libraries or other “full-blown” packages. Say you need the ‘guile-json’ package available on the build side, here’s how you would do it: (use-modules (gnu packages guile)) ;for 'guile-json' (with-extensions (list guile-json) (gexp->derivation "something-with-json" #~(begin (use-modules (json)) ...)))’ Greetings, Maxime. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 260 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#52749: G-expressions don't consistently preserve #nil 2021-12-23 4:25 bug#52749: G-expressions don't consistently preserve #nil Philip McGrath 2021-12-23 6:59 ` Maxime Devos @ 2021-12-23 17:58 ` Maxime Devos 1 sibling, 0 replies; 8+ messages in thread From: Maxime Devos @ 2021-12-23 17:58 UTC (permalink / raw) To: Philip McGrath, 52749 Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]: > G-expressions currently do not consistently preserve the distinction > between #nil and '(), which causes trouble for programs that rely on > that distinction. In particular, the issue affects programs that use > (guix build json), because that library uses #nil to represent the JSON > value `null', whereas it uses '() to represent an empty JSON array. > > The following program exposes the error: > [ > ;...] > > ; This one fails! > (check-equal? (gexp->json-string #~'(@ ("k" . #nil))) > "{\"k\":null}" > "gexp: null in object") A simpler test: Compare this: (cdr (gexp->approximate-sexp #~("stuff" . #nil))) ; output: #nil --- seems like everything is ok? with: (gexp->approximate-sexp #~("stuff" . #nil)) ; output: ("stuff") --- where did the #nil go? I think the idea is that, if you construct a list (a b c . #nil) in elisp, and pass it to Scheme, then Scheme should treat it as a Scheme list, so it should be printed as (a b c) when using Scheme's 'write' or 'display'. Greetings, Maxime. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-01-03 10:51 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-12-23 4:25 bug#52749: G-expressions don't consistently preserve #nil Philip McGrath 2021-12-23 6:59 ` Maxime Devos 2021-12-25 11:13 ` Maxime Devos 2021-12-27 18:38 ` Philip McGrath 2021-12-27 20:24 ` Maxime Devos 2022-01-03 10:28 ` Maxime Devos 2022-01-03 10:49 ` Maxime Devos 2021-12-23 17:58 ` Maxime Devos
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).