Removing program-arities export

Removing program-arities export

[Skyler via Developers list for Guile, the GNU extensibility library]

Hello,

I am working on a patch for a bug I recently found in guile. The `program-arities` function is exported from the `(system vm program)` module and documented in the manual, but it is no longer defined (it was removed in commit 1c33be992e8120abd20add8021e4d91d226f5b6a).  It is only used in the `program-arity function`, which is not called anywhere in the Guile repository (and calling it with valid inputs results in an error due to the `program-arities` being undefined). See the shell session at the end of this email for details.

Removing those functions is simple, but the `arity:` accessors have more users, and since they're just pattern-matching on a list I'm concerned that they have been used for the new functions (eg, find-program-arity) so I'm not sure if it's safe to just remove them. In particular, `arity->arguments-alist` uses them unconditionally. However, `arity->arguments-alist` only has two callers, `program-arguments-alist` and `program-arguments-alists` (that's not a typo, the second one is pluralized), which call it conditionally. The conditions are different but both require that the given code passes the `primitive-code?` predicate. These functions are called from a few different places so it'll take a little more time to figure out if it's safe to remove them (or possibly just remove the logic conditioned by `primitive-code?`, if that is no longer in use, and leaving the rest of the function intact).

In solidarity,
Skyler

user@foreign-guix /d/shm> cat channels.scm
%default-channels

user@foreign-guix /d/shm> guix time-machine --channels=./channels.scm -- shell --container --nesting guile-next guile-readline
user@foreign-guix /dev/shm [env]$ guix describe
  guix 4a4a8c9
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 4a4a8c9d2621d4871c4b9c8857bfb0e6e53d7e56

user@foreign-guix /dev/shm [env]$ guile -q
GNU Guile 3.0.9-0.3b76a30
Copyright (C) 1995-2024 Free Software Foundation, Inc.
                                                                                                              
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
                                                                                   
Enter `,help' for help.
scheme@(guile-user)> (module-variable (resolve-module '(system vm program)) 'program-arities)
$1 = #<variable 734b7c40e940 value: #<undefined>>

scheme@(guile-user)> (use-modules (system vm program))
scheme@(guile-user)> (program-arity car #f)
ice-9/boot-9.scm:1676:22: In procedure raise-exception:
Unbound variable: program-arities

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.

RE: Removing program-arities export

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

>It is only used in the `program-arity function`, which is not called anywhere in the Guile repository (and calling it with valid inputs results in an error due to the `program-arities` being undefined).

This is untrue, it (i.e. program-arities) is also used in the manual.

>Removing those functions is simple, but the `arity:` accessors have more users, […]

It shouldn’t be removed, rather a replacement should be provided somewhere.
Regardless of the changes to the VM, compiled procedures still exist, and by definition that’s what a ‘program’ is, so ‘programs’ still exist, and hence many of the procedures in that module still make sense, including  program-arities. In particular, arity information is still recorded (in .guile.arities), so it should be made / kept accessible somewhere.

Best regards,
Maxime Devos


[-- Attachment #2: Type: text/html, Size: 2333 bytes --]

RE: Removing program-arities export

[Skyler via Developers list for Guile, the GNU extensibility library]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On Sunday, January 5th, 2025 at 6:07 PM, Maxime Devos <maximedevos@telenet.be> wrote:

>>It is only used in the `program-arity function`, which is not called anywhere in the Guile repository (and calling it with valid inputs results in an error due to the `program-arities` being undefined).
>
> This is untrue, it (i.e. program-arities) is also used in the manual.

Fair point.

>>Removing those functions is simple, but the `arity:` accessors have more users, […]
>
> It shouldn’t be removed, rather a replacement should be provided somewhere.
>
> Regardless of the changes to the VM, compiled procedures still exist, and by definition that’s what a ‘program’ is, so ‘programs’ still exist, and hence many of the procedures in that module still make sense, including program-arities. In particular, arity information is still recorded (in .guile.arities), so it should be made / kept accessible somewhere.

I agree that this information should be available. I should have explicated my assumption that the similarly-named function "find-program-arities" was a suitable replacement. It is defined in "system/vm/debug.scm" and references .guile.arities. I will interrogate this assumption more thoroughly before submitting a patch, the email was meant as an early notification that this is something that I'm starting to look at in case someone else is already looking at it. I am fine with changing the direction of the patch if that turns out to make sense (perhaps by turning program-arities an alias for find-program-arities).

>

[-- Attachment #2: Type: text/html, Size: 2938 bytes --]

RE: Removing program-arities export

[Skyler Ferris via Developers list for Guile, the GNU extensibility library]

(switching to the correct sending address; I was previously sending from skyvine due to a misconfiguration in my client)

It looks like there are 2 potential candidates for aliasing, but both of them have some problems. When aliasing to "find-program-arities" I am unable to get the arity of any procedure which passes the "primitive-code?" predicate; it just returns #f. When aliasing to "program-arguments-alists" I get results for primitive code, but they are not accurate. For example, try calling it on the "count" procedure from SRFI 1. It is using the "fallback" helper which hardcodes unknown values for missing information. Also, the alists it returns do not provide the information needed for "arity:start" and "arity:end".

I believe that program-arguments-alists is the intended replacement for program-arities, based on this timeline of relevant commits:

- eb2bc00fb3863986927f0bade97487209b6d6a5b (16 May 2013): Added program-arguments-alist, commit message states "New helper, implemented also for RTL procedures."
- 27337b6373954e1a975d97d0bf06b5c03d65b64d (18 October 2013): Added fallback logic, commit message states "Fix up various arity-related things for primitives, which don't use ELF arity info."
- 8bd261baaa96eba005517eef5fb8d5d08f22720a (also 18 October 2013): Exported program-arguments-alist, commit message states "Export this interface.  Fall back to grovelling through procedure-minimum-arity if the program has no arities, as might be the case for continuations."
- (1c33be992e8120abd20add8021e4d91d226f5b6a, 8 November 2013): Removed program-arities implementation, commit message states "Remove old program code." The header of the commit states "Remove stack programs, objcode, and the old VM."

I have 2 concerns that I want to address. The first concern is the bad user experience created by calling a procedure which is advertised in the manual only to be told that it does not exist. I consider this to be an urgent problem and would like to find a path to fix this quickly. Based on the above I still believe that removing program-arity (and updating the manual accordingly) makes sense. However, if we want to keep this as a separate interface I could do something like throw an error that explicitly states "this is currently unimplemented, we are aware of the problem and are looking at it." I don't have a strong opinion about keeping it or removing it. I just want to make sure that callers have a good experience with whatever APIs exist.

The second concern is ensuring that arity information is available for all procedures. However, I would like to delay that discussion until after I spend more time studying how different programming languages handle arguments.

Re: yeRemoving program-arities export

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

Skyler Ferris via "Developers list for Guile, the GNU extensibility library" <guile-devel@gnu.org> writes:

> I have 2 concerns that I want to address. The first concern is the bad
> user experience created by calling a procedure which is advertised in
> the manual only to be told that it does not exist. I consider this to
> be an urgent problem and would like to find a path to fix this
> quickly. Based on the above I still believe that removing
> program-arity (and updating the manual accordingly) makes sense.

Given that the program-arity / program-arities is documented, the more
pressing problem is that programs using it would be broken right now.
People may be holding off from updating Guile because they don’t have
the time to investigate breakage in programs that are working perfectly
fine in the old version. Which is the typical case for useful tools.

The worst user experience is seeing a working program break when
updating a dependency: if the docs are wrong, you’re still in the
context of your tool, but if it breaks on update, all that context has
been flushed out already — often years ago.

Is it viable to wrap the procedure and exactly preserve the API, so
existing programs keep working? If it’s a bit slower, that should not
cause pain (because hardware and Guile got faster in the meantime) but
it should not break.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: yeRemoving program-arities export

[Skyler Ferris via Developers list for Guile, the GNU extensibility library]

> Given that the program-arity / program-arities is documented, the more
> pressing problem is that programs using it would be broken right now.
> People may be holding off from updating Guile because they don’t have
> the time to investigate breakage in programs that are working perfectly
> fine in the old version. Which is the typical case for useful tools.
> 
> The worst user experience is seeing a working program break when
> updating a dependency: if the docs are wrong, you’re still in the
> context of your tool, but if it breaks on update, all that context has
> been flushed out already — often years ago.

Fair point. A lot of projects transition APIs by providing deprecation notices, but I am not aware of Guile doing that for this API. This is a strong argument against removing it immediately.
 
> Is it viable to wrap the procedure and exactly preserve the API, so
> existing programs keep working? If it’s a bit slower, that should not
> cause pain (because hardware and Guile got faster in the meantime) but
> it should not break.

I believe this is possible so long as these programs do not need to access arity information from procedures which pass the "primitive-code?" predicate. I am not sure what this predicate is actually testing. The implementation of "is_primitive_code" in "libguile/programs.c" is looking at a list of "code_arena" structs which I am not familiar with. The implementation of "find-program-arities" in "module/system/vm/debug.scm" is searching through loaded ELF images to find the one which contains the given program (and always fails to find one when the program passes "primitive-code?"). Based on the commit message in 27337b6373954e1a975d97d0bf06b5c03d65b64d, it seems like this behavior is expected. What puzzles me the most is that functions defined at the scheme level in the same file are not consistent in whether or not they are primitive. For example, "count" in SRFI 1 is primitive but "unzip1" is not. My best guess is that it has something to do with JIT compiled functions, but that truly is a guess.

Also, I have not looked at the output of "arity:start" and "arity:end". I'm not sure how I would go about testing these, but it looks like the struct returned from "find-program-arities" contains the same information. This procedure also has an optional "context" argument; I have only tried using the default value.

This is the code I'm currently using to try things out. The final line produces a lot of output, you can just comment it out for a quieter script.

(use-modules (system vm debug) (system vm program))

(define program-module (resolve-module '(system vm program)))

(module-set! program-module 'program-arities
  (compose find-program-arities program-code))

(module-set! program-module 'arity:start             arity-low-pc)
(module-set! program-module 'arity:end               arity-high-pc)
(module-set! program-module 'arity:nreq              arity-nreq)
(module-set! program-module 'arity:nopt              arity-nopt)
(module-set! program-module 'arity:rest?             arity-has-rest?)
(module-set! program-module 'arity:kw                arity-keyword-args)
(module-set! program-module 'arity:allow-other-keys? arity-allow-other-keys?)

(define* (uut req #:optional opt0 opt1
                  #:key key0 key1 key2
                  #:allow-other-keys
                  #:rest rest)
  (cons* req opt0 opt1 key0 key1 key2 rest))

(define arities (program-arities uut))
(define arity   (program-arity uut #f))

(format #t "Number of arities:  ~s~%" (length                  arities))
(format #t "Required arguments: ~s~%" (arity:nreq              arity))
(format #t "Optional arguments: ~s~%" (arity:nopt              arity))
(format #t "Rest argmuent?      ~s~%" (arity:rest?             arity))
(format #t "Keyword arguments:  ~s~%" (arity:kw                arity))
(format #t "Other keys?         ~s~%" (arity:allow-other-keys? arity))
(newline)

(define uut
  (case-lambda
    ((single) single)
    ((du o) (list du o))
    ((tr i o) (list tr i o))))

(format #t "Arities from a case-lambda: ~s~%" (length (program-arities uut)))
(newline)

(define (generate-arity-trier proc)
  (lambda (sym var)
    (let ((val (variable-ref var)))
      (when (procedure? val)
        (format #t "Arities of ~s: ~s~a~%"
                sym
                (and=> (proc val) length)
                (if (primitive-code? (program-code val)) " (primitive)" ""))))))

; use any of the below procedures in module-for-each line at the end
(define try-program-arity (generate-arity-trier program-arities))
(define try-program-arguments-alist (generate-arity-trier program-arguments-alist))

(define (check-for-inconsistency sym var)
  (let ((val (variable-ref var)))
    (when (and (procedure?      val)
               (program-arities val)
               (primitive-code? (program-code val)))
      (format #t "~s is inconsistent!~%" sym))))

(module-for-each try-program-arity (resolve-interface '(srfi srfi-1)))

Re: yeRemoving program-arities export

[Skyler Ferris via Developers list for Guile, the GNU extensibility library]

Hi again,

Can I clarify whether you raised this point as a concern in principle or if you use/maintain/are aware of a project that is currently unable to upgrade due to this problem? In the latter case, I would appreciate if you could answer the following questions (and/or pass them along to others that they might be relevant to):

1. Is it important that you are able to use this API (program-arity, program-arities) on any arbitrary function or only a specific subset of functions?

2. Do use use the arity:start and arity:end accessors? If so, could you describe what this information is being used for?

Thanks,
Skyler (they/them)

RE: Removing program-arities export

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4582 bytes --]

> It looks like there are 2 potential candidates for aliasing, but both of them have some problems. When aliasing to "find-program-arities" I am unable to get the arity of any procedure which passes the "primitive-code?" predicate; it just returns #f. When aliasing to "program-arguments-alists" I get results for primitive code, but they are not accurate. For example, try calling it on the "count" procedure from SRFI 1. It is using the "fallback" helper which hardcodes unknown values for missing information. Also, the alists it returns do not provide the information needed for "arity:start" and "arity:end".

There is no such thing as ‘program-arguments-alists’ AFAICT, there is only ‘program-arguments-alist’ (singular instead of plural is key here)

You can’t implement it in terms of ‘find-program-arguments-alist’, because that procedure output format can only accomodate single-arity procedures, whereas program-arities returns all the arities. However, ‘find-program-arities’ looks fine (it actually wouldn’t work for general procedures since ‘find-program-arities’ has “FIXME: handle case-lambda arities”, but that comment makes clear it is _supposed_ for those).

‘count’ from SRFI 1 is not primitive code. Maybe there is also another implementation in (guile), but ‘count’ from the SRFI 1 implementation is just a Scheme procedure defined in Scheme by ‘define’. I don’t know what Guile does _currently_, but is used to be that the information _is_ available: type (display f) for any procedure f that is a (compiled) case-lambda with multiple branches (e.g. count), and you would see all the arities. That was in Guile 3.0, which postdates the mentioned commits.

I’m not sure, but I think primitive procedures aren’t technically programs (and if they were technically programs, they probably wouldn’t have an ELF section). I don’t know if the arity information is recorded anywhere, but at least during the definition of the primitive procedure the arity information is passed to the primitive procedure-constructing C function (look for gsubr). Perhaps there is a hint there on where arity information of primitive procedures is stored (if anywhere). It might also be the case that even before those commits, arity information of such procedures weren’t available at runtime (can test with (display insert-primitive-procedure-here)).

>I believe that program-arguments-alists is the intended replacement for program-arities, based on this timeline of relevant commits: […]

There is no such thing as program-arguments-alists, as far as I can tell, though if it existed that seems a good replacement. And program-arguments-alist (singular) is not a replacement, since it only returns a _single_ arity.

Based on that timeline, I think a mistake was made – whether by accidentally removing program-arities as well, or by intentionally choosing to remove program-arities (without providing a replacement).

>The second concern is ensuring that arity information is available for all procedures. However, I would like to delay that discussion until after I spend more time studying how different programming languages handle arguments.

The interpreter probably constructs procedures by (lambda args [first deconstruct args, then do relevant actions]) closures (*), so it probably looks like a compiled procedure with a rest argument to the current implementation. (*) at least, that’s how I would implement a Scheme interpreter in Scheme.

You could probably modify the interpreter to record arity information as a ‘procedure property’, and then modify the arity querying procedures to look for this information when available, and prefer it above. You would need to investigate the performance impact (especially the performance impact on the bootstrapping process (i.e. compile without any precompiled .go)) (Inconveniently there doesn’t seem to be a standard way of testing for such.)

If there is some relevant performance impact, I have a rather ugly method for eliminating it during bootstrapping (at cost of not having proper arity information for interpreted things during bootstrapping) if you are interested.

(Also, technically there are multiple Scheme interpreters in Guile for technical reasons – a Scheme interpreter written in Scheme (“meta-circular evaluator’), and one written in C. The C implementation exists for bootstrapping the Scheme Scheme interpreter, so it’s of less interest to get arity information in there.)

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 7240 bytes --]

RE: Removing program-arities export

[Skyler Ferris via Developers list for Guile, the GNU extensibility library]

(note: all REPL sessions were performed using the environment described at the end of this email)

> There is no such thing as ‘program-arguments-alists’ AFAICT, there is only ‘program-arguments-alist’ (singular instead of plural is key here)
>
> You can’t implement it in terms of ‘find-program-arguments-alist’, because that procedure output format can only accomodate single-arity procedures, whereas program-arities returns all the arities. However, ‘find-program-arities’ looks fine (it actually wouldn’t work for general procedures since ‘find-program-arities’ has “FIXME: handle case-lambda arities”, but that comment makes clear it is _supposed_ for those).

I think we might be looking at different versions of the repository. I'm on commit 8ac25f3faeb92c31c94f2bc09f4b29780df383fe, which is the most recent commit on the branch "main" in the repository located at https://git.sv.gnu.org/git/guile.git. In this version of the code, both program-arguments-alist and program-arguments-alists are defined (and exported) in modules/system/program.scm. Additionally, find-program-arities works for case-lambda arities (at least, the comment is no longer there and the number of returned entries is correct).

Although, I am noticing that the alist functions don't include the information provided by alist:start and alist:end (the start and end addresses of the procedure definition in the relevant ELF image).

scheme@(guile-user)> (pretty-print (program-arguments-alists map))
(((required f l) (optional) (keyword) (allow-other-keys? . #f) (rest . #f))
 ((required f l1 l2) (optional) (keyword) (allow-other-keys? . #f) (rest . #f))
 ((required f l1) (optional) (keyword) (allow-other-keys? . #f) (rest . rest)))
scheme@(guile-user)> (length (find-program-arities (program-code map)))
$4 = 3

> ‘count’ from SRFI 1 is not primitive code. Maybe there is also another implementation in (guile), but ‘count’ from the SRFI 1 implementation is just a Scheme procedure defined in Scheme by ‘define’. I don’t know what Guile does _currently_, but is used to be that the information _is_ available: type (display f) for any procedure f that is a (compiled) case-lambda with multiple branches (e.g. count), and you would see all the arities. That was in Guile 3.0, which postdates the mentioned commits.
>
> I’m not sure, but I think primitive procedures aren’t technically programs (and if they were technically programs, they probably wouldn’t have an ELF section). I don’t know if the arity information is recorded anywhere, but at least during the definition of the primitive procedure the arity information is passed to the primitive procedure-constructing C function (look for gsubr). Perhaps there is a hint there on where arity information of primitive procedures is stored (if anywhere). It might also be the case that even before those commits, arity information of such procedures weren’t available at runtime (can test with (display insert-primitive-procedure-here)).

I don't know what the "primitive-code?" predicate is actually testing, and I agree it is surprising (based purely on the name) that it returns true when "count" is given. Perhaps I am calling it incorrectly. Regardless, for both this procedure and others which pass the "primitive-code?" predicate "find-program-arities" returns #f.

scheme@(guile-user)> (primitive-code? (program-code count))
$5 = #t
scheme@(guile-user)> (find-program-arities (program-code count))
$6 = #f

> You could probably modify the interpreter to record arity information as a ‘procedure property’, and then modify the arity querying procedures to look for this information when available, and prefer it above.
There is actually a place where arity information is recorded as a procedure property - line 576 of module/ice-9/eval.scm (on the same commit as mentioned above). However, I have not found a function which actually has this property so I'm not sure what circumstances exercise this code path.

> If there is some relevant performance impact, I have a rather ugly method for eliminating it during bootstrapping (at cost of not having proper arity information for interpreted things during bootstrapping) if you are interested.
>
> (Also, technically there are multiple Scheme interpreters in Guile for technical reasons – a Scheme interpreter written in Scheme (“meta-circular evaluator’), and one written in C. The C implementation exists for bootstrapping the Scheme Scheme interpreter, so it’s of less interest to get arity information in there.)

Good to know! I'm not super familiar with the bootstrapping process so I haven't looked at this aspect yet. I'll keep you in mind when I do!

> You would need to investigate the performance impact (especially the performance impact on the bootstrapping process (i.e. compile without any precompiled .go)) (Inconveniently there doesn’t seem to be a standard way of testing for such.)

My hope is that I can test this locally by using Guix's --with-patch flag to perform the build. The precompiled files should not be used according to Guix's model, but it is technically possible to use them and I haven't looked at the build definition that closely. However, I'm not sure if my computing environment is representative of the typical computing environment or the subset of computing environments that are most relevant to people building Guile from scratch.

Shell environment:

$ guix time-machine --channels=./channels.scm -- shell --container --nesting guile-next guile-readline coreutils
user@foreign-guix /dev/shm [env]$ cat channels.scm
(use-modules (guix ci))
(list (channel-with-substitutes-available
       %default-guix-channel
       "https://ci.guix.gnu.org"))
user@foreign-guix /dev/shm [env]$ guix describe
  guix d663d4b
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: d663d4b6cae2289bcdd8bdcd02d58ed30769f1e2
user@foreign-guix /dev/shm [env]$ guile
GNU Guile 3.0.9-0.3b76a30
Copyright (C) 1995-2024 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (ice-9 pretty-print) (system vm debug) (system vm program) (srfi srfi-1))