Getting all symbols in a Scheme file as a list

Getting all symbols in a Scheme file as a list

[Christina O'Donnell]

Hi,

I'm trying to write a Guile script to trace symbol definition and reference
between modules in a large Guile repo (GNU/Guix), for the purposes of large
scale refactoring.

I'm wondering how I could programmatically get all the values in a 
Scheme file
as an S-expression. From the manual, I know that the REPL has meta keyword
',binding' and ',apropos'
<https://www.gnu.org/software/guile/manual/guile.html#index-apropos> 
which let
you search and list all bindings in accessible to a module. These are 
exactly
what I need only, since they're meta-commands, they don't produce Scheme
expressions.

Does anyone have any pointers? Should I go down the route of,

    (open-input-pipe (string-append "guile -l" file "-c ,binding"))

? Seems a little bit baroque to me, I'd expect a simpler way of doing 
it. Any
libraries anyone knows of?

Kind regards,
  - Christina

RE: Getting all symbols in a Scheme file as a list

[M]

>Onderwerp: Getting all symbols in a Scheme file as a list

The following will give you a list of all the symbols (unless I made some syntax errors) (also replace ‘ by a proper quote, e-mail program is corrupting it):

(use-modules (ice-9 match))
(define (all-symbols s-exp)
  (match s-exp
    ((? symbol? a) (list a))
    ((a . b) (append (all-symbols a) (all-symbols b)))
   ;; insert rules for vectors and arrays
    […]
    (_ ‘())))
[ also open file with open-input-file, apply ‘read’ to the port and pass the result to all-symbols, in a loop, and append the results together. ].

Doesn’t seem useful for what you are mentioning later, though …

>I'm trying to write a Guile script to trace symbol definition and reference
between modules in a large Guile repo (GNU/Guix), for the purposes of large
scale refactoring.

There is no such thing as symbol definitions in Scheme – you can make symbols with symbol->string, but you can’t define a symbol to anything, symbols simply are.

You can, however, define variables, which have a symbol as name (and that name may depend on context in case of hygienic macros or renamed imports/exports in modules).

>I'm wondering how I could programmatically get all the values in a 
Scheme file
as an S-expression. From the manual, I know that the REPL has meta keyword
',binding' and ',apropos'
<https://www.gnu.org/software/guile/manual/guile.html#index-apropos> 
which let
you search and list all bindings in accessible to a module. These are 
exactly
what I need only, since they're meta-commands, they don't produce Scheme
expressions.

>Does anyone have any pointers? Should I go down the route of,

>    (open-input-pipe (string-append "guile -l" file "-c ,binding"))

>? Seems a little bit baroque to me, I'd expect a simpler way of doing 
it. Any
libraries anyone knows of?

No. The quoting is incorrect when file has spaces, \, …, guile in PATH might not be the Guile that is being run, … (For the former, consider open-input-pipe*.)

You can probably find a simpler way of doing it by locating where “,binding” is implemented (in the Guile source code). It probably uses the module reflection API. This API is documented in

https://www.gnu.org/software/guile/manual/html_node/Module-System-Reflection.html

However, the documentation is incomplete – IIRC there is a procedure ‘module-bindings’ to find a list of top-level definitions, but it doesn’t seem to be documented.

To find what modules a module uses, there is module-uses, though I doubt it is reliable w.r.t. optimisation, inlining, uses of (@ (module name) variable), …

So, instead, I would propose to instead work  on the Tree-IL level (https://www.gnu.org/software/guile/manual/html_node/Tree_002dIL.html). In particular, see <toplevel-define> for definitions and <module-ref>,<module-set!> for uses of other modules.

This also allows for more fine-grained information – e.g. these <module-ref> etc. objects contain the location in the source code, and if a <module-ref> is inside a <toplevel-define> then you know that it is the procedure (assuming it is a procedure) of the <toplevel-define> that uses the variable of the <module-ref>.

(See https://www.gnu.org/software/guile/manual/html_node/The-Scheme-Compiler.html for how to compile stuff – also, IIRC, the compilation procedure accepts ports instead of only S-expressions, despite what the example suggests.)

Best regards,
Maxime Devos

Re: Getting all symbols in a Scheme file as a list

[Christina O'Donnell]

Hi,

> There is no such thing as symbol definitions in Scheme – you can make 
> symbols with symbol->string, but you can’t define a symbol to 
> anything, symbols simply are.
>
> You can, however, define variables, which have a symbol as name (and 
> that name may depend on context in case of hygienic macros or renamed 
> imports/exports in modules).
Thanks for the clarification. I'll try to remember this.

> You can probably find a simpler way of doing it by locating where 
> “,binding” is implemented (in the Guile source code). It probably uses 
> the module reflection API. This API is documented in
>
> https://www.gnu.org/software/guile/manual/html_node/Module-System-Reflection.html
>
> However, the documentation is incomplete – IIRC there is a procedure 
> ‘module-bindings’ to find a list of top-level definitions, but it 
> doesn’t seem to be documented.
>
Got it! The REPL uses module-for-each, in (ice-9 boot-9), which is 
exported publicly:

(define (module-for-each proc module)
   "Call PROC on each symbol in MODULE, with arguments of (SYMBOL 
VARIABLE)."
   (hash-for-each proc (module-obarray module)))

This is called from (system repl command):

(define-meta-command (binding repl)
   "binding
List current bindings."
   (module-for-each (lambda (k v) (format #t "~23A ~A\n" k v))
                    (current-module)))

That'll do it for me!

> To find what modules a module uses, there is module-uses, though I 
> doubt it is reliable w.r.t. optimisation, inlining, uses of (@ (module 
> name) variable), …
>
> So, instead, I would propose to instead work  on the Tree-IL level 
> (https://www.gnu.org/software/guile/manual/html_node/Tree_002dIL.html). 
> In particular, see <toplevel-define> for definitions and 
> <module-ref>,<module-set!> for uses of other modules.
>
> This also allows for more fine-grained information – e.g. these 
> <module-ref> etc. objects contain the location in the source code, and 
> if a <module-ref> is inside a <toplevel-define> then you know that it 
> is the procedure (assuming it is a procedure) of the <toplevel-define> 
> that uses the variable of the <module-ref>.
>
> (See 
> https://www.gnu.org/software/guile/manual/html_node/The-Scheme-Compiler.html 
> for how to compile stuff – also, IIRC, the compilation procedure 
> accepts ports instead of only S-expressions, despite what the example 
> suggests.)
>
Thanks. I know what I'm read next.

Thank you so much for your help!
  - Christina

On 04/02/2024 21:03, M wrote:
>
> >I'm trying to write a Guile script to trace symbol definition and 
> reference
>
> between modules in a large Guile repo (GNU/Guix), for the purposes of 
> large
>
> scale refactoring.
>
> There is no such thing as symbol definitions in Scheme – you can make 
> symbols with symbol->string, but you can’t define a symbol to 
> anything, symbols simply are.
>
> You can, however, define variables, which have a symbol as name (and 
> that name may depend on context in case of hygienic macros or renamed 
> imports/exports in modules).
>
> >I'm wondering how I could programmatically get all the values in a
>
> Scheme file
>
> as an S-expression. From the manual, I know that the REPL has meta keyword
>
> ',binding' and ',apropos'
>
> <https://www.gnu.org/software/guile/manual/guile.html#index-apropos>
>
> which let
>
> you search and list all bindings in accessible to a module. These are
>
> exactly
>
> what I need only, since they're meta-commands, they don't produce Scheme
>
> expressions.
>
> >Does anyone have any pointers? Should I go down the route of,
>
> >    (open-input-pipe (string-append "guile -l" file "-c ,binding"))
>
> >? Seems a little bit baroque to me, I'd expect a simpler way of doing
>
> it. Any
>
> libraries anyone knows of?
>
> No. The quoting is incorrect when file has spaces, \, …, guile in PATH 
> might not be the Guile that is being run, … (For the former, consider 
> open-input-pipe*.)
>
> You can probably find a simpler way of doing it by locating where 
> “,binding” is implemented (in the Guile source code). It probably uses 
> the module reflection API. This API is documented in
>
> https://www.gnu.org/software/guile/manual/html_node/Module-System-Reflection.html
>
> However, the documentation is incomplete – IIRC there is a procedure 
> ‘module-bindings’ to find a list of top-level definitions, but it 
> doesn’t seem to be documented.
>
> To find what modules a module uses, there is module-uses, though I 
> doubt it is reliable w.r.t. optimisation, inlining, uses of (@ (module 
> name) variable), …
>
> So, instead, I would propose to instead work  on the Tree-IL level 
> (https://www.gnu.org/software/guile/manual/html_node/Tree_002dIL.html). 
> In particular, see <toplevel-define> for definitions and 
> <module-ref>,<module-set!> for uses of other modules.
>
> This also allows for more fine-grained information – e.g. these 
> <module-ref> etc. objects contain the location in the source code, and 
> if a <module-ref> is inside a <toplevel-define> then you know that it 
> is the procedure (assuming it is a procedure) of the <toplevel-define> 
> that uses the variable of the <module-ref>.
>
> (See 
> https://www.gnu.org/software/guile/manual/html_node/The-Scheme-Compiler.html 
> for how to compile stuff – also, IIRC, the compilation procedure 
> accepts ports instead of only S-expressions, despite what the example 
> suggests.)
>
> Best regards,
>
> Maxime Devos
>

Re: Getting all symbols in a Scheme file as a list

[Matt Wette]

On 2/4/24 11:50 AM, Christina O'Donnell wrote:
> I'm trying to write a Guile script to trace symbol definition and 
> reference
> between modules in a large Guile repo (GNU/Guix), for the purposes of 
> large
> scale refactoring.

> ? Seems a little bit baroque to me, I'd expect a simpler way of doing 
> it. Any
> libraries anyone knows of?

Maybe use something from (system vm elf ) on the .go files?

$ cd .../ice-9/
$ nm safe.go
00000000000002ac r
0000000000000248 r make-safe-module
00000000000001c4 r safe-environment


I use guile elf to extract info from C-sourced files, but maybe some 
tidbits may help:

   (let* ((bv (call-with-input-file filename
                (lambda (port)
                  (get-bytevector-n port (and=> (stat port) stat:size)))))
          (elf (parse-elf bv))
    (pretty-print (elf-named-syms elf))

(define (elf-named-syms elf)
   (define (less a b) (< (elf-symbol-value (cdr a)) (elf-symbol-value 
(cdr b))))
   (let* ((.symtab (elf-section-by-name elf ".symtab"))
          (strtab (elf-section elf (elf-section-link .symtab)))
          (tbl-len (elf-symbol-table-len .symtab)))
     (let loop ((symd '()) (ix 0))
       (if (< ix tbl-len)
           (let* ((sym (elf-symbol-table-ref elf .symtab ix strtab))
                  (name (elf-symbol-name sym))
                  (nlen (string-length name)))
             (loop (if (positive? nlen) (acons name sym symd) symd) (1+ 
ix)))
           (sort symd less)))))