Some issues with guile

Some issues with guile

[Nikolaos Chatzikonstantinou]

Hello list,

I want to talk about some issues I've encountered with Guile. I'll
quickly summarize the points and I'll expand below.

1. Can't connect geiser from emacs to a remote REPL server unless
versions match.
2. Documentation extraction sucks.
3. Programs/libraries that use Guile as an extension language are
incompatible with one another.

If anyone can help with these points I would appreciate it. Keep in
mind that I am not a seasoned schemer nor intimately familiar with the
GNU Guile manual and it is possible that I've misunderstood something.

1. The easiest OS to hack on Guile is Guix. I don't run Guix so I
spinned a VM with it, where I installed guile-next for the
bleeding-edge guile. I attempted to run `guile --listen` in the VM and
`connect-to-guile` in the (Debian) host, but I'm getting errors, it's
not working. I don't quite understand why. The errors seem to be
related to version incompatibility, but I don't understand what if
anything my host guile package has to do with the guest VM guile
package.

2. There's an issue with documentating source code. The best system
I've seen is Rust's, but sphinx and Doxygen work fine too. At the very
least, a documentation system should have the following features:
  i. document all language constructs
  ii. markup (i.e. code blocks, links to other items)
  iii. exportable
Of course there can be more features, such as unit tests in
documentation, but I don't consider them essential. I don't know what
Guile does. I know there's `guild doc-snarf` and
`(ice-9 documentation)` with docstrings, as well as the package
documentá (<https://luis-felipe.gitlab.io/guile-documenta/>.) These don't work:
  - `guild doc-snarf` does something clumsily ONLY with
    double-semicolon comments. Triple-semicolon file headers that
    explain the purpose of a module are ignored, for example. It's not
    clear what the utility of the output of this tool is.
  - `object-documentation` from `(ice-9 documentation)` only seems to
    work with docstrings of functions, although it claims to work with
    macros too, suggesting that the `documentation` property should be
    set. But how? It's not explained. The info page on "Object
    Properties" makes it seem like this suffices: (set! (documentation
    my-function) "the docstring.")  but I can't get it to work like
    that. Docstrings cannot document files. Maybe they can document
    macros, variables, and modules at least? But the docstring format
    is raw, there is no markup!
  - documentá in its page does not include an example of how it works!
    Not a line of code to explain to the user which documentation is
    extracted. I could not understand how to use it.

3. I use GNU GDB for debugging. Sometimes there's a desire to be able
to interpret byte values in memory. GNU poke allows for this sort of
thing. Both programs are very similar in how they behave, and in
theory it should be possible to run poke commands from gdb, especially
since there's libpoke for this exact purpose. However, this is not
possible as far as I've been told, and the reason is that both libpoke
and gdb load guile, and therefore, gdb cannot load libpoke in memory
because of some issue with the Guile garbage collector! This is
terribly disappointing. Can anyone clarify what is going on with this
sort of thing? Workarounds so far include compiling gdb without guile
support (everyone uses python for scripting in gdb anyway) or dumping
byte ranges from gdb and examining them from the poke process
separately.

Re: Some issues with guile

[Luis Felipe]

[-- Attachment #1.1.1: Type: text/plain, Size: 1863 bytes --]

Hi Nikolaos,

El 26/04/24 a las 7:05, Nikolaos Chatzikonstantinou escribió:
> 2. Documentation extraction sucks.
[...]
>    - documentá in its page does not include an example of how it works!
>      Not a line of code to explain to the user which documentation is
>      extracted. I could not understand how to use it.

Yeah, I didn't want to include how to document code in Documentá. 
Instead, I wanted to propose adding that documentation to Guile's 
documentation and link to it from Documentá. But I haven't made the time 
to write the proposed section.

Currently, Documentá can extract module documentation and procedure 
documentation. It also documents variables, record types, and macros 
exported by modules, but it simply lists them (record type fields are 
listed too), it doesn't extract any particular documentation added by 
human code writers. I haven't found, and in some cases investigated, a 
way to properly document variables, macros, record types and GOOPS 
clases using human-written documentation strings. But I want to have 
that too.

You can see examples of module and procedure documentation in 
Documentá's source 
(https://codeberg.org/luis-felipe/guile-documenta/src/branch/trunk/documenta). 
Note that for module documentation Documentá supports the conventional 
format

   ;;; Commentary:

   ;;; Your module documentation here ↓

   ;;; Code:

   ;;; Your code here ↓

Guile Scheme comments in the "Commentary" section are considered module 
documentation. You can use block comments too, the ones surrounded in #| 
... |#, instead of multiple line comments (see Documentá source code for 
examples).

Finally, better structuring, indexing and linking of generated API 
documentation are planned. I also want to explore exporting to Org format.

Hope that helps,

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 2881 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Some issues with guile

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:

> Of course there can be more features, such as unit tests in
> documentation, but I don't consider them essential. I don't know what

I was missing these badly when I started, so I wrote support for them
myself:

Usage:
https://hg.sr.ht/~arnebab/wisp/browse/examples/doctests-test.scm?rev=tip
Module:
https://hg.sr.ht/~arnebab/wisp/browse/examples/doctests.scm?rev=tip
Minimal example:
https://hg.sr.ht/~arnebab/wisp/browse/examples/doctests-testone.scm?rev=tip

These don’t put the tests into the documentation (where it would be
prone to parsing complexities — I got burned by Python ☺) but instead
attaches them as procedure properties (which is possible, because in
Scheme code and data are uniform: you can write code as a
datastructure).

Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Some issues with guile

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 4156 bytes --]

On 2024-04-26 03:05:51 -0400, Nikolaos Chatzikonstantinou wrote:
> Hello list,
>
> I want to talk about some issues I've encountered with Guile. I'll
> quickly summarize the points and I'll expand below.
>
> 1. Can't connect geiser from emacs to a remote REPL server unless
> versions match.
> 2. Documentation extraction sucks.
> 3. Programs/libraries that use Guile as an extension language are
> incompatible with one another.
>
> If anyone can help with these points I would appreciate it. Keep in
> mind that I am not a seasoned schemer nor intimately familiar with the
> GNU Guile manual and it is possible that I've misunderstood something.
>
> 1. The easiest OS to hack on Guile is Guix. I don't run Guix so I
> spinned a VM with it, where I installed guile-next for the
> bleeding-edge guile. I attempted to run `guile --listen` in the VM and
> `connect-to-guile` in the (Debian) host, but I'm getting errors, it's
> not working. I don't quite understand why. The errors seem to be
> related to version incompatibility, but I don't understand what if
> anything my host guile package has to do with the guest VM guile
> package.
>
> 2. There's an issue with documentating source code. The best system
> I've seen is Rust's, but sphinx and Doxygen work fine too. At the very
> least, a documentation system should have the following features:
>   i. document all language constructs
>   ii. markup (i.e. code blocks, links to other items)
>   iii. exportable
> Of course there can be more features, such as unit tests in
> documentation, but I don't consider them essential. I don't know what
> Guile does. I know there's `guild doc-snarf` and
> `(ice-9 documentation)` with docstrings, as well as the package
> documentá (<https://luis-felipe.gitlab.io/guile-documenta/>.) These don't work:
>   - `guild doc-snarf` does something clumsily ONLY with
>     double-semicolon comments. Triple-semicolon file headers that
>     explain the purpose of a module are ignored, for example. It's not
>     clear what the utility of the output of this tool is.
>   - `object-documentation` from `(ice-9 documentation)` only seems to
>     work with docstrings of functions, although it claims to work with
>     macros too, suggesting that the `documentation` property should be
>     set. But how? It's not explained. The info page on "Object
>     Properties" makes it seem like this suffices: (set! (documentation
>     my-function) "the docstring.")  but I can't get it to work like
>     that. Docstrings cannot document files. Maybe they can document
>     macros, variables, and modules at least?

What you want is:

    (set-object-property! foo 'documentation "Contains a @code{'bar}.")

> But the docstring format is raw, there is no markup!

You can write them in whatever markup you want, I personally use texinfo format
(next version of geiser will be able to process and display it nicely).

>   - documentá in its page does not include an example of how it works!
>     Not a line of code to explain to the user which documentation is
>     extracted. I could not understand how to use it.
>
> 3. I use GNU GDB for debugging. Sometimes there's a desire to be able
> to interpret byte values in memory. GNU poke allows for this sort of
> thing. Both programs are very similar in how they behave, and in
> theory it should be possible to run poke commands from gdb, especially
> since there's libpoke for this exact purpose. However, this is not
> possible as far as I've been told, and the reason is that both libpoke
> and gdb load guile, and therefore, gdb cannot load libpoke in memory
> because of some issue with the Guile garbage collector! This is
> terribly disappointing. Can anyone clarify what is going on with this
> sort of thing? Workarounds so far include compiling gdb without guile
> support (everyone uses python for scripting in gdb anyway) or dumping
> byte ranges from gdb and examining them from the poke process
> separately.
>

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Some issues with guile

[Nikolaos Chatzikonstantinou]

On Fri, Apr 26, 2024 at 11:21 AM Luis Felipe <sirgazil@zoho.com> wrote:
>
> Hi Nikolaos,

Hello Luis!

> El 26/04/24 a las 7:05, Nikolaos Chatzikonstantinou escribió:
> > 2. Documentation extraction sucks.
> [...]
> >    - documentá in its page does not include an example of how it works!
> >      Not a line of code to explain to the user which documentation is
> >      extracted. I could not understand how to use it.
>
> Yeah, I didn't want to include how to document code in Documentá.
> Instead, I wanted to propose adding that documentation to Guile's
> documentation and link to it from Documentá. But I haven't made the time
> to write the proposed section.

Just add /something/ with a visible TODO that your wish is to have
guile document it instead. It should be prominent too, not buried 10
layers deep. You could just say "read the source code of documentá for
examples." When I looked at your documentation, I spent about 10
minutes trying to figure this out, and I was frustrated when I
couldn't find any examples. The user is left thinking they're an idiot
(they very well may be!) for not RTFM well enough and frustrated,
unlikely to look back at documentá...

> Currently, Documentá can extract module documentation and procedure
> documentation. It also documents variables, record types, and macros
> exported by modules, but it simply lists them (record type fields are
> listed too), it doesn't extract any particular documentation added by
> human code writers. I haven't found, and in some cases investigated, a
> way to properly document variables, macros, record types and GOOPS
> clases using human-written documentation strings. But I want to have
> that too.

What is the issue with this?

    ;;; my favorite constant
    (define magic 42)

Regards,
Nikolaos Chatzikonstantinou

Re: Some issues with guile

[Nikolaos Chatzikonstantinou]

On Fri, Apr 26, 2024 at 4:39 PM Tomas Volf <~@wolfsden.cz> wrote:
>
> On 2024-04-26 03:05:51 -0400, Nikolaos Chatzikonstantinou wrote:
> > <snip complaints about Guile's documentation extraction>
> >   - `object-documentation` from `(ice-9 documentation)` only seems to
> >     work with docstrings of functions, although it claims to work with
> >     macros too, suggesting that the `documentation` property should be
> >     set. But how? It's not explained. The info page on "Object
> >     Properties" makes it seem like this suffices: (set! (documentation
> >     my-function) "the docstring.")  but I can't get it to work like
> >     that. Docstrings cannot document files. Maybe they can document
> >     macros, variables, and modules at least?
>
> What you want is:
>
>     (set-object-property! foo 'documentation "Contains a @code{'bar}.")

Okay, so this can document objects. I propose that a good-enough
solution is to document symbols. This can document functions,
variables, macros, modules (although it seems that modules live in a
different namespace?) I am not sure how to document GOOP-specific
stuff, but AT LEAST we have something.

> > But the docstring format is raw, there is no markup!
>
> You can write them in whatever markup you want, I personally use texinfo format
> (next version of geiser will be able to process and display it nicely).

I can use any format but will there be compatibility with the tooling?
Texinfo -- fine, but there's no examples in the guile manual of such
things. It's impossible to think that the user should pick up such
ideas on their own.

You can see for example what happens under
guile/module/ice-9/streams.scm, a random file I picked from the Guile
package. Not a single procedure is docstring-documented. There's some
documentation in a giant block containing things such as
    ;; (stream-null? stream)
    ;;  - yes.
(who knows what that means?) Other procedures are more fortunate,
    ;; (stream-map proc stream0 ...)
    ;;  - like `map', except returns a stream of results, and not a list.
Note here the use of `' when writing `map'. I am not sure if this is
texinfo related; I don't think it is. I know it works in elisp to link
to other items. The entire block starts with
    ;; Use:
Is this markup? who knows. But let's look at another file, threads.scm
in the same directory. Every procedure is documented with docstrings
in texinfo format! Well, let's look at regex.scm. It uses triple
semicolons for
    ;;; Code:
as opposed to streams.scm's double semicolons. Why? Why not!

I don't mean to just complain. There needs to be some documentation
consistency and once established it needs to be championed, and that's
what I'm trying to accomplish...

Regards,
Nikolaos Chatzikonstantinou

Re: Some issues with guile

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:

> I don't mean to just complain. There needs to be some documentation
> consistency and once established it needs to be championed, and that's
> what I'm trying to accomplish...

Would you like to start with sending some small and easy to review
patches? As long as they are small enough and uncontroversial (and don’t
break anything), I can commit and push them.

Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Some issues with guile

[Linas Vepstas]

On Sat, Apr 27, 2024 at 2:47 AM Nikolaos Chatzikonstantinou <
nchatz314@gmail.com> wrote:

> On Fri, Apr 26, 2024 at 4:39 PM Tomas Volf <~@wolfsden.cz> wrote:
> >
> > What you want is:
> >
> >     (set-object-property! foo 'documentation "Contains a @code{'bar}.")
>
> Okay, so this can document objects. I propose that a good-enough
> solution is to document symbols.


 (define foo 42)
(set-object-property! foo 'documentation "my foo thing")
,a foo
(guile-user): foo
,d foo
my foo thing
(define (bar) (list 'a))
(set-object-property! bar 'documentation "this bar does stuff")
,a bar
(guile-user): bar #<procedure bar ()>
(guile): module-obarray-ref #<procedure module-obarray-ref (ob key)>
...
,d bar
this bar does stuff

where ,a is short for ,apropos and ,d is short for ,describe

-- Linas

Re: Some issues with guile

[Luis Felipe]

[-- Attachment #1.1.1: Type: text/plain, Size: 1205 bytes --]

Hi Linas,

El 27/04/24 a las 17:35, Linas Vepstas escribió:
> On Sat, Apr 27, 2024 at 2:47 AM Nikolaos Chatzikonstantinou <
> nchatz314@gmail.com> wrote:
>
>> On Fri, Apr 26, 2024 at 4:39 PM Tomas Volf <~@wolfsden.cz> wrote:
>>> What you want is:
>>>
>>>      (set-object-property! foo 'documentation "Contains a @code{'bar}.")
>> Okay, so this can document objects. I propose that a good-enough
>> solution is to document symbols.
>
>   (define foo 42)
> (set-object-property! foo 'documentation "my foo thing")
> ,a foo
> (guile-user): foo
> ,d foo
> my foo thing
> (define (bar) (list 'a))
> (set-object-property! bar 'documentation "this bar does stuff")
> ,a bar
> (guile-user): bar #<procedure bar ()>
> (guile): module-obarray-ref #<procedure module-obarray-ref (ob key)>
> ...
> ,d bar
> this bar does stuff
>
> where ,a is short for ,apropos and ,d is short for ,describe

If I understand correctly, though, the "set-object-property!" procedure 
is part of a legacy interface. The manual recommends using object 
properties as shown in the Object Properties section instead 
(https://www.gnu.org/software/guile/manual/html_node/Object-Properties.html).



[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 2881 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Some issues with guile

[Luis Felipe]

[-- Attachment #1.1.1: Type: text/plain, Size: 2304 bytes --]

El 27/04/24 a las 7:28, Nikolaos Chatzikonstantinou escribió:
> On Fri, Apr 26, 2024 at 11:21 AM Luis Felipe <sirgazil@zoho.com> wrote:
>> Hi Nikolaos,
> Hello Luis!
>
>> El 26/04/24 a las 7:05, Nikolaos Chatzikonstantinou escribió:
>>> 2. Documentation extraction sucks.
>> [...]
>>>     - documentá in its page does not include an example of how it works!
>>>       Not a line of code to explain to the user which documentation is
>>>       extracted. I could not understand how to use it.
>> Yeah, I didn't want to include how to document code in Documentá.
>> Instead, I wanted to propose adding that documentation to Guile's
>> documentation and link to it from Documentá. But I haven't made the time
>> to write the proposed section.
> Just add /something/ with a visible TODO that your wish is to have
> guile document it instead. It should be prominent too, not buried 10
> layers deep. You could just say "read the source code of documentá for
> examples." When I looked at your documentation, I spent about 10
> minutes trying to figure this out, and I was frustrated when I
> couldn't find any examples. The user is left thinking they're an idiot
> (they very well may be!) for not RTFM well enough and frustrated,
> unlikely to look back at documentá...

I'll see what I can do to make it easier :)

>> Currently, Documentá can extract module documentation and procedure
>> documentation. It also documents variables, record types, and macros
>> exported by modules, but it simply lists them (record type fields are
>> listed too), it doesn't extract any particular documentation added by
>> human code writers. I haven't found, and in some cases investigated, a
>> way to properly document variables, macros, record types and GOOPS
>> clases using human-written documentation strings. But I want to have
>> that too.
> What is the issue with this?
>
>      ;;; my favorite constant
>      (define magic 42)

That I don't know any existing mechanism to tell that the comment is the 
documentation of the variable. I'd prefer something like Elisp 
(https://www.gnu.org/software/emacs/manual/html_node/elisp/Defining-Variables.html):

   defvar symbol [value [doc-string]]

   (defvar bar 23
     "The normal weight of a bar.")



[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 2881 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Some issues with guile

[Nikolaos Chatzikonstantinou]

On Sat, Apr 27, 2024 at 1:35 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
> On Sat, Apr 27, 2024 at 2:47 AM Nikolaos Chatzikonstantinou <nchatz314@gmail.com> wrote:
>>
>> On Fri, Apr 26, 2024 at 4:39 PM Tomas Volf <~@wolfsden.cz> wrote:
>> >
>> > What you want is:
>> >
>> >     (set-object-property! foo 'documentation "Contains a @code{'bar}.")
>>
>> Okay, so this can document objects. I propose that a good-enough
>> solution is to document symbols.
>
>
>  (define foo 42)
> (set-object-property! foo 'documentation "my foo thing")
> ,d foo
> my foo thing

You will also get the same response for
    ,d 42
because you've documented the value object and not the symbol object.
The function used by ,d is object-documentation by (ice-9
documentation). When procedures are documented via docstrings, the
documentation is set to the lambda itself (and as far as I can tell,
not via object properties.)

That's why I suggested that for variables, macros, etc, we can
document the symbol. However, I'm realizing there's another issue,
which has to do with module usage, i.e. symbols can be renamed. To
have consistent documentation follow the symbols around correctly, the
use-modules macro needs to be rewritten.

Regards,
Nikolaos Chatzikonstantinou