Compile skribilo doc containing guix channel references

Compile skribilo doc containing guix channel references

[Phil]

Hi all,

I'd like to document a private guix channel using Guile-based document
framework: Skribilo - https://www.nongnu.org/skribilo/.

The idea is that the guix channel repo will contain a skb file which can
then reference code in the channel by setting the source-path and then
using the :definition in the source function to reference bits of the
package I want to talk about in my document.

(source :language [:tab 8] [:definition] [:stop] [:start] [:file])

This works really well, but it got me thinking - what would be even
cooler would be if I could import modules in my channel and actually
execute channel functions to generate document output as part of the
document compilation process, rather than just display the package code!

The problem with this is that in order to import modules in my channel,
I'd also have to have the guix core modules on my load-path at skribilo
compilation time, otherwise skribilo complains when my channel makes use
of guix modules.

For example, if I just add the channel packages without guix core
modules at the top of my skribilo document:
(add-to-load-path (string-append (dirname (current-filename)) "/packages"))

I understandably get something like the below:
no code for module (guix git)

Normally when scripting for guix you'd avoid the above problem by using
'guix repl -- some-script.scm' but here I'm using the skribilo
executable, *not* guix as the entry point - it's not entirely clear to me
how to get the required guix modules onto my load path?

As an ill-advised hack I tried the following under guix repl:
(use-modules (gnu packages))
(%package-module-path)

Then I added the returned paths via add-to-load-path at the top of my skb
file.  This resulted in:
$ GUILE_AUTO_COMPILE=0 guix environment --ad-hoc skribilo -- skribilo
--target=html --source-path=.. guix-packages.skb -o index.html
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
/gnu/store/8bab6fga7858whgd11f0wlsi2zsp4yr2-guix-module-union/share/guile/site/3.0/guix/packages.scm:361:5:
Unknown #object: "#~"

My guess is there is a better approach - any suggestions?  Perhaps I can
call the skribilo compiler from inside "guix repl" for example?

Cheers,
Phil.

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Phil writes:

> My guess is there is a better approach - any suggestions?  Perhaps I can
> call the skribilo compiler from inside "guix repl" for example?

To somewhat answer my own question in case it's useful to anyone else or
if anyone has any suggestion on how to do it better - there are 2 ways
I've found to do this.  The most obvious is to abandon the skribilo DSL
and write the document directly as a scheme file which will evaluate and
write itself out as HTML - it can then be called as: guix repl -- foo.scm:

(with-output-to-file "foo.html"
  (lambda ()
    (evaluate-document d e))))

See skribilo docs for full example.

This isn't ideal because you can't use a the skribe format which isn't
pure scheme.  To do that we have to call the scribilo compiler's main entry
point from inside scheme rather than from the command line - the below
example worked for me, but it's quite fiddly to get it to work.  It is
also called from guix repl.

To give a very simple example the document can use guix functions to
reference infomation in the channel I'm documenting:

(p [The ,(package-name my-package) package is set to ,(package-version my-package).])

The below script has to be one level above the package directory in my
channel, and the script must be called the directory which houses it.

The output should also go to the same directory.  I then moved the
output HTML afterwards, rather than putting the target path in --output
- this makes sure the links work.

The hardcoding of the load-path is also not ideal, I suspect this can be
improved.

(add-to-load-path "/gnu/store/ymvqgkljzykfv4g7vbw36pgghfjbhnps-skribilo-0.9.5/share/guile/site/3.0")
(add-to-load-path "/gnu/store/2160nv245ys30vxvhpjxgcbjm9rq67gw-guile-reader-0.6.3/share/guile/site/3.0")

(define local-repo-clone-path
  (dirname (current-filename)))
 
(define local-packages-clone-path
  (string-append local-repo-clone-path "/packages"))

(add-to-load-path local-packages-clone-path)

(use-modules (skribilo condition))

(call-with-skribilo-error-catch/exit
  (λ ()
    (apply (module-ref (resolve-interface '(skribilo)) 'skribilo)
           `("--target=html"
             ,(string-append "--source-path=" local-packages-clone-path)
             ,(string-append local-repo-clone-path "/docs/my-channel-skribilo-doc.skb")
             "--output=index.html"))))

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Hi,

Phil <phil@beadling.co.uk> skribis:

> The idea is that the guix channel repo will contain a skb file which can
> then reference code in the channel by setting the source-path and then
> using the :definition in the source function to reference bits of the
> package I want to talk about in my document.
>
> (source :language [:tab 8] [:definition] [:stop] [:start] [:file])
>
> This works really well, but it got me thinking - what would be even
> cooler would be if I could import modules in my channel and actually
> execute channel functions to generate document output as part of the
> document compilation process, rather than just display the package code!

I like that.  :-)

We toyed with Skribilo/Guix integration in this paper:

  https://hal.inria.fr/hal-01161771/en
  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/reppar-2015/reproducible-hpc.skb

(Note that it starts with ugly ‘%load-path’ hacks.  :-))

> Normally when scripting for guix you'd avoid the above problem by using
> 'guix repl -- some-script.scm' but here I'm using the skribilo
> executable, *not* guix as the entry point - it's not entirely clear to me
> how to get the required guix modules onto my load path?

I would suggest using your own script, executed with ‘guix repl’,
instead of the ‘skribilo’ executable.  Something like you describe in
your second message looks good!

Thanks,
Ludo’.

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Ludovic Courtès writes:

> We toyed with Skribilo/Guix integration in this paper:
>
>  https://hal.inria.fr/hal-01161771/en
>  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/reppar-2015/reproducible-hpc.skb
>
> (Note that it starts with ugly ‘%load-path’ hacks.  :-))

Thanks for the reply and the links, good to see another example.

> I would suggest using your own script, executed with ‘guix repl’,
> instead of the ‘skribilo’ executable.  Something like you describe in
> your second message looks good!

I've got the repl approach working without the load path hacks now -
creating an --ad-hoc environment containing both guile and skribilo was
all that was required in the end.  If I can make the approach completely
generic I'll post the code somewhere with an example of use.

What's cool is it is then possible to reference packages without even
having the skribilo document co-located in the same repo as the
packages.  For example if I have a package in guix called "my-package":

(define my-package-str
  (with-output-to-string
    (lambda () (pretty-print (package->code my-package)))))

You can then reference this in skribilo like this:

(prog (source :language scheme my-pacakge-str))

This works great, but I have come-up against one issue I have a proposed
fix for.

package->code uses a function called source->code, to output the
scheme representation of the source.  However source->code assumes that
the source can only be an "origin" record, but it's also valid that
source is a "git-checkout" record.  In particular, if the source repo URL
is an ssh address rather than an https address using git-checkout is the
correct approach - for example:

(source
  (git-checkout
    (url "ssh://git@myserver/foo/bar.git")
    (commit "abcdef12345")))

We make extensive use of this, combined with an ssh-agent, to automate
authentication on private repositories.

Supporting git-checkout in source->code is easy enough - I've written a
fix in our guix channel and it works well, happy to contribute this to
guix-patches too if it makes sense?

The proposed fix can been see here:
https://github.com/quantiletechnologies/qt-guix/commit/49900405617fdf6a2e4ab6f4004306a2770221e0

Finally, I suspect Guix could also be interestingly combined with Haunt
to produce and serve HTML generated from Guix content too:
https://dthompson.us/manuals/haunt/index.html

Cheers,
Phil.

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Phil writes:

> Hi,
>
> Ludovic Courtès writes:
>
>> We toyed with Skribilo/Guix integration in this paper:
>>
>>  https://hal.inria.fr/hal-01161771/en
>>  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/reppar-2015/reproducible-hpc.skb
>>
>> (Note that it starts with ugly ‘%load-path’ hacks.  :-))

At the risk of cross-posting from skribilo list, whilst our internal
documentation has to remain private, we also have a public fork of guix
which would benefit from some documentation explaining the changes we've made
to guix core code that we hope one day to push upstream - so I've taken
the same technique and applied it here - so people can see a working example:

This generates the docs with guix imports - see compile-command in the header:
https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/compile-docs.scm

This is the document - it's less ambitious than my internal version in
terms of generating content from guix - but has a few examples of
generating content from channels:
https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/qtdocs/qt-guix.skb

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Phil <phil@beadling.co.uk> skribis:

> At the risk of cross-posting from skribilo list, whilst our internal
> documentation has to remain private, we also have a public fork of guix
> which would benefit from some documentation explaining the changes we've made
> to guix core code that we hope one day to push upstream - so I've taken
> the same technique and applied it here - so people can see a working example:
>
> This generates the docs with guix imports - see compile-command in the header:
> https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/compile-docs.scm
>
> This is the document - it's less ambitious than my internal version in
> terms of generating content from guix - but has a few examples of
> generating content from channels:
> https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/qtdocs/qt-guix.skb

Nice!  Including channel info in the document like you do here is
probably a major use case; it also makes a lot of sense in the context
of reproducible research workflows.

Looking forward to discussing the ‘qt-guix’ improvements too.  :-)

Ludo’.

Re: Compile skribilo doc containing guix channel references

[zimoun]

Hi,

On Sat, 05 Nov 2022 at 19:03, Ludovic Courtès <ludo@gnu.org> wrote:

>> This generates the docs with guix imports - see compile-command in the header:
>> https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/compile-docs.scm
>>
>> This is the document - it's less ambitious than my internal version in
>> terms of generating content from guix - but has a few examples of
>> generating content from channels:
>> https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/qtdocs/qt-guix.skb

I am missing how the Skribilo file is it compiled?  At “guix pull” time?
Or manually when running the script compile-docs.scm?

> Nice!  Including channel info in the document like you do here is
> probably a major use case; it also makes a lot of sense in the context
> of reproducible research workflows.

Ludo, what do you have in mind about reproducible research workflow?


Cheers,
simon

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi Simon,

zimoun writes:

> I am missing how the Skribilo file is it compiled?  At “guix pull” time?
> Or manually when running the script compile-docs.scm?
>

At the moment compilation is only manual and from inside the repo clone:
guix environment skribilo guile -- guix repl -- compile-docs.scm

Making it part of guix pull is an interesting idea given this is where
standard package compilation takes place.  For this to work any code
references would have to be of the form:
(use-modules (my-module foo)
(package->code my-package)

rather than direct access to scheme files in the repo clone using:
(source :file "foo.scm" :definiton 'my-package)

The advantage of using pacakge->code is that the documentation
compilation is then decoupled from the repo clone and can be generated
anywhere guix can be run (just like guix pull).

The 2 disadvantages are 1) Files in the repo, but not part of the guix
module system cannot be referenced, 2) The pretty-printing of packages
generated using package->code does not perfectly reproduce the format in
the repo clone - although they are functionally identical.

For example inputs are referenced fully qualified in-situ - eg
(@ (foo bar) baz), rather than just referencing baz in the inputs, and
importing (foo bar) at the top of the module.  Comments etc, are also
lost too.

If there was a nice way of referencing the uncompiled scheme files in
the channel without needing the repo cloned that would be perfect IMHO.
I don't think this is possible tho?

Re: Compile skribilo doc containing guix channel references

[zimoun]

Hi Phil,

On dim., 06 nov. 2022 at 22:12, Phil <phil@beadling.co.uk> wrote:

> At the moment compilation is only manual and from inside the repo clone:
> guix environment skribilo guile -- guix repl -- compile-docs.scm

I see.  Interesting as a way to document the channel.  Because I find
hard to write and read Skribilo mark-up format, I would use Org and
org-babel; but that’s another story. :-)

Well, this leads me to 2 questions: :-)

 1. Is it possible to have a reader in the style of Racket Scribble [1]?

 2. Does it make to sense to hook “guix pull” to generate some outputs
 (HTML and Info, say) if the channel contains a specific folder?


1: <https://docs.racket-lang.org/scribble/index.html>

Cheers,
simon

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 05 Nov 2022 at 19:03, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Nice!  Including channel info in the document like you do here is
>> probably a major use case; it also makes a lot of sense in the context
>> of reproducible research workflows.
>
> Ludo, what do you have in mind about reproducible research workflow?

In a published paper, it’d be useful to include channel information, and
Skribilo/Guix integration is one way to do that.  For the convenience of
people who do not use Guix, some might also find it to include
approximate package metadata such as name/version pairs, as in that 2015
paper.

Ludo’.

Re: Compile skribilo doc containing guix channel references

[zimoun]

Hi Ludo,

On lun., 07 nov. 2022 at 10:21, Ludovic Courtès <ludo@gnu.org> wrote:

> In a published paper, it’d be useful to include channel information, and
> Skribilo/Guix integration is one way to do that.  For the convenience of
> people who do not use Guix, some might also find it to include
> approximate package metadata such as name/version pairs, as in that 2015
> paper.

About the channel information to include in a published paper, yeah
Skribilo/Guix is one (non-straightforwardly?) way to get “guix describe
-f channels”. ;-)

Is it possible to have a Skribilo reader which would use less
parenthesis?  Similar to Racket Scribble?

Well, if I understand correctly, «the context of reproducible research
workflows» somehow means using Skribilo as authoring system for
producing “active papers“, right?


Cheers,
simon

PS: About the name/version pairs, are we not trying to explain at length
that it is the wrong way to obtain reproducibility?  Aside it is not
clear what is the depth of name/version pairs to provide when speaking
about reproducibility, no?

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> About the channel information to include in a published paper, yeah
> Skribilo/Guix is one (non-straightforwardly?) way to get “guix describe
> -f channels”. ;-)

Sure, one can achieve the same thing without Skribilo.

> Is it possible to have a Skribilo reader which would use less
> parenthesis?  Similar to Racket Scribble?

Definitely, there’s already a couple of them and we could add one with a
syntax similar to Texinfo or Scribble.

> Well, if I understand correctly, «the context of reproducible research
> workflows» somehow means using Skribilo as authoring system for
> producing “active papers“, right?

Right.

> PS: About the name/version pairs, are we not trying to explain at length
> that it is the wrong way to obtain reproducibility?  Aside it is not
> clear what is the depth of name/version pairs to provide when speaking
> about reproducibility, no?

Indeed! :-)  We can do both: explain why it’s wrong and provide a way to
do it for those who want/need to do it anyway.

Ludo’.

Re: Compile skribilo doc containing guix channel references

[zimoun]

Hi Ludo,

On jeu., 10 nov. 2022 at 12:58, Ludovic Courtès <ludo@gnu.org> wrote:

>> Is it possible to have a Skribilo reader which would use less
>> parenthesis?  Similar to Racket Scribble?
>
> Definitely, there’s already a couple of them and we could add one with a
> syntax similar to Texinfo or Scribble.

Do you mean Skribe syntax and Outline syntax [1] by “a couple of them”?
Or others?

Well, if it is possible to implement something similar to Texinfo, why
not similar to LaTeX which is more popular for scientific authoring
system than all the others.  For what it is worth, I would never invest
in something different from LaTeX syntax for publishing. :-)

1: https://www.nongnu.org/skribilo/doc/user-3.html


Cheers,
simon

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On jeu., 10 nov. 2022 at 12:58, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>> Is it possible to have a Skribilo reader which would use less
>>> parenthesis?  Similar to Racket Scribble?
>>
>> Definitely, there’s already a couple of them and we could add one with a
>> syntax similar to Texinfo or Scribble.
>
> Do you mean Skribe syntax and Outline syntax [1] by “a couple of them”?
> Or others?

Yes, plus Gemtext recently added by Arun.

> Well, if it is possible to implement something similar to Texinfo, why
> not similar to LaTeX which is more popular for scientific authoring
> system than all the others.

Because LaTeX (really: TeX) has the wrong abstraction level.

But maybe that’s more of a discussion for skribilo-users@nongnu.org.  :-)

Ludo’.

Re: Compile skribilo doc containing guix channel references

[Ludovic Courtès]

Hi,

Phil <phil@beadling.co.uk> skribis:

> The proposed fix can been see here:
> https://github.com/quantiletechnologies/qt-guix/commit/49900405617fdf6a2e4ab6f4004306a2770221e0

Sure, that makes sense; you can send it to guix-patches.

I’d suggest using #:autoload for (guix git) and adding a test case in
‘tests/print.scm’.

Thanks,
Ludo’.