Literate programming

Literate programming

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

What do you think about literate programming [1,2,3]?

I've never tried it, but I like the idea.  WEB and Noweb look a bit
outdated, but there is also Babel [4,5,6].

Would you like to give it a try?  If so, we can start with
'guix-package' and 'guix-build' because they are already documented.

[1] https://en.wikipedia.org/wiki/Literate_programming
[2] http://www-cs-faculty.stanford.edu/~uno/lp.html
[3] http://www.cs.tufts.edu/~nr/noweb/intro.html
[4] http://orgmode.org/worg/org-contrib/babel/intro.html#literate-programming
[5] http://orgmode.org/worg/org-contrib/babel/how-to-use-Org-Babel-for-R.html
[6] http://kimavcrp.blogspot.com/2012/05/literate-programming-in-clojure-table.html

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Literate programming

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> What do you think about literate programming [1,2,3]?

I don’t think it’s appropriate here.  We have (or should have) user doc
for command-line tools, a manual for the API, as well as an on-line
reference in the form of docstrings; the code itself is reasonably
commented, I think, and in a functional style that hopefully simplifies
reasoning a bit (I know there’s more to it than just writing functional
code, but still, it’s better than TeX macros ;-)).

WDYT?

Ludo’.

Re: Literate programming

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

>> What do you think about literate programming [1,2,3]?

> I don’t think it’s appropriate here.  We have (or should have) user doc
> for command-line tools, a manual for the API, as well as an on-line
> reference in the form of docstrings; the code itself is reasonably
> commented, I think, and in a functional style that hopefully simplifies
> reasoning a bit (I know there’s more to it than just writing functional
> code, but still, it’s better than TeX macros ;-)).

> WDYT?

Here is what I found yesterday: [1,2].  Those (texinfo ...) modules
provide various functions to generate html, pdf, and info from a single
source file.  If different files are used, it's easy to end up with
undocumented code [3] or to have a documentation for nonexistent
functions [4].

There might be some problems (I've only checked a couple of functions).
Though, I hope that we'll give it a try.

The only drawback I see right now is the need to adapt the existing
modules.  But I'm willing to do it.

[1] http://wingolog.org/archives/2004/07/25/literate-programming-with-guile-lib
[2] http://wingolog.org/archives/2006/11/17/high-on-sodium-vapor
[3] https://lists.gnu.org/archive/html/guile-user/2007-06/msg00046.html
[4] https://lists.gnu.org/archive/html/bug-guile/2013-04/msg00000.html

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Literate programming

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> Here is what I found yesterday: [1,2].  Those (texinfo ...) modules
> provide various functions to generate html, pdf, and info from a single
> source file.

[...]

> [1] http://wingolog.org/archives/2004/07/25/literate-programming-with-guile-lib
> [2] http://wingolog.org/archives/2006/11/17/high-on-sodium-vapor

Guile-Lib’s method was eventually integrated in Guile 2.0, to generate
the documentation of modules that were imported from Guile-Lib, such as
the texinfo and sxml modules.  The result, as can be seen in Guile’s
manual up to 2.0.7, was of poor quality compared to the rest of the
manual.

That’s why the upcoming 2.0.8 (due in a couple of days) got rid of
that.  NEWS reads this (written by Andy):

  ** An end to the generated-documentation experiment.

  When Guile 2.0 imported some modules from Guile-Lib, they came with a
  system that generated documentation from docstrings and module
  commentaries.  This produced terrible documentation.  We finally bit the
  bullet and incorporated these modules into the main text, and will be
  improving them manually over time, as is the case with SXML.  Help is
  appreciated.

So yes, you’re right that synchronizing code (and docstrings) with a
separate Texinfo manual can lead to discrepancies.  Yet, “good” manuals
can hardly be automatically generated.

For Guix, perhaps a middle ground approach could be used.  For instance,
we would keep the manual in its current form, but extract docstrings for
internal modules in separate .texi files, which would be @included in
the right places of the manual.  (It would be an improvement over the
current situation where almost none of the API documentation appears in
the manual.)  We would still write appropriate text to provide context
and cross-references above the raw function @deffn lists.

WDYT?

Thanks,
Ludo’.

Re: Literate programming

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

> The result, as can be seen in Guile’s manual up to 2.0.7, was of poor
> quality compared to the rest of the manual.

Are you talking about the pdf versions?  Could you show an example?

> We finally bit the bullet and incorporated these modules into the main
> text, and will be improving them manually over time, as is the case
> with SXML.

I don't understand this part.  Is it trying to say that it was necessary
to rewrite the relevant parts of the manual by hand?  Is it trying to
say that the sources were moved somewhere from Guile-Lib?  Is it about
the code or the manual?

> For Guix, perhaps a middle ground approach could be used.  For instance,
> we would keep the manual in its current form, but extract docstrings for
> internal modules in separate .texi files, which would be @included in
> the right places of the manual.  (It would be an improvement over the
> current situation where almost none of the API documentation appears in
> the manual.)

Good, I agree.

> We would still write appropriate text to provide context and
> cross-references above the raw function @deffn lists.

OK.  But I hope that the glue text won't be technical but philosophical
(to avoid mistakes).  For instance, it can explain the rationale or talk
about the GNU system.

Each module should be self-contained.  It should be possible to import
it somewhere without a problem.  Basically, the current version of the
manual follows the same approach; each module has its own node:
"Invoking guix-daemon," "Invoking guix package," and so forth.

So it's only necessary to represent this information using docstrings
and commentary (i.e., a module-level documentation).

Could you explore the possibilities of the literate programming system?
So we can identify the missing bits.  Of course, I'll do the same.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Literate programming

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

>> The result, as can be seen in Guile’s manual up to 2.0.7, was of poor
>> quality compared to the rest of the manual.
>
> Are you talking about the pdf versions?

It’s poor in terms of presentation, not in terms of typography.

> Could you show an example?

http://www.gnu.org/software/guile/manual/html_node/Standard-Library.html

(Watch it before 2.0.8 is out.)

>> We finally bit the bullet and incorporated these modules into the main
>> text, and will be improving them manually over time, as is the case
>> with SXML.
>
> I don't understand this part.  Is it trying to say that it was necessary
> to rewrite the relevant parts of the manual by hand?  Is it trying to
> say that the sources were moved somewhere from Guile-Lib?  Is it about
> the code or the manual?

It’s about the manual.  Initially, each of the sections above was
entirely extracted from the corresponding module: the module contained a
comment at the top that would give a rough overview, and the rest was
just taken from docstrings.

The result looked poor mostly because there is little introductory
material, few cross-references, and the modules and procedures are
listed in alphabetical order, which is rarely the right order to
introduce and connect concepts.

>> For Guix, perhaps a middle ground approach could be used.  For instance,
>> we would keep the manual in its current form, but extract docstrings for
>> internal modules in separate .texi files, which would be @included in
>> the right places of the manual.  (It would be an improvement over the
>> current situation where almost none of the API documentation appears in
>> the manual.)
>
> Good, I agree.
>
>> We would still write appropriate text to provide context and
>> cross-references above the raw function @deffn lists.
>
> OK.  But I hope that the glue text won't be technical but philosophical
> (to avoid mistakes).  For instance, it can explain the rationale or talk
> about the GNU system.

Yes, the extraction would be for the APIs, which are currently almost
undocumented in the manual.

> Each module should be self-contained.  It should be possible to import
> it somewhere without a problem.  Basically, the current version of the
> manual follows the same approach; each module has its own node:
> "Invoking guix-daemon," "Invoking guix package," and so forth.

These are not modules, these are commands.

I wouldn’t want to automatically extract command-line doc.  First,
because --help needs to be concise, whereas the manual can give
additional details (compare ‘guix package --help’ and “Invoking guix
packages”.)  Second, because the manual should include cross-references
and examples to make things more understandable.

> So it's only necessary to represent this information using docstrings
> and commentary (i.e., a module-level documentation).
>
> Could you explore the possibilities of the literate programming system?
> So we can identify the missing bits.  Of course, I'll do the same.

What I have in mind is just API documentation.  This is easily done by
traversing the exported bindings, and calling ‘procedure-documentation’.

There’s a complication: currently docstrings in Guile must be Info-style
plain text, not Texinfo (this may change in the future.)  So one would
need to infer Texinfo markup from the docstrings.

Anyway, a clarification: I find it OK to extract docstrings today,
because it’s better than nothing; but to me, a good manual should be
more than a mere paragraph followed by a list of docstrings.

Thanks,
Ludo’.

Re: Literate programming

[Nikita Karetnikov]

[-- Attachment #1.1: Type: text/plain, Size: 1530 bytes --]

> I wouldn’t want to automatically extract command-line doc.  First,
> because --help needs to be concise, whereas the manual can give
> additional details (compare ‘guix package --help’ and “Invoking guix
> packages”.)  Second, because the manual should include cross-references
> and examples to make things more understandable.

Here is how I see the process: one creates a command-line tool and uses
Commentary to document it.  Then the person should manually @include the
output of extracted and converted Commentary (e.g., 'filename.texi'),
which isn't created at this point.  After that it's only necessary to
run 'make' to compile 'filename.texi' and 'guix.texi'.  (Note that it's
possible to use cross-references.)

I've created a rough example.  'doc/hash-texi.scm' shows how to extract
and convert Commentary.  The same thing should be done for all scripts.
So each 'file.scm' should have a corresponding 'file.texi'.  Other files
in 'guix' will need a different extraction scheme because it's necessary
to get docstrings as well.  I'd like to write a couple of procedures and
put them into a makefile.  Still, it'll be necessary to @include files
manually.

I guess this sounds like a lot of work, but the same approach is also
used with LaTeX [1].  You have a main file, a style file, and other files
which will be included into the main file.

I'd also like to find a way to automatically create a table of contents.

[1] https://en.wikibooks.org/wiki/LaTeX/Modular_Documents


[-- Attachment #1.2: autodoc.diff --]
[-- Type: text/x-diff, Size: 5287 bytes --]

diff --git a/doc/guix.texi b/doc/guix.texi
index 9147f43..17f7e14 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -1214,36 +1214,7 @@ the @code{package-derivation} procedure of the @code{(guix packages)}
 module, and to the @code{build-derivations} procedure of the @code{(guix
 store)} module.
 
-@node Invoking guix hash
-@section Invoking @command{guix hash}
-
-The @command{guix hash} command allows to check the integrity of a file.
-It is primarily a convenience tool for anyone contributing to the
-distribution: it computes the cryptographic hash of a file, which can be
-used in the definition of a package (@pxref{Defining Packages}).
-
-The general syntax is:
-
-@example
-guix hash @var{option} @var{file}
-@end example
-
-@command{guix hash} has the following option:
-
-@table @code
-
-@item --format=@var{fmt}
-@itemx -f @var{fmt}
-Write the hash in the given format.
-
-Supported formats: @code{nix-base32}, @code{base32}, @code{base16}
-(@code{hex} and @code{hexadecimal} can be used as well).
-
-If the @option{--format} option is not specified, @command{guix hash}
-will output the hash in @code{nix-base32}.  This representation is used
-in the definitions of packages.
-
-@end table
+@include hash.texi
 
 @node Invoking guix refresh
 @section Invoking @command{guix refresh}
diff --git a/doc/hash-texi.scm b/doc/hash-texi.scm
new file mode 100644
index 0000000..801f92f
--- /dev/null
+++ b/doc/hash-texi.scm
@@ -0,0 +1,29 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(use-modules (texinfo serialize)
+             (ice-9 session))
+
+;; 'string->stexi' and 'module-commentary' are used instead of
+;; 'module-stexi-documentation' because we do not want to mention 'guix-hash'
+;; and the names of sections.
+(call-with-output-file "doc/hash.texi"
+  (lambda (out)
+    (format out "~a~%"
+            (stexi->texi ((@@ (texinfo reflection) string->stexi)
+                          (module-commentary '(guix scripts hash)))))))
\ No newline at end of file
diff --git a/doc/hash.texi b/doc/hash.texi
new file mode 100644
index 0000000..a6469da
--- /dev/null
+++ b/doc/hash.texi
@@ -0,0 +1,35 @@
+
+@c %start of fragment
+
+@node Invoking guix hash
+@section Invoking @command{guix hash}
+The @command{guix hash} command allows to check the integrity of a file.
+It is primarily a convenience tool for anyone contributing to the
+distribution: it computes the cryptographic hash of a file, which can be
+used in the definition of a package (@pxref{Defining Packages}).
+
+The general syntax is:
+
+@example 
+ guix hash @var{option} @var{file}
+@end example
+
+@command{guix hash} has the following option:
+
+@table @code
+@item --format=@var{fmt}
+@itemx -f @var{fmt}
+Write the hash in the given format.
+
+Supported formats: @code{nix-base32}, @code{base32}, @code{base16}
+(@code{hex} and @code{hexadecimal} can be used as well).
+
+If the @option{--format} option is not specified, @command{guix hash}
+will output the hash in @code{nix-base32}.  This representation is used
+in the definitions of packages.
+
+@end table
+
+
+@c %end of fragment
+
diff --git a/guix/scripts/hash.scm b/guix/scripts/hash.scm
index deded63..41346ac 100644
--- a/guix/scripts/hash.scm
+++ b/guix/scripts/hash.scm
@@ -30,6 +30,41 @@
     #:export (guix-hash))
 
 \f
+;;; Commentary:
+;;
+;; @node Invoking guix hash
+;; @section Invoking @command{guix hash}
+;;
+;; The @command{guix hash} command allows to check the integrity of a file.
+;; It is primarily a convenience tool for anyone contributing to the
+;; distribution: it computes the cryptographic hash of a file, which can be
+;; used in the definition of a package (@pxref{Defining Packages}).
+;;
+;; The general syntax is:
+;;
+;; @example
+;; guix hash @var{option} @var{file}
+;; @end example
+;;
+;; @command{guix hash} has the following option:
+;;
+;; @table @code
+;;
+;; @item --format=@var{fmt}
+;; @itemx -f @var{fmt}
+;; Write the hash in the given format.
+;;
+;; Supported formats: @code{nix-base32}, @code{base32}, @code{base16}
+;; (@code{hex} and @code{hexadecimal} can be used as well).
+;;
+;; If the @option{--format} option is not specified, @command{guix hash}
+;; will output the hash in @code{nix-base32}.  This representation is used
+;; in the definitions of packages.
+;;
+;; @end table
+;;
+;;; Code:
+
 ;;;
 ;;; Command-line options.
 ;;;

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Literate programming

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> --- a/guix/scripts/hash.scm
> +++ b/guix/scripts/hash.scm
> @@ -30,6 +30,41 @@
>      #:export (guix-hash))
>  
>  \f
> +;;; Commentary:
> +;;
> +;; @node Invoking guix hash
> +;; @section Invoking @command{guix hash}
> +;;
> +;; The @command{guix hash} command allows to check the integrity of a file.
> +;; It is primarily a convenience tool for anyone contributing to the
> +;; distribution: it computes the cryptographic hash of a file, which can be
> +;; used in the definition of a package (@pxref{Defining Packages}).

Honestly I’m reluctant to this approach.  First because currently
“commentary” is supposed to plain text, so using markup there would be
an inconvenience (for instance, when viewing it in Geiser, it doesn’t
get automatically converted from Texinfo.)

Second, it doesn’t buy us anything in terms of automation.  It even
requires some thought to get the build system right (remember that the
GCS mandate that .info files be distributed with tarballs.)

Furthermore, this is really what the failed experiment in Guile did.

Lastly, info "(standards) Doc Strings and Manuals":

     The only good way to use documentation strings in writing a good
  manual is to use them as a source of information for writing good text.

Sorry for being so unenthusiastic.

Ludo’.