[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

* doc/contributing.texi (Source Tree Structure): New node.
* doc/guix.texi (Programming Interface): Add cross-reference.

Change-Id: I141a1f4d806ae5f72c7a246e18c14dc63056a607
---
 doc/contributing.texi | 235 +++++++++++++++++++++++++++++++++++++++++-
 doc/guix.texi         |   3 +-
 2 files changed, 236 insertions(+), 2 deletions(-)

Hello there!

This is a long overdue section for contributors, very much inspired
by Josselin's talk last year:

  https://10years.guix.gnu.org/video/guixy-guile-the-derivation-factory-a-tour-of-the-guix-source-tree/

Thoughts?

Ludo'.

diff --git a/doc/contributing.texi b/doc/contributing.texi
index a7d91724fb..ff7065ad2a 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -23,7 +23,8 @@ Contributing
 * Building from Git::           The latest and greatest.
 * Running Guix Before It Is Installed::  Hacker tricks.
 * The Perfect Setup::           The right tools.
-* Alternative Setups::          Other posible tools that do the job.
+* Alternative Setups::          Other possible tools that do the job.
+* Source Tree Structure::       Source code guided tour.
 * Packaging Guidelines::        Growing the distribution.
 * Coding Style::                Hygiene of the contributor.
 * Submitting Patches::          Share your work.
@@ -546,6 +547,238 @@ Vim and NeoVim
 process and inject your code there live (sadly it's not packaged in Guix yet).
 
 
+@node Source Tree Structure
+@section Source Tree Structure
+
+@cindex structure, of the source tree
+Now that we're all set, let's look at the source tree.  Overall, the
+Guix source tree contains almost exclusively Guile @dfn{modules}, each
+of which can be seen as an independent library (@pxref{Modules,,, guile,
+GNU Guile Reference Manual}).
+
+The following table gives an overview of the main directories and what
+they contain.  Remember that in Guile, each module name is derived from
+its file name---e.g., the module in file@file{guix/packages.scm} is
+called @code{(guix packages)}.
+
+@table @file
+@item guix
+This is the location of core Guix mechanisms.  A few examples:
+
+@table @code
+@item (guix store)
+Connecting to and interacting with the build daemon (@pxref{The Store}).
+@item (guix derivations)
+Creating derivations (@pxref{Derivations}).
+@item (guix gexps)
+Writing G-expressions (@pxref{G-Expressions}).
+@item (guix packages)
+Defining packages and origins (@pxref{package Reference}).
+@item (guix download)
+@itemx (guix git-download)
+The @code{url-fetch} and @code{git-fetch} origin download methods
+(@pxref{origin Reference}).
+@item (guix swh)
+Fetching source code from the
+@uref{https://archive.softwareheritage.org,Software Heritage archive}.
+@item (guix search-paths)
+Implementing search paths (@pxref{Search Paths}).
+@item (guix build-system)
+The build system interface (@pxref{Build Systems}).
+@item (guix profiles)
+Implementing profiles.
+@end table
+
+@cindex build system, directory structure
+@item guix/build-system
+This directory contains specific build system implementations
+(@pxref{Build Systems}), such as:
+
+@table @code
+@item (guix build-system gnu)
+the GNU build system;
+@item (guix build-system cmake)
+the CMake build system;
+@item (guix build-system pyproject)
+The Python ``pyproject'' build system.
+@end table
+
+@item guix/build
+This contains code generally used on the ``build side''
+(@pxref{G-Expressions, strata of code}).  This includes code used to
+build packages or other operating system components, as well as
+utilities:
+
+@table @code
+@item (guix build utils)
+Utilities for package definitions and more (@pxref{Build Utilities}).
+@item (guix build gnu-build-system)
+@itemx (guix build cmake-build-system)
+@itemx (guix build pyproject-build-system)
+Implementation of build systems, and in particular definition of their
+build phases (@pxref{Build Phases}).
+@item (guix build syscalls)
+Interface to the C library and to Linux system calls.
+@end table
+
+@cindex command-line tools, as Guile modules
+@cindex command modules
+@item guix/scripts
+This contains modules corresponding to @command{guix} sub-commands.  For
+example, the @code{(guix scripts shell)} module exports the
+@code{guix-shell} procedure, which directly corresponds to the
+@command{guix shell} command (@pxref{Invoking guix shell}).
+
+@cindex importer modules
+@item guix/import
+This contains supporting code for the importers and updaters
+(@pxref{Invoking guix import}, and @pxref{Invoking guix refresh}).  For
+example, @code{(guix import pypi)} defines the interface to PyPI, which
+is used by the @code{guix import pypi} command.
+@end table
+
+The directories we have seen so far all live under @file{guix/}.  The
+other important place is the @code{gnu/} directory, which contains
+primarily package definitions as well as libraries and tools for Guix
+System (@pxref{System Configuration}) and Guix Home (@pxref{Home
+Configuration}).
+
+@table @file
+@cindex package modules
+@item gnu/packages
+This is by far the most crowded directory of the source tree: it
+contains @dfn{package modules} that export package definitions
+(@pxref{Package Modules}).  A few examples:
+
+@table @code
+@item (gnu packages base)
+Module providing ``base'' packages: @code{glibc}, @code{coreutils},
+@code{grep}, etc.
+@item (gnu packages guile)
+Guile and core Guile packages.
+@item (gnu packages linux)
+The Linux-libre kernel and related packages.
+@item (gnu packages python)
+Python and core Python packages.
+@item (gnu packages python-xyz)
+Miscellaneous Python packages (we were not very creative).
+@end table
+
+In any case, you can jump to a package definition using @command{guix
+edit} (@pxref{Invoking guix edit}) and view its location with
+@command{guix show} (@pxref{Invoking guix package}).
+
+@findex search-patches
+@item gnu/packages/patches
+This directory contains patches applied against packages and obtained
+using the @code{search-patches} procedure.
+
+@item gnu/services
+This contains service definitions, primarily for Guix System
+(@pxref{Services}) but some of them are adapted and reused for Guix Home
+as we will see below.  Examples:
+
+@table @code
+@item (gnu services)
+The service framework itself, which defines the service and service type
+data types (@pxref{Service Composition}).
+@item (gnu services base)
+``Base'' services (@pxref{Base Services}).
+@item (gnu services desktop)
+``Desktop'' services (@pxref{Desktop Services}).
+@item (gnu services shepherd)
+Support for Shepherd services (@pxref{Shepherd Services}).
+@end table
+
+You can jump to a service definition using @command{guix system edit}
+and view its location with @command{guix system search} (@pxref{Invoking
+guix system}).
+
+@item gnu/system
+These are core Guix System modules, such as:
+
+@table @code
+@item (gnu system)
+Defines @code{operating-system} (@pxref{operating-system Reference}).
+@item (gnu system file-systems)
+Defines @code{file-system} (@pxref{File Systems}).
+@item (gnu system mapped-devices)
+Defines @code{mapped-device} (@pxref{Mapped Devices}).
+@end table
+
+@item gnu/build
+These are modules that are either used on the ``build side'' when
+building operating systems or packages, or at run time by operating
+systems.
+
+@table @code
+@item (gnu build accounts)
+Creating @file{/etc/passwd}, @file{/etc/shadow}, etc. (@pxref{User
+Accounts}).
+@item (gnu build activation)
+Activating an operating system at boot time or reconfiguration time.
+@item (gnu build file-systems)
+Searching, checking, and mounting file systems.
+@item (gnu build linux-boot)
+@itemx (gnu build hurd-boot)
+Booting GNU/Linux and GNU/Hurd operating systems.
+@item (gnu build linux-initrd)
+Creating a Linux initial RAM disk (@pxref{Initial RAM Disk}).
+@end table
+
+@item gnu/home
+This contains all things Guix Home (@pxref{Home Configuration});
+examples:
+
+@table @code
+@item (gnu home services)
+Core services such as @code{home-files-service-type}.
+@item (gnu home services ssh)
+SSH-related services (@pxref{Secure Shell}).
+@end table
+
+@item gnu/installer
+This contains the text-mode graphical system installer (@pxref{Guided
+Graphical Installation}).
+
+@item gnu/machine
+These are the @dfn{machine abstractions} used by @command{guix deploy}
+(@pxref{Invoking guix deploy}).
+
+@item gnu/tests
+This contains system tests---tests that spawn virtual machines to check
+that system services work as expected (@pxref{Running the Test Suite}).
+@end table
+
+Last, there's also a few directories that contain files that are
+@emph{not} Guile modules:
+
+@table @file
+@item tests
+These are unit tests, each file corresponding more or less to one
+module, in particular @code{(guix @dots{})} modules (@pxref{Running the
+Test Suite}).
+
+@item doc
+This is the documentation in the form of Texinfo files: this manual and
+the Cookbook.
+
+@item po
+This is the location of translations of Guix itself, of package synopses
+and descriptions, of the manual, and of the cookbook (@pxref{Translating
+Guix}).
+
+@item etc
+Miscellaneous files: shell completions, support for systemd and other
+init systems, Git hooks, etc.
+@end table
+
+With all this, a fair chunk of your operating system is at your
+fingertips!  Beyond @command{grep} and @command{git grep}, @pxref{The
+Perfect Setup} on how to navigate code from your editor, and
+@pxref{Using Guix Interactively} for information on how to use
+interactively use Scheme modules.  Enjoy!
+
 @node Packaging Guidelines
 @section Packaging Guidelines
 
diff --git a/doc/guix.texi b/doc/guix.texi
index 87fe9f803c..fa26d78187 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7677,7 +7677,8 @@ Programming Interface
 that build results @emph{derive} from them.
 
 This chapter describes all these APIs in turn, starting from high-level
-package definitions.
+package definitions.  @xref{Source Tree Structure}, for a more general
+overview of the source code.
 
 @menu
 * Package Modules::             Packages from the programmer's viewpoint.

base-commit: 9d9bb8955a939b89b5b28f1071b70ed9f9a54f8c
-- 
2.41.0

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Hi Ludo.

Good that I now have finally watched Josselin’s talk.

This section seems to be a good idea in general, but the examples are an
opinionated selection.  I believe the list should be small.  Listing
individual modules directly in ~/src/guix/guix directory is not useful
to newcomers or often self-explanatory.  gnu/installer, gnu/system,
gnu/build, gnu/machine is not useful to know.  po is better not talked
about, otherwise people edit translations directly in po, when they
should use Weblate.

Ludovic Courtès <ludo@gnu.org> writes:
> +The following table gives an overview of the main directories and what
> +they contain.  Remember that in Guile, each module name is derived from
> +its file name---e.g., the module in file@file{guix/packages.scm} is

file@file should be @file


> +called @code{(guix packages)}.
> +
> +@table @file
> +@item guix
> +This is the location of core Guix mechanisms.  A few examples:

What does core mean?  Perhaps (guix …) should be listed after (gnu …)
and defined as the Guix mechanisms that do not belong in gnu?  Not quite
sure either.

> […]
> +With all this, a fair chunk of your operating system is at your
> +fingertips!  Beyond @command{grep} and @command{git grep}, @pxref{The
> +Perfect Setup} on how to navigate code from your editor, and
> +@pxref{Using Guix Interactively} for information on how to use
> +interactively use Scheme modules.  Enjoy!

Remove duplicate “use”.

Regards,
Florian

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

Hello!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> This section seems to be a good idea in general, but the examples are an
> opinionated selection.  I believe the list should be small.  Listing
> individual modules directly in ~/src/guix/guix directory is not useful
> to newcomers or often self-explanatory.  gnu/installer, gnu/system,
> gnu/build, gnu/machine is not useful to know.

Are you suggesting to remove the examples?

If that’s the case, we might as well not add the section or it would
become a bullet list with three items (guix/, gnu/, and tests/), no?

The selection is opinionated, indeed.  My thought was that it was a way
to connect the dots among things presented in the manual and to provide
a “guided tour” pretty much like Josselin’s talk.

> po is better not talked about, otherwise people edit translations
> directly in po, when they should use Weblate.

Not sure; there’s a link to “Translating Guix” that explains what to do.

>> +@table @file
>> +@item guix
>> +This is the location of core Guix mechanisms.  A few examples:
>
> What does core mean?

The examples were meant to illustrate what is meant by “core”.  Do you
think some other adjective or a longer description would help?

> Perhaps (guix …) should be listed after (gnu …)  and defined as the
> Guix mechanisms that do not belong in gnu?  Not quite sure either.

The order I chose is (roughly) from lower-level to higher-level:

(guix store) -> (guix derivation) -> (guix packages) -> …
… -> (gnu packages) -> (gnu system) -> …

Does that make sense?

Thanks for your feedback!

Ludo’.

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Hi Ludo.

Josselin’s talk is different in that it is a talk of more than 30
minutes.  In so much time, it can give more detailed guidance to almost
the whole guix source tree, even including build-aux and nix.  Josselin
also gives hints to use git grep (like you) but also to read the
commentary at the top of the file.  This may be a helpful hint to
someone starting out, but someone starting out maybe does not want to
read as much as a complete talk.  If they wanted it all, then better
link to Josselin’s talk.

Ludovic Courtès <ludo@gnu.org> writes:
> The order I chose is (roughly) from lower-level to higher-level:
>
> (guix store) -> (guix derivation) -> (guix packages) -> …
> … -> (gnu packages) -> (gnu system) -> …
>
> Does that make sense?

In your section the modules directly in (guix …) appeared unsorted to
me.  Could you explicitly state this order in the manual section?

Nice things like (guix swh) or (gnu system), (gnu build), (gnu
installer), (gnu machine), or po, still seem not useful for the general
populace to me.


> Are you suggesting to remove the examples?

I like tests and gnu/tests.  Also that your section lists (guix
build-system …) and the occasional duality with (guix build
…-build-system), Rightfully you also list non-dual utilities like (guix
build syscalls) to make clear not all in (guix build) is about build
systems.

The explanation about guix/scripts containing the entry points is
useful.

>> What does core mean?
>
> The examples were meant to illustrate what is meant by “core”.  Do you
> think some other adjective or a longer description would help?
>
>> Perhaps (guix …) should be listed after (gnu …)  and defined as the
>> Guix mechanisms that do not belong in gnu?  Not quite sure either.

Josselin called the distinction between (guix …) and (gnu …) murky,
explaining that most of (guix …) must not import (gnu …) except by
module-ref, while (guix scripts …) and such can just use-modules (gnu
…).  To me, gnu/packages.scm looks like core as well, but it rightfully
is in gnu.

Regards,
Florian

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]

Hello!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> Josselin’s talk is different in that it is a talk of more than 30
> minutes.  In so much time, it can give more detailed guidance to almost
> the whole guix source tree, even including build-aux and nix.  Josselin
> also gives hints to use git grep (like you) but also to read the
> commentary at the top of the file.  This may be a helpful hint to
> someone starting out, but someone starting out maybe does not want to
> read as much as a complete talk.  If they wanted it all, then better
> link to Josselin’s talk.

Right.

> Ludovic Courtès <ludo@gnu.org> writes:
>> The order I chose is (roughly) from lower-level to higher-level:
>>
>> (guix store) -> (guix derivation) -> (guix packages) -> …
>> … -> (gnu packages) -> (gnu system) -> …
>>
>> Does that make sense?
>
> In your section the modules directly in (guix …) appeared unsorted to
> me.  Could you explicitly state this order in the manual section?

Good idea, will do.

> Nice things like (guix swh) or (gnu system), (gnu build), (gnu
> installer), (gnu machine), or po, still seem not useful for the general
> populace to me.

This is in the “Contributing” chapter, so we’re talking about a subset
of the general populace.  :-)

You might argue that few current contributors care about the modules you
mention, but by exposing the structure of the code, my hope is that more
people would dare take a look and fiddle with it.

[...]

>> The examples were meant to illustrate what is meant by “core”.  Do you
>> think some other adjective or a longer description would help?
>>
>>> Perhaps (guix …) should be listed after (gnu …)  and defined as the
>>> Guix mechanisms that do not belong in gnu?  Not quite sure either.
>
> Josselin called the distinction between (guix …) and (gnu …) murky,
> explaining that most of (guix …) must not import (gnu …) except by
> module-ref, while (guix scripts …) and such can just use-modules (gnu
> …).  To me, gnu/packages.scm looks like core as well, but it rightfully
> is in gnu.

I think “murky” is a strong word, or at least it shouldn’t be
interpreted as meaning that the guix/gnu distinction is arbitrary.  I’ll
try to clarify that as well.

I was going to send a v2 but I’m not sure the changes I made fully
address your concerns:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1388 bytes --]

diff --git a/doc/contributing.texi b/doc/contributing.texi
index ff7065ad2a..18f3705a43 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -563,7 +563,9 @@ Source Tree Structure
 
 @table @file
 @item guix
-This is the location of core Guix mechanisms.  A few examples:
+This is the location of core Guix mechanisms.  To illustrate what is
+meant by ``core'', here are a few examples, starting from low-level
+tools and going towards higher-level tools:
 
 @table @code
 @item (guix store)
@@ -638,10 +640,16 @@ Source Tree Structure
 @end table
 
 The directories we have seen so far all live under @file{guix/}.  The
-other important place is the @code{gnu/} directory, which contains
+other important place is the @file{gnu/} directory, which contains
 primarily package definitions as well as libraries and tools for Guix
 System (@pxref{System Configuration}) and Guix Home (@pxref{Home
-Configuration}).
+Configuration}), all of which build upon functionality provided by
+@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
+@dots{})}  modules must generally not depend on @code{(gnu @dots{})}
+modules, with one notable exception: @code{(guix build-system @dots{})}
+modules may look up packages at run time---e.g., @code{(guix
+build-system cmake)} needs to access the @code{cmake} variable at run
+time.}.
 
 @table @file
 @cindex package modules

[-- Attachment #3: Type: text/plain, Size: 52 bytes --]


Let me know what you think!

Thanks,
Ludo’.

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Hi Ludo.

Ludovic Courtès <ludo@gnu.org> writes:
>> Nice things like (guix swh) or (gnu system), (gnu build), (gnu
>> installer), (gnu machine), or po, still seem not useful for the general
>> populace to me.
>
> This is in the “Contributing” chapter, so we’re talking about a subset
> of the general populace.  :-)
>
> You might argue that few current contributors care about the modules you
> mention, but by exposing the structure of the code, my hope is that more
> people would dare take a look and fiddle with it.

Your reply makes clear that emphasis on (guix swh) was intentional.  If
SWH is central to Guix, then you are right mentioning it.  I had not
recognized and only considered it a nice, well-integrated bonus.

Still I would prefer if (gnu system), (gnu build), (gnu installer), (gnu
machine), and especially po, were not part of the list.  I expect that
most contributors want to provide a package or (home) service with docs
and tests.  They will not customize the operating-system record type.

> [...]
>
>>> The examples were meant to illustrate what is meant by “core”.  Do you
>>> think some other adjective or a longer description would help?
>>>
>>>> Perhaps (guix …) should be listed after (gnu …)  and defined as the
>>>> Guix mechanisms that do not belong in gnu?  Not quite sure either.
>>
>> Josselin called the distinction between (guix …) and (gnu …) murky,
>> explaining that most of (guix …) must not import (gnu …) except by
>> module-ref, while (guix scripts …) and such can just use-modules (gnu
>> …).  To me, gnu/packages.scm looks like core as well, but it rightfully
>> is in gnu.
>
> I think “murky” is a strong word, or at least it shouldn’t be
> interpreted as meaning that the guix/gnu distinction is arbitrary.  I’ll
> try to clarify that as well.

Hmm what is the difference between, let’s say, (gnu packages) and (guix
package)?


> @@ -638,10 +640,16 @@ Source Tree Structure
>  @end table
>  
>  The directories we have seen so far all live under @file{guix/}.  The
> -other important place is the @code{gnu/} directory, which contains
> +other important place is the @file{gnu/} directory, which contains
>  primarily package definitions as well as libraries and tools for Guix
>  System (@pxref{System Configuration}) and Guix Home (@pxref{Home
> -Configuration}).
> +Configuration}), all of which build upon functionality provided by
> +@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
> +@dots{})}  modules must generally not depend on @code{(gnu @dots{})}
> +modules, with one notable exception: @code{(guix build-system @dots{})}
> +modules may look up packages at run time---e.g., @code{(guix
> +build-system cmake)} needs to access the @code{cmake} variable at run
> +time.}.

I think the (guix build-system @dots{}) never use (gnu …)?  scripts and
importers do.

Regards,
Florian

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3141 bytes --]

Hello Ludo'

thank you for this patch, I find it super useful

I don't have specific comments, just general ones.

Ludovic Courtès <ludo@gnu.org> writes:

> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

[...]

>> but someone starting out maybe does not want to read as much as a
>> complete talk.  If they wanted it all, then better link to Josselin’s
>> talk.
>
> Right.

I appreciate Josselin’s talk very much, but in general I prefer studying
written information instead of "attenting to" a talk; I guess I'm not
the only one :-)

[...]

>> Nice things like (guix swh) or (gnu system), (gnu build), (gnu
>> installer), (gnu machine), or po, still seem not useful for the general
>> populace to me.
>
> This is in the “Contributing” chapter, so we’re talking about a subset
> of the general populace.  :-)

I don't know what general populance is :-) but I agree with you that
explaining the structure of the code is interesting for /some/ people,
like me for example.

Also, please don't (dis)miss the usefulness of such documentation as a
sort of meta-literate programming, probably also useful for expert Guix
programmers too, IMHO

> You might argue that few current contributors care about the modules you
> mention, but by exposing the structure of the code, my hope is that more
> people would dare take a look and fiddle with it.

Also, we should not assume what are the insterests of most contributors
or in general of readers, since everyone have it's own interests and
goals in studying Guix.

A section like this one does not have to be "perfect" at first "shot",
it can be improved by expert Guix persons when they feel "inspired"; I
already find the patches you sent useful.

It's true that (Guile) programmers could study the source code to
understand how it works but knowing the general _architecture is very
helpful.

Last but not least, I have a feeling it's time to split the Guix manual
in two: one for users, like "GNU Guix User Manual" and one for
programmers [1], like "GNU Guix Programmer Manual", with chapters like
"Programming Interface", "Platforms", "Bootstrapping", "Porting",
"Contributing"... but that's another story, OT here.

[...]

>> Josselin called the distinction between (guix …) and (gnu …) murky,
>> explaining that most of (guix …) must not import (gnu …) except by
>> module-ref, while (guix scripts …) and such can just use-modules (gnu
>> …).  To me, gnu/packages.scm looks like core as well, but it rightfully
>> is in gnu.
>
> I think “murky” is a strong word, or at least it shouldn’t be
> interpreted as meaning that the guix/gnu distinction is arbitrary.  I’ll
> try to clarify that as well.

IMHO the very fact that Josselin find that distinction "murky"
(arbitrary? not well defined? not much clear?) is an indication that we
need a section like the one you are proposing :-D

[...]

Thank you for your work! Gio'


[1] ehrm... using Guix _is_ programming but well, I hope you got the
idea :-)

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Thank you Giovanni for voicing support; I’m afraid I sounded more
opposed than I am.

Giovanni Biscuolo <g@xelera.eu> writes:
> Also, please don't (dis)miss the usefulness of such documentation as a
> sort of meta-literate programming, probably also useful for expert Guix
> programmers too, IMHO

Do you think the following controversial parts should stay?  But what is
their audience, and why explain them but not the directories build-aux
and nix and some other not so important files in guix?

‘gnu/system’
     These are core Guix System modules, such as:

     ‘(gnu system)’
          Defines ‘operating-system’ (*note operating-system
          Reference::).
     ‘(gnu system file-systems)’
          Defines ‘file-system’ (*note File Systems::).
     ‘(gnu system mapped-devices)’
          Defines ‘mapped-device’ (*note Mapped Devices::).

‘gnu/build’
     These are modules that are either used on the “build side” when
     building operating systems or packages, or at run time by operating
     systems.

     ‘(gnu build accounts)’
          Creating ‘/etc/passwd’, ‘/etc/shadow’, etc.  (*note User
          Accounts::).
     ‘(gnu build activation)’
          Activating an operating system at boot time or reconfiguration
          time.
     ‘(gnu build file-systems)’
          Searching, checking, and mounting file systems.
     ‘(gnu build linux-boot)’
     ‘(gnu build hurd-boot)’
          Booting GNU/Linux and GNU/Hurd operating systems.
     ‘(gnu build linux-initrd)’
          Creating a Linux initial RAM disk (*note Initial RAM Disk::).

[…]
‘gnu/installer’
     This contains the text-mode graphical system installer (*note
     Guided Graphical Installation::).

‘gnu/machine’
     These are the “machine abstractions” used by ‘guix deploy’ (*note
     Invoking guix deploy::).

[…]
‘po’
     This is the location of translations of Guix itself, of package
     synopses and descriptions, of the manual, and of the cookbook
     (*note Translating Guix::).

> Last but not least, I have a feeling it's time to split the Guix manual
> in two: one for users, like "GNU Guix User Manual" and one for
> programmers [1], like "GNU Guix Programmer Manual", with chapters like
> "Programming Interface", "Platforms", "Bootstrapping", "Porting",
> "Contributing"... but that's another story, OT here.

Hmm why?  But yes, another story.

Regards,
Florian

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>>> Nice things like (guix swh) or (gnu system), (gnu build), (gnu
>>> installer), (gnu machine), or po, still seem not useful for the general
>>> populace to me.
>>
>> This is in the “Contributing” chapter, so we’re talking about a subset
>> of the general populace.  :-)
>>
>> You might argue that few current contributors care about the modules you
>> mention, but by exposing the structure of the code, my hope is that more
>> people would dare take a look and fiddle with it.

[...]

> Still I would prefer if (gnu system), (gnu build), (gnu installer), (gnu
> machine), and especially po, were not part of the list.  I expect that
> most contributors want to provide a package or (home) service with docs
> and tests.  They will not customize the operating-system record type.

I disagree here.  This section is intended for people willing to
contribute to Guix or to learn about it beyond packages (perhaps that
intention should be more clearly stated though; perhaps that’s the crux
of our difference of interpretation?).  I wouldn’t assume that this or
that part is not worthy.

If the section is deemed too long, it probably makes sense to trim it a
bit, but I don’t find it this long.

Or we can use different examples, though I would keep those that are
already documented elsewhere in the manual (like (gnu system)).

WDYT?

>> I think “murky” is a strong word, or at least it shouldn’t be
>> interpreted as meaning that the guix/gnu distinction is arbitrary.  I’ll
>> try to clarify that as well.
>
> Hmm what is the difference between, let’s say, (gnu packages) and (guix
> package)?

(guix packages) defines a <package> type and associated mechanisms (the
“package Reference” section).

(gnu packages) lets you browse packages defined in (gnu packages …),
etc.

The former is abstract; the latter is about concrete package
definitions.

>> +@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
>> +@dots{})}  modules must generally not depend on @code{(gnu @dots{})}
>> +modules, with one notable exception: @code{(guix build-system @dots{})}
>> +modules may look up packages at run time---e.g., @code{(guix
>> +build-system cmake)} needs to access the @code{cmake} variable at run
>> +time.}.
>
> I think the (guix build-system @dots{}) never use (gnu …)?

They do, as in the ‘cmake’ example above.

> scripts and importers do.

Oh right, that’s true.  So there’s more than one notable exception.  :-)

Ludo’.

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Hello, thank you for moving this to a resolution.

Ludovic Courtès <ludo@gnu.org> writes:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
>> Ludovic Courtès <ludo@gnu.org> writes:
>>>> Nice things like (guix swh) or (gnu system), (gnu build), (gnu
>>>> installer), (gnu machine), or po, still seem not useful for the general
>>>> populace to me.
>>>
>>> This is in the “Contributing” chapter, so we’re talking about a subset
>>> of the general populace.  :-)
>>>
>>> You might argue that few current contributors care about the modules you
>>> mention, but by exposing the structure of the code, my hope is that more
>>> people would dare take a look and fiddle with it.
>
> [...]
>
>> Still I would prefer if (gnu system), (gnu build), (gnu installer), (gnu
>> machine), and especially po, were not part of the list.  I expect that
>> most contributors want to provide a package or (home) service with docs
>> and tests.  They will not customize the operating-system record type.
>
> This section is intended for people willing to
> contribute to Guix or to learn about it beyond packages (perhaps that
> intention should be more clearly stated though; perhaps that’s the crux
> of our difference of interpretation?).

This is the misunderstanding.  It would help if the audience is clear,
so other readers can skip the section.


> If the section is deemed too long, it probably makes sense to trim it a
> bit, but I don’t find it this long.
>
> Or we can use different examples, though I would keep those that are
> already documented elsewhere in the manual (like (gnu system)).
>
> WDYT?

Okay, people might be curious about directories and therefore look at
these not immediately important directories.  Then the reason the
directory nix is not talked about is that we seek to get rid of nix?

That there are other sections is not a good reason, however.  But it
also does not seem like it was your criterion of inclusion.

> ‘po’
>      This is the location of translations of Guix itself, of package
>      synopses and descriptions, of the manual, and of the cookbook
>      (*note Translating Guix::).

Could you mention directly that translations are pulled from Weblate?



>>> I think “murky” is a strong word, or at least it shouldn’t be
>>> interpreted as meaning that the guix/gnu distinction is arbitrary.  I’ll
>>> try to clarify that as well.
>>
>> Hmm what is the difference between, let’s say, (gnu packages) and (guix
>> package)?
>
> (guix packages) defines a <package> type and associated mechanisms (the
> “package Reference” section).
>
> (gnu packages) lets you browse packages defined in (gnu packages …),
> etc.
>
> The former is abstract; the latter is about concrete package
> definitions.

I see, but this is unlike (gnu system), which is equally abstract.
There is a tendency, but case-by-case it seems murky.


>>> +@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
>>> +@dots{})}  modules must generally not depend on @code{(gnu @dots{})}
>>> +modules, with one notable exception: @code{(guix build-system @dots{})}
>>> +modules may look up packages at run time---e.g., @code{(guix
>>> +build-system cmake)} needs to access the @code{cmake} variable at run
>>> +time.}.
>>
>> I think the (guix build-system @dots{}) never use (gnu …)?
>
> They do, as in the ‘cmake’ example above.

Only by module-ref.


>
>> scripts and importers do.
>
> Oh right, that’s true.  So there’s more than one notable exception.  :-)
>
> Ludo’.

Regards,
Florian

[bug#69587] [PATCH] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

Hello!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> This section is intended for people willing to
>> contribute to Guix or to learn about it beyond packages (perhaps that
>> intention should be more clearly stated though; perhaps that’s the crux
>> of our difference of interpretation?).
>
> This is the misunderstanding.  It would help if the audience is clear,
> so other readers can skip the section.

Great, I’ll do that.

>> If the section is deemed too long, it probably makes sense to trim it a
>> bit, but I don’t find it this long.
>>
>> Or we can use different examples, though I would keep those that are
>> already documented elsewhere in the manual (like (gnu system)).
>>
>> WDYT?
>
> Okay, people might be curious about directories and therefore look at
> these not immediately important directories.  Then the reason the
> directory nix is not talked about is that we seek to get rid of nix?

Very good point, I’ll add it (did my subconscious want me to forget the
project’s roots?).

>> ‘po’
>>      This is the location of translations of Guix itself, of package
>>      synopses and descriptions, of the manual, and of the cookbook
>>      (*note Translating Guix::).
>
> Could you mention directly that translations are pulled from Weblate?

Sure.

>>> Hmm what is the difference between, let’s say, (gnu packages) and (guix
>>> package)?
>>
>> (guix packages) defines a <package> type and associated mechanisms (the
>> “package Reference” section).
>>
>> (gnu packages) lets you browse packages defined in (gnu packages …),
>> etc.
>>
>> The former is abstract; the latter is about concrete package
>> definitions.
>
> I see, but this is unlike (gnu system), which is equally abstract.
> There is a tendency, but case-by-case it seems murky.

Yeah okay, maybe.  :-)

I’ll work on a new version.

Thanks for your patience,
Ludo’.

[bug#69587] [PATCH v2] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

* doc/contributing.texi (Source Tree Structure): New node.
* doc/guix.texi (Programming Interface): Add cross-reference.

Change-Id: I141a1f4d806ae5f72c7a246e18c14dc63056a607
---
 doc/contributing.texi | 252 +++++++++++++++++++++++++++++++++++++++++-
 doc/guix.texi         |   3 +-
 2 files changed, 253 insertions(+), 2 deletions(-)

Hi!

Changes since v1:

  • Clarify the intended audience upfront.

  • Mention the nix/ directory.

  • Explain that .po files are taken from Weblate.

  • Explain what “core mechanisms” means.

  • Clarify guix/gnu module name space distinction.

Ludo’.

diff --git a/doc/contributing.texi b/doc/contributing.texi
index a7d91724fb..17ecf4a04c 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -23,7 +23,8 @@ Contributing
 * Building from Git::           The latest and greatest.
 * Running Guix Before It Is Installed::  Hacker tricks.
 * The Perfect Setup::           The right tools.
-* Alternative Setups::          Other posible tools that do the job.
+* Alternative Setups::          Other possible tools that do the job.
+* Source Tree Structure::       Source code guided tour.
 * Packaging Guidelines::        Growing the distribution.
 * Coding Style::                Hygiene of the contributor.
 * Submitting Patches::          Share your work.
@@ -546,6 +547,255 @@ Vim and NeoVim
 process and inject your code there live (sadly it's not packaged in Guix yet).
 
 
+@node Source Tree Structure
+@section Source Tree Structure
+
+@cindex structure, of the source tree
+If you're willing to contribute to Guix beyond packages, or if you'd
+like to learn how it all fits together, this section provides a guided
+tour in the code base that you may find useful.
+
+Overall, the Guix source tree contains almost exclusively Guile
+@dfn{modules}, each of which can be seen as an independent library
+(@pxref{Modules,,, guile, GNU Guile Reference Manual}).
+
+The following table gives an overview of the main directories and what
+they contain.  Remember that in Guile, each module name is derived from
+its file name---e.g., the module in file@file{guix/packages.scm} is
+called @code{(guix packages)}.
+
+@table @file
+@item guix
+This is the location of core Guix mechanisms.  To illustrate what is
+meant by ``core'', here are a few examples, starting from low-level
+tools and going towards higher-level tools:
+
+@table @code
+@item (guix store)
+Connecting to and interacting with the build daemon (@pxref{The Store}).
+@item (guix derivations)
+Creating derivations (@pxref{Derivations}).
+@item (guix gexps)
+Writing G-expressions (@pxref{G-Expressions}).
+@item (guix packages)
+Defining packages and origins (@pxref{package Reference}).
+@item (guix download)
+@itemx (guix git-download)
+The @code{url-fetch} and @code{git-fetch} origin download methods
+(@pxref{origin Reference}).
+@item (guix swh)
+Fetching source code from the
+@uref{https://archive.softwareheritage.org,Software Heritage archive}.
+@item (guix search-paths)
+Implementing search paths (@pxref{Search Paths}).
+@item (guix build-system)
+The build system interface (@pxref{Build Systems}).
+@item (guix profiles)
+Implementing profiles.
+@end table
+
+@cindex build system, directory structure
+@item guix/build-system
+This directory contains specific build system implementations
+(@pxref{Build Systems}), such as:
+
+@table @code
+@item (guix build-system gnu)
+the GNU build system;
+@item (guix build-system cmake)
+the CMake build system;
+@item (guix build-system pyproject)
+The Python ``pyproject'' build system.
+@end table
+
+@item guix/build
+This contains code generally used on the ``build side''
+(@pxref{G-Expressions, strata of code}).  This includes code used to
+build packages or other operating system components, as well as
+utilities:
+
+@table @code
+@item (guix build utils)
+Utilities for package definitions and more (@pxref{Build Utilities}).
+@item (guix build gnu-build-system)
+@itemx (guix build cmake-build-system)
+@itemx (guix build pyproject-build-system)
+Implementation of build systems, and in particular definition of their
+build phases (@pxref{Build Phases}).
+@item (guix build syscalls)
+Interface to the C library and to Linux system calls.
+@end table
+
+@cindex command-line tools, as Guile modules
+@cindex command modules
+@item guix/scripts
+This contains modules corresponding to @command{guix} sub-commands.  For
+example, the @code{(guix scripts shell)} module exports the
+@code{guix-shell} procedure, which directly corresponds to the
+@command{guix shell} command (@pxref{Invoking guix shell}).
+
+@cindex importer modules
+@item guix/import
+This contains supporting code for the importers and updaters
+(@pxref{Invoking guix import}, and @pxref{Invoking guix refresh}).  For
+example, @code{(guix import pypi)} defines the interface to PyPI, which
+is used by the @code{guix import pypi} command.
+@end table
+
+The directories we have seen so far all live under @file{guix/}.  The
+other important place is the @file{gnu/} directory, which contains
+primarily package definitions as well as libraries and tools for Guix
+System (@pxref{System Configuration}) and Guix Home (@pxref{Home
+Configuration}), all of which build upon functionality provided by
+@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
+@dots{})}  modules must generally not depend on @code{(gnu @dots{})}
+modules, with one notable exception: @code{(guix build-system @dots{})}
+modules may look up packages at run time---e.g., @code{(guix
+build-system cmake)} needs to access the @code{cmake} variable at run
+time.}.
+
+@table @file
+@cindex package modules
+@item gnu/packages
+This is by far the most crowded directory of the source tree: it
+contains @dfn{package modules} that export package definitions
+(@pxref{Package Modules}).  A few examples:
+
+@table @code
+@item (gnu packages base)
+Module providing ``base'' packages: @code{glibc}, @code{coreutils},
+@code{grep}, etc.
+@item (gnu packages guile)
+Guile and core Guile packages.
+@item (gnu packages linux)
+The Linux-libre kernel and related packages.
+@item (gnu packages python)
+Python and core Python packages.
+@item (gnu packages python-xyz)
+Miscellaneous Python packages (we were not very creative).
+@end table
+
+In any case, you can jump to a package definition using @command{guix
+edit} (@pxref{Invoking guix edit}) and view its location with
+@command{guix show} (@pxref{Invoking guix package}).
+
+@findex search-patches
+@item gnu/packages/patches
+This directory contains patches applied against packages and obtained
+using the @code{search-patches} procedure.
+
+@item gnu/services
+This contains service definitions, primarily for Guix System
+(@pxref{Services}) but some of them are adapted and reused for Guix Home
+as we will see below.  Examples:
+
+@table @code
+@item (gnu services)
+The service framework itself, which defines the service and service type
+data types (@pxref{Service Composition}).
+@item (gnu services base)
+``Base'' services (@pxref{Base Services}).
+@item (gnu services desktop)
+``Desktop'' services (@pxref{Desktop Services}).
+@item (gnu services shepherd)
+Support for Shepherd services (@pxref{Shepherd Services}).
+@end table
+
+You can jump to a service definition using @command{guix system edit}
+and view its location with @command{guix system search} (@pxref{Invoking
+guix system}).
+
+@item gnu/system
+These are core Guix System modules, such as:
+
+@table @code
+@item (gnu system)
+Defines @code{operating-system} (@pxref{operating-system Reference}).
+@item (gnu system file-systems)
+Defines @code{file-system} (@pxref{File Systems}).
+@item (gnu system mapped-devices)
+Defines @code{mapped-device} (@pxref{Mapped Devices}).
+@end table
+
+@item gnu/build
+These are modules that are either used on the ``build side'' when
+building operating systems or packages, or at run time by operating
+systems.
+
+@table @code
+@item (gnu build accounts)
+Creating @file{/etc/passwd}, @file{/etc/shadow}, etc. (@pxref{User
+Accounts}).
+@item (gnu build activation)
+Activating an operating system at boot time or reconfiguration time.
+@item (gnu build file-systems)
+Searching, checking, and mounting file systems.
+@item (gnu build linux-boot)
+@itemx (gnu build hurd-boot)
+Booting GNU/Linux and GNU/Hurd operating systems.
+@item (gnu build linux-initrd)
+Creating a Linux initial RAM disk (@pxref{Initial RAM Disk}).
+@end table
+
+@item gnu/home
+This contains all things Guix Home (@pxref{Home Configuration});
+examples:
+
+@table @code
+@item (gnu home services)
+Core services such as @code{home-files-service-type}.
+@item (gnu home services ssh)
+SSH-related services (@pxref{Secure Shell}).
+@end table
+
+@item gnu/installer
+This contains the text-mode graphical system installer (@pxref{Guided
+Graphical Installation}).
+
+@item gnu/machine
+These are the @dfn{machine abstractions} used by @command{guix deploy}
+(@pxref{Invoking guix deploy}).
+
+@item gnu/tests
+This contains system tests---tests that spawn virtual machines to check
+that system services work as expected (@pxref{Running the Test Suite}).
+@end table
+
+Last, there's also a few directories that contain files that are
+@emph{not} Guile modules:
+
+@table @file
+@item nix
+This is the C++ implementation of @command{guix-daemon}, inherited from
+Nix (@pxref{Invoking guix-daemon}).
+
+@item tests
+These are unit tests, each file corresponding more or less to one
+module, in particular @code{(guix @dots{})} modules (@pxref{Running the
+Test Suite}).
+
+@item doc
+This is the documentation in the form of Texinfo files: this manual and
+the Cookbook.  @xref{Writing a Texinfo File,,, texinfo, GNU Texinfo},
+for information on Texinfo markup language.
+
+@item po
+This is the location of translations of Guix itself, of package synopses
+and descriptions, of the manual, and of the cookbook.  Note that
+@file{.po} files that live here are pulled directly from Weblate
+(@pxref{Translating Guix}).`
+
+@item etc
+Miscellaneous files: shell completions, support for systemd and other
+init systems, Git hooks, etc.
+@end table
+
+With all this, a fair chunk of your operating system is at your
+fingertips!  Beyond @command{grep} and @command{git grep}, @pxref{The
+Perfect Setup} on how to navigate code from your editor, and
+@pxref{Using Guix Interactively} for information on how to use
+interactively use Scheme modules.  Enjoy!
+
 @node Packaging Guidelines
 @section Packaging Guidelines
 
diff --git a/doc/guix.texi b/doc/guix.texi
index 20f007b1c0..138d53602f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7680,7 +7680,8 @@ Programming Interface
 that build results @emph{derive} from them.
 
 This chapter describes all these APIs in turn, starting from high-level
-package definitions.
+package definitions.  @xref{Source Tree Structure}, for a more general
+overview of the source code.
 
 @menu
 * Package Modules::             Packages from the programmer's viewpoint.

base-commit: b7eb1a8116b2caee7acf26fb963ae998fbdb4253
-- 
2.41.0

[bug#69587] [PATCH v2] doc: Add “Source Tree Structure” section.

[pelzflorian (Florian Pelz)]

Hello, some small things to fix:

Ludovic Courtès <ludo@gnu.org> writes:
> +The following table gives an overview of the main directories and what
> +they contain.  Remember that in Guile, each module name is derived from
> +its file name---e.g., the module in file@file{guix/packages.scm}

The file@file should be @file, I presume.



> […]
> +The directories we have seen so far all live under @file{guix/}.  The
> +other important place is the @file{gnu/} directory, which contains
> +primarily package definitions as well as libraries and tools for Guix
> +System (@pxref{System Configuration}) and Guix Home (@pxref{Home
> +Configuration}), all of which build upon functionality provided by
> +@code{(guix @dots{})} modules@footnote{For this reason, @code{(guix
> +@dots{})}  modules

The double space is inappropriate.


> must generally not depend on @code{(gnu @dots{})}
> +modules, with one notable exception:

Perhaps better mention all three exceptions, not only build-system but
also direct #:use-module in scripts like pack and to a lesser degree
import.


> +@item po
> +This is the location of translations of Guix itself, of package synopses
> +and descriptions, of the manual, and of the cookbook.  Note that
> +@file{.po} files that live here are pulled directly from Weblate
> +(@pxref{Translating Guix}).`

At the end is a stray `.


> +With all this, a fair chunk of your operating system is at your
> +fingertips!  Beyond @command{grep} and @command{git grep}, @pxref{The
> +Perfect Setup} on how to navigate code from your editor, and
> +@pxref{Using Guix Interactively} for information on how to use
> +interactively use Scheme modules.  Enjoy!

The word “use” is duplicated.

Otherwise I’m happy and it LGTM.  Thank you!

Regards,
Florian

bug#69587: [PATCH v2] doc: Add “Source Tree Structure” section.

[Ludovic Courtès]

Hi,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> Otherwise I’m happy and it LGTM.  Thank you!

Fixed all the typos (oops!), added more exceptions in the guix/gnu
footnote, and pushed as d67e4f0f9b10c7ddac8fb0ca68cbf1d6ad0a6e5d.

Thanks again for your patience!

Ludo’.