[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[Attila Lendvai]

---
 doc/contributing.texi | 91 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index 207efc4ee6..d36b6e66e0 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -29,6 +29,7 @@ choice.
 * Tracking Bugs and Patches::   Keeping it all organized.
 * Commit Access::               Pushing to the official repository.
 * Updating the Guix Package::   Updating the Guix package definition.
+* Working on Shepherd::         Modifying and testing Shepherd.
 * Translating Guix::            Make Guix speak your native language.
 @end menu
 
@@ -1697,6 +1698,96 @@ This check can be disabled, @emph{at your own peril}, by setting the
 this variable is set, the updated package source is also added to the
 store.  This is used as part of the release process of Guix.
 
+@node Working on Shepherd
+@section Working on Shepherd
+
+This chapter documents how to modify and test GNU@tie{}Shepherd
+(@pxref{Shepherd Services}) in the Guix environment.
+
+There are two @emph{manifestations} of Shepherd in a Guix System:
+
+@table @code
+
+@item @emph{The Shepherd process}
+The init process first started by the kernel (i.e. running as PID 1).
+This is a Guile executable that is executing the @code{main} function of
+the Shepherd codebase.  Among other things, this is the process
+responsible for starting and stopping Guix System services (i.e. daemon
+processes).
+
+@item @emph{The Shepherd API}
+The Scheme code of Shepherd, which is a dependency of certain packages
+and the Guix codebase itself.  A typical example of this is the Scheme
+code implementing a Guix System service, e.g. the OpenSSH server service
+(see @code{openssh-shepherd-service}).
+
+@end table
+
+Modifying the latter results in the recompilation of several dependant
+packages, and it takes too long to be a reasonable edit-compile-test
+cycle.  But starting up a VM that merely uses a customized Shepherd init
+process is a relatively quick operation.
+
+Luckily, not all changes to Shepherd require the recompilation of all
+its dependencies.  The rule of thumb here is that:
+
+@itemize
+
+@item
+if you are making changes to the public API of Shepherd (i.e. anything
+that may have compile-time effects on dependant packages, like adding or
+removing public functions, or changing public macros, etc.), then you
+will need to go through a full recompilation, so that the the Guix
+codebase, and the dependant packages can observe the changes while they
+are being compiled.
+
+@item
+if you're only working on Shepherd's implementation (e.g. making
+Shepherd's error handling more bullet proof), then it's enough to only
+recompile Shepherd itself, and use the resulting package as the one that
+gets started as the init process.
+
+@end itemize
+
+The @ref{Shepherd Services, @code{shepherd-configuration}} section
+documents how you can replace the Shepherd process by specifying a
+custom Shepherd package for an @code{operating-system} object.  To get a
+customized Shepherd package, you can simply make a copy of it in
+@file{gnu/packages/admin.scm}, and change the @code{source} and
+@code{version} field along these lines:
+
+@lisp
+(define-public shepherd-dev-pid-1
+  (package
+    (name "shepherd")
+    (version "dev-pid-1")
+    (source (git-checkout
+             (url "file:///my/path/shepherd/")))
+    ...
+    ))
+@end lisp
+
+To modify and use a new Shepherd API, you can change the @code{source}
+and @code{version} field of the @code{shepherd} package in
+@file{gnu/packages/admin.scm} along these lines:
+
+@lisp
+(define-public shepherd     ; do not change this
+  (package
+    (name "shepherd")
+    (version "dev")
+    (source (git-checkout
+             (url "file:///my/path/shepherd/")
+             (commit "[a commit hash]")))
+    ...
+    ))
+@end lisp
+
+To avoid excessive recompilation times, we pick a specific commit in the
+latter, and only update it as needed.  But the former will pick up any
+newly recorded commit when we issue a @command{guix system vm
+/path/to/my-test.scm}.
+
 @cindex translation
 @cindex l10n
 @cindex i18n
-- 
2.34.0

[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

Attila Lendvai schreef op ma 28-02-2022 om 19:51 [+0100]:
> +Luckily, not all changes to Shepherd require the recompilation of all
> +its dependencies.  The rule of thumb here is that:

‘not all changes require’ -> ‘most changes do not require’?

> +@itemize
> +
> +@item
> +if you are making changes to the public API of Shepherd (i.e. anything
> +that may have compile-time effects on dependant packages, like adding or
> +removing public functions, or changing public macros, etc.), then you
> +will need to go through a full recompilation, so that the the Guix
> +codebase, and the dependant packages can observe the changes while they
> +are being compiled.

I don't think this is necessary?  If I change the API a little, then
why would packages only using 'sbin/halt', 'sbin/reboot' and
'sbin/shutdown' of the 'shepherd' package have to be recompiled?

Even if the API is changed, the package still uses the old shepherd
package with the old API, so no recompilation necessary.

Also, even if the API is changed, then 'guix system reconfigure ...'
would pick up the modified shepherd, and shepherd services modules
would be compiled against the shepherd from the shepherd-configuration
record (see 'shepherd-boot-gexp', 'shepherd-configuration-file' and
'scm->go' in (gnu services shepherd)).

> +@item
> +if you're only working on Shepherd's implementation (e.g. making
> +Shepherd's error handling more bullet proof), then it's enough to only
> +recompile Shepherd itself, and use the resulting package as the one that
> +gets started as the init process.
> +

So I don't think the distinction between API and implementation is
necessary here.  (Feel free to correct in you have observed the
contrary!)

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

Attila Lendvai schreef op ma 28-02-2022 om 19:51 [+0100]:
> +custom Shepherd package for an @code{operating-system} object.  To get a
> +customized Shepherd package, you can simply make a copy of it in
> +@file{gnu/packages/admin.scm}, and change the @code{source} and
> +@code{version} field along these lines:
> +
> +@lisp
> +(define-public shepherd-dev-pid-1
> +  (package
> +    (name "shepherd")
> +    (version "dev-pid-1")
> +    (source (git-checkout
> +             (url "file:///my/path/shepherd/")))
> +    ...
> +    ))


There's no need to make a copy, we can use inheritance here
to reduce duplication:

(define my-shepherd
  (package
    (inherit shepherd)
    (source (git-checkout
              (url "file:///my/path/shepherd/"))))

Also, we're missing autoconf & friends here?
To help with shepherd hacking, I think it would be ok to add them to
the standard 'shepherd' package with a nice comment.  If not, a
'shepherd-next' package or the like could be defined, a variant
of 'shepherd' for building from git?

Also, the documentation is being inconsistent with versions here.
In one place, you're using "dev-pid-1", and in another, "dev".
These versions also seem a bit odd for not being version _numbers_,
which doesn't mesh well with '17.4.3 Version Numbers'.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[pelzflorian (Florian Pelz)]

Thank you Attila.

On Mon, Feb 28, 2022 at 07:51:16PM +0100, Attila Lendvai wrote:
> +@item @emph{The Shepherd API}
> +The Scheme code of Shepherd, which is a dependency of certain packages
> +and the Guix codebase itself.  A typical example of this is the Scheme
> +code implementing a Guix System service, e.g. the OpenSSH server service
> +(see @code{openssh-shepherd-service}).

The last line reads like a @ref clickable link.


> +To modify and use a new Shepherd API, you can change the @code{source}
> +and @code{version} field of the @code{shepherd} package in
> +@file{gnu/packages/admin.scm} along these lines:
> +
> +@lisp
> +(define-public shepherd     ; do not change this
> +  (package
> +    (name "shepherd")
> +    (version "dev")
> +    (source (git-checkout
> +             (url "file:///my/path/shepherd/")
> +             (commit "[a commit hash]")))
> +    ...
> +    ))
> +@end lisp
> +
> +To avoid excessive recompilation times, we pick a specific commit in the
> +latter, and only update it as needed.  But the former will pick up any
> +newly recorded commit when we issue a @command{guix system vm
> +/path/to/my-test.scm}.

Don’t you need to update the sha256 hash so Guix picks up that it
needs to rebuild shepherd?  Unless you change the version each time to
a new value.

I wonder if you can just graft, i.e. introduce a
(replacement my-shepherd) field in the

(define-public shepherd     ; do not change this
  (package
    (name "shepherd")
    (replacement my-shepherd)
    …

I have not tested and grafting may be unreliable.

Regards,
Florian

[bug#54199] [PATCH] doc: Add 'Working on Shepherd' section.

[Attila Lendvai]

FTR, this is waiting for the fate of:

https://issues.guix.gnu.org/54216

i'll update this patch after that one is settled.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Reason is not automatic. Those who deny it cannot be conquered by it. Do not count on them. Leave them alone.”
	— Ayn Rand (1905–1982)