unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Building, packaging and updating Guix with confidence
@ 2022-07-06 20:01 Josselin Poiret
  2022-07-07 12:06 ` Zhu Zihao
                   ` (2 more replies)
  0 siblings, 3 replies; 11+ messages in thread
From: Josselin Poiret @ 2022-07-06 20:01 UTC (permalink / raw)
  To: guix-devel

Hello everyone,

This is going to be a long post, but that's mostly because it includes a
write-up of how many things currently function.  Let's start with the
TLDR:

I think we need to rethink the way we build and package Guix, so that
a) the guix package is always current compared to the guix that contains
it;
b) we can say with confidence that the development, in-tree Guix,
the `guix pull` and the guix package are all built exactly in the same
way, with *extremely minor* differences in areas such as commit/version
embedding.

I know this isn't a light proposition, but this would solve a swath of
issues, included but not limited to: the system-wide guix not working
well when interacting with a `guix pull`ed guix, having to update the
guix package twice for the installer to pick up needed changes, issues
only showing up for people using the system-wide Guix that the local
development setup doesn't catch (eg. [1]), installer images never being
actually "current", installer images not carrying the closure of
%bare-bones-os corresponding to the guix version it will use, not being
able to test installer/guix-daemon/guix-build-process changes without
tinkering with the guix package definition, etc.

Two recent things have brought today's subject to the table:
* a recent surge of problems related to the new manifest format (4) that
rendered some users' Guixes unusable;
* me working on adding a C Guile extension to Guix that would add
an interface for posix_spawn, resolving some of our issues with resource
leakage and deadlocks in inferiors (as well as in other places, I'm
certain (yes I'm looking at the installer)).

The elephant in the room in both of these situations is the myriad of
different ways we have of building a Guix, their versioning, and how
these different products interact between each other on a running
system.  Let's start with the first case.

Commit 4ff12d1de7cd617b791996ee7ca1240660b4c20e changed the version of
(internal) profile manifests produced by guix commands to 4, along with
other changes.  Commit 06493e738825598447f5b45d7100ca7eff8b669d updated
the guix package (the one in gnu/packages/package-management.scm) to a
commit that includes the former one.  Now comes the issue: systems
installed using a commit between those two would produce a system
profile that uses manifest 4, but with a Guix that cannot read such a
manifest.  This won't be an issue for people that have already installed
their system and `guix pull`'d, but for fresh systems `guix pull` and
`guix describe` will both bail out because they want to read the
manifest of the profile they're running from since it might contain
their commit SHA (guix pull uses it to prevent downgrades), but don't
understand it. [Notes about why it searches there later on] This
effectively locks the user out from updating their guix.

In the second case, I am in the process of adding a very simple Guile C
extension to Guix that only requires to wrap a simple libc function.
The C code itself took approx. 5% of my time on it, while adding the
magical invocations for the Autotools took 35%, and now testing the
changes is taking 60%, because I need to test that `guix pull` works,
that the local development setup works, and that the guix package works,
all with a local change that cannot be authenticated and that also
cannot be referred to with a git-reference [more on that later] since
it's entirely local.  This is causing me many headaches, and I don't
think that even after manually testing this with handwritten hacks so
that these changes actually do appear everywhere, I will be confident
enough for them to be merged.

In any case, it's extremely difficult for someone who hasn't just stared
at the code for a long time to discern the different issues that might
arise when adding non-trivial changes to Guix.  Let me try to spell out
the different ways that we build Guix:

a) The local, development way.  This uses autotools, with configure.ac
and configure-daemon.ac as autoconf inputs, and Makefile.am, gnu/local.mk and
nix/local.mk as automake inputs. Basically, that means `autoreconf`, `./configure`
followed by `make`, the standard gnu-build-system fare.  This is the
number 1 way that guix developers build their copy with, and the one
they interact the most with.  Currently, the resulting Guix doesn't know
any of its provenance info, ie. guix/config.scm isn't properly filled out.

b) The guix package, gnu-build-system way.  This is quite similar to a),
except that guix/config.scm is properly filled out, fill out paths to
hardcoded compressors, and wrap `guix` so that it finds the proper guile
and extensions. We also disable some failing tests because of the build
container.  This package is used by `guix system` to install the
system-wide guix to the system profile, which is used by users that
haven't guix pull'd yet.  This is necessarily *always* out of date,
because we can't see commits in the future :p.

c) The `guix pull` way.  This is a bit of an hybrid.  Internally, it
amounts to eval'ing build-aux/build-self.scm inside of the currently
running guix, which returns a procedure which, given a source code
checkout of Guix, returns a derivation that builds the new Guix.  This
itself relies on (guix self) where most of the building code is.  The
scheme modules are compiled in an ad-hoc manner (not following
Makefile.am), and files are included without consulting the Makefile.
This is why `guix pull` users are not affected by [1].  But then, there
still is an issue with the guix daemon (and in the future the C
extensions), which is C code.  Since we're not pretending to know how to
universally configure C packages, we rely on the guix-daemon package
defined in gnu/packages/package-management.scm of the future Guix, which
inherits from the guix package itself, and then builds itself roughly
following b), meaning the daemon is still out-of-date.

This way, you can see why many issues arise:  the double update for the
installer issue is because the guix that will be installed in the final
system corresponds to the guix package that's defined inside the guix on
the installer OS, itself corresponding to the guix package defined
installed the one running `guix system image` (or similar).  So if we
want an update to hit the system-wide Guix for the end-user after
installing, we need 2 linked updates.

What I personally think, is that we should rationalize the way we
interact with Guix source: a running Guix should always be able to hold
a reference to its source.  The guix package or future equivalent
(ie. good for internal consumption) should always refer to that same
source, but that will also require factoring the daemon (and extensions)
out of the repository, so that the C code doesn't get compiled again on
every unrelated commit.  Finally, and I think this is the most
challenging one, we should try to keep the differences between a) and c)
to the minimum, meaning that one way of building has to go.  This is a
big change, conceptually and technically, and I understand that this
might be way more complicated that we'd like, but I think this needs to
be done at some point.

WDYT?

[1] https://issues.guix.gnu.org/52572
    (20211217222522.2440-1-dev@jpoiret.xyz)

-- 
Josselin Poiret


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2022-08-19 10:22 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-06 20:01 Building, packaging and updating Guix with confidence Josselin Poiret
2022-07-07 12:06 ` Zhu Zihao
2022-07-07 14:34   ` Josselin Poiret
2022-07-17 16:52     ` bokr
2022-07-21 16:10       ` Josselin Poiret
2022-07-21 16:18         ` Maxime Devos
2022-07-26  1:09         ` Bengt Richter
2022-08-18 13:19           ` zimoun
2022-07-18 11:03 ` Ludovic Courtès
2022-08-18 12:52 ` zimoun
2022-08-19 10:21   ` Josselin Poiret

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).