From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id WO8QL8DrxWKzTQEAbAwnHQ (envelope-from ) for ; Wed, 06 Jul 2022 22:08:32 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id QP8OL8DrxWLxHAEAauVa8A (envelope-from ) for ; Wed, 06 Jul 2022 22:08:32 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 81A99D944 for ; Wed, 6 Jul 2022 22:08:32 +0200 (CEST) Received: from localhost ([::1]:38946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o9BK3-0007W2-7W for larch@yhetil.org; Wed, 06 Jul 2022 16:08:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34736) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o9BD4-0003aZ-Pr for guix-devel@gnu.org; Wed, 06 Jul 2022 16:01:20 -0400 Received: from jpoiret.xyz ([206.189.101.64]:54218) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o9BCz-0007ZW-Sf for guix-devel@gnu.org; Wed, 06 Jul 2022 16:01:18 -0400 Received: from authenticated-user (jpoiret.xyz [206.189.101.64]) by jpoiret.xyz (Postfix) with ESMTPA id 25EB1184F27 for ; Wed, 6 Jul 2022 20:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jpoiret.xyz; s=dkim; t=1657137669; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=mQpCuzLb5EdSX15c51BKZD6NFjjMfaktnpU6aZuRius=; b=i8+IR5OZXnCJj4GBUcDMTy2Pm0N154Kz8xi+P8pmCBDFVMStjXlLYs0DNma7R0SE/12/Uv m/xPKE2d92xYmG5M5KPI175m7iDSBAzBY3/fOyR/2Y6T8j0yMDIi5/5MXvYTzwCy6CtrIa ES41JHj8JvR+BeL5YOOhn5kKSevOWNiwRB74TupA3WFathFB6ij6NtpwYwTEVBzvtw0z/O R/5rskok5kvZLGxnUOpNGKNUQ9NCxe5pEm70WulOmnXEBQEfw5C+/XPQKlWGp4eZeWZTnt fF4TwZ0uSK57IFkQWcv8Vu+vTRhfrsfKEEw/j1W/TWZ3Sl+4zX7vZrnZ94Zfww== From: Josselin Poiret To: guix-devel@gnu.org Subject: Building, packaging and updating Guix with confidence Date: Wed, 06 Jul 2022 22:01:07 +0200 Message-ID: <87let6roxo.fsf@jpoiret.xyz> MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Bar: / Received-SPF: pass client-ip=206.189.101.64; envelope-from=dev@jpoiret.xyz; helo=jpoiret.xyz X-Spam_score_int: 24 X-Spam_score: 2.4 X-Spam_bar: ++ X-Spam_report: (2.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.499, FROM_SUSPICIOUS_NTLD_FP=1.997, PDS_OTHER_BAD_TLD=1.997, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1657138112; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=mQpCuzLb5EdSX15c51BKZD6NFjjMfaktnpU6aZuRius=; b=EqRPrazoXcTRVR4fgn9jVFijA27emZeC2znQh/Ho1oUtcGoU/ilOXGomLvii/w3HjM29T9 4RW7lLgUESoe3XWWBP+JeVeQ0yIVqKOwttTW9kUejVQGFOsykoXQCJotEQLZqv/vYunZjm gkdn/ZF0zDo83z6eUJZI34hLyDkH+zEIj4iW/ZTLR+TJ9EWPiu6k6QEicneH0rwpsbGy0L g1pNEdKoNInEFHDHKZzYwNhNe6AEZMG9venbeh3cm4+kka9r0AThS+mZrxvbSufWUV8tM7 iieFetexD+o8O76hbg1NYVAPNqjFOX1NdoYcb95xBlwmXlEEOwN7DxRxheI4hg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1657138112; a=rsa-sha256; cv=none; b=iaWSd4SYW4XFr6pvlBdeHwPCAESuq9FuW45QnbGVISWdseneTidxau1btiu6XnQu1pdjEq GmfDIOhu7F725K0i/1b/0qqjkK+OKY4aZmM7iLPJpBAdAmfWfjXlN7barp6Hf0nsRkRJs+ hvdKHI7FMNUpz89jdrocM2gAS2yTzkOxcywbDm0LMxNHaeKcmNrOYzUMJN5oXy5V0Nx0Oz aed2XuMq8rVYS+6DC1bcmstk3miUT/Sj5883ke8vNoF2BNtElxLOBNm1fraIf4D15xnI/e NbFwae23zQUqFqCh+axbBkZBmyeQdajdZUvKsrmNJM4f7Nt2RS+aMp64h7aDUg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=jpoiret.xyz header.s=dkim header.b=i8+IR5OZ; dmarc=pass (policy=reject) header.from=jpoiret.xyz; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.45 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=jpoiret.xyz header.s=dkim header.b=i8+IR5OZ; dmarc=pass (policy=reject) header.from=jpoiret.xyz; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 81A99D944 X-Spam-Score: -5.45 X-Migadu-Scanner: scn0.migadu.com X-TUID: uzCReyD7V41X Hello everyone, This is going to be a long post, but that's mostly because it includes a write-up of how many things currently function. Let's start with the TLDR: I think we need to rethink the way we build and package Guix, so that a) the guix package is always current compared to the guix that contains it; b) we can say with confidence that the development, in-tree Guix, the `guix pull` and the guix package are all built exactly in the same way, with *extremely minor* differences in areas such as commit/version embedding. I know this isn't a light proposition, but this would solve a swath of issues, included but not limited to: the system-wide guix not working well when interacting with a `guix pull`ed guix, having to update the guix package twice for the installer to pick up needed changes, issues only showing up for people using the system-wide Guix that the local development setup doesn't catch (eg. [1]), installer images never being actually "current", installer images not carrying the closure of %bare-bones-os corresponding to the guix version it will use, not being able to test installer/guix-daemon/guix-build-process changes without tinkering with the guix package definition, etc. Two recent things have brought today's subject to the table: * a recent surge of problems related to the new manifest format (4) that rendered some users' Guixes unusable; * me working on adding a C Guile extension to Guix that would add an interface for posix_spawn, resolving some of our issues with resource leakage and deadlocks in inferiors (as well as in other places, I'm certain (yes I'm looking at the installer)). The elephant in the room in both of these situations is the myriad of different ways we have of building a Guix, their versioning, and how these different products interact between each other on a running system. Let's start with the first case. Commit 4ff12d1de7cd617b791996ee7ca1240660b4c20e changed the version of (internal) profile manifests produced by guix commands to 4, along with other changes. Commit 06493e738825598447f5b45d7100ca7eff8b669d updated the guix package (the one in gnu/packages/package-management.scm) to a commit that includes the former one. Now comes the issue: systems installed using a commit between those two would produce a system profile that uses manifest 4, but with a Guix that cannot read such a manifest. This won't be an issue for people that have already installed their system and `guix pull`'d, but for fresh systems `guix pull` and `guix describe` will both bail out because they want to read the manifest of the profile they're running from since it might contain their commit SHA (guix pull uses it to prevent downgrades), but don't understand it. [Notes about why it searches there later on] This effectively locks the user out from updating their guix. In the second case, I am in the process of adding a very simple Guile C extension to Guix that only requires to wrap a simple libc function. The C code itself took approx. 5% of my time on it, while adding the magical invocations for the Autotools took 35%, and now testing the changes is taking 60%, because I need to test that `guix pull` works, that the local development setup works, and that the guix package works, all with a local change that cannot be authenticated and that also cannot be referred to with a git-reference [more on that later] since it's entirely local. This is causing me many headaches, and I don't think that even after manually testing this with handwritten hacks so that these changes actually do appear everywhere, I will be confident enough for them to be merged. In any case, it's extremely difficult for someone who hasn't just stared at the code for a long time to discern the different issues that might arise when adding non-trivial changes to Guix. Let me try to spell out the different ways that we build Guix: a) The local, development way. This uses autotools, with configure.ac and configure-daemon.ac as autoconf inputs, and Makefile.am, gnu/local.mk and nix/local.mk as automake inputs. Basically, that means `autoreconf`, `./configure` followed by `make`, the standard gnu-build-system fare. This is the number 1 way that guix developers build their copy with, and the one they interact the most with. Currently, the resulting Guix doesn't know any of its provenance info, ie. guix/config.scm isn't properly filled out. b) The guix package, gnu-build-system way. This is quite similar to a), except that guix/config.scm is properly filled out, fill out paths to hardcoded compressors, and wrap `guix` so that it finds the proper guile and extensions. We also disable some failing tests because of the build container. This package is used by `guix system` to install the system-wide guix to the system profile, which is used by users that haven't guix pull'd yet. This is necessarily *always* out of date, because we can't see commits in the future :p. c) The `guix pull` way. This is a bit of an hybrid. Internally, it amounts to eval'ing build-aux/build-self.scm inside of the currently running guix, which returns a procedure which, given a source code checkout of Guix, returns a derivation that builds the new Guix. This itself relies on (guix self) where most of the building code is. The scheme modules are compiled in an ad-hoc manner (not following Makefile.am), and files are included without consulting the Makefile. This is why `guix pull` users are not affected by [1]. But then, there still is an issue with the guix daemon (and in the future the C extensions), which is C code. Since we're not pretending to know how to universally configure C packages, we rely on the guix-daemon package defined in gnu/packages/package-management.scm of the future Guix, which inherits from the guix package itself, and then builds itself roughly following b), meaning the daemon is still out-of-date. This way, you can see why many issues arise: the double update for the installer issue is because the guix that will be installed in the final system corresponds to the guix package that's defined inside the guix on the installer OS, itself corresponding to the guix package defined installed the one running `guix system image` (or similar). So if we want an update to hit the system-wide Guix for the end-user after installing, we need 2 linked updates. What I personally think, is that we should rationalize the way we interact with Guix source: a running Guix should always be able to hold a reference to its source. The guix package or future equivalent (ie. good for internal consumption) should always refer to that same source, but that will also require factoring the daemon (and extensions) out of the repository, so that the C code doesn't get compiled again on every unrelated commit. Finally, and I think this is the most challenging one, we should try to keep the differences between a) and c) to the minimum, meaning that one way of building has to go. This is a big change, conceptually and technically, and I understand that this might be way more complicated that we'd like, but I think this needs to be done at some point. WDYT? [1] https://issues.guix.gnu.org/52572 (20211217222522.2440-1-dev@jpoiret.xyz) -- Josselin Poiret