bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.

[Janneke Nieuwenhuizen <janneke@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

Ludovic Courtès writes:

Hi!

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Similar to the Makefile.am change, this breaks-up packages into 26 chunks
>> when building on 32bit.  Also force garbage collection.
>>
>> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
>> Use it when building on a "i586" or "i686" cpu.
>
> .go files of a Guix built with this new (guix self) would not be found:
>
> newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0

Hmm, how did I miss this?

> In fact, guix/*.go is entirely missing, it seems:

Right...that makes sense now that I look at the code again.

Wait...I probably even experienced this breakage after running `guix
pull' on the Hurd but failed to notice its cause, the missing guix/*.go,
and ascribed it to "something" being broken on Hurd.  So the good news
is that we'll most probably have guix pull work on the Hurd after fixing
this!

--8<---------------cut here---------------start------------->8---
             (cute partition <> files)
             (lambda (regex)
               (cute string-match regex <>))
             (cute string-append "^gnu/packages/" <>)
--8<---------------cut here---------------end--------------->8---

it still has the "gnu/packages" directory name hardcoded here.  (I even
believe this was kind-of intentional, meaning not to split-up builds of
other directories...).  The result is, of course, that partition always
produces an empty result for any '("guix/foo.scm") files!

It is fixed by doing

--8<---------------cut here---------------start------------->8---
             (cute string-append "^" directory "/" <>)
--8<---------------cut here---------------end--------------->8---

i.e., also splitting guix/ 26 ways.

> I’m reverting for now.

Thank you for looking into and taking care of this mess I made, sorry!

> Note that (guix self) is very sensitive, so we should test it
> thoroughly.

Using the attached patch, I ran

--8<---------------cut here---------------start------------->8---
$ guix pull --branch=master --url=$PWD
[..]
08:11:02 janneke@drakenpad:~/src/guix/master 
$ guix describe
Git checkout:
  repository: /home/janneke/src/guix/master/
  branch: master
  commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3
08:11:04 janneke@drakenpad:~/src/guix/master 
$ guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1
--8<---------------cut here---------------end--------------->8---

> Here I used ‘make as-derivation’ and ran the resulting
> ‘guix’ command.

Ah, nice that guix works too for me too now

--8<---------------cut here---------------start------------->8---
$ make as-derivation
[..]
/gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06
08:13:15 janneke@drakenpad:~/src/guix/master [env]
$ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1
--8<---------------cut here---------------end--------------->8---

Greetings,
Janneke


[-- Attachment #2: v3-0001-self-Build-guix-and-gnu-packages-directories-in-2.patch --]
[-- Type: text/x-patch, Size: 3600 bytes --]

From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001
Message-ID: <a639a64d590067d3fe928be3ecc9d2d4cc2e8df3.1692770000.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26
 steps.

Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into 26 chunks.
---
 guix/self.scm | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..e5a85eaad6 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-71)
                        (ice-9 match)
                        (ice-9 format)
+                       (ice-9 regex)
                        (ice-9 threads)
                        (guix build compile)
                        (guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            ;; Split gnu/packages in 26 chunks to avoid OOM errors
+            (let* ((chunks (map (compose
+                                 (cute partition <> files)
+                                 (lambda (regex)
+                                   (cute string-match regex <>))
+                                 (cute string-append "^" directory "/" <>)
+                                 (cute make-string 1 <>)
+                                 integer->char
+                                 (cute + (char->integer #\a) <>))
+                                (iota 26)))
+                   (chunks (filter pair? chunks)))
+              (for-each
+               (lambda (chunck)
+                 ;; Hide compilation warnings.
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   (compile-files directory #$output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 164 bytes --]


-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com