bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Janneke Nieuwenhuizen]

Hi!

Initially writter for the Hurd (see hurd-team
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=hurd-team&id=06438c2a7edf1f0c90392858fc86480c6e5c2195>,
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=hurd-team&id=5623ab503fa70eab95e6f859e82c89581c3f69f4>)
This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit
issue, rather than a Hurd problem
(<https://logs.guix.gnu.org/guix/2023-08-22.log#182716>,
<https://ci.guix.gnu.org/build/1870536/log/raw>).

I didn't submit these for the Hurd yet, as I didn't really feel comfortable
with the 26-way split.  I tried a 5-way split and that still gave OOM errors
on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah.

Also, the 5-way split in the Makefile.am will produce percentages over
100 which suggests a typo...but I couldn't spot it.

Lastly, these patches feel like a workaround for Guile / libgc memory
management?  Otoh, being able to build Guix on 32bit hosts is kinda nice
too... So yeah.

Greetings,
Janneke

Janneke Nieuwenhuizen (2):
  build: Build gnu/packages/*.go in five steps.
  self: Build gnu/packages/*.go in 26 steps on 32bit.

 Makefile.am   | 62 ++++++++++++++++++++++++++++++++++++++++++++-------
 guix/self.scm | 31 ++++++++++++++++++++++++--
 2 files changed, 83 insertions(+), 10 deletions(-)


base-commit: c655231b72ac28b5a433069fcf86a835c9c83691
-- 
2.41.0

bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps.

[Janneke Nieuwenhuizen]

This breaks-up packages into five chunks of ~150,000 lines, allowing guix
build --target=i586-pc-gnu from an x86 host.

This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294.

* Makefile.am (first_half): Rename to...
(first_quart): ...this, and also split into...
(second_quart): ...this.
(third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables.
(make-packages3-go, make-packages4-go): New targets.
(make-packages-go): Add them.
---
 Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 54 insertions(+), 8 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 738532f839..7c8c215f86 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -776,19 +776,45 @@ endef
 # in <https://issues.guix.gnu.org/48963>.  Each 'eval' call below creates a
 # 'make-*-go' phony target that builds the corresponding subset.
 
-first_half :=								\
-  gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d%	\
-  gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h%	\
-  gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l%
+first_fifth :=								\
+  gnu/packages/a% gnu/packages/b% gnu/packages/c%
+
+second_fifth :=								\
+  gnu/packages/d% gnu/packages/e% gnu/packages/f%
+
+third_fifth :=								\
+  gnu/packages/g%  gnu/packages/h% gnu/packages/i%
+
+fourth_fifth :=								\
+  gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% 	\
+  gnu/packages/n% gnu/packages/o%
 
 MODULES_CORE      := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES)))
-MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES))
-MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES)))
-MODULES_PACKAGES  := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)
+MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES))
+MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES))
+MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES))
+MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES))
+MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth)	\
+			$(third_fifth) $(fourth_fifth),			\
+			$(filter gnu/packages/%,$(MODULES)))
+MODULES_PACKAGES  := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)		\
+			$(MODULES_PACKAGES3) $(MODULES_PACKAGES4)	\
+			$(MODULES_PACKAGES5)
 MODULES_SYSTEM    := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
 MODULES_CLI       := $(filter guix/scripts/%,$(MODULES))
 MODULES_PO        := guix/build/po.scm
 
+print-p1:
+	@echo $(MODULES_PACKAGES1)
+print-p2:
+	@echo $(MODULES_PACKAGES2)
+print-p3:
+	@echo $(MODULES_PACKAGES3)
+print-p4:
+	@echo $(MODULES_PACKAGES4)
+print-p5:
+	@echo $(MODULES_PACKAGES5)
+
 $(eval $(call guile-compilation-rule,make-core-go,	\
   $(MODULES_CORE) guix/config.scm $(dist_noinst_DATA),	\
   0))
@@ -804,7 +830,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go,	\
   $(words $(MODULES_CORE) $(MODULES_PACKAGES1))))
 .PHONY: make-packages2-go
 
-make-packages-go: make-packages1-go make-packages2-go
+$(eval $(call guile-compilation-rule,make-packages3-go,				\
+  $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go,	\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2))))
+.PHONY: make-packages3-go
+
+$(eval $(call guile-compilation-rule,make-packages4-go,			\
+  $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go	\
+    make-packages3-go,							\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)	\
+    $(MODULES_PACKAGES3))))
+.PHONY: make-packages4-go
+
+$(eval $(call guile-compilation-rule,make-packages5-go,			\
+  $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go	\
+    make-packages3-go make-packages4-go,				\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)	\
+    $(MODULES_PACKAGES3) $(MODULES_PACKAGES4))))
+.PHONY: make-packages5-go
+
+make-packages-go: make-packages1-go make-packages2-go \
+  make-packages3-go make-packages4-go make-packages5-go
 .PHONY: make-packages-go
 
 $(eval $(call guile-compilation-rule,make-system-go,	\
-- 
2.41.0

bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit.

[Janneke Nieuwenhuizen]

Similar to the Makefile.am change, this breaks-up packages into 26 chunks
when building on 32bit.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
Use it when building on a "i586" or "i686" cpu.
---
 guix/self.scm | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..474715fe7a 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-71)
                        (ice-9 match)
                        (ice-9 format)
+                       (ice-9 regex)
                        (ice-9 threads)
                        (guix build compile)
                        (guix build utils))
@@ -1251,6 +1255,26 @@ (define* (compiled-modules name module-tree module-files
                              #:report-load report-load
                              #:report-compilation report-compilation)))
 
+          (define (process-directory/32bit directory files output)
+            (let* ((chunks (map (compose
+                                 (cute partition <> files)
+                                 (lambda (regex)
+                                   (cute string-match regex <>))
+                                 (cute string-append "^gnu/packages/" <>)
+                                 (cute make-string 1 <>)
+                                 integer->char
+                                 (cute + (char->integer #\a) <>))
+                                (iota 26)))
+                   (chunks (filter pair? chunks)))
+              (for-each
+               (lambda (chunck)
+                 (compile-files directory #$output chunck
+                                #:workers (parallel-job-count)
+                                #:report-load report-load
+                                #:report-compilation report-compilation)
+                 (gc))
+               chunks)))
+
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)
 
@@ -1277,7 +1301,10 @@ (define* (compiled-modules name module-tree module-files
 
           (mkdir #$output)
           (chdir #+module-tree)
-          (process-directory "." '#+module-files #$output)
+          (match (string-split %host-type #\-)
+            (((or "i586" "i686") rest ...)
+             (process-directory/32bit "." '#+module-files #$output))
+            (_ (process-directory "." '#+module-files #$output)))
           (newline))))
 
   (computed-file name build
-- 
2.41.0

bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

> * Makefile.am (first_half): Rename to...
[..]
> +print-p1:
> +	@echo $(MODULES_PACKAGES1)
> +print-p2:
> +	@echo $(MODULES_PACKAGES2)
> +print-p3:
> +	@echo $(MODULES_PACKAGES3)
> +print-p4:
> +	@echo $(MODULES_PACKAGES4)
> +print-p5:
> +	@echo $(MODULES_PACKAGES5)
> +

...I mean to remove this debugging code, but it might come in handy for
review.

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps.

[Janneke Nieuwenhuizen]

This breaks-up packages into five chunks of ~150,000 lines, allowing guix
build --target=i586-pc-gnu from an x86 host.

This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294.

* Makefile.am (first_half): Rename to...
(first_quart): ...this, and also split into...
(second_quart): ...this.
(third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables.
(make-packages3-go, make-packages4-go): New targets.
(make-packages-go): Add them.
---
 Makefile.am | 51 +++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 43 insertions(+), 8 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 738532f839..922913355c 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -776,15 +776,30 @@ endef
 # in <https://issues.guix.gnu.org/48963>.  Each 'eval' call below creates a
 # 'make-*-go' phony target that builds the corresponding subset.
 
-first_half :=								\
-  gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d%	\
-  gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h%	\
-  gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l%
+first_fifth :=								\
+  gnu/packages/a% gnu/packages/b% gnu/packages/c%
+
+second_fifth :=								\
+  gnu/packages/d% gnu/packages/e% gnu/packages/f%
+
+third_fifth :=								\
+  gnu/packages/g%  gnu/packages/h% gnu/packages/i%
+
+fourth_fifth :=								\
+  gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% 	\
+  gnu/packages/n% gnu/packages/o%
 
 MODULES_CORE      := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES)))
-MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES))
-MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES)))
-MODULES_PACKAGES  := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)
+MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES))
+MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES))
+MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES))
+MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES))
+MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth)	\
+			$(third_fifth) $(fourth_fifth),			\
+			$(filter gnu/packages/%,$(MODULES)))
+MODULES_PACKAGES  := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)		\
+			$(MODULES_PACKAGES3) $(MODULES_PACKAGES4)	\
+			$(MODULES_PACKAGES5)
 MODULES_SYSTEM    := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
 MODULES_CLI       := $(filter guix/scripts/%,$(MODULES))
 MODULES_PO        := guix/build/po.scm
@@ -804,7 +819,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go,	\
   $(words $(MODULES_CORE) $(MODULES_PACKAGES1))))
 .PHONY: make-packages2-go
 
-make-packages-go: make-packages1-go make-packages2-go
+$(eval $(call guile-compilation-rule,make-packages3-go,				\
+  $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go,	\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2))))
+.PHONY: make-packages3-go
+
+$(eval $(call guile-compilation-rule,make-packages4-go,			\
+  $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go	\
+    make-packages3-go,							\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)	\
+    $(MODULES_PACKAGES3))))
+.PHONY: make-packages4-go
+
+$(eval $(call guile-compilation-rule,make-packages5-go,			\
+  $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go	\
+    make-packages3-go make-packages4-go,				\
+  $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)	\
+    $(MODULES_PACKAGES3) $(MODULES_PACKAGES4))))
+.PHONY: make-packages5-go
+
+make-packages-go: make-packages1-go make-packages2-go \
+  make-packages3-go make-packages4-go make-packages5-go
 .PHONY: make-packages-go
 
 $(eval $(call guile-compilation-rule,make-system-go,	\

base-commit: c655231b72ac28b5a433069fcf86a835c9c83691
-- 
2.41.0

bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.

[Janneke Nieuwenhuizen]

Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
"gnu/packages" into 26 chunks.
---
 guix/self.scm | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..151fa9f452 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-71)
                        (ice-9 match)
                        (ice-9 format)
+                       (ice-9 regex)
                        (ice-9 threads)
                        (guix build compile)
                        (guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            ;; Split gnu/packages in 26 chunks to avoid OOM errors
+            (let* ((chunks (map (compose
+                                 (cute partition <> files)
+                                 (lambda (regex)
+                                   (cute string-match regex <>))
+                                 (cute string-append "^gnu/packages/" <>)
+                                 (cute make-string 1 <>)
+                                 integer->char
+                                 (cute + (char->integer #\a) <>))
+                                (iota 26)))
+                   (chunks (filter pair? chunks)))
+              (for-each
+               (lambda (chunck)
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   ;; Hide compilation warnings.
+                   (compile-files directory #$output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)
-- 
2.41.0

bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

> diff --git a/guix/self.scm b/guix/self.scm
> index 81a36e007f..151fa9f452 100644
[..]

> +                 (parameterize ((current-warning-port (%make-void-port "w")))

meant to move this comment

> +                   ;; Hide compilation warnings.

above the parameterize; fixed locally


Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

After a LGTM from Ludo and discussing with Maxim on IRC, pushed to
master as 5898b2e8a3dbf7797e83b39a2783c5b543015725

Maxim suggested (<https://logs.guix.gnu.org/guix/2023-08-22.log#211631>)
we could maybe use Guile in make to split building .go files in
Makefile.am algorithmically like we do in guix/self.scm.

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Ludovic Courtès]

Hi,

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

> Similar to the Makefile.am change, this breaks-up packages into 26 chunks
> when building on 32bit.  Also force garbage collection.
>
> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
> Use it when building on a "i586" or "i686" cpu.

.go files of a Guix built with this new (guix self) would not be found:

--8<---------------cut here---------------start------------->8---
newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0
newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/share/guile/site/3.0/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/run/current-system/profile/lib/guile/3.0/site-ccache/guix/ui.go", {st_mode=S_IFREG|0444, st_size=530053, ...}, 0) = 0
--8<---------------cut here---------------end--------------->8---

In fact, guix/*.go is entirely missing, it seems:

$ --8<---------------cut here---------------start------------->8---
ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix
ls: cannot access '/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix': No such file or directory
$ ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/gnu
packages/
--8<---------------cut here---------------end--------------->8---

With the patch, without the patch:

--8<---------------cut here---------------start------------->8---
$ find -L /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union -name \*.go | wc -l
719
$ find -L  /gnu/store/4dm864f7iif1vg9sz355gdqkgb7air98-guix-module-union -name \*.go | wc -l
1188
--8<---------------cut here---------------end--------------->8---

The problem would then manifest like this:

--8<---------------cut here---------------start------------->8---
$ /gnu/store/57x6wrf78147lif1ss0ck2z3a0323b9l-guix-20230822.21/bin/guix build hello
error: license:arphic-1999: unbound variable
hint: Did you forget a `use-modules' form?

error: googletest: unbound variable
hint: Did you forget a `use-modules' form?

[…]

Throw to key `unbound-variable' with args `("resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f)'.
Backtrace:
In guix/store.scm:
   659:37 19 (thunk)
   1298:8 18 (call-with-build-handler #<procedure 7f82c427ac60 at guix/ui.scm:1196:2 (continu…> …)
In guix/scripts/build.scm:
    584:2 17 (_)
In srfi/srfi-1.scm:
   673:15 16 (append-map _ _ . _)
   586:17 15 (map1 ((argument . "hello") (build-mode . 0) (graft? . #t) (substitutes? . #t) # …))
In guix/scripts/build.scm:
   604:31 14 (_ _)
In gnu/packages.scm:
    485:2 13 (%find-package "hello" "hello" #f)
    365:6 12 (find-best-packages-by-name _ _)
   295:56 11 (_ "hello" _)
In unknown file:
          10 (force #<promise #<procedure 7f82c6c5eee0 at gnu/packages.scm:285:18 ()>>)
In gnu/packages.scm:
   242:33  9 (fold-packages #<procedure 7f82c21125f0 at gnu/packages.scm:286:35 (p r)> #<vlis…> …)
In guix/discovery.scm:
   158:11  8 (all-modules _ #:warn _)
In srfi/srfi-1.scm:
   460:18  7 (fold #<procedure 7f82c6cc7f00 at guix/discovery.scm:142:8 (spec result)> _ ((. #)))
In guix/discovery.scm:
   148:19  6 (_ _ ())
    115:5  5 (scheme-modules _ _ #:warn _)
In srfi/srfi-1.scm:
   691:23  4 (filter-map #<procedure 7f82c6cc7de0 at guix/discovery.scm:115:16 (file)> _ . _)
In guix/discovery.scm:
   123:24  3 (_ . _)
In guix/ui.scm:
    343:2  2 (report-unbound-variable-error _ #:frame _)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `match-error' with args `("match" "no matching pattern" (unbound-variable "resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f))'.
--8<---------------cut here---------------end--------------->8---

This is because an old (guix licenses) module would be loaded.

I’m reverting for now.

Note that (guix self) is very sensitive, so we should test it
thoroughly.  Here I used ‘make as-derivation’ and ran the resulting
‘guix’ command.

Ludo’.

bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.

[Janneke Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

Ludovic Courtès writes:

Hi!

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Similar to the Makefile.am change, this breaks-up packages into 26 chunks
>> when building on 32bit.  Also force garbage collection.
>>
>> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
>> Use it when building on a "i586" or "i686" cpu.
>
> .go files of a Guix built with this new (guix self) would not be found:
>
> newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0

Hmm, how did I miss this?

> In fact, guix/*.go is entirely missing, it seems:

Right...that makes sense now that I look at the code again.

Wait...I probably even experienced this breakage after running `guix
pull' on the Hurd but failed to notice its cause, the missing guix/*.go,
and ascribed it to "something" being broken on Hurd.  So the good news
is that we'll most probably have guix pull work on the Hurd after fixing
this!

--8<---------------cut here---------------start------------->8---
             (cute partition <> files)
             (lambda (regex)
               (cute string-match regex <>))
             (cute string-append "^gnu/packages/" <>)
--8<---------------cut here---------------end--------------->8---

it still has the "gnu/packages" directory name hardcoded here.  (I even
believe this was kind-of intentional, meaning not to split-up builds of
other directories...).  The result is, of course, that partition always
produces an empty result for any '("guix/foo.scm") files!

It is fixed by doing

--8<---------------cut here---------------start------------->8---
             (cute string-append "^" directory "/" <>)
--8<---------------cut here---------------end--------------->8---

i.e., also splitting guix/ 26 ways.

> I’m reverting for now.

Thank you for looking into and taking care of this mess I made, sorry!

> Note that (guix self) is very sensitive, so we should test it
> thoroughly.

Using the attached patch, I ran

--8<---------------cut here---------------start------------->8---
$ guix pull --branch=master --url=$PWD
[..]
08:11:02 janneke@drakenpad:~/src/guix/master 
$ guix describe
Git checkout:
  repository: /home/janneke/src/guix/master/
  branch: master
  commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3
08:11:04 janneke@drakenpad:~/src/guix/master 
$ guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1
--8<---------------cut here---------------end--------------->8---

> Here I used ‘make as-derivation’ and ran the resulting
> ‘guix’ command.

Ah, nice that guix works too for me too now

--8<---------------cut here---------------start------------->8---
$ make as-derivation
[..]
/gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06
08:13:15 janneke@drakenpad:~/src/guix/master [env]
$ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1
--8<---------------cut here---------------end--------------->8---

Greetings,
Janneke


[-- Attachment #2: v3-0001-self-Build-guix-and-gnu-packages-directories-in-2.patch --]
[-- Type: text/x-patch, Size: 3600 bytes --]

From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001
Message-ID: <a639a64d590067d3fe928be3ecc9d2d4cc2e8df3.1692770000.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26
 steps.

Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into 26 chunks.
---
 guix/self.scm | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..e5a85eaad6 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-71)
                        (ice-9 match)
                        (ice-9 format)
+                       (ice-9 regex)
                        (ice-9 threads)
                        (guix build compile)
                        (guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            ;; Split gnu/packages in 26 chunks to avoid OOM errors
+            (let* ((chunks (map (compose
+                                 (cute partition <> files)
+                                 (lambda (regex)
+                                   (cute string-match regex <>))
+                                 (cute string-append "^" directory "/" <>)
+                                 (cute make-string 1 <>)
+                                 integer->char
+                                 (cute + (char->integer #\a) <>))
+                                (iota 26)))
+                   (chunks (filter pair? chunks)))
+              (for-each
+               (lambda (chunck)
+                 ;; Hide compilation warnings.
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   (compile-files directory #$output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 164 bytes --]


-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time.

[Janneke Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]

Janneke Nieuwenhuizen writes:

Hi!

> Ludovic Courtès writes:
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
[..]

> it still has the "gnu/packages" directory name hardcoded here.  (I even
> believe this was kind-of intentional, meaning not to split-up builds of
> other directories...).  The result is, of course, that partition always
> produces an empty result for any '("guix/foo.scm") files!
>
> It is fixed by doing
>
>              (cute string-append "^" directory "/" <>)

Well, almost...In my test script it worked, but directory is "." in the
actual self.scm.

So with v3 no .go files get built at all...and then a naive `guix build
hello' seems to work!; `guix shell' for example fails.  Comparing the
list of files from `find /gnu/store/...-guix-20230823.08 -follow' is
also helpful.

I realised that there's no need to use the name-based splitting hack
that Makefile.am uses and changed the creation of chunks to simply use a
maximum chunk length of 25 files.

New version attached.

Greetings,
Janneke


[-- Attachment #2: v4-0001-self-Build-directories-in-chunks-of-max-25-files-.patch --]
[-- Type: text/x-patch, Size: 3288 bytes --]

From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
 time.

Similar to split build of make-go in Makefile.am, this breaks-up building
directories into chunks of max 25 files.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into chunks of max 25 files.
---
 guix/self.scm | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..fc2dfd8131 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
                        (ice-9 match)
                        (ice-9 format)
                        (ice-9 threads)
@@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            (let* ((size 25)                      ;compile max 25 files a time
+                   (chunks (unfold
+                            (lambda (seed) (< (length seed) size)) ;p
+                            (cute take <> size)                    ;f
+                            (cute drop <> size)                    ;g
+                            files                                  ;seed
+                            list)))                                ;tail
+              (for-each
+               (lambda (chunck)
+                 ;; Hide compilation warnings.
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   (compile-files directory output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 164 bytes --]


-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

Hi!

> Ludovic Courtès writes:
>
> Hi!
>
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
[..]
>> In fact, guix/*.go is entirely missing, it seems:
>
> Right...that makes sense now that I look at the code again.
>
> Wait...I probably even experienced this breakage after running `guix
> pull' on the Hurd but failed to notice its cause, the missing guix/*.go,
> and ascribed it to "something" being broken on Hurd.  So the good news
> is that we'll most probably have guix pull work on the Hurd after fixing
> this!

So, some good news at last; I can confirm that using v4 of this patch we
now have "guix pull", pulling from a local git directory, now fully
working on the Hurd!  (When pulling from a git url, I get "Illegal
instruction" while receiving objects.)

See log below.

Greetings,
Janneke

/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=http://gitlab.com/janneke/guix --branch=wip-hurd
Updating channel 'guix' from Git repository at 'http://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix install nss-certs
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
   nss-certs 3.88.1

substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/z1jyx8lyhgr1gykiky5wjh5mncwvp6ls-profile.drv
  /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv
  /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv
  /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv

71.6 MB will be downloaded
 certdata2pem.c  2KiB                                 23KiB/s 00:00 [##################] 100.0%
 nss-3.88.1.tar.gz  68.3MiB                          2.1MiB/s 00:32 [##################] 100.0%
building /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv...
building /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv...
building /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
killing process 174: Invalid argument
killing process 175: Invalid argument
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ bash -login
root@guixydevel ~/src/guix/wip-hurd# 
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
receiving objects  49% [##################################                                   ]Illegal instruction (core dumped)
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd
Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'...
guix pull: error: Git error: cannot locate remote-tracking branch 'origin/keyring'
root@guixydevel ~/src/guix/wip-hurd# git branch keyring origin/keyring
branch 'keyring' set up to track 'origin/keyring'.
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd
Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'...
Authenticating channel 'guix', commits 9edb3f6 to beb2704 (26 new commits)...
guix pull: warning: pulled channel 'guix' from a mirror of https://git.savannah.gnu.org/git/guix.git, which might be stale
Building from this channel:
  guix      /root/src/guix/wip-hurd	beb2704
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
building /gnu/store/67p25fiinjvm1h0ga8bays6d0kyy0dp4-config.scm.drv...
building /gnu/store/iy189p9rd62ymac23l83h6vz86sx775y-git.scm.drv...
building /gnu/store/w5ld3vgpsd7im25zqjrk7rxpw3j6km3j-hash.scm.drv...
building /gnu/store/151iwm03mirbpbgsrrmxnpr97bj5w1ps-module-import.drv...
building /gnu/store/sdnlhfdqrg0kkh33adg4dcb15qnx4205-module-import.drv...
building /gnu/store/s835yp2frsy9mf4ha7q0b3mrg4mc78sc-module-import-compiled.drv...
building /gnu/store/sc3ik9br6k1fzkczg51qr54d9ilkl268-module-import-compiled.drv...
building /gnu/store/yn9hi0kmyygk29s79fa44nqh0wqip82n-compute-guix-derivation.drv...
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
 bash                                                       389KiB/s 00:02 | 907KiB transferred
 dbus-1.14.0                                                1.2MiB/s 00:00 | 355KiB transferred
 guile-json-3.5.0                                            5.9MiB/s 00:00 | 61KiB transferre|
 guix-1.4.0-12.329069a-checkout                            2.2MiB/s 00:09 | 20.3MiB transferred
 libdaemon-0.14                                           +inf.0TiB/s 00:00 | 39KiB transferred
 libevent-2.1.12                                            167KiB/s 00:03 | 463KiB transferred
 mkdir                                                      155KiB/s 00:03 | 432KiB transferred
 avahi-0.8                                                  229KiB/s 00:02 | 466KiB transferred
 tar                                                        3.2MiB/s 00:00 | 840KiB transferred
 guile-avahi-0.4.1                                            26KiB/s 00:03 | 73KiB transferred
 xz                                                         670KiB/s 00:01 | 529KiB transferred
building /gnu/store/nr3r518p14rj0672q37c320fkd00r01y-guix-daemon-1.4.0-12.329069a.drv...
\ 'build' phaseild-log 286 52
  CXX      nix/nix-daemon/guix_daemon-guix-daemon.o
| 'build' phase47
  CXX      nix/libstore/libstore_a-pathlocks.o
/ 'make-dynamic-linker-cache' phas/
killing process 283: Invalid argument
killing process 284: Invalid argument
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/k6z2f845lqzkz2h4q60w089rzl7p7ay7-profile.drv
  /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv
  /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv
  /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv
  /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv
  /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv
  /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv
  /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv
  /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv
  /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv
  /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv
  /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv
  /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv
  /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv
  /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv
  /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv
  /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv
  /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv
  /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv
  /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv
  /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv
  /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv
  /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv
  /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv
  /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv
  /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv
  /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv
  /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv
  /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv
  /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv
  /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv
  /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv
  /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv
  /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv
  /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv
  /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv
  /gnu/store/50fk0z57vprxh13pp000alz6200lm9r6-profile.drv
  /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv

5 items will be downloaded
 module-import-compiled                                      800KiB/s 00:00 | 64KiB transferred
 guile-wrapper                                                160KiB/s 00:00 | 3KiB transferred
 module-import-compiled                                      541KiB/s 00:00 | 81KiB transferred
 module-import-compiled                                     860KiB/s 00:00 | 138KiB transferred
 module-import-compiled                                      328KiB/s 00:00 | 10KiB transferred
building /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv...
building /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv...
building /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv...
building /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv...
building /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv...
building /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv...
building /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv...
building /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv...
building /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv...
building /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv...
building /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv...
building /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv...
building /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv...
building /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv...
building /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv...
building /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv...
building /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv...
building /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv...
building /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv...
building /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv...
building /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv...
building /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv...
building /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv...
building /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv...
building /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv...
building /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv...
building /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv...
building /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv...
building /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv...
building /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv...
building /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv...
building /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv...
building /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv...
building /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv...
building /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:

     GUIX_PROFILE="/root/.config/guix/current"
     . "$GUIX_PROFILE/etc/profile"

Alternately, see `guix package --search-paths -p "/root/.config/guix/current"'.


hint: After setting `PATH', run `hash guix' to make sure your shell refers to
`/root/.config/guix/current/bin/guix'.

root@guixydevel ~/src/guix/wip-hurd#      GUIX_PROFILE="/root/.config/guix/current"
     . "$GUIX_PROFILE/etc/profile"
~/src/guix/wip-hurd# type -p guix
/root/.config/guix/current/bin/guix
root@guixydevel ~/src/guix/wip-hurd# guix --version
guix (GNU Guix) beb27046836d60974390f6a940167e8b5def1066
Copyright (C) 2023 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
root@guixydevel ~/src/guix/wip-hurd# guix describe
Generation 1	Aug 24 2023 01:34:21	(current)
  guix beb2704
    repository URL: /root/src/guix/wip-hurd
    branch: wip-hurd
    commit: beb27046836d60974390f6a940167e8b5def1066
root@guixydevel ~/src/guix/wip-hurd# guix shell -D guix
hint: Consider passing the `--check' option once to make sure your shell does not clobber
environment variables.

root@guixydevel ~/src/guix/wip-hurd [env]# guix build hello
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
The following file will be downloaded:
  /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1
substituting /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1...
downloading from http://dezyne.org:8181/nar/zstd/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 ...
 hello-2.12.1                                                297KiB/s 00:00 | 74KiB transferred

/gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1
root@guixydevel ~/src/guix/wip-hurd [env]# 

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]


Janneke Nieuwenhuizen <janneke@gnu.org> writes:

> So, some good news at last; I can confirm that using v4 of this patch we
> now have "guix pull", pulling from a local git directory, now fully
> working on the Hurd!

That’s awesome! Thank you for your work!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Ludovic Courtès]

Hi!

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
> From: Janneke Nieuwenhuizen <janneke@gnu.org>
> Date: Thu, 22 Jun 2023 08:30:25 +0200
> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
>  time.
>
> Similar to split build of make-go in Makefile.am, this breaks-up building
> directories into chunks of max 25 files.  Also force garbage collection.

The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
for each chunk of files: each process starts with an empty heap, which
is not the case here as we reuse the same process.

However, (guix self) is already splitting gnu/packages/*.scm in two
pieces: ‘guix-packages-base’ and ‘guix-packages’.  The former is the
closure of (gnu packages base), and the latter contains the remaining
files.  Unfortunately this is uneven:

--8<---------------cut here---------------start------------->8---
$ readlink -f $(type -P guix)
/gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base
/gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
/gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
/gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
$ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base  -type f |wc -l
361
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$
/gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
64
$ guix describe
Generation 271  Aug 20 2023 23:48:59    (current)
  guix a0f5885
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05
--8<---------------cut here---------------end--------------->8---

Maxime Devos looked into this a while back:

  https://issues.guix.gnu.org/54539

> * guix/self.scm (compiled-modules)[process-directory]: Split building of
> directories into chunks of max 25 files.

[...]

>            (define (process-directory directory files output)
> -            ;; Hide compilation warnings.
> -            (parameterize ((current-warning-port (%make-void-port "w")))
> -              (compile-files directory #$output files
> -                             #:workers (parallel-job-count)
> -                             #:report-load report-load
> -                             #:report-compilation report-compilation)))
> +            (let* ((size 25)                      ;compile max 25 files a time
> +                   (chunks (unfold
> +                            (lambda (seed) (< (length seed) size)) ;p
> +                            (cute take <> size)                    ;f
> +                            (cute drop <> size)                    ;g
> +                            files                                  ;seed
> +                            list)))                                ;tail
> +              (for-each
> +               (lambda (chunck)

s/chunck/chunk/

Can you confirm that this reduces memory usage observably?  One way to
check that would be to print (gc-stats) from ‘process-directory’, with
and without the change.  Could you give it a try?

Intuitively, I don’t see why it would eat less memory; maybe peak memory
usage is lower because we do less at once?

Also, I think we should remove the explicit (gc) call: it should not be
necessary, and if we depend on that, something’s wrong.

Anyhow, thanks for tackling this issue!

Ludo’.

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Janneke Nieuwenhuizen]

Ludovic Courtès writes:

Hello!

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
>> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
>> From: Janneke Nieuwenhuizen <janneke@gnu.org>
>> Date: Thu, 22 Jun 2023 08:30:25 +0200
>> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
>>  time.
>>
>> Similar to split build of make-go in Makefile.am, this breaks-up building
>> directories into chunks of max 25 files.  Also force garbage collection.
>
> The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
> for each chunk of files: each process starts with an empty heap, which
> is not the case here as we reuse the same process.

Right.

> However, (guix self) is already splitting gnu/packages/*.scm in two
> pieces: ‘guix-packages-base’ and ‘guix-packages’.  The former is the
> closure of (gnu packages base), and the latter contains the remaining
> files.  Unfortunately this is uneven:

Okay...

> $ readlink -f $(type -P guix)
> /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
> $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base
> /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
> /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
> /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
> $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base  -type f |wc -l
> 361
> $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$
> /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
> ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
> 64
> $ guix describe
> Generation 271  Aug 20 2023 23:48:59    (current)
>   guix a0f5885
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05
>
> Maxime Devos looked into this a while back:
>
>   https://issues.guix.gnu.org/54539

Oh my....

>> * guix/self.scm (compiled-modules)[process-directory]: Split building of
>> directories into chunks of max 25 files.
>> +              (for-each
>> +               (lambda (chunck)
>
> s/chunck/chunk/

Oops, fixed.

> Can you confirm that this reduces memory usage observably?  One way to
> check that would be to print (gc-stats) from ‘process-directory’, with
> and without the change.  Could you give it a try?

What a good and seemingly simple question.  After a week of
instrumentation and testing, my answer can only be: I tried, and maybe.
(see below).

> Intuitively, I don’t see why it would eat less memory; maybe peak memory
> usage is lower because we do less at once?

Okay...

> Also, I think we should remove the explicit (gc) call: it should not be
> necessary, and if we depend on that, something’s wrong.

> Anyhow, thanks for tackling this issue!

Hehe.  You've probably seen Josselin's recent GraphML backend effort
that might really help to address this?  I'm afraid this patch can maybe
only postpone what really needs to be done...

There is gc-stats output from a successful `guix pull' or `make
as-derivation' on Guix/Hurd, that I can show you, and I've tried more
than 20 times; it always fails (OOM, hang, spontaneous reset, ...).

Below is a typical output of gc-stats on the Hurd for building self.scm,
when heap-size peaks (using the the max 25 files patch):

--8<---------------cut here---------------start------------->8---
((gc-time-taken . 1530)
 (heap-size . 2,625,474,560)
 (heap-free-size . 1127989248)
 (heap-total-allocated . 1337029496)
 (heap-allocated-since-gc . 28728)
 (protected-objects . 28)
 (gc-times . 324))
--8<---------------cut here---------------end--------------->8---

notice that it's *much* bigger (more than twice) than my findings on
linux-64 below.  I have no idea why this is of what it might mean...

So I turned to Guix GNU/Linux to get some gc-stat measurements.  What
you see below is the maximum head-size at any point (I also have
heap-total-allocated but I think that's irrelevant? and initially didn't
use a script that measured the time).

--8<---------------cut here---------------start------------->8---
* guix/self.scm: Vanilla, not chunked; print gc-stats.
((gc-time-taken . 27319485051)
 (heap-size . 1,360,330,752)
 (heap-free-size . 285,696,000)
 (heap-total-allocated . 74,067,590,944)
 (heap-allocated-since-gc . 186,250,144)
 (protected-objects . 28)
 (gc-times . 464))
real	24m36.643s

* guix/self.scm: Split building of directories into 26 chunks; print gc-stats.
 (heap-size . 1,131,298,816)

* guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats.
 (heap-size . 1,121,116,160)

* guix/self.scm: Chunks of 25 files; run gc; print gc-stats.
 (heap-size . 1,066,725,376)

* guix/self.scm: Chunks of 50 files; no gc; print gc-stats.
 (heap-size . 1,299,230,720)
real	26m40.708s

* guix/self.scm: Chunks of 25 files; no gc; print gc-stats.
 (heap-size . 1,024,045,056)  ; 1st run
real	28m4.451s

* guix/self.scm: Chunks of 10 files; no gc; print gc-stats.
 (heap-size . 1,077,895,168)
real	30m14.049s
--8<---------------cut here---------------end--------------->8---

...strangely enough, if we assume that these statistics translate to the
Hurd, using chunks of max 25 files seems to be a sort of sweet spot?
25% less peak memory (~300MB), "only" 12% (3"45') slower...  though not
great for GNU/Linux users...

I have produced a handful of successful `guix pull's (from a local
checked-out worktree) using the 26-way split and chunks of max-25 files
patches, but sadly also many more attempts failed.  Initially, when
creating this patch series, I was convinced this fixed building on the
Hurd, but I'm much less enthusiastic now.

So I still have a slight preference for using the latest max-25-files
patch, but I'm sorry to say that I cannot back it up with tangible data.
All in all a rather discouraging week with much effort spent for little
gain.  Hopefully Josselin can do some of his magic here :)

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

Hi!

> Ludovic Courtès writes:
>
> Hello!
>
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
[..]
>> Anyhow, thanks for tackling this issue!
>
> Hehe.  You've probably seen Josselin's recent GraphML backend effort
> that might really help to address this?  I'm afraid this patch can maybe
> only postpone what really needs to be done...
>
> There is gc-stats output from a successful `guix pull' or `make
> as-derivation' on Guix/Hurd, that I can show you, and I've tried more
> than 20 times; it always fails (OOM, hang, spontaneous reset, ...).
>
> Below is a typical output of gc-stats on the Hurd for building self.scm,
> when heap-size peaks (using the the max 25 files patch):
>
> ((gc-time-taken . 1530)
>  (heap-size . 2,625,474,560)
>  (heap-free-size . 1127989248)
>  (heap-total-allocated . 1337029496)
>  (heap-allocated-since-gc . 28728)
>  (protected-objects . 28)
>  (gc-times . 324))
>
>
> notice that it's *much* bigger (more than twice) than my findings on
> linux-64 below.  I have no idea why this is of what it might mean...
>
> So I turned to Guix GNU/Linux to get some gc-stat measurements.  What
> you see below is the maximum head-size at any point (I also have
> heap-total-allocated but I think that's irrelevant? and initially didn't
> use a script that measured the time).
>
> * guix/self.scm: Vanilla, not chunked; print gc-stats.
> ((gc-time-taken . 27319485051)
>  (heap-size . 1,360,330,752)
>  (heap-free-size . 285,696,000)
>  (heap-total-allocated . 74,067,590,944)
>  (heap-allocated-since-gc . 186,250,144)
>  (protected-objects . 28)
>  (gc-times . 464))
> real	24m36.643s
>
> * guix/self.scm: Split building of directories into 26 chunks; print gc-stats.
>  (heap-size . 1,131,298,816)
>
> * guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats.
>  (heap-size . 1,121,116,160)
>
> * guix/self.scm: Chunks of 25 files; run gc; print gc-stats.
>  (heap-size . 1,066,725,376)
>
> * guix/self.scm: Chunks of 50 files; no gc; print gc-stats.
>  (heap-size . 1,299,230,720)
> real	26m40.708s
>
> * guix/self.scm: Chunks of 25 files; no gc; print gc-stats.
>  (heap-size . 1,024,045,056)  ; 1st run
> real	28m4.451s
>
> * guix/self.scm: Chunks of 10 files; no gc; print gc-stats.
>  (heap-size . 1,077,895,168)
> real	30m14.049s
>
> ...strangely enough, if we assume that these statistics translate to the
> Hurd, using chunks of max 25 files seems to be a sort of sweet spot?
> 25% less peak memory (~300MB), "only" 12% (3"45') slower...  though not
> great for GNU/Linux users...
>
> I have produced a handful of successful `guix pull's (from a local
> checked-out worktree) using the 26-way split and chunks of max-25 files
> patches, but sadly also many more attempts failed.  Initially, when
> creating this patch series, I was convinced this fixed building on the
> Hurd, but I'm much less enthusiastic now.
>
> So I still have a slight preference for using the latest max-25-files
> patch, but I'm sorry to say that I cannot back it up with tangible data.
> All in all a rather discouraging week with much effort spent for little
> gain.  Hopefully Josselin can do some of his magic here :)

Anyway, I've finally

--8<---------------cut here---------------start------------->8---
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b2cc649999ed97b62517dcfb7cf21ee731c97487>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f2cfb4a85c82882c571274573fd7ff646d380b63>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=41df5c5289c970b40a002a32dcfdcb61ddc5a2f5>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bd303443be97c5aa5f8696f6f1ecc5e40b919874>
--8<---------------cut here---------------end--------------->8---

managed to get all remaining necessary dependencies built to run `guix
system reconfigure' on the Hurd*.  The only patch that we need that's not
on master yet is this one.

I'm still rooting for Josselin's graphml cycle analysis and/or lechner
Felix's bespoke make rewrite will render this hack obsolete real soon
now...

If there are no objections I'll go forward and install this patch by the
end of the weekend.

Greetings,
Janneke

*) using the hurd-team branch, grub install still fails, apparently
   because "someone" imagines /dev/wd0 to be /dev/hd0, but the new
   system derivation is being enabled.

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

Hi!

> Janneke Nieuwenhuizen writes:
>
>> Ludovic Courtès writes:

> If there are no objections I'll go forward and install this patch by the
> end of the weekend.

So, I've been running guix pull from hurd-team on the Hurd all day.
Starting with the max-25 files split without (gc), it failed 5
consecutive times building guix-packages-base.  Because the initial
patch worked pretty well and had the (gc) call (and yes, also used the
26-way split), I re-added (gc), and now the fourth attempt succeeded.
After `guix-packages-base' built, `guix-packages' then built right away.

I seem to remember guix-packages was the main problem when I wrote the
first version of this patch some four months ago, but that one split the
list of packages 26-way of course.  That suggests that the (combined)
compilation of certain files is especially problematic, or that it has
since become more problematic (<https://issues.guix.gnu.org/66063>).

On the other hand, we may be able to use more than 4GiB of memory on the
Hurd some time soon (and after we upgrade and compile using
--enable-pae)
<https://lists.gnu.org/archive/html/bug-hurd/2023-08/msg00090.html>.

Pushed to master as 658de25e990a56265e2398a3737e9cf1eb57af68

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com