* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. @ 2023-08-22 17:17 Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen ` (2 more replies) 0 siblings, 3 replies; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:17 UTC (permalink / raw) To: 65456 Cc: Christopher Baines, Josselin Poiret, Ludovic Courtès, Mathieu Othacehe, Ricardo Wurmus, Simon Tournier, Tobias Geerinckx-Rice Hi! Initially writter for the Hurd (see hurd-team <https://git.savannah.gnu.org/cgit/guix.git/commit/?h=hurd-team&id=06438c2a7edf1f0c90392858fc86480c6e5c2195>, <https://git.savannah.gnu.org/cgit/guix.git/commit/?h=hurd-team&id=5623ab503fa70eab95e6f859e82c89581c3f69f4>) This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit issue, rather than a Hurd problem (<https://logs.guix.gnu.org/guix/2023-08-22.log#182716>, <https://ci.guix.gnu.org/build/1870536/log/raw>). I didn't submit these for the Hurd yet, as I didn't really feel comfortable with the 26-way split. I tried a 5-way split and that still gave OOM errors on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah. Also, the 5-way split in the Makefile.am will produce percentages over 100 which suggests a typo...but I couldn't spot it. Lastly, these patches feel like a workaround for Guile / libgc memory management? Otoh, being able to build Guix on 32bit hosts is kinda nice too... So yeah. Greetings, Janneke Janneke Nieuwenhuizen (2): build: Build gnu/packages/*.go in five steps. self: Build gnu/packages/*.go in 26 steps on 32bit. Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++------- guix/self.scm | 31 ++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 10 deletions(-) base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps. 2023-08-22 17:17 bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Janneke Nieuwenhuizen @ 2023-08-22 17:19 ` Janneke Nieuwenhuizen 2023-08-22 17:33 ` Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit Janneke Nieuwenhuizen 2023-08-22 17:49 ` bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen 2 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:19 UTC (permalink / raw) To: 65456 This breaks-up packages into five chunks of ~150,000 lines, allowing guix build --target=i586-pc-gnu from an x86 host. This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294. * Makefile.am (first_half): Rename to... (first_quart): ...this, and also split into... (second_quart): ...this. (third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables. (make-packages3-go, make-packages4-go): New targets. (make-packages-go): Add them. --- Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 54 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index 738532f839..7c8c215f86 100644 --- a/Makefile.am +++ b/Makefile.am @@ -776,19 +776,45 @@ endef # in <https://issues.guix.gnu.org/48963>. Each 'eval' call below creates a # 'make-*-go' phony target that builds the corresponding subset. -first_half := \ - gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \ - gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \ - gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l% +first_fifth := \ + gnu/packages/a% gnu/packages/b% gnu/packages/c% + +second_fifth := \ + gnu/packages/d% gnu/packages/e% gnu/packages/f% + +third_fifth := \ + gnu/packages/g% gnu/packages/h% gnu/packages/i% + +fourth_fifth := \ + gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \ + gnu/packages/n% gnu/packages/o% MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES))) -MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES)) -MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES))) -MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) +MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES)) +MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES)) +MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES)) +MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES)) +MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \ + $(third_fifth) $(fourth_fifth), \ + $(filter gnu/packages/%,$(MODULES))) +MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \ + $(MODULES_PACKAGES5) MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI := $(filter guix/scripts/%,$(MODULES)) MODULES_PO := guix/build/po.scm +print-p1: + @echo $(MODULES_PACKAGES1) +print-p2: + @echo $(MODULES_PACKAGES2) +print-p3: + @echo $(MODULES_PACKAGES3) +print-p4: + @echo $(MODULES_PACKAGES4) +print-p5: + @echo $(MODULES_PACKAGES5) + $(eval $(call guile-compilation-rule,make-core-go, \ $(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \ 0)) @@ -804,7 +830,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES1)))) .PHONY: make-packages2-go -make-packages-go: make-packages1-go make-packages2-go +$(eval $(call guile-compilation-rule,make-packages3-go, \ + $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)))) +.PHONY: make-packages3-go + +$(eval $(call guile-compilation-rule,make-packages4-go, \ + $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3)))) +.PHONY: make-packages4-go + +$(eval $(call guile-compilation-rule,make-packages5-go, \ + $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4)))) +.PHONY: make-packages5-go + +make-packages-go: make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go make-packages5-go .PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ -- 2.41.0 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps. 2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen @ 2023-08-22 17:33 ` Janneke Nieuwenhuizen 0 siblings, 0 replies; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:33 UTC (permalink / raw) To: 65456 Janneke Nieuwenhuizen writes: > * Makefile.am (first_half): Rename to... [..] > +print-p1: > + @echo $(MODULES_PACKAGES1) > +print-p2: > + @echo $(MODULES_PACKAGES2) > +print-p3: > + @echo $(MODULES_PACKAGES3) > +print-p4: > + @echo $(MODULES_PACKAGES4) > +print-p5: > + @echo $(MODULES_PACKAGES5) > + ...I mean to remove this debugging code, but it might come in handy for review. -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit. 2023-08-22 17:17 bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen @ 2023-08-22 17:19 ` Janneke Nieuwenhuizen 2023-08-22 21:51 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès 2023-08-22 17:49 ` bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen 2 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:19 UTC (permalink / raw) To: 65456 Cc: Christopher Baines, Josselin Poiret, Ludovic Courtès, Mathieu Othacehe, Ricardo Wurmus, Simon Tournier, Tobias Geerinckx-Rice Similar to the Makefile.am change, this breaks-up packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define. Use it when building on a "i586" or "i686" cpu. --- guix/self.scm | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..474715fe7a 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1251,6 +1255,26 @@ (define* (compiled-modules name module-tree module-files #:report-load report-load #:report-compilation report-compilation))) + (define (process-directory/32bit directory files output) + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^gnu/packages/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation) + (gc)) + chunks))) + (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) @@ -1277,7 +1301,10 @@ (define* (compiled-modules name module-tree module-files (mkdir #$output) (chdir #+module-tree) - (process-directory "." '#+module-files #$output) + (match (string-split %host-type #\-) + (((or "i586" "i686") rest ...) + (process-directory/32bit "." '#+module-files #$output)) + (_ (process-directory "." '#+module-files #$output))) (newline)))) (computed-file name build -- 2.41.0 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. 2023-08-22 17:19 ` bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit Janneke Nieuwenhuizen @ 2023-08-22 21:51 ` Ludovic Courtès 2023-08-23 6:16 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Ludovic Courtès @ 2023-08-22 21:51 UTC (permalink / raw) To: Janneke Nieuwenhuizen Cc: Josselin Poiret, Tobias Geerinckx-Rice, Simon Tournier, Mathieu Othacehe, 65456, Ricardo Wurmus, Christopher Baines Hi, Janneke Nieuwenhuizen <janneke@gnu.org> skribis: > Similar to the Makefile.am change, this breaks-up packages into 26 chunks > when building on 32bit. Also force garbage collection. > > * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define. > Use it when building on a "i586" or "i686" cpu. .go files of a Guix built with this new (guix self) would not be found: --8<---------------cut here---------------start------------->8--- newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0 newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/share/guile/site/3.0/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/current-system/profile/lib/guile/3.0/site-ccache/guix/ui.go", {st_mode=S_IFREG|0444, st_size=530053, ...}, 0) = 0 --8<---------------cut here---------------end--------------->8--- In fact, guix/*.go is entirely missing, it seems: $ --8<---------------cut here---------------start------------->8--- ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix ls: cannot access '/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix': No such file or directory $ ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/gnu packages/ --8<---------------cut here---------------end--------------->8--- With the patch, without the patch: --8<---------------cut here---------------start------------->8--- $ find -L /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union -name \*.go | wc -l 719 $ find -L /gnu/store/4dm864f7iif1vg9sz355gdqkgb7air98-guix-module-union -name \*.go | wc -l 1188 --8<---------------cut here---------------end--------------->8--- The problem would then manifest like this: --8<---------------cut here---------------start------------->8--- $ /gnu/store/57x6wrf78147lif1ss0ck2z3a0323b9l-guix-20230822.21/bin/guix build hello error: license:arphic-1999: unbound variable hint: Did you forget a `use-modules' form? error: googletest: unbound variable hint: Did you forget a `use-modules' form? […] Throw to key `unbound-variable' with args `("resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f)'. Backtrace: In guix/store.scm: 659:37 19 (thunk) 1298:8 18 (call-with-build-handler #<procedure 7f82c427ac60 at guix/ui.scm:1196:2 (continu…> …) In guix/scripts/build.scm: 584:2 17 (_) In srfi/srfi-1.scm: 673:15 16 (append-map _ _ . _) 586:17 15 (map1 ((argument . "hello") (build-mode . 0) (graft? . #t) (substitutes? . #t) # …)) In guix/scripts/build.scm: 604:31 14 (_ _) In gnu/packages.scm: 485:2 13 (%find-package "hello" "hello" #f) 365:6 12 (find-best-packages-by-name _ _) 295:56 11 (_ "hello" _) In unknown file: 10 (force #<promise #<procedure 7f82c6c5eee0 at gnu/packages.scm:285:18 ()>>) In gnu/packages.scm: 242:33 9 (fold-packages #<procedure 7f82c21125f0 at gnu/packages.scm:286:35 (p r)> #<vlis…> …) In guix/discovery.scm: 158:11 8 (all-modules _ #:warn _) In srfi/srfi-1.scm: 460:18 7 (fold #<procedure 7f82c6cc7f00 at guix/discovery.scm:142:8 (spec result)> _ ((. #))) In guix/discovery.scm: 148:19 6 (_ _ ()) 115:5 5 (scheme-modules _ _ #:warn _) In srfi/srfi-1.scm: 691:23 4 (filter-map #<procedure 7f82c6cc7de0 at guix/discovery.scm:115:16 (file)> _ . _) In guix/discovery.scm: 123:24 3 (_ . _) In guix/ui.scm: 343:2 2 (report-unbound-variable-error _ #:frame _) In ice-9/boot-9.scm: 1685:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `match-error' with args `("match" "no matching pattern" (unbound-variable "resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f))'. --8<---------------cut here---------------end--------------->8--- This is because an old (guix licenses) module would be loaded. I’m reverting for now. Note that (guix self) is very sensitive, so we should test it thoroughly. Here I used ‘make as-derivation’ and ran the resulting ‘guix’ command. Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. 2023-08-22 21:51 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès @ 2023-08-23 6:16 ` Janneke Nieuwenhuizen 2023-08-23 9:41 ` bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time Janneke Nieuwenhuizen 2023-08-24 5:33 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 0 siblings, 2 replies; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-23 6:16 UTC (permalink / raw) To: Ludovic Courtès Cc: Josselin Poiret, Tobias Geerinckx-Rice, Simon Tournier, Mathieu Othacehe, 65456, Ricardo Wurmus, Christopher Baines [-- Attachment #1: Type: text/plain, Size: 3212 bytes --] Ludovic Courtès writes: Hi! > Janneke Nieuwenhuizen <janneke@gnu.org> skribis: > >> Similar to the Makefile.am change, this breaks-up packages into 26 chunks >> when building on 32bit. Also force garbage collection. >> >> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define. >> Use it when building on a "i586" or "i686" cpu. > > .go files of a Guix built with this new (guix self) would not be found: > > newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0 Hmm, how did I miss this? > In fact, guix/*.go is entirely missing, it seems: Right...that makes sense now that I look at the code again. Wait...I probably even experienced this breakage after running `guix pull' on the Hurd but failed to notice its cause, the missing guix/*.go, and ascribed it to "something" being broken on Hurd. So the good news is that we'll most probably have guix pull work on the Hurd after fixing this! --8<---------------cut here---------------start------------->8--- (cute partition <> files) (lambda (regex) (cute string-match regex <>)) (cute string-append "^gnu/packages/" <>) --8<---------------cut here---------------end--------------->8--- it still has the "gnu/packages" directory name hardcoded here. (I even believe this was kind-of intentional, meaning not to split-up builds of other directories...). The result is, of course, that partition always produces an empty result for any '("guix/foo.scm") files! It is fixed by doing --8<---------------cut here---------------start------------->8--- (cute string-append "^" directory "/" <>) --8<---------------cut here---------------end--------------->8--- i.e., also splitting guix/ 26 ways. > I’m reverting for now. Thank you for looking into and taking care of this mess I made, sorry! > Note that (guix self) is very sensitive, so we should test it > thoroughly. Using the attached patch, I ran --8<---------------cut here---------------start------------->8--- $ guix pull --branch=master --url=$PWD [..] 08:11:02 janneke@drakenpad:~/src/guix/master $ guix describe Git checkout: repository: /home/janneke/src/guix/master/ branch: master commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 08:11:04 janneke@drakenpad:~/src/guix/master $ guix build hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- > Here I used ‘make as-derivation’ and ran the resulting > ‘guix’ command. Ah, nice that guix works too for me too now --8<---------------cut here---------------start------------->8--- $ make as-derivation [..] /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06 08:13:15 janneke@drakenpad:~/src/guix/master [env] $ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix build hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- Greetings, Janneke [-- Attachment #2: v3-0001-self-Build-guix-and-gnu-packages-directories-in-2.patch --] [-- Type: text/x-patch, Size: 3600 bytes --] From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001 Message-ID: <a639a64d590067d3fe928be3ecc9d2d4cc2e8df3.1692770000.git.janneke@gnu.org> From: Janneke Nieuwenhuizen <janneke@gnu.org> Date: Thu, 22 Jun 2023 08:30:25 +0200 Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of directories into 26 chunks. --- guix/self.scm | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..e5a85eaad6 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files (force-output)) (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + ;; Split gnu/packages in 26 chunks to avoid OOM errors + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^" directory "/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + ;; Hide compilation warnings. + (parameterize ((current-warning-port (%make-void-port "w"))) + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea -- 2.41.0 [-- Attachment #3: Type: text/plain, Size: 164 bytes --] -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time. 2023-08-23 6:16 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen @ 2023-08-23 9:41 ` Janneke Nieuwenhuizen 2023-08-24 14:42 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès 2023-08-24 5:33 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 1 sibling, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-23 9:41 UTC (permalink / raw) To: Ludovic Courtès Cc: Josselin Poiret, Tobias Geerinckx-Rice, Simon Tournier, Mathieu Othacehe, 65456, Ricardo Wurmus, Christopher Baines [-- Attachment #1: Type: text/plain, Size: 1062 bytes --] Janneke Nieuwenhuizen writes: Hi! > Ludovic Courtès writes: >> Janneke Nieuwenhuizen <janneke@gnu.org> skribis: >> [..] > it still has the "gnu/packages" directory name hardcoded here. (I even > believe this was kind-of intentional, meaning not to split-up builds of > other directories...). The result is, of course, that partition always > produces an empty result for any '("guix/foo.scm") files! > > It is fixed by doing > > (cute string-append "^" directory "/" <>) Well, almost...In my test script it worked, but directory is "." in the actual self.scm. So with v3 no .go files get built at all...and then a naive `guix build hello' seems to work!; `guix shell' for example fails. Comparing the list of files from `find /gnu/store/...-guix-20230823.08 -follow' is also helpful. I realised that there's no need to use the name-based splitting hack that Makefile.am uses and changed the creation of chunks to simply use a maximum chunk length of 25 files. New version attached. Greetings, Janneke [-- Attachment #2: v4-0001-self-Build-directories-in-chunks-of-max-25-files-.patch --] [-- Type: text/x-patch, Size: 3288 bytes --] From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org> From: Janneke Nieuwenhuizen <janneke@gnu.org> Date: Thu, 22 Jun 2023 08:30:25 +0200 Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a time. Similar to split build of make-go in Makefile.am, this breaks-up building directories into chunks of max 25 files. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of directories into chunks of max 25 files. --- guix/self.scm | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..fc2dfd8131 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) (ice-9 match) (ice-9 format) (ice-9 threads) @@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-files (force-output)) (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + (let* ((size 25) ;compile max 25 files a time + (chunks (unfold + (lambda (seed) (< (length seed) size)) ;p + (cute take <> size) ;f + (cute drop <> size) ;g + files ;seed + list))) ;tail + (for-each + (lambda (chunck) + ;; Hide compilation warnings. + (parameterize ((current-warning-port (%make-void-port "w"))) + (compile-files directory output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea -- 2.41.0 [-- Attachment #3: Type: text/plain, Size: 164 bytes --] -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. 2023-08-23 9:41 ` bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time Janneke Nieuwenhuizen @ 2023-08-24 14:42 ` Ludovic Courtès 2023-09-01 12:48 ` Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Ludovic Courtès @ 2023-08-24 14:42 UTC (permalink / raw) To: Janneke Nieuwenhuizen Cc: Josselin Poiret, 65456, Simon Tournier, Mathieu Othacehe, Tobias Geerinckx-Rice, Ricardo Wurmus, Christopher Baines Hi! Janneke Nieuwenhuizen <janneke@gnu.org> skribis: >>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 > Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org> > From: Janneke Nieuwenhuizen <janneke@gnu.org> > Date: Thu, 22 Jun 2023 08:30:25 +0200 > Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a > time. > > Similar to split build of make-go in Makefile.am, this breaks-up building > directories into chunks of max 25 files. Also force garbage collection. The big difference with ‘make-go’ is that ‘make-go’ spawns a new process for each chunk of files: each process starts with an empty heap, which is not the case here as we reuse the same process. However, (guix self) is already splitting gnu/packages/*.scm in two pieces: ‘guix-packages-base’ and ‘guix-packages’. The former is the closure of (gnu packages base), and the latter contains the remaining files. Unfortunately this is uneven: --8<---------------cut here---------------start------------->8--- $ readlink -f $(type -P guix) /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -type f |wc -l 361 $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$ /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l 64 $ guix describe Generation 271 Aug 20 2023 23:48:59 (current) guix a0f5885 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05 --8<---------------cut here---------------end--------------->8--- Maxime Devos looked into this a while back: https://issues.guix.gnu.org/54539 > * guix/self.scm (compiled-modules)[process-directory]: Split building of > directories into chunks of max 25 files. [...] > (define (process-directory directory files output) > - ;; Hide compilation warnings. > - (parameterize ((current-warning-port (%make-void-port "w"))) > - (compile-files directory #$output files > - #:workers (parallel-job-count) > - #:report-load report-load > - #:report-compilation report-compilation))) > + (let* ((size 25) ;compile max 25 files a time > + (chunks (unfold > + (lambda (seed) (< (length seed) size)) ;p > + (cute take <> size) ;f > + (cute drop <> size) ;g > + files ;seed > + list))) ;tail > + (for-each > + (lambda (chunck) s/chunck/chunk/ Can you confirm that this reduces memory usage observably? One way to check that would be to print (gc-stats) from ‘process-directory’, with and without the change. Could you give it a try? Intuitively, I don’t see why it would eat less memory; maybe peak memory usage is lower because we do less at once? Also, I think we should remove the explicit (gc) call: it should not be necessary, and if we depend on that, something’s wrong. Anyhow, thanks for tackling this issue! Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. 2023-08-24 14:42 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès @ 2023-09-01 12:48 ` Janneke Nieuwenhuizen 2023-09-16 15:16 ` Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-09-01 12:48 UTC (permalink / raw) To: Ludovic Courtès Cc: Josselin Poiret, 65456, Simon Tournier, Mathieu Othacehe, Tobias Geerinckx-Rice, Ricardo Wurmus, Christopher Baines, Janneke Nieuwenhuizen Ludovic Courtès writes: Hello! > Janneke Nieuwenhuizen <janneke@gnu.org> skribis: > >>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 >> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org> >> From: Janneke Nieuwenhuizen <janneke@gnu.org> >> Date: Thu, 22 Jun 2023 08:30:25 +0200 >> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a >> time. >> >> Similar to split build of make-go in Makefile.am, this breaks-up building >> directories into chunks of max 25 files. Also force garbage collection. > > The big difference with ‘make-go’ is that ‘make-go’ spawns a new process > for each chunk of files: each process starts with an empty heap, which > is not the case here as we reuse the same process. Right. > However, (guix self) is already splitting gnu/packages/*.scm in two > pieces: ‘guix-packages-base’ and ‘guix-packages’. The former is the > closure of (gnu packages base), and the latter contains the remaining > files. Unfortunately this is uneven: Okay... > $ readlink -f $(type -P guix) > /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base > /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base > /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source > /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules > $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -type f |wc -l > 361 > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$ > /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages > ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l > 64 > $ guix describe > Generation 271 Aug 20 2023 23:48:59 (current) > guix a0f5885 > repository URL: https://git.savannah.gnu.org/git/guix.git > branch: master > commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05 > > Maxime Devos looked into this a while back: > > https://issues.guix.gnu.org/54539 Oh my.... >> * guix/self.scm (compiled-modules)[process-directory]: Split building of >> directories into chunks of max 25 files. >> + (for-each >> + (lambda (chunck) > > s/chunck/chunk/ Oops, fixed. > Can you confirm that this reduces memory usage observably? One way to > check that would be to print (gc-stats) from ‘process-directory’, with > and without the change. Could you give it a try? What a good and seemingly simple question. After a week of instrumentation and testing, my answer can only be: I tried, and maybe. (see below). > Intuitively, I don’t see why it would eat less memory; maybe peak memory > usage is lower because we do less at once? Okay... > Also, I think we should remove the explicit (gc) call: it should not be > necessary, and if we depend on that, something’s wrong. > Anyhow, thanks for tackling this issue! Hehe. You've probably seen Josselin's recent GraphML backend effort that might really help to address this? I'm afraid this patch can maybe only postpone what really needs to be done... There is gc-stats output from a successful `guix pull' or `make as-derivation' on Guix/Hurd, that I can show you, and I've tried more than 20 times; it always fails (OOM, hang, spontaneous reset, ...). Below is a typical output of gc-stats on the Hurd for building self.scm, when heap-size peaks (using the the max 25 files patch): --8<---------------cut here---------------start------------->8--- ((gc-time-taken . 1530) (heap-size . 2,625,474,560) (heap-free-size . 1127989248) (heap-total-allocated . 1337029496) (heap-allocated-since-gc . 28728) (protected-objects . 28) (gc-times . 324)) --8<---------------cut here---------------end--------------->8--- notice that it's *much* bigger (more than twice) than my findings on linux-64 below. I have no idea why this is of what it might mean... So I turned to Guix GNU/Linux to get some gc-stat measurements. What you see below is the maximum head-size at any point (I also have heap-total-allocated but I think that's irrelevant? and initially didn't use a script that measured the time). --8<---------------cut here---------------start------------->8--- * guix/self.scm: Vanilla, not chunked; print gc-stats. ((gc-time-taken . 27319485051) (heap-size . 1,360,330,752) (heap-free-size . 285,696,000) (heap-total-allocated . 74,067,590,944) (heap-allocated-since-gc . 186,250,144) (protected-objects . 28) (gc-times . 464)) real 24m36.643s * guix/self.scm: Split building of directories into 26 chunks; print gc-stats. (heap-size . 1,131,298,816) * guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats. (heap-size . 1,121,116,160) * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. (heap-size . 1,066,725,376) * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. (heap-size . 1,299,230,720) real 26m40.708s * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. (heap-size . 1,024,045,056) ; 1st run real 28m4.451s * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. (heap-size . 1,077,895,168) real 30m14.049s --8<---------------cut here---------------end--------------->8--- ...strangely enough, if we assume that these statistics translate to the Hurd, using chunks of max 25 files seems to be a sort of sweet spot? 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not great for GNU/Linux users... I have produced a handful of successful `guix pull's (from a local checked-out worktree) using the 26-way split and chunks of max-25 files patches, but sadly also many more attempts failed. Initially, when creating this patch series, I was convinced this fixed building on the Hurd, but I'm much less enthusiastic now. So I still have a slight preference for using the latest max-25-files patch, but I'm sorry to say that I cannot back it up with tangible data. All in all a rather discouraging week with much effort spent for little gain. Hopefully Josselin can do some of his magic here :) Greetings, Janneke -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. 2023-09-01 12:48 ` Janneke Nieuwenhuizen @ 2023-09-16 15:16 ` Janneke Nieuwenhuizen 2023-09-18 4:52 ` Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-09-16 15:16 UTC (permalink / raw) To: Ludovic Courtès Cc: Josselin Poiret, 65456, Simon Tournier, Mathieu Othacehe, Tobias Geerinckx-Rice, Ricardo Wurmus, Christopher Baines Janneke Nieuwenhuizen writes: Hi! > Ludovic Courtès writes: > > Hello! > >> Janneke Nieuwenhuizen <janneke@gnu.org> skribis: >> >>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 [..] >> Anyhow, thanks for tackling this issue! > > Hehe. You've probably seen Josselin's recent GraphML backend effort > that might really help to address this? I'm afraid this patch can maybe > only postpone what really needs to be done... > > There is gc-stats output from a successful `guix pull' or `make > as-derivation' on Guix/Hurd, that I can show you, and I've tried more > than 20 times; it always fails (OOM, hang, spontaneous reset, ...). > > Below is a typical output of gc-stats on the Hurd for building self.scm, > when heap-size peaks (using the the max 25 files patch): > > ((gc-time-taken . 1530) > (heap-size . 2,625,474,560) > (heap-free-size . 1127989248) > (heap-total-allocated . 1337029496) > (heap-allocated-since-gc . 28728) > (protected-objects . 28) > (gc-times . 324)) > > > notice that it's *much* bigger (more than twice) than my findings on > linux-64 below. I have no idea why this is of what it might mean... > > So I turned to Guix GNU/Linux to get some gc-stat measurements. What > you see below is the maximum head-size at any point (I also have > heap-total-allocated but I think that's irrelevant? and initially didn't > use a script that measured the time). > > * guix/self.scm: Vanilla, not chunked; print gc-stats. > ((gc-time-taken . 27319485051) > (heap-size . 1,360,330,752) > (heap-free-size . 285,696,000) > (heap-total-allocated . 74,067,590,944) > (heap-allocated-since-gc . 186,250,144) > (protected-objects . 28) > (gc-times . 464)) > real 24m36.643s > > * guix/self.scm: Split building of directories into 26 chunks; print gc-stats. > (heap-size . 1,131,298,816) > > * guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats. > (heap-size . 1,121,116,160) > > * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. > (heap-size . 1,066,725,376) > > * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. > (heap-size . 1,299,230,720) > real 26m40.708s > > * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. > (heap-size . 1,024,045,056) ; 1st run > real 28m4.451s > > * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. > (heap-size . 1,077,895,168) > real 30m14.049s > > ...strangely enough, if we assume that these statistics translate to the > Hurd, using chunks of max 25 files seems to be a sort of sweet spot? > 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not > great for GNU/Linux users... > > I have produced a handful of successful `guix pull's (from a local > checked-out worktree) using the 26-way split and chunks of max-25 files > patches, but sadly also many more attempts failed. Initially, when > creating this patch series, I was convinced this fixed building on the > Hurd, but I'm much less enthusiastic now. > > So I still have a slight preference for using the latest max-25-files > patch, but I'm sorry to say that I cannot back it up with tangible data. > All in all a rather discouraging week with much effort spent for little > gain. Hopefully Josselin can do some of his magic here :) Anyway, I've finally --8<---------------cut here---------------start------------->8--- <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b2cc649999ed97b62517dcfb7cf21ee731c97487> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f2cfb4a85c82882c571274573fd7ff646d380b63> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=41df5c5289c970b40a002a32dcfdcb61ddc5a2f5> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bd303443be97c5aa5f8696f6f1ecc5e40b919874> --8<---------------cut here---------------end--------------->8--- managed to get all remaining necessary dependencies built to run `guix system reconfigure' on the Hurd*. The only patch that we need that's not on master yet is this one. I'm still rooting for Josselin's graphml cycle analysis and/or lechner Felix's bespoke make rewrite will render this hack obsolete real soon now... If there are no objections I'll go forward and install this patch by the end of the weekend. Greetings, Janneke *) using the hurd-team branch, grub install still fails, apparently because "someone" imagines /dev/wd0 to be /dev/hd0, but the new system derivation is being enabled. -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. 2023-09-16 15:16 ` Janneke Nieuwenhuizen @ 2023-09-18 4:52 ` Janneke Nieuwenhuizen 0 siblings, 0 replies; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-09-18 4:52 UTC (permalink / raw) To: Ludovic Courtès Cc: Josselin Poiret, Simon Tournier, Mathieu Othacehe, Tobias Geerinckx-Rice, 65456-done, Ricardo Wurmus, Christopher Baines Janneke Nieuwenhuizen writes: Hi! > Janneke Nieuwenhuizen writes: > >> Ludovic Courtès writes: > If there are no objections I'll go forward and install this patch by the > end of the weekend. So, I've been running guix pull from hurd-team on the Hurd all day. Starting with the max-25 files split without (gc), it failed 5 consecutive times building guix-packages-base. Because the initial patch worked pretty well and had the (gc) call (and yes, also used the 26-way split), I re-added (gc), and now the fourth attempt succeeded. After `guix-packages-base' built, `guix-packages' then built right away. I seem to remember guix-packages was the main problem when I wrote the first version of this patch some four months ago, but that one split the list of packages 26-way of course. That suggests that the (combined) compilation of certain files is especially problematic, or that it has since become more problematic (<https://issues.guix.gnu.org/66063>). On the other hand, we may be able to use more than 4GiB of memory on the Hurd some time soon (and after we upgrade and compile using --enable-pae) <https://lists.gnu.org/archive/html/bug-hurd/2023-08/msg00090.html>. Pushed to master as 658de25e990a56265e2398a3737e9cf1eb57af68 Greetings, Janneke -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. 2023-08-23 6:16 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 2023-08-23 9:41 ` bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time Janneke Nieuwenhuizen @ 2023-08-24 5:33 ` Janneke Nieuwenhuizen 2023-08-24 6:44 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-24 5:33 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Josselin Poiret, 65456 Janneke Nieuwenhuizen writes: Hi! > Ludovic Courtès writes: > > Hi! > >> Janneke Nieuwenhuizen <janneke@gnu.org> skribis: >> [..] >> In fact, guix/*.go is entirely missing, it seems: > > Right...that makes sense now that I look at the code again. > > Wait...I probably even experienced this breakage after running `guix > pull' on the Hurd but failed to notice its cause, the missing guix/*.go, > and ascribed it to "something" being broken on Hurd. So the good news > is that we'll most probably have guix pull work on the Hurd after fixing > this! So, some good news at last; I can confirm that using v4 of this patch we now have "guix pull", pulling from a local git directory, now fully working on the Hurd! (When pulling from a git url, I get "Illegal instruction" while receiving objects.) See log below. Greetings, Janneke /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=http://gitlab.com/janneke/guix --branch=wip-hurd Updating channel 'guix' from Git repository at 'http://gitlab.com/janneke/guix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix install nss-certs guix install: warning: Consider running 'guix pull' followed by 'guix package -u' to get up-to-date packages and security updates. The following package will be installed: nss-certs 3.88.1 substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/z1jyx8lyhgr1gykiky5wjh5mncwvp6ls-profile.drv /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv 71.6 MB will be downloaded certdata2pem.c 2KiB 23KiB/s 00:00 [##################] 100.0% nss-3.88.1.tar.gz 68.3MiB 2.1MiB/s 00:32 [##################] 100.0% building /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv... building /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv... building /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv... building CA certificate bundle... listing Emacs sub-directories... building fonts directory... building directory of Info manuals... building profile with 1 package... killing process 174: Invalid argument killing process 175: Invalid argument /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ bash -login root@guixydevel ~/src/guix/wip-hurd# root@guixydevel ~/src/guix/wip-hurd# guix pull --url=https://gitlab.com/janneke/guix --branch=wip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'... receiving objects 49% [################################## ]Illegal instruction (core dumped) root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'... guix pull: error: Git error: cannot locate remote-tracking branch 'origin/keyring' root@guixydevel ~/src/guix/wip-hurd# git branch keyring origin/keyring branch 'keyring' set up to track 'origin/keyring'. root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'... Authenticating channel 'guix', commits 9edb3f6 to beb2704 (26 new commits)... guix pull: warning: pulled channel 'guix' from a mirror of https://git.savannah.gnu.org/git/guix.git, which might be stale Building from this channel: guix /root/src/guix/wip-hurd beb2704 substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% building /gnu/store/67p25fiinjvm1h0ga8bays6d0kyy0dp4-config.scm.drv... building /gnu/store/iy189p9rd62ymac23l83h6vz86sx775y-git.scm.drv... building /gnu/store/w5ld3vgpsd7im25zqjrk7rxpw3j6km3j-hash.scm.drv... building /gnu/store/151iwm03mirbpbgsrrmxnpr97bj5w1ps-module-import.drv... building /gnu/store/sdnlhfdqrg0kkh33adg4dcb15qnx4205-module-import.drv... building /gnu/store/s835yp2frsy9mf4ha7q0b3mrg4mc78sc-module-import-compiled.drv... building /gnu/store/sc3ik9br6k1fzkczg51qr54d9ilkl268-module-import-compiled.drv... building /gnu/store/yn9hi0kmyygk29s79fa44nqh0wqip82n-compute-guix-derivation.drv... substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% bash 389KiB/s 00:02 | 907KiB transferred dbus-1.14.0 1.2MiB/s 00:00 | 355KiB transferred guile-json-3.5.0 5.9MiB/s 00:00 | 61KiB transferre| guix-1.4.0-12.329069a-checkout 2.2MiB/s 00:09 | 20.3MiB transferred libdaemon-0.14 +inf.0TiB/s 00:00 | 39KiB transferred libevent-2.1.12 167KiB/s 00:03 | 463KiB transferred mkdir 155KiB/s 00:03 | 432KiB transferred avahi-0.8 229KiB/s 00:02 | 466KiB transferred tar 3.2MiB/s 00:00 | 840KiB transferred guile-avahi-0.4.1 26KiB/s 00:03 | 73KiB transferred xz 670KiB/s 00:01 | 529KiB transferred building /gnu/store/nr3r518p14rj0672q37c320fkd00r01y-guix-daemon-1.4.0-12.329069a.drv... \ 'build' phaseild-log 286 52 CXX nix/nix-daemon/guix_daemon-guix-daemon.o | 'build' phase47 CXX nix/libstore/libstore_a-pathlocks.o / 'make-dynamic-linker-cache' phas/ killing process 283: Invalid argument killing process 284: Invalid argument substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% The following derivations will be built: /gnu/store/k6z2f845lqzkz2h4q60w089rzl7p7ay7-profile.drv /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv /gnu/store/50fk0z57vprxh13pp000alz6200lm9r6-profile.drv /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv 5 items will be downloaded module-import-compiled 800KiB/s 00:00 | 64KiB transferred guile-wrapper 160KiB/s 00:00 | 3KiB transferred module-import-compiled 541KiB/s 00:00 | 81KiB transferred module-import-compiled 860KiB/s 00:00 | 138KiB transferred module-import-compiled 328KiB/s 00:00 | 10KiB transferred building /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv... building /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv... building /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv... building /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv... building /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv... building /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv... building /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv... building /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv... building /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv... building /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv... building /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv... building /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv... building /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv... building /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv... building /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv... building /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv... building /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv... building /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv... building /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv... building /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv... building /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv... building /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv... building /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv... building /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv... building /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv... building /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv... building /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv... building /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv... building /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv... building /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv... building /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv... building /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv... building /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv... building /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv... building /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv... building CA certificate bundle... listing Emacs sub-directories... building fonts directory... building directory of Info manuals... building profile with 1 package... building /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv... building package cache... building profile with 1 package... hint: Consider setting the necessary environment variables by running: GUIX_PROFILE="/root/.config/guix/current" . "$GUIX_PROFILE/etc/profile" Alternately, see `guix package --search-paths -p "/root/.config/guix/current"'. hint: After setting `PATH', run `hash guix' to make sure your shell refers to `/root/.config/guix/current/bin/guix'. root@guixydevel ~/src/guix/wip-hurd# GUIX_PROFILE="/root/.config/guix/current" . "$GUIX_PROFILE/etc/profile" ~/src/guix/wip-hurd# type -p guix /root/.config/guix/current/bin/guix root@guixydevel ~/src/guix/wip-hurd# guix --version guix (GNU Guix) beb27046836d60974390f6a940167e8b5def1066 Copyright (C) 2023 the Guix authors License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. root@guixydevel ~/src/guix/wip-hurd# guix describe Generation 1 Aug 24 2023 01:34:21 (current) guix beb2704 repository URL: /root/src/guix/wip-hurd branch: wip-hurd commit: beb27046836d60974390f6a940167e8b5def1066 root@guixydevel ~/src/guix/wip-hurd# guix shell -D guix hint: Consider passing the `--check' option once to make sure your shell does not clobber environment variables. root@guixydevel ~/src/guix/wip-hurd [env]# guix build hello substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% The following file will be downloaded: /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 substituting /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1... downloading from http://dezyne.org:8181/nar/zstd/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 ... hello-2.12.1 297KiB/s 00:00 | 74KiB transferred /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 root@guixydevel ~/src/guix/wip-hurd [env]# -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. 2023-08-24 5:33 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen @ 2023-08-24 6:44 ` Dr. Arne Babenhauserheide 0 siblings, 0 replies; 17+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-24 6:44 UTC (permalink / raw) To: Janneke Nieuwenhuizen; +Cc: ludo, 65456, dev [-- Attachment #1: Type: text/plain, Size: 371 bytes --] Janneke Nieuwenhuizen <janneke@gnu.org> writes: > So, some good news at last; I can confirm that using v4 of this patch we > now have "guix pull", pulling from a local git directory, now fully > working on the Hurd! That’s awesome! Thank you for your work! Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps. 2023-08-22 17:17 bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit Janneke Nieuwenhuizen @ 2023-08-22 17:49 ` Janneke Nieuwenhuizen 2023-08-22 17:49 ` bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps Janneke Nieuwenhuizen 2 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:49 UTC (permalink / raw) To: 65456 This breaks-up packages into five chunks of ~150,000 lines, allowing guix build --target=i586-pc-gnu from an x86 host. This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294. * Makefile.am (first_half): Rename to... (first_quart): ...this, and also split into... (second_quart): ...this. (third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables. (make-packages3-go, make-packages4-go): New targets. (make-packages-go): Add them. --- Makefile.am | 51 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index 738532f839..922913355c 100644 --- a/Makefile.am +++ b/Makefile.am @@ -776,15 +776,30 @@ endef # in <https://issues.guix.gnu.org/48963>. Each 'eval' call below creates a # 'make-*-go' phony target that builds the corresponding subset. -first_half := \ - gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \ - gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \ - gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l% +first_fifth := \ + gnu/packages/a% gnu/packages/b% gnu/packages/c% + +second_fifth := \ + gnu/packages/d% gnu/packages/e% gnu/packages/f% + +third_fifth := \ + gnu/packages/g% gnu/packages/h% gnu/packages/i% + +fourth_fifth := \ + gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \ + gnu/packages/n% gnu/packages/o% MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES))) -MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES)) -MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES))) -MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) +MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES)) +MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES)) +MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES)) +MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES)) +MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \ + $(third_fifth) $(fourth_fifth), \ + $(filter gnu/packages/%,$(MODULES))) +MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \ + $(MODULES_PACKAGES5) MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI := $(filter guix/scripts/%,$(MODULES)) MODULES_PO := guix/build/po.scm @@ -804,7 +819,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES1)))) .PHONY: make-packages2-go -make-packages-go: make-packages1-go make-packages2-go +$(eval $(call guile-compilation-rule,make-packages3-go, \ + $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)))) +.PHONY: make-packages3-go + +$(eval $(call guile-compilation-rule,make-packages4-go, \ + $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3)))) +.PHONY: make-packages4-go + +$(eval $(call guile-compilation-rule,make-packages5-go, \ + $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4)))) +.PHONY: make-packages5-go + +make-packages-go: make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go make-packages5-go .PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. 2023-08-22 17:49 ` bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen @ 2023-08-22 17:49 ` Janneke Nieuwenhuizen 2023-08-22 17:55 ` Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:49 UTC (permalink / raw) To: 65456 Cc: Christopher Baines, Josselin Poiret, Ludovic Courtès, Mathieu Othacehe, Ricardo Wurmus, Simon Tournier, Tobias Geerinckx-Rice Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of "gnu/packages" into 26 chunks. --- guix/self.scm | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..151fa9f452 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files (force-output)) (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + ;; Split gnu/packages in 26 chunks to avoid OOM errors + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^gnu/packages/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + (parameterize ((current-warning-port (%make-void-port "w"))) + ;; Hide compilation warnings. + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) -- 2.41.0 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. 2023-08-22 17:49 ` bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps Janneke Nieuwenhuizen @ 2023-08-22 17:55 ` Janneke Nieuwenhuizen 2023-08-22 19:28 ` Janneke Nieuwenhuizen 0 siblings, 1 reply; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 17:55 UTC (permalink / raw) To: 65456 Janneke Nieuwenhuizen writes: > diff --git a/guix/self.scm b/guix/self.scm > index 81a36e007f..151fa9f452 100644 [..] > + (parameterize ((current-warning-port (%make-void-port "w"))) meant to move this comment > + ;; Hide compilation warnings. above the parameterize; fixed locally Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. 2023-08-22 17:55 ` Janneke Nieuwenhuizen @ 2023-08-22 19:28 ` Janneke Nieuwenhuizen 0 siblings, 0 replies; 17+ messages in thread From: Janneke Nieuwenhuizen @ 2023-08-22 19:28 UTC (permalink / raw) To: 65456-done Janneke Nieuwenhuizen writes: After a LGTM from Ludo and discussing with Maxim on IRC, pushed to master as 5898b2e8a3dbf7797e83b39a2783c5b543015725 Maxim suggested (<https://logs.guix.gnu.org/guix/2023-08-22.log#211631>) we could maybe use Guile in make to split building .go files in Makefile.am algorithmically like we do in guix/self.scm. Greetings, Janneke -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2023-09-18 4:54 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-22 17:17 bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen 2023-08-22 17:33 ` Janneke Nieuwenhuizen 2023-08-22 17:19 ` bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit Janneke Nieuwenhuizen 2023-08-22 21:51 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès 2023-08-23 6:16 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 2023-08-23 9:41 ` bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time Janneke Nieuwenhuizen 2023-08-24 14:42 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès 2023-09-01 12:48 ` Janneke Nieuwenhuizen 2023-09-16 15:16 ` Janneke Nieuwenhuizen 2023-09-18 4:52 ` Janneke Nieuwenhuizen 2023-08-24 5:33 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen 2023-08-24 6:44 ` Dr. Arne Babenhauserheide 2023-08-22 17:49 ` bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen 2023-08-22 17:49 ` bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps Janneke Nieuwenhuizen 2023-08-22 17:55 ` Janneke Nieuwenhuizen 2023-08-22 19:28 ` Janneke Nieuwenhuizen
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).