From: Janneke Nieuwenhuizen <janneke@gnu.org>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: Josselin Poiret <dev@jpoiret.xyz>,
Tobias Geerinckx-Rice <me@tobias.gr>,
Simon Tournier <zimon.toutoune@gmail.com>,
Mathieu Othacehe <othacehe@gnu.org>,
65456@debbugs.gnu.org, Ricardo Wurmus <rekado@elephly.net>,
Christopher Baines <guix@cbaines.net>
Subject: bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time.
Date: Wed, 23 Aug 2023 11:41:31 +0200 [thread overview]
Message-ID: <87h6oq6qpg.fsf_-_@gnu.org> (raw)
In-Reply-To: <871qfu8erx.fsf_-_@gnu.org> (Janneke Nieuwenhuizen's message of "Wed, 23 Aug 2023 08:16:18 +0200")
[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]
Janneke Nieuwenhuizen writes:
Hi!
> Ludovic Courtès writes:
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
[..]
> it still has the "gnu/packages" directory name hardcoded here. (I even
> believe this was kind-of intentional, meaning not to split-up builds of
> other directories...). The result is, of course, that partition always
> produces an empty result for any '("guix/foo.scm") files!
>
> It is fixed by doing
>
> (cute string-append "^" directory "/" <>)
Well, almost...In my test script it worked, but directory is "." in the
actual self.scm.
So with v3 no .go files get built at all...and then a naive `guix build
hello' seems to work!; `guix shell' for example fails. Comparing the
list of files from `find /gnu/store/...-guix-20230823.08 -follow' is
also helpful.
I realised that there's no need to use the name-based splitting hack
that Makefile.am uses and changed the creation of chunks to simply use a
maximum chunk length of 25 files.
New version attached.
Greetings,
Janneke
[-- Attachment #2: v4-0001-self-Build-directories-in-chunks-of-max-25-files-.patch --]
[-- Type: text/x-patch, Size: 3288 bytes --]
From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
time.
Similar to split build of make-go in Makefile.am, this breaks-up building
directories into chunks of max 25 files. Also force garbage collection.
* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into chunks of max 25 files.
---
guix/self.scm | 27 ++++++++++++++++++++-------
1 file changed, 20 insertions(+), 7 deletions(-)
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..fc2dfd8131 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-files
'((guix build compile)
(guix build utils)))
#~(begin
- (use-modules (srfi srfi-26)
+ (use-modules (srfi srfi-1)
+ (srfi srfi-26)
(ice-9 match)
(ice-9 format)
(ice-9 threads)
@@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-files
(force-output))
(define (process-directory directory files output)
- ;; Hide compilation warnings.
- (parameterize ((current-warning-port (%make-void-port "w")))
- (compile-files directory #$output files
- #:workers (parallel-job-count)
- #:report-load report-load
- #:report-compilation report-compilation)))
+ (let* ((size 25) ;compile max 25 files a time
+ (chunks (unfold
+ (lambda (seed) (< (length seed) size)) ;p
+ (cute take <> size) ;f
+ (cute drop <> size) ;g
+ files ;seed
+ list))) ;tail
+ (for-each
+ (lambda (chunck)
+ ;; Hide compilation warnings.
+ (parameterize ((current-warning-port (%make-void-port "w")))
+ (compile-files directory output chunck
+ #:workers (parallel-job-count)
+ #:report-load report-load
+ #:report-compilation report-compilation))
+ (gc))
+ chunks)))
(setvbuf (current-output-port) 'line)
(setvbuf (current-error-port) 'line)
base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
--
2.41.0
[-- Attachment #3: Type: text/plain, Size: 164 bytes --]
--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com
next prev parent reply other threads:[~2023-08-23 9:42 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-22 17:17 bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Janneke Nieuwenhuizen
2023-08-22 17:19 ` bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen
2023-08-22 17:33 ` Janneke Nieuwenhuizen
2023-08-22 17:19 ` bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit Janneke Nieuwenhuizen
2023-08-22 21:51 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès
2023-08-23 6:16 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen
2023-08-23 9:41 ` Janneke Nieuwenhuizen [this message]
2023-08-24 14:42 ` bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts Ludovic Courtès
2023-09-01 12:48 ` Janneke Nieuwenhuizen
2023-09-16 15:16 ` Janneke Nieuwenhuizen
2023-09-18 4:52 ` Janneke Nieuwenhuizen
2023-08-24 5:33 ` bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps Janneke Nieuwenhuizen
2023-08-24 6:44 ` Dr. Arne Babenhauserheide
2023-08-22 17:49 ` bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps Janneke Nieuwenhuizen
2023-08-22 17:49 ` bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps Janneke Nieuwenhuizen
2023-08-22 17:55 ` Janneke Nieuwenhuizen
2023-08-22 19:28 ` Janneke Nieuwenhuizen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h6oq6qpg.fsf_-_@gnu.org \
--to=janneke@gnu.org \
--cc=65456@debbugs.gnu.org \
--cc=dev@jpoiret.xyz \
--cc=guix@cbaines.net \
--cc=ludo@gnu.org \
--cc=me@tobias.gr \
--cc=othacehe@gnu.org \
--cc=rekado@elephly.net \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.