From: ludo@gnu.org (Ludovic Courtès)
To: Eric Bavier <ericbavier@centurylink.net>
Cc: 27284@debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@gmail.com>
Subject: bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile).
Date: Sun, 22 Oct 2017 22:10:12 -0700 [thread overview]
Message-ID: <874lqqv497.fsf@gnu.org> (raw)
In-Reply-To: <20171022175253.2b72da76@centurylink.net> (Eric Bavier's message of "Sun, 22 Oct 2017 17:52:53 -0500")
[-- Attachment #1: Type: text/plain, Size: 501 bytes --]
Eric Bavier <ericbavier@centurylink.net> skribis:
> On Sun, 22 Oct 2017 19:51:14 -0700
> ludo@gnu.org (Ludovic Courtès) wrote:
[...]
>> > It Would Be Nice if the functionality worked for older versions of make
>> > that people might have on their systems.
>>
>> With the patch I posted, Scheme compilation would always use one thread
>> per core, which is what it currently does in ‘master’.
Oops, that was not quite true: it would use one thread.
I fixed it like this:
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2368 bytes --]
diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 4aa4ac9b9..c7ca5a6f6 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -19,6 +19,7 @@
(use-modules (ice-9 match)
(ice-9 threads)
+ (srfi srfi-1)
(guix build compile)
(guix build utils))
@@ -50,24 +51,32 @@ to 'make'."
(match flags
(#f (current-processor-count))
(flags
- (let loop ((flags (string-tokenize flags)))
- (match flags
- (()
- 1)
- (("-j" (= string->number count) _ ...)
- (if (integer? count)
- count
- (current-processor-count)))
- ((head tail ...)
- (if (string-prefix? "-j" head)
- (match (string-drop head 2)
- (""
- (current-processor-count))
- ((= string->number count)
- (if (integer? count)
- count
- (current-processor-count))))
- (loop tail))))))))
+ (let ((initial-flags (string-tokenize flags)))
+ (let loop ((flags initial-flags))
+ (match flags
+ (()
+ ;; Note: GNU make prior to version 4.2 would hide "-j" flags from
+ ;; $MAKEFLAGS. Thus, check for a "--jobserver" flag here and
+ ;; assume we're using all cores if specified.
+ (if (any (lambda (flag)
+ (string-prefix? "--jobserver" flag))
+ initial-flags)
+ (current-processor-count) ;GNU make < 4.2
+ 1)) ;sequential make
+ (("-j" (= string->number count) _ ...)
+ (if (integer? count)
+ count
+ (current-processor-count)))
+ ((head tail ...)
+ (if (string-prefix? "-j" head)
+ (match (string-drop head 2)
+ (""
+ (current-processor-count))
+ ((= string->number count)
+ (if (integer? count)
+ count
+ (current-processor-count))))
+ (loop tail)))))))))
;; Install a SIGINT handler to give unwind handlers in 'compile-file' an
;; opportunity to run upon SIGINT and to remove temporary output files.
[-- Attachment #3: Type: text/plain, Size: 1182 bytes --]
>> > Using the jobserver directly would require quite a bit of work for the
>> > current patch set, but I wonder if there is another way to determine
>> > the -jN parameter for make<4.2 that we could use. Maybe simply
>> > polling the jobserver fds at the start?
>>
>> AIUI the job server does not reveal how many jobs are allowed. It
>> merely grants you an execution token.
>>
>> Or did you have something else in mind?
>
> The idea (hardely tested) would be to read tokens from the input fd
> until it blocks, do the scheme compiles with however many tokens were
> read, then write them back out. Crude, I guess, and probably error
> prone; compile-all.scm could be invoked when make has job tokens tied
> up building the daemon source...
>
> Anyhow, the current patch works well for me with a recent make. I'm
> content leaving further improvements to a future hypothetical hacker. :)
Sounds reasonable. Let’s make sure the hypothetical hacker has enough
on their plate. :-)
I’ve pushed this and the easy parts of this patch series, with commit
d298c815e638581d466222f3a883b280f019b368 as the tip.
Thanks for the review!
Ludo’.
next prev parent reply other threads:[~2017-10-23 5:11 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-08 8:39 bug#27284: Memory leak in 'guix pull' or 'make' in guix source ng0
2017-06-08 15:02 ` ng0
2017-09-19 20:48 ` Ludovic Courtès
2017-09-20 2:40 ` Maxim Cournoyer
2017-09-20 11:42 ` Ludovic Courtès
2017-09-20 18:29 ` Maxim Cournoyer
2017-09-20 20:12 ` Ludovic Courtès
2017-09-21 14:43 ` Maxim Cournoyer
2017-09-23 18:14 ` Taylan Ulrich Bayırlı/Kammer
2017-09-24 19:44 ` Ludovic Courtès
2017-09-25 21:00 ` Maxim Cournoyer
2017-10-20 16:05 ` bug#27284: [PATCH 0/8] 'guix pull' creates several derivations Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile) Ludovic Courtès
2017-10-22 21:22 ` Maxim Cournoyer
2017-10-23 1:50 ` Ludovic Courtès
2017-10-22 21:42 ` Eric Bavier
2017-10-23 2:51 ` Ludovic Courtès
2017-10-22 22:52 ` Eric Bavier
2017-10-23 5:10 ` Ludovic Courtès [this message]
2017-10-20 16:05 ` bug#27284: [PATCH 2/8] build: Honor make's '-j' flag Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 3/8] discovery: Move 'file-name->module-name' to (guix modules) Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 4/8] gexp: Add 'file-union' Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 5/8] gexp: Add 'directory-union' Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 6/8] union: Parametrize the symlink procedure Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 7/8] gexp: 'directory-union' has a #:quiet? parameter Ludovic Courtès
2017-10-20 16:05 ` bug#27284: [PATCH 8/8] DRAFT Add (guix self) and use it when pulling Ludovic Courtès
2017-10-22 20:05 ` Maxim Cournoyer
2017-10-27 23:49 ` Ludovic Courtès
2017-11-21 22:26 ` bug#27284: [PATCH 0/8] 'guix pull' creates several derivations Ludovic Courtès
2017-11-21 22:56 ` Ludovic Courtès
2017-12-11 10:52 ` bug#27284: [PATCH 0/4] 'guix pull' reloads modules, second try Ludovic Courtès
2017-12-11 10:52 ` bug#27284: [PATCH 1/4] gnu: Fix ambiguous 'zip' reference Ludovic Courtès
2017-12-11 10:52 ` bug#27284: [PATCH 2/4] gexp: 'computed-file' has a new #:guile parameter Ludovic Courtès
2017-12-11 10:52 ` bug#27284: [PATCH 3/4] Add (guix self) and use it when pulling Ludovic Courtès
2017-12-18 14:57 ` Ludovic Courtès
2018-03-27 9:14 ` bug#27284: ‘guix pull’ builds using multiple derivations Ludovic Courtès
2018-03-27 14:33 ` Ludovic Courtès
2018-03-27 19:25 ` Nils Gillmann
2018-03-27 20:51 ` Ludovic Courtès
2018-04-08 16:37 ` Ludovic Courtès
2018-04-09 19:53 ` Ricardo Wurmus
2018-04-10 21:53 ` bug#27284: ‘guix pull’ broken on Guile 2.0 Ludovic Courtès
2018-04-10 23:18 ` bug#31117: " Ludovic Courtès
2018-04-14 17:39 ` Ricardo Wurmus
2017-12-11 10:52 ` bug#27284: [PATCH 4/4] pull: Reload modules before doing anything else Ludovic Courtès
2017-11-12 21:33 ` bug#27284: Memory leak in 'guix pull' or 'make' in guix source Ludovic Courtès
2017-11-13 8:59 ` Ricardo Wurmus
2017-11-13 9:28 ` Ludovic Courtès
2017-11-13 14:09 ` Ricardo Wurmus
2017-11-13 17:48 ` Ricardo Wurmus
2017-11-14 7:54 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=874lqqv497.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=27284@debbugs.gnu.org \
--cc=ericbavier@centurylink.net \
--cc=maxim.cournoyer@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.