From: "Ludovic Courtès" <ludo@gnu.org>
To: Arne Babenhauserheide <arne_bab@web.de>
Cc: 36630@debbugs.gnu.org
Subject: [bug#36630] [PATCH] guix: parallelize building the manual-database
Date: Mon, 15 Jul 2019 18:12:27 +0200 [thread overview]
Message-ID: <878sszl1jo.fsf@gnu.org> (raw)
In-Reply-To: <20190712214245.23857-1-arne_bab@web.de> (Arne Babenhauserheide's message of "Fri, 12 Jul 2019 23:42:45 +0200")
Hi Arne,
Arne Babenhauserheide <arne_bab@web.de> skribis:
> * guix/profiles.scm (manual-database): par-map over the entries. This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O. The order of the entries stays the same since write-mandb-database sorts
> them.
I would think the whole process is largely I/O-bound. Did you try
measuring differences?
I picked the manual-database derivation returned for:
guix environment --ad-hoc jupyter python-ipython python-ipykernel -n
(It has 3,046 entries.)
On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
‘master’, and 53s with this patch.
However, it will definitely not scale linearly, so we should probably
cap at 2 or 4 threads. WDYT?
Another issue with the patch is that the [n/total] counter does not grow
monotically now: it might temporally go backwards. Consequently, at
-v1, users will see a progress bar that hesitates and occasionally goes
backward, which isn’t great.
This would need to fix it with a mutex-protected global counter.
All in all, I’m not sure this is worth the complexity.
WDYT?
Thanks,
Ludo’.
next prev parent reply other threads:[~2019-07-15 16:13 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-12 21:42 [bug#36630] [PATCH] guix: parallelize building the manual-database Arne Babenhauserheide
2019-07-15 16:12 ` Ludovic Courtès [this message]
2019-07-15 23:32 ` Arne Babenhauserheide
2019-07-16 21:14 ` Ludovic Courtès
2019-07-17 22:06 ` Arne Babenhauserheide
2019-07-18 8:55 ` Ludovic Courtès
2019-07-18 8:57 ` Ludovic Courtès
2019-07-18 10:59 ` Arne Babenhauserheide
2019-07-18 13:46 ` Ludovic Courtès
2019-07-18 20:03 ` Arne Babenhauserheide
2019-10-23 20:01 ` Arne Babenhauserheide
2020-03-31 13:02 ` bug#36630: " Ludovic Courtès
2019-10-23 19:59 ` [bug#36630] [PATCH] use two threads to build man-pages and secure output msgs with mutex Arne Babenhauserheide
2019-10-27 16:27 ` Arne Babenhauserheide
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=878sszl1jo.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=36630@debbugs.gnu.org \
--cc=arne_bab@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.