From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:470:142:3::10]:57704)
 by lists.gnu.org with esmtp (Exim 4.86_2)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1hn3b9-0004I1-Nl
 for guix-patches@gnu.org; Mon, 15 Jul 2019 12:13:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1hn3b8-0002IH-NO
 for guix-patches@gnu.org; Mon, 15 Jul 2019 12:13:07 -0400
Received: from debbugs.gnu.org ([209.51.188.43]:40022)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1hn3b8-0002IA-JP
 for guix-patches@gnu.org; Mon, 15 Jul 2019 12:13:06 -0400
Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1hn3b8-0007lG-B7
 for guix-patches@gnu.org; Mon, 15 Jul 2019 12:13:06 -0400
Subject: [bug#36630] [PATCH] guix: parallelize building the manual-database
Resent-Message-ID: <handler.36630.B36630.156320716429778@debbugs.gnu.org>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>
References: <20190712214245.23857-1-arne_bab@web.de>
Date: Mon, 15 Jul 2019 18:12:27 +0200
In-Reply-To: <20190712214245.23857-1-arne_bab@web.de> (Arne Babenhauserheide's
 message of "Fri, 12 Jul 2019 23:42:45 +0200")
Message-ID: <878sszl1jo.fsf@gnu.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
List-Id: <guix-patches.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/guix-patches>,
 <mailto:guix-patches-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/guix-patches>
List-Post: <mailto:guix-patches@gnu.org>
List-Help: <mailto:guix-patches-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/guix-patches>,
 <mailto:guix-patches-request@gnu.org?subject=subscribe>
Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org
Sender: "Guix-patches" <guix-patches-bounces+kyle=kyleam.com@gnu.org>
To: Arne Babenhauserheide <arne_bab@web.de>
Cc: 36630@debbugs.gnu.org

Hi Arne,

Arne Babenhauserheide <arne_bab@web.de> skribis:

> * guix/profiles.scm (manual-database): par-map over the entries.  This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O.  The order of the entries stays the same since write-mandb-database =
sorts
> them.

I would think the whole process is largely I/O-bound.  Did you try
measuring differences?

I picked the manual-database derivation returned for:

  guix environment --ad-hoc jupyter python-ipython python-ipykernel -n

(It has 3,046 entries.)

On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
=E2=80=98master=E2=80=99, and 53s with this patch.

However, it will definitely not scale linearly, so we should probably
cap at 2 or 4 threads.  WDYT?

Another issue with the patch is that the [n/total] counter does not grow
monotically now: it might temporally go backwards.  Consequently, at
-v1, users will see a progress bar that hesitates and occasionally goes
backward, which isn=E2=80=99t great.

This would need to fix it with a mutex-protected global counter.

All in all, I=E2=80=99m not sure this is worth the complexity.

WDYT?

Thanks,
Ludo=E2=80=99.