From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:40353) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hnATr-0003C6-By for guix-patches@gnu.org; Mon, 15 Jul 2019 19:34:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hnATq-00069d-73 for guix-patches@gnu.org; Mon, 15 Jul 2019 19:34:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40573) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hnATq-00069U-3Z for guix-patches@gnu.org; Mon, 15 Jul 2019 19:34:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hnATp-0008RG-UV for guix-patches@gnu.org; Mon, 15 Jul 2019 19:34:01 -0400 Subject: [bug#36630] [PATCH] guix: parallelize building the manual-database Resent-Message-ID: References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> From: Arne Babenhauserheide In-reply-to: <878sszl1jo.fsf@gnu.org> Date: Tue, 16 Jul 2019 01:32:46 +0200 Message-ID: <877e8ig9gh.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 36630@debbugs.gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludo=E2=80=99, Ludovic Court=C3=A8s writes: >> * guix/profiles.scm (manual-database): par-map over the entries. This >> distributes the load roughly equally over all cores and avoids blocking = on >> I/O. The order of the entries stays the same since write-mandb-database= sorts >> them. > > I would think the whole process is largely I/O-bound. Did you try > measuring differences? I did not measure the difference in build-time, but I did check the system load. Without this patch, one of my cores is under full load. With this patch all 12 hyperthreads have a mean load of 50%. > I picked the manual-database derivation returned for: > guix environment --ad-hoc jupyter python-ipython python-ipykernel -n > (It has 3,046 entries.) How exactly did you run the derivation? I=E2=80=99d like to check it if you= can give me the exact commandline to run (a command I can run repeatedly). > On a SSD and with a hot cache, on my 4-core laptop, I get 74s with > =E2=80=98master=E2=80=99, and 53s with this patch. I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and an N= VMe M.2 disk, so it is likely that it would not be disk-bound for me at 4 threads. > However, it will definitely not scale linearly, so we should probably > cap at 2 or 4 threads. WDYT? Looking at the underlying action, this seems to be a task that scales pretty well. It just unpacks files into the disk-cache. It should also not consume much memory, so I don=E2=80=99t see a reason to artificially limit the number of threads. > Another issue with the patch is that the [n/total] counter does not grow > monotically now: it might temporally go backwards. Consequently, at > -v1, users will see a progress bar that hesitates and occasionally goes > backward, which isn=E2=80=99t great. It typically jumps forward in the beginning and then stalls until the first manual page is finished. Since par-map uses a global queue of futures to process, and since the output is the first part of (compute-entry =E2=80=A6), I don=E2=80=99t expe= ct the progress to move backwards in ways a user sees: It could only move backwards during the initial step where all threads start at the same time, and there the initial output should be overwritten fast enough to not be noticeable. > This would need to fix it with a mutex-protected global counter. A global counter would be pretty bad for scaling. As it is, this code needs no communication between processes besides returning the final result, so it behaves exactly like a normal map, aside from being faster. So I=E2=80=99d prefer to accept the forward-jumping. > All in all, I=E2=80=99m not sure this is worth the complexity. > > WDYT? Given that building manual pages is the most timeconsuming part when installing a small tool into my profile, I think it is worth the complexity. Especially because most of the complexity is being taken care of by (ice-9 threads par-map). Best wishes, Arne =2D- Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0tDSAACgkQE++NRSQD w+uuzQ//SIjIPq5kpaNC/3EgntRa5nIt+yKfT23K2juzCaYCAVUjnwHw4IXCCcSi c2vX8heNXMFcLeML65E6ldao72WejeJ5tyXdbwjvexIQgvddUvG5vXpJfVVnrfPb JPsjgaNj3DWdZRdq1lTMghlIpGrnRym+XIlzMB4BFHaiI1V5mBJ6IUk3ICddSqQo QxOvEcI3oHuDh6C55UkQrrClcO0KDYNxG631EXB6/jCLNd7UzSq3DKtQNmHGRiQq JyN2g29XipNyFp5HZ284FlegBoiO/OWseZoJ9s61KaWb9TKHeDeb7j4Yroj+MHxc lpYoD58hVyJHb0Cgy0PtWIaQIa6WwWVG1OGStDKTTkLD5a8aZqwk0I8luuzvftV2 NdZYf047HYpbABQuVR+VLpS+ofkRy5GJCpSgJwpYzko8aI71LWDexipIs34LJTEE RJ2QZ8WM7vUGBFRqjztOi0kxOYArRAkBR+aBY+dTOG+DAU/5Bitm8goVxuq9THob MZQeRRCC4b/KC4lMB3QkREXf2U6FfLOpC0Y0+OHQMV4WBs/5Wf+3Qhqa3oy/NIZW oodu4nAV4sytcdT9lVJuJqXmW6Vqy1UdcDry/FQbUjocJa4HN/BoRg2q5C5TL9Om +B4CEXoxjfa7KLEL1dxEEqt+xcXajX6ys1t14FELqxr/OKJGdYmIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdLQ0gAAoJENzPDbMLwQVICe0EAJarRMWw leLwx4Q6quii/FUJK85wPqbB3cAPupTyS2v9pso/QIGkd2ku/NkksWEowa1PbhRx V1PhD1AVFZ8aW1eQjz5aeh49g1udv0+IrOzetdm9Bzsv+zuweswzhZUJXgM1MMni ILP1PcMcQEMPcYkC1+BiYOxZMgEip+3U72fN =883H -----END PGP SIGNATURE----- --=-=-=--