From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qL/RIFZi8GDCXgAAgWs5BA (envelope-from ) for ; Thu, 15 Jul 2021 18:29:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id ANK4GlZi8GBBYAAAB5/wlQ (envelope-from ) for ; Thu, 15 Jul 2021 16:29:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A08A912A05 for ; Thu, 15 Jul 2021 18:29:09 +0200 (CEST) Received: from localhost ([::1]:58054 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m44EW-0000Yi-Qd for larch@yhetil.org; Thu, 15 Jul 2021 12:29:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51386) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m44EQ-0000YK-7W for guix-patches@gnu.org; Thu, 15 Jul 2021 12:29:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:38011) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m44EP-0002SR-W8 for guix-patches@gnu.org; Thu, 15 Jul 2021 12:29:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m44EP-0004bL-SH for guix-patches@gnu.org; Thu, 15 Jul 2021 12:29:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49421] [PATCH] profiles: Optimise 'fonts-dir-file'. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 15 Jul 2021 16:29:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49421 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 49421@debbugs.gnu.org Received: via spool by 49421-submit@debbugs.gnu.org id=B49421.162636650917646 (code B ref 49421); Thu, 15 Jul 2021 16:29:01 +0000 Received: (at 49421) by debbugs.gnu.org; 15 Jul 2021 16:28:29 +0000 Received: from localhost ([127.0.0.1]:49556 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m44Dt-0004aY-1G for submit@debbugs.gnu.org; Thu, 15 Jul 2021 12:28:29 -0400 Received: from andre.telenet-ops.be ([195.130.132.53]:57620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m44Dn-0004aK-Ig for 49421@debbugs.gnu.org; Thu, 15 Jul 2021 12:28:27 -0400 Received: from [172.20.10.4] ([213.119.236.63]) by andre.telenet-ops.be with bizsmtp id VUUM250031Nl63101UUM5H; Thu, 15 Jul 2021 18:28:22 +0200 Message-ID: <5ec75ed429133cbc73dc7916959302033fe70a13.camel@telenet.be> From: Maxime Devos Date: Thu, 15 Jul 2021 18:28:16 +0200 In-Reply-To: <87a6myoyl5.fsf@gnu.org> References: <87a6myoyl5.fsf@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-f8AOwVr+vtZtUE5SEUdt" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1626366502; bh=TtwnhMXV00Kw+fJqCpIEb6tcxMTFImCVAS6szYMXJp8=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=aUZyer5L/fQRWeNuE8NwVomG1v3Qknkyg3xvkRyjmTYZUCPv+WULW8YrN9ruwD6NQ JvNAT00Yf2jtM/AwdaRms9EDQa9jtnspIZGrJnvvnqhMo87xVJ8ZcNyvfg7TOW5RrD JZjAYn258WHWbCacBsjEoT7L6GeB4uLwiqMQEHHJVB9Gc91JYjr3ZRJRZWXI4VkjCd HWedZ6ob0Wp4Y5087xMRlJthmOoxWcsDt0RGCYUK3GBBbDxhBg0GHch/akjYb9Jriq IIRdBrPNS6Qf6ebpbrdKcoJsFEPIWpK0E8t8hvAh5FiCcGD0OuOIqT5uzHXpCLAum3 v3ihFuypZNyJg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1626366549; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=TtwnhMXV00Kw+fJqCpIEb6tcxMTFImCVAS6szYMXJp8=; b=RtsnAA57YFLr5Nkw/56CS7bxaWoo/Go6lqLu4g+D/kvlJk7rrb7N6yQ5DWAQfuo4FS5Hmp rbFIWK958h2yY6IHW8a+TpjqJO/DIQZHpjj2gfx1XeifBIaTLN484Wnc2qpAeWbKgU5gf8 kH7AjXmHRkIrUyWPlOLuDJn5R7aurirx2Sqk8/sWhhrFf0k7TDrymN42elZ8eC++0jVUNJ aVoQFnDAThRQ1SODmWvykfqvYAFfM+2HLYaf6E86SBAPrkvT2XRnBGcqB10DQNposgmCnV 1q2vf0qe/9JIZvl48Gk4fUlc7CObm3gscLV7T1uJhFFlMwlNX09rFmpUhMjBdw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1626366549; a=rsa-sha256; cv=none; b=Lg/L8D3fAKd5evGVtF2UWWGIC498yzjq7EzW8WeoErKx/x4tUVV2aHMiLjx8PspnMqkzP4 Iqg+1zqyVuwRbA2DiX18m30GHwKVU4apT7FqWcnQIXDuccqu13sj9asTBVGpWwCOaFdgLj 2fQLcWP8UUwd/DDK6NxkRy0krO7Yiqs0NffvtxvJlGb4yLZy0N9moSbbSeOg4zig2yUvy9 s7c7pDcU3IRNPHfR6rqmPfyVOT4j7QJ3msQB3IH/rnku0/hErKn0zgPQ5WV9lbqK/07/J+ auD26BLvvHTZtS9KAANZ9xivMII36LHpgCy8AFXKe+nrDHl6EwwhIRjAUUuHcQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=aUZyer5L; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -3.40 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=aUZyer5L; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: A08A912A05 X-Spam-Score: -3.40 X-Migadu-Scanner: scn0.migadu.com X-TUID: Yv1TQzEYF0BT --=-f8AOwVr+vtZtUE5SEUdt Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op wo 07-07-2021 om 18:52 [+0200]: > Hi, >=20 > Maxime Devos skribis: >=20 > > These two patches should speed up profile generation > > by optimising the 'fonts-dir-file' hook. The first patch > > is the most important; from the commit message: > >=20 > > =E2=80=98Only let the build G-exp refer to inputs that might actually > > have fonts. That way, if the list of fonts in the manifest > > didn't change and the new manifest is built, the font files > > will not be rebuilt.=E2=80=99 > >=20 > > The second patch doesn't bring much. > >=20 > > To test, you can test whether things like ... > >=20 > > $ ./pre-inst-env guix environment --ad-hoc --pure lagrange -- lagrange > >=20 > > still work (lagrange is a graphical application using fonts). > > Not sure what a good =E2=80=98benchmark=E2=80=99 would be. > >=20 > > Unfortunately, this does not help with the "guix package -i" case, > > as in that case, the code doesn't have access to all the package object= s, > > and will have to satisfy itself with the store paths, in which case the > > profile code pessimistically assumes the store item has fonts ... > >=20 > > I suppose this restriction could be lifted if/when the gs-fonts package > > is renamed to font-ghostscript or something like that ... >=20 > Do you know what=E2=80=99s taking time in the =E2=80=98fonts-dir-file=E2= =80=99 hook? > (On my x86_64 laptop with SSD, it runs in 4.4s on a cold cache for a > profile containing 14 font packages, and in 1.2s on a warm cache.) I noticed =E2=80=98building fonts directories=E2=80=99 was taking long (9 m= inutes or so, when using "guix package -r two packages that don't have fonts"), so I took the opportunity to "strace" the process, though I was a little la= te. Some statistics: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 27,14 1,676551 117 14297 rename 19,60 1,211012 13 90829 8 lstat 18,34 1,133182 63 17787 15 statx 11,80 0,729185 51 14297 link 6,65 0,411065 68510 6 wait4 5,08 0,313632 10 28651 chmod 3,66 0,226258 14 15114 readlink 2,18 0,134913 8 15193 utimensat 1,78 0,110257 32 3431 rmdir 0,91 0,056504 7 7370 getdents64 0,82 0,050817 3 14295 getpid 0,47 0,029074 7 3736 openat 0,46 0,028547 8 3479 3428 unlink 0,36 0,021940 5 3747 close 0,26 0,016355 4 3704 fstat 0,16 0,009752 10 898 lchown 0,12 0,007242 4 1800 geteuid 0,07 0,004203 62 67 read 0,06 0,003512 3 900 getegid 0,03 0,001959 326 6 clone 0,02 0,001133 18 62 write 0,01 0,000488 3 127 1 fcntl 0,01 0,000451 3 136 lseek 0,00 0,000197 14 14 mkdir 0,00 0,000065 13 5 select 0,00 0,000045 5 8 chown 0,00 0,000044 7 6 brk 0,00 0,000027 13 2 munmap 0,00 0,000023 11 2 pipe 0,00 0,000013 6 2 stat 0,00 0,000007 7 1 futex 0,00 0,000004 2 2 getuid 0,00 0,000000 0 1 rt_sigreturn ------ ----------- ----------- --------- --------- ---------------- 100,00 6,178457 25 239975 3452 total And a part of the log: lstat("/gnu/store/.links/[...]2i", {st_mode=3DS_IFLNK|0777, st_size=3D124, = ...}) =3D 0 lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", {st_mode=3DS_IFDIR|0555, st_size=3D4112, ...> chmod("/gnu/store/[..]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS= /Regular/283", 040755) =3D 0 getpid() =3D 26371 link("/gnu/store/.links/[..]2i", "/gnu/store/.tmp-link-26371-632297102") = =3D 0 rename("/gnu/store/.tmp-link-26371-632297102", "/gnu/store/[...]nm-fonts-di= r/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", {st_mode=3DS_IFDIR|0755, st_size=3D4112, ...> chmod("/gnu/store/[...]-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/= Regular/283", 040555) =3D 0 utimensat(AT_FDCWD, "/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-= CSS/TeX/png/AMS/Regular/283", [{tv_sec=3D1626363034, tv_ns> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/2223.png", {st_mode=3DS_IFLNK|0777, st_size=3D> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/2223.png", {st_mode=3DS_IFLNK|0777, st_size=3D> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/2223.png", {st_mode=3DS_IFLNK|0777, st_size=3D> readlink("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png= /AMS/Regular/283/2223.png", "/gnu/store/bf582s56ldb3y34p> statx(AT_FDCWD, "/gnu/store/.links/[...]v3", AT_STATX_SYNC_AS_STAT|AT_SYMLI= NK_NOFOLLOW, 0, {stx_mask=3DSTATX_ALL|> lstat("/gnu/store/.links/[...]v3", {st_mode=3DS_IFLNK|0777, st_size=3D124, = ...}) =3D 0 lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", {st_mode=3DS_IFDIR|0555, st_size=3D4112, ...> chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", 040755) =3D 0 getpid() =3D 26371 link("/gnu/store/.links/[...]v3", "/gnu/store/.tmp-link-26371-114005693") = =3D 0 rename("/gnu/store/.tmp-link-26371-114005693", "/gnu/store/[...]nm-fonts-di= r/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", {st_mode=3DS_IFDIR|0755, st_size=3D4112, ...> chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", 040555) =3D 0 utimensat(AT_FDCWD, "/gnu/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/Te= X/png/AMS/Regular/283", [{tv_sec=3D1626363034, tv_ns> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/22CC.png", {st_mode=3DS_IFLNK|0777, st_size=3D> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/22CC.png", {st_mode=3DS_IFLNK|0777, st_size=3D> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283/22CC.png", {st_mode=3DS_IFLNK|0777, st_size=3D> readlink("/gnu/store/h6681qq9s5kqjvdzkhqhddqbjj3q48nm-fonts-dir/share/fonts= /mathjax/HTML-CSS/TeX/png/AMS/Regular/283/22CC.png", "/gnu/store/bf582s56ld= b3y34p> statx(AT_FDCWD, "/gnu/store/.links/[...]my", AT_STATX_SYNC_AS_STAT|AT_SYMLI= NK_NOFOLLOW, 0, {stx_mask=3DSTATX_ALL|> lstat("/gnu/store/.links/[...]my", {st_mode=3DS_IFLNK|0777, st_size=3D124, = ...}) =3D 0 lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", {st_mode=3DS_IFDIR|0555, st_size=3D4112, ...> chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AM= S/Regular/283", 040755) =3D 0 getpid() =3D 26371 It seems like =E2=80=98duplication=E2=80=99 is taking a lot of time! (See nix/libstore/optimise-store.cc and (guix store deduplication).) > IIUC, the hook completes almost instantaneously when creating a profile > without fonts because (null? fonts-dirs) is true. >=20 > In other cases, one option to speed things up (and possibly remove the > hook=E2=80=99s dependency on mkfontdir) would be to create =E2=80=98fonts= .dir=E2=80=99 and > =E2=80=98fonts.scale=E2=80=99 in a build phase of each font package, *pro= vided* there=E2=80=99s > a cheap way to assemble several such files, such as mere concatenation. > That way, the profile hook would only need to assemble these files as > opposed to traversing all the font files. Looking at ~/.guix-profile/share/fonts/truetype/fonts.dir, it seems to be simply a list of entries lke DejaVuSans-Bold.ttf -misc-dejavu sans-bold-r-normal--0-0-0-0-p-0-koi8-ru with additionally as first line the number of fonts. It appears to be sorted by font file name (DejaVuSans-Bold.ttf) though, though that only seems to be to not depend on file system ordering. Not quite _mere_ concatenation, but should be easy enough, I think. (Likewise for fonts.scaale) I would need to look at the code of "mkfontscale" and "mkfontdir" to be sur= e though. > Given that =E2=80=98fonts.dir=E2=80=99 and =E2=80=98fonts.scale=E2=80=99 = are mere text files with one > line per font, this approach looks doable. >=20 > WDYT? That seems a good idea, reducing the amount of 'stat'ing, 'readdir'ing, and 'read'ing (mkfontscale needs to actually read the font files) which should help on spinning disks (and other disks, but on SSDs it's prob= ably less noticable). > > From 4fe1e30e33c01be9fd17cf240732b3351c7b0fa4 Mon Sep 17 00:00:00 2001 > > From: Maxime Devos > > Date: Mon, 5 Jul 2021 18:55:31 +0200 > > Subject: [PATCH 1/2] profiles: Optimise 'fonts-dir-file'. > >=20 > > Only let the build G-exp refer to inputs that might actually > > have fonts. That way, if the list of fonts in the manifest > > didn't change and the new manifest is built, the font files > > will not be rebuilt. > >=20 > > * guix/profiles.scm > > (fonts-dir-file)[has-fonts?]: New predicate. > > (fonts-dir-file)[relevant-inputs]: New variable. > > (fonts-dir-file)[build]: Use 'relevant-inputs' instead of > > 'manifest-inputs'. > > * doc/contributing.texi (Fonts): Note the 'fonts-' naming > > convention is technically important now. > > * gnu/packages/ghostscript.scm (gs-fonts): Work-around the > > package name contravening the convention. >=20 > To me this approach should be a last resort because it relies on > conventions and manual annotations, and it=E2=80=99s easy to forget these= and > get incorrect results (we could end up having to deal with reports like > =E2=80=9CI installed font X, why doesn=E2=80=99t in show up in the font l= ist?=E2=80=9D). AFAIK, all font packages in Guix (except gs-fonts) are currently correctly-named. They aren't going to spontanuously rename theirselves. So to me, the only potential problem seems to be _new_ font packages ignoring the conventions. But new packages should always be reviewed and tested, so such reports don't seem plausible to me. (Also, the manual annotation would go away once 'gs-fonts' is renamed. I'll run "echo /gnu/store/*/share/fonts > atchie-fonts" though to see if there are other packages not following the convention.) > > + ;; In the upstream 'guix' channel, font packages should > > + ;; be named font-SOMETHING. But if another channel > > + ;; names its fonts differently but uses font-build-system, > > + ;; accepting that seems friendly. > > + (eq? 'font (build-system-name (package-build-system thing)= )) >=20 > In general =E2=80=98build-system-name=E2=80=99 shouldn=E2=80=99t be used;= the name is meant to > be a debugging aid. I could do (eq? font-build-system (packag-build-system thing)) instead or drop this clause. > Thanks, > Ludo=E2=80=99. --=-f8AOwVr+vtZtUE5SEUdt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYPBiIBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7vYQAP4p5PTgHgfrNPzE6lB/cIAg8WQP 2UTmyDLOQNOQ8qnVLwEAvkqoSME3w0PP5KnZloAY5BsT0wd2rzaWpBkRzWE4NA4= =1Vm6 -----END PGP SIGNATURE----- --=-f8AOwVr+vtZtUE5SEUdt--