From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cLozLlHoNWEygwEAgWs5BA (envelope-from ) for ; Mon, 06 Sep 2021 12:07:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id IEcUKlHoNWEELgAAbx9fmQ (envelope-from ) for ; Mon, 06 Sep 2021 10:07:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5EF789B23 for ; Mon, 6 Sep 2021 12:07:13 +0200 (CEST) Received: from localhost ([::1]:40246 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNBWy-0007jh-BQ for larch@yhetil.org; Mon, 06 Sep 2021 06:07:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44418) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNBWo-0007i0-IR for guix-patches@gnu.org; Mon, 06 Sep 2021 06:07:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40430) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNBWo-0008HU-8v for guix-patches@gnu.org; Mon, 06 Sep 2021 06:07:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mNBWo-0005pu-3H for guix-patches@gnu.org; Mon, 06 Sep 2021 06:07:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50384] [PATCH v2] Optimise search-patch (reducing I/O) Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 06 Sep 2021 10:07:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50384 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun , Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 50384@debbugs.gnu.org Received: via spool by 50384-submit@debbugs.gnu.org id=B50384.163092277322360 (code B ref 50384); Mon, 06 Sep 2021 10:07:02 +0000 Received: (at 50384) by debbugs.gnu.org; 6 Sep 2021 10:06:13 +0000 Received: from localhost ([127.0.0.1]:51974 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNBW0-0005oZ-Tv for submit@debbugs.gnu.org; Mon, 06 Sep 2021 06:06:13 -0400 Received: from laurent.telenet-ops.be ([195.130.137.89]:49160) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNBVy-0005oM-AC for 50384@debbugs.gnu.org; Mon, 06 Sep 2021 06:06:11 -0400 Received: from butterfly.local ([213.132.129.254]) by laurent.telenet-ops.be with bizsmtp id qa672500C5VU3ct01a68iy; Mon, 06 Sep 2021 12:06:08 +0200 Message-ID: <3491bccdaa15281fa254c2b788121cb42ea81adb.camel@telenet.be> From: Maxime Devos Date: Mon, 06 Sep 2021 12:06:00 +0200 In-Reply-To: <864kayjdwz.fsf@gmail.com> References: <8900fa8c8eef7f72fc97adc2408be26c88de7803.camel@telenet.be> <87tuj0xaja.fsf@gnu.org> <0ec7f0270fcccec730808f9210f074cd5339961f.camel@telenet.be> <864kayjdwz.fsf@gmail.com> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-UKv/eq1qQMChClGAB6nC" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1630922768; bh=KdPIZOIIfP0vwU8cNjUKWuc75Um5mMgg23CO1Lt6ZAs=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=NMbf86vbfd3Wlu/ZD0yv8xYiMO5+XIOQsIEsS8iy0AeDPU9Pgbqa7wW8pHhyutkHU 852I5+G+EJopKik5OQDfkeE+gbMbVWFWARKUlD9HdB7CkJhUQTkNB2DxGQr1Pe/vsd uj5ZTl1fqqep2ylt9zHZsbqVCJtUtZ5iC/+1nnNbKgMdniDdscK2Y6h3SxtGey0qlV kVOfLIC8t2dYz05epoiyjlt1kUOtL1CJ5YzsgPtVP0YviRC6N0ArBQmgbmpf5fvLqj d2Q5sfk0n3Z93Au2gwV8G4len76xKhbWZ0G52yLkU1xXT6yyAvSqKHQrq4hyGPxWuf sAqK9TBqvrHEA== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630922833; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=KdPIZOIIfP0vwU8cNjUKWuc75Um5mMgg23CO1Lt6ZAs=; b=RMFVWoyQGo6AC/v3YtRoCoz0EDpilGgsI+T+sfa82HoQQ5MP3CSpHUpSBF/ahaHEbDTY3f f7/NCnSW8KuiOy8yfnJ2IKYXOqXc90bH9eHmza67umu3esidJImHFxpamogLfbHkMdTpu+ btnUHFmSL467CAT2PALbgL5uyhzT/tDrjnUZQzca5g/BC938fFDshQtHRDrX3pGRDcat7W LijiwNm2e1/rlEW0LY+hb744TyCWBKGa5xWd91+LQTobqaoMzxZuOoyBu7DCXcFXBooljD F1bblLDWyVsuROqWVv9T1CpWzmt+qnEmIkOF2JYcb7j4Aku6iMl5NVJL38CZPw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630922833; a=rsa-sha256; cv=none; b=ipGUX8NABQwyh+T2FP3JvL+5ME7HvSrtV4C2RUcBHffPUJsKt5vDb9VcVkpypwnvB+s8dE jlN8sdzgwQWDIhygK3JcmyA0usF7vOHLX/lpJhybLtUIphm+yN5KDii4CCu4JIw77OyJb1 npwq1hovK3ziKqxnKr0OhF3pzkG5QhZdezHmJJhPnVghTJW/VxtRSdhBxYPEgVOV4huRdw SFsZvTj7CeQsfPu4fxWkY+nUclE7w3e7gmNbLCgfpy2S9Pb1cy64aZtY787dYDXisc3bim 56oAb12rVtbjabO2W/cXTgwYl7XO3JKZyoGW3GFloec7K+hAdOiGwjovP6X/tw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=NMbf86vb; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -3.41 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=NMbf86vb; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 5EF789B23 X-Spam-Score: -3.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: qEwjxCRjS34h --=-UKv/eq1qQMChClGAB6nC Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable zimoun schreef op ma 06-09-2021 om 10:39 [+0200]: > Hi Maxime, >=20 > Thanks for looking at optimising stuff. :-) >=20 > On Sun, 05 Sep 2021 at 21:48, Maxime Devos wrote= : >=20 > > Let's compare the numbers again! This time, I've run > >=20 > > echo powersave |sudo tee /sys/devices/system/cpu/cpufreq/policy{0,1,2,3= }/scaling_governor > >=20 > > to make sure the CPU frequency doesn't change. On a hot (disk) cache: > >=20 > > # After the patch series > > time GUIX_PROFILING=3D"rpc gc" ./the-optimised-guix/bin/guix build -d -= -no-grafts pigx >=20 > [...] >=20 > > So on a hot disk cache, there doesn't appear to be any improvement > > (except for =E2=80=98time spent in GC=E2=80=99 -- presumably that's due= to the optimisations > > to guix/base32.scm). >=20 > Which kind of disk do you have? SSD, spinning HDD, other? A spinning disk, presumably a HDD. FWIW, it's a =E2=80=98TOSHIBA MQ01ABD10= 0 (AX1P2C)=E2=80=99 that has been =E2=80=98on=E2=80=99 for 10 months and 10 days, according to = =E2=80=98SMART-data and selftests=E2=80=99 I just noticed it has =E2=80=982304 bad sectors=E2=80=99. Maybe I should m= ake backups and run file system checks? >=20 > > What about a cold cache? > >=20 > > # After the patch series > >=20 > > sync && echo 3 | sudo tee /proc/sys/vm/drop_caches > > ./the-optimised-guix/bin/guix --help > > time GUIX_PROFILING=3D"rpc gc" ./the-optimised-guix/bin/guix build -d -= -no-grafts pigx > > /gnu/store/fq6x8d2vcm6sbjkimg7g8kcgb4c5xv1b-pigx-0.0.3.drv > > Remote procedure call summary: 5949 RPCs > > built-in-builders ... 1 > > add-to-store ... 3 > > add-to-store/tree ... 26 > > add-temp-root ... 195 > > valid-path? ... 195 > > add-text-to-store ... 5529 > > Garbage collection statistics: > > heap size: 93.85 MiB > > allocated: 312.03 MiB > > GC times: 17 > > time spent in GC: 3.37 seconds (23% of user time) > >=20 > > real 1m39,178s > > user 0m14,557s > > sys 0m0,990s >=20 > How the average (against 3 examples) looks like? I'll try to optimise more things and report the average for the v3.=20 > > It seems that if the disk cache is cold, the time-to-derivation decreas= es > > a little by this patch series. Much less than I had hoped for though; = I'll > > have to look into other areas for interesting performance gains ... >=20 > Please update the number using your diff showed in [1]. :-) >=20 >=20 > 1: Are you referring to: > Oops, I forgot to include the following change to build-aux/compile-all.s= cm: > (or (not (file-exists? go)) > (file-mtime - (any (cut file-mtime) extra-dependencies)))) > + (any (lambda (dependency) > + (or (not (file-exists? dependency)) > + (file-mtime