From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id B9BvGkeM019HEAAA0tVLHw (envelope-from ) for ; Fri, 11 Dec 2020 15:12:07 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id WFSxFUeM01/sFAAAB5/wlQ (envelope-from ) for ; Fri, 11 Dec 2020 15:12:07 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E53F49404E8 for ; Fri, 11 Dec 2020 15:12:06 +0000 (UTC) Received: from localhost ([::1]:35656 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1knk5V-0006Y3-Pd for larch@yhetil.org; Fri, 11 Dec 2020 10:12:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:49840) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1knk3W-0004HS-7K for bug-guix@gnu.org; Fri, 11 Dec 2020 10:10:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59272) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1knk3V-0000cs-Sv for bug-guix@gnu.org; Fri, 11 Dec 2020 10:10:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1knk3V-0007j7-NG for bug-guix@gnu.org; Fri, 11 Dec 2020 10:10:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#44760: [PATCH 00/15] Speed up 'guix system init' & co. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 11 Dec 2020 15:10:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44760 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 44760@debbugs.gnu.org Received: via spool by 44760-submit@debbugs.gnu.org id=B44760.160769938029564 (code B ref 44760); Fri, 11 Dec 2020 15:10:01 +0000 Received: (at 44760) by debbugs.gnu.org; 11 Dec 2020 15:09:40 +0000 Received: from localhost ([127.0.0.1]:42559 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knk3A-0007gk-HW for submit@debbugs.gnu.org; Fri, 11 Dec 2020 10:09:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56534) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knk38-0007gM-MP for 44760@debbugs.gnu.org; Fri, 11 Dec 2020 10:09:39 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:47525) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1knk32-0000S7-Oz; Fri, 11 Dec 2020 10:09:32 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59324 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1knk30-0004Tz-Tn; Fri, 11 Dec 2020 10:09:32 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 11 Dec 2020 16:09:10 +0100 Message-Id: <20201211150919.18435-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <87h7pkffzy.fsf@inria.fr> References: <87h7pkffzy.fsf@inria.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.80 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: E53F49404E8 X-Spam-Score: -1.80 X-Migadu-Scanner: scn1.migadu.com X-TUID: Lnrmiy3jy4F5 Hi there! Here’s a long and rather boring patch series to address and a bit more. To avoid traversing store items repeatedly as described in the issue above, the strategy here is to gradually move the reset-timestamps and deduplicate phases as part of the file copying process, such that each file is accessed only once. Consequently, the kitchen sink that ‘register-items’ once was is now very focused. Furthermore, it changes ‘guix system init’ so that it reuses the already-known store item hashes when populating the target database instead of re-traversing store items. On my laptop (SSD, warm cache, derivations already built), the command: guix system init gnu/system/examples/bare-bones.tmpl /tmp/sys goes from 32s to 22s, a 33% improvement. Feedback welcome! Ludo’. Ludovic Courtès (15): serialization: 'fold-archive' notifies about directory processing completion. serialization: 'restore-file' sets canonical timestamp and permissions. nar: Deduplicate files right as they are restored. store-copy: 'populate-store' resets timestamps. image: 'register-closure' assumes already-reset timestamps. database: Remove #:reset-timestamps? from 'register-items'. store-copy: 'populate-store' can optionally deduplicate files. image: 'register-closure' leaves it up to the caller to deduplicate. database: Remove #:deduplicate? from 'register-items'. guix system: 'init' copies, resets timestamps, and deduplicates at once. database: Remove #:deduplicate? and #:reset-timestamps? from 'register-path'. system: 'init' does not recompute the hash of each store item. database: Remove 'register-path'. database: Honor 'SOURCE_DATE_EPOCH'. deduplicate: Create the '.links' directory lazily. .dir-locals.el | 1 + gnu/build/image.scm | 16 +- gnu/build/install.scm | 3 +- gnu/build/linux-initrd.scm | 3 +- gnu/build/vm.scm | 14 +- gnu/system/install.scm | 12 +- gnu/system/linux-initrd.scm | 10 +- guix/build/store-copy.scm | 133 ++++++++++++---- guix/nar.scm | 8 +- guix/scripts/archive.scm | 2 + guix/scripts/challenge.scm | 1 + guix/scripts/pack.scm | 276 +++++++++++++++++----------------- guix/scripts/system.scm | 64 ++++---- guix/serialization.scm | 36 +++-- guix/store/database.scm | 58 ++----- guix/store/deduplication.scm | 167 ++++++++++++++------ tests/gexp.scm | 20 ++- tests/guix-archive.sh | 4 +- tests/nar.scm | 21 ++- tests/store-database.scm | 18 ++- tests/store-deduplication.scm | 20 ++- 21 files changed, 544 insertions(+), 343 deletions(-) -- 2.29.2