From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id OL5bOtEtXGfIjAAA62LTzQ:P1 (envelope-from ) for ; Fri, 13 Dec 2024 12:51:30 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id OL5bOtEtXGfIjAAA62LTzQ (envelope-from ) for ; Fri, 13 Dec 2024 13:51:30 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=BATIrWC2; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Seal: i=2; s=key1; d=yhetil.org; t=1734094289; a=rsa-sha256; cv=pass; b=kSbB50JT83buDxtZsCWZzOKL4KMIHJVSdgkvmSJWsnQv+qd23wTfOIvgryrwVPvnvmrVr+ MfcIjJrYpScDU7ScnOKlndtWoppqm3d8UDC/wTFKoatEe/VEOqKN9zBpAMsZpObAbjj1GK N6Ty/0KZz1E+6a0yOZb46IYqEKi32y3xeYMpdB85pgwapik47OVKYJFR2EEeOCBEmguTjX XdselBAKqLTeU8bdU4npEnXsCmjp8upDlIt/ZQPFyQSRkG8zkHtUrCJm6KyM5gdTzynVbd QU8Y2wsjp+4Iw6du3pWpe5rXn34eR8UzXVS9O1W98Wp1+su1E+5bfJAi9ktfrg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=BATIrWC2; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1734094289; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=40qoam4M0fu0T4zz4ymdzOso0SvwVWBChgJnW0SSIfM=; b=pQ0fHzehv73mg8IL8gH6hpHqXsAgVF8e5c1c4fNhMP3SNeYlcFcelKIkubgS8WDy/rOD/r h+Gm4q1DltRQHEwnyEiRQJCRHm7L0u08q28T84FbThex4ELwuK1aJZhoMF+no6glxGHzNy EfYIDX/AebUUrL8xF+7p0QmbrpMwVn8XpCiBd09Tf+hg95woA8BHeDsEMuw8fsg7X2QNmQ 9pRkjUCCUOp1ahvqsTndvuCBXMHrDquLGXqS7N01n8WxpR/l0ojWcC8dBUl60UnyzOzgm+ J72JVQetzdPVQfh31pQhBmcKt+Q5VkqSaw5ZiIf57gCvfGHVG/9yoKGcapYhAw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8E09719AE0 for ; Fri, 13 Dec 2024 13:51:29 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tM580-0005qq-L9; Fri, 13 Dec 2024 07:50:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tM57y-0005qU-ML for guix-devel@gnu.org; Fri, 13 Dec 2024 07:50:43 -0500 Received: from sender4-of-o51.zoho.com ([136.143.188.51]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tM57v-000160-O1 for guix-devel@gnu.org; Fri, 13 Dec 2024 07:50:42 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1734094232; cv=none; d=zohomail.com; s=zohoarc; b=Bbllmq32XQibEb3eLYfi67PI5BMI8shZTHsaBBqoS/NRkV9Y4OjP26P0+OFE1jNps/yjsPBbO5K6grNamPZVGm5IBvd2shX1RWTbBAJpMDU9E+CfWoaWZz8wnUO4ovtLVVZhzfWRjN2fwPrMIRHUG/4LJk9KwaZqqLrc7oC9OsA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1734094232; h=Content-Type:Date:Date:From:From:MIME-Version:Message-ID:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=40qoam4M0fu0T4zz4ymdzOso0SvwVWBChgJnW0SSIfM=; b=WwrK9KUr3/S2YtU+IvgXcGz4RFfTSX6Wv3vtaHR7a7priMOUcBBZ+F+d/4OtZ2zr9A8DjqFlVNuKOfgXk1b/fcvKnaFfQbX+4CUS1B3dmCIZbevjGjS58nbVsPdgMBISiHxChesy24QIaFavq+f4lfYbRVxTf1KJyGK8wTP9Htw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1734094232; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Subject:Subject:Date:Date:Message-ID:MIME-Version:Content-Type:Message-Id:Reply-To:Cc; bh=40qoam4M0fu0T4zz4ymdzOso0SvwVWBChgJnW0SSIfM=; b=BATIrWC2H4CwaysZSoaEM+HaAL5L4Y8j5Dv9wL92Oy7Lvu5RR2AnCbZ0PLVJmz+d OBzsISrYlejTKlQmLk4rT1Yt5tuzi3ZRAzeXMk9JUbqb/T/n032Zr/X+zL4Ds6kxPyb nolrEVl+eyBweOw+Xg1Vm8SLPBGwDwKLlRn7Rr2Q= Received: by mx.zohomail.com with SMTPS id 1734094229880412.10774594313546; Fri, 13 Dec 2024 04:50:29 -0800 (PST) From: Ricardo Wurmus To: guix-devel@gnu.org Subject: Speed up grafts by storing reference offset in index User-Agent: mu4e 1.12.7; emacs 29.4 Date: Fri, 13 Dec 2024 13:50:27 +0100 Message-ID: <87o71frbl8.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -5.97 X-Spam-Score: -5.97 X-Migadu-Queue-Id: 8E09719AE0 X-Migadu-Scanner: mx12.migadu.com X-TUID: BokdqlrpRlnM Hello Guix, grafts can be a little slow for a number of reasons: - they are not substituted, because the assumption is that it is preferrable to rewrite references locally instead of downloading a big archive with the modified file. Local computations on x86_64 are often acceptable, but on aarch64 systems they can be very slow indeed. - when a lot of grafts need to be applied, many files need to be rewritten - big files take longer to read and thus to rewrite Since it is December and I'm in a silly mood here is a silly idea: would it make sense to shift parts of the grafting work to an offloadable build? Here's what I imagine: - on the build farms build an additional derivation for a references file. The references file is an S-expression containing a list of tuples of the form (FILE-NAME OFFSET). Each of these tuples identifies the location of a single reference at the recorded byte OFFSET in FILE-NAME. - when computing grafts, don't search the local files sequentially for references but look them up in the references file. Instead of computing the reference file substitute it from a build server. Alternatively, change the format for substitutes and record reference locations there, so that the local store database can also store reference locations. It already stores references (for things like "guix gc -R"), so maybe we could store just a little extra information to allow us to seek to the offset directly. As you can tell, I haven't thought this through, and there are a number of different places where a feature like this could live. What do you think? -- Ricardo