From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 8Mr/MJ4eXmdsoQAAe85BDQ:P1 (envelope-from ) for ; Sun, 15 Dec 2024 00:11:11 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id 8Mr/MJ4eXmdsoQAAe85BDQ (envelope-from ) for ; Sun, 15 Dec 2024 01:11:10 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=iDlUILK5; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1734221470; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=xRXCEAHyZLbeNRFPfHGrPIM+TiCVPCwjDJ6rdalj2Vs=; b=Hq+dpIt33unfb46NaSNAZDxzhWlWdomGLavqjHf8SAEUxmKBQMIaREGvmm4P4eA+w3VM1q I7dRTQXxF6OGBkKm3NiJJwriSyrVWlicVZEoQKsY+t7oVDvmthT110bgVy7HtrBCQzWda9 pkGvz+PumN6GQEqjtf5bXquROOJYpstA1PazxDe8crGvljFplGZVPpEJOrEgGjqZZpDZbR cu2Q2WG05q+czlSIer0gs+lwZqJ4hbiL0CnllAWerzffxWo+HSmnEXMh615H4KbT0ilUCx 70VBxJ1QpyCCWx9tD0R1rycdFEP6In3SnvMilE3h7sK982hB42tCIJWSnPto5g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=iDlUILK5; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1734221470; a=rsa-sha256; cv=none; b=B82D+HgD9lcqTZQ9UUyvztVw3OAqyCrn/YR2EPQJMmXPY5aHMQka4IwlMXnNRelBNEToJf YTa5viwSonL57gKBLaAhiMCr3NzvvMwvkecnx83e3gFz0T95ZDDgkvq2QoKvcNnJNsotYE cQxHO8U5/ohmb5Y/r2zxh714X8W9ASLVYj/JtW1dZEQ+tnovZP7pbH/i1nYet9beaY4yLB eJYNMSLHfj/UOQyENbu5iW28VVLmXv5N3H6jg00IdmRrNcWpiX7ZfMqLf+L8HJSvkMxeMF 2vCj9bdsME/x1E6HkoRMTIxGQ3X99Hq1wZqLayaFaYGuq23261qyxwkHLRBIMQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8825753879 for ; Sun, 15 Dec 2024 01:11:10 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tMcDf-0000mP-U9; Sat, 14 Dec 2024 19:10:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tMcDd-0000lz-Pn for guix-devel@gnu.org; Sat, 14 Dec 2024 19:10:45 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tMcDd-0001oW-DN; Sat, 14 Dec 2024 19:10:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=xRXCEAHyZLbeNRFPfHGrPIM+TiCVPCwjDJ6rdalj2Vs=; b=iDlUILK5oxODiaLTx6ab ztpRWbo2GUzFkLuRBhBMO9u7PwPCGYWkg/6Q02ntqDdsI6oDz1mrwdZt9XsjvrfXLmi3hetpdV8fW nB01rQhH4jeqKV8yTaZp5TdNlls3zqGuf8lD5YWamDOreP8PzVtZoSTbX5BBxkzQMbqDJx0pQf1FP LPfuIdsyhRDXjAYXQuVtUCqamWNLMlj/gERGxqO8kR1U14eN89Gf9B0P3UwKD1CyE/Mw+GybGZIxP 7UogLbIkw3xz5ChNDkiSZtCac44Q1lT3iP48lqjKILXz6nn268YjoPX77whWNW079lehA9HV8v1JK Ed7yUMcH9jZ6xg==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Cc: guix-devel@gnu.org Subject: Re: Speed up grafts by storing reference offset in index In-Reply-To: <87o71frbl8.fsf@elephly.net> (Ricardo Wurmus's message of "Fri, 13 Dec 2024 13:50:27 +0100") References: <87o71frbl8.fsf@elephly.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Quintidi 25 Frimaire an 233 de la =?utf-8?Q?R=C3=A9v?= =?utf-8?Q?olution=2C?= jour du Grillon X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 15 Dec 2024 01:10:43 +0100 Message-ID: <871py9kdq4.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 8825753879 X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -8.35 X-Spam-Score: -8.35 X-TUID: yV+MZeFw5/AN Hi! Ricardo Wurmus skribis: > Since it is December and I'm in a silly mood here is a silly idea: would > it make sense to shift parts of the grafting work to an offloadable > build? Here's what I imagine: > > - on the build farms build an additional derivation for a references > file. The references file is an S-expression containing a list of > tuples of the form (FILE-NAME OFFSET). Each of these tuples > identifies the location of a single reference at the recorded byte > OFFSET in FILE-NAME. > > - when computing grafts, don't search the local files sequentially for > references but look them up in the references file. Instead of > computing the reference file substitute it from a build server. This sounds quite ambitious and it=E2=80=99s unclear that this would be beneficial (it would be beneficial *if* scanning for references is substantially more expensive than just copying the part of the file that would be scanned, and it=E2=80=99s far from obvious that this holds.) I have another, more down-to-earth proposal: ungraft more often! That=E2= =80=99s the spirit of the auto-ungraft manifest and jobset: =E2=80=A6 but it doesn=E2=80=99t quite w= ork as expected because of =E2=80=98rust-ring=E2=80=99 shenanigans: . (Making grafting faster would still be welcome, but I=E2=80=99d rather look= for a =E2=80=9Clocal=E2=80=9D optimization in the code itself.) Ludo=E2=80=99.