From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id wPgWOk0ukmSxeQEASxT56A (envelope-from ) for ; Wed, 21 Jun 2023 00:55:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 0N4MOk0ukmS9VwEAauVa8A (envelope-from ) for ; Wed, 21 Jun 2023 00:55:09 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BCD1D3ED66 for ; Wed, 21 Jun 2023 00:55:09 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBkF8-0008DK-N8; Tue, 20 Jun 2023 18:54:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBkF6-0008Cw-H0 for guix-devel@gnu.org; Tue, 20 Jun 2023 18:54:32 -0400 Received: from mx1.riseup.net ([198.252.153.129]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBkF4-0000dC-QU for guix-devel@gnu.org; Tue, 20 Jun 2023 18:54:32 -0400 Received: from fews01-sea.riseup.net (fews01-sea-pn.riseup.net [10.0.1.109]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.riseup.net (Postfix) with ESMTPS id 4Qm2511SgDzDrhC; Tue, 20 Jun 2023 22:54:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1687301669; bh=sh529IBaSRiUGKK1b2kcXMOvBzASloZhuuDnDWWDu80=; h=From:To:Cc:Subject:Date:From; b=ffRn7jLfICZfjxAVEsYqhoOzxNKc7H0JbN3qKeY8+3STfxAeqiPpBg/KEz4ayqtFz 26RpI/MMlB1cwgJBZy9Rbtg+DY4+3UOkCtSQn1y16Ok1sDTs3Snjd414bO6ScYh7fk cBy1aSimotNQKT5rw5gb89brW10ma+givWjzaLJ8= X-Riseup-User-ID: E1494B174B3E54E366FC8974CF313531522A634DC5AC7EB5F90BEC044A4F2AA2 Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews01-sea.riseup.net (Postfix) with ESMTPSA id 4Qm2503cBDzJn8G; Tue, 20 Jun 2023 22:54:28 +0000 (UTC) From: Csepp To: guix-devel@gnu.org Cc: pukkamustard Subject: distributed substitutes: file slicing Date: Wed, 21 Jun 2023 00:44:06 +0200 Message-ID: <87jzvxoicg.fsf@riseup.net> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=198.252.153.129; envelope-from=raingloom@riseup.net; helo=mx1.riseup.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1687301709; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=sh529IBaSRiUGKK1b2kcXMOvBzASloZhuuDnDWWDu80=; b=Op3gVApv+sjy7UpwLPYbFqg8n4mUrdnO4iOWiZWRZojD2VXUfBU+bafQ+UVzXeqrM/WhVo TTxHkSu1LM1CoMj6Grb35wuGkc1zLGskh6Ze95D01P/3YnV2I8tGi+B3NTyYTUvAMM43OD dyiQcWnsl7yOHijdv4k4dSgY1D3bE8utqNWpjuSg/ZsvMOjG5NlDUakQfud5qR+WYen9rN aA9+XY/YdIeAfd1YyEXFD1AcTiCk/0zCUvIARxTgi+rwCNpVEHk1YORBDf4mM6bpE+KhgA 05QZdwmjurrg2amZrFGGwwun0mdbyoHvQXwmsV4dWxJXHZDqesvTFEBwCv8sBw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=ffRn7jLf; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1687301709; a=rsa-sha256; cv=none; b=dx1ttwvKFxn4E+hQIBs56eYqIADB3ce6Nik4xT/6W2HFfWX4gQpltxXGxM3CuTYF9MHC7S jBvpb2vwa+kUz9EQCtRmUyNPZNcNkp5Uk6qait8+IBNIT32dWnNiPRBMGjTTxQu7x8Cwl8 VsYPXF+w3BLgwyhpv7U0Ms9iNMki907lPgrPl6UAf0ExV6i4RR8/wvXStZ5D3+46l6OZlD rNQuXK+2UbNmIuXd/vUqs0dQnZ5cw2OdjtUma8fb0un8cIuGa0IPcxVXLjCvN+OBK9mOPY fufcjVfOF9WT5HwXVzdNtk2h5szpSXc7anfl/+jj1uCtQWF+MTH7XzO9ca7YDQ== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -6.58 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=ffRn7jLf; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: BCD1D3ED66 X-Spam-Score: -6.58 X-TUID: TpZGbkGMk0Ox I have a question / suggestion about the distributed substitutes project: would downloads be split into uniformly sized chunks or could the sizes vary? Specifically, in an extreme case where an update introduced a single extra byte at the beginning of a file, would that result in completely new chunks? An alternative I've been thinking about is this: find the store references in a file and split it along these references, optionally apply further chunking to the non-reference blobs. It's probably best to do this at the NAR level?? Storing reference offsets is already something that we should be doing to speed other operations up, so this could tie in nicely with that.