From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id KLRbE1nnQWa3AAAAqHPOHw:P1 (envelope-from ) for ; Mon, 13 May 2024 12:11:37 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id KLRbE1nnQWa3AAAAqHPOHw (envelope-from ) for ; Mon, 13 May 2024 12:11:37 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=frsWhmhw; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1715595097; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=MA8XKeP6+jP6u3AJSD+5h+T8XN14wKaetTkznCASlQ4=; b=UfTNffHC0xnH9q9RLxzbVYFneL/8CvI6Riz3kOFvtgbEFqh7H3FThlPShdI6t11h8Y3GWR dFwKZFqf50gHSX/d1Ug3BXzwEnuNPRxOZi3eOv2eB6p/mIb+aXXhCcqqfS+tYLwtAlV5GR UGOVpz4r5oTVTFugcHpd0D2p7h4XpEqGAz1ltN8+dlNQPhbMaD7kOFf0idkJ7nhlXx0sdD 69Zcld4DInrNEWN1KzSipcZX6zalIPiw16V7x9Becty+nG61IV8H4g7OGqMH34oqpnrz7M 7QTXtIBNLKGcpZPlkZg9Cn0Dg28y+2Q1Gbit0m2HlHFFUQK+gckGMHWvMHwmkA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=frsWhmhw; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1715595097; a=rsa-sha256; cv=none; b=PqpDNEYPtxBOYmLQxbl5bQ10JqFWKyxWHnAC3r/tzO2zG3SrQqXmWNQvy4cy7VwRfnsHHX eCU5qjOiih/m4Q/lRCQAwzUwa+Njs5qgsJWXf6tnAOToWgTtY9cfWY8EMK/NtjZcf4F4GS pqTUjJJ1ZDN/1+F5FnqvBvdE+ARnFd2rBbjaa3cLl2F/TwPGW2Flq/vMMyXK9KWI2T3F/7 +OMTEYNY8EEASmhIYOGSpnTiuR8hqQzV2WM63HDMf17wYRWvL2ML41GsYLZiv0zbcpq1Fd YuIp+ZQXVkIncVr4imLnN2kVua3RZwNhVGHLFVeYOujac1/KAD6o6YP6EPiafA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 245BB60FE5 for ; Mon, 13 May 2024 12:11:37 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s6SeD-0005n1-VE; Mon, 13 May 2024 06:11:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6Se5-0005kz-N5 for bug-guix@gnu.org; Mon, 13 May 2024 06:11:02 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s6Se5-0006Ml-Ck for bug-guix@gnu.org; Mon, 13 May 2024 06:11:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s6Se5-0008Bl-Nx for bug-guix@gnu.org; Mon, 13 May 2024 06:11:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 13 May 2024 10:11:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: 70877@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171559504731463 (code B ref 70877); Mon, 13 May 2024 10:11:01 +0000 Received: (at 70877) by debbugs.gnu.org; 13 May 2024 10:10:47 +0000 Received: from localhost ([127.0.0.1]:60017 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6Sdq-0008BP-UP for submit@debbugs.gnu.org; Mon, 13 May 2024 06:10:47 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50268) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6Sdo-0008BF-Ld for 70877@debbugs.gnu.org; Mon, 13 May 2024 06:10:45 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6Sdi-000665-1c; Mon, 13 May 2024 06:10:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=MA8XKeP6+jP6u3AJSD+5h+T8XN14wKaetTkznCASlQ4=; b=frsWhmhwbaFMfE1wcpNc lurQLXmmek3Y7865LyNEe2JcGraREgVRRrT3oJUBvPiaQomKTaIjVBKVlgfh23Yw4MMpeE/PYsZpJ g8uun1CCmpgVmIxtpUROrG6pXCEPI4yQXBdWp5HgQcUQhYnTbe1ZTD83vuj8FE8241U7/ghGRJ4n5 DY6fEZxw701QQTfZsI/4BTat/Ci/WLZa9vxS6g5jAF2lDACLaK45RBk9Os/5aCFU3FAFdneCWkV1E CsNR4uVx7VXpzqJftaJKT0Iq5kNuUsJX/svuqN0E5NeCJixBHUpyOQ/hzz+NOnC6p2acmVmiBB7P7 xZ+ELhCQf3eNvQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <87msowpqm2.fsf@elephly.net> (Ricardo Wurmus's message of "Sat, 11 May 2024 12:52:53 +0200") References: <87msowpqm2.fsf@elephly.net> Date: Mon, 13 May 2024 12:10:34 +0200 Message-ID: <87v83ic99h.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -7.27 X-Spam-Score: -7.27 X-Migadu-Queue-Id: 245BB60FE5 X-Migadu-Scanner: mx13.migadu.com X-TUID: 9Jzonzp+D688 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Thanks for the bug report and nice reproducer! Ricardo Wurmus skribis: > The guix-daemon's libutil/util.cc uses copy_file_range to copy a > downloaded file into the store. copy_file_range fails on files larger > than 4GB with an error like this: > > guix build: error: short write in copy_file_range `15' to `16': No su= ch file or directory > > The man page for copy_file_range says that it could return EFBIG when > the range exceeds the maximum range. The daemon code does not check any > limits and will attempt to copy the whole file. > > I believe our code ought to check the value of st.size and fall back to > a boring copy if it exceeds some "reasonable" value. The goal leading to this error message looks like this: copy_file_range(15, NULL, 16, NULL, 4294967297, 0) =3D 2147479552 =E2=80=A6 which is precisely 2 GiB - 4 KiB. Reading the man page, it=E2=80=99s entirely fine: like =E2=80=98write=E2=80= =99, =E2=80=98copy_file_range=E2=80=99 might copy less than asked for, so it=E2= =80=99s really a mistake of mine to assume that short writes can=E2=80=99t happen. Presumab= ly there=E2=80=99s an internal limit here we=E2=80=99re reaching that explains= why it won=E2=80=99t copy more than 2 GiB at once. With the following change, we get: newfstatat(15, "", {st_mode=3DS_IFREG|0644, st_size=3D4294967297, ...}, A= T_EMPTY_PATH) =3D 0 copy_file_range(15, NULL, 16, NULL, 4294967297, 0) =3D 2147479552 copy_file_range(15, NULL, 16, NULL, 2147487745, 0) =3D 2147479552 copy_file_range(15, NULL, 16, NULL, 8193, 0) =3D 8193 fchown(16, 30001, 30000) =3D 0 Could you confirm that it works for you? Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-daemon-Loop-over-copy_file_range-upon-short-writes.patch Content-Transfer-Encoding: quoted-printable Content-Description: the patch >From efd9f3383756df9959651125c0f2e2e769630851 Mon Sep 17 00:00:00 2001 Message-ID: From: =3D?UTF-8?q?Ludovic=3D20Court=3DC3=3DA8s?=3D Date: Mon, 13 May 2024 12:02:30 +0200 Subject: [PATCH] =3D?UTF-8?q?daemon:=3D20Loop=3D20over=3D20=3DE2=3D80=3D98c= opy=3D5Ffile=3D5F?=3D =3D?UTF-8?q?range=3DE2=3D80=3D99=3D20upon=3D20short=3D20writes.?=3D MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit Fixes . * nix/libutil/util.cc (copyFile): Loop over =E2=80=98copy_file_range=E2=80= =99 instead of throwing upon short write. Reported-by: Ricardo Wurmus Change-Id: Id7b8a65ea59006c2d91bc23732309a68665b9ca0 --- nix/libutil/util.cc | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/nix/libutil/util.cc b/nix/libutil/util.cc index 578d6572934..3206dea11b1 100644 --- a/nix/libutil/util.cc +++ b/nix/libutil/util.cc @@ -397,9 +397,14 @@ static void copyFile(int sourceFd, int destinationFd) } else { if (result < 0) throw SysError(format("copy_file_range `%1%' to `%2%'") % sourceFd % = destinationFd); - if (result < st.st_size) - throw SysError(format("short write in copy_file_range `%1%' to `%2%'") - % sourceFd % destinationFd); + + /* If 'copy_file_range' copied less than requested, try again. */ + for (ssize_t copied =3D result; copied < st.st_size; copied +=3D result) { + result =3D copy_file_range(sourceFd, NULL, destinationFd, NULL, + st.st_size - copied, 0); + if (result < 0) + throw SysError(format("copy_file_range `%1%' to `%2%'") % sourceFd % des= tinationFd); + } } } =20 base-commit: 89cd778f6a45cd9b43a4dc1f236dcd0a87af955c --=20 2.41.0 --=-=-=--