From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Corrupt .drv files Date: Mon, 11 Jul 2016 23:49:08 +0200 Message-ID: <87twfv26cr.fsf_-_@gnu.org> References: <20160711081921.GA1651@solar> <87k2gsjooh.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50688) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bMj4n-0001iG-Dc for guix-devel@gnu.org; Mon, 11 Jul 2016 17:49:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bMj4j-00082q-4R for guix-devel@gnu.org; Mon, 11 Jul 2016 17:49:16 -0400 In-Reply-To: <87k2gsjooh.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 11 Jul 2016 15:21:18 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Andreas Enge Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable (Moving to guix-devel.) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Andreas Enge skribis: > >> guix archive: error: build failed: error parsing derivation `/gnu/store/= k49lwfwgs8wcamys5qzn8c5n2zk0prc1-tcl8.6.4-src.tar.xz.drv': expected string = `Derive([' > > It looks like the store on this machine is corrupt. Indeed, the daemon doesn=E2=80=99t attempt to atomically write files coming= from an add-to-store RPC, which includes .drv files. So I think that if you pull the plug before the .drv has been flushed to disk but after the .drv has been marked as valid in the SQLite database (which is likely to happen in a timely fashion because SQLite does the =E2=80=98fdatasync=E2=80=99 dance appropriately), then you end up with a tr= uncated .drv file. (This is acknowledged by the comment in =E2=80=98LocalStore::registerValidPaths=E2=80=99, which can call =E2=80=98s= ync=E2=80=99, a sledgehammer.) The attached patch should fix it. I think the performance overhead of the extra =E2=80=98fdatasync=E2=80=99 should be OK but I haven=E2=80=99t ma= de any measurements. Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/nix/libstore/local-store.cc b/nix/libstore/local-store.cc index 347e8a7..80566ea 100644 --- a/nix/libstore/local-store.cc +++ b/nix/libstore/local-store.cc @@ -1391,7 +1391,7 @@ Path LocalStore::addToStoreFromDump(const string & dump, const string & name, StringSource source(dump); restorePath(dstPath, source); } else - writeFile(dstPath, dump); + writeFileAtomically(dstPath, dump); canonicalisePathMetaData(dstPath, -1); diff --git a/nix/libutil/util.cc b/nix/libutil/util.cc index c077544..e33824b 100644 --- a/nix/libutil/util.cc +++ b/nix/libutil/util.cc @@ -272,6 +272,22 @@ void writeFile(const Path & path, const string & s) writeFull(fd, s); } +void writeFileAtomically(const Path & path, const string & s) +{ + auto tmpPath = path + ".tmp"; + AutoCloseFD fd = open(tmpPath.c_str(), O_WRONLY | O_TRUNC | O_CREAT, 0666); + if (fd == -1) + throw SysError(format("opening file '%1%'") % tmpPath); + writeFull(fd, s); +#if _POSIX_SYNCHRONIZED_IO > 0 + if (fdatasync(fd) != 0) + throw SysError(format("flushing file '%1%'") % tmpPath); +#endif + fd.close(); + if (rename(tmpPath.c_str(), path.c_str()) != 0) + throw SysError(format("renaming '%1%' to '%2'") % tmpPath % path); +} + string readLine(int fd) { diff --git a/nix/libutil/util.hh b/nix/libutil/util.hh index e84d64d..829f90b 100644 --- a/nix/libutil/util.hh +++ b/nix/libutil/util.hh @@ -86,6 +86,9 @@ string readFile(const Path & path, bool drain = false); /* Write a string to a file. */ void writeFile(const Path & path, const string & s); +/* Same, but do it atomically. */ +void writeFileAtomically(const Path & path, const string & s); + /* Read a line from a file descriptor. */ string readLine(int fd); --=-=-=--