From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail Date: Tue, 28 Nov 2017 14:30:59 +0100 Message-ID: <87d1421qek.fsf@gnu.org> References: <877ewf18d4.fsf@gnu.org> <87o9ppoabw.fsf@gnu.org> <20171002182208.GB10773@jasmine.lan> <878tgt721q.fsf@gnu.org> <20171020211700.GA32355@jasmine.lan> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57473) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eJfzf-0005B2-Ta for bug-guix@gnu.org; Tue, 28 Nov 2017 08:32:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eJfzW-0006hR-6r for bug-guix@gnu.org; Tue, 28 Nov 2017 08:32:11 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:53165) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eJfzW-0006hM-27 for bug-guix@gnu.org; Tue, 28 Nov 2017 08:32:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1eJfzV-0004nb-OZ for bug-guix@gnu.org; Tue, 28 Nov 2017 08:32:01 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <20171020211700.GA32355@jasmine.lan> (Leo Famulari's message of "Fri, 20 Oct 2017 17:17:00 -0400") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Leo Famulari Cc: 28659@debbugs.gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Leo Famulari skribis: > On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Court=C3=A8s wrote: >> Right. Jan suggested checking the content-addressed mirrors *before* >> the real upstream address. That would address the problem of upstream >> sources modified in-place, but at the cost of privacy/self-sufficiency >> as you note. (Though it=E2=80=99s not really making =E2=80=9Cprivacy=E2= =80=9D any worse in this >> case: it=E2=80=99s gnu.org vs. github.com.) > > Yeah, I don't personally think there is a privacy issue with fetching > sources from our mirrors at gnu.org, or other domains we control. > >> Perhaps we should make content-addressed mirrors configurable in a way >> that=E2=80=99s orthogonal to derivations, something similar in spirit to >> --substitute-urls? The difficulty is that content-addressed mirrors are >> not just URLs; see (guix download). >> >> Thoughts? > > I do think we should make it so that users don't suffer from unreliable > upstream sources when we know the sources are available on our servers > (or the Nix mirror), even with --no-substitutes. The more I think about it, the more I=E2=80=99m inclined to simply move content-addressed mirrors to the front of the list. This means that users, in practice, would be fetching all the source from mirror.hydra.gnu.org. The main issue is making it configurable. Currently the content-addressed mirror configuration for regular files in (guix download) looks like this: --8<---------------cut here---------------start------------->8--- (define %content-addressed-mirrors ;; List of content-addressed mirrors. Each mirror is represented as a ;; procedure that takes a file name, an algorithm (symbol) and a hash ;; (bytevector), and returns a URL or #f. ;; Note: Avoid 'https' to mitigate . ;; TODO: Add more. '(list (lambda (file algo hash) ;; Files served by 'guix publish' are accessible under a single ;; hash algorithm. (string-append "http://mirror.hydra.gnu.org/file/" file "/" (symbol->string algo) "/" (bytevector->nix-base32-string hash))) (lambda (file algo hash) ;; 'tarballs.nixos.org' supports several algorithms. (string-append "http://tarballs.nixos.org/" (symbol->string algo) "/" (bytevector->nix-base32-string hash))))) --8<---------------cut here---------------end--------------->8--- That for VCS checkouts in (guix build download-nar) looks like this: --8<---------------cut here---------------start------------->8--- (define (urls-for-item item) "Return the fallback nar URL for ITEM--e.g., \"/gnu/store/cabbag3=E2=80=A6-foo-1.2-checkout\"." ;; Here we hard-code nar URLs without checking narinfos. That's probably= OK ;; though. ;; TODO: Use HTTPS? The downside is the extra dependency. (let ((bases '("http://mirror.hydra.gnu.org/guix" "http://berlin.guixsd.org")) (item (basename item))) (append (map (cut string-append <> "/nar/gzip/" item) bases) (map (cut string-append <> "/nar/" item) bases)))) --8<---------------cut here---------------end--------------->8--- The latter could be expressed by a command-line flag. In fact it=E2=80=99s= the same as --substitute-urls. (Time passes=E2=80=A6) Thinking more about it, why not simply always enable substitutes for fixed-output derivations, like this: --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index d68e8b2bc..03a8f5080 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation() /* We are first going to try to create the invalid output paths through substitutes. If that doesn't work, we'll build - them. */ - if (settings.useSubstitutes && substitutesAllowed(drv)) + them. Always enable substitutes for fixed-output derivations to + protect against disappearing files and in-place modifications on + upstream sites. */ + if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv)) foreach (PathSet::iterator, i, invalidOutputs) addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair)); --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable This solves all our problems and makes download-nar.scm useless. As an added bonus, it provides a improves the UI since we now always see: --8<---------------cut here---------------start------------->8--- 0.1 MB will be downloaded: /gnu/store/plx9848n6waj6zghn3d54ybx8ihcn23k-guile-git-0.0-4.951a32c-chec= kout --8<---------------cut here---------------end--------------->8--- =E2=80=A6 instead of: --8<---------------cut here---------------start------------->8--- The following derivation will be built: /gnu/store/y86rlb6pdm35im7q02y6479ca84zwylz-guile-git-000.0-4.951a32c-ch= eckout.drv --8<---------------cut here---------------end--------------->8--- The downside is that it still requires one to authorize the server=E2=80=99s key, although it=E2=80=99s in theory unnecessary since it=E2=80=99s content= addressed. I=E2=80=99m not sure how to solve that because =E2=80=98guix substitute=E2= =80=99 doesn=E2=80=99t know that it=E2=80=99s substituting a fixed-output derivation. I suppose we=E2= =80=99d need to modify the =E2=80=9Cprotocol=E2=80=9D between guix-daemon and =E2=80=98g= uix substitute=E2=80=99. Thoughts? Ludo=E2=80=99. --=-=-=--