bug#39575: guix time-machine fails when a tarball was modified in-place

bug#39575: guix time-machine fails when a tarball was modified in-place

[Jan Nieuwenhuizen]

Hi,

Trying to travel back to Sun Apr 7 22:07:14 2019 +0200 (commit
56e95d54d209c2428f970d65d9b27ae4168449ad) to re-create mcrl2-minimal by
doing

--8<---------------cut here---------------start------------->8---
guix time-machine --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- environment --ad-hoc mcrl2-minimal
--8<---------------cut here---------------end--------------->8---

fails with

--8<---------------cut here---------------start------------->8---
building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
|offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
- 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
  expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
  actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
build of /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv failed
View build log at '/var/log/guix/drvs/cj/im33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv.bz2'.
cannot build derivation `/gnu/store/p6gfcdacjcqf2br0zwsyzx1chfvg9gxi-harfbuzz-2.4.0.drv': 1 dependencies couldn't be built
killing process 5083
--8<---------------cut here---------------end--------------->8---

The recipe for harfbuzz has a sha256 that used to be valid in April, but
hasn't been valid anymore since May, as this fix

--8<---------------cut here---------------start------------->8---
commit a8bb8fccd82a10a46f127b2235675b4f6cbaaf98
Author: Marius Bakke <mbakke@fastmail.com>
Date:   Sat May 4 18:01:12 2019 +0200

    gnu: harfbuzz: Update source hash.
    
    The previous tarball was modified in-place; see
    <https://github.com/harfbuzz/harfbuzz/issues/1641>.
    
    * gnu/packages/gtk.scm (harfbuzz)[source](sha256): Update.
--8<---------------cut here---------------end--------------->8---

shows.  Thoughts?

Greetings,
janneke


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,


On Wed, 12 Feb 2020 at 14:44, Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> Trying to travel back to Sun Apr 7 22:07:14 2019 +0200 (commit
> 56e95d54d209c2428f970d65d9b27ae4168449ad) to re-create mcrl2-minimal by
> doing
>
> --8<---------------cut here---------------start------------->8---
> guix time-machine --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- environment --ad-hoc mcrl2-minimal
> --8<---------------cut here---------------end--------------->8---

Even the simple:

--8<---------------cut here---------------start------------->8---
guix time-machine --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- help
--8<---------------cut here---------------end--------------->8---

> fails with
>
> --8<---------------cut here---------------start------------->8---
> building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
> downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
> |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
> - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
>   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
> hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
> build of /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv failed
> View build log at '/var/log/guix/drvs/cj/im33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv.bz2'.
> cannot build derivation `/gnu/store/p6gfcdacjcqf2br0zwsyzx1chfvg9gxi-harfbuzz-2.4.0.drv': 1 dependencies couldn't be built
> killing process 5083
> --8<---------------cut here---------------end--------------->8---

same for e.g., this commit:

--8<---------------cut here---------------start------------->8---
 guix time-machine --commit=ae528aaf19f3828d3d7d204b15570800e1bbf100 -- help
--8<---------------cut here---------------end--------------->8---


> The recipe for harfbuzz has a sha256 that used to be valid in April, but
> hasn't been valid anymore since May, as this fix
>
> --8<---------------cut here---------------start------------->8---
> commit a8bb8fccd82a10a46f127b2235675b4f6cbaaf98
> Author: Marius Bakke <mbakke@fastmail.com>
> Date:   Sat May 4 18:01:12 2019 +0200
>
>     gnu: harfbuzz: Update source hash.
>
>     The previous tarball was modified in-place; see
>     <https://github.com/harfbuzz/harfbuzz/issues/1641>.
>
>     * gnu/packages/gtk.scm (harfbuzz)[source](sha256): Update.
> --8<---------------cut here---------------end--------------->8---
>
> shows.  Thoughts?

Therefore, all the commits between the introduction of harfbuzz with
the old sha256 (commit 2da9b81837fd1e6f08a10952784d3358be982855) and
the commit updating to the new sha256 should be broken.
Roughly speaking, all the commits between April, 7th and the May, 4th;
i.e., 1100+ commits, isn't it?


Well, this ask an interesting question: how Guix can fallback when
upstream is doing wrong?


Considering this 'harbuzz' issue, is it possible to rebuild the old
tarball and push it to SoftWare Heritage? Then when a sha mismatch
happens, fallback and try to fetch it from SWH?


WDYT?

Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
> downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
> |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
> - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
>   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
> hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'

The problem here is really that we fall back to content-addressed
mirrors instead of using them directly:

  https://issues.guix.gnu.org/issue/28659

The file itself is still available on our machines though, and you can
get it with:

  guix download -o harfbuzz-2.4.0.tar.bz2 \
  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l

  guix download file://$PWD/harfbuzz-2.4.0.tar.bz2

After that, re-running ‘guix time-machine’ should work.

Using ci.guix.gnu.org for substitutes should have the same effect.

HTH,
Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi Ludo,

On Thu, 13 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

> The problem here is really that we fall back to content-addressed
> mirrors instead of using them directly:
>
>   https://issues.guix.gnu.org/issue/28659

Thank you for the pointer. Good to see that the problem is almost addressed.
I will try to understand the discussion and see what is the status of
the proposed patch.


> The file itself is still available on our machines though, and you can
> get it with:

It is an half cooked solution because the Guix project cannot archive
all; for example in term of store resources.

The content-addressed mirror should be SWH, IMHO.

Well, once sources.json will be up, it should be almost done for the future.
But we still need to push all the correct sources that are on
ci.guix.gnu.org; at least all the url based source of package.

Do you have suggestion for a plan?


>   guix download -o harfbuzz-2.4.0.tar.bz2 \
>   https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>
>   guix download file://$PWD/harfbuzz-2.4.0.tar.bz2
>
> After that, re-running ‘guix time-machine’ should work.

Thank you. This should fix the harfbuzz mismatch issue. Cool! :-)


> Using ci.guix.gnu.org for substitutes should have the same effect.

Hum? I thought that I used ci.guix.gnu.org as substitutes... soI need to check.


Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1102 bytes --]

Hello Ludo'

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> The problem here is really that we fall back to content-addressed
> mirrors instead of using them directly:
>
>   https://issues.guix.gnu.org/issue/28659

Given the natute (AFAIU) of this issue is the very same of the bug you
mention, shouldn't this bug be merged (forcemerged?) whith #28659?

If you agree and have no time, I can try doing this

> The file itself is still available on our machines though, and you can
> get it with:
>
>   guix download -o harfbuzz-2.4.0.tar.bz2 \
>   https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>
>   guix download file://$PWD/harfbuzz-2.4.0.tar.bz2
>
> After that, re-running ‘guix time-machine’ should work.

Does make it sense to make this workaround more general and add this as
a Cookbook or blog entry? If the patch you propose in #28659 solve any
issue and is ready to be merget of course it's not worth the effort

[...]

Thanks! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi Giovanni,

On Fri, 14 Feb 2020 at 11:07, Giovanni Biscuolo <g@xelera.eu> wrote:
> Ludovic Courtès <ludo@gnu.org> writes:

> > The problem here is really that we fall back to content-addressed
> > mirrors instead of using them directly:
> >
> >   https://issues.guix.gnu.org/issue/28659
>
> Given the natute (AFAIU) of this issue is the very same of the bug you
> mention, shouldn't this bug be merged (forcemerged?) whith #28659?

AFAIU, there is 2 issues:
 1. how to do with the particular case of harfbuzz -- bug#39575
 2. how to solve the general case of unreliable sources -- bug#28659

So instead of merging the bugs (case 2.), I would like to solve 1.,
mark it as done and report to bug#28659. It will ease to follow
because the thread in bug#28659 is already heavy. :-)



All the best,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi Ludo,

On Thu, 13 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi,
>
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
> > building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
> > downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
> > |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
> > - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
> >   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
> >   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
> > hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
>
> The file itself is still available on our machines though, and you can
> get it with:
>
>   guix download -o harfbuzz-2.4.0.tar.bz2 \
>   https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l

Maybe I miss a point, but the file we need is the old one, not the new
one, i.e., the one with the expected hash
1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch. And I should do
wrong but ci.guix.gnu.org does not have this file -- otherwise it will
find it because of substitutes mechanism.

--8<---------------cut here---------------start------------->8---
 $ guix download -o /tmp/harfbuzz-old.tar.bz2 \
 https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch

Starting download of /tmp/harfbuzz-old.tar.bz2
From https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch...
download failed
"https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch"
404 "Not Found"
failed to download "/tmp/harfbuzz-old.tar.bz2" from
"https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch"
guix download: error: open-file: No such file or directory:
"/tmp/harfbuzz-old.tar.bz2"
--8<---------------cut here---------------end--------------->8---



Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 13 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> Hi,
>>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>> > building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
>> > downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
>> > |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
>> > - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
>> >   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>> >   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>> > hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
>>
>> The file itself is still available on our machines though, and you can
>> get it with:
>>
>>   guix download -o harfbuzz-2.4.0.tar.bz2 \
>>   https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>
> Maybe I miss a point, but the file we need is the old one, not the new
> one, i.e., the one with the expected hash
> 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch.

Oops, my bad.

> And I should do wrong but ci.guix.gnu.org does not have this file --
> otherwise it will find it because of substitutes mechanism.
>
>  $ guix download -o /tmp/harfbuzz-old.tar.bz2 \
>  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch

I checked on a bunch of machines and couldn’t find it.

Everyone, please check whether you have
/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2 and
so share!

Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> On Thu, 13 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>>>
>>> Hi,
>>>
>>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>>
>>> > building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
>>> > downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
>>> > |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
>>> > - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
>>> >   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>>> >   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>>> > hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
>>>
>>> The file itself is still available on our machines though, and you can
>>> get it with:
>>>
>>>   guix download -o harfbuzz-2.4.0.tar.bz2 \
>>>   https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>>
>> Maybe I miss a point, but the file we need is the old one, not the new
>> one, i.e., the one with the expected hash
>> 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch.
>
> Oops, my bad.
>
>> And I should do wrong but ci.guix.gnu.org does not have this file --
>> otherwise it will find it because of substitutes mechanism.
>>
>>  $ guix download -o /tmp/harfbuzz-old.tar.bz2 \
>>  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>
> I checked on a bunch of machines and couldn’t find it.
>
> Everyone, please check whether you have
> /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2 and
> so share!

What about

    https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2

(The strange thing being here, that snapshot.debian.org does not provide
a copy of the the in-place rewritten upstream tarball, either on
2019-05-06 or later.)

So, this now becomes the recipe

    wget -O harfbuzz-2.4.0.tar.bz2 https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
    guix download $PWD/harfbuzz-2.4.0.tar.bz2
    guix time-machine --commit=56e95d54d209c2428f970d65d9b27ae4168449ad --no-offload -- help

that i'm trying now, and for now it looks fine (lots of stuff to build,
i'll report success or failure when it's done).

It seems, however, that for offload builds to work the guix download
needs to be repeated on the offload build farm machines too?

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> What about
>
>     https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2

Good idea.

> So, this now becomes the recipe
>
>     wget -O harfbuzz-2.4.0.tar.bz2 https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
>     guix download $PWD/harfbuzz-2.4.0.tar.bz2
>     guix time-machine --commit=56e95d54d209c2428f970d65d9b27ae4168449ad --no-offload -- help
>
> that i'm trying now, and for now it looks fine (lots of stuff to build,
> i'll report success or failure when it's done).

OK!

> It seems, however, that for offload builds to work the guix download
> needs to be repeated on the offload build farm machines too?

No, I don’t think so, because the head node copies all the inputs to
build machines before it actually offloads the build.

Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,

On Fri, 14 Feb 2020 at 14:51, Ludovic Courtès <ludo@gnu.org> wrote:
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> > What about
> >
> >     https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
>
> Good idea.

Cool!
But how do you determine the "date", i.e., this reference '20190406T212022Z' ?

Could it be automated?


Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Jan, Simon,

Janneke 写道：
> https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2

This is a wonderful resource!  Thank you, Janneke (and Debian)!

zimoun 写道：
> Cool!
> But how do you determine the "date", i.e., this reference 
> '20190406T212022Z' ?

You'd take the timestamp immediately preceding your desired (Guix) 
commit's date, or something like that.  The fact that git commit 
dates aren't linear shouldn't hurt here.

> Could it be automated?

Not without parsing HTML to get the valid timestamps: 
<https://snapshot.debian.org/archive/debian/?year=2020&month=2>.

Also, this doesn't seem to be a supported service yet[0]:

  “This is an implementation for a possible snapshot.debian.org 
  service.
   It's not yet finished, it's more a prototype/proof of concept 
   to show
   and learn what we want and can provide.  So far it seems to 
   actually work.”

Still really cool,

T G-R

[0]: https://salsa.debian.org/snapshot-team/snapshot

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#39575: guix time-machine fails when a tarball was modified in-place

[Bengt Richter]

On +2020-02-15 21:01:36 +0100, Tobias Geerinckx-Rice via Bug reports for GNU Guix wrote:
> Jan, Simon,
> 
> Janneke 写道：
> > https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
> 
> This is a wonderful resource!  Thank you, Janneke (and Debian)!
> 
> zimoun 写道：
> > Cool!
> > But how do you determine the "date", i.e., this reference
> > '20190406T212022Z' ?
> 
> You'd take the timestamp immediately preceding your desired (Guix) commit's
> date, or something like that.  The fact that git commit dates aren't linear
> shouldn't hurt here.
> 
> > Could it be automated?
> 
> Not without parsing HTML to get the valid timestamps:
> <https://snapshot.debian.org/archive/debian/?year=2020&month=2>.
>

You may not need to parse the html fully if the part you need is
isolatable into delimited scopes that you can successively narrow.

For example, I while back I wanted a command I could type to get
the url of the latest linux kernel at kernel.org:

stable-kernel.scm -h 
--8<---------------cut here---------------start------------->8---
Usage: stable-kernel-scm [ -h ]
       -h for this message
          (without args):
       go to https://www.kernel.org/ to wget page,
       extract URL of latest stable release tarball
       and write that URL to stdout.
--8<---------------cut here---------------end--------------->8---
(oops, I see I din't use $0 in the usage text -- should be .scm, not -scm)

I offer it below [1], with the thought that you could probably
modify (not to mention improve :-) it to get the timestamps you want.
Especially if you could get them to make the narrow context unique enough
that it's delimiters can delimit it in one shot.

The page at kernel.org is apparently stable enough that this still works,
but YMMV until the snapshot page is similarly stable. (You could ask
them to make it easy :)

> Also, this doesn't seem to be a supported service yet[0]:
> 
>  “This is an implementation for a possible snapshot.debian.org  service.
>   It's not yet finished, it's more a prototype/proof of concept   to show
>   and learn what we want and can provide.  So far it seems to   actually
> work.”
> 
> Still really cool,
> 
> T G-R
> 
> [0]: https://salsa.debian.org/snapshot-team/snapshot

HTH or is useful some way.
-- 
Regards,
Bengt Richter

[1]
--8<---------------cut here---------------start------------->8---
#!/usr/bin/bash
exec guile -e main -s "$0" "$@"
!#
;;;; stable-kernel.scm
;;;; goes to https://www.kernel.org/ to wget page, then
;;;; extracts name of latest stable release tarball to stdout

;;;;
(define (usage)
  (format (current-error-port)
	  (string-join
	   '(
	    "Usage: stable-kernel-scm [ -h ]"
	    "       -h for this message"
	    "          (without args):"
            "       go to https://www.kernel.org/ to wget page,"
	    "       extract URL of latest stable release tarball"
	    "       and write that URL to stdout."
	    "")
	   "\n")))

(use-modules (ice-9 format))
(use-modules (ice-9 rdelim))
(use-modules (ice-9 popen))
(use-modules (ice-9 textual-ports))
(use-modules (ice-9 and-let-star))
(use-modules (ice-9 regex))

(define (extract-delimited str s-beg s-end)
  (and-let*
   ((ix-beg (string-contains str s-beg))
    (ix-post-beg (+ ix-beg (string-length s-beg)))
    (ix-end   (string-contains str s-end ix-post-beg)))
   (substring str ix-post-beg  ix-end)))

(define kernel-url "https://www.kernel.org/")

(define (get-kern-name)
  (let*((cmd-kern (string-append "wget -q -O - " kernel-url))
	(p-inp (open-input-pipe cmd-kern))
	(wgot-pinp-str (get-string-all p-inp))
	(extracted-table-releases
	 (extract-delimited wgot-pinp-str
			    "<table id=\"releases\">"
			    "</table>"))
	(extracted-stable-tarball-anchor
	 (extract-delimited extracted-table-releases
			    "<td>stable:</td>"
			    ">tarball<"))
	(extracted-stable-href
	 (extract-delimited extracted-stable-tarball-anchor
			    "<a href=\""
			    "\"")))
   (begin
    extracted-stable-href)))

(define (main args)
  (begin
    (set! args (cdr args)) ;; always dump callee arg
    (if (not (pair? args))
	(set! args '("-do-default") ))

    ;; simple opts...
    (cond
     ((string-prefix? "-h" (car args))
      (begin (usage)
	     (exit)))
     
     ((string-prefix? "-do-default" (car args))
      (let ((kern-name (get-kern-name)))
	(display kern-name)(newline)))

     (#t
      (begin
	(format (current-error-port) "Error: bad opt: '~a'\n" (car args))
	(usage)
	(exit))))))
--8<---------------cut here---------------end--------------->8---

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,

On Sat, 15 Feb 2020 at 21:01, Tobias Geerinckx-Rice <me@tobias.gr> wrote:

> Janneke 写道：
> > https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
>
> This is a wonderful resource!  Thank you, Janneke (and Debian)!
>
> zimoun 写道：
> > Cool!
> > But how do you determine the "date", i.e., this reference
> > '20190406T212022Z' ?
>
> You'd take the timestamp immediately preceding your desired (Guix)
> commit's date, or something like that.  The fact that git commit
> dates aren't linear shouldn't hurt here.

You assume that Debian packs packages as fast as Guix, I mean on the
same schedule which is a strong assumption IMHO.
For example, if it was the contrary and the "new" release of harfbuzz
2.4.0 were missing, then would Debian be helpful?


> Also, this doesn't seem to be a supported service yet[0]:
>
>   “This is an implementation for a possible snapshot.debian.org
>   service.
>    It's not yet finished, it's more a prototype/proof of concept
>    to show
>    and learn what we want and can provide.  So far it seems to
>    actually work.”
>
> Still really cool,

Yes, still cool! :-)


Thanks,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

On Mon, Feb 17, 2020 at 09:47:41AM +0100, zimoun wrote:
> Hi,
> 
> On Sat, 15 Feb 2020 at 21:01, Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> 
> > Janneke 写道：
> > > https://snapshot.debian.org/archive/debian/20190406T212022Z/pool/main/h/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2
> >
> > This is a wonderful resource!  Thank you, Janneke (and Debian)!
> >
> > zimoun 写道：
> > > Cool!
> > > But how do you determine the "date", i.e., this reference
> > > '20190406T212022Z' ?
> >
> > You'd take the timestamp immediately preceding your desired (Guix)
> > commit's date, or something like that.  The fact that git commit
> > dates aren't linear shouldn't hurt here.
> 
> You assume that Debian packs packages as fast as Guix, I mean on the
> same schedule which is a strong assumption IMHO.
> For example, if it was the contrary and the "new" release of harfbuzz
> 2.4.0 were missing, then would Debian be helpful?
> 
> 

We could first try
mirror://debian/pool/main/harfbuzz/harfbuzz_2.4.0.orig.tar.bz2

and then scrape https://snapshot.debian.org/package/harfbuzz/ for
2.4.0-1 and then parse the website for harfbuzz_2.4.0.orig.tar.bz2. Or
for just 'orig.tar'

> > Also, this doesn't seem to be a supported service yet[0]:
> >
> >   “This is an implementation for a possible snapshot.debian.org
> >   service.
> >    It's not yet finished, it's more a prototype/proof of concept
> >    to show
> >    and learn what we want and can provide.  So far it seems to
> >    actually work.”
> >
> > Still really cool,
> 
> Yes, still cool! :-)
> 
> 
> Thanks,
> simon
> 
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#39575: guix time-machine fails when a tarball was modified in-place

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

zimoun 写道：
> You assume that Debian packs packages as fast as Guix

Indeed I do!  :-D

Efraim's solution sounds reasonable.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

HI Jan,

On Fri, 14 Feb 2020 at 14:24, Jan Nieuwenhuizen <janneke@gnu.org> wrote:

This command

> >>  $ guix download -o /tmp/harfbuzz-old.tar.bz2 \
> >>  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch

now works.


However, this command

$ guix time-machine \
     --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- help

still fails for me with the message:

--8<---------------cut here---------------start------------->8---
[...]
building /gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv...
- 'check' phasebuilder for
`/gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv' failed
with exit code 1
build of /gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv failed
View build log at
'/var/log/guix/drvs/gg/lbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv.bz2'.
cannot build derivation
`/gnu/store/yqpxm07zm0mirrdvl2c4qvf8biyzg468-guix-56e95d54d.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/7z7p0m7abi246gzigw8as2q3w33k1n31-profile.drv': 1
dependencies couldn't be built
guix time-machine: error: build of
`/gnu/store/7z7p0m7abi246gzigw8as2q3w33k1n31-profile.drv' failed
--8<---------------cut here---------------end--------------->8---

The log /var/log/guix/drvs/gg/lbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv.bz2
is not meaningful for me... but I can report it here.


> that i'm trying now, and for now it looks fine (lots of stuff to build,
> i'll report success or failure when it's done).

Well, is it a success or a failure for you?


Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Jan Nieuwenhuizen]

zimoun writes:

Hi Simon,

> On Fri, 14 Feb 2020 at 14:24, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>
> This command
>
>> >>  $ guix download -o /tmp/harfbuzz-old.tar.bz2 \
>> >>  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>
> now works.
>
>
> However, this command
>
> $ guix time-machine \
>      --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- help
>
> still fails for me with the message:
>
> [...]
> building /gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv...
> - 'check' phasebuilder for
> `/gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv' failed
> with exit code 1
> build of /gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv failed
> View build log at
> '/var/log/guix/drvs/gg/lbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv.bz2'.
> cannot build derivation
> `/gnu/store/yqpxm07zm0mirrdvl2c4qvf8biyzg468-guix-56e95d54d.drv': 1
> dependencies couldn't be built
> cannot build derivation
> `/gnu/store/7z7p0m7abi246gzigw8as2q3w33k1n31-profile.drv': 1
> dependencies couldn't be built
> guix time-machine: error: build of
> `/gnu/store/7z7p0m7abi246gzigw8as2q3w33k1n31-profile.drv' failed
>
> The log /var/log/guix/drvs/gg/lbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv.bz2
> is not meaningful for me... but I can report it here.
>
>
>> that i'm trying now, and for now it looks fine (lots of stuff to build,
>> i'll report success or failure when it's done).
>
> Well, is it a success or a failure for you?

For me, pythohn-minimal fails to build

    build-started /gnu/store/s0lw23myd3hvpw28sffkhz8b30x1hcz0-python-minimal-3.7.3.drv - x86_64-linux /var/log/guix/drvs/s0//lw23myd3hvpw28sffkhz8b30x1hcz0-python-minimal-3.7.3.drv.bz2 20827

    ======================================================================
    FAIL: test_register_chain (test.test_faulthandler.FaultHandlerTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_faulthandler.py", line 724, in test_register_chain
        self.check_register(chain=True)
      File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_faulthandler.py", line 702, in check_register
        self.assertEqual(exitcode, 0)
    AssertionError: -11 != 0

    ----------------------------------------------------------------------

    Ran 42 tests in 18.782s

    FAILED (failures=1, skipped=4)
    1 test failed again:
        test_faulthandler

    == Tests result: FAILURE then FAILURE ==

    382 tests OK.

    1 test failed:
        test_faulthandler

Not sure what to do here.  Could this be a (harmless) coincident?

Greetings
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,

Some follow ups.


For me, now, the command

> > $ guix time-machine \
> >      --commit=56e95d54d209c2428f970d65d9b27ae4168449ad -- help

does not fail anymore at the Guile step:

--8<---------------cut here---------------start------------->8---
  building /gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv...
  - 'check' phasebuilder for
  `/gnu/store/gglbrs8j0iq8ygh55inwfvpwb5z2x254-guile-2.2.4.drv' failed
--8<---------------cut here---------------end--------------->8---

because it was about "FAIL: test-out-of-memory".


Well, now, I am failing at the Python 3.7.3 step too:

--8<---------------cut here---------------start------------->8---
  /gnu/store/s0lw23myd3hvpw28sffkhz8b30x1hcz0-python-minimal-3.7.3.drv failed
--8<---------------cut here---------------end--------------->8---


However, the python error seems about TLS:

--8<---------------cut here---------------start------------->8---
test.test_asyncio.test_windows_utils (unittest.loader.ModuleSkipped)
... test test_asyncio failed
skipped 'Windows only'

======================================================================
ERROR: test_start_tls_server_1
(test.test_asyncio.test_sslproto.SelectorStartTLSTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_asyncio/test_sslproto.py",
line 507, in test_start_tls_server_1
    self.loop.run_until_complete(run_main())
  File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/asyncio/base_events.py",
line 584, in run_until_complete
    return future.result()
  File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_asyncio/test_sslproto.py",
line 502, in run_main
    loop=self.loop, timeout=self.TIMEOUT)
  File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/asyncio/tasks.py",
line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
--8<---------------cut here---------------end--------------->8---


> Not sure what to do here.  Could this be a (harmless) coincident?

Me neither.


Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Well, now, I am failing at the Python 3.7.3 step too:
>
>   /gnu/store/s0lw23myd3hvpw28sffkhz8b30x1hcz0-python-minimal-3.7.3.drv failed
>
>
>
> However, the python error seems about TLS:
>
> test.test_asyncio.test_windows_utils (unittest.loader.ModuleSkipped)
> ... test test_asyncio failed
> skipped 'Windows only'
>
> ======================================================================
> ERROR: test_start_tls_server_1
> (test.test_asyncio.test_sslproto.SelectorStartTLSTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_asyncio/test_sslproto.py",
> line 507, in test_start_tls_server_1
>     self.loop.run_until_complete(run_main())
>   File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/asyncio/base_events.py",
> line 584, in run_until_complete
>     return future.result()
>   File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/test/test_asyncio/test_sslproto.py",
> line 502, in run_main
>     loop=self.loop, timeout=self.TIMEOUT)
>   File "/tmp/guix-build-python-minimal-3.7.3.drv-0/Python-3.7.3/Lib/asyncio/tasks.py",
> line 423, in wait_for
>     raise futures.TimeoutError()
> concurrent.futures._base.TimeoutError

It seems to be timing-sensitive, is it deterministic?

One lesson here is that we should keep substitutes for a longer amount
of time—we actually have a lot of storage space on berlin so we should
investigate what happened.  (The NixOS folks currently keep substitutes
forever but they found it’s starting to be expensive…)

Thanks,
Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

zimoun 写道：
> Maybe I miss a point, but the file we need is the old one, not 
> the new
> one, i.e., the one with the expected hash
> 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch. And I 
> should do

~ λ guix download 
https://www.tobias.gr/guix/harfbuzz-2.4.0.tar.bz2
Starting download of /tmp/guix-file.JSWxOl
From https://www.tobias.gr/guix/harfbuzz-2.4.0.tar.bz2...
 ...4.0.tar.bz2  17.1MiB 
 6.5MiB/s 00:03 [##################] 100.0%
/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2
1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch

Enjoy,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> zimoun 写道：
>> Maybe I miss a point, but the file we need is the old one, not the
>> new
>> one, i.e., the one with the expected hash
>> 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch. And I should
>> do
>
> ~ λ guix download https://www.tobias.gr/guix/harfbuzz-2.4.0.tar.bz2

Thanks, you saved us!

Now we have it here:

  https://ci.guix.gnu.org/file/harfbuzz-2.4.0.tar.bz2/sha256/1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch

I’ve also registered a GC root.

Anyway, everything will be so much better when SWH archives tarballs!

Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,

On Fri, 14 Feb 2020 at 14:14, Ludovic Courtès <ludo@gnu.org> wrote:
> Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> > ~ λ guix download https://www.tobias.gr/guix/harfbuzz-2.4.0.tar.bz2
>
> Thanks, you saved us!

Thank you! :-)


> Anyway, everything will be so much better when SWH archives tarballs!

The future will be better. :-)
Even some details need to be discussed: frequency of source.json
generation, frequency of the SWH crawler ingest it, etc.

How could all the archives living in ci.guix.gnu.org be sent to SWH?
Because IMHO the of Berlin (or other) is to archive the world but to
build it. :-)


Cheers,
simon

bug#39575: guix time-machine fails when a tarball was modified in-place

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> building /gnu/store/cjim33x0q1bv1ppkv3qijvr1pvsn4y0q-harfbuzz-2.4.0.tar.bz2.drv...
>> downloading from https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-2.4.0.tar.bz2...
>> |offloading build of /gnu/store/6fgg1irkcvqyb4f9f8n0nzi5gknyqhfn-gcc-mesboot1-4.7.4.drv to 'kluit.dezyne.org'
>> - 'build' phasesha256 hash mismatch for /gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2:
>>   expected hash: 1mpah6kwqid1kxsj4rwqsniivqbrx231j65v51yncx6s0dch0dch
>>   actual hash:   0vrkvdlmihdg62a4c6h5kx27khc33xmb95l50zgnwnavvpwyyw5l
>> hash mismatch for store item '/gnu/store/b4cdp9sp44848348lrpzbfafhmjqf8nr-harfbuzz-2.4.0.tar.bz2'
>
> The problem here is really that we fall back to content-addressed
> mirrors instead of using them directly:
>
>   https://issues.guix.gnu.org/issue/28659

Wait, what happened here; you finally proposed a patch two years ago and
nothing happened/we all forgot to follow up?

I cannot determine if possibly we were hoping to "wait" for the guile
build daemon?

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:

[...]

>> The problem here is really that we fall back to content-addressed
>> mirrors instead of using them directly:
>>
>>   https://issues.guix.gnu.org/issue/28659
>
> Wait, what happened here; you finally proposed a patch two years ago and
> nothing happened/we all forgot to follow up?

I think we forgot, indeed.

One thing I don’t quite like about the patch is the fact that ‘guix
substitutes’ connects to the daemon in ‘content-addressed-item?’.

Also, one could argue that we’d steer users towards downloading from our
server, which could be a privacy concern (probably not a strong argument
since one can easily change the substitute URLs.)

Thoughts?

Ludo’.

bug#28659: bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi,

On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

> Also, one could argue that we’d steer users towards downloading from our
> server, which could be a privacy concern (probably not a strong argument
> since one can easily change the substitute URLs.)

I am not following the privacy concern.
What do you mean?

Cheers,
simon

bug#28659: bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Also, one could argue that we’d steer users towards downloading from our
>> server, which could be a privacy concern (probably not a strong argument
>> since one can easily change the substitute URLs.)
>
> I am not following the privacy concern.
> What do you mean?

I mean that by default, someone who’s disabled substitutes (presumably
out of security or privacy concerns) would find themself downloading
source code from ci.guix.gnu.org instead of various upstream sites.

Ludo’.

bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

Hi Ludo,

On Sun, 16 Feb 2020 at 11:59, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> > On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

> >> Also, one could argue that we’d steer users towards downloading from our
> >> server, which could be a privacy concern (probably not a strong argument
> >> since one can easily change the substitute URLs.)
> >
> > I am not following the privacy concern.
> > What do you mean?
>
> I mean that by default, someone who’s disabled substitutes (presumably
> out of security or privacy concerns) would find themself downloading
> source code from ci.guix.gnu.org instead of various upstream sites.

I do not see the difference between mirroring and traveling back in
time with missing upstream sources.
And because it is content-addressed, it seems even more secure than
downloading from a upstream URL, IMHO.
If one trusts Guix, then an attacker needs to corrupt in the same time
the Guix history and Berlin (and/or any other farm).
If one does not trust Guix, why does they use the recipe coming from
Guix? To be precise, this person has to check all the recipes of all
the dependencies.

Well, I do not see a security concern because we are talking about
serving the sources.
It is another story when the substitutes serve the results of the
build (binaries); because one does not have any strong guarantee that
the substitute serves the expected binaries.

By privacy concern, do you mean that Guix could collect who downloads
what; in a central fashion? Which is not the case when one downloads
from several distributed upstream sources. Right?
Well, I am not convinced because the case of missing upstream source
is rare. And it is easy to protect against such collecting data
process.
In paranoid mode, traveling back in time is becoming difficult because
of the reliability of the sources; I mean if the sources were
reliable, SWH would not exist. ;-) The solution should be an IPFS /
GNUnet / full distributed archive... which is not ready... yet! :-)


Well, maybe for the TODO list of the time-machine: add an option to
allow substitutes *only* for the sources (substitutes meaning
ci.guix.gnu.org and/or SWH). If this option does not exist yet. ;-)


Cheers,
simon

bug#28659: bug#39575: guix time-machine fails when a tarball was modified in-place

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sun, 16 Feb 2020 at 11:59, Ludovic Courtès <ludo@gnu.org> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>> > On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> >> Also, one could argue that we’d steer users towards downloading from our
>> >> server, which could be a privacy concern (probably not a strong argument
>> >> since one can easily change the substitute URLs.)
>> >
>> > I am not following the privacy concern.
>> > What do you mean?
>>
>> I mean that by default, someone who’s disabled substitutes (presumably
>> out of security or privacy concerns) would find themself downloading
>> source code from ci.guix.gnu.org instead of various upstream sites.

[...]

> By privacy concern, do you mean that Guix could collect who downloads
> what; in a central fashion? Which is not the case when one downloads
> from several distributed upstream sources. Right?

Exactly.  But like I wrote above, I don’t think it’s a strong argument.

What remains is the issue with ‘content-addressed-item?’, then.

Ludo’.

bug#28659: bug#39575: guix time-machine fails when a tarball was modified in-place

[zimoun]

On Mon, 17 Feb 2020 at 15:40, Ludovic Courtès <ludo@gnu.org> wrote:

> Exactly.  But like I wrote above, I don’t think it’s a strong argument.

I agree and the big picture depends on the audience.
Scientific communities would be fine with centralized archives such as
SWH. And only centralized archives IMHO can provide a reliable "long
term" support which is the point for that communities. (Quote because
not clearly defined what it is. :-))
Other communities would prefer distributed archive such as IPFS or
GNUnet but 1. it still needs some work and 2. the "long term" is not
guarantee by nature, IMHO. But it is probably not an issue for that
communities.


> What remains is the issue with ‘content-addressed-item?’, then.

I agree.
The bridge with SWH is in good shape, IMHO.
And the pending IPFS patch would deserve more love. :-) Maybe soon...



Cheers,
simon

bug#28659: Content-addressed mirror is not used upon invalid hash

[zimoun]

Hi,

On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

> One thing I don’t quite like about the patch is the fact that ‘guix
> substitutes’ connects to the daemon in ‘content-addressed-item?’.

What is the status of this patch [1] following the recent discussion about
tar “disarchive” and SWH?

Related:
 - http://issues.guix.gnu.org/issue/39575
 - http://issues.guix.gnu.org/42162
 - https://git.ngyro.com/disarchive/
 
All the best,
simon

[1] http://issues.guix.gnu.org/issue/28659#26

bug#28659: Content-addressed mirror is not used upon invalid hash

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> One thing I don’t quite like about the patch is the fact that ‘guix
>> substitutes’ connects to the daemon in ‘content-addressed-item?’.
>
> What is the status of this patch [1] following the recent discussion about
> tar “disarchive” and SWH?
>
> Related:
>  - http://issues.guix.gnu.org/issue/39575
>  - http://issues.guix.gnu.org/42162
>  - https://git.ngyro.com/disarchive/

Thanks for the reminder.  I don’t think Timothy’s work changes anything
wrt. to this issue: it would still need to be addressed.

Ludo’.

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Jan Nieuwenhuizen]

Hi!

As reported by laertus on irc[0]: guix pull on 0.13 without substitutes fails

      guix pull

    Starting download of /tmp/guix-file.3r6cH0
    From https://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz...
     ….tar.gz                                   5.7MiB/s 00:02 | 13.6MiB transferred
    unpacking '/gnu/store/sginfwnrcfqn1far31gmzlaffd8xlxyy-guix-latest.tar.gz'...

    Starting download of /gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz
    From https://github.com/libgit2/libgit2/archive/v0.25.1.tar.gz...
    following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.25.1'...
     v0.25.1                                     6.1MiB/s 00:01 | 4.1MiB transferred
    output path `/gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz' should have sha256 hash `1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s', instead has `0ywcxw1mwd56c8qc14hbx31bf198gxck3nja3laxyglv7l57qp26'
    cannot build derivation `/gnu/store/z1ky970mnamnbairnpyxxb72qnc485zq-libgit2-0.25.1.drv': 1 dependencies couldn't be built
    cannot build derivation `/gnu/store/rl7ms8rmbywvydy4qf656g1sdfxafb7r-guile-git-0.0-2.06f9fc3.drv': 1 dependencies couldn't be built
    guix pull: error: build failed: build of `/gnu/store/rl7ms8rmbywvydy4qf656g1sdfxafb7r-guile-git-0.0-2.06f9fc3.drv' failed

because the libgit2-0.25.1 content hash does not check out.

I verified this on version-0.13.  The same goes for 0.26.0 on master

    $ guix build -S libgit2 --no-substitutes
    The following derivations will be built:
       /gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv
       /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv
    @ build-started /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv - x86_64-linux /var/log/guix/drvs/mg//h4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv.bz2

    Starting download of /gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz
    From https://github.com/libgit2/libgit2/archive/v0.26.0.tar.gz...
    following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.26.0'...
     v0.26.0  4.5MiB                    3.1MiB/s 00:01 [####################] 100.0%
    sha256 hash mismatch for output path `/gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz'
      expected: 1fdk9yhwvl1w1z71ykzcvgh4nsf8scxcbclz5anh98zpplmhmisa
      actual:   1b3figbhp5l83vd37vq6j2narrq4yl9pfw6mw0px0dzb1hz3jqka
    @ build-failed /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv - 1 sha256 hash mismatch for output path `/gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz'
      expected: 1fdk9yhwvl1w1z71ykzcvgh4nsf8scxcbclz5anh98zpplmhmisa
      actual:   1b3figbhp5l83vd37vq6j2narrq4yl9pfw6mw0px0dzb1hz3jqka
    cannot build derivation `/gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv': 1 dependencies couldn't be built
    guix build: error: build failed: build of `/gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv' failed

I found no apparent difference in the content

    -r--r--r-- 1 janneke janneke  4252130 Oct  1 09:08 c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz
    -rw-r--r-- 1 janneke janneke  4252139 Oct  1 09:09 NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz
    -rw-r--r-- 1 janneke janneke 16363520 Oct  1 09:14 c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar
    -rw-r--r-- 1 janneke janneke 16363520 Oct  1 09:14 NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar

but there's this difference between the tar balls...

    12:13:57 janneke@dundal:~/src/guix-0.13 
    $ cmp -l c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar
    13122049   0 157
    13122050   0 162
    13122051   0 151
    13122052   0 147
    13122053   0 151
    13122054   0 156
    13122055   0  57
    13122490  57   0
    13122491 157   0
    13122492 162   0
    13122493 151   0
    13122494 147   0
    13122495 151   0
    13122496 156   0
    13270529   0 157
    13270530   0 162
    13270531   0 151
    13270532   0 147
    13270533   0 151
    13270534   0 156
    13270535   0  57
    13270972  57   0
    13270973 157   0
    13270974 162   0
    13270975 151   0
    13270976 147   0
    13270977 151   0
    13270978 156   0
    13294081   0 157
    13294082   0 162
    13294083   0 151
    13294084   0 147
    13294085   0 151
    13294086   0 156
    13294087   0  57
    13294519  57   0
    13294520 157   0
    13294521 162   0
    13294522 151   0
    13294523 147   0
    13294524 151   0
    13294525 156   0

janneke

[0] https://gnunet.org/bot/log/guix/2017-10-01#T1517584

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Ludovic Courtès]

Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> As reported by laertus on irc[0]: guix pull on 0.13 without substitutes fails

I just checked and we do have substitutes, but I understand it doesn’t
help here.

>       guix pull
>
>     Starting download of /tmp/guix-file.3r6cH0
>     From https://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz...
>      ….tar.gz                                   5.7MiB/s 00:02 | 13.6MiB transferred
>     unpacking '/gnu/store/sginfwnrcfqn1far31gmzlaffd8xlxyy-guix-latest.tar.gz'...
>
>     Starting download of /gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz
>     From https://github.com/libgit2/libgit2/archive/v0.25.1.tar.gz...
>     following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.25.1'...
>      v0.25.1                                     6.1MiB/s 00:01 | 4.1MiB transferred
>     output path `/gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz' should have sha256 hash `1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s', instead has `0ywcxw1mwd56c8qc14hbx31bf198gxck3nja3laxyglv7l57qp26'

What’s sad here is that we do have the right tarball at:

  https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

The problem is that the hash check is performed by guix-daemon itself,
not by “guix perform-download”.  So when guix-daemon diagnoses a hash
mismatch, it’s too late and we cannot try again and use the
content-addressed mirror.

A crude but helpful fix would be to have perform-download compute the
hash by itself and act accordingly.  It’s crude because that means that
we’d be computing the hash twice: once in ‘guix perform-download’ and a
second time in guix-daemon.  For archives below ~20 MiB it’s probably OK
though.

Thoughts?

In the future, with the daemon written in Guile, it’s one area where we
could achieve better integration and coordination among the various
pieces.

Ludo’.

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Mon, Oct 02, 2017 at 05:09:39PM +0200, Ludovic Courtès wrote:
> What’s sad here is that we do have the right tarball at:
> 
>   https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

It seems to me that there are several reasons someone may choose not to
use substitutes. Some of those reasons (reproducibility and security
concerns) are obviated for fixed-output derivations like upstream
sources, and I think it would be fine to still use substitutes for these
derivations.

But the motivations of privacy, self-sufficiency, etc are not addressed
by that idea.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Oct 02, 2017 at 05:09:39PM +0200, Ludovic Courtès wrote:
>> What’s sad here is that we do have the right tarball at:
>> 
>>   https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

Just to be clear: this URL is not that of a substitute, but that of a
content-addressed file (corresponding to the output of a fixed-output
derivation.)

> It seems to me that there are several reasons someone may choose not to
> use substitutes. Some of those reasons (reproducibility and security
> concerns) are obviated for fixed-output derivations like upstream
> sources, and I think it would be fine to still use substitutes for these
> derivations.
>
> But the motivations of privacy, self-sufficiency, etc are not addressed
> by that idea.

Right.  Jan suggested checking the content-addressed mirrors *before*
the real upstream address.  That would address the problem of upstream
sources modified in-place, but at the cost of privacy/self-sufficiency
as you note.  (Though it’s not really making “privacy” any worse in this
case: it’s gnu.org vs. github.com.)

Perhaps we should make content-addressed mirrors configurable in a way
that’s orthogonal to derivations, something similar in spirit to
--substitute-urls?  The difficulty is that content-addressed mirrors are
not just URLs; see (guix download).

Thoughts?

Ludo’.

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:
> Right.  Jan suggested checking the content-addressed mirrors *before*
> the real upstream address.  That would address the problem of upstream
> sources modified in-place, but at the cost of privacy/self-sufficiency
> as you note.  (Though it’s not really making “privacy” any worse in this
> case: it’s gnu.org vs. github.com.)

Yeah, I don't personally think there is a privacy issue with fetching
sources from our mirrors at gnu.org, or other domains we control.

> Perhaps we should make content-addressed mirrors configurable in a way
> that’s orthogonal to derivations, something similar in spirit to
> --substitute-urls?  The difficulty is that content-addressed mirrors are
> not just URLs; see (guix download).
>
> Thoughts?

I do think we should make it so that users don't suffer from unreliable
upstream sources when we know the sources are available on our servers
(or the Nix mirror), even with --no-substitutes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 3524 bytes --]

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:
>> Right.  Jan suggested checking the content-addressed mirrors *before*
>> the real upstream address.  That would address the problem of upstream
>> sources modified in-place, but at the cost of privacy/self-sufficiency
>> as you note.  (Though it’s not really making “privacy” any worse in this
>> case: it’s gnu.org vs. github.com.)
>
> Yeah, I don't personally think there is a privacy issue with fetching
> sources from our mirrors at gnu.org, or other domains we control.
>
>> Perhaps we should make content-addressed mirrors configurable in a way
>> that’s orthogonal to derivations, something similar in spirit to
>> --substitute-urls?  The difficulty is that content-addressed mirrors are
>> not just URLs; see (guix download).
>>
>> Thoughts?
>
> I do think we should make it so that users don't suffer from unreliable
> upstream sources when we know the sources are available on our servers
> (or the Nix mirror), even with --no-substitutes.

The more I think about it, the more I’m inclined to simply move
content-addressed mirrors to the front of the list.  This means that
users, in practice, would be fetching all the source from
mirror.hydra.gnu.org.

The main issue is making it configurable.  Currently the
content-addressed mirror configuration for regular files in (guix
download) looks like this:

--8<---------------cut here---------------start------------->8---
(define %content-addressed-mirrors
  ;; List of content-addressed mirrors.  Each mirror is represented as a
  ;; procedure that takes a file name, an algorithm (symbol) and a hash
  ;; (bytevector), and returns a URL or #f.
  ;; Note: Avoid 'https' to mitigate <http://bugs.gnu.org/22774>.
  ;; TODO: Add more.
  '(list (lambda (file algo hash)
           ;; Files served by 'guix publish' are accessible under a single
           ;; hash algorithm.
           (string-append "http://mirror.hydra.gnu.org/file/"
                          file "/" (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))
         (lambda (file algo hash)
           ;; 'tarballs.nixos.org' supports several algorithms.
           (string-append "http://tarballs.nixos.org/"
                          (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))))
--8<---------------cut here---------------end--------------->8---

That for VCS checkouts in (guix build download-nar) looks like this:

--8<---------------cut here---------------start------------->8---
(define (urls-for-item item)
  "Return the fallback nar URL for ITEM--e.g.,
\"/gnu/store/cabbag3…-foo-1.2-checkout\"."
  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK
  ;; though.
  ;; TODO: Use HTTPS?  The downside is the extra dependency.
  (let ((bases '("http://mirror.hydra.gnu.org/guix"
                 "http://berlin.guixsd.org"))
        (item  (basename item)))
    (append (map (cut string-append <> "/nar/gzip/" item) bases)
            (map (cut string-append <> "/nar/" item) bases))))
--8<---------------cut here---------------end--------------->8---

The latter could be expressed by a command-line flag.  In fact it’s the
same as --substitute-urls.

(Time passes…)

Thinking more about it, why not simply always enable substitutes for
fixed-output derivations, like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 813 bytes --]

diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d68e8b2bc..03a8f5080 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()
 
     /* We are first going to try to create the invalid output paths
        through substitutes.  If that doesn't work, we'll build
-       them. */
-    if (settings.useSubstitutes && substitutesAllowed(drv))
+       them.  Always enable substitutes for fixed-output derivations to
+       protect against disappearing files and in-place modifications on
+       upstream sites.  */
+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))
         foreach (PathSet::iterator, i, invalidOutputs)
             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));
 

[-- Attachment #3: Type: text/plain, Size: 1081 bytes --]


This solves all our problems and makes download-nar.scm useless.

As an added bonus, it provides a improves the UI since we now always
see:

--8<---------------cut here---------------start------------->8---
0.1 MB will be downloaded:
   /gnu/store/plx9848n6waj6zghn3d54ybx8ihcn23k-guile-git-0.0-4.951a32c-checkout
--8<---------------cut here---------------end--------------->8---

… instead of:

--8<---------------cut here---------------start------------->8---
The following derivation will be built:
   /gnu/store/y86rlb6pdm35im7q02y6479ca84zwylz-guile-git-000.0-4.951a32c-checkout.drv
--8<---------------cut here---------------end--------------->8---

The downside is that it still requires one to authorize the server’s
key, although it’s in theory unnecessary since it’s content addressed.
I’m not sure how to solve that because ‘guix substitute’ doesn’t know
that it’s substituting a fixed-output derivation.  I suppose we’d need
to modify the “protocol” between guix-daemon and ‘guix substitute’.

Thoughts?

Ludo’.

bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Thinking more about it, why not simply always enable substitutes for
> fixed-output derivations, like this:
>
> diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
> index d68e8b2bc..03a8f5080 100644
> --- a/nix/libstore/build.cc
> +++ b/nix/libstore/build.cc
> @@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()
>  
>      /* We are first going to try to create the invalid output paths
>         through substitutes.  If that doesn't work, we'll build
> -       them. */
> -    if (settings.useSubstitutes && substitutesAllowed(drv))
> +       them.  Always enable substitutes for fixed-output derivations to
> +       protect against disappearing files and in-place modifications on
> +       upstream sites.  */
> +    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))
>          foreach (PathSet::iterator, i, invalidOutputs)
>              addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));

[...]

> The downside is that it still requires one to authorize the server’s
> key, although it’s in theory unnecessary since it’s content addressed.
> I’m not sure how to solve that because ‘guix substitute’ doesn’t know
> that it’s substituting a fixed-output derivation.  I suppose we’d need
> to modify the “protocol” between guix-daemon and ‘guix substitute’.

I looked at how to address this by having ‘guix substitute’
automatically determine whether it’s being asked for a content-addressed
item or not.  The guts of it is this procedure:

  (define* (content-addressed-item? item hash
                                    #:key (hash-algo 'sha256))
    "Return true if ITEM, a store file name, is definitely a content-addressed
  item (result of a fixed-output derivation) with the given HASH of type
  HASH-ALGO, false otherwise.

  Note: This procedure is useful when the deriver of ITEM is unknown.  In other
  cases, the recommended approach is to check 'fixed-output-derivation?' on the
  deriver."
    ;; XXX: This returns #f for "text" items produced by 'add-text-to-store'.
    ;; There's not much we can do because the file name for these is a function
    ;; of their content.
    (let ((name (store-path-package-name item)))
      (or (string=? item (fixed-output-path name hash #:recursive? #f
                                            #:hash-algo hash-algo))
          (string=? item (fixed-output-path name hash #:recursive? #t
                                            #:hash-algo hash-algo)))))

It works as expected for the result of “recursive fixed-output
derivations”—i.e., fixed-output derivations that produce a directory,
such as VCS checkouts.

However it doesn’t work for fixed-output derivations that produce a flat
file, such as origins with the ‘url-fetch’ method.  The reason is
because in the case of non-recursive derivations, the store file name is
computed as a function of the file hash, not as a function of the nar
hash, whereas narinfos only contains the nar hash (the thing that ‘guix
hash -r’ computes.)

So I think we have to communicate more info from the daemon to ‘guix
substitute’.

Ludo’.

bug#28659: Always enable substitutes for fixed-output derivations

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> So I think we have to communicate more info from the daemon to ‘guix
> substitute’.

The attached patch addresses that by simply calling out to the daemon to
determine whether we’re dealing with a content-addressed item.

To summarize, the new behavior is that substitutes are always enabled
for fixed-output derivations.  That way, people willing to build
everything from source can still use ‘--no-substitutes’ and yet be able
to retrieve source code without being penalized compared to someone
enabling substitutes wholesale.

Of course, when substitutes are missing, we fall back to regular
downloads or VCS checkouts.  It is also still possible to choose where
substitutes are downloaded from, using ‘--substitute-urls’, or even to
pass an empty list of URLs.

Feedback welcome!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-substitute-Always-allow-substitutes-for-fixed-output.patch --]
[-- Type: text/x-patch, Size: 5940 bytes --]

From aab42bcb212698bc1f61beb9f321ffbd751f36f5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 15 Dec 2017 09:57:04 +0100
Subject: [PATCH 1/2] substitute: Always allow substitutes for fixed-output
 derivation results.

Fixes <https://bugs.gnu.org/28659>.

* guix/scripts/substitute.scm (content-addressed-item?): New procedure.
(valid-narinfo?): Use it.
* nix/libstore/build.cc (DerivationGoal::haveDerivation): Always make a
substitution goal when 'fixedOutput' is true.
* tests/substitute.scm ("query unsigned narinfo for content-addressed
item"): New test.
---
 guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++++++++-
 nix/libstore/build.cc       |  6 ++++--
 tests/substitute.scm        | 24 +++++++++++++++++++++++-
 3 files changed, 57 insertions(+), 4 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 2fd2bf810..670a9b4dd 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -25,6 +25,9 @@
   #:use-module (guix config)
   #:use-module (guix records)
   #:use-module ((guix serialization) #:select (restore-file))
+  #:use-module ((guix derivations)
+                #:select (read-derivation-from-file
+                          fixed-output-derivation?))
   #:use-module (guix hash)
   #:use-module (guix base32)
   #:use-module (guix base64)
@@ -406,10 +409,36 @@ No authentication and authorization checks are performed here!"
        (let ((above-signature (string-take contents index)))
          (sha256 (string->utf8 above-signature)))))))
 
+(define* (content-addressed-item? item)
+  "Return true if ITEM is content-addressed---i.e., if ITEM is the result of a
+fixed-output derivation."
+  (guard (c ((nix-connection-error? c)
+             ;; We failed to connect, maybe because we have the wrong
+             ;; GUIX_DAEMON_SOCKET?  Let's conservatively assume that
+             ;; nothing's content-addressed.
+             #f))
+    (with-store store
+      (match (valid-derivers store item)
+        (()
+         ;; If there are no valid derivers it's most likely because ITEM is a
+         ;; source (added with 'add-to-store' or similar).  Nevertheless,
+         ;; since we can't be certain, return #f.
+         #f)
+        ((drv . _)
+         (fixed-output-derivation?
+          (read-derivation-from-file drv)))))))
+
 (define* (valid-narinfo? narinfo #:optional (acl (current-acl))
                          #:key verbose?)
-  "Return #t if NARINFO's signature is not valid."
+  "Return #t if NARINFO is \"valid\"---signed by an authorized key, or
+designating a content-addressed item."
   (or %allow-unauthenticated-substitutes?
+
+      ;; If NARINFO designates a content-addressed item, there's no point
+      ;; authenticating it.  Don't explicitly check 'narinfo-hash' for
+      ;; integrity: this will be done by the daemon once we've downloaded it.
+      (content-addressed-item? (narinfo-path narinfo))
+
       (let ((hash      (narinfo-sha256 narinfo))
             (signature (narinfo-signature narinfo))
             (uri       (uri->string (narinfo-uri narinfo))))
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d68e8b2bc..03a8f5080 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()
 
     /* We are first going to try to create the invalid output paths
        through substitutes.  If that doesn't work, we'll build
-       them. */
-    if (settings.useSubstitutes && substitutesAllowed(drv))
+       them.  Always enable substitutes for fixed-output derivations to
+       protect against disappearing files and in-place modifications on
+       upstream sites.  */
+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))
         foreach (PathSet::iterator, i, invalidOutputs)
             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));
 
diff --git a/tests/substitute.scm b/tests/substitute.scm
index 0ad624795..03579b9f1 100644
--- a/tests/substitute.scm
+++ b/tests/substitute.scm
@@ -21,15 +21,17 @@
   #:use-module (guix scripts substitute)
   #:use-module (guix base64)
   #:use-module (guix hash)
+  #:use-module (guix derivations)
   #:use-module (guix serialization)
   #:use-module (guix pk-crypto)
   #:use-module (guix pki)
   #:use-module (guix config)
   #:use-module (guix base32)
-  #:use-module ((guix store) #:select (%store-prefix))
+  #:use-module ((guix store) #:select (%store-prefix with-store))
   #:use-module ((guix ui) #:select (guix-warning-port))
   #:use-module ((guix build utils)
                 #:select (mkdir-p delete-file-recursively))
+  #:use-module (guix tests)
   #:use-module (guix tests http)
   #:use-module (rnrs bytevectors)
   #:use-module (rnrs io ports)
@@ -241,6 +243,26 @@ a file for NARINFO."
            (lambda ()
              (guix-substitute "--query"))))))))
 
+(test-assert "query unsigned narinfo for content-addressed item"
+  (with-store store
+    (let* ((hash (sha256 (random-bytevector 128)))
+           (drv  (derivation store "content-addressed"
+                             "builtin:download" '()
+                             #:hash-algo 'sha256 #:hash hash)))
+      (define output
+        (with-output-to-string
+          (lambda ()
+            (with-derivation-narinfo drv (sha256 => hash)
+              (with-input-from-string (string-append "have "
+                                                     (derivation->output-path drv))
+                (lambda ()
+                  (set! (@@ (guix scripts substitute)
+                            %allow-unauthenticated-substitutes?)
+                    #f)
+                  (guix-substitute "--query")))))))
+
+      (string=? (string-trim-both output) (derivation->output-path drv)))))
+
 (test-quit "substitute, no signature"
     "no valid substitute"
   (with-narinfo %narinfo
-- 
2.15.1


[-- Attachment #3: 0002-Revert-download-Download-a-nar-when-a-VCS-checkout-f.patch --]
[-- Type: text/x-patch, Size: 14302 bytes --]

From 9bcf90b99a79f9f3e126cde5fe1cf51b0dfa58aa Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 15 Dec 2017 10:03:39 +0100
Subject: [PATCH 2/2] Revert "download: Download a nar when a VCS checkout
 fails."

This reverts commit 37ce440dcffa9ff4f5401bacbc9619bd8ea561c1, which is
useless now that substitutes are always enabled for content-addressed
items.
---
 Makefile.am                 |   1 -
 guix/build/download-nar.scm | 125 --------------------------------------------
 guix/cvs-download.scm       |  38 ++++----------
 guix/git-download.scm       |  37 +++----------
 guix/hg-download.scm        |  36 ++++---------
 5 files changed, 26 insertions(+), 211 deletions(-)
 delete mode 100644 guix/build/download-nar.scm

diff --git a/Makefile.am b/Makefile.am
index 85b9ab36d..d2660b0a7 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -110,7 +110,6 @@ MODULES =					\
   guix/ui.scm					\
   guix/build/ant-build-system.scm		\
   guix/build/download.scm			\
-  guix/build/download-nar.scm			\
   guix/build/cargo-build-system.scm		\
   guix/build/cmake-build-system.scm		\
   guix/build/dub-build-system.scm		\
diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm
deleted file mode 100644
index 13f01fb1e..000000000
--- a/guix/build/download-nar.scm
+++ /dev/null
@@ -1,125 +0,0 @@
-;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
-;;;
-;;; This file is part of GNU Guix.
-;;;
-;;; GNU Guix is free software; you can redistribute it and/or modify it
-;;; under the terms of the GNU General Public License as published by
-;;; the Free Software Foundation; either version 3 of the License, or (at
-;;; your option) any later version.
-;;;
-;;; GNU Guix is distributed in the hope that it will be useful, but
-;;; WITHOUT ANY WARRANTY; without even the implied warranty of
-;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-;;; GNU General Public License for more details.
-;;;
-;;; You should have received a copy of the GNU General Public License
-;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
-
-(define-module (guix build download-nar)
-  #:use-module (guix build download)
-  #:use-module (guix build utils)
-  #:use-module (guix serialization)
-  #:use-module (guix zlib)
-  #:use-module (guix progress)
-  #:use-module (web uri)
-  #:use-module (srfi srfi-11)
-  #:use-module (srfi srfi-26)
-  #:use-module (ice-9 format)
-  #:use-module (ice-9 match)
-  #:export (download-nar))
-
-;;; Commentary:
-;;;
-;;; Download a normalized archive or "nar", similar to what 'guix substitute'
-;;; does.  The intent here is to use substitute servers as content-addressed
-;;; mirrors of VCS checkouts.  This is mostly useful for users who have
-;;; disabled substitutes.
-;;;
-;;; Code:
-
-(define (urls-for-item item)
-  "Return the fallback nar URL for ITEM--e.g.,
-\"/gnu/store/cabbag3…-foo-1.2-checkout\"."
-  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK
-  ;; though.
-  ;; TODO: Use HTTPS?  The downside is the extra dependency.
-  (let ((bases '("http://mirror.hydra.gnu.org/guix"
-                 "http://berlin.guixsd.org"))
-        (item  (basename item)))
-    (append (map (cut string-append <> "/nar/gzip/" item) bases)
-            (map (cut string-append <> "/nar/" item) bases))))
-
-(define (restore-gzipped-nar port item size)
-  "Restore the gzipped nar read from PORT, of SIZE bytes (compressed), to
-ITEM."
-  ;; Since PORT is typically a non-file port (for instance because 'http-get'
-  ;; returns a delimited port), create a child process so we're back to a file
-  ;; port that can be passed to 'call-with-gzip-input-port'.
-  (match (pipe)
-    ((input . output)
-     (match (primitive-fork)
-       (0
-        (dynamic-wind
-          (const #t)
-          (lambda ()
-            (close-port output)
-            (close-port port)
-            (catch #t
-              (lambda ()
-                (call-with-gzip-input-port input
-                  (cut restore-file <> item)))
-              (lambda (key . args)
-                (print-exception (current-error-port)
-                                 (stack-ref (make-stack #t) 1)
-                                 key args)
-                (primitive-exit 1))))
-          (lambda ()
-            (primitive-exit 0))))
-       (child
-        (close-port input)
-        (dump-port* port output
-                    #:reporter (progress-reporter/file item size
-                                                       #:abbreviation
-                                                       store-path-abbreviation))
-        (close-port output)
-        (newline)
-        (match (waitpid child)
-          ((_ . status)
-           (unless (zero? status)
-             (error "nar decompression failed" status)))))))))
-
-(define (download-nar item)
-  "Download and extract the normalized archive for ITEM.  Return #t on
-success, #f otherwise."
-  ;; Let progress reports go through.
-  (setvbuf (current-error-port) _IONBF)
-  (setvbuf (current-output-port) _IONBF)
-
-  (let loop ((urls (urls-for-item item)))
-    (match urls
-      ((url rest ...)
-       (format #t "Trying content-addressed mirror at ~a...~%"
-               (uri-host (string->uri url)))
-       (let-values (((port size)
-                     (catch #t
-                       (lambda ()
-                         (http-fetch (string->uri url)))
-                       (lambda args
-                         (values #f #f)))))
-         (if (not port)
-             (loop rest)
-             (begin
-               (if size
-                   (format #t "Downloading from ~a (~,2h MiB)...~%" url
-                           (/ size (expt 2 20.)))
-                   (format #t "Downloading from ~a...~%" url))
-               (if (string-contains url "/gzip")
-                   (restore-gzipped-nar port item size)
-                   (begin
-                     ;; FIXME: Add progress report.
-                     (restore-file port item)
-                     (close-port port)))
-               #t))))
-      (()
-       #f))))
diff --git a/guix/cvs-download.scm b/guix/cvs-download.scm
index 8b46f8ef8..85744c5b5 100644
--- a/guix/cvs-download.scm
+++ b/guix/cvs-download.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2014 Sree Harsha Totakura <sreeharsha@totakura.in>
 ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
 ;;;
@@ -23,7 +23,6 @@
   #:use-module (guix gexp)
   #:use-module (guix store)
   #:use-module (guix monads)
-  #:use-module (guix modules)
   #:use-module (guix packages)
   #:use-module (ice-9 match)
   #:export (cvs-reference
@@ -60,35 +59,16 @@
   "Return a fixed-output derivation that fetches REF, a <cvs-reference>
 object.  The output is expected to have recursive hash HASH of type
 HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
-  (define zlib
-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))
-
-  (define config.scm
-    (scheme-file "config.scm"
-                 #~(begin
-                     (define-module (guix config)
-                       #:export (%libz))
-
-                     (define %libz
-                       #+(file-append zlib "/lib/libz")))))
-
-  (define modules
-    (cons `((guix config) => ,config.scm)
-          (delete '(guix config)
-                  (source-module-closure '((guix build cvs)
-                                           (guix build download-nar))))))
   (define build
-    (with-imported-modules modules
+    (with-imported-modules '((guix build cvs)
+                             (guix build utils))
       #~(begin
-          (use-modules (guix build cvs)
-                       (guix build download-nar))
-
-          (or (cvs-fetch '#$(cvs-reference-root-directory ref)
-                         '#$(cvs-reference-module ref)
-                         '#$(cvs-reference-revision ref)
-                         #$output
-                         #:cvs-command (string-append #+cvs "/bin/cvs"))
-              (download-nar #$output)))))
+          (use-modules (guix build cvs))
+          (cvs-fetch '#$(cvs-reference-root-directory ref)
+                     '#$(cvs-reference-module ref)
+                     '#$(cvs-reference-revision ref)
+                     #$output
+                     #:cvs-command (string-append #+cvs "/bin/cvs")))))
 
   (mlet %store-monad ((guile (package->derivation guile system)))
     (gexp->derivation (or name "cvs-checkout") build
diff --git a/guix/git-download.scm b/guix/git-download.scm
index 731e549b3..7397cbe7f 100644
--- a/guix/git-download.scm
+++ b/guix/git-download.scm
@@ -25,7 +25,6 @@
   #:use-module (guix monads)
   #:use-module (guix records)
   #:use-module (guix packages)
-  #:use-module (guix modules)
   #:autoload   (guix build-system gnu) (standard-packages)
   #:use-module (ice-9 match)
   #:use-module (ice-9 popen)
@@ -78,31 +77,12 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
         (standard-packages)
         '()))
 
-  (define zlib
-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))
-
-  (define config.scm
-    (scheme-file "config.scm"
-                 #~(begin
-                     (define-module (guix config)
-                       #:export (%libz))
-
-                     (define %libz
-                       #+(file-append zlib "/lib/libz")))))
-
-  (define modules
-    (cons `((guix config) => ,config.scm)
-          (delete '(guix config)
-                  (source-module-closure '((guix build git)
-                                           (guix build utils)
-                                           (guix build download-nar))))))
-
   (define build
-    (with-imported-modules modules
+    (with-imported-modules '((guix build git)
+                             (guix build utils))
       #~(begin
           (use-modules (guix build git)
                        (guix build utils)
-                       (guix build download-nar)
                        (ice-9 match))
 
           ;; The 'git submodule' commands expects Coreutils, sed,
@@ -112,13 +92,12 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
                                            (((names dirs) ...)
                                             dirs)))
 
-          (or (git-fetch (getenv "git url") (getenv "git commit")
-                         #$output
-                         #:recursive? (call-with-input-string
-                                          (getenv "git recursive?")
-                                        read)
-                         #:git-command (string-append #+git "/bin/git"))
-              (download-nar #$output)))))
+          (git-fetch (getenv "git url") (getenv "git commit")
+                     #$output
+                     #:recursive? (call-with-input-string
+                                      (getenv "git recursive?")
+                                    read)
+                     #:git-command (string-append #+git "/bin/git")))))
 
   (mlet %store-monad ((guile (package->derivation guile system)))
     (gexp->derivation (or name "git-checkout") build
diff --git a/guix/hg-download.scm b/guix/hg-download.scm
index 6b25b87b6..842098090 100644
--- a/guix/hg-download.scm
+++ b/guix/hg-download.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -22,7 +22,6 @@
   #:use-module (guix store)
   #:use-module (guix monads)
   #:use-module (guix records)
-  #:use-module (guix modules)
   #:use-module (guix packages)
   #:autoload   (guix build-system gnu) (standard-packages)
   #:use-module (ice-9 match)
@@ -60,35 +59,18 @@
   "Return a fixed-output derivation that fetches REF, a <hg-reference>
 object.  The output is expected to have recursive hash HASH of type
 HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
-  (define zlib
-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))
-
-  (define config.scm
-    (scheme-file "config.scm"
-                 #~(begin
-                     (define-module (guix config)
-                       #:export (%libz))
-
-                     (define %libz
-                       #+(file-append zlib "/lib/libz")))))
-
-  (define modules
-    (cons `((guix config) => ,config.scm)
-          (delete '(guix config)
-                  (source-module-closure '((guix build hg)
-                                           (guix build download-nar))))))
-
   (define build
-    (with-imported-modules modules
+    (with-imported-modules '((guix build hg)
+                             (guix build utils))
       #~(begin
           (use-modules (guix build hg)
-                       (guix build download-nar))
+                       (guix build utils)
+                       (ice-9 match))
 
-          (or (hg-fetch '#$(hg-reference-url ref)
-                        '#$(hg-reference-changeset ref)
-                        #$output
-                        #:hg-command (string-append #+hg "/bin/hg"))
-              (download-nar #$output)))))
+          (hg-fetch '#$(hg-reference-url ref)
+                    '#$(hg-reference-changeset ref)
+                    #$output
+                    #:hg-command (string-append #+hg "/bin/hg")))))
 
   (mlet %store-monad ((guile (package->derivation guile system)))
     (gexp->derivation (or name "hg-checkout") build
-- 
2.15.1

bug#28659: Content-addressed mirror is not used upon invalid hash

[zimoun]

Hi Ludo,

On Fri, 15 Dec 2017 at 10:30, ludo@gnu.org (Ludovic Courtès) wrote:

>> So I think we have to communicate more info from the daemon to ‘guix
>> substitute’.
>
> The attached patch addresses that by simply calling out to the daemon to
> determine whether we’re dealing with a content-addressed item.

WDYT to rebase this patch [1] and resubmit to guix-patches in order to
get more attention and so potential feedback and/or review?

1: <https://issues.guix.gnu.org/issue/28659#26>


Cheers,
simon