bug#33419: guix package is not showing that the checksum is mismatching

bug#33419: guix package is not showing that the checksum is mismatching

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 2672 bytes --]

(Note: this is on a private channel of me, but I think this is
unrelated, it could happen with any Guix package).

I updated a package-definition and forgot to update the Checksum. When
then updating, it just fails, and the new, nice, logging-reduced UI
doesn't tell:

$ guix package -u guix-tools
substitute: updating list of substitutes from 'https://berlin.guixsd.org'...   0
[..]
building /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv...
build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
-guix package: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed

The log was not very helpful:


Initialized empty Git repository in /gnu/store/s399g9f1k19v01rs992w5dl6aif87har-
From https://gitlab.com/hoebjo/guix-tools
 * branch            4f17b792a3882f10ca9496b2d9aeb12b97ee93e2 -> FETCH_HEAD
Note: checking out 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 4f17b79 gdev: Add more package to environment.
environment variable `PATH' unset




Only when I explicitly built it, the error was shown:

$ guix build guix-tools

HEAD is now at 4f17b79 gdev: Add more package to environment.
environment variable `PATH' unset
output path `/gnu/store/s399g9f1k19v01rs992w5dl6aif87har-guix-tools-0.1.0-2.4f17b79-checkout' should have r:sha256 hash `1j4q43hk7jrys7zsmws37g6w7babzshfdb1s5myl7qwr3mcx6hnf', instead has `0b61q29915b4i6adidx3zixnx3m0zp58rjbfa9byqcz13szznc52'
build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
cannot build derivation `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed


Is it intended that this build-failure detail is not shown? I suppose
not. At least in a log it should be shown.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#33419: guix package is not showing that the checksum is mismatching

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 3087 bytes --]

On Sun, Nov 18, 2018 at 03:04:33PM +0100, Björn Höfling wrote:
> I updated a package-definition and forgot to update the Checksum. When
> then updating, it just fails, and the new, nice, logging-reduced UI
> doesn't tell:

Yes, this is a classic "gotcha" of Guix package development.

In Guix, those "things" for which you provide a hash are called
"fixed-output derivations":

https://www.gnu.org/software/guix/manual/en/html_node/Derivations.html

Unlike regular derivations, we know in advance what the output of the
derivation will be. Therefore, it does not matter to us how it is built
(from source, downloaded over HTTP, downloaded with Git, found in
/gnu/store, etc).

When building fixed-output derivations, Guix first looks in /gnu/store
to see if they are already built. If so, it just uses what it finds
there.

So, if you give the wrong hash, as you did in your example, Guix will
use that wrong source code.

This failure mode is often discovered by people who used `guix download`
to calculate a source hash, but then put the wrong URI in their package
definition. Because the source is already in /gnu/store, the URI is not
tested and their package doesn't work for anybody else.

It may also happen when the source is only found on the
content-addressed mirrors provided by Nix; no file-name checking is
performed in that case.

> $ guix package -u guix-tools
> substitute: updating list of substitutes from 'https://berlin.guixsd.org'...   0
> [..]
> building /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv...
> build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
> View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> -guix package: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed

[...]

> Only when I explicitly built it, the error was shown:
> 
> $ guix build guix-tools
> 
> HEAD is now at 4f17b79 gdev: Add more package to environment.
> environment variable `PATH' unset
> output path `/gnu/store/s399g9f1k19v01rs992w5dl6aif87har-guix-tools-0.1.0-2.4f17b79-checkout' should have r:sha256 hash `1j4q43hk7jrys7zsmws37g6w7babzshfdb1s5myl7qwr3mcx6hnf', instead has `0b61q29915b4i6adidx3zixnx3m0zp58rjbfa9byqcz13szznc52'
> build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
> View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> cannot build derivation `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv': 1 dependencies couldn't be built
> guix build: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed
> 
> Is it intended that this build-failure detail is not shown? I suppose
> not. At least in a log it should be shown.

I agree that `guix package` should print the relevant error.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33419: guix package is not showing that the checksum is mismatching

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 4501 bytes --]

Hi Leo,

On Sun, 18 Nov 2018 17:59:48 -0500
Leo Famulari <leo@famulari.name> wrote:

> On Sun, Nov 18, 2018 at 03:04:33PM +0100, Björn Höfling wrote:
> > I updated a package-definition and forgot to update the Checksum.
> > When then updating, it just fails, and the new, nice,
> > logging-reduced UI doesn't tell:  
> 
> Yes, this is a classic "gotcha" of Guix package development.
> 
> In Guix, those "things" for which you provide a hash are called
> "fixed-output derivations":
> 
> https://www.gnu.org/software/guix/manual/en/html_node/Derivations.html
> 
> Unlike regular derivations, we know in advance what the output of the
> derivation will be. Therefore, it does not matter to us how it is
> built (from source, downloaded over HTTP, downloaded with Git, found
> in /gnu/store, etc).
> 
> When building fixed-output derivations, Guix first looks in /gnu/store
> to see if they are already built. If so, it just uses what it finds
> there.
> 
> So, if you give the wrong hash, as you did in your example, Guix will
> use that wrong source code.
> 
> This failure mode is often discovered by people who used `guix
> download` to calculate a source hash, but then put the wrong URI in
> their package definition. Because the source is already
> in /gnu/store, the URI is not tested and their package doesn't work
> for anybody else.
> 
> It may also happen when the source is only found on the
> content-addressed mirrors provided by Nix; no file-name checking is
> performed in that case.
> 
> > $ guix package -u guix-tools
> > substitute: updating list of substitutes from
> > 'https://berlin.guixsd.org'...   0 [..]
> > building /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv...
> > build
> > of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv
> > failed View build log at
> > '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> > -guix package: error: build failed: build of
> > `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv'
> > failed  
> 
> [...]
> 
> > Only when I explicitly built it, the error was shown:
> > 
> > $ guix build guix-tools
> > 
> > HEAD is now at 4f17b79 gdev: Add more package to environment.
> > environment variable `PATH' unset
> > output path
> > `/gnu/store/s399g9f1k19v01rs992w5dl6aif87har-guix-tools-0.1.0-2.4f17b79-checkout'
> > should have r:sha256 hash
> > `1j4q43hk7jrys7zsmws37g6w7babzshfdb1s5myl7qwr3mcx6hnf', instead has
> > `0b61q29915b4i6adidx3zixnx3m0zp58rjbfa9byqcz13szznc52' build
> > of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv
> > failed View build log at
> > '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> > cannot build derivation
> > `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv':
> > 1 dependencies couldn't be built guix build: error: build failed:
> > build of
> > `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv'
> > failed
> > 
> > Is it intended that this build-failure detail is not shown? I
> > suppose not. At least in a log it should be shown.  
> 
> I agree that `guix package` should print the relevant error.


thank you very much for your lengthy explanations. I'm just no longer
sure we talk about the same problem :-) 

What you described is the fact that somethings wrong in the URL from
beginning but someone seeded the store with 'guix download'. Or the URL
404s out but the sources are all available in the
Content-Addressed-Network. Then Guix  still downloads it by hash
(though it is not locally searching in the store by hash, cmp this bug
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32245).

But that's not what I was referring to here: My point was:
"guix build" failed, and it was failing because the hash of the (then
newly downloaded) sources did not match the expected ones. And "guix
build" is saying so in its output.

But: A "guix package" (with -u or -i) is of cause also failing. But it
hides the cause. And that is nice, the new UI is meant to hide all
those details and backtraces from the user. It refers to the log for
those details. But in the referenced log you cannot find the cause
either. It is not fully equivalent with the "guix build" logs.

Does that make sense?

Björn




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#33419: guix package is not showing that the checksum is mismatching

[Ludovic Courtès]

Hello,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> $ guix package -u guix-tools
> substitute: updating list of substitutes from 'https://berlin.guixsd.org'...   0
> [..]
> building /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv...
> build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
> View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> -guix package: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed

[...]

> $ guix build guix-tools
>
> HEAD is now at 4f17b79 gdev: Add more package to environment.
> environment variable `PATH' unset
> output path `/gnu/store/s399g9f1k19v01rs992w5dl6aif87har-guix-tools-0.1.0-2.4f17b79-checkout' should have r:sha256 hash `1j4q43hk7jrys7zsmws37g6w7babzshfdb1s5myl7qwr3mcx6hnf', instead has `0b61q29915b4i6adidx3zixnx3m0zp58rjbfa9byqcz13szznc52'
> build of /gnu/store/vy3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv failed
> View build log at '/var/log/guix/drvs/vy/3s1y7bv1w6d8gmp5b10xppy9skbgkd-guix-tools-0.1.0-2.4f17b79-checkout.drv.bz2'.
> cannot build derivation `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv': 1 dependencies couldn't be built
> guix build: error: build failed: build of `/gnu/store/iwma3gq778n32mqz6y7277g67nvx1abb-guix-tools-0.1.0-2.4f17b79.drv' failed
>
>
> Is it intended that this build-failure detail is not shown? I suppose
> not. At least in a log it should be shown.

I suspect you’re running an “old” guix-daemon, aren’t you?

Could you try again with an up-to-date daemon (and client)?  The code
that prints hash mismatch errors when talking to recent daemons is in
(guix status).

Thanks,
Ludo’.

bug#33419: guix package is not showing that the checksum is mismatching

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 3166 bytes --]

On Mon, 19 Nov 2018 21:30:46 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hello,
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> > Is it intended that this build-failure detail is not shown? I
> > suppose not. At least in a log it should be shown.  
> 
> I suspect you’re running an “old” guix-daemon, aren’t you?
> 
> Could you try again with an up-to-date daemon (and client)?  The code
> that prints hash mismatch errors when talking to recent daemons is in
> (guix status).

Indeed! Short story: After really updating the daemon/root's Guix, it
works now.

Long story:

I have a really old daemon here:

$ tail -n 2 /etc/init/guix-daemon.conf 

exec /gnu/store/5drb0ijbszvy8xmps89qcav1p4vy9wqr-guix-0.11.0/bin/guix-daemon --build-users-group=guixbuild

I have no clue how to update it correctly and it was always a mystery,
so I read in the manual how to do that nowadays:

https://guix.info/manual/en/Binary-Installation.html#Binary-Installation

and did:

$ sudo
cp /root/.guix-profile/lib/upstart/system/guix-daemon.conf /etc/init/guix-daemon.conf
$ tail -n 2 /etc/init/guix-daemon.conf 

exec /var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon --build-users-group=guixbuild

$ sudo systemctl stop guix-daemon
$ sudo systemctl start guix-daemon
$ ps -aux | grep guix
root      6995  0.1  0.0  27976  3940 ?        Ss   07:30   0:00 /var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon --substitute-urls=https://berlinguixsd.org https://mirror.hydra.gnu.org --build-users-group=guixbuild
bjoern    6998  0.0  0.0  15780   948 pts/16   S+   07:30   0:00 grep --color=auto guix

Now, let's introduce the wrong checksum again, pull, update:

$ guix pull
$ guix package -u guix-tools

[...]

I was about to say that it still doesn't work, until I thought maybe my
guix in ~root/.guix-profile could also be a bit old:

$ sudo guix package -I | grep guix

guix    0.13.0  out     /gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0

Uh, better than 0.11, but still not the newest version. But how to
get to a new one? Now, a `guix pull` for root failed with some error. So
I did a 

/home/bjoern/.config/guix/current/bin/guix pull --commit=<the-one-of-my-normal-user>

That worked, it updated guix.

After guix package -u, restarting the daemon again, I'm happy to see
the error directly on (as unpriviledged user bjoern: "guix package -u
guix-tools"):

substitute: updating substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
building /gnu/store/kx97aqz66lpibxjiw3g8k7xhz5mmniyp-guix-tools-0.1.0-3.295d0a0-checkout.drv...
r:sha256 hash mismatch for /gnu/store/9d0djawpyjdl6jp900agmqfxrbk2vcra-guix-tools-0.1.0-3.295d0a0-checkout:
  expected hash: 11111111y50jcakricz36n1n5m99fxv8gxmk9ba3g0zfsl1a0918
  actual hash:   06z0jhxpy50jcakricz36n1n5m99fxv8gxmk9ba3g0zfsl1a0918
hash mismatch for store item '/gnu/store/9d0djawpyjdl6jp900agmqfxrbk2vcra-guix-tools-0.1.0-3.295d0a0-checkout'
build of /gnu/store/kx97aqz66lpibxjiw3g8k7xhz5mmniyp-guix-tools-0.1.0-3.295d0a0-checkout.drv failed


Thanks, closing this bug.

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#33419: guix package is not showing that the checksum is mismatching

[Ludovic Courtès]

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> On Mon, 19 Nov 2018 21:30:46 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hello,
>> 
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>
>> > Is it intended that this build-failure detail is not shown? I
>> > suppose not. At least in a log it should be shown.  
>> 
>> I suspect you’re running an “old” guix-daemon, aren’t you?
>> 
>> Could you try again with an up-to-date daemon (and client)?  The code
>> that prints hash mismatch errors when talking to recent daemons is in
>> (guix status).
>
> Indeed! Short story: After really updating the daemon/root's Guix, it
> works now.

Cool!

> Long story:

Pffew, quite a journey.  Hopefully upgrading the daemon on foreign
distro will be easier and more natural with
<https://issues.guix.info/issue/33111>.

Thanks,
Ludo’.