guix time-machine, broken hash in an old package definition, a workaround?

guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 2287 bytes --]

Hi list,

It appears that the package bundles at CRAN are not stable. The bundle
contents can change, and so its hash. I’ve encountered such a problem
three times, I think, already, so it’s presumably systemic. The latest
(minimal working) example:

$ cat manifest.scm

(specifications->manifest
  '("r-foreign"))

$ cat channel-specs.scm

(list (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (commit
          "d81fb2ae9443994ae5dd1cb5837276fad63f842c")))

$ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- environment -C --pure --manifest=manifest.scm

Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
building /gnu/store/ixqarfdy375cqwkwgvm9z8cadw87f6gg-foreign_0.8-75.tar.gz.drv...
downloading from http://cran.r-project.org/src/contrib/Archive/foreign/foreign_0.8-75.tar.gz...
/sha256 hash mismatch for /gnu/store/z6hkz6r1i5cgzjxc5m883lgh1xvjff8s-foreign_0.8-75.tar.gz:
  expected hash: 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl
  actual hash:   1c888wrn9xf94lp7w9kjw5l8fnarrkv5pi1px5rfnybm1qlysdx5
hash mismatch for store item '/gnu/store/z6hkz6r1i5cgzjxc5m883lgh1xvjff8s-foreign_0.8-75.tar.gz'
build of /gnu/store/ixqarfdy375cqwkwgvm9z8cadw87f6gg-foreign_0.8-75.tar.gz.drv failed
View build log at '/var/log/guix/drvs/ix/qarfdy375cqwkwgvm9z8cadw87f6gg-foreign_0.8-75.tar.gz.drv.bz2'.
cannot build derivation `/gnu/store/vx9187mcdj041sr76bivzbnggr6ajm4q-r-foreign-0.8-75.drv': 1 dependencies couldn't be built
guix environment: error: build of `/gnu/store/vx9187mcdj041sr76bivzbnggr6ajm4q-r-foreign-0.8-75.drv' failed

I tried

. adding another channel with r-foreign redefined in hope that the one in guix
would get masked

. adding --with-input=r-foreign=r-foreign-redefined (the latter defined
in another channel)

. adding -L to the environment (also to time-machine) pointing to a
directory with r-foreign redefined

. an inferior

These attempts were either ignored by guix or resulted in

guix time-machine: error: got unexpected path `Backtrace:' from substituter

Is there a recommended way to deal with such a situation? Thank you for
taking a look.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

Sadly, nothing can be done.  Well, I hope that someone will show me I am
wrong. :-)

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c \
       -- weather r-foreign \
       --substitute-urls="https://ci.guix.gnu.org https://bayfront.guix.gnu.org https://guix.cbaines.net https://guix.tobias.gr https://builds.guix-patches.cbaines.net"
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
guile: warning: failed to install locale
hint: Consider installing the `glibc-utf8-locales' or `glibc-locales' package
and defining `GUIX_LOCPATH', along these lines:

     guix package -i glibc-utf8-locales
     export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

See the "Application Setup" section in the manual, for more info.

computing 1 package derivations for x86_64-linux...
looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  .0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.000 seconds per request (0.0 seconds in total)
  2770.1 requests per second

  0.0% (0 out of 1) of the missing items are queued
  660 queued builds
      i586-gnu: 5 (.8%)
      aarch64-linux: 634 (96.1%)
      armhf-linux: 1 (.2%)
      x86_64-linux: 18 (2.7%)
      i686-linux: 2 (.3%)
  build rate: 111.65 builds per hour
      x86_64-linux: 37.52 builds per hour
      aarch64-linux: 70.38 builds per hour
      i686-linux: 4.14 builds per hour
looking for 1 store items on https://bayfront.guix.gnu.org...
updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
https://bayfront.guix.gnu.org
  .0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.176 seconds per request (0.2 seconds in total)
  5.7 requests per second

  0.0% (0 out of 1) of the missing items are queued
  93 queued builds
      x86_64-linux: 93 (100.0%)
  'https://bayfront.guix.gnu.org/api/latestbuilds?nr=1000' returned 504 ("Gateway Time-out")
looking for 1 store items on https://guix.cbaines.net...
updating substitutes from 'https://guix.cbaines.net'... 100.0%
https://guix.cbaines.net
  .0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  1.751 seconds per request (1.8 seconds in total)
  0.6 requests per second
  (continuous integration information unavailable)
looking for 1 store items on https://guix.tobias.gr...
updating substitutes from 'https://guix.tobias.gr'... 100.0%
https://guix.tobias.gr
  .0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.210 seconds per request (0.2 seconds in total)
  4.8 requests per second
  (continuous integration information unavailable)
looking for 1 store items on https://builds.guix-patches.cbaines.net...
updating substitutes from 'https://builds.guix-patches.cbaines.net'... 100.0%
https://builds.guix-patches.cbaines.net
  .0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.182 seconds per request (0.2 seconds in total)
  5.5 requests per second
  (continuous integration information unavailable)
--8<---------------cut here---------------end--------------->8---

Well, maybe someone has the tarball in their personal store and could
share it.  Otherwise, it is game over.

The story about “guix time-machine” and tarball is far to be complete
and robust.  It is impossible to fix the upstream in-place replacement.
The only thing the Guix project can do is store these tarballs on their
machine; it is what it is commonly done but time to time edge cases
happen, sadly.

Instead of archiving the upstream source code on machines from Guix
projects, the idea is to use Software Heritage infrastructure as
archivist.  It works pretty well for Git source because the mapping
(content-address) between Git ID, SWH-ID, and NAR hash is almost
straightforward.  But, this mapping is not as straightforward for
tarballs because of metadata.  That’s the aim of the project Disarchive;
still experimental.

https://git.ngyro.com/disarchive-db/


One way to locally fix is: checkout the Guix repo at commit
d81fb2ae9443994ae5dd1cb5837276fad63f842c, then tweak the checksum, build
from this source, and use:

  ./pre-inst-env guix environment -C -m manifest.scm


All the best,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

On Wed, Jan 13, 2021 at 05:24:39PM +0100, zimoun wrote:

> One way to locally fix is: checkout the Guix repo at commit
> d81fb2ae9443994ae5dd1cb5837276fad63f842c, then tweak the checksum, build
> from this source, and use:
>
>   ./pre-inst-env guix environment -C -m manifest.scm

Hi simon,

Thanks for the tip. I may give it a try if there is no other solution
available at the moment.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Leo Famulari]

On Wed, Jan 13, 2021 at 02:22:23PM +0100, Wiktor Żelazny wrote:
> These attempts were either ignored by guix or resulted in
> 
> guix time-machine: error: got unexpected path `Backtrace:' from substituter

I don't think this error is related to time-machine and unstable source
code. It appears to be <https://bugs.gnu.org/45828>.

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

On Wed, Jan 13, 2021 at 01:57:06PM -0500, Leo Famulari wrote:
> On Wed, Jan 13, 2021 at 02:22:23PM +0100, Wiktor Żelazny wrote:
> > These attempts were either ignored by guix or resulted in
> >
> > guix time-machine: error: got unexpected path `Backtrace:' from substituter
>
> I don't think this error is related to time-machine and unstable source
> code. It appears to be <https://bugs.gnu.org/45828>.

Hi Leo,

Yeah, there may be a connection. Is --no-substitutes a way to avoid this
error?

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On Wed, Jan 13, 2021 at 08:37:30PM +0100, Wiktor Żelazny wrote:
> Yeah, there may be a connection. Is --no-substitutes a way to avoid this
> error?

Yes, the problem is in the substituter so --no-substitutes will avoid
it. You might spend a loooooong time building things but it depends on
the specific package(s).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Wed, Jan 13, 2021 at 03:44:41PM -0500, Leo Famulari wrote:

> You might spend a loooooong time building things but it depends on the
> specific package(s).

I would argue that while using time-machine one does not expect
substitutes to be available, any way, since the old binaries are removed
by the substitute provider as the time goes by. It that’s not true, a
possibility to avoid the long build time problem, I think, is to perform
`guix time-machine --commit --channels -- build --no-substitutes` for
the offending package, only, before `environment`, the latter with the
substitutes permitted.

I will retry now my attempts to fix the r-foreign problem with
--no-substitutes added.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Thu, 14 Jan 2021 at 09:30, Wiktor Żelazny <wz@freeshell.de> wrote:

> I will retry now my attempts to fix the r-foreign problem with
> --no-substitutes added.

I guess it will not change your problem at hand: the missing tarball.

If I understand correctly, your initial message describes 2 issues:

 - hash mismatch
 - substitutes error

About the hash mismatch, game over with time-machine.  Well, you have to
do the “time-machine” by hand where the simplest IMHO is what I proposed.

The substitutes error seems transient.  Rebuild locally all you need vs
wait the fix: I do not know what would be the fastest. :-)


All the best,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 2019 bytes --]

On Thu, Jan 14, 2021 at 10:48:35AM +0100, zimoun wrote:

> I guess it will not change your problem at hand: the missing tarball.

A tarball is there, just a different one. I would like to make guix
accept it.

> About the hash mismatch, game over with time-machine.

Are you sure? I remember a situation where a package was defined in my
private channel. Then, someone committed a definition for the same
package to guix, but the definition in the private channel was still
given a priority while performing `guix package` operations.

Isn’t it possible to obtain analogous behavior with time-machine and
have a definition with a current hash defined in a private channel
replace the original one? Isn’t it the purpose of inferiors? Not a long
time ago, there was this [1] thread discussed here, which looks related
to my problem — a need for package definition shuffling. An inferior
gave me the substitutes error, so I’m hoping to make some progress once
I remove this obstacle.

[1]: https://lists.gnu.org/archive/html/help-guix/2020-11/msg00230.html

If I don’t manage to make the different channels “communicate with each
other”, I can try substituting the input r-foreign definition from the
guix channel with one with another version, which is even closer to the
theme of the cited thread. I don’t care much about the r-foreign
version, but I care about the version (and the binary) of r and the R
package stack that I use in that environment, and r happens to depend on
r-foreign as an input.

> Well, you have to do the “time-machine” by hand where the simplest
> IMHO is what I proposed.

If the above fails, I will.

> The substitutes error seems transient.  Rebuild locally all you need
> vs wait the fix: I do not know what would be the fastest. :-)

For testing a solution to the hash mismatch problem, it suffices to
build r, which uses r-foreign as an input. I will decide later about
what to do next.

Thank you for your support,

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Thu, 14 Jan 2021 at 20:00, Wiktor Żelazny <wz@freeshell.de> wrote:

>> About the hash mismatch, game over with time-machine.
>
> Are you sure? I remember a situation where a package was defined in my
> private channel. Then, someone committed a definition for the same
> package to guix, but the definition in the private channel was still
> given a priority while performing `guix package` operations.

It depends on what you want at the end: only the package r-foreign or
some packages depending on r-foreign.

For only the package r-foreign, it is trivial:

--8<---------------cut here---------------start------------->8---
$ cat pkgs/fix.scm                    
(define-module (fix)                                                            
  #:use-module (guix packages)                                                  
  #:use-module (guix download)                                                  
  #:use-module (guix build-system r)                                            
  #:use-module (gnu packages statistics))                                       
                                                                                
(define-public r-foreign-fixed                                                  
  (package                                                                      
    (inherit r-foreign)                                                         
    (name "r-foreign")                                                    
    (version "0.8-75")                                                          
    (source                                                                     
     (origin                                                                    
       (method url-fetch)                                                       
       (uri (cran-uri "foreign" version))                                       
       (sha256                                                                  
        (base32                                                                 
         "1c888wrn9xf94lp7w9kjw5l8fnarrkv5pi1px5rfnybm1qlysdx5"))))))

$  guix time-machine --commit=d81fb2a -- build -L pkgs r-foreign-
guix build: avertissement : spécification du paquet « r-foreign » ambiguë       
guix build: avertissement : choix de r-foreign@0.8-75 à l'emplacement pkgs/fix.\
scm:8:2
[…]
/gnu/store/d1vfvx9y6cada0fcl8adx0gslk8iz4sc-r-foreign-0.8-75
--8<---------------cut here---------------end--------------->8---

And you can put this package in a manifest file as you did and simply
run:

  guix time-machine -C channels.scm \
       -- environment -C -L pkgs -m manifest.scm

But I am doubtful that is what you really want.  Instead, I guess you
want packages that depends on r-foreign, as r for instance.  Let take
r-hmisc and r-rio for simplicity.

As you see, there is an ambiguity.  If the symbol ‘r-foreign‘ is defined
in the module (fix), then the module (gnu packages statistics) cannot be
used so you need to declare all the recipe.  And if ’inherit‘ is used
then the symbol ‘r-foreign‘ cannot be defined twice.

Moreover, the symbol you really want is the one in (fix).  But the
package r-hmisc refers to the one in (gnu packages statistics).  And the
package r-rio also refers to the one in (gnu packages statistics)
because on the top of the file, as you can see, there is #:use-module
(gnu packages statistics).

Game over? :-)  No wait…

> If I don’t manage to make the different channels “communicate with each
> other”, I can try substituting the input r-foreign definition from the
> guix channel with one with another version, which is even closer to the
> theme of the cited thread. I don’t care much about the r-foreign
> version, but I care about the version (and the binary) of r and the R
> package stack that I use in that environment, and r happens to depend on
> r-foreign as an input.

…this trick works:

--8<---------------cut here---------------start------------->8---
$  guix time-machine --commit=d81fb2a \
        -- build -L pkgs r-hmisc r-rio --with-input=r-foreign=r-foreign 
[…]
guix build: avertissement : spécification du paquet « r-foreign » ambiguë       
guix build: avertissement : choix de r-foreign@0.8-75 à l'emplacement pkgs/fix.\
scm:8:2
[…]
/gnu/store/b64i6d3vsyss7154j1dgvc8rr7k4wzqs-r-rio-0.5.16
/gnu/store/w0lpix3yjlzsb9kh32hsg0lp1igrk1y9-r-hmisc-4.3-0
--8<---------------cut here---------------end--------------->8---

If you want you avoid the ambiguity, you can instead rename the package
as you want, for instance r-foreign-new and just type:

  --with-input=r-foreign=r-foreign-new

> For testing a solution to the hash mismatch problem, it suffices to
> build r, which uses r-foreign as an input. I will decide later about
> what to do next.

Well, first be careful because you will rebuild a lot of R packages
because r-foreign is almost a root package.  Second, you will not get
the exact R packages as they were at the time of d81fb2a; which is for
me Game Over! :-)


Hope that helps,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]

On Thu, Jan 14, 2021 at 09:29:48PM +0100, zimoun wrote:

> But I am doubtful that is what you really want.  Instead, I guess you
> want packages that depends on r-foreign, as r for instance.  Let take
> r-hmisc and r-rio for simplicity.

Hi,

Thank you for the great explanation.

> --8<---------------cut here---------------start------------->8---
> $  guix time-machine --commit=d81fb2a \
>         -- build -L pkgs r-hmisc r-rio --with-input=r-foreign=r-foreign 
> […]
> guix build: avertissement : spécification du paquet « r-foreign » ambiguë       
> guix build: avertissement : choix de r-foreign@0.8-75 à l'emplacement pkgs/fix.\
> scm:8:2
> […]
> /gnu/store/b64i6d3vsyss7154j1dgvc8rr7k4wzqs-r-rio-0.5.16
> /gnu/store/w0lpix3yjlzsb9kh32hsg0lp1igrk1y9-r-hmisc-4.3-0
> --8<---------------cut here---------------end--------------->8---
>
> If you want you avoid the ambiguity, you can instead rename the package
> as you want, for instance r-foreign-new and just type:
>
>   --with-input=r-foreign=r-foreign-new

This actually looks like one of the approaches that I tried before
starting this thread, but with `environment` substituted for `build`. Is
it possible that `guix environment` ignores --with-input? `guix
environment --help-transform` lists it.

It is also possible that I did something a bit differently than in your
example (devil in the details). I would need to compare the presented
approach with mine.

> you will not get the exact R packages as they were at the time of
> d81fb2a;

Can you, please, elaborate on that? Do you mean by that that the
different r-foreign will result in a different r, and that will
propagate to the packages, as they depend on r? But R does not compile
the R code in the packages while they are being installed, does it? Am I
missing something? If it were the issue wouldn’t it occur also in your
`./pre-inst-env` approach?

A new idea: I just checked “CRAN Time Machine” at MRAN. The tarball with
the 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl hash is there.
I guess I can use `build --with-source=` now, maybe even `environment
--with-source=r-foreign=`? Perhaps a more elegant solution would be to
define r-foreign-fixed, as you describe above, yet this time leaving the
hash, but changing the URL. Are there philosophical reasons for not
using MRAN?

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Fri, 15 Jan 2021 at 21:18, Wiktor Żelazny <wz@freeshell.de> wrote:
> On Thu, Jan 14, 2021 at 09:29:48PM +0100, zimoun wrote:

>> you will not get the exact R packages as they were at the time of
>> d81fb2a;
>
> Can you, please, elaborate on that? Do you mean by that that the
> different r-foreign will result in a different r, and that will

Yes.  Bit-to-bit different but you can expect functionally similar.

> propagate to the packages, as they depend on r? But R does not compile
> the R code in the packages while they are being installed, does it? Am I
> missing something? If it were the issue wouldn’t it occur also in your
> `./pre-inst-env` approach?

Same root of problem, same consequence. :-)

This upstream bad practise is not fixable; whatever the mean.


> A new idea: I just checked “CRAN Time Machine” at MRAN. The tarball with
> the 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl hash is there.
> I guess I can use `build --with-source=` now, maybe even `environment
> --with-source=r-foreign=`? Perhaps a more elegant solution would be to
> define r-foreign-fixed, as you describe above, yet this time leaving the
> hash, but changing the URL. Are there philosophical reasons for not
> using MRAN?

Oh, thanks for the pointer.  Yeah, in this case it is possible to use
--with-source, maybe combined with another trick to populate your store
with the expected by d81fb2a tarball.  And then simply use “guix
time-machine” as usual.

The API needs to be checked, maybe it should be possible to automatise
the fallback: try Guix build farm, then upstream, then SWH, then CRAN
for R build system.  Maybe. :-)


All the best,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 1888 bytes --]

On Fri, Jan 15, 2021 at 09:48:20PM +0100, zimoun wrote:
> On Fri, 15 Jan 2021 at 21:18, Wiktor Żelazny <wz@freeshell.de> wrote:
>
> > A new idea: I just checked “CRAN Time Machine” at MRAN. The tarball with
> > the 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl hash is there.
> > I guess I can use `build --with-source=` now, maybe even `environment
> > --with-source=r-foreign=`?
>
> Oh, thanks for the pointer.  Yeah, in this case it is possible to use
> --with-source, maybe combined with another trick to populate your store
> with the expected by d81fb2a tarball.

Hello again,

Unfortunately, it’s not the end of the story:

$ curl -o foreign_0.8-75.tar.gz https://cran.microsoft.com/snapshot/2020-01-27/src/contrib/foreign_0.8-75.tar.gz
$ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- \
   build --with-source=r-foreign=foreign_0.8-75.tar.gz r-foreign
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
building /gnu/store/xzsw0wiw223s07szh18kqlq2ynmmvlp8-r-foreign-75.drv...
[building]
successfully built /gnu/store/xzsw0wiw223s07szh18kqlq2ynmmvlp8-r-foreign-75.drv
The following graft will be made:
   /gnu/store/ha6y2wpk2a9s237lk8jzhfg9pawfp12v-r-foreign-75.drv
applying 1 graft for /gnu/store/ha6y2wpk2a9s237lk8jzhfg9pawfp12v-r-foreign-75.drv...
grafting '/gnu/store/617x95s9ykd0va7r06g187y3dbwa3ddh-r-foreign-75' -> '/gnu/store/jvh9zyym3xmzy94g37gx5klg8rx72vfj-r-foreign-75'...
successfully built /gnu/store/ha6y2wpk2a9s237lk8jzhfg9pawfp12v-r-foreign-75.drv
/gnu/store/jvh9zyym3xmzy94g37gx5klg8rx72vfj-r-foreign-75

As you can see, there’s “-75” rather than “0.8-75” in the paths. Am I
doing something wrong or, perhaps, is this a Guix bug?

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Mon, Jan 18, 2021 at 09:57:32AM +0100, Wiktor Żelazny wrote:

> $ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- \
>    build --with-source=r-foreign=foreign_0.8-75.tar.gz r-foreign
> /gnu/store/jvh9zyym3xmzy94g37gx5klg8rx72vfj-r-foreign-75

A step forward:

$ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- \
   build --with-source=r-foreign@0.8-75=foreign_0.8-75.tar.gz r-foreign@0.8-75

results in a path with the correct version. However, subsequent `guix
time-machine -- environment` ignores is, downloads the source from CRAN,
and complaints about the hash.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

On Fri, Jan 15, 2021 at 09:19:01PM +0100, Wiktor Żelazny wrote:

> A new idea: I just checked “CRAN Time Machine” at MRAN. The tarball
> with the 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl hash is
> there.

A solution with an inferior:

in guix-packages.git/local/cran.scm (guix-packages.git is a local git repo):

   (define-public r-foreign-fixed
     (package (inherit r-foreign)
        (version "0.8-75-fixed")
        (source
           (origin
           (method url-fetch)
           (uri "https://cran.microsoft.com/snapshot/2020-01-27/src/contrib/foreign_0.8-75.tar.gz")
           (sha256
              (base32
                 "0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl"))))))

$ cat manifest.scm

   (use-modules (guix inferior) (guix channels)
                (srfi srfi-1))   ;for 'first'

   (define channels
     ;; This is the revision from which we want to
     ;; extract r-foreign.
     (list (channel
            (name 'guix)
            (url "https://git.savannah.gnu.org/git/guix.git")
            (commit
             "d81fb2ae9443994ae5dd1cb5837276fad63f842c"))
           (channel
            (name 'local)
            (url "./guix-packages.git"))))

   (define inferior
      ;; An inferior representing the above revision
      (inferior-for-channels channels))

   ;; Now create a manifest with the current "r" package
   ;; and the substituted "r-foreign" package.
   (packages->manifest
    (list (first (lookup-inferior-packages inferior "r-foreign" "0.8-75-fixed"))
          (specification->package "r")))

More stuff can be added to the manifest by appending other
`(specification->package "<package>")` elements to the list.

$ cat channel-specs.scm

  (list (channel
        (name 'local)
        (url "./guix-packages.git"))
      (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (commit
          "d81fb2ae9443994ae5dd1cb5837276fad63f842c")))

$ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- \
  environment -C --pure --manifest=manifest.scm

I extracted and adapted this from my larger package definition
repository and manifest, and did not test the resulting minimum version,
but you get the idea. Perhaps, this should go to the manual.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Wed, 20 Jan 2021 at 10:35, Wiktor Żelazny <wz@freeshell.de> wrote:
> On Fri, Jan 15, 2021 at 09:19:01PM +0100, Wiktor Żelazny wrote:
>
>> A new idea: I just checked “CRAN Time Machine” at MRAN. The tarball
>> with the 0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl hash is
>> there.
>
> A solution with an inferior:

Your solution could be error prone, IMHO.

> in guix-packages.git/local/cran.scm (guix-packages.git is a local git repo):
>
>    (define-public r-foreign-fixed
>      (package (inherit r-foreign)
>         (version "0.8-75-fixed")
>         (source
>            (origin
>            (method url-fetch)
>            (uri "https://cran.microsoft.com/snapshot/2020-01-27/src/contrib/foreign_0.8-75.tar.gz")
>            (sha256
>               (base32
>                  "0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl"))))))

Cool if Microsoft support long time archive of CRAN packages.  Well, it
seems possible to use it as fallback.

>    (packages->manifest
>     (list (first (lookup-inferior-packages inferior "r-foreign" "0.8-75-fixed"))
>           (specification->package "r")))

Here, the package “r-foreign” come from d81fb2a and so it is built using
the R build system from d81fb2a.

However, the package “r” come from the current Guix, i.e., the Guix when
the manifest is called.  Therefore, the “r-foreign” and “r” packages
could be incompatible.  For example, if “r-foreign” is built with R 3.x
and “r” with R 3.y maybe these two R versions use a different bytecode
for their VMs.

> $ guix time-machine --commit=d81fb2ae9443994ae5dd1cb5837276fad63f842c --channels=channel-specs.scm -- \
>   environment -C --pure --manifest=manifest.scm

Specifying --commit and --channels is redundant.  Other said, the
--commit is not necessary because it is already provided by your
’channel-specs.scm’.  But that’s a detail. :-)

This command create an inferior at commit fixed by ’channel-specs.scm’
and in this inferior, it runs ’manifest.scm’.

The file ’manifest.scm’ creates another inferior at commit d81fb2a to
build “r-foreign” and then build “r” in the current inferior (the one
specified by ’channel-specs.scm’).

By “chance”, the file ’channel-specs.scm’ and ’manifest.scm’ points to
the same commit.  However, the inferior in ’manifest.scm’ is not
necessary.  And even, from my point of view, it is a bad idea.

Inferiors in ’manifest.scm’ are used when you want to put some packages
from different Guix commits in the same profile.  And it is not what you
want here; if I understand correctly your problem.


All the best,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 3313 bytes --]

On Wed, Jan 20, 2021 at 11:15:17AM +0100, zimoun wrote:

> Cool if Microsoft support long time archive of CRAN packages.  Well, it
> seems possible to use it as fallback.

Hi,

Thanks for the review. You could say the same thing about Software
Heritage. You never know. I was considering starting a new thread
questioning the sufficiency of the manifest and channels to reproduce an
environment (you can find this kind of statement in Konrad Hinsen’s
“Reproducible computations with Guix” last year’s GuixHPC blog post, for
instance). Looks like you also want to archive the source code, just in
case.

> >    (packages->manifest
> >     (list (first (lookup-inferior-packages inferior "r-foreign" "0.8-75-fixed"))
> >           (specification->package "r")))
>
> Here, the package “r-foreign” come from d81fb2a and so it is built using
> the R build system from d81fb2a.

I do want d81fb2a r-foreign be built by d81fb2a R build system.

> However, the package “r” come from the current Guix, i.e., the Guix when
> the manifest is called.

It is called from the time machine, so it’s also d81fb2a, right? That’s
what I want.

> Specifying --commit and --channels is redundant.  Other said, the
> --commit is not necessary because it is already provided by your
> ’channel-specs.scm’.  But that’s a detail. :-)

Thanks. I thought that --channels referred to the package definitions,
whereas --commit to the guix version that processes them (yes, I know
that the definitions are a part of guix, but still).

Does `environment -C` imply `environment --pure`, as well?

> By “chance”, the file ’channel-specs.scm’ and ’manifest.scm’ points to
> the same commit.

This is intentional. I want to have an environment reflecting the
d81fb2a state of things, so that’s why I’m consistent with the commit.

> However, the inferior in ’manifest.scm’ is not necessary.

Why? If it’s not there, you’re facing the hash mismatch problem, aren’t
you? Please, explain.

> Inferiors in ’manifest.scm’ are used when you want to put some packages
> from different Guix commits in the same profile.  And it is not what you
> want here; if I understand correctly your problem.

If a tool designed for some purpose turns out to be suitable for other
purposes, as well, I see no reason for not using it for latter. It’s a
bit like Unix philosophy.

I use an inferior to substitute a guix package also in my config.scm. I
added some patch or something (I don’t remember), so the definition is
loaded from another channel, rather than another guix commit. Maybe I
even borrowed this trick from someone who had been showing it on this
mailing list.

   (define wz-channel
     (cons* (channel
             (name 'guix-wz)
             (url "file:///home/w/guix/guix-wz-git"))
            %default-channels))

   (define spectrwm-wz
     (first
      (lookup-inferior-packages
       (inferior-for-channels wz-channel)
       "spectrwm-wz" "3.2.0")))  ; I should upgrade it, I guess

   (packages (append (list
                      ;; dadada
                      spectrwm-wz)
                     %base-packages))

It works for me. If there are better solutions, I’ll be happy to learn.

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Wed, 20 Jan 2021 at 13:26, Wiktor Żelazny <wz@freeshell.de> wrote:
> On Wed, Jan 20, 2021 at 11:15:17AM +0100, zimoun wrote:

> > Cool if Microsoft support long time archive of CRAN packages.  Well, it
> > seems possible to use it as fallback.

> Thanks for the review. You could say the same thing about Software
> Heritage. You never know. I was considering starting a new thread

I was thinking to add MRAN as fallback for CRAN packages.  I will give a look.

The current issue with Software Heritage is that the tarballs story is
not ready yet.  Roughly speaking, now it is not possible to fallback
to SWH for tarballs.


> > Specifying --commit and --channels is redundant.  Other said, the
> > --commit is not necessary because it is already provided by your
> > ’channel-specs.scm’.  But that’s a detail. :-)
>
> Thanks. I thought that --channels referred to the package definitions,
> whereas --commit to the guix version that processes them (yes, I know
> that the definitions are a part of guix, but still).

I do not know if there is a precedence and what happens if the file
passed to the --channels option provides a commit and in the same time
--commit provides another commit.

> Does `environment -C` imply `environment --pure`, as well?

--pure simply clear all the environment variables.
--container runs a fresh container, i.e., nothing inherited.

> > By “chance”, the file ’channel-specs.scm’ and ’manifest.scm’ points to
> > the same commit.
>
> This is intentional. I want to have an environment reflecting the
> d81fb2a state of things, so that’s why I’m consistent with the commit.

My point is the inferior done by your manifest is not necessary for that.
Somehow, you are running this:

  guix time-machine --commit=d81fb2a \
          -- time-machine --commit=d81fb2a \
          -- build r-foreign@fixed

This command line is almost equivalent to the inferior in your manifest file.

> > However, the inferior in ’manifest.scm’ is not necessary.
>
> Why? If it’s not there, you’re facing the hash mismatch problem, aren’t
> you? Please, explain.

No.  Because you need to build 'r-foreign@fixed' and not 'r-foreign'

I bet that removing the inferior still works, for example:

--8<---------------cut here---------------start------------->8---
$ cat manifest.scm
;; Maybe adding modules

(define-public r-foreign-fixed
  (package (inherit r-foreign)
           (version "0.8-75-fixed")
           (source
            (origin
              (method url-fetch)
              (uri
"https://cran.microsoft.com/snapshot/2020-01-27/src/contrib/foreign_0.8-75.tar.gz")
              (sha256
               (base32
                "0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl"))))))

(specifications->manifest
 (list
  "r"
  "r-foreign@0.8-75-fixed"))

$ guix time-machine --commit=d81fb2a \
           -- environment -m manifest.scm
--8<---------------cut here---------------end--------------->8---

(As previously shown in this thread.)


> > Inferiors in ’manifest.scm’ are used when you want to put some packages
> > from different Guix commits in the same profile.  And it is not what you
> > want here; if I understand correctly your problem.
>
> If a tool designed for some purpose turns out to be suitable for other
> purposes, as well, I see no reason for not using it for latter. It’s a
> bit like Unix philosophy.

Obviously!  You can shot in your foot, too. :-)

My comment is to explain that your use of "inferior" is far to be
optimal and not necessary to solve your problem.


> It works for me. If there are better solutions, I’ll be happy to learn.

For example, see here
<https://lists.gnu.org/archive/html/help-guix/2021-01/msg00108.html>


All the best,
simon

Re: guix time-machine, broken hash in an old package definition, a workaround?

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 2579 bytes --]

On Wed, Jan 20, 2021 at 04:03:09PM +0100, zimoun wrote:

> I was thinking to add MRAN as fallback for CRAN packages.  I will give
> a look.

Hi simon,

Would be cool, however for MRAN you also need the snapshot date. Would
it be feasible to extract it from the commit date? There are dates at
CRAN, but for the archived package versions these are “Last modified”.
There is also “Date/Publication:” field in the tarball, but you wouldn’t
trust a tarball with a hash mismatch.

> I bet that removing the inferior still works, for example:
>
> --8<---------------cut here---------------start------------->8---
> $ cat manifest.scm
> ;; Maybe adding modules
>
> (define-public r-foreign-fixed
>   (package (inherit r-foreign)
>            (version "0.8-75-fixed")
>            (source
>             (origin
>               (method url-fetch)
>               (uri
> "https://cran.microsoft.com/snapshot/2020-01-27/src/contrib/foreign_0.8-75.tar.gz")
>               (sha256
>                (base32
>                 "0g4mi101srjbl17ydb2hl3854m3xj0llj6861lfr30sp08nkqavl"))))))
>
> (specifications->manifest
>  (list
>   "r"
>   "r-foreign@0.8-75-fixed"))
>
> $ guix time-machine --commit=d81fb2a \
>            -- environment -m manifest.scm
> --8<---------------cut here---------------end--------------->8---

I cannot check it. This approach works, but for some mysterious reason
it also works when I remove the r-foreign-fixed definition and constrain
the manifest to r. Without the definition, I would expect guix to try
building r-foreign from CRAN. I thought that maybe guix treated
r-foreign@0.8-75 and r-foreign@0.8-75-fixed as exchangeable because of
the same hash, even if the versions and URIs differed, and so did not
try to build r-foreign@0.8-75, but used r-foreign@0.8-75-fixed from the
store. However, with `guix time-machine … -- build r-foreign@0.8-75`,
I’m getting a different output than for `guix time-machine … -- build
r-foreign@0.8-75-fixed`. I tried `guix gc <path to r>` to force the
rebuild, but I got the “still alive” error, even though I had exited the
environment.

I will just trust your expertise on that, and keep your solution. I can
always go back to the inferior if it turns out to fail when I encounter
the hash mismatch problem sometime in the future.

> (As previously shown in this thread.)

Sorry, I somehow hard-coded in my head that those were “game over”’s and
did not review them after MRAN had “changed the game”.

Have a nice weekend,

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: guix time-machine, broken hash in an old package definition, a workaround?

[zimoun]

Hi,

On Fri, 22 Jan 2021 at 12:36, Wiktor Żelazny <wz@freeshell.de> wrote:

> Would be cool, however for MRAN you also need the snapshot date. Would
> it be feasible to extract it from the commit date? There are dates at

Extract date from ~/.cache/guix/checkouts/pj... and author date of the
commit provided at the time-machine should the good one to provide to
MRAN.


> CRAN, but for the archived package versions these are “Last modified”.
> There is also “Date/Publication:” field in the tarball, but you wouldn’t
> trust a tarball with a hash mismatch.

About trust and mismatch, I would say: it depends.  You can still
download the new 'r-foreign@0.85' served by CRAN with the mismatch and
audit by hand.  Well, that's another story.

> I cannot check it. This approach works, but for some mysterious reason
> it also works when I remove the r-foreign-fixed definition and constrain
> the manifest to r. Without the definition, I would expect guix to try
> building r-foreign from CRAN. I thought that maybe guix treated
> r-foreign@0.8-75 and r-foreign@0.8-75-fixed as exchangeable because of
> the same hash, even if the versions and URIs differed, and so did not
> try to build r-foreign@0.8-75, but used r-foreign@0.8-75-fixed from the
> store. However, with `guix time-machine … -- build r-foreign@0.8-75`,
> I’m getting a different output than for `guix time-machine … -- build
> r-foreign@0.8-75-fixed`. I tried `guix gc <path to r>` to force the
> rebuild, but I got the “still alive” error, even though I had exited the
> environment.

To rebuild, the easiest is the option "build --check".

> I will just trust your expertise on that, and keep your solution. I can
> always go back to the inferior if it turns out to fail when I encounter
> the hash mismatch problem sometime in the future.

As I see it, there are 2 options:

 _ option 1: write the package definition with the new MRAN source (or
with the CRAN source but with the new checksum hash), and a manifest
file.  Then run

  $ guix time-machine --commit=d81fb2a \
              -- environment -m manifest.scm

The trick here is to use "--with-input"; somehow the graph has to be
rewritten.  At the manifest level, it is something with
"transform-package-inputs"

 _ option 2: use another package transformation:
"transformation-package-sources".  The new API is simpler with
"package-with-source", so the manifest could contain...

   (packages->manifest
     (cons
        (package-with-source r-foreign
"https://cran.microsoft.com/snapshot/2020-01-27/")
        (map specification->package
           (list "r"
                  "r-another-package"))))

...But the issue is that "package-with-source" was not so simple at
commit d81fb2a time.  And the way is "transformation-package-sources"
even if I am not convinced it is simpler than create by hand the
correct 'r-foreign' package with option 1.


> Have a nice weekend,

Have a nice week-end too! :-)

All the best,
simon