Problem with guix copy

Problem with guix copy

[Emmanuel Medernach]

Hello,

I have problems with guix copy: I have 2 machines Machine_A and 
Machine_B. They both have the same channels.  Machine_B is slow (I don't 
want to recompile there, I would like to reuse the same packages from 
Machine_A).  Machine_A and Machine_B have limited connectivity, 
Machine_A has only ssh access (I cannot use it as a substitute with guix 
publish) and Machine_B has nothing.

I installed bazel on Machine_A:

Machine_A # guix install bazel
Machine_A # guix build bazel
/gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
Machine_A # readlink -f $GUIX_EXTRA_PROFILES/bazel/profile/bin/bazel
/gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0/bin/bazel

I tried guix copy, but I have 2 problems:

First, Machine_B asks for a different hash that the one I have on 
Machine_A (even if both have the same channels):

Machine_B # guix copy --from=Machine_A --dry-run bazel
guix copy: error: no such item on remote host 'Machine_A': 
/gnu/store/6n69a11yjydwhb7p8kb6qg20nj5cxrg1-bazel-6.4.0

Second, when I try to copy the package with the hash from Machine_A I 
get an error:

Machine_B # guix copy --from=Machine_A --dry-run 
/gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
retrieving 1 store item from 'Machine_A'...
guix copy: error: implementation cannot deal with > 32-bit integers

How to use guix copy to transfer the same packages from one machine to 
another ?

Here are my current channels:

(list (channel
         (name 'guix-science)
         (url "https://github.com/guix-science/guix-science.git")
         (branch "master")
         (commit
          "61836bfd9c3a01b696142aabb625b0851d6fd536")
         (introduction
          (make-channel-introduction
           "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
           (openpgp-fingerprint
            "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
        (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (branch "master")
         (commit
          "7319b4d5286d31a9c6a889e81af72308efdaab41")
         (introduction
          (make-channel-introduction
           "9edb3f66fd807b096b48283debdcddccfea34bad"
           (openpgp-fingerprint
            "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))

Thanks for your help

Best regards,

Emmanuel Medernach

Re: Problem with guix copy

[Emmanuel Medernach]

Le 15/03/2024 à 10:14, Emmanuel Medernach a écrit :
> Hello,
>
> I have problems with guix copy: I have 2 machines Machine_A and 
> Machine_B. They both have the same channels.  Machine_B is slow (I 
> don't want to recompile there, I would like to reuse the same packages 
> from Machine_A).  Machine_A and Machine_B have limited connectivity, 
> Machine_A has only ssh access (I cannot use it as a substitute with 
> guix publish) and Machine_B has nothing.
>
> I installed bazel on Machine_A:
>
>
If that could help, here is a minimal manifest and command I use to 
install it in an extra profile:

(specifications->manifest
  '("glibc-locales"
    "bazel"
    ))

# guix package --manifest="$HOME"/manifests/bazel.scm 
--profile="$GUIX_EXTRA_PROFILES"/bazel/profile


> Machine_A # guix build bazel
> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
> Machine_A # readlink -f $GUIX_EXTRA_PROFILES/bazel/profile/bin/bazel
> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0/bin/bazel
>
> I tried guix copy, but I have 2 problems:
>
> First, Machine_B asks for a different hash that the one I have on 
> Machine_A (even if both have the same channels):
>
> Machine_B # guix copy --from=Machine_A --dry-run bazel
> guix copy: error: no such item on remote host 'Machine_A': 
> /gnu/store/6n69a11yjydwhb7p8kb6qg20nj5cxrg1-bazel-6.4.0
>
> Second, when I try to copy the package with the hash from Machine_A I 
> get an error:
>
> Machine_B # guix copy --from=Machine_A --dry-run 
> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
> retrieving 1 store item from 'Machine_A'...
> guix copy: error: implementation cannot deal with > 32-bit integers
>
> How to use guix copy to transfer the same packages from one machine to 
> another ?
>
> Here are my current channels:
>
> (list (channel
>         (name 'guix-science)
>         (url "https://github.com/guix-science/guix-science.git")
>         (branch "master")
>         (commit
>          "61836bfd9c3a01b696142aabb625b0851d6fd536")
>         (introduction
>          (make-channel-introduction
>           "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
>           (openpgp-fingerprint
>            "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
>        (channel
>         (name 'guix)
>         (url "https://git.savannah.gnu.org/git/guix.git")
>         (branch "master")
>         (commit
>          "7319b4d5286d31a9c6a889e81af72308efdaab41")
>         (introduction
>          (make-channel-introduction
>           "9edb3f66fd807b096b48283debdcddccfea34bad"
>           (openpgp-fingerprint
>            "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))
>
> Thanks for your help
>
> Best regards,
>
> Emmanuel Medernach
>

Re: Problem with guix copy

[Emmanuel Medernach]

Le 15/03/2024 à 10:21, Emmanuel Medernach a écrit :
>
> Le 15/03/2024 à 10:14, Emmanuel Medernach a écrit :
>> Hello,
>>
>> I have problems with guix copy: I have 2 machines Machine_A and 
>> Machine_B. They both have the same channels.  Machine_B is slow (I 
>> don't want to recompile there, I would like to reuse the same 
>> packages from Machine_A).  Machine_A and Machine_B have limited 
>> connectivity, Machine_A has only ssh access (I cannot use it as a 
>> substitute with guix publish) and Machine_B has nothing.
>>
>> I installed bazel on Machine_A:
>>
>>
> If that could help, here is a minimal manifest and command I use to 
> install it in an extra profile:
>
> (specifications->manifest
>  '("glibc-locales"
>    "bazel"
>    ))
>
> # guix package --manifest="$HOME"/manifests/bazel.scm 
> --profile="$GUIX_EXTRA_PROFILES"/bazel/profile
>
>
>> Machine_A # guix build bazel
>> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
>> Machine_A # readlink -f $GUIX_EXTRA_PROFILES/bazel/profile/bin/bazel
>> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0/bin/bazel
>>
>> I tried guix copy, but I have 2 problems:
>>
>> First, Machine_B asks for a different hash that the one I have on 
>> Machine_A (even if both have the same channels):
>>
I tried a guix pull as a user and as root on both machines.

Now on both machines guix and guix-daemon used are the same, here are 
both paths:

/gnu/store/m79ld7h62wjbx6kc2jip73lmh8lfc8qs-guix-command
/gnu/store/z642sncd5jgj4bc3p18rdqwdkq30p833-guix-daemon

When I try a guix copy the requested hash is still different from the 
one on Machine_A (and it wishes to recompile it):

Machine_B # guix copy --dry-run --from=Machine_A bazel
The following derivations would be built:
   /gnu/store/0lscmi07b2y0aw6hscbgm2h7nqwc6sb1-bazel-6.4.0.drv
/gnu/store/gssvcsrhq6j2pifavjzw00h8wqlx05km-java-commons-compress-1.21.drv
   /gnu/store/l8z82vbw16nq70w3fnjj4jkwk389lkbh-java-xz-1.9.drv
/gnu/store/h03jjkqcps6xw0x4hgxph4rwynzg7n07-bazel-6.4.0-dist.tar.xz.drv
/gnu/store/var9aniprgbyg2w0i1fhgzsmjbqdv5cp-bazel-6.4.0-dist.zip.drv
/gnu/store/ggg0r1jxp8vvk3sfva6kgc0vsrrlzzb1-zipbomb-bazel-6.4.0-dist.zip.drv
681.5 MB would be downloaded:
   /gnu/store/anzqriqq31ai85ifmn2iiisqyqrs2388-openjdk-11.0.22-jdk
   /gnu/store/qjnlrsvp0j04hhbxh8a4xi7d76bg1s47-openjdk-9.181-jdk
   /gnu/store/ng26zmv0w15cq6zg8v13604labaxllz2-gtk+-2.24.33
   /gnu/store/gab2vx6lrncbpb8w351rwhjwfd30496w-icedtea-3.19.0-jdk
   /gnu/store/wjg1al20skmbczzmli1q8zg6ayq0dhnn-grpc-1.34.0

What am I missing ?

Thanks for your help

Best regards,

Emmanuel Medernach


>> Machine_B # guix copy --from=Machine_A --dry-run bazel
>> guix copy: error: no such item on remote host 'Machine_A': 
>> /gnu/store/6n69a11yjydwhb7p8kb6qg20nj5cxrg1-bazel-6.4.0
>>
>> Second, when I try to copy the package with the hash from Machine_A I 
>> get an error:
>>
>> Machine_B # guix copy --from=Machine_A --dry-run 
>> /gnu/store/5i872l0ijm7vd9vk2knn67ai3daa6q68-bazel-6.4.0
>> retrieving 1 store item from 'Machine_A'...
>> guix copy: error: implementation cannot deal with > 32-bit integers
>>
>> How to use guix copy to transfer the same packages from one machine 
>> to another ?
>>
>> Here are my current channels:
>>
>> (list (channel
>>         (name 'guix-science)
>>         (url "https://github.com/guix-science/guix-science.git")
>>         (branch "master")
>>         (commit
>>          "61836bfd9c3a01b696142aabb625b0851d6fd536")
>>         (introduction
>>          (make-channel-introduction
>>           "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
>>           (openpgp-fingerprint
>>            "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
>>        (channel
>>         (name 'guix)
>>         (url "https://git.savannah.gnu.org/git/guix.git")
>>         (branch "master")
>>         (commit
>>          "7319b4d5286d31a9c6a889e81af72308efdaab41")
>>         (introduction
>>          (make-channel-introduction
>>           "9edb3f66fd807b096b48283debdcddccfea34bad"
>>           (openpgp-fingerprint
>>            "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))
>>
>> Thanks for your help
>>
>> Best regards,
>>
>> Emmanuel Medernach
>>
>

Re: Problem with guix copy

[Andreas Enge]

Hello,

Am Fri, Mar 15, 2024 at 02:58:23PM +0100 schrieb Emmanuel Medernach:
> Now on both machines guix and guix-daemon used are the same, here are both
> paths:
> /gnu/store/m79ld7h62wjbx6kc2jip73lmh8lfc8qs-guix-command
> /gnu/store/z642sncd5jgj4bc3p18rdqwdkq30p833-guix-daemon
> When I try a guix copy the requested hash is still different from the one on
> Machine_A (and it wishes to recompile it):

just a wild guess, but could it be that you did a "guix gc" on Machine A
and that there are grafts in play? In that case the ungrafted package may
have been deleted, and will then be recomputed.

Maybe you could try the same operation adding --no-grafts to the command.

Andreas

Re: Problem with guix copy

[Emmanuel Medernach]

Le 15/03/2024 à 15:10, Andreas Enge a écrit :
> Hello,
>
> Am Fri, Mar 15, 2024 at 02:58:23PM +0100 schrieb Emmanuel Medernach:
>> Now on both machines guix and guix-daemon used are the same, here are both
>> paths:
>> /gnu/store/m79ld7h62wjbx6kc2jip73lmh8lfc8qs-guix-command
>> /gnu/store/z642sncd5jgj4bc3p18rdqwdkq30p833-guix-daemon
>> When I try a guix copy the requested hash is still different from the one on
>> Machine_A (and it wishes to recompile it):
> just a wild guess, but could it be that you did a "guix gc" on Machine A
> and that there are grafts in play? In that case the ungrafted package may
> have been deleted, and will then be recomputed.
>
> Maybe you could try the same operation adding --no-grafts to the command.

Thanks for your idea, here is the result:

Machine_A # guix package --manifest="$HOME"/manifests/bazel.scm 
--no-grafts --no-substitutes
Machine_A # guix package --list-installed | grep bazel
bazel           6.4.0   out 
/gnu/store/8xmcsk5jrkxb06whgj0fif0mrxvhyvj6-bazel-6.4.0

The hash has changed but is still not the one requested by Machine_B 
with guix copy (same message there)

I cannot publish from Machine_A to Machine_B (but I don't know if that 
would be better or not)

Emmanuel


> Andreas
>

Re: Problem with guix copy

[Emmanuel Medernach]

Le 15/03/2024 à 15:37, Emmanuel Medernach a écrit :
>
> Le 15/03/2024 à 15:10, Andreas Enge a écrit :
>> Hello,
>>
>> Am Fri, Mar 15, 2024 at 02:58:23PM +0100 schrieb Emmanuel Medernach:
>>> Now on both machines guix and guix-daemon used are the same, here 
>>> are both
>>> paths:
>>> /gnu/store/m79ld7h62wjbx6kc2jip73lmh8lfc8qs-guix-command
>>> /gnu/store/z642sncd5jgj4bc3p18rdqwdkq30p833-guix-daemon
>>> When I try a guix copy the requested hash is still different from 
>>> the one on
>>> Machine_A (and it wishes to recompile it):
>> just a wild guess, but could it be that you did a "guix gc" on Machine A
>> and that there are grafts in play? In that case the ungrafted package 
>> may
>> have been deleted, and will then be recomputed.
>>
>> Maybe you could try the same operation adding --no-grafts to the 
>> command.
>
> Thanks for your idea, here is the result:
>
> Machine_A # guix package --manifest="$HOME"/manifests/bazel.scm 
> --no-grafts --no-substitutes
> Machine_A # guix package --list-installed | grep bazel
> bazel           6.4.0   out 
> /gnu/store/8xmcsk5jrkxb06whgj0fif0mrxvhyvj6-bazel-6.4.0
>
> The hash has changed but is still not the one requested by Machine_B 
> with guix copy (same message there)
>
> I cannot publish from Machine_A to Machine_B (but I don't know if that 
> would be better or not)
>
Ok for the time being I give up and will compile bazel on Machine_B, but 
I am curious why the hash requested from Machine_B is not the one built 
on Machine_A ?

The problem is that I will have many machines like Machine_B and I would 
like to avoid recompilation of packages on all of them if possible.

Emmanuel

Re: Problem with guix copy

[Simon Tournier]

Hi Emmanuel,

> Machine_B # guix copy --from=Machine_A bazel --dry-run
> The following derivation would be built:
>    /gnu/store/0lscmi07b2y0aw6hscbgm2h7nqwc6sb1-bazel-6.4.0.drv

Could you share this derivation file?  And the one from Machine A?

To be precise, on Machine B, you should get:

        $ guix time-machine -q -C channels.scm -- build --no-grafts bazel -d
        /gnu/store/dnwdzl84cxgk232cwh8843f3dhqmyd69-bazel-6.4.0.drv

And on Machine A, you get another one, right?

For instance, run this command on both machines:

    cp                                                                      \
      $(guix time-machine -q -C channels.scm -- build --no-grafts bazel -d) \
      /tmp/machine-X.drv

where X is A or B, respectively.

These files track exactly how the item is built (derived).  Therefore,
since the hash of the output differs, you should differ somewhere.
Previously, you checked about the inputs, now we need to compare the
complete derivation.


Cheers,
simon

PS: Please note that the content of two store items might be bit-to-bit
    identical with different store item paths (hash).
    
    Consider the example with the exact same source where the difference
    is just one comment somewhere in the code.  Then, their hashes are
    different.  However, once compiled the binaries are identical
    (compiler usually removes comments).

    Since the hash of one input (source) differs, then the store item
    paths are different but the content of these both store items are
    bit-to-bit identical.

    The machinery to track down the origin of the difference is all
    these derivation .drv files.

Re: Problem with guix copy

[Emmanuel Medernach]

Le 19/03/2024 à 11:26, Simon Tournier a écrit :
> Hi Emmanuel,
>
>> Machine_B # guix copy --from=Machine_A bazel --dry-run
>> The following derivation would be built:
>>     /gnu/store/0lscmi07b2y0aw6hscbgm2h7nqwc6sb1-bazel-6.4.0.drv
> Could you share this derivation file?  And the one from Machine A?
>
> To be precise, on Machine B, you should get:
>
>          $ guix time-machine -q -C channels.scm -- build --no-grafts bazel -d
>          /gnu/store/dnwdzl84cxgk232cwh8843f3dhqmyd69-bazel-6.4.0.drv
>
> And on Machine A, you get another one, right?

Hi Simon,

Bazel is now compiled on both machines, but I have similar issue with 
tensorflow

Both have the same derivation file:

Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
--no-grafts tensorflow -d
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

Machine_A # md5sum 
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
c7833f974f217ada62b3eb6cdccee11f 
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv


Machine_B # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
--no-grafts tensorflow -d
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

Machine_B # md5sum 
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
c7833f974f217ada62b3eb6cdccee11f 
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv


The derivation file and channels are both the same on both machines.

But I cannot copy tensorflow from Machine_A to Machine_B:

Machine_A # guix package --list-installed | grep tensorflow
tensorflow              2.13.1          out 
/gnu/store/jghvlb5dz4sy1p0cd1qx552r1ldj33wi-tensorflow-2.13.1

Machine_B # guix copy --from=<Machine_A> tensorflow --dry-run
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 
100.0%
The following derivations would be built:
/gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
/gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv
/gnu/store/h3zcigb52z7m6yyjl785w3a5ls1cjjc7-python-jax-0.4.20.drv
/gnu/store/h9m0ix80glph19k4pfqh130nmkil8f39-python-jaxlib-0.4.20.drv
/gnu/store/m4icdavpvbryliz34ai8817bb1d3l3xr-python-jaxlib-0.4.20.drv


Without dry-run he would try to recompile tensorflow on Machine_B which 
results in a memory crash.

Am I doing something wrong ?

Why 'guix copy' derivation is 
/gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv 
instead of 
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv ?

Cheers,

Emmanuel


> For instance, run this command on both machines:
>
>      cp                                                                      \
>        $(guix time-machine -q -C channels.scm -- build --no-grafts bazel -d) \
>        /tmp/machine-X.drv
>
> where X is A or B, respectively.
>
> These files track exactly how the item is built (derived).  Therefore,
> since the hash of the output differs, you should differ somewhere.
> Previously, you checked about the inputs, now we need to compare the
> complete derivation.
>
>
> Cheers,
> simon
>
> PS: Please note that the content of two store items might be bit-to-bit
>      identical with different store item paths (hash).
>      
>      Consider the example with the exact same source where the difference
>      is just one comment somewhere in the code.  Then, their hashes are
>      different.  However, once compiled the binaries are identical
>      (compiler usually removes comments).
>
>      Since the hash of one input (source) differs, then the store item
>      paths are different but the content of these both store items are
>      bit-to-bit identical.
>
>      The machinery to track down the origin of the difference is all
>      these derivation .drv files.

Re: Problem with guix copy

[Emmanuel Medernach]

Le 19/03/2024 à 16:40, Emmanuel Medernach a écrit :
>
> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- 
> build --no-grafts tensorflow -d
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>
> Machine_A # md5sum 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
> c7833f974f217ada62b3eb6cdccee11f 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>

> But I cannot copy tensorflow from Machine_A to Machine_B:
>
> Machine_A # guix package --list-installed | grep tensorflow
> tensorflow              2.13.1          out 
> /gnu/store/jghvlb5dz4sy1p0cd1qx552r1ldj33wi-tensorflow-2.13.1
>
> Machine_B # guix copy --from=<Machine_A> tensorflow --dry-run
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> substitute: updating substitutes from 
> 'https://bordeaux.guix.gnu.org'... 100.0%
> The following derivations would be built:
> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
> /gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv 
>
> /gnu/store/h3zcigb52z7m6yyjl785w3a5ls1cjjc7-python-jax-0.4.20.drv
> /gnu/store/h9m0ix80glph19k4pfqh130nmkil8f39-python-jaxlib-0.4.20.drv
> /gnu/store/m4icdavpvbryliz34ai8817bb1d3l3xr-python-jaxlib-0.4.20.drv
>

>
> Am I doing something wrong ?
>
> Why 'guix copy' derivation is 
> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv 
> instead of 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv ?
>
Here is the diff between the 2 derivations:

1,2c1,2
< 
Derive([("out","/gnu/store/yhr2vxh26acz3pzpjlph4f82ffrw6icv-tensorflow-2.13.1","",""
< 
),("python","/gnu/store/fyjpsvfdmkdimd1vx6d99ik01z3x3j93-tensorflow-2.13.1-python","",""
---
 > 
Derive([("out","/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1","",""
 > 
),("python","/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python","",""
42a43
 > 
),("/gnu/store/50vi4ywi79irbg71l814xqkww4k1afh2-python-jaxlib-0.4.20.drv",["out"]
61a63
 > 
),("/gnu/store/8avai81995rj5h7906nzgyrnrpvqi6if-tensorflow-2.13.1-bazel-deps.tar.xz.drv",["out"]
64d65
< ),("/gnu/store/8jfr297mkpv698d4y1cq270a7f03wyh6-bazel-6.3.2.drv",["out"]
75a77
 > ),("/gnu/store/an1hq0y3npd66plwghgnccm4sl7lyvka-bazel-6.3.2.drv",["out"]
111d112
< 
),("/gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv",["out"]
123d123
< 
),("/gnu/store/h3zcigb52z7m6yyjl785w3a5ls1cjjc7-python-jax-0.4.20.drv",["out"]
127d126
< 
),("/gnu/store/h9m0ix80glph19k4pfqh130nmkil8f39-python-jaxlib-0.4.20.drv",["out"]
167a167
 > 
),("/gnu/store/lvbjhnrdgl5i5z558b1zpvj4vjqzdm4g-python-jax-0.4.20.drv",["out"]
257,258c257,258
< 
)],["/gnu/store/a5pi7l07irynidrfkiw7x44s8angi2fh-tensorflow-2.13.1-builder","/gnu/store/wkdx6qdj2aikwg6xygymz65ck978xym6-module-import"],"x86_64-linux","/gnu/store/g8p09w6r78hhkl2rv1747pcp9zbk6fxv-guile-3.0.9/bin/guile",["--no-auto-compile","-L","/gnu/store/wkdx6qdj2aikwg6xygymz65ck978xym6-module-import","-C","/gnu/store/yj6xd6zqbn32zks8cg2nm1sw537zbcsy-module-import-compiled","/gnu/store/a5pi7l07irynidrfkiw7x44s8angi2fh-tensorflow-2.13.1-builder"],[("out","/gnu/store/yhr2vxh26acz3pzpjlph4f82ffrw6icv-tensorflow-2.13.1"
< 
),("python","/gnu/store/fyjpsvfdmkdimd1vx6d99ik01z3x3j93-tensorflow-2.13.1-python"
---
 > 
)],["/gnu/store/r200hqk2jn68qr5wqj6762dpqbns3vqj-tensorflow-2.13.1-builder","/gnu/store/wkdx6qdj2aikwg6xygymz65ck978xym6-module-import"],"x86_64-linux","/gnu/store/g8p09w6r78hhkl2rv1747pcp9zbk6fxv-guile-3.0.9/bin/guile",["--no-auto-compile","-L","/gnu/store/wkdx6qdj2aikwg6xygymz65ck978xym6-module-import","-C","/gnu/store/yj6xd6zqbn32zks8cg2nm1sw537zbcsy-module-import-compiled","/gnu/store/r200hqk2jn68qr5wqj6762dpqbns3vqj-tensorflow-2.13.1-builder"],[("out","/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1"
 > 
),("python","/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python"

> Cheers,
>
> Emmanuel

Re: Problem with guix copy

[Simon Tournier]

Hi Emmanuel,

On mar., 19 mars 2024 at 16:40, Emmanuel Medernach <Emmanuel.Medernach@iphc.cnrs.fr> wrote:

> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
> --no-grafts tensorflow -d
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>
> Machine_A # md5sum 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
> c7833f974f217ada62b3eb6cdccee11f 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>
> Machine_B # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
> --no-grafts tensorflow -d
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>
> Machine_B # md5sum 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
> c7833f974f217ada62b3eb6cdccee11f 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

The 2 derivations are the exact same.  Since they build tensorflow
without grafts, it means the issue comes from grafts, no?


> Machine_A # guix package --list-installed | grep tensorflow
> tensorflow              2.13.1          out 
> /gnu/store/jghvlb5dz4sy1p0cd1qx552r1ldj33wi-tensorflow-2.13.1

This information is useless, I think.  What is needed is the derivation
for grafting.  Something like:

Machine_A# guix time-machine -q -C ~/.config/guix/channels.scm -- build tensorflow -d



> Machine_B # guix copy --from=<Machine_A> tensorflow --dry-run
[…]
> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv

[...]

> Without dry-run he would try to recompile tensorflow on Machine_B which 
> results in a memory crash.

Yeah, that’s expected because it needs the non-grafted item.


> Why 'guix copy' derivation is 
> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv 
> instead of 
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv ?

Well, on Machine_A, these

    # guix build tensorflow -d --without-grafts
    # guix build tensorflow -d

should produce two different derivations.  Right?  On my machine, I get:

--8<---------------cut here---------------start------------->8---
$ guix time-machine -q -C channels.scm -- build tensorflow@2 -d --no-grafts -n
  /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

$ guix time-machine -q -C channels.scm -- build tensorflow@2 -d -n
  /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
--8<---------------cut here---------------end--------------->8---


What does “guix copy --no-grafts”?


Well, to be sure we are on the same wavelength: there are 4 derivations,

 + Machine_A: the real build (--no-grafts) and the grafted
 + Machine_B: idem and idem

So the debug seems:

 + check the no-grafts derivations are the same on both machines
 + check the grafted derivations are the same on both machines
 + check that “guix copy” copy both (no-grafts and grafted)

Emacs provides a mode for easing the manipulation of derivation and
Scheme builder files.  If you do not use Emacs, “guix gc” can be helpful
for tracking the derivations, I guess.

Cheers,
simon

Re: Problem with guix copy

[Emmanuel Medernach]

Le 21/03/2024 à 23:19, Simon Tournier a écrit :
> Hi Emmanuel,
>
> On mar., 19 mars 2024 at 16:40, Emmanuel Medernach <Emmanuel.Medernach@iphc.cnrs.fr> wrote:
>
>> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build
>> --no-grafts tensorflow -d
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>>
>> Machine_A # md5sum
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>> c7833f974f217ada62b3eb6cdccee11f
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>>
>> Machine_B # guix time-machine -q -C ~/.config/guix/channels.scm -- build
>> --no-grafts tensorflow -d
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>>
>> Machine_B # md5sum
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>> c7833f974f217ada62b3eb6cdccee11f
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
> The 2 derivations are the exact same.  Since they build tensorflow
> without grafts, it means the issue comes from grafts, no?
>
>
>> Machine_A # guix package --list-installed | grep tensorflow
>> tensorflow              2.13.1          out
>> /gnu/store/jghvlb5dz4sy1p0cd1qx552r1ldj33wi-tensorflow-2.13.1
> This information is useless, I think.  What is needed is the derivation
> for grafting.  Something like:
>
> Machine_A# guix time-machine -q -C ~/.config/guix/channels.scm -- build tensorflow -d
>
>
>
>> Machine_B # guix copy --from=<Machine_A> tensorflow --dry-run
> […]
>> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
> [...]
>
>> Without dry-run he would try to recompile tensorflow on Machine_B which
>> results in a memory crash.
> Yeah, that’s expected because it needs the non-grafted item.
>
>
>> Why 'guix copy' derivation is
>> /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
>> instead of
>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv ?
> Well, on Machine_A, these
>
>      # guix build tensorflow -d --without-grafts
>      # guix build tensorflow -d
>
> should produce two different derivations.  Right?  On my machine, I get:
>
> --8<---------------cut here---------------start------------->8---
> $ guix time-machine -q -C channels.scm -- build tensorflow@2 -d --no-grafts -n
>    /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv
>
> $ guix time-machine -q -C channels.scm -- build tensorflow@2 -d -n
>    /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
> --8<---------------cut here---------------end--------------->8---
>
>
> What does “guix copy --no-grafts”?
>
>
> Well, to be sure we are on the same wavelength: there are 4 derivations,
>
>   + Machine_A: the real build (--no-grafts) and the grafted
>   + Machine_B: idem and idem
>
> So the debug seems:
>
>   + check the no-grafts derivations are the same on both machines
>   + check the grafted derivations are the same on both machines
>   + check that “guix copy” copy both (no-grafts and grafted)
>
> Emacs provides a mode for easing the manipulation of derivation and
> Scheme builder files.  If you do not use Emacs, “guix gc” can be helpful
> for tracking the derivations, I guess.

Hi Simon,

Here are the results:

Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
tensorflow -d --no-grafts
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
tensorflow --no-grafts
/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1
/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python

Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
tensorflow -d
/gnu/store/vivxpmrpkh1wq5pnw28vq9c51qd3vwf6-tensorflow-2.13.1.drv

Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build 
tensorflow
/gnu/store/jghvlb5dz4sy1p0cd1qx552r1ldj33wi-tensorflow-2.13.1
/gnu/store/1pzdf2rfifgwdw92g8438mhqvzx948s5-tensorflow-2.13.1-python

For  all  of  this  store elements  guix  copy  on
Machine B  tried to recompile tensorflow.  I ended
up  to copy  the whole  profile, this  copy worked
fine:

Machine_B # guix copy --from=MachineA /gnu/store/...-profile
Machine_B # guix package --list-installed | grep tensorflow
tensorflow              2.13.1          out 
/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1

For now I will copy whole profile instead of store
elements.

Thanks for your help,

Cheers,

Emmanuel

> Cheers,
> simon

Re: Problem with guix copy

[Simon Tournier]

Hi Emmanuel,

On lun., 25 mars 2024 at 11:13, Emmanuel Medernach <Emmanuel.Medernach@iphc.cnrs.fr> wrote:

>>> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build --no-grafts tensorflow -d
>>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

>>> Machine_B # guix time-machine -q -C ~/.config/guix/channels.scm -- build --no-grafts tensorflow -d
>>> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

[...]

> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build tensorflow -d --no-grafts
> /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv

Well, If I open the derivation
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv the
header reads:

--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1","","")
  ,("python","/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python","","")]
--8<---------------cut here---------------end--------------->8---

And you get:

> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build tensorflow --no-grafts
> /gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1
> /gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python

Therefore, all seems consistent.  IIUC, it is also consistent on both
machines (see quote above from earlier message).

It means that the manual “--no-grafts” is OK.


The question is thus, is this derivation:

> Machine_A # guix time-machine -q -C ~/.config/guix/channels.scm -- build tensorflow -d
> /gnu/store/vivxpmrpkh1wq5pnw28vq9c51qd3vwf6-tensorflow-2.13.1.drv

built using
/gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv or
using another one?  Anyway. :-)


Now, if I run on my laptop:

--8<---------------cut here---------------start------------->8---
$ guix time-machine -q -C channels.scm -- build tensorflow@2.13 -n
[...]
The following derivations would be built:
  /gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
  /gnu/store/8jfr297mkpv698d4y1cq270a7f03wyh6-bazel-6.3.2.drv
  /gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv
  /gnu/store/h3zcigb52z7m6yyjl785w3a5ls1cjjc7-python-jax-0.4.20.drv
  /gnu/store/h9m0ix80glph19k4pfqh130nmkil8f39-python-jaxlib-0.4.20.drv
  /gnu/store/m4icdavpvbryliz34ai8817bb1d3l3xr-python-jaxlib-0.4.20.drv
  /gnu/store/ha0hl2j607g0wanxqbkwrcfy6yz3k2xq-python-jaxlib-0.4.20-bazel-deps.tar.xz.drv
--8<---------------cut here---------------end--------------->8---

the output mentions another no-grafts derivation, i.e.,
/gnu/store/4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv.

This derivation 4g2whbzgn5nqs55x9iqgg23jdapf1srk-tensorflow-2.13.1.drv
is not the derivation for grafting but the derivation for building the
no-grafts.

Well, I think the issue with “guix copy” is coming from something
similar.  There is a mismatch between what is computed. Let give a look.

Let transform the one-line file to multi-lines:

--8<---------------cut here---------------start------------->8---
$ cat /gnu/store/s6fm43503ra6yxclqvk80gfiw4zxbp91-tensorflow-2.13.1.drv | sed 's/,/\n,/g' > /tmp/manual
--8<---------------cut here---------------end--------------->8---

And diff the both files (see complete raw below).  Inline comments…

        2c2
        < ,"/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1"
        ---
        > ,"/gnu/store/yhr2vxh26acz3pzpjlph4f82ffrw6icv-tensorflow-2.13.1"
        6c6
        < ,"/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python"
        ---
        > ,"/gnu/store/fyjpsvfdmkdimd1vx6d99ik01z3x3j93-tensorflow-2.13.1-python"

That’s expected.  Because this hash is the result of all the others,
i.e., something that comes next; for instance,

        90,91d89
        < ,("/gnu/store/50vi4ywi79irbg71l814xqkww4k1afh2-python-jaxlib-0.4.20.drv"
        < ,["out"])
        130,131d127
        < ,("/gnu/store/8avai81995rj5h7906nzgyrnrpvqi6if-tensorflow-2.13.1-bazel-deps.tar.xz.drv"
        < ,["out"])
        135a132,133
        > ,("/gnu/store/8jfr297mkpv698d4y1cq270a7f03wyh6-bazel-6.3.2.drv"
        > ,["out"])
        158,159d155
        < ,("/gnu/store/an1hq0y3npd66plwghgnccm4sl7lyvka-bazel-6.3.2.drv"
        < ,["out"])
        229a226,227
        > ,("/gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv"
        > ,["out"])

Hum, it seems:

 + the order is different;
 + the derivations are also different

Ok, the order might be one the problem.  And that could be a bug for
“guix copy”.  And because the derivations are not the same, then the
item is different.

Let consider tensorflow-2.13.1-bazel-deps.tar which is “fixed-output”:
it means the hash (NAR SHA256) is known beforehand.  Therefore the hash
of the store item should be same.  Well, let open these two derivations;
they both read:

        Derive
        ([("out","/gnu/store/rdpxkb8gl0ym0vrz8viklc0rwnjkn7cl-tensorflow-2.13.1-bazel-deps.tar.xz",
        "sha256","487a1b6685767e8d9fc0a23a2e82d2d7be305e1bd5aeaa4acd1599bc92e3142f")]

It means the derivations are different but their result is the exact
same item.


Well, I will do further tests. :-)


Cheers,
simon


--8<---------------cut here---------------start------------->8---
2c2
< ,"/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1"
---
> ,"/gnu/store/yhr2vxh26acz3pzpjlph4f82ffrw6icv-tensorflow-2.13.1"
6c6
< ,"/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python"
---
> ,"/gnu/store/fyjpsvfdmkdimd1vx6d99ik01z3x3j93-tensorflow-2.13.1-python"
90,91d89
< ,("/gnu/store/50vi4ywi79irbg71l814xqkww4k1afh2-python-jaxlib-0.4.20.drv"
< ,["out"])
130,131d127
< ,("/gnu/store/8avai81995rj5h7906nzgyrnrpvqi6if-tensorflow-2.13.1-bazel-deps.tar.xz.drv"
< ,["out"])
135a132,133
> ,("/gnu/store/8jfr297mkpv698d4y1cq270a7f03wyh6-bazel-6.3.2.drv"
> ,["out"])
158,159d155
< ,("/gnu/store/an1hq0y3npd66plwghgnccm4sl7lyvka-bazel-6.3.2.drv"
< ,["out"])
229a226,227
> ,("/gnu/store/g90vfnx128ifhgy0n1zmlscjrn57l8xm-tensorflow-2.13.1-bazel-deps.tar.xz.drv"
> ,["out"])
251a250,251
> ,("/gnu/store/h3zcigb52z7m6yyjl785w3a5ls1cjjc7-python-jax-0.4.20.drv"
> ,["out"])
257a258,259
> ,("/gnu/store/h9m0ix80glph19k4pfqh130nmkil8f39-python-jaxlib-0.4.20.drv"
> ,["out"])
338,339d339
< ,("/gnu/store/lvbjhnrdgl5i5z558b1zpvj4vjqzdm4g-python-jax-0.4.20.drv"
< ,["out"])
519c519
< ,["/gnu/store/r200hqk2jn68qr5wqj6762dpqbns3vqj-tensorflow-2.13.1-builder"
---
> ,["/gnu/store/a5pi7l07irynidrfkiw7x44s8angi2fh-tensorflow-2.13.1-builder"
528c528
< ,"/gnu/store/r200hqk2jn68qr5wqj6762dpqbns3vqj-tensorflow-2.13.1-builder"]
---
> ,"/gnu/store/a5pi7l07irynidrfkiw7x44s8angi2fh-tensorflow-2.13.1-builder"]
530c530
< ,"/gnu/store/3mq3q11ripx53xg60shbshk7cr470yfx-tensorflow-2.13.1")
---
> ,"/gnu/store/yhr2vxh26acz3pzpjlph4f82ffrw6icv-tensorflow-2.13.1")
532c532
< ,"/gnu/store/qc234r07i3ndgrpa0fdkpkbg4d20h7i1-tensorflow-2.13.1-python")])
\ No newline at end of file
---
> ,"/gnu/store/fyjpsvfdmkdimd1vx6d99ik01z3x3j93-tensorflow-2.13.1-python")])
\ No newline at end of file
--8<---------------cut here---------------end--------------->8---