Package definition hash calculation

Package definition hash calculation

[Zelphir Kaltstahl]

Hello Guix users!

I feel a bit stupid to ask about this topic again, however, to me it is not 
really clear, what I need to do, when calculating the hash of a package, so that 
I can write it in the package definition.

I have a project (https://notabug.org/ZelphirKaltstahl/guile-fslib), which I 
have packaged before, but that was already a year ago or so, and I forgot the 
precise process involving the hashes.

I have the following questions:

(1) When I edit the `guix.scm` file and change the hash in there, make a tarball 
release on notabug, and then run `guix download <tarball of release>`, I get a 
new hash. If I edit the guix.scm file again and repeat the process, I get a new 
hash … endless loop of getting a new hash and changing the file accordingly. My 
guess is, that this is, because `guix download` does not exclude the `guix.scm` 
file. I would have to manually make a `tar.gz` and upload that as a release to 
notabug and then reference that. – Is this correct?

(2) I guess I should be using `guix hash --exclude-vcs --serializer=nar 
--format=??? .` instead, since my package definition makes use of the 
`git-fetch` method of fetching the package. I had totally forgotten about this, 
until I searched in old e-mails, reading old replies to previous questions I 
asked on this mailing list. I think it could be made clearer in the docs, which 
command to use in which case. However, now I am not sure which `--format=` I 
should use. I would guess `base32`, because in my package definition it says 
`(sha256 (base32 "..."))`. Is this correct? Or is the default fine?

(3) What is the recommended way to update a package's source code and then "in 
one go" calculate the hash, update the `guix.scm` and make a proper release, 
which only has the appropriate files in the tarball?

(4) Should a release tarball contain a `guix.scm` package definition? (My guess 
is not, since the hash in that file changes and that would change the tarball. 
Maybe I am overlooking things/magic though.)

I am feeling like I am stuck in what should be a simple process, because I still 
have some points that are unclear to me. I try updating my guide to packaging a 
pure Guile package when I learn new things, so that I can read up next time I 
want to make a release or a new package, but a few things are still missing or 
unclear.

Thank you for all your help!

Best regards,
Zelphir

-- 
repositories: https://notabug.org/ZelphirKaltstahl

Re: Package definition hash calculation

[Julien Lepiller]

When you use guix download, or url-fetch, the hash is computed over the entire file, whether it's a tarball that contains other files or whatever else doesn't matter. You can't exclude files from inside the tarball. It's just the checksum of the file.

What you describes sounds like you're downloading a tarball that's generated from your master instead of a particular commit. So everytime you push a change to guix.scm, it's a new commit and a different tar.gz (different checksum). So you're always chasing after the correct checksum, which won't work.

So you can have a guix.scm in your repo, but it can't refer to a generated tarball from master. Instead, you could make it refer to master and not have to provide a hash like so:

(source (git-checkout (url "https://…")))

No more chasing afcer master :)

On July 9, 2022 1:09:27 PM GMT+02:00, Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote:
>Hello Guix users!
>
>I feel a bit stupid to ask about this topic again, however, to me it is not really clear, what I need to do, when calculating the hash of a package, so that I can write it in the package definition.
>
>I have a project (https://notabug.org/ZelphirKaltstahl/guile-fslib), which I have packaged before, but that was already a year ago or so, and I forgot the precise process involving the hashes.
>
>I have the following questions:
>
>(1) When I edit the `guix.scm` file and change the hash in there, make a tarball release on notabug, and then run `guix download <tarball of release>`, I get a new hash. If I edit the guix.scm file again and repeat the process, I get a new hash … endless loop of getting a new hash and changing the file accordingly. My guess is, that this is, because `guix download` does not exclude the `guix.scm` file. I would have to manually make a `tar.gz` and upload that as a release to notabug and then reference that. – Is this correct?
>
>(2) I guess I should be using `guix hash --exclude-vcs --serializer=nar --format=??? .` instead, since my package definition makes use of the `git-fetch` method of fetching the package. I had totally forgotten about this, until I searched in old e-mails, reading old replies to previous questions I asked on this mailing list. I think it could be made clearer in the docs, which command to use in which case. However, now I am not sure which `--format=` I should use. I would guess `base32`, because in my package definition it says `(sha256 (base32 "..."))`. Is this correct? Or is the default fine?
>
>(3) What is the recommended way to update a package's source code and then "in one go" calculate the hash, update the `guix.scm` and make a proper release, which only has the appropriate files in the tarball?
>
>(4) Should a release tarball contain a `guix.scm` package definition? (My guess is not, since the hash in that file changes and that would change the tarball. Maybe I am overlooking things/magic though.)
>
>I am feeling like I am stuck in what should be a simple process, because I still have some points that are unclear to me. I try updating my guide to packaging a pure Guile package when I learn new things, so that I can read up next time I want to make a release or a new package, but a few things are still missing or unclear.
>
>Thank you for all your help!
>
>Best regards,
>Zelphir
>
>-- 
>repositories: https://notabug.org/ZelphirKaltstahl
>
>

Re: Package definition hash calculation

[Zelphir Kaltstahl]

Hello Julien!

I did create the release tag on the latest commit, which was also the master 
branch. I thought there was some clever trick to get Guix to ignore the 
`guix.scm` file, when calculating the hash sum, but apparently that is not the 
case. I still wonder how I made the first version of the package though. I did 
not know about `git-checkout` as `source` method. Thanks for that!

I am unsure, whether my project must have a `guix.scm` file or not, to be a 
valid Guix package in the end, when I add it to (update the entry in) 
`gnu/packages/guile-xyz.scm`. Maybe a `guix.scm` in the project is not even needed.

Anyway, the idea makes sense to track the master using a checkout without a hash 
sum, so that I don't have the problem of the hash changing any longer. Just not 
sure it will work for updating the package in guix. I will try it.

There is no information about `git-checkout`: 
https://guix.gnu.org/manual/en/html_node/origin-Reference.html Is it elsewhere 
in the docs?

Best regards,
Zelphir

On 7/9/22 13:44, Julien Lepiller wrote:
> When you use guix download, or url-fetch, the hash is computed over the entire 
> file, whether it's a tarball that contains other files or whatever else 
> doesn't matter. You can't exclude files from inside the tarball. It's just the 
> checksum of the file.
>
> What you describes sounds like you're downloading a tarball that's generated 
> from your master instead of a particular commit. So everytime you push a 
> change to guix.scm, it's a new commit and a different tar.gz (different 
> checksum). So you're always chasing after the correct checksum, which won't work.
>
> So you can have a guix.scm in your repo, but it can't refer to a generated 
> tarball from master. Instead, you could make it refer to master and not have 
> to provide a hash like so:
>
> (source (git-checkout (url "https://…")))
>
> No more chasing afcer master :)
>
> On July 9, 2022 1:09:27 PM GMT+02:00, Zelphir Kaltstahl 
> <zelphirkaltstahl@posteo.de> wrote:
>
>     Hello Guix users!
>
>     I feel a bit stupid to ask about this topic again, however, to me it is not really clear, what I need to do, when calculating the hash of a package, so that I can write it in the package definition.
>
>     I have a project (https://notabug.org/ZelphirKaltstahl/guile-fslib),  which I have packaged before, but that was already a year ago or so, and I forgot the precise process involving the hashes.
>
>     I have the following questions:
>
>     (1) When I edit the `guix.scm` file and change the hash in there, make a tarball release on notabug, and then run `guix download <tarball of release>`, I get a new hash. If I edit the guix.scm file again and repeat the process, I get a new hash … endless loop of getting a new hash and changing the file accordingly. My guess is, that this is, because `guix download` does not exclude the `guix.scm` file. I would have to manually make a `tar.gz` and upload that as a release to notabug and then reference that. – Is this correct?
>
>     (2) I guess I should be using `guix hash --exclude-vcs --serializer=nar --format=??? .` instead, since my package definition makes use of the `git-fetch` method of fetching the package. I had totally forgotten about this, until I searched in old e-mails, reading old replies to previous questions I asked on this mailing list. I think it could be made clearer in the docs, which command to use in which case. However, now I am not sure which `--format=` I should use. I would guess `base32`, because in my package definition it says `(sha256 (base32 "..."))`. Is this correct? Or is the default fine?
>
>     (3) What is the recommended way to update a package's source code and then "in one go" calculate the hash, update the `guix.scm` and make a proper release, which only has the appropriate files in the tarball?
>
>     (4) Should a release tarball contain a `guix.scm` package definition? (My guess is not, since the hash in that file changes and that would change the tarball. Maybe I am overlooking things/magic though.)
>
>     I am feeling like I am stuck in what should be a simple process, because I still have some points that are unclear to me. I try updating my guide to packaging a pure Guile package when I learn new things, so that I can read up next time I want to make a release or a new package, but a few things are still missing or unclear.
>
>     Thank you for all your help!
>
>     Best regards,
>     Zelphir
>
>     -- repositories: https://notabug.org/ZelphirKaltstahl
>
-- 
repositories:https://notabug.org/ZelphirKaltstahl

Re: Package definition hash calculation

[Julien Lepiller]

I don't think it's documented.

I use guix.scm in my projects, so it's supported. I don't understand your issue with the hash then, because now it sounds that you're cloning the same commit everytime, so how do you get a different hash?

Le 11 juillet 2022 11:19:25 GMT+02:00, Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> a écrit :
>Hello Julien!
>
>I did create the release tag on the latest commit, which was also the master branch. I thought there was some clever trick to get Guix to ignore the `guix.scm` file, when calculating the hash sum, but apparently that is not the case. I still wonder how I made the first version of the package though. I did not know about `git-checkout` as `source` method. Thanks for that!
>
>I am unsure, whether my project must have a `guix.scm` file or not, to be a valid Guix package in the end, when I add it to (update the entry in) `gnu/packages/guile-xyz.scm`. Maybe a `guix.scm` in the project is not even needed.
>
>Anyway, the idea makes sense to track the master using a checkout without a hash sum, so that I don't have the problem of the hash changing any longer. Just not sure it will work for updating the package in guix. I will try it.
>
>There is no information about `git-checkout`: https://guix.gnu.org/manual/en/html_node/origin-Reference.html Is it elsewhere in the docs?
>
>Best regards,
>Zelphir
>
>On 7/9/22 13:44, Julien Lepiller wrote:
>> When you use guix download, or url-fetch, the hash is computed over the entire file, whether it's a tarball that contains other files or whatever else doesn't matter. You can't exclude files from inside the tarball. It's just the checksum of the file.
>> 
>> What you describes sounds like you're downloading a tarball that's generated from your master instead of a particular commit. So everytime you push a change to guix.scm, it's a new commit and a different tar.gz (different checksum). So you're always chasing after the correct checksum, which won't work.
>> 
>> So you can have a guix.scm in your repo, but it can't refer to a generated tarball from master. Instead, you could make it refer to master and not have to provide a hash like so:
>> 
>> (source (git-checkout (url "https://…")))
>> 
>> No more chasing afcer master :)
>> 
>> On July 9, 2022 1:09:27 PM GMT+02:00, Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote:
>> 
>>     Hello Guix users!
>> 
>>     I feel a bit stupid to ask about this topic again, however, to me it is not really clear, what I need to do, when calculating the hash of a package, so that I can write it in the package definition.
>> 
>>     I have a project (https://notabug.org/ZelphirKaltstahl/guile-fslib),  which I have packaged before, but that was already a year ago or so, and I forgot the precise process involving the hashes.
>> 
>>     I have the following questions:
>> 
>>     (1) When I edit the `guix.scm` file and change the hash in there, make a tarball release on notabug, and then run `guix download <tarball of release>`, I get a new hash. If I edit the guix.scm file again and repeat the process, I get a new hash … endless loop of getting a new hash and changing the file accordingly. My guess is, that this is, because `guix download` does not exclude the `guix.scm` file. I would have to manually make a `tar.gz` and upload that as a release to notabug and then reference that. – Is this correct?
>> 
>>     (2) I guess I should be using `guix hash --exclude-vcs --serializer=nar --format=??? .` instead, since my package definition makes use of the `git-fetch` method of fetching the package. I had totally forgotten about this, until I searched in old e-mails, reading old replies to previous questions I asked on this mailing list. I think it could be made clearer in the docs, which command to use in which case. However, now I am not sure which `--format=` I should use. I would guess `base32`, because in my package definition it says `(sha256 (base32 "..."))`. Is this correct? Or is the default fine?
>> 
>>     (3) What is the recommended way to update a package's source code and then "in one go" calculate the hash, update the `guix.scm` and make a proper release, which only has the appropriate files in the tarball?
>> 
>>     (4) Should a release tarball contain a `guix.scm` package definition? (My guess is not, since the hash in that file changes and that would change the tarball. Maybe I am overlooking things/magic though.)
>> 
>>     I am feeling like I am stuck in what should be a simple process, because I still have some points that are unclear to me. I try updating my guide to packaging a pure Guile package when I learn new things, so that I can read up next time I want to make a release or a new package, but a few things are still missing or unclear.
>> 
>>     Thank you for all your help!
>> 
>>     Best regards,
>>     Zelphir
>> 
>>     -- repositories: https://notabug.org/ZelphirKaltstahl
>> 
>-- 
>repositories:https://notabug.org/ZelphirKaltstahl

Re: Package definition hash calculation

[Zelphir Kaltstahl]

I was updating the `guix.scm` and then calculating the hash, and then updating 
the `guix.scm` and then calculating the hash and so on. It's not because I don't 
understand, how a hash works, but because I thought, that:

* I am calculating the hash in the wrong way. ("Could be there is a way, which 
ignores the `guix.scm` when calculating the hash, but I am doing it wrong.")

* I thought, that I must have a `guix.scm` file in the repository.

* I did not know about `git-checkout` as a `source` method.

Now I know: One can have a guix.scm file in the repo, if it does not change at 
every commit, and the means to do that is `git-checkout`. The guix package 
definition for `gnu/packages/guile-xyz.scm` can be different and can use 
`url-fetch` or `git-fetch`.

Thanks for clearing up my confusion!

Regards,
Zelphir

On 7/11/22 12:18, Julien Lepiller wrote:
> I don't think it's documented.
>
> I use guix.scm in my projects, so it's supported. I don't understand your 
> issue with the hash then, because now it sounds that you're cloning the same 
> commit everytime, so how do you get a different hash?
>
> Le 11 juillet 2022 11:19:25 GMT+02:00, Zelphir Kaltstahl 
> <zelphirkaltstahl@posteo.de> a écrit :
>
>     Hello Julien!
>
>     I did create the release tag on the latest commit, which was also the
>     master branch. I thought there was some clever trick to get Guix to ignore
>     the `guix.scm` file, when calculating the hash sum, but apparently that is
>     not the case. I still wonder how I made the first version of the package
>     though. I did not know about `git-checkout` as `source` method. Thanks for
>     that!
>
>     I am unsure, whether my project must have a `guix.scm` file or not, to be
>     a valid Guix package in the end, when I add it to (update the entry in)
>     `gnu/packages/guile-xyz.scm`. Maybe a `guix.scm` in the project is not
>     even needed.
>
>     Anyway, the idea makes sense to track the master using a checkout without
>     a hash sum, so that I don't have the problem of the hash changing any
>     longer. Just not sure it will work for updating the package in guix. I
>     will try it.
>
>     There is no information about `git-checkout`:
>     https://guix.gnu.org/manual/en/html_node/origin-Reference.html Is it
>     elsewhere in the docs?
>
>     Best regards,
>     Zelphir
>
>     On 7/9/22 13:44, Julien Lepiller wrote:
>>     When you use guix download, or url-fetch, the hash is computed over the
>>     entire file, whether it's a tarball that contains other files or whatever
>>     else doesn't matter. You can't exclude files from inside the tarball.
>>     It's just the checksum of the file.
>>
>>     What you describes sounds like you're downloading a tarball that's
>>     generated from your master instead of a particular commit. So everytime
>>     you push a change to guix.scm, it's a new commit and a different tar.gz
>>     (different checksum). So you're always chasing after the correct
>>     checksum, which won't work.
>>
>>     So you can have a guix.scm in your repo, but it can't refer to a
>>     generated tarball from master. Instead, you could make it refer to master
>>     and not have to provide a hash like so:
>>
>>     (source (git-checkout (url "https://…")))
>>
>>     No more chasing afcer master :)
>>
>>     On July 9, 2022 1:09:27 PM GMT+02:00, Zelphir Kaltstahl
>>     <zelphirkaltstahl@posteo.de> wrote:
>>
>>         Hello Guix users!
>>
>>         I feel a bit stupid to ask about this topic again, however, to me it is not really clear, what I need to do, when calculating the hash of a package, so that I can write it in the package definition.
>>
>>         I have a project (https://notabug.org/ZelphirKaltstahl/guile-fslib),  which I have packaged before, but that was already a year ago or so, and I forgot the precise process involving the hashes.
>>
>>         I have the following questions:
>>
>>         (1) When I edit the `guix.scm` file and change the hash in there, make a tarball release on notabug, and then run `guix download <tarball of release>`, I get a new hash. If I edit the guix.scm file again and repeat the process, I get a new hash … endless loop of getting a new hash and changing the file accordingly. My guess is, that this is, because `guix download` does not exclude the `guix.scm` file. I would have to manually make a `tar.gz` and upload that as a release to notabug and then reference that. – Is this correct?
>>
>>         (2) I guess I should be using `guix hash --exclude-vcs --serializer=nar --format=??? .` instead, since my package definition makes use of the `git-fetch` method of fetching the package. I had totally forgotten about this, until I searched in old e-mails, reading old replies to previous questions I asked on this mailing list. I think it could be made clearer in the docs, which command to use in which case. However, now I am not sure which `--format=` I should use. I would guess `base32`, because in my package definition it says `(sha256 (base32 "..."))`. Is this correct? Or is the default fine?
>>
>>         (3) What is the recommended way to update a package's source code and then "in one go" calculate the hash, update the `guix.scm` and make a proper release, which only has the appropriate files in the tarball?
>>
>>         (4) Should a release tarball contain a `guix.scm` package definition? (My guess is not, since the hash in that file changes and that would change the tarball. Maybe I am overlooking things/magic though.)
>>
>>         I am feeling like I am stuck in what should be a simple process, because I still have some points that are unclear to me. I try updating my guide to packaging a pure Guile package when I learn new things, so that I can read up next time I want to make a release or a new package, but a few things are still missing or unclear.
>>
>>         Thank you for all your help!
>>
>>         Best regards,
>>         Zelphir
>>
>>         -- repositories: https://notabug.org/ZelphirKaltstahl
>>
>     -- 
>     repositories:https://notabug.org/ZelphirKaltstahl
>
-- 
repositories:https://notabug.org/ZelphirKaltstahl