What's the state of (guix build download-nar)?

What's the state of (guix build download-nar)?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

Hey!

As part of the work on bordeaux.guix.gnu.org, I've stumbled upon (guix
build download-nar). I'm not sure I understand it very well, but it
relates to downloading nars from berlin.guix.gnu.org.

I guess this raises two things in my mind. I'm not sure this'll work
well given the not so recent changes to ci.guix.gnu.org. This module
looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
nars are available, and gzipped ones are no longer available.

The second question is around the relation to bordeaux.guix.gnu.org, is
it worth trying to support fetching nars from bordeaux.guix.gnu.org
here, and what would be the best way of doing that? It wouldn't be too
hard to add support for decomressed nars I think if that's a good
approach?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: What's the state of (guix build download-nar)?

[Ludovic Courtès]

Hi Chris!

Christopher Baines <mail@cbaines.net> skribis:

> I guess this raises two things in my mind. I'm not sure this'll work
> well given the not so recent changes to ci.guix.gnu.org. This module
> looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
> nars are available, and gzipped ones are no longer available.
>
> The second question is around the relation to bordeaux.guix.gnu.org, is
> it worth trying to support fetching nars from bordeaux.guix.gnu.org
> here, and what would be the best way of doing that? It wouldn't be too
> hard to add support for decomressed nars I think if that's a good
> approach?

Uh, that module needs love indeed.

Currently it’s used by some of the (guix VCS-download) modules.  I think
we should just update to (1) use lzip instead of gzip, and (2) have it
check ci.guix.gnu.org + bordeaux.guix.gnu.org.

Would you like to give it a try?

Thanks,
Ludo’.

Re: What's the state of (guix build download-nar)?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi Chris!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I guess this raises two things in my mind. I'm not sure this'll work
>> well given the not so recent changes to ci.guix.gnu.org. This module
>> looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
>> nars are available, and gzipped ones are no longer available.
>>
>> The second question is around the relation to bordeaux.guix.gnu.org, is
>> it worth trying to support fetching nars from bordeaux.guix.gnu.org
>> here, and what would be the best way of doing that? It wouldn't be too
>> hard to add support for decomressed nars I think if that's a good
>> approach?
>
> Uh, that module needs love indeed.
>
> Currently it’s used by some of the (guix VCS-download) modules.  I think
> we should just update to (1) use lzip instead of gzip, and (2) have it
> check ci.guix.gnu.org + bordeaux.guix.gnu.org.
>
> Would you like to give it a try?

That approach sounds good to me, I'll have a look at drafting a patch.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: What's the state of (guix build download-nar)?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Chris!
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> I guess this raises two things in my mind. I'm not sure this'll work
>>> well given the not so recent changes to ci.guix.gnu.org. This module
>>> looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
>>> nars are available, and gzipped ones are no longer available.
>>>
>>> The second question is around the relation to bordeaux.guix.gnu.org, is
>>> it worth trying to support fetching nars from bordeaux.guix.gnu.org
>>> here, and what would be the best way of doing that? It wouldn't be too
>>> hard to add support for decomressed nars I think if that's a good
>>> approach?
>>
>> Uh, that module needs love indeed.
>>
>> Currently it’s used by some of the (guix VCS-download) modules.  I think
>> we should just update to (1) use lzip instead of gzip, and (2) have it
>> check ci.guix.gnu.org + bordeaux.guix.gnu.org.
>>
>> Would you like to give it a try?
>
> That approach sounds good to me, I'll have a look at drafting a patch.

I've sent a patch for this now: https://issues.guix.gnu.org/62129

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: What's the state of (guix build download-nar)?

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Chris!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I guess this raises two things in my mind. I'm not sure this'll work
>> well given the not so recent changes to ci.guix.gnu.org. This module
>> looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
>> nars are available, and gzipped ones are no longer available.
>>
>> The second question is around the relation to bordeaux.guix.gnu.org, is
>> it worth trying to support fetching nars from bordeaux.guix.gnu.org
>> here, and what would be the best way of doing that? It wouldn't be too
>> hard to add support for decomressed nars I think if that's a good
>> approach?
>
> Uh, that module needs love indeed.
>
> Currently it’s used by some of the (guix VCS-download) modules.  I think
> we should just update to (1) use lzip instead of gzip, and (2) have it
> check ci.guix.gnu.org + bordeaux.guix.gnu.org.

How about using zstd?  I'm proposing it instead of lzip, because long
term, I think we want to reduce the size of our storage requirements and
offer a single compression type for our NARs, which zstd would be ideal
(it's faster and compresses close enough to lzip).

-- 
Thanks,
Maxim

Re: What's the state of (guix build download-nar)?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Chris!
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> I guess this raises two things in my mind. I'm not sure this'll work
>>> well given the not so recent changes to ci.guix.gnu.org. This module
>>> looks to rely on gzipped or uncompressed nars. I'm not sure uncompressed
>>> nars are available, and gzipped ones are no longer available.
>>>
>>> The second question is around the relation to bordeaux.guix.gnu.org, is
>>> it worth trying to support fetching nars from bordeaux.guix.gnu.org
>>> here, and what would be the best way of doing that? It wouldn't be too
>>> hard to add support for decomressed nars I think if that's a good
>>> approach?
>>
>> Uh, that module needs love indeed.
>>
>> Currently it’s used by some of the (guix VCS-download) modules.  I think
>> we should just update to (1) use lzip instead of gzip, and (2) have it
>> check ci.guix.gnu.org + bordeaux.guix.gnu.org.
>
> How about using zstd?  I'm proposing it instead of lzip, because long
> term, I think we want to reduce the size of our storage requirements and
> offer a single compression type for our NARs, which zstd would be ideal
> (it's faster and compresses close enough to lzip).

I haven't looked at the code, but I don't know of a specific reason why
multiple compression options can't be supported here.

Anyway, at least for bordeaux.guix.gnu.org (which already just stores a
single compression for nars), supporting lzip is a must as that's the
compression used. I'm not sure there's a good reason to change that,
although that does depend on how offering some zstd compressed nars
plays out in terms of performance for users.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: What's the state of (guix build download-nar)?

[Simon Tournier]

Hi,

On jeu., 09 févr. 2023 at 09:16, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> Currently it’s used by some of the (guix VCS-download) modules.  I think
>> we should just update to (1) use lzip instead of gzip, and (2) have it
>> check ci.guix.gnu.org + bordeaux.guix.gnu.org.
>
> How about using zstd?  I'm proposing it instead of lzip, because long
> term, I think we want to reduce the size of our storage requirements and
> offer a single compression type for our NARs, which zstd would be ideal
> (it's faster and compresses close enough to lzip).

I guess it’s the same direction as this thread [1],

        When substitute download + decompression is CPU-bound
        Ludovic Courtès <ludo@gnu.org>
        Mon, 14 Dec 2020 23:20:17 +0100
        id:87im94qbby.fsf@gnu.org

especially this Guillaume’s message [2] comparing various methods.  The
possible agenda [3] concluding the thread was, quoting:

        We could do that.  I suppose a possible agenda would be:

          1. Start providing zstd susbstitutes anytime.  However, most clients
             will keep choosing lzip because it usually compresses better.

          2. After the next release, stop providing lzip substitutes and provide
             only gzip + zstd-19.

        This option has the advantage that it wouldn’t break any installation.
        It’s not as nice as the ability to choose a download strategy, as we
        discussed earlier, but implementing that download strategy sounds
        tricky.

Well, to be honest, I am a bit lost about the compression methods;
especially when also considering this old blog post [4].

As the subject of this thread is asking: “What's the state of (guix
build download-nar)?” ;-)

1: https://yhetil.org/guix/87im94qbby.fsf@gnu.org/#r
2: https://yhetil.org/guix/87ft3d2fge.fsf@yamatai
3: https://yhetil.org/guix/87bld9j651.fsf@gnu.org
4: https://guix.gnu.org/en/blog/2019/substitutes-are-now-available-as-lzip/


Cheers,
simon