Caching calls to download-to-store?

Caching calls to download-to-store?

[Ricardo Wurmus]

Hi Guix,

the CRAN updater downloads every upstream archive three times.  The
importer memoizes calls to the download procedure, but one download that
is out of reach is the very first.

It is triggered by package-update/url-fetch in (guix upstream).  It uses
download-to-store, which will unconditionally fetch whatever URL it is
given.

I would very much like to parameterize download-to-store, so that we can
optionally cache the results of its invocation.

What do you think?

-- 
Ricardo

Re: Caching calls to download-to-store?

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> the CRAN updater downloads every upstream archive three times.  The
> importer memoizes calls to the download procedure, but one download that
> is out of reach is the very first.
>
> It is triggered by package-update/url-fetch in (guix upstream).  It uses
> download-to-store, which will unconditionally fetch whatever URL it is
> given.

Why three times (and not two)?

> I would very much like to parameterize download-to-store, so that we can
> optionally cache the results of its invocation.

I’m not sure exactly how to do that, but why not.

Another option would be to register GC roots (or mere symlinks) in
~/.cache for things that are downloaded, and then to use
‘If-Modified-Since’ plus some default TTL to determine whether to
re-download things.

HTH,
Ludo’.

Re: Caching calls to download-to-store?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> the CRAN updater downloads every upstream archive three times.  The
>> importer memoizes calls to the download procedure, but one download that
>> is out of reach is the very first.
>>
>> It is triggered by package-update/url-fetch in (guix upstream).  It uses
>> download-to-store, which will unconditionally fetch whatever URL it is
>> given.
>
> Why three times (and not two)?

Not entirely sure, but it's tracked here

   https://issues.guix.gnu.org/48792
   https://issues.guix.gnu.org/44256

I'm already caching some downloads in the updater, but it's not always
working as intended.  (I have an unfinished fix for this locally.)

>> I would very much like to parameterize download-to-store, so that we can
>> optionally cache the results of its invocation.
>
> I’m not sure exactly how to do that, but why not.
>
> Another option would be to register GC roots (or mere symlinks) in
> ~/.cache for things that are downloaded, and then to use
> ‘If-Modified-Since’ plus some default TTL to determine whether to
> re-download things.

That's a good idea.  Currently, we download to arbitrary temporary
directories.  The downloads performed by importers vanish immediately.

-- 
Ricardo

Re: Caching calls to download-to-store?

[Simon Tournier]

Hi,

On Thu, 28 Nov 2024 at 11:21, Ricardo Wurmus <rekado@elephly.net> wrote:

>> Why three times (and not two)?

Yeah three times, weird.

> Not entirely sure, but it's tracked here
>
>    https://issues.guix.gnu.org/48792
>    https://issues.guix.gnu.org/44256
>
> I'm already caching some downloads in the updater, but it's not always
> working as intended.  (I have an unfinished fix for this locally.)

Ah cool!

>>> I would very much like to parameterize download-to-store, so that we can
>>> optionally cache the results of its invocation.
>>
>> I’m not sure exactly how to do that, but why not.
>>
>> Another option would be to register GC roots (or mere symlinks) in
>> ~/.cache for things that are downloaded, and then to use
>> ‘If-Modified-Since’ plus some default TTL to determine whether to
>> re-download things.
>
> That's a good idea.  Currently, we download to arbitrary temporary
> directories.  The downloads performed by importers vanish immediately.

If we populate some ~/.cache/guix then what would trigger the clean up
of such cache?

Cheers,
simon