how to customize mirror list used in custom channels?

how to customize mirror list used in custom channels?

[zimoun]

Hi,

I have a custom channel defining a package.  The source of this package
is mirrored.

In ’(guix download)’ module, the list of mirrors are provided by the
list ’%mirrors’.  So I would like to add mine, e.g.,

        (cons
         '(mine
           "https://fast-server.org/sources/"
           "hhtps://kikoo.io/path/to/something/")
         %mirrors)

and then in my package definition,

        (uri "mirror://mine/stuff.tar.gz")

Somehow, I would like to replace the list %mirrors by an extended one.

Well, I am stuck because I am puzzled by the Guile module system. :-)


How to customize the mirror list?

Cheers,
simon

Re: how to customize mirror list used in custom channels?

[Ludovic Courtès]

Hi! :-)

zimoun <zimon.toutoune@gmail.com> skribis:

> In ’(guix download)’ module, the list of mirrors are provided by the
> list ’%mirrors’.  So I would like to add mine, e.g.,
>
>         (cons
>          '(mine
>            "https://fast-server.org/sources/"
>            "hhtps://kikoo.io/path/to/something/")
>          %mirrors)
>
> and then in my package definition,
>
>         (uri "mirror://mine/stuff.tar.gz")
>
> Somehow, I would like to replace the list %mirrors by an extended one.

The easiest solution is to avoid mirror:// and pass a list of regular
URLs:

  (origin
    (method url-fetch)
    (uri (list URL1 URL2 URL3))
    (base32 …))

See for example how ‘qtbase’ does it.

Now, if you really want to extend the set of things recognized, you
could write variant of ‘url-fetch’ that passes a different #:mirrors
argument to ‘built-in-download’.  Maybe ‘url-fetch’ could have a
#:mirrors parameter to simplify it.

But really, the URL list should do the job most of the time.

Ludo’.

Re: how to customize mirror list used in custom channels?

[zimoun]

Hi,

On Sat, 26 Nov 2022 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:

>   (origin
>     (method url-fetch)
>     (uri (list URL1 URL2 URL3))
>     (base32 …))

Neat!  This is nice for one or two packages.


> See for example how ‘qtbase’ does it.

Well, it looks like it is a manual implementation of the Guix mirror
mechanism.


> Now, if you really want to extend the set of things recognized, you
> could write variant of ‘url-fetch’ that passes a different #:mirrors
> argument to ‘built-in-download’.  Maybe ‘url-fetch’ could have a
> #:mirrors parameter to simplify it.

Why is it not possible to extend ’%mirrors?  For instance, it is
possible to extend ’%load-path’.  Why not %mirrors?


Cheers,
simon

Re: how to customize mirror list used in custom channels?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 26 Nov 2022 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Now, if you really want to extend the set of things recognized, you
>> could write variant of ‘url-fetch’ that passes a different #:mirrors
>> argument to ‘built-in-download’.  Maybe ‘url-fetch’ could have a
>> #:mirrors parameter to simplify it.
>
> Why is it not possible to extend ’%mirrors?

What I described above is one way to extend it, just not via ‘set!’.

Ludo’.

Re: how to customize mirror list used in custom channels?

[zimoun]

Hi,

On Mon, 28 Nov 2022 at 11:49, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> On Sat, 26 Nov 2022 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:
>
> [...]
>
>>> Now, if you really want to extend the set of things recognized, you
>>> could write variant of ‘url-fetch’ that passes a different #:mirrors
>>> argument to ‘built-in-download’.  Maybe ‘url-fetch’ could have a
>>> #:mirrors parameter to simplify it.
>>
>> Why is it not possible to extend ’%mirrors?
>
> What I described above is one way to extend it, just not via ‘set!’.

Just to be sure, one way is this extension mechanism,

--8<---------------cut here---------------start------------->8---
(define (qt-urls component version)
  "Return a list of URLs for VERSION of the Qt5 COMPONENT."
  ;; We can't use a mirror:// scheme because these URLs are not exact copies:
  ;; the layout differs between them.
  (list (string-append "https://download.qt.io/official_releases/qt/"
                       (version-major+minor version) "/" version
                       "/submodules/" component "-everywhere-opensource-src-"
                       version ".tar.xz")
        (string-append "https://download.qt.io/official_releases/qt/"
                       (version-major+minor version) "/" version
                       "/submodules/" component "-everywhere-src-"
                       version ".tar.xz")
        (string-append "https://download.qt.io/archive/qt/"
                       (version-major+minor version) "/" version
                       "/submodules/" component "-everywhere-opensource-src-"
                       version ".tar.xz")
        (let ((directory (string-append "qt5" (string-drop component 2))))
          (string-append "http://sources.buildroot.net/" directory "/"
                         component "-everywhere-opensource-src-" version ".tar.xz"))
        (string-append "https://distfiles.macports.org/qt5/"
                       component "-everywhere-opensource-src-" version ".tar.xz")))
--8<---------------cut here---------------end--------------->8---

then,

    (uri (qt-urls name version))

Right?  Well, for what it is worth, it does not appear to me consistent
with the rest.  We could do the same for everything, and remove
"mirror://" after all. :-)

The other one way is to implement a variant of url-fetch method.  What
would be the interface?  Where the extended list of mirrors would be
provided?


Cheers,
simon

Re: how to customize mirror list used in custom channels?

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> Hi,
>
> On Mon, 28 Nov 2022 at 11:49, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi,
>>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> On Sat, 26 Nov 2022 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> [...]
>>
>>>> Now, if you really want to extend the set of things recognized, you
>>>> could write variant of ‘url-fetch’ that passes a different #:mirrors
>>>> argument to ‘built-in-download’.  Maybe ‘url-fetch’ could have a
>>>> #:mirrors parameter to simplify it.
>>>
>>> Why is it not possible to extend ’%mirrors?
>>
>> What I described above is one way to extend it, just not via ‘set!’.
>
> Just to be sure, one way is this extension mechanism,

I was referring to #:mirrors etc. above.

[...]

> The other one way is to implement a variant of url-fetch method.  What
> would be the interface?  Where the extended list of mirrors would be
> provided?

The more I write, the closer I get to an actual implementation.  :-)
If ‘url-fetch’ takes a #:mirrors, that should allow us to extend the set
of mirrors quite conveniently, along the lines of:

  (define (my-url-fetch . args)
    (apply url-fetch (append args '(#:mirrors …))))

HTH!

Ludo’.

Re: how to customize mirror list used in custom channels?

[zimoun]

Hi,

On Tue, 29 Nov 2022 at 09:41, Ludovic Courtès <ludo@gnu.org> wrote:

> The more I write, the closer I get to an actual implementation.  :-)
> If ‘url-fetch’ takes a #:mirrors, that should allow us to extend the set
> of mirrors quite conveniently, along the lines of:
>
>   (define (my-url-fetch . args)
>     (apply url-fetch (append args '(#:mirrors …))))

Well, the ’url-fetch’ defined in (guix build download) already takes
#:mirrors, no?

--8<---------------cut here---------------start------------->8---
(define* (url-fetch url file
                    #:key
                    (timeout 10) (verify-certificate? #t)
                    (mirrors '()) (content-addressed-mirrors '())
                    (disarchive-mirrors '())
                    (hashes '())
                    print-build-trace?)
--8<---------------cut here---------------end--------------->8---

So, I guess you are talking about ’url-fetch*’ from (guix download) and
exported as ’url-fetch’.  Right?  The one which put the bits in the
store. :-)

Does something like that would make sense?

--8<---------------cut here---------------start------------->8---
diff --git a/guix/download.scm b/guix/download.scm
index 2e9ecb43fc..2497ea0f75 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -499,6 +499,7 @@ (define %download-fallback-test
 (define* (url-fetch* url hash-algo hash
                      #:optional name
                      #:key (system (%current-system))
+                     (mirrors %mirrors)
                      (guile (default-guile))
                      executable?)
   "Return a fixed-output derivation that fetches data from URL (a string, or a
@@ -519,7 +520,8 @@ (define file-name
       (_
        (basename url))))
 
-  (let ((uri (and (string? url) (string->uri url))))
+  (let ((uri (and (string? url) (string->uri url)))
+        (mirror-file (plain-file "mirrors" (object->string mirrors))))
     (if (or (and (string? url) (not uri))
             (and uri (memq (uri-scheme uri) '(#f file))))
         (interned-file (if uri (uri-path uri) url)
@@ -540,7 +542,7 @@ (define file-name
                              #:hash-algo hash-algo
                              #:hash hash
                              #:executable? executable?
-                             #:mirrors %mirror-file
+                             #:mirrors mirror-file
                              #:content-addressed-mirrors
                              (match (%download-fallback-test)
                                ((or #f 'content-addressed-mirrors)
--8<---------------cut here---------------end--------------->8---

This change would a world rebuild, right?

Last, note that ’download-to-store’ uses %mirrors so it seems
straightforward here to add ’#:key (mirrors %mirrors)’, I guess.


Cheers,
simon

Re: how to customize mirror list used in custom channels?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Does something like that would make sense?

Yes, that’s the idea.

> diff --git a/guix/download.scm b/guix/download.scm
> index 2e9ecb43fc..2497ea0f75 100644
> --- a/guix/download.scm
> +++ b/guix/download.scm
> @@ -499,6 +499,7 @@ (define %download-fallback-test
>  (define* (url-fetch* url hash-algo hash
>                       #:optional name
>                       #:key (system (%current-system))
> +                     (mirrors %mirrors)
>                       (guile (default-guile))
>                       executable?)
>    "Return a fixed-output derivation that fetches data from URL (a string, or a
> @@ -519,7 +520,8 @@ (define file-name
>        (_
>         (basename url))))
>  
> -  (let ((uri (and (string? url) (string->uri url))))
> +  (let ((uri (and (string? url) (string->uri url)))
> +        (mirror-file (plain-file "mirrors" (object->string mirrors))))

There are subtleties though: we don’t want to create a new <plain-file>
at each call as this would lead to poor performance (lack of
memoization).

> This change would a world rebuild, right?

No, not at all.

Thanks,
Ludo’.