[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[zimoun]

Format discussed here <https://forge.softwareheritage.org/D2025#51269>.

* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
 website/apps/packages/builder.scm | 62 +++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..5279096 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; Initially written by sirgazil
 ;;; who waives all copyright interest on this file.
@@ -37,6 +38,8 @@
   #:use-module (haunt page)
   #:use-module (haunt utils)
   #:use-module (srfi srfi-1)
+  #:use-module ((web uri) #:select (string->uri uri->string uri-scheme))
+  #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
@@ -70,6 +73,7 @@
   (flatten
    (list
     (index-builder)
+    (sources-json-builder)
     (packages-json-builder)
     (packages-builder)
     (package-list-builder))))
@@ -84,6 +88,64 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
+(define (sources-json-builder)
+  "Return a JSON page listing all the sources."
+  (define (origin->json origin)
+    (define method
+      (origin-method origin))
+
+    (define uri                         ;represented as string
+      (origin-uri origin))
+
+    (define (mirror->url uri)
+      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
+
+    (define (resolve urls)
+      (let* ((url (car urls))
+             (uri (string->uri url))
+             (rest (cdr urls)))
+        (case (uri-scheme uri)
+          ((mirror) (mirror->url uri))
+          ((http) url)
+          ((https) url)
+          (else
+           (if (null? rest)
+               url
+               (resolve rest))))))
+
+    `((type . ,(cond ((eq? url-fetch method) 'url)
+                     ((eq? git-fetch method) 'git)
+                     ((eq? svn-fetch method) 'svn)
+                     (else                   #nil)))
+      ,@(cond ((eq? url-fetch method)
+               `(("url" . ,(match uri
+				  ((? string? url) (mirror->url (string->uri url)))
+				  ((urls ...) (resolve urls))))))
+              ((eq? git-fetch method)
+               `(("git_url" . ,(git-reference-url uri))))
+              ((eq? svn-fetch method)
+               `(("svn_url" . ,(svn-reference-url uri))))
+              (else '()))
+      ,@(if (eq? method git-fetch)
+            `(("git_ref" . ,(git-reference-commit uri)))
+            '())
+      ,@(if (eq? method svn-fetch)
+            `(("svn_revision" . ,(svn-reference-revision
+                                  uri)))
+            '())))
+
+  (define (package->json package)
+    `(,@(if (origin? (package-source package))
+            (origin->json (package-source package))
+            `(("type" . "no-origin")
+              ("name" . ,(package-name package))))))
+
+  (make-page "sources.json"
+             `(("sources" . ,(list->vector (map package->json (all-packages))))
+               ("version" . "1"))
+             scm->json))
+
+
 (define (packages-json-builder)
   "Return a JSON page listing all packages."
   (define (origin->json origin)
-- 
2.23.0

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.

[...]

> +(define (sources-json-builder)
> +  "Return a JSON page listing all the sources."

Please add a reference to
<https://forge.softwareheritage.org/D2025#51269> here.

> +  (define (origin->json origin)
> +    (define method
> +      (origin-method origin))
> +
> +    (define uri                         ;represented as string
> +      (origin-uri origin))
> +
> +    (define (mirror->url uri)
> +      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> +
> +    (define (resolve urls)
> +      (let* ((url (car urls))
> +             (uri (string->uri url))
> +             (rest (cdr urls)))
> +        (case (uri-scheme uri)
> +          ((mirror) (mirror->url uri))
> +          ((http) url)
> +          ((https) url)
> +          (else
> +           (if (null? rest)
> +               url
> +               (resolve rest))))))
> +
> +    `((type . ,(cond ((eq? url-fetch method) 'url)
> +                     ((eq? git-fetch method) 'git)
> +                     ((eq? svn-fetch method) 'svn)
> +                     (else                   #nil)))
> +      ,@(cond ((eq? url-fetch method)
> +               `(("url" . ,(match uri
> +				  ((? string? url) (mirror->url (string->uri url)))
> +				  ((urls ...) (resolve urls))))))
> +              ((eq? git-fetch method)
> +               `(("git_url" . ,(git-reference-url uri))))
> +              ((eq? svn-fetch method)
> +               `(("svn_url" . ,(svn-reference-url uri))))
> +              (else '()))
> +      ,@(if (eq? method git-fetch)
> +            `(("git_ref" . ,(git-reference-commit uri)))
> +            '())
> +      ,@(if (eq? method svn-fetch)
> +            `(("svn_revision" . ,(svn-reference-revision
> +                                  uri)))
> +            '())))

Could you, in a first patch, move ‘origin->json’ out of
‘packages-json-builder’, and in a second patch, add mirror-expansion
feature?

For mirror:// expansion, I think you can just write something like:

  (define uris
    (append-map (cut maybe-expand-mirrors <> %mirrors)
                (match url
                  ((_ ...) (map string->uri url))
                  (_       (list (string->uri url))))))

and then pick the first element of the list.

In parallel, I’d recommend suggesting a format change or addition that
would allow us to provide all the URLs.  :-)

With those changes in place, I think we’ll be ready to go!

Thanks for working on it, Simon!

Ludo’.

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[zimoun]

Hi Ludo,

Thank you for the review.


On Fri, 14 Feb 2020 at 09:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:

> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (sources-json-builder): New procedure.
>
> [...]
>
> > +(define (sources-json-builder)
> > +  "Return a JSON page listing all the sources."
>
> Please add a reference to
> <https://forge.softwareheritage.org/D2025#51269> here.

ok


> > +  (define (origin->json origin)
> > +    (define method
> > +      (origin-method origin))
> > +
> > +    (define uri                         ;represented as string
> > +      (origin-uri origin))
> > +
> > +    (define (mirror->url uri)
> > +      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> > +
> > +    (define (resolve urls)
> > +      (let* ((url (car urls))
> > +             (uri (string->uri url))
> > +             (rest (cdr urls)))
> > +        (case (uri-scheme uri)
> > +          ((mirror) (mirror->url uri))
> > +          ((http) url)
> > +          ((https) url)
> > +          (else
> > +           (if (null? rest)
> > +               url
> > +               (resolve rest))))))
> > +
> > +    `((type . ,(cond ((eq? url-fetch method) 'url)
> > +                     ((eq? git-fetch method) 'git)
> > +                     ((eq? svn-fetch method) 'svn)
> > +                     (else                   #nil)))
> > +      ,@(cond ((eq? url-fetch method)
> > +               `(("url" . ,(match uri
> > +                               ((? string? url) (mirror->url (string->uri url)))
> > +                               ((urls ...) (resolve urls))))))
> > +              ((eq? git-fetch method)
> > +               `(("git_url" . ,(git-reference-url uri))))
> > +              ((eq? svn-fetch method)
> > +               `(("svn_url" . ,(svn-reference-url uri))))
> > +              (else '()))
> > +      ,@(if (eq? method git-fetch)
> > +            `(("git_ref" . ,(git-reference-commit uri)))
> > +            '())
> > +      ,@(if (eq? method svn-fetch)
> > +            `(("svn_revision" . ,(svn-reference-revision
> > +                                  uri)))
> > +            '())))
>
> Could you, in a first patch, move ‘origin->json’ out of
> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> feature?

Yes, I will try but I am a bit lost. There is:
 - packages-json-builder that I did not modified
 - sources-json-builder which the "adaptation" of the former to output
to the sources.json format.

Well, do you want a refactor of 'origin->json' shared by the 2
"{sources,packages}-json-builder"?


> For mirror:// expansion, I think you can just write something like:
>
>   (define uris
>     (append-map (cut maybe-expand-mirrors <> %mirrors)
>                 (match url
>                   ((_ ...) (map string->uri url))
>                   (_       (list (string->uri url))))))
>
> and then pick the first element of the list.

Yes, it is better. :-)


> In parallel, I’d recommend suggesting a format change or addition that
> would allow us to provide all the URLs.  :-)

Ok :-)


> With those changes in place, I think we’ll be ready to go!

Working on updating the package Julia, I have seen that some patches
are 'origin' and live for example upstream
(https://blablab/project.git/patches/fancy-name.patch) and the current
patch will not list them. Therefore, if upstream disappears and/or
change in-place the patches, Guix would not be able to re-build in the
future (time-machine).

I am trying to implement a recursive exporter.
Well, I will try to make that the v2 contains your suggestions and all
the patches.



Cheers,
simon

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>> Could you, in a first patch, move ‘origin->json’ out of
>> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> feature?
>
> Yes, I will try but I am a bit lost. There is:
>  - packages-json-builder that I did not modified
>  - sources-json-builder which the "adaptation" of the former to output
> to the sources.json format.
>
> Well, do you want a refactor of 'origin->json' shared by the 2
> "{sources,packages}-json-builder"?

Yes, exactly.  No reason to have two copies of that code.

> Working on updating the package Julia, I have seen that some patches
> are 'origin' and live for example upstream
> (https://blablab/project.git/patches/fancy-name.patch) and the current
> patch will not list them. Therefore, if upstream disappears and/or
> change in-place the patches, Guix would not be able to re-build in the
> future (time-machine).

Right.  I guess you can implement that in the next version.  What you
propose is already nice to have.

Ludo’.

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[zimoun]

Hi Ludo,

On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:

> >> Could you, in a first patch, move ‘origin->json’ out of
> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> feature?
> >
> > Yes, I will try but I am a bit lost. There is:
> >  - packages-json-builder that I did not modified
> >  - sources-json-builder which the "adaptation" of the former to output
> > to the sources.json format.
> >
> > Well, do you want a refactor of 'origin->json' shared by the 2
> > "{sources,packages}-json-builder"?
>
> Yes, exactly.  No reason to have two copies of that code.

The minor "issue" is the 'match' piece of 'origin->json':

--8<---------------cut here---------------start------------->8---
               `(("url" . ,(match uri
                             ((? string? url) (vector url))
                             ((urls ...) (list->vector urls))))))
--8<---------------cut here---------------end--------------->8---

Because in the 'sources.json' it is always one unique string.
However, in 'packages.json' it is always a vector of one or several
elements. And in the previous patch, the job of 'resolve' was to
uniquify these elements for 'sources.json'.

For example, see the package 'adns' in 'packages.json'

--8<---------------cut here---------------start------------->8---
{
    "name": "adns",
    "version": "1.5.1",
    "source": {
      "type": "url",
      "url": [
        "mirror://gnu/adns/adns-1.5.1.tar.gz",
        "http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
      ]
    },
    "synopsis": "Asynchronous DNS client library and utilities",
    "homepage": "https://www.gnu.org/software/adns/",
    "location": "gnu/packages/adns.scm:32"
  },
--8<---------------cut here---------------end--------------->8---



Just to be on the same wavelength, does 'packages.json' stay as it is
(vector for the source url field)?
Or is 'packages.json' modified to record only one url for the source
url field with mirror:// expanded?


Cheers,
simon

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> >> Could you, in a first patch, move ‘origin->json’ out of
>> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> >> feature?
>> >
>> > Yes, I will try but I am a bit lost. There is:
>> >  - packages-json-builder that I did not modified
>> >  - sources-json-builder which the "adaptation" of the former to output
>> > to the sources.json format.
>> >
>> > Well, do you want a refactor of 'origin->json' shared by the 2
>> > "{sources,packages}-json-builder"?
>>
>> Yes, exactly.  No reason to have two copies of that code.
>
> The minor "issue" is the 'match' piece of 'origin->json':
>
>                `(("url" . ,(match uri
>                              ((? string? url) (vector url))
>                              ((urls ...) (list->vector urls))))))
>
>
> Because in the 'sources.json' it is always one unique string.
> However, in 'packages.json' it is always a vector of one or several
> elements. And in the previous patch, the job of 'resolve' was to
> uniquify these elements for 'sources.json'.

Ah, good point.  What about adding a keyword parameter to ‘origin->json’
that would tell whether to return a single URL or a list thereof?

> For example, see the package 'adns' in 'packages.json'
>
> {
>     "name": "adns",
>     "version": "1.5.1",
>     "source": {
>       "type": "url",
>       "url": [
>         "mirror://gnu/adns/adns-1.5.1.tar.gz",
>         "http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
>       ]
>     },
>     "synopsis": "Asynchronous DNS client library and utilities",
>     "homepage": "https://www.gnu.org/software/adns/",
>     "location": "gnu/packages/adns.scm:32"
>   },
>
>
>
> Just to be on the same wavelength, does 'packages.json' stay as it is
> (vector for the source url field)?

I think ‘packages.json’ should still return a list of URLs.

I’ve also added a comment asking whether ‘sources.json’ could be changed
to contain a list of URLs:

  https://forge.softwareheritage.org/D2025#64037

> Or is 'packages.json' modified to record only one url for the source
> url field with mirror:// expanded?

Yeah, I think we can expand mirror URLs.

Thanks,
Ludo’.

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[zimoun]

Hi Ludo,

On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> > On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> >> zimoun <zimon.toutoune@gmail.com> skribis:

> >> >> Could you, in a first patch, move ‘origin->json’ out of
> >> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> >> feature?
> >> >
> >> > Yes, I will try but I am a bit lost. There is:
> >> >  - packages-json-builder that I did not modified
> >> >  - sources-json-builder which the "adaptation" of the former to output
> >> > to the sources.json format.
> >> >
> >> > Well, do you want a refactor of 'origin->json' shared by the 2
> >> > "{sources,packages}-json-builder"?
> >>
> >> Yes, exactly.  No reason to have two copies of that code.
> >
> > The minor "issue" is the 'match' piece of 'origin->json':
> >
> >                `(("url" . ,(match uri
> >                              ((? string? url) (vector url))
> >                              ((urls ...) (list->vector urls))))))
> >
> >
> > Because in the 'sources.json' it is always one unique string.
> > However, in 'packages.json' it is always a vector of one or several
> > elements. And in the previous patch, the job of 'resolve' was to
> > uniquify these elements for 'sources.json'.
>
> Ah, good point.  What about adding a keyword parameter to ‘origin->json’
> that would tell whether to return a single URL or a list thereof?

Sharing the same code.

 - packages.json: as it is now.
 - sources.json: expand mirror and resolve to display only URL.

I am going for that.


> > Just to be on the same wavelength, does 'packages.json' stay as it is
> > (vector for the source url field)?
>
> I think ‘packages.json’ should still return a list of URLs.

Ok.


> I’ve also added a comment asking whether ‘sources.json’ could be changed
> to contain a list of URLs:
>
>   https://forge.softwareheritage.org/D2025#64037

I have seen. :-)
Even, all the mirrors list should be expanded, not only the first one.


> > Or is 'packages.json' modified to record only one url for the source
> > url field with mirror:// expanded?
>
> Yeah, I think we can expand mirror URLs.

So, 'mirror://' will become a list (from '%mirrors') of URLs, right?


Cheers,
simon

[bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

[...]

>> > Or is 'packages.json' modified to record only one url for the source
>> > url field with mirror:// expanded?
>>
>> Yeah, I think we can expand mirror URLs.
>
> So, 'mirror://' will become a list (from '%mirrors') of URLs, right?

Yup!

Thanks,
Ludo’.

[bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list.

[zimoun]

* website/apps/packages/builder.scm (origin->json): New procedure.
---
 website/apps/packages/builder.scm | 34 ++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..d3a777e 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; Initially written by sirgazil
 ;;; who waives all copyright interest on this file.
@@ -37,13 +38,16 @@
   #:use-module (haunt page)
   #:use-module (haunt utils)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix svn-download)
   #:use-module (guix utils)                       ;location
+  #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (json)
   #:use-module (ice-9 match)
+  #:use-module ((web uri) #:select (string->uri uri->string))
   #:export (builder))
 
 
@@ -84,33 +88,43 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (packages-json-builder)
-  "Return a JSON page listing all packages."
-  (define (origin->json origin)
+(define (origin->json origin)
     (define method
       (origin-method origin))
 
+    (define uri                         ;represented as string
+      (origin-uri origin))
+
+    (define (resolve urls)
+      (map uri->string
+           (append-map (cut maybe-expand-mirrors <> %mirrors)
+                       (map string->uri urls))))
+
     `((type . ,(cond ((eq? url-fetch method) 'url)
                      ((eq? git-fetch method) 'git)
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(match (origin-uri origin)
-                             ((? string? url) (vector url))
-                             ((urls ...) (list->vector urls))))))
+               `(("url" . ,(list->vector
+                            (resolve
+                             (match uri
+                               ((? string? url) (list url))
+                               ((urls ...) urls)))))))
               ((eq? git-fetch method)
-               `(("git_url" . ,(git-reference-url (origin-uri origin)))))
+               `(("git_url" . ,(git-reference-url uri))))
               ((eq? svn-fetch method)
-               `(("svn_url" . ,(svn-reference-url (origin-uri origin)))))
+               `(("svn_url" . ,(svn-reference-url uri))))
               (else '()))
       ,@(if (eq? method git-fetch)
-            `(("git_ref" . ,(git-reference-commit (origin-uri origin))))
+            `(("git_ref" . ,(git-reference-commit uri)))
             '())
       ,@(if (eq? method svn-fetch)
             `(("svn_revision" . ,(svn-reference-revision
-                                  (origin-uri origin))))
+                                  uri)))
             '())))
 
+(define (packages-json-builder)
+  "Return a JSON page listing all packages."
   (define (package->json package)
     (define cpe-name
       (assoc-ref (package-properties package) 'cpe-name))
-- 
2.25.0

[bug#39547] [PATCH v2 2/2] website: Provide JSON sources list used by Software Heritage.

[zimoun]

Format discussed here <https://forge.softwareheritage.org/D2025#51269>.

* website/apps/packages/builder.scm (origin->json): Add list modifier.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
 website/apps/packages/builder.scm | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index d3a777e..49721f6 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -74,6 +74,7 @@
   (flatten
    (list
     (index-builder)
+    (sources-json-builder)
     (packages-json-builder)
     (packages-builder)
     (package-list-builder))))
@@ -88,7 +89,7 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (origin->json origin)
+(define (origin->json origin transformer)
     (define method
       (origin-method origin))
 
@@ -105,7 +106,7 @@
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(list->vector
+               `(("url" . ,(transformer
                             (resolve
                              (match uri
                                ((? string? url) (list url))
@@ -136,7 +137,7 @@
       ,@(if cpe-name `(("cpe_name" . ,cpe-name)) '())
       ,@(if cpe-version `(("cpe_version" . ,cpe-version)) '())
       ,@(if (origin? (package-source package))
-            `(("source" . ,(origin->json (package-source package))))
+            `(("source" . ,(origin->json (package-source package) list->vector)))
             '())
       ("synopsis" . ,(package-synopsis package))
       ,@(if (package-home-page package)
@@ -155,6 +156,21 @@
 	     (list->vector (map package->json (all-packages)))
              scm->json))
 
+(define (sources-json-builder)
+  "Return a JSON page listing all the sources.
+
+See <https://forge.softwareheritage.org/D2025#51269>."
+  (define (package->json package)
+    `(,@(if (origin? (package-source package))
+            (origin->json (package-source package) car)
+            `(("type" . "no-origin")
+              ("name" . ,(package-name package))))))
+
+  (make-page "sources.json"
+             `(("sources" . ,(list->vector (map package->json (all-packages))))
+               ("version" . "1"))
+             scm->json))
+
 (define (index-builder)
   "Return a Haunt page listing some random packages."
   (define (sample n from)
-- 
2.25.0

[bug#39547] [PATCH v3] sources.json: array instead of list

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

Hi Ludo,

Attached, the tiny modification to output the list (array) of URLs
instead of the first one.
The version number is still '1' because I do not know yet if lewo would bump it.
Note also that "transformer" is not useful any more because
'packages-json-builder' and 'sources-json-builder' returns both
vectors now.


Cheers,
simon

[-- Attachment #2: v3-0001-website-Refactor-and-resolve-mirror-of-JSON-packa.patch --]
[-- Type: text/x-patch, Size: 3716 bytes --]

From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Mon, 10 Feb 2020 17:52:13 +0100
Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
 package list.

* website/apps/packages/builder.scm (origin->json): New procedure.
---
 website/apps/packages/builder.scm | 34 ++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..d3a777e 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; Initially written by sirgazil
 ;;; who waives all copyright interest on this file.
@@ -37,13 +38,16 @@
   #:use-module (haunt page)
   #:use-module (haunt utils)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix svn-download)
   #:use-module (guix utils)                       ;location
+  #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (json)
   #:use-module (ice-9 match)
+  #:use-module ((web uri) #:select (string->uri uri->string))
   #:export (builder))
 
 
@@ -84,33 +88,43 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (packages-json-builder)
-  "Return a JSON page listing all packages."
-  (define (origin->json origin)
+(define (origin->json origin)
     (define method
       (origin-method origin))
 
+    (define uri                         ;represented as string
+      (origin-uri origin))
+
+    (define (resolve urls)
+      (map uri->string
+           (append-map (cut maybe-expand-mirrors <> %mirrors)
+                       (map string->uri urls))))
+
     `((type . ,(cond ((eq? url-fetch method) 'url)
                      ((eq? git-fetch method) 'git)
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(match (origin-uri origin)
-                             ((? string? url) (vector url))
-                             ((urls ...) (list->vector urls))))))
+               `(("url" . ,(list->vector
+                            (resolve
+                             (match uri
+                               ((? string? url) (list url))
+                               ((urls ...) urls)))))))
               ((eq? git-fetch method)
-               `(("git_url" . ,(git-reference-url (origin-uri origin)))))
+               `(("git_url" . ,(git-reference-url uri))))
               ((eq? svn-fetch method)
-               `(("svn_url" . ,(svn-reference-url (origin-uri origin)))))
+               `(("svn_url" . ,(svn-reference-url uri))))
               (else '()))
       ,@(if (eq? method git-fetch)
-            `(("git_ref" . ,(git-reference-commit (origin-uri origin))))
+            `(("git_ref" . ,(git-reference-commit uri)))
             '())
       ,@(if (eq? method svn-fetch)
             `(("svn_revision" . ,(svn-reference-revision
-                                  (origin-uri origin))))
+                                  uri)))
             '())))
 
+(define (packages-json-builder)
+  "Return a JSON page listing all packages."
   (define (package->json package)
     (define cpe-name
       (assoc-ref (package-properties package) 'cpe-name))
-- 
2.25.0


[-- Attachment #3: v3-0002-website-Provide-JSON-sources-list-used-by-Softwar.patch --]
[-- Type: text/x-patch, Size: 2326 bytes --]

From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Tue, 18 Feb 2020 13:25:14 +0100
Subject: [PATCH v3 2/2] website: Provide JSON sources list used by Software
 Heritage.

Format discussed here <https://forge.softwareheritage.org/D2025#51269>.

* website/apps/packages/builder.scm (origin->json): Add list modifier.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
 website/apps/packages/builder.scm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index d3a777e..3fc1285 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -74,6 +74,7 @@
   (flatten
    (list
     (index-builder)
+    (sources-json-builder)
     (packages-json-builder)
     (packages-builder)
     (package-list-builder))))
@@ -88,7 +89,7 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (origin->json origin)
+(define* (origin->json origin #:optional (transformer list->vector))
     (define method
       (origin-method origin))
 
@@ -105,7 +106,7 @@
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(list->vector
+               `(("url" . ,(transformer
                             (resolve
                              (match uri
                                ((? string? url) (list url))
@@ -155,6 +156,21 @@
 	     (list->vector (map package->json (all-packages)))
              scm->json))
 
+(define (sources-json-builder)
+  "Return a JSON page listing all the sources.
+
+See <https://forge.softwareheritage.org/D2025#51269>."
+  (define (package->json package)
+    `(,@(if (origin? (package-source package))
+            (origin->json (package-source package))
+            `(("type" . "no-origin")
+              ("name" . ,(package-name package))))))
+
+  (make-page "sources.json"
+             `(("sources" . ,(list->vector (map package->json (all-packages))))
+               ("version" . "1"))
+             scm->json))
+
 (define (index-builder)
   "Return a Haunt page listing some random packages."
   (define (sample n from)
-- 
2.25.0

[bug#39547] [PATCH v3] sources.json: array instead of list

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> Attached, the tiny modification to output the list (array) of URLs
> instead of the first one.
> The version number is still '1' because I do not know yet if lewo would bump it.
> Note also that "transformer" is not useful any more because
> 'packages-json-builder' and 'sources-json-builder' returns both
> vectors now.

Cool.

> From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Mon, 10 Feb 2020 17:52:13 +0100
> Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
>  package list.
>
> * website/apps/packages/builder.scm (origin->json): New procedure.

LGTM!

> From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Tue, 18 Feb 2020 13:25:14 +0100
> Subject: [PATCH v3 2/2] website: Provide JSON sources list used by Software
>  Heritage.
>
> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (origin->json): Add list modifier.
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.

Nitpick: no need to repeat the file name here.

Also: please mention the addition to the ‘builder’ procedure.

> +(define* (origin->json origin #:optional (transformer list->vector))

You can remove this parameter, as you wrote.

I can make these changes and push it on your behalf if you want, let me
know!

Thanks,
Ludo’.

[bug#39547] [PATCH v3] sources.json: array instead of list

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

Hi Ludo,

Yes please push.

Cheers,
simon

PS: i am typing with my smartphone because i am sleeping in a hospital
(Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
after a shock to my neck during a basket ball party with some friends.
Everything is on the process but I cannot say when i will have access to my
laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:

On Friday, 6 March 2020, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Attached, the tiny modification to output the list (array) of URLs
> > instead of the first one.
> > The version number is still '1' because I do not know yet if lewo would
> bump it.
> > Note also that "transformer" is not useful any more because
> > 'packages-json-builder' and 'sources-json-builder' returns both
> > vectors now.
>
> Cool.
>
> > From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune@gmail.com>
> > Date: Mon, 10 Feb 2020 17:52:13 +0100
> > Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
> >  package list.
> >
> > * website/apps/packages/builder.scm (origin->json): New procedure.
>
> LGTM!
>
> > From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune@gmail.com>
> > Date: Tue, 18 Feb 2020 13:25:14 +0100
> > Subject: [PATCH v3 2/2] website: Provide JSON sources list used by
> Software
> >  Heritage.
> >
> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (origin->json): Add list modifier.
> > * website/apps/packages/builder.scm (sources-json-builder): New
> procedure.
>
> Nitpick: no need to repeat the file name here.
>
> Also: please mention the addition to the ‘builder’ procedure.
>
> > +(define* (origin->json origin #:optional (transformer list->vector))
>
> You can remove this parameter, as you wrote.
>
> I can make these changes and push it on your behalf if you want, let me
> know!
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2888 bytes --]

bug#39547: [PATCH v3] sources.json: array instead of list

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Yes please push.

Done!  <https://guix.gnu.org/sources.json> should show up within an hour.

> PS: i am typing with my smartphone because i am sleeping in a hospital
> (Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
> after a shock to my neck during a basket ball party with some friends.
> Everything is on the process but I cannot say when i will have access to my
> laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:

Ouch, I wish you quick recovery!

Take care of yourself,
Ludo’.

[bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?

[zimoun]

Hi,

Some upstream sources could disappear between the moment when the
package enters in Guix and the moment when SWH ingests it. (As an edge
case, let consider the recent harfbuzz one.)

Now the format 'sources.json' should accept an array. Does it make
sense to put in this array the content addressed mirrors
(ci.guix.gnu.org and tarball.nixos.org)?

What do you think?


Cheers,
simon

[bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> Some upstream sources could disappear between the moment when the
> package enters in Guix and the moment when SWH ingests it. (As an edge
> case, let consider the recent harfbuzz one.)
>
> Now the format 'sources.json' should accept an array. Does it make
> sense to put in this array the content addressed mirrors
> (ci.guix.gnu.org and tarball.nixos.org)?

Yes, I think ‘sources.json’ should contain, for each item, a list of
URLs rather than a single URL, and mirror:// URLs should be expanded.

However, ‘sources.json’ is currently specified to contain a single URL,
so we’re stuck with a single URL for now.

Is that what you meant?

Thanks,
Ludo’.

[bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?

[zimoun]

Hi Ludo,

On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Some upstream sources could disappear between the moment when the
> > package enters in Guix and the moment when SWH ingests it. (As an edge
> > case, let consider the recent harfbuzz one.)
> >
> > Now the format 'sources.json' should accept an array. Does it make
> > sense to put in this array the content addressed mirrors
> > (ci.guix.gnu.org and tarball.nixos.org)?
>
> Yes, I think ‘sources.json’ should contain, for each item, a list of
> URLs rather than a single URL, and mirror:// URLs should be expanded.

Yes, now it is accepted [1] and it becomes the format of the
'sources.json' file.

[1] https://forge.softwareheritage.org/D2025#65063


The patch set v2 supports the first discussion of the format (single
url); which is not relevant anymore.
The patch set attached in v3 [2] supports the array of URLs.

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35



> However, ‘sources.json’ is currently specified to contain a single URL,
> so we’re stuck with a single URL for now.
>
> Is that what you meant?

What I mean is: append %content-addressed-mirrors for each package in
the list of URLs.



Cheers,
simon

[bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?

[Ludovic Courtès]

Hi Simon!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Yes, I think ‘sources.json’ should contain, for each item, a list of
>> URLs rather than a single URL, and mirror:// URLs should be expanded.
>
> Yes, now it is accepted [1] and it becomes the format of the
> 'sources.json' file.
>
> [1] https://forge.softwareheritage.org/D2025#65063

Excellent!

> The patch set v2 supports the first discussion of the format (single
> url); which is not relevant anymore.
> The patch set attached in v3 [2] supports the array of URLs.
>
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35

Oh sorry, I hadn’t seen it.

>> However, ‘sources.json’ is currently specified to contain a single URL,
>> so we’re stuck with a single URL for now.
>>
>> Is that what you meant?
>
> What I mean is: append %content-addressed-mirrors for each package in
> the list of URLs.

Now I understand.  :-)  So yes, sure, the array of URLs in
‘sources.json’ could include a ci.guix.gnu.org URL and maybe the other
ones in there too.  For a subsequent patch I guess?

Thanks!

Ludo’.