* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
@ 2020-02-10 17:04 zimoun
2020-02-14 8:40 ` Ludovic Courtès
` (3 more replies)
0 siblings, 4 replies; 18+ messages in thread
From: zimoun @ 2020-02-10 17:04 UTC (permalink / raw)
To: 39547; +Cc: zimoun
Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
website/apps/packages/builder.scm | 62 +++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..5279096 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; Initially written by sirgazil
;;; who waives all copyright interest on this file.
@@ -37,6 +38,8 @@
#:use-module (haunt page)
#:use-module (haunt utils)
#:use-module (srfi srfi-1)
+ #:use-module ((web uri) #:select (string->uri uri->string uri-scheme))
+ #:use-module ((guix build download) #:select (maybe-expand-mirrors))
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix git-download)
@@ -70,6 +73,7 @@
(flatten
(list
(index-builder)
+ (sources-json-builder)
(packages-json-builder)
(packages-builder)
(package-list-builder))))
@@ -84,6 +88,64 @@
;; Maximum number of packages shown on /packages.
30)
+(define (sources-json-builder)
+ "Return a JSON page listing all the sources."
+ (define (origin->json origin)
+ (define method
+ (origin-method origin))
+
+ (define uri ;represented as string
+ (origin-uri origin))
+
+ (define (mirror->url uri)
+ (uri->string (car (maybe-expand-mirrors uri %mirrors))))
+
+ (define (resolve urls)
+ (let* ((url (car urls))
+ (uri (string->uri url))
+ (rest (cdr urls)))
+ (case (uri-scheme uri)
+ ((mirror) (mirror->url uri))
+ ((http) url)
+ ((https) url)
+ (else
+ (if (null? rest)
+ url
+ (resolve rest))))))
+
+ `((type . ,(cond ((eq? url-fetch method) 'url)
+ ((eq? git-fetch method) 'git)
+ ((eq? svn-fetch method) 'svn)
+ (else #nil)))
+ ,@(cond ((eq? url-fetch method)
+ `(("url" . ,(match uri
+ ((? string? url) (mirror->url (string->uri url)))
+ ((urls ...) (resolve urls))))))
+ ((eq? git-fetch method)
+ `(("git_url" . ,(git-reference-url uri))))
+ ((eq? svn-fetch method)
+ `(("svn_url" . ,(svn-reference-url uri))))
+ (else '()))
+ ,@(if (eq? method git-fetch)
+ `(("git_ref" . ,(git-reference-commit uri)))
+ '())
+ ,@(if (eq? method svn-fetch)
+ `(("svn_revision" . ,(svn-reference-revision
+ uri)))
+ '())))
+
+ (define (package->json package)
+ `(,@(if (origin? (package-source package))
+ (origin->json (package-source package))
+ `(("type" . "no-origin")
+ ("name" . ,(package-name package))))))
+
+ (make-page "sources.json"
+ `(("sources" . ,(list->vector (map package->json (all-packages))))
+ ("version" . "1"))
+ scm->json))
+
+
(define (packages-json-builder)
"Return a JSON page listing all packages."
(define (origin->json origin)
--
2.23.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-10 17:04 [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage zimoun
@ 2020-02-14 8:40 ` Ludovic Courtès
2020-02-14 9:04 ` zimoun
2020-02-18 12:32 ` [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list zimoun
` (2 subsequent siblings)
3 siblings, 1 reply; 18+ messages in thread
From: Ludovic Courtès @ 2020-02-14 8:40 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hello,
zimoun <zimon.toutoune@gmail.com> skribis:
> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.
[...]
> +(define (sources-json-builder)
> + "Return a JSON page listing all the sources."
Please add a reference to
<https://forge.softwareheritage.org/D2025#51269> here.
> + (define (origin->json origin)
> + (define method
> + (origin-method origin))
> +
> + (define uri ;represented as string
> + (origin-uri origin))
> +
> + (define (mirror->url uri)
> + (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> +
> + (define (resolve urls)
> + (let* ((url (car urls))
> + (uri (string->uri url))
> + (rest (cdr urls)))
> + (case (uri-scheme uri)
> + ((mirror) (mirror->url uri))
> + ((http) url)
> + ((https) url)
> + (else
> + (if (null? rest)
> + url
> + (resolve rest))))))
> +
> + `((type . ,(cond ((eq? url-fetch method) 'url)
> + ((eq? git-fetch method) 'git)
> + ((eq? svn-fetch method) 'svn)
> + (else #nil)))
> + ,@(cond ((eq? url-fetch method)
> + `(("url" . ,(match uri
> + ((? string? url) (mirror->url (string->uri url)))
> + ((urls ...) (resolve urls))))))
> + ((eq? git-fetch method)
> + `(("git_url" . ,(git-reference-url uri))))
> + ((eq? svn-fetch method)
> + `(("svn_url" . ,(svn-reference-url uri))))
> + (else '()))
> + ,@(if (eq? method git-fetch)
> + `(("git_ref" . ,(git-reference-commit uri)))
> + '())
> + ,@(if (eq? method svn-fetch)
> + `(("svn_revision" . ,(svn-reference-revision
> + uri)))
> + '())))
Could you, in a first patch, move ‘origin->json’ out of
‘packages-json-builder’, and in a second patch, add mirror-expansion
feature?
For mirror:// expansion, I think you can just write something like:
(define uris
(append-map (cut maybe-expand-mirrors <> %mirrors)
(match url
((_ ...) (map string->uri url))
(_ (list (string->uri url))))))
and then pick the first element of the list.
In parallel, I’d recommend suggesting a format change or addition that
would allow us to provide all the URLs. :-)
With those changes in place, I think we’ll be ready to go!
Thanks for working on it, Simon!
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-14 8:40 ` Ludovic Courtès
@ 2020-02-14 9:04 ` zimoun
2020-02-14 10:20 ` Ludovic Courtès
0 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-02-14 9:04 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 39547
Hi Ludo,
Thank you for the review.
On Fri, 14 Feb 2020 at 09:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (sources-json-builder): New procedure.
>
> [...]
>
> > +(define (sources-json-builder)
> > + "Return a JSON page listing all the sources."
>
> Please add a reference to
> <https://forge.softwareheritage.org/D2025#51269> here.
ok
> > + (define (origin->json origin)
> > + (define method
> > + (origin-method origin))
> > +
> > + (define uri ;represented as string
> > + (origin-uri origin))
> > +
> > + (define (mirror->url uri)
> > + (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> > +
> > + (define (resolve urls)
> > + (let* ((url (car urls))
> > + (uri (string->uri url))
> > + (rest (cdr urls)))
> > + (case (uri-scheme uri)
> > + ((mirror) (mirror->url uri))
> > + ((http) url)
> > + ((https) url)
> > + (else
> > + (if (null? rest)
> > + url
> > + (resolve rest))))))
> > +
> > + `((type . ,(cond ((eq? url-fetch method) 'url)
> > + ((eq? git-fetch method) 'git)
> > + ((eq? svn-fetch method) 'svn)
> > + (else #nil)))
> > + ,@(cond ((eq? url-fetch method)
> > + `(("url" . ,(match uri
> > + ((? string? url) (mirror->url (string->uri url)))
> > + ((urls ...) (resolve urls))))))
> > + ((eq? git-fetch method)
> > + `(("git_url" . ,(git-reference-url uri))))
> > + ((eq? svn-fetch method)
> > + `(("svn_url" . ,(svn-reference-url uri))))
> > + (else '()))
> > + ,@(if (eq? method git-fetch)
> > + `(("git_ref" . ,(git-reference-commit uri)))
> > + '())
> > + ,@(if (eq? method svn-fetch)
> > + `(("svn_revision" . ,(svn-reference-revision
> > + uri)))
> > + '())))
>
> Could you, in a first patch, move ‘origin->json’ out of
> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> feature?
Yes, I will try but I am a bit lost. There is:
- packages-json-builder that I did not modified
- sources-json-builder which the "adaptation" of the former to output
to the sources.json format.
Well, do you want a refactor of 'origin->json' shared by the 2
"{sources,packages}-json-builder"?
> For mirror:// expansion, I think you can just write something like:
>
> (define uris
> (append-map (cut maybe-expand-mirrors <> %mirrors)
> (match url
> ((_ ...) (map string->uri url))
> (_ (list (string->uri url))))))
>
> and then pick the first element of the list.
Yes, it is better. :-)
> In parallel, I’d recommend suggesting a format change or addition that
> would allow us to provide all the URLs. :-)
Ok :-)
> With those changes in place, I think we’ll be ready to go!
Working on updating the package Julia, I have seen that some patches
are 'origin' and live for example upstream
(https://blablab/project.git/patches/fancy-name.patch) and the current
patch will not list them. Therefore, if upstream disappears and/or
change in-place the patches, Guix would not be able to re-build in the
future (time-machine).
I am trying to implement a recursive exporter.
Well, I will try to make that the v2 contains your suggestions and all
the patches.
Cheers,
simon
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-14 9:04 ` zimoun
@ 2020-02-14 10:20 ` Ludovic Courtès
2020-02-17 17:59 ` zimoun
0 siblings, 1 reply; 18+ messages in thread
From: Ludovic Courtès @ 2020-02-14 10:20 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
>> Could you, in a first patch, move ‘origin->json’ out of
>> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> feature?
>
> Yes, I will try but I am a bit lost. There is:
> - packages-json-builder that I did not modified
> - sources-json-builder which the "adaptation" of the former to output
> to the sources.json format.
>
> Well, do you want a refactor of 'origin->json' shared by the 2
> "{sources,packages}-json-builder"?
Yes, exactly. No reason to have two copies of that code.
> Working on updating the package Julia, I have seen that some patches
> are 'origin' and live for example upstream
> (https://blablab/project.git/patches/fancy-name.patch) and the current
> patch will not list them. Therefore, if upstream disappears and/or
> change in-place the patches, Guix would not be able to re-build in the
> future (time-machine).
Right. I guess you can implement that in the next version. What you
propose is already nice to have.
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-14 10:20 ` Ludovic Courtès
@ 2020-02-17 17:59 ` zimoun
2020-02-18 8:24 ` Ludovic Courtès
0 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-02-17 17:59 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 39547
Hi Ludo,
On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> >> Could you, in a first patch, move ‘origin->json’ out of
> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> feature?
> >
> > Yes, I will try but I am a bit lost. There is:
> > - packages-json-builder that I did not modified
> > - sources-json-builder which the "adaptation" of the former to output
> > to the sources.json format.
> >
> > Well, do you want a refactor of 'origin->json' shared by the 2
> > "{sources,packages}-json-builder"?
>
> Yes, exactly. No reason to have two copies of that code.
The minor "issue" is the 'match' piece of 'origin->json':
--8<---------------cut here---------------start------------->8---
`(("url" . ,(match uri
((? string? url) (vector url))
((urls ...) (list->vector urls))))))
--8<---------------cut here---------------end--------------->8---
Because in the 'sources.json' it is always one unique string.
However, in 'packages.json' it is always a vector of one or several
elements. And in the previous patch, the job of 'resolve' was to
uniquify these elements for 'sources.json'.
For example, see the package 'adns' in 'packages.json'
--8<---------------cut here---------------start------------->8---
{
"name": "adns",
"version": "1.5.1",
"source": {
"type": "url",
"url": [
"mirror://gnu/adns/adns-1.5.1.tar.gz",
"http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
]
},
"synopsis": "Asynchronous DNS client library and utilities",
"homepage": "https://www.gnu.org/software/adns/",
"location": "gnu/packages/adns.scm:32"
},
--8<---------------cut here---------------end--------------->8---
Just to be on the same wavelength, does 'packages.json' stay as it is
(vector for the source url field)?
Or is 'packages.json' modified to record only one url for the source
url field with mirror:// expanded?
Cheers,
simon
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-17 17:59 ` zimoun
@ 2020-02-18 8:24 ` Ludovic Courtès
2020-02-18 8:38 ` zimoun
0 siblings, 1 reply; 18+ messages in thread
From: Ludovic Courtès @ 2020-02-18 8:24 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hi!
zimoun <zimon.toutoune@gmail.com> skribis:
> On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> >> Could you, in a first patch, move ‘origin->json’ out of
>> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> >> feature?
>> >
>> > Yes, I will try but I am a bit lost. There is:
>> > - packages-json-builder that I did not modified
>> > - sources-json-builder which the "adaptation" of the former to output
>> > to the sources.json format.
>> >
>> > Well, do you want a refactor of 'origin->json' shared by the 2
>> > "{sources,packages}-json-builder"?
>>
>> Yes, exactly. No reason to have two copies of that code.
>
> The minor "issue" is the 'match' piece of 'origin->json':
>
> `(("url" . ,(match uri
> ((? string? url) (vector url))
> ((urls ...) (list->vector urls))))))
>
>
> Because in the 'sources.json' it is always one unique string.
> However, in 'packages.json' it is always a vector of one or several
> elements. And in the previous patch, the job of 'resolve' was to
> uniquify these elements for 'sources.json'.
Ah, good point. What about adding a keyword parameter to ‘origin->json’
that would tell whether to return a single URL or a list thereof?
> For example, see the package 'adns' in 'packages.json'
>
> {
> "name": "adns",
> "version": "1.5.1",
> "source": {
> "type": "url",
> "url": [
> "mirror://gnu/adns/adns-1.5.1.tar.gz",
> "http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
> ]
> },
> "synopsis": "Asynchronous DNS client library and utilities",
> "homepage": "https://www.gnu.org/software/adns/",
> "location": "gnu/packages/adns.scm:32"
> },
>
>
>
> Just to be on the same wavelength, does 'packages.json' stay as it is
> (vector for the source url field)?
I think ‘packages.json’ should still return a list of URLs.
I’ve also added a comment asking whether ‘sources.json’ could be changed
to contain a list of URLs:
https://forge.softwareheritage.org/D2025#64037
> Or is 'packages.json' modified to record only one url for the source
> url field with mirror:// expanded?
Yeah, I think we can expand mirror URLs.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-18 8:24 ` Ludovic Courtès
@ 2020-02-18 8:38 ` zimoun
2020-02-18 8:43 ` Ludovic Courtès
0 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-02-18 8:38 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 39547
Hi Ludo,
On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> > On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> >> zimoun <zimon.toutoune@gmail.com> skribis:
> >> >> Could you, in a first patch, move ‘origin->json’ out of
> >> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> >> feature?
> >> >
> >> > Yes, I will try but I am a bit lost. There is:
> >> > - packages-json-builder that I did not modified
> >> > - sources-json-builder which the "adaptation" of the former to output
> >> > to the sources.json format.
> >> >
> >> > Well, do you want a refactor of 'origin->json' shared by the 2
> >> > "{sources,packages}-json-builder"?
> >>
> >> Yes, exactly. No reason to have two copies of that code.
> >
> > The minor "issue" is the 'match' piece of 'origin->json':
> >
> > `(("url" . ,(match uri
> > ((? string? url) (vector url))
> > ((urls ...) (list->vector urls))))))
> >
> >
> > Because in the 'sources.json' it is always one unique string.
> > However, in 'packages.json' it is always a vector of one or several
> > elements. And in the previous patch, the job of 'resolve' was to
> > uniquify these elements for 'sources.json'.
>
> Ah, good point. What about adding a keyword parameter to ‘origin->json’
> that would tell whether to return a single URL or a list thereof?
Sharing the same code.
- packages.json: as it is now.
- sources.json: expand mirror and resolve to display only URL.
I am going for that.
> > Just to be on the same wavelength, does 'packages.json' stay as it is
> > (vector for the source url field)?
>
> I think ‘packages.json’ should still return a list of URLs.
Ok.
> I’ve also added a comment asking whether ‘sources.json’ could be changed
> to contain a list of URLs:
>
> https://forge.softwareheritage.org/D2025#64037
I have seen. :-)
Even, all the mirrors list should be expanded, not only the first one.
> > Or is 'packages.json' modified to record only one url for the source
> > url field with mirror:// expanded?
>
> Yeah, I think we can expand mirror URLs.
So, 'mirror://' will become a list (from '%mirrors') of URLs, right?
Cheers,
simon
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage.
2020-02-18 8:38 ` zimoun
@ 2020-02-18 8:43 ` Ludovic Courtès
0 siblings, 0 replies; 18+ messages in thread
From: Ludovic Courtès @ 2020-02-18 8:43 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
> On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
[...]
>> > Or is 'packages.json' modified to record only one url for the source
>> > url field with mirror:// expanded?
>>
>> Yeah, I think we can expand mirror URLs.
>
> So, 'mirror://' will become a list (from '%mirrors') of URLs, right?
Yup!
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list.
2020-02-10 17:04 [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage zimoun
2020-02-14 8:40 ` Ludovic Courtès
@ 2020-02-18 12:32 ` zimoun
2020-02-18 12:32 ` [bug#39547] [PATCH v2 2/2] website: Provide JSON sources list used by Software Heritage zimoun
2020-03-02 17:24 ` [bug#39547] [PATCH v3] sources.json: array instead of list zimoun
2020-03-03 18:00 ` [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)? zimoun
3 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-02-18 12:32 UTC (permalink / raw)
To: 39547; +Cc: zimoun
* website/apps/packages/builder.scm (origin->json): New procedure.
---
website/apps/packages/builder.scm | 34 ++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 10 deletions(-)
diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..d3a777e 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; Initially written by sirgazil
;;; who waives all copyright interest on this file.
@@ -37,13 +38,16 @@
#:use-module (haunt page)
#:use-module (haunt utils)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix svn-download)
#:use-module (guix utils) ;location
+ #:use-module ((guix build download) #:select (maybe-expand-mirrors))
#:use-module (json)
#:use-module (ice-9 match)
+ #:use-module ((web uri) #:select (string->uri uri->string))
#:export (builder))
@@ -84,33 +88,43 @@
;; Maximum number of packages shown on /packages.
30)
-(define (packages-json-builder)
- "Return a JSON page listing all packages."
- (define (origin->json origin)
+(define (origin->json origin)
(define method
(origin-method origin))
+ (define uri ;represented as string
+ (origin-uri origin))
+
+ (define (resolve urls)
+ (map uri->string
+ (append-map (cut maybe-expand-mirrors <> %mirrors)
+ (map string->uri urls))))
+
`((type . ,(cond ((eq? url-fetch method) 'url)
((eq? git-fetch method) 'git)
((eq? svn-fetch method) 'svn)
(else #nil)))
,@(cond ((eq? url-fetch method)
- `(("url" . ,(match (origin-uri origin)
- ((? string? url) (vector url))
- ((urls ...) (list->vector urls))))))
+ `(("url" . ,(list->vector
+ (resolve
+ (match uri
+ ((? string? url) (list url))
+ ((urls ...) urls)))))))
((eq? git-fetch method)
- `(("git_url" . ,(git-reference-url (origin-uri origin)))))
+ `(("git_url" . ,(git-reference-url uri))))
((eq? svn-fetch method)
- `(("svn_url" . ,(svn-reference-url (origin-uri origin)))))
+ `(("svn_url" . ,(svn-reference-url uri))))
(else '()))
,@(if (eq? method git-fetch)
- `(("git_ref" . ,(git-reference-commit (origin-uri origin))))
+ `(("git_ref" . ,(git-reference-commit uri)))
'())
,@(if (eq? method svn-fetch)
`(("svn_revision" . ,(svn-reference-revision
- (origin-uri origin))))
+ uri)))
'())))
+(define (packages-json-builder)
+ "Return a JSON page listing all packages."
(define (package->json package)
(define cpe-name
(assoc-ref (package-properties package) 'cpe-name))
--
2.25.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH v2 2/2] website: Provide JSON sources list used by Software Heritage.
2020-02-18 12:32 ` [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list zimoun
@ 2020-02-18 12:32 ` zimoun
0 siblings, 0 replies; 18+ messages in thread
From: zimoun @ 2020-02-18 12:32 UTC (permalink / raw)
To: 39547; +Cc: zimoun
Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
* website/apps/packages/builder.scm (origin->json): Add list modifier.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
website/apps/packages/builder.scm | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index d3a777e..49721f6 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -74,6 +74,7 @@
(flatten
(list
(index-builder)
+ (sources-json-builder)
(packages-json-builder)
(packages-builder)
(package-list-builder))))
@@ -88,7 +89,7 @@
;; Maximum number of packages shown on /packages.
30)
-(define (origin->json origin)
+(define (origin->json origin transformer)
(define method
(origin-method origin))
@@ -105,7 +106,7 @@
((eq? svn-fetch method) 'svn)
(else #nil)))
,@(cond ((eq? url-fetch method)
- `(("url" . ,(list->vector
+ `(("url" . ,(transformer
(resolve
(match uri
((? string? url) (list url))
@@ -136,7 +137,7 @@
,@(if cpe-name `(("cpe_name" . ,cpe-name)) '())
,@(if cpe-version `(("cpe_version" . ,cpe-version)) '())
,@(if (origin? (package-source package))
- `(("source" . ,(origin->json (package-source package))))
+ `(("source" . ,(origin->json (package-source package) list->vector)))
'())
("synopsis" . ,(package-synopsis package))
,@(if (package-home-page package)
@@ -155,6 +156,21 @@
(list->vector (map package->json (all-packages)))
scm->json))
+(define (sources-json-builder)
+ "Return a JSON page listing all the sources.
+
+See <https://forge.softwareheritage.org/D2025#51269>."
+ (define (package->json package)
+ `(,@(if (origin? (package-source package))
+ (origin->json (package-source package) car)
+ `(("type" . "no-origin")
+ ("name" . ,(package-name package))))))
+
+ (make-page "sources.json"
+ `(("sources" . ,(list->vector (map package->json (all-packages))))
+ ("version" . "1"))
+ scm->json))
+
(define (index-builder)
"Return a Haunt page listing some random packages."
(define (sample n from)
--
2.25.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH v3] sources.json: array instead of list
2020-02-10 17:04 [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage zimoun
2020-02-14 8:40 ` Ludovic Courtès
2020-02-18 12:32 ` [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list zimoun
@ 2020-03-02 17:24 ` zimoun
2020-03-06 11:01 ` Ludovic Courtès
2020-03-03 18:00 ` [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)? zimoun
3 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-03-02 17:24 UTC (permalink / raw)
To: 39547, Ludovic Courtès
[-- Attachment #1: Type: text/plain, Size: 337 bytes --]
Hi Ludo,
Attached, the tiny modification to output the list (array) of URLs
instead of the first one.
The version number is still '1' because I do not know yet if lewo would bump it.
Note also that "transformer" is not useful any more because
'packages-json-builder' and 'sources-json-builder' returns both
vectors now.
Cheers,
simon
[-- Attachment #2: v3-0001-website-Refactor-and-resolve-mirror-of-JSON-packa.patch --]
[-- Type: text/x-patch, Size: 3716 bytes --]
From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Mon, 10 Feb 2020 17:52:13 +0100
Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
package list.
* website/apps/packages/builder.scm (origin->json): New procedure.
---
website/apps/packages/builder.scm | 34 ++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 10 deletions(-)
diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..d3a777e 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; Initially written by sirgazil
;;; who waives all copyright interest on this file.
@@ -37,13 +38,16 @@
#:use-module (haunt page)
#:use-module (haunt utils)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix svn-download)
#:use-module (guix utils) ;location
+ #:use-module ((guix build download) #:select (maybe-expand-mirrors))
#:use-module (json)
#:use-module (ice-9 match)
+ #:use-module ((web uri) #:select (string->uri uri->string))
#:export (builder))
@@ -84,33 +88,43 @@
;; Maximum number of packages shown on /packages.
30)
-(define (packages-json-builder)
- "Return a JSON page listing all packages."
- (define (origin->json origin)
+(define (origin->json origin)
(define method
(origin-method origin))
+ (define uri ;represented as string
+ (origin-uri origin))
+
+ (define (resolve urls)
+ (map uri->string
+ (append-map (cut maybe-expand-mirrors <> %mirrors)
+ (map string->uri urls))))
+
`((type . ,(cond ((eq? url-fetch method) 'url)
((eq? git-fetch method) 'git)
((eq? svn-fetch method) 'svn)
(else #nil)))
,@(cond ((eq? url-fetch method)
- `(("url" . ,(match (origin-uri origin)
- ((? string? url) (vector url))
- ((urls ...) (list->vector urls))))))
+ `(("url" . ,(list->vector
+ (resolve
+ (match uri
+ ((? string? url) (list url))
+ ((urls ...) urls)))))))
((eq? git-fetch method)
- `(("git_url" . ,(git-reference-url (origin-uri origin)))))
+ `(("git_url" . ,(git-reference-url uri))))
((eq? svn-fetch method)
- `(("svn_url" . ,(svn-reference-url (origin-uri origin)))))
+ `(("svn_url" . ,(svn-reference-url uri))))
(else '()))
,@(if (eq? method git-fetch)
- `(("git_ref" . ,(git-reference-commit (origin-uri origin))))
+ `(("git_ref" . ,(git-reference-commit uri)))
'())
,@(if (eq? method svn-fetch)
`(("svn_revision" . ,(svn-reference-revision
- (origin-uri origin))))
+ uri)))
'())))
+(define (packages-json-builder)
+ "Return a JSON page listing all packages."
(define (package->json package)
(define cpe-name
(assoc-ref (package-properties package) 'cpe-name))
--
2.25.0
[-- Attachment #3: v3-0002-website-Provide-JSON-sources-list-used-by-Softwar.patch --]
[-- Type: text/x-patch, Size: 2326 bytes --]
From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Tue, 18 Feb 2020 13:25:14 +0100
Subject: [PATCH v3 2/2] website: Provide JSON sources list used by Software
Heritage.
Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
* website/apps/packages/builder.scm (origin->json): Add list modifier.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
website/apps/packages/builder.scm | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index d3a777e..3fc1285 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -74,6 +74,7 @@
(flatten
(list
(index-builder)
+ (sources-json-builder)
(packages-json-builder)
(packages-builder)
(package-list-builder))))
@@ -88,7 +89,7 @@
;; Maximum number of packages shown on /packages.
30)
-(define (origin->json origin)
+(define* (origin->json origin #:optional (transformer list->vector))
(define method
(origin-method origin))
@@ -105,7 +106,7 @@
((eq? svn-fetch method) 'svn)
(else #nil)))
,@(cond ((eq? url-fetch method)
- `(("url" . ,(list->vector
+ `(("url" . ,(transformer
(resolve
(match uri
((? string? url) (list url))
@@ -155,6 +156,21 @@
(list->vector (map package->json (all-packages)))
scm->json))
+(define (sources-json-builder)
+ "Return a JSON page listing all the sources.
+
+See <https://forge.softwareheritage.org/D2025#51269>."
+ (define (package->json package)
+ `(,@(if (origin? (package-source package))
+ (origin->json (package-source package))
+ `(("type" . "no-origin")
+ ("name" . ,(package-name package))))))
+
+ (make-page "sources.json"
+ `(("sources" . ,(list->vector (map package->json (all-packages))))
+ ("version" . "1"))
+ scm->json))
+
(define (index-builder)
"Return a Haunt page listing some random packages."
(define (sample n from)
--
2.25.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?
2020-02-10 17:04 [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage zimoun
` (2 preceding siblings ...)
2020-03-02 17:24 ` [bug#39547] [PATCH v3] sources.json: array instead of list zimoun
@ 2020-03-03 18:00 ` zimoun
2020-03-05 16:19 ` Ludovic Courtès
3 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-03-03 18:00 UTC (permalink / raw)
To: 39547, Ludovic Courtès
Hi,
Some upstream sources could disappear between the moment when the
package enters in Guix and the moment when SWH ingests it. (As an edge
case, let consider the recent harfbuzz one.)
Now the format 'sources.json' should accept an array. Does it make
sense to put in this array the content addressed mirrors
(ci.guix.gnu.org and tarball.nixos.org)?
What do you think?
Cheers,
simon
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?
2020-03-03 18:00 ` [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)? zimoun
@ 2020-03-05 16:19 ` Ludovic Courtès
2020-03-06 9:26 ` zimoun
0 siblings, 1 reply; 18+ messages in thread
From: Ludovic Courtès @ 2020-03-05 16:19 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hello!
zimoun <zimon.toutoune@gmail.com> skribis:
> Some upstream sources could disappear between the moment when the
> package enters in Guix and the moment when SWH ingests it. (As an edge
> case, let consider the recent harfbuzz one.)
>
> Now the format 'sources.json' should accept an array. Does it make
> sense to put in this array the content addressed mirrors
> (ci.guix.gnu.org and tarball.nixos.org)?
Yes, I think ‘sources.json’ should contain, for each item, a list of
URLs rather than a single URL, and mirror:// URLs should be expanded.
However, ‘sources.json’ is currently specified to contain a single URL,
so we’re stuck with a single URL for now.
Is that what you meant?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?
2020-03-05 16:19 ` Ludovic Courtès
@ 2020-03-06 9:26 ` zimoun
2020-03-06 10:57 ` Ludovic Courtès
0 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-03-06 9:26 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 39547
Hi Ludo,
On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Some upstream sources could disappear between the moment when the
> > package enters in Guix and the moment when SWH ingests it. (As an edge
> > case, let consider the recent harfbuzz one.)
> >
> > Now the format 'sources.json' should accept an array. Does it make
> > sense to put in this array the content addressed mirrors
> > (ci.guix.gnu.org and tarball.nixos.org)?
>
> Yes, I think ‘sources.json’ should contain, for each item, a list of
> URLs rather than a single URL, and mirror:// URLs should be expanded.
Yes, now it is accepted [1] and it becomes the format of the
'sources.json' file.
[1] https://forge.softwareheritage.org/D2025#65063
The patch set v2 supports the first discussion of the format (single
url); which is not relevant anymore.
The patch set attached in v3 [2] supports the array of URLs.
[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35
> However, ‘sources.json’ is currently specified to contain a single URL,
> so we’re stuck with a single URL for now.
>
> Is that what you meant?
What I mean is: append %content-addressed-mirrors for each package in
the list of URLs.
Cheers,
simon
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)?
2020-03-06 9:26 ` zimoun
@ 2020-03-06 10:57 ` Ludovic Courtès
0 siblings, 0 replies; 18+ messages in thread
From: Ludovic Courtès @ 2020-03-06 10:57 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hi Simon!
zimoun <zimon.toutoune@gmail.com> skribis:
> On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:
[...]
>> Yes, I think ‘sources.json’ should contain, for each item, a list of
>> URLs rather than a single URL, and mirror:// URLs should be expanded.
>
> Yes, now it is accepted [1] and it becomes the format of the
> 'sources.json' file.
>
> [1] https://forge.softwareheritage.org/D2025#65063
Excellent!
> The patch set v2 supports the first discussion of the format (single
> url); which is not relevant anymore.
> The patch set attached in v3 [2] supports the array of URLs.
>
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35
Oh sorry, I hadn’t seen it.
>> However, ‘sources.json’ is currently specified to contain a single URL,
>> so we’re stuck with a single URL for now.
>>
>> Is that what you meant?
>
> What I mean is: append %content-addressed-mirrors for each package in
> the list of URLs.
Now I understand. :-) So yes, sure, the array of URLs in
‘sources.json’ could include a ci.guix.gnu.org URL and maybe the other
ones in there too. For a subsequent patch I guess?
Thanks!
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH v3] sources.json: array instead of list
2020-03-02 17:24 ` [bug#39547] [PATCH v3] sources.json: array instead of list zimoun
@ 2020-03-06 11:01 ` Ludovic Courtès
2020-03-07 22:17 ` zimoun
0 siblings, 1 reply; 18+ messages in thread
From: Ludovic Courtès @ 2020-03-06 11:01 UTC (permalink / raw)
To: zimoun; +Cc: 39547
Hi!
zimoun <zimon.toutoune@gmail.com> skribis:
> Attached, the tiny modification to output the list (array) of URLs
> instead of the first one.
> The version number is still '1' because I do not know yet if lewo would bump it.
> Note also that "transformer" is not useful any more because
> 'packages-json-builder' and 'sources-json-builder' returns both
> vectors now.
Cool.
> From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Mon, 10 Feb 2020 17:52:13 +0100
> Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
> package list.
>
> * website/apps/packages/builder.scm (origin->json): New procedure.
LGTM!
> From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Tue, 18 Feb 2020 13:25:14 +0100
> Subject: [PATCH v3 2/2] website: Provide JSON sources list used by Software
> Heritage.
>
> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (origin->json): Add list modifier.
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.
Nitpick: no need to repeat the file name here.
Also: please mention the addition to the ‘builder’ procedure.
> +(define* (origin->json origin #:optional (transformer list->vector))
You can remove this parameter, as you wrote.
I can make these changes and push it on your behalf if you want, let me
know!
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [bug#39547] [PATCH v3] sources.json: array instead of list
2020-03-06 11:01 ` Ludovic Courtès
@ 2020-03-07 22:17 ` zimoun
2020-03-09 9:53 ` bug#39547: " Ludovic Courtès
0 siblings, 1 reply; 18+ messages in thread
From: zimoun @ 2020-03-07 22:17 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 39547@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]
Hi Ludo,
Yes please push.
Cheers,
simon
PS: i am typing with my smartphone because i am sleeping in a hospital
(Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
after a shock to my neck during a basket ball party with some friends.
Everything is on the process but I cannot say when i will have access to my
laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:
On Friday, 6 March 2020, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Attached, the tiny modification to output the list (array) of URLs
> > instead of the first one.
> > The version number is still '1' because I do not know yet if lewo would
> bump it.
> > Note also that "transformer" is not useful any more because
> > 'packages-json-builder' and 'sources-json-builder' returns both
> > vectors now.
>
> Cool.
>
> > From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune@gmail.com>
> > Date: Mon, 10 Feb 2020 17:52:13 +0100
> > Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
> > package list.
> >
> > * website/apps/packages/builder.scm (origin->json): New procedure.
>
> LGTM!
>
> > From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune@gmail.com>
> > Date: Tue, 18 Feb 2020 13:25:14 +0100
> > Subject: [PATCH v3 2/2] website: Provide JSON sources list used by
> Software
> > Heritage.
> >
> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (origin->json): Add list modifier.
> > * website/apps/packages/builder.scm (sources-json-builder): New
> procedure.
>
> Nitpick: no need to repeat the file name here.
>
> Also: please mention the addition to the ‘builder’ procedure.
>
> > +(define* (origin->json origin #:optional (transformer list->vector))
>
> You can remove this parameter, as you wrote.
>
> I can make these changes and push it on your behalf if you want, let me
> know!
>
> Thanks,
> Ludo’.
>
[-- Attachment #2: Type: text/html, Size: 2888 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* bug#39547: [PATCH v3] sources.json: array instead of list
2020-03-07 22:17 ` zimoun
@ 2020-03-09 9:53 ` Ludovic Courtès
0 siblings, 0 replies; 18+ messages in thread
From: Ludovic Courtès @ 2020-03-09 9:53 UTC (permalink / raw)
To: zimoun; +Cc: 39547@debbugs.gnu.org
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
> Yes please push.
Done! <https://guix.gnu.org/sources.json> should show up within an hour.
> PS: i am typing with my smartphone because i am sleeping in a hospital
> (Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
> after a shock to my neck during a basket ball party with some friends.
> Everything is on the process but I cannot say when i will have access to my
> laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:
Ouch, I wish you quick recovery!
Take care of yourself,
Ludo’.
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2020-03-09 9:54 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-02-10 17:04 [bug#39547] [PATCH] website: Provide JSON sources list used by Software Heritage zimoun
2020-02-14 8:40 ` Ludovic Courtès
2020-02-14 9:04 ` zimoun
2020-02-14 10:20 ` Ludovic Courtès
2020-02-17 17:59 ` zimoun
2020-02-18 8:24 ` Ludovic Courtès
2020-02-18 8:38 ` zimoun
2020-02-18 8:43 ` Ludovic Courtès
2020-02-18 12:32 ` [bug#39547] [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON package list zimoun
2020-02-18 12:32 ` [bug#39547] [PATCH v2 2/2] website: Provide JSON sources list used by Software Heritage zimoun
2020-03-02 17:24 ` [bug#39547] [PATCH v3] sources.json: array instead of list zimoun
2020-03-06 11:01 ` Ludovic Courtès
2020-03-07 22:17 ` zimoun
2020-03-09 9:53 ` bug#39547: " Ludovic Courtès
2020-03-03 18:00 ` [bug#39547] Addition of %content-addressed-mirrors to sources.json (SWH)? zimoun
2020-03-05 16:19 ` Ludovic Courtès
2020-03-06 9:26 ` zimoun
2020-03-06 10:57 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).