[bug#41690] [PATCH 0/1] Add svn-multi, hg, url/{tarbomb, zipbomb} to {sources, packages}.json

[bug#41690] [PATCH 0/1] Add svn-multi, hg, url/{tarbomb, zipbomb} to {sources, packages}.json

[zimoun]

Dear,

This patch applies against commit 4efa5ce of guix-artwork.  It adds missing
sources; in preparation to SWH fetcher.  Let me know if it is preferable to
split it in 3 parts: svn-multi, hg and url bombs.


The source of packages is summarized by,

--8<---------------cut here---------------start------------->8---
guix build -f /tmp/origin/lister.scm \
    | grep '#<procedure' | cut -d' ' -f2 | sort | uniq -c | sort
      1 bzr-fetch
      3 cvs-fetch
      9 url-fetch/tarbomb
     24 url-fetch/zipbomb
     28 hg-fetch
     30 computed-origin-method
     67 no-origin
    115 svn-fetch
    135 svn-multi-fetch
   3574 git-fetch
   9690 url-fetch
--8<---------------cut here---------------end--------------->8---



These sources are still unarchivable by SWH:

      1 bzr-fetch
      3 cvs-fetch
     30 computed-origin-method
     67 no-origin


Well, the 4 "historical" packages are

--8<---------------cut here---------------start------------->8---
guix build -f /tmp/origin/lister.scm \
    | grep ';;' | grep -E '(cvs|bzr)'
;; "bzr-fetch" 	# "libmemcached"
;; "cvs-fetch" 	# "emacs-w3m"
;; "cvs-fetch" 	# "gnu-standards"
;; "cvs-fetch" 	# "tidy"
--8<---------------cut here---------------end--------------->8---

and "guix refresh -l" says

--8<---------------cut here---------------start------------->8---
libmemcached
Building the following 2 packages would ensure 2 dependent packages are rebuilt: python-pylibmc@1.6.1 python2-pylibmc@1.6.1

emacs-w3m
No dependents other than itself: emacs-w3m@2018-11-11

gnu-standards
No dependents other than itself: gnu-standards@2018-02-18

tidy
Building the following 5 packages would ensure 9 dependent packages are rebuilt: hoedown@3.0.7 hugin@2019.2.0 emacs-telega@0.6.0-0.ae09592 arcanist@0.0.0-1.45a8d22 pumpa@0.9.3
--8<---------------cut here---------------end--------------->8---


What do we do for these 4 packages?  Especially libmemcached and tidy?
Last, what about the 30 'computed-origin-method' packages?


All the best,
simon


--8<---------------cut here---------------start------------->8---
(define-module (lister)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix svn-download)
  #:use-module (guix hg-download)
  #:use-module (guix cvs-download)
  #:use-module (guix bzr-download)
  #:use-module (gnu packages))

(fold-packages (lambda (package result)
                 (let ((method
                        (if (origin? (package-source package))
                            (let* ((method (origin-method
                                            (package-source package)))
                                   (pkg (package-name package))
                                   (print (lambda (p m)
                                            (format #t ";; ~s \t# ~s\n" m p))))
                              (cond
                               ((eq? method bzr-fetch)
                                (print pkg "bzr-fetch"))
                               ((eq? method cvs-fetch)
                                (print pkg "cvs-fetch"))
                               ((eq? method svn-multi-fetch)
                                (print pkg "svn-multi-fetch"))
                               ((eq? method hg-fetch)
                                (print pkg "hg-fetch")))
                              method)
                            (begin
                              (format #t ";; no-origin \t# ~s\n" (package-name
                                                                  package))
                              "#<procedure no-origin >"))))
                   (format #t "~s\n" method)))
               #f)


(format #t "\n\n")
(specification->package "hello")
--8<---------------cut here---------------end--------------->8---



zimoun (1):
  website: Add fetch methods to JSON sources and packages list.

 website/apps/packages/builder.scm | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

-- 
2.26.2

[bug#41690] [PATCH 1/1] website: Add fetch methods to JSON sources and packages list.

[zimoun]

* website/apps/packages/builder.scm (origin->json): Add 'url-fetch/tarbomb',
  'url-fetc/zipbomb', 'svn-multi-fetch' and 'hg-fetch' methods.
---
 website/apps/packages/builder.scm | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 85494a6..d2bccd7 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -43,6 +43,7 @@
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix svn-download)
+  #:use-module (guix hg-download)
   #:use-module (guix utils)                       ;location
   #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (json)
@@ -102,11 +103,17 @@
          (append-map (cut maybe-expand-mirrors <> %mirrors)
                      (map string->uri urls))))
 
-  `((type . ,(cond ((eq? url-fetch method) 'url)
+  `((type . ,(cond ((or (eq? url-fetch method)
+                        (eq? url-fetch/tarbomb method)
+                        (eq? url-fetch/zipbomb method)) 'url)
                    ((eq? git-fetch method) 'git)
-                   ((eq? svn-fetch method) 'svn)
+                   ((or (eq? svn-fetch method)
+                        (eq? svn-multi-fetch method)) 'svn)
+                   ((eq? hg-fetch method) 'hg)
                    (else                   #nil)))
-    ,@(cond ((eq? url-fetch method)
+    ,@(cond ((or (eq? url-fetch method)
+                 (eq? url-fetch/tarbomb method)
+                 (eq? url-fetch/zipbomb method))
              `(("url" . ,(list->vector
                           (resolve
                            (match uri
@@ -116,13 +123,22 @@
              `(("git_url" . ,(git-reference-url uri))))
             ((eq? svn-fetch method)
              `(("svn_url" . ,(svn-reference-url uri))))
+            ((eq? svn-multi-fetch method)
+             `(("svn_url" . ,(svn-multi-reference-url uri))))
+            ((eq? hg-fetch method)
+             `(("hg_url" . ,(hg-reference-url uri))))
             (else '()))
     ,@(if (eq? method git-fetch)
           `(("git_ref" . ,(git-reference-commit uri)))
           '())
     ,@(if (eq? method svn-fetch)
-          `(("svn_revision" . ,(svn-reference-revision
-                                uri)))
+          `(("svn_revision" . ,(svn-reference-revision uri)))
+          '())
+    ,@(if (eq? method svn-multi-fetch)
+          `(("svn_revision" . ,(svn-multi-reference-revision uri)))
+          '())
+    ,@(if (eq? method hg-fetch)
+          `(("hg_changeset" . ,(hg-reference-changeset uri)))
           '())))
 
 (define (packages-json-builder)
-- 
2.26.2

bug#41690: [PATCH 0/1] Add svn-multi, hg, url/{tarbomb, zipbomb} to {sources, packages}.json

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> The source of packages is summarized by,
>
> guix build -f /tmp/origin/lister.scm \
>     | grep '#<procedure' | cut -d' ' -f2 | sort | uniq -c | sort
>       1 bzr-fetch
>       3 cvs-fetch
>       9 url-fetch/tarbomb
>      24 url-fetch/zipbomb
>      28 hg-fetch
>      30 computed-origin-method
>      67 no-origin
>     115 svn-fetch
>     135 svn-multi-fetch
>    3574 git-fetch
>    9690 url-fetch

Interesting.

> * website/apps/packages/builder.scm (origin->json): Add 'url-fetch/tarbomb',
>   'url-fetc/zipbomb', 'svn-multi-fetch' and 'hg-fetch' methods.

Applied, thank you!

Ludo’.