[bug#62202] [PATCH 0/21] Juliahub import script.

[bug#62202] [PATCH 0/21] Juliahub import script.

[Nicolas Graves via Guix-patches via]

 Hi guix!

 Took me quite more time than I would've liked, but I have a usable
 juliahub scheme import script!

 It seems there's still one edge case that isn't covered and revolves
 around when Julia packagers don't properly tag their git repos (I've
 only seen the case with SnoopPrecompile). There's the possibility to
 rely on tree commit hashes from the General repository (since this is a
 valid way to identify/store a git repo), but that needs some major
 changes in the way latest-repository-commit works. Otherwise, it needs
 to be done by hand. It might also not work for subpackages in
 directories that are up-to-date on juliahub but not yet on github, I
 haven't met this case yet.

 I'm sending a patch series in the coming minutes.

It's detailed since I haven't swauased all commits, for readability, but
I can squash it further if necessary.

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH 01/21] import: juliahub: first script draft.

[Nicolas Graves via Guix-patches via]

---
 guix/import/go.scm               |   6 +-
 guix/import/juliahub.scm         | 183 +++++++++++++++++++++++++++++++
 guix/scripts/import.scm          |   2 +-
 guix/scripts/import/juliahub.scm | 107 ++++++++++++++++++
 4 files changed, 295 insertions(+), 3 deletions(-)
 create mode 100644 guix/import/juliahub.scm
 create mode 100644 guix/scripts/import/juliahub.scm

diff --git a/guix/import/go.scm b/guix/import/go.scm
index 69937f8a4d..f264715fbd 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -503,9 +503,11 @@ (define (transform-version version)
         '(string-append "v" version)
         '(go-version->git-ref version))))
 
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+                      #:optional transform-version)
   "Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
   (case vcs-type
     ((git)
      (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..efe6abbb24
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,183 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+  #:use-module (ice-9 textual-ports)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 string-fun)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-1)
+  #:use-module (guix http-client)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix base16)
+  #:use-module (guix base32)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module ((guix licenses) #:prefix license:)
+
+  #:export (juliahub->guix-package))
+
+(define (juliahub-uri name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (port (http-fetch url #:text? #t))
+         (_ (get-line port))
+         (meta (get-line port))
+         (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+         (redirect (match:substring (string-match regex meta))))
+    (close-port port)
+    (string-drop redirect 4)))
+
+(define (juliahub-url name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (uri (juliahub-uri name)))
+    (string-append url uri "/")))
+
+(define (juliahub-slug-version name)
+  (let* ((uri (juliahub-uri name))
+         (slug (string-take uri 5))
+         (latest-version (string-drop uri 6)))
+    `(,slug ,latest-version)))
+
+(define (json->juliahub-direct-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (juliahub-dependency-direct? dep)
+               dep
+               #f)))
+       (vector->list vector))))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+  json->juliahub-package
+  (homepage juliahub-package-homepage) ;string
+  (readme juliahub-package-readme) ;string
+  ;; (slug juliahub-package-slug) ;string
+  (version juliahub-package-version) ;string
+  (description juliahub-package-description) ;string
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+   ;; (lambda (vector)
+     ;; (map json->juliahub-dependency (vector->list vector))))
+  (url juliahub-package-url) ;string
+  (uuid juliahub-package-uuid) ;string
+  (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+  make-juliahub-dependency juliahub-dependency?
+  json->juliahub-dependency
+  (direct? juliahub-dependency-direct? "direct") ;boolean
+  (name juliahub-dependency-name) ;string
+  (uuid juliahub-dependency-uuid) ;string
+  (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+  ;; (slug juliahub-dependency-slug) ;string
+
+(define (julia-name->guix-name name)
+  (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+  "Return a <juliahub-package> record for package NAME, or #f on failure."
+  (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
+         json->juliahub-package))
+
+(define (make-julia-sexp name version uri hash home-page synopsis description
+                         dependencies licenses)
+  "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+  `(package
+     (name ,(julia-name->guix-name name))
+     (version ,version)
+     (source (origin
+               (method url-fetch)
+               (uri ,uri)
+               (sha256
+                (base32
+                 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+                 ;; ,(bytevector->nix-base32-string hash)
+                 ))))
+     (build-system julia-build-system)
+     ,@(if (null? dependencies)
+           '()
+           `((inputs
+              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                           dependencies)))))
+     (synopsis ,synopsis)
+     (description ,description)
+     (home-page ,home-page)
+     (license ,(match licenses
+                 (() #f)
+                 ((license) (license->symbol license))
+                 (_ `(list ,@(map license->symbol licenses)))))))
+
+(define* (juliahub->guix-package package-name
+                                 #:key version #:allow-other-keys)
+  "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+  (let ((package (if version
+                      (juliahub-fetch package-name version)
+                      (juliahub-fetch package-name))))
+    (if package
+        (let* ((dependencies-names
+                (map juliahub-dependency-name
+                     (juliahub-package-dependencies package)))
+               (licenses
+                (map spdx-string->license
+                     (list (juliahub-package-license package)))))
+          (values (make-julia-sexp
+                   package-name
+                   (juliahub-package-version package)
+                   (juliahub-package-url package)
+                   "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+                   (juliahub-package-homepage package)
+                   (juliahub-package-description package)
+                   (beautify-description (juliahub-package-readme package))
+                   (juliahub-package-dependencies package)
+                   licenses)
+                  dependencies-names))
+        (values #f '()))))
+
+(define* (import-release package #:key (version #f))
+  "Return an <upstream-source> for the latest release of PACKAGE."
+  (let* ((package-name (guix-package->juliahub-name package))
+         (package      (juliahub-fetch package-name))
+         (version  (or version (juliahub-version gem)))
+         (url      (rubyjuliahubs-uri gem-name version)))
+    (upstream-source
+     (package (package-name package))
+     (version version)
+     (urls (list url)))))
+
+(define %juliahub-updater
+  (upstream-updater
+   (name 'juliahub)
+   (description "Updater for Juliahub packages")
+   (pred juliahub-package?)
+   (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+  (recursive-import package-name
+                    #:repo '()
+                    #:repo->guix-package juliahub->guix-package
+                    #:guix-name ruby-package-name
+                    #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index f84a964a53..ef4e0b9cc6 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
 
 (define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
                     "gem" "go" "cran" "crate" "texlive" "json" "opam"
-                    "minetest" "elm" "hexpm"))
+                    "minetest" "elm" "hexpm" "juliahub"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import juliahub)
+  #:use-module (guix scripts import)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 receive)
+  #:export (guix-import-juliahub))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME.  Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (display (G_ "
+  -r, --recursive        generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import gem")))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
+         %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (parse-command-line args %options (list %default-options)
+                        #:build-options? #f))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                             (('argument . value)
+                              value)
+                             (_ #f))
+                           (reverse opts))))
+    (match args
+      ((spec)
+       (receive (package-name package-version)
+           (package-name->name+version spec)
+         (let ((code (if (assoc-ref opts 'recursive)
+                         (map (match-lambda
+                                ((and ('package ('name name) . rest) pkg)
+                                 `(define-public ,(string->symbol name)
+                                    ,pkg))
+                                (_ #f))
+                              (juliahub-recursive-import package-name package-version))
+                         (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+                           (if sexp sexp #f)))))
+           (match code
+             ((or #f '(#f))
+              (leave (G_ "failed to download meta-data for package '~a'~%")
+                     package-name))
+             (_ code)))))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.39.2

[bug#62202] [PATCH 02/21] import: utils: Change git->origin function to git->origin+version.

[Nicolas Graves via Guix-patches via]

---
 guix/import/go.scm    |  2 +-
 guix/import/utils.scm | 50 +++++++++++++++++++++++++++----------------
 2 files changed, 33 insertions(+), 19 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index f264715fbd..3be82ed164 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -510,7 +510,7 @@ (define* (vcs->origin vcs-type vcs-repo-url version
 which takes version as an input."
   (case vcs-type
     ((git)
-     (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
+      (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
     ((hg)
      `(origin
         (method hg-fetch)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 529d7f11f8..171dca54e8 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -65,6 +65,7 @@ (define-module (guix import utils)
 
             url-fetch
             guix-hash-url
+            git->origin+dir
             git->origin
 
             package-names->package-inputs
@@ -156,9 +157,12 @@ (define (guix-hash-url filename)
   "Return the hash of FILENAME in nix-base32 format."
   (bytevector->nix-base32-string (file-sha256 filename)))
 
-(define* (git->origin repo-url ref #:optional ref->commit)
-  "Generate the `origin' block of a package depending on the git source
-control system. REPO-URL or REF can be null."
+(define* (git->origin+dir repo-url ref #:optional ref->commit)
+  "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary.  REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
   (let-values (((directory commit)
                 (with-store store
                   (latest-repository-commit store repo-url #:ref ref))))
@@ -172,21 +176,31 @@ (define* (git->origin repo-url ref #:optional ref->commit)
                        version)
                       (_
                        (ref->commit version)))))
-      `(origin
-         (method git-fetch)
-         (uri (git-reference
-               (url ,(and (not (eq? repo-url 'null)) repo-url))
-               (commit ,vcommit)))
-         (file-name (git-file-name name version))
-         (sha256
-          (base32
-           ,(if (pair? ref)
-                (bytevector->nix-base32-string
-                 (file-hash* directory
-                             ;; 'git-fetch' already filtered out the '.git' directory.
-                             #:select? (const #true)
-                             #:recursive? #true))
-                #f)))))))
+      (values
+       `(origin
+          (method git-fetch)
+          (uri (git-reference
+                (url ,(and (not (eq? repo-url 'null)) repo-url))
+                (commit ,vcommit)))
+          (file-name (git-file-name name version))
+          (sha256
+           (base32
+            ,(if (pair? ref)
+                 (bytevector->nix-base32-string
+                  (file-hash* directory
+                              ;; 'git-fetch' already filtered out the '.git' directory.
+                              #:select? (const #true)
+                              #:recursive? #true))
+                 #f))))
+       directory))))
+
+(define* (git->origin repo-url ref #:optional ref->commit)
+  "Returns a generated `origin' block of a package depending on the git source
+control system.  REPO-URL or REF can be null. REF->COMMIT can be a function or
+#t, in which case the commit matching ref is used. If REF->COMMIT is not used,
+the value inside REF is used."
+  (let-values (((origin _) (git->origin+dir repo-url ref ref->commit)))
+    origin))
 
 (define %spdx-license-identifiers
   ;; https://spdx.org/licenses/
-- 
2.39.2

[bug#62202] [PATCH 03/21] import: juliahub: Add support for native-inputs.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 105 ++++++++++++++++++++++++---------------
 1 file changed, 64 insertions(+), 41 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index efe6abbb24..4544dee980 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -20,13 +20,14 @@ (define-module (guix import juliahub)
   #:use-module (ice-9 textual-ports)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 match)
+  #:use-module (ice-9 streams)
   #:use-module (ice-9 string-fun)
-  #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-11)
   #:use-module (guix http-client)
   #:use-module (guix import utils)
   #:use-module (guix import json)
-  #:use-module (guix base16)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
@@ -66,19 +67,42 @@ (define (json->juliahub-direct-dependencies vector)
                #f)))
        (vector->list vector))))
 
+(define (ini-list->extra-dependencies lst)
+  (match lst
+    (('(extras) ooo ...)
+     (extra-list->extra-dependencies ooo))
+    (((tag) ooo ...)
+     (ini-list->extra-dependencies ooo))
+    ((attribute '= value ooo ...)
+     (ini-list->extra-dependencies ooo))
+    ('()
+     '())))
+
+(define (extra-list->extra-dependencies lst)
+  (match lst
+    ((attribute '= value ooo ...)
+     `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
+    (((tag) ooo ...)
+     '())
+    ('()
+     '())))
+
+(define (parse-extra-dependencies directory)
+  (let* ((port (open-input-file (string-append directory "/Project.toml")))
+         (ini-list (stream->list (port->stream port read))))
+    (close-port port)
+    (ini-list->extra-dependencies ini-list)))
+
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   json->juliahub-package
   (homepage juliahub-package-homepage) ;string
   (readme juliahub-package-readme) ;string
-  ;; (slug juliahub-package-slug) ;string
   (version juliahub-package-version) ;string
   (description juliahub-package-description) ;string
-  (dependencies
-   juliahub-package-dependencies "deps"
+  (direct-dependencies
+   juliahub-package-direct-dependencies "deps"
    json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
-   ;; (lambda (vector)
-     ;; (map json->juliahub-dependency (vector->list vector))))
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -90,7 +114,6 @@ (define-json-mapping <juliahub-dependency>
   (name juliahub-dependency-name) ;string
   (uuid juliahub-dependency-uuid) ;string
   (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
-  ;; (slug juliahub-dependency-slug) ;string
 
 (define (julia-name->guix-name name)
   (string-append "julia-" (snake-case name)))
@@ -100,27 +123,25 @@ (define* (juliahub-fetch name #:key (version #f))
   (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
          json->juliahub-package))
 
-(define (make-julia-sexp name version uri hash home-page synopsis description
-                         dependencies licenses)
+(define (make-julia-sexp name source home-page synopsis description
+                         direct-dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSES."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
-     (source (origin
-               (method url-fetch)
-               (uri ,uri)
-               (sha256
-                (base32
-                 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
-                 ;; ,(bytevector->nix-base32-string hash)
-                 ))))
+     (source ,source)
      (build-system julia-build-system)
-     ,@(if (null? dependencies)
+     ,@(if (null? direct-dependencies)
            '()
-           `((inputs
+           `((propagated-inputs
               (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           dependencies)))))
+                           direct-dependencies)))))
+     ,@(if (null? test-dependencies-names)
+           '()
+           `((native-inputs
+              (list ,@(map julia-name->guix-name test-dependencies-names)))))
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
@@ -135,26 +156,28 @@ (define* (juliahub->guix-package package-name
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
   (let ((package (if version
-                      (juliahub-fetch package-name version)
-                      (juliahub-fetch package-name))))
+                     (juliahub-fetch package-name version)
+                     (juliahub-fetch package-name))))
     (if package
-        (let* ((dependencies-names
-                (map juliahub-dependency-name
-                     (juliahub-package-dependencies package)))
-               (licenses
-                (map spdx-string->license
-                     (list (juliahub-package-license package)))))
-          (values (make-julia-sexp
-                   package-name
-                   (juliahub-package-version package)
-                   (juliahub-package-url package)
-                   "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
-                   (juliahub-package-homepage package)
-                   (juliahub-package-description package)
-                   (beautify-description (juliahub-package-readme package))
-                   (juliahub-package-dependencies package)
-                   licenses)
-                  dependencies-names))
+        (let-values (((source directory)
+                     (git->origin+dir url `(tag-or-commit . ,package-version))))
+          (let* ((dependencies-names
+                  (map juliahub-dependency-name
+                       (juliahub-package-direct-dependencies package)))
+                 (licenses
+                  (map spdx-string->license
+                       (list (juliahub-package-license package))))
+                 (test-dependencies-names (parse-extra-dependencies directory)))
+            (values (make-julia-sexp
+                     package-name
+                     source
+                     (juliahub-package-homepage package)
+                     (juliahub-package-description package)
+                     (beautify-description (juliahub-package-readme package))
+                     (juliahub-package-direct-dependencies package)
+                     test-dependencies-names
+                     licenses)
+                    (append dependencies-names test-dependencies))))
         (values #f '()))))
 
 (define* (import-release package #:key (version #f))
-- 
2.39.2

[bug#62202] [PATCH 04/21] import: juliahub: Correct source parsing.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4544dee980..4c3ceed109 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -160,7 +160,11 @@ (define* (juliahub->guix-package package-name
                      (juliahub-fetch package-name))))
     (if package
         (let-values (((source directory)
-                     (git->origin+dir url `(tag-or-commit . ,package-version))))
+                      (git->origin+dir
+                       (juliahub-package-url package)
+                       `(tag-or-commit
+                         . ,(string-append
+                             "v" (juliahub-package-version package))))))
           (let* ((dependencies-names
                   (map juliahub-dependency-name
                        (juliahub-package-direct-dependencies package)))
-- 
2.39.2

[bug#62202] [PATCH 05/21] import: juliahub: Add indirect dependencies.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4c3ceed109..fb361a0044 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -67,6 +67,16 @@ (define (json->juliahub-direct-dependencies vector)
                #f)))
        (vector->list vector))))
 
+(define (json->juliahub-indirect-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (not (juliahub-dependency-direct? dep))
+               dep
+               #f)))
+       (vector->list vector))))
+
 (define (ini-list->extra-dependencies lst)
   (match lst
     (('(extras) ooo ...)
@@ -103,6 +113,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   (direct-dependencies
    juliahub-package-direct-dependencies "deps"
    json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+  (indirect-dependencies
+   juliahub-package-indirect-dependencies "deps"
+   json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -181,7 +194,7 @@ (define* (juliahub->guix-package package-name
                      (juliahub-package-direct-dependencies package)
                      test-dependencies-names
                      licenses)
-                    (append dependencies-names test-dependencies))))
+                    (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
 (define* (import-release package #:key (version #f))
-- 
2.39.2

[bug#62202] [PATCH 06/21] import: juliahub: Add updater and recursive-importer.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fb361a0044..c38c830caa 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -34,7 +34,9 @@ (define-module (guix import juliahub)
   #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
 
-  #:export (juliahub->guix-package))
+  #:export (juliahub->guix-package
+            %juliahub-updater
+            juliahub-recursive-import))
 
 (define (juliahub-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
@@ -197,16 +199,26 @@ (define* (juliahub->guix-package package-name
                     (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
+(define (guix-package->juliahub-name package)
+  (let* ((url (juliahub-package-url package))
+         (git-name (car (last-pair (string-split url #\/))))
+         (ungitted-name (if (string-suffix? ".git" git-name)
+                            (string-drop-right git-name 4)
+                            git-name))
+         (package-name (if (string-suffix? ".jl" ungitted-name)
+                            (string-drop-right ungitted-name 4)
+                            ungitted-name)))
+    package-name))
+
 (define* (import-release package #:key (version #f))
   "Return an <upstream-source> for the latest release of PACKAGE."
   (let* ((package-name (guix-package->juliahub-name package))
          (package      (juliahub-fetch package-name))
-         (version  (or version (juliahub-version gem)))
-         (url      (rubyjuliahubs-uri gem-name version)))
+         (version  (or version (juliahub-package-version package))))
     (upstream-source
      (package (package-name package))
      (version version)
-     (urls (list url)))))
+     (urls (list (juliahub-package-url package))))))
 
 (define %juliahub-updater
   (upstream-updater
@@ -219,5 +231,5 @@ (define* (juliahub-recursive-import package-name #:optional version)
   (recursive-import package-name
                     #:repo '()
                     #:repo->guix-package juliahub->guix-package
-                    #:guix-name ruby-package-name
+                    #:guix-name julia-name->guix-name
                     #:version version))
-- 
2.39.2

[bug#62202] [PATCH 07/21] import: juliahub: Filter out julia stdlibs.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 47 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caa..af08f3d698 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
             %juliahub-updater
             juliahub-recursive-import))
 
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+  (list "julia"
+        "ArgTools"
+        "Artifacts"
+        "Base64"
+        "CRC32c"
+        "FileWatching"
+        "Libdl"
+        "Logging"
+        "Mmap"
+        "NetworkOptions"
+        "SHA"
+        "Serialization"
+        "Sockets"
+        "Unicode"
+        "DelimitedFiles"
+        "LinearAlgebra"
+        "Markdown"
+        "Printf"
+        "Random"
+        "Tar"
+        "Dates"
+        "Distributed"
+        "Future"
+        "InteractiveUtils"
+        "LibGit2"
+        "Profile"
+        "SparseArrays"
+        "UUIDs"
+        "REPL"
+        "SharedArrays"
+        "Statistics"
+        "SuiteSparse"
+        "TOML"
+        "Test"
+        "LibCURL"
+        "Downloads"
+        "Pkg"
+        "LazyArtifacts"))
+
 (define (juliahub-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
          (port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (juliahub-dependency-direct? dep)
+           (if (and (juliahub-dependency-direct? dep)
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (not (juliahub-dependency-direct? dep))
+           (if (and (not (juliahub-dependency-direct? dep))
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
-- 
2.39.2

[bug#62202] [PATCH 08/21] import: juliahub: Simplify juliahub dependency management.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 42 ++++++++++++++--------------------------
 1 file changed, 14 insertions(+), 28 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index af08f3d698..b1eeb736a8 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -100,26 +100,13 @@ (define (juliahub-slug-version name)
          (latest-version (string-drop uri 6)))
     `(,slug ,latest-version)))
 
-(define (json->juliahub-direct-dependencies vector)
+(define (json->juliahub-dependencies vector)
   (if (vector? vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (and (juliahub-dependency-direct? dep)
-                    (not (member (juliahub-dependency-name dep)
-                                 %julia-stdlibs)))
-               dep
-               #f)))
-       (vector->list vector))))
-
-(define (json->juliahub-indirect-dependencies vector)
-  (if (vector? vector)
-      (filter-map
-       (lambda (el)
-         (let ((dep (json->juliahub-dependency el)))
-           (if (and (not (juliahub-dependency-direct? dep))
-                    (not (member (juliahub-dependency-name dep)
-                                 %julia-stdlibs)))
+           (if (not (member (juliahub-dependency-name dep)
+                            %julia-stdlibs))
                dep
                #f)))
        (vector->list vector))))
@@ -157,12 +144,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   (readme juliahub-package-readme) ;string
   (version juliahub-package-version) ;string
   (description juliahub-package-description) ;string
-  (direct-dependencies
-   juliahub-package-direct-dependencies "deps"
-   json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
-  (indirect-dependencies
-   juliahub-package-indirect-dependencies "deps"
-   json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-dependencies) ;list of <juliahub-dependency>
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -184,7 +168,7 @@ (define* (juliahub-fetch name #:key (version #f))
          json->juliahub-package))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         direct-dependencies test-dependencies-names licenses)
+                         dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
 VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSES."
@@ -193,11 +177,13 @@ (define (make-julia-sexp name source home-page synopsis description
      (version ,version)
      (source ,source)
      (build-system julia-build-system)
-     ,@(if (null? direct-dependencies)
-           '()
-           `((propagated-inputs
-              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           direct-dependencies)))))
+     ,@(let ((direct-dependencies
+              (filter julia-dependency-direct? dependencies)))
+         (if (null? direct-dependencies)
+             '()
+             `((propagated-inputs
+                (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                             direct-dependencies))))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
-- 
2.39.2

[bug#62202] [PATCH 09/21] import: juliahub: Improve dependency management.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b1eeb736a8..fc25ba1d42 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -168,22 +168,20 @@ (define* (juliahub-fetch name #:key (version #f))
          json->juliahub-package))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         dependencies test-dependencies-names licenses)
+                         direct-dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSES."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
      (source ,source)
      (build-system julia-build-system)
-     ,@(let ((direct-dependencies
-              (filter julia-dependency-direct? dependencies)))
-         (if (null? direct-dependencies)
-             '()
-             `((propagated-inputs
-                (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                             direct-dependencies))))))
+     ,@(if (null? direct-dependencies)
+           '()
+           `((propagated-inputs
+              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                           direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
@@ -211,9 +209,12 @@ (define* (juliahub->guix-package package-name
                        `(tag-or-commit
                          . ,(string-append
                              "v" (juliahub-package-version package))))))
-          (let* ((dependencies-names
+          (let* ((direct-dependencies
+                  (filter juliahub-dependency-direct?
+                          (juliahub-package-dependencies package)))
+                 (dependencies-names
                   (map juliahub-dependency-name
-                       (juliahub-package-direct-dependencies package)))
+                       direct-dependencies))
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
@@ -224,7 +225,7 @@ (define* (juliahub->guix-package package-name
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
                      (beautify-description (juliahub-package-readme package))
-                     (juliahub-package-direct-dependencies package)
+                     direct-dependencies
                      test-dependencies-names
                      licenses)
                     (append dependencies-names test-dependencies-names))))
-- 
2.39.2

[bug#62202] [PATCH 10/21] import: juliahub: Add functions to parse the git repo for a git tag.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 56 insertions(+), 6 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fc25ba1d42..5327e92325 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -26,6 +26,7 @@ (define-module (guix import juliahub)
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-11)
   #:use-module (guix http-client)
+  #:use-module (guix git)
   #:use-module (guix import utils)
   #:use-module (guix import json)
   #:use-module (guix base32)
@@ -38,6 +39,48 @@ (define-module (guix import juliahub)
             %juliahub-updater
             juliahub-recursive-import))
 
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-version, and
+;; import this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag, and where looking for the commit is tedious and artificial. In these
+;; cases, we introduce the tree-commit which is available in the Versions.toml
+;; file in the General repository. This is equivalent to a commit, since we have
+;; a unique hash of the listing of files and directories, thus it can be used to
+;; identify the state of a repository.
+
+(define %general-base-url
+  "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+  (let ((folder (string-capitalize (string-take package-name 1))))
+    (string-append
+     %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-list->alist lst)
+  (match lst
+    ((attribute '= value ooo ...)
+     `((,attribute . ,value) ,@(ini-list->alist ooo)))
+    ('()
+     '())))
+
+(define (ini-fetch url)
+  (let* ((port (http-fetch url #:text? #t))
+         (ini-list (stream->list (port->stream port read))))
+    (close-port port)
+    (ini-list->alist ini-list)))
+
+(define (latest-git-tag repo)
+  (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+         (last-git-tag (last (string-split last-ref #\/))))
+    (string-drop last-git-tag 1)))
+
 ;; To update, see file sysimg.jl
 (define %julia-stdlibs
   (list "julia"
@@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description
                  ((license) (license->symbol license))
                  (_ `(list ,@(map license->symbol licenses)))))))
 
+;; TODO handle subdir case properly.
+
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let ((package (if version
-                     (juliahub-fetch package-name version)
-                     (juliahub-fetch package-name))))
+  (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+         (subdir (assoc-ref package-toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+         (package (if version
+                      (juliahub-fetch package-name version)
+                      (if tag
+                          (juliahub-fetch package-name tag)
+                          (juliahub-fetch package-name)))))
     (if package
         (let-values (((source directory)
                       (git->origin+dir
@@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name
 
 (define (guix-package->juliahub-name package)
   (let* ((url (juliahub-package-url package))
-         (git-name (car (last-pair (string-split url #\/))))
+         (git-name (last (string-split url #\/)))
          (ungitted-name (if (string-suffix? ".git" git-name)
                             (string-drop-right git-name 4)
                             git-name))
          (package-name (if (string-suffix? ".jl" ungitted-name)
-                            (string-drop-right ungitted-name 4)
-                            ungitted-name)))
+                           (string-drop-right ungitted-name 4)
+                           ungitted-name)))
     package-name))
 
 (define* (import-release package #:key (version #f))
-- 
2.39.2

[bug#62202] [PATCH 11/21] import: juliahub: Improve test dependencies parsing.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 5327e92325..2ea461b72a 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,31 +154,27 @@ (define (json->juliahub-dependencies vector)
                #f)))
        (vector->list vector))))
 
-(define (ini-list->extra-dependencies lst)
+(define (ini-list->test-dependencies lst)
   (match lst
-    (('(extras) ooo ...)
-     (extra-list->extra-dependencies ooo))
-    (((tag) ooo ...)
-     (ini-list->extra-dependencies ooo))
-    ((attribute '= value ooo ...)
-     (ini-list->extra-dependencies ooo))
+    (('test '= ooo ...)
+     `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
+    ((value ooo ...)
+     (ini-list->test-dependencies ooo))
     ('()
-     '())))
+      '())))
 
-(define (extra-list->extra-dependencies lst)
+(define (test-list->test-dependencies lst)
   (match lst
-    ((attribute '= value ooo ...)
-     `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
-    (((tag) ooo ...)
-     '())
+    ((('unquote value) ooo ...)
+     `(,value ,@(test-list->test-dependencies ooo)))
     ('()
-     '())))
+      '())))
 
-(define (parse-extra-dependencies directory)
+(define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
          (ini-list (stream->list (port->stream port read))))
     (close-port port)
-    (ini-list->extra-dependencies ini-list)))
+    (ini-list->test-dependencies ini-list)))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -268,7 +264,7 @@ (define* (juliahub->guix-package package-name
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
-                 (test-dependencies-names (parse-extra-dependencies directory)))
+                 (test-dependencies-names (parse-test-dependencies directory)))
             (values (make-julia-sexp
                      package-name
                      source
-- 
2.39.2

[bug#62202] [PATCH 12/21] import: juliahub: Handle the case where we have a subdirectory.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 2ea461b72a..b646f93295 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -233,14 +233,12 @@ (define (make-julia-sexp name source home-page synopsis description
                  ((license) (license->symbol license))
                  (_ `(list ,@(map license->symbol licenses)))))))
 
-;; TODO handle subdir case properly.
-
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
          (subdir (assoc-ref package-toml 'subdir))
          (tag (latest-git-tag (assoc-ref package-toml 'repo)))
          (package (if version
@@ -264,7 +262,11 @@ (define* (juliahub->guix-package package-name
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
-                 (test-dependencies-names (parse-test-dependencies directory)))
+                 (test-dependencies-names
+                  (if subdir
+                      (parse-test-dependencies
+                       (string-append subdir "/" directory))
+                      (parse-test-dependencies directory))))
             (values (make-julia-sexp
                      package-name
                      source
-- 
2.39.2

[bug#62202] [PATCH 13/21] import: juliahub: Add support for versions for juliahub-fetch.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b646f93295..6ce0487dba 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -122,7 +122,7 @@ (define %julia-stdlibs
         "Pkg"
         "LazyArtifacts"))
 
-(define (juliahub-uri name)
+(define (juliahub-redirect-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
          (port (http-fetch url #:text? #t))
          (_ (get-line port))
@@ -134,11 +134,11 @@ (define (juliahub-uri name)
 
 (define (juliahub-url name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
-         (uri (juliahub-uri name)))
+         (uri (juliahub-redirect-uri name)))
     (string-append url uri "/")))
 
-(define (juliahub-slug-version name)
-  (let* ((uri (juliahub-uri name))
+(define (juliahub-slug+version name)
+  (let* ((uri (juliahub-redirect-uri name))
          (slug (string-take uri 5))
          (latest-version (string-drop uri 6)))
     `(,slug ,latest-version)))
@@ -203,8 +203,12 @@ (define (julia-name->guix-name name)
 
 (define* (juliahub-fetch name #:key (version #f))
   "Return a <juliahub-package> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
-         json->juliahub-package))
+  (let ((url (if version
+                 (string-append "https://docs.juliahub.com/" name "/"
+                                (car (juliahub-slug+version name)) "/"
+                                version "/pkg.json")
+                 (string-append (juliahub-url name) "pkg.json"))))
+    (and=> (json-fetch url) json->juliahub-package)))
 
 (define (make-julia-sexp name source home-page synopsis description
                          direct-dependencies test-dependencies-names licenses)
@@ -242,9 +246,9 @@ (define* (juliahub->guix-package package-name
          (subdir (assoc-ref package-toml 'subdir))
          (tag (latest-git-tag (assoc-ref package-toml 'repo)))
          (package (if version
-                      (juliahub-fetch package-name version)
+                      (juliahub-fetch package-name #:version version)
                       (if tag
-                          (juliahub-fetch package-name tag)
+                          (juliahub-fetch package-name #:version tag)
                           (juliahub-fetch package-name)))))
     (if package
         (let-values (((source directory)
-- 
2.39.2

[bug#62202] [PATCH 14/21] import: juliahub: Filter out stdlibs from test-dependencies.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 6ce0487dba..1c7b029296 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -174,7 +174,8 @@ (define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
          (ini-list (stream->list (port->stream port read))))
     (close-port port)
-    (ini-list->test-dependencies ini-list)))
+    (filter (lambda (x) (not (member x %julia-stdlibs)))
+            (ini-list->test-dependencies ini-list))))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
-- 
2.39.2

[bug#62202] [PATCH 15/21] import: juliahub: More robust toml regex parser.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 29 ++++++++---------------------
 1 file changed, 8 insertions(+), 21 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 1c7b029296..3985d8d0be 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,28 +154,15 @@ (define (json->juliahub-dependencies vector)
                #f)))
        (vector->list vector))))
 
-(define (ini-list->test-dependencies lst)
-  (match lst
-    (('test '= ooo ...)
-     `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
-    ((value ooo ...)
-     (ini-list->test-dependencies ooo))
-    ('()
-      '())))
-
-(define (test-list->test-dependencies lst)
-  (match lst
-    ((('unquote value) ooo ...)
-     `(,value ,@(test-list->test-dependencies ooo)))
-    ('()
-      '())))
-
 (define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
-         (ini-list (stream->list (port->stream port read))))
+         (project.toml (get-string-all port))
+         (regex "\ntest = \\[.*\\]")
+         (deps (match:substring (string-match regex project.toml)))
+         (pure (string-delete (list->char-set (list #\" #\ )) deps)))
     (close-port port)
     (filter (lambda (x) (not (member x %julia-stdlibs)))
-            (ini-list->test-dependencies ini-list))))
+            (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -243,9 +230,9 @@ (define* (juliahub->guix-package package-name
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
-         (subdir (assoc-ref package-toml 'subdir))
-         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+  (let* ((package.toml (ini-fetch (general-url package-name "Package.toml")))
+         (subdir (assoc-ref package.toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package.toml 'repo)))
          (package (if version
                       (juliahub-fetch package-name #:version version)
                       (if tag
-- 
2.39.2

[bug#62202] [PATCH 16/21] import: juliahub: Beautify description.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 60 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 56 insertions(+), 4 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 3985d8d0be..338f042441 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -32,8 +32,10 @@ (define-module (guix import juliahub)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
-  #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (json)
+  #:use-module (htmlprag)
+  #:use-module (sxml transform)
 
   #:export (juliahub->guix-package
             %juliahub-updater
@@ -164,6 +166,53 @@ (define (parse-test-dependencies directory)
     (filter (lambda (x) (not (member x %julia-stdlibs)))
             (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
 
+(define %juliahub-beautify-description-rules
+  `((h1 *preorder*   . ,(lambda args #f))
+    (h2 *preorder*   . ,(lambda args #f))
+    (h3 *preorder*   . ,(lambda args #f))
+    (h4 *preorder*   . ,(lambda args #f))
+    (hr *preorder*   . ,(lambda args #f))
+    (span *preorder* . ,(lambda args #f))
+    (img *preorder*  . ,(lambda args #f))
+    (pre *preorder*  . ,(lambda args #f))
+    (div *preorder*  . ,(lambda args #f))
+    (table *preorder* . ,(lambda args #f))
+    (imgalt *preorder* . ,(lambda args #f))
+    (@ *preorder* . ,(lambda args #f))
+    (*TOP*        . ,(lambda args (cdr args)))
+    (p            . ,(lambda args (cdr args)))
+    (em           . ,(lambda args (cdr args)))
+    (strong       . ,(lambda args (cdr args)))
+    (a            . ,(lambda args
+                       (match args
+                         ((tag link ref)
+                          (if ref ref #f))
+                         (_ #f))))
+    (ul           . ,(lambda args
+                       `("@itemize" ,@(cdr args) "\n@end itemize")))
+    (ol           . ,(lambda args
+                       `("@enumerate" ,@(cdr args) "@end enumerate")))
+    (blockquote   . ,(lambda args
+                       `("@quotation" ,@(cdr args) "@end quotation")))
+    (li           . ,(lambda args
+                       `("\n@item" ,@(cdr args))))
+    (code         . ,(lambda args
+                       (string-append
+                        "@code{"
+                        (string-join (cdr args) " ")
+                        "}")))
+    (*text*       . ,(lambda (tag x) x))
+    (*default*    . ,(lambda (tag . body)
+                       (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+  (string-join
+   (filter (lambda (x) (if (equal? x " ") #f x))
+           (flatten
+            (pre-post-order (html->sxml description)
+                            %juliahub-beautify-description-rules)))
+   " "))
+
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   json->juliahub-package
@@ -211,8 +260,9 @@ (define (make-julia-sexp name source home-page synopsis description
      ,@(if (null? direct-dependencies)
            '()
            `((propagated-inputs
-              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           direct-dependencies)))))
+              (list
+               ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                      direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
@@ -264,7 +314,9 @@ (define* (juliahub->guix-package package-name
                      source
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
-                     (beautify-description (juliahub-package-readme package))
+                     ((compose beautify-description
+                               juliahub-beautify-description)
+                      (juliahub-package-readme package))
                      direct-dependencies
                      test-dependencies-names
                      licenses)
-- 
2.39.2

[bug#62202] [PATCH 17/21] import: juliahub: Fix license management.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 338f042441..e4b26bea34 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -248,10 +248,10 @@ (define* (juliahub-fetch name #:key (version #f))
     (and=> (json-fetch url) json->juliahub-package)))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         direct-dependencies test-dependencies-names licenses)
+                         direct-dependencies test-dependencies-names license)
   "Return the `package' s-expression for a Julia package with the given NAME,
 VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
-TEST-DEPENDENCIES-NAMES and LICENSES."
+TEST-DEPENDENCIES-NAMES and LICENSE."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
@@ -270,10 +270,8 @@ (define (make-julia-sexp name source home-page synopsis description
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
-     (license ,(match licenses
-                 (() #f)
-                 ((license) (license->symbol license))
-                 (_ `(list ,@(map license->symbol licenses)))))))
+     (license
+      ,(if license (spdx-string->license license) #f))))
 
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
@@ -301,9 +299,6 @@ (define* (juliahub->guix-package package-name
                  (dependencies-names
                   (map juliahub-dependency-name
                        direct-dependencies))
-                 (licenses
-                  (map spdx-string->license
-                       (list (juliahub-package-license package))))
                  (test-dependencies-names
                   (if subdir
                       (parse-test-dependencies
@@ -319,7 +314,7 @@ (define* (juliahub->guix-package package-name
                       (juliahub-package-readme package))
                      direct-dependencies
                      test-dependencies-names
-                     licenses)
+                     (juliahub-package-license package))
                     (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
-- 
2.39.2

[bug#62202] [PATCH 18/21] import: juliahub: Fix version management.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index e4b26bea34..94d4ae8233 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -247,10 +247,10 @@ (define* (juliahub-fetch name #:key (version #f))
                  (string-append (juliahub-url name) "pkg.json"))))
     (and=> (json-fetch url) json->juliahub-package)))
 
-(define (make-julia-sexp name source home-page synopsis description
+(define (make-julia-sexp name version source home-page synopsis description
                          direct-dependencies test-dependencies-names license)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSE."
   `(package
      (name ,(julia-name->guix-name name))
@@ -306,6 +306,7 @@ (define* (juliahub->guix-package package-name
                       (parse-test-dependencies directory))))
             (values (make-julia-sexp
                      package-name
+                     (juliahub-package-version package)
                      source
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
-- 
2.39.2

[bug#62202] [PATCH 19/21] import: juliahub: Fix undefined homepages.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 94d4ae8233..06574db724 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -303,12 +303,16 @@ (define* (juliahub->guix-package package-name
                   (if subdir
                       (parse-test-dependencies
                        (string-append subdir "/" directory))
-                      (parse-test-dependencies directory))))
+                      (parse-test-dependencies directory)))
+                 (homepage (juliahub-package-homepage package)))
             (values (make-julia-sexp
                      package-name
                      (juliahub-package-version package)
                      source
-                     (juliahub-package-homepage package)
+                     (match homepage
+                       ("" (juliahub-package-url package))
+                       ((? string?) homepage)
+                       (_ (juliahub-package-url package)))
                      (juliahub-package-description package)
                      ((compose beautify-description
                                juliahub-beautify-description)
-- 
2.39.2

[bug#62202] [PATCH 20/21] import: utils: Rule out texinfo common syntax from @ escape.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 5 +----
 guix/import/utils.scm    | 8 +++++++-
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 06574db724..0d3f89ad61 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -197,10 +197,7 @@ (define %juliahub-beautify-description-rules
     (li           . ,(lambda args
                        `("\n@item" ,@(cdr args))))
     (code         . ,(lambda args
-                       (string-append
-                        "@code{"
-                        (string-join (cdr args) " ")
-                        "}")))
+                       `("@code{" ,@(cdr args) "}")))
     (*text*       . ,(lambda (tag x) x))
     (*default*    . ,(lambda (tag . body)
                        (cons tag body)))))
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 171dca54e8..5ed1dfd815 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -376,7 +376,13 @@ (define* (beautify-description description #:optional (length 80))
                    (cut string-trim-both <> #\')
                    ;; Escape single @ to prevent it from being understood as
                    ;; invalid Texinfo syntax.
-                   (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)))))
+                   (lambda (word)
+                     (if
+                      (member word '("@itemize" "@item" "@end" "@quotation"
+                                     "@enumerate" "@code" "@code{"))
+                      word
+                      ((cut regexp-substitute/global
+                            #f "@" <> 'pre "@@" 'post) word)))))))
          (words
           (string-tokenize (string-trim-both description)
                            (char-set-complement
-- 
2.39.2

[bug#62202] [PATCH 21/21] import: juliahub: output package names as symbols.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 0d3f89ad61..3e5735b950 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -258,12 +258,15 @@ (define (make-julia-sexp name version source home-page synopsis description
            '()
            `((propagated-inputs
               (list
-               ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+               ,@(map (compose string->symbol
+                               julia-name->guix-name
+                               juliahub-dependency-name)
                       direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
-              (list ,@(map julia-name->guix-name test-dependencies-names)))))
+              (list ,@(map (compose string->symbol julia-name->guix-name)
+                           test-dependencies-names)))))
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
-- 
2.39.2

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi Nicolas,

Sorry for the delay. (I was in holidays \o/ :-))

I am not able to apply the series.  Could you rebase it or provide here
by email the commit against which the series apply.  Thanks in advance.


On mer., 15 mars 2023 at 13:47, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

>  Took me quite more time than I would've liked, but I have a usable
>  juliahub scheme import script!

Really cool!  Thank you!


>  It seems there's still one edge case that isn't covered and revolves
>  around when Julia packagers don't properly tag their git repos (I've
>  only seen the case with SnoopPrecompile). There's the possibility to
>  rely on tree commit hashes from the General repository (since this is a
>  valid way to identify/store a git repo), but that needs some major
>  changes in the way latest-repository-commit works. Otherwise, it needs
>  to be done by hand. It might also not work for subpackages in
>  directories that are up-to-date on juliahub but not yet on github, I
>  haven't met this case yet.
>
>  I'm sending a patch series in the coming minutes.

Well, I have not read all series. :-)

Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Ludovic Courtès]

Hi!

Nicolas Graves <ngraves@ngraves.fr> skribis:

>  Took me quite more time than I would've liked, but I have a usable
>  juliahub scheme import script!
>
>  It seems there's still one edge case that isn't covered and revolves
>  around when Julia packagers don't properly tag their git repos (I've
>  only seen the case with SnoopPrecompile). There's the possibility to
>  rely on tree commit hashes from the General repository (since this is a
>  valid way to identify/store a git repo), but that needs some major
>  changes in the way latest-repository-commit works. Otherwise, it needs
>  to be done by hand. It might also not work for subpackages in
>  directories that are up-to-date on juliahub but not yet on github, I
>  haven't met this case yet.
>
>  I'm sending a patch series in the coming minutes.
>
> It's detailed since I haven't swauased all commits, for readability, but
> I can squash it further if necessary.

I’ll let Simon comment on the actual code since I’m not a Julia person.
:-)

Some more general comments:

  • Please make sure to document it in ‘doc/guix.texi’ under “Invoking
    guix import”, following the same template as the others there.

  • Please write ‘tests/juliahub.scm’.  I recommend the same strategy as
    ‘tests/cpan.scm’, which is to mock the upstream HTTP server.

  • Prefer (srfi srfi-41) over (ice-9 streams) (see rationale at
    <https://srfi.schemers.org/srfi-41/srfi-41.html>).

  • Prefer (srfi srfi-71) over (srfi srfi-11) for multiple-value
    bindings.

Thanks,
Ludo’.

[bug#62202] [PATCH 0/21] Juliahub import script.

[Ludovic Courtès]

Hello Nicolas & Simon!

What should we do about this importer?  Looks like useful code to me!

Thanks,
Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

> Hi!
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>>  Took me quite more time than I would've liked, but I have a usable
>>  juliahub scheme import script!
>>
>>  It seems there's still one edge case that isn't covered and revolves
>>  around when Julia packagers don't properly tag their git repos (I've
>>  only seen the case with SnoopPrecompile). There's the possibility to
>>  rely on tree commit hashes from the General repository (since this is a
>>  valid way to identify/store a git repo), but that needs some major
>>  changes in the way latest-repository-commit works. Otherwise, it needs
>>  to be done by hand. It might also not work for subpackages in
>>  directories that are up-to-date on juliahub but not yet on github, I
>>  haven't met this case yet.
>>
>>  I'm sending a patch series in the coming minutes.
>>
>> It's detailed since I haven't swauased all commits, for readability, but
>> I can squash it further if necessary.
>
> I’ll let Simon comment on the actual code since I’m not a Julia person.
> :-)
>
> Some more general comments:
>
>   • Please make sure to document it in ‘doc/guix.texi’ under “Invoking
>     guix import”, following the same template as the others there.
>
>   • Please write ‘tests/juliahub.scm’.  I recommend the same strategy as
>     ‘tests/cpan.scm’, which is to mock the upstream HTTP server.
>
>   • Prefer (srfi srfi-41) over (ice-9 streams) (see rationale at
>     <https://srfi.schemers.org/srfi-41/srfi-41.html>).
>
>   • Prefer (srfi srfi-71) over (srfi srfi-11) for multiple-value
>     bindings.
>
> Thanks,
> Ludo’.

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi Ludo,

Sorry for the delay, I was again in holidays. :-)

On Tue, 08 Aug 2023 at 17:24, Ludovic Courtès <ludo@gnu.org> wrote:

> What should we do about this importer?  Looks like useful code to me!

Yeah, it looks nice.  As I said [1], “I am not able to apply the
series.”  And since the series is 21 patches, I have been lazy.


1: https://yhetil.org/guix/87v8i7k6uv.fsf@gmail.com


Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

Hello Nicolas,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

> Yeah, it looks nice.  As I said [1], “I am not able to apply the
> series.”  And since the series is 21 patches, I have been lazy.

I just want to add that on 2023-09-13 Simon explained failed attempts to
apply this patch series in a message to guix-devel [1] (starting from
the string "I am not speaking on the vacuum of an hypothetical
problem").

I quote some output Simon got:

>> Using the usual “git am -3s” from Emacs Debbugs, then I get:
>> 
>> --8<---------------cut here---------------start------------->8---
>> Applying: import: juliahub: first script draft.
>> error: sha1 information is lacking or useless (guix/import/go.scm).
>> error: could not build fake ancestor
>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>> Patch failed at 0001 import: juliahub: first script draft.
>> When you have resolved this problem, run "git am --continue".
>> If you prefer to skip this patch, run "git am --skip" instead.
>> To restore the original branch and stop patching, run "git am --abort".
>> --8<---------------cut here---------------end--------------->8---

Please see the original message [1] for details.

Sorry I can't help resolving this problem right now and/or send an
updated patch series.

Nicolas could you please try to apply it yourself to see if you are
succeful and eventually send a V2 patch series?

Thanks! Gio'


[1] id:874jjzfhx0.fsf@gmail.com https://yhetil.org/guix/874jjzfhx0.fsf@gmail.com/

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

[bug#62202] [PATCH 0/21] Juliahub import script.

[Nicolas Graves via Guix-patches via]

On 2023-09-15 11:45, Giovanni Biscuolo wrote:

> Hello Nicolas,
>
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
> [...]
>
>> Yeah, it looks nice.  As I said [1], “I am not able to apply the
>> series.”  And since the series is 21 patches, I have been lazy.
>
> I just want to add that on 2023-09-13 Simon explained failed attempts to
> apply this patch series in a message to guix-devel [1] (starting from
> the string "I am not speaking on the vacuum of an hypothetical
> problem").
>
> I quote some output Simon got:
>
>>> Using the usual “git am -3s” from Emacs Debbugs, then I get:
>>> 
>>> --8<---------------cut here---------------start------------->8---
>>> Applying: import: juliahub: first script draft.
>>> error: sha1 information is lacking or useless (guix/import/go.scm).
>>> error: could not build fake ancestor
>>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>>> Patch failed at 0001 import: juliahub: first script draft.
>>> When you have resolved this problem, run "git am --continue".
>>> If you prefer to skip this patch, run "git am --skip" instead.
>>> To restore the original branch and stop patching, run "git am --abort".
>>> --8<---------------cut here---------------end--------------->8---
>
> Please see the original message [1] for details.
>
> Sorry I can't help resolving this problem right now and/or send an
> updated patch series.
>
> Nicolas could you please try to apply it yourself to see if you are
> succeful and eventually send a V2 patch series?

Hi Giovanni,

Sorry for the lack of work to get this merged. I must still have this
branch locally, I'll try to output a V2.

The issue I had with the use is that my factorization of some functions
broke the Go importer, so either I have to fix that or drop the
factorization and duplicate the code in the juliahub importer.

I'll try to work on that this weekend, thanks for asking.

Nicolas 
>
> Thanks! Gio'
>
>
> [1] id:874jjzfhx0.fsf@gmail.com https://yhetil.org/guix/874jjzfhx0.fsf@gmail.com/

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi Nicolas,

On Fri, 15 Sept 2023 at 15:32, Nicolas Graves <ngraves@ngraves.fr> wrote:

> Sorry for the lack of work to get this merged. I must still have this
> branch locally, I'll try to output a V2.

Oh, thank you.  I am really sorry for the burden.

> I'll try to work on that this weekend, thanks for asking.

Previously, I have asked too and you have probably missed it. :-)  And
I was in the mood to go via some boring manual work this week.  I am
very happy if it can be avoided and you are able to find some v2.

Have a nice week-end.

Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Nicolas Graves via Guix-patches via]

On 2023-09-15 16:01, Simon Tournier wrote:

> Hi Nicolas,
>
> On Fri, 15 Sept 2023 at 15:32, Nicolas Graves <ngraves@ngraves.fr> wrote:
>
>> Sorry for the lack of work to get this merged. I must still have this
>> branch locally, I'll try to output a V2.
>
> Oh, thank you.  I am really sorry for the burden.
>
>> I'll try to work on that this weekend, thanks for asking.
>
> Previously, I have asked too and you have probably missed it. :-)  And
> I was in the mood to go via some boring manual work this week.  I am
> very happy if it can be avoided and you are able to find some v2.
>
> Have a nice week-end.

Couldn't find time this wkend, but it's on my backlog, I'll try to do
that soon. 
>
> Cheers,
> simon

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v2 01/23] DRAFT guix: import: go: Add optional transform-version to vcs->origin.

[Simon Tournier]

---
 guix/import/go.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1ebe7..d1c6f68d9c9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -513,7 +513,8 @@ (define* (git-checkout-hash url reference algorithm)
                                           `(tag-or-commit . ,reference)))))
     (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
 
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+                      #:optional transform-version)
   "Generate the `origin' block of a package depending on what type of source
 control system is being used."
   (case vcs-type

base-commit: 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d
-- 
2.38.1

[bug#62202] [PATCH v2 02/23] DRAFT TODO guix: import: utils: Add git->origin+dir.

[Simon Tournier]

---
 guix/import/utils.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde73..189facfcf823 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -63,6 +63,8 @@ (define-module (guix import utils)
 
             url-fetch
             guix-hash-url
+            git->origin+dir
+            ;; git->origin
 
             package-names->package-inputs
             maybe-inputs
-- 
2.38.1

[bug#62202] [PATCH v2 03/23] DRAFT TODO: guix: import: utils: Fix corner cases beautify-descritption.

[Simon Tournier]

* guix/import/utils.scm (beautify-descritption): Fix.
---
 guix/import/utils.scm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 189facfcf823..01d9861f279c 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -339,6 +339,16 @@ (define* (beautify-description description #:optional (length 80))
                    ;; Escape single @ to prevent it from being understood as
                    ;; invalid Texinfo syntax.
                    (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+
+                   ;; Remove cut above
+                   ;; (lambda (word)
+                   ;;   (if
+                   ;;    (member word '("@itemize" "@item" "@end" "@quotation"
+                   ;;                   "@enumerate" "@code" "@code{"))
+                   ;;    word
+                   ;;    ((cut regexp-substitute/global
+                   ;;          #f "@" <> 'pre "@@" 'post) word)))))))
+
                    ;; Wrap camelCase or PascalCase words in @code{...}.
                    (lambda (word)
                      (let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
-- 
2.38.1

[bug#62202] [PATCH v2 04/23] DRAFT import: Add julia.

[Simon Tournier]

* guix/scripts/import.scm (importers): Add juliahub.
* guix/import/juliahub.scm: New file.
* guix/scripts/import/juliahub.scm: New file.
* Makefile.am: Add them.
---
 Makefile.am                      |   2 +
 guix/import/juliahub.scm         | 183 +++++++++++++++++++++++++++++++
 guix/scripts/import.scm          |   2 +-
 guix/scripts/import/juliahub.scm | 107 ++++++++++++++++++
 4 files changed, 293 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/juliahub.scm
 create mode 100644 guix/scripts/import/juliahub.scm

diff --git a/Makefile.am b/Makefile.am
index 922913355c45..70b06c1a793f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -289,6 +289,7 @@ MODULES =					\
   guix/import/hackage.scm			\
   guix/import/hexpm.scm				\
   guix/import/json.scm				\
+  guix/import/juliahub.scm			\
   guix/import/kde.scm				\
   guix/import/launchpad.scm   			\
   guix/import/minetest.scm   			\
@@ -344,6 +345,7 @@ MODULES =					\
   guix/scripts/import/hackage.scm		\
   guix/scripts/import/hexpm.scm			\
   guix/scripts/import/json.scm  		\
+  guix/scripts/import/juliahub.scm  		\
   guix/scripts/import/minetest.scm  		\
   guix/scripts/import/opam.scm			\
   guix/scripts/import/pypi.scm			\
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 000000000000..efe6abbb2481
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,183 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+  #:use-module (ice-9 textual-ports)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 string-fun)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-1)
+  #:use-module (guix http-client)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix base16)
+  #:use-module (guix base32)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module ((guix licenses) #:prefix license:)
+
+  #:export (juliahub->guix-package))
+
+(define (juliahub-uri name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (port (http-fetch url #:text? #t))
+         (_ (get-line port))
+         (meta (get-line port))
+         (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+         (redirect (match:substring (string-match regex meta))))
+    (close-port port)
+    (string-drop redirect 4)))
+
+(define (juliahub-url name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (uri (juliahub-uri name)))
+    (string-append url uri "/")))
+
+(define (juliahub-slug-version name)
+  (let* ((uri (juliahub-uri name))
+         (slug (string-take uri 5))
+         (latest-version (string-drop uri 6)))
+    `(,slug ,latest-version)))
+
+(define (json->juliahub-direct-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (juliahub-dependency-direct? dep)
+               dep
+               #f)))
+       (vector->list vector))))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+  json->juliahub-package
+  (homepage juliahub-package-homepage) ;string
+  (readme juliahub-package-readme) ;string
+  ;; (slug juliahub-package-slug) ;string
+  (version juliahub-package-version) ;string
+  (description juliahub-package-description) ;string
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+   ;; (lambda (vector)
+     ;; (map json->juliahub-dependency (vector->list vector))))
+  (url juliahub-package-url) ;string
+  (uuid juliahub-package-uuid) ;string
+  (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+  make-juliahub-dependency juliahub-dependency?
+  json->juliahub-dependency
+  (direct? juliahub-dependency-direct? "direct") ;boolean
+  (name juliahub-dependency-name) ;string
+  (uuid juliahub-dependency-uuid) ;string
+  (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+  ;; (slug juliahub-dependency-slug) ;string
+
+(define (julia-name->guix-name name)
+  (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+  "Return a <juliahub-package> record for package NAME, or #f on failure."
+  (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
+         json->juliahub-package))
+
+(define (make-julia-sexp name version uri hash home-page synopsis description
+                         dependencies licenses)
+  "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+  `(package
+     (name ,(julia-name->guix-name name))
+     (version ,version)
+     (source (origin
+               (method url-fetch)
+               (uri ,uri)
+               (sha256
+                (base32
+                 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+                 ;; ,(bytevector->nix-base32-string hash)
+                 ))))
+     (build-system julia-build-system)
+     ,@(if (null? dependencies)
+           '()
+           `((inputs
+              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                           dependencies)))))
+     (synopsis ,synopsis)
+     (description ,description)
+     (home-page ,home-page)
+     (license ,(match licenses
+                 (() #f)
+                 ((license) (license->symbol license))
+                 (_ `(list ,@(map license->symbol licenses)))))))
+
+(define* (juliahub->guix-package package-name
+                                 #:key version #:allow-other-keys)
+  "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+  (let ((package (if version
+                      (juliahub-fetch package-name version)
+                      (juliahub-fetch package-name))))
+    (if package
+        (let* ((dependencies-names
+                (map juliahub-dependency-name
+                     (juliahub-package-dependencies package)))
+               (licenses
+                (map spdx-string->license
+                     (list (juliahub-package-license package)))))
+          (values (make-julia-sexp
+                   package-name
+                   (juliahub-package-version package)
+                   (juliahub-package-url package)
+                   "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+                   (juliahub-package-homepage package)
+                   (juliahub-package-description package)
+                   (beautify-description (juliahub-package-readme package))
+                   (juliahub-package-dependencies package)
+                   licenses)
+                  dependencies-names))
+        (values #f '()))))
+
+(define* (import-release package #:key (version #f))
+  "Return an <upstream-source> for the latest release of PACKAGE."
+  (let* ((package-name (guix-package->juliahub-name package))
+         (package      (juliahub-fetch package-name))
+         (version  (or version (juliahub-version gem)))
+         (url      (rubyjuliahubs-uri gem-name version)))
+    (upstream-source
+     (package (package-name package))
+     (version version)
+     (urls (list url)))))
+
+(define %juliahub-updater
+  (upstream-updater
+   (name 'juliahub)
+   (description "Updater for Juliahub packages")
+   (pred juliahub-package?)
+   (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+  (recursive-import package-name
+                    #:repo '()
+                    #:repo->guix-package juliahub->guix-package
+                    #:guix-name ruby-package-name
+                    #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 4ddd8d46a121..74c26e347335 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
 
 (define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
                     "gem" "go" "cran" "crate" "texlive" "json" "opam"
-                    "minetest" "elm" "hexpm"))
+                    "minetest" "elm" "hexpm" "juliahub"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 000000000000..1317c67aa342
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import juliahub)
+  #:use-module (guix scripts import)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 receive)
+  #:export (guix-import-juliahub))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME.  Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (display (G_ "
+  -r, --recursive        generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import gem")))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
+         %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (parse-command-line args %options (list %default-options)
+                        #:build-options? #f))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                             (('argument . value)
+                              value)
+                             (_ #f))
+                           (reverse opts))))
+    (match args
+      ((spec)
+       (receive (package-name package-version)
+           (package-name->name+version spec)
+         (let ((code (if (assoc-ref opts 'recursive)
+                         (map (match-lambda
+                                ((and ('package ('name name) . rest) pkg)
+                                 `(define-public ,(string->symbol name)
+                                    ,pkg))
+                                (_ #f))
+                              (juliahub-recursive-import package-name package-version))
+                         (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+                           (if sexp sexp #f)))))
+           (match code
+             ((or #f '(#f))
+              (leave (G_ "failed to download meta-data for package '~a'~%")
+                     package-name))
+             (_ code)))))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.38.1

[bug#66088] [PATCH v2 05/23] DRAFT import: juliahub: Add support for native-inputs.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 105 ++++++++++++++++++++++++---------------
 1 file changed, 64 insertions(+), 41 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index efe6abbb2481..4544dee98016 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -20,13 +20,14 @@ (define-module (guix import juliahub)
   #:use-module (ice-9 textual-ports)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 match)
+  #:use-module (ice-9 streams)
   #:use-module (ice-9 string-fun)
-  #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-11)
   #:use-module (guix http-client)
   #:use-module (guix import utils)
   #:use-module (guix import json)
-  #:use-module (guix base16)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
@@ -66,19 +67,42 @@ (define (json->juliahub-direct-dependencies vector)
                #f)))
        (vector->list vector))))
 
+(define (ini-list->extra-dependencies lst)
+  (match lst
+    (('(extras) ooo ...)
+     (extra-list->extra-dependencies ooo))
+    (((tag) ooo ...)
+     (ini-list->extra-dependencies ooo))
+    ((attribute '= value ooo ...)
+     (ini-list->extra-dependencies ooo))
+    ('()
+     '())))
+
+(define (extra-list->extra-dependencies lst)
+  (match lst
+    ((attribute '= value ooo ...)
+     `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
+    (((tag) ooo ...)
+     '())
+    ('()
+     '())))
+
+(define (parse-extra-dependencies directory)
+  (let* ((port (open-input-file (string-append directory "/Project.toml")))
+         (ini-list (stream->list (port->stream port read))))
+    (close-port port)
+    (ini-list->extra-dependencies ini-list)))
+
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   json->juliahub-package
   (homepage juliahub-package-homepage) ;string
   (readme juliahub-package-readme) ;string
-  ;; (slug juliahub-package-slug) ;string
   (version juliahub-package-version) ;string
   (description juliahub-package-description) ;string
-  (dependencies
-   juliahub-package-dependencies "deps"
+  (direct-dependencies
+   juliahub-package-direct-dependencies "deps"
    json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
-   ;; (lambda (vector)
-     ;; (map json->juliahub-dependency (vector->list vector))))
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -90,7 +114,6 @@ (define-json-mapping <juliahub-dependency>
   (name juliahub-dependency-name) ;string
   (uuid juliahub-dependency-uuid) ;string
   (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
-  ;; (slug juliahub-dependency-slug) ;string
 
 (define (julia-name->guix-name name)
   (string-append "julia-" (snake-case name)))
@@ -100,27 +123,25 @@ (define* (juliahub-fetch name #:key (version #f))
   (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
          json->juliahub-package))
 
-(define (make-julia-sexp name version uri hash home-page synopsis description
-                         dependencies licenses)
+(define (make-julia-sexp name source home-page synopsis description
+                         direct-dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSES."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
-     (source (origin
-               (method url-fetch)
-               (uri ,uri)
-               (sha256
-                (base32
-                 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
-                 ;; ,(bytevector->nix-base32-string hash)
-                 ))))
+     (source ,source)
      (build-system julia-build-system)
-     ,@(if (null? dependencies)
+     ,@(if (null? direct-dependencies)
            '()
-           `((inputs
+           `((propagated-inputs
               (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           dependencies)))))
+                           direct-dependencies)))))
+     ,@(if (null? test-dependencies-names)
+           '()
+           `((native-inputs
+              (list ,@(map julia-name->guix-name test-dependencies-names)))))
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
@@ -135,26 +156,28 @@ (define* (juliahub->guix-package package-name
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
   (let ((package (if version
-                      (juliahub-fetch package-name version)
-                      (juliahub-fetch package-name))))
+                     (juliahub-fetch package-name version)
+                     (juliahub-fetch package-name))))
     (if package
-        (let* ((dependencies-names
-                (map juliahub-dependency-name
-                     (juliahub-package-dependencies package)))
-               (licenses
-                (map spdx-string->license
-                     (list (juliahub-package-license package)))))
-          (values (make-julia-sexp
-                   package-name
-                   (juliahub-package-version package)
-                   (juliahub-package-url package)
-                   "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
-                   (juliahub-package-homepage package)
-                   (juliahub-package-description package)
-                   (beautify-description (juliahub-package-readme package))
-                   (juliahub-package-dependencies package)
-                   licenses)
-                  dependencies-names))
+        (let-values (((source directory)
+                     (git->origin+dir url `(tag-or-commit . ,package-version))))
+          (let* ((dependencies-names
+                  (map juliahub-dependency-name
+                       (juliahub-package-direct-dependencies package)))
+                 (licenses
+                  (map spdx-string->license
+                       (list (juliahub-package-license package))))
+                 (test-dependencies-names (parse-extra-dependencies directory)))
+            (values (make-julia-sexp
+                     package-name
+                     source
+                     (juliahub-package-homepage package)
+                     (juliahub-package-description package)
+                     (beautify-description (juliahub-package-readme package))
+                     (juliahub-package-direct-dependencies package)
+                     test-dependencies-names
+                     licenses)
+                    (append dependencies-names test-dependencies))))
         (values #f '()))))
 
 (define* (import-release package #:key (version #f))
-- 
2.38.1

[bug#66077] [PATCH v2 06/23] DRAFT import: juliahub: Correct source parsing.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4544dee98016..4c3ceed10904 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -160,7 +160,11 @@ (define* (juliahub->guix-package package-name
                      (juliahub-fetch package-name))))
     (if package
         (let-values (((source directory)
-                     (git->origin+dir url `(tag-or-commit . ,package-version))))
+                      (git->origin+dir
+                       (juliahub-package-url package)
+                       `(tag-or-commit
+                         . ,(string-append
+                             "v" (juliahub-package-version package))))))
           (let* ((dependencies-names
                   (map juliahub-dependency-name
                        (juliahub-package-direct-dependencies package)))
-- 
2.38.1

[bug#62202] [PATCH v2 07/23] DRAFT import: juliahub: Add indirect dependencies.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4c3ceed10904..fb361a004435 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -67,6 +67,16 @@ (define (json->juliahub-direct-dependencies vector)
                #f)))
        (vector->list vector))))
 
+(define (json->juliahub-indirect-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (not (juliahub-dependency-direct? dep))
+               dep
+               #f)))
+       (vector->list vector))))
+
 (define (ini-list->extra-dependencies lst)
   (match lst
     (('(extras) ooo ...)
@@ -103,6 +113,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   (direct-dependencies
    juliahub-package-direct-dependencies "deps"
    json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+  (indirect-dependencies
+   juliahub-package-indirect-dependencies "deps"
+   json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -181,7 +194,7 @@ (define* (juliahub->guix-package package-name
                      (juliahub-package-direct-dependencies package)
                      test-dependencies-names
                      licenses)
-                    (append dependencies-names test-dependencies))))
+                    (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
 (define* (import-release package #:key (version #f))
-- 
2.38.1

[bug#66090] [PATCH v2 08/23] DRAFT import: juliahub: Add updater and recursive-importer.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fb361a004435..c38c830caaa0 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -34,7 +34,9 @@ (define-module (guix import juliahub)
   #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
 
-  #:export (juliahub->guix-package))
+  #:export (juliahub->guix-package
+            %juliahub-updater
+            juliahub-recursive-import))
 
 (define (juliahub-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
@@ -197,16 +199,26 @@ (define* (juliahub->guix-package package-name
                     (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
+(define (guix-package->juliahub-name package)
+  (let* ((url (juliahub-package-url package))
+         (git-name (car (last-pair (string-split url #\/))))
+         (ungitted-name (if (string-suffix? ".git" git-name)
+                            (string-drop-right git-name 4)
+                            git-name))
+         (package-name (if (string-suffix? ".jl" ungitted-name)
+                            (string-drop-right ungitted-name 4)
+                            ungitted-name)))
+    package-name))
+
 (define* (import-release package #:key (version #f))
   "Return an <upstream-source> for the latest release of PACKAGE."
   (let* ((package-name (guix-package->juliahub-name package))
          (package      (juliahub-fetch package-name))
-         (version  (or version (juliahub-version gem)))
-         (url      (rubyjuliahubs-uri gem-name version)))
+         (version  (or version (juliahub-package-version package))))
     (upstream-source
      (package (package-name package))
      (version version)
-     (urls (list url)))))
+     (urls (list (juliahub-package-url package))))))
 
 (define %juliahub-updater
   (upstream-updater
@@ -219,5 +231,5 @@ (define* (juliahub-recursive-import package-name #:optional version)
   (recursive-import package-name
                     #:repo '()
                     #:repo->guix-package juliahub->guix-package
-                    #:guix-name ruby-package-name
+                    #:guix-name julia-name->guix-name
                     #:version version))
-- 
2.38.1

[bug#62202] [PATCH v2 09/23] DRAFT import: juliahub: Filter out julia stdlibs.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 47 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caaa0..af08f3d698d0 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
             %juliahub-updater
             juliahub-recursive-import))
 
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+  (list "julia"
+        "ArgTools"
+        "Artifacts"
+        "Base64"
+        "CRC32c"
+        "FileWatching"
+        "Libdl"
+        "Logging"
+        "Mmap"
+        "NetworkOptions"
+        "SHA"
+        "Serialization"
+        "Sockets"
+        "Unicode"
+        "DelimitedFiles"
+        "LinearAlgebra"
+        "Markdown"
+        "Printf"
+        "Random"
+        "Tar"
+        "Dates"
+        "Distributed"
+        "Future"
+        "InteractiveUtils"
+        "LibGit2"
+        "Profile"
+        "SparseArrays"
+        "UUIDs"
+        "REPL"
+        "SharedArrays"
+        "Statistics"
+        "SuiteSparse"
+        "TOML"
+        "Test"
+        "LibCURL"
+        "Downloads"
+        "Pkg"
+        "LazyArtifacts"))
+
 (define (juliahub-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
          (port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (juliahub-dependency-direct? dep)
+           (if (and (juliahub-dependency-direct? dep)
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (not (juliahub-dependency-direct? dep))
+           (if (and (not (juliahub-dependency-direct? dep))
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
-- 
2.38.1

[bug#66087] [PATCH v2 09/23] DRAFT import: juliahub: Filter out julia stdlibs.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 47 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caaa0..af08f3d698d0 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
             %juliahub-updater
             juliahub-recursive-import))
 
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+  (list "julia"
+        "ArgTools"
+        "Artifacts"
+        "Base64"
+        "CRC32c"
+        "FileWatching"
+        "Libdl"
+        "Logging"
+        "Mmap"
+        "NetworkOptions"
+        "SHA"
+        "Serialization"
+        "Sockets"
+        "Unicode"
+        "DelimitedFiles"
+        "LinearAlgebra"
+        "Markdown"
+        "Printf"
+        "Random"
+        "Tar"
+        "Dates"
+        "Distributed"
+        "Future"
+        "InteractiveUtils"
+        "LibGit2"
+        "Profile"
+        "SparseArrays"
+        "UUIDs"
+        "REPL"
+        "SharedArrays"
+        "Statistics"
+        "SuiteSparse"
+        "TOML"
+        "Test"
+        "LibCURL"
+        "Downloads"
+        "Pkg"
+        "LazyArtifacts"))
+
 (define (juliahub-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
          (port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (juliahub-dependency-direct? dep)
+           (if (and (juliahub-dependency-direct? dep)
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (not (juliahub-dependency-direct? dep))
+           (if (and (not (juliahub-dependency-direct? dep))
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
                dep
                #f)))
        (vector->list vector))))
-- 
2.38.1

[bug#66076] [PATCH v2 10/23] DRAFT import: juliahub: Simplify juliahub dependency management.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 42 ++++++++++++++--------------------------
 1 file changed, 14 insertions(+), 28 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index af08f3d698d0..b1eeb736a824 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -100,26 +100,13 @@ (define (juliahub-slug-version name)
          (latest-version (string-drop uri 6)))
     `(,slug ,latest-version)))
 
-(define (json->juliahub-direct-dependencies vector)
+(define (json->juliahub-dependencies vector)
   (if (vector? vector)
       (filter-map
        (lambda (el)
          (let ((dep (json->juliahub-dependency el)))
-           (if (and (juliahub-dependency-direct? dep)
-                    (not (member (juliahub-dependency-name dep)
-                                 %julia-stdlibs)))
-               dep
-               #f)))
-       (vector->list vector))))
-
-(define (json->juliahub-indirect-dependencies vector)
-  (if (vector? vector)
-      (filter-map
-       (lambda (el)
-         (let ((dep (json->juliahub-dependency el)))
-           (if (and (not (juliahub-dependency-direct? dep))
-                    (not (member (juliahub-dependency-name dep)
-                                 %julia-stdlibs)))
+           (if (not (member (juliahub-dependency-name dep)
+                            %julia-stdlibs))
                dep
                #f)))
        (vector->list vector))))
@@ -157,12 +144,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   (readme juliahub-package-readme) ;string
   (version juliahub-package-version) ;string
   (description juliahub-package-description) ;string
-  (direct-dependencies
-   juliahub-package-direct-dependencies "deps"
-   json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
-  (indirect-dependencies
-   juliahub-package-indirect-dependencies "deps"
-   json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-dependencies) ;list of <juliahub-dependency>
   (url juliahub-package-url) ;string
   (uuid juliahub-package-uuid) ;string
   (license juliahub-package-license)) ;string
@@ -184,7 +168,7 @@ (define* (juliahub-fetch name #:key (version #f))
          json->juliahub-package))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         direct-dependencies test-dependencies-names licenses)
+                         dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
 VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSES."
@@ -193,11 +177,13 @@ (define (make-julia-sexp name source home-page synopsis description
      (version ,version)
      (source ,source)
      (build-system julia-build-system)
-     ,@(if (null? direct-dependencies)
-           '()
-           `((propagated-inputs
-              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           direct-dependencies)))))
+     ,@(let ((direct-dependencies
+              (filter julia-dependency-direct? dependencies)))
+         (if (null? direct-dependencies)
+             '()
+             `((propagated-inputs
+                (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                             direct-dependencies))))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
-- 
2.38.1

[bug#66092] [PATCH v2 11/23] DRAFT import: juliahub: Improve dependency management.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b1eeb736a824..fc25ba1d4220 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -168,22 +168,20 @@ (define* (juliahub-fetch name #:key (version #f))
          json->juliahub-package))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         dependencies test-dependencies-names licenses)
+                         direct-dependencies test-dependencies-names licenses)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSES."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
      (source ,source)
      (build-system julia-build-system)
-     ,@(let ((direct-dependencies
-              (filter julia-dependency-direct? dependencies)))
-         (if (null? direct-dependencies)
-             '()
-             `((propagated-inputs
-                (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                             direct-dependencies))))))
+     ,@(if (null? direct-dependencies)
+           '()
+           `((propagated-inputs
+              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                           direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
@@ -211,9 +209,12 @@ (define* (juliahub->guix-package package-name
                        `(tag-or-commit
                          . ,(string-append
                              "v" (juliahub-package-version package))))))
-          (let* ((dependencies-names
+          (let* ((direct-dependencies
+                  (filter juliahub-dependency-direct?
+                          (juliahub-package-dependencies package)))
+                 (dependencies-names
                   (map juliahub-dependency-name
-                       (juliahub-package-direct-dependencies package)))
+                       direct-dependencies))
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
@@ -224,7 +225,7 @@ (define* (juliahub->guix-package package-name
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
                      (beautify-description (juliahub-package-readme package))
-                     (juliahub-package-direct-dependencies package)
+                     direct-dependencies
                      test-dependencies-names
                      licenses)
                     (append dependencies-names test-dependencies-names))))
-- 
2.38.1

[bug#62202] [PATCH v2 12/23] DRAFT import: juliahub: Add functions to parse the git repo for a git tag.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 56 insertions(+), 6 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fc25ba1d4220..5327e923251d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -26,6 +26,7 @@ (define-module (guix import juliahub)
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-11)
   #:use-module (guix http-client)
+  #:use-module (guix git)
   #:use-module (guix import utils)
   #:use-module (guix import json)
   #:use-module (guix base32)
@@ -38,6 +39,48 @@ (define-module (guix import juliahub)
             %juliahub-updater
             juliahub-recursive-import))
 
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-version, and
+;; import this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag, and where looking for the commit is tedious and artificial. In these
+;; cases, we introduce the tree-commit which is available in the Versions.toml
+;; file in the General repository. This is equivalent to a commit, since we have
+;; a unique hash of the listing of files and directories, thus it can be used to
+;; identify the state of a repository.
+
+(define %general-base-url
+  "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+  (let ((folder (string-capitalize (string-take package-name 1))))
+    (string-append
+     %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-list->alist lst)
+  (match lst
+    ((attribute '= value ooo ...)
+     `((,attribute . ,value) ,@(ini-list->alist ooo)))
+    ('()
+     '())))
+
+(define (ini-fetch url)
+  (let* ((port (http-fetch url #:text? #t))
+         (ini-list (stream->list (port->stream port read))))
+    (close-port port)
+    (ini-list->alist ini-list)))
+
+(define (latest-git-tag repo)
+  (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+         (last-git-tag (last (string-split last-ref #\/))))
+    (string-drop last-git-tag 1)))
+
 ;; To update, see file sysimg.jl
 (define %julia-stdlibs
   (list "julia"
@@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description
                  ((license) (license->symbol license))
                  (_ `(list ,@(map license->symbol licenses)))))))
 
+;; TODO handle subdir case properly.
+
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let ((package (if version
-                     (juliahub-fetch package-name version)
-                     (juliahub-fetch package-name))))
+  (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+         (subdir (assoc-ref package-toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+         (package (if version
+                      (juliahub-fetch package-name version)
+                      (if tag
+                          (juliahub-fetch package-name tag)
+                          (juliahub-fetch package-name)))))
     (if package
         (let-values (((source directory)
                       (git->origin+dir
@@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name
 
 (define (guix-package->juliahub-name package)
   (let* ((url (juliahub-package-url package))
-         (git-name (car (last-pair (string-split url #\/))))
+         (git-name (last (string-split url #\/)))
          (ungitted-name (if (string-suffix? ".git" git-name)
                             (string-drop-right git-name 4)
                             git-name))
          (package-name (if (string-suffix? ".jl" ungitted-name)
-                            (string-drop-right ungitted-name 4)
-                            ungitted-name)))
+                           (string-drop-right ungitted-name 4)
+                           ungitted-name)))
     package-name))
 
 (define* (import-release package #:key (version #f))
-- 
2.38.1

[bug#66089] [PATCH v2 13/23] DRAFT import: juliahub: Improve test dependencies parsing.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 5327e923251d..2ea461b72aba 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,31 +154,27 @@ (define (json->juliahub-dependencies vector)
                #f)))
        (vector->list vector))))
 
-(define (ini-list->extra-dependencies lst)
+(define (ini-list->test-dependencies lst)
   (match lst
-    (('(extras) ooo ...)
-     (extra-list->extra-dependencies ooo))
-    (((tag) ooo ...)
-     (ini-list->extra-dependencies ooo))
-    ((attribute '= value ooo ...)
-     (ini-list->extra-dependencies ooo))
+    (('test '= ooo ...)
+     `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
+    ((value ooo ...)
+     (ini-list->test-dependencies ooo))
     ('()
-     '())))
+      '())))
 
-(define (extra-list->extra-dependencies lst)
+(define (test-list->test-dependencies lst)
   (match lst
-    ((attribute '= value ooo ...)
-     `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
-    (((tag) ooo ...)
-     '())
+    ((('unquote value) ooo ...)
+     `(,value ,@(test-list->test-dependencies ooo)))
     ('()
-     '())))
+      '())))
 
-(define (parse-extra-dependencies directory)
+(define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
          (ini-list (stream->list (port->stream port read))))
     (close-port port)
-    (ini-list->extra-dependencies ini-list)))
+    (ini-list->test-dependencies ini-list)))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -268,7 +264,7 @@ (define* (juliahub->guix-package package-name
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
-                 (test-dependencies-names (parse-extra-dependencies directory)))
+                 (test-dependencies-names (parse-test-dependencies directory)))
             (values (make-julia-sexp
                      package-name
                      source
-- 
2.38.1

[bug#62202] [PATCH v2 14/23] DRAFT import: juliahub: Handle the case where we have a subdirectory.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 2ea461b72aba..b646f9329562 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -233,14 +233,12 @@ (define (make-julia-sexp name source home-page synopsis description
                  ((license) (license->symbol license))
                  (_ `(list ,@(map license->symbol licenses)))))))
 
-;; TODO handle subdir case properly.
-
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
          (subdir (assoc-ref package-toml 'subdir))
          (tag (latest-git-tag (assoc-ref package-toml 'repo)))
          (package (if version
@@ -264,7 +262,11 @@ (define* (juliahub->guix-package package-name
                  (licenses
                   (map spdx-string->license
                        (list (juliahub-package-license package))))
-                 (test-dependencies-names (parse-test-dependencies directory)))
+                 (test-dependencies-names
+                  (if subdir
+                      (parse-test-dependencies
+                       (string-append subdir "/" directory))
+                      (parse-test-dependencies directory))))
             (values (make-julia-sexp
                      package-name
                      source
-- 
2.38.1

[bug#62202] [PATCH v2 15/23] DRAFT import: juliahub: Add support for versions for juliahub-fetch.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b646f9329562..6ce0487dba38 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -122,7 +122,7 @@ (define %julia-stdlibs
         "Pkg"
         "LazyArtifacts"))
 
-(define (juliahub-uri name)
+(define (juliahub-redirect-uri name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
          (port (http-fetch url #:text? #t))
          (_ (get-line port))
@@ -134,11 +134,11 @@ (define (juliahub-uri name)
 
 (define (juliahub-url name)
   (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
-         (uri (juliahub-uri name)))
+         (uri (juliahub-redirect-uri name)))
     (string-append url uri "/")))
 
-(define (juliahub-slug-version name)
-  (let* ((uri (juliahub-uri name))
+(define (juliahub-slug+version name)
+  (let* ((uri (juliahub-redirect-uri name))
          (slug (string-take uri 5))
          (latest-version (string-drop uri 6)))
     `(,slug ,latest-version)))
@@ -203,8 +203,12 @@ (define (julia-name->guix-name name)
 
 (define* (juliahub-fetch name #:key (version #f))
   "Return a <juliahub-package> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
-         json->juliahub-package))
+  (let ((url (if version
+                 (string-append "https://docs.juliahub.com/" name "/"
+                                (car (juliahub-slug+version name)) "/"
+                                version "/pkg.json")
+                 (string-append (juliahub-url name) "pkg.json"))))
+    (and=> (json-fetch url) json->juliahub-package)))
 
 (define (make-julia-sexp name source home-page synopsis description
                          direct-dependencies test-dependencies-names licenses)
@@ -242,9 +246,9 @@ (define* (juliahub->guix-package package-name
          (subdir (assoc-ref package-toml 'subdir))
          (tag (latest-git-tag (assoc-ref package-toml 'repo)))
          (package (if version
-                      (juliahub-fetch package-name version)
+                      (juliahub-fetch package-name #:version version)
                       (if tag
-                          (juliahub-fetch package-name tag)
+                          (juliahub-fetch package-name #:version tag)
                           (juliahub-fetch package-name)))))
     (if package
         (let-values (((source directory)
-- 
2.38.1

[bug#62202] [PATCH v2 16/23] DRAFT import: juliahub: Filter out stdlibs from test-dependencies.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 6ce0487dba38..1c7b02929634 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -174,7 +174,8 @@ (define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
          (ini-list (stream->list (port->stream port read))))
     (close-port port)
-    (ini-list->test-dependencies ini-list)))
+    (filter (lambda (x) (not (member x %julia-stdlibs)))
+            (ini-list->test-dependencies ini-list))))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
-- 
2.38.1

[bug#66086] [PATCH v2 17/23] DRAFT import: juliahub: More robust toml regex parser.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 29 ++++++++---------------------
 1 file changed, 8 insertions(+), 21 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 1c7b02929634..3985d8d0be44 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,28 +154,15 @@ (define (json->juliahub-dependencies vector)
                #f)))
        (vector->list vector))))
 
-(define (ini-list->test-dependencies lst)
-  (match lst
-    (('test '= ooo ...)
-     `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
-    ((value ooo ...)
-     (ini-list->test-dependencies ooo))
-    ('()
-      '())))
-
-(define (test-list->test-dependencies lst)
-  (match lst
-    ((('unquote value) ooo ...)
-     `(,value ,@(test-list->test-dependencies ooo)))
-    ('()
-      '())))
-
 (define (parse-test-dependencies directory)
   (let* ((port (open-input-file (string-append directory "/Project.toml")))
-         (ini-list (stream->list (port->stream port read))))
+         (project.toml (get-string-all port))
+         (regex "\ntest = \\[.*\\]")
+         (deps (match:substring (string-match regex project.toml)))
+         (pure (string-delete (list->char-set (list #\" #\ )) deps)))
     (close-port port)
     (filter (lambda (x) (not (member x %julia-stdlibs)))
-            (ini-list->test-dependencies ini-list))))
+            (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
 
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -243,9 +230,9 @@ (define* (juliahub->guix-package package-name
   "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
 `package' s-expression corresponding to that package, or #f on failure.
 Optionally include a VERSION string to fetch a specific version juliahub."
-  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
-         (subdir (assoc-ref package-toml 'subdir))
-         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+  (let* ((package.toml (ini-fetch (general-url package-name "Package.toml")))
+         (subdir (assoc-ref package.toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package.toml 'repo)))
          (package (if version
                       (juliahub-fetch package-name #:version version)
                       (if tag
-- 
2.38.1

[bug#66079] [PATCH v2 18/23] DRAFT import: juliahub: Beautify description.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 60 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 56 insertions(+), 4 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 3985d8d0be44..338f0424414c 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -32,8 +32,10 @@ (define-module (guix import juliahub)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
-  #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (json)
+  #:use-module (htmlprag)
+  #:use-module (sxml transform)
 
   #:export (juliahub->guix-package
             %juliahub-updater
@@ -164,6 +166,53 @@ (define (parse-test-dependencies directory)
     (filter (lambda (x) (not (member x %julia-stdlibs)))
             (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
 
+(define %juliahub-beautify-description-rules
+  `((h1 *preorder*   . ,(lambda args #f))
+    (h2 *preorder*   . ,(lambda args #f))
+    (h3 *preorder*   . ,(lambda args #f))
+    (h4 *preorder*   . ,(lambda args #f))
+    (hr *preorder*   . ,(lambda args #f))
+    (span *preorder* . ,(lambda args #f))
+    (img *preorder*  . ,(lambda args #f))
+    (pre *preorder*  . ,(lambda args #f))
+    (div *preorder*  . ,(lambda args #f))
+    (table *preorder* . ,(lambda args #f))
+    (imgalt *preorder* . ,(lambda args #f))
+    (@ *preorder* . ,(lambda args #f))
+    (*TOP*        . ,(lambda args (cdr args)))
+    (p            . ,(lambda args (cdr args)))
+    (em           . ,(lambda args (cdr args)))
+    (strong       . ,(lambda args (cdr args)))
+    (a            . ,(lambda args
+                       (match args
+                         ((tag link ref)
+                          (if ref ref #f))
+                         (_ #f))))
+    (ul           . ,(lambda args
+                       `("@itemize" ,@(cdr args) "\n@end itemize")))
+    (ol           . ,(lambda args
+                       `("@enumerate" ,@(cdr args) "@end enumerate")))
+    (blockquote   . ,(lambda args
+                       `("@quotation" ,@(cdr args) "@end quotation")))
+    (li           . ,(lambda args
+                       `("\n@item" ,@(cdr args))))
+    (code         . ,(lambda args
+                       (string-append
+                        "@code{"
+                        (string-join (cdr args) " ")
+                        "}")))
+    (*text*       . ,(lambda (tag x) x))
+    (*default*    . ,(lambda (tag . body)
+                       (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+  (string-join
+   (filter (lambda (x) (if (equal? x " ") #f x))
+           (flatten
+            (pre-post-order (html->sxml description)
+                            %juliahub-beautify-description-rules)))
+   " "))
+
 ;; Julia package.
 (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
   json->juliahub-package
@@ -211,8 +260,9 @@ (define (make-julia-sexp name source home-page synopsis description
      ,@(if (null? direct-dependencies)
            '()
            `((propagated-inputs
-              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
-                           direct-dependencies)))))
+              (list
+               ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+                      direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
@@ -264,7 +314,9 @@ (define* (juliahub->guix-package package-name
                      source
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
-                     (beautify-description (juliahub-package-readme package))
+                     ((compose beautify-description
+                               juliahub-beautify-description)
+                      (juliahub-package-readme package))
                      direct-dependencies
                      test-dependencies-names
                      licenses)
-- 
2.38.1

[bug#66080] [PATCH v2 19/23] DRAFT import: juliahub: Fix license management.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 338f0424414c..e4b26bea340a 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -248,10 +248,10 @@ (define* (juliahub-fetch name #:key (version #f))
     (and=> (json-fetch url) json->juliahub-package)))
 
 (define (make-julia-sexp name source home-page synopsis description
-                         direct-dependencies test-dependencies-names licenses)
+                         direct-dependencies test-dependencies-names license)
   "Return the `package' s-expression for a Julia package with the given NAME,
 VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
-TEST-DEPENDENCIES-NAMES and LICENSES."
+TEST-DEPENDENCIES-NAMES and LICENSE."
   `(package
      (name ,(julia-name->guix-name name))
      (version ,version)
@@ -270,10 +270,8 @@ (define (make-julia-sexp name source home-page synopsis description
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
-     (license ,(match licenses
-                 (() #f)
-                 ((license) (license->symbol license))
-                 (_ `(list ,@(map license->symbol licenses)))))))
+     (license
+      ,(if license (spdx-string->license license) #f))))
 
 (define* (juliahub->guix-package package-name
                                  #:key version #:allow-other-keys)
@@ -301,9 +299,6 @@ (define* (juliahub->guix-package package-name
                  (dependencies-names
                   (map juliahub-dependency-name
                        direct-dependencies))
-                 (licenses
-                  (map spdx-string->license
-                       (list (juliahub-package-license package))))
                  (test-dependencies-names
                   (if subdir
                       (parse-test-dependencies
@@ -319,7 +314,7 @@ (define* (juliahub->guix-package package-name
                       (juliahub-package-readme package))
                      direct-dependencies
                      test-dependencies-names
-                     licenses)
+                     (juliahub-package-license package))
                     (append dependencies-names test-dependencies-names))))
         (values #f '()))))
 
-- 
2.38.1

[bug#62202] [PATCH v2 20/23] DRAFT import: juliahub: Fix version management.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index e4b26bea340a..94d4ae823334 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -247,10 +247,10 @@ (define* (juliahub-fetch name #:key (version #f))
                  (string-append (juliahub-url name) "pkg.json"))))
     (and=> (json-fetch url) json->juliahub-package)))
 
-(define (make-julia-sexp name source home-page synopsis description
+(define (make-julia-sexp name version source home-page synopsis description
                          direct-dependencies test-dependencies-names license)
   "Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
 TEST-DEPENDENCIES-NAMES and LICENSE."
   `(package
      (name ,(julia-name->guix-name name))
@@ -306,6 +306,7 @@ (define* (juliahub->guix-package package-name
                       (parse-test-dependencies directory))))
             (values (make-julia-sexp
                      package-name
+                     (juliahub-package-version package)
                      source
                      (juliahub-package-homepage package)
                      (juliahub-package-description package)
-- 
2.38.1

[bug#66085] [PATCH v2 21/23] DRAFT import: juliahub: Fix undefined homepages.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 94d4ae823334..06574db724fe 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -303,12 +303,16 @@ (define* (juliahub->guix-package package-name
                   (if subdir
                       (parse-test-dependencies
                        (string-append subdir "/" directory))
-                      (parse-test-dependencies directory))))
+                      (parse-test-dependencies directory)))
+                 (homepage (juliahub-package-homepage package)))
             (values (make-julia-sexp
                      package-name
                      (juliahub-package-version package)
                      source
-                     (juliahub-package-homepage package)
+                     (match homepage
+                       ("" (juliahub-package-url package))
+                       ((? string?) homepage)
+                       (_ (juliahub-package-url package)))
                      (juliahub-package-description package)
                      ((compose beautify-description
                                juliahub-beautify-description)
-- 
2.38.1

[bug#62202] [PATCH v2 22/23] DRAFT import: juliahub: output package names as symbols.

[Simon Tournier]

From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
 guix/import/juliahub.scm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 06574db724fe..be874b193b28 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -261,12 +261,15 @@ (define (make-julia-sexp name version source home-page synopsis description
            '()
            `((propagated-inputs
               (list
-               ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+               ,@(map (compose string->symbol
+                               julia-name->guix-name
+                               juliahub-dependency-name)
                       direct-dependencies)))))
      ,@(if (null? test-dependencies-names)
            '()
            `((native-inputs
-              (list ,@(map julia-name->guix-name test-dependencies-names)))))
+              (list ,@(map (compose string->symbol julia-name->guix-name)
+                           test-dependencies-names)))))
      (synopsis ,synopsis)
      (description ,description)
      (home-page ,home-page)
-- 
2.38.1

[bug#62202] [PATCH v2 23/23] DRAFT guix: import: julia: Fix beautify.

[Simon Tournier]

---
 guix/import/juliahub.scm | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index be874b193b28..3e5735b9503f 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -197,10 +197,7 @@ (define %juliahub-beautify-description-rules
     (li           . ,(lambda args
                        `("\n@item" ,@(cdr args))))
     (code         . ,(lambda args
-                       (string-append
-                        "@code{"
-                        (string-join (cdr args) " ")
-                        "}")))
+                       `("@code{" ,@(cdr args) "}")))
     (*text*       . ,(lambda (tag x) x))
     (*default*    . ,(lambda (tag . body)
                        (cons tag body)))))
-- 
2.38.1

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi Nicolas,

On Mon, 18 Sep 2023 at 11:31, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

> Couldn't find time this wkend, but it's on my backlog, I'll try to do
> that soon.

No worry.  I did it manually. :-)

I have sent v2 which is just your patches applied the top of
3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.

My plan is to give a look to the series this week.

Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

> No worry.  I did it manually. :-)

out of curiosity, what do you mean by "manually" please?  I mean: ho did
you got to apply the patches?

I'm asking because it could be helpful to others

> I have sent v2 which is just your patches applied the top of
> 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.
>
> My plan is to give a look to the series this week.

thank you!

Happy hacking! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi,

On Tue, 19 Sep 2023 at 08:23, Giovanni Biscuolo <g@xelera.eu> wrote:

> out of curiosity, what do you mean by "manually" please?  I mean: ho did
> you got to apply the patches?

I mean literally. ;-)

Open the patch, read it and then open the file, modify it, commit.
Repeat.

Else, I just pipe the message with “git am -3s”.

Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Nicolas Graves via Guix-patches via]

On 2023-09-18 20:06, Simon Tournier wrote:

> Hi Nicolas,
>
> On Mon, 18 Sep 2023 at 11:31, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:
>
>> Couldn't find time this wkend, but it's on my backlog, I'll try to do
>> that soon.
>
> No worry.  I did it manually. :-)
>
> I have sent v2 which is just your patches applied the top of
> 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.
>
> My plan is to give a look to the series this week.

Just note that in my experience, the juliahub importer worked quite
well (there was sometimes errors, mainly annoying with recursive imports
when the git repository didn't have any tag IIRC, not sure if I had
fixed that or not). The other annoying thing was that the factorization
broke the go importer IIRC.

But these fixes shouldn't be too hard compared to the initial effort to
make the importer. 

Thanks a lot and good luck, sorry, quite hard to find time recently. 

> Cheers,
> simon

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH 0/21] Juliahub import script.

[Simon Tournier]

Hi,

On Tue, 19 Sep 2023 at 08:51, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

>> My plan is to give a look to the series this week.

[...]

> Thanks a lot and good luck, sorry, quite hard to find time recently. 

I pause a bit.  If someone is motivated, feel free to pick it.  Else I
will resume when I will be able to dedicate some time for that.

Cheers,
simon

[bug#62202] [PATCH 0/21] Juliahub import script.

[Nicolas Graves via Guix-patches via]

On 2023-10-02 11:34, Simon Tournier wrote:

> Hi,
>
> On Tue, 19 Sep 2023 at 08:51, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:
>
>>> My plan is to give a look to the series this week.
>
> [...]
>
>> Thanks a lot and good luck, sorry, quite hard to find time recently. 
>
> I pause a bit.  If someone is motivated, feel free to pick it.  Else I
> will resume when I will be able to dedicate some time for that.

Will probably try to give it a go, either this evening or during winter
vacation. Could you point out if some changes other than properly
rebasing have been done at the point of the DRAFT V2 ? Thanks! 


>
> Cheers,
> simon
>
>
>

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v3 1/4] import: utils: Add function git->origin.

[Nicolas Graves via Guix-patches via]

* guix/import/utils.scm: (git->origin): Add function.

* guix/import/elpa.scm
(download-git-repository): Remove function download-git-repository.
(git-repository->origin): Remove function git-repository->origin.
(ref): Add function ref.
(melpa-recipe->origin): Use functions git->origin and ref.

* guix/import/go.scm
(git-checkout-hash): Remove function git-checkout-hash.
(transform-version): Add function transform-version.
(vcs->origin): Use functions git->origin and transform-version. Add
optional argument transform-version.

* tests/import/go.scm
(go-module->guix-package): Adapt test case to changes in guix/import/go.scm.

* guix/import/minetest.scm
(download-git-repository): Remove function download-git-repository.
(make-minetest-sexp): Use function git->origin.

* tests/minetest.scm
(make-package-sexp): Use function git->origin.
(example-package): Adapt test-case to git->origin.

* guix/import/composer.scm
(make-php-sexp): Use function git->origin.

Change-Id: Ied05a63bdd60fbafe26fbbb4e115ff6f0bb9db3c
---
 guix/import/composer.scm | 86 ++++++++++++++--------------------------
 guix/import/elpa.scm     | 43 ++++++--------------
 guix/import/go.scm       | 57 ++++++++------------------
 guix/import/minetest.scm | 28 ++-----------
 guix/import/utils.scm    | 39 ++++++++++++++++++
 tests/go.scm             | 29 ++++++++++----
 tests/minetest.scm       | 15 ++-----
 7 files changed, 127 insertions(+), 170 deletions(-)

diff --git a/guix/import/composer.scm b/guix/import/composer.scm
index 1ad608964b..dabc5423ed 100644
--- a/guix/import/composer.scm
+++ b/guix/import/composer.scm
@@ -19,22 +19,17 @@
 (define-module (guix import composer)
   #:use-module (ice-9 match)
   #:use-module (json)
-  #:use-module (guix hash)
-  #:use-module (guix base32)
-  #:use-module (guix build git)
-  #:use-module (guix build utils)
-  #:use-module (guix build-system)
   #:use-module (guix build-system composer)
+  #:use-module ((guix download) #:select (download-to-store))
   #:use-module (guix import json)
   #:use-module (guix import utils)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix memoization)
   #:use-module (guix packages)
-  #:use-module (guix serialization)
+  #:use-module (guix store)
   #:use-module (guix upstream)
   #:use-module (guix utils)
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:export (composer->guix-package
             %composer-updater
@@ -141,55 +136,34 @@ (define (make-php-sexp composer-package)
          (dependencies (map php-package-name
                             (composer-package-require composer-package)))
          (dev-dependencies (map php-package-name
-                                (composer-package-dev-require composer-package)))
-         (git? (equal? (composer-source-type source) "git")))
-    ((if git? call-with-temporary-directory call-with-temporary-output-file)
-     (lambda* (temp #:optional port)
-       (and (if git?
-               (begin
-                 (mkdir-p temp)
-                 (git-fetch (composer-source-url source)
-                            (composer-source-reference source)
-                            temp))
-               (url-fetch (composer-source-url source) temp))
-            `(package
-               (name ,(composer-package-name composer-package))
-               (version ,(composer-package-version composer-package))
-               (source
-                (origin
-                  ,@(if git?
-                        `((method git-fetch)
-                          (uri (git-reference
-                                (url ,(if (string-suffix?
-                                           ".git"
-                                           (composer-source-url source))
-                                          (string-drop-right
-                                           (composer-source-url source)
-                                           (string-length ".git"))
-                                          (composer-source-url source)))
-                                (commit ,(composer-source-reference source))))
-                          (file-name (git-file-name name version))
-                          (sha256
-                           (base32
-                            ,(bytevector->nix-base32-string
-                              (file-hash* temp)))))
-                        `((method url-fetch)
-                          (uri ,(composer-source-url source))
-                          (sha256 (base32 ,(guix-hash-url temp)))))))
-               (build-system composer-build-system)
-               ,@(if (null? dependencies)
-                     '()
-                     `((inputs
-                        (list ,@(map string->symbol dependencies)))))
-               ,@(if (null? dev-dependencies)
-                     '()
-                     `((native-inputs
-                        (list ,@(map string->symbol dev-dependencies)))))
-               (synopsis "")
-               (description ,(composer-package-description composer-package))
-               (home-page ,(composer-package-homepage composer-package))
-               (license ,(or (composer-package-license composer-package)
-                             'unknown-license!))))))))
+                                (composer-package-dev-require composer-package))))
+    `(package
+       (name ,(composer-package-name composer-package))
+       (version ,(composer-package-version composer-package))
+       (source
+        ,(if (string= (composer-source-type source) "git")
+             (git->origin (composer-source-url source)
+                          `(tag-or-commit . ,(composer-source-reference source)))
+             (let* ((source (composer-source-url source))
+                    (tarball (with-store store (download-to-store store source))))
+               `(origin
+                  (method url-fetch)
+                  (uri ,source)
+                  (sha256 (base32 ,(guix-hash-url tarball)))))))
+       (build-system composer-build-system)
+       ,@(if (null? dependencies)
+             '()
+             `((inputs
+                (list ,@(map string->symbol dependencies)))))
+       ,@(if (null? dev-dependencies)
+             '()
+             `((native-inputs
+                (list ,@(map string->symbol dev-dependencies)))))
+       (synopsis "")
+       (description ,(composer-package-description composer-package))
+       (home-page ,(composer-package-homepage composer-package))
+       (license ,(or (composer-package-license composer-package)
+                     'unknown-license!)))))
 
 (define composer->guix-package
   (memoize
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index d1855b3698..a755387242 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -208,11 +209,6 @@ (define* (fetch-elpa-package name #:optional (repo 'gnu))
                             url)))
       (_ #f))))
 
-(define* (download-git-repository url ref)
-  "Fetch the given REF from the Git repository at URL."
-  (with-store store
-    (latest-repository-commit store url #:ref ref)))
-
 (define (package-name->melpa-recipe package-name)
   "Fetch the MELPA recipe for PACKAGE-NAME, represented as an alist from
 keywords to values."
@@ -232,28 +228,15 @@ (define (data->recipe data)
     (close-port port)
     (data->recipe (cons ':name data))))
 
-(define (git-repository->origin recipe url)
-  "Fetch origin details from the Git repository at URL for the provided MELPA
-RECIPE."
-  (define ref
-    (cond
-     ((assoc-ref recipe #:branch)
-      => (lambda (branch) (cons 'branch branch)))
-     ((assoc-ref recipe #:commit)
-      => (lambda (commit) (cons 'commit commit)))
-     (else
-      '())))
-
-  (let-values (((directory commit) (download-git-repository url ref)))
-    `(origin
-       (method git-fetch)
-       (uri (git-reference
-             (url ,url)
-             (commit ,commit)))
-       (sha256
-        (base32
-         ,(bytevector->nix-base32-string
-           (file-hash* directory #:recursive? #true)))))))
+(define (ref recipe)
+  "Create REF from MELPA RECIPE."
+  (cond
+   ((assoc-ref recipe #:branch)
+    => (lambda (branch) (cons 'branch branch)))
+   ((assoc-ref recipe #:commit)
+    => (lambda (commit) (cons 'commit commit)))
+   (else
+    '())))
 
 (define* (melpa-recipe->origin recipe)
   "Fetch origin details from the MELPA recipe and associated repository for
@@ -264,9 +247,9 @@ (define (gitlab-repo->url repo)
     (string-append "https://gitlab.com/" repo ".git"))
 
   (match (assq-ref recipe ':fetcher)
-    ('github (git-repository->origin recipe (github-repo->url (assq-ref recipe ':repo))))
-    ('gitlab (git-repository->origin recipe (gitlab-repo->url (assq-ref recipe ':repo))))
-    ('git    (git-repository->origin recipe (assq-ref recipe ':url)))
+    ('github (git->origin (github-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+    ('gitlab (git->origin (gitlab-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+    ('git    (git->origin (assq-ref recipe ':url) (ref recipe)))
     (#f #f)   ; if we're not using melpa then this stops us printing a warning
     (_ (warning (G_ "unsupported MELPA fetcher: ~a, falling back to unstable MELPA source~%")
                 (assq-ref recipe ':fetcher))
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..6e2ce2ed00 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -514,49 +515,24 @@ (define (module-meta-data-repo-url meta-data goproxy)
       goproxy
       (module-meta-repo-root meta-data)))
 
-(define* (git-checkout-hash url reference algorithm)
-  "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or
-tag."
-  (define cache
-    (string-append (or (getenv "TMPDIR") "/tmp")
-                   "/guix-import-go-"
-                   (passwd:name (getpwuid (getuid)))))
+;; This is done because the version field of the package, which the generated
+;; quoted expression refers to, has been stripped of any 'v' prefixed.
+(define (transform-version version)
+  (let ((plain-version? (string=? version (go-version->git-ref version)))
+        (v-prefixed?    (string-prefix? "v" version)))
+    (if (and plain-version? v-prefixed?)
+        '(string-append "v" version)
+        '(go-version->git-ref version))))
 
-  ;; Use a custom cache to avoid cluttering the default one under
-  ;; ~/.cache/guix, but choose one under /tmp so that it's persistent across
-  ;; subsequent "guix import" invocations.
-  (mkdir-p cache)
-  (chmod cache #o700)
-  (let-values (((checkout commit _)
-                (parameterize ((%repository-cache-directory cache))
-                  (update-cached-checkout url
-                                          #:ref
-                                          `(tag-or-commit . ,reference)))))
-    (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+                      #:key (transform-version #f))
   "Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
   (case vcs-type
     ((git)
-     (let ((plain-version? (string=? version (go-version->git-ref version)))
-           (v-prefixed?    (string-prefix? "v" version)))
-       `(origin
-          (method git-fetch)
-          (uri (git-reference
-                (url ,vcs-repo-url)
-                ;; This is done because the version field of the package,
-                ;; which the generated quoted expression refers to, has been
-                ;; stripped of any 'v' prefixed.
-                (commit ,(if (and plain-version? v-prefixed?)
-                             '(string-append "v" version)
-                             '(go-version->git-ref version)))))
-          (file-name (git-file-name name version))
-          (sha256
-           (base32
-            ,(bytevector->nix-base32-string
-              (git-checkout-hash vcs-repo-url (go-version->git-ref version)
-                                 (hash-algorithm sha256))))))))
+     (git->origin vcs-repo-url `(tag-or-commit . ,version)
+                  #:ref->commit transform-version))
     ((hg)
      `(origin
         (method hg-fetch)
@@ -649,7 +625,8 @@ (define* (go-module->guix-package module-path #:key
         (name ,guix-name)
         (version ,(strip-v-prefix version*))
         (source
-         ,(vcs->origin vcs-type vcs-repo-url version*))
+         ,(vcs->origin vcs-type vcs-repo-url version*
+                       #:transform-version transform-version))
         (build-system go-build-system)
         (arguments
          (list ,@(if (version>? min-go-version (package-version (go-package)))
diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm
index 5ea6e023ce..65ef242431 100644
--- a/guix/import/minetest.scm
+++ b/guix/import/minetest.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021, 2022 Maxime Devos <maximedevos@telenet.be>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,7 +33,6 @@ (define-module (guix import minetest)
   #:use-module (guix import utils)
   #:use-module (guix import json)
   #:use-module (json)
-  #:use-module (guix base32)
   #:use-module (guix git)
   #:use-module ((guix git-download) #:prefix download:)
   #:use-module (guix hash)
@@ -277,12 +277,6 @@ (define url (string-append (%contentdb-api) "packages/?type=" type
 
 \f
 
-;; XXX copied from (guix import elpa)
-(define* (download-git-repository url ref)
-  "Fetch the given REF from the Git repository at URL."
-  (with-store store
-    (latest-repository-commit store url #:ref ref)))
-
 (define (make-minetest-sexp author/name version repository commit
                             inputs home-page synopsis
                             description media-license license)
@@ -293,24 +287,8 @@ (define (make-minetest-sexp author/name version repository commit
      (name ,(contentdb->package-name author/name))
      (version ,version)
      (source
-       (origin
-         (method git-fetch)
-         (uri (git-reference
-                (url ,repository)
-                (commit ,commit)))
-         (sha256
-          (base32
-           ;; The git commit is not always available.
-           ,(and commit
-                 (bytevector->nix-base32-string
-                  (file-hash*
-                   (download-git-repository repository
-                                            `(commit . ,commit))
-                   ;; 'download-git-repository' already filtered out the '.git'
-                   ;; directory.
-                   #:select? (const #true)
-                   #:recursive? #true)))))
-         (file-name (git-file-name name version))))
+      ,(git->origin
+        repository `(tag-or-commit . ,commit) #:ref->commit #t))
      (build-system minetest-mod-build-system)
      ,@(maybe-propagated-inputs (map contentdb->package-name inputs))
      (home-page ,home-page)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde..47254539a1 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2022 Alice Brenon <alice.brenon@ens-lyon.fr>
 ;;; Copyright © 2022 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -39,6 +40,8 @@ (define-module (guix import utils)
   #:use-module (guix packages)
   #:use-module (guix discovery)
   #:use-module (guix build-system)
+  #:use-module (guix git)
+  #:use-module (guix hash)
   #:use-module ((guix i18n) #:select (G_))
   #:use-module (guix store)
   #:use-module (guix download)
@@ -63,6 +66,7 @@ (define-module (guix import utils)
 
             url-fetch
             guix-hash-url
+            git->origin
 
             package-names->package-inputs
             maybe-inputs
@@ -161,6 +165,41 @@ (define (guix-hash-url filename)
   "Return the hash of FILENAME in nix-base32 format."
   (bytevector->nix-base32-string (file-sha256 filename)))
 
+(define* (git->origin repo-url ref #:key (ref->commit #f))
+  "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary.  REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
+  (let* ((version (and (pair? ref) (cdr ref)))
+         (directory commit
+                    (if version
+                        (with-store store
+                          (latest-repository-commit store repo-url
+                                                    #:ref (if version ref '())))
+                        (values #f #f)))
+         (vcommit (match ref->commit
+                    (#t    commit)
+                    (#f    version)
+                    ((? procedure?) (ref->commit version))
+                    (_     #f))))
+    (values
+     `(origin
+        (method git-fetch)
+        (uri (git-reference
+              (url ,(and (not (eq? repo-url 'null)) repo-url))
+              (commit ,vcommit)))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          ,(and version  ; Version or commit is not always available.
+                (bytevector->nix-base32-string
+                 (file-hash* directory
+                             ;; 'git-fetch' already filtered out '.git'.
+                             #:select? (const #true)
+                             #:recursive? #true))))))
+     directory)))
+
 (define %spdx-license-identifiers
   ;; https://spdx.org/licenses/
   ;; The gfl1.0, nmap, repoze
diff --git a/tests/go.scm b/tests/go.scm
index d2e8846b30..4644e1bd1c 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,7 +25,6 @@ (define-module (tests-import-go)
   #:use-module (guix base32)
   #:use-module (guix build-system go)
   #:use-module (guix import go)
-  #:use-module (guix base32)
   #:use-module ((guix utils) #:select (call-with-temporary-directory))
   #:use-module (guix tests)
   #:use-module (ice-9 match)
@@ -403,13 +403,26 @@ (define (mock-http-get testcase)
             (mock-http-get fixtures-go-check-test))
          (mock ((guix http-client) http-fetch
                 (mock-http-fetch fixtures-go-check-test))
-             (mock ((guix git) update-cached-checkout
-                    (lambda* (url #:key ref)
-                      ;; Return an empty directory and its hash.
-                      (values checkout
-                              (nix-base32-string->bytevector
-                               "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")
-                              #f)))
+             (mock ((guix import utils) git->origin
+                    ;; Mock an empty directory by replacing hash.
+                    (lambda* (repo-url ref #:key (ref->commit #f))
+                      (let* ((version (if (pair? ref)
+                                          (cdr ref)
+                                          #f))
+                             (vcommit (match ref->commit
+                                        (#t    commit)
+                                        (#f    version)
+                                        ((? procedure?) (ref->commit version))
+                                        (_     #f))))
+                        `(origin
+                           (method git-fetch)
+                           (uri (git-reference
+                                 (url ,(and (not (eq? repo-url 'null)) repo-url))
+                                 (commit ,vcommit)))
+                           (file-name (git-file-name name version))
+                           (sha256
+                            (base32
+                             "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"))))))
                  (go-module->guix-package* "github.com/go-check/check")))))))
 
 (test-end "go")
diff --git a/tests/minetest.scm b/tests/minetest.scm
index 78469bf95b..7ff72dbfdc 100644
--- a/tests/minetest.scm
+++ b/tests/minetest.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -57,15 +58,7 @@ (define* (make-package-sexp #:key
   `(package
      (name ,guix-name)
      (version ,version)
-     (source
-      (origin
-        (method git-fetch)
-        (uri (git-reference
-              (url ,(and (not (eq? repo 'null)) repo))
-              (commit #f)))
-        (sha256
-         (base32 #f))
-        (file-name (git-file-name name version))))
+     (source ,(git->origin repo #f))
      (build-system minetest-mod-build-system)
      ,@(maybe-propagated-inputs inputs)
      (home-page ,home-page)
@@ -419,8 +412,8 @@ (define* (example-package #:key
            (uri (git-reference
                  (url repo)
                  (commit commit #;"808f9ffbd3106da4c92d2367b118b98196c9e81e")))
-           (sha256 #f) ; not important for the following tests
-           (file-name (git-file-name name version)))
+           (file-name (git-file-name name version))
+           (sha256 #f)) ; not important for the following tests
          source))
     (build-system minetest-mod-build-system)
     (license #f)
-- 
2.41.0

[bug#62202] [PATCH v3 2/4] import: Add juliahub importer.

[Nicolas Graves via Guix-patches via]

---
 doc/guix.texi                    |  27 +++
 guix/import/juliahub.scm         | 309 +++++++++++++++++++++++++++++++
 guix/scripts/import.scm          |   2 +-
 guix/scripts/import/juliahub.scm | 107 +++++++++++
 4 files changed, 444 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/juliahub.scm
 create mode 100644 guix/scripts/import/juliahub.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index b742a3d5b2..f50bb3f328 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14678,6 +14678,33 @@ guix import hexpm cf@@0.3.0
 
 Additional options include:
 
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
+@item juliahub
+@cindex juliahub
+Import metadata from both the General
+@uref{https://github.com/JuliaRegistries/General} and Juliahub
+@uref{https://juliahub.com} Julia package repositories, as in this
+example:
+
+@example
+guix import juliahub Cthulhu@@2.8.9
+@end example
+
+The supplied package name must have the same case as in the
+aforementioned package repositories, and the version used must be an
+exact version (e.g. @code{2.8.9} instead of @code{2.8}). The command
+will also fail in the case of a Julia package that doesn't use a git
+tag.
+
+Additional options include:
+
 @table @code
 @item --recursive
 @itemx -r
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..ab838b6035
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,309 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+  #:use-module (ice-9 textual-ports)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-71)
+  #:use-module (guix http-client)
+  #:use-module (guix git)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix base32)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module ((guix licenses) #:prefix license:)
+
+  #:export (juliahub->guix-package
+            %juliahub-updater
+            juliahub-recursive-import))
+
+
+;; JuliaHub API.
+(define (juliahub-redirect-uri name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (port (http-fetch url #:text? #t))
+         (_ (get-line port))
+         (meta (get-line port))
+         (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+         (redirect (match:substring (string-match regex meta))))
+    (close-port port)
+    (string-drop redirect 4)))
+
+(define (juliahub-url name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (uri (juliahub-redirect-uri name)))
+    (string-append url uri "/")))
+
+;; General package repository.
+(define %general-base-url
+  "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+  (let ((folder (string-capitalize (string-take package-name 1))))
+    (string-append
+     %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-line->alist line)
+  (let* ((l (string-split line #\=))
+         (attribute (string->symbol (string-drop-right (car l) 1)))
+         (value (string-drop (string-drop-right (cadr l) 1) 2)))
+    `(,attribute . ,value)))
+
+(define (ini-fetch url)
+  (let* ((port (http-fetch url #:text? #t))
+         (raw (get-string-all port))
+         (lines (drop-right (string-split raw #\newline) 1)))
+    (close-port port)
+    (map ini-line->alist lines)))
+
+;; Filtering out julia-stdlibs.
+;; To update them, see file sysimg.jl.
+(define %julia-stdlibs
+  (list "julia"
+        "ArgTools"
+        "Artifacts"
+        "Base64"
+        "CRC32c"
+        "FileWatching"
+        "Libdl"
+        "Logging"
+        "Mmap"
+        "NetworkOptions"
+        "SHA"
+        "Serialization"
+        "Sockets"
+        "Unicode"
+        "DelimitedFiles"
+        "LinearAlgebra"
+        "Markdown"
+        "Printf"
+        "Random"
+        "Tar"
+        "Dates"
+        "Distributed"
+        "Future"
+        "InteractiveUtils"
+        "LibGit2"
+        "Profile"
+        "SparseArrays"
+        "UUIDs"
+        "REPL"
+        "SharedArrays"
+        "Statistics"
+        "SuiteSparse"
+        "TOML"
+        "Test"
+        "LibCURL"
+        "Downloads"
+        "Pkg"
+        "LazyArtifacts"))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+  json->juliahub-package
+  (homepage juliahub-package-homepage) ;string
+  (readme juliahub-package-readme) ;string
+  (version juliahub-package-version) ;string
+  (description juliahub-package-description) ;string
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-dependencies) ;list of <juliahub-dependency>
+  (url juliahub-package-url) ;string
+  (uuid juliahub-package-uuid) ;string
+  (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+  make-juliahub-dependency juliahub-dependency?
+  json->juliahub-dependency
+  (direct? juliahub-dependency-direct? "direct") ;boolean
+  (name juliahub-dependency-name) ;string
+  (uuid juliahub-dependency-uuid) ;string
+  (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+
+(define (julia-name->guix-name name)
+  (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+  "Return a <juliahub-package> record for package NAME, or #f on failure."
+  (let* ((uri (juliahub-redirect-uri name))
+         (slug (string-take uri 5))
+         (url (if version
+                  (string-append "https://docs.juliahub.com/" name "/"
+                                 slug "/" version "/pkg.json")
+                  (string-append (juliahub-url name) "pkg.json"))))
+    (and=> (json-fetch url) json->juliahub-package)))
+
+(define (make-julia-sexp name version source home-page synopsis description
+                         direct-dependencies test-dependencies-names license)
+  "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSE."
+  `(package
+     (name ,(julia-name->guix-name name))
+     (version ,version)
+     (source ,source)
+     (build-system julia-build-system)
+     ,@(if (null? direct-dependencies)
+           '()
+           `((propagated-inputs
+              (list ,@(map (compose string->symbol
+                                    julia-name->guix-name
+                                    juliahub-dependency-name)
+                           direct-dependencies)))))
+     ,@(if (null? test-dependencies-names)
+           '()
+           `((native-inputs
+              (list ,@(map (compose string->symbol julia-name->guix-name)
+                           test-dependencies-names)))))
+     (synopsis ,synopsis)
+     (description ,description)
+     (home-page ,home-page)
+     (license ,(if license (spdx-string->license license) #f))))
+
+;; Dependencies helpers.
+(define (json->juliahub-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (and (juliahub-dependency-direct? dep)
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
+               dep
+               #f)))
+       (vector->list vector))))
+
+(define (parse-test-dependencies directory)
+  (let* ((port (open-input-file (string-append directory "/Project.toml")))
+         (project.toml (get-string-all port))
+         (regex "\ntest = \\[.*\\]")
+         (deps (match:substring (string-match regex project.toml)))
+         (pure (string-delete (list->char-set (list #\" #\ )) deps)))
+    (close-port port)
+    (filter (lambda (x) (not (member x %julia-stdlibs)))
+            (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-tag, and import
+;; this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag. In this case, the script fails, but it seems quite rare. We could
+;; introduce the tree-commit which is available in the Versions.toml file in the
+;; General repository. This can be used to identify the state of a repository,
+;; since we have a unique hash of the listing of files and directories.
+
+(define (latest-git-tag repo)
+  (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+         (last-git-tag (last (string-split last-ref #\/))))
+    (string-drop last-git-tag 1)))
+
+(define* (juliahub->guix-package package-name
+                                 #:key version #:allow-other-keys)
+  "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
+         (subdir (assoc-ref package-toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+         (package (if version
+                      (juliahub-fetch package-name #:version version)
+                      (if tag
+                          (juliahub-fetch package-name #:version tag)
+                          (juliahub-fetch package-name)))))
+    (if package
+        (let* ((source directory
+                       (git->origin
+                        (juliahub-package-url package)
+                        `(tag-or-commit
+                          . ,(string-append
+                              "v" (juliahub-package-version package)))))
+               (direct-dependencies
+                (filter juliahub-dependency-direct?
+                        (juliahub-package-dependencies package)))
+               (dependencies-names (map juliahub-dependency-name
+                                        direct-dependencies))
+               (test-dependencies-names
+                (if subdir
+                    (parse-test-dependencies
+                     (string-append subdir "/" directory))
+                    (parse-test-dependencies directory)))
+               (homepage (juliahub-package-homepage package)))
+          (values (make-julia-sexp
+                   package-name
+                   (juliahub-package-version package)
+                   source
+                   (match homepage
+                     ("" (juliahub-package-url package))
+                     ((? string?) homepage)
+                     (_ (juliahub-package-url package)))
+                   (juliahub-package-description package)
+                   (beautify-description
+                    (juliahub-package-readme package))
+                   direct-dependencies
+                   test-dependencies-names
+                   (juliahub-package-license package))
+                  (append dependencies-names test-dependencies-names)))
+    (values #f '()))))
+
+;; We must use the url to get a name with the true case of juliahub/general.
+(define (guix-package->juliahub-name package)
+  (let* ((url (juliahub-package-url package))
+         (git-name (last (string-split url #\/)))
+         (ungitted-name (if (string-suffix? ".git" git-name)
+                            (string-drop-right git-name 4)
+                            git-name))
+         (package-name (if (string-suffix? ".jl" ungitted-name)
+                           (string-drop-right ungitted-name 4)
+                           ungitted-name)))
+    package-name))
+
+(define* (import-release package #:key (version #f))
+  "Return an <upstream-source> for the latest release of PACKAGE."
+  (let* ((package-name (guix-package->juliahub-name package))
+         (package      (juliahub-fetch package-name))
+         (version  (or version (juliahub-package-version package))))
+    (upstream-source
+     (package (package-name package))
+     (version version)
+     (urls (list (juliahub-package-url package))))))
+
+(define %juliahub-updater
+  (upstream-updater
+   (name 'juliahub)
+   (description "Updater for Juliahub packages")
+   (pred juliahub-package?)
+   (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+  (recursive-import package-name
+                    #:repo '()
+                    #:repo->guix-package juliahub->guix-package
+                    #:guix-name julia-name->guix-name
+                    #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index d2a1cee56e..8926c9610f 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
 
 (define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
                     "gem" "go" "cran" "crate" "texlive" "json" "opam"
-                    "minetest" "elm" "hexpm" "composer"))
+                    "minetest" "elm" "hexpm" "composer" "juliahub"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import juliahub)
+  #:use-module (guix scripts import)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 receive)
+  #:export (guix-import-juliahub))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME.  Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (display (G_ "
+  -r, --recursive        generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import gem")))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
+         %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (parse-command-line args %options (list %default-options)
+                        #:build-options? #f))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                             (('argument . value)
+                              value)
+                             (_ #f))
+                           (reverse opts))))
+    (match args
+      ((spec)
+       (receive (package-name package-version)
+           (package-name->name+version spec)
+         (let ((code (if (assoc-ref opts 'recursive)
+                         (map (match-lambda
+                                ((and ('package ('name name) . rest) pkg)
+                                 `(define-public ,(string->symbol name)
+                                    ,pkg))
+                                (_ #f))
+                              (juliahub-recursive-import package-name package-version))
+                         (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+                           (if sexp sexp #f)))))
+           (match code
+             ((or #f '(#f))
+              (leave (G_ "failed to download meta-data for package '~a'~%")
+                     package-name))
+             (_ code)))))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.41.0

[bug#62202] [PATCH v3 3/4] import: juliahub: Beautify description.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 53 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 51 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index ab838b6035..e4540de06d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -30,8 +30,10 @@ (define-module (guix import juliahub)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
-  #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (json)
+  #:use-module (htmlprag)
+  #:use-module (sxml transform)
 
   #:export (juliahub->guix-package
             %juliahub-updater
@@ -76,6 +78,53 @@ (define (ini-fetch url)
     (close-port port)
     (map ini-line->alist lines)))
 
+;; Beautify description.
+(define %juliahub-beautify-description-rules
+  `((h1 *preorder*   . ,(lambda args #f))
+    (h2 *preorder*   . ,(lambda args #f))
+    (h3 *preorder*   . ,(lambda args #f))
+    (h4 *preorder*   . ,(lambda args #f))
+    (h5 *preorder*   . ,(lambda args #f))
+    (h6 *preorder*   . ,(lambda args #f))
+    (hr *preorder*   . ,(lambda args #f))
+    (span *preorder* . ,(lambda args #f))
+    (img *preorder*  . ,(lambda args #f))
+    (pre *preorder*  . ,(lambda args #f))
+    (div *preorder*  . ,(lambda args #f))
+    (table *preorder* . ,(lambda args #f))
+    (imgalt *preorder* . ,(lambda args #f))
+    (@ *preorder* . ,(lambda args #f))
+    (*TOP*        . ,(lambda args (cdr args)))
+    (p            . ,(lambda args (cdr args)))
+    (em           . ,(lambda args (cdr args)))
+    (strong       . ,(lambda args (cdr args)))
+    (a            . ,(lambda args
+                       (match args
+                         ((tag link ref)
+                          (if ref ref #f))
+                         (_ #f))))
+    (ul           . ,(lambda args
+                       `("@itemize" ,@(cdr args) "\n@end itemize")))
+    (ol           . ,(lambda args
+                       `("@enumerate" ,@(cdr args) "@end enumerate")))
+    (blockquote   . ,(lambda args
+                       `("@quotation" ,@(cdr args) "@end quotation")))
+    (li           . ,(lambda args
+                       `("\n@item" ,@(cdr args))))
+    (code         . ,(lambda args
+                       `("@code{" ,@(cdr args) "}")))
+    (*text*       . ,(lambda (tag x) x))
+    (*default*    . ,(lambda (tag . body)
+                       (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+  (string-join
+   (filter (lambda (x) (if (equal? x " ") #f x))
+           (flatten
+            (pre-post-order (html->sxml description)
+                            %juliahub-beautify-description-rules)))
+   " "))
+
 ;; Filtering out julia-stdlibs.
 ;; To update them, see file sysimg.jl.
 (define %julia-stdlibs
@@ -264,7 +313,7 @@ (define* (juliahub->guix-package package-name
                      ((? string?) homepage)
                      (_ (juliahub-package-url package)))
                    (juliahub-package-description package)
-                   (beautify-description
+                   ((compose beautify-description juliahub-beautify-description)
                     (juliahub-package-readme package))
                    direct-dependencies
                    test-dependencies-names
-- 
2.41.0

[bug#62202] [PATCH v3 4/4] import: utils: Rule out texinfo common syntax from @ escape.

[Nicolas Graves via Guix-patches via]

---
 guix/import/utils.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 47254539a1..57e4ec0ce7 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -375,7 +375,13 @@ (define* (beautify-description description #:optional (length 80))
                    (cut string-trim-both <> #\')
                    ;; Escape single @ to prevent it from being understood as
                    ;; invalid Texinfo syntax.
-                   (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+                   (lambda (word)
+                     (if  ; Rule out some valid Texinfo syntax.
+                      (member word '("@itemize" "@item" "@end" "@quotation"
+                                     "@enumerate" "@code" "@code{"))
+                      word
+                      ((cut regexp-substitute/global
+                            #f "@" <> 'pre "@@" 'post) word)))
                    ;; Wrap camelCase or PascalCase words in @code{...}.
                    (lambda (word)
                      (let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
-- 
2.41.0

[bug#62202] [PATCH v4 1/6] import: utils: Add function git->origin.

[Nicolas Graves via Guix-patches via]

* guix/import/utils.scm: (git->origin): Add function.

* guix/import/elpa.scm
(download-git-repository): Remove function download-git-repository.
(git-repository->origin): Remove function git-repository->origin.
(ref): Add function ref.
(melpa-recipe->origin): Use functions git->origin and ref.

* guix/import/go.scm
(git-checkout-hash): Remove function git-checkout-hash.
(transform-version): Add function transform-version.
(vcs->origin): Use functions git->origin and transform-version. Add
optional argument transform-version.

* tests/import/go.scm
(go-module->guix-package): Adapt test case to changes in guix/import/go.scm.

* guix/import/minetest.scm
(download-git-repository): Remove function download-git-repository.
(make-minetest-sexp): Use function git->origin.

* tests/minetest.scm
(make-package-sexp): Use function git->origin.
(example-package): Adapt test-case to git->origin.

* guix/import/composer.scm
(make-php-sexp): Use function git->origin.

Change-Id: Ied05a63bdd60fbafe26fbbb4e115ff6f0bb9db3c
---
 guix/import/composer.scm | 86 ++++++++++++++--------------------------
 guix/import/elpa.scm     | 43 ++++++--------------
 guix/import/go.scm       | 57 ++++++++------------------
 guix/import/minetest.scm | 28 ++-----------
 guix/import/utils.scm    | 39 ++++++++++++++++++
 tests/go.scm             | 29 ++++++++++----
 tests/minetest.scm       | 15 ++-----
 7 files changed, 127 insertions(+), 170 deletions(-)

diff --git a/guix/import/composer.scm b/guix/import/composer.scm
index 1ad608964b..dabc5423ed 100644
--- a/guix/import/composer.scm
+++ b/guix/import/composer.scm
@@ -19,22 +19,17 @@
 (define-module (guix import composer)
   #:use-module (ice-9 match)
   #:use-module (json)
-  #:use-module (guix hash)
-  #:use-module (guix base32)
-  #:use-module (guix build git)
-  #:use-module (guix build utils)
-  #:use-module (guix build-system)
   #:use-module (guix build-system composer)
+  #:use-module ((guix download) #:select (download-to-store))
   #:use-module (guix import json)
   #:use-module (guix import utils)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix memoization)
   #:use-module (guix packages)
-  #:use-module (guix serialization)
+  #:use-module (guix store)
   #:use-module (guix upstream)
   #:use-module (guix utils)
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:export (composer->guix-package
             %composer-updater
@@ -141,55 +136,34 @@ (define (make-php-sexp composer-package)
          (dependencies (map php-package-name
                             (composer-package-require composer-package)))
          (dev-dependencies (map php-package-name
-                                (composer-package-dev-require composer-package)))
-         (git? (equal? (composer-source-type source) "git")))
-    ((if git? call-with-temporary-directory call-with-temporary-output-file)
-     (lambda* (temp #:optional port)
-       (and (if git?
-               (begin
-                 (mkdir-p temp)
-                 (git-fetch (composer-source-url source)
-                            (composer-source-reference source)
-                            temp))
-               (url-fetch (composer-source-url source) temp))
-            `(package
-               (name ,(composer-package-name composer-package))
-               (version ,(composer-package-version composer-package))
-               (source
-                (origin
-                  ,@(if git?
-                        `((method git-fetch)
-                          (uri (git-reference
-                                (url ,(if (string-suffix?
-                                           ".git"
-                                           (composer-source-url source))
-                                          (string-drop-right
-                                           (composer-source-url source)
-                                           (string-length ".git"))
-                                          (composer-source-url source)))
-                                (commit ,(composer-source-reference source))))
-                          (file-name (git-file-name name version))
-                          (sha256
-                           (base32
-                            ,(bytevector->nix-base32-string
-                              (file-hash* temp)))))
-                        `((method url-fetch)
-                          (uri ,(composer-source-url source))
-                          (sha256 (base32 ,(guix-hash-url temp)))))))
-               (build-system composer-build-system)
-               ,@(if (null? dependencies)
-                     '()
-                     `((inputs
-                        (list ,@(map string->symbol dependencies)))))
-               ,@(if (null? dev-dependencies)
-                     '()
-                     `((native-inputs
-                        (list ,@(map string->symbol dev-dependencies)))))
-               (synopsis "")
-               (description ,(composer-package-description composer-package))
-               (home-page ,(composer-package-homepage composer-package))
-               (license ,(or (composer-package-license composer-package)
-                             'unknown-license!))))))))
+                                (composer-package-dev-require composer-package))))
+    `(package
+       (name ,(composer-package-name composer-package))
+       (version ,(composer-package-version composer-package))
+       (source
+        ,(if (string= (composer-source-type source) "git")
+             (git->origin (composer-source-url source)
+                          `(tag-or-commit . ,(composer-source-reference source)))
+             (let* ((source (composer-source-url source))
+                    (tarball (with-store store (download-to-store store source))))
+               `(origin
+                  (method url-fetch)
+                  (uri ,source)
+                  (sha256 (base32 ,(guix-hash-url tarball)))))))
+       (build-system composer-build-system)
+       ,@(if (null? dependencies)
+             '()
+             `((inputs
+                (list ,@(map string->symbol dependencies)))))
+       ,@(if (null? dev-dependencies)
+             '()
+             `((native-inputs
+                (list ,@(map string->symbol dev-dependencies)))))
+       (synopsis "")
+       (description ,(composer-package-description composer-package))
+       (home-page ,(composer-package-homepage composer-package))
+       (license ,(or (composer-package-license composer-package)
+                     'unknown-license!)))))
 
 (define composer->guix-package
   (memoize
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index d1855b3698..a755387242 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -208,11 +209,6 @@ (define* (fetch-elpa-package name #:optional (repo 'gnu))
                             url)))
       (_ #f))))
 
-(define* (download-git-repository url ref)
-  "Fetch the given REF from the Git repository at URL."
-  (with-store store
-    (latest-repository-commit store url #:ref ref)))
-
 (define (package-name->melpa-recipe package-name)
   "Fetch the MELPA recipe for PACKAGE-NAME, represented as an alist from
 keywords to values."
@@ -232,28 +228,15 @@ (define (data->recipe data)
     (close-port port)
     (data->recipe (cons ':name data))))
 
-(define (git-repository->origin recipe url)
-  "Fetch origin details from the Git repository at URL for the provided MELPA
-RECIPE."
-  (define ref
-    (cond
-     ((assoc-ref recipe #:branch)
-      => (lambda (branch) (cons 'branch branch)))
-     ((assoc-ref recipe #:commit)
-      => (lambda (commit) (cons 'commit commit)))
-     (else
-      '())))
-
-  (let-values (((directory commit) (download-git-repository url ref)))
-    `(origin
-       (method git-fetch)
-       (uri (git-reference
-             (url ,url)
-             (commit ,commit)))
-       (sha256
-        (base32
-         ,(bytevector->nix-base32-string
-           (file-hash* directory #:recursive? #true)))))))
+(define (ref recipe)
+  "Create REF from MELPA RECIPE."
+  (cond
+   ((assoc-ref recipe #:branch)
+    => (lambda (branch) (cons 'branch branch)))
+   ((assoc-ref recipe #:commit)
+    => (lambda (commit) (cons 'commit commit)))
+   (else
+    '())))
 
 (define* (melpa-recipe->origin recipe)
   "Fetch origin details from the MELPA recipe and associated repository for
@@ -264,9 +247,9 @@ (define (gitlab-repo->url repo)
     (string-append "https://gitlab.com/" repo ".git"))
 
   (match (assq-ref recipe ':fetcher)
-    ('github (git-repository->origin recipe (github-repo->url (assq-ref recipe ':repo))))
-    ('gitlab (git-repository->origin recipe (gitlab-repo->url (assq-ref recipe ':repo))))
-    ('git    (git-repository->origin recipe (assq-ref recipe ':url)))
+    ('github (git->origin (github-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+    ('gitlab (git->origin (gitlab-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+    ('git    (git->origin (assq-ref recipe ':url) (ref recipe)))
     (#f #f)   ; if we're not using melpa then this stops us printing a warning
     (_ (warning (G_ "unsupported MELPA fetcher: ~a, falling back to unstable MELPA source~%")
                 (assq-ref recipe ':fetcher))
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..6e2ce2ed00 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -514,49 +515,24 @@ (define (module-meta-data-repo-url meta-data goproxy)
       goproxy
       (module-meta-repo-root meta-data)))
 
-(define* (git-checkout-hash url reference algorithm)
-  "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or
-tag."
-  (define cache
-    (string-append (or (getenv "TMPDIR") "/tmp")
-                   "/guix-import-go-"
-                   (passwd:name (getpwuid (getuid)))))
+;; This is done because the version field of the package, which the generated
+;; quoted expression refers to, has been stripped of any 'v' prefixed.
+(define (transform-version version)
+  (let ((plain-version? (string=? version (go-version->git-ref version)))
+        (v-prefixed?    (string-prefix? "v" version)))
+    (if (and plain-version? v-prefixed?)
+        '(string-append "v" version)
+        '(go-version->git-ref version))))
 
-  ;; Use a custom cache to avoid cluttering the default one under
-  ;; ~/.cache/guix, but choose one under /tmp so that it's persistent across
-  ;; subsequent "guix import" invocations.
-  (mkdir-p cache)
-  (chmod cache #o700)
-  (let-values (((checkout commit _)
-                (parameterize ((%repository-cache-directory cache))
-                  (update-cached-checkout url
-                                          #:ref
-                                          `(tag-or-commit . ,reference)))))
-    (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+                      #:key (transform-version #f))
   "Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
   (case vcs-type
     ((git)
-     (let ((plain-version? (string=? version (go-version->git-ref version)))
-           (v-prefixed?    (string-prefix? "v" version)))
-       `(origin
-          (method git-fetch)
-          (uri (git-reference
-                (url ,vcs-repo-url)
-                ;; This is done because the version field of the package,
-                ;; which the generated quoted expression refers to, has been
-                ;; stripped of any 'v' prefixed.
-                (commit ,(if (and plain-version? v-prefixed?)
-                             '(string-append "v" version)
-                             '(go-version->git-ref version)))))
-          (file-name (git-file-name name version))
-          (sha256
-           (base32
-            ,(bytevector->nix-base32-string
-              (git-checkout-hash vcs-repo-url (go-version->git-ref version)
-                                 (hash-algorithm sha256))))))))
+     (git->origin vcs-repo-url `(tag-or-commit . ,version)
+                  #:ref->commit transform-version))
     ((hg)
      `(origin
         (method hg-fetch)
@@ -649,7 +625,8 @@ (define* (go-module->guix-package module-path #:key
         (name ,guix-name)
         (version ,(strip-v-prefix version*))
         (source
-         ,(vcs->origin vcs-type vcs-repo-url version*))
+         ,(vcs->origin vcs-type vcs-repo-url version*
+                       #:transform-version transform-version))
         (build-system go-build-system)
         (arguments
          (list ,@(if (version>? min-go-version (package-version (go-package)))
diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm
index 5ea6e023ce..65ef242431 100644
--- a/guix/import/minetest.scm
+++ b/guix/import/minetest.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021, 2022 Maxime Devos <maximedevos@telenet.be>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,7 +33,6 @@ (define-module (guix import minetest)
   #:use-module (guix import utils)
   #:use-module (guix import json)
   #:use-module (json)
-  #:use-module (guix base32)
   #:use-module (guix git)
   #:use-module ((guix git-download) #:prefix download:)
   #:use-module (guix hash)
@@ -277,12 +277,6 @@ (define url (string-append (%contentdb-api) "packages/?type=" type
 
 \f
 
-;; XXX copied from (guix import elpa)
-(define* (download-git-repository url ref)
-  "Fetch the given REF from the Git repository at URL."
-  (with-store store
-    (latest-repository-commit store url #:ref ref)))
-
 (define (make-minetest-sexp author/name version repository commit
                             inputs home-page synopsis
                             description media-license license)
@@ -293,24 +287,8 @@ (define (make-minetest-sexp author/name version repository commit
      (name ,(contentdb->package-name author/name))
      (version ,version)
      (source
-       (origin
-         (method git-fetch)
-         (uri (git-reference
-                (url ,repository)
-                (commit ,commit)))
-         (sha256
-          (base32
-           ;; The git commit is not always available.
-           ,(and commit
-                 (bytevector->nix-base32-string
-                  (file-hash*
-                   (download-git-repository repository
-                                            `(commit . ,commit))
-                   ;; 'download-git-repository' already filtered out the '.git'
-                   ;; directory.
-                   #:select? (const #true)
-                   #:recursive? #true)))))
-         (file-name (git-file-name name version))))
+      ,(git->origin
+        repository `(tag-or-commit . ,commit) #:ref->commit #t))
      (build-system minetest-mod-build-system)
      ,@(maybe-propagated-inputs (map contentdb->package-name inputs))
      (home-page ,home-page)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde..47254539a1 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2022 Alice Brenon <alice.brenon@ens-lyon.fr>
 ;;; Copyright © 2022 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -39,6 +40,8 @@ (define-module (guix import utils)
   #:use-module (guix packages)
   #:use-module (guix discovery)
   #:use-module (guix build-system)
+  #:use-module (guix git)
+  #:use-module (guix hash)
   #:use-module ((guix i18n) #:select (G_))
   #:use-module (guix store)
   #:use-module (guix download)
@@ -63,6 +66,7 @@ (define-module (guix import utils)
 
             url-fetch
             guix-hash-url
+            git->origin
 
             package-names->package-inputs
             maybe-inputs
@@ -161,6 +165,41 @@ (define (guix-hash-url filename)
   "Return the hash of FILENAME in nix-base32 format."
   (bytevector->nix-base32-string (file-sha256 filename)))
 
+(define* (git->origin repo-url ref #:key (ref->commit #f))
+  "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary.  REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
+  (let* ((version (and (pair? ref) (cdr ref)))
+         (directory commit
+                    (if version
+                        (with-store store
+                          (latest-repository-commit store repo-url
+                                                    #:ref (if version ref '())))
+                        (values #f #f)))
+         (vcommit (match ref->commit
+                    (#t    commit)
+                    (#f    version)
+                    ((? procedure?) (ref->commit version))
+                    (_     #f))))
+    (values
+     `(origin
+        (method git-fetch)
+        (uri (git-reference
+              (url ,(and (not (eq? repo-url 'null)) repo-url))
+              (commit ,vcommit)))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          ,(and version  ; Version or commit is not always available.
+                (bytevector->nix-base32-string
+                 (file-hash* directory
+                             ;; 'git-fetch' already filtered out '.git'.
+                             #:select? (const #true)
+                             #:recursive? #true))))))
+     directory)))
+
 (define %spdx-license-identifiers
   ;; https://spdx.org/licenses/
   ;; The gfl1.0, nmap, repoze
diff --git a/tests/go.scm b/tests/go.scm
index d2e8846b30..4644e1bd1c 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
 ;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,7 +25,6 @@ (define-module (tests-import-go)
   #:use-module (guix base32)
   #:use-module (guix build-system go)
   #:use-module (guix import go)
-  #:use-module (guix base32)
   #:use-module ((guix utils) #:select (call-with-temporary-directory))
   #:use-module (guix tests)
   #:use-module (ice-9 match)
@@ -403,13 +403,26 @@ (define (mock-http-get testcase)
             (mock-http-get fixtures-go-check-test))
          (mock ((guix http-client) http-fetch
                 (mock-http-fetch fixtures-go-check-test))
-             (mock ((guix git) update-cached-checkout
-                    (lambda* (url #:key ref)
-                      ;; Return an empty directory and its hash.
-                      (values checkout
-                              (nix-base32-string->bytevector
-                               "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")
-                              #f)))
+             (mock ((guix import utils) git->origin
+                    ;; Mock an empty directory by replacing hash.
+                    (lambda* (repo-url ref #:key (ref->commit #f))
+                      (let* ((version (if (pair? ref)
+                                          (cdr ref)
+                                          #f))
+                             (vcommit (match ref->commit
+                                        (#t    commit)
+                                        (#f    version)
+                                        ((? procedure?) (ref->commit version))
+                                        (_     #f))))
+                        `(origin
+                           (method git-fetch)
+                           (uri (git-reference
+                                 (url ,(and (not (eq? repo-url 'null)) repo-url))
+                                 (commit ,vcommit)))
+                           (file-name (git-file-name name version))
+                           (sha256
+                            (base32
+                             "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"))))))
                  (go-module->guix-package* "github.com/go-check/check")))))))
 
 (test-end "go")
diff --git a/tests/minetest.scm b/tests/minetest.scm
index 78469bf95b..7ff72dbfdc 100644
--- a/tests/minetest.scm
+++ b/tests/minetest.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -57,15 +58,7 @@ (define* (make-package-sexp #:key
   `(package
      (name ,guix-name)
      (version ,version)
-     (source
-      (origin
-        (method git-fetch)
-        (uri (git-reference
-              (url ,(and (not (eq? repo 'null)) repo))
-              (commit #f)))
-        (sha256
-         (base32 #f))
-        (file-name (git-file-name name version))))
+     (source ,(git->origin repo #f))
      (build-system minetest-mod-build-system)
      ,@(maybe-propagated-inputs inputs)
      (home-page ,home-page)
@@ -419,8 +412,8 @@ (define* (example-package #:key
            (uri (git-reference
                  (url repo)
                  (commit commit #;"808f9ffbd3106da4c92d2367b118b98196c9e81e")))
-           (sha256 #f) ; not important for the following tests
-           (file-name (git-file-name name version)))
+           (file-name (git-file-name name version))
+           (sha256 #f)) ; not important for the following tests
          source))
     (build-system minetest-mod-build-system)
     (license #f)
-- 
2.41.0

[bug#62202] [PATCH v4 2/6] import: Add juliahub importer.

[Nicolas Graves via Guix-patches via]

---
 doc/guix.texi                    |  27 +++
 guix/import/juliahub.scm         | 309 +++++++++++++++++++++++++++++++
 guix/scripts/import.scm          |   2 +-
 guix/scripts/import/juliahub.scm | 107 +++++++++++
 4 files changed, 444 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/juliahub.scm
 create mode 100644 guix/scripts/import/juliahub.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index c71d7e94cf..6baa726517 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14782,6 +14782,33 @@ guix import hexpm cf@@0.3.0
 
 Additional options include:
 
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
+@item juliahub
+@cindex juliahub
+Import metadata from both the General
+@uref{https://github.com/JuliaRegistries/General} and Juliahub
+@uref{https://juliahub.com} Julia package repositories, as in this
+example:
+
+@example
+guix import juliahub Cthulhu@@2.8.9
+@end example
+
+The supplied package name must have the same case as in the
+aforementioned package repositories, and the version used must be an
+exact version (e.g. @code{2.8.9} instead of @code{2.8}). The command
+will also fail in the case of a Julia package that doesn't use a git
+tag.
+
+Additional options include:
+
 @table @code
 @item --recursive
 @itemx -r
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..ab838b6035
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,309 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+  #:use-module (ice-9 textual-ports)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
+  #:use-module (srfi srfi-71)
+  #:use-module (guix http-client)
+  #:use-module (guix git)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix base32)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module ((guix licenses) #:prefix license:)
+
+  #:export (juliahub->guix-package
+            %juliahub-updater
+            juliahub-recursive-import))
+
+
+;; JuliaHub API.
+(define (juliahub-redirect-uri name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (port (http-fetch url #:text? #t))
+         (_ (get-line port))
+         (meta (get-line port))
+         (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+         (redirect (match:substring (string-match regex meta))))
+    (close-port port)
+    (string-drop redirect 4)))
+
+(define (juliahub-url name)
+  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+         (uri (juliahub-redirect-uri name)))
+    (string-append url uri "/")))
+
+;; General package repository.
+(define %general-base-url
+  "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+  (let ((folder (string-capitalize (string-take package-name 1))))
+    (string-append
+     %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-line->alist line)
+  (let* ((l (string-split line #\=))
+         (attribute (string->symbol (string-drop-right (car l) 1)))
+         (value (string-drop (string-drop-right (cadr l) 1) 2)))
+    `(,attribute . ,value)))
+
+(define (ini-fetch url)
+  (let* ((port (http-fetch url #:text? #t))
+         (raw (get-string-all port))
+         (lines (drop-right (string-split raw #\newline) 1)))
+    (close-port port)
+    (map ini-line->alist lines)))
+
+;; Filtering out julia-stdlibs.
+;; To update them, see file sysimg.jl.
+(define %julia-stdlibs
+  (list "julia"
+        "ArgTools"
+        "Artifacts"
+        "Base64"
+        "CRC32c"
+        "FileWatching"
+        "Libdl"
+        "Logging"
+        "Mmap"
+        "NetworkOptions"
+        "SHA"
+        "Serialization"
+        "Sockets"
+        "Unicode"
+        "DelimitedFiles"
+        "LinearAlgebra"
+        "Markdown"
+        "Printf"
+        "Random"
+        "Tar"
+        "Dates"
+        "Distributed"
+        "Future"
+        "InteractiveUtils"
+        "LibGit2"
+        "Profile"
+        "SparseArrays"
+        "UUIDs"
+        "REPL"
+        "SharedArrays"
+        "Statistics"
+        "SuiteSparse"
+        "TOML"
+        "Test"
+        "LibCURL"
+        "Downloads"
+        "Pkg"
+        "LazyArtifacts"))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+  json->juliahub-package
+  (homepage juliahub-package-homepage) ;string
+  (readme juliahub-package-readme) ;string
+  (version juliahub-package-version) ;string
+  (description juliahub-package-description) ;string
+  (dependencies
+   juliahub-package-dependencies "deps"
+   json->juliahub-dependencies) ;list of <juliahub-dependency>
+  (url juliahub-package-url) ;string
+  (uuid juliahub-package-uuid) ;string
+  (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+  make-juliahub-dependency juliahub-dependency?
+  json->juliahub-dependency
+  (direct? juliahub-dependency-direct? "direct") ;boolean
+  (name juliahub-dependency-name) ;string
+  (uuid juliahub-dependency-uuid) ;string
+  (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+
+(define (julia-name->guix-name name)
+  (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+  "Return a <juliahub-package> record for package NAME, or #f on failure."
+  (let* ((uri (juliahub-redirect-uri name))
+         (slug (string-take uri 5))
+         (url (if version
+                  (string-append "https://docs.juliahub.com/" name "/"
+                                 slug "/" version "/pkg.json")
+                  (string-append (juliahub-url name) "pkg.json"))))
+    (and=> (json-fetch url) json->juliahub-package)))
+
+(define (make-julia-sexp name version source home-page synopsis description
+                         direct-dependencies test-dependencies-names license)
+  "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSE."
+  `(package
+     (name ,(julia-name->guix-name name))
+     (version ,version)
+     (source ,source)
+     (build-system julia-build-system)
+     ,@(if (null? direct-dependencies)
+           '()
+           `((propagated-inputs
+              (list ,@(map (compose string->symbol
+                                    julia-name->guix-name
+                                    juliahub-dependency-name)
+                           direct-dependencies)))))
+     ,@(if (null? test-dependencies-names)
+           '()
+           `((native-inputs
+              (list ,@(map (compose string->symbol julia-name->guix-name)
+                           test-dependencies-names)))))
+     (synopsis ,synopsis)
+     (description ,description)
+     (home-page ,home-page)
+     (license ,(if license (spdx-string->license license) #f))))
+
+;; Dependencies helpers.
+(define (json->juliahub-dependencies vector)
+  (if (vector? vector)
+      (filter-map
+       (lambda (el)
+         (let ((dep (json->juliahub-dependency el)))
+           (if (and (juliahub-dependency-direct? dep)
+                    (not (member (juliahub-dependency-name dep)
+                                 %julia-stdlibs)))
+               dep
+               #f)))
+       (vector->list vector))))
+
+(define (parse-test-dependencies directory)
+  (let* ((port (open-input-file (string-append directory "/Project.toml")))
+         (project.toml (get-string-all port))
+         (regex "\ntest = \\[.*\\]")
+         (deps (match:substring (string-match regex project.toml)))
+         (pure (string-delete (list->char-set (list #\" #\ )) deps)))
+    (close-port port)
+    (filter (lambda (x) (not (member x %julia-stdlibs)))
+            (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-tag, and import
+;; this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag. In this case, the script fails, but it seems quite rare. We could
+;; introduce the tree-commit which is available in the Versions.toml file in the
+;; General repository. This can be used to identify the state of a repository,
+;; since we have a unique hash of the listing of files and directories.
+
+(define (latest-git-tag repo)
+  (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+         (last-git-tag (last (string-split last-ref #\/))))
+    (string-drop last-git-tag 1)))
+
+(define* (juliahub->guix-package package-name
+                                 #:key version #:allow-other-keys)
+  "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+  (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
+         (subdir (assoc-ref package-toml 'subdir))
+         (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+         (package (if version
+                      (juliahub-fetch package-name #:version version)
+                      (if tag
+                          (juliahub-fetch package-name #:version tag)
+                          (juliahub-fetch package-name)))))
+    (if package
+        (let* ((source directory
+                       (git->origin
+                        (juliahub-package-url package)
+                        `(tag-or-commit
+                          . ,(string-append
+                              "v" (juliahub-package-version package)))))
+               (direct-dependencies
+                (filter juliahub-dependency-direct?
+                        (juliahub-package-dependencies package)))
+               (dependencies-names (map juliahub-dependency-name
+                                        direct-dependencies))
+               (test-dependencies-names
+                (if subdir
+                    (parse-test-dependencies
+                     (string-append subdir "/" directory))
+                    (parse-test-dependencies directory)))
+               (homepage (juliahub-package-homepage package)))
+          (values (make-julia-sexp
+                   package-name
+                   (juliahub-package-version package)
+                   source
+                   (match homepage
+                     ("" (juliahub-package-url package))
+                     ((? string?) homepage)
+                     (_ (juliahub-package-url package)))
+                   (juliahub-package-description package)
+                   (beautify-description
+                    (juliahub-package-readme package))
+                   direct-dependencies
+                   test-dependencies-names
+                   (juliahub-package-license package))
+                  (append dependencies-names test-dependencies-names)))
+    (values #f '()))))
+
+;; We must use the url to get a name with the true case of juliahub/general.
+(define (guix-package->juliahub-name package)
+  (let* ((url (juliahub-package-url package))
+         (git-name (last (string-split url #\/)))
+         (ungitted-name (if (string-suffix? ".git" git-name)
+                            (string-drop-right git-name 4)
+                            git-name))
+         (package-name (if (string-suffix? ".jl" ungitted-name)
+                           (string-drop-right ungitted-name 4)
+                           ungitted-name)))
+    package-name))
+
+(define* (import-release package #:key (version #f))
+  "Return an <upstream-source> for the latest release of PACKAGE."
+  (let* ((package-name (guix-package->juliahub-name package))
+         (package      (juliahub-fetch package-name))
+         (version  (or version (juliahub-package-version package))))
+    (upstream-source
+     (package (package-name package))
+     (version version)
+     (urls (list (juliahub-package-url package))))))
+
+(define %juliahub-updater
+  (upstream-updater
+   (name 'juliahub)
+   (description "Updater for Juliahub packages")
+   (pred juliahub-package?)
+   (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+  (recursive-import package-name
+                    #:repo '()
+                    #:repo->guix-package juliahub->guix-package
+                    #:guix-name julia-name->guix-name
+                    #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index d2a1cee56e..8926c9610f 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
 
 (define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
                     "gem" "go" "cran" "crate" "texlive" "json" "opam"
-                    "minetest" "elm" "hexpm" "composer"))
+                    "minetest" "elm" "hexpm" "composer" "juliahub"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import juliahub)
+  #:use-module (guix scripts import)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 receive)
+  #:export (guix-import-juliahub))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME.  Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (display (G_ "
+  -r, --recursive        generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import gem")))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
+         %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (parse-command-line args %options (list %default-options)
+                        #:build-options? #f))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                             (('argument . value)
+                              value)
+                             (_ #f))
+                           (reverse opts))))
+    (match args
+      ((spec)
+       (receive (package-name package-version)
+           (package-name->name+version spec)
+         (let ((code (if (assoc-ref opts 'recursive)
+                         (map (match-lambda
+                                ((and ('package ('name name) . rest) pkg)
+                                 `(define-public ,(string->symbol name)
+                                    ,pkg))
+                                (_ #f))
+                              (juliahub-recursive-import package-name package-version))
+                         (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+                           (if sexp sexp #f)))))
+           (match code
+             ((or #f '(#f))
+              (leave (G_ "failed to download meta-data for package '~a'~%")
+                     package-name))
+             (_ code)))))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.41.0

[bug#62202] [PATCH v4 3/6] import: juliahub: Beautify description.

[Nicolas Graves via Guix-patches via]

---
 guix/import/juliahub.scm | 53 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 51 insertions(+), 2 deletions(-)

diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index ab838b6035..e4540de06d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -30,8 +30,10 @@ (define-module (guix import juliahub)
   #:use-module (guix base32)
   #:use-module (guix packages)
   #:use-module (guix upstream)
-  #:use-module (json)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (json)
+  #:use-module (htmlprag)
+  #:use-module (sxml transform)
 
   #:export (juliahub->guix-package
             %juliahub-updater
@@ -76,6 +78,53 @@ (define (ini-fetch url)
     (close-port port)
     (map ini-line->alist lines)))
 
+;; Beautify description.
+(define %juliahub-beautify-description-rules
+  `((h1 *preorder*   . ,(lambda args #f))
+    (h2 *preorder*   . ,(lambda args #f))
+    (h3 *preorder*   . ,(lambda args #f))
+    (h4 *preorder*   . ,(lambda args #f))
+    (h5 *preorder*   . ,(lambda args #f))
+    (h6 *preorder*   . ,(lambda args #f))
+    (hr *preorder*   . ,(lambda args #f))
+    (span *preorder* . ,(lambda args #f))
+    (img *preorder*  . ,(lambda args #f))
+    (pre *preorder*  . ,(lambda args #f))
+    (div *preorder*  . ,(lambda args #f))
+    (table *preorder* . ,(lambda args #f))
+    (imgalt *preorder* . ,(lambda args #f))
+    (@ *preorder* . ,(lambda args #f))
+    (*TOP*        . ,(lambda args (cdr args)))
+    (p            . ,(lambda args (cdr args)))
+    (em           . ,(lambda args (cdr args)))
+    (strong       . ,(lambda args (cdr args)))
+    (a            . ,(lambda args
+                       (match args
+                         ((tag link ref)
+                          (if ref ref #f))
+                         (_ #f))))
+    (ul           . ,(lambda args
+                       `("@itemize" ,@(cdr args) "\n@end itemize")))
+    (ol           . ,(lambda args
+                       `("@enumerate" ,@(cdr args) "@end enumerate")))
+    (blockquote   . ,(lambda args
+                       `("@quotation" ,@(cdr args) "@end quotation")))
+    (li           . ,(lambda args
+                       `("\n@item" ,@(cdr args))))
+    (code         . ,(lambda args
+                       `("@code{" ,@(cdr args) "}")))
+    (*text*       . ,(lambda (tag x) x))
+    (*default*    . ,(lambda (tag . body)
+                       (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+  (string-join
+   (filter (lambda (x) (if (equal? x " ") #f x))
+           (flatten
+            (pre-post-order (html->sxml description)
+                            %juliahub-beautify-description-rules)))
+   " "))
+
 ;; Filtering out julia-stdlibs.
 ;; To update them, see file sysimg.jl.
 (define %julia-stdlibs
@@ -264,7 +313,7 @@ (define* (juliahub->guix-package package-name
                      ((? string?) homepage)
                      (_ (juliahub-package-url package)))
                    (juliahub-package-description package)
-                   (beautify-description
+                   ((compose beautify-description juliahub-beautify-description)
                     (juliahub-package-readme package))
                    direct-dependencies
                    test-dependencies-names
-- 
2.41.0

[bug#62202] [PATCH v4 4/6] import: utils: Rule out texinfo common syntax from @ escape.

[Nicolas Graves via Guix-patches via]

---
 guix/import/utils.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 47254539a1..57e4ec0ce7 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -375,7 +375,13 @@ (define* (beautify-description description #:optional (length 80))
                    (cut string-trim-both <> #\')
                    ;; Escape single @ to prevent it from being understood as
                    ;; invalid Texinfo syntax.
-                   (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+                   (lambda (word)
+                     (if  ; Rule out some valid Texinfo syntax.
+                      (member word '("@itemize" "@item" "@end" "@quotation"
+                                     "@enumerate" "@code" "@code{"))
+                      word
+                      ((cut regexp-substitute/global
+                            #f "@" <> 'pre "@@" 'post) word)))
                    ;; Wrap camelCase or PascalCase words in @code{...}.
                    (lambda (word)
                      (let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
-- 
2.41.0

[bug#62202] [PATCH v4 5/6] tests: go: Add mock-git->origin function.

[Nicolas Graves via Guix-patches via]

* tests/go.scm
(mock-git->origin): Add function.
(go-module->guix-package test): Use mock-git->origin.

Change-Id: I9f9de814fc6c9dcf9772507c9e5bf32c16d81784
---
 tests/go.scm | 41 +++++++++++++++++++++--------------------
 1 file changed, 21 insertions(+), 20 deletions(-)

diff --git a/tests/go.scm b/tests/go.scm
index 4644e1bd1c..cca27506e6 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -371,6 +371,26 @@ (define (mock-http-get testcase)
           (values response-header body)
           (error "mocked http-get Unexpected URL: " url)))))
 
+;; Mock an empty directory by replacing hash.
+(define* (mock-git->origin repo-url ref #:key (ref->commit #f))
+  (let* ((version (if (pair? ref)
+                      (cdr ref)
+                      #f))
+         (vcommit (match ref->commit
+                    (#t    commit)
+                    (#f    version)
+                    ((? procedure?) (ref->commit version))
+                    (_     #f))))
+    `(origin
+       (method git-fetch)
+       (uri (git-reference
+             (url ,(and (not (eq? repo-url 'null)) repo-url))
+             (commit ,vcommit)))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32
+         "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))))
+
 (test-equal "go-module->guix-package"
   '(package
      (name "go-github-com-go-check-check")
@@ -403,26 +423,7 @@ (define (mock-http-get testcase)
             (mock-http-get fixtures-go-check-test))
          (mock ((guix http-client) http-fetch
                 (mock-http-fetch fixtures-go-check-test))
-             (mock ((guix import utils) git->origin
-                    ;; Mock an empty directory by replacing hash.
-                    (lambda* (repo-url ref #:key (ref->commit #f))
-                      (let* ((version (if (pair? ref)
-                                          (cdr ref)
-                                          #f))
-                             (vcommit (match ref->commit
-                                        (#t    commit)
-                                        (#f    version)
-                                        ((? procedure?) (ref->commit version))
-                                        (_     #f))))
-                        `(origin
-                           (method git-fetch)
-                           (uri (git-reference
-                                 (url ,(and (not (eq? repo-url 'null)) repo-url))
-                                 (commit ,vcommit)))
-                           (file-name (git-file-name name version))
-                           (sha256
-                            (base32
-                             "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"))))))
+             (mock ((guix import utils) git->origin mock-git->origin)
                  (go-module->guix-package* "github.com/go-check/check")))))))
 
 (test-end "go")
-- 
2.41.0

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

* tests/juliahub.scm : Add unit tests juliahub-redirect,
julia-general-registry-parsing, juliahub-fetch.

Change-Id: Ief5133b49a15d0bfc72bb357321126d296ba2802
---
 tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)
 create mode 100644 tests/juliahub.scm

diff --git a/tests/juliahub.scm b/tests/juliahub.scm
new file mode 100644
index 0000000000..21fdc3a4eb
--- /dev/null
+++ b/tests/juliahub.scm
@@ -0,0 +1,185 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Summary
+;; Tests for guix/import/juliahub.scm
+
+(define-module (tests-import-juliahub)
+  #:use-module (guix base32)
+  #:use-module (json)
+  #:use-module (guix import juliahub)
+  #:use-module (guix import utils)
+  #:use-module ((guix utils) #:select (call-with-temporary-directory))
+  #:use-module (guix tests)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-19)
+  #:use-module (srfi srfi-64)
+  #:use-module (srfi srfi-71)
+  #:use-module (web response))
+
+;; XXX: Copied from tests/go.scm
+
+(define (mock-http-fetch testcase)
+  (lambda (url . rest)
+    (let ((body (assoc-ref testcase url)))
+      (if body
+          (open-input-string body)
+          (error "mocked http-fetch Unexpected URL: " url)))))
+
+(define (mock-http-get testcase)
+  (lambda (url . rest)
+    (let ((body (assoc-ref testcase url))
+          (response-header
+             (build-response
+                #:version '(1 . 1)
+                #:code 200
+                #:reason-phrase "Ok"
+                #:headers `(
+                            (content-type text/html (charset . "utf-8"))
+                            (date . ,(make-date 0 10 58 12 6 3 2021 0))
+                            (transfer-encoding (chunked)))
+                #:port #f
+                #:validate-headers? #t)))
+      (if body
+          (values response-header body)
+          (error "mocked http-get Unexpected URL: " url)))))
+
+;; Mock an empty directory by replacing hash.
+(define* (mock-git->origin repo-url ref #:key (ref->commit #f))
+  (let* ((version (if (pair? ref)
+                      (cdr ref)
+                      #f))
+         (vcommit (match ref->commit
+                    (#t    commit)
+                    (#f    version)
+                    ((? procedure?) (ref->commit version))
+                    (_     #f))))
+    `(origin
+       (method git-fetch)
+       (uri (git-reference
+             (url ,(and (not (eq? repo-url 'null)) repo-url))
+             (commit ,vcommit)))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32
+         "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))))
+
+;; Fixtures.
+
+(define fixture-pkg.json
+  (scm->json-string
+   `(("hosted_uri" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+     ("installable" . "missing")
+     ("license" . "GPL-3.0")
+     ("tags" . #("my-tag"))
+     ("uuid" . "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
+     ("url" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+     ("contributors" . #((("url" . "https://github.com/me") ("type" . "User") ("contributions" . 1) ("name" . "me"))))
+     ("success" . #t)
+     ("reversedeps" . #())
+     ("pkgeval"
+      ("report_url" . "https://github.com/JuliaCI/NanosoldierReports/blob/master/pkgeval/by_date/2024-02/01/report.md")
+      ("version" ("minor" . 18) ("patch" . 1) ("build" . #()) ("prerelease" . #()) ("major" . 1))
+      ("reason" . "time_limit")
+      ("log_url" . "https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2024-02/01/MyPackage.jl.primary.log")
+      ("status" . "fail")
+      ("duration" . 0))
+     ("stargazers_count" . 0)
+     ("deps" . #((("slug" . "3FQLY")
+                  ("registry" . "")
+                  ("versions" . #("0.0.0" "1.6.0-1"))
+                  ("uuid" . "de0858da-6303-5e67-8744-51eddeeeb8d7")
+                  ("name" . "Printf")
+                  ("direct" . #t))
+                 (("slug" . "SIw1t")
+                  ("registry" . "")
+                  ("versions" . #("*" "0.0.0" "1"))
+                  ("uuid" . "cf7118a7-6976-5b1a-9a39-7adc72f591a4")
+                  ("name" . "UUIDs")
+                  ("direct" . #f))
+                 (("slug" . "THl1k")
+                  ("registry" . "General")
+                  ("versions" . #("0.7" "1" "1-1.10" "1.0-1.5" "1.3.0-1" "1.6.0-1" "1.9.0-1" "1.9.4-1"))
+                  ("uuid" . "1222c4b2-2114-5bfd-aeef-88e4692bbb3e")
+                  ("name" . "julia")
+                  ("direct" . #f))
+                 (("slug" . "THl1k")
+                  ("registry" . "General")
+                  ("versions" . #("1.6.0-1"))
+                  ("uuid" . "1222c4b2-2114-5bfd-aeef-88e4692bbb3e")
+                  ("name" . "julia")
+                  ("direct" . #t))))
+     ("description" . "My description")
+     ("version" . "1.0.0")
+     ("documenter_errored" . "missing")
+     ("slug" . "MySlg")
+     ("owner" . "me")
+     ("release_date" . "Feb 2024")
+     ("name" . "MyPackage.jl")
+     ("readme" . "My readme")
+     ("homepage" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+     ("doctype" . "hosted")
+     ("license_url" . "/docs/MyPackage/MySlg/1.0.0/_packagesource/LICENSE.md"))))
+
+(define fixtures-juliahub-check-test
+  `(("https://docs.juliahub.com/MyPackage/" . "\
+<head>
+<meta http-equiv=\"refresh\" content=\"0; url=MySlg/1.0.0\" />
+</head>")
+    ("https://raw.githubusercontent.com/JuliaRegistries/General/master\
+/M/MyPackage/Package.toml" . "\
+name = \"MyPackage\"
+uuid = \"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa\"
+repo = \"https://git.savannah.gnu.org/cgit/MyPackage.git\"
+")
+    ("https://docs.juliahub.com/MyPackage/MySlg/1.0.0/pkg.json" . ,fixture-pkg.json)))
+
+(test-begin "juliahub")
+
+;;; Unit tests for (guix import juliahub)
+
+(test-equal "juliahub-redirect"
+  "https://docs.juliahub.com/MyPackage/MySlg/1.0.0/"
+  (mock ((web client) http-get
+         (mock-http-get fixtures-juliahub-check-test))
+        (mock ((guix http-client) http-fetch
+               (mock-http-fetch fixtures-juliahub-check-test))
+              ((@@ (guix import juliahub) juliahub-url) "MyPackage"))))
+
+(test-equal "julia-general-registry-parsing"
+  "https://git.savannah.gnu.org/cgit/MyPackage.git"
+  (mock ((web client) http-get
+         (mock-http-get fixtures-juliahub-check-test))
+        (mock ((guix http-client) http-fetch
+               (mock-http-fetch fixtures-juliahub-check-test))
+              (assoc-ref
+               ((@@ (guix import juliahub) ini-fetch)
+                ((@@ (guix import juliahub) general-url) "MyPackage" "Package.toml"))
+               'repo))))
+
+(test-equal "juliahub-fetch"
+  #t
+  (mock ((web client) http-get
+         (mock-http-get fixtures-juliahub-check-test))
+        (mock ((guix http-client) http-fetch
+               (mock-http-fetch fixtures-juliahub-check-test))
+              (mock ((guix import utils) git->origin mock-git->origin)
+                    ((@@ (guix import juliahub) juliahub-package?)
+                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
+
+(test-end "juliahub")
-- 
2.41.0

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Ludovic Courtès]

Hi,

As part of this v4, I would recommend merging patches 2, 3, and 6, such
that there’s a single self-contained patch adding ‘guix import
juliahub’.  (That’s what we usually do and I find it clearer because we
immediately see what goes together.)

Nicolas Graves <ngraves@ngraves.fr> skribis:

> * tests/juliahub.scm : Add unit tests juliahub-redirect,
> julia-general-registry-parsing, juliahub-fetch.

Just “New file.”

Some of the other files lack a commit log; we can add it for you, but
it’d be great if you could do it upfront.

> ---
>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++

Please add it to ‘Makefile.am’.

[...]

> +(define (mock-http-fetch testcase)
> +  (lambda (url . rest)
> +    (let ((body (assoc-ref testcase url)))
> +      (if body
> +          (open-input-string body)
> +          (error "mocked http-fetch Unexpected URL: " url)))))
> +
> +(define (mock-http-get testcase)
> +  (lambda (url . rest)
> +    (let ((body (assoc-ref testcase url))
> +          (response-header
> +             (build-response
> +                #:version '(1 . 1)

I strongly encourage using ‘with-http-server’ using the same strategy
that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
is very sensitive to inlining, plus you sorta have to make assumptions
about the code path to be able to mock the right things.)

> +(test-equal "juliahub-fetch"
> +  #t
> +  (mock ((web client) http-get
> +         (mock-http-get fixtures-juliahub-check-test))
> +        (mock ((guix http-client) http-fetch
> +               (mock-http-fetch fixtures-juliahub-check-test))
> +              (mock ((guix import utils) git->origin mock-git->origin)
> +                    ((@@ (guix import juliahub) juliahub-package?)
> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))

Checking for ‘juliahub-package?’ doesn’t tell us much; what about
checking the whole package, similar to what is done in other importer
tests?

Thanks,
Ludo’.

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

On 2024-04-01 22:50, Ludovic Courtès wrote:

> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’.  (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)

Will do.
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.

Sorry for these past contribution, I do it now.
>
>> ---
>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.

Will do.
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url)))
>> +      (if body
>> +          (open-input-string body)
>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url))
>> +          (response-header
>> +             (build-response
>> +                #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)
>
>> +(test-equal "juliahub-fetch"
>> +  #t
>> +  (mock ((web client) http-get
>> +         (mock-http-get fixtures-juliahub-check-test))
>> +        (mock ((guix http-client) http-fetch
>> +               (mock-http-fetch fixtures-juliahub-check-test))
>> +              (mock ((guix import utils) git->origin mock-git->origin)
>> +                    ((@@ (guix import juliahub) juliahub-package?)
>> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?

Couldn't manage to get it to work, don't remember why exactly but it was
related to gexps. Will retry it when moving to 'with-http-server' as
advised. 

>
> Thanks,
> Ludo’.

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

On 2024-04-01 22:50, Ludovic Courtès wrote:

> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’.  (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.
>
>> ---
>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url)))
>> +      (if body
>> +          (open-input-string body)
>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> +  (lambda (url . rest)
>> +    (let ((body (assoc-ref testcase url))
>> +          (response-header
>> +             (build-response
>> +                #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)

I can't however mock a git server, right? I still must mock at least the
git repo instead of getting it through a custom server, or is there a
better solution here?

>
>> +(test-equal "juliahub-fetch"
>> +  #t
>> +  (mock ((web client) http-get
>> +         (mock-http-get fixtures-juliahub-check-test))
>> +        (mock ((guix http-client) http-fetch
>> +               (mock-http-fetch fixtures-juliahub-check-test))
>> +              (mock ((guix import utils) git->origin mock-git->origin)
>> +                    ((@@ (guix import juliahub) juliahub-package?)
>> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?
>
> Thanks,
> Ludo’.

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

On 2024-04-09 09:29, Nicolas Graves wrote:

> On 2024-04-01 22:50, Ludovic Courtès wrote:
>
>> Hi,
>>
>> As part of this v4, I would recommend merging patches 2, 3, and 6, such
>> that there’s a single self-contained patch adding ‘guix import
>> juliahub’.  (That’s what we usually do and I find it clearer because we
>> immediately see what goes together.)
>>
>> Nicolas Graves <ngraves@ngraves.fr> skribis:
>>
>>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>>> julia-general-registry-parsing, juliahub-fetch.
>>
>> Just “New file.”
>>
>> Some of the other files lack a commit log; we can add it for you, but
>> it’d be great if you could do it upfront.
>>
>>> ---
>>>  tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>>
>> Please add it to ‘Makefile.am’.
>>
>> [...]
>>
>>> +(define (mock-http-fetch testcase)
>>> +  (lambda (url . rest)
>>> +    (let ((body (assoc-ref testcase url)))
>>> +      (if body
>>> +          (open-input-string body)
>>> +          (error "mocked http-fetch Unexpected URL: " url)))))
>>> +
>>> +(define (mock-http-get testcase)
>>> +  (lambda (url . rest)
>>> +    (let ((body (assoc-ref testcase url))
>>> +          (response-header
>>> +             (build-response
>>> +                #:version '(1 . 1)
>>
>> I strongly encourage using ‘with-http-server’ using the same strategy
>> that’s used in ‘tests/pypi.scm’ and others instead of mocking.  (‘mock’
>> is very sensitive to inlining, plus you sorta have to make assumptions
>> about the code path to be able to mock the right things.)
>
> I can't however mock a git server, right? I still must mock at least the
> git repo instead of getting it through a custom server, or is there a
> better solution here?

It's actually simpler than I thought, but there's an impediment in guile
http server implementation that doesn't allow me to push this effort to
the end.

https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

I'm currently writing it, it'll result in a handy helper for tests, such
as :

(with-git-forge  ; spawns a dumb but functional git server
      '(("MyPackage" . ((add "a.txt" "A")
                        (commit "First commit")
                        (tag "v1.0.0" "Release 1.0"))))
    (with-julia-test-servers
        `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
          ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
          ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
           ,(lambda (port) (display (fixture-pkg.json) port)))
          ("/general/M/MyPackage/Package.toml" 200
           ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
      (juliahub->guix-package "MyPackage")))


However, for that I'll need the http server to be able to respond with a
         (content-type . (application/x-git-upload-pack-advertisement))
header to git. But in guile's web server implementation, this is not
possible because of sanitize-response's charset addition, which is not
configurable. 

That's outside my field, how can we progress further ? We do indeed need
such a server to properly test juliahub since we go get the tag from the
actual repo (this is justified in the patch series).

_____________________________________________________________________________
;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories

(define (call-with-temporary-git-repositories names+directives proc)
  "Call PROC with populated git temporary directories as per NAMES+DIRECTIVES;
close the directories and delete them when leaving the dynamic extent of this
call."
  (call-with-temporary-directory
   (lambda (directory)
     (for-each
      (match-lambda
        ((name . directives)
         (populate-git-repository
          (string-append directory "/" name ".git") directives)))
      names+directives)
     (proc directory))))

(define %git-forge-port
  ;; TCP port to use for the dumb git server.
  ;; If 0, the OS will automatically choose
  ;; a port.
  (make-parameter 0))

(define (binary-file-dump file)
  "Return a procedure that dumps binary FILE to the given port."
  (lambda (output)
    (call-with-input-file file
      (lambda (input)
        (put-bytevector output (get-bytevector-all input)))
      #:binary #t)))

(define (serialize-git-ref ref oid)
  (format #f "~a     ~a\n" oid ref))

(define (refs->alist repo refs)
  (let ((repository (repository-open repo)))
    (map
     (lambda (ref)
       (cons ref (oid->string (reference-name->oid repository ref))))
     refs)))

(define* (call-with-git-forge repositories+directives thunk)
  "Call THUNK with a running GIT test forge, i.e. an HTTP server implementing
the git dumb protocol (see
https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols) running.
This server behaves like a GIT forge with the repositories constructed from
REPOSITORIES+DIRECTIVES.  Each element of REPOSITORIES+DIRECTIVES must be a
tuple containing a repository name and a list of DIRECTIVES.

%git-forge-port will be set to the port listened at
The port listened at will be set for the dynamic extent of THUNK."
  (call-with-temporary-git-repositories
   repositories+directives
   (lambda (dir-with-repos)
     (define responses+data
       (let ((repos (scandir dir-with-repos
                             (lambda (name)
                               (not (member name '("." "..")))))))
         (append-map
          (lambda (relative-repo)
            (let* ((name (string-drop-right relative-repo (string-length ".git")))
                   (repo (string-append dir-with-repos "/" relative-repo)))
              `((,(string-append "/" name ".git/info/refs")
                 200
                 ((content-type . (application/x-git-upload-pack-advertisement)))
                 ,((@ (gnu services configuration) generic-serialize-alist)
                   string-append
                   serialize-git-ref
                   (refs->alist repo (remote-refs repo))))
                (,(string-append "/" name ".git/HEAD")
                 200
                 "ref: refs/heads/master")
                ,@(map
                   (lambda (object)
                     `(,(string-append "/" name ".git/objects/"
                                       (string-take-right object 41))
                       200
                       ,(binary-file-dump
                         (string-append repo "/.git/objects/" object))))
                   (find-files (string-append repo "/.git/objects")))
                (,(string-append "/" name ".git/objects/info/http-alternates")
                 200
                 "")
                (,(string-append "/" name ".git/objects/info/packs")
                 200 ""))))
          repos)))

     (parameterize ((%http-server-port (%git-forge-port)))
       (call-with-http-server (pk 'responses+data responses+data) thunk)))))

(define-syntax with-git-forge
  (syntax-rules ()
    ((_ repositories+directives body ...)
     (call-with-git-forge repositories+directives (lambda () body ...)))))
__________________________________________________________________________________

>>
>>> +(test-equal "juliahub-fetch"
>>> +  #t
>>> +  (mock ((web client) http-get
>>> +         (mock-http-get fixtures-juliahub-check-test))
>>> +        (mock ((guix http-client) http-fetch
>>> +               (mock-http-fetch fixtures-juliahub-check-test))
>>> +              (mock ((guix import utils) git->origin mock-git->origin)
>>> +                    ((@@ (guix import juliahub) juliahub-package?)
>>> +                     ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>>
>> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
>> checking the whole package, similar to what is done in other importer
>> tests?
>>
>> Thanks,
>> Ludo’.

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Ludovic Courtès]

Hi,

Nicolas Graves <ngraves@ngraves.fr> skribis:

> I'm currently writing it, it'll result in a handy helper for tests, such
> as :
>
> (with-git-forge  ; spawns a dumb but functional git server
>       '(("MyPackage" . ((add "a.txt" "A")
>                         (commit "First commit")
>                         (tag "v1.0.0" "Release 1.0"))))
>     (with-julia-test-servers
>         `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>           ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>           ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
>            ,(lambda (port) (display (fixture-pkg.json) port)))
>           ("/general/M/MyPackage/Package.toml" 200
>            ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
>       (juliahub->guix-package "MyPackage")))

Nice!

> However, for that I'll need the http server to be able to respond with a
>          (content-type . (application/x-git-upload-pack-advertisement))
> header to git. But in guile's web server implementation, this is not
> possible because of sanitize-response's charset addition, which is not
> configurable. 

D’oh.  I’m not sure what this is about; it’s definitely possible to add
pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
instance), but if it’s going too far, then maybe you should go back to
procedure mocks.

> That's outside my field, how can we progress further ? We do indeed need
> such a server to properly test juliahub since we go get the tag from the
> actual repo (this is justified in the patch series).
>
> _____________________________________________________________________________
> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories

This work’s not lost: we can definitely switch back to it when the
limitations you encountered has been fixed on the Guile side, and/or in
other, simpler cases.

Thanks!

Ludo’.

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

On 2024-04-16 18:52, Ludovic Courtès wrote:

> Hi,
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> I'm currently writing it, it'll result in a handy helper for tests, such
>> as :
>>
>> (with-git-forge  ; spawns a dumb but functional git server
>>       '(("MyPackage" . ((add "a.txt" "A")
>>                         (commit "First commit")
>>                         (tag "v1.0.0" "Release 1.0"))))
>>     (with-julia-test-servers
>>         `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>>           ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>>           ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
>>            ,(lambda (port) (display (fixture-pkg.json) port)))
>>           ("/general/M/MyPackage/Package.toml" 200
>>            ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
>>       (juliahub->guix-package "MyPackage")))
>
> Nice!
>
>> However, for that I'll need the http server to be able to respond with a
>>          (content-type . (application/x-git-upload-pack-advertisement))
>> header to git. But in guile's web server implementation, this is not
>> possible because of sanitize-response's charset addition, which is not
>> configurable. 
>
> D’oh.  I’m not sure what this is about; it’s definitely possible to add
> pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
> instance), but if it’s going too far, then maybe you should go back to
> procedure mocks.

Just to write that in a clearer way:

libgit2, which is behind guile-git, expects this exact header or fails,
with no additional charset field. The sanitize-response function in the
guile web server implementation on its side ensures that a charset field
is added no matter what.

So indeed we can set

   (content-type . (application/x-git-upload-pack-advertisement))
    
but in reality guile-git // libgit2 will read

   (content-type . (application/x-git-upload-pack-advertisement
                     (charset . "utf-8"))
                     
and will fail in this case. 

Since it's exterior (expected by libgit2), I thought the more relevant
would be to allow more flexibility on the guile web server side. But if
you think it's way better to ask on the libgit2 or guile-git side, I'll
be happy to.  WDYT ?

>
>> That's outside my field, how can we progress further ? We do indeed need
>> such a server to properly test juliahub since we go get the tag from the
>> actual repo (this is justified in the patch series).
>>
>> _____________________________________________________________________________
>> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories
>
> This work’s not lost: we can definitely switch back to it when the
> limitations you encountered has been fixed on the Guile side, and/or in
> other, simpler cases.
>
> Thanks!
>
> Ludo’.

-- 
Best regards,
Nicolas Graves

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Ludovic Courtès]

Nicolas Graves <ngraves@ngraves.fr> skribis:

> libgit2, which is behind guile-git, expects this exact header or fails,
> with no additional charset field. The sanitize-response function in the
> guile web server implementation on its side ensures that a charset field
> is added no matter what.
>
> So indeed we can set
>
>    (content-type . (application/x-git-upload-pack-advertisement))
>     
> but in reality guile-git // libgit2 will read
>
>    (content-type . (application/x-git-upload-pack-advertisement
>                      (charset . "utf-8"))
>                      
> and will fail in this case. 

Oh I see, I had misunderstood that.

Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
is #f or a bytevector.  Maybe we could do that?

Thanks,
Ludo’.

[bug#62202] [PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).

[Nicolas Graves via Guix-patches via]

On 2024-04-17 10:51, Ludovic Courtès wrote:

> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> libgit2, which is behind guile-git, expects this exact header or fails,
>> with no additional charset field. The sanitize-response function in the
>> guile web server implementation on its side ensures that a charset field
>> is added no matter what.
>>
>> So indeed we can set
>>
>>    (content-type . (application/x-git-upload-pack-advertisement))
>>     
>> but in reality guile-git // libgit2 will read
>>
>>    (content-type . (application/x-git-upload-pack-advertisement
>>                      (charset . "utf-8"))
>>                      
>> and will fail in this case. 
>
> Oh I see, I had misunderstood that.
>
> Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
> is #f or a bytevector.  Maybe we could do that?

I've just taken a look at your suggestion. That's possible although not
elegant.

I stumbled accross a bigger issue however: the dumb protocol is
implemented for git but not for libgit2, thus is not present in
guile-git.

https://github.com/libgit2/libgit2/issues/6609

The smart protocol is however way too complex / unadapted for such a
light use in tests, and requires more that a simple http server easily
configured.

That's sad, such a light implementation would've been quite convenient
in my juliahub case. 

>
> Thanks,
> Ludo’.
>
>
>

-- 
Best regards,
Nicolas Graves