* [PATCH 0/1] Go importer
@ 2018-04-26 16:22 Rouby Pierre-Antoine
2018-04-27 7:45 ` [PATCH 1/1] import: Add gopkg importer Rouby Pierre-Antoine
2018-05-02 20:04 ` [PATCH 0/1] Go importer Leo Famulari
0 siblings, 2 replies; 11+ messages in thread
From: Rouby Pierre-Antoine @ 2018-04-26 16:22 UTC (permalink / raw)
To: guix-devel
Hi Guix !
This patch is a importer for go packages, it's use git repository and the
'Gopkg.toml' file (https://golang.github.io/dep/docs/Gopkg.toml.html).
Example with totally random package in Go language ;)
---Start-----------------------------------------------------------------------
$ guix import gopkg \
https://gitlab.com/ContinuousEvolution/continuous-evolution.git \
0.8.1
(define-public go-github-com-fsouza-go-dockerclient
(let ((commit "master") (revision "0"))
(package
(name "go-github-com-fsouza-go-dockerclient")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/fsouza/go-dockerclient.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"0viwysdg5i9dgdazks9wx0f93180j72x3cq9fx2synaqvqa7ldr4"))))
(build-system go-build-system)
(arguments
'(#:import-path
"github.com/fsouza/go-dockerclient"))
(native-inputs `())
(home-page
"https://github.com/fsouza/go-dockerclient")
(synopsis "XXX")
(description "XXX")
(license #f))))
(define-public go-github-com-sirupsen-logrus
(let ((commit "v1.0.4") (revision "0"))
(package
(name "go-github-com-sirupsen-logrus")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/sirupsen/logrus.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"0s8s8wvshmh7cb2f4fqnibqrpxahbaydyvskn3xsrl7z2a5wwajz"))))
(build-system go-build-system)
(arguments
'(#:import-path "github.com/sirupsen/logrus"))
(native-inputs `())
(home-page "https://github.com/sirupsen/logrus")
(synopsis "XXX")
(description "XXX")
(license #f))))
(define-public go-github-com-heroku-docker-registry-client
(let ((commit "master") (revision "0"))
(package
(name "go-github-com-heroku-docker-registry-client")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/heroku/docker-registry-client.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"0db2wdgizhg71hn3f3zvan3knb8yglr0njd3mh1jn70ab7277hll"))))
(build-system go-build-system)
(arguments
'(#:import-path
"github.com/heroku/docker-registry-client"))
(native-inputs `())
(home-page
"https://github.com/heroku/docker-registry-client")
(synopsis "XXX")
(description "XXX")
(license #f))))
(define-public go-github-com-blang-semver
(let ((commit "3.5.1") (revision "0"))
(package
(name "go-github-com-blang-semver")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/blang/semver.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"19pli07y5592g4dyjyj0jq5rn548vc3fz0qg3624vm1j5828p1c2"))))
(build-system go-build-system)
(arguments
'(#:import-path "github.com/blang/semver"))
(native-inputs `())
(home-page "https://github.com/blang/semver")
(synopsis "XXX")
(description "XXX")
(license #f))))
(define-public go-github-com-pelletier-go-toml
(let ((commit "1.0.1") (revision "0"))
(package
(name "go-github-com-pelletier-go-toml")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/pelletier/go-toml.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"1n8na0yg90gm0rpifmzrby5r385vvd62cdam3ls7ssy02bjvfw15"))))
(build-system go-build-system)
(arguments
'(#:import-path "github.com/pelletier/go-toml"))
(native-inputs `())
(home-page
"https://github.com/pelletier/go-toml")
(synopsis "XXX")
(description "XXX")
(license #f))))
(define-public go-gitlab-com-continuousevolution-continuous-evolution
(let ((commit "0.8.1") (revision "0"))
(package
(name "go-gitlab-com-continuousevolution-continuous-evolution")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://gitlab.com/ContinuousEvolution/continuous-evolution.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"1bnr0hz36afvrb1cdnaasrvvvf1zfnf59d0m3pbdsy49md2q17q7"))))
(build-system go-build-system)
(arguments
'(#:import-path
"gitlab.com/ContinuousEvolution/continuous-evolution"))
(native-inputs
`(("go-github-com-fsouza-go-dockerclient"
,go-github-com-fsouza-go-dockerclient)
("go-github-com-sirupsen-logrus"
,go-github-com-sirupsen-logrus)
("go-github-com-heroku-docker-registry-client"
,go-github-com-heroku-docker-registry-client)
("go-github-com-blang-semver"
,go-github-com-blang-semver)
("go-github-com-pelletier-go-toml"
,go-github-com-pelletier-go-toml)))
(home-page
"https://gitlab.com/ContinuousEvolution/continuous-evolution")
(synopsis "XXX")
(description "XXX")
(license #f))))
---end-------------------------------------------------------------------------
I start to work on this patch to help me to create Guix package for
'gitlab-runner' (https://gitlab.com/gitlab-org/gitlab-runner/). But at this
point the patch it's not totally functional.
It's able to create the package and the package for each dependencies defined
in 'Gopkg.toml'.
The code operation is not so difficult:
1 - Clone the repository
2 - Read the 'toml' file while dependence definition are found
2.1 - Get pieces of information on dependence
2.2 - Create package and add dependence in list of dependencies
2.3 - Clone dependence and get the repository hash
2.4 - Continue to point 2
3 - Create principal the package and get the repository hash
4 - Return all packages
But at this time it's contains lot of HACK, for example we skip the 'k8s.io'
'golang.org' and 'cloud.google.com' URLs, because the name cannot be parse to
URL also simply than 'GitHub' or 'GitLab' URLs.
Some interrogation:
1 - Not totally sure the 'toml' file is the best solution. Very reliable ?
2 - Someone have an idea of how to get 'k8s.io', 'golang.org' and
'cloud.google.com' dependencies ?
3 - Other comment ?
Pierre-Antoine Rouby
Rouby Pierre-Antoine (1):
import: Add gopkg importer.
Makefile.am | 1 +
guix/import/gopkg.scm | 294 ++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/gopkg.scm | 99 ++++++++++++
4 files changed, 395 insertions(+), 1 deletion(-)
create mode 100644 guix/import/gopkg.scm
create mode 100644 guix/scripts/import/gopkg.scm
--
2.17.0
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/1] import: Add gopkg importer.
2018-04-26 16:22 [PATCH 0/1] Go importer Rouby Pierre-Antoine
@ 2018-04-27 7:45 ` Rouby Pierre-Antoine
2018-05-02 20:04 ` [PATCH 0/1] Go importer Leo Famulari
1 sibling, 0 replies; 11+ messages in thread
From: Rouby Pierre-Antoine @ 2018-04-27 7:45 UTC (permalink / raw)
To: guix-devel
* guix/import/gopkg.scm: New file.
* guix/scripts/import/gopkg.scm: New file.
* guix/scripts/import.scm: Add 'gopkg'.
* Makefile.am: Add 'gopkg' importer in modules list.
---
Makefile.am | 1 +
guix/import/gopkg.scm | 294 ++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/gopkg.scm | 99 ++++++++++++
4 files changed, 395 insertions(+), 1 deletion(-)
create mode 100644 guix/import/gopkg.scm
create mode 100644 guix/scripts/import/gopkg.scm
diff --git a/Makefile.am b/Makefile.am
index 9f134c970..e103517fc 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -183,6 +183,7 @@ MODULES = \
guix/import/hackage.scm \
guix/import/elpa.scm \
guix/import/texlive.scm \
+ guix/import/gopkg.scm \
guix/scripts.scm \
guix/scripts/download.scm \
guix/scripts/perform-download.scm \
diff --git a/guix/import/gopkg.scm b/guix/import/gopkg.scm
new file mode 100644
index 000000000..451e94a8e
--- /dev/null
+++ b/guix/import/gopkg.scm
@@ -0,0 +1,294 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import gopkg)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module ((ice-9 rdelim) #:select (read-line))
+ #:use-module (git)
+ #:use-module (guix hash)
+ #:use-module (guix base32)
+ #:use-module (guix serialization)
+ #:use-module (guix utils)
+ #:use-module (guix build utils)
+ #:use-module (srfi srfi-11)
+ #:use-module ((guix licenses) #:prefix license:)
+ #:export (gopkg->guix-package))
+
+(define (vcs-file? file stat)
+ (case (stat:type stat)
+ ((directory)
+ (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))
+ ((regular)
+ ;; Git sub-modules have a '.git' file that is a regular text file.
+ (string=? (basename file) ".git"))
+ (else
+ #f)))
+
+(define (file-hash->base32 file)
+ (let-values (((port get-hash) (open-sha256-port)))
+ (write-file file port #:select? (negate vcs-file?))
+ (force-output port)
+ (bytevector->nix-base32-string (get-hash))))
+
+(define (append-inputs inputs name)
+ (append inputs
+ (list
+ (list name
+ (list 'unquote
+ (string->symbol name))))))
+
+(define (package-name url)
+ (string-downcase
+ (string-append "go-"
+ (string-replace-substring
+ (string-replace-substring url
+ "/" "-")
+ "." "-"))))
+
+(define (cut-url url)
+ (string-replace-substring
+ (cond
+ ((string-match "http://" url)
+ (string-replace-substring url "http://" ""))
+ ((string-match "https://" url)
+ (string-replace-substring url "https://" ""))
+ ((string-match "git://" url)
+ (string-replace-substring url "git://" ""))
+ (else
+ (values url)))
+ ".git" ""))
+
+(define (url-to-path url)
+ (string-replace-substring
+ (string-append "/tmp/"
+ (cut-url url))
+ "." "-"))
+
+;; HACK system exec
+(define (git-checkout directory commit)
+ (let ((command (string-append "cd " directory " &&"
+ "git checkout " commit
+ " > /dev/null 2> /dev/null"))) ; HACK no command output
+ (if (not (or (equal? commit "0")
+ (equal? commit "XXX")
+ (equal? commit "master")))
+ (system command))))
+
+(define (git-clone url commit)
+ (define (clone-in-dir url directory)
+ (mkdir-p directory)
+ (clone url directory (clone-init-options))
+ (git-checkout directory commit)
+ (values directory))
+
+ (let ((directory (url-to-path url)))
+ (if (not (file-exists? (string-append directory)))
+ (clone-in-dir url directory)
+ (values directory))))
+
+(define (comment? line)
+ (eq? (string-ref (string-trim line) 0) #\#))
+
+(define (attribute? line str)
+ (equal? (string-trim-right
+ (string-trim
+ (car (string-split line #\=)))) str))
+
+(define (attribute-by-name line name)
+ (string-trim
+ (string-replace-substring
+ (string-replace-substring
+ line (string-append name " = ")
+ "")
+ "\"" "")))
+
+(define (make-go-sexp->package packages dependencies
+ name url version revision
+ commit str-license home-page
+ git-url is-dep hash)
+ (define (package-inputs)
+ (if (not is-dep)
+ (values dependencies)
+ '()))
+
+ (values
+ `(define-public ,(string->symbol name)
+ (let ((commit ,commit)
+ (revision ,revision))
+ (package
+ (name ,name)
+ (version (git-version ,version revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,git-url)
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,hash))))
+ (build-system go-build-system)
+ (arguments
+ '(#:import-path ,url))
+ (native-inputs ,(list 'quasiquote (package-inputs)))
+ (home-page ,home-page)
+ (synopsis "XXX")
+ (description "XXX")
+ (license #f))))))
+
+(define (create-package->packages+dependencies packages dependencies
+ url version
+ revision commit
+ containt is-dep)
+ (let ((synopsis "XXX")
+ (description "XXX")
+ (license "XXX")
+ (name (package-name url))
+ (home-page (string-append "https://" url))
+ (git-url (string-append "https://" url ".git"))
+ (hash (file-hash->base32
+ (git-clone (string-append "https://"
+ url ".git")
+ commit))))
+ (values
+ (append packages
+ (list (make-go-sexp->package packages dependencies
+ name url version
+ revision commit license
+ home-page git-url
+ is-dep hash)))
+ (if containt
+ (append-inputs dependencies name)
+ dependencies))))
+
+(define (website? url)
+ (car (string-split url #\/)))
+
+(define (parse-dependencies->packages+dependencies port constraint
+ packages dependencies)
+ (let ((url "XXX")
+ (version "0.0.0")
+ (revision "0")
+ (commit "XXX"))
+ (define (loop port url commit packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((string-null? (string-trim line)) ; Empty line
+ (if (not (or (equal? "k8s.io" (website? url)) ; HACK bypass k8s
+ (equal? "golang.org" (website? url)) ; HACK bypass golang
+ (equal? "cloud.google.com" (website? url)))) ; HACK bypass cloud.google
+ (create-package->packages+dependencies packages dependencies
+ url version revision
+ commit
+ constraint #t)
+ (values packages dependencies)))
+ ((comment? line) ; Comment
+ (loop port url commit
+ packages dependencies))
+ ((attribute? line "name") ; Name
+ (loop port
+ (attribute-by-name line "name")
+ commit
+ packages dependencies))
+ ((attribute? line "revision") ; Revision
+ (loop port
+ url
+ (attribute-by-name line "revision")
+ packages dependencies))
+ ((attribute? line "version") ; Version
+ (loop port
+ url
+ (attribute-by-name line "version")
+ packages dependencies))
+ ((attribute? line "branch") ; Branch
+ (loop port
+ url
+ (attribute-by-name line "branch")
+ packages dependencies))
+ ((string-match "=" line) ; Other options
+ (loop port url commit
+ packages dependencies))
+ (else (loop port url commit
+ packages dependencies)))))
+ (loop port url commit
+ packages dependencies)))
+
+(define (parse-toml->packages+dependencies port packages dependencies)
+ "Read toml file on 'port' and return all dependencies packages sexp and list of
+constraint dependencies."
+ (define (loop port packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((string-null? line) ; Empty line
+ (loop port packages dependencies))
+ ((comment? line) ; Comment
+ (loop port packages dependencies))
+ ((equal? line "[prune]") ; Ignored
+ (loop port packages dependencies))
+ ((equal? "[[constraint]]" line) ; Direct dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #t
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ ((equal? "[[override]]" line) ; Dependencies of dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #f
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ (else (loop port packages dependencies)))))
+ (loop port packages dependencies))
+
+(define (gopkg-dep->packages+dependencies path)
+ "Open toml file if exist and parse it and return packages sexp and
+dependencies list. Or return two empty list if file not found."
+ (if (file-exists? path)
+ (let ((port (open-input-file path)))
+ (let-values (((packages dependencies)
+ (parse-toml->packages+dependencies port
+ '()
+ '())))
+ (close-port port)
+ (values packages dependencies)))
+ (values '() '())))
+
+(define (gopkg->guix-package url branch)
+ "Create package for git repository dans branch verison and all dependencies sexp packages with
+Gopkg.toml file."
+ (let ((output (url-to-path url))
+ (name (package-name (cut-url url)))
+ (version "0.0.0")
+ (revision "0"))
+ (git-clone url branch)
+
+ (let-values (((packages dependencies)
+ (gopkg-dep->packages+dependencies
+ (string-append output
+ "/Gopkg.toml"))))
+ (let-values (((packages dependencies)
+ (create-package->packages+dependencies packages dependencies
+ (cut-url url) version
+ revision branch
+ #f #f)))
+ (values packages)))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 67bc7a755..3c55bfaff 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -74,7 +74,7 @@ rather than \\n."
;;;
(define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
- "cran" "crate" "texlive" "json"))
+ "cran" "crate" "texlive" "json" "gopkg"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/gopkg.scm b/guix/scripts/import/gopkg.scm
new file mode 100644
index 000000000..f513779ed
--- /dev/null
+++ b/guix/scripts/import/gopkg.scm
@@ -0,0 +1,99 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import gopkg)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import gopkg)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-gopkg))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import gopkg PACKAGE-URL BRANCH
+Import and convert the git repo with toml file to guix package using
+PACKAGE-URL and matching BRANCH.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gopkg")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-gopkg . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((package-url branch)
+ (let ((sexp (gopkg->guix-package package-url branch)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ ((package-url)
+ (let ((sexp (gopkg->guix-package package-url "master")))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.17.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-04-26 16:22 [PATCH 0/1] Go importer Rouby Pierre-Antoine
2018-04-27 7:45 ` [PATCH 1/1] import: Add gopkg importer Rouby Pierre-Antoine
@ 2018-05-02 20:04 ` Leo Famulari
2018-06-04 8:18 ` Pierre-Antoine Rouby
1 sibling, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2018-05-02 20:04 UTC (permalink / raw)
To: Rouby Pierre-Antoine; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]
On Thu, Apr 26, 2018 at 06:22:38PM +0200, Rouby Pierre-Antoine wrote:
> This patch is a importer for go packages, it's use git repository and the
> 'Gopkg.toml' file (https://golang.github.io/dep/docs/Gopkg.toml.html).
Neat! I didn't have a chance to review the code yet but I'm happy to see
progress in this area. The examples look promising! I'll review it
properly soon...
I've noticed several different dependency manifest formats for Go
software "in the wild". So, this one is specific to Go software using
the Gopkg tool?
Skimming the patch, I noticed a few places that I think could use some
explanatory comments. For example, the git-checkout procedure. Since a
few days have passed, it might be a good time for you to read it over
and write some commentary on the sections that are beginning to become
confusing, as all code does with time :)
Nitpick...
> (define-public go-github-com-fsouza-go-dockerclient
> (let ((commit "master") (revision "0"))
Does "master" refer to the branch? If so, we should avoid using it,
since its value will probably change over time. We should stick to
commits and tags. Tags can be changed but this is rarely done.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-05-02 20:04 ` [PATCH 0/1] Go importer Leo Famulari
@ 2018-06-04 8:18 ` Pierre-Antoine Rouby
2018-07-11 19:04 ` Leo Famulari
0 siblings, 1 reply; 11+ messages in thread
From: Pierre-Antoine Rouby @ 2018-06-04 8:18 UTC (permalink / raw)
To: Leo Famulari; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 378 bytes --]
Hi, Leo
> From: "Leo Famulari" <leo@famulari.name>
> I've noticed several different dependency manifest formats for Go
> software "in the wild". So, this one is specific to Go software using
> the Gopkg tool?
Yes, Gopkg it's specific for Go software, it's not used by all Go
projects, but lot of big project use it.
Attached an update of importer.
Thanks for comments.
P-A
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-gopkg-importer.patch --]
[-- Type: text/x-patch; name=0001-import-Add-gopkg-importer.patch, Size: 20312 bytes --]
From 3e9b4fa12811432fdd7a4d6330f9093dcc72d25a Mon Sep 17 00:00:00 2001
From: Rouby Pierre-Antoine <pierre-antoine.rouby@inria.fr>
Date: Thu, 26 Apr 2018 15:05:23 +0200
Subject: [PATCH] import: Add gopkg importer.
* guix/import/gopkg.scm: New file.
* guix/scripts/import/gopkg.scm: New file.
* guix/scripts/import.scm: Add 'gopkg'.
* Makefile.am: Add 'gopkg' importer in modules list.
---
Makefile.am | 1 +
guix/import/gopkg.scm | 384 ++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/gopkg.scm | 99 +++++++++
4 files changed, 485 insertions(+), 1 deletion(-)
create mode 100644 guix/import/gopkg.scm
create mode 100644 guix/scripts/import/gopkg.scm
diff --git a/Makefile.am b/Makefile.am
index 9f134c970..e103517fc 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -183,6 +183,7 @@ MODULES = \
guix/import/hackage.scm \
guix/import/elpa.scm \
guix/import/texlive.scm \
+ guix/import/gopkg.scm \
guix/scripts.scm \
guix/scripts/download.scm \
guix/scripts/perform-download.scm \
diff --git a/guix/import/gopkg.scm b/guix/import/gopkg.scm
new file mode 100644
index 000000000..200d9ffd3
--- /dev/null
+++ b/guix/import/gopkg.scm
@@ -0,0 +1,384 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import gopkg)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module ((ice-9 rdelim) #:select (read-line))
+ #:use-module (srfi srfi-11)
+ #:use-module (texinfo string-utils) ; transform-string
+ #:use-module (guix hash)
+ #:use-module (guix base32)
+ #:use-module (guix serialization)
+ #:use-module (guix utils)
+ #:use-module (guix build utils)
+ #:use-module ((guix licenses) #:prefix license:)
+ #:export (gopkg->guix-package))
+
+;;
+;; Directory.
+;;
+
+(define (get-new-tmp-dir)
+ "Return new temp directory."
+ (let ((tmp "/tmp/guix-import-gopkg"))
+ (define (new num)
+ (let ((new-dir (string-append tmp "-" (number->string num))))
+ (if (file-exists? new-dir)
+ (new (+ num 1))
+ new-dir)))
+ (if (file-exists? tmp)
+ (new 0)
+ tmp)))
+
+(define tmp-dir (get-new-tmp-dir))
+
+;;
+;; Git.
+;;
+
+(define (vcs-file? file stat)
+ (case (stat:type stat)
+ ((directory)
+ (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))
+ ((regular)
+ ;; Git sub-modules have a '.git' file that is a regular text file.
+ (string=? (basename file) ".git"))
+ (else
+ #f)))
+
+(define (file->hash-base32 file)
+ "Return hash of FILE in nix base32 sha256 format. If FILE is a directory,
+exclude vcs files."
+ (let-values (((port get-hash) (open-sha256-port)))
+ (write-file file port #:select? (negate vcs-file?))
+ (force-output port)
+ (bytevector->nix-base32-string (get-hash))))
+
+(define (git->hash url commit file)
+ "Clone git repository and return FILE hash in nix base32 sha256 format."
+ (if (not (file-exists? file))
+ (git-fetch url commit file #:recursive? #f))
+ (file->hash-base32 file))
+
+(define (git-ref->commit path tag)
+ "Return commit number coresponding to git TAG. Return \"XXX\" if tag is not
+found."
+ (define (loop port)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (begin
+ (close-port port)
+ "XXX"))
+ ((string-match tag line) ; Match tag
+ (let ((commit (car (string-split (transform-string line #\tab " ")
+ #\ ))))
+ commit))
+ (else ; Else
+ (loop port)))))
+
+ (let ((file (if (file-exists? (string-append path "/.git/packed-refs"))
+ (string-append path "/.git/packed-refs")
+ (string-append path "/.git/FETCH_HEAD"))))
+ (loop (open-input-file file))))
+
+(define* (git-fetch url commit directory
+ #:key (git-command "git") recursive?)
+ "Fetch COMMIT from URL into DIRECTORY. COMMIT must be a valid Git commit
+identifier. When RECURSIVE? is true, all the sub-modules of URL are fetched,
+recursively. Return #t on success, #f otherwise."
+ ;; Disable TLS certificate verification. The hash of the checkout is known
+ ;; in advance anyway.
+ (setenv "GIT_SSL_NO_VERIFY" "true")
+
+ (mkdir-p directory)
+
+ (with-directory-excursion directory
+ (invoke git-command "init")
+ (invoke git-command "remote" "add" "origin" url)
+ (if (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
+ (invoke git-command "checkout" "FETCH_HEAD")
+ (begin
+ (invoke git-command "fetch" "origin")
+ (if (not (zero? (system* git-command "checkout" commit)))
+ (let ((commit-hash (git-ref->commit directory commit)))
+ (invoke git-command "checkout" "master")
+ (if (not (equal? "XXX" commit-hash)) ;HACK else stay on master
+ (zero? (system* git-command "checkout" commit-hash))))
+ #t)))))
+
+;;
+;; Append attributes.
+;;
+
+(define (append-inputs inputs name)
+ "Return list with new input corresponding to package NAME."
+ (let ((unquote-name (list 'unquote (string->symbol name))))
+ (append inputs (list (list name unquote-name)))))
+
+;;
+;; Parse attributes.
+;;
+
+(define (url->package-name url)
+ "Compute URL and return package name."
+ (let* ((url-no-slash (string-replace-substring url "/" "-"))
+ (url-no-slash-no-dot (string-replace-substring url-no-slash
+ "." "-")))
+ (string-downcase (string-append "go-" url-no-slash-no-dot))))
+
+(define (cut-url url)
+ "Return URL without protocol prefix and git file extension."
+ (string-replace-substring
+ (cond
+ ((string-match "http://" url)
+ (string-replace-substring url "http://" ""))
+ ((string-match "https://" url)
+ (string-replace-substring url "https://" ""))
+ ((string-match "git://" url)
+ (string-replace-substring url "git://" ""))
+ (else
+ url))
+ ".git" ""))
+
+(define (url->path url)
+ "Return directory path corresponding to URL."
+ (string-replace-substring
+ (string-append tmp-dir "/"
+ (cut-url url))
+ "." "-"))
+
+(define (url->dn url)
+ "Return the web site DN form url 'gnu.org/software/guix' --> 'gnu.org'"
+ (car (string-split url #\/)))
+
+(define (url->git-url url)
+ (string-append "https://" url ".git"))
+
+(define (comment? line)
+ "Return #t if LINE start with comment delimiter, else return #f."
+ (eq? (string-ref (string-trim line) 0) #\#))
+
+(define (empty-line? line)
+ "Return #t if LINE is empty, else #f."
+ (string-null? (string-trim line)))
+
+(define (attribute? line attribute)
+ "Return #t if LINE contain ATTRIBUTE."
+ (equal? (string-trim-right
+ (string-trim
+ (car (string-split line #\=)))) attribute))
+
+(define (attribute-by-name line name)
+ "Return attribute value corresponding to NAME."
+ (let* ((line-no-attribut-name (string-replace-substring
+ line
+ (string-append name " = ") ""))
+ (value-no-double-quote (string-replace-substring
+ line-no-attribut-name
+ "\"" "")))
+ (string-trim value-no-double-quote)))
+
+;;
+;; Packages functions.
+;;
+
+(define (make-go-sexp->package packages dependencies
+ name url version revision
+ commit str-license home-page
+ git-url is-dep? hash)
+ "Create Guix sexp package for Go software NAME. Return new package sexp."
+ (define (package-inputs)
+ (if (not is-dep?)
+ `((native-inputs ,(list 'quasiquote dependencies)))
+ '()))
+
+ (values
+ `(define-public ,(string->symbol name)
+ (let ((commit ,commit)
+ (revision ,revision))
+ (package
+ (name ,name)
+ (version (git-version ,version revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,git-url)
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,hash))))
+ (build-system go-build-system)
+ (arguments
+ '(#:import-path ,url))
+ ,@(package-inputs)
+ (home-page ,home-page)
+ (synopsis "XXX")
+ (description "XXX")
+ (license #f))))))
+
+(define (create-package->packages+dependencies packages dependencies
+ url version
+ revision commit
+ constraint? is-dep?)
+ "Return packages and dependencies with new package sexp corresponding to
+URL."
+ (let ((name (url->package-name url))
+ (home-page (string-append "https://" url))
+ (git-url (url->git-url url))
+ (synopsis "XXX")
+ (description "XXX")
+ (license "XXX"))
+ (let ((hash (git->hash (url->git-url url)
+ commit
+ (url->path git-url)))
+ (commit-hash (if (< (string-length commit) 40)
+ (git-ref->commit (url->path
+ (url->git-url url))
+ commit)
+ commit)))
+ (values
+ (append packages
+ (list
+ (make-go-sexp->package packages dependencies
+ name url version
+ revision commit-hash
+ license home-page
+ git-url is-dep? hash)))
+ (if constraint?
+ (append-inputs dependencies name)
+ dependencies)))))
+
+(define (parse-dependencies->packages+dependencies port constraint?
+ packages dependencies)
+ "Parse one dependencies in PORT, and return packages and dependencies list."
+ (let ((url "XXX")
+ (version "0.0.0")
+ (revision "0")
+ (commit "XXX"))
+ (define (loop port url commit packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (if (not (or (equal? "k8s.io" (url->dn url)) ; HACK bypass k8s
+ (equal? "golang.org" (url->dn url)) ; HACK bypass golang
+ (equal? "cloud.google.com" (url->dn url)))) ; HACK bypass cloud.google
+ (create-package->packages+dependencies packages dependencies
+ url version revision
+ commit
+ constraint? #t)
+ (values packages dependencies)))
+ ((comment? line) ; Comment
+ (loop port url commit
+ packages dependencies))
+ ((attribute? line "name") ; Name
+ (loop port
+ (attribute-by-name line "name")
+ commit
+ packages dependencies))
+ ((attribute? line "revision") ; Revision
+ (loop port
+ url
+ (attribute-by-name line "revision")
+ packages dependencies))
+ ((attribute? line "version") ; Version
+ (loop port
+ url
+ (attribute-by-name line "version")
+ packages dependencies))
+ ((attribute? line "branch") ; Branch
+ (loop port
+ url
+ (attribute-by-name line "branch")
+ packages dependencies))
+ ((string-match "=" line) ; Other options
+ (loop port url commit
+ packages dependencies))
+ (else (loop port url commit
+ packages dependencies)))))
+ (loop port url commit
+ packages dependencies)))
+
+(define (parse-toml->packages+dependencies port packages dependencies)
+ "Read toml file on PORT and return all dependencies packages sexp and list
+of constraint dependencies."
+ (define (loop port packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (loop port packages dependencies))
+ ((comment? line) ; Comment
+ (loop port packages dependencies))
+ ((equal? line "[prune]") ; Ignored
+ (loop port packages dependencies))
+ ((equal? "[[constraint]]" line) ; Direct dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #t
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ ((equal? "[[override]]" line) ; Dependencies of dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #f
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ (else (loop port packages dependencies)))))
+ (loop port packages dependencies))
+
+(define (gopkg-dep->packages+dependencies path)
+ "Open toml file if exist and parse it and return packages sexp and
+dependencies list. Or return two empty list if file not found."
+ (if (file-exists? path)
+ (let ((port (open-input-file path)))
+ (let-values (((packages dependencies)
+ (parse-toml->packages+dependencies port
+ '() '())))
+ (close-port port)
+ (values packages dependencies)))
+ (values '() '())))
+
+;;
+;; Entry point.
+;;
+
+(define (gopkg->guix-package url branch)
+ "Create package for git repository dans branch verison and all dependencies
+sexp packages with Gopkg.toml file."
+ (let ((output (url->path url))
+ (name (url->package-name (cut-url url)))
+ (version "0.0.0")
+ (revision "0"))
+ (git-fetch url branch output #:recursive? #f)
+
+ (let-values (((packages dependencies)
+ (gopkg-dep->packages+dependencies
+ (string-append output
+ "/Gopkg.toml"))))
+ (let-values (((packages dependencies)
+ (create-package->packages+dependencies packages dependencies
+ (cut-url url) version
+ revision branch
+ #f #f)))
+ (values packages)))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 67bc7a755..3c55bfaff 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -74,7 +74,7 @@ rather than \\n."
;;;
(define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
- "cran" "crate" "texlive" "json"))
+ "cran" "crate" "texlive" "json" "gopkg"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/gopkg.scm b/guix/scripts/import/gopkg.scm
new file mode 100644
index 000000000..f513779ed
--- /dev/null
+++ b/guix/scripts/import/gopkg.scm
@@ -0,0 +1,99 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import gopkg)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import gopkg)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-gopkg))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import gopkg PACKAGE-URL BRANCH
+Import and convert the git repo with toml file to guix package using
+PACKAGE-URL and matching BRANCH.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gopkg")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-gopkg . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((package-url branch)
+ (let ((sexp (gopkg->guix-package package-url branch)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ ((package-url)
+ (let ((sexp (gopkg->guix-package package-url "master")))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.17.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-06-04 8:18 ` Pierre-Antoine Rouby
@ 2018-07-11 19:04 ` Leo Famulari
2018-07-18 13:11 ` Pierre-Antoine Rouby
0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2018-07-11 19:04 UTC (permalink / raw)
To: Pierre-Antoine Rouby; +Cc: guix-devel
[-- Attachment #1.1: Type: text/plain, Size: 2491 bytes --]
On Mon, Jun 04, 2018 at 10:18:53AM +0200, Pierre-Antoine Rouby wrote:
> Attached an update of importer.
Thanks! I'm sorry my response is so late.
> From 3e9b4fa12811432fdd7a4d6330f9093dcc72d25a Mon Sep 17 00:00:00 2001
> From: Rouby Pierre-Antoine <pierre-antoine.rouby@inria.fr>
> Date: Thu, 26 Apr 2018 15:05:23 +0200
> Subject: [PATCH] import: Add gopkg importer.
>
> * guix/import/gopkg.scm: New file.
> * guix/scripts/import/gopkg.scm: New file.
> * guix/scripts/import.scm: Add 'gopkg'.
> * Makefile.am: Add 'gopkg' importer in modules list.
I wonder which of the new files needs to be added to Makefile.am? My
Autotools knowledge is not very strong...
> +(define (get-new-tmp-dir)
> + "Return new temp directory."
> + (let ((tmp "/tmp/guix-import-gopkg"))
> + (define (new num)
> + (let ((new-dir (string-append tmp "-" (number->string num))))
> + (if (file-exists? new-dir)
> + (new (+ num 1))
> + new-dir)))
> + (if (file-exists? tmp)
> + (new 0)
> + tmp)))
> +
> +(define tmp-dir (get-new-tmp-dir))
I noticed a couple issues with this code. First, the names of the
temporary directories are predictable (they use an incrementing
integer). Second, the temporary files are not deleted after the importer
runs. I've attached a modified patch that addresses this by using ((guix
utils) call-with-temporary-directory), which should address these
problems. [0]
> +(define* (git-fetch url commit directory
> + #:key (git-command "git") recursive?)
> + "Fetch COMMIT from URL into DIRECTORY. COMMIT must be a valid Git commit
> +identifier. When RECURSIVE? is true, all the sub-modules of URL are fetched,
> +recursively. Return #t on success, #f otherwise."
> + ;; Disable TLS certificate verification. The hash of the checkout is known
> + ;; in advance anyway.
> + (setenv "GIT_SSL_NO_VERIFY" "true")
My patch turns certificate verification back on. When importing, the
hash of the checkout is not known in advance.
And finally I added some brief documentation to the manual. Maybe there
could be further clean-up and code deduplication with other parts of the
Guix codebase, but I think it's better to have this importer in Guix
now.
What do you think of my patch? Does it still work for you?
[0] Actually, the temp directories will not be cleaned up due to
<https://bugs.gnu.org/32126>, but I think this will eventually be fixed.
[-- Attachment #1.2: 0001-import-Add-gopkg-importer.patch --]
[-- Type: text/plain, Size: 21413 bytes --]
From 50c12ff1770286fae00dac469cad3af4a9df1070 Mon Sep 17 00:00:00 2001
From: Rouby Pierre-Antoine <pierre-antoine.rouby@inria.fr>
Date: Thu, 26 Apr 2018 15:05:23 +0200
Subject: [PATCH] import: Add gopkg importer.
* guix/import/gopkg.scm: New file.
* guix/scripts/import/gopkg.scm: New file.
* guix/scripts/import.scm: Add 'gopkg'.
* Makefile.am: Add 'gopkg' importer in modules list.
Co-authored-by: Leo Famulari <leo@famulari.name>
---
Makefile.am | 1 +
doc/guix.texi | 9 +-
guix/import/gopkg.scm | 352 ++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/gopkg.scm | 99 ++++++++++
5 files changed, 461 insertions(+), 2 deletions(-)
create mode 100644 guix/import/gopkg.scm
create mode 100644 guix/scripts/import/gopkg.scm
diff --git a/Makefile.am b/Makefile.am
index 618d1653e..a93a280b2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -186,6 +186,7 @@ MODULES = \
guix/import/hackage.scm \
guix/import/elpa.scm \
guix/import/texlive.scm \
+ guix/import/gopkg.scm \
guix/scripts.scm \
guix/scripts/download.scm \
guix/scripts/perform-download.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index a8e53a530..07100cd25 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20,7 +20,7 @@ Copyright @copyright{} 2014, 2015, 2016 Alex Kost@*
Copyright @copyright{} 2015, 2016 Mathieu Lirzin@*
Copyright @copyright{} 2014 Pierre-Antoine Rault@*
Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer@*
-Copyright @copyright{} 2015, 2016, 2017 Leo Famulari@*
+Copyright @copyright{} 2015, 2016, 2017, 2018 Leo Famulari@*
Copyright @copyright{} 2015, 2016, 2017, 2018 Ricardo Wurmus@*
Copyright @copyright{} 2016 Ben Woodcroft@*
Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
@@ -6667,6 +6667,13 @@ Import metadata from the crates.io Rust package repository
@cindex OCaml
Import metadata from the @uref{https://opam.ocaml.org/, OPAM} package
repository used by the OCaml community.
+
+@item gopkg
+@cindex gopkg
+@cindex Golang
+@cindex Go
+Import metadata from the @uref{https://gopkg.in/, gopkg} package
+versioning service used by some Go software.
@end table
The structure of the @command{guix import} code is modular. It would be
diff --git a/guix/import/gopkg.scm b/guix/import/gopkg.scm
new file mode 100644
index 000000000..c2b72616a
--- /dev/null
+++ b/guix/import/gopkg.scm
@@ -0,0 +1,352 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import gopkg)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module ((ice-9 rdelim) #:select (read-line))
+ #:use-module (srfi srfi-11)
+ #:use-module (texinfo string-utils) ; transform-string
+ #:use-module (guix hash)
+ #:use-module (guix base32)
+ #:use-module (guix serialization)
+ #:use-module (guix utils)
+ #:use-module (guix build utils)
+ #:use-module ((guix licenses) #:prefix license:)
+ #:export (gopkg->guix-package))
+
+(define (vcs-file? file stat)
+ (case (stat:type stat)
+ ((directory)
+ (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))
+ ((regular)
+ ;; Git sub-modules have a '.git' file that is a regular text file.
+ (string=? (basename file) ".git"))
+ (else
+ #f)))
+
+(define (file->hash-base32 file)
+ "Return hash of FILE in nix base32 sha256 format. If FILE is a directory,
+exclude vcs files."
+ (let-values (((port get-hash) (open-sha256-port)))
+ (write-file file port #:select? (negate vcs-file?))
+ (force-output port)
+ (bytevector->nix-base32-string (get-hash))))
+
+(define (git->hash url commit file)
+ "Clone git repository and return FILE hash in nix base32 sha256 format."
+ (if (not (file-exists? file))
+ (git-fetch url commit file #:recursive? #f))
+ (file->hash-base32 file))
+
+(define (git-ref->commit path tag)
+ "Return commit number coresponding to git TAG. Return \"XXX\" if tag is not
+found."
+ (define (loop port)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (begin
+ (close-port port)
+ "XXX"))
+ ((string-match tag line) ; Match tag
+ (let ((commit (car (string-split (transform-string line #\tab " ")
+ #\ ))))
+ commit))
+ (else ; Else
+ (loop port)))))
+
+ (let ((file (if (file-exists? (string-append path "/.git/packed-refs"))
+ (string-append path "/.git/packed-refs")
+ (string-append path "/.git/FETCH_HEAD"))))
+ (loop (open-input-file file))))
+
+(define* (git-fetch url commit directory
+ #:key (git-command "git") recursive?)
+ "Fetch COMMIT from URL into DIRECTORY. COMMIT must be a valid Git commit
+identifier. When RECURSIVE? is true, all the sub-modules of URL are fetched,
+recursively. Return #t on success, #f otherwise."
+ (mkdir-p directory)
+
+ (with-directory-excursion directory
+ (invoke git-command "init")
+ (invoke git-command "remote" "add" "origin" url)
+ (if (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
+ (invoke git-command "checkout" "FETCH_HEAD")
+ (begin
+ (invoke git-command "fetch" "origin")
+ (if (not (zero? (system* git-command "checkout" commit)))
+ (let ((commit-hash (git-ref->commit directory commit)))
+ (invoke git-command "checkout" "master")
+ (if (not (equal? "XXX" commit-hash)) ;HACK else stay on master
+ (zero? (system* git-command "checkout" commit-hash))))
+ #t)))))
+
+;;
+;; Append attributes.
+;;
+
+(define (append-inputs inputs name)
+ "Return list with new input corresponding to package NAME."
+ (let ((unquote-name (list 'unquote (string->symbol name))))
+ (append inputs (list (list name unquote-name)))))
+
+;;
+;; Parse attributes.
+;;
+
+(define (url->package-name url)
+ "Compute URL and return package name."
+ (let* ((url-no-slash (string-replace-substring url "/" "-"))
+ (url-no-slash-no-dot (string-replace-substring url-no-slash
+ "." "-")))
+ (string-downcase (string-append "go-" url-no-slash-no-dot))))
+
+(define (cut-url url)
+ "Return URL without protocol prefix and git file extension."
+ (string-replace-substring
+ (cond
+ ((string-match "http://" url)
+ (string-replace-substring url "http://" ""))
+ ((string-match "https://" url)
+ (string-replace-substring url "https://" ""))
+ ((string-match "git://" url)
+ (string-replace-substring url "git://" ""))
+ (else
+ url))
+ ".git" ""))
+
+(define (url->dn url)
+ "Return the web site DN form url 'gnu.org/software/guix' --> 'gnu.org'"
+ (car (string-split url #\/)))
+
+(define (url->git-url url)
+ (string-append "https://" url ".git"))
+
+(define (comment? line)
+ "Return #t if LINE start with comment delimiter, else return #f."
+ (eq? (string-ref (string-trim line) 0) #\#))
+
+(define (empty-line? line)
+ "Return #t if LINE is empty, else #f."
+ (string-null? (string-trim line)))
+
+(define (attribute? line attribute)
+ "Return #t if LINE contain ATTRIBUTE."
+ (equal? (string-trim-right
+ (string-trim
+ (car (string-split line #\=)))) attribute))
+
+(define (attribute-by-name line name)
+ "Return attribute value corresponding to NAME."
+ (let* ((line-no-attribut-name (string-replace-substring
+ line
+ (string-append name " = ") ""))
+ (value-no-double-quote (string-replace-substring
+ line-no-attribut-name
+ "\"" "")))
+ (string-trim value-no-double-quote)))
+
+;;
+;; Packages functions.
+;;
+
+(define (make-go-sexp->package packages dependencies
+ name url version revision
+ commit str-license home-page
+ git-url is-dep? hash)
+ "Create Guix sexp package for Go software NAME. Return new package sexp."
+ (define (package-inputs)
+ (if (not is-dep?)
+ `((native-inputs ,(list 'quasiquote dependencies)))
+ '()))
+
+ (values
+ `(define-public ,(string->symbol name)
+ (let ((commit ,commit)
+ (revision ,revision))
+ (package
+ (name ,name)
+ (version (git-version ,version revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,git-url)
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,hash))))
+ (build-system go-build-system)
+ (arguments
+ '(#:import-path ,url))
+ ,@(package-inputs)
+ (home-page ,home-page)
+ (synopsis "XXX")
+ (description "XXX")
+ (license #f))))))
+
+(define (create-package->packages+dependencies packages dependencies
+ url version directory
+ revision commit
+ constraint? is-dep?)
+ "Return packages and dependencies with new package sexp corresponding to
+URL."
+ (let ((name (url->package-name url))
+ (home-page (string-append "https://" url))
+ (git-url (url->git-url url))
+ (synopsis "XXX")
+ (description "XXX")
+ (license "XXX"))
+ (let ((hash (git->hash (url->git-url url)
+ commit
+ directory))
+ (commit-hash (if (< (string-length commit) 40)
+ (git-ref->commit directory
+ commit)
+ commit)))
+ (values
+ (append packages
+ (list
+ (make-go-sexp->package packages dependencies
+ name url version
+ revision commit-hash
+ license home-page
+ git-url is-dep? hash)))
+ (if constraint?
+ (append-inputs dependencies name)
+ dependencies)))))
+
+(define (parse-dependencies->packages+dependencies port constraint?
+ packages dependencies)
+ "Parse one dependencies in PORT, and return packages and dependencies list."
+ (let ((url "XXX")
+ (version "0.0.0")
+ (revision "0")
+ (commit "XXX"))
+ (define (loop port url commit packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (if (not (or (equal? "k8s.io" (url->dn url)) ; HACK bypass k8s
+ (equal? "golang.org" (url->dn url)) ; HACK bypass golang
+ (equal? "cloud.google.com" (url->dn url)))) ; HACK bypass cloud.google
+ (create-package->packages+dependencies packages dependencies
+ url version port revision
+ commit
+ constraint? #t)
+ (values packages dependencies)))
+ ((comment? line) ; Comment
+ (loop port url commit
+ packages dependencies))
+ ((attribute? line "name") ; Name
+ (loop port
+ (attribute-by-name line "name")
+ commit
+ packages dependencies))
+ ((attribute? line "revision") ; Revision
+ (loop port
+ url
+ (attribute-by-name line "revision")
+ packages dependencies))
+ ((attribute? line "version") ; Version
+ (loop port
+ url
+ (attribute-by-name line "version")
+ packages dependencies))
+ ((attribute? line "branch") ; Branch
+ (loop port
+ url
+ (attribute-by-name line "branch")
+ packages dependencies))
+ ((string-match "=" line) ; Other options
+ (loop port url commit
+ packages dependencies))
+ (else (loop port url commit
+ packages dependencies)))))
+ (loop port url commit
+ packages dependencies)))
+
+(define (parse-toml->packages+dependencies port packages dependencies)
+ "Read toml file on PORT and return all dependencies packages sexp and list
+of constraint dependencies."
+ (define (loop port packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (loop port packages dependencies))
+ ((comment? line) ; Comment
+ (loop port packages dependencies))
+ ((equal? line "[prune]") ; Ignored
+ (loop port packages dependencies))
+ ((equal? "[[constraint]]" line) ; Direct dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #t
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ ((equal? "[[override]]" line) ; Dependencies of dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #f
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ (else (loop port packages dependencies)))))
+ (loop port packages dependencies))
+
+(define (gopkg-dep->packages+dependencies path)
+ "Open toml file if exist and parse it and return packages sexp and
+dependencies list. Or return two empty list if file not found."
+ (if (file-exists? path)
+ (let ((port (open-input-file path)))
+ (let-values (((packages dependencies)
+ (parse-toml->packages+dependencies port
+ '() '())))
+ (close-port port)
+ (values packages dependencies)))
+ (values '() '())))
+
+;;
+;; Entry point.
+;;
+
+(define (gopkg->guix-package url branch)
+ "Create package for git repository dans branch verison and all dependencies
+sexp packages with Gopkg.toml file."
+ (let ((name (url->package-name (cut-url url)))
+ (version "0.0.0")
+ (revision "0"))
+ (call-with-temporary-directory
+ (lambda (directory)
+ (git-fetch url branch directory #:recursive? #f)
+
+ (let-values (((packages dependencies)
+ (gopkg-dep->packages+dependencies
+ (string-append directory
+ "/Gopkg.toml"))))
+ (let-values (((packages dependencies)
+ (create-package->packages+dependencies packages dependencies
+ (cut-url url) version
+ directory
+ revision branch
+ #f #f)))
+ (values packages)))))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 0b326e104..56b34971e 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -75,7 +75,7 @@ rather than \\n."
;;;
(define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
- "cran" "crate" "texlive" "json" "opam"))
+ "cran" "crate" "texlive" "json" "opam" "gopkg"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/gopkg.scm b/guix/scripts/import/gopkg.scm
new file mode 100644
index 000000000..9a39e58d7
--- /dev/null
+++ b/guix/scripts/import/gopkg.scm
@@ -0,0 +1,99 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import gopkg)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import gopkg)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-gopkg))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import gopkg PACKAGE-URL BRANCH
+Import and convert the Git repository with TOML file to a Guix package
+using PACKAGE-URL and matching BRANCH.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gopkg")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-gopkg . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((package-url branch)
+ (let ((sexp (gopkg->guix-package package-url branch)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ ((package-url)
+ (let ((sexp (gopkg->guix-package package-url "master")))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.18.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-11 19:04 ` Leo Famulari
@ 2018-07-18 13:11 ` Pierre-Antoine Rouby
2018-07-18 17:07 ` Leo Famulari
0 siblings, 1 reply; 11+ messages in thread
From: Pierre-Antoine Rouby @ 2018-07-18 13:11 UTC (permalink / raw)
To: Leo Famulari; +Cc: guix-devel
Hi Leo,
----- Original Message -----
> From: "Leo Famulari" <leo@famulari.name>
>> * guix/import/gopkg.scm: New file.
>> * guix/scripts/import/gopkg.scm: New file.
>> * guix/scripts/import.scm: Add 'gopkg'.
>> * Makefile.am: Add 'gopkg' importer in modules list.
>
> I wonder which of the new files needs to be added to Makefile.am? My
> Autotools knowledge is not very strong...
Oups, yes guix/scripts/import/gopkg.scm need to added to Makefile.am, my bad.
> I noticed a couple issues with this code. First, the names of the
> temporary directories are predictable (they use an incrementing
> integer). Second, the temporary files are not deleted after the importer
> runs. I've attached a modified patch that addresses this by using ((guix
> utils) call-with-temporary-directory), which should address these
> problems. [0]
> What do you think of my patch? Does it still work for you?
I thinks the modifications doesn't works on my computer.
---------------------------------------------------------------------------
Backtrace:
13 (apply-smob/1 #<catch-closure 9410a0>)
In ice-9/boot-9.scm:
705:2 12 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
619:8 11 (_ #(#(#<directory (guile-user) 9f8140>)))
In guix/ui.scm:
1579:12 10 (run-guix-command _ . _)
In guix/scripts/import.scm:
115:11 9 (guix-import . _)
In guix/scripts/import/gopkg.scm:
85:19 8 (guix-import-gopkg . _)
In guix/utils.scm:
633:8 7 (call-with-temporary-directory #<procedure f21e20 at ic…>)
In unknown file:
6 (_ #<procedure f27f40 at ice-9/eval.scm:330:13 ()> #<p…> …)
5 (_ #<procedure f27c40 at ice-9/eval.scm:330:13 ()> #<p…> …)
4 (_ #<procedure f29800 at ice-9/eval.scm:330:13 ()> #<p…> …)
In ice-9/eval.scm:
298:34 3 (_ #(#(#(#<directory (guix import gopkg) df78c0>) # …) …))
619:8 2 (_ #(#(#<directory (guix import gopkg) df78c0> #<i…>) …))
In guix/serialization.scm:
270:25 1 (write-file #<input: /tmp/guix-directory.H1Ut1e/Gopkg.…> …)
In unknown file:
0 (lstat #<input: /tmp/guix-directory.H1Ut1e/Gopkg.toml 15>)
ERROR: In procedure lstat:
Wrong type (expecting string): #<input: /tmp/guix-directory.H1Ut1e/Gopkg.toml 15>
---------------------------------------------------------------------------
--
Pierre-Antoine Rouby
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-18 13:11 ` Pierre-Antoine Rouby
@ 2018-07-18 17:07 ` Leo Famulari
2018-07-19 6:56 ` Pierre-Antoine Rouby
0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2018-07-18 17:07 UTC (permalink / raw)
To: Pierre-Antoine Rouby; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 1655 bytes --]
On Wed, Jul 18, 2018 at 03:11:36PM +0200, Pierre-Antoine Rouby wrote:
> I thinks the modifications doesn't works on my computer.
>
> ---------------------------------------------------------------------------
> Backtrace:
> 13 (apply-smob/1 #<catch-closure 9410a0>)
> In ice-9/boot-9.scm:
> 705:2 12 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> In ice-9/eval.scm:
> 619:8 11 (_ #(#(#<directory (guile-user) 9f8140>)))
> In guix/ui.scm:
> 1579:12 10 (run-guix-command _ . _)
> In guix/scripts/import.scm:
> 115:11 9 (guix-import . _)
> In guix/scripts/import/gopkg.scm:
> 85:19 8 (guix-import-gopkg . _)
> In guix/utils.scm:
> 633:8 7 (call-with-temporary-directory #<procedure f21e20 at ic…>)
> In unknown file:
> 6 (_ #<procedure f27f40 at ice-9/eval.scm:330:13 ()> #<p…> …)
> 5 (_ #<procedure f27c40 at ice-9/eval.scm:330:13 ()> #<p…> …)
> 4 (_ #<procedure f29800 at ice-9/eval.scm:330:13 ()> #<p…> …)
> In ice-9/eval.scm:
> 298:34 3 (_ #(#(#(#<directory (guix import gopkg) df78c0>) # …) …))
> 619:8 2 (_ #(#(#<directory (guix import gopkg) df78c0> #<i…>) …))
> In guix/serialization.scm:
> 270:25 1 (write-file #<input: /tmp/guix-directory.H1Ut1e/Gopkg.…> …)
> In unknown file:
> 0 (lstat #<input: /tmp/guix-directory.H1Ut1e/Gopkg.toml 15>)
>
> ERROR: In procedure lstat:
> Wrong type (expecting string): #<input: /tmp/guix-directory.H1Ut1e/Gopkg.toml 15>
> ---------------------------------------------------------------------------
What package are you trying to import here?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-18 17:07 ` Leo Famulari
@ 2018-07-19 6:56 ` Pierre-Antoine Rouby
2018-07-19 21:38 ` Leo Famulari
0 siblings, 1 reply; 11+ messages in thread
From: Pierre-Antoine Rouby @ 2018-07-19 6:56 UTC (permalink / raw)
To: Leo Famulari; +Cc: guix-devel
> From: "Leo Famulari" <leo@famulari.name>
> What package are you trying to import here?
I trying to import 'gitlab-runner' (https://gitlab.com/gitlab-org/gitlab-runner)
with tag 'v10.6.0'.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-19 6:56 ` Pierre-Antoine Rouby
@ 2018-07-19 21:38 ` Leo Famulari
2018-07-25 8:48 ` Pierre-Antoine Rouby
0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2018-07-19 21:38 UTC (permalink / raw)
To: Pierre-Antoine Rouby; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 4588 bytes --]
On Thu, Jul 19, 2018 at 08:56:03AM +0200, Pierre-Antoine Rouby wrote:
> I trying to import 'gitlab-runner' (https://gitlab.com/gitlab-org/gitlab-runner)
> with tag 'v10.6.0'.
Okay, thanks. I can reproduce the error:
------
$ ./pre-inst-env guix import gopkg https://gitlab.com/gitlab-org/gitlab-runner v10.6.0
Initialized empty Git repository in /tmp/guix-directory.silSgB/.git/
warning: redirecting to https://gitlab.com/gitlab-org/gitlab-runner.git/
remote: Counting objects: 10694, done.
remote: Compressing objects: 100% (8261/8261), done.
remote: Total 10694 (delta 1974), reused 8461 (delta 1826)
Receiving objects: 100% (10694/10694), 29.12 MiB | 2.57 MiB/s, done.
Resolving deltas: 100% (1974/1974), done.
From https://gitlab.com/gitlab-org/gitlab-runner
* tag v10.6.0 -> FETCH_HEAD
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at a3543a2 Update v10.6.0 changelog
Backtrace:
14 (apply-smob/1 #<catch-closure 1af03e0>)
In ice-9/boot-9.scm:
705:2 13 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
619:8 12 (_ #(#(#<directory (guile-user) 1b88140>)))
In guix/ui.scm:
1579:12 11 (run-guix-command _ . _)
In guix/scripts/import.scm:
115:11 10 (guix-import . _)
In guix/scripts/import/gopkg.scm:
85:19 9 (guix-import-gopkg . _)
In guix/utils.scm:
633:8 8 (call-with-temporary-directory #<procedure 1fc0900 at g…>)
In guix/import/gopkg.scm:
343:22 7 (_ "/tmp/guix-directory.silSgB")
322:22 6 (gopkg-dep->packages+dependencies _)
303:22 5 (parse-toml->packages+dependencies _ _ _)
216:16 4 (create-package->packages+dependencies () () "github.c…" …)
47:4 3 (file->hash-base32 #<input: /tmp/guix-directory.silSgB/…>)
In guix/serialization.scm:
343:6 2 (dump #<input: /tmp/guix-directory.silSgB/Gopkg.toml 13>)
275:34 1 (_ _)
In unknown file:
0 (lstat #<input: /tmp/guix-directory.silSgB/Gopkg.toml 13>)
ERROR: In procedure lstat:
Wrong type (expecting string): #<input: /tmp/guix-directory.silSgB/Gopkg.toml 13>
------
It does work with some other packages. I guess the difference is whether
or not the package has a 'Gopkg.toml' file. For example, this works:
------
$ ./pre-inst-env guix import gopkg https://gopkg.in/redsync.v1
Initialized empty Git repository in /tmp/guix-directory.6rwYux/.git/
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 0), reused 10 (delta 0), pack-reused 0
Unpacking objects: 100% (13/13), done.
From https://gopkg.in/redsync.v1
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 98dabdf Merge pull request #2 from rayzyar/master
(define-public go-gopkg-in-redsync-v1
(let ((commit
"98dabdf1c8574561bf976911c14ff652c47b1ddf")
(revision "0"))
(package
(name "go-gopkg-in-redsync-v1")
(version (git-version "0.0.0" revision commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://gopkg.in/redsync.v1.git")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"1ir3xj8rz2igw5ciha2nhkvs55w0yf2k1w5xyizy8d0r4xh3x1p5"))))
(build-system go-build-system)
(arguments
'(#:import-path "gopkg.in/redsync.v1"))
(native-inputs `())
(home-page "https://gopkg.in/redsync.v1")
(synopsis "XXX")
(description "XXX")
(license #f))))
------
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-19 21:38 ` Leo Famulari
@ 2018-07-25 8:48 ` Pierre-Antoine Rouby
2018-07-26 14:12 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Pierre-Antoine Rouby @ 2018-07-25 8:48 UTC (permalink / raw)
To: Leo Famulari; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 351 bytes --]
Hi leo,
> From: "Leo Famulari" <leo@famulari.name>
> On Thu, Jul 19, 2018 at 08:56:03AM +0200, Pierre-Antoine Rouby wrote:
>> I trying to import 'gitlab-runner' (https://gitlab.com/gitlab-org/gitlab-runner)
>> with tag 'v10.6.0'.
>
> Okay, thanks. I can reproduce the error:
I have fix this issue (patch attached), let me know if that work for you.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-gopkg-importer.patch --]
[-- Type: text/x-patch; name=0001-import-Add-gopkg-importer.patch, Size: 21849 bytes --]
From 2f97b62beeacc58935a86d08a1635c082d078189 Mon Sep 17 00:00:00 2001
From: Rouby Pierre-Antoine <pierre-antoine.rouby@inria.fr>
Date: Wed, 25 Jul 2018 10:34:44 +0200
Subject: [PATCH] import: Add gopkg importer.
* guix/import/gopkg.scm: New file.
* guix/scripts/import/gopkg.scm: New file.
* guix/scripts/import.scm: Add 'gopkg'.
* Makefile.am: Add 'gopkg' importer in modules list.
Co-authored-by: Leo Famulari <leo@famulari.name>
---
Makefile.am | 12 +-
doc/guix.texi | 9 +-
guix/import/gopkg.scm | 354 ++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/gopkg.scm | 99 ++++++++++
5 files changed, 469 insertions(+), 7 deletions(-)
create mode 100644 guix/import/gopkg.scm
create mode 100644 guix/scripts/import/gopkg.scm
diff --git a/Makefile.am b/Makefile.am
index 6733f4f89..7d557f0c8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -167,7 +167,7 @@ MODULES = \
guix/build/rpath.scm \
guix/build/cvs.scm \
guix/build/svn.scm \
- guix/build/syscalls.scm \
+ guix/build/syscalls.scm \
guix/build/gremlin.scm \
guix/build/emacs-utils.scm \
guix/build/java-utils.scm \
@@ -185,8 +185,9 @@ MODULES = \
guix/import/cabal.scm \
guix/import/cran.scm \
guix/import/hackage.scm \
- guix/import/elpa.scm \
- guix/import/texlive.scm \
+ guix/import/elpa.scm \
+ guix/import/texlive.scm \
+ guix/import/gopkg.scm \
guix/scripts.scm \
guix/scripts/download.scm \
guix/scripts/perform-download.scm \
@@ -210,8 +211,9 @@ MODULES = \
guix/scripts/import/gnu.scm \
guix/scripts/import/nix.scm \
guix/scripts/import/hackage.scm \
- guix/scripts/import/elpa.scm \
- guix/scripts/import/texlive.scm \
+ guix/scripts/import/elpa.scm \
+ guix/scripts/import/texlive.scm \
+ guix/scripts/import/gopkg.scm \
guix/scripts/environment.scm \
guix/scripts/publish.scm \
guix/scripts/edit.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 84347d156..9b796554c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20,7 +20,7 @@ Copyright @copyright{} 2014, 2015, 2016 Alex Kost@*
Copyright @copyright{} 2015, 2016 Mathieu Lirzin@*
Copyright @copyright{} 2014 Pierre-Antoine Rault@*
Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer@*
-Copyright @copyright{} 2015, 2016, 2017 Leo Famulari@*
+Copyright @copyright{} 2015, 2016, 2017, 2018 Leo Famulari@*
Copyright @copyright{} 2015, 2016, 2017, 2018 Ricardo Wurmus@*
Copyright @copyright{} 2016 Ben Woodcroft@*
Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
@@ -6740,6 +6740,13 @@ Import metadata from the crates.io Rust package repository
@cindex OCaml
Import metadata from the @uref{https://opam.ocaml.org/, OPAM} package
repository used by the OCaml community.
+
+@item gopkg
+@cindex gopkg
+@cindex Golang
+@cindex Go
+Import metadata from the @uref{https://gopkg.in/, gopkg} package
+versioning service used by some Go software.
@end table
The structure of the @command{guix import} code is modular. It would be
diff --git a/guix/import/gopkg.scm b/guix/import/gopkg.scm
new file mode 100644
index 000000000..739261882
--- /dev/null
+++ b/guix/import/gopkg.scm
@@ -0,0 +1,354 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import gopkg)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module ((ice-9 rdelim) #:select (read-line))
+ #:use-module (srfi srfi-11)
+ #:use-module (texinfo string-utils) ; transform-string
+ #:use-module (guix hash)
+ #:use-module (guix base32)
+ #:use-module (guix serialization)
+ #:use-module (guix utils)
+ #:use-module (guix build utils)
+ #:use-module ((guix licenses) #:prefix license:)
+ #:export (gopkg->guix-package))
+
+(define (vcs-file? file stat)
+ (case (stat:type stat)
+ ((directory)
+ (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))
+ ((regular)
+ ;; Git sub-modules have a '.git' file that is a regular text file.
+ (string=? (basename file) ".git"))
+ (else
+ #f)))
+
+(define (file->hash-base32 file)
+ "Return hash of FILE in nix base32 sha256 format. If FILE is a directory,
+exclude vcs files."
+ (let-values (((port get-hash) (open-sha256-port)))
+ (write-file file port #:select? (negate vcs-file?))
+ (force-output port)
+ (bytevector->nix-base32-string (get-hash))))
+
+(define (git->hash url commit file)
+ "Clone git repository and return FILE hash in nix base32 sha256 format."
+ (if (not (file-exists? (string-append file "/.git")))
+ (git-fetch url commit file #:recursive? #f))
+ (file->hash-base32 file))
+
+(define (git-ref->commit path tag)
+ "Return commit number coresponding to git TAG. Return \"XXX\" if tag is not
+found."
+ (define (loop port)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (begin
+ (close-port port)
+ "XXX"))
+ ((string-match tag line) ; Match tag
+ (let ((commit (car (string-split (transform-string line #\tab " ")
+ #\ ))))
+ commit))
+ (else ; Else
+ (loop port)))))
+
+ (let ((file (if (file-exists? (string-append path "/.git/packed-refs"))
+ (string-append path "/.git/packed-refs")
+ (string-append path "/.git/FETCH_HEAD"))))
+ (loop (open-input-file file))))
+
+(define* (git-fetch url commit directory
+ #:key (git-command "git") recursive?)
+ "Fetch COMMIT from URL into DIRECTORY. COMMIT must be a valid Git commit
+identifier. When RECURSIVE? is true, all the sub-modules of URL are fetched,
+recursively. Return #t on success, #f otherwise."
+ (mkdir-p directory)
+
+ (with-directory-excursion directory
+ (invoke git-command "init")
+ (invoke git-command "remote" "add" "origin" url)
+ (if (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
+ (invoke git-command "checkout" "FETCH_HEAD")
+ (begin
+ (invoke git-command "fetch" "origin")
+ (if (not (zero? (system* git-command "checkout" commit)))
+ (let ((commit-hash (git-ref->commit directory commit)))
+ (invoke git-command "checkout" "master")
+ (if (not (equal? "XXX" commit-hash)) ;HACK else stay on master
+ (zero? (system* git-command "checkout" commit-hash))))
+ #t)))))
+
+;;
+;; Append attributes.
+;;
+
+(define (append-inputs inputs name)
+ "Return list with new input corresponding to package NAME."
+ (let ((unquote-name (list 'unquote (string->symbol name))))
+ (append inputs (list (list name unquote-name)))))
+
+;;
+;; Parse attributes.
+;;
+
+(define (url->package-name url)
+ "Compute URL and return package name."
+ (let* ((url-no-slash (string-replace-substring url "/" "-"))
+ (url-no-slash-no-dot (string-replace-substring url-no-slash
+ "." "-")))
+ (string-downcase (string-append "go-" url-no-slash-no-dot))))
+
+(define (cut-url url)
+ "Return URL without protocol prefix and git file extension."
+ (string-replace-substring
+ (cond
+ ((string-match "http://" url)
+ (string-replace-substring url "http://" ""))
+ ((string-match "https://" url)
+ (string-replace-substring url "https://" ""))
+ ((string-match "git://" url)
+ (string-replace-substring url "git://" ""))
+ (else
+ url))
+ ".git" ""))
+
+(define (url->dn url)
+ "Return the web site DN form url 'gnu.org/software/guix' --> 'gnu.org'"
+ (car (string-split url #\/)))
+
+(define (url->git-url url)
+ (string-append "https://" url ".git"))
+
+(define (comment? line)
+ "Return #t if LINE start with comment delimiter, else return #f."
+ (eq? (string-ref (string-trim line) 0) #\#))
+
+(define (empty-line? line)
+ "Return #t if LINE is empty, else #f."
+ (string-null? (string-trim line)))
+
+(define (attribute? line attribute)
+ "Return #t if LINE contain ATTRIBUTE."
+ (equal? (string-trim-right
+ (string-trim
+ (car (string-split line #\=)))) attribute))
+
+(define (attribute-by-name line name)
+ "Return attribute value corresponding to NAME."
+ (let* ((line-no-attribut-name (string-replace-substring
+ line
+ (string-append name " = ") ""))
+ (value-no-double-quote (string-replace-substring
+ line-no-attribut-name
+ "\"" "")))
+ (string-trim value-no-double-quote)))
+
+;;
+;; Packages functions.
+;;
+
+(define (make-go-sexp->package packages dependencies
+ name url version revision
+ commit str-license home-page
+ git-url is-dep? hash)
+ "Create Guix sexp package for Go software NAME. Return new package sexp."
+ (define (package-inputs)
+ (if (not is-dep?)
+ `((native-inputs ,(list 'quasiquote dependencies)))
+ '()))
+
+ (values
+ `(define-public ,(string->symbol name)
+ (let ((commit ,commit)
+ (revision ,revision))
+ (package
+ (name ,name)
+ (version (git-version ,version revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,git-url)
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,hash))))
+ (build-system go-build-system)
+ (arguments
+ '(#:import-path ,url))
+ ,@(package-inputs)
+ (home-page ,home-page)
+ (synopsis "XXX")
+ (description "XXX")
+ (license #f))))))
+
+(define (create-package->packages+dependencies packages dependencies
+ url version directory
+ revision commit
+ constraint? is-dep?)
+ "Return packages and dependencies with new package sexp corresponding to
+URL."
+ (call-with-temporary-directory
+ (lambda (dir)
+ (let ((name (url->package-name url))
+ (home-page (string-append "https://" url))
+ (git-url (url->git-url url))
+ (synopsis "XXX")
+ (description "XXX")
+ (license "XXX"))
+ (let ((hash (git->hash (url->git-url url)
+ commit
+ dir))
+ (commit-hash (if (< (string-length commit) 40)
+ (git-ref->commit dir
+ commit)
+ commit)))
+ (values
+ (append packages
+ (list
+ (make-go-sexp->package packages dependencies
+ name url version
+ revision commit-hash
+ license home-page
+ git-url is-dep? hash)))
+ (if constraint?
+ (append-inputs dependencies name)
+ dependencies)))))))
+
+(define (parse-dependencies->packages+dependencies port constraint?
+ packages dependencies)
+ "Parse one dependencies in PORT, and return packages and dependencies list."
+ (let ((url "XXX")
+ (version "0.0.0")
+ (revision "0")
+ (commit "XXX"))
+ (define (loop port url commit packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (if (not (or (equal? "k8s.io" (url->dn url)) ; HACK bypass k8s
+ (equal? "golang.org" (url->dn url)) ; HACK bypass golang
+ (equal? "cloud.google.com" (url->dn url)))) ; HACK bypass cloud.google
+ (create-package->packages+dependencies packages dependencies
+ url version port revision
+ commit
+ constraint? #t)
+ (values packages dependencies)))
+ ((comment? line) ; Comment
+ (loop port url commit
+ packages dependencies))
+ ((attribute? line "name") ; Name
+ (loop port
+ (attribute-by-name line "name")
+ commit
+ packages dependencies))
+ ((attribute? line "revision") ; Revision
+ (loop port
+ url
+ (attribute-by-name line "revision")
+ packages dependencies))
+ ((attribute? line "version") ; Version
+ (loop port
+ url
+ (attribute-by-name line "version")
+ packages dependencies))
+ ((attribute? line "branch") ; Branch
+ (loop port
+ url
+ (attribute-by-name line "branch")
+ packages dependencies))
+ ((string-match "=" line) ; Other options
+ (loop port url commit
+ packages dependencies))
+ (else (loop port url commit
+ packages dependencies)))))
+ (loop port url commit
+ packages dependencies)))
+
+(define (parse-toml->packages+dependencies port packages dependencies)
+ "Read toml file on PORT and return all dependencies packages sexp and list
+of constraint dependencies."
+ (define (loop port packages dependencies)
+ (let ((line (read-line port)))
+ (cond
+ ((eof-object? line) ; EOF
+ (values packages dependencies))
+ ((empty-line? line) ; Empty line
+ (loop port packages dependencies))
+ ((comment? line) ; Comment
+ (loop port packages dependencies))
+ ((equal? line "[prune]") ; Ignored
+ (loop port packages dependencies))
+ ((equal? "[[constraint]]" line) ; Direct dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #t
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ ((equal? "[[override]]" line) ; Dependencies of dependencies
+ (let-values (((packages dependencies)
+ (parse-dependencies->packages+dependencies port #f
+ packages
+ dependencies)))
+ (loop port packages dependencies)))
+ (else (loop port packages dependencies)))))
+ (loop port packages dependencies))
+
+(define (gopkg-dep->packages+dependencies path)
+ "Open toml file if exist and parse it and return packages sexp and
+dependencies list. Or return two empty list if file not found."
+ (if (file-exists? path)
+ (let ((port (open-input-file path)))
+ (let-values (((packages dependencies)
+ (parse-toml->packages+dependencies port
+ '() '())))
+ (close-port port)
+ (values packages dependencies)))
+ (values '() '())))
+
+;;
+;; Entry point.
+;;
+
+(define (gopkg->guix-package url branch)
+ "Create package for git repository dans branch verison and all dependencies
+sexp packages with Gopkg.toml file."
+ (let ((name (url->package-name (cut-url url)))
+ (version "0.0.0")
+ (revision "0"))
+ (call-with-temporary-directory
+ (lambda (directory)
+ (git-fetch url branch directory #:recursive? #f)
+
+ (let-values (((packages dependencies)
+ (gopkg-dep->packages+dependencies
+ (string-append directory
+ "/Gopkg.toml"))))
+ (let-values (((packages dependencies)
+ (create-package->packages+dependencies packages dependencies
+ (cut-url url) version
+ directory
+ revision branch
+ #f #f)))
+ (values packages)))))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 0b326e104..56b34971e 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -75,7 +75,7 @@ rather than \\n."
;;;
(define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
- "cran" "crate" "texlive" "json" "opam"))
+ "cran" "crate" "texlive" "json" "opam" "gopkg"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/gopkg.scm b/guix/scripts/import/gopkg.scm
new file mode 100644
index 000000000..9a39e58d7
--- /dev/null
+++ b/guix/scripts/import/gopkg.scm
@@ -0,0 +1,99 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import gopkg)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import gopkg)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-gopkg))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import gopkg PACKAGE-URL BRANCH
+Import and convert the Git repository with TOML file to a Guix package
+using PACKAGE-URL and matching BRANCH.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gopkg")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-gopkg . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((package-url branch)
+ (let ((sexp (gopkg->guix-package package-url branch)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ ((package-url)
+ (let ((sexp (gopkg->guix-package package-url "master")))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-url))
+ sexp))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.18.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] Go importer
2018-07-25 8:48 ` Pierre-Antoine Rouby
@ 2018-07-26 14:12 ` Ludovic Courtès
0 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2018-07-26 14:12 UTC (permalink / raw)
To: Pierre-Antoine Rouby; +Cc: guix-devel
Hello!
Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr> skribis:
>> From: "Leo Famulari" <leo@famulari.name>
>> On Thu, Jul 19, 2018 at 08:56:03AM +0200, Pierre-Antoine Rouby wrote:
>>> I trying to import 'gitlab-runner' (https://gitlab.com/gitlab-org/gitlab-runner)
>>> with tag 'v10.6.0'.
>>
>> Okay, thanks. I can reproduce the error:
>
> I have fix this issue (patch attached), let me know if that work for you.
Seems to me that this importer is almost ready. Some comments:
> From 2f97b62beeacc58935a86d08a1635c082d078189 Mon Sep 17 00:00:00 2001
> From: Rouby Pierre-Antoine <pierre-antoine.rouby@inria.fr>
> Date: Wed, 25 Jul 2018 10:34:44 +0200
> Subject: [PATCH] import: Add gopkg importer.
>
> * guix/import/gopkg.scm: New file.
> * guix/scripts/import/gopkg.scm: New file.
> * guix/scripts/import.scm: Add 'gopkg'.
> * Makefile.am: Add 'gopkg' importer in modules list.
Please mention the guix.texi changes as well.
It would be good to have unit tests in tests/go.scm for the Toml parser
and/or for the whole importer, like we do in tests/gem.scm, for
instance.
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -167,7 +167,7 @@ MODULES = \
> guix/build/rpath.scm \
> guix/build/cvs.scm \
> guix/build/svn.scm \
> - guix/build/syscalls.scm \
> + guix/build/syscalls.scm \
Please remove whitespace changes.
> +@item gopkg
> +@cindex gopkg
> +@cindex Golang
> +@cindex Go
> +Import metadata from the @uref{https://gopkg.in/, gopkg} package
> +versioning service used by some Go software.
A few words on limitations (if any ;-)), or mentioning that it’s
recursive by default (right?), and an example would be nice.
> +(define (vcs-file? file stat)
With “TODO: Factorize” please. :-)
> +(define (file->hash-base32 file)
> + "Return hash of FILE in nix base32 sha256 format. If FILE is a directory,
> +exclude vcs files."
> + (let-values (((port get-hash) (open-sha256-port)))
> + (write-file file port #:select? (negate vcs-file?))
Shouldn’t it be #:select? vcs-file? ?
> +(define (git->hash url commit file)
> + "Clone git repository and return FILE hash in nix base32 sha256 format."
> + (if (not (file-exists? (string-append file "/.git")))
> + (git-fetch url commit file #:recursive? #f))
> + (file->hash-base32 file))
> +
> +(define (git-ref->commit path tag)
> + "Return commit number coresponding to git TAG. Return \"XXX\" if tag is not
> +found."
> + (define (loop port)
> + (let ((line (read-line port)))
> + (cond
> + ((eof-object? line) ; EOF
> + (begin
> + (close-port port)
> + "XXX"))
> + ((string-match tag line) ; Match tag
> + (let ((commit (car (string-split (transform-string line #\tab " ")
> + #\ ))))
> + commit))
> + (else ; Else
> + (loop port)))))
> +
> + (let ((file (if (file-exists? (string-append path "/.git/packed-refs"))
> + (string-append path "/.git/packed-refs")
> + (string-append path "/.git/FETCH_HEAD"))))
> + (loop (open-input-file file))))
> +
> +(define* (git-fetch url commit directory
> + #:key (git-command "git") recursive?)
> + "Fetch COMMIT from URL into DIRECTORY. COMMIT must be a valid Git commit
> +identifier. When RECURSIVE? is true, all the sub-modules of URL are fetched,
> +recursively. Return #t on success, #f otherwise."
> + (mkdir-p directory)
> +
> + (with-directory-excursion directory
> + (invoke git-command "init")
> + (invoke git-command "remote" "add" "origin" url)
> + (if (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
> + (invoke git-command "checkout" "FETCH_HEAD")
> + (begin
> + (invoke git-command "fetch" "origin")
> + (if (not (zero? (system* git-command "checkout" commit)))
> + (let ((commit-hash (git-ref->commit directory commit)))
> + (invoke git-command "checkout" "master")
> + (if (not (equal? "XXX" commit-hash)) ;HACK else stay on master
> + (zero? (system* git-command "checkout" commit-hash))))
> + #t)))))
I would highly recommend Guile-Git for all this (or (guix git)) since
it’s already a hard dependency, but we can leave that for later.
> +(define (cut-url url)
> + "Return URL without protocol prefix and git file extension."
> + (string-replace-substring
> + (cond
> + ((string-match "http://" url)
> + (string-replace-substring url "http://" ""))
> + ((string-match "https://" url)
> + (string-replace-substring url "https://" ""))
> + ((string-match "git://" url)
> + (string-replace-substring url "git://" ""))
> + (else
> + url))
> + ".git" ""))
Use (uri-path (string->uri url)) to get the “path” part of the URL. And
then perhaps:
(if (string-suffix? ".git" path)
(string-drop-right path 4)
path)
Because ‘string-replace-substring’ would replace “.git” in the middle of
the URL as well.
> +(define (url->dn url)
> + "Return the web site DN form url 'gnu.org/software/guix' --> 'gnu.org'"
> + (car (string-split url #\/)))
(uri-host (string->uri url))
> +(define (url->git-url url)
> + (string-append "https://" url ".git"))
(uri->string uri)
To me this suggests the code should manipulate URI objects internally
(info "(guile) URIs") rather than URLs as strings.
> +(define (comment? line)
> + "Return #t if LINE start with comment delimiter, else return #f."
> + (eq? (string-ref (string-trim line) 0) #\#))
(string-ref X 0) fails if X is the empty string. So rather:
(string-prefix? "#" (string-trim line))
> +(define (empty-line? line)
> + "Return #t if LINE is empty, else #f."
> + (string-null? (string-trim line)))
Should use ‘string-trim-both’, not ‘string-trim’.
> +(define (attribute? line attribute)
> + "Return #t if LINE contain ATTRIBUTE."
> + (equal? (string-trim-right
> + (string-trim
> + (car (string-split line #\=)))) attribute))
No car please. :-)
(match (string-split (string-trim-both line) #\=)
((key value)
(string=? key attribute))
(_
#f))
> +(define (attribute-by-name line name)
> + "Return attribute value corresponding to NAME."
> + (let* ((line-no-attribut-name (string-replace-substring
> + line
> + (string-append name " = ") ""))
> + (value-no-double-quote (string-replace-substring
> + line-no-attribut-name
> + "\"" "")))
> + (string-trim value-no-double-quote)))
Use ‘match’ along the same lines as above (in fact it’s probably enough
to keep ‘attribute-by-name’ and get rid of ‘attribute?’).
> +(define (make-go-sexp->package packages dependencies
> + name url version revision
> + commit str-license home-page
> + git-url is-dep? hash)
> + "Create Guix sexp package for Go software NAME. Return new package sexp."
> + (define (package-inputs)
> + (if (not is-dep?)
> + `((native-inputs ,(list 'quasiquote dependencies)))
> + '()))
> +
> + (values
> + `(define-public ,(string->symbol name)
> + (let ((commit ,commit)
> + (revision ,revision))
> + (package
> + (name ,name)
> + (version (git-version ,version revision commit))
> + (source (origin
> + (method git-fetch)
> + (uri (git-reference
> + (url ,git-url)
> + (commit commit)))
> + (file-name (git-file-name name version))
> + (sha256
> + (base32
> + ,hash))))
> + (build-system go-build-system)
> + (arguments
> + '(#:import-path ,url))
> + ,@(package-inputs)
> + (home-page ,home-page)
> + (synopsis "XXX")
> + (description "XXX")
> + (license #f))))))
No need for ‘values’ here.
> +(define (create-package->packages+dependencies packages dependencies
> + url version directory
> + revision commit
> + constraint? is-dep?)
> + "Return packages and dependencies with new package sexp corresponding to
> +URL."
Use keyword arguments here and try to explain the important parameters
in the docstring.
> +(define (parse-dependencies->packages+dependencies port constraint?
> + packages dependencies)
The name (same with the procedures above) looks weird. I would call it
either ‘parse-XYZ’ or ‘XYZ->ABC’.
The rest LGTM!
Pierre-Antoine I know you’ll be leaving Inria and Guix-HPC shortly, so
if you don’t get around to it, hopefully one of us will do this final
pass of polishing.
Thank you!
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-07-26 14:13 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-26 16:22 [PATCH 0/1] Go importer Rouby Pierre-Antoine
2018-04-27 7:45 ` [PATCH 1/1] import: Add gopkg importer Rouby Pierre-Antoine
2018-05-02 20:04 ` [PATCH 0/1] Go importer Leo Famulari
2018-06-04 8:18 ` Pierre-Antoine Rouby
2018-07-11 19:04 ` Leo Famulari
2018-07-18 13:11 ` Pierre-Antoine Rouby
2018-07-18 17:07 ` Leo Famulari
2018-07-19 6:56 ` Pierre-Antoine Rouby
2018-07-19 21:38 ` Leo Famulari
2018-07-25 8:48 ` Pierre-Antoine Rouby
2018-07-26 14:12 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).