all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Go importer - help with git-fetch
@ 2019-08-06 10:23 swedebugia
  2019-08-06 10:26 ` Pierre Neidhardt
  2019-08-06 12:56 ` Ricardo Wurmus
  0 siblings, 2 replies; 8+ messages in thread
From: swedebugia @ 2019-08-06 10:23 UTC (permalink / raw)
  To: guix-devel


[-- Attachment #1.1: Type: text/plain, Size: 2595 bytes --]

Hi :)

I'm stuck trying to write a git-fetch -> hash procedure for the go importer.

This is what I am trying to do in pseudocode:

(base32-nix-hash (sha256 (git-fetch url commit temp-path)))

I got this far by using (guix build git):

(display (let* ((name "github.com/gogo/protobuf")
                 (url (go-name->url name))
                 (commit "28a6bbf47e48e0b2220b2a244750b660c83d4942"))
            (with-store store
              (run-with-store store
                (let ((path "test"))
                  (git-fetch url commit path))))))

But I'm pretty new to this derivation 2nd order code stuff and I don't 
understand the errors.

It gives me:

Initialized empty Git repository in /home/egil/src/guix/test2/.git/
remote: Enumerating objects: 924, done.
remote: Counting objects: 100% (924/924), done.
remote: Compressing objects: 100% (745/745), done.
remote: Total 924 (delta 476), reused 330 (delta 132), pack-reused 0
Receiving objects: 100% (924/924), 1.45 MiB | 944.00 KiB/s, done.
Resolving deltas: 100% (476/476), done.
 From https://github.com/gogo/protobuf
  * branch            28a6bbf47e48e0b2220b2a244750b660c83d4942 -> FETCH_HEAD
Note: checking out 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

   git checkout -b <new-branch-name>

HEAD is now at 28a6bbf [merge] Fix merge of non-nullable slices (#569)
Backtrace:
            8 (apply-smob/1 #<catch-closure 1f4e1c0>)
In ice-9/boot-9.scm:
     705:2  7 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
     619:8  6 (_ #(#(#<directory (guile-user) 1fe0140>)))
In ice-9/boot-9.scm:
    2312:4  5 (save-module-excursion _)
   3831:12  4 (_)
In guix/import/go.scm:
    262:11  3 (_)
In guix/store.scm:
    623:10  2 (call-with-store #<procedure 2607960 at guix/import/go.…>)
   1803:24  1 (run-with-store #<store-connection 256.99 29f4ae0> _ # _ …)
In unknown file:
            0 (_ #<store-connection 256.99 29f4ae0>)

ERROR: Wrong type to apply: #t

What does this error mean?

I looked into derivations.scm and found out about a port-sha256 from 
gcrypt - is that usable here to compute the hash?

How do I get it to download to a temporary path?

-- 
Cheers Swedebugia

[-- Attachment #1.2: Type: text/html, Size: 3796 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-import-Add-golang-importer-via-the-Go-search-AP.patch --]
[-- Type: text/x-patch; name="0001-guix-import-Add-golang-importer-via-the-Go-search-AP.patch", Size: 15665 bytes --]

From b7e99f057039e01a327c2f620b7202f7bc762d6c Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Thu, 1 Aug 2019 18:54:11 +0200
Subject: [PATCH] guix: import: Add golang importer via the Go-search API.

* guix/import/github.scm (fetch-readme, fetch-license)
  (fetch-latest-commit, headers, http-url?): Add support for /commits,
  /license, and /readme Github APIv3 endpoints.
  (export): Export fetch-readme, fetch-license & fetch-latest-commit.
  (github-user-slash-repository): Use http-url? for better error reporting.
* guix/import/go.scm: New file.
---
 guix/import/github.scm |  66 +++++++++--
 guix/import/go.scm     | 255 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 311 insertions(+), 10 deletions(-)
 create mode 100644 guix/import/go.scm

diff --git a/guix/import/github.scm b/guix/import/github.scm
index fa23fa4c0..b889da69a 100644
--- a/guix/import/github.scm
+++ b/guix/import/github.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2019 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2019 swedebugia <swedebugia@riseup.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,7 +34,11 @@
   #:use-module (guix upstream)
   #:use-module (guix http-client)
   #:use-module (web uri)
-  #:export (%github-updater))
+  #:export (%github-updater
+            fetch-latest-commit
+            fetch-license
+            latest-released-version
+            fetch-readme))
 
 (define (find-extension url)
   "Return the extension of the archive e.g. '.tar.gz' given a URL, or
@@ -115,19 +120,66 @@ URL of the form 'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
     ((_ owner project . rest)
      (string-append (basename project ".git")))))
 
+(define (http-url? url)
+  ;; We only support Github urls beginning with http.
+  (string-prefix? "http" url))
+
 (define (github-user-slash-repository url)
   "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
 repository separated by a forward slash, from a string URL of the form
 'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
-  (match (string-split (uri-path (string->uri url)) #\/)
-    ((_ owner project . rest)
-     (string-append owner "/" (basename project ".git")))))
+  (if (http-url? url)
+   (match (string-split (uri-path (string->uri url)) #\/)
+     ((_ owner project . rest)
+      (string-append owner "/" (basename project ".git"))))
+   (error "Not a valid url.")))
 
 (define %github-token
   ;; Token to be passed to Github.com to avoid the 60-request per hour
   ;; limit, or #f.
   (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
 
+(define headers
+    ;; Ask for version 3 of the API as suggested at
+    ;; <https://developer.github.com/v3/>.
+    `((Accept . "application/vnd.github.v3+json")
+      (user-agent . "GNU Guile")))
+
+(define (fetch-readme url)
+  "Return a file with the README if any from a github repository url."
+  (let ((readme-url
+         (string-append "https://api.github.com/repos/"
+                        (github-user-slash-repository url)
+                        "/readme")))
+    "Get json, extract and fetch the raw url."
+    (let ((data (json-fetch readme-url #:headers headers)))
+      (http-fetch (assoc-ref data "download_url")))))
+
+(define (fetch-license url)
+  "Return the license json if any from a github repository url. This contains
+the SPDX id among other things."
+  (let ((license-url
+         (string-append "https://api.github.com/repos/"
+                        (github-user-slash-repository url)
+                        "/license")))
+    (json-fetch license-url #:headers headers)))
+
+(define (fetch-latest-commit url)
+  "Get the latest commit-id."
+  (let ((commit-url
+         (string-append "https://api.github.com/repos/"
+                        (github-user-slash-repository url)
+                        "/commits")))
+    ;; This might be able to implement using only match
+    (assoc-ref
+     (match (vector->list (json-fetch commit-url))
+       (()                                ;empty
+        (error "No commits"))
+       ;; Pick the latest one
+       (((_ . x) . _) x)
+       )
+     "sha")))
+
 (define (fetch-releases-or-tags url)
   "Fetch the list of \"releases\" or, if it's empty, the list of tags for the
 repository at URL.  Return the corresponding JSON dictionaries (alists),
@@ -149,12 +201,6 @@ empty list."
                    (github-user-slash-repository url)
                    "/tags"))
 
-  (define headers
-    ;; Ask for version 3 of the API as suggested at
-    ;; <https://developer.github.com/v3/>.
-    `((Accept . "application/vnd.github.v3+json")
-      (user-agent . "GNU Guile")))
-
   (define (decorate url)
     (if (%github-token)
         (string-append url "?access_token=" (%github-token))
diff --git a/guix/import/go.scm b/guix/import/go.scm
new file mode 100644
index 000000000..4f1aee26d
--- /dev/null
+++ b/guix/import/go.scm
@@ -0,0 +1,255 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019 swedebugia <swedebugia@riseup.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import go)
+  #:use-module (ice-9 hash-table)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 textual-ports)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-2) ;and-let
+  #:use-module (guix utils)
+  #:use-module ((guix git-download) #:prefix download:)
+  #:use-module (guix import github)
+  #:use-module (guix import json)
+  #:use-module (guix import utils)
+  #:use-module (guix packages)
+  #:use-module (guix http-client)
+  #:use-module (web uri))
+
+;;; Commentary:
+;;; This utilizes the https://go-search.org/infoapi API.
+;;; This API contains no licenses or versions. We fetch those from github when
+;;; possible.
+
+;;; Code:
+
+(define (go-name->url name)
+  "Takes a go-name on the form github.com/andyleap/go-ssb and turns it into
+https://github.com/andyleap/go-ssb"
+  (string-append "https://" name))
+
+;; from opam.scm - should probably be factored out to utils.scm
+(define (substitute-char str what with)
+  (string-join (string-split str what) with))
+
+(define (go-name->guix-name name)
+  "Takes a go-name e.g. on the form github.com/x/y and turns it into
+go-github-com-x-y"
+  (substitute-char
+   (substitute-char
+    (cond
+     ;;((equal? name "ocamlfind") "ocaml-findlib")
+     ;;((string-prefix? "ocaml" name) name)
+     ((string-prefix? "github.com/" name) (string-append "go-github-com-" (substring name 11)))
+     ((string-prefix? "golang.org/x/" name) (string-append "go-golang-org-" (substring name 11)))
+     ((string-prefix? "cryptoscope.co/go/" name) (string-append "go-cryptoscope-co-" (substring name 11)))
+     (else (string-append "go-" name)))
+    #\_ "-")
+   #\/ "-"))
+;;(display (go-name->guix-name "golang.org/x/text/transform"))
+
+(define (fetch-data name)
+  "Fetches data about imports and description"
+  (json-fetch (string-append "https://go-search.org/api"
+                             "?action=package&id=" name)))
+;;(display (hash-table->alist (fetch-data "golang.org/x/text/transform")))
+
+(define (synopsis name)
+  (and-let* ((data (fetch-data name)))
+    (if (assoc-ref data "Synopsis")
+        (assoc-ref data "Synopsis")
+        ;; If synopsis is empty get the description instead
+        (assoc-ref data "Description"))))
+
+;;(display (synopsis "golang.org/x/text/transform"))
+
+;; Github projects enable us to get the license and readme
+(define (github-url? url)
+  (->bool (string-prefix? "https://github.com/" url)))
+
+(define (string->license name)
+  "Get SPDX-id from github if github-url"
+  (and-let* ((url (go-name->url name))
+            (github-url? url)
+            (data (fetch-license url))
+            (hasht (assoc-ref data "license"))
+            (str (string-downcase (assoc-ref hasht "spdx_id"))))
+    (cond
+     ((equal? str "gpl-3.0") '(license:gpl-3))
+     (else `(,string-append "license:" ,str)))))
+
+;;(display (string->license "github.com/andyleap/go-ssb"))
+
+(define (readme name)
+  "We get the first 1000 characters for the description"
+  (and-let* ((url (go-name->url name))
+            (github-url? url))
+    (get-string-n (fetch-readme url) 1000)))
+
+(define (description name)
+  (and-let* ((data (fetch-data name)))
+    (if (assoc-ref data "Synopsis")
+        ;; Synopsis is non-empty.
+        (if (assoc-ref data "Description")
+            (assoc-ref data "Description")
+            ;; Description is empty
+            (readme name))
+        ;; Synopsis is empty and the description from GSAPI has been used as
+        ;; synopsis, get the readme instead
+        (readme name))))
+
+;;(display (description "golang.org/x/text/transform"))
+
+;; Versions are tricky because the go-ecosystem does not rely on them at
+;; all. We get the latest released or tagged version from github and fall
+;; backto the latest commit.
+(define (version name)
+  "Get the latest release or tag if any."
+  (and-let* ((url (go-name->url name))
+             (github-url? url))
+    (latest-released-version url name)))
+
+;;(display (version "github.com/andyleap/go-ssb"))
+
+(define (commit name)
+  "Get latest commit-id"
+  (and-let* ((url (go-name->url name))
+             (github-url? url))
+    (fetch-latest-commit url)))
+
+;;(display (commit "github.com/andyleap/go-ssb"))
+
+(define (dependencies name)
+  (and-let* ((data (fetch-data name)))
+    ;; Join with (assoc-ref data "TestImports")?
+    (assoc-ref data "Imports")))
+
+;;(display (dependencies "golang.org/x/text/transform"))
+
+(define (test-dependencies name)
+  (and-let* ((data (fetch-data name)))
+    ;; Join with (assoc-ref data "TestImports")?
+    (assoc-ref data "TestImports")))
+
+;; this is from ocaml.scm
+(define (dependencies->inputs dependencies)
+  "Transform the list of dependencies in a list of inputs."
+  (if (not dependencies)
+    '()
+    (map (lambda (input)
+           (list input (list 'unquote (string->symbol input))))
+         (map go-name->guix-name dependencies))))
+
+;;(display (dependencies->inputs (dependencies "github.com/andyleap/go-ssb")))
+
+(define (go->guix-package name)
+  (let ((version (version name)))
+    (if (equal? version #t)
+        ;; Got release or tag
+        (let ((source-url (go-name->url name))
+              (commit version)
+              (inputs (dependencies->inputs (dependencies name)))
+              (synopsis (synopsis name))
+              (description (description name)))
+          ;; This is broken because of git-fetch from git-download does not at
+          ;; all work like the similar url-fetch-procedure.
+          (call-with-temporary-output-file
+           (lambda (temp port)
+             (and (download:git-fetch (download:git-reference
+                                       (url source-url)
+                                       (commit commit))
+                                      "sha256" "dummy-hash")
+                  `(package
+                     (name ,(go-name->guix-name name))
+                     (version ,version)
+                     (source
+                      (origin
+                        (method git-fetch)
+                        (uri (git-reference
+                              (url ,source-url)
+                              (commit ,commit)))
+                        (file-name (git-file-name name version))
+                        (sha256 (base32 ,(guix-hash-url temp)))))
+                     (build-system go-build-system)
+                     ,@(if (null? inputs)
+                           '()
+                           `((inputs ,(list 'quasiquote inputs))))
+                     (home-page ,source-url)
+                     (synopsis ,synopsis)
+                     (description ,description)
+                     (license ,@(string->license name)))))))
+        ;; No release or tag, fall back to latest commit
+        (let ((source-url (go-name->url name))
+              (commit (commit name))
+              (inputs (dependencies->inputs (dependencies name)))
+              (synopsis (synopsis name))
+              (description (description name)))
+          ;; This is broken because of git-fetch from git-download does not at
+          ;; all work like the similar url-fetch-procedure.
+          (call-with-temporary-output-file
+           (lambda (temp port)
+             (and (download:git-fetch (download:git-reference
+                                       (url source-url)
+                                       (commit commit))
+                                      "sha256" "dummy-hash")
+                  `(package
+                     (name ,(go-name->guix-name name))
+                     (version ,version)
+                     (source
+                      (origin
+                        (method git-fetch)
+                        (uri (git-reference
+                              (url ,source-url)
+                              (commit ,commit)))
+                        (file-name (git-file-name name version))
+                        (sha256 (base32 ,(guix-hash-url temp)))))
+                     (build-system go-build-system)
+                     ,@(if (null? inputs)
+                           '()
+                           `((inputs ,(list 'quasiquote inputs))))
+                     (home-page ,source-url)
+                     (synopsis ,synopsis)
+                     (description ,description)
+                     (license ,@(string->license name))))))))))
+
+(go->guix-package "github.com/gogo/protobuf")
+
+;; Debug
+#;
+(display (let* ((name "github.com/gogo/protobuf")
+               (url (go-name->url name))
+               (commit name))
+           (call-with-temporary-output-file
+           (lambda (temp port)
+             (download:git-fetch (download:git-reference
+                                  (url url)
+                                  (commit commit))
+                                 "sha256" commit temp)))))
+
+
+#;
+(display (let* ((name "github.com/gogo/protobuf")
+               (url (go-name->url name))
+               (commit name))
+           (download:git-fetch (download:git-reference
+                                (url url)
+                                (commit commit))
+"sha256" commit name)))
+
+;; Both of these returns procedures like this:#<procedure 1b72f50 at guix/git-download.scm:145:2 (state)>
+;; How do we get it to return something we can compute the hash from?
-- 
2.19.2


^ permalink raw reply related	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2019-08-06 22:03 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-08-06 10:23 Go importer - help with git-fetch swedebugia
2019-08-06 10:26 ` Pierre Neidhardt
2019-08-06 12:45   ` swedebugia
2019-08-06 12:56 ` Ricardo Wurmus
2019-08-06 13:04   ` Jakob L. Kreuze
2019-08-06 20:26     ` swedebugia
2019-08-06 20:46       ` Jakob L. Kreuze
2019-08-06 22:02       ` Ricardo Wurmus

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.