Guix & IPFS

Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

Hi Guix!

I'm in the process of packaging IPFS.
It has about 100 dependencies, most of which are currently missing in
Guix, so that's gonna take me some hours :p
Thankfully Go is rather straightforward to package so that should be
alright.

That said, the case of IPFS of particular: it uses it's own dependency
manager "gx" which retrieves the specific versions of its deps over the
IFPS protocol.

The build recipe is as follows (from the project's Rules.mk):

--8<---------------cut here---------------start------------->8---
install_unsupported:
	@echo "note: this command has yet to be tested to build in the system you are using"
	@echo "installing gx"
	go get -v -u github.com/whyrusleeping/gx
	go get -v -u github.com/whyrusleeping/gx-go
	@echo check gx and gx-go
	gx -v && gx-go -v
	@echo downloading dependencies
	gx install --global
	@echo "installing go-ipfs"
	go install -v -tags nofuse ./cmd/ipfs
--8<---------------cut here---------------end--------------->8---

This raises a few questions:

- IPFS is rather peculiar about its dependency versions.
If I package all those dependencies in Guix, I should use the same
versions.  But then we should make sure that no one is going to update
those packages independently, which would break the strict versioning of
IPFS.

Else I could package all those deps as usual (i.e. the latest version) and then for
the IPFS package I would use the native inputs as follows:

  ("go-path" ,(package (inherit go-path)
                       (version "1.1.9")
                       (source (origin...
                                  (base32 <new-hash-here>)))))

I should probably define a function of PACKAGE VERSION BASE32 arguments.

- gx retrieves the deps over IPFS: how important is this?  Should we do
  the same?  What about using "gx"?  We could also implement a new
  "gx-fetch" method fully written in Guile.

Thoughts?

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

On Mon, Oct 15, 2018 at 09:59:04AM +0200, Pierre Neidhardt wrote:
> Hi Guix!
> 
> I'm in the process of packaging IPFS.
> It has about 100 dependencies, most of which are currently missing in
> Guix, so that's gonna take me some hours :p
> Thankfully Go is rather straightforward to package so that should be
> alright.
> 
> That said, the case of IPFS of particular: it uses it's own dependency
> manager "gx" which retrieves the specific versions of its deps over the
> IFPS protocol.
> 
> The build recipe is as follows (from the project's Rules.mk):
> 
> --8<---------------cut here---------------start------------->8---
> install_unsupported:
> 	@echo "note: this command has yet to be tested to build in the system you are using"
> 	@echo "installing gx"
> 	go get -v -u github.com/whyrusleeping/gx
> 	go get -v -u github.com/whyrusleeping/gx-go
> 	@echo check gx and gx-go
> 	gx -v && gx-go -v
> 	@echo downloading dependencies
> 	gx install --global
> 	@echo "installing go-ipfs"
> 	go install -v -tags nofuse ./cmd/ipfs
> --8<---------------cut here---------------end--------------->8---
> 
> This raises a few questions:
> 
> - IPFS is rather peculiar about its dependency versions.
> If I package all those dependencies in Guix, I should use the same
> versions.  But then we should make sure that no one is going to update
> those packages independently, which would break the strict versioning of
> IPFS.

I think a guix channel is the appropriate way to manage these versions
(channels are all about versions in my opinion) and I can provide a
binary substitute server. Still we can merge current versions in
mainline.

Pj.

Re: Guix & IPFS

[Pjotr Prins]

On Mon, Oct 15, 2018 at 09:59:04AM +0200, Pierre Neidhardt wrote:
> Hi Guix!
> 
> I'm in the process of packaging IPFS.
> It has about 100 dependencies, most of which are currently missing in
> Guix, so that's gonna take me some hours :p
> Thankfully Go is rather straightforward to package so that should be
> alright.
> 
> That said, the case of IPFS of particular: it uses it's own dependency
> manager "gx" which retrieves the specific versions of its deps over the
> IFPS protocol.

One thing gx does is make git a true blockchain with interesting
properties. In particular you can not edit git history and sharing
becomes more trust worthy. Distributed sharing will become possible
too.

  https://medium.com/@mycoralhealth/learn-to-securely-share-files-on-the-blockchain-with-ipfs-219ee47df54c

We should certainly consider it. Even for our own repos.

Pj.

Re: Guix & IPFS

[Ludovic Courtès]

Hello!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I'm in the process of packaging IPFS.
> It has about 100 dependencies, most of which are currently missing in
> Guix, so that's gonna take me some hours :p

FWIW Pierre-Antoine (who was working with me as an intern on Guix-HPC)
has given me the last revision of the gopkg importer, which I’m planning
to include soon.  It might be useful to you!

> - IPFS is rather peculiar about its dependency versions.
> If I package all those dependencies in Guix, I should use the same
> versions.  But then we should make sure that no one is going to update
> those packages independently, which would break the strict versioning of
> IPFS.

This seems to be the same for all Go packages: they specify the exact
commit of each of their dependencies, AIUI.

In practice it may be that upgrading will often be fine, but it does
mean that every time we upgrade a Go package, we have to build
everything “guix refresh -l GO-PACKAGE” reports to make sure.

> - gx retrieves the deps over IPFS: how important is this?  Should we do
>   the same?  What about using "gx"?  We could also implement a new
>   "gx-fetch" method fully written in Guile.

In Guix dependencies are listed statically in the ‘inputs’ fields, so I
suppose we have to run it once to retrieve that list and then “encode”
it as ‘inputs’, no?

HTH,
Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1929 bytes --]


> FWIW Pierre-Antoine (who was working with me as an intern on Guix-HPC)
> has given me the last revision of the gopkg importer, which I’m planning
> to include soon.  It might be useful to you!

Absolutely!  Could you share the code now so that I can start using it?  Thanks!

> This seems to be the same for all Go packages: they specify the exact
> commit of each of their dependencies, AIUI.

No, Go programs typically refer only to repositories, not to their commit.  This
is precisely why it's a problem.  Golang's dependency management was not very
well designed I'm afraid.

Misunderstanding?

> In practice it may be that upgrading will often be fine, but it does
> mean that every time we upgrade a Go package, we have to build
> everything “guix refresh -l GO-PACKAGE” reports to make sure.

Yup, but it's also the case with any other package, isn't it?

> > - gx retrieves the deps over IPFS: how important is this?  Should we do
> >   the same?  What about using "gx"?  We could also implement a new
> >   "gx-fetch" method fully written in Guile.
>
> In Guix dependencies are listed statically in the ‘inputs’ fields, so I
> suppose we have to run it once to retrieve that list and then “encode”
> it as ‘inputs’, no?

Sorry, I meant "ipfs-fetch", so we might be talking about different things :p

- Network: it would be nice to be able to retrieve packages over IPFS.  We can
  do this without IPFS, just like "gx" does.

- Importer: I don't know how Pierre-Antoine retrieves dependencies, but
  technically it's enough to download the package and run some "go ..." command
  to list deps.  In the case of go-ipfs (and possibly other Go packages using
  gx), we can only list the deps with "gx deps...".  Note that in both bases the
  package does not have to be built, if I'm not mistaken.

Makes sense?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Gábor Boskovits]

Pierre Neidhardt <mail@ambrevar.xyz> ezt írta (időpont: 2018. okt.
15., H, 12:40):
>
>
> > FWIW Pierre-Antoine (who was working with me as an intern on Guix-HPC)
> > has given me the last revision of the gopkg importer, which I’m planning
> > to include soon.  It might be useful to you!
>
> Absolutely!  Could you share the code now so that I can start using it?  Thanks!
>
> > This seems to be the same for all Go packages: they specify the exact
> > commit of each of their dependencies, AIUI.
>
> No, Go programs typically refer only to repositories, not to their commit.  This
> is precisely why it's a problem.  Golang's dependency management was not very
> well designed I'm afraid.
>

I'm not much into go, but it seems, that most of the times it is safe
to use another version of dependencies.

> Misunderstanding?
>
> > In practice it may be that upgrading will often be fine, but it does
> > mean that every time we upgrade a Go package, we have to build
> > everything “guix refresh -l GO-PACKAGE” reports to make sure.
>
> Yup, but it's also the case with any other package, isn't it?
>

Sometimes it is considered safe to assume that a given update does not break
reverse dependencies, but this might lead to unexpected breakage nevertheless.
Sometimes building all revdeps are prohibitively resource consuming.

> > > - gx retrieves the deps over IPFS: how important is this?  Should we do
> > >   the same?  What about using "gx"?  We could also implement a new
> > >   "gx-fetch" method fully written in Guile.
> >
> > In Guix dependencies are listed statically in the ‘inputs’ fields, so I
> > suppose we have to run it once to retrieve that list and then “encode”
> > it as ‘inputs’, no?
>
> Sorry, I meant "ipfs-fetch", so we might be talking about different things :p
>
> - Network: it would be nice to be able to retrieve packages over IPFS.  We can
>   do this without IPFS, just like "gx" does.
>

This seems to be orthogonal to the other issues mentioned here, and I believe it
should be implemented orthogonally (i.e. have it as an option to all,
even unrelated packages).
It would be great if integration with guix publish would be possible.

> - Importer: I don't know how Pierre-Antoine retrieves dependencies, but
>   technically it's enough to download the package and run some "go ..." command
>   to list deps.  In the case of go-ipfs (and possibly other Go packages using
>   gx), we can only list the deps with "gx deps...".  Note that in both bases the
>   package does not have to be built, if I'm not mistaken.
>
> Makes sense?
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

> I'm not much into go, but it seems, that most of the times it is safe
> to use another version of dependencies.

This is unrelated to Go: it's safe as long as there is no API breakage.  My
understanding of the IPFS project is that they don't want to make that kind of
assumptions.

> This seems to be orthogonal to the other issues mentioned here, and I believe it
> should be implemented orthogonally (i.e. have it as an option to all,
> even unrelated packages).
> It would be great if integration with guix publish would be possible.

Absolutely.  Just to be clear, the reason I've mentioned both in the same thread
is because "gx" does both: managed versioned dependencies over IPFS.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2308 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> FWIW Pierre-Antoine (who was working with me as an intern on Guix-HPC)
>> has given me the last revision of the gopkg importer, which I’m planning
>> to include soon.  It might be useful to you!
>
> Absolutely!  Could you share the code now so that I can start using it?  Thanks!

Attached!  Leo reviewed an earlier version of this code already.

>> This seems to be the same for all Go packages: they specify the exact
>> commit of each of their dependencies, AIUI.
>
> No, Go programs typically refer only to repositories, not to their commit.  This
> is precisely why it's a problem.  Golang's dependency management was not very
> well designed I'm afraid.

I was referring to the “gopkg” way of handling dependencies (with those
“.toml” files), which is what the importer implements.

Now, there are apparently several dependency management tools in use, so
I guess my comment only applies to the gopkg way, but I’m not well
versed into Go.

>> In practice it may be that upgrading will often be fine, but it does
>> mean that every time we upgrade a Go package, we have to build
>> everything “guix refresh -l GO-PACKAGE” reports to make sure.
>
> Yup, but it's also the case with any other package, isn't it?

Of course :-), but I mean it’s even more important here due to the way
upstream handles this (no semantic versioning and in fact, no releases.)

> Sorry, I meant "ipfs-fetch", so we might be talking about different things :p

Oh OK.  ‘ipfs-fetch’ surely makes a lot of sense.  (What makes even more
sense is substitutes over IPFS!)

> - Network: it would be nice to be able to retrieve packages over IPFS.  We can
>   do this without IPFS, just like "gx" does.

Sure.

> - Importer: I don't know how Pierre-Antoine retrieves dependencies, but
>   technically it's enough to download the package and run some "go ..." command
>   to list deps.  In the case of go-ipfs (and possibly other Go packages using
>   gx), we can only list the deps with "gx deps...".  Note that in both bases the
>   package does not have to be built, if I'm not mistaken.

The importer parses ‘Gopkg.toml’ files, which may or may not be useful
for IPFS and its dependencies.

Ludo’.


[-- Attachment #2: 0001-import-Add-gopkg-importer.patch --]
[-- Type: text/x-patch, Size: 22155 bytes --]

From 96ba90097f3b4a7f502c826d9a0385472870a5c9 Mon Sep 17 00:00:00 2001
From: Pierre-Antoine Rouby <contact@parouby.fr>
Date: Sun, 9 Sep 2018 16:44:53 +0200
Subject: [PATCH] import: Add gopkg importer.

* Makefile.am: Add 'gopkg' importer in modules list.
* doc/guix.texi: Add 'gopkg' importer documentation.
* guix/import/gopkg.scm: New file.
* guix/scripts/import.scm: Add 'gopkg'.
* guix/scripts/import/gopkg.scm: New file.
---
 Makefile.am                   |  10 +-
 doc/guix.texi                 |   9 +-
 guix/import/gopkg.scm         | 356 ++++++++++++++++++++++++++++++++++
 guix/scripts/import.scm       |   2 +-
 guix/scripts/import/gopkg.scm |  99 ++++++++++
 5 files changed, 470 insertions(+), 6 deletions(-)
 create mode 100644 guix/import/gopkg.scm
 create mode 100644 guix/scripts/import/gopkg.scm

diff --git a/Makefile.am b/Makefile.am
index 5c8639d66..26a86c9ad 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -187,8 +187,9 @@ MODULES =					\
   guix/import/cabal.scm				\
   guix/import/cran.scm				\
   guix/import/hackage.scm			\
-  guix/import/elpa.scm   			\
-  guix/import/texlive.scm   			\
+  guix/import/elpa.scm				\
+  guix/import/texlive.scm			\
+  guix/import/gopkg.scm				\
   guix/scripts.scm				\
   guix/scripts/download.scm			\
   guix/scripts/perform-download.scm		\
@@ -213,8 +214,9 @@ MODULES =					\
   guix/scripts/import/gnu.scm			\
   guix/scripts/import/nix.scm			\
   guix/scripts/import/hackage.scm		\
-  guix/scripts/import/elpa.scm  		\
-  guix/scripts/import/texlive.scm  		\
+  guix/scripts/import/elpa.scm			\
+  guix/scripts/import/texlive.scm		\
+  guix/scripts/import/gopkg.scm			\
   guix/scripts/environment.scm			\
   guix/scripts/publish.scm			\
   guix/scripts/edit.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 19a497c74..ab866b778 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20,7 +20,7 @@ Copyright @copyright{} 2014, 2015, 2016 Alex Kost@*
 Copyright @copyright{} 2015, 2016 Mathieu Lirzin@*
 Copyright @copyright{} 2014 Pierre-Antoine Rault@*
 Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer@*
-Copyright @copyright{} 2015, 2016, 2017 Leo Famulari@*
+Copyright @copyright{} 2015, 2016, 2017, 2018 Leo Famulari@*
 Copyright @copyright{} 2015, 2016, 2017, 2018 Ricardo Wurmus@*
 Copyright @copyright{} 2016 Ben Woodcroft@*
 Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
@@ -7049,6 +7049,13 @@ Import metadata from the crates.io Rust package repository
 @cindex OCaml
 Import metadata from the @uref{https://opam.ocaml.org/, OPAM} package
 repository used by the OCaml community.
+
+@item gopkg
+@cindex gopkg
+@cindex Golang
+@cindex Go
+Import metadata from the @uref{https://gopkg.in/, gopkg} package
+versioning service used by some Go software.
 @end table
 
 The structure of the @command{guix import} code is modular.  It would be
diff --git a/guix/import/gopkg.scm b/guix/import/gopkg.scm
new file mode 100644
index 000000000..204bf18ab
--- /dev/null
+++ b/guix/import/gopkg.scm
@@ -0,0 +1,356 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import gopkg)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module ((ice-9 rdelim) #:select (read-line))
+  #:use-module (srfi srfi-11)
+  #:use-module (texinfo string-utils) ; transform-string
+  #:use-module (gcrypt hash)
+  ;; #:use-module (guix hash)
+  #:use-module (guix base32)
+  #:use-module (guix serialization)
+  #:use-module (guix utils)
+  #:use-module (guix build utils)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:export (gopkg->guix-package))
+
+(define (vcs-file? file stat)
+  ;; TODO: Factorize
+  (case (stat:type stat)
+    ((directory)
+     (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))
+    ((regular)
+     ;; Git sub-modules have a '.git' file that is a regular text file.
+     (string=? (basename file) ".git"))
+    (else
+     #f)))
+
+(define (file->hash-base32 file)
+  "Return hash of FILE in nix base32 sha256 format.  If FILE is a directory,
+exclude vcs files."
+  (let-values (((port get-hash) (open-sha256-port)))
+    (write-file file port #:select? (negate vcs-file?))
+    (force-output port)
+    (bytevector->nix-base32-string (get-hash))))
+
+(define (git->hash url commit file)
+  "Clone git repository and return FILE hash in nix base32 sha256 format."
+  (if (not (file-exists? (string-append file "/.git")))
+      (git-fetch url commit file #:recursive? #f))
+  (file->hash-base32 file))
+
+(define (git-ref->commit path tag)
+  "Return commit number coresponding to git TAG.  Return \"XXX\" if tag is not
+found."
+  (define (loop port)
+    (let ((line (read-line port)))
+      (cond
+       ((eof-object? line)              ; EOF
+        (begin
+          (close-port port)
+          "XXX"))
+       ((string-match tag line)         ; Match tag
+        (let ((commit (car (string-split (transform-string line #\tab " ")
+                                         #\ ))))
+          commit))
+       (else                            ; Else
+        (loop port)))))
+
+  (let ((file (if (file-exists? (string-append path "/.git/packed-refs"))
+                  (string-append path "/.git/packed-refs")
+                  (string-append path "/.git/FETCH_HEAD"))))
+    (loop (open-input-file file))))
+
+(define* (git-fetch url commit directory
+                    #:key (git-command "git") recursive?)
+  "Fetch COMMIT from URL into DIRECTORY.  COMMIT must be a valid Git commit
+identifier.  When RECURSIVE? is true, all the sub-modules of URL are fetched,
+recursively.  Return #t on success, #f otherwise."
+  (mkdir-p directory)
+  
+  (with-directory-excursion directory
+    (invoke git-command "init")
+    (invoke git-command "remote" "add" "origin" url)
+    (if (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
+        (invoke git-command "checkout" "FETCH_HEAD")
+        (begin
+          (invoke git-command "fetch" "origin")
+          (if (not (zero? (system* git-command "checkout" commit)))
+              (let ((commit-hash (git-ref->commit directory commit)))
+                (invoke git-command "checkout" "master")
+                (if (not (equal? "XXX" commit-hash)) ;HACK else stay on master
+                    (zero? (system* git-command "checkout" commit-hash))))
+              #t)))))
+
+;;
+;; Append attributes.
+;;
+
+(define (append-inputs inputs name)
+  "Return list with new input corresponding to package NAME."
+  (let ((unquote-name (list 'unquote (string->symbol name))))
+    (append inputs (list (list name unquote-name)))))
+
+;;
+;; Parse attributes.
+;;
+
+(define (url->package-name url)
+  "Compute URL and return package name."
+  (let* ((url-no-slash (string-replace-substring url "/" "-"))
+         (url-no-slash-no-dot (string-replace-substring url-no-slash
+                                                        "." "-")))
+    (string-downcase (string-append "go-" url-no-slash-no-dot))))
+
+(define (cut-url url)
+  "Return URL without protocol prefix and git file extension."
+  (string-replace-substring
+   (cond
+    ((string-match "http://"  url)
+     (string-replace-substring url "http://" ""))
+    ((string-match "https://" url)
+     (string-replace-substring url "https://" ""))
+    ((string-match "git://"   url)
+     (string-replace-substring url "git://" ""))
+    (else
+     url))
+   ".git" ""))
+
+(define (url->dn url)
+  "Return the web site DN form url 'gnu.org/software/guix' --> 'gnu.org'"
+  (car (string-split url #\/)))
+
+(define (url->git-url url)
+  (string-append "https://" url ".git"))
+
+(define (comment? line)
+  "Return #t if LINE start with comment delimiter, else return #f."
+  (eq? (string-ref (string-trim line) 0) #\#))
+
+(define (empty-line? line)
+  "Return #t if LINE is empty, else #f."
+  (string-null? (string-trim line)))
+
+(define (attribute? line attribute)
+  "Return #t if LINE contain ATTRIBUTE."
+  (equal? (string-trim-right
+           (string-trim
+            (car (string-split line #\=)))) attribute))
+
+(define (attribute-by-name line name)
+  "Return attribute value corresponding to NAME."
+  (let* ((line-no-attribut-name (string-replace-substring
+                                 line
+                                 (string-append name " = ") ""))
+         (value-no-double-quote (string-replace-substring
+                                 line-no-attribut-name
+                                 "\"" "")))
+    (string-trim value-no-double-quote)))
+
+;;
+;; Packages functions.
+;;
+
+(define (make-go-sexp->package packages dependencies
+                               name url version revision
+                               commit str-license home-page
+                               git-url is-dep? hash)
+  "Create Guix sexp package for Go software NAME. Return new package sexp."
+  (define (package-inputs)
+    (if (not is-dep?)
+        `((native-inputs ,(list 'quasiquote dependencies)))
+        '()))
+
+  (values
+   `(define-public ,(string->symbol name)
+      (let ((commit ,commit)
+            (revision ,revision))
+        (package
+          (name ,name)
+          (version (git-version ,version revision commit))
+          (source (origin
+                    (method git-fetch)
+                    (uri (git-reference
+                          (url ,git-url)
+                          (commit commit)))
+                    (file-name (git-file-name name version))
+                    (sha256
+                     (base32
+                      ,hash))))
+          (build-system go-build-system)
+          (arguments
+           '(#:import-path ,url))
+          ,@(package-inputs)
+          (home-page ,home-page)
+          (synopsis "XXX")
+          (description "XXX")
+          (license #f))))))
+
+(define (create-package->packages+dependencies packages dependencies
+                                               url version directory
+                                               revision commit
+                                               constraint? is-dep?)
+  "Return packages and dependencies with new package sexp corresponding to
+URL."
+  (call-with-temporary-directory
+   (lambda (dir)
+     (let ((name      (url->package-name url))
+           (home-page (string-append "https://" url))
+           (git-url   (url->git-url url))
+           (synopsis    "XXX")
+           (description "XXX")
+           (license     "XXX"))
+       (let ((hash (git->hash (url->git-url url)
+                              commit
+                              dir))
+             (commit-hash (if (< (string-length commit) 40)
+                              (git-ref->commit dir
+                                               commit)
+                              commit)))
+         (values
+          (append packages
+                  (list
+                   (make-go-sexp->package packages dependencies
+                                          name url version
+                                          revision commit-hash
+                                          license home-page
+                                          git-url is-dep? hash)))
+          (if constraint?
+              (append-inputs dependencies name)
+              dependencies)))))))
+
+(define (parse-dependencies->packages+dependencies port constraint?
+                                                   packages dependencies)
+  "Parse one dependencies in PORT, and return packages and dependencies list."
+  (let ((url "XXX")
+        (version "0.0.0")
+        (revision "0")
+        (commit "XXX"))
+    (define (loop port url commit packages dependencies)
+      (let ((line (read-line port)))
+        (cond
+         ((eof-object? line)            ; EOF
+          (values packages dependencies))
+         ((empty-line? line)                               ; Empty line
+          (if (not (or (equal? "k8s.io" (url->dn url))     ; HACK bypass k8s
+                       (equal? "golang.org" (url->dn url)) ; HACK bypass golang
+                       (equal? "cloud.google.com" (url->dn url)))) ; HACK bypass cloud.google
+              (create-package->packages+dependencies packages dependencies
+                                                     url version port revision
+                                                     commit
+                                                     constraint? #t)
+              (values packages dependencies)))
+         ((comment? line)               ; Comment
+          (loop port url commit
+                packages dependencies))
+         ((attribute? line "name")      ; Name
+          (loop port
+                (attribute-by-name line "name")
+                commit
+                packages dependencies))
+         ((attribute? line "revision")  ; Revision
+          (loop port
+                url
+                (attribute-by-name line "revision")
+                packages dependencies))
+         ((attribute? line "version")   ; Version
+          (loop port
+                url
+                (attribute-by-name line "version")
+                packages dependencies))
+         ((attribute? line "branch")    ; Branch
+          (loop port
+                url
+                (attribute-by-name line "branch")
+                packages dependencies))
+         ((string-match "=" line)       ; Other options
+          (loop port url commit
+                packages dependencies))
+         (else (loop port url commit
+                     packages dependencies)))))
+    (loop port url commit
+          packages dependencies)))
+
+(define (parse-toml->packages+dependencies port packages dependencies)
+  "Read toml file on PORT and return all dependencies packages sexp and list
+of constraint dependencies."
+  (define (loop port packages dependencies)
+    (let ((line (read-line port)))
+      (cond
+       ((eof-object? line)              ; EOF
+        (values packages dependencies))
+       ((empty-line? line)              ; Empty line
+        (loop port packages dependencies))
+       ((comment? line)                 ; Comment
+        (loop port packages dependencies))
+       ((equal? line "[prune]")         ; Ignored
+        (loop port packages dependencies))
+       ((equal? "[[constraint]]" line)  ; Direct dependencies
+        (let-values (((packages dependencies)
+                      (parse-dependencies->packages+dependencies port #t
+                                                                 packages
+                                                                 dependencies)))
+          (loop port packages dependencies)))
+       ((equal? "[[override]]" line)    ; Dependencies of dependencies
+        (let-values (((packages dependencies)
+                      (parse-dependencies->packages+dependencies port #f
+                                                                 packages
+                                                                 dependencies)))
+          (loop port packages dependencies)))
+       (else (loop port packages dependencies)))))
+  (loop port packages dependencies))
+
+(define (gopkg-dep->packages+dependencies path)
+  "Open toml file if exist and parse it and return packages sexp and
+dependencies list. Or return two empty list if file not found."
+  (if (file-exists? path)
+      (let ((port (open-input-file path)))
+        (let-values (((packages dependencies)
+                      (parse-toml->packages+dependencies port
+                                                         '() '())))
+          (close-port port)
+          (values packages dependencies)))
+      (values '() '())))
+
+;;
+;; Entry point.
+;;
+
+(define (gopkg->guix-package url branch)
+  "Create package for git repository dans branch verison and all dependencies
+sexp packages with Gopkg.toml file."
+  (let ((name (url->package-name (cut-url url)))
+        (version "0.0.0")
+        (revision "0"))
+    (call-with-temporary-directory
+     (lambda (directory)
+       (git-fetch url branch directory #:recursive? #f)
+
+       (let-values (((packages dependencies)
+                     (gopkg-dep->packages+dependencies
+                      (string-append directory
+                                     "/Gopkg.toml"))))
+         (let-values (((packages dependencies)
+                       (create-package->packages+dependencies packages dependencies
+                                                              (cut-url url) version
+                                                              directory
+                                                              revision branch
+                                                              #f #f)))
+           (values packages)))))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 0b326e104..56b34971e 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -75,7 +75,7 @@ rather than \\n."
 ;;;
 
 (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
-                    "cran" "crate" "texlive" "json" "opam"))
+                    "cran" "crate" "texlive" "json" "opam" "gopkg"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/gopkg.scm b/guix/scripts/import/gopkg.scm
new file mode 100644
index 000000000..9a39e58d7
--- /dev/null
+++ b/guix/scripts/import/gopkg.scm
@@ -0,0 +1,99 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import gopkg)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import gopkg)
+  #:use-module (guix scripts import)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 format)
+  #:export (guix-import-gopkg))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import gopkg PACKAGE-URL BRANCH
+Import and convert the Git repository with TOML file to a Guix package
+using PACKAGE-URL and matching BRANCH.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import gopkg")))
+         %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-gopkg . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (args-fold* args %options
+                (lambda (opt name arg result)
+                  (leave (G_ "~A: unrecognized option~%") name))
+                (lambda (arg result)
+                  (alist-cons 'argument arg result))
+                %default-options))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                            (('argument . value)
+                             value)
+                            (_ #f))
+                           (reverse opts))))
+    (match args
+      ((package-url branch)
+       (let ((sexp (gopkg->guix-package package-url branch)))
+         (unless sexp
+           (leave (G_ "failed to download meta-data for package '~a'~%")
+                  package-url))
+         sexp))
+      ((package-url)
+       (let ((sexp (gopkg->guix-package package-url "master")))
+         (unless sexp
+           (leave (G_ "failed to download meta-data for package '~a'~%")
+                  package-url))
+         sexp))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.18.0

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 217 bytes --]


> Now, there are apparently several dependency management tools in use,

Yup, and it's still not fixed :(

Thanks for the patch, I'll try to use it and report :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

The gopkg patch seems to work on repository where .toml is missing, despite some
missing fields (e.g. synopsis), but that's OK.

Back to IPFS: I have some bad news :(
The import path are hard-coded to use "gx/ipfs...", for instance

--8<---------------cut here---------------start------------->8---
import (
	cid "gx/ipfs/QmPSQnBKM9g7BaUcZCvswUJVscQ1ipjmwxN5PXCjkp9EQ7/go-cid"
)
--8<---------------cut here---------------end--------------->8---

so this won't build directly.  I see two possibilities:

1. We patch IPFS to replace all "gx/ipfs/.*/go-cid" with
  "github.com/ipfs/go-cid", etc.  Thus leaving the responsibility to Guix to
  package the right version.

2. We package "gx" and run "gx install --global" before the build phase.

Method 2 goes in the direction of IPFS but opposite Guix, while method 1 is the
other way around.

Method 2 will be easier, and technically IPFS is here to give us a guarantee of
reproducibility.  What do you people think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

On Mon, Oct 15, 2018 at 01:40:45PM +0200, Pierre Neidhardt wrote:
> > I'm not much into go, but it seems, that most of the times it is safe
> > to use another version of dependencies.
> 
> This is unrelated to Go: it's safe as long as there is no API breakage.  My
> understanding of the IPFS project is that they don't want to make that kind of
> assumptions.

The real issue is that we can not assume APIs are safe. It is actually a
wider problem. Every dependency is a new dimension and any combination
thereof may or may not work as expected. Things break. This is why
running tests is so important. If tests pass you can make an
assumption that at least the tested stuff works.

To be best theoretical approach is to use the exact same version
developers are using. But that is not even realistic because
dependencies themselves may be developed on different versions ;).
I.e., these are different teams of developers.

We can just hope the combination of dependencies we have works. This
is why distributions like Debian have such a long testing cycle. They
want to test the combinations of dependencies through and through.
Users of unstable and testing are the actual testers.  Stable API's?
Hmmm. Some are better than others.

For deployment, to fixate dependencies, guix channels offer a great
solution. GUIX_PACKAGE_PATH with a fixated guix tree also works. Guix
as a rolling distribution generally works, but we are using untested
combinations of software. And once in a while you hit a problem. 

The GO people have their own solution for fixating dependencies to
avoid this problem. I don't think it is a bad one, only problem it is
that they created their own ecosystem.

Pj.

Fwd: Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 2510 bytes --]


Re-including everyone in the conversation:

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> > Otherwise I'll just go with the first method and patch the IPFS source to use
>> > the local dependencies.
>>
>> Also fine.
>
> Hmm, on second thought, I'm not so sure.  It's a slippery slope and in the long
> run, I doubt we could easily make sure the Guix packaged dep corresponds to the
> IPFS-path dep in the source.
>
> There is a conversation upstream:
>
> 	https://github.com/ipfs/go-ipfs/issues/5054
>
> To quote one of the developers:
>
>> > We need to build IPFS downstream in Debian from reusable packaged libraries.
>> Don't do that. We havent tested and verified ipfs with those dependencies. We
>> have tested and verified ipfs with exactly the dependencies that we ship it
>> with. Building ipfs with other dependencies is just begging for something to
>> go wrong that will be super annoying for me to have to debug. If debian starts
>> shipping an ipfs built with dependencies of their choosing, my response to any
>> bug report from a debian user will be "please use a version of ipfs compiled
>> from canonical source"
>
> So I think the best idea at the moment is to write a gx-fetcher.
> It could look like the following:
>
> --8<---------------cut here---------------start------------->8---
> (define-public ipfs-go-ipfs-config
>   (let ((ipfs-hash "QmSoYrBMibm2T3LupaLuez7LPGnyrJwdRxvTfPUyCp691u"))
>     (package
>       (name "ipfs-go-ipfs-config")
>       (version (gx-version "0.2.13" revision ipfs-hash))
>       (source
>        (origin
>          (method gx-fetch)
>          (uri (gx-reference ipfs-hash))
>          (file-name (gx-file-name name version))
>          (sha256
>           (base32
>            "0qk6fshgdmhp8dip2ksm13j6nywi41m9mn0czkvmw6b697z85l2r"))))
>       (build-system go-build-system)
>       (arguments
>        '(#:unpack-path (string-append  "gx/ipfs/" ipfs-hash "/go-ipfs-config")
>          #:import-path (string-append  "gx/ipfs/" ipfs-hash "/go-ipfs-config")))
>       (home-page
>        "https://github.com/ipfs/go-ipfs-config")
>       (synopsis "...")
>       (description "...")
>       (license license:expat))))
> --8<---------------cut here---------------end--------------->8---
>
> What about a gx-build-system?  It would inherit from go-build-system and
> automate the (string-append "gx/ipfs" ...) part.  Don't know if it's worth it.


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[bill-auger]

FWIW, that reads to me like: "this program is extremely brittle and
unfit for distribution in any distro - here, take our blob instead"

perhaps guix is in a unique position to accommodate such constrained
dependency requirements; but such caveat warnings are indicative of a
project that is still in the beta stage, so i would not be very eager to
package it

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

I look at it differently: they are very well aware of the flaws of most package
managers (e.g. non-reproducibility), including the one of the Go language, and
this project is big/important enough that it needs a proper system.

"gx" was their way to go, which works very well for their purpose, but
unfortunately it has some friction with the Guix model.

IPFS and Guix can work well together if we provide a gx-fetcher.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Back to IPFS: I have some bad news :(
> The import path are hard-coded to use "gx/ipfs...", for instance
>
> import (
> 	cid "gx/ipfs/QmPSQnBKM9g7BaUcZCvswUJVscQ1ipjmwxN5PXCjkp9EQ7/go-cid"
> )
>
> so this won't build directly.  I see two possibilities:
>
> 1. We patch IPFS to replace all "gx/ipfs/.*/go-cid" with
>   "github.com/ipfs/go-cid", etc.  Thus leaving the responsibility to Guix to
>   package the right version.
>
> 2. We package "gx" and run "gx install --global" before the build phase.

That won’t work: there’s not network access in the build environment in
order to maximize reproducibility.

So I suppose Method #1 is all we have…

Thoughts?

Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]


> That won’t work: there’s not network access in the build environment in
> order to maximize reproducibility.
> 
> So I suppose Method #1 is all we have…
> 
> Thoughts?

Hi Ludo, did you see the forwarded message I've send today?  (Sorry, part of the
conversation was taken off-list by accident).

I'm suggesting writing a gx-fetcher / gx-build-system so that the "imports"
in the code will find the packages, and so that we don't go against upstream's
tracking of dependencies.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

On Tue, Oct 16, 2018 at 01:51:29PM +0200, Ludovic Courtès wrote:
> That won’t work: there’s not network access in the build environment in
> order to maximize reproducibility.

IPFS-gx, if I am correct, is just like git for downloading sources. So
we could do gx-fetch in the download phase.

> So I suppose Method #1 is all we have…
> 
> Thoughts?

Nix, these days, has a two phase build. In the second phase you get network
access where you can run network tests. That is kinda interesting.

Pj.

Re: Guix & IPFS

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

On Mon, Oct 15, 2018 at 12:40:18PM +0200, Pierre Neidhardt wrote:
> No, Go programs typically refer only to repositories, not to their commit.  This
> is precisely why it's a problem.  Golang's dependency management was not very
> well designed I'm afraid.
> 
> Misunderstanding?

I think you and Ludo talked past each a bit. To clarify:

Go code refers to dependencies by the URL of their repositories. For
example, the code will import the modules it depends on like this:

------
import ("github.com/thejerf/suture"
        "github.com/rcrowley/go-metrics")
------

The Go compiler will actually fetch these Git repos if they are not
already available locally, cloning and using the HEAD commit (?!).

But in practice, any complex Go software will use a dependency
management tool that also specifies the Git commit to use, because...

... there is basically no concept of "releases" or "stable APIs" in Go
world, although there are 3rd party tools that try to fill the gap.
Since nobody is thinking about releasing or maintaining an API, they
don't, and it becomes important to use the right Git commit.

My opinion is that we should use the "correct" Git commit every time,
even it means we have to package several versions of the same Go
library, or use channels, or Scheme macros, etc. Otherwise we will have
a lot of debugging to do, and upstream developers will start to resent
us when they get bug reports. With Guile, we have the tools to achieve
this. I've been doing it "manually" for the Syncthing package.

We discussed this previously:

https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00030.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3337 bytes --]

I've started writing "gx-download.scm", it's on the wip-ipfs branch.

On the ipfs-go-ipfs-cmdkit example, I get this error:

--8<---------------cut here---------------start------------->8---
> ./pre-inst-env guix build -K ipfs-go-ipfs-cmdkit
;;; note: source file /home/ambrevar/projects/guix/guix/gx-download.scm
;;;       newer than compiled /home/ambrevar/.cache/guile/ccache/2.2-LE-8-3.A/home/ambrevar/projects/guix/guix/gx-download.scm.go
substitute: updating substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
building /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv...
[ 1/ 2] Compiling './guix/build/gx.scm'...
[ 2/ 2] Compiling './guix/build/utils.scm'...
successfully built /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv
building /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv...
what language will the project be in? initializing package 3nbb94ssalhxdas7vz2bv8n6dpz3mdw2-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout...
where should users go to report issues? notice: no helper tool found for 
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
ERROR: (install): failed to fetch package: QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky: Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided 
Backtrace:
           2 (primitive-load "/gnu/store/gpnysp8v5nv1l3yklbv0y5fvb0r?")
In guix/build/gx.scm:
    52:10  1 (gx-fetch _ _ #:gx-command _)
In unknown file:
           0 (scm-error misc-error #f "~A" ("Cannot import gx pack?") ?)

ERROR: In procedure scm-error:
Cannot import gx package
note: keeping build directory `/tmp/guix-build-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv-1'
builder for `/gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv' failed with exit code 1
build of /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv failed
View build log at '/var/log/guix/drvs/pw/jnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv.bz2'.
cannot build derivation `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv' failed
--8<---------------cut here---------------end--------------->8---

My understanding is that I need to pass certificates to the fetcher.
Does anyone know how to do that?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

Hello,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> Nix, these days, has a two phase build. In the second phase you get network
> access where you can run network tests. That is kinda interesting.

Interesting, do you have links to documentation or code?

Thanks,
Ludo.

Re: Guix & IPFS

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
> retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
> ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
> retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
> ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
> ERROR: (install): failed to fetch package: QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky: Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided 
> Backtrace:
>            2 (primitive-load "/gnu/store/gpnysp8v5nv1l3yklbv0y5fvb0r?")
> In guix/build/gx.scm:
>     52:10  1 (gx-fetch _ _ #:gx-command _)
> In unknown file:
>            0 (scm-error misc-error #f "~A" ("Cannot import gx pack?") ?)
>
> ERROR: In procedure scm-error:
> Cannot import gx package
> note: keeping build directory `/tmp/guix-build-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv-1'
> builder for `/gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv' failed wi

As discussed before (I think?), builds are performed in an isolated
environment without network access—this is one of the measures taken to
guarantee build reproducibility and statelessness.

So what you’re doing here (running “gx” in a derivation) cannot work.

HTH,
Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]


> As discussed before (I think?), builds are performed in an isolated
> environment without network access—this is one of the measures taken to
> guarantee build reproducibility and statelessness.
> 
> So what you’re doing here (running “gx” in a derivation) cannot work.

No, the above error comes from the "gx-fetch" code I've just pushed to the
wip-ipfs branch.  Downloaders obviously have network access, but maybe I've set
it up wrong.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]

Does anyone have some insights to share about downloaders? :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

Hi!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> As discussed before (I think?), builds are performed in an isolated
>> environment without network access—this is one of the measures taken to
>> guarantee build reproducibility and statelessness.
>> 
>> So what you’re doing here (running “gx” in a derivation) cannot work.
>
> No, the above error comes from the "gx-fetch" code I've just pushed to the
> wip-ipfs branch.  Downloaders obviously have network access, but maybe I've set
> it up wrong.

To be precise, builders of fixed-output derivations (derivations for
which the hash of the result is known in advance) have network access.

In practice these are downloaders, but it’s because they are
fixed-output derivations that the daemon is more liberal and provides
them with network access.

How did you define ‘gx-fetch’?

Thanks,
Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 320 bytes --]


> To be precise, builders of fixed-output derivations (derivations for
> which the hash of the result is known in advance) have network access.

Oh, I did not know this detail.  Thanks for pointing it out!

> How did you define ‘gx-fetch’?

It's on the wip-ipfs2 branch.  Find the (broken) patch attached.

[-- Attachment #1.2: 0001-gx-download-DRAFT.patch --]
[-- Type: text/x-patch, Size: 8045 bytes --]

From 376a9fa08fa3ffb5a5ab0980acf75abdfc797486 Mon Sep 17 00:00:00 2001
From: Pierre Neidhardt <mail@ambrevar.xyz>
Date: Wed, 17 Oct 2018 18:56:38 +0200
Subject: [PATCH] gx-download (DRAFT)

---
 guix/build/gx.scm    |  60 ++++++++++++++++++++
 guix/gx-download.scm | 131 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 191 insertions(+)
 create mode 100644 guix/build/gx.scm
 create mode 100644 guix/gx-download.scm

diff --git a/guix/build/gx.scm b/guix/build/gx.scm
new file mode 100644
index 000000000..4ba0197b4
--- /dev/null
+++ b/guix/build/gx.scm
@@ -0,0 +1,60 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build gx)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 popen)
+  #:export (gx-fetch))
+
+;;; Commentary:
+;;;
+;;; This is the build-side support code of (guix gx-download).  It allows a
+;;; gx hash to be fetched.
+;;;
+;;; Code:
+
+(define* (gx-fetch hash directory
+                   #:key (gx-command "gx"))
+  "Fetch IPFS HASH into DIRECTORY.  HASH must be a valid IPFS hash.
+Return #t on success, #f otherwise."
+
+  (mkdir-p directory)
+
+  (with-directory-excursion directory
+    ;; TODO: Silence verbose output.
+
+    ;; Initialization is interactive, but we can shut it up by piping it to
+    ;; nothing.
+    (let ((port (open-pipe* OPEN_WRITE gx-command "init")))
+      (display "\n" port)
+      (if (not (eqv? 0 (status:exit-val (close-pipe port))))
+          (error "Cannot initialize gx package")))
+
+    ;; Fetch to the "vendor" directory.
+    (let ((port (open-pipe* OPEN_WRITE gx-command "import" "--local" hash)))
+      (display "N\n" port)
+      (if (not (eqv? 0 (status:exit-val (close-pipe port))))
+          (error "Cannot import gx package")))
+
+    (delete-file "package.json")
+    (mkdir-p   "gx/ipfs")
+    (rename-file (string-append  "vendor/gx/ipfs/" hash) (string-append "gx/ipfs/" hash))
+    (delete-file-recursively "vendor")
+    #t))
+
+;;; gx.scm ends here
diff --git a/guix/gx-download.scm b/guix/gx-download.scm
new file mode 100644
index 000000000..4acf7bf61
--- /dev/null
+++ b/guix/gx-download.scm
@@ -0,0 +1,131 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix gx-download)
+  #:use-module (guix gexp)
+  #:use-module (guix store)
+  #:use-module (guix monads)
+  #:use-module (guix records)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (guix modules)
+  ;; #:autoload   (guix build-system gnu) (standard-packages)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 vlist)
+  #:use-module (srfi srfi-1)
+  #:export (gx-reference
+            gx-reference?
+            gx-reference-hash
+
+            gx-fetch
+            gx-version
+            gx-file-name))
+
+;;; Commentary:
+;;;
+;;; An <origin> method that uses gx to fetch a specific hash over IPFS.
+;;; See https://github.com/whyrusleeping/gx.
+;;; The hash is specified with a <gx-reference> object.
+;;;
+;;; Code:
+
+(define-record-type* <gx-reference>
+  gx-reference make-gx-reference
+  gx-reference?
+  (hash gx-reference-hash))
+
+(define (gx-package)
+  "Return the default gx package."
+  (let ((distro (resolve-interface '(gnu packages ipfs))))
+    (module-ref distro 'gx)))
+
+(define* (gx-fetch ref hash-algo hash
+                    #:optional name
+                    #:key (system (%current-system)) (guile (default-guile))
+                    (gx (gx-package)))
+  "Return a fixed-output derivation that fetches REF, a <gx-reference>
+object.  The output is expected to have recursive hash HASH of type
+HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
+  ;; (define inputs
+  ;;   ;; When doing 'git clone --recursive', we need sed, grep, etc. to be
+  ;;   ;; available so that 'git submodule' works.
+  ;;   ;; (if (git-reference-recursive? ref)
+  ;;   ;;     (standard-packages)
+  ;;   ;;     '())
+  ;;   )
+
+  ;; (define zlib
+  ;;   (module-ref (resolve-interface '(gnu packages compression)) 'zlib))
+
+  ;; (define config.scm
+  ;;   (scheme-file "config.scm"
+  ;;                #~(begin
+  ;;                    (define-module (guix config)
+  ;;                      #:export (%libz))
+
+  ;;                    (define %libz
+  ;;                      #+(file-append zlib "/lib/libz")))))
+
+  ;; (define modules
+  ;;   (cons `((guix config) => ,config.scm)
+  ;;         (delete '(guix config)
+  ;;                 (source-module-closure '((guix build git)
+  ;;                                          (guix build utils)
+  ;;                                          (guix build download-nar))))))
+
+  (define build
+    (with-imported-modules '((guix build gx)
+                             (guix build utils))
+      #~(begin
+          (use-modules (guix build gx)
+                       ;; (guix build utils)
+                       ;; (guix build download-nar)
+                       ;; (ice-9 match)
+                       )
+
+          ;; The 'git submodule' commands expects Coreutils, sed,
+          ;; grep, etc. to be in $PATH.
+          ;; (set-path-environment-variable "PATH" '("bin")
+          ;;                                (match '#+inputs
+          ;;                                  (((names dirs outputs ...) ...)
+          ;;                                   dirs)))
+
+          (or (gx-fetch '#$(gx-reference-hash ref)
+                         #$output
+                         #:gx-command (string-append #+gx "/bin/gx"))
+              ;; (download-nar #$output)
+              ))))
+
+  (mlet %store-monad ((guile (package->derivation guile system)))
+    (gexp->derivation (or name "gx-checkout") build
+                      #:system system
+                      #:local-build? #t
+                      #:hash-algo hash-algo
+                      #:hash hash
+                      #:recursive? #t
+                      #:guile-for-build guile)))
+
+(define (gx-version version revision hash)
+  "Return the version string for packages using gx-download."
+  (string-append version "-" revision "." (string-take hash 7)))
+
+(define (gx-file-name name version)
+  "Return the file-name for packages using gx-download."
+  (string-append name "-" version "-checkout"))
+
+;;; gx-download.scm ends here
-- 
2.19.1


[-- Attachment #1.3: Type: text/plain, Size: 48 bytes --]


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

Hello!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> To be precise, builders of fixed-output derivations (derivations for
>> which the hash of the result is known in advance) have network access.
>
> Oh, I did not know this detail.  Thanks for pointing it out!

[...]

> +  (mlet %store-monad ((guile (package->derivation guile system)))
> +    (gexp->derivation (or name "gx-checkout") build
> +                      #:system system
> +                      #:local-build? #t
> +                      #:hash-algo hash-algo
> +                      #:hash hash
> +                      #:recursive? #t
> +                      #:guile-for-build guile)))

As long as ‘hash-algo’ and ‘hash’ are true, it does define a
fixed-output derivation, so the builder should indeed have network
access.

What does ‘gx’ tell you in the builder?

HTH,
Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3289 bytes --]


> What does ‘gx’ tell you in the builder?

Sorry, I don't understand what you are asking.  The following is the output I
get when I attempt to build a package using gx-fetch:

--8<---------------cut here---------------start------------->8---
> ./pre-inst-env guix build -K ipfs-go-ipfs-cmdkit
;;; note: source file /home/ambrevar/projects/guix/guix/gx-download.scm
;;;       newer than compiled /home/ambrevar/.cache/guile/ccache/2.2-LE-8-3.A/home/ambrevar/projects/guix/guix/gx-download.scm.go
substitute: updating substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
building /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv...
[ 1/ 2] Compiling './guix/build/gx.scm'...
[ 2/ 2] Compiling './guix/build/utils.scm'...
successfully built /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv
building /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv...
what language will the project be in? initializing package 3nbb94ssalhxdas7vz2bv8n6dpz3mdw2-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout...
where should users go to report issues? notice: no helper tool found for 
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a second...
ERROR: from shell.Get(): Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided
ERROR: (install): failed to fetch package: QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky: Post https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true: x509: failed to load system roots and no roots provided 
Backtrace:
           2 (primitive-load "/gnu/store/gpnysp8v5nv1l3yklbv0y5fvb0r?")
In guix/build/gx.scm:
    52:10  1 (gx-fetch _ _ #:gx-command _)
In unknown file:
           0 (scm-error misc-error #f "~A" ("Cannot import gx pack?") ?)

ERROR: In procedure scm-error:
Cannot import gx package
note: keeping build directory `/tmp/guix-build-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv-1'
builder for `/gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv' failed with exit code 1
build of /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv failed
View build log at '/var/log/guix/drvs/pw/jnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv.bz2'.
cannot build derivation `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv' failed
--8<---------------cut here---------------end--------------->8---

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3733 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> ezt írta (időpont: 2018. okt. 25., Cs
20:11):

>
> > What does ‘gx’ tell you in the builder?
>
> Sorry, I don't understand what you are asking.  The following is the
> output I
> get when I attempt to build a package using gx-fetch:
>
> --8<---------------cut here---------------start------------->8---
> > ./pre-inst-env guix build -K ipfs-go-ipfs-cmdkit
> ;;; note: source file /home/ambrevar/projects/guix/guix/gx-download.scm
> ;;;       newer than compiled
> /home/ambrevar/.cache/guile/ccache/2.2-LE-8-3.A/home/ambrevar/projects/guix/guix/gx-download.scm.go
> substitute: updating substitutes from 'https://mirror.hydra.gnu.org'...
> 100.0%
> building
> /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv...
> [ 1/ 2] Compiling './guix/build/gx.scm'...
> [ 2/ 2] Compiling './guix/build/utils.scm'...
> successfully built
> /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv
> building
> /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv...
> what language will the project be in? initializing package
> 3nbb94ssalhxdas7vz2bv8n6dpz3mdw2-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout...
> where should users go to report issues? notice: no helper tool found for
> ERROR: from shell.Get(): Post
> https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true:
> x509: failed to load system roots and no roots provided
>
Maybe I am just stating the obvious, but this error seems to be certificate
related, i think network access is ok, but the certificate cant be
validated, as no root cert is found.

> retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a
> second...
> ERROR: from shell.Get(): Post
> https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true:
> x509: failed to load system roots and no roots provided
> retrying fetch QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky after a
> second...
> ERROR: from shell.Get(): Post
> https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true:
> x509: failed to load system roots and no roots provided
> ERROR: (install): failed to fetch package:
> QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky: Post
> https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true:
> x509: failed to load system roots and no roots provided
> Backtrace:
>            2 (primitive-load "/gnu/store/gpnysp8v5nv1l3yklbv0y5fvb0r?")
> In guix/build/gx.scm:
>     52:10  1 (gx-fetch _ _ #:gx-command _)
> In unknown file:
>            0 (scm-error misc-error #f "~A" ("Cannot import gx pack?") ?)
>
> ERROR: In procedure scm-error:
> Cannot import gx package
> note: keeping build directory
> `/tmp/guix-build-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv-1'
> builder for
> `/gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv'
> failed with exit code 1
> build of
> /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv
> failed
> View build log at
> '/var/log/guix/drvs/pw/jnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv.bz2'.
> cannot build derivation
> `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv':
> 1 dependencies couldn't be built
> guix build: error: build failed: build of
> `/gnu/store/1prmx5jgzm4wzm23zzwbc1af0r593zhj-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r.drv'
> failed
> --8<---------------cut here---------------end--------------->8---
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
>

[-- Attachment #2: Type: text/html, Size: 5191 bytes --]

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]


> Maybe I am just stating the obvious, but this error seems to be certificate
> related, i think network access is ok, but the certificate cant be
> validated, as no root cert is found.

That was my first clue as well! :)

But looking at other downloaders, only url-fetch has some x509 provisions.  I'm
not too sure about its mechanics however.  For instance, there is this snippet:

--8<---------------cut here---------------start------------->8---
(define %x509-certificate-directory
  ;; The directory where X.509 authority PEM certificates are stored.
  (make-parameter (or (getenv "GUIX_TLS_CERTIFICATE_DIRECTORY")
                      (getenv "SSL_CERT_DIR"))))  ;like OpenSSL
--8<---------------cut here---------------end--------------->8---

But url-fetch is quite complex and I've little time to study it at the moment :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Ludovic Courtès]

Hello,

Gábor Boskovits <boskovits@gmail.com> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> ezt írta (időpont: 2018. okt. 25., Cs
> 20:11):
>
>>
>> > What does ‘gx’ tell you in the builder?
>>
>> Sorry, I don't understand what you are asking.  The following is the
>> output I
>> get when I attempt to build a package using gx-fetch:
>>
>> --8<---------------cut here---------------start------------->8---
>> > ./pre-inst-env guix build -K ipfs-go-ipfs-cmdkit
>> ;;; note: source file /home/ambrevar/projects/guix/guix/gx-download.scm
>> ;;;       newer than compiled
>> /home/ambrevar/.cache/guile/ccache/2.2-LE-8-3.A/home/ambrevar/projects/guix/guix/gx-download.scm.go
>> substitute: updating substitutes from 'https://mirror.hydra.gnu.org'...
>> 100.0%
>> building
>> /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv...
>> [ 1/ 2] Compiling './guix/build/gx.scm'...
>> [ 2/ 2] Compiling './guix/build/utils.scm'...
>> successfully built
>> /gnu/store/rnwbwvks2imzzs5av5p0ihab6qynfgi8-module-import-compiled.drv
>> building
>> /gnu/store/pwjnh2sp30plxbw12zmwg4p5w0wfh5fy-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout.drv...
>> what language will the project be in? initializing package
>> 3nbb94ssalhxdas7vz2bv8n6dpz3mdw2-ipfs-go-ipfs-cmdkit-1.1.3-1.QmSP88r-checkout...
>> where should users go to report issues? notice: no helper tool found for
>> ERROR: from shell.Get(): Post
>> https://ipfs.io/api/v0/get?arg=QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky&create=true:
>> x509: failed to load system roots and no roots provided
>>
> Maybe I am just stating the obvious, but this error seems to be certificate
> related, i think network access is ok, but the certificate cant be
> validated, as no root cert is found.

Yes, that’s probably what’s happening.

Pierre, perhaps it’s just a matter of setting some environment variable
(SSL_CERT_DIR and/or SSL_CERT_FILE?) pointing to X.509 certificates such
as those provided by the ‘nss-certs’ package?

HTH,
Ludo’.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]

I found a solution to package IPFS without gx, so go-ipfs has just hit master!

I'll put the "gx-download" on hold since it's not used for anything else at the moment.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

On Sun, Dec 09, 2018 at 09:19:20PM +0100, Pierre Neidhardt wrote:
> I found a solution to package IPFS without gx, so go-ipfs has just hit master!

That is excellent news. That means we have a way of distributing
materials in a robust fashion.

Pj.

Re: Guix & IPFS

[swedebugia]

On 2018-12-10 02:52, Pjotr Prins wrote:
> On Sun, Dec 09, 2018 at 09:19:20PM +0100, Pierre Neidhardt wrote:
>> I found a solution to package IPFS without gx, so go-ipfs has just hit master!
> 
> That is excellent news. That means we have a way of distributing
> materials in a robust fashion.

Congratulations Pierre! And Guix!

Now it is time to finish polishing a service/services for it no?

How is that work progressing?

-- 
Cheers
Swedebugia

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 150 bytes --]


> Now it is time to finish polishing a service/services for it no?

Would be a good idea indeed!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

IPFS is working like a charm. Thanks Pierre a.o!

Pj.

Re: Guix & IPFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Ludo and Me have played with it at the Reproducible Build summit, and it works
quite well indeed!

Ludo is working Guile bindings.

I'll implement the service later, but I'd really like to get the first contract
money first.  Alicia told me there were some issues with the University because
they issue the payment on a monthly basis or something.  Alicia is working out a
solution, but that's a pity I've got to wait this long :(

Anyways, have fun with IPFS! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix & IPFS

[Pjotr Prins]

Here is an interesting application of IPFS to serve a static website:

  https://withblue.ink/2019/03/20/hugo-and-ipfs-how-this-blog-works-and-scales.html