* Improving Elixir support
@ 2017-12-03 6:20 Pjotr Prins
2017-12-11 0:04 ` nee
0 siblings, 1 reply; 3+ messages in thread
From: Pjotr Prins @ 2017-12-03 6:20 UTC (permalink / raw)
To: Guix-devel
I am forwarding this from the bug tracker because I believe it is of
general interest to those working on Elixir/Erlang.
http://lists.gnu.org/archive/html/bug-guix/2017-12/msg00019.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Improving Elixir support
2017-12-03 6:20 Improving Elixir support Pjotr Prins
@ 2017-12-11 0:04 ` nee
2017-12-11 7:41 ` Pjotr Prins
0 siblings, 1 reply; 3+ messages in thread
From: nee @ 2017-12-11 0:04 UTC (permalink / raw)
To: guix-devel
[-- Attachment #1: Type: text/plain, Size: 2673 bytes --]
Hello, I had a short look the elixir/erlang package systems.
I'm sending my current WIP status.
It seems the following things will be required:
1. hex.pm importer
https://hex.pm is the main package repository for erlang and elixir.
I already wrote a basic importer. Hex.pm has a simple json api that
provides everything an importer needs.
I still have to write an updater.
The downloads provided by repo.hex.pm are ".tar" files.
These tars contain:
- some metadata in an erlang specific config format that is already
available via the json api.
- contents.tar.gz which contains the actual package.
- a hash that is also available via the api and the webinterface. It
looks like a sha256 hash. I haven't found anything that matches it.
I would assume it relates to the contents.tar.gz somehow.
2. a url-fetch/hex.pm orgin fetch method that unpacks the nested
contents.tar.gz that hex.pm serves. This is already done.
3. build rebar3 from source
rebar3 is the main build tool for erlang packages.
I started working on this. I download the dependencies from hex.pm
and unpack them to _build/default. The problem is that rebar3 needs the
full hex.pm package index (around 6Mbytes of binary data)
to (I think) verify the package hashes.
I looked at how nix solves this, and they have a snapshot of the
hex.pm package-index mirrored in a github repository and update it
whenever they need a newer version of a hex.pm package.
I don't really like that solution, since all packages depend on this
central snapshot, and it has to be updated whenever you want to package
or update a new hex.pm package that isn't in the old snapshot.
Maybe rebar3 can be patched to not lookup the package registry, and
just accept whatever is in the current _build directory.
4. rebar3 build system
It's probably similar to building rebar3 from source.
5. mix build system
Mix is the build tool used by elixir. It's already built in the elixir
package. Dependencies seem to be installed locally in the ./deps
directory of a project. There are also so called archives. I'm not
sure yet if they can be used as dependencies.
I'm expecting the same registry problem as with rebar3, but haven't
checked, yet.
Interesting environment variables:
- MIX_ENV # set to "prod" to build a production target
- MIX_HOME
- MIX_ARCHIVES # path where archives and loaded from and installed to
- MIX_REBAR # override the rebar executable
https://hexdocs.pm/mix/Mix.html#module-environment-variables
Interesting commands:
- mix deps.check
- mix compile
- mix archive.build
- mix archive.install
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-unfinished-elixir-and-erlang-buildsystem-and-hex.patch --]
[-- Type: text/x-patch; name="0001-WIP-unfinished-elixir-and-erlang-buildsystem-and-hex.patch", Size: 15225 bytes --]
From ed2829549b8d7ca1cb34737f73ba9b6ea129d499 Mon Sep 17 00:00:00 2001
From: nee <nee.git@cock.li>
Date: Mon, 11 Dec 2017 00:49:14 +0100
Subject: [PATCH] WIP unfinished elixir and erlang buildsystem and hex importer
---
Makefile.am | 2 +
gnu/packages/erlang.scm | 58 +++++++++++++++++++
guix/download.scm | 41 ++++++++++++++
guix/import/hex.scm | 133 ++++++++++++++++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/hex.scm | 7 +++
6 files changed, 242 insertions(+), 1 deletion(-)
create mode 100644 guix/import/hex.scm
create mode 100644 guix/scripts/import/hex.scm
diff --git a/Makefile.am b/Makefile.am
index ddbf7a798..a07a5dc6e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -160,6 +160,7 @@ MODULES = \
guix/import/cabal.scm \
guix/import/cran.scm \
guix/import/hackage.scm \
+ guix/import/hex.scm \
guix/import/elpa.scm \
guix/import/texlive.scm \
guix/scripts.scm \
@@ -184,6 +185,7 @@ MODULES = \
guix/scripts/import/gnu.scm \
guix/scripts/import/nix.scm \
guix/scripts/import/hackage.scm \
+ guix/scripts/import/hex.scm \
guix/scripts/import/elpa.scm \
guix/scripts/import/texlive.scm \
guix/scripts/environment.scm \
diff --git a/gnu/packages/erlang.scm b/gnu/packages/erlang.scm
index 770ed715b..bccb835ea 100644
--- a/gnu/packages/erlang.scm
+++ b/gnu/packages/erlang.scm
@@ -19,6 +19,7 @@
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (gnu packages erlang)
+ #:use-module (ice-9 match)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix build-system gnu)
#:use-module (guix download)
@@ -176,3 +177,60 @@ built-in support for concurrency, distribution and fault tolerance.")
;; have other licenses. See 'system/COPYRIGHT' in the source distribution.
(license (list license:asl2.0 license:bsd-2 license:bsd-3 license:expat
license:lgpl2.0+ license:tcl/tk license:zlib))))
+
+;;; TODO patch rebar3 to not check for the registry and just accept whater it finds in _build
+(define-public rebar3
+ (let ((deps '(("erlware_commons" "1.0.4" "18j8bin8q9dcbwfgkyc66231wkyji6ddjnfqsjnfh6zvvfal8kar")
+ ("ssl_verify_fun" "1.1.3" "1zljxashfhqmiscmf298vhr880ppwbgi2rl3nbnyvsfn0mjhw4if")
+ ("certifi" "2.0.0" "075v7cvny52jbhnskchd3fp68fxgp7qfvdls0haamcycxrn0dipx")
+ ("providers" "1.7.0" "19p4rbsdx9lm2ihgvlhxyld1q76kxpd7qwyqxxsgmhl5r8ln3rlb")
+ ("getopt" "1.0.1" "174mb46c2qd1f4a7507fng4vvscjh1ds7rykfab5rdnfp61spqak")
+ ("bbmustache" "1.3.0" "042pfgss8kscq6ssg8gix8ccmdsrx0anjczsbrn2a6c36ljrx2p6")
+ ("relx" "3.24.3" "1g90ssg5gnnqs77n7l68nmv5a9fjcrxd0jb259vmff9mnpf1d1p3")
+ ("cf" "0.2.2" "08cvy7skn5d2k4manlx5k3anqgjdvajjhc5jwxbaszxw34q3na28")
+ ("cth_readable" "1.3.2" "0g9a6b7zi4azglafd9lfhn5a7vbbafg3s09l092zhcmk9jlgf866")
+ ("eunit_formatters" "0.5.0" "1jb3hzb216r29x2h4pcjwfmx1k81431rgh5v0mp4x5146hhvmj6n"))))
+ (package
+ (name "rebar3")
+ (version "3.4.7")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/erlang/rebar3/archive/"
+ version ".tar.gz"))
+ (sha256
+ (base32
+ "11rxb8a62hd5pwnhskmbxl8c0iwqchf6v9iwvzyfxrzjvv6xkcqr"))))
+ (build-system gnu-build-system)
+ (arguments `(#:phases (modify-phases %standard-phases
+ (delete 'configure)
+ (replace 'build
+ (lambda* (#:key inputs #:allow-other-keys)
+ ;; prepare dependencies
+ (for-each (lambda (name)
+ (mkdir-p (string-append "_build/default/lib/" name))
+ (copy-recursively (assoc-ref inputs name)
+ (string-append "_build/default/lib/" name)))
+ (list ,@(map car (warn "DEPS" deps))))
+ (setenv "HOME" (getcwd))
+ (zero? (system* "./bootstrap"))))
+ (delete 'check) ;TODO package it
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let ((out (assoc-ref outputs "out")))
+ (mkdir-p (string-append out "/bin"))
+ (copy-file "rebar3" (string-append out "/bin/"))))))))
+ (inputs `(("erlang" ,erlang)
+ ,@(map (match-lambda
+ ((name version hash)
+ (list
+ name
+ (origin (method url-fetch/hex.pm)
+ (uri (string-append "https://repo.hex.pm/tarballs/" name "-" version ".tar"))
+ (sha256 (base32 hash))
+ (file-name (string-append name "-" version ".tar.gz"))))))
+ deps)))
+ (home-page "https://rebar3.org")
+ (synopsis "erlang package build tool")
+ (description "Rebar3 is an Erlang tool that makes it easy to create, develop,
+and release Erlang libraries, applications, and systems in a repeatable manner.")
+ (license license:asl2.0))))
diff --git a/guix/download.scm b/guix/download.scm
index 8a0b19c01..32803b44d 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -37,6 +37,7 @@
url-fetch
url-fetch/tarbomb
url-fetch/zipbomb
+ url-fetch/hex.pm
download-to-store))
;;; Commentary:
@@ -630,6 +631,46 @@ own. This helper makes it easier to deal with \"zip bombs\"."
#$drv)))
#:local-build? #t)))
+(define* (url-fetch/hex.pm url hash-algo hash
+ #:optional name
+ #:key (system (%current-system))
+ (guile (default-guile)))
+ "Similar to 'url-fetch' but unpack the contents.tar.gz contained in the url.tar.
+ This helper makes it easier to deal with \"hex.pm tars\"."
+ (define file-name
+ (match url
+ ((head _ ...)
+ (basename head))
+ (_
+ (basename url))))
+ (define gzip
+ (module-ref (resolve-interface '(gnu packages compression)) 'gzip))
+ (define tar
+ (module-ref (resolve-interface '(gnu packages base)) 'tar))
+
+ (mlet %store-monad ((drv (url-fetch url hash-algo hash
+ (string-append "hex.pm-"
+ (or name file-name))
+ #:system system
+ #:guile guile)))
+ ;; Take the hex.pm tar, and unpack both
+ (gexp->derivation (or name file-name)
+ #~(begin
+ (mkdir #$output)
+ (setenv "PATH" (string-append #$gzip "/bin"))
+ (chdir #$output)
+ (zero? (system* (string-append #$tar "/bin/tar")
+ "xf" #$drv))
+ ;; delete a bunch of hex.pm metadata we don't need
+ (delete-file "CHECKSUM")
+ (delete-file "metadata.config")
+ (delete-file "VERSION")
+ ;; unpack the actual package content
+ (zero? (system* (string-append #$tar "/bin/tar")
+ "xf" "contents.tar.gz"))
+ (delete-file "contents.tar.gz"))
+ #:local-build? #t)))
+
(define* (download-to-store store url #:optional (name (basename url))
#:key (log (current-error-port)) recursive?
(verify-certificate? #t))
diff --git a/guix/import/hex.scm b/guix/import/hex.scm
new file mode 100644
index 000000000..86d960e76
--- /dev/null
+++ b/guix/import/hex.scm
@@ -0,0 +1,133 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Federico Beffa <nee-git@hidamari.blue>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import hex)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-34)
+ #:use-module (srfi srfi-35)
+ #:use-module (guix import json)
+ #:use-module (guix import hackage)
+ #:use-module (guix memoization)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (guix store)
+ #:use-module (guix base32)
+ #:use-module (guix hash)
+ #:use-module (guix import utils)
+ #:use-module ((guix download) #:select (download-to-store url-fetch))
+ #:export (hex-fetch))
+
+(define %packages-url "https://hex.pm/api/packages/")
+(define %tarball-url "https://repo.hex.pm/tarballs/")
+(define %package-name-prefix "hex-")
+
+(define (hex-name->guix-name name)
+ (if (string-prefix? %package-name-prefix name)
+ (snake-case name)
+ (string-append %package-name-prefix (snake-case name))))
+
+(define (make-input-entry name)
+ (list name (list 'unquote (string->symbol name))))
+
+(define (as-list item/s)
+ (if (list? item/s)
+ item/s
+ (list item/s)))
+
+(define (list-or-single item/s)
+ (if (null? (cdr item/s))
+ (car item/s)
+ items))
+
+;;; TODO take version as second arg
+(define (hex-fetch package-name)
+ (define url (string-append %packages-url package-name))
+ (define json (json-fetch url))
+ (define meta (assoc-ref json "meta"))
+ (define latest-release (car (assoc-ref json "releases")))
+ (define latest-release-info-url (assoc-ref latest-release "url"))
+ (define latest-release-version (assoc-ref latest-release "version"))
+ (define release-json (json-fetch latest-release-info-url))
+ (define release-meta (assoc-ref release-json "meta"))
+ ;; TODO maybe also check the versions and optional
+ (define inputs (map car (assoc-ref release-json "requirements")))
+ (define tarball-name (string-append package-name "-" latest-release-version ".tar"))
+ (define source-url (string-append %tarball-url tarball-name))
+ (define tarball (with-store store
+ (download-to-store store source-url)))
+ `(package
+ (name ,(hex-name->guix-name package-name))
+ (version ,latest-release-version)
+ (source (origin
+ (method url-fetch/hex.pm)
+ (uri (string-append ,@(list (string-append %tarball-url package-name "-")
+ 'version ".tar")))
+ (sha256
+ (base32
+ ,(if tarball
+ (bytevector->nix-base32-string (file-sha256 tarball))
+ "failed to download tar archive")))))
+ (build-system hex-build-system)
+ (inputs ,(list 'quasiquote
+ (map (lambda (input-hex)
+ (define input-guix (hex-name->guix-name input-hex))
+ (make-input-entry input-guix))
+ inputs)))
+ (native-inputs ,(list 'quasiquote
+ (map (lambda (input-hex)
+ (define input-guix (match input-hex
+ ("rebar3" "rebar3")
+ ("mix" "elixir")
+ (input (hex-name->guix-name input))))
+ (make-input-entry input-guix))
+ (match (assoc-ref release-meta "build_tools")
+ (#f '())
+ ((? list? lst) lst)
+ (single-tool (list single-tool))))))
+ (home-page "")
+ (synopsis "")
+ (description ,(assoc-ref meta "description"))
+ (license ,(list-or-single
+ (map string->license
+ (as-list (or (assoc-ref meta "licenses") '())))))))
+
+(define string->license
+ ;; copied from the hackage importer, slightly edited some names
+ ;; TODO: find a complete list of licences on hex.pm
+ (match-lambda
+ ("GPL-2" 'gpl2)
+ ("GPL-3" 'gpl3)
+ ("GPL" "'gpl??")
+ ("AGPL-3" 'agpl3)
+ ("AGPL" "'agpl??")
+ ("LGPL-2.1" 'lgpl2.1)
+ ("LGPL-3" 'lgpl3)
+ ("LGPL" "'lgpl??")
+ ("BSD2" 'bsd-2)
+ ("BSD3" 'bsd-3)
+ ("MIT" 'expat)
+ ("ISC" 'isc)
+ ("MPL" 'mpl2.0)
+ ("CC0-1.0" 'cc0)
+ ("Apache 2.0" 'asl2.0)
+ ((x) (string->license x))
+ ((lst ...) `(list ,@(map string->license lst)))
+ (_ #f)))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 67bc7a755..5cb3f5c83 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -74,7 +74,7 @@ rather than \\n."
;;;
(define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
- "cran" "crate" "texlive" "json"))
+ "hex" "cran" "crate" "texlive" "json"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/hex.scm b/guix/scripts/import/hex.scm
new file mode 100644
index 000000000..f84f87a50
--- /dev/null
+++ b/guix/scripts/import/hex.scm
@@ -0,0 +1,7 @@
+(define-module (guix scripts import hex)
+ #:use-module (guix import hex)
+ #:export (guix-import-hex))
+
+(define (guix-import-hex . args)
+ (warn "hex import" args)
+ (hex-fetch (car args)))
--
2.14.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: Improving Elixir support
2017-12-11 0:04 ` nee
@ 2017-12-11 7:41 ` Pjotr Prins
0 siblings, 0 replies; 3+ messages in thread
From: Pjotr Prins @ 2017-12-11 7:41 UTC (permalink / raw)
To: nee; +Cc: guix-devel
On Mon, Dec 11, 2017 at 01:04:19AM +0100, nee wrote:
> Hello, I had a short look the elixir/erlang package systems.
> I'm sending my current WIP status.
Good progress! Just thinking out-of-the-box here, but almost all
Elixir packages are pretty straightforward. Can we not just unpack
them and compile them? I think there is too much magic behind mix and
all that. All you really want is to compile them and have them on the
path.
Pj.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-12-11 7:44 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-03 6:20 Improving Elixir support Pjotr Prins
2017-12-11 0:04 ` nee
2017-12-11 7:41 ` Pjotr Prins
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).