[bug#47187] [PATCH] gnu: Add c-lightning.

[bug#47187] [PATCH] gnu: Add c-lightning.

[ZmnSCPxj via Guix-patches via]

Good morning Guix developers,

C-Lightning is one of the earliest living implementations of the Lightning
Network protocol.
I am one of the developers of this implementation.

I tested this as follows:

* Extracted the package `c-lightning` into a `c-lightning.scm` file.
* Ran `guix build --rounds=4 -f c-lightning.scm` on an X86-64 server to
  check for reproducability issues.
* Ran the compiled `lightningd` with `lightningd --version`,
  `lightningd --help`, and the developer-secret undocumented
  `lightningd --test-daemons-only` and checked they work correctly.
* Ran the compiled `lightningd` with `lightningd --proxy=127.0.0.1:9050
  --bind-addr=127.0.0.1:9735 --addr=statictor:127.0.0.1:9051
  --always-use-proxy=true` on a server with Bitcoin and Tor installed,
  and checked that it indeed created an LN wallet and started operating
  the node.
  * This also tests the SQLITE database backend and that the sub-daemons
    are accessible and runnable.
* From an existing Lightning Network node, did a `lightning-cli connect`
  to the new node running on the above command, and confirmed that the
  above new node was connected to, and that it started downloading the
  gossip map (the first thing every new node does when it initially gets
  connected to the network).
* On a separate X86-64 machine with the same Guix commit version, copied
  the `c-lightning.scm` file and reran `guix build --rounds=4 -f
  c-lightning.scm` as well.
* Checked that the same `/gnu/store` path was produced on the separate
  machine as on the first test server.
* Checked that this command produced the exact same text on both computers:
  `find ${GNU_STORE_PATH} -type f | sort | xargs -n1 sha256sum`.
* Extracted the package `c-lightning-postgresql` into a
  `c-lightning-postgresql.scm` file.
* Ran `guix build --rounds=4 -f c-lightning-postgresql.scm` on a X86-64
  server to check for reproducibility issues.
* Chekced that `strings ${GNU_STORE_PATH}/bin/lightningd` for the
  `c-lightning-postgresql` had references to `PQ*` functions, and that
  the `c-lightning` version had none, meaning that `c-lightning-postgresql`
  indeed included PostgreSQL support.

Thus, PostgreSQL support is not fully tested yet, but I believe the
produced binaries will work correctly as a C-Lightning instance.
Therefore I would like to submit this patch for inclusion into Guix.

Some notes:

* C-Lightning includes an extensive suite of tests, but these require
  several additional python packages, some of which (e.g.
  `python-bitcoinlib`) are not yet in Guix.
  The tests are extensive and *very long*, running the entire test
  suite can take an hour or more depending on exact settings.
* I took advantage of the functional nature of Guix and disabled the
  relative-path `BINTOPKGLIBEXECDIR`, making it an absolute path, as
  it also incidentally solves the issue of "upgrading C-lightning may
  cause strange problems in a running instance".
  See patch for more details.
* Our `configure` and `Makefile` are weird and I should probably go
  fix those upstream.

Regards,
ZmnSCPxj

From 490aa608765228fbb6bb64b9fa679bb4c8145001 Mon Sep 17 00:00:00 2001
From: ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>
Date: Mon, 15 Mar 2021 11:53:35 +0000
Subject: [PATCH] gnu: Add c-lightning.

* gnu/packages/finance.scm (c-lightning, c-lightning-postgresql): New variables.
---
 gnu/packages/finance.scm | 153 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 153 insertions(+)

diff --git a/gnu/packages/finance.scm b/gnu/packages/finance.scm
index e6df40c168..4b565ddfeb 100644
--- a/gnu/packages/finance.scm
+++ b/gnu/packages/finance.scm
@@ -22,6 +22,7 @@
 ;;; Copyright © 2020 Vinicius Monego <monego@posteo.net>
 ;;; Copyright © 2020 Carlo Holl <carloholl@gmail.com>
 ;;; Copyright © 2020 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2021 ZmnSCPxj jxPCSnmZ <ZmnSCPxj@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -57,6 +58,7 @@
   #:use-module (gnu packages aidc)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages base)
+  #:use-module (gnu packages bash)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages check)
   #:use-module (gnu packages compression)
@@ -99,6 +101,7 @@
   #:use-module (gnu packages qt)
   #:use-module (gnu packages readline)
   #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages sqlite)
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages textutils)
   #:use-module (gnu packages time)
@@ -1676,3 +1679,152 @@ Interface (UI) for the hledger accounting system.  It can be used as a
 local, single-user UI, or as a multi-user UI for viewing, adding, and
 editing on the Web.")
     (license license:gpl3)))
+
+(define-public c-lightning
+  (package
+    (name "c-lightning")
+    (version "0.9.3")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (string-append
+               "https://github.com/ElementsProject/lightning/releases/download/"
+               "v" version "/clightning-v" version ".zip"))
+        (sha256
+          (base32 "1chqzxcqpr49vbayrw4213lznmyw4lcghcdh6afxbk4bxlhkjmml"))))
+    (build-system gnu-build-system)
+    (native-inputs
+      `(("autoconf" ,autoconf)
+        ("automake" ,automake)
+        ("bash" ,bash)
+        ("libtool" ,libtool)
+        ("pkg-config" ,pkg-config)
+        ("python" ,python)
+        ("python-mako" ,python-mako)
+        ("sed" ,sed)
+        ("unzip" ,unzip)
+        ("which" ,which)))
+    (inputs
+      `(("gmp" ,gmp)
+        ("sqlite" ,sqlite)
+        ("zlib" ,zlib)))
+    (arguments
+      ;; Tests exist, but need a lot of Python packages (some not available
+      ;; on Guix) and they are incompatible with our BINTOPKGLIBEXECDIR hack.
+      `(#:tests? #f
+        #:phases
+        (modify-phases %standard-phases
+          (add-before 'configure 'patch-makefile
+            (lambda _
+              (substitute* "Makefile"
+                ;; The C-lightning Makefile uses the PYTHONPATH
+                ;; variable, which causes Guix builds to fail
+                ;; since the environment variable is masked by
+                ;; the Makefile variable.
+                ;; Insert the contents of the variable into the
+                ;; Makefile.
+                (("^PYTHONPATH=")
+                 (string-append
+                   "PYTHONPATH="
+                   (getenv "PYTHONPATH")
+                   ":"))
+                ;; C-lightning will spawn a number of other
+                ;; processes from binaries installed in its
+                ;; libexecdir.
+                ;; It normally uses relative paths so that
+                ;; users can move around the installation
+                ;; location.
+                ;; However, it does have the drawback that if
+                ;; the installation location is overwritten
+                ;; with a newer version while an existing
+                ;; instance is still running, any new
+                ;; sub-processes launched will be the new
+                ;; version, which is likely incompatible with
+                ;; the running instance.
+                ;; Since Guix would not allow the store
+                ;; directory to be moved anyway, we use an
+                ;; absolute path instead in the below
+                ;; substitution.
+                ;; With Guix and an absolute path, even if a
+                ;; running instance was launched from a
+                ;; profile and that profile is upgraded to a
+                ;; newer version, the running instance will
+                ;; refer to the absolute store directory
+                ;; containing binaries of the running version.
+                (("BINTOPKGLIBEXECDIR=.*$")
+                 "BINTOPKGLIBEXECDIR='\"'\"$(pkglibexecdir)\"'\"'\n"))))
+          ;; C-lightning configure is unusual, it does not understand
+          ;; the standard parameters Guix passes in, so, provide those
+          ;; by env variables.
+          (replace 'configure
+            (lambda* (#:key outputs inputs (configure-flags '())
+                      #:allow-other-keys)
+              (let* ((bash    (string-append (assoc-ref inputs "bash") "/bin/bash"))
+                     (python  (assoc-ref inputs "python"))
+                     (prefix  (assoc-ref outputs "out"))
+                     (flags   (cons*
+                                "CC=gcc"
+                                (string-append "--prefix=" prefix)
+                                configure-flags)))
+                (setenv "PYTHON" (string-append python "/bin/python3"))
+                (setenv "CONFIG_SHELL" bash)
+                (setenv "SHELL" bash)
+                (format #t "build directory: ~s~%" (getcwd))
+                (format #t "configure flags: ~s~%" flags)
+                (apply invoke bash
+                       "./configure"
+                       flags))))
+          ;; Rather than call the configure script of its external
+          ;; libraries from its own configure script, the external
+          ;; configure are created and called at build time.
+          ;; Unfortunately, it is a single Makefile stanza which
+          ;; does the autoreconf, configure, and make.
+          ;; This means we cannot "cut" through here by creating
+          ;; the external configure before this step (the Makefile
+          ;; stanza will recreate and overwrite the external
+          ;; configure), we have to modify the autogen.sh scripts
+          ;; of the external libraries instead so that they
+          ;; patch the shebangs after autoreconf.
+          (add-before 'build 'fix-autoreconf
+            (lambda _
+              (substitute* "external/libsodium/autogen.sh"
+                (("exec autoreconf(.*)$" exec-autoreconf flags)
+                 (string-append
+                   "autoreconf" flags
+                   "sed 's:/bin/sh:" (getenv "SHELL") ":g' < configure > configure.tmp\n"
+                   "mv configure.tmp configure\n"
+                   "chmod +x configure\n"
+                   "exit 0\n")))
+              (substitute* "external/libwally-core/tools/autogen.sh"
+                (("autoreconf(.*)$" autoreconf)
+                 (string-append
+                   autoreconf
+                   "sed 's:/bin/sh:" (getenv "SHELL") ":g' < configure > configure.tmp\n"
+                   "mv configure.tmp configure\n"
+                   "chmod +x configure\n"))))))))
+    (home-page "https://github.com/ElementsProject/lightning")
+    (synopsis "Lightweight Lightning Network protocol implementation in C")
+    (description
+      "c-lightning is a lightweight, highly customizable, and standard
+compliant implementation of the Lightning Network protocol.")
+    (license license:expat)))
+
+(define-public c-lightning-postgresql
+  (package
+    (inherit c-lightning)
+    (name "c-lightning-postgresql")
+    (inputs
+      `(("postgresql" ,postgresql)
+        ;; C-Lightning requires SQLITE3 as of 0.9.3, and will
+        ;; fail to build if it is not found.
+        ;; (The configure script will allow PostgreSQL without
+        ;; SQLITE3 but some build tool of C-Lightning fails if
+        ;; SQLITE3 is not found.)
+        ,@(package-inputs c-lightning)))
+    (description
+      "c-lightning is a lightweight, highly customizable, and standard
+compliant implementation of the Lightning Network protocol.
+
+This package includes support for using a PostgreSQL database to back
+your node; you will need to set up the PostgreSQL separately and pass
+in its details using an appropriate flag setting.")))
--
2.30.1

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Hello!

Thanks for the patch!

I built it and it was successful, however, few comments:

The various hacks in the recipe to get a working build seem troublesome
maintenance-wise, is it possible you think since you are a developer
that you make the build scripts a bit more standard in the project?

I see there's some vendored libraries: gheap, jsmn, libbacktrace,
libsodium, libwally-core ; Is it possible to unvendor those and create
separate packages for each of them? Or is there a requirement of strict
behavior here, either way, you could also create separate packages for
each of them even if those are specific to c-lightning. It would make
the main recipe cleaner, since as far as I can see it's also why the
build system is so stubborn to GNU Guix? That it also attempts to build
vendored libraries?

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

Additionally, please do enable the test suite it's really valuable in
GNU Guix.

I can help packaging the necessary Python dependencies, also we have a
Python importer, e.g. "$ guix import pypi -r <pkg>" to help us go
faster at it.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[ZmnSCPxj via Guix-patches via]

Good morning Leo,

> The various hacks in the recipe to get a working build seem troublesome
> maintenance-wise, is it possible you think since you are a developer
> that you make the build scripts a bit more standard in the project?

Already working on that, but 0.10.0 will release soon and I doubt my changes will make it by then, so Guix might get c-lightning to 0.10.1 or later if we wait for it.

> I see there's some vendored libraries: gheap, jsmn, libbacktrace,
> libsodium, libwally-core ; Is it possible to unvendor those and create
> separate packages for each of them? Or is there a requirement of strict
> behavior here, either way, you could also create separate packages for
> each of them even if those are specific to c-lightning. It would make
> the main recipe cleaner, since as far as I can see it's also why the
> build system is so stubborn to GNU Guix? That it also attempts to build
> vendored libraries?

I think it is more the custom `configure` script.
The custom `configure` script also compiles a ***C*** program that then generates the configuration for the `ccan` library of the project, meaning cross-compilation requires special work for C-Lightning.
And `configure` does not call `configure` of the included libraries as well.

Yes, it is true that there is something of a requirement of a strict behavior here, I suppose it is possible to use `git-fetch` instead of `url-fetch` for our external libraries.
How do I generate `guix hash` for `git-fetch`ed `source`s?
However it also means that every release of C-Lightning I have to go double-check what git commit to use for each library (though `jsmn` and `libbacktrace` at least seem very stable).

But it looks to me that unvendoring will require more extensive patching of the `Makefile` and an even larger maintenance burden on Guix side?

`libwally-core` itself depends on another library `libsecp256k1`, so I suppose it must transitively be unvendored as well.

> Additionally, please do enable the test suite it's really valuable in
> GNU Guix.

The test suite is fairly large and can take a significant amount of time to run in full.

In addition, part of the test includes checks which take advantage of `BINTOPKGLIBEXECDIR` relative path we normally use, which I want to disable for Guix at least since the relative path only has an advantage if the user wants to move the installation after-the-fact to a different location (and on Guix the "installation" path cannot be moved anyway).
Using an absolute `BINTOPKGLIBEXECDIR` gives an advantage as mentioned in the comments that a profile being upgraded from one version of C-Lightning to another does not cause problems for daemons currently running off the profile.
The test can be disabled (but not easily), I suppose.

> I can help packaging the necessary Python dependencies, also we have a
> Python importer, e.g. "$ guix import pypi -r <pkg>" to help us go
> faster at it.

Please do, I am not very familiar with any Python infrastructure and am primarily a C programmer here, I just barely hack together some kind of test in Python.


Regards,
ZmnSCPxj

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2820 bytes --]

On Tue, 2021-03-16 at 11:27 +0000, ZmnSCPxj wrote:
> Already working on that, but 0.10.0 will release soon and I doubt my
> changes will make it by then, so Guix might get c-lightning to 0.10.1
> or later if we wait for it.

Awesome!

> Yes, it is true that there is something of a requirement of a strict
> behavior here, I suppose it is possible to use `git-fetch` instead of
> `url-fetch` for our external libraries.

Yes you can use git-fetch, to make sure we are on the same page, are
you speaking of strict behavior requirements like for Bitcoin Core's
consensus code?

> How do I generate `guix hash` for `git-fetch`ed `source`s?

Actually what I do is put a wrong hash in and then copy the "actual
hash" from the error. I havent found another way but this definitely
feels subpar and prevents much verification before putting in the hash,
better suggestions welcome.

> However it also means that every release of C-Lightning I have to go
> double-check what git commit to use for each library (though `jsmn`
> and `libbacktrace` at least seem very stable).
> 
> But it looks to me that unvendoring will require more extensive
> patching of the `Makefile` and an even larger maintenance burden on
> Guix side?

Unvendoring is more or less a policy because we must be able to audit
each piece of software separately for freedom issues (licenses,
violations to the GNU FSDG), and it eases work for security-patching
also.

> `libwally-core` itself depends on another library `libsecp256k1`, so
> I suppose it must transitively be unvendored as well.

Probably, we already have a libsecp256k1 specially for Bitcoin Core.

> The test suite is fairly large and can take a significant amount of
> time to run in full.

That's fine.

> In addition, part of the test includes checks which take advantage of
> `BINTOPKGLIBEXECDIR` relative path we normally use, which I want to
> disable for Guix at least since the relative path only has an
> advantage if the user wants to move the installation after-the-fact
> to a different location (and on Guix the "installation" path cannot
> be moved anyway).
> Using an absolute `BINTOPKGLIBEXECDIR` gives an advantage as
> mentioned in the comments that a profile being upgraded from one
> version of C-Lightning to another does not cause problems for daemons
> currently running off the profile.
> The test can be disabled (but not easily), I suppose.

Problematic or unrealistically expensive tests can be disabled yes.

> Please do, I am not very familiar with any Python infrastructure and
> am primarily a C programmer here, I just barely hack together some
> kind of test in Python.

If you can list the Python dependencies and their version I can look at
packaging them.

> Regards,
> ZmnSCPxj

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[ZmnSCPxj via Guix-patches via]

Good morning Leo,

> > Yes, it is true that there is something of a requirement of a strict
> > behavior here, I suppose it is possible to use `git-fetch` instead of
> > `url-fetch` for our external libraries.
>
> Yes you can use git-fetch, to make sure we are on the same page, are
> you speaking of strict behavior requirements like for Bitcoin Core's
> consensus code?

Well we need to produce signatures and transactions that pass Bitcoin Core signature validation at least, so it is best to use a version of `libsecp256k1` (which produces signatures) that we know works, as well as `libwally-core` (which produces transactions).
I would personally use the `libsecp256k1` version that `libwally-core` vendors in as well, since there may be specific interactions between `libwally-core` and `libsecp256k1` that may be different if we use the Bitcoin Core version of `libsecp256k1`.

For `libsodium`, at least the hashing has to work correctly, but I think it is simple enough that strict behavior requirements are not so onerous.
Indeed, we usually get this from the OS (but we need a later feature than that available on some old Ubuntu versions, which is why it got vendored in), so I should probably "just" add it as an input.

>
> > How do I generate `guix hash` for `git-fetch`ed `source`s?
>
> Actually what I do is put a wrong hash in and then copy the "actual
> hash" from the error. I havent found another way but this definitely
> feels subpar and prevents much verification before putting in the hash,
> better suggestions welcome.

Haha I shall do so as well.

>
> > However it also means that every release of C-Lightning I have to go
> > double-check what git commit to use for each library (though `jsmn`
> > and `libbacktrace` at least seem very stable).
> > But it looks to me that unvendoring will require more extensive
> > patching of the `Makefile` and an even larger maintenance burden on
> > Guix side?
>
> Unvendoring is more or less a policy because we must be able to audit
> each piece of software separately for freedom issues (licenses,
> violations to the GNU FSDG), and it eases work for security-patching
> also.

I understand.

This will require a largish amount of work I think.

Would this technique be acceptable?

* `add-before 'configure 'unvendor-externals`
  * `rm -rf` the vendored externals.
  * `ln -s` the needed `.h` and `.la`/`.a`/`.so` files from the `inputs` to the expected paths within the `external/` directory.

?

> > Please do, I am not very familiar with any Python infrastructure and
> > am primarily a C programmer here, I just barely hack together some
> > kind of test in Python.
>
> If you can list the Python dependencies and their version I can look at
> packaging them.

We have a `requirements.txt` file which contains this, I duplicate below:

```
# Dependencies required to build and test c-lightning
https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp36-cp36m-linux_x86_64.whl; 'linux' in sys_platform and python_version == '3.6'
https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-linux_x86_64.whl; 'linux' in sys_platform and python_version == '3.7'
https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl; sys_platform == 'darwin' and python_version == '3.7'
mrkd ~= 0.1.6
Mako ~= 1.1.3

# Dependencies from pyln-client
Sphinx ~= 3.4.0
flake8==3.7.8
recommonmark>=0.7.*
sphinx-rtd-theme==0.4.2
sphinxcontrib-websupport==1.1.0
tqdm==4.32.2

# Dependencies from pyln-testing
Flask==1.1.*
cheroot==8.5.*
ephemeral-port-reserve==1.1.1
filelock==3.0.*
flaky ~= 3.7.0
psutil==5.7.*
psycopg2-binary==2.8.*
pytest-rerunfailures==9.1.1
pytest-timeout ~= 1.4.2
pytest-xdist ~= 2.2.0
pytest==6.1.*
python-bitcoinlib==0.11.*

# Dependencies from pyln-proto
base58 ~= 2.0.1
bitstring ~= 3.1.6
coincurve ~= 13.0.0
cryptography ~= 3.2
mypy ~= 0.790
pysocks ~= 1.7.1

# Dependencies from pyln-spec
# None
```

Incidentally, we also install some Python modules.
How do I "properly" export the Python modules within Guix?

Regards,
ZmnSCPxj

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2326 bytes --]

On Wed, 2021-03-17 at 03:42 +0000, ZmnSCPxj wrote:
> I understand.
> 
> This will require a largish amount of work I think.
> 
> Would this technique be acceptable?
> 
> * `add-before 'configure 'unvendor-externals`
>   * `rm -rf` the vendored externals.
>   * `ln -s` the needed `.h` and `.la`/`.a`/`.so` files from the
> `inputs` to the expected paths within the `external/` directory.
> 
> ?

Maybe simply add an option like --with-system-libbacktrace etc. in the
build system? And yes remove externals, I think that you should use a
snippet in the origin field instead of a phase for that.

> 
> We have a `requirements.txt` file which contains this, I duplicate
> below:
> 
> ```
> # Dependencies required to build and test c-lightning
> https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp36-cp36m-linux_x86_64.whl; 'linux' in sys_platform and python_version ==
> '3.6'
> https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-linux_x86_64.whl; 'linux' in sys_platform and python_version ==
> '3.7'
> https://github.com/ElementsProject/libwally-core/releases/download/release_0.8.0/wallycore-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl; sys_platform == 'darwin' and python_version ==
> '3.7'
> mrkd ~= 0.1.6
> Mako ~= 1.1.3
> 
> # Dependencies from pyln-client
> Sphinx ~= 3.4.0
> flake8==3.7.8
> recommonmark>=0.7.*
> sphinx-rtd-theme==0.4.2
> sphinxcontrib-websupport==1.1.0
> tqdm==4.32.2
> 
> # Dependencies from pyln-testing
> Flask==1.1.*
> cheroot==8.5.*
> ephemeral-port-reserve==1.1.1
> filelock==3.0.*
> flaky ~= 3.7.0
> psutil==5.7.*
> psycopg2-binary==2.8.*
> pytest-rerunfailures==9.1.1
> pytest-timeout ~= 1.4.2
> pytest-xdist ~= 2.2.0
> pytest==6.1.*
> python-bitcoinlib==0.11.*
> 
> # Dependencies from pyln-proto
> base58 ~= 2.0.1
> bitstring ~= 3.1.6
> coincurve ~= 13.0.0
> cryptography ~= 3.2
> mypy ~= 0.790
> pysocks ~= 1.7.1
> 
> # Dependencies from pyln-spec
> # None
> ```

Thank you, I am going to go through them.

> Incidentally, we also install some Python modules.
> How do I "properly" export the Python modules within Guix?

I will ask someone else here but you might have to mix the python-
build-system in.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[ZmnSCPxj via Guix-patches via]

Good morning Leo,

> On Wed, 2021-03-17 at 03:42 +0000, ZmnSCPxj wrote:
>
> > I understand.
> > This will require a largish amount of work I think.
> > Would this technique be acceptable?
> >
> > -   `add-before 'configure 'unvendor-externals`
> >     -   `rm -rf` the vendored externals.
> >     -   `ln -s` the needed `.h` and `.la`/`.a`/`.so` files from the
> >         `inputs` to the expected paths within the `external/` directory.
> >
> >
> > ?
>
> Maybe simply add an option like --with-system-libbacktrace etc. in the
> build system?

That would not be simple I think, but let me see what can be done.
For one, it would be fairly difficult to test it outside of a patched Guix, as the packages are not usually available in most distros (which is why we ship the source build with them); e.g. we do not know how to actually find the "system" libbacktrace as no existing system I know of actually ships the libbacktrace we use (Debian ships a libbacktrace for android development, and is not the same package as what we use).
Of the external packages we use, only libsodium is available on Debian (and older Debian/Ubuntu, as mentioned, have an older version that does not have a functionality we require, which is why it is included in our source build as well), so that is the only external we have that we can plausibly say "we can use the system version", because all the others are not available on most systems.

Also --- we would be somewhat wary of taking the "system" `libsecp256k1`, as this is very consensus-critical, and we know that our program works with a very specific version, but cannot assure this to be true if the "system" `libsecp256k1` is not the exact version.
(Note that we cannot take the `libsecp256k1` from `bitcoin-core` because (1) the `bitcoin-core` package uses a vendored `libsecp256k1`, the separate Guix `libsecp256k1` is actually only used by `electrum` and related projects and (2) the interface `bitcoin-core` uses may be different from the interface we use, it would require review before we would be able to assure that the interfaces they use are exactly the same; this would be mitigated by running the full test suite as it also tests consensus-criticality.)
Is there an easy way to ensure that any `libsecp256k1` that gets fed into the `c-lightning` build as an `inputs` is of a specific `git` commit?

> And yes remove externals, I think that you should use a
> snippet in the origin field instead of a phase for that.

Hmm do you mean something like:

    (origin
      ; whatever...
      (snippet
        '(begin
           (delete-file-recursively "external/gheap")
           (delete-file-recursively "external/jsmn")
           (delete-file-recursively "external/libbacktrace")
           (delete-file-recursively "external/libsodium")
           (delete-file-recursively "external/libwally-core"))))

?

> > Incidentally, we also install some Python modules.
> > How do I "properly" export the Python modules within Guix?
>
> I will ask someone else here but you might have to mix the python-
> build-system in.

How do you mix in a build system?
Would `python-pyqt` be a good example of such "mix in"?

Regards,
ZmnSCPxj

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 3761 bytes --]

On Thu, 2021-03-18 at 16:54 +0000, ZmnSCPxj wrote:
> That would not be simple I think, but let me see what can be done.
> For one, it would be fairly difficult to test it outside of a patched
> Guix, as the packages are not usually available in most distros
> (which is why we ship the source build with them); e.g. we do not
> know how to actually find the "system" libbacktrace as no existing
> system I know of actually ships the libbacktrace we use (Debian ships
> a libbacktrace for android development, and is not the same package
> as what we use).
> Of the external packages we use, only libsodium is available on
> Debian (and older Debian/Ubuntu, as mentioned, have an older version
> that does not have a functionality we require, which is why it is
> included in our source build as well), so that is the only external
> we have that we can plausibly say "we can use the system version",
> because all the others are not available on most systems.

We can package every of those in GNU Guix, there's not as much
bureaucracy as Debian when it comes to inclusion, as long as Free
Software and freedom-non-controversial. The "system" thing is just so
handy for us build-system wise we can just provide it as an input and
it will find things automatically, I don't say that "system" lookup
should imply to take any available library in general, in the context
of GNU Guix we can give it a particular version and stick to it, unlike
Debian's who can't have or with great difficulty, multiple versions of
the same library installed.

> Also --- we would be somewhat wary of taking the "system"
> `libsecp256k1`, as this is very consensus-critical, and we know that
> our program works with a very specific version, but cannot assure
> this to be true if the "system" `libsecp256k1` is not the exact
> version.
> (Note that we cannot take the `libsecp256k1` from `bitcoin-core`
> because (1) the `bitcoin-core` package uses a vendored
> `libsecp256k1`, the separate Guix `libsecp256k1` is actually only
> used by `electrum` and related projects and (2) the interface
> `bitcoin-core` uses may be different from the interface we use, it
> would require review before we would be able to assure that the
> interfaces they use are exactly the same; this would be mitigated by
> running the full test suite as it also tests consensus-criticality.)
> Is there an easy way to ensure that any `libsecp256k1` that gets fed
> into the `c-lightning` build as an `inputs` is of a specific `git`
> commit?

Yes we can package yet another variant like Bitcoin Core at a specific
commit for c-lightning.

> Hmm do you mean something like:
> 
>     (origin
>       ; whatever...
>       (snippet
>         '(begin
>            (delete-file-recursively "external/gheap")
>            (delete-file-recursively "external/jsmn")
>            (delete-file-recursively "external/libbacktrace")
>            (delete-file-recursively "external/libsodium")
>            (delete-file-recursively "external/libwally-core"))))
> 
> ?

Yes!

> How do you mix in a build system?
> Would `python-pyqt` be a good example of such "mix in"?

Maybe? I'm no expert here.

> Regards,
> ZmnSCPxj

FYI, I'm not as available for GNU Guix as I'd like, I am looking at the
Python deps, just it will take me some time to actually do it (not at
home now). I see python-mako is already packaged, python-wallycore I am
packaging it, python-mrkd also looking.

Also if we want to provide the Python bindings (?) to c-lightning we
also have the option of creating a separate package based on the same
sources but with different build-system (like python-build-system), and
also it can have c-lightning as an input. 

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[ZmnSCPxj via Guix-patches via]

Good morning Leo,

> On Thu, 2021-03-18 at 16:54 +0000, ZmnSCPxj wrote:
>
> > That would not be simple I think, but let me see what can be done.
> > For one, it would be fairly difficult to test it outside of a patched
> > Guix, as the packages are not usually available in most distros
> > (which is why we ship the source build with them); e.g. we do not
> > know how to actually find the "system" libbacktrace as no existing
> > system I know of actually ships the libbacktrace we use (Debian ships
> > a libbacktrace for android development, and is not the same package
> > as what we use).
> > Of the external packages we use, only libsodium is available on
> > Debian (and older Debian/Ubuntu, as mentioned, have an older version
> > that does not have a functionality we require, which is why it is
> > included in our source build as well), so that is the only external
> > we have that we can plausibly say "we can use the system version",
> > because all the others are not available on most systems.
>
> We can package every of those in GNU Guix, there's not as much
> bureaucracy as Debian when it comes to inclusion, as long as Free
> Software and freedom-non-controversial. The "system" thing is just so
> handy for us build-system wise we can just provide it as an input and
> it will find things automatically, I don't say that "system" lookup
> should imply to take any available library in general, in the context
> of GNU Guix we can give it a particular version and stick to it, unlike
> Debian's who can't have or with great difficulty, multiple versions of
> the same library installed.

Okay.

It is somewhat unclear to me however how to discover where the library is located.
As far as I understand how autotools does it, it looks at some standard locations on `/`, but I am uncertain if that is how it actually works.

For example if `libwally-core` is separately packaged, obviously it would install into a `/gnu/store/*` via a `--prefix` override.
And if the dependent package were using autotools, it is somehow able to magically find where the `lib*.a` file is.
However, C-Lightning does not actually use autotools (its build system predated my participation, I offered to port to autotools but this was declined), so I somehow need to emulate what autotools does in `--use-system-libraries` mode.

I probably have to go hack some autotools package that trivially depends on `libwally-core` and see what the exact shell script is generated.

> Also if we want to provide the Python bindings (?) to c-lightning we
> also have the option of creating a separate package based on the same
> sources but with different build-system (like python-build-system), and
> also it can have c-lightning as an input.

Note that AFAIK the Python bindings do not actually invoke any of the C-Lightning binaries, they just open the UNIX socket to the c-lightning daemon, so I think there is no need for `c-lightning` itself as an input.

Regards,
ZmnSCPxj

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

Hello!

> Okay.
> 
> It is somewhat unclear to me however how to discover where the
> library is located.
> As far as I understand how autotools does it, it looks at some
> standard locations on `/`, but I am uncertain if that is how it
> actually works.
> 
> For example if `libwally-core` is separately packaged, obviously it
> would install into a `/gnu/store/*` via a `--prefix` override.
> And if the dependent package were using autotools, it is somehow able
> to magically find where the `lib*.a` file is.
> However, C-Lightning does not actually use autotools (its build
> system predated my participation, I offered to port to autotools but
> this was declined), so I somehow need to emulate what autotools does
> in `--use-system-libraries` mode.
> 
> I probably have to go hack some autotools package that trivially
> depends on `libwally-core` and see what the exact shell script is
> generated.

pkg-config if your deps provide .pc files! Otherwise I am actually not
sure of the lookup strategies.

> Note that AFAIK the Python bindings do not actually invoke any of the
> C-Lightning binaries, they just open the UNIX socket to the c-
> lightning daemon, so I think there is no need for `c-lightning`
> itself as an input.

Alright!

> Regards,
> ZmnSCPxj

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47187] [PATCH] gnu: Add c-lightning.

[Léo Le Bouter via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

Just for your information, I am quite busy on a GNOME upgrade now, we
have the 1.2.1 release soon also, some security issues to solve, I told
you earlier I would help but at the end I think it may come a bit later
rather than sooner. Hang out on the IRC channel #guix on Freenode for
more real-time collaboration if that's of interest to you!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]