From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 0KGMFfhjS2U0pAAA9RJhRA:P1 (envelope-from ) for ; Wed, 08 Nov 2023 11:33:28 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 0KGMFfhjS2U0pAAA9RJhRA (envelope-from ) for ; Wed, 08 Nov 2023 11:33:28 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C470A50A52 for ; Wed, 8 Nov 2023 11:33:18 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=deeplinks-com.20230601.gappssmtp.com header.s=20230601 header.b=WczWB9QC; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1699439598; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=AGjz9VH6/p0jzkfYLyn+LDVQKvQ8Rsk41/CUa//3WcA=; b=XxRApF8Pmz4vrMLAv4OI0awPdQUSOkbk1doYr5XXYpeTg2pKxJ+GnCDJcB8RZhv4zmEAHq MCGuTWlCTEihggVhv2ckYOYMbO9Vhu/f+7bh+oZ0FweMJKXPm+HgSFWHOI351ucUKLwJDy PSbTcCic5+Dv30DfVRbK2jOgZ0FrL5TwI2yK6bIJOK9Uh5JniYU6VPl3GIl+F5mg3RAbJ+ If+s/BDm83j8Lc5S4Ei0YeEyIvCTRysCndDdQE896rhSOozJbBoK+6FVo56e12zkqDZV8/ I6EoK8y6tV+sjQMQVBMGMsbBVJ5cZCpRpHEj0hriXJdP2eaDDY932M1QHczdXw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=deeplinks-com.20230601.gappssmtp.com header.s=20230601 header.b=WczWB9QC; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1699439598; a=rsa-sha256; cv=none; b=pZDrUSywRT6LVjUwjv72UqVrijiq5MtCEEGI43v0ZBopZvCJp3LQXQM0YVeMBdi0o/qYNw 2qMOMquHJc+8KUYy74fYZK5kCsKlzUt6ECCWdcNC/aBT9k1u987fG2n8iVZzpPfxzcw8+v SXW3+gu0X/lDg2nLm3UO3z6BTluoPgyDj3J0pNvErCng1qRY64P5sUzZ98snv3GETQFZsV LNNUJKKXB12VolAek1YSEday0KgKEFAffa79EPTecbnPDqRN1YhCwDIJpFUHSCMaRdV9/t p5TtKYsSGz2KEz4MeUvviIqQw62eY7N5UgbKnbOCehhGw+4pSiB3nJAFIaN5RA== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0eoQ-0000Xs-3h; Wed, 08 Nov 2023 04:25:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0eoO-0000XU-Fd for guix-patches@gnu.org; Wed, 08 Nov 2023 04:25:24 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0eoO-0005OK-7S for guix-patches@gnu.org; Wed, 08 Nov 2023 04:25:24 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1r0ep0-0002z2-8h for guix-patches@gnu.org; Wed, 08 Nov 2023 04:26:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#66801] [PATCH 0/5] build Erlang packages with dependencies References: <68117eb2b3e0e6adcc7449d878e602c7b831ffee.1698524350.git.phfrohring@deeplinks.com> In-Reply-To: <68117eb2b3e0e6adcc7449d878e602c7b831ffee.1698524350.git.phfrohring@deeplinks.com> Resent-From: Pierre-Henry =?UTF-8?Q?Fr=C3=B6hring?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 08 Nov 2023 09:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66801 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 66801@debbugs.gnu.org Cc: Pierre-Henry =?UTF-8?Q?Fr=C3=B6hring?= Received: via spool by 66801-submit@debbugs.gnu.org id=B66801.169943556011460 (code B ref 66801); Wed, 08 Nov 2023 09:26:02 +0000 Received: (at 66801) by debbugs.gnu.org; 8 Nov 2023 09:26:00 +0000 Received: from localhost ([127.0.0.1]:43998 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r0eox-0002yl-6k for submit@debbugs.gnu.org; Wed, 08 Nov 2023 04:25:59 -0500 Received: from mail-lf1-x12d.google.com ([2a00:1450:4864:20::12d]:52561) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r0eou-0002yV-Vh for 66801@debbugs.gnu.org; Wed, 08 Nov 2023 04:25:58 -0500 Received: by mail-lf1-x12d.google.com with SMTP id 2adb3069b0e04-507bd64814fso9033298e87.1 for <66801@debbugs.gnu.org>; Wed, 08 Nov 2023 01:25:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=deeplinks-com.20230601.gappssmtp.com; s=20230601; t=1699435512; x=1700040312; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=AGjz9VH6/p0jzkfYLyn+LDVQKvQ8Rsk41/CUa//3WcA=; b=WczWB9QCodxZanw3qmkfTtfcx0sHCNtpHTI/iBynDCWyRM9s3kgjq4oKDR7gibWz1E gqNWNd6sAqZ+IAv3cl+aYxZmgY9Hv581UwnF94gVqhaWywFXbZ8+sqJPdrkjjGjeFh4G V5tjG0tvPiicJaurg2SB+/gpHJTJhQBs9XiW4yyNlsfWV6Lyw68qkw8l2jcuscYJikEb 2ZUcj3qqUjChYEL1QDsB4uHEAYj76U9AJEwYUX4v91x0+me3WJhv8eKQd5qBbRpzxPwu NuFEsk126DSvygs+yd4Q8reCit6Cx5cCv5D7jcV1abtOKf0OT2la3v8qjW1rdahMKxxm 16+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699435512; x=1700040312; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AGjz9VH6/p0jzkfYLyn+LDVQKvQ8Rsk41/CUa//3WcA=; b=MuQAh3RrrrRkNcsV1fxnEZKSKGEIGtOElF5dBCXQuiFht0JhiE4xhchZCLF/CAVHwp vFWISmkGufm/CJJqazhSCydjjf2uGj37l0Ceg5h/tIBjpYY4ZbsUlyqxhW3UhGVaeXtK qQuhbeh08CYanDXssYA528KDiaJ8ht4Po0/8ZQ8HUk3IuvMlJPC4K1ypx+eOB5Sp+5BO 4J02Mn5B/QcWzfap0xxgjLgqe4uQjvXKuPH+pD3E8rpLZ2Cz8Q2pD2rBvjccncXLXHhG sxZIOfTtL2y6S/sVpD0jP8xyE35Vsi+0NjV1AN6Fd2m8MxdvAacKZ3jk71z5bBzy4D+a VIeQ== X-Gm-Message-State: AOJu0YzVHbRfXxHEvUgGoX3nQvJqifOf18oe9O+3haF/+IfV/7V6OOem amFDGojQ4z29o3rkMEM3sF0OdUzhHsnv72CUvdo= X-Google-Smtp-Source: AGHT+IEmGkIrzSKj5Igme3wEHFhTozIUbNiCzmSFsxF52HjcCRZPgHgGqFWPMyw7vg3cnAWLEQkuSg== X-Received: by 2002:a19:5e04:0:b0:4fe:8c1d:9e81 with SMTP id s4-20020a195e04000000b004fe8c1d9e81mr531894lfb.36.1699435511316; Wed, 08 Nov 2023 01:25:11 -0800 (PST) Received: from doug.com ([185.244.73.162]) by smtp.gmail.com with ESMTPSA id o8-20020a05600c510800b00407752f5ab6sm18841789wms.6.2023.11.08.01.25.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Nov 2023 01:25:10 -0800 (PST) From: Pierre-Henry =?UTF-8?Q?Fr=C3=B6hring?= Date: Wed, 8 Nov 2023 10:22:34 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -1.11 X-Spam-Score: -1.11 X-Migadu-Queue-Id: C470A50A52 X-Migadu-Scanner: mx13.migadu.com X-TUID: eMKqI+qmFv6/ Introduction ════════════ • The objective is to introduce a `mix-build-system' for building Elixir packages. • It is first necessary to have `rebar-build-system' to build Erlang packages. • We start by showing a build procedure using a package named `lager' that operates without the need for network access. • We generalize the build procedure to any Erlang package by using [Checkout Dependencies]. • To align this build procedure with Guix, we ensure that for any Erlang Guix package, all its dependencies, including the transitive ones, can be identified. • We show that the source code for each dependency is retrievable from the store. • We modify `rebar.scm' and `rebar-build-system.scm' to build procedure into Guix. • The packages defined in `erlang-xyz.scm' serve as the test cases for this implementation. • The changes to the Guix source code are minimal. While the primary goal is to validate the approach, it may be necessary to resolve stylistic issues or potential bugs. [Checkout Dependencies] Bash ════ Rebar3 compiles and tests Lager without network access ────────────────────────────────────────────────────── There is a directory and an environment such that `rebar3 compile' and `rebar3 test' compile and test the [lager package] without network access. [lager package] Sources are fetched and unpacked ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ #! /usr/bin/env bash │ set -euo pipefail │ IFS=$'\n\t' │ │ tmpdir=$(mktemp -d) │ cd $tmpdir │ wget -O source.tar https://repo.hex.pm/tarballs/lager-3.9.2.tar │ tar xf source.tar │ tar xf contents.tar.gz └──── Dependencies are fetched ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ rebar3 compile │ rebar3 eunit └──── Checkouts are built ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ #+begin_src bash mkdir _checkouts mv _build/default/lib/goldrush _checkouts rm -rf _build #+end_srcè The directory is built ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ . │ ├── _checkouts │ │   └── goldrush │ │   ├── ebin │ │   │   ├── … │ │   │   └── gr_sup.beam │ │   ├── hex_metadata.config │ │   ├── LICENSE │ │   ├── priv │ │   │   └── edoc.css │ │   ├── README.org │ │   ├── rebar.config │ │   ├── rebar.lock │ │   └── src │ │   ├── … │ │   └── gr_sup.erl │ ├── CHECKSUM │ ├── contents.tar.gz │ ├── include │ │   └── lager.hrl │ ├── LICENSE │ ├── metadata.config │ ├── priv │ │   └── edoc.css │ ├── README.md │ ├── rebar.config │ ├── rebar.config.script │ ├── rebar.lock │ ├── source.tar │ ├── src │ │   ├── … │ │   └── lager_util.erl │ └── VERSION │ │ 9 directories, 73 files └──── The environment is defined ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ (specifications->manifest '("coreutils" "rebar3")) └──── Lager is compiled and tested without network access ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ guix shell -C -m manifest.scm -- rebar3 compile │ guix shell -C -m manifest.scm -- rebar3 eunit └──── Erlang can find and use the compiled module ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┌──── │ ERL_LIBS=_build/default/lib erl -noshell -eval 'io:format("~p~n", [code:which(lager)]).' -s init stop └──── ┌──── │ "_build/default/lib/lager/ebin/lager.beam" └──── Discussion ══════════ Several observations can be made, and alternative approaches can be proposed, as listed below. Meanwhile, with the implementation described above, it becomes feasible to build Erlang packages with Guix, which is an essential step towards building Elixir packages. Wasted Computations ─────────────────── This build procedure needs to re-build of all Erlang dependencies for an Erlang package, even if they have been previously built. This is due to the `rebar3' build system's refusal to operate without network access unless the dependencies are checked out. If checked out, it insists on rebuilding them from sources. Expensive computations ────────────────────── For a given Erlang dependency, it is necessary to retreive all its transitive dependencies which might be expensive to compute. See: [all-transitive-inputs]. [all-transitive-inputs] <./guix/build-system/rebar.scm::(define (all-transitive-inputs> Wasted Space ──────────── The source code for all Erlang dependencies of a specific Erlang package must be available on the build machine, whereas it might be assumed that the build artifacts alone would suffice. Alternatives ──────────── Liliana Marie Pinker has suggested several alternatives listed below. Store sources along the build artifacts ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ From [issue/66801]: keep the sources in lib/erlang/lib/lib-name/src so that it gets symlinked by the phase we have. Though at that point we can surely go with a less surprising install directory. [issue/66801] Write the sources to a special source output ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ From [issue/66801]: You could alternatively also write the sources to a special source output if and only if one such output is requested via the outputs field. Again, I'd hazard a guess that this would be a very standard output for anything that needs to go into mix-build-system and thus not worth the split after all. [issue/66801] Patch Rebar3 ╌╌╌╌╌╌╌╌╌╌╌╌ From [issue/66801]: you could patch mix to only look for compiled stuff and not sources. That would tackle the issue at the root instead of trying to work around it, with the caveat being that we would need to maintain this patch ourselves if upstream doesn't accept it. [issue/66801] Pierre-Henry Fröhring (5): guix: build-system: rebar: build Erlang packages with dependencies. gnu: Add erlang-goldrush. gnu: Add erlang-lager. gnu: Add erlang-unicode-util-compat. gnu: Add erlang-idna. gnu/packages/erlang-xyz.scm | 111 +++++++++++++++ guix/build-system/rebar.scm | 223 ++++++++++++++++++++++-------- guix/build/rebar-build-system.scm | 43 +++--- 3 files changed, 300 insertions(+), 77 deletions(-) create mode 100644 gnu/packages/erlang-xyz.scm base-commit: 4dfbc536689b07e56aead3dd864b8af54613d091 -- 2.41.0