[bug#66801] [PATCH 1/5] guix: build-system: rebar: build Erlang packages with dependencies.

["Pierre-Henry Fröhring" <phfrohring@deeplinks.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2004 bytes --]

      This loses the previous erlang-depends logic, which unpackaged the
      already compiled packages.  I really don't think we should have
erlang be
      yet another rust that heats up the planet compiling leftpad over and
over
      again.  Can we somehow have that cake?

Okay, perhaps it would have been better to start working on the Mix
building system
and inform you simultaneously, instead of sending patches all at once. So,
let's
start anew.

I have taken into account all of your suggestions and have crafted this
patch
introducing the mix-build-system. Yes, there are ChangeLogs now. Currently,
the
"Machete" package is compiled and tested without repeatedly compiling
leftpad — i.e.,
considering pure Elixir packages up to transitive dependencies, it works as
expected. This is new.

As I'm learning how all this works, I'm not entirely sure that updating the
rebar
build system is absolutely necessary. I suppose we will understand more as
we add an
increasing number of Elixir packages, especially when incorporating Elixir
packages
with Erlang dependencies.

So, here's what's coming up:

1. A patch that introduces the mix-build-system variable.
2. Several patches that introduce Elixir packages, up to elixir-machete,
which is one
   of the few packages that includes tests in the tarball.

I guess the next steps are as follows:

1. To determine if the approach is acceptable. Will we boil the world by
compiling
   left-pad endlessly?
2. If the approach is deemed acceptable, the next step is to evaluate the
code. Is
   the code riddled with delicious regexes?
3. If the first two points are verified, it might be worthwhile to continue
   developing this implementation until Phoenix is successfully compiled
and tested
   as expected.

For reference, I have attached a file named `mix-build-system.org'.
This file contains a description of a prototype in Bash that
illustrates the approach taken in this patch.

Cheers.

[-- Attachment #1.2: Type: text/html, Size: 2218 bytes --]

[-- Attachment #2: mix-build-system.org --]
[-- Type: application/octet-stream, Size: 12495 bytes --]

# begin_properties
#+PROPERTY: header-args :noweb yes
# end_properties


#+TITLE: Mix build system


* Introduction
The aim is to introduce a ~mix build system~, a standardized method that enables Guix
to construct Elixir packages. Initially, we establish a Bash procedure for building
Elixir packages. We then explore the adaptation of this procedure to Guix.


* A procedure to build Elixir packages
The build procedure is divided into two steps: ~build_directory~ and
~compile_test_install~. These steps use arguments that mirror the project configuration
options outlined in [[https://hexdocs.pm/mix/1.14/Mix.Project.html][Mix.Project]].
#+begin_src bash
guix shell -D -f guix.scm -- bash build_directory \
     --archive <filename> \
     --build-per-environment <boolean> \
     --log <boolean> \
     --out <path> \
     --deps-path <path> \
     --test <boolean> \
     --workdir <path>

guix shell -C -D -f guix.scm -- bash compile_test_install \
     --archive <filename> \
     --build-per-environment <boolean> \
     --log <boolean> \
     --out <path> \
     --deps-path <path> \
     --test <boolean> \
     --workdir <path>
#+end_src
- ~archive~ refers to the name of the tarball to be retrieved from [[https://hex.pm/]], for example, ~phoenix-1.7.10.tar~.
- ~build-per-environment~ is a setting detailed in [[https://hexdocs.pm/mix/1.14/Mix.Project.html][Mix.Project]].
- ~log~ indicates whether the execution should be logged.
- ~test~ determines whether to run tests.
- ~out~ specifies the directory path where the package will be installed.
- ~deps~ is a configuration option explained in [[https://hexdocs.pm/mix/1.14/Mix.Project.html][Mix.Project]].
- ~workdir~ is either the absolute or relative path to the current directory where the build process will take place.


** Environment
:PROPERTIES:
:ID:       5f55ed2d-6301-460b-bc78-c4100e95e274
:END:

Except for ~tree~, all native inputs are necessary to execute the build
procedure. These inputs are listed in the ~native-inputs~ property of the package in
~guix.scm~.
#+caption: guix.scm
#+begin_src scheme :tangle guix.scm
(use-modules (guix build-system gnu)
             (gnu packages)
             (guix packages)
             (gnu packages base)
             (gnu packages elixir)
             (gnu packages erlang)
             (gnu packages admin))

(define glibc-utf8-locales
  (make-glibc-utf8-locales glibc
                           #:locales (list "en_US")
                           #:name "glibc-utf8-locales"))

(package
  (name "")
  (version "")
  (source #f)
  (build-system gnu-build-system)
  (native-inputs (list
                  glibc-utf8-locales
                  coreutils
                  which
                  elixir
                  erlang
                  rebar3
                  tree
                  ))
  (synopsis "")
  (description "")
  (home-page "")
  (license #f))
#+end_src

** The ~build_directory~ Step
The ~build_directory~ step involves using the network to fetch any external data
required for compiling and testing the package. Each step of the script is detailed
in the following subsections.
#+name: build_directory
#+begin_src bash :shebang "#! /usr/bin/env bash" :tangle build_directory :exports none
<<script-conf>>

<<arguments>>

<<debug>>

<<workdir>>

<<env>>

<<hex>>

<<rebar3>>

<<archive>>

<<unpack>>

<<deps>>

<<show:0>>
#+end_src


*** The Bash script is configured
#+name: script-conf
#+begin_src bash
set -euo pipefail
IFS=$'\n\t'
#+end_src


*** Arguments
#+name: arguments
#+begin_src bash
DEPS_DIR="deps"
LOG="false"
BUILD_PER_ENVIRONMENT="true"
while [ "${1:-}" != "" ]; do
  case "$1" in
    --archive)
        shift
        ARCHIVE=$1
      ;;
    --log)
        shift
        LOG=$1
      ;;
    --build-per-environment)
        shift
        BUILD_PER_ENVIRONMENT=$1
      ;;
    --out)
        shift
        OUT=$1
      ;;
    --deps-path)
        shift
        DEPS_DIR=$1
      ;;
    --test)
        shift
        TEST=$1
      ;;
    --workdir)
        shift
        WORKDIR=$1
      ;;
    ,*)
      echo "Invalid argument: $1"
      exit 1
  esac
  shift
done
#+end_src


*** Debug mode
#+name: debug
#+begin_src bash
[[ $LOG == "true" ]] && set -x
#+end_src


*** Work directory
#+name: workdir
#+begin_src bash
mkdir -p $WORKDIR
cd $WORKDIR
#+end_src


*** Environment variables
#+name: env
#+begin_src bash :exports none
<<env:0>>

<<env:1>>

<<env:2>>

<<env:3>>

<<env:4>>

<<env:5>>
#+end_src

The current locale must be set to a ~UTF-8~ compatible format; otherwise, the Erlang VM
will emit a warning: "Warning: The VM is running with the native name encoding of
latin1, which may cause Elixir to malfunction, as it expects UTF-8. Please ensure
your locale is set to UTF-8 (which can be verified by running 'locale' in your
shell)."
#+name: env:0
#+begin_src bash
export LC_ALL=en_US.UTF-8
#+end_src

~mix~ expects to have access to a few local directories.
#+name: env:1
#+begin_src bash
export MIX_HOME=$PWD/mix-home
export MIX_ARCHIVES=$MIX_HOME/archives
mkdir -p "$MIX_ARCHIVES"
#+end_src

~mix~ expects to install dependencies sources in ~DEPS_DIR~.
#+name: env:2
#+begin_src bash
mkdir -p $DEPS_DIR
#+end_src

~mix~ installs compilations artifacts under the ~BUILD_DIR~ directory.
#+name: env:3
#+begin_src bash
export BUILD_DIR=_build
mkdir -p "$BUILD_DIR"
#+end_src

~mix~ expects to fetch tarballs from ~HEX_PM~.
#+name: env:4
#+begin_src bash
export HEX_PM=https://repo.hex.pm/tarballs
#+end_src

This procedure installs compilation artifacts under ~ELIXIR_LIB~. ~1.14~ is the version
number of the current Elixir used.
#+name: env:5
#+begin_src bash
export ELIXIR_LIB=lib/elixir/1.14
#+end_src


*** Hex is installed
Hex must be installed under ~MIX_ARCHIVES~; otherwise, ~mix~ will refuse to compile.
#+name: hex
#+begin_src bash
mix local.hex --force
[[ $LOG == "true" ]] && tree $MIX_HOME
#+end_src


*** Rebar3 is installed
Rebar3 must be installed under ~MIX_ARCHIVES~; otherwise, ~mix~ will refuse to compile if
Erlang dependencies are present.
#+name: rebar3
#+begin_src bash
mix local.rebar --force rebar3 $(which rebar3)
[[ $LOG == "true" ]] && tree $MIX_HOME
#+end_src


*** The archive is fetched
#+name: archive
#+begin_src bash
if [[ -f "../$ARCHIVE" ]]; then
    cp "../$ARCHIVE" ./
else
    wget $HEX_PM/$ARCHIVE >/dev/null
    cp "$ARCHIVE" ../
fi
ls -al "$ARCHIVE"
#+end_src


*** The archive is unpacked
Sources are unpacked in the current directory.
#+name: unpack
#+begin_src bash
tar xf $ARCHIVE
tar xvf contents.tar.gz
#+end_src


*** Dependencies are installed
:PROPERTIES:
:ID:       9526e510-90cf-4595-bcad-4411ea822c23
:END:

#+name: deps
#+begin_src bash :exports none
<<deps:0>>

<<deps:1>>

<<deps:2>>
#+end_src

~MIX_ENVS~ represents all environments that should be considered for building and
testing the package.
#+name: deps:0
#+begin_src bash
if [[ "$BUILD_PER_ENVIRONMENT" == "true" ]]; then
    MIX_ENVS=( "prod" )
    if [[ $TEST == "true" ]]; then
        MIX_ENVS+=( "test" )
    fi
else
    MIX_ENVS=( "shared" )
fi
#+end_src

then, for each ~MIX_ENV~, we get the dependencies and compile them. For each
environment ~MIX_ENV~, the result will be installed in ~_build/$MIX_ENV~.
#+name: deps:1
#+begin_src bash
for profile in ${MIX_ENVS[@]}; do
    export MIX_ENV=$profile

    mix deps.get --only $MIX_ENV
    [[ $LOG == "true" ]] && tree $DEPS_DIR

    mix deps.compile
    [[ $LOG == "true" ]] && tree $BUILD_DIR
done
#+end_src

then, we remove the sources as we only need the compiled dependencies.
#+name: deps:2
#+begin_src bash
rm -rf $DEPS_DIR
#+end_src


*** What has been built is shown
#+name: show:0
#+begin_src bash
tree -L 3 ${BUILD_DIR}
tree -L 3 ${MIX_HOME}
#+end_src


** The ~compile_test_install~ step
The ~compile_test_install~ step compiles, tests, and installs the package within a
container.
#+name: compile_test_install
#+begin_src bash :shebang "#! /usr/bin/env bash" :tangle compile_test_install :exports none
<<script-conf>>

<<arguments>>

<<debug>>

<<workdir>>

<<env>>

<<test>>

<<compile>>

<<install>>

<<show:1>>
#+end_src


*** The package is tested
The compilation environment ~MIX_ENV~ is computed then the package is compiled,
installed in ~_build/$MIX_ENV~ and tested.
#+name: test
#+begin_src bash
if [[ "$TEST" == "true" ]]; then
    if [[ "$BUILD_PER_ENVIRONMENT" == "true" ]]; then
        export MIX_ENV=test
    else
        export MIX_ENV=shared
    fi
    TEST_DIR=$BUILD_DIR/$MIX_ENV
    mix test --no-deps-check
    [[ $LOG == "true" ]] && tree $TEST_DIR
fi
#+end_src


*** The package is built
The compilation environment ~MIX_ENV~ is computed then the package is compiled and
installed in ~_build/$MIX_ENV~.
#+name: compile
#+begin_src bash
if [[ "$BUILD_PER_ENVIRONMENT" == "true" ]]; then
    export MIX_ENV=prod
else
    export MIX_ENV=shared
fi
PROD_DIR=$BUILD_DIR/$MIX_ENV
mix compile --no-deps-check
[[ $LOG == "true" ]] && tree $PROD_DIR
#+end_src


*** The package is installed
The Elixir version number ~X.Y~ is computed and the built package is installed under
~$OUT/lib/elixir/X.Y/~.
#+name: install
#+begin_src bash
NAME="${ARCHIVE%%-*}"
INSTALL_DIR=$OUT/${ELIXIR_LIB}/${NAME}
mkdir -p $INSTALL_DIR
# …/* because we don't want to copy hidden files.
cp -vLrf $PROD_DIR/${ELIXIR_LIB%%/*}/${NAME}/* $INSTALL_DIR
#+end_src


*** What has been built is shown
#+name: show:1
#+begin_src bash
tree -L 3 $BUILD_DIR
tree -L 4 $OUT
#+end_src


* A few packages have been built
** ~makeup~
#+begin_src bash
WORKDIR=$PWD/workdir
INSTALLDIR=$WORKDIR/install
rm -rf $WORKDIR $INSTALLDIR

guix shell -D -f guix.scm -- bash build_directory \
     --archive makeup-1.1.1.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test false \
     --workdir $WORKDIR

guix shell -C -D -f guix.scm -- bash compile_test_install \
     --archive makeup-1.1.1.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test false \
     --workdir $WORKDIR
#+end_src


** ~machete~
#+begin_src bash
WORKDIR=$PWD/workdir
INSTALLDIR=$WORKDIR/install
rm -rf $WORKDIR $INSTALLDIR

guix shell -D -f guix.scm -- bash build_directory \
     --archive machete-0.2.8.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test true \
     --workdir $WORKDIR

guix shell -C -D -f guix.scm -- bash compile_test_install \
     --archive machete-0.2.8.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test true \
     --workdir $WORKDIR
#+end_src


** ~phoenix~
#+begin_src bash
WORKDIR=$PWD/workdir
INSTALLDIR=$WORKDIR/install
rm -rf $WORKDIR $INSTALLDIR

guix shell -D -f guix.scm -- bash build_directory \
     --archive phoenix-1.7.10.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test false \
     --workdir $WORKDIR

guix shell -C -D -f guix.scm -- bash compile_test_install \
     --archive phoenix-1.7.10.tar \
     --log true \
     --out $INSTALLDIR \
     --deps-path deps \
     --test false \
     --workdir $WORKDIR
#+end_src


* Adapting the procedure to Guix
The goal is to answer the question: given the above procedure, how to adapt it to
Guix? We examine each step and, when necessary, map it to Guix.
1) The build procedure should match the ~mix-build-system~ variable.
2) For all Elixir package, the ~native-inputs~ of ~guix.scm~ except ~tree~ should be in
   default native-inputs.
3) The Elixir version number should be computed from the Elixir used to compile.
4) Arguments should be provided: ~#:tests?~, ~#:deps-path~, ~#:build-per-environment~.
5) A phase should install ~Hex~.
6) A phase should install ~Rebar3~.
7) The ~unpack~ phase should be adapted to tarballs and source checkouts.
8) All Erlang and Elixir packages should be deduced from all the inputs of the
   package being built and installed in the necessary build directories to mimic the
   [[id:9526e510-90cf-4595-bcad-4411ea822c23][Dependencies are installed]].


* Conclusion
  A preliminary build procedure for Elixir packages, designed for integration into
  Guix, has been presented. The issue of bit reproducibility within the build process
  requires further investigation. If the current methodology is deemed satisfactory,
  the subsequent step will be to implement this procedure in the Guix build system.


* Annex
** ~build_directory~
#+begin_src bash
<<build_directory>>
#+end_src


** ~compile_test_install~
#+begin_src bash
<<compile_test_install>>
#+end_src


# Local Variables:
# fill-column: 85
# End: