GWL as a build-automation

GWL as a build-automation

[Olivier Dion via]

Hi,

I've been hacking around for the past weeks on a build system that uses
Guix.  The more I write it, the more I find similarities with what GWL
already does/provides.


So I'm thinking, one could easily do something like this for a simple
Guile project:

--8<---------------cut here---------------start------------->8---
process compile-guile-source (with dot-scm)
  package "guile"
  inputs  : file dot-scm
  outputs : file : string-replace-substring dot-scm ".scm" ".go"
  # {
    guild compile --output {{outputs}} {{inputs}}
  }
--8<---------------cut here---------------end--------------->8---

And this could also be done for other projects, e.g.:

--8<---------------cut here---------------start------------->8---
process compile-c-source (with dot-c)
  package "gcc-toolchain"
  inputs  : file dot-c
  outputs : file : string-replace-substring dot-c ".c" ".o"
  # {
    gcc -c -o {{outputs}} {{inputs}}
  }
--8<---------------cut here---------------end--------------->8---


I know that GWL focuses on scientific workflows, but I think that it can
go beyond its original goal.

Thoughts?

-- 
Olivier Dion
oldiob.dev

Re: GWL as a build-automation

[Ricardo Wurmus]

Hi Olivier,

> So I'm thinking, one could easily do something like this for a simple
> Guile project:
>
> process compile-guile-source (with dot-scm)
>   package "guile"
>   inputs  : file dot-scm
>   outputs : file : string-replace-substring dot-scm ".scm" ".go"
>   # {
>     guild compile --output {{outputs}} {{inputs}}
>   }
>
>
> And this could also be done for other projects, e.g.:
>
> process compile-c-source (with dot-c)
>   package "gcc-toolchain"
>   inputs  : file dot-c
>   outputs : file : string-replace-substring dot-c ".c" ".o"
>   # {
>     gcc -c -o {{outputs}} {{inputs}}
>   }

It is not entirely surprising to me that the GWL can express this,
because it has really simple abstractions: that of a process and that of
a workflow consisting of processes.

What I do find a little surprising, though, is that there is an apparent
need for declaring processes like this.  The Scheme Shell (scsh), for
example, takes on a similar problem, but the abstraction is different;
less focused on inputs and outputs and instead more geared to
integrating Shell script idioms with plain Scheme.

Personally, I find the GWL too complex for mere process abstractions and
lacking in features that are common in Shell scripts.  Notably, it punts
on improving the syntax of the command; it shrugs and just uses a string
to express the whole command (with a little variable interpolation).

Perhaps there is space for a different tool that takes lessons from the
GWL and Scsh alike, with a focus on command composition and shell
abstractions.  Perhaps that tool already exists and is called Metabash:

  https://github.com/artyom-poptsov/metabash

:)

-- 
Ricardo

Re: GWL as a build-automation

[zimoun]

Hi,

I am late to the party. :-)

On Mon, 06 Jun 2022 at 10:50, Ricardo Wurmus <rekado@elephly.net> wrote:

> It is not entirely surprising to me that the GWL can express this,
> because it has really simple abstractions: that of a process and that of
> a workflow consisting of processes.

[...]

> Perhaps there is space for a different tool that takes lessons from the
> GWL and Scsh alike, with a focus on command composition and shell
> abstractions.  Perhaps that tool already exists and is called Metabash:
>
>   https://github.com/artyom-poptsov/metabash

From my understanding, metabash allows to remotely run processes, i.e.,
distribute the pipeline.  Somehow, it could be see as an extension of
Scsh.

However, a pipeline is a linear sequence of processes.  When a workflow
is a DAG of processes.  Therefore, it would appear difficult to me to be
able to express a build-system using only pipelines.

Last, it appears to me expected that GWL could be considered as a
build-system.  A scientific workflow system [1] (as GWL) is just a
specialized implementation to deal with a graph of dependencies.
Software folks speak about the venerable Make as build automation
workflow, while bioinfo folks speak about a specific Python
implementation SnakeMake as data analysis workflow. Just the same
concepts but viewed by different communities. :-)

If I might, an interesting analysis of different strategies for dealing
with the graph of dependencies is done in the paper «Build systems à la
carte» [2].  It presents the various abstractions using Haskell
notations.

1: <https://en.wikipedia.org/wiki/Scientific_workflow_system>
2: <https://doi.org/10.1017/S0956796820000088>


Cheers,
simon