What support is there for distributed builds?

What support is there for distributed builds?

[Josh Marshall]

Hello all,

A colleague of mine has a set of scripts and program pipeline thing he
regularly uses to generate a few analysis files.  It just so happens that
how he handles this works identically to software packaging with a custom
build system.

Currently, his work has been put into a higher level language called
nextflow.  It is fairly analogous to gwl.  From this, what is the support
like for using external distributed computing tool, and gwl for building
packages?  This would be analogous to trying to build an utterly massive
codebase in a cloud environment.

Re: What support is there for distributed builds?

[zimoun]

Hi Josh,

Thank you for your email.

On Fri, 29 Nov 2019 at 23:53, Josh Marshall
<joshua.r.marshall.1991@gmail.com> wrote:

> A colleague of mine has a set of scripts and program pipeline thing he
> regularly uses to generate a few analysis files.  It just so happens that
> how he handles this works identically to software packaging with a custom
> build system.

Yes, GWL is based on similar ideas ;-)


> Currently, his work has been put into a higher level language called
> nextflow.  It is fairly analogous to gwl.  From this, what is the support
> like for using external distributed computing tool, and gwl for building
> packages?  This would be analogous to trying to build an utterly massive
> codebase in a cloud environment.

The advantages of GWL is to describe the workflow and all the tools
used by this very pipeline are automatically built by Guix itself.
Everything is under controlled... or almost. ;-)
At lot of pieces are still missing to be really usable. Well, (real)
life intervened. :-)

Feel free to wake up the mailing list gwl-devel@gnu.org. It needs love
and care. :-)


Maybe a good entry point about GWL is the PDF in [1]. Or this thread
[2]. Recent updates: thread [3] continuing [4].
Well, speaking about distribution, a patch [5] needs review, tests, etc.

Initially, GWL was designed to describe workflows used in
bioinformatics analysis. But an experimental workflow to build a
website has been tested [6]. Then I do not find where Pjotr suggested
to use GWL to replace Make (or related) but I am not sure GWL will go
to this direction; there is already too much build system [7]. :-)

What do you mean by "external distributed computing tool"?

Do not hesitate to ask more details, here or on gwl-devel@gnu.org.

Hope that helps.
simon


[1] https://lists.gnu.org/archive/html/guix-devel/2016-10/msg01248.html
[2] https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html
[3] https://lists.gnu.org/archive/html/gwl-devel/2019-05/msg00007.html
[4] https://lists.gnu.org/archive/html/gwl-devel/2019-06/msg00003.html
[5] https://lists.gnu.org/archive/html/gwl-devel/2019-01/msg00025.html
[6] https://lists.gnu.org/archive/html/gwl-devel/2019-07/msg00007.html
[7] https://github.com/snowleopard/build