Re: Getting rid of build tools

[Pjotr Prins <pjotr.public12@thebird.nl>]

On Tue, Dec 29, 2015 at 04:37:16PM +0100, Ludovic Courtès wrote:
> The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
> in the paper would be quite easy to write using our current APIs.

I am thinking about this and maybe I'll give it a go at some point. A
functional approach to buildtools incorporating Hashes makes a lot of
sense. Guile can do composition, so that ticks a box. Guile has
futures, that is another box ticked.

Rather than calculating a dependency graph explitely I would like to
use futures to have the Guile compiler implicitly create a graph which
gets executed in parallel. I am not sure Guile can handle that type of
parallelism at a large scale, even though I find futures in the docs

  https://www.gnu.org/software/guile/manual/html_node/Futures.html

Eelco's Maak paper, interestingly, also mentions the 1 second
limitation of make as a potential source of race conditions. I would
like to have the option of using either time stamps or hash values to
decide whether a compilation step has completed succesfully. SHA
values arguably take longer to compute (actually pfff is a fast
alternative for large data https://github.com/pfff/pfff), but I want
Hash values for the next thing:

I would like to have cluster compilation where jobs are put out in an
opportunistic fashion. I.e., we assume semi-unlimited resources and
place jobs multiple times. The one job that completes first is the one
we take (the others may get killed). This would allow for longer
running jobs to be put out to a heterogeneous environment (think
cluster w. NFS, think cloud, think grid, think phone). For build environents
this could already be of interest, for bioinformatics large data
compute it could be brilliant. It may look wasteful, but truth is most
compute environments are happily idle and have differing bottlenecks
which would help speed up compilation (fastest one wins).

Anyone interested in this project? You know, there are quite a few
workflow initiatives (recent one the CWL) and build tool initiatives
(CMake and sbt, for example, are fairly recent), but I believe they
miss out on real software engineering solutions! Maybe we can get it
right. My question, is any really talented programmer here interested
to work on this and write a prototype? If you are a student we can
even try and get some of the work funded.