Getting rid of build tools

Getting rid of build tools

[Pjotr Prins]

Hi all,

2015 was a good year for GNU Guix - Guix has made immense progress. Some
thoughts for 2016:

I am a software developer by trade and for years I have struggled with
build systems, such as configure/automake, cmake, Ruby RVM/bundler,
Python virtualenv etc. etc. You probably know I have already ditched
RVM/bundler and virtualenv for GNU Guix which is great :)

Recently it dawned on me that for programming with GNU Guix there is
no longer a need for configure/automake and cmake either! These tools
really try to address the problem of targetting different (posix)
build environments. If I only target GNU Guix I think a simple make
will do again because there are only a few final targets (test, debug,
install) and GNU Guix resolves all dependencies. This greatly
simplifies the task of the software developer.

I am not going to let tears over losing these complex build tools. And
being a Linux guy I am happy to only target Linux. The different
virtualization solutions make deployments on different systems quite
easy anyway and trivial with Guix because it comes with all
dependencies.

Even so, my prediction is that eventually other systems will be
targeted too.  Even though there currently is not much GNU Guix
initiative outside Linux/Hurd I think people will start working on
other ports. The Guix/Nix back-end already runs on the BSDs, for
example. So, it is mostly a matter of adapting the Guix front-end.

Happy hacking in 2016 :)

Pj.

Re: Getting rid of build tools

[Christopher Allan Webber]

Pjotr Prins writes:

> Hi all,
>
> 2015 was a good year for GNU Guix - Guix has made immense progress. Some
> thoughts for 2016:
>
> I am a software developer by trade and for years I have struggled with
> build systems, such as configure/automake, cmake, Ruby RVM/bundler,
> Python virtualenv etc. etc. You probably know I have already ditched
> RVM/bundler and virtualenv for GNU Guix which is great :)
>
> Recently it dawned on me that for programming with GNU Guix there is
> no longer a need for configure/automake and cmake either! These tools
> really try to address the problem of targetting different (posix)
> build environments. If I only target GNU Guix I think a simple make
> will do again because there are only a few final targets (test, debug,
> install) and GNU Guix resolves all dependencies. This greatly
> simplifies the task of the software developer.
>
> I am not going to let tears over losing these complex build tools. And
> being a Linux guy I am happy to only target Linux. The different
> virtualization solutions make deployments on different systems quite
> easy anyway and trivial with Guix because it comes with all
> dependencies.
>
> Even so, my prediction is that eventually other systems will be
> targeted too.  Even though there currently is not much GNU Guix
> initiative outside Linux/Hurd I think people will start working on
> other ports. The Guix/Nix back-end already runs on the BSDs, for
> example. So, it is mostly a matter of adapting the Guix front-end.

I appreciate and share your pro-guix enthusiasm, and I share your
frustration fighting build and packaging systems, though I think there
are some good reasons to have the traditional build tools, with and
without Guix.  Portability is one nice one, but:

 - Compiling only the files needed during development is important
 - It's nice to be able to have a standard way to switch on/off features
   (see emacs with and without X packages in Guix)
 - Finding what to link to at compile time

Other things I'm sure (standardizing how to run tests and stuff saves
some time while packaging, etc).

I guess these things could be done through Guix itself.  What I'd rather
see though is a "./configure && make" compatible interface system for
package building which uses Guile as its configuration language.
Automake and friends are great when they're working, but I'd love a
system that relied less on string-macro expansion and etc when I have to
debug them.

I think that would be an interesting project.  It would be nice to see
it happen outside of Guix though, so it could be used more generally.

 - Chris

Re: Getting rid of build tools

[Pjotr Prins]

On Mon, Dec 28, 2015 at 09:06:40AM -0600, Christopher Allan Webber wrote:
> I guess these things could be done through Guix itself.  What I'd rather
> see though is a "./configure && make" compatible interface system for
> package building which uses Guile as its configuration language.
> Automake and friends are great when they're working, but I'd love a
> system that relied less on string-macro expansion and etc when I have to
> debug them.
> 
> I think that would be an interesting project.  It would be nice to see
> it happen outside of Guix though, so it could be used more generally.

I agree. It is actually in line with Eelco Dolstra's thinking more
than 10 years ago (he created 'Maak'), though he did not use Guile.

I am not saying we don't need build tools altogether - I'll still use
make and rake, for example. But we certainly don't need all the
baggage that comes with the current set of 'complex' Makefile
generation tools that are automake and cmake. I guess it should be
somewhere in between :). Yes a configurable Guile solution would be
excellent.

Pj.

Re: Getting rid of build tools

[Pjotr Prins]

On Mon, Dec 28, 2015 at 08:05:36PM +0100, Pjotr Prins wrote:
> On Mon, Dec 28, 2015 at 09:06:40AM -0600, Christopher Allan Webber wrote:
> I agree. It is actually in line with Eelco Dolstra's thinking more
> than 10 years ago (he created 'Maak'), though he did not use Guile.
> 
> I am not saying we don't need build tools altogether - I'll still use
> make and rake, for example. But we certainly don't need all the
> baggage that comes with the current set of 'complex' Makefile
> generation tools that are automake and cmake. I guess it should be
> somewhere in between :). Yes a configurable Guile solution would be
> excellent.

For anyone interested, Eelco's description of Maak, a functional make,
is here:

  http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf

-- 

Re: Getting rid of build tools

[Pjotr Prins]

On Tue, Dec 29, 2015 at 02:47:23AM +0100, Pjotr Prins wrote:
> For anyone interested, Eelco's description of Maak, a functional make,
> is here:
> 
>   http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf

This is actually an interesting topic. GNU Make does a number of
things well (note, btw, that it has Guile support), but it has
problems too. In short, in addition to 

1. The sucky non-functional macro system 

2. Make files are hard to read (and maintain)

3. Does dependencies on time stamps

the last is a real problem on cluster builds, which becomes visible
with bioinformatics pipelines using make.

I propose we need a lazy functional system which (optionally) uses
hash values rather than time stamps to decide what needs to be
(re)built. Two years ago I wrote up a design for a Ruby DSL that uses
futures to define dependencies. Futures make for easier reading of
dependencies (you can ignore the Ruby syntax part)

  https://github.com/pjotrp/bioinformatics/blob/master/doc/design/workflow.md

Pj.

Re: Getting rid of build tools

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> This is actually an interesting topic. GNU Make does a number of
> things well (note, btw, that it has Guile support), but it has
> problems too. In short, in addition to 
>
> 1. The sucky non-functional macro system 
>
> 2. Make files are hard to read (and maintain)
>
> 3. Does dependencies on time stamps
>
> the last is a real problem on cluster builds, which becomes visible
> with bioinformatics pipelines using make.

Why is this a problem?

Snakemake (like make but with Python syntax) is very popular here and
often used for pipelines.

~~ Ricardo

Re: Getting rid of build tools

[Pjotr Prins]

On Tue, Dec 29, 2015 at 08:22:53AM +0100, Ricardo Wurmus wrote:
> > 3. Does dependencies on time stamps
> >
> > the last is a real problem on cluster builds, which becomes visible
> > with bioinformatics pipelines using make.
> 
> Why is this a problem?
> 
> Snakemake (like make but with Python syntax) is very popular here and
> often used for pipelines.

After trying we discarded snakemake for the same reason. The time
stamps are whole seconds. Cluster builds over NFS get confused and
unreliable. At least in our experience. I think the problem is that
the granularity of a second is too low, combined with the NFS/time
synchronization of machines.

Anyway, time stamps also really don't tell you whether a file is
complete or not.

Pj.
-- 

Re: Getting rid of build tools

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> I am not saying we don't need build tools altogether - I'll still use
> make and rake, for example. But we certainly don't need all the
> baggage that comes with the current set of 'complex' Makefile
> generation tools that are automake and cmake.

The main purpose of Automake is to generate makefiles that (1) contain a
lot of non-trivial rules that Make alone does not provide, such as
dependency tracking for C source and headers, and (2) to generate the
rules specified in the GNU Coding Standards that and users have come to
expect:

  http://www.gnu.org/prep/standards/html_node/Standard-Targets.html

I think this remains relevant.

Ludo’.

Re: Getting rid of build tools

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> For anyone interested, Eelco's description of Maak, a functional make,
> is here:
>
>   http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf

It would be interesting to see what a Maak translated to Guile + Guix
would look like.

The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
in the paper would be quite easy to write using our current APIs.

Ludo’.

Re: Getting rid of build tools

[Pjotr Prins]

On Tue, Dec 29, 2015 at 04:33:47PM +0100, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > I am not saying we don't need build tools altogether - I'll still use
> > make and rake, for example. But we certainly don't need all the
> > baggage that comes with the current set of 'complex' Makefile
> > generation tools that are automake and cmake.
> 
> The main purpose of Automake is to generate makefiles that (1) contain a
> lot of non-trivial rules that Make alone does not provide, such as
> dependency tracking for C source and headers, and (2) to generate the
> rules specified in the GNU Coding Standards that and users have come to
> expect:
> 
>   http://www.gnu.org/prep/standards/html_node/Standard-Targets.html
> 
> I think this remains relevant.

Sure, but with less targets, thanks to GNU Guix, we could get a much
simpler system. Automake and cmake make my head hurt every time I deal
with them.

Pj.
-- 

Re: Getting rid of build tools

[Pjotr Prins]

On Tue, Dec 29, 2015 at 04:37:16PM +0100, Ludovic Courtès wrote:
> >
> >   http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf
> 
> It would be interesting to see what a Maak translated to Guile + Guix
> would look like.
> 
> The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
> in the paper would be quite easy to write using our current APIs.

Absolutely. If some ambituous hacker wants to try I am sure we are
happy to support the project. Maybe it is an idea for GSoC too.

Pj.

Re: Getting rid of build tools

[Christopher Allan Webber]

Ludovic Courtès writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> For anyone interested, Eelco's description of Maak, a functional make,
>> is here:
>>
>>   http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf
>
> It would be interesting to see what a Maak translated to Guile + Guix
> would look like.
>
> The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
> in the paper would be quite easy to write using our current APIs.
>
> Ludo’.

Reading this... it's interesting.  It looks like Nix and Maak don't
share code, but do share ideas and contributors (and a founder)?

So, assuming we had that theoretical guix-make, would it be usable
without running Guix as a full package manager, or without running the
daemon necessarily?  That might be more interesting, and in a way might
encourage more people to install Guix on their machines.  If it can't be
done in a clean way, we shouldn't consider forcing that.

I worry a bit that not everyone will be willing to devote to using
/gnu/store/ and having the daemon on their distribution for building.

 - Chris

Re: Getting rid of build tools

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:
>
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>
>>> For anyone interested, Eelco's description of Maak, a functional make,
>>> is here:
>>>
>>>   http://nixos.org/~eelco/pubs/iscsd-scm11-submitted.pdf
>>
>> It would be interesting to see what a Maak translated to Guile + Guix
>> would look like.
>>
>> The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
>> in the paper would be quite easy to write using our current APIs.
>>
>> Ludo’.
>
> Reading this... it's interesting.  It looks like Nix and Maak don't
> share code, but do share ideas and contributors (and a founder)?

The Nix and Maak languages are basically the same, and ISTR Eelco’s
thesis mentions something Maak-like built on top of Nix.

> So, assuming we had that theoretical guix-make, would it be usable
> without running Guix as a full package manager, or without running the
> daemon necessarily?

No, I was just thinking of something built using Guix as it currently
exists.

Ludo’.

Re: Getting rid of build tools

[Pjotr Prins]

On Tue, Dec 29, 2015 at 04:37:16PM +0100, Ludovic Courtès wrote:
> The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations
> in the paper would be quite easy to write using our current APIs.

I am thinking about this and maybe I'll give it a go at some point. A
functional approach to buildtools incorporating Hashes makes a lot of
sense. Guile can do composition, so that ticks a box. Guile has
futures, that is another box ticked.

Rather than calculating a dependency graph explitely I would like to
use futures to have the Guile compiler implicitly create a graph which
gets executed in parallel. I am not sure Guile can handle that type of
parallelism at a large scale, even though I find futures in the docs

  https://www.gnu.org/software/guile/manual/html_node/Futures.html

Eelco's Maak paper, interestingly, also mentions the 1 second
limitation of make as a potential source of race conditions. I would
like to have the option of using either time stamps or hash values to
decide whether a compilation step has completed succesfully. SHA
values arguably take longer to compute (actually pfff is a fast
alternative for large data https://github.com/pfff/pfff), but I want
Hash values for the next thing:

I would like to have cluster compilation where jobs are put out in an
opportunistic fashion. I.e., we assume semi-unlimited resources and
place jobs multiple times. The one job that completes first is the one
we take (the others may get killed). This would allow for longer
running jobs to be put out to a heterogeneous environment (think
cluster w. NFS, think cloud, think grid, think phone). For build environents
this could already be of interest, for bioinformatics large data
compute it could be brilliant. It may look wasteful, but truth is most
compute environments are happily idle and have differing bottlenecks
which would help speed up compilation (fastest one wins).

Anyone interested in this project? You know, there are quite a few
workflow initiatives (recent one the CWL) and build tool initiatives
(CMake and sbt, for example, are fairly recent), but I believe they
miss out on real software engineering solutions! Maybe we can get it
right. My question, is any really talented programmer here interested
to work on this and write a prototype? If you are a student we can
even try and get some of the work funded.

Re: Getting rid of build tools

[Pjotr Prins]

On Mon, Dec 28, 2015 at 09:06:40AM -0600, Christopher Allan Webber wrote:
> I guess these things could be done through Guix itself.  What I'd rather
> see though is a "./configure && make" compatible interface system for
> package building which uses Guile as its configuration language.
> Automake and friends are great when they're working, but I'd love a
> system that relied less on string-macro expansion and etc when I have to
> debug them.

Interestingly, the Guile extension of GNU Make can address this pretty
much. See

  https://www.gnu.org/software/make/manual/html_node/Guile-Example.html#Guile-Example

The way I read this is that string-macro expansion and make file
generation (autotools or cmake) can be replaced with Guile
scripts. If one only targets GNU Guix these can be fairly simple -
much of complexity actually comes from targeting different
architecture and deployment systems.

> I think that would be an interesting project.  It would be nice to see
> it happen outside of Guix though, so it could be used more generally.

Build tools such as automake and cmake are complex beasts. What they
actually do is target multiple build environments by generating make
files using a complex list of string expansions. Complexity, in
general, means hard to understand, change and maintain. The bad thing
is that almost every software project has to deal with this complexity
to make the software deployable.

What is wrong with these build tools? In short:

1. Complex - and therefore hard to understand and maintain
2. Simplistic language - simple string expansion systems are not nice and not DRY
3. Race conditions - make's time stamps are not suitable for building across servers
4. Different combinations of options are badly handled
5. No intermediate representation

The Guile extension to GNU Make addresses 1 and 2, but not 3-5.

This could be a largish project - not exactly what I need right now -
but I will try and look for others to help out. Maybe in time we'll
get something. I think we can start simple with Guix and the Guile
extension of GNU Make and replace autotools/cmake/rake first in a few
projects I have.

Pj.