Autodafe is "production" ready

Autodafe is "production" ready

[jbranso]

Hey Guix!

Have you heard about autodafe?  I just did today.

So it looks like autodafe (1) converts autotools into 
simple makefiles. Quoting from it's hacker's guide (2):  
"The proximate goal of this project is to eliminate autotools's
 piles of intermediate products and scripts that create a 
jungle in which exploits like the xz crack of 2024 can hide.
 The wider goal is to abolish the festering complexity sink 
that autotools recipes have become."  Sounds like a fairly cool
goal eh?

I know that guix has wanted to create a "guile based build 
system" to replace autotools (autodafe is written in 
python 3).  It sounds like Eric's work is moving to 
supplant and/or fix or repace autotools.  Eric is there
anything Guix can do for you to help you with your goal?
We do love our guile, so we will probably encourage you
to use some scheme code.  :)

Richard do you have any advice for Eric in how to "fix or
replace" autotools?

Ludo, does guix need to do anything to get ready for some
software wanting to use autodafe  ?  Is there a way we 
could use autodafe to help us get started on a guile based
build tool?

Thanks,

Joshua


1) https://www.phoronix.com/news/Autodafe-1.0-Released 

2) https://gitlab.com/esr/autodafe/-/blob/master/hacking.adoc

Re: Autodafe is "production" ready

[Eric S. Raymond]

jbranso@dismail.de <jbranso@dismail.de>:
> Hey Guix!
> 
> Have you heard about autodafe?  I just did today.
> 
> So it looks like autodafe (1) converts autotools into 
> simple makefiles. Quoting from it's hacker's guide (2):  
> "The proximate goal of this project is to eliminate autotools's
>  piles of intermediate products and scripts that create a 
> jungle in which exploits like the xz crack of 2024 can hide.
>  The wider goal is to abolish the festering complexity sink 
> that autotools recipes have become."  Sounds like a fairly cool
> goal eh?

I should emphasize that the project was *directly* motivated by the xz
Trojan. That's what, in my mind, moved autotools from "chronic
irritation" to "Someone should kill it with fire, *now*, and I guess
that will be me".  Took me eight weeks of balls-to-the-wall hacking.

The demand for some solution to supply-chain attacks through autotools
is urgent. I've been getting a particular earful about it from a
friend who is on the Gentoo packager lists; they want to be able to
throw out autotools *yesterday*. I have no doubt similar conversations
are going on within other packaging groups.

> I know that guix has wanted to create a "guile based build 
> system" to replace autotools (autodafe is written in 
> python 3).  It sounds like Eric's work is moving to 
> supplant and/or fix or repace autotools.  Eric is there
> anything Guix can do for you to help you with your goal?
> We do love our guile, so we will probably encourage you
> to use some scheme code.  :)

I looked into using Guile embedded in Make as an engine;
I'm a fan of Scheme myself.  There are two reasons I didn't
go that route:

1. Practical: Guile support is normally disablled in the packaged
versions of gmake on Debian, Ubuntu, Red Hat, etc.

That means that for my purpose of fielding a prompt replacement for
autotools, Guile might as well not be there at all.  The demand for a
usable drop-in replacement for autotools won't wait for the number of
packaging cycles it would take to get all the distributions on board
with enabling embedded Guile-in-make and then deploying it.

2. Architectural: I have a promary goal of making the build
flow in the replacement for autotools more auditable, with fewer
places an attacker can hide exploit components.

As much as I personally like Scheme, it would be a move in the wrong
direction from an audiability perspective even if it were
available. Too few people can read Scheme, or want to be able to.

What Scheme might have done if it were actually available I have
managed to cover with 661 lines of standalone Python 3. Under
the requirements, that is in every way a better solution,
even if not having an excuse to hack some Lisp makes me
a little wistful.

> Richard do you have any advice for Eric in how to "fix or
> replace" autotools?
> 
> Ludo, does guix need to do anything to get ready for some
> software wanting to use autodafe  ?  Is there a way we 
> could use autodafe to help us get started on a guile based
> build tool?

You'll make a better decision about that if you understand
what I've actually done.

There are three tools in the autodafe suite.  Two of them are one-time
transition aids; the third replaces autotools configure and is meant
to be checked into project repositories and shipped with them.

deconfig analyzes a code tree, removes guards abound POSIX features,
and reports a minimal set of guard symbols for non-POSIX
extensions. On a typical large old project, assuming POSIX reduces the
complexity of the configuration about an order of magnitude - you go
from hundreds of goard symbols to dozens.

Yes, this does mean you risk losing back-portability to big-iron
Unixes from the last century.  One of the design premises of autodafe
is that in 2024 this is an acceptable trade for a huge
reduction in the complexity and obscurity of the build recipe.

makemake takes a Makefuke generated by autotools and tries to rurn it
into a standalone Makefile that can be read and modifies by humans -
removing unused boilerplate and rearranging parts so the control
variables of the build are easy to modify.

These are two transion aids. They aren't needed onece you have your
nuild conversion done.

autodafe configure replaces the autotools generated configure script,
but is much, *much* smaller and easier to read than those are. It
reads magic comments in your Makefile (which you add during the
conversion process - deconfig will typically generates most of them
for you) to know which testrs to do to discover the capabilities of
your host ststem.  Here's what the example from the documentation
looks like:

# CHECK_ENABLE(ipv6)
# CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPCNT)
# CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPIDLE)
# CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPINTVL)
# CHECK_HAVE(getopt.h, getopt_long)
# CHECK_HAVE(_GNU_SOURCE, unistd.h, getresuid)
# CHECK_LIB(m) REQUIRED
# CHECK_LIB(nsl)
# CHECK_LIB(resolv)
# CHECK_LIB(rt)
# CHECK_HAVE(linux/if/tun.h)
# CHECK_HAVE(linux/ipv6.h)
# CHECK_HAVE(_GNU_SOURCE, string.h, memrchr)
# CHECK_HAVE(_GNU_SOURCE, unistd.h, setresuid)
# CHECK_SIZEOF(int)
# CHECK_SIZEOF(int *)
# CHECK_SIZEOF(long)
# CHECK_SIZEOF(long long)
# CHECK_SIZEOF(sys/types.h, off_t)
# CHECK_SIZEOF(pthread.h, pthread_t)
# CHECK_SIZEOF(stddef.h, size_t)

The effect is that the Makefile becomes the single point
of truth for the build, repl;acing configure.ac.

The entire transition process is described in detail here:

https://gitlab.com/esr/autodafe/-/blob/master/configure.adoc?ref_type=heads

> Thanks,
> 
> Joshua
> 
> 
> 1) https://www.phoronix.com/news/Autodafe-1.0-Released 
> 
> 2) https://gitlab.com/esr/autodafe/-/blob/master/hacking.adoc

-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Autodafe is "production" ready

[jbranso]

June 2, 2024 at 5:39 PM, "Eric S. Raymond" <esr@thyrsus.com> wrote:



> 
> jbranso@dismail.de <jbranso@dismail.de>:
> 
> > 
> > Hey Guix!
> > 
> >  
> > 
> >  Have you heard about autodafe? I just did today.
> > 
> >  
> > 
> >  So it looks like autodafe (1) converts autotools into 
> > 
> >  simple makefiles. Quoting from it's hacker's guide (2):  
> > 
> >  "The proximate goal of this project is to eliminate autotools's
> > 
> >  piles of intermediate products and scripts that create a 
> > 
> >  jungle in which exploits like the xz crack of 2024 can hide.
> > 
> >   The wider goal is to abolish the festering complexity sink 
> > 
> >  that autotools recipes have become."  Sounds like a fairly cool
> > 
> >  goal eh?
> > 
> 
> I should emphasize that the project was *directly* motivated by the xz
> 
> Trojan. That's what, in my mind, moved autotools from "chronic
> 
> irritation" to "Someone should kill it with fire, *now*, and I guess
> 
> that will be me". Took me eight weeks of balls-to-the-wall hacking.
> 
> The demand for some solution to supply-chain attacks through autotools
> 
> is urgent. I've been getting a particular earful about it from a
> 
> friend who is on the Gentoo packager lists; they want to be able to
> 
> throw out autotools *yesterday*. I have no doubt similar conversations
> 
> are going on within other packaging groups.
> 
> > 
> > I know that guix has wanted to create a "guile based build 
> > 
> >  system" to replace autotools (autodafe is written in 
> > 
> >  python 3).  It sounds like Eric's work is moving to 
> > 
> >  supplant and/or fix or repace autotools.  Eric is there
> > 
> >  anything Guix can do for you to help you with your goal?
> > 
> >  We do love our guile, so we will probably encourage you
> > 
> >  to use some scheme code.  :)
> > 
> 
> I looked into using Guile embedded in Make as an engine;
> 
> I'm a fan of Scheme myself. There are two reasons I didn't
> 
> go that route:
> 
> 1. Practical: Guile support is normally disablled in the packaged
> 
> versions of gmake on Debian, Ubuntu, Red Hat, etc.
> 
> That means that for my purpose of fielding a prompt replacement for
> 
> autotools, Guile might as well not be there at all. The demand for a
> 
> usable drop-in replacement for autotools won't wait for the number of
> 
> packaging cycles it would take to get all the distributions on board
> 
> with enabling embedded Guile-in-make and then deploying it.
> 
> 2. Architectural: I have a promary goal of making the build
> 
> flow in the replacement for autotools more auditable, with fewer
> 
> places an attacker can hide exploit components.
> 
> As much as I personally like Scheme, it would be a move in the wrong
> 
> direction from an audiability perspective even if it were
> 
> available. Too few people can read Scheme, or want to be able to.
> 
> What Scheme might have done if it were actually available I have
> 
> managed to cover with 661 lines of standalone Python 3. Under
> 
> the requirements, that is in every way a better solution,
> 
> even if not having an excuse to hack some Lisp makes me
> 
> a little wistful.
> 
> > 
> > Richard do you have any advice for Eric in how to "fix or
> > 
> >  replace" autotools?
> > 
> >  
> > 
> >  Ludo, does guix need to do anything to get ready for some
> > 
> >  software wanting to use autodafe  ?  Is there a way we 
> > 
> >  could use autodafe to help us get started on a guile based
> > 
> >  build tool?
> > 
> 
> You'll make a better decision about that if you understand
> 
> what I've actually done.
> 
> There are three tools in the autodafe suite. Two of them are one-time
> 
> transition aids; the third replaces autotools configure and is meant
> 
> to be checked into project repositories and shipped with them.
> 
> deconfig analyzes a code tree, removes guards abound POSIX features,
> 
> and reports a minimal set of guard symbols for non-POSIX
> 
> extensions. On a typical large old project, assuming POSIX reduces the
> 
> complexity of the configuration about an order of magnitude - you go
> 
> from hundreds of goard symbols to dozens.
> 
> Yes, this does mean you risk losing back-portability to big-iron
> 
> Unixes from the last century. One of the design premises of autodafe
> 
> is that in 2024 this is an acceptable trade for a huge
> 
> reduction in the complexity and obscurity of the build recipe.
> 
> makemake takes a Makefuke generated by autotools and tries to rurn it
> 
> into a standalone Makefile that can be read and modifies by humans -
> 
> removing unused boilerplate and rearranging parts so the control
> 
> variables of the build are easy to modify.
> 
> These are two transion aids. They aren't needed onece you have your
> 
> nuild conversion done.
> 
> autodafe configure replaces the autotools generated configure script,
> 
> but is much, *much* smaller and easier to read than those are. It
> 
> reads magic comments in your Makefile (which you add during the
> 
> conversion process - deconfig will typically generates most of them
> 
> for you) to know which testrs to do to discover the capabilities of
> 
> your host ststem. Here's what the example from the documentation
> 
> looks like:
> 
> # CHECK_ENABLE(ipv6)
> 
> # CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPCNT)
> 
> # CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPIDLE)
> 
> # CHECK_DECL(sys/socket.h, netinet/in.h, netinet/tcp.h, TCP_KEEPINTVL)
> 
> # CHECK_HAVE(getopt.h, getopt_long)
> 
> # CHECK_HAVE(_GNU_SOURCE, unistd.h, getresuid)
> 
> # CHECK_LIB(m) REQUIRED
> 
> # CHECK_LIB(nsl)
> 
> # CHECK_LIB(resolv)
> 
> # CHECK_LIB(rt)
> 
> # CHECK_HAVE(linux/if/tun.h)
> 
> # CHECK_HAVE(linux/ipv6.h)
> 
> # CHECK_HAVE(_GNU_SOURCE, string.h, memrchr)
> 
> # CHECK_HAVE(_GNU_SOURCE, unistd.h, setresuid)
> 
> # CHECK_SIZEOF(int)
> 
> # CHECK_SIZEOF(int *)
> 
> # CHECK_SIZEOF(long)
> 
> # CHECK_SIZEOF(long long)
> 
> # CHECK_SIZEOF(sys/types.h, off_t)
> 
> # CHECK_SIZEOF(pthread.h, pthread_t)
> 
> # CHECK_SIZEOF(stddef.h, size_t)
> 
> The effect is that the Makefile becomes the single point
> 
> of truth for the build, repl;acing configure.ac.
> 
> The entire transition process is described in detail here:
> 
> https://gitlab.com/esr/autodafe/-/blob/master/configure.adoc?ref_type=heads
> 
> > 
> > Thanks,
> > 
> >  
> > 
> >  Joshua
> > 
> >  
> > 
> >  
> > 
> >  1) https://www.phoronix.com/news/Autodafe-1.0-Released 
> > 
> >  
> > 
> >  2) https://gitlab.com/esr/autodafe/-/blob/master/hacking.adoc
> > 
> 
> -- 
> 
>  <a href="http://www.catb.org/~esr/%22>Eric http://www.catb.org/~esr/%22%3EEric  S. Raymond</a>
>


Eric thanks for the super quick response!  I really enjoyed reading your
previous email.  Thanks for explaining the decisions that you made when
you wrote autodafe.  I suppose autodafe doesn't have a logo yet.  

If there are any good guix artists out there, then you could probably
help Eric by creating a logo for autodafe!  Eric would you mind if I 
took your previous email and turn it into a blog post on
https://gnucode.me  ?

Thanks,

Joshua

P.S.  The Cathedral and the Bazaar was a really fun read for me!

Re: Autodafe is "production" ready

[Eric S. Raymond]

jbranso@dismail.de <jbranso@dismail.de>:
> Eric thanks for the super quick response!  I really enjoyed reading your
> previous email.  Thanks for explaining the decisions that you made when
> you wrote autodafe.  I suppose autodafe doesn't have a logo yet.  

It does, actually. I use the astrological symbol for the minor planet
makemake.  You can see it on the repository page at https://gitlab.com/esr/autodafe

>                                         Eric would you mind if I 
> took your previous email and turn it into a blog post on
> https://gnucode.me  ?

Go right ahead!

You can mine this for more material:

https://gitlab.com/esr/autodafe/-/blob/master/de-autoconfiscation.adoc?ref_type=heads
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Autodafe is "production" ready

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 4793 bytes --]

On 2024-06-02 19:02:46 +0000, jbranso@dismail.de wrote:
> Hey Guix!
>
> Have you heard about autodafe?  I just did today.
>
> So it looks like autodafe (1) converts autotools into
> simple makefiles. Quoting from it's hacker's guide (2):
> "The proximate goal of this project is to eliminate autotools's
>  piles of intermediate products and scripts that create a
> jungle in which exploits like the xz crack of 2024 can hide.
>  The wider goal is to abolish the festering complexity sink
> that autotools recipes have become."  Sounds like a fairly cool
> goal eh?

Goal might be cool, but the implementation seems to leave few things to be
desired.  I tried to follow the steps[0] for one project of mine, but got stuck
on 8.  The makemake just hangs, with no CPU usage and no debug output (even with
-ddddddddd).  I killed it after few minutes.

The ./configure seems to lack --program-... family of options, but hopefully
that will be added later (I hope, they are useful).  The renaming of
PACKAGE_VERSION to just VERSION seem somewhat unnecessary (maybe I just missed
the reason?).

EDIT: Ah, if I run `makemake .' instead of just `makemake', it does work right
away.  Interesting choice.  However now I get:

    $ make clean
    Makefile:29: warning: overriding recipe for target 'DEBUG'
    Makefile:28: warning: ignoring old recipe for target 'DEBUG'
    Makefile:30: warning: overriding recipe for target 'DEBUG'
    Makefile:29: warning: ignoring old recipe for target 'DEBUG'
    Makefile:31: warning: overriding recipe for target 'DEBUG'
    Makefile:30: warning: ignoring old recipe for target 'DEBUG'
    Makefile:32: warning: overriding recipe for target 'DEBUG'
    Makefile:31: warning: ignoring old recipe for target 'DEBUG'
    Makefile:33: warning: overriding recipe for target 'DEBUG'
    Makefile:32: warning: ignoring old recipe for target 'DEBUG'
    Makefile:34: warning: overriding recipe for target 'DEBUG'
    Makefile:33: warning: ignoring old recipe for target 'DEBUG'
    Makefile:35: warning: overriding recipe for target 'DEBUG'
    Makefile:34: warning: ignoring old recipe for target 'DEBUG'
    Makefile:36: warning: overriding recipe for target 'DEBUG'
    Makefile:35: warning: ignoring old recipe for target 'DEBUG'
    Makefile:37: warning: overriding recipe for target 'DEBUG'
    Makefile:36: warning: ignoring old recipe for target 'DEBUG'
    Makefile:38: warning: overriding recipe for target 'DEBUG'
    Makefile:37: warning: ignoring old recipe for target 'DEBUG'
    Makefile:39: warning: overriding recipe for target 'DEBUG'
    Makefile:38: warning: ignoring old recipe for target 'DEBUG'
    Makefile:42: warning: overriding recipe for target 'DEBUG'
    Makefile:39: warning: ignoring old recipe for target 'DEBUG'
    Makefile:43: warning: overriding recipe for target 'DEBUG'
    Makefile:42: warning: ignoring old recipe for target 'DEBUG'
    Makefile:44: warning: overriding recipe for target 'DEBUG'
    Makefile:43: warning: ignoring old recipe for target 'DEBUG'
    Makefile:45: warning: overriding recipe for target 'DEBUG'
    Makefile:44: warning: ignoring old recipe for target 'DEBUG'
    Makefile:54: warning: overriding recipe for target 'DEBUG'
    Makefile:45: warning: ignoring old recipe for target 'DEBUG'
    Makefile:55: warning: overriding recipe for target 'DEBUG'
    Makefile:54: warning: ignoring old recipe for target 'DEBUG'
    Makefile:56: *** multiple target patterns.  Stop.

This is where I gave up.

>
> I know that guix has wanted to create a "guile based build
> system" to replace autotools (autodafe is written in
> python 3).  It sounds like Eric's work is moving to
> supplant and/or fix or repace autotools.  Eric is there
> anything Guix can do for you to help you with your goal?
> We do love our guile, so we will probably encourage you
> to use some scheme code.  :)
>
> Richard do you have any advice for Eric in how to "fix or
> replace" autotools?
>
> Ludo, does guix need to do anything to get ready for some
> software wanting to use autodafe  ?  Is there a way we
> could use autodafe to help us get started on a guile based
> build tool?

Not Ludo, but since it still tries to provide the `./configure && make && make
install' interface, should it not mostly just work?  Did you run into any
specific issues?

>
> Thanks,
>
> Joshua
>
>
> 1) https://www.phoronix.com/news/Autodafe-1.0-Released
>
> 2) https://gitlab.com/esr/autodafe/-/blob/master/hacking.adoc
>

Have a nice day,
Tomas Volf

0: https://gitlab.com/esr/autodafe/-/blob/master/de-autoconfiscation.adoc

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]