Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

[James Luke]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

I'm interested in making cross-compilation of emacs easier. Or rather,
making cross-compilation of temacs easier, such that one can run temacs on
the host machine to finish the build. This would make it easier to create
reproducible builds (particularly with Guix/NixOS as the build machine),
and means one doesn't need a C compiler on the host. Right now the problem
I'm running into is caused by build intermediates that must be built then
run on the builder: the programs "lib-src/make-docfile.c" and
"lib-src/make-fingerprint.c".

These intermediates both use gnulib (with #include <config.h> etc), but
configure.ac only configures gnulib for the host machine. So the headers,
#defines, and library are all wrong for the build machine and compilation
fails. To fix this, I'd like to rewrite make-fingerprint.c and
make-docfile.c to avoid non-portable constructs and thus the dependency on
gnulib, but I don't know if this is considered acceptable for a GNU
project. The alternative is a larger overhaul of the build system, with a
nested configure script (to run configure for the builder when doing a
cross build), another makefile, and another gnulib tree (for the modules
needed only for the intermediates). I suspect the latter would be more
fragile and produce a much larger maintenance burden than the former.

Would there be any objections to the simple rewrite?

[-- Attachment #2: Type: text/html, Size: 1532 bytes --]

Re: Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

[Eli Zaretskii]

> From: James Luke <james92856@gmail.com>
> Date: Fri, 25 Jun 2021 19:22:10 -0700
> 
> I'm interested in making cross-compilation of emacs easier. Or rather, making cross-compilation of temacs
> easier, such that one can run temacs on the host machine to finish the build. This would make it easier to
> create reproducible builds (particularly with Guix/NixOS as the build machine), and means one doesn't need
> a C compiler on the host. Right now the problem I'm running into is caused by build intermediates that must
> be built then run on the builder: the programs "lib-src/make-docfile.c" and "lib-src/make-fingerprint.c".

Please tell more about the idea.  Is the intent to run part of the
build on the "build" system, producing temacs, and then continue on
the "host(=target)" system by running the built temacs there?  If so,
I don't understand the "doesn't need a C compiler on the host" part,
given that we now have native-compilation in Emacs.

Also, which system(s) would you like to target that require
cross-compilation?

> These intermediates both use gnulib (with #include <config.h> etc), but configure.ac only configures gnulib
> for the host machine. So the headers, #defines, and library are all wrong for the build machine and
> compilation fails. To fix this, I'd like to rewrite make-fingerprint.c and make-docfile.c to avoid non-portable
> constructs and thus the dependency on gnulib

I don't think this is a good idea: it's a lot of work, and basically
you will end up with the same portability shims, just incorporated
into the sources.

Why not modify the build process such that it configures Gnulib twice:
once for the build system and another one for the host/target system?
We already compile Gnulib twice in the unexec configuration (which
will probably be removed in the future version, but we still have it
meanwhile).  Building Gnulib twice doesn't sound like such a bad
situation, especially given that we already do so in some cases.

Re: Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

[Stefan Monnier]

> These intermediates both use gnulib (with #include <config.h> etc), but
> configure.ac only configures gnulib for the host machine. So the headers,
> #defines, and library are all wrong for the build machine and compilation
> fails. To fix this, I'd like to rewrite make-fingerprint.c and
> make-docfile.c to avoid non-portable constructs and thus the dependency on
> gnulib, but I don't know if this is considered acceptable for a GNU
> project.

Have you looked at what that would take?
E.g. how do you plan to deal with the sha256 code used by make-fingerprint.c
and normally provided by gnulib?

How 'bout doing "a bit of both": i.e. use a separate gnulib+autoconf for
those two executables, but patch them when it's easy to try and minimize
the reliance on gnulib and autoconf?


        Stefan

Re: Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

[James Luke]

On Fri, 25 Jun 2021 at 23:26, Eli Zaretskii <eliz@gnu.org> wrote:
> Please tell more about the idea.  Is the intent to run part of the
> build on the "build" system, producing temacs, and then continue on
> the "host(=target)" system by running the built temacs there?

Yes, that was the notion.

>If so,
> I don't understand the "doesn't need a C compiler on the host" part,
> given that we now have native-compilation in Emacs.

I think you've outed me as a dunderhead. :-)
I was under the mistaken impression that libgccjit was standalone and
did not rely on the system's gcc. This revelation scuttles my main
reason for pursuing the idea, which was to limit the need to set up a
build system (compiler, make, etc) on the host. As such, I don't plan
on pursuing the idea further.

> Also, which system(s) would you like to target that require
> cross-compilation?

No system that actually *required* cross-compilation. The idea was to
make reproducible builds easier, and provide a convenience for testing
and distribution. My initial interest was catching problems on Windows
and making it easier to do regular development snapshots.

> I don't think this is a good idea: it's a lot of work, and basically
> you will end up with the same portability shims, just incorporated
> into the sources.

I took an (admittedly brief) look at make-docfile.c, and I'm pretty
sure it's possible rewrite it so that it only requires a few functions
from stdio. I suspect platform-specific #ifdefs would have been quite
small and maintainable. Of course, it's possible I underestimated the
number of functions required, the number of #ifdefs required, or just
how sneaky the portability issues could be.

It turned out that make-fingerprint was more of a challenge. It is
possible to replace it with a small shell script that uses the the
"base32" and "sha256" commands, but I discovered those are not
available on all build platforms emacs supports. (The meat of the
script would have been along the lines of `cat temacs | base32 | sed
's/<placeholder fingerprint>/<fingerprint>/' | base32 -d >
temacs-fingerprinted`. Which is slightly silly, but surprisingly
efficient.)

Even if I hadn't been mistaken about libgccjit, the make-fingerprint
problem would have likely toppled the idea of a "portable rewrite" and
I would have gone the recursive configure route.

Thanks for the input. :-)

Re: Make cross-compiling temacs easier: drop gnulib for build intermediates, or use recursive autoconf?

[Eli Zaretskii]

> From: James Luke <james92856@gmail.com>
> Date: Sat, 26 Jun 2021 15:56:56 -0700
> Cc: emacs-devel@gnu.org
> 
> >If so,
> > I don't understand the "doesn't need a C compiler on the host" part,
> > given that we now have native-compilation in Emacs.
> 
> I think you've outed me as a dunderhead. :-)
> I was under the mistaken impression that libgccjit was standalone and
> did not rely on the system's gcc.

AFAIK, it doesn't require gcc the driver, but it does use cc1, the
assembler, and the linker.

> No system that actually *required* cross-compilation. The idea was to
> make reproducible builds easier, and provide a convenience for testing
> and distribution. My initial interest was catching problems on Windows
> and making it easier to do regular development snapshots.

You could do that, but such builds will not support native
compilation.

> > I don't think this is a good idea: it's a lot of work, and basically
> > you will end up with the same portability shims, just incorporated
> > into the sources.
> 
> I took an (admittedly brief) look at make-docfile.c, and I'm pretty
> sure it's possible rewrite it so that it only requires a few functions
> from stdio. I suspect platform-specific #ifdefs would have been quite
> small and maintainable.

Not sure how you looked.  I added the "-t -t" switches to the link
command line in lib-src, and saw that an MS-Windows build of those
programs uses getopt.c, memmem.c, and sha256.c modules for
make-fingerprint; and binary-io.c and c-type.c modules for
make-docfile.  How would you avoid using those?

> It turned out that make-fingerprint was more of a challenge. It is
> possible to replace it with a small shell script that uses the the
> "base32" and "sha256" commands, but I discovered those are not
> available on all build platforms emacs supports. (The meat of the
> script would have been along the lines of `cat temacs | base32 | sed
> 's/<placeholder fingerprint>/<fingerprint>/' | base32 -d >
> temacs-fingerprinted`. Which is slightly silly, but surprisingly
> efficient.)

The MSYS installation on my MS-Windows box has neither the base32 nor
the base256 commands, FWIW.