Building Emacs on MacOS with clang+LTO fails

Building Emacs on MacOS with clang+LTO fails

[dhruva]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

I am trying to build emacs using clang 8 (from home-brew) with link time
optimization enabled.

Emacs configured as:
LD=lld CC=clang CFLAGS="-flto" ../../git/emacs/configure --with-mailutils
--with-ns

I get an error in building lib-src/make-fingerprint:
make[2]: Nothing to be done for `info'.
  CCLD     make-fingerprint
ld: reference to bitcode symbol '_rpl_getopt' which LTO has not compiled in
'_main' from /var/folders/5c/5w9bqxts6gl2pgfgdpzq94n0nhqj5f/T/cc-ebc71c.o
for architecture x86_64
clang-8: error: linker command failed with exit code 1 (use -v to see
invocation)
make[1]: *** [make-fingerprint] Error 1
make: *** [lib-src] Error 2

The following patch fixes the build:
$ git diff
diff --git a/lib-src/make-fingerprint.c b/lib-src/make-fingerprint.c
index 79bd007a5f..d00dd26c9f 100644
--- a/lib-src/make-fingerprint.c
+++ b/lib-src/make-fingerprint.c
@@ -65,7 +65,7 @@ main (int argc, char **argv)
 {
   int c;
   bool raw = false;
-  while (0 <= (c = getopt (argc, argv, "rh")))
+  while (0 <= (c = getopt_long (argc, argv, "rh", NULL, NULL)))
     {
       switch (c)
         {

[-- Attachment #2: Type: text/html, Size: 2278 bytes --]

Re: Building Emacs on MacOS with clang+LTO fails

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

dhruva wrote:
> I am trying to build emacs using clang 8 (from home-brew) with link time
> optimization enabled.

As INSTALL says, link time optimization seems to make Emacs slower and/or 
crashier even when Emacs builds. Plus as you've discovered, Emacs tends not to 
build. So I don't recommend LTO unless your job is to fix LTO bugs.

> Emacs configured as:
> LD=lld CC=clang CFLAGS="-flto" ../../git/emacs/configure --with-mailutils
> --with-ns

I don't think LD matters.

I suggest using the --enable-link-time-optimization flag of 'configure' instead.

For what it's worth, I just now tried building Emacs master on Fedora 30 as follows:

./configure --enable-link-time-optimization CC=clang

and found a recently-introduced bug with fingerprinting that is a different bug 
from your problem. I installed the attached patch to work around it. As this 
patch suggests, LTO is not often tested with Emacs.

>    CCLD     make-fingerprint
> ld: reference to bitcode symbol '_rpl_getopt' which LTO has not compiled in
> '_main' from /var/folders/5c/5w9bqxts6gl2pgfgdpzq94n0nhqj5f/T/cc-ebc71c.o
> for architecture x86_64
> clang-8: error: linker command failed with exit code 1 (use -v to see
> invocation)

If make-fingerprint links without LTO and fails to link with LTO, it's most 
likely a bug in the way Emacs is using the linker (or a bug in the compiler or 
linker), which surely would need to be investigated further on your platform.

> -  while (0 <= (c = getopt (argc, argv, "rh")))
> +  while (0 <= (c = getopt_long (argc, argv, "rh", NULL, NULL)))

This doesn't look right, for a couple of reasons. First, getopt_long doesn't 
document what it does wit a null 4th arg. More importantly, why is this change 
needed, and not similar changes in other getopt calls? It'd be better to debug 
the actual linking problem than to try to paper over it.

[-- Attachment #2: 0001-Port-new-fingerprinting-scheme-to-clang-LTO.patch --]
[-- Type: text/x-patch, Size: 2409 bytes --]

From ebecafbd19e2ba55ba90bfc9f7de88f4742479ad Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 4 May 2019 13:15:29 -0700
Subject: [PATCH] Port new fingerprinting scheme to clang + LTO

* lib-src/make-fingerprint.c (main): Don't consider multiple
instances of the fingerprint to be an error, as this can
happen with clang and -flto.  Instead, replace all instances
of the fingerprint.  There is a tiny chance that this will
silently corrupt the Emacs executable.

This patch suggests that we should go back to fingerprinting
the inputs to the linker instead of its output, as the new
fingerprinting scheme is unnecessarily complicated and this
complexity reduces reliability. The old scheme (i.e., before
commit 2019-05-14T23:31:24Z!eggert@cs.ucla.edu) was simpler
and more portable and good enough, and it's looking like it
would be less trouble in practice than the new scheme.
---
 lib-src/make-fingerprint.c | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/lib-src/make-fingerprint.c b/lib-src/make-fingerprint.c
index 79bd007a5f..5779e0d274 100644
--- a/lib-src/make-fingerprint.c
+++ b/lib-src/make-fingerprint.c
@@ -140,29 +140,25 @@ main (int argc, char **argv)
     }
   else
     {
-      char *finger = memmem (buf, chunksz, fingerprint, sizeof fingerprint);
-      if (!finger)
-	{
-	  fprintf (stderr, "%s: %s: missing fingerprint\n", prog, file);
-	  return EXIT_FAILURE;
-	}
-      else if (memmem (finger + 1, buf + chunksz - (finger + 1),
-		       fingerprint, sizeof fingerprint))
-	{
-	  fprintf (stderr, "%s: %s: two occurrences of fingerprint\n",
-		   prog, file);
-	  return EXIT_FAILURE;
-	}
+      bool fingered = false;
 
-      if (fseeko (f, finger - buf, SEEK_SET) != 0)
+      for (char *finger = buf;
+	   (finger = memmem (finger, buf + chunksz - finger,
+			     fingerprint, sizeof fingerprint));
+	   finger++)
 	{
-	  perror (file);
-	  return EXIT_FAILURE;
+	  if (! (fseeko (f, finger - buf, SEEK_SET) == 0
+		 && fwrite (digest, 1, sizeof digest, f) == sizeof digest))
+	    {
+	      perror (file);
+	      return EXIT_FAILURE;
+	    }
+	  fingered = true;
 	}
 
-      if (fwrite (digest, 1, sizeof digest, f) != sizeof digest)
+      if (!fingered)
 	{
-	  perror (file);
+	  fprintf (stderr, "%s: %s: missing fingerprint\n", prog, file);
 	  return EXIT_FAILURE;
 	}
     }
-- 
2.17.1