Lightning fast build

Lightning fast build

[Gerd Möllmann]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 455 bytes --]

I just noticed that building Emacs master from scratch, i.e. what I do 
from time to time

     git clean -xdf \
         && ./autogen.sh \
         && ./configure --with-native-compilation \
         && bear -- make \
         && make install

went down from around 8 minutes to 4:30, and everything seems to still 
be working normally.  (This is macos 12.5.1, M1 chip.)

Kudos to whoever made that happen!


[-- Attachment #1.1.1.2: Type: text/html, Size: 867 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3211 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Lightning fast build

[Alan Mackenzie]

Hello, Gerd.

On Fri, Aug 19, 2022 at 14:44:51 +0200, Gerd Möllmann wrote:
> I just noticed that building Emacs master from scratch, i.e. what I do 
> from time to time

>      git clean -xdf \
>          && ./autogen.sh \
>          && ./configure --with-native-compilation \
>          && bear -- make \
>          && make install

> went down from around 8 minutes to 4:30, and everything seems to still 
> be working normally.  (This is macos 12.5.1, M1 chip.)

> Kudos to whoever made that happen!

I think that was mainly Lars.  I welcome the speed-up, too.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lightning fast build

[Pankaj Jangid]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>
> I just noticed that building Emacs master from scratch, i.e. what I do
> from time to time
>
>     git clean -xdf \
>         && ./autogen.sh \
>         && ./configure --with-native-compilation \
>         && bear -- make \
>         && make install
>
> went down from around 8 minutes to 4:30, and everything seems to still
> be working normally.  (This is macos 12.5.1, M1 chip.)
>
> Kudos to whoever made that happen!

Yes. Now it is,

real	1m22.072s
user	8m44.482s
sys	0m48.667s

with,

    git clean -fdx \
    && make -j`nproc` \
    	 configure="--prefix=$HOME/.local --with-pgtk CFLAGS='-O0 -g3'"

It was more than 2.5m earlier.

Re: Lightning fast build

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

Hello Gerd. Out of curiosity, and speculation, is your use of 'bear'
intended to make a compilation database of Emacs sources for use with a LSP
server?

If so, how is that going? Last time I tried to do that, the server (clangd,
i think) was still very confused in many files.

João

On Fri, Aug 19, 2022, 13:45 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> I just noticed that building Emacs master from scratch, i.e. what I do
> from time to time
>
>     git clean -xdf \
>         && ./autogen.sh \
>         && ./configure --with-native-compilation \
>         && bear -- make \
>         && make install
>
> went down from around 8 minutes to 4:30, and everything seems to still be
> working normally.  (This is macos 12.5.1, M1 chip.)
>
> Kudos to whoever made that happen!
>

[-- Attachment #2: Type: text/html, Size: 1504 bytes --]

Re: Lightning fast build

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

>
> Hello Gerd. Out of curiosity, and speculation, is your use of 'bear'
> intended to make a compilation database of Emacs sources for use with
> a LSP server?

Yes, exactly.

> If so, how is that going? Last time I tried to do that, the server (clangd, i think) was still very confused in many files.

I can't complain.

I'm using lsp-mode with clangd and bear because that were the first
things I tried, and all of them just worked for me.

I've also installed llvm14 in the meantime because of reasons and I'm
using

 '(lsp-clients-clangd-executable "/opt/homebrew/opt/llvm/bin/clangd")

which says 

~/emacs/master/ > /opt/homebrew/opt/llvm/bin/clangd --version 
Homebrew clangd version 14.0.6

But it also worked for me before with the clangd coming with Xcode,
which is version 13 something.

In case it matters, another thing I did right from the start is

;; Activate Lsp in all C-like buffers.
(add-hook 'c-mode-common-hook 'lsp-deferred)

    lsp-deferred is an autoloaded Lisp closure in ‘lsp-mode.el’.

    (lsp-deferred)

    Entry point that defers server startup until buffer is visible.
    ‘lsp-deferred’ will wait until the buffer is visible before invoking ‘lsp’.
    This avoids overloading the server with many files when starting Emacs.

All this is on a Mac mini/M1, macOS 12.5.1 with 16G RAM.  Others
mentioned they had problems with clangd, I believe on GNU/Linux, not
sure about that.  But as I said, I personally can't complain.

Emacs + bear + compile_commands.json + clangd

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

João Távora <joaotavora@gmail.com> writes:

> Hello Gerd. Out of curiosity, and speculation, is your use of 'bear'
> intended to make a compilation database of Emacs sources for use with a LSP
> server?
>
> If so, how is that going? Last time I tried to do that, the server (clangd,
> i think) was still very confused in many files.
>
> João

Hey João, I had issues with Emacs source and clangd, especially with
header files.  The issue is that many of Emacs' C header files are not
"self contained" so they can't be compiled alone.  Also, `bear` will
only generate compile commands for .c files, and clangd uses heuristics
to guess a compile command for nearby .h files.  It does a reasonable
job in some projects, but not Emacs due to the "self contained" issue.
Biggest issue: when editing an .h file clangd won't include <config.h>
for you.

I now do this:

    ./configure <whatever>
    bear --force-wrapper -- make -j$CPUS
    emacs-fixup-compile-commands.py

where emacs-fixup-compile-commands.py is attached.  Bonus to anyone
rewrites it in elisp.  ;-)

The idea is to find the .c file with the most similar name to each .h
file, then form a compile command for the .h by hacking the .c file's
command line, adding:

   -Wno-unused-macros -include config.h

...this way, when clangd "compiles" an Emacs .h file it includes
config.h first (which fixes many issues), and won't complain about
macros going unused in the file.

With this, Emacs + clangd is pretty seamless for me.

P.S. I recently had to start passing "--force-wrapper" to bear.  Some
Emacs builds steps were failing with bear's LD_PRELOAD hack.  Maybe the
Emacs modules stuff?  Or tests?  I didn't look into it, but
"--force-wrapper" fixed it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-fixup-compile-commands.py --]
[-- Type: text/x-python, Size: 2448 bytes --]

#!/usr/bin/env python3

import copy
import json
import glob
import os
import sys


def levenshteinDistance(s1, s2):
    if len(s1) > len(s2):
        s1, s2 = s2, s1

    distances = range(len(s1) + 1)
    for i2, c2 in enumerate(s2):
        distances_ = [i2 + 1]
        for i1, c1 in enumerate(s1):
            if c1 == c2:
                distances_.append(distances[i1])
            else:
                distances_.append(
                    1 + min((distances[i1], distances[i1 + 1], distances_[-1]))
                )
        distances = distances_
    return distances[-1]


def Read():
    with open("compile_commands.json") as file:
        return json.load(file)


def Write(compile_commands):
    with open("compile_commands.json", "w") as file:
        json.dump(compile_commands, file, sort_keys=True, indent=2)


def Headers():
    # return [f for f in os.listdir('src') if f.endswith('.h')]
    return glob.glob(os.path.join(os.getcwd(), "src/*.h"))


def NeedsConfigH(header):
    """Returns True if header needs config.h to function"""
    if header.endswith("/config.h"):
        return False
    with open(header) as file:
        return "INLINE_HEADER_BEGIN" in file.read()


def FindSimilarCommand(compile_commands, header):
    header = os.path.join(os.getcwd(), header)
    bestDistance = sys.maxsize
    bestCommand = None
    for command in compile_commands:
        distance = levenshteinDistance(header, command["file"])
        if distance < bestDistance:
            bestDistance = distance
            bestCommand = command
    return bestCommand


def AdaptTemplate(command, header):
    command = copy.deepcopy(command)
    if "output" in command:
        del command["output"]
    src = os.path.relpath(command["file"], command["directory"])
    index = command["arguments"].index(src)
    new_src = os.path.relpath(header, command["directory"])
    command["arguments"][index : index + 1] = [
        "-Wno-unused-macros",
        "-include",
        "config.h",
        new_src,
    ]
    command["file"] = header
    return command


compile_commands = Read()
new_compile_commands = copy.copy(compile_commands)
for header in Headers():
    if NeedsConfigH(header):
        similar = FindSimilarCommand(compile_commands, header)
        if similar["file"] == header:
            continue
        adapted = AdaptTemplate(similar, header)
        new_compile_commands.append(adapted)
Write(new_compile_commands)

Re: Emacs + bear + compile_commands.json + clangd

[Gerd Möllmann]

Matt Armstrong <matt@rfc20.org> writes:

> Hey João, I had issues with Emacs source and clangd, especially with
> header files.  The issue is that many of Emacs' C header files are not
> "self contained" so they can't be compiled alone.  Also, `bear` will
> only generate compile commands for .c files, and clangd uses
> heuristics to guess a compile command for nearby .h files.  It does a
> reasonable job in some projects, but not Emacs due to the "self
> contained" issue.  Biggest issue: when editing an .h file clangd won't
> include <config.h> for you.

I'm wondering how you use clangd.  Is this lsp-mode or eglot?  I'm using
lsp-mode ATM, just because I tried it first, and it worked for me.

> I now do this:
>
>     ./configure <whatever>
>     bear --force-wrapper -- make -j$CPUS
>     emacs-fixup-compile-commands.py
>
> where emacs-fixup-compile-commands.py is attached.  Bonus to anyone
> rewrites it in elisp.  ;-)

I've never had to use --force-wrapper or alter compile_commands.json.
My current version is bear 3.0.20, on macOS 12.6.  What's your system?

P.S.

WRT to seamless, I have to add, for macOS: This was true for me with
Xcode 13, and is no longer with Xcode 14.  Both clangd and lldb crash
pretty frequently :-(.

Re: Emacs + bear + compile_commands.json + clangd

[Matt Armstrong]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Matt Armstrong <matt@rfc20.org> writes:
>
>> Hey João, I had issues with Emacs source and clangd, especially with
>> header files.  The issue is that many of Emacs' C header files are not
>> "self contained" so they can't be compiled alone.  Also, `bear` will
>> only generate compile commands for .c files, and clangd uses
>> heuristics to guess a compile command for nearby .h files.  It does a
>> reasonable job in some projects, but not Emacs due to the "self
>> contained" issue.  Biggest issue: when editing an .h file clangd won't
>> include <config.h> for you.
>
> I'm wondering how you use clangd.  Is this lsp-mode or eglot?  I'm using
> lsp-mode ATM, just because I tried it first, and it worked for me.

I use eglot because I tried it first and it worked for me.  ;-)

I tried eglot first because I like the general design philosophy of
integration with core Emacs packages and a more conservative approach to
the UI.  You barely know when eglot is running!


>> I now do this:
>>
>>     ./configure <whatever>
>>     bear --force-wrapper -- make -j$CPUS
>>     emacs-fixup-compile-commands.py
>>
>> where emacs-fixup-compile-commands.py is attached.  Bonus to anyone
>> rewrites it in elisp.  ;-)
>
> I've never had to use --force-wrapper or alter compile_commands.json.

Regarding compile_commands.json, if I use the compile_commands.json
generated by bear, with eglot, with clangd version "Debian clangd
version 14.0.6-2", it works well for .c files but not .h files.

E.g. I open src/buffer.h and I get these errors in flymake (this is a
small subset):

   26   9 error    e-f-b    clang [unknown_typename]: In included file: unknown type name 'INLINE_HEADER_BEGIN'
    1   0 error    e-f-b    clang [fatal_too_many_errors]: Too many errors emitted, stopping now
  283   4 error    e-f-b    clang [unknown_typename]: Unknown type name 'INTERVAL'
  722   9 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'PSEUDOVECTORP' is invalid in C99
  877   5 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'NILP' is invalid in C99
 1079  26 error    e-f-b    clang [expr_not_ice]: Expression is not an integer constant expression
 1256   7 error    e-f-b    clang [unknown_typename]: Unknown type name 'INTERVAL'
 1573  14 warning  e-f-b    clang [-Wint-conversion]: Incompatible integer to pointer conversion initializing 'Lisp_Object' (aka 'struct Lisp_X *') with an expression of type 'int'
 1573  21 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'CHAR_TABLE_REF' is invalid in C99

I always assumed that this was becuase clangd didn't know how to compile
src/buffer.h and is unable to figure it out:

    $ grep -F buffer.h compile_commands.json
    <nothing printed>

After I run my emacs-fixup-compile-commands.py I have this in
compile_commands.json:

  {
    "arguments": [
      "/usr/bin/gcc",

[...blah blah blah you get the point...]

      "-Wno-unused-macros",
      "-Wno-unused-function",
      "-include",
      "config.h",
      "buffer.h"
    ],
    "directory": "/home/matt/git/emacs-noverlay/src",
    "file": "/home/matt/git/emacs-noverlay/src/buffer.h"
  },

and clangd works fine.

Given all this I have no idea how your experience with clangd could
different, but I'm not a clangd expert 🤔.


> My current version is bear 3.0.20, on macOS 12.6.  What's your system?

bear 3.0.20 on Debian/Bullseye (testing)

My issue may have been related to building with Address Sanitizer, which
I think also (might) play games with LD_PRELOAD.  I tried
--force-wrapper and it worked for me.  ;-)


> P.S.
>
> WRT to seamless, I have to add, for macOS: This was true for me with
> Xcode 13, and is no longer with Xcode 14.  Both clangd and lldb crash
> pretty frequently :-(.

Maybe using the clangd from homebrew would work better for you?

Re: Emacs + bear + compile_commands.json + clangd

[Gerd Möllmann]

Matt Armstrong <matt@rfc20.org> writes:

>> I've never had to use --force-wrapper or alter compile_commands.json.
>
> Regarding compile_commands.json, if I use the compile_commands.json
> generated by bear, with eglot, with clangd version "Debian clangd
> version 14.0.6-2", it works well for .c files but not .h files.
>
> E.g. I open src/buffer.h and I get these errors in flymake (this is a
> small subset):
>
>    26   9 error    e-f-b    clang [unknown_typename]: In included file: unknown type name 'INLINE_HEADER_BEGIN'
>     1   0 error    e-f-b    clang [fatal_too_many_errors]: Too many errors emitted, stopping now
>   283   4 error    e-f-b    clang [unknown_typename]: Unknown type name 'INTERVAL'
>   722   9 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'PSEUDOVECTORP' is invalid in C99
>   877   5 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'NILP' is invalid in C99
>  1079  26 error    e-f-b    clang [expr_not_ice]: Expression is not an integer constant expression
>  1256   7 error    e-f-b    clang [unknown_typename]: Unknown type name 'INTERVAL'
>  1573  14 warning  e-f-b    clang [-Wint-conversion]: Incompatible integer to pointer conversion initializing 'Lisp_Object' (aka 'struct Lisp_X *') with an expression of type 'int'
>  1573  21 warning  e-f-b    clang [-Wimplicit-function-declaration]: Implicit declaration of function 'CHAR_TABLE_REF' is invalid in C99
>
> I always assumed that this was becuase clangd didn't know how to compile
> src/buffer.h and is unable to figure it out:
>
> Given all this I have no idea how your experience with clangd could
> different, but I'm not a clangd expert 🤔.

I'm not an expert for anything, but I don't see such errors here.
Strange.

>> My current version is bear 3.0.20, on macOS 12.6.  What's your system?
>
> bear 3.0.20 on Debian/Bullseye (testing)

Thanks.
>
> My issue may have been related to building with Address Sanitizer, which
> I think also (might) play games with LD_PRELOAD.  I tried
> --force-wrapper and it worked for me.  ;-)

:-).  And BTW, I'm also using ASAN here, almost all the time, and don't
see this either.  This is with the system compiler, which is Apple clang
version 14.0.0 (clang-1400.0.29.102).

Maybe I'll try the lldb from LLVM 15 a try at some point.