Intermitten build problems on Macos

Intermitten build problems on Macos

[Lars Ingebrigtsen]

It seems like half the times I say "git pull; make" on Macos, it all
ends with this:

make[2]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 137
make[1]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 2
/bin/sh: line 1: 31725 Killed: 9               EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp -l autoload --eval '(setq autoload-ensure-writable t)' --eval '(setq autoload-builtin-package-versions t)' --eval '(setq generated-autoload-file (expand-file-name (unmsys--file-name "loaddefs.el")))' -f batch-update-autoloads . ./calc ./calendar ./cedet ./cedet/ede ./cedet/semantic ./cedet/semantic/analyze ./cedet/semantic/bovine ./cedet/semantic/decorate ./cedet/semantic/symref ./cedet/semantic/wisent ./cedet/srecode ./emacs-lisp ./emulation ./erc ./eshell ./gnus ./image ./international ./language ./leim ./leim/ja-dic ./leim/quail ./mail ./mh-e ./net ./nxml ./org ./play ./progmodes ./textmodes ./url ./vc
make[2]: *** [loaddefs.el] Error 137
make[1]: *** [../lisp/loaddefs.el] Error 2
make: *** [src] Error 2
make: *** Waiting for unfinished jobs....


If I say "make bootstrap", then it builds fine.

Is anybody seeing the same?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Intermitten build problems on Macos

[Philipp]

> Am 09.11.2021 um 06:21 schrieb Lars Ingebrigtsen <larsi@gnus.org>:
> 
> It seems like half the times I say "git pull; make" on Macos, it all
> ends with this:
> 
> make[2]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 137
> make[1]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 2
> /bin/sh: line 1: 31725 Killed: 9               EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp -l autoload --eval '(setq autoload-ensure-writable t)' --eval '(setq autoload-builtin-package-versions t)' --eval '(setq generated-autoload-file (expand-file-name (unmsys--file-name "loaddefs.el")))' -f batch-update-autoloads . ./calc ./calendar ./cedet ./cedet/ede ./cedet/semantic ./cedet/semantic/analyze ./cedet/semantic/bovine ./cedet/semantic/decorate ./cedet/semantic/symref ./cedet/semantic/wisent ./cedet/srecode ./emacs-lisp ./emulation ./erc ./eshell ./gnus ./image ./international ./language ./leim ./leim/ja-dic ./leim/quail ./mail ./mh-e ./net ./nxml ./org ./play ./progmodes ./textmodes ./url ./vc
> make[2]: *** [loaddefs.el] Error 137
> make[1]: *** [../lisp/loaddefs.el] Error 2
> make: *** [src] Error 2
> make: *** Waiting for unfinished jobs....
> 
> 
> If I say "make bootstrap", then it builds fine.
> 
> Is anybody seeing the same?

Yes, this happens to me all the time.  Deleting bootstrap-emacs also fixes it temporarily.  The problem is generally that bootstrap-emacs is occasionally not properly codesigned.  See e.g. this crash report:

Process:               bootstrap-emacs [44123]
Path:                  /Users/USER/*/bootstrap-emacs
Identifier:            bootstrap-emacs
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        Exited process [44122]
Responsible:           Terminal [14989]
User ID:               501

Date/Time:             2021-11-07 16:31:28.3870 +0100
OS Version:            macOS 12.0.1 (21A559)
Report Version:        12
Anonymous UUID:        [...]

Sleep/Wake UUID:       [...]

Time Awake Since Boot: 520000 seconds
Time Since Wake:       3082 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x0000000105084008
Exception Codes:       0x0000000000000032, 0x0000000105084008
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 2 

Re: Intermitten build problems on Macos

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 09 Nov 2021 06:21:39 +0100
> 
> It seems like half the times I say "git pull; make" on Macos, it all
> ends with this:
> 
> make[2]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 137
> make[1]: *** [../lisp/emacs-lisp/cl-generic.elc] Error 2
> /bin/sh: line 1: 31725 Killed: 9               EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp -l autoload --eval '(setq autoload-ensure-writable t)' --eval '(setq autoload-builtin-package-versions t)' --eval '(setq generated-autoload-file (expand-file-name (unmsys--file-name "loaddefs.el")))' -f batch-update-autoloads . ./calc ./calendar ./cedet ./cedet/ede ./cedet/semantic ./cedet/semantic/analyze ./cedet/semantic/bovine ./cedet/semantic/decorate ./cedet/semantic/symref ./cedet/semantic/wisent ./cedet/srecode ./emacs-lisp ./emulation ./erc ./eshell ./gnus ./image ./international ./language ./leim ./leim/ja-dic ./leim/quail ./mail ./mh-e ./net ./nxml ./org ./play ./progmodes ./textmodes ./url ./vc
> make[2]: *** [loaddefs.el] Error 137

Doesn't error 137 mean out of memory?  "Killed 9" also points in that
direction: it's the OOM killer, I think.  Maybe you should enlarge the
VM size of your box?

Re: Intermitten build problems on Macos

[Lars Ingebrigtsen]

Philipp <p.stephani2@gmail.com> writes:

> Yes, this happens to me all the time.  Deleting bootstrap-emacs also
> fixes it temporarily.  The problem is generally that bootstrap-emacs
> is occasionally not properly codesigned.  See e.g. this crash report:

Hm...   So is the problem that this bit doesn't always run?  Or...  is
run at the wrong time?

temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(EMACSRES) \
[...]
ifeq ($(DO_CODESIGN),yes)
	codesign -s - -f $@.tmp
endif

Unfortunately, the Macos build doesn't fail for me now, so I can't test.
I'll try to poke at this more the next time it fails.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Intermitten build problems on Macos

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Doesn't error 137 mean out of memory?  "Killed 9" also points in that
> direction: it's the OOM killer, I think.  Maybe you should enlarge the
> VM size of your box?

This is on my physical Apple laptop (with 16GB memory), so it sounds
unlikely to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Intermitten build problems on Macos

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Unfortunately, the Macos build doesn't fail for me now, so I can't test.
> I'll try to poke at this more the next time it fails.

It failed for me now again, and it seems like codesigning the
bootstrap-emacs executable (again?) made the problem go away, so I've
pushed a change that does that.  We'll see whether that really fixes the
problem, or whether it needs further tweaks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Intermitten build problems on Macos

[Philipp]

> Am 10.11.2021 um 00:39 schrieb Lars Ingebrigtsen <larsi@gnus.org>:
> 
> Philipp <p.stephani2@gmail.com> writes:
> 
>> Yes, this happens to me all the time.  Deleting bootstrap-emacs also
>> fixes it temporarily.  The problem is generally that bootstrap-emacs
>> is occasionally not properly codesigned.  See e.g. this crash report:
> 
> Hm...   So is the problem that this bit doesn't always run?  Or...  is
> run at the wrong time?
> 
> temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(EMACSRES) \
> [...]
> ifeq ($(DO_CODESIGN),yes)
> 	codesign -s - -f $@.tmp
> endif
> 
> Unfortunately, the Macos build doesn't fail for me now, so I can't test.
> I'll try to poke at this more the next time it fails.

I don't quite understand where the problem lies exactly.  The Makefile rule doesn't look wrong per se, but it does appear to result in binaries that are occasionally not codesigned.

Maybe it's worth investigating a more robust approach: Instead of having make-fingerprint modify the binary directly, have it write out the fingerprint into a generated C source file, then compile that and link it against the rest of the object files using the system linker.  That approach should be more portable, as it only relies on the system linker to produce correct binaries.

Re: Intermitten build problems on Macos

[Lars Ingebrigtsen]

Philipp <p.stephani2@gmail.com> writes:

>> temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(EMACSRES) \
>> [...]
>> ifeq ($(DO_CODESIGN),yes)
>> 	codesign -s - -f $@.tmp
>> endif
>> 
>> Unfortunately, the Macos build doesn't fail for me now, so I can't test.
>> I'll try to poke at this more the next time it fails.
>
> I don't quite understand where the problem lies exactly.  The Makefile
> rule doesn't look wrong per se, but it does appear to result in
> binaries that are occasionally not codesigned.

I added another codesigning bit to the bootstrap-emacs rule, and I
haven't seen the build problem since.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no