strange build problem with missing -by.el and -wy.el files

strange build problem with missing -by.el and -wy.el files

[Gregor Zattler]

Dear fellow emacs users, since a week or two I have problems
building Emacs from the git repo.

This is because of missing *.el files which should be
generated for cedet/semantic, see below.

My last attempt was with emacs master as of commit
a691e811e23dee48674e9e0716429074fdbe3d7b.  Normally I build
with a few non-default build options but in order to get rid
of the problems in a pristine git worktree all I did was

make V=1

to get the most default build.  This calls ./autogen.sh,
./configure etc and even this fails like so:

[...]
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/dep.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/doc.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/ede-grammar.el
Loading /home/grfz/src/emacs-master-next/lisp/cedet/semantic/grm-wy-boot.el (source)...
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/edit.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/find.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/format.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/fw.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/grammar.el
Loading /home/grfz/src/emacs-master-next/lisp/cedet/semantic/grm-wy-boot.el (source)...
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/grm-wy-boot.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/html.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/ia.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/ia-sb.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/idle.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/imenu.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/java.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/lex.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/lex-spp.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/loaddefs.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/mru-bookmark.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/sb.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/scope.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/senator.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/sort.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/symref.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/tag.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/tag-file.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/tag-ls.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/tag-write.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/texi.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/util.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/util-modes.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/wisent.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/analyze/complete.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/analyze/debug.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/analyze/fcn.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/analyze/refs.el
'../src/emacs' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  -f batch-byte-compile cedet/semantic/bovine/c.el

In toplevel form:
cedet/semantic/bovine/c.el:35:2: Error: Cannot open load file: No such file or directory, semantic/bovine/c-by
make[4]: *** [Makefile:332: cedet/semantic/bovine/c.elc] Error 1
make[4]: Leaving directory '/home/grfz/src/emacs-master-next/lisp'
make[3]: *** [Makefile:366: compile-main] Error 2
make[3]: Leaving directory '/home/grfz/src/emacs-master-next/lisp'
make[2]: *** [Makefile:531: lisp] Error 2
make[2]: Leaving directory '/home/grfz/src/emacs-master-next'
make[2]: Entering directory '/home/grfz/src/emacs-master-next'
***
*** "make all" failed with exit status 2.
***
*** You could try to:
*** - run "make bootstrap", which might fix the problem
*** - run "make V=1", which displays the full commands invoked by make,
***   to further investigate the problem
***
make[2]: *** [Makefile:413: advice-on-failure] Error 2
make[2]: Leaving directory '/home/grfz/src/emacs-master-next'
make[1]: *** [Makefile:370: all] Error 2
make[1]: Leaving directory '/home/grfz/src/emacs-master-next'
make: *** [GNUmakefile:95: default] Error 2

Line 35 in lisp/cedet/semantic/bovine/c.el reads:

(require 'semantic/bovine/c-by)

There is no semantic/bovine/c-by.el in the tree, but
cedet/Changelog.1 says it was removed in 2013-11-30 together
with other generated files.


In
https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-11/msg00277.html
Paul Eggert wrote:

"One hypothesis is that c-by.el depends on some other .el file being compiled into *.elc form (rather than being present in only source form), and that using 'make -j5' causes 'make' on my platform to (unluckily) arrange for c-by.el to be compiled before the other *.elc file (whatever it is) is available.


c-by.el starts this way:

(require 'semantic/lex)
(eval-when-compile (require 'semantic/bovine))

so perhaps we should arrange for all the .elc files needed by semantic/lex and/or semantic/bovine to be present, before compiling c-by.el."


but    make -j 1   did not help.

So I copied several .el files which were not generated from
a not so old installation of emacs, like so:

gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/bovine/c-by.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/bovine/c-by.el
gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/bovine/make-by.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/bovine/make-by.el
gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/bovine/scm-by.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/bovine/scm-by.el
gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/wisent/javat-wy.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/wisent/javat-wy.el
gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/wisent/js-wy.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/wisent/js-wy.el
gunzip < /usr/local/stow/emacs-22bee93d92567a1b01ebf7354089e6695e134611--2022-09-03T21-17-50+02-00/share/emacs/29.0.50/lisp/cedet/semantic/wisent/python-wy.el.gz > ~/src/emacs-master-next/lisp/cedet/semantic/wisent/python-wy.el

But now the build fails with

cedet/srecode/cpp.el:31:2: Error: Cannot open load file: No such file or directory, srecode/srt-wy

but there is no lisp/cedet/semantic/srecode directory.


Any ideas, what's happening here?






Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: strange build problem with missing -by.el and -wy.el files

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Date: Mon, 31 Oct 2022 23:09:45 +0100
> 
> In toplevel form:
> cedet/semantic/bovine/c.el:35:2: Error: Cannot open load file: No such file or directory, semantic/bovine/c-by
> make[4]: *** [Makefile:332: cedet/semantic/bovine/c.elc] Error 1
> make[4]: Leaving directory '/home/grfz/src/emacs-master-next/lisp'
> make[3]: *** [Makefile:366: compile-main] Error 2
> make[3]: Leaving directory '/home/grfz/src/emacs-master-next/lisp'
> make[2]: *** [Makefile:531: lisp] Error 2
> make[2]: Leaving directory '/home/grfz/src/emacs-master-next'
> make[2]: Entering directory '/home/grfz/src/emacs-master-next'
> ***
> *** "make all" failed with exit status 2.
> ***
> *** You could try to:
> *** - run "make bootstrap", which might fix the problem
> *** - run "make V=1", which displays the full commands invoked by make,
> ***   to further investigate the problem
> ***
> make[2]: *** [Makefile:413: advice-on-failure] Error 2
> make[2]: Leaving directory '/home/grfz/src/emacs-master-next'
> make[1]: *** [Makefile:370: all] Error 2
> make[1]: Leaving directory '/home/grfz/src/emacs-master-next'
> make: *** [GNUmakefile:95: default] Error 2
> 
> Line 35 in lisp/cedet/semantic/bovine/c.el reads:
> 
> (require 'semantic/bovine/c-by)
> 
> There is no semantic/bovine/c-by.el in the tree, but
> cedet/Changelog.1 says it was removed in 2013-11-30 together
> with other generated files.

It was removed from Git, but it gets regenerated as part of the
build.

> c-by.el starts this way:

So you do have c-by.el in the tree?  Now I'm confused.

> so perhaps we should arrange for all the .elc files needed by semantic/lex and/or semantic/bovine to be present, before compiling c-by.el."
> 
> 
> but    make -j 1   did not help.
> 
> So I copied several .el files which were not generated from
> a not so old installation of emacs, like so:

That's the wrong way of handling the problem.

You need to understand why c-by.el is not generated as part of the
build.  It's supposed to be generated by the rules in
admin/grammars/Makefile.in, which run early on during the build.  It
is run via the rule in lisp/Makefile.in for the target "semantic",
look around line 415 in that file.  Why doesn't that happen in your
case?

Re: strange build problem with missing -by.el and -wy.el files

[Gregor Zattler]

Hi Eli,
* Eli Zaretskii <eliz@gnu.org> [2022-11-01; 09:04 +02]:
>> From: Gregor Zattler <telegraph@gmx.net>
>> Date: Mon, 31 Oct 2022 23:09:45 +0100
[...]
>> Line 35 in lisp/cedet/semantic/bovine/c.el reads:
>>
>> (require 'semantic/bovine/c-by)
>>
>> There is no semantic/bovine/c-by.el in the tree, but
>> cedet/Changelog.1 says it was removed in 2013-11-30 together
>> with other generated files.
>
> It was removed from Git, but it gets regenerated as part of the
> build.
>
>> c-by.el starts this way:
>
> So you do have c-by.el in the tree?  Now I'm confused.

This is a quote from a bug report email by Paul
Eggert I quoted.  It was another problem but also abut
the [bw]y.el files being produced earlier in the build
and some race condition.


[...]
>> So I copied several .el files which were not generated from
>> a not so old installation of emacs, like so:
>
> That's the wrong way of handling the problem.
>
> You need to understand why c-by.el is not generated as part of the
> build.  It's supposed to be generated by the rules in
> admin/grammars/Makefile.in, which run early on during the build.  It
> is run via the rule in lisp/Makefile.in for the target "semantic",
> look around line 415 in that file.  Why doesn't that happen in your
> case?

Because I get "Args out of range: "master", 0, 7"
errors when building these files:

0 ((e39537ea1e...)) grfz@no:~/src/emacs-master-next/lisp$ make V=1 semantic
make -C ../admin/grammars all EMACS="../../src/emacs"
make[1]: Entering directory '/home/grfz/src/emacs-master-next/admin/grammars'
[ ! -f "../../lisp/cedet/semantic/bovine/c-by.el" ] || chmod +w "../../lisp/cedet/semantic/bovine/c-by.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/bovine/grammar -f bovine-batch-make-parser -o "../../lisp/cedet/semantic/bovine/c-by.el" c.by
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/bovine/make-by.el" ] || chmod +w "../../lisp/cedet/semantic/bovine/make-by.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/bovine/grammar -f bovine-batch-make-parser -o "../../lisp/cedet/semantic/bovine/make-by.el" make.by
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/bovine/scm-by.el" ] || chmod +w "../../lisp/cedet/semantic/bovine/scm-by.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/bovine/grammar -f bovine-batch-make-parser -o "../../lisp/cedet/semantic/bovine/scm-by.el" scheme.by
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/grammar-wy.el" ] || chmod +w "../../lisp/cedet/semantic/grammar-wy.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/wisent/grammar -f wisent-batch-make-parser -o "../../lisp/cedet/semantic/grammar-wy.el" grammar.wy
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/wisent/javat-wy.el" ] || chmod +w "../../lisp/cedet/semantic/wisent/javat-wy.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/wisent/grammar -f wisent-batch-make-parser -o "../../lisp/cedet/semantic/wisent/javat-wy.el" java-tags.wy
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/wisent/js-wy.el" ] || chmod +w "../../lisp/cedet/semantic/wisent/js-wy.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/wisent/grammar -f wisent-batch-make-parser -o "../../lisp/cedet/semantic/wisent/js-wy.el" js.wy
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/semantic/wisent/python-wy.el" ] || chmod +w "../../lisp/cedet/semantic/wisent/python-wy.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/wisent/grammar -f wisent-batch-make-parser -o "../../lisp/cedet/semantic/wisent/python-wy.el" python.wy
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
[ ! -f "../../lisp/cedet/srecode/srt-wy.el" ] || chmod +w "../../lisp/cedet/srecode/srt-wy.el"
"../../src/emacs" -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l semantic/wisent/grammar -f wisent-batch-make-parser -o "../../lisp/cedet/srecode/srt-wy.el" srecode-template.wy
Loading semantic/grm-wy-boot...
Args out of range: "master", 0, 7
make[1]: Leaving directory '/home/grfz/src/emacs-master-next/admin/grammars'
0 ((e39537ea1e...)) grfz@no:~/src/emacs-master-next/lisp$


But I have no clue why this happens.

Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: strange build problem with missing -by.el and -wy.el files

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Date: Tue, 01 Nov 2022 16:38:53 +0100
> 
> > You need to understand why c-by.el is not generated as part of the
> > build.  It's supposed to be generated by the rules in
> > admin/grammars/Makefile.in, which run early on during the build.  It
> > is run via the rule in lisp/Makefile.in for the target "semantic",
> > look around line 415 in that file.  Why doesn't that happen in your
> > case?
> 
> Because I get "Args out of range: "master", 0, 7"
> errors when building these files:

So you need to understand why this happens.  Perhaps some stale files
from previous builds, or some other problem?  E.g., what happens if
you delete lisp/cedet/semantic/grm-wy-boot.elc?