Git master head build failure?

Git master head build failure?

[Tim Cross]

Trying to build from current git head (rev:168d2628b23333) and getting
the following error

In toplevel form:
cedet/semantic/bovine/c.el:35:1: Error: Loading file /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.elc failed to provide feature ‘semantic/bovine/c-by’
make[2]: *** [Makefile:279: cedet/semantic/bovine/c.elc] Error 1
make[2]: Leaving directory '/home/tim/git/emacs/lisp'
make[1]: *** [Makefile:302: compile-main] Error 2
make[1]: Leaving directory '/home/tim/git/emacs/lisp'
make: *** [Makefile:398: lisp] Error 2

Tried the following -

git pull
make maintainer-clean
./autogen.sh
./configure
make

also tried make bootstrap.

This is in a repo which previously had the emacs-27 branch checked out.
Now has the master branch checked out.

Is this a known issue or have I missed something?

Platform is Linux (Ubuntu 20.10).

-- 
Tim Cross

Re: Git master head build failure?

[Stefan Monnier]

> In toplevel form:
> cedet/semantic/bovine/c.el:35:1: Error: Loading file
> /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.elc failed to provide
> feature ‘semantic/bovine/c-by’
[...]
> This is in a repo which previously had the emacs-27 branch checked out.
> Now has the master branch checked out.
> Is this a known issue or have I missed something?

I think it's known to the extent that I have seen such problems every
once in a while, but I don't think I've seen an actual bug report for
it yet.

IIUC the problem is that our "make clean" rules don't clean hard enough.
I suspect for example that

    rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el

might help,


        Stefan

Re: Git master head build failure?

[Tim Cross]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> In toplevel form:
>> cedet/semantic/bovine/c.el:35:1: Error: Loading file
>> /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.elc failed to provide
>> feature ‘semantic/bovine/c-by’
> [...]
>> This is in a repo which previously had the emacs-27 branch checked out.
>> Now has the master branch checked out.
>> Is this a known issue or have I missed something?
>
> I think it's known to the extent that I have seen such problems every
> once in a while, but I don't think I've seen an actual bug report for
> it yet.
>
> IIUC the problem is that our "make clean" rules don't clean hard enough.
> I suspect for example that
>
>     rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el
>
> might help,
>

OK, thanks Stefan. I will give that a go.

I thought 'git status' would show me anything 'left' after a clean that
was not part of the repo, but now I notice that file is included in the
.gitignore list, so would not show up.

I wonder if it would be possible for make maintainer-clean to somehow
look at what is in .gitignore and ensure anything in there is also
removed? Would be good if at least that target returned the repo to an
exact replica of what you would get if you did a fresh clone. 


-- 
Tim Cross

Re: Git master head build failure?

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Thu, 01 Apr 2021 14:10:50 +1100
> Cc: emacs-devel@gnu.org
> 
> I wonder if it would be possible for make maintainer-clean to somehow
> look at what is in .gitignore and ensure anything in there is also
> removed? Would be good if at least that target returned the repo to an
> exact replica of what you would get if you did a fresh clone. 

First, you should be using extraclean, not maintainer-clean; see the
top-level Makefile.in for the documentation of the various clean
targets.

And second, we should simply make sure extraclean removes all the
generated files, as documented, and keep it that way.  Looking into
.gitignore means either building its interpreter into our build system
(a non-trivial job, AFAIU), or using Git itself to do the job, which
would mean the job cannot be done without Git -- a disadvantage, IMO.

Re: Git master head build failure?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

On Thu, 1 Apr 2021 at 18:33, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Tim Cross <theophilusx@gmail.com>
> > Date: Thu, 01 Apr 2021 14:10:50 +1100
> > Cc: emacs-devel@gnu.org
> >
> > I wonder if it would be possible for make maintainer-clean to somehow
> > look at what is in .gitignore and ensure anything in there is also
> > removed? Would be good if at least that target returned the repo to an
> > exact replica of what you would get if you did a fresh clone.
>
> First, you should be using extraclean, not maintainer-clean; see the
> top-level Makefile.in for the documentation of the various clean
> targets.
>
> And second, we should simply make sure extraclean removes all the
> generated files, as documented, and keep it that way.  Looking into
> .gitignore means either building its interpreter into our build system
> (a non-trivial job, AFAIU), or using Git itself to do the job, which
> would mean the job cannot be done without Git -- a disadvantage, IMO.
>

In that case, either extraclean should be added to the make help output or
if the preferred mechanism is to read Makefile.in, then remove make help.

-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1850 bytes --]

Re: Git master head build failure?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

On Thu, Apr 01, 2021 at 07:57:58PM +1100, Tim Cross wrote:
> On Thu, 1 Apr 2021 at 18:33, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > From: Tim Cross <theophilusx@gmail.com>
> > > Date: Thu, 01 Apr 2021 14:10:50 +1100
> > > Cc: emacs-devel@gnu.org
> > >
> > > I wonder if it would be possible for make maintainer-clean to somehow
> > > look at what is in .gitignore and ensure anything in there is also
> > > removed? Would be good if at least that target returned the repo to an
> > > exact replica of what you would get if you did a fresh clone.

If you want /that/, then don't ask make -- ask git :-)

Look into `git clean'. Lots of options for cases where you want it actually
to remove files (-f) or just to tell you (-n), whether to remove only ignored
files (-X) or everything not known by the repo (-x) and so on. Decisions,
decisions :-)

I usually go with -fx. Teaches me hard lessons whenever my living room is
untidy :-D

I think one should strive to keep repo and build system as orthogonal as
possible (although they both fight hard against that ;-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Git master head build failure?

[Daniel Brooks]

Eli Zaretskii <eliz@gnu.org> writes:
> And second, we should simply make sure extraclean removes all the
> generated files, as documented, and keep it that way.  Looking into
> .gitignore means either building its interpreter into our build system
> (a non-trivial job, AFAIU), or using Git itself to do the job, which
> would mean the job cannot be done without Git -- a disadvantage, IMO.

There are external libraries that implement it, if anyone wants an
alternative or an inspiration. One example that I know of is the
MIT–licensed Rust crate “ignore”
(https://docs.rs/ignore/0.4.17/ignore/), but note that it only gives you
the files that aren’t ignored rather than all the ones that are.

It’s not all that complicated. In the end it boils down to repeated
application of a 30–line function, plus some parsing of course.
(https://docs.rs/globset/0.4.6/src/globset/glob.rs.html#159-186)

I would say that depending on Git for this doesn’t seem like too big of
a burden; anyone without git will be no worse off than before. Just
print a warning that mentions “git clean” and suggests installing git or
doing it by hand.

db48x

Re: Git master head build failure?

[Gregor Zattler]

Hi Tim, emacs developers,
* Tim Cross <theophilusx@gmail.com> [2021-04-01; 14:10]:
> removed? Would be good if at least that target returned the repo to an
> exact replica of what you would get if you did a fresh clone.

this you can achieve with

rm -rf * ; git checkout -f

Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Git master head build failure?

[Andreas Schwab]

On Apr 01 2021, Gregor Zattler wrote:

> Hi Tim, emacs developers,
> * Tim Cross <theophilusx@gmail.com> [2021-04-01; 14:10]:
>> removed? Would be good if at least that target returned the repo to an
>> exact replica of what you would get if you did a fresh clone.
>
> this you can achieve with

git clean -dfx

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Git master head build failure?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 31 Mar 2021 23:04:39 -0400
> Cc: emacs-devel@gnu.org
> 
> I suspect for example that
> 
>     rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el
> 
> might help,

This should be added to the 'extraclean' target, IMO (together with
all the other generated files in lisp/cedet).

Re: Git master head build failure?

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Wed, 31 Mar 2021 23:04:39 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> I suspect for example that
>> 
>>     rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el
>> 
>> might help,
>
> This should be added to the 'extraclean' target, IMO (together with
> all the other generated files in lisp/cedet).

Do you mean 'maintainer-clean'? I wasn't aware of 'extraclean' and it is
not listed in 'make help'. Help does say about maintainer-clean

'delete almost everything that can be regenerated'

which I think would include those CEDET files (and probably many of the
other files listed in .gitignore). My expectation, which may be wrong,
is that after running 'make maintainer-clean' you would have a really
'clean' repository that would be equivalent to creating a fresh clone. I
guess the key word in the above is 'almost'. 

BTW Stefan was correct. Removing that file fixed the build issue.

-- 
Tim Cross

Re: Git master head build failure?

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> Date: Thu, 01 Apr 2021 19:34:12 +1100
> 
> >>     rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el
> >> 
> >> might help,
> >
> > This should be added to the 'extraclean' target, IMO (together with
> > all the other generated files in lisp/cedet).
> 
> Do you mean 'maintainer-clean'?

no, I meant 'extracklean'.

> I wasn't aware of 'extraclean' and it is not listed in 'make help'.

Another nit to fix, I guess.

> Help does say about maintainer-clean
> 
> 'delete almost everything that can be regenerated'

That "almost" is there for a reason.

> which I think would include those CEDET files (and probably many of the
> other files listed in .gitignore).

Not in this project, no.  maintainer-clean leaves some generated files
that we decided not to remove unless explicitly required.  Switching
to a branch that has diverged significantly from the previous one is a
relatively rare use case, so removing those files would mean
unnecessarily longer builds.

Re: Git master head build failure?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> I wasn't aware of 'extraclean' and it is not listed in 'make help'.
>
> Another nit to fix, I guess.

Does this look okay?

diff --git a/GNUmakefile b/GNUmakefile
index f27163840b..0df5a9abba 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -50,6 +50,8 @@ help:
 	@echo "make distclean        -- delete all build and configuration files,"
 	@echo "                         leave only files included in source
distribution"
 	@echo "make maintainer-clean -- delete almost everything that can be
regenerated"
+	@echo "make extraclean       -- delete everything that can be
regenerated, as well as"
+	@echo "                         backup and autosave files"
 	@echo "make bootstrap        -- delete all compiled files to force a
new bootstrap"
 	@echo "                         from a clean slate, then build in
the normal way"
 	@echo "make uninstall        -- remove files installed by 'make install'"

Re: Git master head build failure?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 1 Apr 2021 03:56:54 -0700
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I wasn't aware of 'extraclean' and it is not listed in 'make help'.
> >
> > Another nit to fix, I guess.
> 
> Does this look okay?

Thanks, but this should go together with fixing our makefile.in files
to follow fit.  I've now done that on the master branch.

Re: Git master head build failure?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, but this should go together with fixing our makefile.in files
> to follow fit.  I've now done that on the master branch.

Thanks.

Re: Git master head build failure?

[Eli Zaretskii]

> Date: Thu, 01 Apr 2021 12:00:34 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > I wasn't aware of 'extraclean' and it is not listed in 'make help'.
> 
> Another nit to fix, I guess.

Now done.

Re: Git master head build failure?

[Eli Zaretskii]

> Date: Thu, 01 Apr 2021 10:25:52 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: theophilusx@gmail.com, emacs-devel@gnu.org
> 
> >     rm /home/tim/git/emacs/lisp/cedet/semantic/bovine/c-by.el
> > 
> > might help,
> 
> This should be added to the 'extraclean' target, IMO (together with
> all the other generated files in lisp/cedet).

Actually, c-by.el would have been deleted already if you said "make
extraclean".

Re: Git master head build failure?

[Stefan Monnier]

> This should be added to the 'extraclean' target, IMO (together with
> all the other generated files in lisp/cedet).

I can never remember which clean target does what (and reading their
description has usually not been sufficient to be able to predict which
files would remain when I wasn't sure).  Luckily I don't think it
matters much because I never invoke such targets.

But what does matter is what happens when we `make bootstrap`, because
this is a much more common need than `make maintainer-clean` or `make
extraclean`.


        Stefan

Re: Git master head build failure?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 01 Apr 2021 09:03:53 -0400
> 
> > This should be added to the 'extraclean' target, IMO (together with
> > all the other generated files in lisp/cedet).
> 
> I can never remember which clean target does what (and reading their
> description has usually not been sufficient to be able to predict which
> files would remain when I wasn't sure).  Luckily I don't think it
> matters much because I never invoke such targets.
> 
> But what does matter is what happens when we `make bootstrap`, because
> this is a much more common need than `make maintainer-clean` or `make
> extraclean`.

I never use "make bootstrap": does that mean it isn't important
either? ;-)

Seriously, though: this depends on your workflows.  AFAIU most people
do "make bootstrap" to solve problems with Lisp macro expansions, and
for that forcing them to regenerate every single file that isn't
versioned is not the best idea.  If you want to start from a pristine
clone, just clone a new worktree.

Re: Git master head build failure?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIU most people do "make bootstrap" to solve problems with Lisp
> macro expansions, and for that forcing them to regenerate every single
> file that isn't versioned is not the best idea.

What do you use instead of bootstrap in this case?
"make mostlyclean && make"?

Re: Git master head build failure?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 1 Apr 2021 08:32:25 -0500
> Cc: theophilusx@gmail.com, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIU most people do "make bootstrap" to solve problems with Lisp
> > macro expansions, and for that forcing them to regenerate every single
> > file that isn't versioned is not the best idea.
> 
> What do you use instead of bootstrap in this case?
> "make mostlyclean && make"?

No, I remove the offending *.elc files and say "make" again.

Re: Git master head build failure?

[Stefan Monnier]

> I never use "make bootstrap": does that mean it isn't important either? ;-)

I almost never *run* `make bootstrap` either, but I do *use* it quite
frequently in emails as the standard tool to solve other people's
build problems.

> Seriously, though: this depends on your workflows.  AFAIU most people
> do "make bootstrap" to solve problems with Lisp macro expansions, and
> for that forcing them to regenerate every single file that isn't
> versioned is not the best idea.  If you want to start from a pristine
> clone, just clone a new worktree.

I agree there's a tension.  But I'm beginning to think that
bootstrap-clean should remove the handful of generated *-by.el and
*-wy.el files (tho an alternative would be to improve the dependency
info so that they get regenerated when their generator's code changes,
which would arguably be even better).


        Stefan

Re: Git master head build failure?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 01 Apr 2021 11:32:18 -0400
> Cc: theophilusx@gmail.com, emacs-devel@gnu.org
> 
> I'm beginning to think that bootstrap-clean should remove the
> handful of generated *-by.el and *-wy.el files

Why is that?

"make bootstrap" isn't for building from a pristine tree, that's not
its purpose.  Perhaps we want such a Make target, but then let's not
overload "bootstrap" with such an additional semantics, let's have a
separate target for that.  Then we will have solved the tension simply
and painlessly.

extraclean and admin/grammars [was Re: Git master head build failure?]

[Glenn Morris]

Eli Zaretskii wrote:

> "make bootstrap" isn't for building from a pristine tree, that's not
> its purpose.

I consider that to be the purpose of bootstrap.
I disagree with the description of extraclean as "and all generated files".

Emacs use of extraclean is really an abuse - see comments in the
top-level Makefile. IIRC, the only differences between extraclean
and bootstrap-clean are:

1) bootstrap-clean does not delete the generated leim files.
This is because they are slow to rebuild and the sources rarely change.

2) bootstrap-clean does not delete the generated semantic grammars.
These are not slow to rebuild. As my comment in admin/grammars/Makefile.in
says, these could easily be deleted in bootstrap-clean.
I don't even really remember why I did it that way now.

Since several people seem to have an issue lately with these files
getting corrupted, I have a good mind to make that change.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> 1) bootstrap-clean does not delete the generated leim files.
> This is because they are slow to rebuild and the sources rarely change.
>
> 2) bootstrap-clean does not delete the generated semantic grammars.
> These are not slow to rebuild. As my comment in admin/grammars/Makefile.in
> says, these could easily be deleted in bootstrap-clean.
> I don't even really remember why I did it that way now.
>
> Since several people seem to have an issue lately with these files
> getting corrupted, I have a good mind to make that change.

My preference would be for "make bootstrap" to delete these files, too
-- so that we have one simple, easy to remember recipe to give users who
have build problems.

Eli suggested adding an additional build target for this, but I'm not
sure I understood the reasoning behind it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> My preference would be for "make bootstrap" to delete these files, too
> -- so that we have one simple, easy to remember recipe to give users who
> have build problems.

Here's timing numbers from my compile rig:

$ make extraclean; time make -j16

real	2m53.046s
user	13m38.421s
sys	0m44.558s

$ time make -j16 bootstrap

real	2m46.672s
user	12m47.264s
sys	0m44.395s

So the difference isn't very large.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Fri, 2 Apr 2021 at 03:19, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> > My preference would be for "make bootstrap" to delete these files, too
> > -- so that we have one simple, easy to remember recipe to give users who
> > have build problems.
>
> Here's timing numbers from my compile rig:
>
> $ make extraclean; time make -j16
>
> real    2m53.046s
> user    13m38.421s
> sys     0m44.558s
>
> $ time make -j16 bootstrap
>
> real    2m46.672s
> user    12m47.264s
> sys     0m44.395s
>
> So the difference isn't very large.
>
>
I was going to post something similar. I found the same when I did a timing
comparison as well. The difference is very small and doesn't provide much
support for justifying having both targets. There may be other reasons to
have both, but build time differences don't seem sufficient, especially
given the potential confusion wrt the different targets.


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Fri, 2 Apr 2021 15:19:02 +1100
> Cc: Glenn Morris <rgm@gnu.org>, Eli Zaretskii <eliz@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,
>  Emacs developers <emacs-devel@gnu.org>
> 
>  $ time make -j16 bootstrap
> 
>  real    2m46.672s
>  user    12m47.264s
>  sys     0m44.395s
> 
>  So the difference isn't very large.
> 
> I was going to post something similar. I found the same when I did a timing comparison as well. The
> difference is very small and doesn't provide much support for justifying having both targets. There may be
> other reasons to have both, but build time differences don't seem sufficient, especially given the potential
> confusion wrt the different targets. 

To the extent that times are at all relevant, you should time on a
slow machine, and without -j16 (which hints on a modern and a very
powerful CPU).  Otherwise the times are completely uninteresting in
this context (although even they already show an extra minute of user
time).

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> To the extent that times are at all relevant, you should time on a
> slow machine, and without -j16 (which hints on a modern and a very
> powerful CPU). 

The percentage increase in time should be somewhat similar.  It's less
than a 10% increase in "make bootstrap" time (which is something you say
you never do).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Tim Cross <theophilusx@gmail.com>,  rgm@gnu.org,
>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 02 Apr 2021 15:41:51 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > To the extent that times are at all relevant, you should time on a
> > slow machine, and without -j16 (which hints on a modern and a very
> > powerful CPU). 
> 
> The percentage increase in time should be somewhat similar.  It's less
> than a 10% increase in "make bootstrap" time (which is something you say
> you never do).

Percentage is not the only thing that counts; absolute times count as
well.  Even an extra minute is long enough to annoy, and on older
machines it will be more.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  theophilusx@gmail.com,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Thu, 01 Apr 2021 18:09:57 +0200
> 
> Eli suggested adding an additional build target for this, but I'm not
> sure I understood the reasoning behind it.

I thought I explained the reasoning: "make bootstrap" is for
bootstrapping Emacs, not for building from a pristine worktree.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I thought I explained the reasoning: "make bootstrap" is for
> bootstrapping Emacs, not for building from a pristine worktree.

I don't understand what you mean the difference between "bootstrapping
Emacs" and "building from a pristine worktree" is.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rgm@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Fri, 02 Apr 2021 15:40:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I thought I explained the reasoning: "make bootstrap" is for
> > bootstrapping Emacs, not for building from a pristine worktree.
> 
> I don't understand what you mean the difference between "bootstrapping
> Emacs" and "building from a pristine worktree" is.

Bootstrapping Emacs means building a working Emacs executable when we
either have no Emacs executable at all or that executable isn't
working or is broken for some reason.

IOW, bootstrapping Emacs doesn't _have_ to delete all generated files,
though it _might_ delete those which are in the way of building a
working Emacs.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Bootstrapping Emacs means building a working Emacs executable when we
> either have no Emacs executable at all or that executable isn't
> working or is broken for some reason.
>
> IOW, bootstrapping Emacs doesn't _have_ to delete all generated files,
> though it _might_ delete those which are in the way of building a
> working Emacs.

I still don't understand what mean here, since it's been demonstrated
several times that (in certain circumstances) people have to say "make
extraclean" to build a working Emacs executable.  So "make bootstrap"
doesn't do what it says on the tin, and the suggestion is to make it do
so.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rgm@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Sat, 03 Apr 2021 15:30:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Bootstrapping Emacs means building a working Emacs executable when we
> > either have no Emacs executable at all or that executable isn't
> > working or is broken for some reason.
> >
> > IOW, bootstrapping Emacs doesn't _have_ to delete all generated files,
> > though it _might_ delete those which are in the way of building a
> > working Emacs.
> 
> I still don't understand what mean here, since it's been demonstrated
> several times that (in certain circumstances) people have to say "make
> extraclean" to build a working Emacs executable.

"make extraclean" is only needed when doing something very drastic,
like switching to a very different branch.  That is an extremely rare
(if not improbable) use case, so IMO having everyone pay for it is
unjustified.

> So "make bootstrap" doesn't do what it says on the tin

But it does: it bootstraps the current configuration of Emacs by doing
what is necessary, but not more than that.

It sounds like you think that "make bootstrap" is equivalent to
checking out a pristine clone.  But that is not the case, at least not
for most people who use that target.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[chad]

[-- Attachment #1: Type: text/plain, Size: 2105 bytes --]

On Sat, Apr 3, 2021 at 7:16 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > I still don't understand what mean here, since it's been demonstrated
> > several times that (in certain circumstances) people have to say "make
> > extraclean" to build a working Emacs executable.
>
> "make extraclean" is only needed when doing something very drastic,
> like switching to a very different branch.  That is an extremely rare
> (if not improbable) use case, so IMO having everyone pay for it is
> unjustified.
>
> > So "make bootstrap" doesn't do what it says on the tin
>
> But it does: it bootstraps the current configuration of Emacs by doing
> what is necessary, but not more than that.
>
> It sounds like you think that "make bootstrap" is equivalent to
> checking out a pristine clone.  But that is not the case, at least not
> for most people who use that target.
>

Without getting too caught up in the details of who-said-what-when, when I
read the Makefile, this text:

# make bootstrap
> #      Removes all the compiled files to force a new bootstrap from a
> #      clean slate, and then build in the normal way.


..it makes me think something much closer to what I think Lars was saying
"make bootstrap" should do than "only use this if you don't have
src/{t,}emacs", which is roughly how I interpret (perhaps wrongly) what Eli
is suggesting "make boostrap" is for. It also surprises me that emacs'
build system needs clean, mostlyclean, distclean, maintainer-clean,
extraclean, and bootstrap, but that overhead seems very low, mostly
confusion around which to use.

Probably, I'm biased by my own use-case, which is roughly: I periodically
(multiple times a week) pull and bootstrap emacs using a custom build
script, so I can test the latest and greatest of emacs-devel. When I do
work on specific changes myself, I use "make all" nearly all the time (I
have memories of needing to bootstrap now and then, but I can't recall the
circumstances). That said, I'm using a moderately performant machine,
(currently 3 years old, before that up to 8 years old) and typically run
-j6.

Hope that helps.
~Chad

[-- Attachment #2: Type: text/html, Size: 2829 bytes --]

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sat, 3 Apr 2021 15:36:51 -0700
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, rgm@gnu.org, Tim Cross <theophilusx@gmail.com>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>, EMACS development team <emacs-devel@gnu.org>
> 
>  # make bootstrap
>  #      Removes all the compiled files to force a new bootstrap from a
>  #      clean slate, and then build in the normal way.
> 
> ..it makes me think something much closer to what I think Lars was saying "make bootstrap" should do than
> "only use this if you don't have src/{t,}emacs", which is roughly how I interpret (perhaps wrongly) what Eli is
> suggesting "make boostrap" is for. It also surprises me that emacs' build system needs clean, mostlyclean,
> distclean, maintainer-clean, extraclean, and bootstrap, but that overhead seems very low, mostly confusion
> around which to use. 

I guess you've missed the "all the compiled files" part?

> Probably, I'm biased by my own use-case, which is roughly: I periodically (multiple times a week) pull and
> bootstrap emacs using a custom build script, so I can test the latest and greatest of emacs-devel. When I do
> work on specific changes myself, I use "make all" nearly all the time (I have memories of needing to
> bootstrap now and then, but I can't recall the circumstances). That said, I'm using a moderately performant
> machine, (currently 3 years old, before that up to 8 years old) and typically run -j6.

I think your workflow is based on old Makefiles, where "make
bootstrap" was needed much more frequently than nowadays.  When we
switched to requiring GNU Make, many changes were made in the
configury stuff that make bootstrap needed much less frequently.  From
now on, I suggest that you just say "make -j6", it should "just work".

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Sun, 4 Apr 2021 at 01:15, Eli Zaretskii <eliz@gnu.org> wrote:

>
> "make extraclean" is only needed when doing something very drastic,
> like switching to a very different branch.  That is an extremely rare
> (if not improbable) use case, so IMO having everyone pay for it is
> unjustified.
>
>
I'm not sure that assumption is correct. You can run into this issue just
changing commits, not switching branches if that commit is old enough. For
example, checking out the emacs-27 tag. I'm also not convinced moving
between the last released version and the current development version and
back again is that unusual and the longer things go between the last
release and current development head, the larger the difference in commits
and greater the potential for this sort of issue.


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1353 bytes --]

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Sun, 4 Apr 2021 09:52:31 +1000
> Cc: Glenn Morris <rgm@gnu.org>, Lars Ingebrigtsen <larsi@gnus.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,
>  Emacs developers <emacs-devel@gnu.org>
> 
> On Sun, 4 Apr 2021 at 01:15, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  "make extraclean" is only needed when doing something very drastic,
>  like switching to a very different branch.  That is an extremely rare
>  (if not improbable) use case, so IMO having everyone pay for it is
>  unjustified.
> 
> I'm not sure that assumption is correct. You can run into this issue just changing commits, not switching
> branches if that commit is old enough. For example, checking out the emacs-27 tag. I'm also not convinced
> moving between the last released version and the current development version and back again is that
> unusual and the longer things go between the last release and current development head, the larger the
> difference in commits and greater the potential for this sort of issue. 

It isn't an assumption: I'm doing all of the above all the time.

Switching between branches in Emacs is not recommended, because they
diverge very quickly.  If you must do that, "make extraclean" is the
way to go.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I still don't understand what mean here, since it's been demonstrated
>> several times that (in certain circumstances) people have to say "make
>> extraclean" to build a working Emacs executable.
>
> "make extraclean" is only needed when doing something very drastic,
> like switching to a very different branch.  That is an extremely rare
> (if not improbable) use case, so IMO having everyone pay for it is
> unjustified.

It doesn't seem like we're going to get any further here?  My opinion is
that "make bootstrap" should guarantee a build (to the best of our
abilities).  Yours is that it shouldn't.

I don't have the stamina to debate the issue further.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Stefan Monnier]

> Emacs use of extraclean is really an abuse - see comments in the
> top-level Makefile. IIRC, the only differences between extraclean
> and bootstrap-clean are:

From a theoretical standpoint, I think `bootstrap-clean` should delete the
files that are both:
A) generated.
B) not faithfully described by the makefile dependencies.

In practice, very few of our generated files have 100% of their
dependencies  faithfully described in our makefiles, so the theory needs
to be taken with a large grain of salt.

Others less theoretical and more pragmatic factors that can be important are
C) used during the generation of other files.
D) changes frequently.

(C) can be important because it can cause not just errors down the line
when using the corresponding functionality, but also errors during the
build itself.

> 1) bootstrap-clean does not delete the generated leim files.
> This is because they are slow to rebuild and the sources rarely change.

AFAIK these don't satisfy B very much, (nor C nor D) so I don't see much
need to delete them in `bootstrap-clean`, indeed.

> 2) bootstrap-clean does not delete the generated semantic grammars.
> These are not slow to rebuild.  As my comment in
> admin/grammars/Makefile.in says, these could easily be deleted in
> bootstrap-clean.  I don't even really remember why I did it that
> way now.

I think for those we could also improve the makefile dependencies.

E.g. a first step could be to move `grammar-wy.el` to `grm-wy-boot.el` like we
do with `ldefs-boot.el` (see patch below) to break the
circular dependency.
WDYT?


        Stefan

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 01 Apr 2021 13:15:56 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, theophilusx@gmail.com, emacs-devel@gnu.org
> 
> E.g. a first step could be to move `grammar-wy.el` to `grm-wy-boot.el` like we
> do with `ldefs-boot.el` (see patch below) to break the
> circular dependency.

What patch?

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Stefan Monnier]

>> E.g. a first step could be to move `grammar-wy.el` to `grm-wy-boot.el` like we
>> do with `ldefs-boot.el` (see patch below) to break the
>> circular dependency.
>
> What patch?

What do you mean "what patch"?  *The* patch, of course.


        Stefan


diff --git a/.gitignore b/.gitignore
index b653ef215b..9fe8ecb594 100644
--- a/.gitignore
+++ b/.gitignore
@@ -88,6 +88,7 @@ lisp/cedet/semantic/wisent/javat-wy.el
 lisp/cedet/semantic/wisent/js-wy.el
 lisp/cedet/semantic/wisent/python-wy.el
 lisp/cedet/srecode/srt-wy.el
+lisp/cedet/semantic/grammar-wy.el
 lisp/eshell/esh-groups.el
 lisp/finder-inf.el
 lisp/leim/ja-dic/
diff --git a/admin/grammars/Makefile.in b/admin/grammars/Makefile.in
index aa09d9edf9..800e31762d 100644
--- a/admin/grammars/Makefile.in
+++ b/admin/grammars/Makefile.in
@@ -48,14 +48,11 @@ BOVINE =
 	${bovinedir}/make-by.el \
 	${bovinedir}/scm-by.el
 
-## FIXME Should include this one too:
-##	${cedetdir}/semantic/grammar-wy.el
-## but semantic/grammar.el (which is what we use to generate grammar-wy.el)
-## requires it!
-WISENT = \
-	${wisentdir}/javat-wy.el \
-	${wisentdir}/js-wy.el \
-	${wisentdir}/python-wy.el \
+WISENT = 				   \
+	${cedetdir}/semantic/grammar-wy.el \
+	${wisentdir}/javat-wy.el 	   \
+	${wisentdir}/js-wy.el 		   \
+	${wisentdir}/python-wy.el 	   \
 	${cedetdir}/srecode/srt-wy.el
 
 ALL = ${BOVINE} ${WISENT}
diff --git a/lisp/cedet/semantic/grammar.el b/lisp/cedet/semantic/grammar.el
index dba289fdd7..782327e617 100644
--- a/lisp/cedet/semantic/grammar.el
+++ b/lisp/cedet/semantic/grammar.el
@@ -31,7 +31,12 @@
 (require 'semantic/format)
 ;; FIXME this is a generated file, but we need to load this file to
 ;; generate it!
-(require 'semantic/grammar-wy)
+;; We need `semantic/grammar-wy.el' but we're also needed to generate
+;; that file from `grammar.wy', so to break the dependency, we keep
+;; a bootstrap copy of `grammar-wy.el' in `grm-wy-boot.el'.
+(eval-and-compile
+  (unless (require 'semantic/grammar-wy nil t)
+    (load "semantic/grm-wy-boot")))
 (require 'semantic/idle)
 (require 'help-fns)
 (require 'semantic/analyze)
diff --git a/lisp/cedet/semantic/grammar-wy.el b/lisp/cedet/semantic/grm-wy-boot.el
similarity index 100%
rename from lisp/cedet/semantic/grammar-wy.el
rename to lisp/cedet/semantic/grm-wy-boot.el

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rgm@gnu.org,  theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 01 Apr 2021 16:13:24 -0400
> 
> >> E.g. a first step could be to move `grammar-wy.el` to `grm-wy-boot.el` like we
> >> do with `ldefs-boot.el` (see patch below) to break the
> >> circular dependency.
> >
> > What patch?
> 
> What do you mean "what patch"?  *The* patch, of course.

Assuming you tested that during bootstrap (which you say you never
do), and assuming there's no better way of breaking the circular
dependency, I'm okay with the change.  But please also change
admin/make-tarball.txt to say that grm-wy-boot.el should be updated
the same way as ldefs-boot.el is.  We should also make sure
grm-wy-boot.el is updated in Git whenever grammar.wy changes.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Stefan Monnier]

> Assuming you tested that during bootstrap (which you say you never
> do), and assuming there's no better way of breaking the circular
> dependency, I'm okay with the change.

There might be a better way (I think providing an ELisp syntax for the
wisent source files could be such a thing), but as long as we can't find
anyone motivated enough to tackle that, it seems like the better way, yes.

> We should also make sure grm-wy-boot.el is updated in Git whenever
> grammar.wy changes.

Done, thanks.


        Stefan

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 12 Apr 2021 22:20:14 -0400
> Cc: rgm@gnu.org, theophilusx@gmail.com, emacs-devel@gnu.org
> 
> > Assuming you tested that during bootstrap (which you say you never
> > do), and assuming there's no better way of breaking the circular
> > dependency, I'm okay with the change.
> 
> There might be a better way (I think providing an ELisp syntax for the
> wisent source files could be such a thing), but as long as we can't find
> anyone motivated enough to tackle that, it seems like the better way, yes.
> 
> > We should also make sure grm-wy-boot.el is updated in Git whenever
> > grammar.wy changes.
> 
> Done, thanks.

After updating from upstream, I see that cedet/semantic/grm-wy-boot.el
is byte-compiled by the build process.  Should it be?

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> Date: Tue, 13 Apr 2021 14:51:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rgm@gnu.org, theophilusx@gmail.com, emacs-devel@gnu.org
> 
> > > We should also make sure grm-wy-boot.el is updated in Git whenever
> > > grammar.wy changes.
> > 
> > Done, thanks.
> 
> After updating from upstream, I see that cedet/semantic/grm-wy-boot.el
> is byte-compiled by the build process.  Should it be?

And on the native-comp branch I see the following warning:

    GEN      ../../lisp/cedet/semantic/grammar-wy.el
  Warning (comp): Cannot look-up eln file as no source file was found for d:/gnu/git/emacs/native-comp/lisp/cedet/semantic/grammar-wy.elc

Is this expected?

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Stefan Monnier]

> And on the native-comp branch I see the following warning:
>
>     GEN      ../../lisp/cedet/semantic/grammar-wy.el
>   Warning (comp): Cannot look-up eln file as no source file was found for d:/gnu/git/emacs/native-comp/lisp/cedet/semantic/grammar-wy.elc
>
> Is this expected?

Hmm... no, I must say I don't know what this is about.  Andrea?


        Stefan

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Stefan Monnier]

> After updating from upstream, I see that cedet/semantic/grm-wy-boot.el
> is byte-compiled by the build process.  Should it be?

To the extent that it is only used during bootstrap, it's not very
useful to compile it, indeed.  But it's harmless, so I think it's better
to leave it like this than to try and avoid compiling it.

I guess we could move those bootstrap files out of `lisp` into their own
dedicated "bootstrap directory", but I'm not sure it's worth the trouble.


        Stefan

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rgm@gnu.org,  theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 13 Apr 2021 08:59:54 -0400
> 
> > After updating from upstream, I see that cedet/semantic/grm-wy-boot.el
> > is byte-compiled by the build process.  Should it be?
> 
> To the extent that it is only used during bootstrap, it's not very
> useful to compile it, indeed.  But it's harmless, so I think it's better
> to leave it like this than to try and avoid compiling it.

Can't we exempt that file in the loop that compile-man target runs in
lisp/Makefile?

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 01 Apr 2021 12:05:25 -0400
> 
> Eli Zaretskii wrote:
> 
> > "make bootstrap" isn't for building from a pristine tree, that's not
> > its purpose.
> 
> I consider that to be the purpose of bootstrap.

I disagree.

> I disagree with the description of extraclean as "and all generated files".

I disagree with your disagreement.

> Since several people seem to have an issue lately with these files
> getting corrupted, I have a good mind to make that change.

Please don't.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Glenn Morris]

I've tidied this up, so it is at least now possible to recommend the
standard maintainer-clean instead of the non-standard extraclean.

Personally I'm sure bootstrap should be equivalent to maintainer-clean,
except maybe for leim/ja-dic. But the generated files from
admin/{charsets,grammars,unidata} are all trivial to recreate,
and I don't see why there has to be a big debate about it.

Re: extraclean and admin/grammars [was Re: Git master head build failure?]

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sun, 04 Apr 2021 12:28:45 -0400
> 
> I've tidied this up, so it is at least now possible to recommend the
> standard maintainer-clean instead of the non-standard extraclean.
> 
> Personally I'm sure bootstrap should be equivalent to maintainer-clean,
> except maybe for leim/ja-dic. But the generated files from
> admin/{charsets,grammars,unidata} are all trivial to recreate,
> and I don't see why there has to be a big debate about it.

That's just rude!  No matter what you think about the subject,
disagreement was clearly voiced, and the least one could expect in a
friendly community is that you talk before you act to the contrary.