bug#16008: semantic/grammar-wy.el cannot be bootstrapped

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Glenn Morris]

Package: emacs
Version: 24.3.50
User: emacs
Usertags: cedet

semantic/grammar-wy.el is a generated file and therefore ideally should
not be kept in the Emacs VCS.

It is generated by semantic/grammar.el.
semantic/grammar.el requires semantic/grammar-wy.el, therefore
it is not possible to bootstrap without semantic/grammar-wy.el already
present.

I should also note that the version of semantic/grammar-wy.el in the
Emacs trunk does not match up with its sources.

It contains:

  (require 'semantic)

Yet if you regenerate it, it contains:

  (require 'semantic/lex)
  (eval-when-compile (require 'semantic/bovine))

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Glenn Morris]

Glenn Morris wrote:

> It is generated by semantic/grammar.el.
> semantic/grammar.el requires semantic/grammar-wy.el, therefore
> it is not possible to bootstrap without semantic/grammar-wy.el already
> present.

I should have said: it is generated by wisent/grammar.el, which requires
semantic/grammar.el, which requires semantic/grammar-wy.el.

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[David Engster]

Glenn Morris writes:
> semantic/grammar-wy.el is a generated file and therefore ideally should
> not be kept in the Emacs VCS.

That's true for all parsers. However, grammar-wy.el is special, as
you've noticed.

> It is generated by semantic/grammar.el.
> semantic/grammar.el requires semantic/grammar-wy.el, therefore
> it is not possible to bootstrap without semantic/grammar-wy.el already
> present.

Yes, it has a circular dependency, because the parser grammar-wy.el
parses grammar files, including itself. In upstream, we have a fallback
parser for this, which is used when grammar-wy.el is not present
yet. Needless to say, you'll soon face chicken/egg problems when you do
changes in grammar.wy, which IMHO cause more pain in the rear than
having generated files under version control. In fact, every time I
worked on grammar.wy, I thought about getting rid of bootstrapping at
compile time and just put all parsers under version control.

> I should also note that the version of semantic/grammar-wy.el in the
> Emacs trunk does not match up with its sources.
>
> It contains:
>
>   (require 'semantic)
>
> Yet if you regenerate it, it contains:
>
>   (require 'semantic/lex)
>   (eval-when-compile (require 'semantic/bovine))

Yes, there may be small differences w.r.t. to requires and such. The
generated parsers should never differ, though.

Problem is: the freeze is near, and I have to do another sync with
upstream, and I'd also really like to get a few things fixed before
that.

-David

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Stefan Monnier]

David Engster [2013-11-30 11:00:47] wrote:
> Glenn Morris writes:
>> semantic/grammar-wy.el is a generated file and therefore ideally should
>> not be kept in the Emacs VCS.
> That's true for all parsers. However, grammar-wy.el is special, as
> you've noticed.
>> It is generated by semantic/grammar.el.
>> semantic/grammar.el requires semantic/grammar-wy.el, therefore
>> it is not possible to bootstrap without semantic/grammar-wy.el already
>> present.
> Yes, it has a circular dependency, because the parser grammar-wy.el
> parses grammar files, including itself.  In upstream, we have
> a fallback parser for this, which is used when grammar-wy.el is not
> present yet.

Of course, another option would be to add a sexp-based syntax for wisent
grammars, so that grammar.wy can be rewritten with that sexp-based syntax
and won't need a wisent parser to parse it any more.

In the mean time I suggest the patch below which I recently sent to
emacs-devel.

Eli Zaretskii [2021-04-03 10:58:12] wrote:
> Assuming you tested that during bootstrap (which you say you never
> do), and assuming there's no better way of breaking the circular
> dependency, I'm okay with the change.  But please also change
> admin/make-tarball.txt to say that grm-wy-boot.el should be updated
> the same way as ldefs-boot.el is.  We should also make sure
> grm-wy-boot.el is updated in Git whenever grammar.wy changes.

Glenn, could you arrange to "auto""-update `grm-wy-boot.el` like you do
for `ldefs-boot.el`?


        Stefan


diff --git a/.gitignore b/.gitignore
index b653ef215b..9fe8ecb594 100644
--- a/.gitignore
+++ b/.gitignore
@@ -88,6 +88,7 @@ lisp/cedet/semantic/wisent/javat-wy.el
 lisp/cedet/semantic/wisent/js-wy.el
 lisp/cedet/semantic/wisent/python-wy.el
 lisp/cedet/srecode/srt-wy.el
+lisp/cedet/semantic/grammar-wy.el
 lisp/eshell/esh-groups.el
 lisp/finder-inf.el
 lisp/leim/ja-dic/
diff --git a/admin/grammars/Makefile.in b/admin/grammars/Makefile.in
index aa09d9edf9..800e31762d 100644
--- a/admin/grammars/Makefile.in
+++ b/admin/grammars/Makefile.in
@@ -48,14 +48,11 @@ BOVINE =
 	${bovinedir}/make-by.el \
 	${bovinedir}/scm-by.el
 
-## FIXME Should include this one too:
-##	${cedetdir}/semantic/grammar-wy.el
-## but semantic/grammar.el (which is what we use to generate grammar-wy.el)
-## requires it!
-WISENT = \
-	${wisentdir}/javat-wy.el \
-	${wisentdir}/js-wy.el \
-	${wisentdir}/python-wy.el \
+WISENT = 				   \
+	${cedetdir}/semantic/grammar-wy.el \
+	${wisentdir}/javat-wy.el 	   \
+	${wisentdir}/js-wy.el 		   \
+	${wisentdir}/python-wy.el 	   \
 	${cedetdir}/srecode/srt-wy.el
 
 ALL = ${BOVINE} ${WISENT}
diff --git a/lisp/cedet/semantic/grammar.el b/lisp/cedet/semantic/grammar.el
index dba289fdd7..782327e617 100644
--- a/lisp/cedet/semantic/grammar.el
+++ b/lisp/cedet/semantic/grammar.el
@@ -31,7 +31,12 @@
 (require 'semantic/format)
 ;; FIXME this is a generated file, but we need to load this file to
 ;; generate it!
-(require 'semantic/grammar-wy)
+;; We need `semantic/grammar-wy.el' but we're also needed to generate
+;; that file from `grammar.wy', so to break the dependency, we keep
+;; a bootstrap copy of `grammar-wy.el' in `grm-wy-boot.el'.  See bug#16008.
+(eval-and-compile
+  (unless (require 'semantic/grammar-wy nil t)
+    (load "semantic/grm-wy-boot")))
 (require 'semantic/idle)
 (require 'help-fns)
 (require 'semantic/analyze)
diff --git a/lisp/cedet/semantic/grammar-wy.el b/lisp/cedet/semantic/grm-wy-boot.el
similarity index 100%
rename from lisp/cedet/semantic/grammar-wy.el
rename to lisp/cedet/semantic/grm-wy-boot.el

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Glenn Morris]

Stefan Monnier wrote:

> In the mean time I suggest the patch below which I recently sent to
> emacs-devel.

Instead of a -boot version, how about regenerating it but not deleting it:

--- a/admin/grammars/Makefile.in
+++ b/admin/grammars/Makefile.in
@@ -51,16 +51,19 @@ BOVINE = \
 	${bovinedir}/make-by.el \
 	${bovinedir}/scm-by.el
 
-## FIXME Should include this one too:
-##	${cedetdir}/semantic/grammar-wy.el
-## but semantic/grammar.el (which is what we use to generate grammar-wy.el)
-## requires it!  https://debbugs.gnu.org/16008
-WISENT = \
+WISENT1 = \
 	${wisentdir}/javat-wy.el \
 	${wisentdir}/js-wy.el \
 	${wisentdir}/python-wy.el \
 	${cedetdir}/srecode/srt-wy.el
 
+DELETE = ${BOVINE} ${WISENT1}
+
+## The last one is special: semantic/grammar.el (which is what generates it)
+## requires it.  https://debbugs.gnu.org/16008
+## So we generate it, but never delete it.
+WISENT = ${WISENT1} ${cedetdir}/semantic/grammar-wy.el
+
 ALL = ${BOVINE} ${WISENT}
 
 .PHONY: all bovine wisent
@@ -107,7 +110,7 @@ distclean:
 bootstrap-clean:
 
 gen-clean:
-	rm -f ${ALL}
+	rm -f ${DELETE}
 
 maintainer-clean: gen-clean distclean
 

> Glenn, could you arrange to "auto""-update `grm-wy-boot.el` like you do
> for `ldefs-boot.el`?

It's admin/update_autogen.

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sun, 04 Apr 2021 20:49:39 -0400
> Cc: 16008@debbugs.gnu.org, cedet-devel@lists.sourceforge.net
> 
> Stefan Monnier wrote:
> 
> > In the mean time I suggest the patch below which I recently sent to
> > emacs-devel.
> 
> Instead of a -boot version, how about regenerating it but not deleting it:

Thanks, but doesn't this get us back to the original situation,
whereby some stale generated *.el files were left behind and failed
the build?

bug#16008: semantic/grammar-wy.el cannot be bootstrapped

[Glenn Morris]

Version: 28.1

Stefan Monnier wrote:

> In the mean time I suggest the patch below which I recently sent to
> emacs-devel.

It seems this patch was applied in 214dfbf, but sadly the discussion
wasn't sent to this bug report.