bug#19206: 25.0.50; CC Mode tracks wrong source files

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Sebastian Wiesner]

CC Mode tracks wrong source files when a CC Mode derived mode is
installed non-interactively.

To reproduce, save the following code as `cc-miscompile.el'

(require 'package)
(require 'cc-defs)

(defun main ()
  (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))

  (setq package-user-dir (make-temp-file "cc-miscompile" 'directory))

  (package-initialize)
  (package-refresh-contents)
  (package-install 'd-mode)

  (require 'd-mode)

  (let ((source (get (intern "c-typedef-decl-kwds" c-lang-constants) 'source)))
    (message "Sources: %S" (mapcar 'car source)))

  (delete-directory package-user-dir 'recursive))

(main)

and run it with `emacs -Q --script cc-miscompile.el'.  The output is as
follows (package.el output shortened for readility):

Contacting host: melpa.org:80
Contacting host: elpa.gnu.org:80
[…]
Sources: (d-mode cc-miscompile cc-langs)

Note that `cc-miscompile' ends up in the source list of
`c-typedef-decl-kwds', even though it never actually calls any `c-*'
functions at all.

Naturally, CC Mode will later try to load this file, and fail if it is
not in the `load-path'.  This effectively breaks installations of D
Mode from non-interactive Emacs sessions.

I did not try to find the culprit.  The CC Mode code is convoluted
beyond my understanding.

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Stefan Monnier]

> I did not try to find the culprit.  The CC Mode code is convoluted
> beyond my understanding.

Agreed, it's madness.  I'd like to streamline it (at some runtime cost),
but that requires major modes that use CC to do

  (require 'cc-langs)

rather than

  (eval-when-compile (require 'cc-langs))

Still, you can already do that and I recommend it since it should reduce
the number of ways the code can fail.  More to the point, it might fix
the problem you're experiencing (maybe cc-miscompile will still be
mentioned in there, but it hopefully won't be any more).


        Stefan

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Alan Mackenzie]

Hello, Sebastian.
In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
> CC Mode tracks wrong source files when a CC Mode derived mode is
> installed non-interactively.

The rest of your post describes your detective work to track down the
problem, which is brilliant.  But you haven't said what the problem itself
is, at least not in high level terms.

What does the file look like which does the non-interactive installation,
when do you see an error, and what is this error message?

> To reproduce, save the following code as `cc-miscompile.el'

> (require 'package)
> (require 'cc-defs)

> (defun main ()
>  (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))

>  (setq package-user-dir (make-temp-file "cc-miscompile" 'directory))

>  (package-initialize)
>  (package-refresh-contents)
>  (package-install 'd-mode)

>  (require 'd-mode)

>  (let ((source (get (intern "c-typedef-decl-kwds" c-lang-constants) 'source)))
>    (message "Sources: %S" (mapcar 'car source)))

>  (delete-directory package-user-dir 'recursive))

> (main)

> and run it with `emacs -Q --script cc-miscompile.el'.  The output is as
> follows (package.el output shortened for readility):

> Contacting host: melpa.org:80
> Contacting host: elpa.gnu.org:80
> [?]
> Sources: (d-mode cc-miscompile cc-langs)

> Note that `cc-miscompile' ends up in the source list of
> `c-typedef-decl-kwds', even though it never actually calls any `c-*'
> functions at all.

The byte compilation of d-mode.el is being done during the loading of
cc-miscompile.el.  This somewhat unusual constellation, I think, is
causing the problem.  When CC Mode determines the file name to put onto
a c-lang-defconst's 'source property, it gives priority to the load file
name, and only when this is nil does it use the byte-compile file name.
(This is in defsubst c-get-current-file in cc-defs.el).  It would seem
this is not the correct priority.

I think swapping the first two arms of the `cond' form in
c-get-current-file may solve the problem.  It's a bit late to try this
tonight, I'll try it tomorrow.

> Naturally, CC Mode will later try to load this file, and fail if it is
> not in the `load-path'.  This effectively breaks installations of D
> Mode from non-interactive Emacs sessions.



> I did not try to find the culprit.  The CC Mode code is convoluted
> beyond my understanding.

The mechanism for the c-lang-defvar's may appear complicated, but it this
concentration of the complexity in a single place that enables the simple,
tabular definition of language dependent constants, even (especially) in
derived modes.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Sebastian Wiesner]

> Am 28.11.2014 um 23:25 schrieb Alan Mackenzie <acm@muc.de>:
> 
> Hello, Sebastian.
> In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
>> CC Mode tracks wrong source files when a CC Mode derived mode is
>> installed non-interactively.
> 
> The rest of your post describes your detective work to track down the
> problem, which is brilliant.  But you haven't said what the problem itself
> is, at least not in high level terms.
> 
> What does the file look like which does the non-interactive installation,
> when do you see an error, and what is this error message?

The “file” that does the non-interactive installation is Cask from https://github.com/cask/cask/.  

I use Cask to install packages into per-project directories, and to run ERT test suites for Emacs Lisp in these “per-project” package environments.

The error occurs when a test case tries to enable D Mode.  It's a standard load file error, pointing to the main script of Cask.

This seems all rather irrelevant to me, though.  The sample code demonstrates the issue quite clearly.

>> I did not try to find the culprit.  The CC Mode code is convoluted
>> beyond my understanding.
> 
> The mechanism for the c-lang-defvar's may appear complicated, but it this
> concentration of the complexity in a single place that enables the simple,
> tabular definition of language dependent constants, even (especially) in
> derived modes.

Well, if you say.  I'm curious, though, what this system would enable me to do, that an ordinary `require' could not?

Greetings,
Sebastian

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Alan Mackenzie]

Hello, again, Sebastian.

On Fri, Nov 28, 2014 at 10:25:42PM -0000, Alan Mackenzie wrote:
> Hello, Sebastian.
> In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
> > CC Mode tracks wrong source files when a CC Mode derived mode is
> > installed non-interactively.

> > To reproduce, save the following code as `cc-miscompile.el'

> > (require 'package)
> > (require 'cc-defs)

> > (defun main ()
> >  (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))

> >  (setq package-user-dir (make-temp-file "cc-miscompile" 'directory))

> >  (package-initialize)
> >  (package-refresh-contents)
> >  (package-install 'd-mode)

> >  (require 'd-mode)

> >  (let ((source (get (intern "c-typedef-decl-kwds" c-lang-constants) 'source)))
> >    (message "Sources: %S" (mapcar 'car source)))

> >  (delete-directory package-user-dir 'recursive))

> > (main)

> > and run it with `emacs -Q --script cc-miscompile.el'.  The output is as
> > follows (package.el output shortened for readility):

> > Contacting host: melpa.org:80
> > Contacting host: elpa.gnu.org:80
> > [?]
> > Sources: (d-mode cc-miscompile cc-langs)

> > Note that `cc-miscompile' ends up in the source list of
> > `c-typedef-decl-kwds', even though it never actually calls any `c-*'
> > functions at all.

OK.  The problem was that CC Mode was using the flag `load-in-progress'
to assume that a CC Mode file was being loaded, for example by a
`require' form inside a compilation of another CC Mode file.  This
assumption breaks down when another file, such as cc-miscompile.el,
while loading, initiates compilation of a CC Mode derivative.

> The byte compilation of d-mode.el is being done during the loading of
> cc-miscompile.el.  This somewhat unusual constellation, I think, is
> causing the problem.  When CC Mode determines the file name to put onto
> a c-lang-defconst's 'source property, it gives priority to the load file
> name, and only when this is nil does it use the byte-compile file name.
> (This is in defsubst c-get-current-file in cc-defs.el).  It would seem
> this is not the correct priority.

> I think swapping the first two arms of the `cond' form in
> c-get-current-file may solve the problem.  It's a bit late to try this
> tonight, I'll try it tomorrow.

No, that wouldn't work.  What I've implemented is when both loading and
byte-compilation are active at the same time, CC Mode walks down the
lisp stack to discover which one of them is actually active.

> > Naturally, CC Mode will later try to load this file, and fail if it is
> > not in the `load-path'.  This effectively breaks installations of D
> > Mode from non-interactive Emacs sessions.

Please try the following patch, which seems to work in the test case you
supplied, in the real situation, and let me know how well it works.  The
d-mode.elc produced can be successfully loaded into the Emacs that built
it.

Thanks for the bug report, and thanks especially for doing so much of the
preliminary detective work.



diff --git a/lisp/progmodes/cc-bytecomp.el b/lisp/progmodes/cc-bytecomp.el
index 1936627..bd2fd34 100644
--- a/lisp/progmodes/cc-bytecomp.el
+++ b/lisp/progmodes/cc-bytecomp.el
@@ -84,13 +84,60 @@
   ;;`(message ,@args)
   )
 
+(defun cc-bytecomp-compiling-or-loading ()
+  ;; Return whether byte-compilation or loading is currently active,
+  ;; returning 'compiling or 'loading or nil.
+  ;; If both are active, the "innermost" activity counts.  Note that
+  ;; compilation can trigger loading (various `require' type forms)
+  ;; and loading can trigger compilation (the package manager does
+  ;; this).  We walk the lisp stack if necessary.
+  (cond
+   ((and load-in-progress
+	 (boundp 'byte-compile-dest-file)
+	 (stringp byte-compile-dest-file))
+    (let ((n 0) elt)
+      (while (and
+	      (setq elt (backtrace-frame n))
+	      (not (and (car elt)
+			(memq (cadr elt)
+			      '(load byte-compile-file
+				     byte-recompile-directory
+				     batch-byte-compile)))))
+	(setq n (1+ n)))
+      (cond
+       ((eq (cadr elt) 'load)
+	'loading)
+       ((memq (cadr elt) '(byte-compile-file
+			   byte-recompile-directory
+			   batch-byte-compile))
+	'compiling)
+       (t				; Can't happen.
+	(message "cc-bytecomp-compiling-or-loading: System flags spuriously set")
+	nil))))
+   (load-in-progress
+    ;; Being loaded.
+    'loading)
+   ((and (boundp 'byte-compile-dest-file)
+	 (stringp byte-compile-dest-file))
+    ;; Being compiled.
+    'compiling)
+   (t
+    ;; Being evaluated interactively.
+    nil)))
+
+(defsubst cc-bytecomp-is-compiling ()
+  "Return non-nil if eval'ed during compilation."
+  (eq (cc-bytecomp-compiling-or-loading) 'compiling))
+
+(defsubst cc-bytecomp-is-loading ()
+  "Return non-nil if eval'ed during loading.
+Nil will be returned if we're in a compilation triggered by the loading."
+  (eq (cc-bytecomp-compiling-or-loading) 'loading))
+
 (defun cc-bytecomp-setup-environment ()
   ;; Eval'ed during compilation to setup variables, functions etc
   ;; declared with `cc-bytecomp-defvar' et al.
-  (if (not load-in-progress)
-      ;; Look at `load-in-progress' to tell whether we're called
-      ;; directly in the file being compiled or just from some file
-      ;; being loaded during compilation.
+  (if (not (cc-bytecomp-is-loading))
       (let (p)
 	(if cc-bytecomp-environment-set
 	    (error "Byte compilation environment already set - \
@@ -138,7 +185,7 @@ perhaps a `cc-bytecomp-restore-environment' is forgotten somewhere"))
 (defun cc-bytecomp-restore-environment ()
   ;; Eval'ed during compilation to restore variables, functions etc
   ;; declared with `cc-bytecomp-defvar' et al.
-  (if (not load-in-progress)
+  (if (not (cc-bytecomp-is-loading))
       (let (p)
 	(setq p cc-bytecomp-unbound-variables)
 	(while p
@@ -282,9 +329,7 @@ use within `eval-when-compile'."
   `(eval-when-compile
      (if (and (fboundp 'cc-bytecomp-is-compiling)
 	      (cc-bytecomp-is-compiling))
-	 (if (or (not load-in-progress)
-		 (not (featurep ,cc-part)))
-	     (cc-bytecomp-load (symbol-name ,cc-part)))
+	     (cc-bytecomp-load (symbol-name ,cc-part))
        (require ,cc-part))))
 
 (defmacro cc-external-require (feature)
@@ -296,12 +341,6 @@ afterwards.  Don't use within `eval-when-compile'."
      (require ,feature)
      (eval-when-compile (cc-bytecomp-setup-environment))))
 
-(defun cc-bytecomp-is-compiling ()
-  "Return non-nil if eval'ed during compilation.  Don't use outside
-`eval-when-compile'."
-  (and (boundp 'byte-compile-dest-file)
-       (stringp byte-compile-dest-file)))
-
 (defmacro cc-bytecomp-defvar (var)
   "Binds the symbol as a variable during compilation of the file,
 to silence the byte compiler.  Don't use within `eval-when-compile'."
@@ -315,8 +354,7 @@ to silence the byte compiler.  Don't use within `eval-when-compile'."
 	      "cc-bytecomp-defvar: Saving %s (as unbound)" ',var)
 	     (setq cc-bytecomp-unbound-variables
 		   (cons ',var cc-bytecomp-unbound-variables))))
-       (if (and (cc-bytecomp-is-compiling)
-		(not load-in-progress))
+       (if (cc-bytecomp-is-compiling)
 	   (progn
 	     (defvar ,var)
 	     (set ',var (intern (concat "cc-bytecomp-ignore-var:"
@@ -344,8 +382,7 @@ at compile time, e.g. for macros and inline functions."
 	     (setq cc-bytecomp-original-functions
 		   (cons (list ',fun nil 'unbound)
 			 cc-bytecomp-original-functions))))
-       (if (and (cc-bytecomp-is-compiling)
-		(not load-in-progress))
+       (if (cc-bytecomp-is-compiling)
 	   (progn
 	     (fset ',fun (intern (concat "cc-bytecomp-ignore-fun:"
 					 (symbol-name ',fun))))
diff --git a/lisp/progmodes/cc-defs.el b/lisp/progmodes/cc-defs.el
index 1d8b8ab..b0e83f3 100644
--- a/lisp/progmodes/cc-defs.el
+++ b/lisp/progmodes/cc-defs.el
@@ -1823,19 +1823,22 @@ system."
 
 (defvar c-lang-const-expansion nil)
 
+;; Ugly hack to pull in the definition of `cc-bytecomp-compiling-or-loading`
+;; from cc-bytecomp to make it available at loadtime.  This is the same
+;; mechanism used in cc-mode.el for `c-populate-syntax-table'.
+(defalias 'cc-bytecomp-compiling-or-loading
+  (cc-eval-when-compile
+    (let ((f (symbol-function 'cc-bytecomp-compiling-or-loading)))
+      (if (byte-code-function-p f) f (byte-compile f)))))
+
 (defsubst c-get-current-file ()
   ;; Return the base name of the current file.
-  (let ((file (cond
-	       (load-in-progress
-		;; Being loaded.
-		load-file-name)
-	       ((and (boundp 'byte-compile-dest-file)
-		     (stringp byte-compile-dest-file))
-		;; Being compiled.
-		byte-compile-dest-file)
-	       (t
-		;; Being evaluated interactively.
-		(buffer-file-name)))))
+  (let* ((c-or-l (cc-bytecomp-compiling-or-loading))
+	 (file
+	  (cond
+	   ((eq c-or-l 'loading) load-file-name)
+	   ((eq c-or-l 'compiling) byte-compile-dest-file)
+	   ((null c-or-l) (buffer-file-name)))))
     (and file
 	 (file-name-sans-extension
 	  (file-name-nondirectory file)))))
@@ -2073,9 +2076,7 @@ quoted."
         (if (or (eq c-lang-const-expansion 'call)
                 (and (not c-lang-const-expansion)
                      (not mode))
-                load-in-progress
-                (not (boundp 'byte-compile-dest-file))
-                (not (stringp byte-compile-dest-file)))
+		(not (cc-bytecomp-is-compiling)))
             ;; Either a straight call is requested in the context, or
             ;; we're in an "uncontrolled" context and got no language,
             ;; or we're not being byte compiled so the compile time
diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
index 68b2d62..22d78b5 100644
--- a/lisp/progmodes/cc-langs.el
+++ b/lisp/progmodes/cc-langs.el
@@ -3252,10 +3252,7 @@ function it returns is byte compiled with all the evaluated results
 from the language constants.  Use the `c-init-language-vars' macro to
 accomplish that conveniently."
 
-  (if (and (not load-in-progress)
-	   (boundp 'byte-compile-dest-file)
-	   (stringp byte-compile-dest-file))
-
+  (if (cc-bytecomp-is-compiling)
       ;; No need to byte compile this lambda since the byte compiler is
       ;; smart enough to detect the `funcall' construct in the
       ;; `c-init-language-vars' macro below and compile it all straight



-- 
Alan Mackenzie (Nuremberg, Germany).

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Sebastian Wiesner]

> Am 30.11.2014 um 19:42 schrieb Alan Mackenzie <acm@muc.de>:
> 
> Hello, again, Sebastian.
> 
> On Fri, Nov 28, 2014 at 10:25:42PM -0000, Alan Mackenzie wrote:
>> Hello, Sebastian.
>> In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
>>> CC Mode tracks wrong source files when a CC Mode derived mode is
>>> installed non-interactively.
> 
>>> To reproduce, save the following code as `cc-miscompile.el'
> 
>>> (require 'package)
>>> (require 'cc-defs)
> 
>>> (defun main ()
>>> (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
> 
>>> (setq package-user-dir (make-temp-file "cc-miscompile" 'directory))
> 
>>> (package-initialize)
>>> (package-refresh-contents)
>>> (package-install 'd-mode)
> 
>>> (require 'd-mode)
> 
>>> (let ((source (get (intern "c-typedef-decl-kwds" c-lang-constants) 'source)))
>>>   (message "Sources: %S" (mapcar 'car source)))
> 
>>> (delete-directory package-user-dir 'recursive))
> 
>>> (main)
> 
>>> and run it with `emacs -Q --script cc-miscompile.el'.  The output is as
>>> follows (package.el output shortened for readility):
> 
>>> Contacting host: melpa.org:80
>>> Contacting host: elpa.gnu.org:80
>>> [?]
>>> Sources: (d-mode cc-miscompile cc-langs)
> 
>>> Note that `cc-miscompile' ends up in the source list of
>>> `c-typedef-decl-kwds', even though it never actually calls any `c-*'
>>> functions at all.
> 
> OK.  The problem was that CC Mode was using the flag `load-in-progress'
> to assume that a CC Mode file was being loaded, for example by a
> `require' form inside a compilation of another CC Mode file.  This
> assumption breaks down when another file, such as cc-miscompile.el,
> while loading, initiates compilation of a CC Mode derivative.
> 
>> The byte compilation of d-mode.el is being done during the loading of
>> cc-miscompile.el.  This somewhat unusual constellation, I think, is
>> causing the problem.  When CC Mode determines the file name to put onto
>> a c-lang-defconst's 'source property, it gives priority to the load file
>> name, and only when this is nil does it use the byte-compile file name.
>> (This is in defsubst c-get-current-file in cc-defs.el).  It would seem
>> this is not the correct priority.
> 
>> I think swapping the first two arms of the `cond' form in
>> c-get-current-file may solve the problem.  It's a bit late to try this
>> tonight, I'll try it tomorrow.
> 
> No, that wouldn't work.  What I've implemented is when both loading and
> byte-compilation are active at the same time, CC Mode walks down the
> lisp stack to discover which one of them is actually active.

Well, if you say… I find this “solution” horrifying, but I am probably just unable to appreciate the full complexity of this issue.

>>> Naturally, CC Mode will later try to load this file, and fail if it is
>>> not in the `load-path'.  This effectively breaks installations of D
>>> Mode from non-interactive Emacs sessions.
> 
> Please try the following patch, which seems to work in the test case you
> supplied, in the real situation, and let me know how well it works.  The
> d-mode.elc produced can be successfully loaded into the Emacs that built
> it.

Do I need to build a patched Emacs to try it?

> diff --git a/lisp/progmodes/cc-bytecomp.el b/lisp/progmodes/cc-bytecomp.el
> index 1936627..bd2fd34 100644
> --- a/lisp/progmodes/cc-bytecomp.el
> +++ b/lisp/progmodes/cc-bytecomp.el
> @@ -84,13 +84,60 @@
>   ;;`(message ,@args)
>   )
> 
> +(defun cc-bytecomp-compiling-or-loading ()
> +  ;; Return whether byte-compilation or loading is currently active,
> +  ;; returning 'compiling or 'loading or nil.
> +  ;; If both are active, the "innermost" activity counts.  Note that
> +  ;; compilation can trigger loading (various `require' type forms)
> +  ;; and loading can trigger compilation (the package manager does
> +  ;; this).  We walk the lisp stack if necessary.
> +  (cond
> +   ((and load-in-progress
> +	 (boundp 'byte-compile-dest-file)
> +	 (stringp byte-compile-dest-file))
> +    (let ((n 0) elt)
> +      (while (and
> +	      (setq elt (backtrace-frame n))
> +	      (not (and (car elt)
> +			(memq (cadr elt)
> +			      '(load byte-compile-file
> +				     byte-recompile-directory
> +				     batch-byte-compile)))))
> +	(setq n (1+ n)))
> +      (cond
> +       ((eq (cadr elt) 'load)
> +	'loading)
> +       ((memq (cadr elt) '(byte-compile-file
> +			   byte-recompile-directory
> +			   batch-byte-compile))
> +	'compiling)
> +       (t				; Can't happen.
> +	(message "cc-bytecomp-compiling-or-loading: System flags spuriously set")
> +	nil))))
> +   (load-in-progress
> +    ;; Being loaded.
> +    'loading)
> +   ((and (boundp 'byte-compile-dest-file)
> +	 (stringp byte-compile-dest-file))
> +    ;; Being compiled.
> +    'compiling)
> +   (t
> +    ;; Being evaluated interactively.
> +    nil)))
> +
> +(defsubst cc-bytecomp-is-compiling ()
> +  "Return non-nil if eval'ed during compilation."
> +  (eq (cc-bytecomp-compiling-or-loading) 'compiling))
> +
> +(defsubst cc-bytecomp-is-loading ()
> +  "Return non-nil if eval'ed during loading.
> +Nil will be returned if we're in a compilation triggered by the loading."
> +  (eq (cc-bytecomp-compiling-or-loading) 'loading))
> +
> (defun cc-bytecomp-setup-environment ()
>   ;; Eval'ed during compilation to setup variables, functions etc
>   ;; declared with `cc-bytecomp-defvar' et al.
> -  (if (not load-in-progress)
> -      ;; Look at `load-in-progress' to tell whether we're called
> -      ;; directly in the file being compiled or just from some file
> -      ;; being loaded during compilation.
> +  (if (not (cc-bytecomp-is-loading))
>       (let (p)
> 	(if cc-bytecomp-environment-set
> 	    (error "Byte compilation environment already set - \
> @@ -138,7 +185,7 @@ perhaps a `cc-bytecomp-restore-environment' is forgotten somewhere"))
> (defun cc-bytecomp-restore-environment ()
>   ;; Eval'ed during compilation to restore variables, functions etc
>   ;; declared with `cc-bytecomp-defvar' et al.
> -  (if (not load-in-progress)
> +  (if (not (cc-bytecomp-is-loading))
>       (let (p)
> 	(setq p cc-bytecomp-unbound-variables)
> 	(while p
> @@ -282,9 +329,7 @@ use within `eval-when-compile'."
>   `(eval-when-compile
>      (if (and (fboundp 'cc-bytecomp-is-compiling)
> 	      (cc-bytecomp-is-compiling))
> -	 (if (or (not load-in-progress)
> -		 (not (featurep ,cc-part)))
> -	     (cc-bytecomp-load (symbol-name ,cc-part)))
> +	     (cc-bytecomp-load (symbol-name ,cc-part))
>        (require ,cc-part))))
> 
> (defmacro cc-external-require (feature)
> @@ -296,12 +341,6 @@ afterwards.  Don't use within `eval-when-compile'."
>      (require ,feature)
>      (eval-when-compile (cc-bytecomp-setup-environment))))
> 
> -(defun cc-bytecomp-is-compiling ()
> -  "Return non-nil if eval'ed during compilation.  Don't use outside
> -`eval-when-compile'."
> -  (and (boundp 'byte-compile-dest-file)
> -       (stringp byte-compile-dest-file)))
> -
> (defmacro cc-bytecomp-defvar (var)
>   "Binds the symbol as a variable during compilation of the file,
> to silence the byte compiler.  Don't use within `eval-when-compile'."
> @@ -315,8 +354,7 @@ to silence the byte compiler.  Don't use within `eval-when-compile'."
> 	      "cc-bytecomp-defvar: Saving %s (as unbound)" ',var)
> 	     (setq cc-bytecomp-unbound-variables
> 		   (cons ',var cc-bytecomp-unbound-variables))))
> -       (if (and (cc-bytecomp-is-compiling)
> -		(not load-in-progress))
> +       (if (cc-bytecomp-is-compiling)
> 	   (progn
> 	     (defvar ,var)
> 	     (set ',var (intern (concat "cc-bytecomp-ignore-var:"
> @@ -344,8 +382,7 @@ at compile time, e.g. for macros and inline functions."
> 	     (setq cc-bytecomp-original-functions
> 		   (cons (list ',fun nil 'unbound)
> 			 cc-bytecomp-original-functions))))
> -       (if (and (cc-bytecomp-is-compiling)
> -		(not load-in-progress))
> +       (if (cc-bytecomp-is-compiling)
> 	   (progn
> 	     (fset ',fun (intern (concat "cc-bytecomp-ignore-fun:"
> 					 (symbol-name ',fun))))
> diff --git a/lisp/progmodes/cc-defs.el b/lisp/progmodes/cc-defs.el
> index 1d8b8ab..b0e83f3 100644
> --- a/lisp/progmodes/cc-defs.el
> +++ b/lisp/progmodes/cc-defs.el
> @@ -1823,19 +1823,22 @@ system."
> 
> (defvar c-lang-const-expansion nil)
> 
> +;; Ugly hack to pull in the definition of `cc-bytecomp-compiling-or-loading`
> +;; from cc-bytecomp to make it available at loadtime.  This is the same
> +;; mechanism used in cc-mode.el for `c-populate-syntax-table'.
> +(defalias 'cc-bytecomp-compiling-or-loading
> +  (cc-eval-when-compile
> +    (let ((f (symbol-function 'cc-bytecomp-compiling-or-loading)))
> +      (if (byte-code-function-p f) f (byte-compile f)))))
> +
> (defsubst c-get-current-file ()
>   ;; Return the base name of the current file.
> -  (let ((file (cond
> -	       (load-in-progress
> -		;; Being loaded.
> -		load-file-name)
> -	       ((and (boundp 'byte-compile-dest-file)
> -		     (stringp byte-compile-dest-file))
> -		;; Being compiled.
> -		byte-compile-dest-file)
> -	       (t
> -		;; Being evaluated interactively.
> -		(buffer-file-name)))))
> +  (let* ((c-or-l (cc-bytecomp-compiling-or-loading))
> +	 (file
> +	  (cond
> +	   ((eq c-or-l 'loading) load-file-name)
> +	   ((eq c-or-l 'compiling) byte-compile-dest-file)
> +	   ((null c-or-l) (buffer-file-name)))))
>     (and file
> 	 (file-name-sans-extension
> 	  (file-name-nondirectory file)))))
> @@ -2073,9 +2076,7 @@ quoted."
>         (if (or (eq c-lang-const-expansion 'call)
>                 (and (not c-lang-const-expansion)
>                      (not mode))
> -                load-in-progress
> -                (not (boundp 'byte-compile-dest-file))
> -                (not (stringp byte-compile-dest-file)))
> +		(not (cc-bytecomp-is-compiling)))
>             ;; Either a straight call is requested in the context, or
>             ;; we're in an "uncontrolled" context and got no language,
>             ;; or we're not being byte compiled so the compile time
> diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
> index 68b2d62..22d78b5 100644
> --- a/lisp/progmodes/cc-langs.el
> +++ b/lisp/progmodes/cc-langs.el
> @@ -3252,10 +3252,7 @@ function it returns is byte compiled with all the evaluated results
> from the language constants.  Use the `c-init-language-vars' macro to
> accomplish that conveniently."
> 
> -  (if (and (not load-in-progress)
> -	   (boundp 'byte-compile-dest-file)
> -	   (stringp byte-compile-dest-file))
> -
> +  (if (cc-bytecomp-is-compiling)
>       ;; No need to byte compile this lambda since the byte compiler is
>       ;; smart enough to detect the `funcall' construct in the
>       ;; `c-init-language-vars' macro below and compile it all straight
> 
> 
> 
> -- 
> Alan Mackenzie (Nuremberg, Germany).

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Alan Mackenzie]

Hello, Sebastian.

On Tue, Dec 02, 2014 at 12:03:21PM +0100, Sebastian Wiesner wrote:

> > Am 30.11.2014 um 19:42 schrieb Alan Mackenzie <acm@muc.de>:

> > Hello, again, Sebastian.

> > On Fri, Nov 28, 2014 at 10:25:42PM -0000, Alan Mackenzie wrote:
> >> Hello, Sebastian.
> >> In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
> >>> CC Mode tracks wrong source files when a CC Mode derived mode is
> >>> installed non-interactively.

[ .... ]

> > OK.  The problem was that CC Mode was using the flag
> > `load-in-progress' to assume that a CC Mode file was being loaded,
> > for example by a `require' form inside a compilation of another CC
> > Mode file.  This assumption breaks down when another file, such as
> > cc-miscompile.el, while loading, initiates compilation of a CC Mode
> > derivative.

[ .... ]

> > No, that wouldn't work.  What I've implemented is when both loading
> > and byte-compilation are active at the same time, CC Mode walks down
> > the lisp stack to discover which one of them is actually active.

> Well, if you say… I find this “solution” horrifying, but I am probably
> just unable to appreciate the full complexity of this issue.

It is horrifying.  I spent quite some time over the weekend searching for
a better solution, without success.  The issue is quite simple: when
loading and compiling are nested in some unknown order, how do you
determine which of them is the "innermost" activity?

But, then again, maybe I could bind the "compilation flags" to nil around
each invocation of `load' inside the CC Mode compilation system.  Yes,
I'll give that a try.

> > Please try the following patch, which seems to work in the test case you
> > supplied, in the real situation, and let me know how well it works.  The
> > d-mode.elc produced can be successfully loaded into the Emacs that built
> > it.

> Do I need to build a patched Emacs to try it?

You need to rebuild CC Mode, but not necessarily the Emacs core.  This
should do the job:

    emacs -Q -batch -f batch-byte-compile path/to/lisp/progmodes/cc-*.el

, or you could just do a `make' from the main Emacs directory, possibly
deleting cc-*.elc first, just to make sure.

But feel free to wait until I've tried the alternative, better, approach
above.  I'll get back to you later.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Stefan Monnier]

> It is horrifying.  I spent quite some time over the weekend searching for
> a better solution, without success.  The issue is quite simple: when
> loading and compiling are nested in some unknown order, how do you
> determine which of them is the "innermost" activity?

No, that is a side issue.  The real issue is: why on earth do we care
about the file name?

We should start by saying that we should never need to answer this
question.  So we should reject whichever "solution" we chose to some
other problem which ends up requiring such horrors.


        Stefan

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Alan Mackenzie]

Hello, Sebastian.

On Tue, Dec 02, 2014 at 12:02:26PM +0000, Alan Mackenzie wrote:
> On Tue, Dec 02, 2014 at 12:03:21PM +0100, Sebastian Wiesner wrote:

> > > Am 30.11.2014 um 19:42 schrieb Alan Mackenzie <acm@muc.de>:

> > > On Fri, Nov 28, 2014 at 10:25:42PM -0000, Alan Mackenzie wrote:
> > >> In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
> > >>> CC Mode tracks wrong source files when a CC Mode derived mode is
> > >>> installed non-interactively.

> [ .... ]

> > > OK.  The problem was that CC Mode was using the flag
> > > `load-in-progress' to assume that a CC Mode file was being loaded,
> > > for example by a `require' form inside a compilation of another CC
> > > Mode file.  This assumption breaks down when another file, such as
> > > cc-miscompile.el, while loading, initiates compilation of a CC Mode
> > > derivative.

> [ .... ]

> > > No, that wouldn't work.  What I've implemented is when both loading
> > > and byte-compilation are active at the same time, CC Mode walks down
> > > the lisp stack to discover which one of them is actually active.

> > Well, if you say… I find this “solution” horrifying, but I am probably
> > just unable to appreciate the full complexity of this issue.

> It is horrifying.  I spent quite some time over the weekend searching for
> a better solution, without success.  The issue is quite simple: when
> loading and compiling are nested in some unknown order, how do you
> determine which of them is the "innermost" activity?

> But, then again, maybe I could bind the "compilation flags" to nil around
> each invocation of `load' inside the CC Mode compilation system.  Yes,
> I'll give that a try.

I gave it a try, a hard and lengthy try, but in the end it wouldn't work.
So, back to the previous scheme.

Please disregard, revert, or discard the previous patch I sent.

Please try out the following patch in the real situation and let me know
whether it works properly.  The patch is to the Emacs master branch.
After applying it, please first explicitly DELETE cc-*.elc (this is
important), and then rebuild them, either with the make command, or
something like:

    emacs -Q -batch -f batch-byte-compile ..../lisp/progmodes/cc-*.el

With this patch I was able to generate d-mode.elc using cc-miscompile.el,
then to load and run it in the same emacs version which generated it.



diff --git a/lisp/progmodes/cc-bytecomp.el b/lisp/progmodes/cc-bytecomp.el
index 2db5a10..926fc98 100644
--- a/lisp/progmodes/cc-bytecomp.el
+++ b/lisp/progmodes/cc-bytecomp.el
@@ -89,13 +89,60 @@
   ;;`(message ,@args)
   )
 
+(defun cc-bytecomp-compiling-or-loading ()
+  ;; Determine whether byte-compilation or loading is currently active,
+  ;; returning 'compiling, 'loading or nil.
+  ;; If both are active, the "innermost" activity counts.  Note that
+  ;; compilation can trigger loading (various `require' type forms)
+  ;; and loading can trigger compilation (the package manager does
+  ;; this).  We walk the lisp stack if necessary.
+  (cond
+   ((and load-in-progress
+	 (boundp 'byte-compile-dest-file)
+	 (stringp byte-compile-dest-file))
+    (let ((n 0) elt)
+      (while (and
+	      (setq elt (backtrace-frame n))
+	      (not (and (car elt)
+			(memq (cadr elt)
+			      '(load require
+				byte-compile-file byte-recompile-directory
+				batch-byte-compile)))))
+	(setq n (1+ n)))
+      (cond
+       ((memq (cadr elt) '(load require))
+	'loading)
+       ((memq (cadr elt) '(byte-compile-file
+			   byte-recompile-directory
+			   batch-byte-compile))
+	'compiling)
+       (t				; Can't happen.
+	(message "cc-bytecomp-compiling-or-loading: System flags spuriously set")
+	nil))))
+   (load-in-progress
+    ;; Being loaded.
+    'loading)
+   ((and (boundp 'byte-compile-dest-file)
+	 (stringp byte-compile-dest-file))
+    ;; Being compiled.
+    'compiling)
+   (t
+    ;; Being evaluated interactively.
+    nil)))
+
+(defsubst cc-bytecomp-is-compiling ()
+  "Return non-nil if eval'ed during compilation."
+  (eq (cc-bytecomp-compiling-or-loading) 'compiling))
+
+(defsubst cc-bytecomp-is-loading ()
+  "Return non-nil if eval'ed during loading.
+Nil will be returned if we're in a compilation triggered by the loading."
+  (eq (cc-bytecomp-compiling-or-loading) 'loading))
+
 (defun cc-bytecomp-setup-environment ()
   ;; Eval'ed during compilation to setup variables, functions etc
   ;; declared with `cc-bytecomp-defvar' et al.
-  (if (not load-in-progress)
-      ;; Look at `load-in-progress' to tell whether we're called
-      ;; directly in the file being compiled or just from some file
-      ;; being loaded during compilation.
+  (if (not (cc-bytecomp-is-loading))
       (let (p)
 	(if cc-bytecomp-environment-set
 	    (error "Byte compilation environment already set - \
@@ -143,7 +190,7 @@ perhaps a `cc-bytecomp-restore-environment' is forgotten somewhere"))
 (defun cc-bytecomp-restore-environment ()
   ;; Eval'ed during compilation to restore variables, functions etc
   ;; declared with `cc-bytecomp-defvar' et al.
-  (if (not load-in-progress)
+  (if (not (cc-bytecomp-is-loading))
       (let (p)
 	(setq p cc-bytecomp-unbound-variables)
 	(while p
@@ -287,8 +334,7 @@ use within `eval-when-compile'."
   `(eval-when-compile
      (if (and (fboundp 'cc-bytecomp-is-compiling)
 	      (cc-bytecomp-is-compiling))
-	 (if (or (not load-in-progress)
-		 (not (featurep ,cc-part)))
+	 (if (not (featurep ,cc-part))
 	     (cc-bytecomp-load (symbol-name ,cc-part)))
        (require ,cc-part))))
 
@@ -301,12 +347,6 @@ afterwards.  Don't use within `eval-when-compile'."
      (require ,feature)
      (eval-when-compile (cc-bytecomp-setup-environment))))
 
-(defun cc-bytecomp-is-compiling ()
-  "Return non-nil if eval'ed during compilation.  Don't use outside
-`eval-when-compile'."
-  (and (boundp 'byte-compile-dest-file)
-       (stringp byte-compile-dest-file)))
-
 (defmacro cc-bytecomp-defvar (var)
   "Binds the symbol as a variable during compilation of the file,
 to silence the byte compiler.  Don't use within `eval-when-compile'."
@@ -320,8 +360,7 @@ to silence the byte compiler.  Don't use within `eval-when-compile'."
 	      "cc-bytecomp-defvar: Saving %s (as unbound)" ',var)
 	     (setq cc-bytecomp-unbound-variables
 		   (cons ',var cc-bytecomp-unbound-variables))))
-       (if (and (cc-bytecomp-is-compiling)
-		(not load-in-progress))
+       (if (cc-bytecomp-is-compiling)
 	   (progn
 	     (defvar ,var)
 	     (set ',var (intern (concat "cc-bytecomp-ignore-var:"
@@ -349,8 +388,7 @@ at compile time, e.g. for macros and inline functions."
 	     (setq cc-bytecomp-original-functions
 		   (cons (list ',fun nil 'unbound)
 			 cc-bytecomp-original-functions))))
-       (if (and (cc-bytecomp-is-compiling)
-		(not load-in-progress))
+       (if (cc-bytecomp-is-compiling)
 	   (progn
 	     (fset ',fun (intern (concat "cc-bytecomp-ignore-fun:"
 					 (symbol-name ',fun))))
diff --git a/lisp/progmodes/cc-defs.el b/lisp/progmodes/cc-defs.el
index 46cb2f9..f92267e 100644
--- a/lisp/progmodes/cc-defs.el
+++ b/lisp/progmodes/cc-defs.el
@@ -1983,19 +1983,22 @@ system."
 
 (defvar c-lang-const-expansion nil)
 
+;; Ugly hack to pull in the definition of `cc-bytecomp-compiling-or-loading`
+;; from cc-bytecomp to make it available at loadtime.  This is the same
+;; mechanism used in cc-mode.el for `c-populate-syntax-table'.
+(defalias 'cc-bytecomp-compiling-or-loading
+  (cc-eval-when-compile
+    (let ((f (symbol-function 'cc-bytecomp-compiling-or-loading)))
+      (if (byte-code-function-p f) f (byte-compile f)))))
+
 (defsubst c-get-current-file ()
   ;; Return the base name of the current file.
-  (let ((file (cond
-	       (load-in-progress
-		;; Being loaded.
-		load-file-name)
-	       ((and (boundp 'byte-compile-dest-file)
-		     (stringp byte-compile-dest-file))
-		;; Being compiled.
-		byte-compile-dest-file)
-	       (t
-		;; Being evaluated interactively.
-		(buffer-file-name)))))
+  (let* ((c-or-l (cc-bytecomp-compiling-or-loading))
+	 (file
+	  (cond
+	   ((eq c-or-l 'loading) load-file-name)
+	   ((eq c-or-l 'compiling) byte-compile-dest-file)
+	   ((null c-or-l) (buffer-file-name)))))
     (and file
 	 (file-name-sans-extension
 	  (file-name-nondirectory file)))))
@@ -2062,6 +2065,9 @@ constant.  A file is identified by its base name."
 	 ;; language constant source definitions.)
 	 (c-lang-const-expansion 'call)
 	 (c-langs-are-parametric t)
+	 (file (intern
+		(or (c-get-current-file)
+		    (error "`c-lang-defconst' can only be used in a file"))))
 	 bindings
 	 pre-files)
 
@@ -2121,9 +2127,14 @@ constant.  A file is identified by its base name."
     ;; definitions for this symbol, to make sure the order in the
     ;; `source' property is correct even when files are loaded out of
     ;; order.
-    (setq pre-files (nreverse
-		     ;; Reverse to get the right load order.
-		     (mapcar 'car (get sym 'source))))
+    (setq pre-files (mapcar 'car (get sym 'source)))
+    (if (memq file pre-files)
+	;; This can happen when the source file (e.g. cc-langs.el) is first
+	;; loaded as source, setting a 'source property entry, and then itself
+	;; being compiled.
+	(setq pre-files (cdr (memq file pre-files))))
+    ;; Reverse to get the right load order.
+    (setq pre-files (nreverse pre-files))
 
     `(eval-and-compile
        (c-define-lang-constant ',name ,bindings
@@ -2233,9 +2244,7 @@ quoted."
         (if (or (eq c-lang-const-expansion 'call)
                 (and (not c-lang-const-expansion)
                      (not mode))
-                load-in-progress
-                (not (boundp 'byte-compile-dest-file))
-                (not (stringp byte-compile-dest-file)))
+		(not (cc-bytecomp-is-compiling)))
             ;; Either a straight call is requested in the context, or
             ;; we're in an "uncontrolled" context and got no language,
             ;; or we're not being byte compiled so the compile time
diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
index 375725e..61e9239 100644
--- a/lisp/progmodes/cc-langs.el
+++ b/lisp/progmodes/cc-langs.el
@@ -3260,10 +3260,7 @@ function it returns is byte compiled with all the evaluated results
 from the language constants.  Use the `c-init-language-vars' macro to
 accomplish that conveniently."
 
-  (if (and (not load-in-progress)
-	   (boundp 'byte-compile-dest-file)
-	   (stringp byte-compile-dest-file))
-
+  (if (cc-bytecomp-is-compiling)
       ;; No need to byte compile this lambda since the byte compiler is
       ;; smart enough to detect the `funcall' construct in the
       ;; `c-init-language-vars' macro below and compile it all straight


-- 
Alan Mackenzie (Nuremberg, Germany).

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Stefan Monnier]

> +(defun cc-bytecomp-compiling-or-loading ()
> +  ;; Determine whether byte-compilation or loading is currently active,
> +  ;; returning 'compiling, 'loading or nil.
> +  ;; If both are active, the "innermost" activity counts.  Note that
> +  ;; compilation can trigger loading (various `require' type forms)
> +  ;; and loading can trigger compilation (the package manager does
> +  ;; this).  We walk the lisp stack if necessary.

I think this deserves a longish and complete explanation of the context,
i.e. *why* do we need to know the file name, in which circumstances it's
used and why other solutions to those problems are worse than this
hideous hack,


        Stefan

bug#19206: 25.0.50; CC Mode tracks wrong source files

[Alan Mackenzie]

In article <mailman.14863.1417170074.1147.bug-gnu-emacs@gnu.org> you wrote:
> CC Mode tracks wrong source files when a CC Mode derived mode is
> installed non-interactively.

Bug fixed.

-- 
Alan Mackenzie (Nuremberg, Germany).