CC Mode: File mode specification error: (void-variable c-font-lock-keywords-3)

CC Mode: File mode specification error: (void-variable c-font-lock-keywords-3)

[Alan Mackenzie]

Hi, Emacs and CC Mode!

Start Emacs 22 with -Q, then visit a file.c.

Emacs displays the message:

  File mode specification error: (void-variable c-font-lock-keywords-3)

.  Probably that variable needs an autoload on it, or something like
that.  Please consider the bug as noted by CC Mode, and that it will be
fixed in the next (minor) release.

-- 
Alan Mackenzie (Munich, Germany)

Problem mit symlinks, locate-library and load-history [Was: <Something else>]

[Alan Mackenzie]

Hi, Emacs!

On Tue, 14 Mar 2006, Alan Mackenzie wrote in an article with Subject: "CC
Mode: File mode specification error: (void-variable
c-font-lock-keywords-3)":

>Hi, Emacs and CC Mode!
>
>Start Emacs 22 with -Q, then visit a file.c.
>
>Emacs displays the message:
>
>  File mode specification error: (void-variable c-font-lock-keywords-3)
>
>.  Probably that variable needs an autoload on it, or something like
>that.  Please consider the bug as noted by CC Mode, and that it will be
>fixed in the next (minor) release.

This problem happens with the form

    (eval-after-load "font-lock" '(require cc-fonts))

in cc-mode.el.  eval-after-load calls (locate-library "font-lock"), which
returns

    "/home/acm/emacs/emacs/lisp/font-lock.elc"

, which it then tries, with (assoc (locate-library file) load-history), to
match against

    "/mnt/hda7/emacs/lisp/font-lock.el"

, and fails.  There are two distinct problems here:

(i) locate-library gives a .elc, but there's a .el in load-history.

(ii) /home/acm/emacs is actually a symbolic link pointing at /mnt/hda7.

My reactions:

(i) The dumped lisp files are byte compiled, so it seems strange indeed
that font-lock.el is record in load-history rather than font-lock.elc.
Is this a bug?

(ii) Why on earth is eval-after-load converting "font-lock" to a full
filename and then searching for that?  Surely it is sufficient that any
old font-lock has been loaded at some time (e.g., at dump time)?

On the other hand, if absolute pathnames are to be used, shouldn't they
first be filtered with file-truename, like this:

*** subr.el	Sun Feb 19 12:50:45 2006
--- subr.acm.el	Sun Mar 19 22:00:15 2006
***************
*** 1378,1384 ****
  	    ;; Emacs for the case that FILE is one of them.
  	    ;; (load-symbol-file-load-history)
  	    (when (locate-library file)
! 	      (assoc (locate-library file) load-history)))
  	  (eval form))))
    form)
  
--- 1378,1384 ----
  	    ;; Emacs for the case that FILE is one of them.
  	    ;; (load-symbol-file-load-history)
  	    (when (locate-library file)
! 	      (assoc (file-truename (locate-library file)) load-history)))
  	  (eval form))))
    form)
  
There are approximately 45 places in ..../lisp where locate-library is
used, and approximately none of them use file-truename.  3 of these are
assoc'king the result with load-history.
 
-- 
Alan Mackenzie (Munich, Germany)

Re: Problem mit symlinks, locate-library and load-history [Was: <Something else>]

[Richard Stallman]

    , and fails.  There are two distinct problems here:

    (i) locate-library gives a .elc, but there's a .el in load-history.

I guess eval-after-load needs to convert .elc to .el.

    (ii) /home/acm/emacs is actually a symbolic link pointing at /mnt/hda7.

I guess eval-after-load needs to call file-chase-links on one or both
of the names.

    (i) The dumped lisp files are byte compiled, so it seems strange indeed
    that font-lock.el is record in load-history rather than font-lock.elc.
    Is this a bug?

The load history is designed to refer to the source files,
and therefore treats a byte compiled file as if it were the source file.

    (ii) Why on earth is eval-after-load converting "font-lock" to a full
    filename and then searching for that?  Surely it is sufficient that any
    old font-lock has been loaded at some time (e.g., at dump time)?

Not necessarily.  There can be multiple files called font-lock.el in
different directories.  We would not want to have such a coincidence
in Emacs itself, but nothing stops users from having them.

    On the other hand, if absolute pathnames are to be used, shouldn't they
    first be filtered with file-truename, like this:

Yes.

    There are approximately 45 places in ..../lisp where locate-library is
    used, and approximately none of them use file-truename.  3 of these are
    assoc'king the result with load-history.

Which three are they?  Where are they?  It sounds like they may all
have a bug.  It could be useful to define a subroutine to do
 	      (assoc (file-truename (locate-library file)) load-history)))
except do it correctly.

Would someone like to do that, and ack?

Re: Problem mit symlinks, locate-library and load-history

[Alan Mackenzie]

Hi, Richard!

[ .... ]

>I guess eval-after-load needs to call file-chase-links on one or both of
>the names.
>
>    (i) The dumped lisp files are byte compiled, so it seems strange
>    indeed that font-lock.el is record in load-history rather than
>    font-lock.elc.  Is this a bug?

>The load history is designed to refer to the source files, and therefore
>treats a byte compiled file as if it were the source file.

After playing around a bit with C-h v and load-file, the top element of
my load-history has this file name:

    "/home/acm/emacs/emacs/lisp/loadhist.elc"

This is inconsistent with the entry for a preloaded lisp file:

    "/mnt/hda7/emacs/lisp/font-lock.el"

[Note:  these files are in the same directory, "/home/acm/emacs" being a
symbolic link pointing at "/mnt/hda7".]

This inconsistency is surely a bug.  The prime cause of this is, IMAO,
the doc string of load-history - it fails to state whether "file.elc" or
"file.el" should be recorded, and also fails to state that a truename,
rather than a name with a symbolic link, is recorded - thus people using
load-history have become confused.

>    (ii) Why on earth is eval-after-load converting "font-lock" to a
>    full filename and then searching for that?  Surely it is sufficient
>    that any old font-lock has been loaded at some time (e.g., at dump
>    time)?

>Not necessarily.  There can be multiple files called font-lock.el in
>different directories.  We would not want to have such a coincidence
>in Emacs itself, but nothing stops users from having them.

Ah.  OK.

>    On the other hand, if absolute pathnames are to be used, shouldn't
>    they first be filtered with file-truename, like this:

>Yes.

#########################################################################
There's another bug in eval-after-load:

Suppose that ..../emacs/lisp/foobar.el exists, but not foobar.elc.

In Emacs 22:
M-x load-file <ret> ..../emacs/lisp/foobar.el
M-x byte-compile-file <ret> ..../emacs/lisp/foobar.el
M-: (eval-after-load "foobar" '(<something>))

The eval-after-load form should trigger immediately.  It doesn't, though,
because locate-library finds "..../emacs/lisp/foobar.elc", but
load-history (correctly) contains "..../emacs/lisp/foobar.el".

Shouldn't eval-after-load strip the ".elc", then loop through
load-suffixes?  (BTW, top marks to the person who decided not to call
this variable "load-suffices".  ;-)

#########################################################################
How about this as a resolution of the problem:

(i) load-history should record the absolute file-truename of the file which
was actually loaded, whether that be ".../directory/file.el",
".../directory/file.elc" or even ".../directory/file".  The files loaded at
dump-time would thus be recorded as .elc, not .el as at present.

(ii) locate-library and friends should remain as they are, returning the
path as recorded in load-list, even if this might contain symbolic links.

(iii) eval-after-load, and suchlike: they should work like my proposed
function loaded-filename (see below).

>    There are approximately 45 places in ..../lisp where locate-library
>    is used, and approximately none of them use file-truename.  3 of
>    these are assoc'king the result with load-history.

>Which three are they?  Where are they?

cus-edit.el L2159
info-xref.el L286
subr.el L1381 (that's eval-after-load)

>It sounds like they may all >have a bug.  It could be useful to define a
>subroutine to do
> 	      (assoc (file-truename (locate-library file)) load-history)))
>except do it correctly.

How about this?

    (defun loaded-filename (filename &optional exact-suffix)
      "Get the absolute file name from which Elisp file FILENAME was loaded.
    Return nil if the file isn't found.  The returned file name will be a
    true name \(i.e. without symbolic links).

    FILENAME is a string.  It may be an absolute file name like
    \"/usr/local/bin/emacs/lisp/foobar\", or a bare file name like
    \"foobar\" or \"foobar.el\".  In the latter case, it will be found only
    if it is in a directory in `load-list'.

    If EXACT-SUFFIX is non-nil, the function searches only for FILENAME.
    Otherwise it also searches for the file names formed by replacing
    FILENAME's suffix, if any, by each of the suffixes in `load-suffixes'.
    ............)

>Would someone like to do that, and ack?

This email is NOT such an ACK.  ;-)

-- 
Alan Mackenzie.

Re: Problem mit symlinks, locate-library and load-history

[Stefan Monnier]

> After playing around a bit with C-h v and load-file, the top element of
> my load-history has this file name:

>     "/home/acm/emacs/emacs/lisp/loadhist.elc"

> This is inconsistent with the entry for a preloaded lisp file:

>     "/mnt/hda7/emacs/lisp/font-lock.el"

> [Note:  these files are in the same directory, "/home/acm/emacs" being a
> symbolic link pointing at "/mnt/hda7".]

> This inconsistency is surely a bug.

It all depends on how we define load-history and its uses.  Basically, we
need to find the file-truename at some point, but I don't think it matters
much whether it's done before putting the filename on load-history or after.
So we should just do it whichever way is more convenient.


        Stefan

Re: Problem mit symlinks, locate-library and load-history [Was: <Something else>]

[Alan Mackenzie]

Hi, Richard!

On Mon, 20 Mar 2006, Richard Stallman wrote:

[ This was in the context of eval-after-load not finding an already
loaded file, because of confusion between suffixes ".el" and ".elc" and
confusion between symbolic links and true names. ] 

>It could be useful to define a subroutine to do
> 	      (assoc (file-truename (locate-library file)) load-history)))
>except do it correctly.

>Would someone like to do that, and ack?

Thinking about this a little more, there is confusion in the above,
certainly when used in eval-after-load:  There is NO REASON why a loaded
file should have to be in load-path, and it seems that (locate-library
file) was used mainly to give a convenient search key for `alloc'.

I therefore propose the following function loaded-filename which does
nothing but search load-history for a file.  It can be called like
this:

(loaded-filename "font-lock.el") => "/home/acm/emacs/emacs/lisp/font-lock.elc"
  (any suffix in `load-suffixes' will do in place of the .el).

(loaded-filename "font-lock") => "/home/acm/emacs/emacs/lisp/font-lock.elc"
  (There needn't be a suffix on the name).

(loaded-filename "font-lock.el" t) => nil
  ("font-lock.el", as such, has not been loaded)

(loaded-filename "~/emacs/emacs/lisp/font-lock") =>
"/home/acm/emacs/emacs/lisp/font-lock.elc"

The functions work fine in GNU.  I'm not sure about MS-Windows (case
folding?) or VMS (version numbers?).

#########################################################################
(defun load-history-filename-element (filename &optional exact-suffix)
  "Get the load-history element which matches the loaded Elisp file FILENAME.
Return nil if the file isn't found.  The parameters have the same meaning as
those of the function `loaded-filename'."
  (let* (supplied-basename-sans-ext
	 supplied-extension		; e.g. ".el" or "", NOT "el".
	 regexp
	 (loads load-history)
	 (load-elt (and loads (car loads))))

    (if (file-name-absolute-p filename)
	(progn
	  (setq filename (file-truename filename))
	  (setq supplied-basename-sans-ext
		(file-name-sans-extension (file-name-nondirectory filename))))
      (if (file-name-directory filename)
	  (error "loaded-filename: Relative file name has directory parts: %s" filename))
      (setq supplied-basename-sans-ext (file-name-sans-extension filename)))
    (setq supplied-extension
	  (if (file-name-extension filename)
	      (concat "." (file-name-extension filename))
	    ""))
    (setq regexp
	  (concat (if (file-name-absolute-p filename) "^" "")
		  (if exact-suffix
		      (regexp-quote filename)
		    (concat supplied-basename-sans-ext
			    (regexp-opt (cons supplied-extension load-suffixes))))
		  "$"))

    (while (and loads
		(or (null (car load-elt))
		    (not (string-match regexp (car load-elt)))
		    (not (string= supplied-basename-sans-ext
				  (file-name-sans-extension
				   (file-name-nondirectory (car load-elt)))))))
      (setq loads (cdr loads)
	    load-elt (and loads (car loads))))
    load-elt))

(defun loaded-filename (filename &optional exact-suffix)
  "Get the absolute file name from which Elisp file FILENAME was loaded.
Return nil if the file isn't found.  The returned file name will be a true
name \(i.e. with all its symbolic links having been chased out).

FILENAME is a string.  It may be an absolute file name like
\"/usr/local/bin/emacs/lisp/foobar\", or a bare file name like \"foobar\" or
\"foobar.el\".  It may not be a relative name containing a \"/\".

If EXACT-SUFFIX is non-nil, the function searches only for FILENAME.
Otherwise it also searches for the file names formed by replacing FILENAME's
suffix, if any, by each of the suffixes in `load-suffixes'."
  (let ((elt (load-history-filename-element filename exact-suffix)))
    (and elt (car elt))))
#########################################################################

-- 
Alan Mackenzie (Munich, Germany)

Re: Problem mit symlinks, locate-library and load-history [Was: <Something else>]

[Richard Stallman]

Your new function seems like a good idea.  For full reliability, the
regexps should use \\` and \\' rather than ^ and $.  And I think it
would be good for load-history-filename-element to use
save-match-data.

Re: Problem mit symlinks, locate-library and load-history [PATCH]

[Alan Mackenzie]

Hi, Richard!

On Mon, 27 Mar 2006, Richard Stallman wrote:

>Your new function seems like a good idea.  For full reliability, the
>regexps should use \\` and \\' rather than ^ and $.  And I think it
>would be good for load-history-filename-element to use
>save-match-data.
>

I updated my Emacs copy yesterday.  It is in ~acm/emacs/emacs, where
~acm/emacs is a symbolic link to /mnt/hda7.

I do: emacs-22.0.50.1 -Q
      M-x load-file <ret> ~acm/emacs/emacs/lisp/progmodes/cc-mode.elc
      C-h v load-history.

Then:
      M-: (eval-after-load "cc-mode" '(beep))  fails.
      M-: (eval-after-load "cc-fonts" '(beep))  beeps.
      M-: (eval-after-load "english" '(beep))  fails.

Here are these file names as they appear in load-history:

1. "/home/acm/emacs/emacs/lisp/progmodes/cc-mode.elc"
2. "/mnt/hda7/emacs/lisp/progmodes/cc-fonts.elc"
3. "/mnt/hda7/emacs/lisp/language/english.elc"

1 and 2 are incompatible.  3 is plain wrong - the actual file is
english.el (english.elc doesn't exist; you can't compile english).

The reason this happens is that in Fload (lread.c):
(i) "language/english" is passed to openp, which returns the full file
  name, including the extension ".el".

(ii) At preload time (`purify-flag'), the extension (".el" or ".elc")
  is stripped from the filename before it's either:
  o - (".elc") loaded; or
  o - (".el") passed to load-with-code-conversion
    (.../international/mule.el).

(iii) At this stage I get too confused to delve further.  The
  (preloaded) file names (relative file names based on ..../emacs/lisp)
  get stored into load-history.  Some of these names have the extension
  ".el", some have no extension, but I don't think any have ".elc".

(iv) When Emacs is started, command-line (startup.el) attempts to restore
  the extensions.  ".el" is left as is, otherwise ".elc" is appended.

This is chaotic - here is a fix:

#########################################################################

The patch here rationalizes load-history.  It (probably) isn't enough on
its own to fix the eval-after-load bugs.  For this I will also need the
function load-history-filename-element and so on, which I proposed on Wed
22 March, and you (RMS) tentatively approved on Mon, 27 Mar 2006 03:36:26
-0500.

With this patch, the names in load-history now look like this:   

1. "/mnt/hda7/emacs/lisp/progmodes/cc-mode.elc"
2. "/mnt/hda7/emacs/lisp/progmodes/cc-fonts.elc"
3. "/mnt/hda7/emacs/lisp/language/english.el"


2006-05-10  Alan Mackenzie  <acm@muc.de>

	* lread.c (Vload_history): Enhance doc-string to say that the file
	name is the absolute truename of the loaded file.

	* lread.c (readevalloop): Call file-truename on the name for
	load-history, except at preloading time.

	* lread.c (Fload): At preloading time, preserve the extension of
	the filename which goes into load-history.  New variable
	hist_file_name.


Index: src/lread.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/lread.c,v
retrieving revision 1.350
diff -c -r1.350 lread.c
*** src/lread.c	27 Feb 2006 02:04:35 -0000	1.350
--- src/lread.c	10 May 2006 10:50:59 -0000
***************
*** 718,725 ****
    register int fd = -1;
    int count = SPECPDL_INDEX ();
    Lisp_Object temp;
!   struct gcpro gcpro1, gcpro2;
!   Lisp_Object found, efound;
    /* 1 means we printed the ".el is newer" message.  */
    int newer = 0;
    /* 1 means we are loading a compiled file.  */
--- 718,725 ----
    register int fd = -1;
    int count = SPECPDL_INDEX ();
    Lisp_Object temp;
!   struct gcpro gcpro1, gcpro2, gcpro3;
!   Lisp_Object found, efound, hist_file_name;
    /* 1 means we printed the ".el is newer" message.  */
    int newer = 0;
    /* 1 means we are loading a compiled file.  */
***************
*** 727,732 ****
--- 727,733 ----
    Lisp_Object handler;
    int safe_p = 1;
    char *fmode = "r";
+   Lisp_Object tmp[2];
  #ifdef DOS_NT
    fmode = "rt";
  #endif /* DOS_NT */
***************
*** 743,749 ****
       the need to gcpro noerror, nomessage and nosuffix.
       (Below here, we care only whether they are nil or not.)
       The presence of this call is the result of a historical accident:
!      it used to be in every file-operations and when it got removed
       everywhere, it accidentally stayed here.  Since then, enough people
       supposedly have things like (load "$PROJECT/foo.el") in their .emacs
       that it seemed risky to remove.  */
--- 744,750 ----
       the need to gcpro noerror, nomessage and nosuffix.
       (Below here, we care only whether they are nil or not.)
       The presence of this call is the result of a historical accident:
!      it used to be in every file-operation and when it got removed
       everywhere, it accidentally stayed here.  Since then, enough people
       supposedly have things like (load "$PROJECT/foo.el") in their .emacs
       that it seemed risky to remove.  */
***************
*** 763,769 ****
    if (SCHARS (file) > 0)
      {
        int size = SBYTES (file);
-       Lisp_Object tmp[2];
  
        found = Qnil;
        GCPRO2 (file, found);
--- 764,769 ----
***************
*** 847,852 ****
--- 847,859 ----
      Vloads_in_progress = Fcons (found, Vloads_in_progress);
    }
  
+   /* Get the name for load-history. */
+   hist_file_name = (! NILP (Vpurify_flag)
+                     ? Fconcat (2, (tmp[0] = Ffile_name_directory (file),
+                                    tmp[1] = Ffile_name_nondirectory (found),
+                                    tmp))
+                     : found) ;
+ 
    if (!bcmp (SDATA (found) + SBYTES (found) - 4,
  	     ".elc", 4))
      /* Load .elc files directly, but not when they are
***************
*** 857,863 ****
  	  struct stat s1, s2;
  	  int result;
  
! 	  GCPRO2 (file, found);
  
  	  if (!safe_to_load_p (fd))
  	    {
--- 864,870 ----
  	  struct stat s1, s2;
  	  int result;
  
! 	  GCPRO3 (file, found, hist_file_name);
  
  	  if (!safe_to_load_p (fd))
  	    {
***************
*** 911,924 ****
  
  	  if (fd >= 0)
  	    emacs_close (fd);
! 	  val = call4 (Vload_source_file_function, found, file,
  		       NILP (noerror) ? Qnil : Qt,
  		       NILP (nomessage) ? Qnil : Qt);
  	  return unbind_to (count, val);
  	}
      }
  
!   GCPRO2 (file, found);
  
  #ifdef WINDOWSNT
    emacs_close (fd);
--- 918,931 ----
  
  	  if (fd >= 0)
  	    emacs_close (fd);
! 	  val = call4 (Vload_source_file_function, found, hist_file_name,
  		       NILP (noerror) ? Qnil : Qt,
  		       NILP (nomessage) ? Qnil : Qt);
  	  return unbind_to (count, val);
  	}
      }
  
!   GCPRO3 (file, found, hist_file_name);
  
  #ifdef WINDOWSNT
    emacs_close (fd);
***************
*** 957,963 ****
    load_descriptor_list
      = Fcons (make_number (fileno (stream)), load_descriptor_list);
    load_in_progress++;
!   readevalloop (Qget_file_char, stream, (! NILP (Vpurify_flag) ? file : found),
  		Feval, 0, Qnil, Qnil, Qnil, Qnil);
    unbind_to (count, Qnil);
  
--- 964,970 ----
    load_descriptor_list
      = Fcons (make_number (fileno (stream)), load_descriptor_list);
    load_in_progress++;
!   readevalloop (Qget_file_char, stream, hist_file_name,
  		Feval, 0, Qnil, Qnil, Qnil, Qnil);
    unbind_to (count, Qnil);
  
***************
*** 1385,1390 ****
--- 1392,1405 ----
  
    GCPRO4 (sourcename, readfun, start, end);
  
+   /* Try to ensure sourcename is a truename, except whilst preloading. */
+   if (NILP (Vpurify_flag) && !NILP (sourcename)
+       && Ffile_name_absolute_p (sourcename))
+     {
+       Lisp_Object Qfile_truename = intern ("file-truename") ;
+       if (!NILP (Ffboundp (Qfile_truename)))
+         sourcename = call1 (Qfile_truename, sourcename) ;
+     }
    LOADHIST_ATTACH (sourcename);
  
    continue_reading_p = 1;
***************
*** 3982,3987 ****
--- 3997,4006 ----
  Each alist element is a list that starts with a file name,
  except for one element (optional) that starts with nil and describes
  definitions evaluated from buffers not visiting files.
+ 
+ The file name is absolute and is the true file name (i.e. it doesn't
+ contain symbolic links) of the loaded file.
+ 
  The remaining elements of each list are symbols defined as variables
  and cons cells of the form `(provide . FEATURE)', `(require . FEATURE)',
  `(defun . FUNCTION)', `(autoload . SYMBOL)', and `(t . SYMBOL)'.



2006-05-10  Alan Mackenzie  <acm@muc.de>

	* startup.el (command-line): For names of preloaded files, don't
	append ".elc" (now done in Fload), and call file-truename on the
	lisp directory.


Index: lisp/startup.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/startup.el,v
retrieving revision 1.407
diff -c -r1.407 startup.el
*** lisp/startup.el	5 May 2006 14:05:54 -0000	1.407
--- lisp/startup.el	10 May 2006 10:51:05 -0000
***************
*** 644,661 ****
  
    ;; Convert preloaded file names to absolute.
    (let ((lisp-dir
! 	 (file-name-directory
! 	  (locate-file "simple" load-path
! 		       (get-load-suffixes)))))
  
      (setq load-history
  	  (mapcar (lambda (elt)
  		    (if (and (stringp (car elt))
  			     (not (file-name-absolute-p (car elt))))
  			(cons (concat lisp-dir
! 				      (car elt)
! 				      (if (string-match "[.]el$" (car elt))
! 					  "" ".elc"))
  			      (cdr elt))
  		      elt))
  		  load-history)))
--- 644,660 ----
  
    ;; Convert preloaded file names to absolute.
    (let ((lisp-dir
! 	 (file-truename
! 	  (file-name-directory
! 	   (locate-file "simple" load-path
! 			(get-load-suffixes))))))
  
      (setq load-history
  	  (mapcar (lambda (elt)
  		    (if (and (stringp (car elt))
  			     (not (file-name-absolute-p (car elt))))
  			(cons (concat lisp-dir
! 				      (car elt))
  			      (cdr elt))
  		      elt))
  		  load-history)))

#########################################################################

-- 
Alan.