bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Eli Zaretskii]

  emacs -Q
  C-x C-f foo.awk RET

Type "# something" (without quotes).  The comment is not highlighted
in the font-lock-comment face, as expected.  comment-start and
comment-start-skip seem to have reasonable values.


In GNU Emacs 24.0.50.8 (x86_64-unknown-linux-gnu, GTK+ Version 2.20.1)
 of 2011-09-05 on fencepost
configured using `configure  '--enable-asserts' '--enable-checking' '--with-gif=no' '--with-tiff=no''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  shell-dirtrack-mode: t
  display-time-mode: t
  show-paren-mode: t
  savehist-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
o n SPC a s SPC p a r t SPC o f SPC t h e SPC s t a 
n d a l o n e SPC i n s t a l l e r SPC i s SPC t h 
e SPC o n l y SPC p r a c t i c a l SPC w a y . ESC 
O B RET ESC < ESC > ESC O D ESC O D SPC t h a t SPC 
d o e s n ' t SPC r e q u i r e SPC P y t h i o n ESC 
O D ESC O D DEL ESC O B ESC O D ESC O D ESC b a SPC 
ESC f SPC i n s t a l l a t i o n ESC q ESC O B C-c 
C-c ESC ! ESC O A RET ESC x ESC O A RET ESC O A RET 
C-s 0 8 : 1 8 : 4 1 ESC O B ESC [ 6 ~ ESC [ 6 ~ ESC 
O B ESC O B ESC O B ESC O B C-x C-x C-w C-x C-s C-x 
k RET C-u g ESC O A RET d d d C-x C-s ESC O B ESC O 
A C-u g ESC O A DEL e w RET d d n d d d d d d d d d 
d d d d d d d d d d d d C-x C-z C-x C-f f o o . a w 
k RET # SPC f o o RET ESC ~ C-x b RET C-x C-z ESC x 
r e p o r t - e m a c s - b u g RET

Recent messages:
Counting new messages...done (24)
Saving file /home/e/eliz/INBOX...
Wrote /home/e/eliz/INBOX [2 times]
Computing summary lines...done
24 new messages read
Showing message 673
Showing message 673...done
No following nondeleted message
(New file)
Modification-flag cleared

Load-path shadows:
None found.

Features:
(shadow emacsbug cc-awk shell pcomplete comint ring newcomment
flyspell ispell multi-isearch vc-bzr cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs qp
rmailsum rmailmm message sendmail regexp-opt format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mailabbrev gmm-utils
mailheader mail-parse rfc2231 rmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils time paren cus-start cus-load time-date savehist
saveplace tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
loaddefs button faces cus-face files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process
dynamic-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

If you delete the compiled version of cc-fonts, it works...

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 9448@debbugs.gnu.org
> Date: Wed, 07 Sep 2011 03:51:32 -0400
> 
> 
> If you delete the compiled version of cc-fonts, it works...

You mean, delete it and never compile it again, yes?

bug#9448: asdf

[Alan Mackenzie]

Hi, Eli and Glenn.

>  emacs -Q
>  C-x C-f foo.awk RET

> Type "# something" (without quotes).  The comment is not highlighted
> in the font-lock-comment face, as expected.  comment-start and
> comment-start-skip seem to have reasonable values.


A bit of binary chopping shows that the bug first became visible after
this change:


  revno: 104401
  committer: Chong Yidong <cyd@stupidchicken.com>
  branch nick: trunk
  timestamp: Sat 2011-05-28 13:16:54 -0400
  message:
    Make CC modes inherit from prog-mode.

    * lisp/progmodes/cc-mode.el (c-mode, c++-mode, objc-mode, java-mode)
    (idl-mode, pike-mode, awk-mode): Inherit from prog-mode.


I cannot see anything in the change with looks remotely suspicious,
however.  Except, of course, that AWK Mode is now constructed by a macro
of some complexity, doubly invoked.

Deleting cc-fonts.elc (thus allowing cc-fonts.el to be loaded) gets
fontification of AWK comments working for me, too.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

Hi, all.

On Wed, Sep 07, 2011 at 05:01:47PM +0000, Alan Mackenzie wrote:

> >  emacs -Q
> >  C-x C-f foo.awk RET

> > Type "# something" (without quotes).  The comment is not highlighted
> > in the font-lock-comment face, as expected.  comment-start and
> > comment-start-skip seem to have reasonable values.


> A bit of binary chopping shows that the bug first became visible after
> this change:


>   revno: 104401
>   committer: Chong Yidong <cyd@stupidchicken.com>
>   branch nick: trunk
>   timestamp: Sat 2011-05-28 13:16:54 -0400
>   message:
>     Make CC modes inherit from prog-mode.

>     * lisp/progmodes/cc-mode.el (c-mode, c++-mode, objc-mode, java-mode)
>     (idl-mode, pike-mode, awk-mode): Inherit from prog-mode.


> I cannot see anything in the change with looks remotely suspicious,
> however.  Except, of course, that AWK Mode is now constructed by a macro
> of some complexity, doubly invoked.

I think I see the problem, it's with the macro `define-derived-mode'.
This creates an empty syntax table with (defvar awk-mode-syntax-table),
and does this before cc-awk.elc has been loaded.  This thus stuffs the
proper definition of awk-mode-syntax-table.

Because of this, ?# doesn't have the open-comment syntax flag, thus
comments can't get fontified.

If I'm right about this, a solution would be to insert

   (makunbound 'awk-mode-syntax-table)
   
at the top of the mode initialisation.  Why on earth does
`define-derived-mode' have the temerity to create an empty, thus useless,
syntax table?  Maybe this should be taken out of the macro.

> Deleting cc-fonts.elc (thus allowing cc-fonts.el to be loaded) gets
> fontification of AWK comments working for me, too.

I still don't have a clue why this is happening.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Stefan Monnier]

> If I'm right about this, a solution would be to insert
>    (makunbound 'awk-mode-syntax-table)

I wouldn't consider this as a solution.  I see a few solutions, tho:
- move the defvar from cc-awk to cc-mode.
- move the define-derived-mode to cc-awk (sounds natural, from my
  "I haven't looked at the code" point of view).
- add a ":syntax-table awk-mode-syntax-table" argument to define-derived-mode.

> at the top of the mode initialisation.  Why on earth does
> `define-derived-mode' have the temerity to create an empty, thus useless,
> syntax table?

It's actually not empty: it inherits from the parent.  This is taken
directly from the original define-derived-mode, which was oddly meant
for end-users.
It also has the advantage to ensure that every major-mode defined with
define-derived-mode will have a foo-mode-syntax-table.

> Maybe this should be taken out of the macro.

Maybe so, yes.  I'm not completely sure.  In any case it's too late to
do that for Emacs-24.  But we could try it out in Emacs-25.


        Stefan

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Stefan Monnier wrote:

> - add a ":syntax-table awk-mode-syntax-table" argument to define-derived-mode.

This doesn't work by itself. When calling `awk-mode', cc-awk.el is not
loaded early enough.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

Hi, Stefan

On Wed, Sep 07, 2011 at 10:12:40PM -0400, Stefan Monnier wrote:
> > If I'm right about this, a solution would be to insert
> >    (makunbound 'awk-mode-syntax-table)

> I wouldn't consider this as a solution.

It doesn't work anyway.  ;-)

> I see a few solutions, tho:
> - move the defvar from cc-awk to cc-mode.

Undesirable.

> - move the define-derived-mode to cc-awk (sounds natural, from my
>   "I haven't looked at the code" point of view).

That's beginning to sound serious.  CC Mode is structured so that the
interface with the "OS" bit of the host Emacs is in cc-mode.el.

> - add a ":syntax-table awk-mode-syntax-table" argument to define-derived-mode.

I've tried :syntax-table nil, and this doesn't generate
awk-mode-syntax-table.  It appears to work.

> > at the top of the mode initialisation.  Why on earth does
> > `define-derived-mode' have the temerity to create an empty, thus useless,
> > syntax table?

> It's actually not empty: it inherits from the parent.  This is taken
> directly from the original define-derived-mode, which was oddly meant
> for end-users.
> It also has the advantage to ensure that every major-mode defined with
> define-derived-mode will have a foo-mode-syntax-table.

> > Maybe this should be taken out of the macro.

> Maybe so, yes.  I'm not completely sure.  In any case it's too late to
> do that for Emacs-24.  But we could try it out in Emacs-25.

OK.

One other thing is bothering me quite a bit.  Just before all this, at
L1530 in cc-mode.el is the line

    (defvar awk-mode-syntax-table)

.  This ought to prevent my solution above from working, but it doesn't.
Glenn, I think you put this line in.  Why?  Would it be OK to take it out
again?

BTW, I've found out why deleting cc-fonts.elc does what it does.  There
is a "(cc-require-when-compile 'cc-awk)" in cc-fonts.el.  When this file
isn't byte compiled, that form loads cc-awk.elc, and this happens before
the bulk of cc-mode.elc is loaded.  That form should probably be removed,
since it no longer serves a purpose.

Glenn, what on earth brought you to try deleting cc-fonts.elc?  It seems
a stroke of genius, but how did you come to consider it?  :-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Alan Mackenzie wrote:

>     (defvar awk-mode-syntax-table)
>
> . This ought to prevent my solution above from working, but it
> doesn't. Glenn, I think you put this line in. Why? Would it be OK to
> take it out again?

A defvar with no initvalue just suppresses a byte-compiler warning about
said variable being undefined. It doesn't actually define the variable,
so I think that line should stay there, to stop the compiler complaining
about an undefined variable in

  (set-syntax-table awk-mode-syntax-table)

(So by the way I think this means cc-bytecomp-defvar is no longer
needed. Also cc-bytecomp-defun can be replaced by declare-function.)

> Glenn, what on earth brought you to try deleting cc-fonts.elc?  It seems
> a stroke of genius, but how did you come to consider it?  :-)

I saw from the ChangeLog that cc-fonts was the most recent part of
cc-mode to be changed, so I tried reverting that commit. To make it take
effect, I deleted the old cc-fonts.elc rather than recompiling it.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

On Thu, Sep 08, 2011 at 02:36:24PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> >     (defvar awk-mode-syntax-table)

> > . This ought to prevent my solution above from working, but it
> > doesn't. Glenn, I think you put this line in. Why? Would it be OK to
> > take it out again?

> A defvar with no initvalue just suppresses a byte-compiler warning about
> said variable being undefined. It doesn't actually define the variable,
> ...

Well, you learn something new every day.  :-)  I'd thought that (defvar
foo nil) and (defvar foo) were the same.  They're not.

> so I think that line should stay there, to stop the compiler complaining
> about an undefined variable in

>   (set-syntax-table awk-mode-syntax-table)

OK.

> (So by the way I think this means cc-bytecomp-defvar is no longer
> needed. Also cc-bytecomp-defun can be replaced by declare-function.)

There's the rub.  The more such "tidying up" changes are made in
savannah, the more difficult it becomes for me to synch changes with the
upstream CC Mode.  There're already > 80 such differences in cc-mode.el
alone.

> > Glenn, what on earth brought you to try deleting cc-fonts.elc?  It seems
> > a stroke of genius, but how did you come to consider it?  :-)

> I saw from the ChangeLog that cc-fonts was the most recent part of
> cc-mode to be changed, so I tried reverting that commit. To make it take
> effect, I deleted the old cc-fonts.elc rather than recompiling it.

And the irony is, AWK code doesn't use cc-fonts.elc for its
fontification; it uses a simple regexp based model.  ;-)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Eli Zaretskii]

> Date: Thu, 8 Sep 2011 21:13:48 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: 9448@debbugs.gnu.org
> 
> There's the rub.  The more such "tidying up" changes are made in
> savannah, the more difficult it becomes for me to synch changes with the
> upstream CC Mode.  There're already > 80 such differences in cc-mode.el
> alone.

How about switching CC Mode repo from CVS to Bazar?  Then merges would
be seamless.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

Morning, Eli.

On Fri, Sep 09, 2011 at 10:01:10AM +0300, Eli Zaretskii wrote:
> > Date: Thu, 8 Sep 2011 21:13:48 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: 9448@debbugs.gnu.org

> > There's the rub.  The more such "tidying up" changes are made in
> > savannah, the more difficult it becomes for me to synch changes with the
> > upstream CC Mode.  There're already > 80 such differences in cc-mode.el
> > alone.

> How about switching CC Mode repo from CVS to Bazar?  Then merges would
> be seamless.

Some time soon I'll probably be forced to change, since SourceForge won't
be supporting CVS for ever.

The time taken is checking each individual change.  Some of them will
break support for older Emacsen.  Ediff works well.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

Hi, everybody.

On Thu, Sep 08, 2011 at 08:33:11AM +0000, Alan Mackenzie wrote:

> I've tried :syntax-table nil [in the define-derived-mode macro], and
> this doesn't generate awk-mode-syntax-table.  It appears to work.

I've just committed this fix and marked the bug as fixed.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Alan Mackenzie wrote:

> I think I see the problem, it's with the macro `define-derived-mode'.
> This creates an empty syntax table with (defvar awk-mode-syntax-table),
> and does this before cc-awk.elc has been loaded.  This thus stuffs the
> proper definition of awk-mode-syntax-table.

That's interesting. I imagine in most cases, all the definitions are in
the same file, eg:

(defvar foo-mode-map ... )
(defvar foo-mode-syntax-table ... )
(define-derived-mode foo-mode ... )

cc-awk is unusual in that the actual mode is defined in cc-mode.el and
requires cc-awk.el as part of its execution.

Another option is to move awk-mode-syntax-table from cc-awk.el to
cc-mode.el (I see the mode-map and abbrev-table are already there).
Unbinding it might erase any user customization.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Alan Mackenzie wrote:

> There's the rub.  The more such "tidying up" changes are made in
> savannah, the more difficult it becomes for me to synch changes with the
> upstream CC Mode.  There're already > 80 such differences in cc-mode.el
> alone.

(defvar foo)

works to silence the compiler since at least Emacs 21.4 (I think).

declare-function works since Emacs 23.1, and exists as a no-op in Emacs
22.2 and 22.3.

I hope that one day cc-bytecomp can just be removed. The part that deals
with "an older CC Mode with outdated macros [being] loaded during
compilation" is clever, but IMO just adds a bunch of complexity to solve
a problem that doesn't need solving. Files should be compiled in batch
mode in a separate Emacs instance.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Alan Mackenzie]

Hi, Glenn.

On Thu, Sep 08, 2011 at 05:50:36PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> > There's the rub.  The more such "tidying up" changes are made in
> > savannah, the more difficult it becomes for me to synch changes with the
> > upstream CC Mode.  There're already > 80 such differences in cc-mode.el
> > alone.

> (defvar foo)

> works to silence the compiler since at least Emacs 21.4 (I think).

There're quite a few other such constructs.  Does it work under XEmacs?
If so which versions.  They're all things which are easy to check, but
cumulatively it's a massive time sink.

> declare-function works since Emacs 23.1, and exists as a no-op in Emacs
> 22.2 and 22.3.

Again, what about XEmacs?

> I hope that one day cc-bytecomp can just be removed. The part that deals
> with "an older CC Mode with outdated macros [being] loaded during
> compilation" is clever, but IMO just adds a bunch of complexity to solve
> a problem that doesn't need solving. Files should be compiled in batch
> mode in a separate Emacs instance.

I'd agree with you about its eventual removal.  Clearly there was a big
problem to solve - something like cc-bytecomp.el doesn't get knocked
together in a couple of hours.  I suspect Martin Stjernholm was having
big problems with his various versions during development, and that
cc-bytecomp is more for me and colleagues rather than ordinary users.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Stefan Monnier]

> ... XEmacs ... it's a massive time sink.

I tend to agree ;-)


        Stefan

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 08 Sep 2011 17:50:36 -0400
> Cc: 9448@debbugs.gnu.org
> 
> Files should be compiled in batch mode in a separate Emacs instance.

Are you saying that emacs-lisp-byte-compile-and-load's interactive use
should be discouraged?  I use it quite frequently to test changes in
the same Emacs session where I make those changes.  If we think it's
not a good idea, then we should deprecate the command and remove the
corresponding menu item.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Eli Zaretskii wrote:

> Are you saying that emacs-lisp-byte-compile-and-load's interactive use
> should be discouraged?  I use it quite frequently to test changes in
> the same Emacs session where I make those changes.

No; I use it like that too. All I'm saying is, when you download
someone's foo-mode to install it on your system, foo-mode's installation
process should compile it using batch mode.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Glenn Morris wrote:

> Eli Zaretskii wrote:
>
>> Are you saying that emacs-lisp-byte-compile-and-load's interactive use
>> should be discouraged?  I use it quite frequently to test changes in
>> the same Emacs session where I make those changes.
>
> No; I use it like that too. All I'm saying is, when you download
> someone's foo-mode to install it on your system, foo-mode's installation
> process should compile it using batch mode.

In fact, I'm not even saying that, because for a single, simple foo.el
compiling it in a running Emacs is probably fine. I'm just saying that
if I were compiling something as complex as cc-mode, I'd definitely do
it in batch-mode, and would not try to support the case where an older
cc-mode is already loaded in the compiling Emacs.

bug#9448: 24.0.50; Comments in Awk scripts not fontified with comment face

[Glenn Morris]

Alan Mackenzie wrote:

> Again, what about XEmacs?

It's your call as to how relevant you think other varieties of Emacs are
to you and to others, and how much effort you think it's worth investing
in supporting them.

If it takes a lot of time to ensure that the latest cc-mode runs on a
bunch of platforms, then personally I think that's time that could be
better spent, in making it work better on fewer.

bug#9448: Bug fixed.

[Alan Mackenzie]

Bug fixed.

-- 
Alan Mackenzie (Nuremberg, Germany).