Re: Compilation order. Help with makefiles, please!

Re: Compilation order. Help with makefiles, please!

[Alan Mackenzie]

Hi, Richard!

On Sat, Aug 18, 2007 at 08:44:52PM -0400, Richard Stallman wrote:
> Can you do something to make recompilation of the cc files reliable?
> We see problems due to recompilation order over and over.

I didn't realise it was that bad.  But it is a massive pain in the
posterior.  I have been attempting to minimise this problem by giving
specific recipes for order of compilation when, for example, cc-langs.el
is changed.  Clearly more is wanted.

> For instance, if you put the macros etc. into one file whose name
> comes alphabetically before the rest, maybe everything will work
> reliably.

Surely the right tool for this job is the (or a?) makefile?  If we're
going to fix the problem, we might as well fix it properly.

I am attempting to modify ..../lisp/Makefile.  However, I'm still
learning about makefiles, and would welcome tips or general help, or
comments about this approach from people here.  At the moment, this is
what I've added to ..../lisp/Makefile:

#########################################################################
#### NEW STOUGH, 2007-08-25
pr=$(lisp)/progmodes
.PHONY: CC-MODE
CC-MODE: $(pr)/cc-align.elc $(pr)/cc-awk.elc  $(pr)/cc-bytecomp.elc  $(pr)/cc-cmds.elc  $(pr)/cc-compat.elc  $(pr)/cc-defs.elc \ $(pr)/cc-engine.elc  $(pr)/cc-fonts.elc  $(pr)/cc-langs.elc  $(pr)/cc-menus.elc  $(pr)/cc-mode.elc  $(pr)/cc-styles.elc  $(pr)\/cc-subword.elc  $(pr)/cc-vars.elc

# "Top level" CC Mode files:
$(pr)/cc-align.elc \
$(pr)/cc-awk.elc \
$(pr)/cc-cmds.elc \
$(pr)/cc-compat.elc \
$(pr)/cc-menus.elc \
$(pr)/cc-styles.elc \
$(pr)/cc-subword.elc \
$(pr)/cc-vars.elc: \
        $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc

# CC Mode files which create, initialise, or use the innard of "language
# variables":
$(pr)/cc-mode.elc $(pr)/cc-engine.elc $(pr)/cc-fonts.elc: $(pr)/cc-langs.elc   # <===========================

# CC Mode file which creates language variables:
$(pr)/cc-langs.elc: \
        $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc

# CC Mode main macro definition file:
$(pr)/cc-defs.elc: $(pr)/cc-bytecomp.elc

#### END OF NEW STOUGH, 2007-08-25
#########################################################################

If I delete cc-langs.elc, then do:

    % make -nf CC-Makefile.mk progmodes/cc-engine.elc

, make reports:

    make: `progmodes/cc-engine.elc' is up to date.

, though I would expect it first to rebuild cc-langs.elc (due to the line
I've marked with <=====).

Thanks in advance for any help!

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Compilation order. Help with makefiles, please!

[Michaël Cadilhac]

[-- Attachment #1.1: Type: text/plain, Size: 727 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> If I delete cc-langs.elc, then do:
>
>     % make -nf CC-Makefile.mk progmodes/cc-engine.elc
>
> , make reports:
>
>     make: `progmodes/cc-engine.elc' is up to date.

IIRC, this is a good example of the pickiness of make(1). Try:

      $ make -nf CC-Makefile.mk $(pwd)/progmodes/cc-engine.elc

(Indeed, you put a $(lisp)/ before each path...)

-- 
 |   Michaël `Micha' Cadilhac       |  Si j'étais sous-secrétaire d'État     |
 |   http://michael.cadilhac.name   |    aux choux farcis, vous entendriez   |
 |   JID/MSN:                       |  beaucoup parler des choux farcis !    |
 `----  michael.cadilhac@gmail.com  |          -- Nicolas Sarkozy       -  --'

[-- Attachment #1.2: Type: application/pgp-signature, Size: 188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Compilation order. Help with makefiles, please!

[Alan Mackenzie]

Hi, Michaël!

On Sun, Aug 26, 2007 at 12:28:14PM +0200, Michaël Cadilhac wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > If I delete cc-langs.elc, then do:

> >     % make -nf CC-Makefile.mk progmodes/cc-engine.elc

> > , make reports:

> >     make: `progmodes/cc-engine.elc' is up to date.

> IIRC, this is a good example of the pickiness of make(1). Try:

>       $ make -nf CC-Makefile.mk $(pwd)/progmodes/cc-engine.elc

> (Indeed, you put a $(lisp)/ before each path...)

Ah!  You mean, make merely does a literal text comparison of the
"filename" one gives it ("progmodes/cc-engine.elc") with the target
($(lisp)/progmodes/cc-engine.elc, which expands to
"/home/acm/emacs/emacs-22/lisp/progmodes/cc-engine.elc").  They don't
match.

What's the best way of handling this?

Anyway, when I do

   % make -nf CC-Makefile.mk /home/acm/emacs/emacs-22/lisp/progmodes/cc-engine.elc

, make replies with this:

   EMACSLOADPATH=/home/acm/emacs/emacs-22/lisp ../src/emacs -batch --no-site-file --multibyte -f batch-byte-compile /home/acm/emacs/emacs-22/lisp/progmodes/cc-defs.el
   EMACSLOADPATH=/home/acm/emacs/emacs-22/lisp ../src/emacs -batch --no-site-file --multibyte -f batch-byte-compile /home/acm/emacs/emacs-22/lisp/progmodes/cc-langs.el
   EMACSLOADPATH=/home/acm/emacs/emacs-22/lisp ../src/emacs -batch --no-site-file --multibyte -f batch-byte-compile /home/acm/emacs/emacs-22/lisp/progmodes/cc-engine.el

, which is what I want.

Many thanks!

>  Michaël `Micha' Cadilhac

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Compilation order. Help with makefiles, please!

[Richard Stallman]

    > For instance, if you put the macros etc. into one file whose name
    > comes alphabetically before the rest, maybe everything will work
    > reliably.

    Surely the right tool for this job is the (or a?) makefile?  If we're
    going to fix the problem, we might as well fix it properly.

A fix in the makefile is better than no fix, but it is troublesome
because it would require major changes in something which is now very
simple.  Practically speaking, we would be much better off if you could
solve this by changes in the cc-mode files themselves.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Help with makefiles, please!

[Alan Mackenzie_JWA]

[-- Attachment #1.1: Type: text/plain, Size: 2303 bytes --]



Hi, Richard!

On Sunday 26th August 2007 you wrote:

>    > For instance, if you put the macros etc. into one file whose name
>    > comes alphabetically before the rest, maybe everything will work
>    > reliably.

>    Surely the right tool for this job is the (or a?) makefile?  If we're
>    going to fix the problem, we might as well fix it properly.

>A fix in the makefile is better than no fix, but it is troublesome
>because it would require major changes in something which is now very
>simple.

Is "major" not an exaggeration?  The sort of change I suggested (recording
the precise dependencies between the cc-*.elc files) would be fairly
localised in effect.  It could set a precedent for further changes,
though.

I think the makefile is simple because it doesn't reflect the inherent
complexity of Emacs (or, at least, of CC Mode), and this is why compiling
CC Mode is troublesome.

>Practically speaking, we would be much better off if you could
>solve this by changes in the cc-mode files themselves.

I can't see how this can be done.  The case I am thinking about is when a
change happens in cc-langs.el, say the introduction of a new language
variable c-foo - this is what I changed in a bug fix last week.  The
sequence of compilations now needed is this:
(i) Compile cc-langs.el; this puts the details of c-foo into a macro in
  cc-langs.elc.
(ii) Compile cc-engine.el and cc-mode.el; from the cc-langs macro, the
  first generates
    (defvar c-foo "Doc string.")
  in cc-engine.elc.  The second generates (in effect), the following in
  CC Mode's initialisation in cc-mode.elc:

    (setq cc-foo
     (cond
      ((eq major mode 'c-mode) "c-foo")
      ((eq major mode 'c++-mode) "c++-foo")
      .....))

0         1         2         3         4         5         6         7  3
Thus changing cc-langs.el makes a recompiliation of cc-{mode,engine}.el
necessary.  I can't see how this can be done sensibly within the CC Mode
files.  Possibly, cc-{mode,engine}.el could, at compilation time, get the
timestamps for cc-langs.el{,c}, compare them, and recursively compile
cc-langs.el if necessary.  But this is a horrible idea.

Or were you thinking of a different troublesome case?

--
Alan Mackenzie (Nuremberg, Germany).

[-- Attachment #1.2: Type: text/html, Size: 4081 bytes --]

[-- Attachment #2: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Help with makefiles, please!

[Richard Stallman]

    Is "major" not an exaggeration?  The sort of change I suggested (record=
    ing
    the precise dependencies between the cc-*.elc files) would be fairly
    localised in effect.  It could set a precedent for further changes,
    though.

Exactly.

    Thus changing cc-langs.el makes a recompiliation of cc-{mode,engine}.el=

    necessary.  I can't see how this can be done sensibly within the CC Mod=
    e
    files.

That is a valid argument.  I guess we do need to put specific files in
lisp/Makefile.in.  I wish it were not necessary, but it is.

Please write it, and show us what the code looks like.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Help with makefiles, please!

[Stefan Monnier]

> That is a valid argument.  I guess we do need to put specific files in
> lisp/Makefile.in.  I wish it were not necessary, but it is.

An "easy" solution is to prefer reading the newer .el files rather than the
older .elc when `requiring' a file during byte-compilation.  This way, in
the example, if cc-engine.el is compiled before cc-langs.el, it's still OK
because it will load the new cc-langs.el rather than the oild cc-langs.elc.


        Stefan

Re: Compilation order. Help with makefiles, please!

[Richard Stallman]

    An "easy" solution is to prefer reading the newer .el files rather than the
    older .elc when `requiring' a file during byte-compilation.

To solve this problem, it would have to load the newer .el file even
if the feature is already provided.

That is quite different from `require', and I don't think we should
change `require' to do that.  But we could add it as a separate new
feature, perhaps called `compilation-require'.

What do you think about that?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Help with makefiles, please!

[Alan Mackenzie_JWA]

[-- Attachment #1.1: Type: text/plain, Size: 1164 bytes --]

Hi, Richard and Stefan,

Richard Stallman <rms@gnu.org> wrote on 2007-08-31 09:34:55:

>     An "easy" solution is to prefer reading the newer .el files rather
>     than the older .elc when `requiring' a file during
>     byte-compilation.

Stefan, the essential issue here is to ensure that cc-{mode,engine}.el
actually get recompiled after a change in cc-langs.el.  This needs stuff
in the makefile.

>     To solve this problem, it would have to load the newer .el file
>     even if the feature is already provided.

There is stuff like this in cc-bytecomp.el - cc-require, cc-provide, and
so on.  I've a feeling that cc-require might load the .el file rather
than the .elc, but I'm not sure at the moment.  Byte compiling
cc-langs.el is a good idea anyway, to check for errors in cc-langs.

> That is quite different from `require', and I don't think we should
> change `require' to do that.  But we could add it as a separate new
> feature, perhaps called `compilation-require'.

> What do you think about that?

cc-bytecomp implements this sort of thing.  Possibly its functionality
could be absorbed into Emacs 23.

--
Alan Mackenzie (Nuremberg, Germany).

[-- Attachment #1.2: Type: text/html, Size: 1589 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Compilation order. Help with makefiles, please!

[Stefan Monnier]

>> An "easy" solution is to prefer reading the newer .el files rather
>> than the older .elc when `requiring' a file during
>> byte-compilation.

> Stefan, the essential issue here is to ensure that cc-{mode,engine}.el
> actually get recompiled after a change in cc-langs.el.  This needs stuff
> in the makefile.

Yes, that's another part of the problemn indeed.  My suggestion just removes
the need to compile cc-langs.el before compiling cc-engine.el but doesn't
help making sure that cc-engine.el gets recompiled at all.

This requires something in the Makefile, but we could extract it
(semi-)automatically by doing a makedepend kind of thing (I had written such
a trick which looks for `require's and `provide's to build the dependencies:
it does suffer from circular dependencies, so either we'd have to live with
complaints from `make' or we'd have to break those dependencies manually).
We could also tweak the byte-compiler to do something like what
"gcc -MD" does, but I suspect this would also suffer from
circular dependencies.


        Stefan

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Help with makefiles, please!

[Richard Stallman]

    > That is quite different from `require', and I don't think we should
    > change `require' to do that.  But we could add it as a separate new
    > feature, perhaps called `compilation-require'.

    > What do you think about that?

    cc-bytecomp implements this sort of thing.  Possibly its functionality
    could be absorbed into Emacs 23.

It sounds like you're saying that this approach would work as a solution.

What do others think of it?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Compilation order. Tentative patch.

[Alan Mackenzie]

On Sat, 18 Aug 2007 at 20:44:52 -0400, Richard Stallman wrote:
> Can you fo something to make recompilation of the cc files reliable?
> We see problems due to recompilation order over and over.
>
> For instance, if you put the macros etc. into one file whose name
> comes alphabetically before the rest, maybe everything will work
> reliably.

And, a bit later in the thread.....

On Thu, Aug 30, 2007 at 03:15:06AM -0400, Richard Stallman wrote:
>     [ACM:]
>     Is "major" not an exaggeration?  The sort of change I suggested
>     (recording the precise dependencies between the cc-*.elc files)
>     would be fairly localised in effect.  It could set a precedent for
>     further changes, though.
> 
> Exactly.
> 
>     Thus changing cc-langs.el makes a recompiliation of
>     cc-{mode,engine}.el necessary.  I can't see how this can be done
>     sensibly within the CC Mode files.
> 
> That is a valid argument.  I guess we do need to put specific files in
> lisp/Makefile.in.  I wish it were not necessary, but it is.
> 
> Please write it, and show us what the code looks like.

OK, here is my tentative patch.  It works, but is lacking in elegance.
Suggestions for increasing the elegance would be welcome.


2007-10-05  Alan Mackenzie  <acm@muc.de>

	* Makefile.in: Rigorously specify the dependencies between the CC
	Mode files.elc.
	(CC-MODE): New target.


Index: Makefile.in
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/Makefile.in,v
retrieving revision 1.85.2.3
diff -c -r1.85.2.3 Makefile.in
*** Makefile.in	19 Aug 2007 00:24:55 -0000	1.85.2.3
--- Makefile.in	5 Oct 2007 21:12:56 -0000
***************
*** 213,225 ****
  	$(emacs) --eval "(batch-byte-recompile-directory 0)" $(lisp)
  
  # CC Mode uses a compile time macro system which causes a compile time
! # dependency in cc-mode.elc on the macros in cc-langs.el and the
! # version string in cc-defs.el.
! $(lisp)/progmodes/cc-mode.elc: \
! 	$(lisp)/progmodes/cc-mode.el \
! 	$(lisp)/progmodes/cc-langs.el \
! 	$(lisp)/progmodes/cc-defs.el
! 	$(emacs) -f batch-byte-compile $(lisp)/progmodes/cc-mode.el
  
  # Update MH-E internal autoloads. These are not to be confused with
  # the autoloads for the MH-E entry points, which are already in
--- 213,246 ----
  	$(emacs) --eval "(batch-byte-recompile-directory 0)" $(lisp)
  
  # CC Mode uses a compile time macro system which causes a compile time
! # dependency in cc-mode.elc, cc-engine.elc, and cc-fonts.elc on the "language
! # variables" in cc-langs.el.  A(lmost) all CC files depend on the macros in
! # cc-defs.elc and the bytecomp stuff in cc-bytecomp.el.
! pr=$(lisp)/progmodes
! .PHONY: CC-MODE
! CC-MODE: $(pr)/cc-align.elc $(pr)/cc-awk.elc $(pr)/cc-bytecomp.elc $(pr)/cc-cmds.elc $(pr)/cc-compat.elc $(pr)/cc-defs.elc $(pr)/cc-engine.elc $(pr)/cc-fonts.elc $(pr)/cc-langs.elc $(pr)/cc-mode.elc $(pr)/cc-menus.elc $(pr)/cc-styles.elc $(pr)/cc-subword.elc $(pr)/cc-vars.elc
! 
! # "Top level" CC Mode files:
! $(pr)/cc-align.elc \
! $(pr)/cc-awk.elc \
! $(pr)/cc-cmds.elc \
! $(pr)/cc-compat.elc \
! $(pr)/cc-menus.elc \
! $(pr)/cc-styles.elc \
! $(pr)/cc-subword.elc \
! $(pr)/cc-vars.elc: \
!     $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc
! 
! # CC Mode files which defvar, initialise, or use the "language variables":
! $(pr)/cc-mode.elc \
! $(pr)/cc-engine.elc \
! $(pr)/cc-fonts.elc: \
!      $(pr)/cc-langs.elc $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc
! 
! # CC Mode file which defines the language variables and their values:
! $(pr)/cc-langs.elc: $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc
! # CC Mode main macro definition file:
! $(pr)/cc-defs.elc: $(pr)/cc-bytecomp.elc
  
  # Update MH-E internal autoloads. These are not to be confused with
  # the autoloads for the MH-E entry points, which are already in

  
-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Compilation order. Tentative patch.

[Mathias Megyei]

Hi Alan,

Alan Mackenzie writes:
 > Suggestions for increasing the elegance would be welcome.
 > 
 > 
 > 2007-10-05  Alan Mackenzie  <acm@muc.de>
 > 
 > 	* Makefile.in: Rigorously specify the dependencies between the CC
 > 	Mode files.elc.
 > 	(CC-MODE): New target.
 > 
 > 
 > Index: Makefile.in
 > ===================================================================
 > RCS file: /cvsroot/emacs/emacs/lisp/Makefile.in,v
 > retrieving revision 1.85.2.3
 > diff -c -r1.85.2.3 Makefile.in
 > *** Makefile.in	19 Aug 2007 00:24:55 -0000	1.85.2.3
 > --- Makefile.in	5 Oct 2007 21:12:56 -0000
 > ***************
 > *** 213,225 ****
 >   	$(emacs) --eval "(batch-byte-recompile-directory 0)" $(lisp)
 >   
 >   # CC Mode uses a compile time macro system which causes a compile time
 > ! # dependency in cc-mode.elc on the macros in cc-langs.el and the
 > ! # version string in cc-defs.el.
 > ! $(lisp)/progmodes/cc-mode.elc: \
 > ! 	$(lisp)/progmodes/cc-mode.el \
 > ! 	$(lisp)/progmodes/cc-langs.el \
 > ! 	$(lisp)/progmodes/cc-defs.el
 > ! 	$(emacs) -f batch-byte-compile $(lisp)/progmodes/cc-mode.el
 >   
 >   # Update MH-E internal autoloads. These are not to be confused with
 >   # the autoloads for the MH-E entry points, which are already in
 > --- 213,246 ----
 >   	$(emacs) --eval "(batch-byte-recompile-directory 0)" $(lisp)
 >   
 >   # CC Mode uses a compile time macro system which causes a compile time
 > ! # dependency in cc-mode.elc, cc-engine.elc, and cc-fonts.elc on the "language
 > ! # variables" in cc-langs.el.  A(lmost) all CC files depend on the macros in
 > ! # cc-defs.elc and the bytecomp stuff in cc-bytecomp.el.
 > ! pr=$(lisp)/progmodes
 > ! .PHONY: CC-MODE
 > ! CC-MODE: $(pr)/cc-align.elc $(pr)/cc-awk.elc $(pr)/cc-bytecomp.elc $(pr)/cc-cmds.elc $(pr)/cc-compat.elc $(pr)/cc-defs.elc $(pr)/cc-engine.elc $(pr)/cc-fonts.elc $(pr)/cc-langs.elc $(pr)/cc-mode.elc $(pr)/cc-menus.elc $(pr)/cc-styles.elc $(pr)/cc-subword.elc $(pr)/cc-vars.elc

 You can omit to list the files cc-bytecomp.elc, cc-defs.elc,
 cc-langs.el, because they appear in the dependency lists of the
 rules below.

 > ! 
 > ! # "Top level" CC Mode files:
 > ! $(pr)/cc-align.elc \
 > ! $(pr)/cc-awk.elc \
 > ! $(pr)/cc-cmds.elc \
 > ! $(pr)/cc-compat.elc \
 > ! $(pr)/cc-menus.elc \
 > ! $(pr)/cc-styles.elc \
 > ! $(pr)/cc-subword.elc \
 > ! $(pr)/cc-vars.elc: \
 > !     $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc

Since $(pr)/cc-defs.elc already depends on $(pr)/cc-bytecomp.elc
there's no need to add the latter to this list.

 > ! 
 > ! # CC Mode files which defvar, initialise, or use the "language variables":
 > ! $(pr)/cc-mode.elc \
 > ! $(pr)/cc-engine.elc \
 > ! $(pr)/cc-fonts.elc: \
 > !      $(pr)/cc-langs.elc $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc

the same here, $(pr)/cc-langs.elc depends on both
cc-defs.elc and cc-bytecomp.elc.

 > ! 
 > ! # CC Mode file which defines the language variables and their values:
 > ! $(pr)/cc-langs.elc: $(pr)/cc-defs.elc $(pr)/cc-bytecomp.elc

the same here

Mathias

Re: Compilation order. Tentative patch.

[Richard Stallman]

What I don't see in your patch is anything to cause the target CC-MODE
to be built (other than a manual request).  That part is needed too.