Recursive compilation?

Recursive compilation?

[Lars Magne Ingebrigtsen]

Sometimes after doing a "bzr update", the Emacs Lisp files won't build
because a new macro is needed from a .el file, but the old .elc file
already exists, so it requires the .elc file, and fails compiling the
.el file, and I delete all the .elc files, and then build everything.

Which takes forever!

So I was wondering whether it would be feasible to compile required
files recursively.

Basically, I envision a function like `M-x
byte-compile-file-recursively' which would work just like the normal
one, but would instrument `require' to call
`byte-compile-file-recursively' on all files that it tries to load and
that are older than their .elc counterparts.

The obvious "gotcha" here is that one probably shouldn't try to
(recursively) byte-compile files in directories that we don't have write
access to.  So if somebody has set their load-path to my directory, and
I have an old .elc file there, then this function shouldn't try to
compile that file, too.

The other caveat is that Makefiles typically first mark all files that
are to be compiled, and then compile them.  This would then possibly
lead to some files being compiled twice.  However, that could be worked
around by `byte-compile-file-recursively' not compiling a file that it's
been asked to compile if the .elc file is newer than the .el file.

So...

I have a feeling that this won't be very difficult to implement, but it
needs meddling with the C layer, since `require' is a C function.  But I
think it would make compiling Emacs after a "bzr update" a lot less
likely to break.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Recursive compilation?

[Stefan Monnier]

> The obvious "gotcha" here is that one probably shouldn't try to
> (recursively) byte-compile files in directories that we don't have write
> access to.  So if somebody has set their load-path to my directory, and
> I have an old .elc file there, then this function shouldn't try to
> compile that file, too.

Right, cyclic require loops can definitely happen in such cases.
But they're easy to break.

> The other caveat is that Makefiles typically first mark all files that
> are to be compiled, and then compile them.  This would then possibly
> lead to some files being compiled twice.  However, that could be worked
> around by `byte-compile-file-recursively' not compiling a file that it's
> been asked to compile if the .elc file is newer than the .el file.

It's also a problem with parallel make.

> I have a feeling that this won't be very difficult to implement, but it
> needs meddling with the C layer, since `require' is a C function.  But I
> think it would make compiling Emacs after a "bzr update" a lot less
> likely to break.

Another approach is to build a set of makefile dependencies from the
`require' calls.  I did that at some point in the past.  It wasn't too
hard to do.  But there are also the cases of preloaded files, as well as
autoloaded macros/functions.

I'm personally using a different hack which doesn't solve all those
problems either but is simple and doesn't suffer from as many problems
in my experience: prefer loading the .el file if it's more recent.


        Stefan

Re: Recursive compilation?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I'm personally using a different hack which doesn't solve all those
> problems either but is simple and doesn't suffer from as many problems
> in my experience: prefer loading the .el file if it's more recent.

I was thinking that perhaps if file a.el requires a macro from b.el, and
b.el isn't compiled, then a.elc would end up in a somewhat less compiled
state than it otherwise would?

But if that's not the case, then perhaps `require', when run via
`byte-compile-file', should just always load the .el file if it's more
recent?  That certainly sounds easy enough to implement --
`byte-compile-file' could just bind a variable to signal that it wanted
this behaviour.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Recursive compilation?

[Stefan Monnier]

>> I'm personally using a different hack which doesn't solve all those
>> problems either but is simple and doesn't suffer from as many problems
>> in my experience: prefer loading the .el file if it's more recent.
> I was thinking that perhaps if file a.el requires a macro from b.el, and
> b.el isn't compiled, then a.elc would end up in a somewhat less compiled
> state than it otherwise would?

When a.el calls an inlinable function from b.el, then the result will
indeed be different (even tho in both cases the function will be
inlined), yet in most cases it shouldn't make much of a difference.

> But if that's not the case, then perhaps `require', when run via
> `byte-compile-file', should just always load the .el file if it's more
> recent?  That certainly sounds easy enough to implement --
> `byte-compile-file' could just bind a variable to signal that it wanted
> this behaviour.

I use the patch below,


        Stefan


=== modified file 'lisp/emacs-lisp/bytecomp.el'
--- lisp/emacs-lisp/bytecomp.el	2011-05-30 17:14:19 +0000
+++ lisp/emacs-lisp/bytecomp.el	2011-05-31 20:48:24 +0000
@@ -1713,6 +1720,12 @@
   (setq byte-compile-last-logged-file nil)
   (let ((byte-compile-current-file filename)
         (byte-compile-current-group nil)
+        ;; Prefer source files over compiled files.  This is so that when
+        ;; several files are changed and recompiled, each new file is
+        ;; properly recompiled with the new macros in the other new files.
+        (load-suffixes (sort (copy-sequence load-suffixes)
+                             (lambda (s1 s2) (and (string-match "\\.elc\\b" s2)
+                                             (string-match "\\.el\\b" s1)))))
 	(set-auto-coding-for-load t)
 	target-file input-buffer output-buffer
 	byte-compile-dest-file)

Re: Recursive compilation?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I use the patch below,

[...]

> +        (load-suffixes (sort (copy-sequence load-suffixes)
> +                             (lambda (s1 s2) (and (string-match "\\.elc\\b" s2)
> +                                             (string-match "\\.el\\b" s1)))))

Well, that was even simpler.  :-)

Perhaps that could be in Emacs by default?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Recursive compilation?

[Dimitri Fontaine]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> So I was wondering whether it would be feasible to compile required
> files recursively.

Do you want bute-code-cache as in the following?

  http://www.emacswiki.org/emacs/byte-code-cache.el

  ;; Description:
  ;; Automatically cache byte-compiled versions of .el files as they're
  ;; loaded, automatically recompiling them as necessary.
  
  ;; Install:
  ;; Load this file as early as possible in ~/.emacs (after setting
  ;; variables, and after custom, if you used custom to set them.)
  ;; Anything loaded after it will be automatically byte-compiled.
  
  ;; This file cannot byte-compile itself, so you should do that
  ;; separately.
  
  ;; Author: Daniel Colascione <dan.colascione@gmail.com>

Regards,
-- 
dim

Re: Recursive compilation?

[Lars Magne Ingebrigtsen]

Dimitri Fontaine <dim@tapoueh.org> writes:

> Do you want bute-code-cache as in the following?
>
>   http://www.emacswiki.org/emacs/byte-code-cache.el

That seems nice, but Stefan's simpler solution would do the trick for
me, and is something that I think Emacs should do by default.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Recursive compilation?

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 03 Jun 2011 23:55:40 +0200
> 
> Dimitri Fontaine <dim@tapoueh.org> writes:
> 
> > Do you want bute-code-cache as in the following?
> >
> >   http://www.emacswiki.org/emacs/byte-code-cache.el
> 
> That seems nice, but Stefan's simpler solution would do the trick for
> me, and is something that I think Emacs should do by default.

There's a controversy about that, because there are legitimate and
important use cases where Stefan's simpler solution causes trouble.

Re: Recursive compilation?

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> There's a controversy about that, because there are legitimate and
> important use cases where Stefan's simpler solution causes trouble.

I think I missed that discussion.  Do you have a recap, by any chance?
:-)

In any case, I think it would make sense to make this an option.  For
instance, a new variable `byte-compile-load-newest-file' defaulting to
nil.  The obvious use case would be for the Emacs makefiles to set that
to t before compiling the .el files, which I'm sure would save us all a
bit of time over the coming years.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Recursive compilation?

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 04 Jun 2011 17:54:41 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There's a controversy about that, because there are legitimate and
> > important use cases where Stefan's simpler solution causes trouble.
> 
> I think I missed that discussion.  Do you have a recap, by any chance?

The one use case I remember is when you are developing a Lisp package
and it is not yet ready for prime time (e.g., loading it would break
Emacs).

> In any case, I think it would make sense to make this an option.  For
> instance, a new variable `byte-compile-load-newest-file' defaulting to
> nil.

If foo.el requires 'bar, and bar.elc is outdated, won't loading bar.el
make the result of compiling foo.el less optimized?

> The obvious use case would be for the Emacs makefiles to set that
> to t before compiling the .el files, which I'm sure would save us all a
> bit of time over the coming years.

Even in the Makefile's, I'm not sure it should be an obvious default:
there could be important use cases when you'd want the warning/error
even when building Emacs, e.g. when making a release or pretest
tarball.

Re: Recursive compilation?

[Glenn Morris]

Lars Magne Ingebrigtsen wrote:

> In any case, I think it would make sense to make this an option.  For
> instance, a new variable `byte-compile-load-newest-file' defaulting to
> nil. 

It's a long-standing feature request:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=2061

Re: Recursive compilation?

[Glenn Morris]

Eli Zaretskii wrote:

> Even in the Makefile's, I'm not sure it should be an obvious default:
> there could be important use cases when you'd want the warning/error
> even when building Emacs, e.g. when making a release or pretest
> tarball.

Releases and pretests (as per admin/make-tarball.txt) are Made The Right Way,
ie by bootstrapping, so it's not an issue for them.

Re: Recursive compilation?

[Stefan Monnier]

>> > There's a controversy about that, because there are legitimate and
>> > important use cases where Stefan's simpler solution causes trouble.
>> I think I missed that discussion.  Do you have a recap, by any chance?
> The one use case I remember is when you are developing a Lisp package
> and it is not yet ready for prime time (e.g., loading it would break Emacs).

If we only change the "newest" behavior for byte-compilation, it means
the above will only apply when byte-compiling a file which requests your
temporarily-broken package.  There are plenty of other ways to break
Emacs with local changes (e.g. changes that aren't ready for prime-time
applied to the Makefiles, or to C files, or ...), and there are plenty
of ways to handle this problem (e.g. "bzr shelve; make; bzr unshelve"),
so I'm really not convinced it's worth the hassle any more.

>> In any case, I think it would make sense to make this an option.
>> For instance, a new variable `byte-compile-load-newest-file'
>> defaulting to nil.
> If foo.el requires 'bar, and bar.elc is outdated, won't loading bar.el
> make the result of compiling foo.el less optimized?

No.  AFAIK the only difference is for inlined functions which will be
inlined in a different way, thus resulting in different results.
I don't know of anyone who bothered to check which result is better, but
my guess is that it's a wash.

>> The obvious use case would be for the Emacs makefiles to set that
>> to t before compiling the .el files, which I'm sure would save us all a
>> bit of time over the coming years.
> Even in the Makefile's, I'm not sure it should be an obvious default:
> there could be important use cases when you'd want the warning/error
> even when building Emacs, e.g. when making a release or pretest
> tarball.

Glenn already pointed out that it's not an issue.

Any other objection?


        Stefan

Re: Recursive compilation?

[Dimitri Fontaine]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>   http://www.emacswiki.org/emacs/byte-code-cache.el
>
> That seems nice, but Stefan's simpler solution would do the trick for
> me, and is something that I think Emacs should do by default.

Maybe we could arrange things so that byte-code-cache gets included?
-- 
dim

Re: Recursive compilation?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

On 6/6/11 1:46 PM, Dimitri Fontaine wrote:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>   http://www.emacswiki.org/emacs/byte-code-cache.el
>>
>> That seems nice, but Stefan's simpler solution would do the trick for
>> me, and is something that I think Emacs should do by default.
> 
> Maybe we could arrange things so that byte-code-cache gets included?

The problem is that byte-code-cache doesn't _quite_ work. Sure, it
handles simple cases well enough, but:

1) it deals with dependency cycles by just not compiling certain files;

2) it breaks inexplicably (i.e., I haven't analyzed it) with nxhtml; and

3) [this may be a spurious effect] code compiled using byte-code-cache
somehow seems slower than code compiled the conventional way. When I
looked at cc-mode, I noticed a slight performance hit. I'd have to look
at it again in more detail to see whether I actually found a problem.

Regardless, the mode is an ugly hack. It shadows LOAD and calls the byte
compiler in a way it was never meant to be called. If we're going to
implement a feature like this, we ought to do it right, and that means
making the compiler reentrant and adding the proper hooks to LOAD.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Recursive compilation?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Any other objection?

The only objection of note is, I think, the one about not wanting things
to go into "production" until you actually byte-compile the file.  But I
don't think that's a very important objection, especially if we include
a `byte-compile-load-newest-file' variable, and default it to t.  Then
people who want the old behaviour can get it if they want it.  But my
guess is that most people would want to load the newest file when
byte-compiling. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/