bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Brent Goodrick]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

The load function will load a .elc file even if the .el file is newer
than the .elc file.  There are warnings issued, but it is likely that
the message is lost in the ton of messages that usually are emitted,
especially during Emacs invocation.

When would the user ever prefer to load the .elc file after having
modified the .el file?

My suggestion is to add a customizable preference variable called,
say, load-prefer-newer-lisp-files (that needs a better name), that is
recognized by the load function (defined in C code). When
load-prefer-newer-lisp-files is non-nil, and when there is a .el and
.elc file, and when the .el file is newer than the .elc file, it loads
the .el file and not the .elc file (and possibly emits a warning of
this action).

Having it be a customizable preference variable avoids surprising
other users who are used to the existing behavior.

I could hack around this by fset'ing `load' to be my own function that
removes the .elc file when the .el file is found to be newer, but that
is an expensive operation involving calling such functions such as
locate-file-internal to find both .el and .elc files, testing their
modification date-time stamps, etc., operations that the `load' C
function performs already.  Hence, why I think it is best to change
the load function to allow for both approaches via a preference
variable.

Thanks!
Brent


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/home/brentg/emacs_from_source/install/share/emacs/23.0.60/etc/DEBUG for instructions.


In GNU Emacs 23.0.60.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.12.11)
 of 2009-01-25 on hungover
Windowing system distributor `The X.Org Foundation', version 11.0.10402000
configured using `configure  '--with-x-toolkit' '--prefix=/home/brentg/emacs_from_source/install''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  erc-list-mode: t
  erc-menu-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-match-mode: t
  erc-button-mode: t
  erc-netsplit-mode: t
  desktop-save-mode: t
  iswitchb-mode: t
  erc-ring-mode: t
  erc-services-mode: t
  erc-networks-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  delete-selection-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: 1
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
C-p C-p C-p C-p M-f M-f SPC f o r m C-e M-b M-f SPC 
t h e SPC . e l SPC f i l e SPC i n t o SPC a SPC . 
e l c SPC f i l e C-d C-a M-q C-n C-n C-n C-n C-n C-n 
C-n C-n C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-n 
C-n C-n C-e C-b SPC f o r SPC d e m o n s t r a t i 
o n SPC p u r o <backspace> p o <backspace> o s e s 
C-a C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n M-f M-f M-f M-f M-f M-f 
M-f M-f M-f M-f M-f M-f M-b M-b M-b M-b f o r SPC M-f 
M-f M-q M-f M-f C-b , SPC w h i c h SPC i s SPC SPC 
<backspace> a SPC l i C-SPC M-b M-b M-b w r o n g SPC 
b e c a u s e SPC t h e SPC . e l c SPC f i l e SPC 
d o e s SPC i n e <backspace> d e e d SPC e x i s t 
M-q C-a C-n C-n C-n C-n C-n C-n M-f M-f M-f M-f M-f 
M-f M-f M-f M-f M-f C-n C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-c 
C-c y e s <return> <down-mouse-1> <mouse-1> M-SPC <return> 
<return> M-x r e p p o r t M-P <return>

Recent messages:
/home/brentg/.authin: 100% (398/398)
334 VXNlcm5hbWU6
334 UGFzc3dvcmQ6
235 2.7.0 Accepted
250 2.1.0 OK k41sm12274699rvb.3
250 2.1.5 OK k41sm12274699rvb.3
354  Go ahead k41sm12274699rvb.3
250 2.0.0 OK 1232907223 k41sm12274699rvb.3
221 2.0.0 closing connection k41sm12274699rvb.3
Sending...done

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Eli Zaretskii]

> From: Brent Goodrick <bgoodr@gmail.com>
> Date: Sun, 25 Jan 2009 18:19:27 -0800
> Cc: 
> 
> When would the user ever prefer to load the .elc file after having
> modified the .el file?

When you are working on the .el file and it is not ready yet, or
buggy.

> My suggestion is to add a customizable preference variable called,
> say, load-prefer-newer-lisp-files (that needs a better name), that is
> recognized by the load function (defined in C code). When
> load-prefer-newer-lisp-files is non-nil, and when there is a .el and
> .elc file, and when the .el file is newer than the .elc file, it loads
> the .el file and not the .elc file (and possibly emits a warning of
> this action).

Yes, but please not now, not before Emacs 23.1 is released.  We are in
feature freeze.

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Brent Goodrick]

Eli Zaretskii writes:
 > Yes, but please not now, not before Emacs 23.1 is released.  We are in
 > feature freeze.

Agreed. I was unaware of the release schedule.

Brent

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Juanma Barranquero]

On Mon, Jan 26, 2009 at 03:19, Brent Goodrick <bgoodr@gmail.com> wrote:

> When would the user ever prefer to load the .elc file after having
> modified the .el file?

Whenever he saves the .el file, but the modifications aren't finished
(perhaps they don't even compile).

In general, if Emacs loads the .el file when it is newer, you lose the
ability to decide whether you want the .el or the .elc. If Emacs
always loads the .elc (if present), you can decide you want the new
code loaded, by compiling the .el.

> I could hack around this by fset'ing `load' to be my own function that
> removes the .elc file when the .el file is found to be newer, but that
> is an expensive operation involving calling such functions such as
> locate-file-internal to find both .el and .elc files, testing their
> modification date-time stamps, etc., operations that the `load' C
> function performs already.

Isn't easier just to compile the .el file? If you're developing or
modifying a package, and want to try it at once, create a macro to
compile it as soon as you save it...

    Juanma

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Brent Goodrick]

Juanma Barranquero writes:
 > On Mon, Jan 26, 2009 at 03:19, Brent Goodrick <bgoodr@gmail.com> wrote:
 > 
 > > When would the user ever prefer to load the .elc file after having
 > > modified the .el file?
 > 
 > Whenever he saves the .el file, but the modifications aren't finished
 > (perhaps they don't even compile).
 > 
 > In general, if Emacs loads the .el file when it is newer, you lose the
 > ability to decide whether you want the .el or the .elc. If Emacs
 > always loads the .elc (if present), you can decide you want the new
 > code loaded, by compiling the .el.

The problem relates to when and how to notify the user that the stale
.elc file is the one being loaded.  During development, I just
`eval-buffer' repeatedly on a .el file, always unaware that there is a
stale .elc file lying in wait to confuse me the next time I reload the
entire Emacs process/session. At init time, I only get a warning,
among a ton of other benign warnings and messages, and that one
critical warning is therefore not seen (of course, it is impractical
to ask the user to read all of those messages). So, since that warning
is not seen, effectively I lose the ability to decide which file, .el
or .elc, to load.

However, your concern is duly noted. Perhaps it is better not to
change that hardcoded behavior in the way I suggested earlier.  How
about this (I'm not stuck on this nomenclature; modify to taste): Add
the following logic to the C `load' function: 

  Before loading either the .el or .elc file, test for the condition
  where the .el file is newer than the .elc file. If it is, then do
  the following:

    See if the `load-hook-stale-byte-compile-handlers' hook variable
    is set to non-nil. When it is non-nil, run the hook variable with
    `run-hook-with-args-until-success'. Each function the user has
    added to that hook variable would do any logic s/he wishes,
    including in my case to popup a minibuffer prompt asking what to
    do. When the hook function thus called returns a 'prefer-el-file
    symbol, `load' then loads the .el file and ignores the .elc
    file. Likewise, when the hook function returns the
    'prefer-elc-file symbol, then load the .elc file but give no
    warning message and ignore the .el file. When nil is returned from
    the `run-hook-with-args-until-success' function, just load the
    .elc file and produce the stale file warning message as is done
    today (i.e., preserve existing behavior).

In my case, I would actually allow the third case in the prompt, which
is to byte compile the file and then return 'prefer-elc-file so that
the newly byte compiled file is then loaded. That way, I don't pay the
byte-compile-upon-every-save penalty as indicated below.
  
 > 
 > > I could hack around this by fset'ing `load' to be my own function that
 > > removes the .elc file when the .el file is found to be newer, but that
 > > is an expensive operation involving calling such functions such as
 > > locate-file-internal to find both .el and .elc files, testing their
 > > modification date-time stamps, etc., operations that the `load' C
 > > function performs already.
 > 
 > Isn't easier just to compile the .el file? If you're developing or
 > modifying a package, and want to try it at once, create a macro to
 > compile it as soon as you save it...

I have tried doing that, but found it unworkable in practice, as
byte-compiling upon each save ended up chewing up too much time during
development (the byte-compile-upon-every-save penalty). Consider that
I save frequently. :)

A cheaper/dirtier arrangement that I have in place now is an
after-save hook that simply deletes the associated .elc file if it
exists. But that is a hack, so I am now trying to get the root problem
addressed in the C code where it exists.

Brent

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Juanma Barranquero]

On Tue, Jan 27, 2009 at 16:54, Brent Goodrick <bgoodr@gmail.com> wrote:

> The problem relates to when and how to notify the user that the stale
> .elc file is the one being loaded.  During development, I just
> `eval-buffer' repeatedly on a .el file, always unaware that there is a
> stale .elc file lying in wait to confuse me the next time I reload the
> entire Emacs process/session.

If you're going to modify/test the package with eval-buffer, you
should make sure there's no .elc sitting around.

> At init time, I only get a warning,
> among a ton of other benign warnings and messages, and that one
> critical warning is therefore not seen (of course, it is impractical
> to ask the user to read all of those messages).

As you know, that's trivially fixed:

(add-hook 'emacs-startup-hook
          (lambda ()
            (with-current-buffer (get-buffer "*Messages*")
              (goto-char (point-min))
              (while (re-search-forward "[Ss]ource file .*? newer" nil t)
                (warn "%s" (buffer-substring (line-beginning-position)
                                             (line-end-position)))))))


> Add the following logic to the C `load' function:
>
>  Before loading either the .el or .elc file, test for the condition
>  where the .el file is newer than the .elc file. If it is, then do
>  the following:
>
>    See if the `load-hook-stale-byte-compile-handlers' hook variable
>    is set to non-nil. When it is non-nil, run the hook variable with
>    `run-hook-with-args-until-success'. Each function the user has
>    added to that hook variable would do any logic s/he wishes,
>    including in my case to popup a minibuffer prompt asking what to
>    do. When the hook function thus called returns a 'prefer-el-file
>    symbol, `load' then loads the .el file and ignores the .elc
>    file. Likewise, when the hook function returns the
>    'prefer-elc-file symbol, then load the .elc file but give no
>    warning message and ignore the .el file. When nil is returned from
>    the `run-hook-with-args-until-success' function, just load the
>    .elc file and produce the stale file warning message as is done
>    today (i.e., preserve existing behavior).

That would work, but it is IMHO too much (interface, not code)
complexity for little gain. In most cases, having a .elc older than
its corresponding .el is a bug (or, let's call it, a temporary
situation), so getting a warning to remind the user about fixing it
seems much more economical.

That said, sometimes I would've liked to have a hook that runs when a
file is loaded; or the ability to defadvice Fload (you can, except
that Fload is also called from C code, for example for autoloads).

> I have tried doing that, but found it unworkable in practice, as
> byte-compiling upon each save ended up chewing up too much time during
> development (the byte-compile-upon-every-save penalty). Consider that
> I save frequently. :)

I didn't mean to byte-compile on saving, I meant to byte-compile on
eval-buffer (or whatever method you use to test your code).

> But that is a hack, so I am now trying to get the root problem
> addressed in the C code where it exists.

I agree that greater control over the loading process could sometimes
be useful; but I don't think this is a compelling use case.

    Juanma

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Kevin Rodgers]

Juanma Barranquero wrote:
> On Tue, Jan 27, 2009 at 16:54, Brent Goodrick <bgoodr@gmail.com> wrote:
...
>> Add the following logic to the C `load' function:
>>
>>  Before loading either the .el or .elc file, test for the condition
>>  where the .el file is newer than the .elc file. If it is, then do
>>  the following:
>>
>>    See if the `load-hook-stale-byte-compile-handlers' hook variable
>>    is set to non-nil. When it is non-nil, run the hook variable with
>>    `run-hook-with-args-until-success'. Each function the user has
>>    added to that hook variable would do any logic s/he wishes,
>>    including in my case to popup a minibuffer prompt asking what to
>>    do. When the hook function thus called returns a 'prefer-el-file
>>    symbol, `load' then loads the .el file and ignores the .elc
>>    file. Likewise, when the hook function returns the
>>    'prefer-elc-file symbol, then load the .elc file but give no
>>    warning message and ignore the .el file. When nil is returned from
>>    the `run-hook-with-args-until-success' function, just load the
>>    .elc file and produce the stale file warning message as is done
>>    today (i.e., preserve existing behavior).
> 
> That would work, but it is IMHO too much (interface, not code)
> complexity for little gain. In most cases, having a .elc older than
> its corresponding .el is a bug (or, let's call it, a temporary
> situation), so getting a warning to remind the user about fixing it
> seems much more economical.
> 
> That said, sometimes I would've liked to have a hook that runs when a
> file is loaded; or the ability to defadvice Fload (you can, except
> that Fload is also called from C code, for example for autoloads).

Yes, before-load-hook and after-load-hook would be generally useful
if load-file-name were set to load's FILE argument when they're run.

-- 
Kevin Rodgers
Denver, Colorado, USA

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Brent Goodrick]

I won't pursue this any further.  You all have given me some useful
workarounds that I will explore further.

Thanks
Brent

On 1/27/09, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Tue, Jan 27, 2009 at 16:54, Brent Goodrick <bgoodr@gmail.com> wrote:
>
>> The problem relates to when and how to notify the user that the stale
>> .elc file is the one being loaded.  During development, I just
>> `eval-buffer' repeatedly on a .el file, always unaware that there is a
>> stale .elc file lying in wait to confuse me the next time I reload the
>> entire Emacs process/session.
>
> If you're going to modify/test the package with eval-buffer, you
> should make sure there's no .elc sitting around.
>
>> At init time, I only get a warning,
>> among a ton of other benign warnings and messages, and that one
>> critical warning is therefore not seen (of course, it is impractical
>> to ask the user to read all of those messages).
>
> As you know, that's trivially fixed:
>
> (add-hook 'emacs-startup-hook
>           (lambda ()
>             (with-current-buffer (get-buffer "*Messages*")
>               (goto-char (point-min))
>               (while (re-search-forward "[Ss]ource file .*? newer" nil t)
>                 (warn "%s" (buffer-substring (line-beginning-position)
>                                              (line-end-position)))))))
>
>
>> Add the following logic to the C `load' function:
>>
>>  Before loading either the .el or .elc file, test for the condition
>>  where the .el file is newer than the .elc file. If it is, then do
>>  the following:
>>
>>    See if the `load-hook-stale-byte-compile-handlers' hook variable
>>    is set to non-nil. When it is non-nil, run the hook variable with
>>    `run-hook-with-args-until-success'. Each function the user has
>>    added to that hook variable would do any logic s/he wishes,
>>    including in my case to popup a minibuffer prompt asking what to
>>    do. When the hook function thus called returns a 'prefer-el-file
>>    symbol, `load' then loads the .el file and ignores the .elc
>>    file. Likewise, when the hook function returns the
>>    'prefer-elc-file symbol, then load the .elc file but give no
>>    warning message and ignore the .el file. When nil is returned from
>>    the `run-hook-with-args-until-success' function, just load the
>>    .elc file and produce the stale file warning message as is done
>>    today (i.e., preserve existing behavior).
>
> That would work, but it is IMHO too much (interface, not code)
> complexity for little gain. In most cases, having a .elc older than
> its corresponding .el is a bug (or, let's call it, a temporary
> situation), so getting a warning to remind the user about fixing it
> seems much more economical.
>
> That said, sometimes I would've liked to have a hook that runs when a
> file is loaded; or the ability to defadvice Fload (you can, except
> that Fload is also called from C code, for example for autoloads).
>
>> I have tried doing that, but found it unworkable in practice, as
>> byte-compiling upon each save ended up chewing up too much time during
>> development (the byte-compile-upon-every-save penalty). Consider that
>> I save frequently. :)
>
> I didn't mean to byte-compile on saving, I meant to byte-compile on
> eval-buffer (or whatever method you use to test your code).
>
>> But that is a hack, so I am now trying to get the root problem
>> addressed in the C code where it exists.
>
> I agree that greater control over the loading process could sometimes
> be useful; but I don't think this is a compelling use case.
>
>     Juanma
>

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Richard M Stallman]

    When would the user ever prefer to load the .elc file after having
    modified the .el file?

If you have written changes which are incomplete and not ready
to be executed.

This feature gives you control over when to start running your
changes: you control it by when you compile the files.  It is
important and it is easy to use.

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Brent Goodrick]

Richard M Stallman writes:
 >     When would the user ever prefer to load the .elc file after having
 >     modified the .el file?
 > 
 > If you have written changes which are incomplete and not ready
 > to be executed.
 > 
 > This feature gives you control over when to start running your
 > changes: you control it by when you compile the files.  It is
 > important and it is easy to use.

See a new proposal I just emailed to this thread which I think
addresses the problem a bit better than just changing the default
`load' behavior as I originally suggested.

Brent

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Stefan Monnier]

> Isn't easier just to compile the .el file? If you're developing or
> modifying a package, and want to try it at once, create a macro to
> compile it as soon as you save it...

A common case for me is "bzr pull" or "cvs update" or "tla star-merge"
to bring in new changes.  I don't look at all the Elisp files brought in
so byte-compile-file is not convenient, and byte-compilation often leads
to odd results because the `require's read the old .elc files.


        Stefan

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Juanma Barranquero]

On Tue, Jan 27, 2009 at 01:49, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> A common case for me is "bzr pull" or "cvs update" or "tla star-merge"
> to bring in new changes.  I don't look at all the Elisp files brought in
> so byte-compile-file is not convenient, and byte-compilation often leads
> to odd results because the `require's read the old .elc files.

I'm used to do "make cvs-update" just about after every "cvs update".

    Juanma

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Richard M Stallman]

    A common case for me is "bzr pull" or "cvs update" or "tla star-merge"
    to bring in new changes.  I don't look at all the Elisp files brought in
    so byte-compile-file is not convenient, and byte-compilation often leads
    to odd results because the `require's read the old .elc files.

Are you saying that in this case you would want to load the .el files
instead?

If so, I wonder if the right solution is to do something special when loading
a file for `require' in compilation.  Perhaps offer to compile them?

bug#2061: 23.0.60; Add preference to force load of Elisp files when they are newer than corresponding byte-compiled file

[Stefan Monnier]

> If so, I wonder if the right solution is to do something special when loading
> a file for `require' in compilation.

Indeed, that's what I do in my local branch, where the bytecompiler's
handling of require gives precedence to the .el files over the .elc files.

> Perhaps offer to compile them?

It needs to work in batch mode.  And byte-compiling them can easily fall
into circularities.


        Stefan