Source file '.../killer-source.el' newer than byte-compiled file

Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Often when I do some changes to Elisp files and
then run the byte-compiler with the following
Makefile [1] I get this message for several
files, but not all of them, and not always:

    Source file '~/path/to/killer-source.el' newer
    than byte-compiled file

But what is the purpose of this message and why
is it so capricious in terms of when
it appears?

What it says isn't radical, because isn't that
the way make(1) works, you have a target, and
a source, and if the source is newer
(i.e. more recently modified) than the target,
that means it has been edited since last
compilation, i.e., (re)compilation
should happen?

Why am I told this specifically, and why for
just certain files, not every one I edit?

Has it to do with files being `require'd before
compilation has gotten that far? So if a.el is
compiled into a.elc, but a requires b, only at
that time b.elc hasn't been compiled *but* b.el
has been edited?

If this theory is correct (?), do you even need
to compile one more time to get rid of the
message? Because at that time a.el hasn't been
changed, so it won't be recompiled, and thus
won't get access to the new b.elc anyway?

Only this in turn doesn't matter either in
practise as long as there is no change to the
interface of b, because when the user uses
b thru a, it'll still get the latest b.elc?


[1] http://user.it.uu.se/~embe8573/emacs-init/Makefile

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Tue, 23 Apr 2019 19:39:34 +0200
> 
> Often when I do some changes to Elisp files and
> then run the byte-compiler with the following
> Makefile [1] I get this message for several
> files, but not all of them, and not always:
> 
>     Source file '~/path/to/killer-source.el' newer
>     than byte-compiled file
> 
> But what is the purpose of this message and why
> is it so capricious in terms of when
> it appears?

Its purpose is to tell you that Emacs loads a .elc file that is older
than the corresponding .el file.  See load-prefer-newer.

> Why am I told this specifically, and why for
> just certain files, not every one I edit?

It depends on the order of byte-compilation.

> Has it to do with files being `require'd before
> compilation has gotten that far? So if a.el is
> compiled into a.elc, but a requires b, only at
> that time b.elc hasn't been compiled *but* b.el
> has been edited?

Yes.

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Eli Zaretskii wrote:

> Its purpose is to tell you that Emacs loads
> a .elc file that is older than the
> corresponding .el file.
> See load-prefer-newer.

OK, so I should have that `t'!

But if so, sometimes the b.el and not the b.elc
will be `require'd by a.el->a.elc (whichever is
correct to say does the the requiring - a.el?).

Is there a downside to this, assuming the
latest version is always the better, and b.elc
*is* compiled eventually?

>> Has it to do with files being `require'd
>> before compilation has gotten that far?
>> So if a.el is compiled into a.elc, but
>> a requires b, only at that time b.elc hasn't
>> been compiled *but* b.el has been edited?
>
> Yes.

If there is a downside, i.e. it _is_ better to
load the byte-compiled code (assuming it to be
newer than the source), what should one do?
touch(1) a.el and byte-compile again?
Because then, there should be no problem with
b.elc being out of date! If this is correct,
perhaps the byte-compiler should tell what file
has the problem to begin with?

Or even not do anything about it (a.el), and
then one would simply compile twice? Hm, maybe
this can get messy if one have lots to
require... - but as long as they are not
circular it should work out fine, right?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

>> Its purpose is to tell you that Emacs loads
>> a .elc file that is older than the
>> corresponding .el file.
>> See load-prefer-newer.
>
> OK, so I should have that `t'! [...]
>
> Is there a downside to this, assuming the
> latest version is always the better, and
> b.elc *is* compiled eventually?

Wait... there is the obvious downside which
wasn't obvious enough for me to think of!
How can one assume that b.el is better than the
old b.elc *before* it has been compiled?

No, this idea stinks!

a.el should hold it compilation until all its
`require'd files are byte-compiled!

But how is that done?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Tue, 23 Apr 2019 20:39:36 +0200
> 
> a.el should hold it compilation until all its
> `require'd files are byte-compiled!
> 
> But how is that done?

You can concoct Make rules for that, but you will have trouble if you
have circular dependencies.

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Eli Zaretskii wrote:

>> a.el should hold it compilation until all
>> its `require'd files are byte-compiled!
>> 
>> But how is that done?
>
> You can concoct Make rules for that

... how? You mean manually?

If so, I'm afraid that's totally out of the
question, as I have 64 `provide's and 284
`require's!

The Makefile [1] don't compile anything on an
explicit file-to-file basis (except for
~/.emacs), everything else it is all shell
pattern matching with all files that are %.el,
should be %.elc as well.

> but you will have trouble if you have
> circular dependencies.

No, I don't have circular dependencies, and
I don't think I can have them, either.
Actually I'm gonna try that right now...

No, I can't:

    In toplevel form:
    test-a.el:1:1:Error: Recursive ‘require’ for feature ‘test-b’

    In toplevel form:
    test-b.el:1:1:Error: Recursive ‘require’ for feature ‘test-a’


[1] http://user.it.uu.se/~embe8573/emacs-init/Makefile

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Nick Dokos]

Emanuel Berg <moasenwood@zoho.eu> writes:

> Eli Zaretskii wrote:
>
>>> a.el should hold it compilation until all
>>> its `require'd files are byte-compiled!
>>> 
>>> But how is that done?
>>
>> You can concoct Make rules for that
>
> ... how? You mean manually?
>
> If so, I'm afraid that's totally out of the
> question, as I have 64 `provide's and 284
> `require's!
>

This is called "topological sorting" and you can use
the tsort(1) program to do that, although you will have
to prepare an input file that describes the dependencies
in the format that tsort expects:

(info "(coreutils) tsort invocation")

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Nick Dokos wrote:

>>> You can concoct Make rules for that
>>
>> ... how? You mean manually? If so, I'm
>> afraid that's totally out of the question,
>> as I have 64 `provide's and 284 `require's!
>>
>
> This is called "topological sorting" and you
> can use the tsort(1) program to do that,
> although you will have to prepare an input
> file that describes the dependencies in the
> format that tsort expects:
>
> (info "(coreutils) tsort invocation")

Cool stuff, straight CS, but if it has to be
done manually, it's out of the question, as
I `require' stuff back and forth all the time!
To have a file to describe that would be so
much extra work, and not only that, but the
worst part, boring and error-prone extra work.
Unless you have a tool to analyze my Elisp,
grep the requires, parse the catch, and direct
it to a file in the format that's needed...

Besides, mustn't I then even hook it to make(1)
as well, or is there yet another tool for that?

So no, I don't think I'll do that, but it is
still interesting, so if you have an example
I'd like to see it, of course :)

OK, hands down, this error doesn't come that
often. When it does, if one could only know
_what files_ require an old .elc, one could
simply touch(1) those one or two files
and recompile!

So again, if the byte-compiler knows this (the
name of the file), it would help if it
outputted them! With errors and warnings, this
always happens. Why not now as well?

However, I think I can modify the Makefile so
that, whenever there is this warning, output
the name of the file that is being compiled!

The simplest solution, right?

But I'd still like to know what it means in
theory as well as possible practice when
a requires an old b, and c requires a new b,
and a, b and c are part of the same
Emacs system?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

> However, I think I can modify the Makefile so
> that, whenever there is this warning, output
> the name of the file that is being compiled!
>
> The simplest solution, right?

In the Makefile [1]

%.elc: %.el
	$(byte-compile) $< $(sed-filter) > $(error-file)
	if [ -s $(error-file) ]; then echo -n "\n$<: "; cat $(error-file); fi
	rm -f $<~

Test run:

$ touch a-require-b.el b-provide-b.el
  c-has-an-error.el d-has-no-error.el
$ emake

a-require-b.el: Source file
‘/home/incal/.emacs.d/emacs-init/b-provide-b.el’
newer than byte-compiled file

c-has-an-error.el: 
In toplevel form:
c-has-an-error.el:1:1:Warning: value returned from (/ 1 0) is unused

SKILLS!


[1] http://user.it.uu.se/~embe8573/emacs-init/Makefile

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Wed, 24 Apr 2019 00:42:42 +0200
> 
> Eli Zaretskii wrote:
> 
> >> a.el should hold it compilation until all
> >> its `require'd files are byte-compiled!
> >> 
> >> But how is that done?
> >
> > You can concoct Make rules for that
> 
> ... how? You mean manually?

That's one way, yes.  Works well for small numbers of dependent files.

> If so, I'm afraid that's totally out of the
> question, as I have 64 `provide's and 284
> `require's!

Then you could try writing a Lisp program to produce Make rules from
the "provide" and "require" lines.  Assuming you don't have
complications like 'eval-when-compile', 'eval-and-compile', etc.

> > but you will have trouble if you have
> > circular dependencies.
> 
> No, I don't have circular dependencies, and
> I don't think I can have them, either.
> Actually I'm gonna try that right now...
> 
> No, I can't:

Not easily, no.

`eval-when-compile' and circular dependencies (was: Re: Source file '.../killer-source.el' newer than byte-compiled file)

[Emanuel Berg]

Eli Zaretskii wrote:

> Assuming you don't have complications like
> 'eval-when-compile', 'eval-and-compile', etc.

No, I don't have them, I think ... no, not
a single one of either!

I don't understand the purpose of them either,
too advanced for my style anyway I think.
From the docstring of `eval-when-compile':

    Like ‘progn’, but evaluates the body at compile
    time if you’re compiling. Thus, the result of
    the body appears to the compiler as
    a quoted constant.

Okay...? ~"The result appears to the compiler
as a quoted constant..." <:|

>>> but you will have trouble if you have
>>> circular dependencies.
>>
>> No, I don't have circular dependencies, and
>> I don't think I can have them, either.
>> Actually I'm gonna try that right now...
>> 
>> No, I can't
>
> Not easily, no.

Why would you want them? And how do you make it
work, if so?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: `eval-when-compile' and circular dependencies (was: Re: Source file '.../killer-source.el' newer than byte-compiled file)

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Thu, 25 Apr 2019 05:25:28 +0200
> 
> >>> but you will have trouble if you have
> >>> circular dependencies.
> >>
> >> No, I don't have circular dependencies, and
> >> I don't think I can have them, either.
> >> Actually I'm gonna try that right now...
> >> 
> >> No, I can't
> >
> > Not easily, no.
> 
> Why would you want them?

We don't, it just happens.

> And how do you make it work, if so?

You could look at Eshell as one example, if you really want to
understand.  The solution involves using eval-when-compile and its
ilk, I think.

Re: `eval-when-compile' and circular dependencies (was: Re: Source file '.../killer-source.el' newer than byte-compiled file)

[Emanuel Berg]

Eli Zaretskii wrote:

> You could look at Eshell as one example, if
> you really want to understand. The solution
> involves using eval-when-compile and its ilk,
> I think.

OK, but this isn't the only use case for
`eval-when-compile', right?

Because I've heard that two wrongs don't
make a ... right!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: `eval-when-compile' and circular dependencies (was: Re: Source file '.../killer-source.el' newer than byte-compiled file)

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Thu, 25 Apr 2019 09:03:05 +0200
> 
> Eli Zaretskii wrote:
> 
> > You could look at Eshell as one example, if
> > you really want to understand. The solution
> > involves using eval-when-compile and its ilk,
> > I think.
> 
> OK, but this isn't the only use case for
> `eval-when-compile', right?

No, of course not.  Its use cases are described in the ELisp manual.

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Tue, 23 Apr 2019 20:32:45 +0200
> 
> Eli Zaretskii wrote:
> 
> > Its purpose is to tell you that Emacs loads
> > a .elc file that is older than the
> > corresponding .el file.
> > See load-prefer-newer.
> 
> OK, so I should have that `t'!

If you really want that.  Personally, I'm not sure.

> Is there a downside to this, assuming the
> latest version is always the better, and b.elc
> *is* compiled eventually?

That assumption might be false.  For example, you may be working on
a.el, and it might have bugs.

> If there is a downside, i.e. it _is_ better to
> load the byte-compiled code (assuming it to be
> newer than the source), what should one do?

Just ignore those messages, really.  They are mostly harmless.  You
could compile the dependent file again if you want, but that is rarely
needed, mainly when they define macros that were changed, and those
macros are used by the files which require a.el.

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Eli Zaretskii wrote:

> Just ignore those messages, really. They are
> mostly harmless. You could compile the
> dependent file again if you want, but that is
> rarely needed, mainly when they define macros
> that were changed, and those macros are used
> by the files which require a.el.

I think you are right, because if you weren't,
this should have broken my system one zillion
times by now, and that hasn't happened, but
I don't understand why. Because think of this
scenario:

First, a.el is the file that `require' b, so
b is `provide'd in b.el.

Both a.el and b.el are edited, a.el w/o bugs,
b.el *with* bugs.

a.el is compiled before b.el is compiled.

When a.el is compiled, it requires b, so,
depending on the variable `load-prefer-newer'
it either gets the buggy b.el _or_ the old
b.elc. Because I never touched the variable,
which defaults to `nil', a.el will get the
old b.elc.

Now, because b.elc exists, it doesn't come with
bugs, as I would have seen that when I compiled
it last time. So a gets a bug-free, but old,
version of b.

Now the compiler comes to b.el and tells me
there is a bug in the file. I fix that bug and
it compiles into a new, bug-free b.elc.
However a.elc is newer than a.el, so a.el won't
recompile to get the new version!

So c.el compiles, and it also requires b.
Now there is no warning from the byte-compiler
since b.elc is newer than b.el. _But_ c got
another version of b.elc than did a!

What does this situation mean?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Source file '.../killer-source.el' newer than byte-compiled file

[Emanuel Berg]

Sometimes (like now) it even says that files in
~/.emacs.d/elpa suffer from this! But I never
touch those source files nor byte-compile them
or remove their byte-compiled versions. So that
must be some unrelated, still similar issue -

    Source file ‘/home/incal/.emacs.d/elpa/slime-20190319.930/slime.el’ newer than byte-compiled file
    Source file ‘/home/incal/.emacs.d/elpa/slime-20190319.930/slime.el’ newer than byte-compiled file

I *do* `require' slime twice in my own Elisp so
the theory in my previous post seems to check
out so far :)

The message is correct as well:

    $ ~/.emacs.d/elpa/slime-20190319.930 ll slime.el slime.elc
    -rw-r--r-- 1 incal incal 286K Apr 19 16:42 slime.el
    -rw-r--r-- 1 incal incal 309K Apr 19 16:32 slime.elc

I suppose I could touch(1) slime.elc if I'm
confident I didn't do anything to slime.el...

Or can one do anything from the package manager
interface, from Emacs?

-- 
underground experts united
http://user.it.uu.se/~embe8573