Updating *.el files and native compilation

Updating *.el files and native compilation

[Eli Zaretskii]

Andrea,

I'd like to better understand what happens with *.eln files when the
corresponding *.el files and the Emacs C code are updated and
recompiled.  There are two use cases here that are relevant:

  . building Emacs as part of development, while keeping old Emacs
    executables

  . updates to locally installed *.el files (e.g., in site-lisp) that
    also target multiple Emacs versions

With byte-compilation, we keep only 1 .elc file for each .el file; if
one starts an Emacs binary which doesn't fit the .elc file, we could
have Lisp errors when invoking the affected Lisp functions, but in
general Emacs should not crash.  However, with native-compilation we
can have several *.eln files in the respective directories (either
native-comp or eln-cache), even if the ABI hash didn't change.  So my
questions are:

  . if the .el file changes in incompatible ways, native-compilation
    will produce a .eln file with a different file name, and each
    Emacs executable will then load the .eln file with which it is
    compatible, is that right?

  . if, for some reason, Emacs loads an incompatible .eln file, then
    some Lisp programs could crash the Emacs session, is that correct?
    If so, how do we make sure such incompatible changes always cause
    a new native compilation that yields a different file name for the
    .eln file?

The upshot of all this is that if one keeps multiple Emacs
executables, it should be safe to invoke each one of them without
risking crashes due to loading incompatible *.eln files that were
produced by other, subtly incompatible Emacs executables.  Is this
indeed safe, or do we have some "gotchas" that still need to be taken
care of?

Thanks.

Re: Updating *.el files and native compilation

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

> Andrea,
>
> I'd like to better understand what happens with *.eln files when the
> corresponding *.el files and the Emacs C code are updated and
> recompiled.  There are two use cases here that are relevant:
>
>   . building Emacs as part of development, while keeping old Emacs
>     executables
>
>   . updates to locally installed *.el files (e.g., in site-lisp) that
>     also target multiple Emacs versions
>
> With byte-compilation, we keep only 1 .elc file for each .el file; if
> one starts an Emacs binary which doesn't fit the .elc file, we could
> have Lisp errors when invoking the affected Lisp functions, but in
> general Emacs should not crash.  However, with native-compilation we
> can have several *.eln files in the respective directories (either
> native-comp or eln-cache), even if the ABI hash didn't change.  So my
> questions are:
>
>   . if the .el file changes in incompatible ways, native-compilation
>     will produce a .eln file with a different file name, and each
>     Emacs executable will then load the .eln file with which it is
>     compatible, is that right?

Hi Eli,

that's correct.

>   . if, for some reason, Emacs loads an incompatible .eln file, then
>     some Lisp programs could crash the Emacs session, is that correct?
>     If so, how do we make sure such incompatible changes always cause
>     a new native compilation that yields a different file name for the
>     .eln file?

Yes but this should not happen, every change that can introduce an
incompatibility has to be accounted in the `comp-abi-hash' computation
and AFAIK ATM it is.

> The upshot of all this is that if one keeps multiple Emacs
> executables, it should be safe to invoke each one of them without
> risking crashes due to loading incompatible *.eln files that were
> produced by other, subtly incompatible Emacs executables.  Is this
> indeed safe, or do we have some "gotchas" that still need to be taken
> care of?

As of today I'm not aware of any gotcha here, if we discover a case of
this we should treat it as bug.

Regards

  Andrea

Re: Updating *.el files and native compilation

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 10 May 2021 07:35:53 +0000
> 
> >   . if, for some reason, Emacs loads an incompatible .eln file, then
> >     some Lisp programs could crash the Emacs session, is that correct?
> >     If so, how do we make sure such incompatible changes always cause
> >     a new native compilation that yields a different file name for the
> >     .eln file?
> 
> Yes but this should not happen, every change that can introduce an
> incompatibility has to be accounted in the `comp-abi-hash' computation
> and AFAIK ATM it is.

Some changes don't require updating comp-abi-hash, but still create
*.eln files with different hashes in its name.  AFAIU, that happens
when the primitives don't change, but the .el file itself changes,
isn't that so?

In any case, are you saying that the only situations where loading and
using a .eln file could crash Emacs are those which are handled by
changing comp-abi-hash?  If so, how can we make sure we never fail to
update comp-abi-hash when that is needed?

> > The upshot of all this is that if one keeps multiple Emacs
> > executables, it should be safe to invoke each one of them without
> > risking crashes due to loading incompatible *.eln files that were
> > produced by other, subtly incompatible Emacs executables.  Is this
> > indeed safe, or do we have some "gotchas" that still need to be taken
> > care of?
> 
> As of today I'm not aware of any gotcha here, if we discover a case of
> this we should treat it as bug.

Sure.

Re: Updating *.el files and native compilation

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 10 May 2021 07:35:53 +0000
>> 
>> >   . if, for some reason, Emacs loads an incompatible .eln file, then
>> >     some Lisp programs could crash the Emacs session, is that correct?
>> >     If so, how do we make sure such incompatible changes always cause
>> >     a new native compilation that yields a different file name for the
>> >     .eln file?
>> 
>> Yes but this should not happen, every change that can introduce an
>> incompatibility has to be accounted in the `comp-abi-hash' computation
>> and AFAIK ATM it is.
>
> Some changes don't require updating comp-abi-hash, but still create
> *.eln files with different hashes in its name.  AFAIU, that happens
> when the primitives don't change, but the .el file itself changes,
> isn't that so?

Correct

> In any case, are you saying that the only situations where loading and
> using a .eln file could crash Emacs are those which are handled by
> changing comp-abi-hash?

Yes

> If so, how can we make sure we never fail to update comp-abi-hash when
> that is needed?

The vast majority of the cases is when some subr is added or a signature
is changed and we account for this automatically.  The rest should be
only about changes specific to the eln load mechanism in comp.c.  I hope
who does that is very well aware of consequences but I don't know if
there's a way we can automatically guard against these changes breaking
the system (I guess there's not).

Regards

  Andrea

Re: Updating *.el files and native compilation

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 10 May 2021 15:44:16 +0000
> 
> > If so, how can we make sure we never fail to update comp-abi-hash when
> > that is needed?
> 
> The vast majority of the cases is when some subr is added or a signature
> is changed and we account for this automatically.  The rest should be
> only about changes specific to the eln load mechanism in comp.c.  I hope
> who does that is very well aware of consequences but I don't know if
> there's a way we can automatically guard against these changes breaking
> the system (I guess there's not).

Maybe you should try writing up the rules for when such a change is
needed.  Don't worry about the English correctness: it is easy to fix
that once the text is written.  But I think no one but you knows the
rules for now, so it should be a good idea to record them.

Thanks.

Re: Updating *.el files and native compilation

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 10 May 2021 15:44:16 +0000
>> 
>> > If so, how can we make sure we never fail to update comp-abi-hash when
>> > that is needed?
>> 
>> The vast majority of the cases is when some subr is added or a signature
>> is changed and we account for this automatically.  The rest should be
>> only about changes specific to the eln load mechanism in comp.c.  I hope
>> who does that is very well aware of consequences but I don't know if
>> there's a way we can automatically guard against these changes breaking
>> the system (I guess there's not).
>
> Maybe you should try writing up the rules for when such a change is
> needed.  Don't worry about the English correctness: it is easy to fix
> that once the text is written.  But I think no one but you knows the
> rules for now, so it should be a good idea to record them.

Okay I can try thinking about something for this, where shall we put it?
In a comment in comp.c (perhaps close to the ABI_VERSION definition)?

  Andrea

Re: Updating *.el files and native compilation

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 10 May 2021 20:45:14 +0000
> 
> > Maybe you should try writing up the rules for when such a change is
> > needed.  Don't worry about the English correctness: it is easy to fix
> > that once the text is written.  But I think no one but you knows the
> > rules for now, so it should be a good idea to record them.
> 
> Okay I can try thinking about something for this, where shall we put it?
> In a comment in comp.c (perhaps close to the ABI_VERSION definition)?

Yes, that would be a good place, I think.