More reliable byte compilation, take 45

More reliable byte compilation, take 45

[Lars Ingebrigtsen]

Various schemes to make byte compilation more reliable has been mooted,
but none have been practical.

To recap: The problem is that if we change a macro in foo.el, and bar.el
uses that macro, bar.el won't be recompiled after changing the macro in
foo.el.

A defensive solution would be to recompile all libraries that require a
library, building a dependency tree from require/autoloads/etc, but the
problem is then that if you edit anything in subr.el, then you have to
recompile all the .el files, basically.  So it's not practical.

So we want a solution that's 1) fairly fast and 2) somewhat reliable.  A
100% solution isn't necessary -- just something that's less "well, just
say 'make bootstrap' then".

So here's the idea of the day:

make macro-recompile

would go through all the .el files and just look for "^\(defmacro ([^
\t\n]+)" (and defsubst and cl-defstruct and probably a couple of other
things, but not a whole lot more), and then do a sha256 of the region
from the start of the match and until the next "^(".  It would maintain
a database of these macro names/hashes.  If a macro changes its hash, it
would then just grep through all the .el files for the macro name, and
delete the .elc files where it fines matches.

The point here is to make it feasible to alter macros and see whether
there's any horrible fall-out immediately, with some confidence.  So it
has to be fast, and not result in too much recompilation.

So writing a little C helper program to do the hashing stuff might
possibly be the way to go here.

Does this sound feasible and useful?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: More reliable byte compilation, take 45

[Clément Pit-Claudel]

On 10/4/21 6:51 AM, Lars Ingebrigtsen wrote:
> Does this sound feasible and useful?

It sounds useful for the development of Emacs itself, yes — but the problem you're describing is one we have with packages on *ELPA, too, and I think this wouldn't help there?

Re: More reliable byte compilation, take 45

[Steingold]

> * Lars Ingebrigtsen <ynefv@tahf.bet> [2021-10-04 12:51:19 +0200]:
>
> A defensive solution would be to recompile all libraries that require a
> library, building a dependency tree from require/autoloads/etc, but the
> problem is then that if you edit anything in subr.el, then you have to
> recompile all the .el files, basically.  So it's not practical.

Nevertheless, this is TRT (and your solution is far too complicated, IMO).

Another solution was adopted by CLISP.
(https://clisp.sourceforge.io/impnotes/require.html#lib-files)

Briefly, compilation of `foo.el` should produce 2 files:

1. `foo.elc`, as now - this is the code whose loading is functionally
   equivalent to loading `foo.el`

2. `foo.ell` (lib) - this contains only the compile-time dependencies
   (i.e., compiled `defvar`, `defconst`, `defmacro`, and `defsubst`
   definitions and function declarations)

When the byte compiler sees `(require 'foo)`, it will check that `foo.ell`
has changed since last loaded and will reload it automatically.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1894
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://jihadwatch.org https://honestreporting.com https://www.memritv.org
A good programmer: language change is easy, editor change is ... huh?!

Re: More reliable byte compilation, take 45

[Lars Ingebrigtsen]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> It sounds useful for the development of Emacs itself, yes — but the
> problem you're describing is one we have with packages on *ELPA, too,
> and I think this wouldn't help there?

No, doesn't help there.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: More reliable byte compilation, take 45

[Lars Ingebrigtsen]

Steingold <sds@gnu.org> writes:

> When the byte compiler sees `(require 'foo)`, it will check that `foo.ell`
> has changed since last loaded and will reload it automatically.

Nothing requires subr.el -- it's prebuilt, but everything uses functions
from it.  Recompiling everything after changing anything in subr.el
would make development impossible.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: More reliable byte compilation, take 45

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 784 bytes --]

Cl¨¦ment Pit-Claudel <cpitclaudel@gmail.com> writes:


A more important fix that is likely critical for the 28.0 release is to
ensure that when native-compilation is on, native compile starts from
the .elc files, rather than the .el files ---
for now I've turned off native compilation because otherwise things
break -- has already been documented via multiple threads a few months
ago.

> On 10/4/21 6:51 AM, Lars Ingebrigtsen wrote:
>> Does this sound feasible and useful?
>
> It sounds useful for the development of Emacs itself, yes ¡ª but the
> problem you're describing is one we have with packages on *ELPA, too,
> and I think this wouldn't help there?
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: More reliable byte compilation, take 45

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 11:16:34 -0700
> 
> A more important fix that is likely critical for the 28.0 release is to
> ensure that when native-compilation is on, native compile starts from
> the .elc files, rather than the .el files ---
> for now I've turned off native compilation because otherwise things
> break -- has already been documented via multiple threads a few months
> ago.

Why do you assume that starting from .elc will have fewer problems
than starting from .el?

Instead of turning off native compilation, I'd urge people to use it
as much as possible and report problems -- that way we will make it
stable much sooner.

Re: More reliable byte compilation, take 45

[Steingold]

> * Lars Ingebrigtsen <ynefv@tahf.bet> [2021-10-04 19:35:36 +0200]:
>
> Steingold <sds@gnu.org> writes:
>
>> When the byte compiler sees `(require 'foo)`, it will check that `foo.ell`
>> has changed since last loaded and will reload it automatically.
>
> Nothing requires subr.el -- it's prebuilt, but everything uses functions
> from it.  Recompiling everything after changing anything in subr.el
> would make development impossible.

Change subr.el.
Recompile it.
See if subr.ell has changed (the compiler should not change subr.ell
unless necessary, including the timestamp).
If it did, everything needs to be recompiled.
If it did not, no action is necessary.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1894
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://www.peaceandtolerance.org/ https://camera.org
This message is rot13 encrypted (twice!); reading it violates DMCA.

Re: More reliable byte compilation, take 45

[Stefan Monnier]

> Why do you assume that starting from .elc will have fewer problems
> than starting from .el?

Converting a `.elc` file to a `.eln` file can be done by a pure function
with no need for any extra information.

In contrast, converting a `.el` to a `.elc` or `.eln` requires extra
information (typically extra files that define macros and functions used
during macro expansion, plus `load-path` settings to find those files, or
commands to load those files, sometimes in a particular order).

Often this extra information follows the usual practice, in which case
Emacs will find that info without us noticing, but not always (in which
case you'll typically need a special Makefile to build the package, or
an extra ELisp file dedicated to compiling the other files).

ELPA has encouraged packages to better follow the usual practice, but
we're still not out of the woods.


        Stefan

Re: More reliable byte compilation, take 45

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: "T.V Raman" <raman@google.com>,  cpitclaudel@gmail.com,
>   emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 15:18:30 -0400
> 
> > Why do you assume that starting from .elc will have fewer problems
> > than starting from .el?
> 
> Converting a `.elc` file to a `.eln` file can be done by a pure function
> with no need for any extra information.

I very much doubt that.

> In contrast, converting a `.el` to a `.elc` or `.eln` requires extra
> information (typically extra files that define macros and functions used
> during macro expansion, plus `load-path` settings to find those files, or
> commands to load those files, sometimes in a particular order).

I have no reason to believe that the problems we see with native
compilation are related to these aspects.  Do you have evidence of
that?

Re: More reliable byte compilation, take 45

[Stefan Monnier]

>> Converting a `.elc` file to a `.eln` file can be done by a pure function
>> with no need for any extra information.
> I very much doubt that.

You might be right, there might be corner cases I'm not taking into
account, but by and large it should be the case.  At least much more so
than for `.el` files where it's clearly a real problem.

> I have no reason to believe that the problems we see with native
> compilation are related to these aspects.

I'm not sure which problems you're referring to.
Within the context of Emacs's own ELisp files, this should never be
an issue.

> Do you have evidence of that?

IIRC, in Raman's case the evidence was that the `.eln` file was calling
a macro as if it were a function, and that this didn't happen when using
the `.elc` file.  These problems typically happen when a `.el` file is
miscompiled because some other ELisp file was not loaded beforehand.

Admittedly, I don't have actual evidence, but it seemed to hint at that.

In any case, the problem with compiling from the `.el` file instead of
from the `.elc` file is real and it's easy to trigger it on purpose.
The only question for me is how often it will bite us out there in the
real world.

There's a chance that it will simply encourage people to fix their ELisp
files to better follow the (undocumented) conventions, so to some degree
I agree that it might be preferable not to fix this issue.


        Stefan

Re: More reliable byte compilation, take 45

[Lars Ingebrigtsen]

Steingold <sds@gnu.org> writes:

> See if subr.ell has changed (the compiler should not change subr.ell
> unless necessary, including the timestamp).

Right, because the subr.ell only contains macros (and the like).  So the
suggestion here is basically the same as mine, only with a lot more
files spread around.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: More reliable byte compilation, take 45

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: raman@google.com,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 16:13:33 -0400
> 
> >> Converting a `.elc` file to a `.eln` file can be done by a pure function
> >> with no need for any extra information.
> > I very much doubt that.
> 
> You might be right, there might be corner cases I'm not taking into
> account, but by and large it should be the case.  At least much more so
> than for `.el` files where it's clearly a real problem.
> 
> > I have no reason to believe that the problems we see with native
> > compilation are related to these aspects.
> 
> I'm not sure which problems you're referring to.

The Subject is "more reliable byte compilation", and we are talking
specifically about more reliable native compilation.  So the problems
I was referring to are problems with compilation reliability.

> > Do you have evidence of that?
> 
> IIRC, in Raman's case the evidence was that the `.eln` file was calling
> a macro as if it were a function, and that this didn't happen when using
> the `.elc` file.  These problems typically happen when a `.el` file is
> miscompiled because some other ELisp file was not loaded beforehand.

And that's all?  Then why not fix that problem right away, it sounds
like something that should be easy to fix.  (Though I'm not sure I
understand why "some other ELisp file was not loaded beforehand" -- is
that a case of a missing 'require' or 'eval-when-compile' or somesuch?)

> In any case, the problem with compiling from the `.el` file instead of
> from the `.elc` file is real and it's easy to trigger it on purpose.

Examples?  Are you saying that it's impossible to solve those
problems?

> There's a chance that it will simply encourage people to fix their ELisp
> files to better follow the (undocumented) conventions, so to some degree
> I agree that it might be preferable not to fix this issue.

Ah, so the problem is with buggy *.el files, and only with them?

Re: More reliable byte compilation, take 45

[Stefan Monnier]

> And that's all?  Then why not fix that problem right away, it sounds
> like something that should be easy to fix.  (Though I'm not sure I
> understand why "some other ELisp file was not loaded beforehand" -- is
> that a case of a missing 'require' or 'eval-when-compile' or somesuch?)

Often the problem is with packages where you basically have to load
all/most the files before you can call any of its functions (because of
mutual dependencies between the files) and where the authors never
compile the code or only ever compile the code from a running session
where the files have already been loaded.

Fixing those can be a pain in the rear, not only because of the amount
of code you need to move between files to disentangle the dependencies,
but also because it requires educating the authors about it (they may
not see the value/importance).

> Ah, so the problem is with buggy *.el files, and only with them?

Yes, tho the authors may disagree about the characterization of "buggy",
in the sense that it works reliably *if* you compile the code by running
their Makefile.


        Stefan

Re: More reliable byte compilation, take 45

[Adam Porter]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> And that's all?  Then why not fix that problem right away, it sounds
>> like something that should be easy to fix.  (Though I'm not sure I
>> understand why "some other ELisp file was not loaded beforehand" -- is
>> that a case of a missing 'require' or 'eval-when-compile' or somesuch?)
>
> Often the problem is with packages where you basically have to load
> all/most the files before you can call any of its functions (because of
> mutual dependencies between the files) and where the authors never
> compile the code or only ever compile the code from a running session
> where the files have already been loaded.
>
> Fixing those can be a pain in the rear, not only because of the amount
> of code you need to move between files to disentangle the dependencies,
> but also because it requires educating the authors about it (they may
> not see the value/importance).
>
>> Ah, so the problem is with buggy *.el files, and only with them?
>
> Yes, tho the authors may disagree about the characterization of "buggy",
> in the sense that it works reliably *if* you compile the code by running
> their Makefile.

FWIW, I mostly solved this kind of problem in my own packages by using
makem.sh to lint compilation, which now compiles each file separately:

https://github.com/alphapapa/makem.sh

It detects project Elisp files automatically, so it doesn't require
writing a Makefile or other configuration, which makes it easier for
package authors to use.

Re: More reliable byte compilation, take 45

[Steingold]

> * Lars Ingebrigtsen <ynefv@tahf.bet> [2021-10-05 09:14:30 +0200]:
>
> Steingold <sds@gnu.org> writes:
>
>> See if subr.ell has changed (the compiler should not change subr.ell
>> unless necessary, including the timestamp).
>
> Right, because the subr.ell only contains macros (and the like).  So
> the suggestion here is basically the same as mine, only with a lot
> more files spread around.

here is your suggestion:

> go through all the .el files and just look for "^\(defmacro ([^
> \t\n]+)" (and defsubst and cl-defstruct and probably a couple of other
> things, but not a whole lot more), and then do a sha256 of the region
> from the start of the match and until the next "^(".  It would maintain
> a database of these macro names/hashes.  If a macro changes its hash, it
> would then just grep through all the .el files for the macro name, and
> delete the .elc files where it fines matches.

I think the extra "compiled header" file is _way_ simpler and more
transparent solution.


-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1894
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://ij.org/ https://ffii.org https://memri.org https://jihadwatch.org
Illiterate?  Write today, for free help!

Re: More reliable byte compilation, take 45

[Steingold]

> * Steingold <fqf@tah.bet> [2021-10-04 12:03:34 -0400]:
>
>> * Lars Ingebrigtsen <ynefv@tahf.bet> [2021-10-04 12:51:19 +0200]:
>>
>> A defensive solution would be to recompile all libraries that require a
>> library, building a dependency tree from require/autoloads/etc, but the
>> problem is then that if you edit anything in subr.el, then you have to
>> recompile all the .el files, basically.  So it's not practical.
>
> Nevertheless, this is TRT (and your solution is far too complicated, IMO).
>
> Another solution was adopted by CLISP.
> (https://clisp.sourceforge.io/impnotes/require.html#lib-files)
>
> Briefly, compilation of `foo.el` should produce 2 files:
>
> 1. `foo.elc`, as now - this is the code whose loading is functionally
>    equivalent to loading `foo.el`
>
> 2. `foo.ell` (lib) - this contains only the compile-time dependencies
>    (i.e., compiled `defvar`, `defconst`, `defmacro`, and `defsubst`
>    definitions and function declarations)

actually, "elh" is more intuitive - this is "compiled header" file,
similar to *.h in C.

> When the byte compiler sees `(require 'foo)`, it will check that `foo.ell`
> has changed since last loaded and will reload it automatically.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1894
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://camera.org https://memri.org https://iris.org.il https://jihadwatch.org
"Sustainability" without "population growth control" is hypocrisy.

Re: More reliable byte compilation, take 45

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: raman@google.com,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 05 Oct 2021 09:27:24 -0400
> 
> > And that's all?  Then why not fix that problem right away, it sounds
> > like something that should be easy to fix.  (Though I'm not sure I
> > understand why "some other ELisp file was not loaded beforehand" -- is
> > that a case of a missing 'require' or 'eval-when-compile' or somesuch?)
> 
> Often the problem is with packages where you basically have to load
> all/most the files before you can call any of its functions (because of
> mutual dependencies between the files) and where the authors never
> compile the code or only ever compile the code from a running session
> where the files have already been loaded.

Wouldn't local no-native-compile be the proper solutions for those
files?

Re: More reliable byte compilation, take 45

[Stefan Monnier]

>> 2. `foo.ell` (lib) - this contains only the compile-time dependencies
>>    (i.e., compiled `defvar`, `defconst`, `defmacro`, and `defsubst`
>>    definitions and function declarations)
>
> actually, "elh" is more intuitive - this is "compiled header" file,
> similar to *.h in C.

Combine the two and use `.hell`?


        Stefan "sorry, couldn't resist"