On Fri, Dec 22, 2017 at 8:30 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> If you mean always keep the source code around in the bytecode file, I'm
> all for that!

No, I'm thinking rather of not keeping the bytecode file around at all,
or to store it in an "internal cache" (which could be kept in the
file-system but could be erased at any point and without any command to
"load a bytecode file").

Sounds like a JIT. Sure, that would address some of the run-time deficiencies I feel are there.


>> > Maybe this could be incorporated into a "safe-load-file" function.
>> Define "safe"
> Okay. Let me call it then "safer" then. And I will define that: detecting
> problems that can be reasonably detected in advance of hitting them instead
> of giving a ¯\_(ツ)_/¯ traceback.

As long as we want to support the case of using a .elc file without the
corresponding .el file, some users would be very annoyed if Emacs-27
prevents them from using a file compiled with Emacs-23 just because
there's a risk that you might hit an error later.

Is there any testing done running emacs code on older bytecode, either in the normal course of testing or before a release? If not, given the expectations that you ascribe to users, there probably should be.

safer-load-file should have the knowledge of what is safe; and this can be as fine grained as one is willing to put into safer-load-file. (Some of this could be determined by the old package and bytecode testing).

If this seems cumbersome, I feel it is part of the responsibility incurred in overhead of compatibility. Other languages  just store a version number in the object file and only allow you to run things that have been compiled by the current major release version compiler.

If you are saying that all Emacs-23 packages and bytecode runs on Emacs-27 then, sure, safer-load-file doesn't need to do anything.


> Recently have come to learn it can be worse because checks are not done on
> bytecode...
> Want to crash emacs immediately without a traceback?  Run
> emacs -batch -Q --eval '(print (#[0 "\300\207" [] 0]))'

Yes, BYTE_CODE_SAFE is not safe enough, agreed.  Patches welcome.

Actually, here, I think the way to go is in the Emacs-in-rust. project. So if I address this, it will probably be via rust. And I will probably include better run-time support for tracebacks that I think is lacking.


> The bytecode file for cl-lib no doubt had in it "Hey, I'm emacs 24."
> and I probably ran that on Emacs 25 where there was an incompatibility
> that can happen between major releases.

I doubt that's the problem [ cl-lib has rather unusual problems in
this respect, and I suspect that you hit one of those, which don't have
anything to do with the bytecode but with the fact that the cl-lib on
GNU ELPA only works on Emacsen that don't come with cl-lib.
It includes hacks to try and avoid the problem, but maybe you've seen
people using odd configurations or older versions of GNU ELPA's cl-lib
where those hacks weren't sufficiently refined.  ]

> If that were the case (and although probably it is not the *only*
> scenario case)  how much nicer would it have been if a safer-load-file
> warned me about running version 24 bytecode.

Do you have actual byte-code generated by Emacs-24's bytecomp.el which
causes a serious error when used on more recent Emacsen?

I think you are missing the point. You said that internal package incompatibilities are kept between major release. So the compile version that is stored in comments in the bytecode file is a proxy for sets of versions of the standard library packages. It's not just about bytecode itself.

A tool like safer-load-file could be told which sets of packages are incompatible and warn when it sees a problem.


For cl-lib, the problem is not that the cl-lib compiled with Emacs-24.1
contains invalid bytecode for Emacs-27, but that for some reason that
cl-lib ended up earlier in the load-path than the cl-lib that comes with
Emacs-27 and it's the bundled cl-lib which should be loaded.

> There are free opcode space available. "apply" could be added is someone
> chooses to add it.

I'm just pointing out differences that illustrate the fact that
tradeoffs were quite different.

Personally I think you are ascribing too much intentionality; this could just as easily be explained by oversight.


> There may be some glibness here. The benefits of using a lower-level
> general-purpose intermediate language like LLVM IR or vmgen is that because
> it a lower level, working with registers and pointers, understands some
> structure layouts, and is more statically typed. So efficiency can be
> gained by specialization.  But if one doesn't break down Lisp_Object and
> uses that in the same way the C interpreter currently does, then I don't
> see why vmgen will be any faster than the current interpreter. (Other than
> the benefit that would also be had by rewriting the interpreter without the
> bloat and compatibility overhead)

The idea is not to rewrite the same thing with vmgen, indeed.  It's to
design a new bytecode (which would hopefully streamline some use cases,
e.g. allow processing function calls more efficiently, maybe also
`apply` more efficiently, which would allow allocating closures with
a single malloc of an object of size proportional to the number of free
variables, ...), and while we're at it, use something like vmgen, so
that we benefit from other people's efforts at improving the efficiency
of the dispatch.

This still all sounds a little loose, and not fully formed.  But if people want to work on this, far be it for me to suggest what others work on. I'm sure there is benefit; and above all, I hope I am not spoiling your fun.


> Thanks for the kind words. It's not something I feel all that knowledgeable
> or qualified to do.

That makes you a good person to write it.
Feel free to ask questions about parts you don't understand or aren't sure.

Sure. When this is more fully finished I'll send out notice.



        Stefan