From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Rocky Bernstein Newsgroups: gmane.emacs.devel Subject: Re: Bytecode interoperability: the good and bad Date: Fri, 22 Dec 2017 12:41:30 -0500 Message-ID: NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a11466198b7cfcb0560f15309" X-Trace: blaine.gmane.org 1513964427 16607 195.159.176.226 (22 Dec 2017 17:40:27 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 22 Dec 2017 17:40:27 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Dec 22 18:40:23 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eSRIv-0003aK-8u for ged-emacs-devel@m.gmane.org; Fri, 22 Dec 2017 18:40:17 +0100 Original-Received: from localhost ([::1]:59705 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eSRKt-0005XQ-Ln for ged-emacs-devel@m.gmane.org; Fri, 22 Dec 2017 12:42:19 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57974) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eSRKA-0005Wz-4M for emacs-devel@gnu.org; Fri, 22 Dec 2017 12:41:35 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eSRK8-0001ha-J2 for emacs-devel@gnu.org; Fri, 22 Dec 2017 12:41:34 -0500 Original-Received: from mail-qk0-x22b.google.com ([2607:f8b0:400d:c09::22b]:33277) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eSRK8-0001hO-Br for emacs-devel@gnu.org; Fri, 22 Dec 2017 12:41:32 -0500 Original-Received: by mail-qk0-x22b.google.com with SMTP id x7so18258320qkb.0 for ; Fri, 22 Dec 2017 09:41:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to; bh=GzeJLE/wufd6xuFvwDtVYzlvKbW0313xqJcLZWZNsB0=; b=Xqvj6AIt38C20rQrbmpOCe+pF1EG+dH9uWaTQThANpBqjPGdLpoF0oCLD0mijhQ0qo LdaMWrc4iGxB3y6dAPlNice9FQ1t8wPvhGQG49Hji+fksVjOM00kZkhh1N32MBG04xIB nmu7dCZPcHY7aPtLhvMWRI9whd3876MjuVtsyIVy1j0Gjkt2ntzupeM0vQsRCo9zVOgR ohEqK3LFv+W8QI82tZ+IYLedLRgCnSifRT/oWnlEJXGbrAafaIBdl5avO3yk6AtY9WkF jzEMaDYiOK+iRc9q2CGMOAc0B2b8ExTUqiy+RghJ6ehfU2ZbtwK80b7pGvLhWD1S07Oa ZIGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=GzeJLE/wufd6xuFvwDtVYzlvKbW0313xqJcLZWZNsB0=; b=sNZMYFZ5TtG8KSYyvmRxX4Hn6zllo4G8qg07zSskdPc4CIf247nNxGiiqiMPRtQjQa YBs0QPAPadX+MFfwyRRqvCALiwFJVLM9bB7muzoSsTiU5X2hEr2r2PStFw1yTMngxENt wMCW7EGX9Fhr55lzSIKPScEJWgzogZWtWaTzoJ12E0OykD17E6NPil3RQhbwSGb3VMP3 +5bAV7A+IhAwZK8caJgX9NYn/p5ce18c5B4TWnOpLKFESj4W9Le1m/ruk9f31PnG6FZ6 nBjUo+xgdC4opxgT6q8ASaZAURg7unH0n2+qkjvyyf5hdNbregjxTh2Z+wGpslbmM7e4 Qe/g== X-Gm-Message-State: AKGB3mIylRKL2P+JET3z8wc4USArp9kLjyiQVuxuLmHHha/LdS/z6U7t 7cn/NRDbM9P3WrXfhF7wywuf8x3qEDGs16XlHH8uGiwQ X-Google-Smtp-Source: ACJfBosEJe4lE8j3tNV0EZ4EkkM1qLxJiuntHrWXtRTcqR8IliVy1nydHvt4P2FrjZhKV/UPo75YVTNGujqMUbXAy14= X-Received: by 10.55.215.144 with SMTP id t16mr21302389qkt.15.1513964491491; Fri, 22 Dec 2017 09:41:31 -0800 (PST) Original-Received: by 10.12.197.8 with HTTP; Fri, 22 Dec 2017 09:41:30 -0800 (PST) X-Google-Sender-Auth: twDsSXlQj6vMeUIDE1yR2rSeupQ X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400d:c09::22b X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:221340 Archived-At: --001a11466198b7cfcb0560f15309 Content-Type: text/plain; charset="UTF-8" On Fri, 22 Dec 2017 09:08:33 -050 Stefan Monnier advised: > In other kinds of bytecode such as the one for C Python, a bytecode > version > > number is stored in the bytecode file. When there is a change to the > > bytecode, that number is changed. > > So far, the only changes that have been made to the byte-code language > is to add new (previously unused) byte codes. So from this perspective > we have always maintained backward compatibility (you can run a .elc > compiled with an older Emacs). > While this is a nice intention, it isn't always true. And it is not with downsides. In the "not true" department, there are instructions 0153 scan_buffer and 0163 set_mark which aren't handled in the current interpreter sources in bytecode.c And as pipcet points out, there is this in lread.c: if (! version || version >= 22) readevalloop (Qget_file_char, &input, hist_file_name, 0, Qnil, Qnil, Qnil, Qnil); else { /* We can't handle a file which was compiled with byte-compile-dynamic by older version of Emacs. */ specbind (Qload_force_doc_strings, Qt); readevalloop (Qget_emacs_mule_file_char, &input, hist_file_name, 0, Qnil, Qnil, Qnil, Qnil); } In the "not without downsides" department, this means that when someone looks at the bytecode interpreter, it is filled with garbage and bloat. This has to have a technology debt associated with it. We do not aim to maintain forward compatibility (so whether a .elc file > compiled with a more recent Emacs will work is not guaranteed), although > it sometimes does work. When encountering an unknown byte-code, Emacs > signals an error, so it shouldn't cause a crash nor "something unintended". > It is likely that the code that purports to handle obsolete (or no longer emitted) instructions is broken, since I doubt any of this behavior is tested. Subtle changes in the semantics of instructions can cause unintended effects. > Compatibility problems with .elc files compiled with other Emacs > versions can also come from macros, and those tend to be more frequent > than the problems introduced by changes to the byte-code. So detecting > a different byte-code version is not sufficient to catch the most common > problems anyway. > My understanding of how this work in a more rational way would be that there shouldn't be incompatible changes between major releases. So I would hope that incompatible macro changes wouldn't happen within a major release but between major releases, the same as I hope would be the case for bytecode changes. If someone is up for it, a possibly interesting program to write might be a bytecode lint and report tool that shows the meta comment in bytecode to describe what version of Emacs the bytecode was compiled under (comparing with the current loaded version), what level of optimization is reported. Possibly a scan over the instructions to look for incompatibility both in the forward and backward direction. It might optionally have knowledge of specific version incompatibilities say because of macro changes between versions. Maybe this could be incorporated into a "safe-load-file" function. > FWIW, I think Emacs deserves a new Elisp compilation system (either > a new kind of bytecode (maybe using something like vmgen), or a JIT or > something): the bytecode we use is basically identical to the one we had > 20 years ago, yet the tradeoffs have changed substantially in the > mean time. > I would be interested in elaboration here about what specific trade offs you mean. >From what I've seen of Emacs Lisp bytecode, I think it would be a bit difficult to use something like vmgen without a lot of effort. In the interpreter for vmgen the objects are basically C kinds of objects, not Lisp Objects. Perhaps that could be negotiated, but it would not be trivial. As for JITing bytecode, haven't there been a couple of efforts in that direction already? Again, this is probably hard. I'm not saying it shouldn't be done. Just that these are very serious projects requiring a lot of effort that would take a bit of time, and might cause instability in the interim. All while Emacs is moving forward on its own. But in any event, a prerequisite for considering doing this is to understand what we got right now. That's why I'm trying to document that more people at least have an understanding of what we are talking about in the replacing or modifying the existing system. Right now I feel that there are only a handful of people who understand bytecode, and even there maybe not in entirety. --001a11466198b7cfcb0560f15309 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= On Fri, 22 Dec 2017 09:08:33 -050 Stefan Monnier advised:

> In other kinds of bytecode such as the one for C Python, a bytecode ve= rsion
> number is stored in the bytecode file.=C2=A0 When there is a change to= the
> bytecode, that number is changed.

So far, the only changes that have been made to the byte-code language
is to add new (previously unused) byte codes.=C2=A0 So from this perspectiv= e
we have always maintained backward compatibility (you can run a .elc
compiled with an older Emacs).

Whi= le this is a nice intention, it isn't always true. And it is not with d= ownsides.

In the "not true" departm= ent, there are instructions 0153 scan_buffer and 0163 set_mark which aren&#= 39;t handled in the current interpreter sources in bytecode.c

And as pipcet points out, there is this in lread.c: 

   
  if (! version || version >=3D 22)
=
    readevalloop (Qget_file_char, &input, hist_file_name,
		  0, Qnil, Qnil, Qnil, Qnil);
  else
    {
      /* We can't handle a file which was compiled with
	 byte-compile-dynamic by older version of Emacs.  */
      specbind (Qload_force_doc_strings, Qt);
      readevalloop (Qget_emacs_mule_file_char, &input, hist_file_name,
		    0, Qnil, Qnil, Qnil, Qnil);
    }

In the "not without downsid= es" department, this means that when someone looks at the bytecode int= erpreter, it is filled with garbage and bloat. This has to have a technolog= y debt associated with it.

We do not aim to maintain forward compatibility (so whether a .elc file
compiled with a more recent Emacs will work is not guaranteed), although it sometimes does work.=C2=A0 When encountering an unknown byte-code, Emacs=
signals an error, so it shouldn't cause a crash nor "something uni= ntended".

It is likely that the co= de that purports to handle obsolete=20 (or no longer emitted) instructions is broken, since I doubt any of this be= havior is=20 tested. Subtle changes in the semantics of instructions can cause unintende= d effects.


Compatibility problems with .elc files compiled with other Emacs
versions can also come from macros, and those tend to be more frequent
than the problems introduced by changes to the byte-code.=C2=A0 So detectin= g
a different byte-code version is not sufficient to catch the most common problems anyway.

My understanding of ho= w this work in a more rational way would be that there shouldn't be inc= ompatible changes between major releases. So I would hope that incompatible= macro changes wouldn't happen within a major release but between major= releases, the same as I hope would be the case for bytecode changes.
=

If someone is up for it, a possibly interesting program= to write might be a bytecode lint and report tool that shows the meta comm= ent in bytecode to describe what version of Emacs the bytecode was compiled= under (comparing with the current loaded version), what level of optimizat= ion is reported. Possibly a scan over the instructions to look for incompat= ibility both in the forward and backward direction.=C2=A0 It might optional= ly have knowledge of specific version incompatibilities say because of macr= o changes between versions.

Maybe this could = be incorporated into a "safe-load-file" function.
=

FWIW, I think Emacs deserves a new Elisp compilation system (either
a new kind of bytecode (maybe using something like vmgen), or a JIT or
something): the bytecode we use is basically identical to the one we had 20 years ago, yet the tradeoffs have changed substantially in the
mean time.

I would=C2=A0 be interested = in elaboration here about what specific=C2=A0 trade offs you mean.

From what I've seen of Emacs Lisp bytecode, I th= ink it would be a bit difficult to use something like vmgen without a lot o= f effort.=C2=A0 In the interpreter for vmgen the objects are basically C ki= nds of objects, not Lisp Objects. Perhaps that could be negotiated, but it = would not be trivial.

As for JITing bytecode, have= n't there been a couple of efforts in that direction already? Again, th= is is probably hard.

I'm not saying it sh= ouldn't be done. Just that these are very serious projects requiring a = lot of effort that would take a bit of time, and might cause instability in= the interim. All while=C2=A0 Emacs is moving forward on its own.
=

But in any event, a prerequisite for considering doing = this is to understand what we got right now. That's why I'm trying = to document that more people at least have an understanding of what we are = talking about in the replacing or modifying the existing system.
=

Right now I feel that there are only a handful of peopl= e who understand bytecode, and even there maybe not in entirety.
=

--001a11466198b7cfcb0560f15309--