bug#39233: .elc file - possibly outdated backward compatibility comments

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Kangas]

In the .elc files on my machine, I see the following:

    ;;; This file uses dynamic docstrings, first added in Emacs 19.29.
    
    ;;; This file does not contain utf-8 non-ASCII characters,
    ;;; and so can be loaded in Emacs versions earlier than 23.

I have the following suggestions/questions:

1. Is it still correct that the byte compiled file can be loaded on
   Emacs 22?  We have (at least) one new opcode not supported by
   Emacs 22 (see e.g. commit 88549ec38).

   And AFAIK, Emacs crashes on incorrect opcodes.

   If the comment is incorrect, we should consider removing or
   changing it.

2. Change the first line to no longer mention Emacs 19.29.  Emacs
   versions before 19.29 are no longer relevant, so there is need to
   say this.

3. Following up on point 2, I also see:
   
       "\n;;; This file uses "
       (if dynamic-docstrings
           "dynamic docstrings, first added in Emacs 19.29"
         "opcodes that do not exist in Emacs 18")
   
   I'm not sure what to do here, but it doesn't seem to make sense to
   mention Emacs 18 in any case.  Of course, this file might use
   opcodes that do not exist in Emacs 23, or perhaps an even later
   version.

Or, these points are all moot because no one cares what these files
says, and it should anyways be kept for hysterical reasons.  I just
don't know.  :-)  So WDYT?

Best regards,
Stefan Kangas

bug#39233: .elc file - possibly outdated backward compatibility comments

[Eli Zaretskii]

Stefan, any comments on these issues?

TIA

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Monnier]

> In the .elc files on my machine, I see the following:
>     ;;; This file uses dynamic docstrings, first added in Emacs 19.29.

It's factually correct, but indeed it not really relevant, we can drop
this to save some bytes.

>     ;;; This file does not contain utf-8 non-ASCII characters,
>     ;;; and so can be loaded in Emacs versions earlier than 23.

Here, this is indeed likely not factually correct: some .elc files might
indeed work in earlier versions, but that will depend on many other
things:
- byte codes have been added in Emacs-24 for the lexical-binding
  support, others were added in Emacs-25, yet more in Emacs-26)
- macros which expand to code which relies on functions/features only
  provided in newer versions).
- changes in the reader (e.g. the readable syntax for hash-tables, or
  that of records).

I think more useful would be to emit a warning any time a .elc file is
loaded into an Emacs that's older than the one with which it
was compiled since that's a case we "don't support" (at least when the
major-version is different).

> 1. Is it still correct that the byte compiled file can be loaded on
>    Emacs 22?

It's quite likely that *some* do.  But I expect that the vast
majority don't.

> 3. Following up on point 2, I also see:
>    
>        "\n;;; This file uses "
>        (if dynamic-docstrings
>            "dynamic docstrings, first added in Emacs 19.29"
>          "opcodes that do not exist in Emacs 18")
>    
>    I'm not sure what to do here, but it doesn't seem to make sense to
>    mention Emacs 18 in any case.  Of course, this file might use
>    opcodes that do not exist in Emacs 23, or perhaps an even later
>    version.

Throw it all away.


        Stefan

bug#39233: .elc file - possibly outdated backward compatibility comments

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 23 Jan 2020 11:26:52 -0500
> Cc: 39233@debbugs.gnu.org
> 
> > In the .elc files on my machine, I see the following:
> >     ;;; This file uses dynamic docstrings, first added in Emacs 19.29.
> 
> It's factually correct, but indeed it not really relevant, we can drop
> this to save some bytes.
> 
> >     ;;; This file does not contain utf-8 non-ASCII characters,
> >     ;;; and so can be loaded in Emacs versions earlier than 23.
> 
> Here, this is indeed likely not factually correct: some .elc files might
> indeed work in earlier versions, but that will depend on many other
> things:
> - byte codes have been added in Emacs-24 for the lexical-binding
>   support, others were added in Emacs-25, yet more in Emacs-26)
> - macros which expand to code which relies on functions/features only
>   provided in newer versions).
> - changes in the reader (e.g. the readable syntax for hash-tables, or
>   that of records).
> 
> I think more useful would be to emit a warning any time a .elc file is
> loaded into an Emacs that's older than the one with which it
> was compiled since that's a case we "don't support" (at least when the
> major-version is different).
> 
> > 1. Is it still correct that the byte compiled file can be loaded on
> >    Emacs 22?
> 
> It's quite likely that *some* do.  But I expect that the vast
> majority don't.
> 
> > 3. Following up on point 2, I also see:
> >    
> >        "\n;;; This file uses "
> >        (if dynamic-docstrings
> >            "dynamic docstrings, first added in Emacs 19.29"
> >          "opcodes that do not exist in Emacs 18")
> >    
> >    I'm not sure what to do here, but it doesn't seem to make sense to
> >    mention Emacs 18 in any case.  Of course, this file might use
> >    opcodes that do not exist in Emacs 23, or perhaps an even later
> >    version.
> 
> Throw it all away.

Will these changes still allow old Emacs binaries to load *.elc files
compiled by Emacs after these changes?  I hope we won't break that use
case.

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Monnier]

> Will these changes still allow old Emacs binaries to load *.elc files
> compiled by Emacs after these changes?

New Emacsen loading old .elc won't be affected.

Old Emacsen loading .elc files compiled by new Emacsen is something that
has always been risky, tho it has sometimes worked.  My proposal is to
make it emit a warning (so it will work as much as it does now, but
will bring to the attention of the user that it's in murky territory).


        Stefan

bug#39233: .elc file - possibly outdated backward compatibility comments

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Will these changes still allow old Emacs binaries to load *.elc files
>> compiled by Emacs after these changes?
>
> New Emacsen loading old .elc won't be affected.
>
> Old Emacsen loading .elc files compiled by new Emacsen is something that
> has always been risky, tho it has sometimes worked.  My proposal is to
> make it emit a warning (so it will work as much as it does now, but
> will bring to the attention of the user that it's in murky territory).

Reading the fix for the compilation problem, it was:

-    (search-forward "\n;;; This file uses")

It's quite possible that I misunderstand the patch, but doesn't this
mean that (until a couple of days ago), Emacs assumed that that string
exists in .elc files unconditionally.  And now it does, so if you try to
use an Emacs from last week to load .elc files from this week, that
week-old Emacs will break?

That doesn't seem optimal.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39233: .elc file - possibly outdated backward compatibility comments

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stefan@marxist.se,  39233@debbugs.gnu.org
> Date: Fri, 24 Jan 2020 09:22:49 -0500
> 
> > Will these changes still allow old Emacs binaries to load *.elc files
> > compiled by Emacs after these changes?
> 
> New Emacsen loading old .elc won't be affected.
> 
> Old Emacsen loading .elc files compiled by new Emacsen is something that
> has always been risky, tho it has sometimes worked.  My proposal is to
> make it emit a warning (so it will work as much as it does now, but
> will bring to the attention of the user that it's in murky territory).

A warning is a (minor) annoyance.  Maybe someone will be annoyed
enough to complain (we shall see), but for the use cases I need this
-- which is running old Emacs binaries to quickly bisect bugs in C
code or in preloaded Lisp -- this doesn't sound like a terrible
problem.  As long as this use case works as well (or as badly) as it
did before, modulo the warning.

I feared that this won't work at all, because the *.elc files lack
something which old Emacs binaries will look for at load time.

Thanks.

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Reading the fix for the compilation problem, it was:
>
> -    (search-forward "\n;;; This file uses")
>
> It's quite possible that I misunderstand the patch, but doesn't this
> mean that (until a couple of days ago), Emacs assumed that that string
> exists in .elc files unconditionally.  And now it does, so if you try to
> use an Emacs from last week to load .elc files from this week, that
> week-old Emacs will break?

That code is only run during byte-compilation, not when loading the
file.  We add the lines, and then use them as an anchor to find the
location to add a version test for Emacs <23 if there are any utf-8
non-ASCII characters.

You could try on Emacs 26.1 `M-x load-file RET cperl-mode.elc RET'
using a file byte-compiled with current master to verify.

Best regards,
Stefan Kangas

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Monnier]

> for the use cases I need this -- which is running old Emacs binaries
> to quickly bisect bugs in C code or in preloaded Lisp -- this doesn't
> sound like a terrible problem.  As long as this use case works as well
> (or as badly) as it did before, modulo the warning.

Yes, it will work just as well/poorly, modulo the warning.

> I feared that this won't work at all, because the *.elc files lack
> something which old Emacs binaries will look for at load time.

No, there should be no change in this respect: when loading a .elc file,
the only thing we look for (other than actually reading&evalling the
sexps) is to look for some kind of "version number" which is encoded in
the 4th byte of the file, a part that we have no intention to
change, AFAIK.


        Stefan

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Monnier]

Here's the patch I propose for `master`.  WDYT?


        Stefan


diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index d35ce66350..c471cc64cf 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2140,50 +2140,9 @@ byte-compile-from-buffer
 	;; Make warnings about unresolved functions
 	;; give the end of the file as their position.
 	(setq byte-compile-last-position (point-max))
-	(byte-compile-warn-about-unresolved-functions))
-      ;; Fix up the header at the front of the output
-      ;; if the buffer contains multibyte characters.
-      (and byte-compile-current-file
-	   (with-current-buffer byte-compile--outbuffer
-	     (byte-compile-fix-header byte-compile-current-file))))
+	(byte-compile-warn-about-unresolved-functions)))
      byte-compile--outbuffer)))
 
-(defun byte-compile-fix-header (_filename)
-  "If the current buffer has any multibyte characters, insert a version test."
-  (when (< (point-max) (position-bytes (point-max)))
-    (goto-char (point-min))
-    ;; Find the comment that describes the version condition.
-    (when (search-forward "\n;;; This file does not contain utf-8" nil t)
-      (narrow-to-region (line-beginning-position) (point-max))
-      ;; Find the first line of ballast semicolons.
-      (search-forward ";;;;;;;;;;")
-      (beginning-of-line)
-      (narrow-to-region (point-min) (point))
-      (let ((old-header-end (point))
-	    (minimum-version "23")
-	    delta)
-        (delete-region (point-min) (point-max))
-        (insert
-         ";;; This file contains utf-8 non-ASCII characters,\n"
-         ";;; and so cannot be loaded into Emacs 22 or earlier.\n"
-         ;; Have to check if emacs-version is bound so that this works
-         ;; in files loaded early in loadup.el.
-         "(and (boundp 'emacs-version)\n"
-         ;; If there is a name at the end of emacs-version,
-         ;; don't try to check the version number.
-         "     (< (aref emacs-version (1- (length emacs-version))) ?A)\n"
-         (format "     (string-lessp emacs-version \"%s\")\n" minimum-version)
-         ;; Because the header must fit in a fixed width, we cannot
-         ;; insert arbitrary-length file names (Bug#11585).
-         "     (error \"`%s' was compiled for "
-         (format "Emacs %s or later\" #$))\n\n" minimum-version))
-        ;; Now compensate for any change in size, to make sure all
-        ;; positions in the file remain valid.
-        (setq delta (- (point-max) old-header-end))
-        (goto-char (point-max))
-        (widen)
-        (delete-char delta)))))
-
 (defun byte-compile-insert-header (_filename outbuffer)
   "Insert a header at the start of OUTBUFFER.
 Call from the source buffer."
@@ -2214,16 +2173,16 @@ byte-compile-insert-header
        ".\n"
        (if dynamic ";;; Function definitions are lazy-loaded.\n"
 	 "")
-       "\n"
-       ;; Note that byte-compile-fix-header may change this.
-       ";;; This file does not contain utf-8 non-ASCII characters,\n"
-       ";;; and so can be loaded in Emacs versions earlier than 23.\n\n"
-       ;; Insert semicolons as ballast, so that byte-compile-fix-header
-       ;; can delete them so as to keep the buffer positions
-       ;; constant for the actual compiled code.
-       ";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n"
-       ";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n"
-       ";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n\n"))))
+       (format "(if (< emacs-major-version %d)\n"
+               ;; Let's allow silently loading into Emacs-27
+               ;; files compiled with Emacs-28.0.NN since the two can
+               ;; be almost identical (e.g. right after cutting the
+               ;; release branch) and people running the development
+               ;; branch can be presumed to know that it's risky anyway.
+               (if (zerop emacs-minor-version)
+                   (1- emacs-major-version) emacs-major-version))
+       (format "    (message \"BEWARE: %%S was compiled by a more recent version of Emacs (%s)\" #$))" emacs-version)
+       "\n\n"))))
 
 (defun byte-compile-output-file-form (form)
   ;; Write the given form to the output buffer, being careful of docstrings

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Here's the patch I propose for `master`.  WDYT?

Looks good to me, but:

> +       (format "    (message \"BEWARE: %%S was compiled by a more recent version of Emacs (%s)\" #$))" emacs-version)

Is this important enough to warrant using `warn' instead?

Best regards,
Stefan Kangas

bug#39233: .elc file - possibly outdated backward compatibility comments

[Stefan Monnier]

>> +       (format "    (message \"BEWARE: %%S was compiled by a more recent
>> version of Emacs (%s)\" #$))" emacs-version)
> Is this important enough to warrant using `warn' instead?

I don't have any strong opinion on this.  I went with `message` because
it seems to be less likely to cause trouble or annoyance: it's
reasonably easy to ignore messages whereas `warn`s are more "in your
face" in my experience.


        Stefan

bug#39233: .elc file - possibly outdated backward compatibility comments

[Glenn Morris]

Stefan Monnier wrote:

> +       (format "(if (< emacs-major-version %d)\n"
> +               ;; Let's allow silently loading into Emacs-27
> +               ;; files compiled with Emacs-28.0.NN since the two can
> +               ;; be almost identical (e.g. right after cutting the
> +               ;; release branch) and people running the development
> +               ;; branch can be presumed to know that it's risky anyway.
> +               (if (zerop emacs-minor-version)
> +                   (1- emacs-major-version) emacs-major-version))
> +       (format "    (message \"BEWARE: %%S was compiled by a more recent version of Emacs (%s)\" #$))" emacs-version)
> +       "\n\n"))))


This looks like an unsuppressible warning using a non-standard format
("BEWARE"), hard-coded into the elc?

I had thought you would be doing this in lread.c, behind an option,
similar to the existing load_dangerous_libraries.

I guess the advantage (...) of your version is that it will have an
immediate effect in pre-existing Emacs versions.

bug#39233: .elc file - possibly outdated backward compatibility comments

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stefan@marxist.se,  39233@debbugs.gnu.org
> Date: Fri, 24 Jan 2020 13:45:34 -0500
> 
> Here's the patch I propose for `master`.  WDYT?

Looks OK, but I know very little about this stuff.

bug#39233: .elc file - possibly outdated backward compatibility comments

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Here's the patch I propose for `master`.  WDYT?

Looks like this was applied at the time, but the bug report was left
open, so I'm closing it now.

commit e4fb95fa18072cedb021a82f7aa0e79fa6ca387a
Author:     Stefan Monnier <monnier@iro.umontreal.ca>
AuthorDate: Sat Mar 7 23:28:12 2020 -0500

    * lisp/emacs-lisp/bytecomp.el: Drop warning for loading into Emacs<23


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no