bug#62317: 28.2; This byte-compiled file behaves wrongly.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

Hi. In the situation below, a byte-compiled init code's evaluation is not completed, but emacs starts as if nothing was wrong.

* How to reproduce:
First, ~/.emacs.d/init.el:
------------------------------------------------------------------------
(eval-and-compile
  (push user-emacs-directory load-path)
  (require 'a))

;; Hereafter is ignored when byte-compiled.

(defvar foo-var 'baz)
(message "Hello, world.")
(pop-to-buffer "*Messages*")
------------------------------------------------------------------------

~/.emacs.d/a.el:
------------------------------------------------------------------------
(set-buffer "*Messages*") ;; or (set-buffer (get-buffer-create "bar"))
(provide 'a)
------------------------------------------------------------------------

Byte compile these two files *in this order*:
$ cd ~/.emacs.d; emacs -Q -batch -f batch-byte-compile a.el init.el

(you can now remove a.elc; it doesn't matter this bug.)
Finally, run emacs as either of these:
$ emacs -Q -l init.elc
$ emacs --no-splash --no-site-file 

* Symptom
As I commented in init.el, the code after `require' is ignored when init.el is byte-compiled. However, emacs starts as if nothing was wrong.

The line of `set-buffer' in a.el seems to trigger this bug. If you delete that line, this bug doesn't occur.

* version and build info:
In short, emacs 28 regression, but I don't know the exact version - My 28.1 was without gcc-jit, and this bug was not there. The 28.2 is built with gcc-jit, and the bug hits me.

Full build info:
------------------------------------------------------------------------
Gentoo Linux
GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.16.0) of 2023-02-02

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --datarootdir=/usr/share
 --disable-silent-rules --docdir=/usr/share/doc/emacs-28.2-r4
 --htmldir=/usr/share/doc/emacs-28.2-r4/html --libdir=/usr/lib64
 --program-suffix=-emacs-28 --includedir=/usr/include/emacs-28
 --infodir=/usr/share/info/emacs-28 --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --without-compress-install --without-hesiod --without-pop
 --with-file-notification=inotify --with-pdumper --enable-acl
 --without-dbus --with-modules --without-gameuser --with-libgmp
 --without-gpm --with-native-compilation --with-json --without-kerberos
 --without-kerberos5 --without-lcms2 --with-xml2 --without-mailutils
 --without-selinux --with-gnutls --without-libsystemd --with-threads
 --with-wide-int --with-sound=no --with-zlib --with-x --without-ns
 --without-gconf --with-gsettings --without-toolkit-scroll-bars
 --without-gif --without-jpeg --without-png --without-rsvg
 --without-tiff --without-xpm --without-imagemagick --with-xft
 --with-cairo --with-harfbuzz --with-libotf --with-m17n-flt
 --with-x-toolkit=gtk3 --without-xwidgets --with-dumping=pdumper
 'CFLAGS=-march=native -O2 -pipe' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
ACL CAIRO FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JSON LIBOTF
LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER SECCOMP
THREADS X11 XDBE XIM GTK3 ZLIB
------------------------------------------------------------------------

Thanks, Emacs developers. Best regards,
Teika 

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

As a workaround, you can wrap `set-buffer' with save-excursion, with-current-buffer, or save-current-buffer. Then it works correctly.

Teika

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

This bug is also related to native-compilation, but I can't demonstrate it in a simple way.

The situation for the init.el that I really use is:
* I suspect that the pair of require - set-buffer causes the bug. At least, the above workaround works, wrapping set-buffer.
* There the existence of init.elc does not matter. Instead, the init-<hash>.eln is the problem. By removing it, the bug disappears.
* But each time I run emacs, an eln is automatically generated aynchronously if it's lacking. (I can't find what causes automatic native compilation.)

Anyway fixing this bug will solve other native-compilation related bugs, if any. 

Regards,
Teika

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> Date: Tue, 28 Mar 2023 18:15:44 +0900 (JST)
> From: Teika Kazura <teika@gmx.com>
> 
> This bug is also related to native-compilation, but I can't demonstrate it in a simple way.
> 
> The situation for the init.el that I really use is:
> * I suspect that the pair of require - set-buffer causes the bug. At least, the above workaround works, wrapping set-buffer.
> * There the existence of init.elc does not matter. Instead, the init-<hash>.eln is the problem. By removing it, the bug disappears.
> * But each time I run emacs, an eln is automatically generated aynchronously if it's lacking. (I can't find what causes automatic native compilation.)

Native compilation is always triggered when Emacs loads a .elc file
for which there is no corresponding .eln.

> Anyway fixing this bug will solve other native-compilation related bugs, if any. 

CC'ing Andrea.

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 28 Mar 2023 18:15:44 +0900 (JST)
>> From: Teika Kazura <teika@gmx.com>
>> 
>> This bug is also related to native-compilation, but I can't demonstrate it in a simple way.
>> 
>> The situation for the init.el that I really use is:
>> * I suspect that the pair of require - set-buffer causes the bug. At least, the above workaround works, wrapping set-buffer.
>> * There the existence of init.elc does not matter. Instead, the init-<hash>.eln is the problem. By removing it, the bug disappears.
>> * But each time I run emacs, an eln is automatically generated aynchronously if it's lacking. (I can't find what causes automatic native compilation.)
>
> Native compilation is always triggered when Emacs loads a .elc file
> for which there is no corresponding .eln.
>
>> Anyway fixing this bug will solve other native-compilation related bugs, if any. 
>
> CC'ing Andrea.

Yes, I guess it's all caused by
263d6c38539691c954f4c3057cbe8d5468499b91.

I'm in a business trip now but I want to fix it ASAP (within the week),
I already started to look into it.

Anyway if we feel this is causing too much disruption I'm happy to
revert the offending commit now WDYT?

Thanks

  Andrea

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Teika Kazura <teika@gmx.com>, 62317@debbugs.gnu.org
> Date: Tue, 28 Mar 2023 12:43:33 +0000
> 
> Anyway if we feel this is causing too much disruption I'm happy to
> revert the offending commit now WDYT?

Yes, maybe we should revert until the solution is found.

Thanks.

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Teika Kazura <teika@gmx.com>, 62317@debbugs.gnu.org
>> Date: Tue, 28 Mar 2023 12:43:33 +0000
>> 
>> Anyway if we feel this is causing too much disruption I'm happy to
>> revert the offending commit now WDYT?
>
> Yes, maybe we should revert until the solution is found.

I agree.  I reverted now the patch series into 29.

Thanks

  Andrea

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

Hi. I found a relation of this bug to native compilation, but there's a new, independent bug. Anyway remember the problem here is the pair of require - set-buffer, and I'm using 28.2. (I know it's not safe. :p)

What surprises is that `native-compile-async' and `batch-native-compile' generate differnt codes. To show it, use the same init.el and a.el above. Byte-compile first a.el, then init.el. Next native-compile init.el. Run emacs, and the above bug appears.

The difference is that (i) if you use native-compile-async, by removing one of init-<hash>.eln or init.elc, the bug disappears, even if the other remains. But (ii) if you use "$ emacs -Q -batch -f batch-native-compile *el", eln in fact does not matter; only the presence of init.elc screws things up.

# Who can expect this?

At the very least, native-compilation has too many undocumented aspects. If you want, I'll open a new bug for this discrepancy.

# It's off-topic for this bug, but for the above sample code, automatic, asynchronous generation of an eln file does not happen, unlike the case of my real init.el. I can't find the reason yet.

Regards,
Teika

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 18:20:45 +0900 (JST)
> Cc: eliz@gnu.org, akrl@sdf.org
> From: Teika Kazura <teika@gmx.com>
> 
> What surprises is that `native-compile-async' and `batch-native-compile' generate differnt codes. To show it, use the same init.el and a.el above. Byte-compile first a.el, then init.el. Next native-compile init.el. Run emacs, and the above bug appears.

This is not surprising at all: native-compile-async compiles in a
separate process with environment unaffected by your customizations,
whereas batch-native-compile compiles in the current session, and thus
inherits all of your customizations.

> The difference is that (i) if you use native-compile-async, by removing one of init-<hash>.eln or init.elc, the bug disappears, even if the other remains. But (ii) if you use "$ emacs -Q -batch -f batch-native-compile *el", eln in fact does not matter; only the presence of init.elc screws things up.
> 
> # Who can expect this?
> 
> At the very least, native-compilation has too many undocumented aspects. If you want, I'll open a new bug for this discrepancy.
> 
> # It's off-topic for this bug, but for the above sample code, automatic, asynchronous generation of an eln file does not happen, unlike the case of my real init.el. I can't find the reason yet.

If the above happens in "emacs -Q", a reproducible recipe will help.
And please try that in Emacs 29 if you can, since we don't plan on any
releases of Emacs 28 with any substantial changes.

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

Sorry for having messed, but native compilation is a separate issue which is to be discussed elsewhere.

The original bug exists for the 29 git head, and for 28.3. [1] It's a pure byte-compilation problem, and the reproduction procedure in Message #5 remains valid. The workaround above works also for 29.

[1] For the 29, I used d0eb12e8d3c9d6f95b8493e05857d583c29dd0fe (2023-03-31) and "28.3" is actually 28.2 + three vulnerebility fixes by my distro, Gentoo Linux. Both are built *without* libgcc, i.e. with --without-native-compilation.

For the native compilation issue, sorry, I can't help. It's difficult for me to recompile repeatedly with --with-native-compilation. (Building 29 took about 3 times longer than 28.)

Thank you,
Teika

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> Date: Sat, 01 Apr 2023 16:36:49 +0900 (JST)
> Cc: eliz@gnu.org, akrl@sdf.org
> From: Teika Kazura <teika@gmx.com>
> 
> Sorry for having messed, but native compilation is a separate issue which is to be discussed elsewhere.
> 
> The original bug exists for the 29 git head, and for 28.3. [1] It's a pure byte-compilation problem, and the reproduction procedure in Message #5 remains valid. The workaround above works also for 29.
> 
> [1] For the 29, I used d0eb12e8d3c9d6f95b8493e05857d583c29dd0fe (2023-03-31) and "28.3" is actually 28.2 + three vulnerebility fixes by my distro, Gentoo Linux. Both are built *without* libgcc, i.e. with --without-native-compilation.
> 
> For the native compilation issue, sorry, I can't help. It's difficult for me to recompile repeatedly with --with-native-compilation. (Building 29 took about 3 times longer than 28.)

Stefan, any ideas?  I think switching to a different buffer inside
eval-and-compile is a bad idea, but maybe I'm missing something.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> * How to reproduce:
> First, ~/.emacs.d/init.el:
> ------------------------------------------------------------------------
> (eval-and-compile
>   (push user-emacs-directory load-path)
>   (require 'a))
>
> ;; Hereafter is ignored when byte-compiled.
>
> (defvar foo-var 'baz)
> (message "Hello, world.")
> (pop-to-buffer "*Messages*")
> ------------------------------------------------------------------------
>
> ~/.emacs.d/a.el:
> ------------------------------------------------------------------------
> (set-buffer "*Messages*") ;; or (set-buffer (get-buffer-create "bar"))
> (provide 'a)
> ------------------------------------------------------------------------

I suspect you can simplify the above to:

    (eval-and-compile (set-buffer "*Messages*"))
    ;; Hereafter is ignored when byte-compiled.
    (message "Hello, world.")

> Stefan, any ideas?  I think switching to a different buffer inside
> `eval-and-compile` is a bad idea, but maybe I'm missing something.

I tend to agree.
[ Side note: (push user-emacs-directory load-path) is also a bad idea.  ]

We could guard against this to some extent, but there will always be
ways for the code executed at compile-time to mess up the state of
the compiler, so I'm not sure where we should draw the line.

FWIW, in my book `set-buffer` is a code smell (usually better replaced
by `with-current-buffer`).

Admittedly, the resulting behavior can be very puzzling&frustrating for
the user, which would tend to argue in favor of trying to at least
detect the problem.
But note that if the code switched to a buffer where point is not at
EOB, we'd probably get helpful error messages during compilation, so
I'm leaning towards considering it a "minor corner case" issue, but
I think the untested patch below would be enough to plug the hole.


        Stefan


diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index 5df1205869c..e22ab94e378 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2376,7 +2376,10 @@ byte-compile-from-buffer
                  (form (read-positioning-symbols inbuffer))
                  (warning (byte-run--unescaped-character-literals-warning)))
             (when warning (byte-compile-warn-x form "%s" warning))
-	    (byte-compile-toplevel-file-form form)))
+            ;; Defend against macros using `set-buffer' or `goto-char'
+            ;; bug#62317.
+            (save-excursion
+	      (byte-compile-toplevel-file-form form))))
 	;; Compile pending forms at end of file.
 	(byte-compile-flush-pending)
 	(byte-compile-warn-about-unresolved-functions)))

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> Cc: 62317@debbugs.gnu.org
> Date: Sat, 01 Apr 2023 12:27:35 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> I think the untested patch below would be enough to plug the hole.

LGTM, thanks.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I think the untested patch below would be enough to plug the hole.
> LGTM, thanks.

You're thinking of it for `emacs-29`, right?
Then I had better test it before I push it :-)


        Stefan

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: teika@gmx.com,  62317@debbugs.gnu.org
> Date: Sat, 01 Apr 2023 19:09:42 -0400
> 
> >> I think the untested patch below would be enough to plug the hole.
> > LGTM, thanks.
> 
> You're thinking of it for `emacs-29`, right?

Not necessarily.

> Then I had better test it before I push it :-)

That's always a good idea.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Kangas]

tags 62317 + patch
thanks

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: teika@gmx.com,  62317@debbugs.gnu.org
>> Date: Sat, 01 Apr 2023 19:09:42 -0400
>>
>> >> I think the untested patch below would be enough to plug the hole.
>> > LGTM, thanks.
>>
>> You're thinking of it for `emacs-29`, right?
>
> Not necessarily.
>
>> Then I had better test it before I push it :-)
>
> That's always a good idea.

It seems like this patch was never installed.

Could we get it installed please?

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It seems like this patch was never installed.

To be honest, I'm not sure it's worth the trouble.

I'd argue that the bug is in having side-effects like that inside
`eval-and/when-compile` (or at the top-level of a `require`d file).
The patch only caters to changes to the `current_buffer` but there can
be many other ways to cause similar troubles and I can't think of a good
way to confine them all.


        Stefan

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It seems like this patch was never installed.
>
> To be honest, I'm not sure it's worth the trouble.
>
> I'd argue that the bug is in having side-effects like that inside
> `eval-and/when-compile` (or at the top-level of a `require`d file).
> The patch only caters to changes to the `current_buffer` but there can
> be many other ways to cause similar troubles and I can't think of a good
> way to confine them all.

Good point.  It seems rather unusual to call `set-buffer' from the
top-level of a required file, to say the least.  And if we accept that,
users can start calling all kinds of weird and wonderful things.

On the other hand, the fix seems innocuous enough.

Eli, what do you think?  Should we close this as wontfix?

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 13 Sep 2023 07:02:38 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, 62317@debbugs.gnu.org, teika@gmx.com
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> It seems like this patch was never installed.
> >
> > To be honest, I'm not sure it's worth the trouble.
> >
> > I'd argue that the bug is in having side-effects like that inside
> > `eval-and/when-compile` (or at the top-level of a `require`d file).
> > The patch only caters to changes to the `current_buffer` but there can
> > be many other ways to cause similar troubles and I can't think of a good
> > way to confine them all.
> 
> Good point.  It seems rather unusual to call `set-buffer' from the
> top-level of a required file, to say the least.  And if we accept that,
> users can start calling all kinds of weird and wonderful things.
> 
> On the other hand, the fix seems innocuous enough.
> 
> Eli, what do you think?  Should we close this as wontfix?

Given that Stefan Monnier himself is not sure, yes, I think we should
close this as wontfix.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Kangas]

tags 62317 + wontfix
close 62317
thanks

Eli Zaretskii <eliz@gnu.org> writes:

>> Eli, what do you think?  Should we close this as wontfix?
>
> Given that Stefan Monnier himself is not sure, yes, I think we should
> close this as wontfix.

Thanks, now done.

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

H Teika,

> I think the untested patch below would be enough to plug the hole.

Could you test that patch to make sure it fixes the problem for you?
You'll need to first apply the patch, then recompile the
`bytecomp.el` file, and only after that re-try compiling the init
file(s) where you saw the problem.


        Stefan

bug#62317: 28.2; This byte-compiled file behaves wrongly.

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> H Teika,
>
>> I think the untested patch below would be enough to plug the hole.
>
> Could you test that patch to make sure it fixes the problem for you?
> You'll need to first apply the patch, then recompile the
> `bytecomp.el` file, and only after that re-try compiling the init
> file(s) where you saw the problem.

Ping.  Teika, did you get a chance to test the patch?

bug#62317: bug #62317: 28.2; This byte-compiled file behaves wrongly.

[Teika Kazura]

Thanks, that patch works for me. To be precise, I didn't byte-compile `bytecomp.el', (because it failed with batch-byte-compile) so instead I simply deleted bytecomp.elc.

BTW
> Admittedly, the resulting behavior can be very puzzling&frustrating for
> the user, which would tend to argue in favor of trying to at least
> detect the problem.

Good observation, ethologically and/or sociologically. :)

>> What surprises is that `native-compile-async' and `batch-native-compile' generate differnt codes.[...]
> This is not surprising at all[...]

IMHO a better bug-reporting guide seems desirable (or bug reporting "standard / protocol"). Or else, silly - censorship - untrained users like me will continue complaining, being a burden to developers. After all, coexistence of el, elc, and eln is too complicating. 

Let me express my gratitude, Eli. Je vous remercie, Stefan; and Grazie to Andrea, though not cc-ed.

Best regards,
Teika

bug#62317: acknowledged by developer

[Teika Kazura]

Thanks all, and sorry for untrained report.

Best wishes,
Teika