bug#55137: Different result when interpreted and when evaluating byte-compiled code

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2269 bytes --]

I'm not absolutely sure if it is a bug or a "feature", but it is
extremely confusing.

To reproduce: store this as file `test.el':

    (defvar special-variable nil)

    (defmacro is-special-as-macro ()
      (special-variable-p 'special-variable))

    (defun is-special-as-function ()
      (is-special-as-macro))

    (print (is-special-as-function))
    (print (eval '(is-special-as-macro)))

Now, from command line:

    $ rm -f test.elc; emacs --batch -l test.el

This loads the file as interpreted Elisp and prints `t' two times,
i.e. always recognizes the variable as special.

Next, byte-compile the file before loading:

    $ rm -f test.elc; emacs --batch --eval "(byte-compile-file
\"test.el\")"; emacs --batch -l test.elc

This prints `nil' and `t', i.e. variable is now considered non-special
by `is-special-as-function'.  From investigating `.elc' file, it is
apparent that this is encoded into it as a macroexpanded constant.
Note that `is-special-as-macro' still works fine, the problem appears
only when it gets macroexpanded during byte-compilation.

So, apparently `defvar' form is sort of "skipped without paying
attention" during byte-compilation.  If I put it into an
`eval-and-compile' form, then macroexpansion does produce expected
result.

I noticed this with code using `iter2' library.  When `iter2-defun'
generator functions get macroexpanded during compilation, built-in
`special-variable-p' is called, because for generator functions this
is important to distinguish between local and dynamic variables.  The
example above is derived from debugging real failure.

Standard `generator' package is also affected.  Here is example code:

    ;;; -*- lexical-binding: t -*-

    (require 'generator)

    (defvar special-variable nil)

    (defun get-special-variable ()
      special-variable)

    (iter-defun buggy ()
      (let ((special-variable t))
        (iter-yield (get-special-variable))))

    (print (iter-next (buggy)))

As before, save as `test.el' and execute the same two commands.
Produced output is different depending on whether the file has been
byte-compiled or not.

Is `eval-and-compile' the proper workaround?  Can things be made less
confusing by noticing declared special variables during
byte-compilation?

Paul

[-- Attachment #2: Type: text/html, Size: 2677 bytes --]

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Wed, 27 Apr 2022 00:16:42 +0200
> 
> I'm not absolutely sure if it is a bug or a "feature", but it is
> extremely confusing.

Thanks, but please tell in what version of Emacs is this, and
preferably show the information about your build that's collected by
"M-x report-emacs-bug".

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

Originally reproduced with 27.2. I now upgraded to 28.1. Pretty sure it is
the same with master.

In GNU Emacs 28.1.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.33,
cairo version 1.16.0)
 of 2022-04-27 built on gonzo
Repository revision: b511f5c05ced05fc7d1e403002bc5ba782879cba
Repository branch: emacs-28
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --with-x-toolkit=gtk2'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND
THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK2 ZLIB

Important settings:
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_NUMERIC: ru_RU.UTF-8
  value of $LC_TIME: de_DE.UTF-8
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8-unix

On Wed, 27 Apr 2022 at 04:27, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Wed, 27 Apr 2022 00:16:42 +0200
> >
> > I'm not absolutely sure if it is a bug or a "feature", but it is
> > extremely confusing.
>
> Thanks, but please tell in what version of Emacs is this, and
> preferably show the information about your build that's collected by
> "M-x report-emacs-bug".
>

[-- Attachment #2: Type: text/html, Size: 1838 bytes --]

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Phil Sainty]

On 2022-04-27 10:16, Paul Pogonyshev wrote:
>     (defmacro is-special-as-macro ()
>       (special-variable-p 'special-variable))

This macro does not expand to code which calls `special-variable-p'.
Rather it calls `special-variable-p' at expansion time, and expands
to the return value of that call (nil or t).

If you byte-compile your code without loading it, then your would-be
`special-variable' doesn't exist, and hence your macro was expanding
to nil rather than t.

Try this:

      (defmacro is-special-as-macro ()
        '(special-variable-p 'special-variable))


-Phil

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

I understand what's happening, I just find this extremely confusing. Can
byte-compilation not notice that there is a `defvar' form and thus count
the variable as special also during compilation?

> Try this:

This cannot be done in the real cases where things break, because they
(`iter-defun', for example) _need_ to know if the variable is special at
the time macro gets expanded (they call internal helper functions that call
`special-variable-p' in turn).

The only workaround I'm aware of is putting `defvar' into an
`eval-and-compile' form (or into a different file, as `require' implicitly
does that). I consider this a workaround rather than a proper solution
since it is very confusing and not even apparent to many users (pretty much
to most of those who never stumbled into this issue). Also, resulting
errors are not self-explanatory and can be anything, including
incomprehensible and seemingly unrelated failures at runtime .
Additionally, you need to know if said variable is going to be used in a
macro that uses `special-variable-p' in its expansion code; in particular,
you need to know the list of such macros (I currently know two related
real-word examples: `iter-defun' and friends from the standard feature
`generator' and `iter2-defun' and friends from library `iter2'). Normally,
you don't even think about putting `defvar' into an `eval-and-compile'.

As mentioned, I think that byte-compilation should be improved to mark
variables as special during compilation time too, so that there is no need
in `eval-and-compile'. I.e. even if this is not considered a bug, it should
count as a feature request.

Paul

On Wed, 27 Apr 2022 at 13:20, Phil Sainty <psainty@orcon.net.nz> wrote:

> On 2022-04-27 10:16, Paul Pogonyshev wrote:
> >     (defmacro is-special-as-macro ()
> >       (special-variable-p 'special-variable))
>
> This macro does not expand to code which calls `special-variable-p'.
> Rather it calls `special-variable-p' at expansion time, and expands
> to the return value of that call (nil or t).
>
> If you byte-compile your code without loading it, then your would-be
> `special-variable' doesn't exist, and hence your macro was expanding
> to nil rather than t.
>
> Try this:
>
>       (defmacro is-special-as-macro ()
>         '(special-variable-p 'special-variable))
>
>
> -Phil
>
>

[-- Attachment #2: Type: text/html, Size: 2915 bytes --]

bug#55137: Different result when interpreted and when evaluating byte-compiled code

[Michael Heerdegen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> This cannot be done in the real cases where things break, because they
> (`iter-defun', for example) _need_ to know if the variable is special
> at the time macro gets expanded (they call internal helper functions
> that call `special-variable-p' in turn).

Do you have a recipe using `iter-defun' that reproduces the problem?

Is it sure that this can't be fixed in generator.el?

Michael.