bug#4845: 23.1.50; Uninterned symbols in .elc files

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Helmut Eller]

In GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.12.11)
loading a file x.el with contents

 (defmacro foo ()
   (let ((sym (make-symbol "bar")))
     `(progn
        (defun ,sym () (message "function %s called" ',sym))
        (,sym))))
 
 (foo)

works as expected:

 shell> emacs -Q -batch -load x.el
 function bar called

However loading the corresponding compiled file signals an error:

 shell> emacs -Q -batch -eval '(byte-compile-file "x.el")' -load x.elc
 Wrote /tmp/x.elc
 Symbol's function definition is void: bar
 [Exit 255]

There is a #1=#:bar in the constant pool of the compiled function but #1
isn't used at the call site.

Helmut

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Marcin Borkowski]

Hi,

this message is just to mention that this bug seems to persist in GNU
Emacs 25.1.50.1.

Best,
mb


On 2009-11-01, at 09:25, Helmut Eller <eller.helmut@gmail.com> wrote:

> In GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.12.11)
> loading a file x.el with contents
>
>  (defmacro foo ()
>    (let ((sym (make-symbol "bar")))
>      `(progn
>         (defun ,sym () (message "function %s called" ',sym))
>         (,sym))))
>  
>  (foo)
>
> works as expected:
>
>  shell> emacs -Q -batch -load x.el
>  function bar called
>
> However loading the corresponding compiled file signals an error:
>
>  shell> emacs -Q -batch -eval '(byte-compile-file "x.el")' -load x.elc
>  Wrote /tmp/x.elc
>  Symbol's function definition is void: bar
>  [Exit 255]
>
> There is a #1=#:bar in the constant pool of the compiled function but #1
> isn't used at the call site.
>
> Helmut

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Alan Mackenzie]

Hello, Marcin and Helmut.

In article <mailman.2481.1453074610.843.bug-gnu-emacs@gnu.org> you wrote:
> Hi,

> this message is just to mention that this bug seems to persist in GNU
> Emacs 25.1.50.1.

I'm convinced this is not a bug, though I could be wrong.

> Best,
> mb


> On 2009-11-01, at 09:25, Helmut Eller <eller.helmut@gmail.com> wrote:

>> In GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.12.11)
>> loading a file x.el with contents

>>  (defmacro foo ()
>>    (let ((sym (make-symbol "bar")))
>>      `(progn
>>         (defun ,sym () (message "function %s called" ',sym))
>>         (,sym))))
>>
>>  (foo)

>> works as expected:

>>  shell> emacs -Q -batch -load x.el
>>  function bar called

>> However loading the corresponding compiled file signals an error:

>>  shell> emacs -Q -batch -eval '(byte-compile-file "x.el")' -load x.elc
>>  Wrote /tmp/x.elc
>>  Symbol's function definition is void: bar
>>  [Exit 255]

For reference, here are the contents of the compiled .elc file:

(defalias 'foo '(macro . #[nil "\301\302!^X\303\304^H\305\306\307\310^HDEF^HCE)\207" [sym make-symbol "bar" progn defun nil message "function %s called" quote] 8]))
(defalias 'bar #[nil "\300\301\302\"\207" [message "function %s called" #1=#:bar] 3])
(#:bar)

Important to note is that "(foo)" is expanded at byte compile time, not
at load time.  The uninterned symbol "bar" is created at compile time
during the expansion of "(foo)", but is not stored in the .elc file
anywhere.  That symbol thus exists only during the compilation of form
"(foo)" - at loading time it no longer exists.

Thus, trying to call "(#:bar)" at load time is doomed to fail.  (Though I
don't fully understand what "#:bar" means.)

There may be a bug in the compiler, in that it compiles a function
definition for the normal symbol 'bar rather than giving an error
message.

To fix the macro `foo', generate code to create the uninterned symbol at
run time rather than at compile time.  Maybe.

>> There is a #1=#:bar in the constant pool of the compiled function but #1
>> isn't used at the call site.

>> Helmut

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#4845: 23.1.50; Uninterned symbols in .elc files

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

>>>>> Alan Mackenzie <acm@muc.de> writes:

>>> (defmacro foo ()
>>> (let ((sym (make-symbol "bar")))
>>> `(progn
>>> (defun ,sym () (message "function %s called" ',sym))
>>> (,sym))))
>>> 
>>> (foo)

>>> However loading the corresponding compiled file signals an error:

shell> emacs -Q -batch -eval '(byte-compile-file "x.el")' -load x.elc
>>> Wrote /tmp/x.elc
>>> Symbol's function definition is void: bar

This example becomes clearer if we use `gensym' instead of `make-symbol
"bar"`.  Here is a macro expansion:

Byte-compiled:

  (progn
    (defun bar68503 nil
      (message "function %s called" '#:bar68503))
    (#:bar68503))

Interpreted:

  (progn
    (defun bar68503 nil
      (message "function %s called" 'bar68503))
    (bar68503))

What's happening here is this:

  1. At compile-time, we're creating a new, uninterned symbol, and we are
     both passing that symbol to defun, and calling it.

  2. `defun', at compilation time, is using the _name_ of that symbol to
     define a function definition for an interned symbol of the same name.

  3. We then try to print and call the uninterned symbol directly. So what we
     are calling, and what defun defined, are not the same symbol.

This works when non-byte-compiled because we don't inline the symbol
reference, making it a reference by name instead of by value. You can trigger
the same error behavior at interpretation time with:

  (defmacro foo ()
    (let ((sym (gensym)))
      `(progn
         (defun ,sym () (message "function %s called" ',sym))
         (funcall (symbol-function (quote ,sym))))))
  
  (foo)

I don't think this is a bug, rather macro expansion doing exactly what it was
told to do. If anything, it's odd that when interpreting, we expand to a
reference by name; but I suppose that makes sense too, given the more dynamic
nature of interpreted code.

The general rule to follow is: Don't leak internal symbols. If you use
`gensym' or `make-symbol', ensure that all dependencies on that symbol occur
entirely within the macro body. `defun' is used to establish global
definitions, so it effectively "exports" the internal symbol, breaking the
abstraction. That it has inconsistent behavior here is due to differences in
the way that byte-compilation inlines references.

I recommend closure of this bug as expected behavior, unless there are further
concerns.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Michael Heerdegen]

John Wiegley <jwiegley@gmail.com> writes:

>   3. We then try to print and call the uninterned symbol directly. So what we
>      are calling, and what defun defined, are not the same symbol.

But theoretically, the byte compiler could do it correctly.

This is how the compiled code looks like currently:

(defalias '#1=#:bar #[0 "[bytes...]" [message "function %s called" #1#] 3])
(#:bar)

If the second line would be

(#1#)

(as the compiler already correctly prints in the function body)

the thing would work AFAIU.  I guess it doesn't because it is a
different expression printed separately.

> I recommend closure of this bug as expected behavior,

I think it's quite unexpected, unless you know about how byte
compilation to files works, and even then...

> unless there are further concerns.

I think this is quite a limitation worth to be fixed, if it is possible
with little effort.

If not, we should at least make the compiler raise an error when it
generates function calls like

    (#:bar)

something like that is not very useful, in no situation.


Michael.

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Helmut Eller]

On Tue, Jan 19 2016, John Wiegley wrote:

> I don't think this is a bug, rather macro expansion doing exactly what it was
> told to do.
 
Interpreted and compiled code behave differently.  That's always a
problem.  Obviously not a big problem.

> The general rule to follow is: Don't leak internal symbols.

I would say "Don't leak non-interned symbols", except that "leaking" is
not a well defined term.

Or alternatively "Keep in mind that uninterned symbols are freshly
created when loading compiled code from files and shared only in certain
situations e.g. for two occurrences inside the same lambda."

The best solution would be to have one constant pool per
compilation-unit instead of a constant pool per function.  But I doubt
that anybody cares enough to implement that.

Helmut

bug#4845: 23.1.50; Uninterned symbols in .elc files

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

>>>>> Helmut Eller <eller.helmut@gmail.com> writes:

> Interpreted and compiled code behave differently. That's always a problem.
> Obviously not a big problem.

The number of ways to coax varying behavior from the two execution modes is
legion. I had to bend over backwards to achieve consistency for use-package
when byte-compiling an init.el file versus loading it un-byte-compiled. So I
don't feel this bug is very urgent.

>>>>> Michael Heerdegen <michael_heerdegen@web.de> writes:

> I think this is quite a limitation worth to be fixed, if it is possible with
> little effort.

Honestly, we have better things to do. If you still think this is a bug, I'm
fine leaving it open until the day when someone wants to champion the cause of
interpreted/compiled equivalence, but until then, I'd rather focus on other
things.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

bug#4845: 23.1.50; Uninterned symbols in .elc files

[John Wiegley]

>>>>> Michael Heerdegen <michael_heerdegen@web.de> writes:

> I think this is quite a limitation worth to be fixed, if it is possible with
> little effort.

If it's possible with little effort, and someone wants to do it, I agree; but
there is no hurry for this one.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Andreas Schwab]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> This is how the compiled code looks like currently:
>
> (defalias '#1=#:bar #[0 "[bytes...]" [message "function %s called" #1#] 3])
> (#:bar)
>
> If the second line would be
>
> (#1#)
>
> (as the compiler already correctly prints in the function body)
>
> the thing would work AFAIU.

It only works inside the same sexp.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#4845: 23.1.50; Uninterned symbols in .elc files

[Stefan Kangas]

tags wontfix
close 4845
thanks

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Helmut Eller <eller.helmut@gmail.com> writes:
>
>> Interpreted and compiled code behave differently. That's always a problem.
>> Obviously not a big problem.
>
> The number of ways to coax varying behavior from the two execution modes is
> legion. I had to bend over backwards to achieve consistency for use-package
> when byte-compiling an init.el file versus loading it un-byte-compiled. So I
> don't feel this bug is very urgent.
>
>>>>>> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> I think this is quite a limitation worth to be fixed, if it is possible with
>> little effort.
>
> Honestly, we have better things to do. If you still think this is a bug, I'm
> fine leaving it open until the day when someone wants to champion the cause of
> interpreted/compiled equivalence, but until then, I'd rather focus on other
> things.

I tend to agree with John here; it doesn't seem worth spending time on
this.  There also have been no further interest in this bug report
within 4 years.

I'm therefore closing this bug report as wontfix.  If anyone disagrees
with that, feel free to reopen.