bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello,

C-h f apply says:

| ...
| With a single argument, call the argument's first element using the
| other elements as args.

Issue 1:  This doc sentence has to be moved after the following example,
because that example:

| Thus, (apply '+ 1 2 '(3 4)) returns 10.

is an example for the more widespread syntax.

That sentence actually explains a special case: it tells that this is
also allowed:

  (apply '(+ 1 2)) ==> 3


Issue 2: The byte compiler currently miscompiles such expressions:

Expected:

  (funcall (lambda () (apply '(+ 1 2)))) ==> 3

but

  (funcall (byte-compile '(lambda () (apply '(+ 1 2)))))
   ~~> Error: Invalid function: (+ 1 2)

AFAIU this is `byte-optimize-apply's fault:

(byte-optimize-apply '(apply '(+ 1 2)))
 ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!


TIA,

Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Drew Adams]

> | With a single argument, call the argument's
> | first element using the other elements as args.

> ...
> That sentence actually explains a special case:
> it tells that this is also allowed:
>   (apply '(+ 1 2)) ==> 3

FWIW, I'm very surprised to find this "feature".
That it's documented suggests it was intended.
But I don't know why it would be.  Does anyone?
What's the use case for not requiring the first
arg to be a function?

No Lisp that I'm aware of, starting with Lisp 1,
has ever supported such a feature - at least not
according to its doc.

Ccing RMS, who might know why, if this has been
in Elisp since Day One or it was added long ago.
___

Aha, I see now - found it.  It was added in Emacs
28.1.

And this text was added then to the Elisp manual:

  ‘apply’ with a single argument is special:
  the first element of the argument, which
  must be a non-empty list, is called as a
  function with the remaining elements as
  individual arguments.  Passing two or more
  arguments will be faster.

And this additional example was added there:

    (apply '(+ 3 4))
         ⇒ 7

But why?  There's nothing in NEWS about this.
Was it even discussed? (Probably, but maybe
only in a bug thread?)

None of the rest of the text is changed -
just what I've mentioned here.

I don't understand why anyone considered it
a good idea to change the behavior of `apply'
(in this way or in any other way).  What
gives?

"Special" is indeed one way to characterize
it.  More like "shocking". ;-)  What's next?

And this bug is about the behavior not even
faithfully reflecting that doc?  Sheesh.

Was this done on April 1?

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Drew Adams <drew.adams@oracle.com> writes:

> FWIW, I'm very surprised to find this "feature".
> That it's documented suggests it was intended.
> But I don't know why it would be.  Does anyone?

Bug#40968 I think.

Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Drew Adams]

> Drew Adams writes:
> 
> > FWIW, I'm very surprised to find this "feature".
> > That it's documented suggests it was intended.
> > But I don't know why it would be.  Does anyone?
> 
> Bug#40968 I think.

Yes, thanks.  I'd forgotten about that,
and that I'd contributed to the thread.

I agree with what I said there ;-),
which is not always the case. ;-)

In particular this:

  Pip>> I think there's consensus, then. 

  me> FTR, FWIW: Not a consensus that includes me.

  me> I'm in favor of making an incompatible
  me> change, to align Emacs with Common Lisp's
  me> more reasonable behavior.

I might have added every other Lisp; it's
not just Common Lisp, AFAIK.

No one has given a good reason _why_ Elisp
ever had this one-off (no other Lisp has
it) behavior, AFAICS.

There's no reason given in that thread.
The thread just concentrates on "fixing"
the odd-ball "special" case.  And the fix
was to mention that case in the doc.

Really too bad.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Michael Heerdegen via "Bug reports for GNU Emacs, the Swiss army knife
of text editors" <bug-gnu-emacs@gnu.org> writes:

> (byte-optimize-apply '(apply '(+ 1 2)))
>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!

I'm adding Mattias to the discussion - he had changed
`byte-optimize-apply' after this new convention (which I find
irritating, too) had been introduced.

I guess since byte compilation of such forms is (at least partly)
broken, not many people are actually using it.

Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Basil L. Contovounesios]

Michael Heerdegen [2024-03-04 06:13 +0100] wrote:

> Michael Heerdegen <bug-gnu-emacs@gnu.org> writes:
>
>> (byte-optimize-apply '(apply '(+ 1 2)))
>>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!
>
> I'm adding Mattias to the discussion - he had changed
> `byte-optimize-apply' after this new convention (which I find
> irritating, too) had been introduced.

Just curious: which convention are you referring to, and why do you say
it's new?  AFAICT the only recent Emacs version which accepted
(apply '(+ 1 2)) without any complaint is Emacs 28.

> I guess since byte compilation of such forms is (at least partly)
> broken, not many people are actually using it.

FWIW, my guilty pleasure is using this calling convention with
a non-literal form, e.g.:

  (define-advice foo (:around (&rest args) my-foo)
    (with-something (apply args)))

This has always byte-compiled without issue.
Of course, I'm not married to this convention.

Thanks,
-- 
Basil

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

"Basil L. Contovounesios" <basil@contovou.net> writes:

> Just curious: which convention are you referring to, and why do you
> say it's new?

The convention that (apply (F . ARGS)) == (apply F . ARGS).

>  AFAICT the only recent Emacs version which accepted
> (apply '(+ 1 2)) without any complaint is Emacs 28.

So it is quite new.

> FWIW, my guilty pleasure is using this calling convention with
> a non-literal form, e.g.:
>
>   (define-advice foo (:around (&rest args) my-foo)
>     (with-something (apply args)))
>
> This has always byte-compiled without issue.

Aha - a user!

No, `byte-optimize-apply' does not mess up this case:

  (byte-optimize-apply '(apply args)) --> (apply args)

so it will compile just fine but

  (byte-optimize-apply '(apply '(+ 1 2)))
   --> (funcall '(+ 1 2) '+ '1 '2) ; broken

That the byte compiler result is (sometimes) broken is a bug in the
optimization code, not in the compiler itself.

Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Basil L. Contovounesios]

Michael Heerdegen [2024-03-04 09:10 +0100] wrote:

> "Basil L. Contovounesios" <basil@contovou.net> writes:
>
>> Just curious: which convention are you referring to, and why do you
>> say it's new?
>
> The convention that (apply (F . ARGS)) == (apply F . ARGS).
>
>>  AFAICT the only recent Emacs version which accepted
>> (apply '(+ 1 2)) without any complaint is Emacs 28.
>
> So it is quite new.

Depends: in interpreted code, it's worked since at least as far back as
Emacs 24.  Emacs 28 is the only version in which it successfully
byte-compiles, and doesn't emit any warnings.  Emacs 29 and 30 choke.

-- 
Basil

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

"Basil L. Contovounesios" <basil@contovou.net> writes:

> FWIW, my guilty pleasure is using this calling convention with
> a non-literal form, e.g.:
>
>   (define-advice foo (:around (&rest args) my-foo)
>     (with-something (apply args)))

Hmm, I realize now that I have used this convention all the time, too,
without thinking about it (it just worked...), in the same
situation.  Instead of - what? - (apply #'funcall args)?  But I would
not say I like it.

Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Basil L. Contovounesios]

Michael Heerdegen [2024-03-04 09:17 +0100] wrote:

> "Basil L. Contovounesios" <basil@contovou.net> writes:
>
>> FWIW, my guilty pleasure is using this calling convention with
>> a non-literal form, e.g.:
>>
>>   (define-advice foo (:around (&rest args) my-foo)
>>     (with-something (apply args)))
>
> Hmm, I realize now that I have used this convention all the time, too,
> without thinking about it (it just worked...), in the same
> situation.  Instead of - what? - (apply #'funcall args)?

No, instead of (apply (car args) (cdr args)), or

  (define-advice foo (:around (fn &rest args) my-foo)
    (with-something (apply fn args)))

-- 
Basil

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Andreas Schwab]

On Mär 04 2024, Basil L. Contovounesios wrote:

> Just curious: which convention are you referring to, and why do you say
> it's new?  AFAICT the only recent Emacs version which accepted
> (apply '(+ 1 2)) without any complaint is Emacs 28.

It was added in commit 8edd4a2b64e, since Emacs 24.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Basil L. Contovounesios]

Andreas Schwab [2024-03-04 10:39 +0100] wrote:

> On Mär 04 2024, Basil L. Contovounesios wrote:
>
>> Just curious: which convention are you referring to, and why do you say
>> it's new?  AFAICT the only recent Emacs version which accepted
>> (apply '(+ 1 2)) without any complaint is Emacs 28.
>
> It was added in commit 8edd4a2b64e, since Emacs 24.

Right, but that doesn't mean literal forms survive byte-compilation:

  $ cd "$(mktemp -d)"
  $ cat << EOF > foo.el
  > ; -*- lexical-binding: t -*-
  > (message "%s" (apply '(1+ 0)))
  > EOF
  $ emacs-24.5 -Q -batch -f batch-byte-compile foo.el

  In toplevel form:
  foo.el:2:1:Warning: `(1+ 0)' is a malformed function
  Wrote /tmp/tmp.aFbeBGQj7q/foo.elc
  $ emacs-24.5 -Q -script foo.elc
  Invalid function: (1+ 0)

By contrast:

  $ emacs-28.2 -Q -batch -f batch-byte-compile foo.el
  $ emacs-28.2 -Q -script foo.elc
  1

-- 
Basil

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Mattias Engdegård]

This was probably just a mistake of mine; a `<=` should have been `<`. Sorry about that, I'll fix.

I think Stefan introduced the 1-arg `apply` semantics as a principled and natural extension to the 2-or-more-arg case, but I'm less convinced. In any case, it's there and documented.

However it is tempting to see the spotty implementation and bugs in many versions as the converse of Hyrum's law that lets us remove the feature instead...

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Mattias Engdegård]

> This was probably just a mistake of mine; a `<=` should have been `<`.

Actually `>=` should have been `>`.
Pushed to master; closing the bug.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Drew Adams]

> Pushed to master; closing the bug.

The real bug was the misguided addition of
this "feature" to Elisp, making it unlike
all other Lisps (AFAIK).

How about we fix that instead?

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Eli Zaretskii]

> Date: Mon, 04 Mar 2024 02:51:42 +0100
> From:  Michael Heerdegen via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> Hello,
> 
> C-h f apply says:
> 
> | ...
> | With a single argument, call the argument's first element using the
> | other elements as args.
> 
> Issue 1:  This doc sentence has to be moved after the following example,
> because that example:
> 
> | Thus, (apply '+ 1 2 '(3 4)) returns 10.
> 
> is an example for the more widespread syntax.
> 
> That sentence actually explains a special case: it tells that this is
> also allowed:
> 
>   (apply '(+ 1 2)) ==> 3
> 
> 
> Issue 2: The byte compiler currently miscompiles such expressions:
> 
> Expected:
> 
>   (funcall (lambda () (apply '(+ 1 2)))) ==> 3
> 
> but
> 
>   (funcall (byte-compile '(lambda () (apply '(+ 1 2)))))
>    ~~> Error: Invalid function: (+ 1 2)
> 
> AFAIU this is `byte-optimize-apply's fault:
> 
> (byte-optimize-apply '(apply '(+ 1 2)))
>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!
> 

Stefan and Mattias, any comments or suggestions?

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Michael Heerdegen via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Stefan and Mattias, any comments or suggestions?

Mattias comment was to fix the issue and close this report - did you
miss that?

Eli, what is your opinion about my issue #1 (the docstring) btw?


Michael.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Mattias Engdegård
>  <mattiase@acm.org>,  69533@debbugs.gnu.org
> Date: Thu, 14 Mar 2024 09:44:26 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Stefan and Mattias, any comments or suggestions?
> 
> Mattias comment was to fix the issue and close this report - did you
> miss that?

I'd like to hear from Stefan as well.

> Eli, what is your opinion about my issue #1 (the docstring) btw?

I agree.  I also think the way the doc string uses "our" is confusing,
and should be reworded.

bug#69533: 30.0.50; Wrong byte compilation of a certain apply syntax

[Mattias Engdegård]

14 mars 2024 kl. 12.13 skrev Eli Zaretskii <eliz@gnu.org>:

>> Eli, what is your opinion about my issue #1 (the docstring) btw?
> 
> I agree.  I also think the way the doc string uses "our" is confusing,
> and should be reworded.

Seems I closed the bug prematurely -- sorry.

Perhaps we should move the example in the `apply` doc string, and that of `funcall`, to shortdoc?