Invalid function error when loading elisp through native compilation

Invalid function error when loading elisp through native compilation

[picnoir]

Hi,

I'm facing a puzzling error that only manifests using native
compilation.

If I evaluate this particular elisp file:
https://github.com/picnoir/my-repo-pins/blob/master/my-repo-pins.el
using eval-buffer (ie. not natively compiled), I can run the following
snippet just fine:

  (my-repo-pins--clone-project "test/test")

However, if I natively compile the same elisp file and load it to Emacs
through emacs-lisp-native-compile-and-load and try to call
my-repo-pins--clone-project the exact same way I was doing above, the
function call errors out with a confusing error message:

  Invalid function: (my-repo-pins--query-github-owner-repo "test" "test"
  (lambda (result) (let ((new-state (if (null result) 'not-found
  result))) (my-repo-pins--update-forges-state "GitHub.com" new-state
  "test/test"))))

I'm not sure how to debug that further. What should I do to find out
what the issue is here? Any idea about what's happening?

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

"picnoir" <picnoir@alternativebit.fr> writes:

>   Invalid function: (my-repo-pins--query-github-owner-repo "test" "test"
>   (lambda (result) (let ((new-state (if (null result) 'not-found
>   result))) (my-repo-pins--update-forges-state "GitHub.com" new-state
>   "test/test"))))

#+begin_src emacs-lisp
(defun my-repo-pins--query-forge-fetchers ..2..
  (let* ((parsed-repo-query
          (my-repo-pins--parse-repo-identifier repo-query))..1..)
    (cond
     ((equal repo-query-kind 'owner-repo)
      (seq-map
       (lambda (forge)
         (let* ((owner (alist-get 'owner parsed-repo-query))..3..)
           (apply `(,fetch-func
                    ,owner
                    ,repo
                    (lambda (result)..3..))))) ...)))))
#+end_src
                  ^^^^^^^^^^^^^^^^^^^^^^^  !

This argument to apply is not a valid function value.  Signature of
`apply' is (apply FUN . ARGS) - you are specifying only one list as first
arg FUN.


Michael.

Re: Invalid function error when loading elisp through native compilation

[tpeplt]

"picnoir" <picnoir@alternativebit.fr> writes:

> Hi,
>
> I'm facing a puzzling error that only manifests using native
> compilation.
>
> If I evaluate this particular elisp file:
> https://github.com/picnoir/my-repo-pins/blob/master/my-repo-pins.el
> using eval-buffer (ie. not natively compiled), I can run the following
> snippet just fine:
>
>   (my-repo-pins--clone-project "test/test")

Note: I am using Emacs 29.2 on GNU/Linux.  I did not get the same
results as you described and part of the reason might be due to
differing versions of Emacs.

1. The URL you provided does not give access to the file
‘my-repo-pins.el’ within Emacs.  Here is a URL that other readers should
be able to use in order to view your source file:

https://raw.githubusercontent.com/picnoir/my-repo-pins/master/my-repo-pins.el

2. After downloading your source file, I loaded it into Emacs using:

  $ emacs -Q my-repo-pins.el

3. I then byte-compiled the file and loaded it into Emacs using the
‘Emacs-Lisp’ menu entry ‘Byte compile and Load’.  (There were a number
of warnings about the use of single quotes in several functions’
docstrings due to a new syntax requirement, but those can be ignored for
this discussion.)

4. I switched to the *scratch* buffer and evaluated the following
expression using C-x C-e:

   (my-repo-pins--clone-project "test/test")

This caused the debugger to be triggered and a stack trace ("backtrace")
of function calls was generated.  That backtrace is included at the end
of this message in case you or any readers might be able to follow it to
point out some error(s).

Note that I did NOT compile the elisp file natively.  The error was
generated after byte-compiling the file only.  The positive point from
this is that backtrace might be useful for finding the source of the
problem, and eliminating it from the natively-compiled version, too.

5. Backtrace:

Debugger entered--Lisp error: (invalid-function (my-repo-pins--query-github-owner-repo "test" "test" (lambda (result) (let ((new-state (if (null result) 'not-found result))) (my-repo-pins--update-forges-state "GitHub.com" new-state "test/test")))))
  (my-repo-pins--query-github-owner-repo "test" "test" (lambda (result) (let ((new-state (if (null result) 'not-found result))) (my-repo-pins--update-forges-state "GitHub.com" new-state "test/test"))))(my-repo-pins--query-github-owner-repo "test" "test" (lambda (result) (let ((new-state (if (null result) 'not-found result))) (my-repo-pins--update-forges-state "GitHub.com" new-state "test/test"))))
  #f(compiled-function (forge) #<bytecode 0x823cf967506dbe8>)(("GitHub.com" (query-user-repo . my-repo-pins--query-github-owner-repo)))
  mapcar(#f(compiled-function (forge) #<bytecode 0x823cf967506dbe8>) (("GitHub.com" (query-user-repo . my-repo-pins--query-github-owner-repo)) ("GitLab.com" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitlab-owner-repo "gitlab.com" owner repo cb))) ("git.sr.ht" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-sourcehut-owner-repo "git.sr.ht" owner repo cb))) ("Codeberg.org" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitea-owner-repo "codeberg.org" owner repo cb)))))
  #f(compiled-function #'sequence #<bytecode 0x1857601701b27f74>)(#f(compiled-function (forge) #<bytecode 0x823cf967506dbe8>) (("GitHub.com" (query-user-repo . my-repo-pins--query-github-owner-repo)) ("GitLab.com" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitlab-owner-repo "gitlab.com" owner repo cb))) ("git.sr.ht" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-sourcehut-owner-repo "git.sr.ht" owner repo cb))) ("Codeberg.org" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitea-owner-repo "codeberg.org" owner repo cb)))))
  apply(#f(compiled-function #'sequence #<bytecode 0x1857601701b27f74>) #f(compiled-function (forge) #<bytecode 0x823cf967506dbe8>) (("GitHub.com" (query-user-repo . my-repo-pins--query-github-owner-repo)) ("GitLab.com" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitlab-owner-repo "gitlab.com" owner repo cb))) ("git.sr.ht" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-sourcehut-owner-repo "git.sr.ht" owner repo cb))) ("Codeberg.org" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitea-owner-repo "codeberg.org" owner repo cb)))) nil)
  seq-map(#f(compiled-function (forge) #<bytecode 0x823cf967506dbe8>) (("GitHub.com" (query-user-repo . my-repo-pins--query-github-owner-repo)) ("GitLab.com" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitlab-owner-repo "gitlab.com" owner repo cb))) ("git.sr.ht" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-sourcehut-owner-repo "git.sr.ht" owner repo cb))) ("Codeberg.org" (query-user-repo lambda (owner repo cb) (my-repo-pins--query-gitea-owner-repo "codeberg.org" owner repo cb)))))
  my-repo-pins--query-forge-fetchers("test/test")
  my-repo-pins--clone-project("test/test")
  (progn (my-repo-pins--clone-project "test/test"))
  elisp--eval-last-sexp(nil)
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  command-execute(eval-last-sexp)

End of Backtrace.

--

Re: Invalid function error when loading elisp through native compilation

[Stefan Monnier via Users list for the GNU Emacs text editor]

>            (apply `(,fetch-func
>                     ,owner
>                     ,repo
>                     (lambda (result)..3..))))) ...)))))
> #+end_src
>                   ^^^^^^^^^^^^^^^^^^^^^^^  !
>
> This argument to apply is not a valid function value.  Signature of
> `apply' is (apply FUN . ARGS) - you are specifying only one list as first
> arg FUN.

FWIW,

    (funcall (byte-compile '(lambda (xs) (apply xs))) '(+ 1 2))
    ==> 3

so it's not really wrong.


        Stefan

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> FWIW,
>
>     (funcall (byte-compile '(lambda (xs) (apply xs))) '(+ 1 2))
>     ==> 3
>
> so it's not really wrong.

Ok,

  (apply '(+ 1 2)) ==> 3

Hmm.  Hmm.

And if I understand the docstring of `apply' correctly:

| With a single argument, call the argument's first element using the
| other elements as args.

this is even not only not really wrong, but documented behavior.

Do you have a native compiling Emacs at hand - can you confirm that
it's a bug in the native compiler?

Michael.

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

Michael Heerdegen via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Do you have a native compiling Emacs at hand - can you confirm that
> it's a bug in the native compiler?

But byte compilation also doesn't get it right (note the difference to
your example - mine includes a complete `apply' call)

  (funcall (byte-compile '(lambda () (apply '(+ 1 2)))))
  ==> grgh

compared to

  (funcall (byte-compile '(lambda () (apply #'+ 1 (list 2)))))
  ==> 3


Michael.

Re: Invalid function error when loading elisp through native compilation

[tpeplt]

tpeplt <tpeplt@gmail.com> writes:

>
> 1. The URL you provided does not give access to the file
> ‘my-repo-pins.el’ within Emacs.  Here is a URL that other readers should
> be able to use in order to view your source file:
>
> https://raw.githubusercontent.com/picnoir/my-repo-pins/master/my-repo-pins.el
>
> 2. After downloading your source file, I loaded it into Emacs using:
>
>   $ emacs -Q my-repo-pins.el
>
> 3. I then byte-compiled the file and loaded it into Emacs using the
> ‘Emacs-Lisp’ menu entry ‘Byte compile and Load’.  (There were a number
> of warnings about the use of single quotes in several functions’
> docstrings due to a new syntax requirement, but those can be ignored for
> this discussion.)
>
> 4. I switched to the *scratch* buffer and evaluated the following
> expression using C-x C-e:
>
>    (my-repo-pins--clone-project "test/test")
>
> This caused the debugger to be triggered and a stack trace ("backtrace")
> of function calls was generated.  That backtrace is included at the end
> of this message in case you or any readers might be able to follow it to
> point out some error(s).
>
> Note that I did NOT compile the elisp file natively.  The error was
> generated after byte-compiling the file only.  The positive point from
> this is that backtrace might be useful for finding the source of the
> problem, and eliminating it from the natively-compiled version, too.
>

The stack trace above is repeatable if these steps above are followed
(and using Emacs 29.2 on GNU/Linux).  And it is repeatable if step 4 is
followed after exiting the debugger and re-evaluating the expression.

However, if the function ‘my-repo-pins--query-forge-fetchers’ is
instrumented, and the expression

   (my-repo-pins--clone-project "test/test")

is evaluated, then the debugger is NOT triggered and the expression
returns the result

   (#<buffer  *http api.github.com:443*-280838> nil nil #<buffer  *http codeberg.org:443*-939883>)

is returned.  Before it is returned, the code displays a buffer with a
prompt:

   "Please select the forge we should clone the project from.
    Press q to close this window."

After type ‘q’, the result above is returned.

What does it mean that there are different results depending on whether
the elisp code has been byte-compiled or has been instrumented?  Is this
an indication of an error in the byte-compiler?

--

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

tpeplt <tpeplt@gmail.com> writes:

> What does it mean that there are different results depending on whether
> the elisp code has been byte-compiled or has been instrumented?  Is this
> an indication of an error in the byte-compiler?

It is.  I'm preparing a bug report.

Michael.

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> [...] so it's not really wrong.

AFAIU, this is the culprit:

  (byte-optimize-apply '(apply '(+ 1 2)))
    ==> (funcall '(+ 1 2) '+ '1 '2)

I opened Bug#69533 for this.

Thanks @picnoir for reporting this.


Michael.

Re: Invalid function error when loading elisp through native compilation

[Michael Heerdegen via Users list for the GNU Emacs text editor]

Michael Heerdegen via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> It is.  I'm preparing a bug report.

Done (Bug#69533).

Meanwhile please use the (apply F . ARGS) syntax instead of
(apply (F . ARGS)).

Michael.

RE: [External] : Re: Invalid function error when loading elisp through native compilation

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

> > It is.  I'm preparing a bug report.
> 
> Done (Bug#69533).
> 
> Meanwhile please use the (apply F . ARGS)
> syntax instead of (apply (F . ARGS)).

I'd say the "bug" was that Someone ever
thought it was a good idea to change
`apply' this way, adding this so-called
"special" case.

Do we really want to fix the behavior to
make the doc for this abomination honest?

Why was this done to our good old `apply',
after almost 7 decades of faithful & loyal
service?  Who's the genius behind this?
And why wasn't this even mentioned in the
Emacs NEWS?

How many still extant flavors of Lisp are
retching when hearing about this now, and
how many extinct flavors are turning over
in their graves?

This feat deserves some kind of award.
I just can't think of what that kind is.

[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 13154 bytes --]