Can the byte-compiler check whether functions passed by name are defined?

Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 2530 bytes --]

Hello!


TL;DR
=====

Is there some possibility to make the byte-compiler check,
whether functions passed to higher-order functions as a symbol are
defined at compile-time, e.g. my-function-1 in

    (mapcar 'my-function-1 some-list)?

I know some (intrusive) solutions, but if there is a built-in solution
or at least an elegant solution I'd strongly prefer that one.


My use-case
===========

When I do programming in emacs lisp, I typically start top-down. In
that process I introduce functions about whose implementation I intend
to think later, and rely on the byte-compiler for pointing out
functions to be implemented.

That approach however breaks down, when using higher-order functions.
If I have a function

    (defun my-function (strings)
      (append (list "start")
      (mapcar 'my-function-1 strings)
      (list "end")))

the byte-compiler will not tell me, that the function is undefined.
What is normally a byte-compiler warning is now a runtime error. So
far I don't know a //good// workaround, though I tried some:

1. Don't use a name, write a lambda in-place. Downside: Seriously
   reduces readability when `my-function-1' is complex or especially,
   when it contains something like

       (mapcar 'my-function-1
         (delete nil
           (mapcar 'my-function-2 strings)))

2. Use a combination of function and variable

       ...
       (mapcar my-function-1 strings)
       ...
       (defconst my-function-1 (lambda () ...

   Downside: Uncommon, therefore probably hard to read for anyone else
   (and probably for myself a year later).

3. Wrap into a lambda.

       (mapcar (lambda (e) (my-function-1 e)) strings)

   Upside: When `my-function-1' is a general function that needs
   more than one argument, this construct seems to be an elegant
   solution when using lexical binding.

   Downside: When, like here, it takes only one argument, it is more
   likely unnecessarily confusing.

4. Wrap the higher-order functions into macros that do the check at
   compile time.

   Downside: I cannot think of a way, that

   - Preserves the interface, including parameter names, and thus
     the readbility of the `describe-function' text.
   - Is future proof in that it doesn't require changing the wrapper
     code when the interface of the function is changed in a
     backward-compatible way.
   - Doesn't interfere with the runtime behaviour of interpreted code
     (e.g. raising warnings only during compilation, but not when
     interpreting the code).


kind regards, Klaus

[-- Attachment #2: Type: text/html, Size: 3809 bytes --]

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>     (mapcar 'my-function-1 some-list)?

We could do it for symbols quoted with #' but not for quoting with '.


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

2013/7/29 Stefan Monnier <monnier@iro.umontreal.ca>

> >     (mapcar 'my-function-1 some-list)?
>
> We could do it for symbols quoted with #' but not for quoting with '.
>
>         Stefan
>

#' would allow checking for a defined function independent of the function,
the function is oassed to, true... Would require however to change coding
practice to using this syntax, with the advantage of preventing warnings
when people don't want the check but the disadvantage, that the warning is
also supressed when people just don't care. I guess it would still be the
best solution, that is backward compatible though.

Another possibilty (probably less good though):

    (eval-and-compile
      (put 'mapcar 'compiler-macro
           (lambda (&rest form)
             (let ((function (nth 1 form)))
               (when (and (macroexp--compiling-p)
          (listp function)
                          (= 2 (length function))
                          (member (nth 0 function) '(function quote))
                          (symbolp (nth 1 function))
                          (not (fboundp (nth 1 function))))
                 (warn "Function not know to be defined: %S" (nth 1
function))))
     (car form))))

    (mapcar 'message '("Hello" "World"))
    (mapcar 'foobar '("Foo" "Bar"))

Some caveats here:
1. Has to be done, though maybe through a macro, for every higher-order
function.
2. `compiler-macro' doesn't seem to be documented anywhere. The arguments
the function gets passed seem to differ from case to case...
3. (warn) doesn't emit warnings to the compilation buffer but to the
separate *Warnings* buffer, making this code only a prrof-of-concept
without practical value.
4. If the quoted funciton is defined in the same file as the higher-order
function it is passed to, the definition of the quoted function must be
both before the first use of the function and inside an (eval-and-compile
..) block, which can only be prevented by changes to the compiler code
anyway.

kind regards, Klaus

[-- Attachment #2: Type: text/html, Size: 3720 bytes --]

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

> #' would allow checking for a defined function independent of the function,
> the function is oassed to, true... Would require however to change coding
> practice to using this syntax, with the advantage of preventing warnings
> when people don't want the check but the disadvantage, that the warning is
> also supressed when people just don't care. I guess it would still be the
> best solution, that is backward compatible though.

The two aren't mutually exclusive.  I'd welcome a patch that adds the
warning when #' is used.

>     (eval-and-compile
>       (put 'mapcar 'compiler-macro
>            (lambda (&rest form)

This should be (lambda (form &rest args)

Warning from a (compiler) macro is a pain in the rear (because the macro
might be called in difference circumstances and because it's not called
at a good time).  But if #' warns, then the compiler-macro can simply
turn 'foo into #'foo and leave the warning to the handle of #'.

> 1. Has to be done, though maybe through a macro, for every higher-order
> function.

Note that macroexp.el already has special handling for the main
higher-order functions (to warn about '(lambda ...)).  So it could be
implemented there.

> 3. (warn) doesn't emit warnings to the compilation buffer but to the
> separate *Warnings* buffer, making this code only a prrof-of-concept
> without practical value.

You can try to use macroexp--warn-and-return.

> 4. If the quoted funciton is defined in the same file as the higher-order
> function it is passed to, the definition of the quoted function must be
> both before the first use of the function and inside an (eval-and-compile
> ..) block, which can only be prevented by changes to the compiler code
> anyway.

That's why the patch for #' warnings needs to be directly in bytecomp.el
rather than in compiler macros.


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Sebastian Wiesner]

2013/7/31 Stefan Monnier <monnier@iro.umontreal.ca>:
>> 1. Has to be done, though maybe through a macro, for every higher-order
>> function.
>
> Note that macroexp.el already has special handling for the main
> higher-order functions (to warn about '(lambda ...)).  So it could be
> implemented there.

I presume the list of “main higher-order” functions is hard-coded
then, isn't it?

Is there a symbol property or "declare" form to mark specific
arguments as function arguments, to have the same warnings for higher
order functions from 3rd party libraries, too?  Similar to how
"docstring" arguments for macros are handled?

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>>> 1. Has to be done, though maybe through a macro, for every higher-order
>>> function.
>> Note that macroexp.el already has special handling for the main
>> higher-order functions (to warn about '(lambda ...)).  So it could be
>> implemented there.
> I presume the list of “main higher-order” functions is hard-coded
> then, isn't it?

Currently, yes.  It could be moved from macroexp.el's main loop to
a bunch of compiler-macros (the code predates the introduction of
compiler macros in core Elisp), if needed.

> Is there a symbol property or "declare" form to mark specific
> arguments as function arguments, to have the same warnings for higher
> order functions from 3rd party libraries, too?  Similar to how
> "docstring" arguments for macros are handled?

No, and I think it would be overkill.  But 3rd party can also use
a compiler-macro to turn the ' into a #'.

The approach I use for '(lambda ...) is to emit warnings in the common
cases, so that coders will slowly learn, which then benefits all cases.

For '(lambda ...) that works well, because removing the ' (or using #')
is often useful (making it possible to byte-compile the code, or using
lexically bound vars), but for single quoted symbols, the benefit is
a lot less clear ("turn 'foo into #'foo to get rid of the silly new
warning" only to then get another warning because the compiler doesn't
understand that `foo' will very much exist by the time we try to call it).


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

[-- Attachment #1.1: Type: text/plain, Size: 3562 bytes --]

2013/8/1 Stefan Monnier <monnier@iro.umontreal.ca>

> >>> 1. Has to be done, though maybe through a macro, for every higher-order
> >>> function.
> >> Note that macroexp.el already has special handling for the main
> >> higher-order functions (to warn about '(lambda ...)).  So it could be
> >> implemented there.
> > I presume the list of “main higher-order” functions is hard-coded
> > then, isn't it?
>
> Currently, yes.  It could be moved from macroexp.el's main loop to
> a bunch of compiler-macros (the code predates the introduction of
> compiler macros in core Elisp), if needed.
>
> > Is there a symbol property or "declare" form to mark specific
> > arguments as function arguments, to have the same warnings for higher
> > order functions from 3rd party libraries, too?  Similar to how
> > "docstring" arguments for macros are handled?
>
> No, and I think it would be overkill.  But 3rd party can also use
> a compiler-macro to turn the ' into a #'.
>
> The approach I use for '(lambda ...) is to emit warnings in the common
> cases, so that coders will slowly learn, which then benefits all cases.
>
> For '(lambda ...) that works well, because removing the ' (or using #')
> is often useful (making it possible to byte-compile the code, or using
> lexically bound vars), but for single quoted symbols, the benefit is
> a lot less clear ("turn 'foo into #'foo to get rid of the silly new
> warning" only to then get another warning because the compiler doesn't
> understand that `foo' will very much exist by the time we try to call it).
>
>
>         Stefan
>

 I have written an implementation of the compile-time check, see the
attached patch to "lisp/bytecomp.el". Since I also introduced as new
declare form for `defun', I have also attached a patch to
"doc/functions.texi".

The code handles both the cases (function SYMBOL) and (quote SYMBOL). In
the latter case it requires a symbol property "higher-order-arguments" to
be set, which should preferably be done by a declare form, but for now is
implemented by checking for a set of common functions (apply, funcall, all
functions I found starting with "map" or "cl-map") for the property and
setting it if it is not yet set. The value of the property should be a list
of integers, which give the positions of the arguments that are expected to
be functions, counting from 0.

If the patch becomes part of the emacs sources, I can also provide a patch
that adds the needed declare forms to the various definitions, though I
don't know how to do it for functions defined in the C-code such as
`mapcar'.

Currently such a function (in this case requiring lexical binding)
definition would read

(defun my-combine (func1 func2)
(declare (higher-order-arguments 0 1)
(lambda (arg)
(funcall func1 (funcall func2 arg))))

or

(defun my-map (func list)
(declare (higher-order 0))
(mapcar func list))

I was thinking of allowing to declare the number of arguments the function
will be passed, but didn't have the time for that detail (yet?).

One thing I am not sure about: I define the handler for the declare form in
"bytecomp.el", hence the handler is not known before loading that file.
When loading a file from source the declare form will therefore cause a
warning. When compiling it doesn't seem to cause issues though.

I couldn't define the handler in "byte-run.el" however, as when I added it
to the declaration of `defun-declaration-alist', it would suddenly be
missing again during compilation.

- Klaus

[-- Attachment #1.2: Type: text/html, Size: 4541 bytes --]

[-- Attachment #2: functions.texi.patch --]
[-- Type: application/octet-stream, Size: 702 bytes --]

--- functions.texi.orig	2013-07-01 10:21:08.404397000 +0200
+++ functions.texi	2013-08-04 14:05:41.516553400 +0200
@@ -1307,6 +1307,13 @@
 which case the warning message gives no extra details).  @var{when}
 should be a string indicating when the function or macro was first
 made obsolete.
+
+@item (higher-order-arguments [@var{pos} ..])
+Tell the byte compiler that the argument at position @var{pos} is
+expected to be a function. @var{pos} starts from 0. When the actual
+parameter is a quoted symbol, the byte compiler will warn about an
+unknown function, if the symbol is not known to be bound to a function
+at compile time. Zero or more @var{pos} can be specified.
 @end table
 @end defmac
 

[-- Attachment #3: bytecomp.el.patch --]
[-- Type: application/octet-stream, Size: 9504 bytes --]

--- bytecomp.el.orig	2013-07-02 22:28:34.936731600 +0200
+++ bytecomp.el	2013-08-04 20:38:38.534651900 +0200
@@ -1496,6 +1496,7 @@
          (byte-compile-const-variables nil)
          (byte-compile-free-references nil)
          (byte-compile-free-assignments nil)
+	 (byte-compile--higher-order--unresolved-functions nil)
          ;;
          ;; Close over these variables so that `byte-compiler-options'
          ;; can change them on a per-file basis.
@@ -1939,6 +1940,7 @@
 	;; Make warnings about unresolved functions
 	;; give the end of the file as their position.
 	(setq byte-compile-last-position (point-max))
+	(byte-compile--higher-order--warn-about-unresolved-function-names)
 	(byte-compile-warn-about-unresolved-functions))
       ;; Fix up the header at the front of the output
       ;; if the buffer contains multibyte characters.
@@ -2960,6 +2962,7 @@
 (defun byte-compile-normal-call (form)
   (when (and (byte-compile-warning-enabled-p 'callargs)
              (symbolp (car form)))
+    (byte-compile--higher-order--check-arguments form)
     (if (memq (car form)
               '(custom-declare-group custom-declare-variable
                                      custom-declare-face))
@@ -4659,6 +4662,170 @@
 	      (indent-to 40)
 	      (insert (int-to-string n) "\n")))
 	(setq i (1+ i))))))
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [BEGINNING] ;;;;;;;;
+
+(defvar byte-compile--higher-order--unresolved-functions nil
+  "A list of functions that were passed by name to higher order functions but where not known to be defined.
+
+Used during compilation to report functions referred to by (quote
+SYMBOL) or (function SYMBOL) that are not known to be defined.
+
+ See also
+`byte-compile--higher-order--warn-about-unresolved-function-names',
+`byte-compile--higher-order--check-arguments'.")
+
+(defun byte-compile--higher-order--warn-about-unresolved-function-names ()
+  "Emit compiler warnings when there are unresolved function a symbols passed to higher order functions.
+See also `byte-compile--higher-order--unresolved-functions'."
+  ;; The logic is similiar to `byte-compile-warn-about-unresolved-functions'.
+  (when (byte-compile-warning-enabled-p 'unresolved)
+    (let ((byte-compile-current-form :end)
+	  not-defined-at-runtime-maybe-symbol-list
+	  resolved-to-function-list
+	  resolved-to-macro-list
+	  unresolved-symbol-list)
+    (dolist (function-name byte-compile--higher-order--unresolved-functions)
+      (cond ((assq function-name byte-compile-function-environment)
+	     (push function-name resolved-to-function-list))
+	    ((assq function-name byte-compile-macro-environment)
+	     (push function-name resolved-to-macro-list))
+	    ((fboundp function-name)
+	     (push function-name not-defined-at-runtime-maybe-symbol-list))
+	    (t (push function-name unresolved-symbol-list))))
+    (dolist (resolved-name resolved-to-function-list)
+      (setq byte-compile--higher-order--unresolved-functions
+	    (delq resolved-name
+		  byte-compile--higher-order--unresolved-functions)))
+    (byte-compile-print-syms 
+     "The symbol `%S' was passed to a function as if the name of a function, but is bound to a macro instead."
+     "The following symbols were passed to a function as if the names functions but are bound to macros instead:"
+     resolved-to-macro-list)
+    (byte-compile-print-syms
+     "The symbol `%S' was used as function name but the function is not known to be defined."
+     "The following symbols were used as function names but the functions are not known to be defined:"
+     unresolved-symbol-list)
+    (byte-compile-print-syms
+     "The function `%S' was used as function name but the function might not be defined at runtime."
+     "The following symbols were used as function names but the functions might not be defined at runtime:"
+     not-defined-at-runtime-maybe-symbol-list))))
+
+(defun byte-compile--higher-order--check-arguments (form)
+  "On FORM, which is a function call, check whether functions passed by name are known to be defined.
+
+If a parameter has the form (function SYMBOL) or it is known to
+be expected to be a function through the `higher-order-arguments'
+property of the symbol (car FORM) and has the form (quote SYMBOL)
+we either warn right away, if SYMBOL is known to be bound to
+something that cannot be passed to higher order functions such as
+special forms and macros. If SYMBOL's function cell is empty, we
+queue the symbol into
+`byte-compile--higher-order--unresolved-functions' for checking
+at the end of the file, see
+`byte-compile--higher-order--warn-about-unresolved-function-names'."
+  (let ((higher-order-arguments
+	 (and (symbolp (car form))
+	      (get (car form) 'higher-order-arguments)))
+	(pos -1))
+    (dolist (subform (cdr form))
+      (setq pos (1+ pos))
+      ;; Check when subform has form (function SYMBOL) or (quote SYMBOL)
+      ;; and position matches one mentioned in higher-order-arguments.
+      (when (or (and (listp subform) ;; case: (function SYMBOL)
+		     (eq (car subform) 'function)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform))))
+		(and (listp subform) ;; case: (quote SYMBOL)
+		     (eq (car subform) 'quote)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform)))
+		     (member pos higher-order-arguments)))
+	(let ((function-name (car (cdr subform))))
+	  ;; Is the function already defined or already known to be
+	  ;; defined in this file? Note that macros are not allowed!
+	  (cond ((assq function-name byte-compile-function-environment)) ;; Do nothing.
+		((assq function-name byte-compile-macro-environment)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		((functionp function-name)) ;; No nothing, is ok.
+		((fboundp function-name)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		(t ;; Unknown symbol
+		 (add-to-list 'byte-compile--higher-order--unresolved-functions
+			      function-name))))))))
+
+;; Placing the entry inside the definition of
+;; `defun-declaration-alist' directly for some reason doesn't work.
+(add-to-list 'defun-declarations-alist
+	     '(higher-order-arguments 
+	       byte-compile--higher-order--declare-form-handler))
+
+(defun byte-compile--higher-order--declare-form-handler (f _args &rest val)
+  "Handler for the (declare (higher-order-arguments [NUM ...])) form, see `defun-declarations-alist'.
+
+F is the name of the function, _ARGS the formal parameter list,
+VAL should be (NUM ..).
+
+LIMITATION: When the function has been used before it was
+defined, a compiler warning is emitted pointing out that prior
+use might not have been checked."
+  ;; Check that VAL is list of integers.
+  (if (or (not (listp val))
+	  (delete nil
+	    (mapcar (lambda (e) (not (integerp e)))
+	      val)))
+      ;; Warn about invalid `declare' form
+      (let ((warn-text 
+	     (concat "(declare ..) form (higher-order-arguments [NUM ..]) with invalid arguments "
+		     (mapconcat (lambda (e) (format "%S" e)) val " "))))
+	(if byte-compile-current-file 
+	    (byte-compile-warn "%s" warn-text)
+	  (warn "%s" warn-text))
+	nil)
+    ;; Return a `put' form, but also perform it at compile time.
+    ;; Limitation: Affacts only higher order functions defined before
+    ;; their first use.
+    (when (assq f byte-compile-unresolved-functions)
+      (byte-compile-warn 
+       "%s"
+       (concat "Higher order function " (symbol-name f) " was used before it was defined. "
+	       "Actual parameters were not checked for validity.")))
+    (list 'eval-and-compile 
+	  (list 'put (list 'quote f) ''higher-order-arguments
+		(list 'quote val)))))
+
+;; Setting the property for certain known functions.
+
+(defmacro byte-compile--higher-order--register-known-functions (pos &rest symbol-list)
+  "For elements of SYMBOL-LIST set the higher-order-arguments property to (POS) when it is not yet set.
+
+Only for backward compatibility use in bytecomp.el. If you want
+to set the property for your own functions use
+the (declare (higher-order [NUM ..])) form."
+  (declare (indent 1))
+  (let (forms)
+    (dolist (function-name symbol-list)
+      (push (list 'if (list 'not (list 'get (list 'quote function-name) ''higher-order-arguments))
+		  (list 'put (list 'quote function-name) ''higher-order-arguments 
+			(list 'quote (list pos))))
+	    forms))
+    (cons 'progn (reverse forms))))
+
+(byte-compile--higher-order--register-known-functions 0
+  ;; Functions defined in C-source.
+  apply funcall map-char-table map-charset-chars map-keymap
+  map-keymap-internal mapatoms mapc mapcar mapconcat maphash
+  ;; Functions defined in standard elisp files
+  map-keymap-sorted map-query-replace-regexp map-y-or-n-p
+  ;; From the deprecated `cl'
+  map mapcan mapcar* mapcon mapl maplist
+  ;; Non-obsolete non-internal functions from `cl-lib'
+  cl-map cl-mapc cl-mapcan cl-mapcar cl-mapcon cl-maphash cl-mapl
+  cl-maplist)
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [END] ;;;;;;;;;;;;;;
+
 \f
 ;; To avoid "lisp nesting exceeds max-lisp-eval-depth" when bytecomp compiles
 ;; itself, compile some of its most used recursive functions (at load time).

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>  I have written an implementation of the compile-time check, see the
> attached patch to "lisp/bytecomp.el". Since I also introduced as new
> declare form for `defun', I have also attached a patch to
> "doc/functions.texi".

That looks a lot more complex than what I expected.
As mentioned, the best option is to start by adding warnings for the
#'symbol form (should be easy: handle it in the same way we handle
warnings for function calls).

Then make the higher-order functions turn 'symbol into #'symbol.
That's important for things like

   (if foo #'a #'b)

> (defun my-combine (func1 func2)
>   (declare (higher-order-arguments 0 1)

Sadly, I defined the `compiler-macro' declaration to take a function
rather than an exp that returns a function, so you can't just
write a function macroexp--rewrite-function-arguments and then use:

  (defun my-combine (func1 func2)
    (declare (compiler-macro (macroexp--rewrite-function-arguments 0 1)))

But you can do something like
    
  (defun my-combine (func1 func2)
    (declare (compiler-macro
              (lambda (body)
                (macroexp--rewrite-function-arguments
                  body (rewrite func1) (rewrite func2)))))

  (defun my-mapcar (func list)
    (declare (compiler-macro
              (lambda (body)
                (macroexp--rewrite-function-arguments
                  body (rewrite func) list))))

Still, this annotation is only needed to turn a ' into a #', so it's not
the most important.
                  
> I couldn't define the handler in "byte-run.el" however, as when I added it
> to the declaration of `defun-declaration-alist', it would suddenly be
> missing again during compilation.

Probably because you didn't re-dump Emacs (byte-run.el is preloaded
into the `emacs' executable so if you change it, you need to rebuild
`emacs').


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

2013/8/4 Stefan Monnier <monnier@iro.umontreal.ca>:
>
> As mentioned, the best option is to start by adding warnings for the
> #'symbol form (should be easy: handle it in the same way we handle
> warnings for function calls).
>
> Then make the higher-order functions turn 'symbol into #'symbol.
> That's important for things like
>
>    (if foo #'a #'b)
>

I am not sure I understand what you have in mind. Is the following
snippet such a case?

    (mapcar (if flag 'f1 'f2) list)
    => (mapcar (if flag #'f1 #'f2) list)

I don't see much point in that, because I don't like the notion that
the above code behaves different from

    (defun select-function (flag f)
      (if flag f 'default-f))
    ...
    (mapcar (select-function flag 'f1) alist)

In my implementation still warnings would be created if the programmer
explicitly writes #'f1, #'f2 in `select-function', which I suppose
would happen with yours too. Handling cases like computed function
names seems fairly complex -- what about e.g.

    (mapcar (catch 'function
              (dolist (f function-list)
                (when (PREDICATE f)
                  (throw 'function f)))
              'default-function)
            list)

When the simple if-case generates a warning, naively I'd expect this
to generate a warning too, but I'd expect it to become impossibly
complex then.


> Sadly, I defined the `compiler-macro' declaration to take a function
> rather than an exp that returns a function, so you can't just
> write a function macroexp--rewrite-function-arguments and then use:
>
>   (defun my-combine (func1 func2)
>     (declare (compiler-macro (macroexp--rewrite-function-arguments 0 1)))
>
> But you can do something like
>
>   (defun my-combine (func1 func2)
>     (declare (compiler-macro
>               (lambda (body)
>                 (macroexp--rewrite-function-arguments
>                   body (rewrite func1) (rewrite func2)))))
>
>   (defun my-mapcar (func list)
>     (declare (compiler-macro
>               (lambda (body)
>                 (macroexp--rewrite-function-arguments
>                   body (rewrite func) list))))
>
> Still, this annotation is only needed to turn a ' into a #', so it's not
> the most important.

Shouldn't the markup for user-written functions be a bit easier?
Something along the lines of using a list of positions (like specified
by my declare form) and having the rewriting, if any, done inside
byte-compile-normal-call or whichever function would be the correct
place. Especially since `compiler-macro' seems to be undocumented...


>> I couldn't define the handler in "byte-run.el" however, as when I added it
>> to the declaration of `defun-declaration-alist', it would suddenly be
>> missing again during compilation.
>
> Probably because you didn't re-dump Emacs (byte-run.el is preloaded
> into the `emacs' executable so if you change it, you need to rebuild
> `emacs').

Indeed I did not, since I did not expect this behaviour. Rebuilding
emacs is fairly slow on windows, so I didn't even think of this
possibility.

Anyway, I find the behaviour confusing. Having code preloaded into
the binary is okay, but having code in installation directory that is
ignored even when it has been changed seems a bit awkward. Or IS it
loaded when changed and just didn't change `defun-declaration-alist',
because it is bound already before loading the changed byte-code?


  - Klaus

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>> As mentioned, the best option is to start by adding warnings for the
>> #'symbol form (should be easy: handle it in the same way we handle
>> warnings for function calls).
>> 
>> Then make the higher-order functions turn 'symbol into #'symbol.
>> That's important for things like
>> 
>> (if foo #'a #'b)

> I am not sure I understand what you have in mind.

That's because my fingers fumbled.  The "That's important for things
like" refers to the "warnings for #'" in the previous paragraph.
The "Then make the higher..." line should have come afterwards.

>     (mapcar (if flag 'f1 'f2) list)
>     => (mapcar (if flag #'f1 #'f2) list)

No, I definitely don't want to get into such fanciness.
But I do want compiler warnings if the coder writes

     (mapcar (if flag #'f1 #'f2) list)

and `f1' or `f2' are unknown.
     
>     (defun select-function (flag f)
>       (if flag f 'default-f))
>     ...
>     (mapcar (select-function flag 'f1) alist)

100% agreement.

> Shouldn't the markup for user-written functions be a bit easier?

I'm not too worried about it, no.  We should instead aim to teach people
to write #'foo instead of 'foo.

Given that we can't get warnings for (if a 'foo 'bar), it's OK if you
won't get warnings for all cases of (higher-order-func arg1 'foo).

> Anyway, I find the behaviour confusing. Having code preloaded into
> the binary is okay, but having code in installation directory that is
> ignored even when it has been changed seems a bit awkward.

Yes, it's a bit confusing.  But the source is still very useful (Emacs
(and the ideals of Free Software) wants you to have very easy access to
the source), so we could arguably get rid of the preloaded .elc files,
but we'd still have to keep the .el file, so in most cases you'd still
be just as confused.

> Or IS it loaded when changed

No, it's not re-loaded (we do have such a "re-load" hack in
byte-compile-refresh-preloaded but it's only used when rebuilding Emacs
to try and minimize the need to bootstrap).

> and just didn't change `defun-declaration-alist', because it is bound
> already before loading the changed byte-code?

Indeed, that's one of the shortcomings of byte-compile-refresh-preloaded.


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

2013/8/5 Stefan Monnier <monnier@iro.umontreal.ca>:
>>     (mapcar (if flag 'f1 'f2) list)
>>     => (mapcar (if flag #'f1 #'f2) list)
>
> No, I definitely don't want to get into such fanciness.
> But I do want compiler warnings if the coder writes
>
>      (mapcar (if flag #'f1 #'f2) list)
>
> and `f1' or `f2' are unknown.

My code does that and indeed that was, what I first implemented. I
then added the second check, where (quote f1) is handled for functions
known to be higher-order.

>> Shouldn't the markup for user-written functions be a bit easier?
>
> I'm not too worried about it, no.  We should instead aim to teach people
> to write #'foo instead of 'foo.

I am not sure about that... On the one hand, I see your point, since
consistently writing #'function is better for the byte compiler, as in
the (if flag #'f1 #'f2) form; If #'f1 is the standard method people
will write (if flag 'f1 'f2) and not get a warning.

On the other hand though, subjectively, #'f1 is a huge deal more
visual clutter and somewhat awkward to type (on a German keyboard at
least). Might be just a training effect, but I also feel that the
hash-quote sequence moves the attention away from the function name,
while 'f1 does not (or #f1, '#f1 for that matter, but those are not
elisp).

  - Klaus

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

2013/8/5 Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>:
> 2013/8/5 Stefan Monnier <monnier@iro.umontreal.ca>:
>>>     (mapcar (if flag 'f1 'f2) list)
>>>     => (mapcar (if flag #'f1 #'f2) list)
>>
>> No, I definitely don't want to get into such fanciness.
>> But I do want compiler warnings if the coder writes
>>
>>      (mapcar (if flag #'f1 #'f2) list)
>>
>> and `f1' or `f2' are unknown.
>
> My code does that and indeed that was, what I first implemented. I
> then added the second check, where (quote f1) is handled for functions
> known to be higher-order.
>
>>> Shouldn't the markup for user-written functions be a bit easier?
>>
>> I'm not too worried about it, no.  We should instead aim to teach people
>> to write #'foo instead of 'foo.
>
> I am not sure about that... On the one hand, I see your point, since
> consistently writing #'function is better for the byte compiler, as in
> the (if flag #'f1 #'f2) form; If #'f1 is the standard method people
> will write (if flag 'f1 'f2) and not get a warning.
>
> On the other hand though, subjectively, #'f1 is a huge deal more
> visual clutter and somewhat awkward to type (on a German keyboard at
> least). Might be just a training effect, but I also feel that the
> hash-quote sequence moves the attention away from the function name,
> while 'f1 does not (or #f1, '#f1 for that matter, but those are not
> elisp).
>
>   - Klaus

Made a new version of the patch, were warnings are emitted for
registered function arguments. Downside: While educational towards
#'FUNCTION notation, it will result in a good deal of warnings all
across even included code. I'd say this results in a risk of people
ignoring warnings alltogether, including those that likely point out
actual errors such as "reference to free variable". For bytecomp.el
alone it results in 22 warnings!

Advantage would be that using such aggressive "reeducation" the the
declare form would be mostly unnecessary.

  - Klaus

[-- Attachment #2: bytecomp.el.patch-2 --]
[-- Type: application/octet-stream, Size: 10394 bytes --]

--- bytecomp.el.orig	2013-07-02 22:28:34.936731600 +0200
+++ bytecomp.el	2013-08-07 13:16:12.843412700 +0200
@@ -284,6 +284,8 @@
   free-vars   references to variables not in the current lexical scope.
   unresolved  calls to unknown functions.
   callargs    function calls with args that don't match the definition.
+  funcargs    higher order function like mapcar used with function argument 
+              as (quote SYMBOL) rather than (function SYMBOL)
   redefine    function name redefined from a macro to ordinary function or vice
               versa, or redefined to take a different number of arguments.
   obsolete    obsolete variables and functions.
@@ -1496,6 +1498,7 @@
          (byte-compile-const-variables nil)
          (byte-compile-free-references nil)
          (byte-compile-free-assignments nil)
+	 (byte-compile--higher-order--unresolved-functions nil)
          ;;
          ;; Close over these variables so that `byte-compiler-options'
          ;; can change them on a per-file basis.
@@ -1939,6 +1942,7 @@
 	;; Make warnings about unresolved functions
 	;; give the end of the file as their position.
 	(setq byte-compile-last-position (point-max))
+	(byte-compile--higher-order--warn-about-unresolved-function-names)
 	(byte-compile-warn-about-unresolved-functions))
       ;; Fix up the header at the front of the output
       ;; if the buffer contains multibyte characters.
@@ -2960,6 +2964,7 @@
 (defun byte-compile-normal-call (form)
   (when (and (byte-compile-warning-enabled-p 'callargs)
              (symbolp (car form)))
+    (byte-compile--higher-order--check-arguments form)
     (if (memq (car form)
               '(custom-declare-group custom-declare-variable
                                      custom-declare-face))
@@ -4659,6 +4664,177 @@
 	      (indent-to 40)
 	      (insert (int-to-string n) "\n")))
 	(setq i (1+ i))))))
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [BEGINNING] ;;;;;;;;
+
+(defvar byte-compile--higher-order--unresolved-functions nil
+  "A list of functions that were passed by name to higher order functions but where not known to be defined.
+
+Used during compilation to report functions referred to by (quote
+SYMBOL) or (function SYMBOL) that are not known to be defined.
+
+ See also
+`byte-compile--higher-order--warn-about-unresolved-function-names',
+`byte-compile--higher-order--check-arguments'.")
+
+(defun byte-compile--higher-order--warn-about-unresolved-function-names ()
+  "Emit compiler warnings when there are unresolved function a symbols passed to higher order functions.
+See also `byte-compile--higher-order--unresolved-functions'."
+  ;; The logic is similiar to `byte-compile-warn-about-unresolved-functions'.
+  (when (byte-compile-warning-enabled-p 'unresolved)
+    (let ((byte-compile-current-form :end)
+	  not-defined-at-runtime-maybe-symbol-list
+	  resolved-to-function-list
+	  resolved-to-macro-list
+	  unresolved-symbol-list)
+    (dolist (function-name byte-compile--higher-order--unresolved-functions)
+      (cond ((assq function-name byte-compile-function-environment)
+	     (push function-name resolved-to-function-list))
+	    ((assq function-name byte-compile-macro-environment)
+	     (push function-name resolved-to-macro-list))
+	    ((fboundp function-name)
+	     (push function-name not-defined-at-runtime-maybe-symbol-list))
+	    (t (push function-name unresolved-symbol-list))))
+    (dolist (resolved-name resolved-to-function-list)
+      (setq byte-compile--higher-order--unresolved-functions
+	    (delq resolved-name
+		  byte-compile--higher-order--unresolved-functions)))
+    (byte-compile-print-syms 
+     "The symbol `%S' was passed to a function as if the name of a function, but is bound to a macro instead."
+     "The following symbols were passed to a function as if the names functions but are bound to macros instead:"
+     resolved-to-macro-list)
+    (byte-compile-print-syms
+     "The symbol `%S' was used as function name but the function is not known to be defined."
+     "The following symbols were used as function names but the functions are not known to be defined:"
+     unresolved-symbol-list)
+    (byte-compile-print-syms
+     "The function `%S' was used as function name but the function might not be defined at runtime."
+     "The following symbols were used as function names but the functions might not be defined at runtime:"
+     not-defined-at-runtime-maybe-symbol-list))))
+
+(defun byte-compile--higher-order--check-arguments (form)
+  "On FORM, which is a function call, check whether functions passed by name are known to be defined.
+
+If a parameter has the form (function SYMBOL) or it is known to
+be expected to be a function through the `higher-order-arguments'
+property of the symbol (car FORM) and has the form (quote SYMBOL)
+we either warn right away, if SYMBOL is known to be bound to
+something that cannot be passed to higher order functions such as
+special forms and macros. If SYMBOL's function cell is empty, we
+queue the symbol into
+`byte-compile--higher-order--unresolved-functions' for checking
+at the end of the file, see
+`byte-compile--higher-order--warn-about-unresolved-function-names'."
+  (let ((higher-order-arguments
+	 (and (symbolp (car form))
+	      (get (car form) 'higher-order-arguments)))
+	(pos -1))
+    (dolist (subform (cdr form))
+      (setq pos (1+ pos))
+      ;; Check when subform has form (function SYMBOL) or (quote SYMBOL)
+      ;; and position matches one mentioned in higher-order-arguments.
+      (when (or (and (listp subform) ;; case: (function SYMBOL)
+		     (eq (car subform) 'function)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform))))
+		(and (listp subform) ;; case: (quote SYMBOL)
+		     (eq (car subform) 'quote)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform)))
+		     (member pos higher-order-arguments)))
+	(let ((function-name (car (cdr subform))))
+	  ;; Give a style warning when form (quote function) was used.
+	  (when (and (byte-compile-warning-enabled-p 'funcargs)
+		     (eq (car subform) 'quote))
+	    (byte-compile-warn 
+	     "%S: Function argument should be passed as #'%S, not %S"
+	     (car form) (car (cdr subform)) subform))
+	  ;; Is the function already defined or already known to be
+	  ;; defined in this file? Note that macros are not allowed!
+	  (cond ((assq function-name byte-compile-function-environment)) ;; Do nothing.
+		((assq function-name byte-compile-macro-environment)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		((functionp function-name)) ;; No nothing, is ok.
+		((fboundp function-name)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		(t ;; Unknown symbol
+		 (add-to-list 'byte-compile--higher-order--unresolved-functions
+			      function-name))))))))
+
+;; Placing the entry inside the definition of
+;; `defun-declaration-alist' directly for some reason doesn't work.
+(add-to-list 'defun-declarations-alist
+	     '(higher-order-arguments 
+	       byte-compile--higher-order--declare-form-handler))
+
+(defun byte-compile--higher-order--declare-form-handler (f _args &rest val)
+  "Handler for the (declare (higher-order-arguments [NUM ...])) form, see `defun-declarations-alist'.
+
+F is the name of the function, _ARGS the formal parameter list,
+VAL should be (NUM ..).
+
+LIMITATION: When the function has been used before it was
+defined, a compiler warning is emitted pointing out that prior
+use might not have been checked."
+  ;; Check that VAL is list of integers.
+  (if (or (not (listp val))
+	  (delete nil
+	    (mapcar (lambda (e) (not (integerp e)))
+	      val)))
+      ;; Warn about invalid `declare' form
+      (let ((warn-text 
+	     (concat "(declare ..) form (higher-order-arguments [NUM ..]) with invalid arguments "
+		     (mapconcat (lambda (e) (format "%S" e)) val " "))))
+	(if byte-compile-current-file 
+	    (byte-compile-warn "%s" warn-text)
+	  (warn "%s" warn-text))
+	nil)
+    ;; Return a `put' form, but also perform it at compile time.
+    ;; Limitation: Affacts only higher order functions defined before
+    ;; their first use.
+    (when (assq f byte-compile-unresolved-functions)
+      (byte-compile-warn 
+       "%s"
+       (concat "Higher order function " (symbol-name f) " was used before it was defined. "
+	       "Actual parameters were not checked for validity.")))
+    (list 'eval-and-compile 
+	  (list 'put (list 'quote f) ''higher-order-arguments
+		(list 'quote val)))))
+
+;; Setting the property for certain known functions.
+
+(defmacro byte-compile--higher-order--register-known-functions (pos &rest symbol-list)
+  "For elements of SYMBOL-LIST set the higher-order-arguments property to (POS) when it is not yet set.
+
+Only for backward compatibility use in bytecomp.el. If you want
+to set the property for your own functions use
+the (declare (higher-order [NUM ..])) form."
+  (declare (indent 1))
+  (let (forms)
+    (dolist (function-name symbol-list)
+      (push (list 'if (list 'not (list 'get (list 'quote function-name) ''higher-order-arguments))
+		  (list 'put (list 'quote function-name) ''higher-order-arguments 
+			(list 'quote (list pos))))
+	    forms))
+    (cons 'progn (reverse forms))))
+
+(byte-compile--higher-order--register-known-functions 0
+  ;; Functions defined in C-source.
+  apply funcall map-char-table map-charset-chars map-keymap
+  map-keymap-internal mapatoms mapc mapcar mapconcat maphash
+  call-interactively
+  ;; Functions defined in standard elisp files
+  map-keymap-sorted map-query-replace-regexp map-y-or-n-p
+  ;; From the deprecated `cl'
+  map mapcan mapcar* mapcon mapl maplist
+  ;; Non-obsolete non-internal functions from `cl-lib'
+  cl-map cl-mapc cl-mapcan cl-mapcar cl-mapcon cl-maphash cl-mapl
+  cl-maplist)
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [END] ;;;;;;;;;;;;;;
+
 \f
 ;; To avoid "lisp nesting exceeds max-lisp-eval-depth" when bytecomp compiles
 ;; itself, compile some of its most used recursive functions (at load time).

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>> (mapcar (if flag #'f1 #'f2) list)
>> and `f1' or `f2' are unknown.

> My code does that and indeed that was, what I first implemented.

Could you point to the place where your code does that?
It should be a few lines of extra code in byte-compile-function-form,
but I couldn't find it in your patch.


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

The patch only adds a single line of code to byte-compile-normal-call
which calls defun byte-compile--higher-order--check-arguments.

@@ -2960,6 +2964,7 @@
 (defun byte-compile-normal-call (form)
   (when (and (byte-compile-warning-enabled-p 'callargs)
              (symbolp (car form)))
+    (byte-compile--higher-order--check-arguments form)
     (if (memq (car form)
               '(custom-declare-group custom-declare-variable
                                      custom-declare-face))
@@ -4659,6 +4664,177 @@


Here the lines


+    (dolist (subform (cdr form))
+      (setq pos (1+ pos))
+      ;; Check when subform has form (function SYMBOL) or (quote SYMBOL)
+      ;; and position matches one mentioned in
higher-order-arguments.+      (when (or (and (listp subform) ;; case:
(function SYMBOL)
+      (when (or (and (listp subform) ;; case: (function SYMBOL)
+                    (eq (car subform) 'function)
+                    (symbolp (car (cdr subform)))
+                    (null (cdr (cdr subform))))
+               (and (listp subform) ;; case: (quote SYMBOL)
+                    (eq (car subform) 'quote)
+                    (symbolp (car (cdr subform)))
+                    (null (cdr (cdr subform)))
+                    (member pos higher-order-arguments)))

decide whether the argument will be considered a quoted function and
corresponding checks will be performed or not.

The first part of the `or' checks for arguments of the form (function
SYMBOL), the second part for arguments of the form (quote SYMBOL)
where for the called function's name, i.e. (car form), the position of
the (quote SYMBOL) argument is one of the integers in the list stored
in the property 'higher-order-arguments (stored to a local variable to
avoided repeated `get' calls). The in new version of the patch
(*.patch-2) additionally a warning is printed in the (quote SYMBOL)
case asking the user to change it to #'SYMBOL.

  - Klaus

2013/8/7 Stefan Monnier <monnier@iro.umontreal.ca>:
>>> (mapcar (if flag #'f1 #'f2) list)
>>> and `f1' or `f2' are unknown.
>
>> My code does that and indeed that was, what I first implemented.
>
> Could you point to the place where your code does that?
> It should be a few lines of extra code in byte-compile-function-form,
> but I couldn't find it in your patch.
>
>
>         Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

> The patch only adds a single line of code to byte-compile-normal-call
> which calls defun byte-compile--higher-order--check-arguments.

But there's no normal call in (if a #'foo1 #'foo2), so how can this work?


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The patch only adds a single line of code to byte-compile-normal-call
>> which calls defun byte-compile--higher-order--check-arguments.

> But there's no normal call in (if a #'foo1 #'foo2), so how can this work?


>         Stefan

I installed the patch below which checks that #'foo refers to a function
that will exist.


        Stefan


=== modified file 'lisp/ChangeLog'
--- lisp/ChangeLog	2013-08-07 16:37:04 +0000
+++ lisp/ChangeLog	2013-08-07 17:33:11 +0000
@@ -1,3 +1,11 @@
+2013-08-07  Stefan Monnier  <monnier@iro.umontreal.ca>
+
+	* emacs-lisp/bytecomp.el: Check existence of f in #'f.
+	(byte-compile-callargs-warn): Use `push'.
+	(byte-compile-arglist-warn): Ignore higher-order "calls".
+	(byte-compile-file-form-autoload): Use `pcase'.
+	(byte-compile-function-form): If quoting a symbol, check that it exists.
+
 2013-08-07  Eli Zaretskii  <eliz@gnu.org>
 
 	* progmodes/dos.el (dos-font-lock-keywords): Rename LINUX to UNIX

=== modified file 'lisp/emacs-lisp/bytecomp.el'
--- lisp/emacs-lisp/bytecomp.el	2013-06-19 07:35:00 +0000
+++ lisp/emacs-lisp/bytecomp.el	2013-08-07 17:20:38 +0000
@@ -1364,7 +1364,10 @@
       ;; This is the first definition.  See if previous calls are compatible.
       (let ((calls (assq name byte-compile-unresolved-functions))
 	    nums sig min max)
-	(when calls
+        (setq byte-compile-unresolved-functions
+              (delq calls byte-compile-unresolved-functions))
+        (setq calls (delq t calls))  ;Ignore higher-order uses of the function.
+	(when (cdr calls)
           (when (and (symbolp name)
                      (eq (function-get name 'byte-optimizer)
                          'byte-compile-inline-expand))
@@ -1382,10 +1385,7 @@
              name
              (byte-compile-arglist-signature-string sig)
              (if (equal sig '(1 . 1)) " arg" " args")
-             (byte-compile-arglist-signature-string (cons min max))))
-
-          (setq byte-compile-unresolved-functions
-                (delq calls byte-compile-unresolved-functions)))))))
+             (byte-compile-arglist-signature-string (cons min max)))))))))
 
 (defvar byte-compile-cl-functions nil
   "List of functions defined in CL.")
@@ -3574,9 +3570,31 @@
 ;; and (funcall (function foo)) will lose with autoloads.
 
 (defun byte-compile-function-form (form)
-  (byte-compile-constant (if (eq 'lambda (car-safe (nth 1 form)))
-                             (byte-compile-lambda (nth 1 form))
-                           (nth 1 form))))
+  (let ((f (nth 1 form)))
+    (when (and (symbolp f)
+               (byte-compile-warning-enabled-p 'callargs))
+      (when (get f 'byte-obsolete-info)
+        (byte-compile-warn-obsolete (car form)))
+
+      ;; Check to see if the function will be available at runtime
+      ;; and/or remember its arity if it's unknown.
+      (or (and (or (fboundp f)          ; Might be a subr or autoload.
+                   (byte-compile-fdefinition (car form) nil))
+               (not (memq f byte-compile-noruntime-functions)))
+          (eq f byte-compile-current-form) ; ## This doesn't work
+                                           ; with recursion.
+          ;; It's a currently-undefined function.
+          ;; Remember number of args in call.
+          (let ((cons (assq f byte-compile-unresolved-functions)))
+            (if cons
+                (or (memq t (cdr cons))
+                    (push t (cdr cons)))
+              (push (list f t)
+                    byte-compile-unresolved-functions)))))
+
+    (byte-compile-constant (if (eq 'lambda (car-safe f))
+                               (byte-compile-lambda f)
+                             f))))
 
 (defun byte-compile-indent-to (form)
   (let ((len (length form)))

Re: Can the byte-compiler check whether functions passed by name are defined?

[Glenn Morris]

Stefan Monnier wrote:

> I installed the patch below which checks that #'foo refers to a function
> that will exist.

It seems overly pessimistic:

foo.el:

(defun foo (&rest args)
  t)

(defun foo2 ()
  (mapcar #'foo '(1 2 3)))

Byte-compile foo.el:

In end of data:
foo.el:6:1:Warning: the function `foo' is not known to be defined.

Re: Can the byte-compiler check whether functions passed by name are defined?

[Glenn Morris]

Glenn Morris wrote:

> It seems overly pessimistic:

Surreal, even:

foo.el:

(defun foo ()
  #'assoc-ignore-case)

In foo:
foo.el:1:8:Warning: `function' is an obsolete variable.

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

> (defun foo (&rest args) t)
> (defun foo2 () (mapcar #'foo '(1 2 3)))
[...]
> foo.el:6:1:Warning: the function `foo' is not known to be defined.

> (defun foo () #'assoc-ignore-case)
[...]
> foo.el:1:8:Warning: `function' is an obsolete variable.

The patch below should fix them, indeed, thanks.


        Stefan


=== modified file 'lisp/emacs-lisp/bytecomp.el'
--- lisp/emacs-lisp/bytecomp.el	2013-08-07 17:33:30 +0000
+++ lisp/emacs-lisp/bytecomp.el	2013-08-08 01:23:52 +0000
@@ -3574,12 +3574,12 @@
     (when (and (symbolp f)
                (byte-compile-warning-enabled-p 'callargs))
       (when (get f 'byte-obsolete-info)
-        (byte-compile-warn-obsolete (car form)))
+        (byte-compile-warn-obsolete f))
 
       ;; Check to see if the function will be available at runtime
       ;; and/or remember its arity if it's unknown.
       (or (and (or (fboundp f)          ; Might be a subr or autoload.
-                   (byte-compile-fdefinition (car form) nil))
+                   (byte-compile-fdefinition f nil))
                (not (memq f byte-compile-noruntime-functions)))
           (eq f byte-compile-current-form) ; ## This doesn't work
                                            ; with recursion.

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 2698 bytes --]

2013/8/8 Stefan Monnier <monnier@iro.umontreal.ca>:
>> (defun foo (&rest args) t)
>> (defun foo2 () (mapcar #'foo '(1 2 3)))
> [...]
>> foo.el:6:1:Warning: the function `foo' is not known to be defined.
>
>> (defun foo () #'assoc-ignore-case)
> [...]
>> foo.el:1:8:Warning: `function' is an obsolete variable.
>
> The patch below should fix them, indeed, thanks.
>
>
>         Stefan
>
>
> === modified file 'lisp/emacs-lisp/bytecomp.el'
> --- lisp/emacs-lisp/bytecomp.el 2013-08-07 17:33:30 +0000
> +++ lisp/emacs-lisp/bytecomp.el 2013-08-08 01:23:52 +0000
> @@ -3574,12 +3574,12 @@
>      (when (and (symbolp f)
>                 (byte-compile-warning-enabled-p 'callargs))
>        (when (get f 'byte-obsolete-info)
> -        (byte-compile-warn-obsolete (car form)))
> +        (byte-compile-warn-obsolete f))
>
>        ;; Check to see if the function will be available at runtime
>        ;; and/or remember its arity if it's unknown.
>        (or (and (or (fboundp f)          ; Might be a subr or autoload.
> -                   (byte-compile-fdefinition (car form) nil))
> +                   (byte-compile-fdefinition f nil))
>                 (not (memq f byte-compile-noruntime-functions)))
>            (eq f byte-compile-current-form) ; ## This doesn't work
>                                             ; with recursion.
>

Applying either of your patches fails for me; Is there some
possibility we can check, that we are actually starting from the same
version of bytecomp.el? I have to admit, I don't have particularily
much experience with patchibg files...

What I intended to check: Does your code allow defining a function
after its first use as #'FUNCTION, as is allowed with normal function
calls (FUNCTION ...)?

>> (funcall (if t #'foo) "Hello World")
>
>(if t #'foo) gets optimized into #'foo.

Okay, I missed that one. If I write

    (funcall (if (some-condition) #'foo) nil)

it warns only about some-condition. I have now moved the call to
`byte-compile--higher-order--check-arguments' to `byte-compile-form'
in my code.

The code

(funcall (if (some-condition) #'foo) nil)
(mapc 'foo nil)
(ignore '(#'bar))

No produces the warnings

-------------------------------------------------------
In toplevel form:
id1379159421d7d15c.el:7:1:Warning: mapc: Function argument should be passed as
    #'foo, not (quote foo)

In end of data:
id1379159421d7d15c.el:11:1:Warning: The symbol `foo' was used as function name
    but the function is not known to be defined.
id1379159421d7d15c.el:11:1:Warning: the function `some-condition' is not known
    to be defined.
Wrote c:/tmp/id1379159421d7d15c.elc
Done.
-------------------------------------------------------


  - Klaus

[-- Attachment #2: bytecomp.el.klaus3.patch --]
[-- Type: application/octet-stream, Size: 13191 bytes --]

--- bytecomp.el.orig	2013-08-08 10:06:58.202295300 +0200
+++ bytecomp.el	2013-08-08 10:40:00.083652500 +0200
@@ -284,6 +284,8 @@
   free-vars   references to variables not in the current lexical scope.
   unresolved  calls to unknown functions.
   callargs    function calls with args that don't match the definition.
+  funcargs    higher order function like mapcar used with function argument 
+              as (quote SYMBOL) rather than (function SYMBOL)
   redefine    function name redefined from a macro to ordinary function or vice
               versa, or redefined to take a different number of arguments.
   obsolete    obsolete variables and functions.
@@ -1364,10 +1366,7 @@
       ;; This is the first definition.  See if previous calls are compatible.
       (let ((calls (assq name byte-compile-unresolved-functions))
 	    nums sig min max)
-        (setq byte-compile-unresolved-functions
-              (delq calls byte-compile-unresolved-functions))
-        (setq calls (delq t calls))  ;Ignore higher-order uses of the function.
-       (when (cdr calls)
+	(when calls
           (when (and (symbolp name)
                      (eq (function-get name 'byte-optimizer)
                          'byte-compile-inline-expand))
@@ -1385,7 +1384,10 @@
              name
              (byte-compile-arglist-signature-string sig)
              (if (equal sig '(1 . 1)) " arg" " args")
-             (byte-compile-arglist-signature-string (cons min max)))))))))
+             (byte-compile-arglist-signature-string (cons min max))))
+
+          (setq byte-compile-unresolved-functions
+                (delq calls byte-compile-unresolved-functions)))))))
 
 (defvar byte-compile-cl-functions nil
   "List of functions defined in CL.")
@@ -1496,6 +1498,7 @@
          (byte-compile-const-variables nil)
          (byte-compile-free-references nil)
          (byte-compile-free-assignments nil)
+	 (byte-compile--higher-order--unresolved-functions nil)
          ;;
          ;; Close over these variables so that `byte-compiler-options'
          ;; can change them on a per-file basis.
@@ -1939,6 +1942,7 @@
 	;; Make warnings about unresolved functions
 	;; give the end of the file as their position.
 	(setq byte-compile-last-position (point-max))
+	(byte-compile--higher-order--warn-about-unresolved-function-names)
 	(byte-compile-warn-about-unresolved-functions))
       ;; Fix up the header at the front of the output
       ;; if the buffer contains multibyte characters.
@@ -2929,6 +2933,8 @@
              (memq fn byte-compile-interactive-only-functions)
              (byte-compile-warn "`%s' used from Lisp code\n\
 That command is designed for interactive use only" fn))
+	(and (byte-compile-warning-enabled-p 'unresolved)
+	     (byte-compile--higher-order--check-arguments form))
         (if (and (fboundp (car form))
                  (eq (car-safe (symbol-function (car form))) 'macro))
             (byte-compile-log-warning
@@ -3574,31 +3580,9 @@
 ;; and (funcall (function foo)) will lose with autoloads.
 
 (defun byte-compile-function-form (form)
-  (let ((f (nth 1 form)))
-    (when (and (symbolp f)
-               (byte-compile-warning-enabled-p 'callargs))
-      (when (get f 'byte-obsolete-info)
-        (byte-compile-warn-obsolete (car form)))
-
-      ;; Check to see if the function will be available at runtime
-      ;; and/or remember its arity if it's unknown.
-      (or (and (or (fboundp f)          ; Might be a subr or autoload.
-                   (byte-compile-fdefinition (car form) nil))
-               (not (memq f byte-compile-noruntime-functions)))
-          (eq f byte-compile-current-form) ; ## This doesn't work
-                                           ; with recursion.
-          ;; It's a currently-undefined function.
-          ;; Remember number of args in call.
-          (let ((cons (assq f byte-compile-unresolved-functions)))
-            (if cons
-                (or (memq t (cdr cons))
-                    (push t (cdr cons)))
-              (push (list f t)
-                    byte-compile-unresolved-functions)))))
-
-    (byte-compile-constant (if (eq 'lambda (car-safe f))
-                               (byte-compile-lambda f)
-                             f))))
+  (byte-compile-constant (if (eq 'lambda (car-safe (nth 1 form)))
+                             (byte-compile-lambda (nth 1 form))
+                           (nth 1 form))))
 
 (defun byte-compile-indent-to (form)
   (let ((len (length form)))
@@ -4681,6 +4665,177 @@
 	      (indent-to 40)
 	      (insert (int-to-string n) "\n")))
 	(setq i (1+ i))))))
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [BEGINNING] ;;;;;;;;
+
+(defvar byte-compile--higher-order--unresolved-functions nil
+  "A list of functions that were passed by name to higher order functions but where not known to be defined.
+
+Used during compilation to report functions referred to by (quote
+SYMBOL) or (function SYMBOL) that are not known to be defined.
+
+ See also
+`byte-compile--higher-order--warn-about-unresolved-function-names',
+`byte-compile--higher-order--check-arguments'.")
+
+(defun byte-compile--higher-order--warn-about-unresolved-function-names ()
+  "Emit compiler warnings when there are unresolved function a symbols passed to higher order functions.
+See also `byte-compile--higher-order--unresolved-functions'."
+  ;; The logic is similiar to `byte-compile-warn-about-unresolved-functions'.
+  (when (byte-compile-warning-enabled-p 'unresolved)
+    (let ((byte-compile-current-form :end)
+	  not-defined-at-runtime-maybe-symbol-list
+	  resolved-to-function-list
+	  resolved-to-macro-list
+	  unresolved-symbol-list)
+    (dolist (function-name byte-compile--higher-order--unresolved-functions)
+      (cond ((assq function-name byte-compile-function-environment)
+	     (push function-name resolved-to-function-list))
+	    ((assq function-name byte-compile-macro-environment)
+	     (push function-name resolved-to-macro-list))
+	    ((fboundp function-name)
+	     (push function-name not-defined-at-runtime-maybe-symbol-list))
+	    (t (push function-name unresolved-symbol-list))))
+    (dolist (resolved-name resolved-to-function-list)
+      (setq byte-compile--higher-order--unresolved-functions
+	    (delq resolved-name
+		  byte-compile--higher-order--unresolved-functions)))
+    (byte-compile-print-syms 
+     "The symbol `%S' was passed to a function as if the name of a function, but is bound to a macro instead."
+     "The following symbols were passed to a function as if the names functions but are bound to macros instead:"
+     resolved-to-macro-list)
+    (byte-compile-print-syms
+     "The symbol `%S' was used as function name but the function is not known to be defined."
+     "The following symbols were used as function names but the functions are not known to be defined:"
+     unresolved-symbol-list)
+    (byte-compile-print-syms
+     "The function `%S' was used as function name but the function might not be defined at runtime."
+     "The following symbols were used as function names but the functions might not be defined at runtime:"
+     not-defined-at-runtime-maybe-symbol-list))))
+
+(defun byte-compile--higher-order--check-arguments (form)
+  "On FORM, which is a function call, check whether functions passed by name are known to be defined.
+
+If a parameter has the form (function SYMBOL) or it is known to
+be expected to be a function through the `higher-order-arguments'
+property of the symbol (car FORM) and has the form (quote SYMBOL)
+we either warn right away, if SYMBOL is known to be bound to
+something that cannot be passed to higher order functions such as
+special forms and macros. If SYMBOL's function cell is empty, we
+queue the symbol into
+`byte-compile--higher-order--unresolved-functions' for checking
+at the end of the file, see
+`byte-compile--higher-order--warn-about-unresolved-function-names'."
+  (let ((higher-order-arguments
+	 (and (symbolp (car form))
+	      (get (car form) 'higher-order-arguments)))
+	(pos -1))
+    (dolist (subform (cdr form))
+      (setq pos (1+ pos))
+      ;; Check when subform has form (function SYMBOL) or (quote SYMBOL)
+      ;; and position matches one mentioned in higher-order-arguments.
+      (when (or (and (listp subform) ;; case: (function SYMBOL)
+		     (eq (car subform) 'function)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform))))
+		(and (listp subform) ;; case: (quote SYMBOL)
+		     (eq (car subform) 'quote)
+		     (symbolp (car (cdr subform)))
+		     (null (cdr (cdr subform)))
+		     (member pos higher-order-arguments)))
+	(let ((function-name (car (cdr subform))))
+	  ;; Give a style warning when form (quote function) was used.
+	  (when (and (byte-compile-warning-enabled-p 'funcargs)
+		     (eq (car subform) 'quote))
+	    (byte-compile-warn 
+	     "%S: Function argument should be passed as #'%S, not %S"
+	     (car form) (car (cdr subform)) subform))
+	  ;; Is the function already defined or already known to be
+	  ;; defined in this file? Note that macros are not allowed!
+	  (cond ((assq function-name byte-compile-function-environment)) ;; Do nothing.
+		((assq function-name byte-compile-macro-environment)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		((functionp function-name)) ;; No nothing, is ok.
+		((fboundp function-name)
+		 (byte-compile-warn "Invalid argument `%S' to function `%S': Not a function."
+				    function-name (car form)))
+		(t ;; Unknown symbol
+		 (add-to-list 'byte-compile--higher-order--unresolved-functions
+			      function-name))))))))
+
+;; Placing the entry inside the definition of
+;; `defun-declaration-alist' directly for some reason doesn't work.
+(add-to-list 'defun-declarations-alist
+	     '(higher-order-arguments 
+	       byte-compile--higher-order--declare-form-handler))
+
+(defun byte-compile--higher-order--declare-form-handler (f _args &rest val)
+  "Handler for the (declare (higher-order-arguments [NUM ...])) form, see `defun-declarations-alist'.
+
+F is the name of the function, _ARGS the formal parameter list,
+VAL should be (NUM ..).
+
+LIMITATION: When the function has been used before it was
+defined, a compiler warning is emitted pointing out that prior
+use might not have been checked."
+  ;; Check that VAL is list of integers.
+  (if (or (not (listp val))
+	  (delete nil
+	    (mapcar (lambda (e) (not (integerp e)))
+	      val)))
+      ;; Warn about invalid `declare' form
+      (let ((warn-text 
+	     (concat "(declare ..) form (higher-order-arguments [NUM ..]) with invalid arguments "
+		     (mapconcat (lambda (e) (format "%S" e)) val " "))))
+	(if byte-compile-current-file 
+	    (byte-compile-warn "%s" warn-text)
+	  (warn "%s" warn-text))
+	nil)
+    ;; Return a `put' form, but also perform it at compile time.
+    ;; Limitation: Affacts only higher order functions defined before
+    ;; their first use.
+    (when (assq f byte-compile-unresolved-functions)
+      (byte-compile-warn 
+       "%s"
+       (concat "Higher order function " (symbol-name f) " was used before it was defined. "
+	       "Actual parameters were not checked for validity.")))
+    (list 'eval-and-compile 
+	  (list 'put (list 'quote f) ''higher-order-arguments
+		(list 'quote val)))))
+
+;; Setting the property for certain known functions.
+
+(defmacro byte-compile--higher-order--register-known-functions (pos &rest symbol-list)
+  "For elements of SYMBOL-LIST set the higher-order-arguments property to (POS) when it is not yet set.
+
+Only for backward compatibility use in bytecomp.el. If you want
+to set the property for your own functions use
+the (declare (higher-order [NUM ..])) form."
+  (declare (indent 1))
+  (let (forms)
+    (dolist (function-name symbol-list)
+      (push (list 'if (list 'not (list 'get (list 'quote function-name) ''higher-order-arguments))
+		  (list 'put (list 'quote function-name) ''higher-order-arguments 
+			(list 'quote (list pos))))
+	    forms))
+    (cons 'progn (reverse forms))))
+
+(byte-compile--higher-order--register-known-functions 0
+  ;; Functions defined in C-source.
+  apply funcall map-char-table map-charset-chars map-keymap
+  map-keymap-internal mapatoms mapc mapcar mapconcat maphash
+  call-interactively
+  ;; Functions defined in standard elisp files
+  map-keymap-sorted map-query-replace-regexp map-y-or-n-p
+  ;; From the deprecated `cl'
+  map mapcan mapcar* mapcon mapl maplist
+  ;; Non-obsolete non-internal functions from `cl-lib'
+  cl-map cl-mapc cl-mapcan cl-mapcar cl-mapcon cl-maphash cl-mapl
+  cl-maplist)
+
+;;;; (DECLARE (HIGHER-ORDER-ARGUMENTS)) AND CHECK [END] ;;;;;;;;;;;;;;
+
 \f
 ;; To avoid "lisp nesting exceeds max-lisp-eval-depth" when bytecomp compiles
 ;; itself, compile some of its most used recursive functions (at load time).

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

> Applying either of your patches fails for me; Is there some
> possibility we can check, that we are actually starting from the same
> version of bytecomp.el? I have to admit, I don't have particularily
> much experience with patchibg files...

Check the code in Emacs's trunk (e.g. http://bzr.savannah.gnu.org/lh/emacs/trunk/annotate/head:/lisp/emacs-lisp/bytecomp.el).

> What I intended to check: Does your code allow defining a function
> after its first use as #'FUNCTION, as is allowed with normal function
> calls (FUNCTION ...)?

Yes: it reuses the code already used for function calls.


        Stefan

Re: Can the byte-compiler check whether functions passed by name are defined?

[Klaus-Dieter Bauer]

2013/8/7 Stefan Monnier <monnier@iro.umontreal.ca>:
>> The patch only adds a single line of code to byte-compile-normal-call
>> which calls defun byte-compile--higher-order--check-arguments.
>
> But there's no normal call in (if a #'foo1 #'foo2), so how can this work?
>
>
>         Stefan

Good point. It does though.

Input file:
--------------------------------------------------
;; -*- mode:emacs-lisp;coding:utf-8-unix;lexical-binding:t;byte-compile-dynamic:t;
-*-

(funcall (if t #'foo) "Hello World")
--------------------------------------------------

Compiler Log:
--------------------------------------------------
Compiling file c:/tmp/id1379159421d7d15c.el at Wed Aug  7 21:53:58 2013

In end of data:
id1379159421d7d15c.el:4:1:Warning: the function `foo' is not known to be
    defined.
--------------------------------------------------

I guess the use of the function just doesn't match its name?

  - Klaus

Re: Can the byte-compiler check whether functions passed by name are defined?

[Stefan Monnier]

> (funcall (if t #'foo) "Hello World")

(if t #'foo) gets optimized into #'foo.


        Stefan