Distinguishing `consp` and `functionp`

Distinguishing `consp` and `functionp`

[Stefan Monnier]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

I've been annoyed at the use of lists to represent function values for
a while now.  For a reason I cannot fathom, I even managed to reproduce
that very same mistake in Emacs-24 with the `(closure ...)` value for
statically scoped interpreted function values.

That was a major blunder.

In any case, I'm playing around with a "fix", making lambda evaluate
(when interpreted) not to (lambda ...) or (closure ...) but to
a self-evaluating value that can be more reliably distinguished.
In the patch below, I just reused the #[...] byte-code objects for that,
putting the function's body where the bytecode string goes and the
function's captured environment where the "constant vector" goes.

It's got several rough edges (most importantly that
`byte-code-function-p` returns non-nil for interpreted function values),
but it seems to work OK so far.

You can't use the patch as-is because it's written against my local
branch, with various local hacks, some of which (partly) remove support
for lexical-binding==nil.
But hopefully, it's readable enough for you to form an opinion.

WDYT?


        Stefan

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: interpreted-function.diff --]
[-- Type: text/x-diff, Size: 28624 bytes --]

diff --git a/lisp/emacs-lisp/byte-opt.el b/lisp/emacs-lisp/byte-opt.el
index 94e467c31b7..a967487dce1 100644
--- a/lisp/emacs-lisp/byte-opt.el
+++ b/lisp/emacs-lisp/byte-opt.el
@@ -160,12 +160,14 @@ byte-compile-inline-expand
        (error "File `%s' didn't define `%s'" (nth 1 fn) name))
       ((and (pred symbolp) (guard (not (eq fn t)))) ;A function alias.
        (byte-compile-inline-expand (cons fn (cdr form))))
-      ((pred byte-code-function-p)
+      ((and (pred byte-code-function-p)
+            (pred compiled-function-p))
        ;; (message "Inlining byte-code for %S!" name)
        ;; The byte-code will be really inlined in byte-compile-unfold-bcf.
        (byte-compile--check-arity-bytecode form fn)
        `(,fn ,@(cdr form)))
-      ((or `(lambda . ,_) `(closure . ,_))
+      ((or `(lambda . ,_) `(closure . ,_)    ;FIXME??
+           (pred byte-code-function-p)) ;But not compiled!
        ;; While byte-compile-unfold-bcf can inline dynbind byte-code into
        ;; letbind byte-code (or any other combination for that matter), we
        ;; can only inline dynbind source into dynbind source or lexbind
@@ -3139,7 +3141,6 @@ byte-optimize-lapcode
 ;;
 (eval-when-compile
  (or (compiled-function-p (symbol-function 'byte-optimize-form))
-     (assq 'byte-code (symbol-function 'byte-optimize-form))
      (let ((byte-optimize nil)
 	   (byte-compile-warnings nil))
        (mapc (lambda (x)
diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index f3a494ab7ae..62e6cac404f 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2928,18 +2928,13 @@ byte-compile-output-as-comment
 
 (defun byte-compile--reify-function (fun)
   "Return an expression which will evaluate to a function value FUN.
-FUN should be either a `lambda' value or a `closure' value."
-  (pcase-let* (((or (and `(lambda ,args . ,body) (let env nil))
-                    `(closure ,env ,args . ,body))
-                fun)
-               (preamble nil)
-               (renv ()))
-    ;; Split docstring and `interactive' form from body.
-    (when (stringp (car body))
-      (push (pop body) preamble))
-    (when (eq (car-safe (car body)) 'interactive)
-      (push (pop body) preamble))
-    (setq preamble (nreverse preamble))
+FUN should be either an interpreted closure."
+  (let ((args (aref fun 0))
+        (body (aref fun 1))
+        (env (aref fun 2))
+        (docstring (function-documentation fun))
+        (iform (interactive-form fun))
+        (renv ()))
     ;; Turn the function's closed vars (if any) into local let bindings.
     (dolist (binding env)
       (cond
@@ -2947,9 +2942,10 @@ byte-compile--reify-function
         (push `(,(car binding) ',(cdr binding)) renv))
        ((eq binding t))
        (t (push `(defvar ,binding) body))))
-    (if (null renv)
-        `(lambda ,args ,@preamble ,@body)
-      `(let ,renv (lambda ,args ,@preamble ,@body)))))
+    (let ((fun `(lambda ,args
+                  ,@(if docstring (list docstring))
+                  ,iform ,@body)))
+      (if (null renv) fun `(let ,renv ,fun)))))
 \f
 ;;;###autoload
 (defun byte-compile (form)
@@ -2975,10 +2971,10 @@ byte-compile
         fun)
        (t
         (let (final-eval)
-          (when (or (symbolp form) (eq (car-safe fun) 'closure))
+          (when (eq (type-of fun) 'interpreted-function)
             ;; `fun' is a function *value*, so try to recover its corresponding
             ;; source code.
-            (setq lexical-binding (eq (car fun) 'closure))
+            (setq lexical-binding (not (null (aref fun 2))))
             (setq fun (byte-compile--reify-function fun))
             (setq final-eval t))
           ;; Expand macros.
@@ -3492,6 +3488,7 @@ byte-compile-form
               (funcall handler form)
             (byte-compile-normal-call form))))
        ((and (byte-code-function-p (car form))
+             (compiled-function-p (car form))
              (memq byte-optimize '(t lap)))
         (byte-compile-unfold-bcf form))
        ((byte-compile-normal-call form)))
diff --git a/lisp/emacs-lisp/cconv.el b/lisp/emacs-lisp/cconv.el
index 305c10c2d7e..5cb2470cf8b 100644
--- a/lisp/emacs-lisp/cconv.el
+++ b/lisp/emacs-lisp/cconv.el
@@ -909,7 +909,7 @@ cconv-fv
                                     (delete-dups cconv--dynbindings)))))
         (cons fvs dyns)))))
 
-(defun cconv-make-interpreted-closure (fun env)
+(defun cconv-make-interpreted-closure (args body env docstring iform)
   "Make a closure for the interpreter.
 This is intended to be called at runtime by the ELisp interpreter (when
 the code has not been compiled).
@@ -918,22 +918,23 @@ cconv-make-interpreted-closure
 i.e. a list whose elements can be either plain symbols (which indicate
 that this symbol should use dynamic scoping) or pairs (SYMBOL . VALUE)
 for the lexical bindings."
-  (cl-assert (eq (car-safe fun) 'lambda))
+  (cl-assert (listp body))
+  (cl-assert (listp args))
   (let ((lexvars (delq nil (mapcar #'car-safe env))))
     (if (or (null lexvars)
             ;; Functions with a `:closure-dont-trim-context' marker
             ;; should keep their whole context untrimmed (bug#59213).
-            (and (eq :closure-dont-trim-context (nth 2 fun))
+            (and (eq :closure-dont-trim-context (car body))
                  ;; Check the function doesn't just return the magic keyword.
-                 (nthcdr 3 fun)))
+                 (cdr body)))
         ;; The lexical environment is empty, or needs to be preserved,
         ;; so there's no need to look for free variables.
         ;; Attempting to replace ,(cdr fun) by a macroexpanded version
         ;; causes bootstrap to fail.
-        `(closure ,env . ,(cdr fun))
+        (make-interpreted-closure args body env docstring iform)
       ;; We could try and cache the result of the macroexpansion and
       ;; `cconv-fv' analysis.  Not sure it's worth the trouble.
-      (let* ((form `#',fun)
+      (let* ((form `#'(lambda ,args ,iform . ,body))
              (expanded-form
               (let ((lexical-binding t) ;; Tell macros which dialect is in use.
 	            ;; Make the macro aware of any defvar declarations in scope.
@@ -942,16 +943,17 @@ cconv-make-interpreted-closure
                          (append env macroexp--dynvars) env)))
                 (macroexpand-all form macroexpand-all-environment)))
              ;; Since we macroexpanded the body, we may as well use that.
-             (expanded-fun-cdr
+             (expanded-fun-body
               (pcase expanded-form
-                (`#'(lambda . ,cdr) cdr)
-                (_ (cdr fun))))
+                (`#'(lambda ,_args ,_iform . ,newbody) newbody)
+                (_ body)))
 
              (dynvars (delq nil (mapcar (lambda (b) (if (symbolp b) b)) env)))
              (fvs (cconv-fv expanded-form lexvars dynvars))
              (newenv (nconc (mapcar (lambda (fv) (assq fv env)) (car fvs))
                             (cdr fvs))))
-        `(closure ,newenv . ,expanded-fun-cdr)))))
+        (make-interpreted-closure args expanded-fun-body newenv
+                                  docstring iform)))))
 
 
 (provide 'cconv)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 25aa2c03a66..2974079bba7 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -758,7 +758,7 @@ cl--generic-compiler
   ;; compiled.  Otherwise the byte-compiler and all the code on
   ;; which it depends needs to be usable before cl-generic is loaded,
   ;; which imposes a significant burden on the bootstrap.
-  (if (consp (lambda (x) (+ x 1)))
+  (if (not (compiled-function-p (lambda (x) (+ x 1))))
       (lambda (exp) (eval exp t))
     ;; But do byte-compile the dispatchers once bootstrap is passed:
     ;; the performance difference is substantial (like a 5x speedup on
diff --git a/lisp/emacs-lisp/cl-preloaded.el b/lisp/emacs-lisp/cl-preloaded.el
index 896fd6b1f18..72113441f85 100644
--- a/lisp/emacs-lisp/cl-preloaded.el
+++ b/lisp/emacs-lisp/cl-preloaded.el
@@ -62,8 +62,9 @@ cl--typeof-types
     (marker integer-or-marker number-or-marker atom)
     (overlay atom) (float number number-or-marker atom)
     (window-configuration atom) (process atom) (window atom)
-    ;; FIXME: Actually `function' isn't true for special forms!
-    (subr function cons-or-function atom)
+    ;; FIXME: Actually `(compiled-)function' isn't true for special forms!
+    (subr compiled-function function cons-or-function atom)
+    (interpreted-function function cons-or-function atom)
     (byte-code-function compiled-function function cons-or-function atom)
     (module-function function cons-or-function atom)
     (buffer atom) (char-table array sequence atom)
diff --git a/lisp/emacs-lisp/cl-print.el b/lisp/emacs-lisp/cl-print.el
index a1627709092..447a4d265e6 100644
--- a/lisp/emacs-lisp/cl-print.el
+++ b/lisp/emacs-lisp/cl-print.el
@@ -237,6 +237,62 @@ cl-print-object
                               'byte-code-function object)))))
     (princ ")" stream)))
 
+(cl-defmethod cl-print-object ((object interpreted-function) stream)
+  (unless stream (setq stream standard-output))
+  (princ "#f(interpreted-function " stream)
+  (let ((args (help-function-arglist object 'preserve-names)))
+    (if args
+        (prin1 args stream)
+      (princ "()" stream)))
+  (if (eq cl-print-compiled 'raw)
+      (let ((button-start
+             (and cl-print-compiled-button
+                  (bufferp stream)
+                  (with-current-buffer stream (1+ (point))))))
+        (princ " " stream)
+        (prin1 object stream)
+        (when button-start
+          (with-current-buffer stream
+            (make-text-button button-start (point)
+                              :type 'help-byte-code
+                              'byte-code-function object))))
+    (pcase (help-split-fundoc (documentation object 'raw) object)
+      ;; Drop args which `help-function-arglist' already printed.
+      (`(,_usage . ,(and doc (guard (stringp doc))))
+       (princ " " stream)
+       (prin1 doc stream)))
+    (let ((inter (interactive-form object)))
+      (when inter
+        (princ " " stream)
+        (cl-print-object
+         (if (eq 'byte-code (car-safe (cadr inter)))
+             `(interactive ,(make-byte-code nil (nth 1 (cadr inter))
+                                            (nth 2 (cadr inter))
+                                            (nth 3 (cadr inter))))
+           inter)
+         stream)))
+    (if (eq cl-print-compiled 'disassemble)
+        (princ
+         (with-temp-buffer
+           (insert "\n")
+           (pp (aref object 1))
+           (buffer-string))
+         stream)
+      (princ " " stream)
+      (let ((button-start (and cl-print-compiled-button
+                               (bufferp stream)
+                               (with-current-buffer stream (point)))))
+        (princ (format "#<sexpcode %#x>" (sxhash object)) stream)
+        (when (eq cl-print-compiled 'static)
+          (princ " " stream)
+          (cl-print-object (aref object 2) stream))
+        (when button-start
+          (with-current-buffer stream
+            (make-text-button button-start (point)
+                              :type 'help-byte-code
+                              'byte-code-function object)))))
+    (princ ")" stream)))
+
 ;; This belongs in oclosure.el, of course, but some load-ordering issues make it
 ;; complicated.
 (cl-defmethod cl-print-object ((object accessor) stream)
diff --git a/lisp/emacs-lisp/oclosure.el b/lisp/emacs-lisp/oclosure.el
index 6d79f9c41f8..4763f2223fd 100644
--- a/lisp/emacs-lisp/oclosure.el
+++ b/lisp/emacs-lisp/oclosure.el
@@ -548,70 +548,51 @@ oclosure--fix-type
   by moving it from the docstring to the environment.
 - For compiled code, this is used as a marker which cconv uses to check that
   immutable fields are indeed not mutated."
-  (if (byte-code-function-p oclosure)
-      ;; Actually, this should never happen since the `cconv.el' should have
-      ;; optimized away the call to this function.
-      oclosure
-    ;; For byte-coded functions, we store the type as a symbol in the docstring
-    ;; slot.  For interpreted functions, there's no specific docstring slot
-    ;; so `Ffunction' turns the symbol into a string.
-    ;; We thus have convert it back into a symbol (via `intern') and then
-    ;; stuff it into the environment part of the closure with a special
-    ;; marker so we can distinguish this entry from actual variables.
-    (cl-assert (eq 'closure (car-safe oclosure)))
-    (let ((typename (nth 3 oclosure))) ;; The "docstring".
-      (cl-assert (stringp typename))
-      (push (cons :type (intern typename))
-            (cadr oclosure))
-      oclosure)))
+  (cl-assert (byte-code-function-p oclosure))
+  ;; This should happen only for interpreted closures since the `cconv.el'
+  ;; should have optimized away the call to this function.
+  oclosure)
 
 (defun oclosure--copy (oclosure mutlist &rest args)
-  (if (byte-code-function-p oclosure)
+  (cl-assert (byte-code-function-p oclosure))
+  (if (stringp (aref oclosure 1))       ;Actual byte-code
       (apply #'make-closure oclosure
              (if (null mutlist)
                  args
                (mapcar (lambda (arg) (if (pop mutlist) (list arg) arg)) args)))
-    (cl-assert (eq 'closure (car-safe oclosure))
-               nil "oclosure not closure: %S" oclosure)
-    (cl-assert (eq :type (caar (cadr oclosure))))
-    (let ((env (cadr oclosure)))
-      `(closure
-           (,(car env)
-            ,@(named-let loop ((env (cdr env)) (args args))
-                (when args
+    (cl-assert (listp (aref oclosure 1)))
+    (cl-assert (symbolp (aref oclosure 4)))
+    (let ((env (aref oclosure 2)))
+      (apply #'make-interpreted-closure
+             (aref oclosure 0)
+             (aref oclosure 1)
+             (named-let loop ((env env) (args args))
+                (if (null args) env
                   (cons (cons (caar env) (car args))
                         (loop (cdr env) (cdr args)))))
-            ,@(nthcdr (1+ (length args)) env))
-           ,@(nthcdr 2 oclosure)))))
+             (nthcdr 4 (append oclosure '()))))))
 
 (defun oclosure--get (oclosure index mutable)
-  (if (byte-code-function-p oclosure)
-      (let* ((csts (aref oclosure 2))
-             (v (aref csts index)))
-        (if mutable (car v) v))
-    (cl-assert (eq 'closure (car-safe oclosure)))
-    (cl-assert (eq :type (caar (cadr oclosure))))
-    (cdr (nth (1+ index) (cadr oclosure)))))
+  (cl-assert (byte-code-function-p oclosure))
+  (let* ((csts (aref oclosure 2)))
+    (if (vectorp csts)
+        (let ((v (aref csts index)))
+          (if mutable (car v) v))
+      (cdr (nth index csts)))))
 
 (defun oclosure--set (v oclosure index)
-  (if (byte-code-function-p oclosure)
-      (let* ((csts (aref oclosure 2))
-             (cell (aref csts index)))
-        (setcar cell v))
-    (cl-assert (eq 'closure (car-safe oclosure)))
-    (cl-assert (eq :type (caar (cadr oclosure))))
-    (setcdr (nth (1+ index) (cadr oclosure)) v)))
+  (cl-assert (byte-code-function-p oclosure))
+  (let ((csts (aref oclosure 2)))
+    (if (vectorp csts)
+        (let ((cell (aref csts index)))
+          (setcar cell v))
+      (setcdr (nth index csts) v))))
 
 (defun oclosure-type (oclosure)
   "Return the type of OCLOSURE, or nil if the arg is not a OClosure."
   (if (byte-code-function-p oclosure)
       (let ((type (and (> (length oclosure) 4) (aref oclosure 4))))
-        (if (symbolp type) type))
-    (and (eq 'closure (car-safe oclosure))
-         (let* ((env (car-safe (cdr oclosure)))
-                (first-var (car-safe env)))
-           (and (eq :type (car-safe first-var))
-                (cdr first-var))))))
+        (if (symbolp type) type))))
 
 (defconst oclosure--accessor-prototype
   ;; Use `oclosure--lambda' to circumvent a bootstrapping problem:
diff --git a/lisp/subr.el b/lisp/subr.el
index b6b6eeac843..ffea44265c9 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -4666,7 +4666,8 @@ compiled-function-p
 Does not distinguish between functions implemented in machine code
 or byte-code."
   (declare (side-effect-free error-free))
-  (or (subrp object) (byte-code-function-p object)))
+  (or (subrp object)
+      (and (byte-code-function-p object) (stringp (aref object 1)))))
 
 (defun field-at-pos (pos)
   "Return the field at position POS, taking stickiness etc into account."
diff --git a/src/alloc.c b/src/alloc.c
index c4d92c9d198..3137e7a7758 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3823,17 +3823,22 @@ and (optional) INTERACTIVE-SPEC.
 usage: (make-byte-code ARGLIST BYTE-CODE CONSTANTS DEPTH &optional DOCSTRING INTERACTIVE-SPEC &rest ELEMENTS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
-  if (! ((FIXNUMP (args[COMPILED_ARGLIST])
-	  || CONSP (args[COMPILED_ARGLIST])
-	  || NILP (args[COMPILED_ARGLIST]))
-	 && STRINGP (args[COMPILED_BYTECODE])
-	 && !STRING_MULTIBYTE (args[COMPILED_BYTECODE])
-	 && VECTORP (args[COMPILED_CONSTANTS])
-	 && FIXNATP (args[COMPILED_STACK_DEPTH])))
+  if (CONSP (args[COMPILED_BYTECODE]))
+    ;                           /* An interpreted closure.  */
+  else if ((FIXNUMP (args[COMPILED_ARGLIST])
+	    || CONSP (args[COMPILED_ARGLIST])
+	    || NILP (args[COMPILED_ARGLIST]))
+	   && STRINGP (args[COMPILED_BYTECODE])
+	   && !STRING_MULTIBYTE (args[COMPILED_BYTECODE])
+	   && VECTORP (args[COMPILED_CONSTANTS])
+	   && FIXNATP (args[COMPILED_STACK_DEPTH]))
+    {
+      /* Bytecode must be immovable.  */
+      pin_string (args[COMPILED_BYTECODE]);
+    }
+  else
     error ("Invalid byte-code object");
 
-  /* Bytecode must be immovable.  */
-  pin_string (args[COMPILED_BYTECODE]);
 
   /* We used to purecopy everything here, if purify-flag was set.  This worked
      OK for Emacs-23, but with Emacs-24's lexical binding code, it can be
diff --git a/src/data.c b/src/data.c
index f780170d683..321f7c5183c 100644
--- a/src/data.c
+++ b/src/data.c
@@ -226,7 +226,9 @@ DEFUN ("type-of", Ftype_of, Stype_of, 1, 1, 0,
         case PVEC_PROCESS: return Qprocess;
         case PVEC_WINDOW: return Qwindow;
         case PVEC_SUBR: return Qsubr;
-        case PVEC_COMPILED: return Qbyte_code_function;
+        case PVEC_COMPILED:
+          return CONSP (AREF (object, 1))
+                 ? Qinterpreted_function : Qbyte_code_function;
         case PVEC_OBARRAY: return Qobarray;
         case PVEC_BUFFER: return Qbuffer;
         case PVEC_CHAR_TABLE: return Qchar_table;
@@ -4183,6 +4185,7 @@ #define PUT_ERROR(sym, tail, msg)			\
   DEFSYM (Qwindow, "window");
   DEFSYM (Qsubr, "subr");
   DEFSYM (Qbyte_code_function, "byte-code-function");
+  DEFSYM (Qinterpreted_function, "interpreted-function");
   DEFSYM (Qbuffer, "buffer");
   DEFSYM (Qframe, "frame");
   DEFSYM (Qvector, "vector");
diff --git a/src/eval.c b/src/eval.c
index 916e9d8353a..b4c43d08e54 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -518,6 +518,30 @@ DEFUN ("quote", Fquote, Squote, 1, UNEVALLED, 0,
   return XCAR (args);
 }
 
+DEFUN ("make-interpreted-closure", Fmake_interpreted_closure,
+       Smake_interpreted_closure, 3, 5, 0,
+       doc: /* Make an interpreted closure.
+BODY should be a list of forms.
+IFORM if non-nil should be of the form (interactive ...).  */)
+  (Lisp_Object args, Lisp_Object body, Lisp_Object env,
+   Lisp_Object docstring, Lisp_Object iform)
+{
+  CHECK_LIST (body);          /* Make sure it's not confused with byte-code! */
+  if (!NILP (iform))
+    {
+      iform = Fcdr (iform);
+      return CALLN (Fmake_byte_code,
+                    args, body, env, Qnil, docstring,
+                    NILP (Fcdr (iform))
+                    ? Fcar (iform)
+                    : CALLN (Fvector, XCAR (iform), XCDR (iform)));
+    }
+  else if (!NILP (docstring))
+    return CALLN (Fmake_byte_code, args, body, env, Qnil, docstring);
+  else
+    return CALLN (Fmake_byte_code, args, body, env);
+}
+
 DEFUN ("function", Ffunction, Sfunction, 1, UNEVALLED, 0,
        doc: /* Like `quote', but preferred for objects which are functions.
 In byte compilation, `function' causes its argument to be handled by
@@ -540,26 +564,46 @@ DEFUN ("function", Ffunction, Sfunction, 1, UNEVALLED, 0,
 	 Also return a "closure" for dynamically scoped functions, just so as to
 	 distinguish lambda source expressions from their evaluated value!  */
       Lisp_Object cdr = XCDR (quoted);
-      Lisp_Object tmp = cdr;
-      if (CONSP (tmp)
-	  && (tmp = XCDR (tmp), CONSP (tmp))
-	  && (tmp = XCAR (tmp), CONSP (tmp))
-	  && (EQ (QCdocumentation, XCAR (tmp))))
-	{
-	  Lisp_Object docstring = eval_sub (Fcar (XCDR (tmp)));
-	  if (SYMBOLP (docstring) && !NILP (docstring))
-	    /* Hack for OClosures: Allow the docstring to be a symbol
-             * (the OClosure's type).  */
-	    docstring = Fsymbol_name (docstring);
-	  CHECK_STRING (docstring);
-	  cdr = Fcons (XCAR (cdr), Fcons (docstring, XCDR (XCDR (cdr))));
-	}
+      Lisp_Object args = Fcar (cdr);
+      cdr = Fcdr (cdr);
+      Lisp_Object docstring = Qnil, iform = Qnil;
+      if (CONSP (cdr))
+        {
+          docstring = XCAR (cdr);
+          if (STRINGP (docstring))
+            {
+              Lisp_Object tmp = XCDR (cdr);
+              /* Beware: The docstring can be also the return value!  */
+              if (!NILP (tmp))
+                cdr = tmp;
+            }
+          else if (CONSP (docstring)
+                   && EQ (QCdocumentation, XCAR (docstring))
+                   && (docstring = eval_sub (Fcar (XCDR (docstring))),
+                       true))
+            cdr = XCDR (cdr);
+          else
+            docstring = Qnil;   /* Not a docstring after all.  */
+        }
+      if (CONSP (cdr))
+        {
+          iform = XCAR (cdr);
+          if (CONSP (iform)
+              && EQ (Qinteractive, XCAR (iform)))
+            cdr = XCDR (cdr);
+          else
+            iform = Qnil;   /* Not an interactive-form after all.  */
+        }
+      if (NILP (cdr))
+        cdr = Fcons (Qnil, Qnil); /* Make sure the body is never empty! */
+
       if (NILP (Vinternal_make_interpreted_closure_function))
-        return Fcons (Qclosure, Fcons (Vinternal_interpreter_environment, cdr));
+        return Fmake_interpreted_closure
+            (args, cdr, Vinternal_interpreter_environment, docstring, iform);
       else
-        return call2 (Vinternal_make_interpreted_closure_function,
-                      Fcons (Qlambda, cdr),
-                      Vinternal_interpreter_environment);
+        return call5 (Vinternal_make_interpreted_closure_function,
+                      args, cdr, Vinternal_interpreter_environment,
+                      docstring, iform);
     }
   else
     /* Simply quote the argument.  */
@@ -3102,7 +3146,7 @@ fetch_and_exec_byte_code (Lisp_Object fun, ptrdiff_t args_template,
 			  ptrdiff_t nargs, Lisp_Object *args)
 {
   if (CONSP (AREF (fun, COMPILED_BYTECODE)))
-    Ffetch_bytecode (fun);
+    return Fprogn (AREF (fun, COMPILED_BYTECODE));
 
   return exec_byte_code (fun, args_template, nargs, args);
 }
@@ -3178,10 +3222,12 @@ funcall_lambda (Lisp_Object fun, ptrdiff_t nargs,
       if (FIXNUMP (syms_left))
 	return fetch_and_exec_byte_code (fun, XFIXNUM (syms_left),
 					 nargs, arg_vector);
-      /* Otherwise the bytecode object uses dynamic binding and the
-	 ARGLIST slot contains a standard formal argument list whose
-	 variables are bound dynamically below.  */
-      lexenv = Qnil;
+      /* Otherwise the bytecode object either is an interpreted closure
+	 or uses dynamic binding and the ARGLIST slot contains a standard
+	 formal argument list whose variables are bound dynamically below.  */
+      lexenv = CONSP (AREF (fun, COMPILED_BYTECODE))
+               ? AREF (fun, COMPILED_CONSTANTS)
+               : Qnil;
     }
 #ifdef HAVE_MODULES
   else if (MODULE_FUNCTIONP (fun))
@@ -3387,7 +3433,8 @@ DEFUN ("fetch-bytecode", Ffetch_bytecode, Sfetch_bytecode,
 
   if (COMPILEDP (object))
     {
-      if (CONSP (AREF (object, COMPILED_BYTECODE)))
+      if (CONSP (AREF (object, COMPILED_BYTECODE))
+	  && INTEGERP (XCDR (AREF (object, COMPILED_BYTECODE))))
 	{
 	  tem = read_doc_string (AREF (object, COMPILED_BYTECODE));
 	  if (! (CONSP (tem) && STRINGP (XCAR (tem))
@@ -4316,7 +4363,7 @@ syms_of_eval (void)
 is temporarily non-nil if `eval-expression-debug-on-error' is non-nil.
 The command `toggle-debug-on-error' toggles this.
 See also the variable `debug-on-quit' and `inhibit-debugger'.  */);
-  Vdebug_on_error = Qt;
+  Vdebug_on_error = Qnil;
 
   DEFVAR_LISP ("debug-ignored-errors", Vdebug_ignored_errors,
     doc: /* List of errors for which the debugger should not be called.
@@ -4449,6 +4496,7 @@ syms_of_eval (void)
   defsubr (&Ssetq);
   defsubr (&Squote);
   defsubr (&Sfunction);
+  defsubr (&Smake_interpreted_closure);
   defsubr (&Sdefault_toplevel_value);
   defsubr (&Sset_default_toplevel_value);
   defsubr (&Sdefvar);
diff --git a/src/lread.c b/src/lread.c
index 35f3a4e97a4..92f4c523239 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -3272,11 +3272,12 @@ bytecode_from_rev_list (Lisp_Object elems, Lisp_Object readcharfun)
   Lisp_Object *vec = XVECTOR (obj)->contents;
   ptrdiff_t size = ASIZE (obj);
 
-  if (!(size >= COMPILED_STACK_DEPTH + 1 && size <= COMPILED_INTERACTIVE + 1
+  if (!(size >= COMPILED_STACK_DEPTH && size <= COMPILED_INTERACTIVE + 1
 	&& (FIXNUMP (vec[COMPILED_ARGLIST])
 	    || CONSP (vec[COMPILED_ARGLIST])
 	    || NILP (vec[COMPILED_ARGLIST]))
-	&& FIXNATP (vec[COMPILED_STACK_DEPTH])))
+	&& (FIXNATP (vec[COMPILED_STACK_DEPTH])
+	     || !STRINGP (vec[COMPILED_BYTECODE]))))
     invalid_syntax ("Invalid byte-code object", readcharfun);
 
   if (load_force_doc_strings
diff --git a/src/regex-emacs.c b/src/regex-emacs.c
index 3f544a35e3e..282cf13edbc 100644
--- a/src/regex-emacs.c
+++ b/src/regex-emacs.c
@@ -2893,7 +2893,7 @@ group_in_compile_stack (compile_stack_type compile_stack, regnum_t regnum)
    To guarantee termination, at each iteration, either LOOP_BEG should
    get bigger, or it should stay the same and P should get bigger.  */
 static bool
-forall_firstchar_1 (re_char *p, re_char *pend,
+forall_firstchar_1 (struct re_pattern_buffer *bufp, re_char *p, re_char *pend,
                     re_char *loop_beg, re_char *loop_end,
                     bool f (const re_char *p, void *arg), void *arg)
 {
@@ -2985,8 +2985,18 @@ forall_firstchar_1 (re_char *p, re_char *pend,
                forward over a subsequent `jump`.  Recognize this pattern
                since that subsequent `jump` is the one that jumps to the
                loop-entry.  */
-	    newp2 = ((re_opcode_t) *newp2 == jump)
-	            ? extract_address (newp2 + 1) : newp2;
+	    if ((re_opcode_t) *newp2 == jump)
+	      {
+	        re_char *p3 = extract_address (newp2 + 1);
+	        /* Only recognize this pattern if one of the two destinations
+	           is going forward, otherwise we'll fall into the pessimistic
+	           "Both destinations go backward" below.
+	           This is important if the `jump` at newp2 is the end of an
+	           outer loop while the `on_failure_jump` is the end of an
+	           inner loop.  */
+	        if (p3 > p_orig || newp1 > p_orig)
+	          newp2 = p3;
+	      }
 
 	  do_twoway_jump:
 	    /* We have to check that both destinations are safe.
@@ -2998,11 +3008,17 @@ forall_firstchar_1 (re_char *p, re_char *pend,
 	      {
 #if ENABLE_CHECKING
 	        fprintf (stderr, "FORALL_FIRSTCHAR: Broken assumption2!!\n");
+	        fprintf (stderr, "Destinations: %ld and %ld!!\n",
+	                 newp1 - bufp->buffer,
+	                 newp2 - bufp->buffer);
+	        fprintf (stderr, "loop_beg = %ld and loop_end = %ld!!\n",
+	                 loop_beg - bufp->buffer,
+	                 loop_end - bufp->buffer);
 #endif
 	        return false;
               }
 
-            if (!forall_firstchar_1 (newp2, pend, loop_beg, loop_end, f, arg))
+            if (!forall_firstchar_1 (bufp, newp2, pend, loop_beg, loop_end, f, arg))
               return false;
 
 	  do_jump:
@@ -3100,7 +3116,7 @@ forall_firstchar (struct re_pattern_buffer *bufp, re_char *p, re_char *pend,
 {
   eassert (!bufp || bufp->used);
   eassert (pend || bufp->used);
-  return forall_firstchar_1 (p, pend,
+  return forall_firstchar_1 (bufp, p, pend,
                              bufp ? bufp->buffer - 1 : p,
                              bufp ? bufp->buffer + bufp->used + 1 : pend,
                              f, arg);

Re: Distinguishing `consp` and `functionp`

[Adam Porter]

Hi Stefan,

> It's got several rough edges (most importantly that
> `byte-code-function-p` returns non-nil for interpreted function values),
> but it seems to work OK so far.

Forgive me if this is noise (i.e. if that is already intended to be 
fixed): If that were not fixed, it would break a few things in non-core 
packages that I'm aware of.

Other than that, and offering no opinion on the patch itself, I'm 
certainly in favor of the intent.  :)

Thanks,
Adam

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> It's got several rough edges (most importantly that
>> `byte-code-function-p` returns non-nil for interpreted function values),
>> but it seems to work OK so far.
>
> Forgive me if this is noise (i.e. if that is already intended to be fixed):
> If that were not fixed, it would break a few things in non-core packages
> that I'm aware of.

Yes, clearly it can't fly without fixing that first :-)


        Stefan

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 25 Jan 2024 18:15:48 -0500
> 
> I've been annoyed at the use of lists to represent function values for
> a while now.  For a reason I cannot fathom, I even managed to reproduce
> that very same mistake in Emacs-24 with the `(closure ...)` value for
> statically scoped interpreted function values.
> 
> That was a major blunder.
> 
> In any case, I'm playing around with a "fix", making lambda evaluate
> (when interpreted) not to (lambda ...) or (closure ...) but to
> a self-evaluating value that can be more reliably distinguished.

Maybe I'm missing something, but I always thought that having code and
data indistinguishable is one of the strong sides of Lisp.  Are we now
going to make this advantage smaller, by deprecating the use of lists
to represent functions?

Or do I misunderstand what you are saying above?  (I admit I didn't
study the patch.)

Re: Distinguishing `consp` and `functionp`

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]

LOn Fri, Jan 26, 2024, 07:32 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Thu, 25 Jan 2024 18:15:48 -0500
> >
> > I've been annoyed at the use of lists to represent function values for
> > a while now.  For a reason I cannot fathom, I even managed to reproduce
> > that very same mistake in Emacs-24 with the `(closure ...)` value for
> > statically scoped interpreted function values.
> >
> > That was a major blunder.
> >
> > In any case, I'm playing around with a "fix", making lambda evaluate
> > (when interpreted) not to (lambda ...) or (closure ...) but to
> > a self-evaluating value that can be more reliably distinguished.
>
> Maybe I'm missing something, but I always thought that having code and
> data indistinguishable is one of the strong sides of Lisp.  Are we now
> going to make this advantage smaller, by deprecating the use of lists
> to represent functions?
>

The printed representation has this "homoiconic" property, yes. Not
necessarily the internal representation resulting from code that is
evaluated or compiled. And not desirably.

I _think_ Stefan is complaining that some objects in the lisp runtime that
are funcallable also respond to consp, and that makes it hard to
distinguish (for purposes of optimization & tooling, I suppose).

So basically:

(funcall (lambda ()))

Is fine and correct, but:

(funcall '(lambda ()))

Shouldn't be fine, yet it is.

I can't check right now, but I think I know of at least one other Lisp that
allows this.

João

[-- Attachment #2: Type: text/html, Size: 2452 bytes --]

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> Is fine and correct, but:
>
> (funcall '(lambda ()))
>
> Shouldn't be fine, yet it is.

That is related but I'm not really interested in disallowing it.

Rather I'm annoyed at the corner cases where

    (functionp (mapcar ...))

can occasionally return t, simply because the returned list happens to
start with the symbol `closure` or `lambda`.

This is one of the motivations why completion tables are defined to
allow "lists of strings" or "alists whose keys can be strings or
symbols" but not "lists of symbols": even though in practice lists of
symbols work fine 99.9% of the time, they are not reliable because
they'd be confused with a function when the first symbol happens to be
`lambda` or `closure`.

There are various other circumstances where we like to accept either
a function or a list of things, but where the test is not reliable and
can give different results depending on whether the code is compiled.


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Fri, Jan 26, 2024 at 9:13 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > Is fine and correct, but:
> >
> > (funcall '(lambda ()))
> >
> > Shouldn't be fine, yet it is.
>
> That is related but I'm not really interested in disallowing it.

Hmmm

> Rather I'm annoyed at the corner cases where
>
>     (functionp (mapcar ...))
>
> can occasionally return t, simply because the returned list happens to
> start with the symbol `closure` or `lambda`.

So you want this to return nil, but still allow 'funcall' of
said returned lists?

Or do you want to disallow

   (funcall (list 'lambda nil))

But still allow

   (funcall (quote (lambda ())))

?

FWIW, SBCL disallows both.  I think Allegro CL allows at least
one of them.

João

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> Rather I'm annoyed at the corner cases where
>>
>>     (functionp (mapcar ...))
>>
>> can occasionally return t, simply because the returned list happens to
>> start with the symbol `closure` or `lambda`.
>
> So you want this to return nil, but still allow 'funcall' of
> said returned lists?

That's right: for compatibility reasons, I think we have to support the
`funcall` case for the foreseeable future (and really, it costs very
little to do so), but I think the `functionp` case doesn't need that
level of backward compatibility.

Tho, you'll note that my patch doesn't actually change `functionp`:
that would be presumably done in a subsequent step.


        Stefan

Re: Distinguishing `consp` and `functionp`

[Daniel Mendler via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Rather I'm annoyed at the corner cases where
>>>
>>>     (functionp (mapcar ...))
>>>
>>> can occasionally return t, simply because the returned list happens to
>>> start with the symbol `closure` or `lambda`.
>>
>> So you want this to return nil, but still allow 'funcall' of
>> said returned lists?
>
> That's right: for compatibility reasons, I think we have to support the
> `funcall` case for the foreseeable future (and really, it costs very
> little to do so), but I think the `functionp` case doesn't need that
> level of backward compatibility.

What about only dropping the list-based closure representation as a
first step, switching it over to a vector-based one?

(functionp '(closure (t) nil t)) => nil (currently t)
(funcall   '(closure (t) nil t)) => error (currently t)

Unfortunately quoted lambdas are still common in packages which have not
been updated for longer. Don't you intent to preserve the following
coherent behavior of functionp and funcall?

(functionp '(lambda () t)) => t
(funcall   '(lambda () t)) => t

Would it make sense to introduce a byte compiler warning, triggered if a
quoted lambda is detected? There will be occasional false positives
however.

>         Stefan

Re: Distinguishing `consp` and `functionp`

[Alan Mackenzie]

Hello, Stefan.

On Thu, Jan 25, 2024 at 18:15:48 -0500, Stefan Monnier wrote:
> I've been annoyed at the use of lists to represent function values for
> a while now.

Why?  Lists are the natural representation, and traditional in Lisp.
They bring all sorts of advantages (like being able to manipulate code
easily in Lisp).

> For a reason I cannot fathom, I even managed to reproduce that very
> same mistake in Emacs-24 with the `(closure ...)` value for statically
> scoped interpreted function values.

> That was a major blunder.

Blunder?  Having (closure ...) alongside (lambda ...) is an
inconvenience, yes, but ...  Why a "blunder"?

> In any case, I'm playing around with a "fix", making lambda evaluate
> (when interpreted) not to (lambda ...) or (closure ...) but to
> a self-evaluating value that can be more reliably distinguished.
> In the patch below, I just reused the #[...] byte-code objects for that,
> putting the function's body where the bytecode string goes and the
> function's captured environment where the "constant vector" goes.

Please don't do that.  If you must invent a new form, then invent a new
symbol to signify it.  Misusing #[...] will lead to problems and extra
work for everybody (as, for example, using the doc string field in
oclosures for something else has led to extra work).

> It's got several rough edges (most importantly that
> `byte-code-function-p` returns non-nil for interpreted function
> values), but it seems to work OK so far.

> You can't use the patch as-is because it's written against my local
> branch, with various local hacks, some of which (partly) remove support
> for lexical-binding==nil.
> But hopefully, it's readable enough for you to form an opinion.

> WDYT?

You haven't given any substantial motivation for such a change.  Being
"annoyed" at something is hardly grounds for making such far reaching
changes.  You haven't said what advantages the change would bring,
beyond saying that the proposed function form will be "more easily
distinguished".

Any packages out there that deal with the internal format of Lisp code
will be severely affected.  At the very least, they'll need amendment to
cope with the new format alongside the current format of code.

For example, my amendments for bug #67455 (Putting position information
into doc strings) are already difficult enough.  This change would add a
whole new layer of indirection into bug #67455's changes.

>         Stefan

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Sat, Jan 27, 2024 at 12:22 AM Daniel Mendler <mail@daniel-mendler.de> wrote:

> What about only dropping the list-based closure representation as a
> first step, switching it over to a vector-based one?
>
> (functionp '(closure (t) nil t)) => nil (currently t)
> (funcall   '(closure (t) nil t)) => error (currently t)

+1 to this.  But not a vector, which can also be spoofed, but a
record (aka vectorlike).

João

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Fri, Jan 26, 2024 at 11:55 PM Stefan Monnier
<monnier@iro.umontreal.ca> wrote:

> That's right: for compatibility reasons, I think we have to support the
> `funcall` case for the foreseeable future (and really, it costs very
> little to do so)

Does it? Doesn't it basically amount to ad-hoc 'eval' with all the
static analysis/type propagation/etc dream-shattering?

João

Re: Distinguishing `consp` and `functionp`

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I've been annoyed at the use of lists to represent function values for
> a while now.  For a reason I cannot fathom, I even managed to reproduce
> that very same mistake in Emacs-24 with the `(closure ...)` value for
> statically scoped interpreted function values.
>
> That was a major blunder.
>
> In any case, I'm playing around with a "fix", making lambda evaluate
> (when interpreted) not to (lambda ...) or (closure ...) but to
> a self-evaluating value that can be more reliably distinguished.
> In the patch below, I just reused the #[...] byte-code objects for that,
> putting the function's body where the bytecode string goes and the
> function's captured environment where the "constant vector" goes.
>
> It's got several rough edges (most importantly that
> `byte-code-function-p` returns non-nil for interpreted function values),
> but it seems to work OK so far.
>
> You can't use the patch as-is because it's written against my local
> branch, with various local hacks, some of which (partly) remove support
> for lexical-binding==nil.
> But hopefully, it's readable enough for you to form an opinion.

I'm fine with it, on the condition that it continues to be possible to
print and read lambdas created by quoted function forms, as:

  #'(lambda () foo)

Needless to say, funcall should continue to accept old list lambdas,
whether functionp regards them as such or not.

Re: Distinguishing `consp` and `functionp`

[Po Lu]

Daniel Mendler via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> What about only dropping the list-based closure representation as a
> first step, switching it over to a vector-based one?
>
> (functionp '(closure (t) nil t)) => nil (currently t)
> (funcall   '(closure (t) nil t)) => error (currently t)
>
> Unfortunately quoted lambdas are still common in packages which have not
> been updated for longer.

There is definitely code in the wild which prints precisely these lists
into files or provides them to subprocesses, so this is not acceptable.

But AFAIU it's not what Stefan proposed either, only modifying the
behavior of functionp and the types of objects generated by the byte
compiler or `function'.

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> I've been annoyed at the use of lists to represent function values for
>> a while now.
> Why?  Lists are the natural representation, and traditional in Lisp.
> They bring all sorts of advantages (like being able to manipulate code
> easily in Lisp).

Not sure how that relates.  When the code is byte-compiled we already
can't access the function as a plain list.  And when it's native
compiled, we literally can't access its internals from Lisp. at all.

> Blunder?  Having (closure ...) alongside (lambda ...) is an
> inconvenience, yes, but ...  Why a "blunder"?

Because we could have easily used #[closure ...] instead and the problem
would be mostly solved by now :-)

> Please don't do that.  If you must invent a new form, then invent a new
> symbol to signify it.  Misusing #[...] will lead to problems and extra
> work for everybody (as, for example, using the doc string field in
> oclosures for something else has led to extra work).

Actually, if you look at the patch, you'll see that the change tends to
reduces the amount of work that's needed elsewhere because docstrings
and interactive-forms are now found at the same place for interpreted
and (byte-)compiled functions.

> Any packages out there that deal with the internal format of Lisp code
> will be severely affected.

Don't know about "severely", but yes they may be affected.  There really
aren't vry many of them, tho.

> For example, my amendments for bug #67455 (Putting position information
> into doc strings) are already difficult enough.

I'll gladly take care of making the necessary adjustements.

> This change would add a whole new layer of indirection into bug
> #67455's changes.

I don't see how/why.  In the worst case, it will replace one special
case with another.


        Stefan

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> I'm fine with it, on the condition that it continues to be possible to
> print and read lambdas created by quoted function forms, as:
>
>   #'(lambda () foo)

The print format will be a sticking point, I expect.

Currently on `master` it typically does not look like what you show
above but like:

    (closure ENV (x y) (message "%S" x) (+ x y))
or
    (closure ENV (x y) DOCSTRING (interactive IFORM) (message "%S" x) (+ x y))

For `prin1` with my PoC it currently looks like:

    #[(x y) ((message "%S" x) (+ x y)) ENV]
or
    #[(x y) ((message "%S" x) (+ x y)) ENV nil DOCSTRING IFORM]

and I'd rather not try to be too clever about it (it should be fairly
rare to use `prin1` on such things).  For `cl-print` we (and users) can
choose, and I currently print it as

    #f(interpreted-function ARGS BODYHASH)

because I basically kept the same code as used for compiled functions
(obeying the same `cl-print-compiled` var) but we could make it look
more "normal", like:

    #f(lambda (x y) (message "%S" x) (+ x y))

[ or #f(λ (x y) (message "%S" x) (+ x y)) 🙂 ]

> Needless to say, funcall should continue to accept old list lambdas,
> whether functionp regards them as such or not.

No disagreement here.


        Stefan

Re: Distinguishing `consp` and `functionp`

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jan 27, 2024 at 09:25:10 -0500, Stefan Monnier wrote:
> >> I've been annoyed at the use of lists to represent function values for
> >> a while now.
> > Why?  Lists are the natural representation, and traditional in Lisp.
> > They bring all sorts of advantages (like being able to manipulate code
> > easily in Lisp).

> Not sure how that relates.  When the code is byte-compiled we already
> can't access the function as a plain list.  And when it's native
> compiled, we literally can't access its internals from Lisp. at all.

Evasive non-answer number 1.  We're not talking about byte or native
compiled forms.  We're talking about interpreted forms.  The advantages
of Lisp forms had to be foregone in the compiled forms to optimise
performance.  But when being able to manipulate forms is more important
than speed, then the interpreted forms are used.  Your plan will make
this more difficult.

Let me repeat, lists are the natural, traditional representation in
Lisp, which bring all sorts of advantages.  Your proposal will reduce
these advantages.  For what gain?

> > Blunder?  Having (closure ...) alongside (lambda ...) is an
> > inconvenience, yes, but ...  Why a "blunder"?

> Because we could have easily used #[closure ...] instead and the problem
> would be mostly solved by now :-)

Evasive non-answer number 2.  What is the advantage of #[closure ...]
over (closure ...)?  I can see only disadvantages.

> > Please don't do that.  If you must invent a new form, then invent a new
> > symbol to signify it.  Misusing #[...] will lead to problems and extra
> > work for everybody (as, for example, using the doc string field in
> > oclosures for something else has led to extra work).

> Actually, if you look at the patch, you'll see that the change tends to
> reduces the amount of work that's needed elsewhere because docstrings
> and interactive-forms are now found at the same place for interpreted
> and (byte-)compiled functions.

Evasive non-answer number 3.  Where in the new structure you put the
components is independent of the symbol you use.  My point wasn't about
the patch as a whole, it was about the symbol to be used for your new
form.  If you press ahead with this abuse of #[...] it means that every
time somebody wants to check for a byte compiled form, they'll
additionally have to check its innards.  #[ ... ] currently has an
unambiguous meaning, and it should stay that way.

> > Any packages out there that deal with the internal format of Lisp code
> > will be severely affected.

> Don't know about "severely", but yes they may be affected.  There really
> aren't vry many of them, tho.

Not "may" but "will".  This is an important point, not something to be
brushed aside and ignored.  Work on the Emacs core will be affected,
too.

> > For example, my amendments for bug #67455 (Putting position information
> > into doc strings) are already difficult enough.

> I'll gladly take care of making the necessary adjustements.

That's just an example to make a point.  _ALL_ similar work on the core
will become more difficult if you press ahead with this change.

> > This change would add a whole new layer of indirection into bug
> > #67455's changes.

> I don't see how/why.

Isn't it obvious?  In place of being able to work on the Lisp form, one
will first have to extract it from the new structure, and then
afterwards write it back to that structure.  What M-: (symbol-function
'foo) currently does will, I think, become more tedious to do.  That is
the extra layer of indirection.

> In the worst case, it will replace one special case with another.

No.  It will just make things more difficult, full stop.  Or can you
identify something in your change which will make development easier?

Again, why?  What is the advantage this proposal of yours is supposed to
bring?  So far, all I've seen in the thread is tiny things about the
ability to do (funcall '(lambda ...)), the ability to cl-prin1 functions
obfuscated and suchlike.  Why?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> Evasive non-answer number 1.

That statement is mildly offensive.
I do not see what in my previous messages would call for it.
[ But it may affect the rest of this message, I'm afraid.  ]

> We're not talking about byte or native compiled forms.  We're talking
> about interpreted forms.  The advantages of Lisp forms had to be
> foregone in the compiled forms to optimise performance.  But when
> being able to manipulate forms is more important than speed, then the
> interpreted forms are used.

Such code that can't be compiled is extremely rare and will usually be
unaffected because its list-representation of a function will usually
not have gone through `eval` at all.

> Your plan will make this more difficult.

I guess I don't really see which kind of code you're talking about:
examples would help.

> Let me repeat, lists are the natural, traditional representation in
> Lisp, which bring all sorts of advantages.

Not sure repetition helps, maybe pointing to evidence would?
Common Lisp, Clojure, and Scheme, all disagree.  AFAIK Lisp Machine Lisp
also disagreed.  And in practice the very vast majority of functions in
ELisp also disagree.

> Your proposal will reduce these advantages.  For what gain?

Since I don't think repetition helps, I'll skip this.

> Evasive non-answer number 3.  Where in the new structure you put the
> components is independent of the symbol you use.  My point wasn't about
> the patch as a whole, it was about the symbol to be used for your new
> form.  If you press ahead with this abuse of #[...] it means that every
> time somebody wants to check for a byte compiled form, they'll
> additionally have to check its innards.  #[ ... ] currently has an
> unambiguous meaning, and it should stay that way.

By "check" you mean with your own eyes?  Or by calling `prin1` and
parsing the result?

Otherwise, I don't understand: while my current patch has the bug that it
makes `byte-code-function-p` return non-nil on interpreted functions,
I already mentioned that it's a problem in the patch.

>> > Any packages out there that deal with the internal format of Lisp code
>> > will be severely affected.
>> Don't know about "severely", but yes they may be affected.  There really
>> aren't vry many of them, tho.
> Not "may" but "will".  This is an important point, not something to be
> brushed aside and ignored.  Work on the Emacs core will be affected,
> too.

The patch *does* touch those places, BTW.

> Isn't it obvious?  In place of being able to work on the Lisp form,
> one will first have to extract it from the new structure, and then
> afterwards write it back to that structure.

Care to point to existing code that extracts data from a function value
and then puts it back (presumably slightly modified)?

> What M-: (symbol-function 'foo) currently does will, I think, become
> more tedious to do.

? `symbol-function` is unaffected.

> That is the extra layer of indirection.

Could it be you're confused?


        Stefan

Re: Distinguishing `consp` and `functionp`

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The print format will be a sticking point, I expect.
>
> Currently on `master` it typically does not look like what you show
> above but like:
>
>     (closure ENV (x y) (message "%S" x) (+ x y))
> or
>     (closure ENV (x y) DOCSTRING (interactive IFORM) (message "%S" x) (+ x y))
>
> For `prin1` with my PoC it currently looks like:
>
>     #[(x y) ((message "%S" x) (+ x y)) ENV]
> or
>     #[(x y) ((message "%S" x) (+ x y)) ENV nil DOCSTRING IFORM]
>
> and I'd rather not try to be too clever about it (it should be fairly
> rare to use `prin1` on such things).  For `cl-print` we (and users) can
> choose, and I currently print it as
>
>     #f(interpreted-function ARGS BODYHASH)
>
> because I basically kept the same code as used for compiled functions
> (obeying the same `cl-print-compiled` var) but we could make it look
> more "normal", like:
>
>     #f(lambda (x y) (message "%S" x) (+ x y))

I'm not so concerned with appearances, so there's no need to go out of
one's way to retain a print representation similar to the old one.

> [ or #f(λ (x y) (message "%S" x) (+ x y)) 🙂 ]

You're killing me...  :-)

Re: Distinguishing `consp` and `functionp`

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The list (lambda (x) (* x x)) is a valid function; `functionp' given that argument should return true.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 27 Jan 2024 19:00:49 -0500
> 
> > Evasive non-answer number 1.
> 
> That statement is mildly offensive.

I agree.  Alan, please avoid such remarks.  They add nothing to the
actual arguments, so these remarks are better dropped before sending,
to keep the discussion technical and constructive.

Re: Distinguishing `consp` and `functionp`

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jan 27, 2024 at 19:00:49 -0500, Stefan Monnier wrote:
> > Evasive non-answer number 1.

> That statement is mildly offensive.

But justified.  You did not answer the points I made and the questions I
put.  You gave completely unsatisfactory "answers" which didn't offer
any information.  You answered like a politician, swerving awkward
points, and derailing the discussion onto topics the questions weren't
concerned with.

The main question I have is why do you want to make your change.  You
have given no substantial reason and evaded my continual questions to
that effect.

As I see it, there is not a lot wrong with the way we handle interpreted
code, and I see your change as worsening it.

> I do not see what in my previous messages would call for it.
> [ But it may affect the rest of this message, I'm afraid.  ]

> > We're not talking about byte or native compiled forms.  We're talking
> > about interpreted forms.  The advantages of Lisp forms had to be
> > foregone in the compiled forms to optimise performance.  But when
> > being able to manipulate forms is more important than speed, then the
> > interpreted forms are used.

> Such code that can't be compiled ....

See, you're doing it again.  I never said nor meant "CAN" be compiled, I
meant code that one wishes not to compile, for purposes of debugging or
understanding.  Was that not clear?  So you've evaded answering my point
about interpreted forms.  I suppose I can't force you to answer.

> .... is extremely rare and will usually be unaffected because its
> list-representation of a function will usually not have gone through
> `eval` at all.

> > Your plan will make this more difficult.

> I guess I don't really see which kind of code you're talking about:
> examples would help.

The kind of code I'm talking about is Lisp code in .el files, and
examples can be found in the directory lisp and its subdirectories.
The functions and macros in these can be evaluated with C-x C-e for
debugging and understanding purposes.

As one example, I've recently had to enhance the backquote source.  This
isn't easy.  I'm currently able to output the lists generated by the
macros without hassle, just using `message'.  That will become more
awkward or impossible after your proposed change, won't it?

> > Let me repeat, lists are the natural, traditional representation in
> > Lisp, which bring all sorts of advantages.

> Not sure repetition helps, maybe pointing to evidence would?

I think 45 years of Emacs should be enough to establish the tradition.

> Common Lisp, Clojure, and Scheme, all disagree.  AFAIK Lisp Machine Lisp
> also disagreed.  And in practice the very vast majority of functions in
> ELisp also disagree.

There's no disagreement.  The two kinds of representation are
complementary.  You're proposing to abolish the list representation, or
at the very least make it harder to use.  That other systems may lack
this representation is no reason for us to lose it.

> > Your proposal will reduce these advantages.  For what gain?

> Since I don't think repetition helps, I'll skip this.

Repetition would be unnecessary if you would just answer the question in
the first place.  I think I've asked that question five or six times
over three posts, and you have avoided answering.  Why?

> > Evasive non-answer number 3.  Where in the new structure you put the
> > components is independent of the symbol you use.  My point wasn't about
> > the patch as a whole, it was about the symbol to be used for your new
> > form.  If you press ahead with this abuse of #[...] it means that every
> > time somebody wants to check for a byte compiled form, they'll
> > additionally have to check its innards.  #[ ... ] currently has an
> > unambiguous meaning, and it should stay that way.

> By "check" you mean with your own eyes?  Or by calling `prin1` and
> parsing the result?

Both.  Also software that manipulates the printed representation of the
form will be made more complicated.

> Otherwise, I don't understand: while my current patch has the bug that it
> makes `byte-code-function-p` return non-nil on interpreted functions,
> I already mentioned that it's a problem in the patch.

Your proposed patch has the bug that it dilutes and weakens the meaning
of #[ ... ].  It's as if you took the meaning of the green traffic
light, which unambiguously means "go" and made it mean "go unless you're
driving a heavy goods vehicle", or something like that.

You still haven't answered my question as to why you're using #[ ... ],
and it seems likely that it was just a matter of convenience while
writing your patch.  It was less work.  Please consider that it might
not be a good idea to use it for real.

> >> > Any packages out there that deal with the internal format of Lisp code
> >> > will be severely affected.
> >> Don't know about "severely", but yes they may be affected.  There really
> >> aren't vry many of them, tho.
> > Not "may" but "will".  This is an important point, not something to be
> > brushed aside and ignored.  Work on the Emacs core will be affected,
> > too.

> The patch *does* touch those places, BTW.

> > Isn't it obvious?  In place of being able to work on the Lisp form,
> > one will first have to extract it from the new structure, and then
> > afterwards write it back to that structure.

> Care to point to existing code that extracts data from a function value
> and then puts it back (presumably slightly modified)?

I already have done.  symbol-function, fset.

> > What M-: (symbol-function 'foo) currently does will, I think, become
> > more tedious to do.

> ? `symbol-function` is unaffected.

So if I write

   (defun foo (bar) (car bar))

into *scratch* and evaluate it, then do M-: (symbol-function 'foo), what
I will get back on my terminal after your change will be exactly

   (closure (t) (bar) (car bar))

, will it?

> > That is the extra layer of indirection.

> Could it be you're confused?

No, it couldn't.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> Date: Sun, 28 Jan 2024 17:26:03 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Stefan.
> 
> On Sat, Jan 27, 2024 at 19:00:49 -0500, Stefan Monnier wrote:
> > > Evasive non-answer number 1.
> 
> > That statement is mildly offensive.
> 
> But justified.

It isn't, not here.  Please stop.  Bonus points for apologizing.

> You did not answer the points I made and the questions I
> put.  You gave completely unsatisfactory "answers" which didn't offer
> any information.  You answered like a politician, swerving awkward
> points, and derailing the discussion onto topics the questions weren't
> concerned with.

All of this can be expressed politely, and given that English is your
first language, you should have no problem coming up with a more
polite wording.  So please do.

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> See, you're doing it again.  I never said nor meant "CAN" be compiled, I
> meant code that one wishes not to compile, for purposes of debugging or
> understanding.  Was that not clear?

No.

<PLONK>

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> Care to point to existing code that extracts data from a function value
>> and then puts it back (presumably slightly modified)?
> I already have done.  symbol-function, fset.

These neither extract data from a function value not put data into
a function value (unless you consider the symbol to be a function value
(which is technically correct), but my patch does not affect the
handling of those function values (they're still just the same symbols),
so it's not relevant).


        Stefan

Re: Distinguishing `consp` and `functionp`

[Alan Mackenzie]

Hello, Eli.

On Sun, Jan 28, 2024 at 19:48:10 +0200, Eli Zaretskii wrote:
> > Date: Sun, 28 Jan 2024 17:26:03 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Hello, Stefan.

> > On Sat, Jan 27, 2024 at 19:00:49 -0500, Stefan Monnier wrote:
> > > > Evasive non-answer number 1.

> > > That statement is mildly offensive.

> > But justified.

> It isn't, not here.  Please stop.  Bonus points for apologizing.

I apologise to you and the rest of the group (excluding SM) for being so
blunt.

> > You did not answer the points I made and the questions I
> > put.  You gave completely unsatisfactory "answers" which didn't offer
> > any information.  You answered like a politician, swerving awkward
> > points, and derailing the discussion onto topics the questions weren't
> > concerned with.

> All of this can be expressed politely, and given that English is your
> first language, you should have no problem coming up with a more
> polite wording.  So please do.

I've been coming up with "polite wording" with Stefan for over 20 years
now, when he's dodged answering questions.  It's never done any good.
What would you suggest I do in such situations?

I'm worried about his proposal for fundamental changes in the Lisp
machine.  I've asked him what the benefit of the proposed change will be
5 or 6 times, mostly politely, and got no meaningful answer any of those
times.

I suspect that there are no advantages to this proposal, only
disadvantages.  But I'm worried that I'll wake up some day to find it
installed on master anyway.  You've also expressed concern, albeit much
more gently.

Perhaps you could find a way of putting the question to Stefan in a way
which he would actually answer.

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> Date: Sun, 28 Jan 2024 19:42:28 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > > That statement is mildly offensive.
> 
> > > But justified.
> 
> > It isn't, not here.  Please stop.  Bonus points for apologizing.
> 
> I apologise to you and the rest of the group (excluding SM) for being so
> blunt.

Thank you.

> > All of this can be expressed politely, and given that English is your
> > first language, you should have no problem coming up with a more
> > polite wording.  So please do.
> 
> I've been coming up with "polite wording" with Stefan for over 20 years
> now, when he's dodged answering questions.  It's never done any good.
> What would you suggest I do in such situations?

Keep doing that.  There's never a better alternative, not IME.

> I'm worried about his proposal for fundamental changes in the Lisp
> machine.  I've asked him what the benefit of the proposed change will be
> 5 or 6 times, mostly politely, and got no meaningful answer any of those
> times.

You got answers, some of them say that your questions and concerns
need to be enriched by more technical details and specific examples.
I suggest to try to come up with those details.

> I suspect that there are no advantages to this proposal, only
> disadvantages.  But I'm worried that I'll wake up some day to find it
> installed on master anyway.  You've also expressed concern, albeit much
> more gently.
> 
> Perhaps you could find a way of putting the question to Stefan in a way
> which he would actually answer.

I did ask a question, and still hope Stefan will answer it when he has
time.

But by and large, when all the questions are asked and answered, we
need to trust one another that we know what we are doing, and that if
we find ourselves in error, we know how to admit that and fix it.  So
if you cannot convince the others, if they still think they are right
after all your questions and concerns, my suggestion is to let the
chips fall as they may, and take it from there.

Re: Distinguishing `consp` and `functionp`

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Currently on `master` it typically does not look like what you show
> above but like:
>
>     (closure ENV (x y) (message "%S" x) (+ x y))
> or
>     (closure ENV (x y) DOCSTRING (interactive IFORM) (message "%S" x) (+ x y))
>
> For `prin1` with my PoC it currently looks like:
>
>     #[(x y) ((message "%S" x) (+ x y)) ENV]
> or
>     #[(x y) ((message "%S" x) (+ x y)) ENV nil DOCSTRING IFORM]
>
> and I'd rather not try to be too clever about it (it should be fairly
> rare to use `prin1` on such things).  For `cl-print` we (and users) can
> choose, and I currently print it as

FWIW, I like one aspect of the print format in SBCL,

    * (setq x (lambda () 1 2 3))
    #<FUNCTION (LAMBDA ()) {70055901BB}>

namely that it explicitly says "function" rather than something like #f,
which is more cryptic (and looks a bit too much like #f in Scheme, but
that's just me perhaps).  It'd be a bit more verbose, of course.

> [ or #f(λ (x y) (message "%S" x) (+ x y)) 🙂 ]

I'm obviously in favor of including a smiley in the print format, but
only if the function makes you feel 🤗

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> Maybe I'm missing something, but I always thought that having code and
> data indistinguishable is one of the strong sides of Lisp.  Are we now
> going to make this advantage smaller, by deprecating the use of lists
> to represent functions?

A function *value* contains code in some representation that depends on
lots of factors.  In practice most function values in most Lisp
implementations are not just plain lists but are somewhat opaque data.
This is also the case in ELisp where most function values are actually
compiled either as bytecode or as native code: we do have tools to look
inside, but they're definitely not plain old lists.

The "code and data indistinguishable" is something that usually refers
to various things:

- the source code (which gives us powerful macros).
- the availability of `read`able print representations of functions (tho
  that doesn't apply to native-compiled functions, sadly).
- the ability to embed any value (including a function) into source code
  via `quote`.

Those three are mostly independent from each other (and are not affected
by my patch).  And if you ask other people, they may give you different
answers.

But looking inside a function *value* (i.e. what is returned at run time
by the evaluation of `#'(lambda ...)`) with `car/cdr` is rarely
supported in the Lisp world.


        Stefan

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sun, 28 Jan 2024 16:27:27 -0500
> 
> > Maybe I'm missing something, but I always thought that having code and
> > data indistinguishable is one of the strong sides of Lisp.  Are we now
> > going to make this advantage smaller, by deprecating the use of lists
> > to represent functions?
> 
> A function *value* contains code in some representation that depends on
> lots of factors.  In practice most function values in most Lisp
> implementations are not just plain lists but are somewhat opaque data.
> This is also the case in ELisp where most function values are actually
> compiled either as bytecode or as native code: we do have tools to look
> inside, but they're definitely not plain old lists.
> 
> The "code and data indistinguishable" is something that usually refers
> to various things:
> 
> - the source code (which gives us powerful macros).
> - the availability of `read`able print representations of functions (tho
>   that doesn't apply to native-compiled functions, sadly).
> - the ability to embed any value (including a function) into source code
>   via `quote`.
> 
> Those three are mostly independent from each other (and are not affected
> by my patch).  And if you ask other people, they may give you different
> answers.
> 
> But looking inside a function *value* (i.e. what is returned at run time
> by the evaluation of `#'(lambda ...)`) with `car/cdr` is rarely
> supported in the Lisp world.

So your patch only affects "looking inside a function value"?

If so, what are the situations where a Lisp program in Emacs would
like or need to "look inside a function value"?

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> If so, what are the situations where a Lisp program in Emacs would
> like or need to "look inside a function value"?

The most obvious cases are in cases of introspection (things like `C-h
f` and underlying functions like `help-function-arglist`).
But there have been some other cases over the years.

E.g. before OClosures:

    (defun kmacro-extract-lambda (mac)
      "Extract kmacro from a kmacro lambda form."
      (let ((mac (cond
                  ((eq (car-safe mac) 'lambda)
                   (let ((e (assoc 'kmacro-exec-ring-item mac)))
                     (car-safe (cdr-safe (car-safe (cdr-safe e))))))
                  ((and (functionp mac)
                        (equal (interactive-form mac) '(interactive "pkmacro")))
                   (let ((r (funcall mac 'kmacro--extract-lambda)))
                     (and (eq (car-safe r) 'kmacro--extract-lambda) (cdr r)))))))
        (and (consp mac)
             (= (length mac) 3)
             (arrayp (car mac))
             mac)))

where the `(assoc 'kmacro-exec-ring-item mac)` looks inside the code of
the function.  Or in Emacs-22's `vc.el`:

    (defun vc-exec-after (code)
      "Eval CODE when the current buffer's process is done.
    If the current buffer has no process, just evaluate CODE.
    Else, add CODE to the process' sentinel."
      (let ((proc (get-buffer-process (current-buffer))))
        (cond
         ;; If there's no background process, just execute the code.
         ;; We used to explicitly call delete-process on exited processes,
         ;; but this led to timing problems causing process output to be
         ;; lost.  Terminated processes get deleted automatically
         ;; anyway. -- cyd
         ((or (null proc) (eq (process-status proc) 'exit))
          (eval code))
         ;; If a process is running, add CODE to the sentinel
         ((eq (process-status proc) 'run)
          (let ((sentinel (process-sentinel proc)))
            (set-process-sentinel proc
              `(lambda (p s)
                 (with-current-buffer ',(current-buffer)
                   (goto-char (process-mark p))
                   ,@(append (cdr (cdr (cdr ;strip off `with-current-buffer buf
                                            ;             (goto-char...)'
                               (car (cdr (cdr ;strip off `lambda (p s)'
                                sentinel))))))
                             (list `(vc-exec-after ',code))))))))
         (t (error "Unexpected process state"))))
      nil)

where the deep nesting of cars and cdrs digs into the guts of a function.

Note that neither of those two cases would be affected by my patch (not
only because they're not with us any more): they look inside functions
they built themselves and that never go through `Ffunction` nor through
the byte-compiler, so they always remain as lists starting with the
`lambda` symbol (never converted a `byte-code-function-p` or a list
starting with `closure` or a native-compiled subr).

I recently came across another instance out in the wild, in the
[Buttercup](http://elpa.nongnu.org/nongnu/buttercup.html) package:

    (defun buttercup--enclosed-expr (fun)
      "Given a zero-arg function FUN, return its unevaluated expression.
    
    The function MUST be byte-compiled or have one of the following
    forms:
    
    \(closure (ENVLIST) () (quote EXPR) (buttercup--mark-stackframe) EXPANDED)
    \(lambda () (quote EXPR) (buttercup--mark-stackframe) EXPR)
    
    and the return value will be EXPR, unevaluated. The quoted EXPR
    is useful if EXPR is a macro call, in which case the `quote'
    ensures access to the un-expanded form."
      (cl-assert (functionp fun) t "Expected FUN to be a function")
      (pcase fun
        ;; This should be the normal case, a closure with unknown enclosed
        ;; variables, empty arglist and a body containing
        ;; * the quoted original expression
        ;; * the stackframe marker
        ;; * the macroexpanded original expression
        (`(closure ,(pred listp) nil
            (quote ,expr) (buttercup--mark-stackframe) ,_expanded)
         expr)
        ;; This a when FUN has not been evaluated.
        ;; Why does that happen?
        ;; A lambda with an empty arglist and a body containing
        ;; * the quoted original expression
        ;; * the stackframe marker
        ;; * the expanded expression
        (`(lambda nil
            (quote ,expr) (buttercup--mark-stackframe) ,_expanded)
         expr)
        ;;; This is when FUN has been byte compiled, as when the entire
        ;;; test file has been byte compiled. Check that it has an empty
        ;;; arglist, that is all that is possible at this point. The
        ;;; return value is byte compiled code, not the original
        ;;; expressions. Also what is possible at this point.
        ((and (pred byte-code-function-p) (guard (member (aref fun 0) '(nil 0))))
         (aref fun 1))
        ;; Error
        (_ (signal 'buttercup-enclosed-expression-error (format "Not a zero-arg one-expression closure: %S" fun)))))

That code will/would be impacted by my patch.  I already sent a patch to
the authors to help them use another solution (in this case OClosures):
https://github.com/jorgenschaefer/emacs-buttercup/issues/241


        Stefan

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 29 Jan 2024 10:19:50 -0500
> 
> > If so, what are the situations where a Lisp program in Emacs would
> > like or need to "look inside a function value"?
> 
> The most obvious cases are in cases of introspection (things like `C-h
> f` and underlying functions like `help-function-arglist`).
> But there have been some other cases over the years.
> [...]

OK, thanks.

And the reason(s) you are "annoyed at the use of lists to represent
function values" are?...  IOW, what will we gain by using your changes
in this matter?

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> And the reason(s) you are "annoyed at the use of lists to represent
> function values" are?...  IOW, what will we gain by using your changes
> in this matter?

Beside taste and philosophical differences, the main motivation is to
make `consp` and `functionp` mutually exclusive so as to eliminate the
risk that a list be considered mistakenly as a function or vice versa.

I mentioned that this risk of confusion is the reason why our completion
functions do not officially support completion tables represented as
lists of symbols (even though in practice they work fine (except when
the first symbol happens to be `lambda` or `closure`)).

It also occurs in various other places where we want to allow either
a function or a list.

E.g. a recent bugfix in YASnippet:

    commit 9228fd983bb9e71d44d406433a46495b22640801
    Author: Marten Lienen <marten.lienen@gmail.com>
    Date:   Mon Jan 22 11:08:44 2024 +0100
    
        * yasnippel.el (yas-buffer-local-condition): Check functionp before consp to allow closures


- Stefan

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 29 Jan 2024 10:41:37 -0500
> 
> > And the reason(s) you are "annoyed at the use of lists to represent
> > function values" are?...  IOW, what will we gain by using your changes
> > in this matter?
> 
> Beside taste and philosophical differences, the main motivation is to
> make `consp` and `functionp` mutually exclusive so as to eliminate the
> risk that a list be considered mistakenly as a function or vice versa.

Is that a serious risk?  In what situations could that happen?

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> Is that a serious risk?  In what situations could that happen?

AFAIK, I gave two examples (one of each, as it turns out).


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Mon, Jan 29, 2024 at 3:42 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > And the reason(s) you are "annoyed at the use of lists to represent
> > function values" are?...  IOW, what will we gain by using your changes
> > in this matter?
>
> Beside taste and philosophical differences, the main motivation is to
> make `consp` and `functionp` mutually exclusive so as to eliminate the
> risk that a list be considered mistakenly as a function or vice versa.

I don't think there is much space for philosophy or debate in noting
for example, that a list as lambda won't get byte-compiled like a
true lambda form.

(byte-compile '(let ((bla (lambda () (+ 42 42))))
                 (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
"\300\207" [84] 1]] 2)


(byte-compile '(let ((bla '(lambda () (+ 42 42))))
                 (funcall bla)));; -> (byte-code "\300 \207" [(lambda
nil (+ 42 42))] 1)

> I mentioned that this risk of confusion is the reason why our completion
> functions do not officially support completion tables represented as
> lists of symbols (even though in practice they work fine (except when
> the first symbol happens to be `lambda` or `closure`)).
>
> It also occurs in various other places where we want to allow either
> a function or a list.
>
> E.g. a recent bugfix in YASnippet:
>
>     commit 9228fd983bb9e71d44d406433a46495b22640801
>     Author: Marten Lienen <marten.lienen@gmail.com>
>     Date:   Mon Jan 22 11:08:44 2024 +0100
>
>         * yasnippel.el (yas-buffer-local-condition): Check functionp before consp to allow closures
>

Yup, thank you very much for that.

Stefan's patch moves us in the right direction if just for the guidance
it gives to ignorant Lispers like the one who introduced that bug circa 2009.

João

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 29 Jan 2024 15:54:04 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I don't think there is much space for philosophy or debate in noting
> for example, that a list as lambda won't get byte-compiled like a
> true lambda form.
> 
> (byte-compile '(let ((bla (lambda () (+ 42 42))))
>                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> "\300\207" [84] 1]] 2)
> 
> 
> (byte-compile '(let ((bla '(lambda () (+ 42 42))))
>                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> nil (+ 42 42))] 1)

I wonder why you considered that IU need this lecture.

> Stefan's patch moves us in the right direction if just for the guidance
> it gives to ignorant Lispers like the one who introduced that bug circa 2009.

I'm glad we have your approval.  What would we do without it?

Re: Distinguishing `consp` and `functionp`

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 29 Jan 2024 15:54:04 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I don't think there is much space for philosophy or debate in noting
> for example, that a list as lambda won't get byte-compiled like a
> true lambda form.
> 
> (byte-compile '(let ((bla (lambda () (+ 42 42))))
>                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> "\300\207" [84] 1]] 2)
> 
> 
> (byte-compile '(let ((bla '(lambda () (+ 42 42))))
>                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> nil (+ 42 42))] 1)

I wonder why you considered that I need this lecture.

> Stefan's patch moves us in the right direction if just for the guidance
> it gives to ignorant Lispers like the one who introduced that bug circa 2009.

I'm glad we have your approval.  What would we do without it?

Re: Distinguishing `consp` and `functionp`

[Andreas Schwab]

On Jan 29 2024, João Távora wrote:

> I don't think there is much space for philosophy or debate in noting
> for example, that a list as lambda won't get byte-compiled like a
> true lambda form.
>
> (byte-compile '(let ((bla (lambda () (+ 42 42))))
>                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> "\300\207" [84] 1]] 2)
>
>
> (byte-compile '(let ((bla '(lambda () (+ 42 42))))
>                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> nil (+ 42 42))] 1)

Functions should be quoted with function, not quote:

ELISP> (byte-compile '(let ((bla #'(lambda () (+ 42 42)))) (funcall bla)))
(byte-code "\300\211 \207"
	   [#f(compiled-function
	       ()
	       #<bytecode 0x1980001536a5>)]
	   2)

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Mon, Jan 29, 2024 at 4:10 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Mon, 29 Jan 2024 15:54:04 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> >
> > I don't think there is much space for philosophy or debate in noting
> > for example, that a list as lambda won't get byte-compiled like a
> > true lambda form.
> >
> > (byte-compile '(let ((bla (lambda () (+ 42 42))))
> >                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> > "\300\207" [84] 1]] 2)
> >
> >
> > (byte-compile '(let ((bla '(lambda () (+ 42 42))))
> >                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> > nil (+ 42 42))] 1)
>
> I wonder why you considered that IU need this lecture.

What lecture?   I wasn't even writing to you, I was writing to
Stefan, who said there are sometimes philosophical reasons for
accepting the latter.  Absolutely uncalled for.

> > Stefan's patch moves us in the right direction if just for the guidance
> > it gives to ignorant Lispers like the one who introduced that bug circa 2009.
>
> I'm glad we have your approval.  What would we do without it?

You are a rude person.

João

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> I don't think there is much space for philosophy or debate in noting
> for example, that a list as lambda won't get byte-compiled like a
> true lambda form.
>
> (byte-compile '(let ((bla (lambda () (+ 42 42))))
>                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> "\300\207" [84] 1]] 2)
>
>
> (byte-compile '(let ((bla '(lambda () (+ 42 42))))
>                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> nil (+ 42 42))] 1)

While it's in the vicinity, this is not directly related to my patch.

>>     commit 9228fd983bb9e71d44d406433a46495b22640801
>>     Author: Marten Lienen <marten.lienen@gmail.com>
>>     Date:   Mon Jan 22 11:08:44 2024 +0100
>>
>>         * yasnippel.el (yas-buffer-local-condition): Check functionp before consp to allow closures
[...]
> Stefan's patch moves us in the right direction if just for the guidance
> it gives to ignorant Lispers like the one who introduced that bug circa 2009.

FWIW, the bug fixed by the above commit was introduced a few days
earlier by yours truly,


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Mon, Jan 29, 2024 at 4:17 PM Andreas Schwab <schwab@suse.de> wrote:
>
> On Jan 29 2024, João Távora wrote:
>
> > I don't think there is much space for philosophy or debate in noting
> > for example, that a list as lambda won't get byte-compiled like a
> > true lambda form.
> >
> > (byte-compile '(let ((bla (lambda () (+ 42 42))))
> >                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> > "\300\207" [84] 1]] 2)
> >
> >
> > (byte-compile '(let ((bla '(lambda () (+ 42 42))))
> >                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> > nil (+ 42 42))] 1)
>
> Functions should be quoted with function, not quote:
>
> ELISP> (byte-compile '(let ((bla #'(lambda () (+ 42 42)))) (funcall bla)))
> (byte-code "\300\211 \207"
>            [#f(compiled-function
>                ()
>                #<bytecode 0x1980001536a5>)]
>            2)


Of course, that's the point.  My point is that even

(let ((bla (list 'lambda nil '(+ 42 42))))
  (functionp bla))

will return t.

In fact even

   (functionp (cddr (list 'foo 'bar 'lambda 'baz))

returns t

Anyway lambda in particular doesn't need to be function-quoted though.
So there's no need to put the #' there if you want a proper function
object.

João

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Mon, Jan 29, 2024 at 4:28 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > I don't think there is much space for philosophy or debate in noting
> > for example, that a list as lambda won't get byte-compiled like a
> > true lambda form.
> >
> > (byte-compile '(let ((bla (lambda () (+ 42 42))))
> >                  (funcall bla))) ;; -> (byte-code "\300\211 \207" [#[0
> > "\300\207" [84] 1]] 2)
> >
> >
> > (byte-compile '(let ((bla '(lambda () (+ 42 42))))
> >                  (funcall bla)));; -> (byte-code "\300 \207" [(lambda
> > nil (+ 42 42))] 1)
>
> While it's in the vicinity, this is not directly related to my patch.

I know, your patch doesn't address this.  But if I understand
correctly it makes at least functionp return nil for the second one.
Or at least makes way for it.  And that's a plus in my book.

>
> >>     commit 9228fd983bb9e71d44d406433a46495b22640801
> >>     Author: Marten Lienen <marten.lienen@gmail.com>
> >>     Date:   Mon Jan 22 11:08:44 2024 +0100
> >>
> >>         * yasnippel.el (yas-buffer-local-condition): Check functionp before consp to allow closures
> [...]
> > Stefan's patch moves us in the right direction if just for the guidance
> > it gives to ignorant Lispers like the one who introduced that bug circa 2009.
>
> FWIW, the bug fixed by the above commit was introduced a few days
> earlier by yours truly,

Alright.  I wouldn't be surprised if I wasn't passing lists as lambdas
around in that extension.  I used to do that, and saw many fall prety
to this, when their lisps allow it.  The result is harder to debug code
and harder to instrument.

Re: Distinguishing `consp` and `functionp`

[Yuri Khan]

On Mon, 29 Jan 2024 at 23:50, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Beside taste and philosophical differences, the main motivation is to
> make `consp` and `functionp` mutually exclusive so as to eliminate the
> risk that a list be considered mistakenly as a function or vice versa.
>
> I mentioned that this risk of confusion is the reason why our completion
> functions do not officially support completion tables represented as
> lists of symbols (even though in practice they work fine (except when
> the first symbol happens to be `lambda` or `closure`)).

A backward-compatible solution for “I want to support lists but there
are already things that have the list shape, with a special symbol at
the head” is to designate a new special symbol, maybe ‘list’, to be
used at the head (and ignored except as the type designator).

(Disclaimer: I am not qualified to have an opinion as to whether
functions-as-lists should continue to be supported.)

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> While it's in the vicinity, this is not directly related to my patch.
> I know, your patch doesn't address this.  But if I understand
> correctly it makes at least functionp return nil for the second one.
> Or at least makes way for it.

The patch I sent doesn't change `functionp`, but indeed makes way for
such a change (and changing `functionp` is the driving force behind it).

> And that's a plus in my book.

:-)

> Alright.  I wouldn't be surprised if I wasn't passing lists as lambdas
> around in that extension.  I used to do that, and saw many fall prety
> to this, when their lisps allow it.  The result is harder to debug code
> and harder to instrument.

I suspect all long-time Lispers have committed such sins at some point.
I'm to blame for the godawful `vc-exec-after` example I sent earlier today.


        Stefan

Re: Distinguishing `consp` and `functionp`

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But looking inside a function *value* (i.e. what is returned at run time
  > by the evaluation of `#'(lambda ...)`) with `car/cdr` is rarely
  > supported in the Lisp world.

Compiled functions have been non-transparent since the 1960s, and they
have to be.  But the kind of function which makes the core of the
concept of Lisp, the interpreted function, can indeed be analyzed that
way.

To alter that would be a major change in the philosophy of Lisp.
It would call for a powerful justification.

The argument that "some other kinds of functions already can't be
introspected, so why not break that for the core kind of function too"
is not logically valid.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Mon, Jan 29, 2024 at 8:00 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> The patch I sent doesn't change `functionp`, but indeed makes way for
> such a change (and changing `functionp` is the driving force behind it).

I've now briefly checked Common Lisp land.  Allegro CL is the
only one approaching Elisp's level of salad.  But at least functionp
does what you want it to do.

CL-USER> (funcall (list 'lambda nil 42))
42 (6 bits, #x2a, #o52, #b101010)
CL-USER> (functionp (list 'lambda nil 42))
NIL

But then there's the common pattern in libraries:

  (let ((value (if (functionp user-visible-variable)
                   (funcall user-visible-variable)
                 user-visible-variable)))
    ...)

Won't this break a user's config containing:

  (setq user-visible-variable '(lambda () 42))

?

> > Alright.  I wouldn't be surprised if I wasn't passing lists as lambdas
> > around in that extension.  I used to do that, and saw many fall prety
> > to this, when their lisps allow it.  The result is harder to debug code
> > and harder to instrument.
>
> I suspect all long-time Lispers have committed such sins at some point.
> I'm to blame for the godawful `vc-exec-after` example I sent earlier today.

I'd guess that's another class of blunder. Confusion about how lists
and lambda work is much more common in newcomers.

You can't forbid (funcall (list 'lambda)) without breaking lots of user
code, but do sth like coopt the existing "lexical-binding" variable to
introduce a  JS-like "strict mode".  I'm betting there's a correlation
between files with lexical-binding=nil and where these funcalls are
attempted or encouraged.

That's if you're interested in fixing this bit at all, which you don't
seem to be.

João

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> CL-USER> (funcall (list 'lambda nil 42))
> 42 (6 bits, #x2a, #o52, #b101010)
> CL-USER> (functionp (list 'lambda nil 42))
> NIL

That matches where I'd like ELisp to be 🙂

> But then there's the common pattern in libraries:
>
>   (let ((value (if (functionp user-visible-variable)
>                    (funcall user-visible-variable)
>                  user-visible-variable)))
>     ...)
>
> Won't this break a user's config containing:
>
>   (setq user-visible-variable '(lambda () 42))
>
> ?

Yup.  My impression is that this is sufficiently rare that we can afford
to break it.  Maybe experience will show me wrong, of course.
We could also consider an intermediate step where `functionp` returns
t but emits a warning.

> You can't forbid (funcall (list 'lambda)) without breaking lots of
> user code,

That's my impression, indeed.

> That's if you're interested in fixing this bit at all, which you don't
> seem to be.

I don't see very much benefit from forbidding (funcall (list 'lambda)),
indeed.  From a purely philosophical point of view, I agree that it
should not be accepted, but in practice the only downsides I can see are:

- it can give a wrong impression to a beginner, encouraging confusion.
- it can occasionally hide an error, making debugging a bit more difficult.

These seem very marginal to me.


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Tue, Jan 30, 2024 at 12:54 PM Stefan Monnier
<monnier@iro.umontreal.ca> wrote:

> Yup.  My impression is that this is sufficiently rare that we can afford
> to break it.  Maybe experience will show me wrong, of course.

I've definitely seen users post configs to Eglot's bug tracker where
lambdas appear incorrectly quoted.  i.e.

  (add-to-list 'eglot-server-programs '(foo-mode . (lambda (..) ...)))

I try to always correct them,  but sometimes it'll just silently work until it
doesn't.

> We could also consider an intermediate step where `functionp` returns
> t but emits a warning.

Indeed, though in that case I'd make the funcall warn. I think it's there
that this proposed runtime warning  ultimately matters and is useful
to help users correct their elisp. Runtime warnings are a bit icky though
:-| but better than nothing.

> - it can give a wrong impression to a beginner, encouraging confusion.
> - it can occasionally hide an error, making debugging a bit more difficult.

Wouldn't you add "complicates type propagation, static analysis and
optimization" to that list?

> These seem very marginal to me.

I've definitely seen this error happen more than once:

(defun call-with-oopsie (fn) (funcall fn))

(defmacro with-oopsie (_nil &rest body)
  ;; usually some much more complicated hairy expansion
  `(call-with-oopsie '(lambda () ,@body)))

(defvar x 42)
(with-oopsie () (+ 42 x)) ;; 84, everything's fine, my macro is great

(let ((y 42)) (with-oopsie () (+ 42 y))) ;; oopsie...

The warning would have made the first call work but signal
something is off.

João

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> Yup.  My impression is that this is sufficiently rare that we can afford
>> to break it.  Maybe experience will show me wrong, of course.
>
> I've definitely seen users post configs to Eglot's bug tracker where
> lambdas appear incorrectly quoted.  i.e.
>
>   (add-to-list 'eglot-server-programs '(foo-mode . (lambda (..) ...)))

Yes, that's very common and that's why I don't think we can drop support
for that in `funcall`.  What I hope is less common is for those cases
to go through `functionp`.

Of course, it also depends on happenstance of how the `functionp` test
is written.  E.g. for `eglot-server-programs` it seems that we
will/would suffer because we start by testing `functionp` instead of
first handling the non-function-like lists.

>> We could also consider an intermediate step where `functionp` returns
>> t but emits a warning.
> Indeed, though in that case I'd make the funcall warn. I think it's there
> that this proposed runtime warning  ultimately matters and is useful
> to help users correct their elisp. Runtime warnings are a bit icky though
> :-| but better than nothing.

We already have compile time warnings at those places where the compiler
easily knows that the list should be a function, but for all those other
cases (like `eglot-server-programs`), we don't have any tool currently
other than run-time warnings.

>> - it can give a wrong impression to a beginner, encouraging confusion.
>> - it can occasionally hide an error, making debugging a bit more difficult.
> Wouldn't you add "complicates type propagation, static analysis and
> optimization" to that list?

No.  All these can be blissfully unaware of what `funcall` does when
presented with a list starting with `lambda`.  In theory, it's true that
analysis&optimization could assume that *after* the `funcall` the
argument was a valid function obeying `functionp`, but I suspect our
analysis/optimization technology is pretty far from that level, and even
if we were able to make that work, the potential for gains from such
information seems vanishingly small.


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3350 bytes --]

On Tue, Jan 30, 2024 at 11:13 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

  E.g. for `eglot-server-programs` it seems that we
> will/would suffer because we start by testing `functionp` instead of
> first handling the non-function-like lists.

Yeah but, I could fix that if I had a time machine :-)
Anyway I tend to think it makes the code simpler to get that functionp case
out of the way first and it doesn't seem I'm alone.  Have you grepped for
the
words "can also be a function" ? It sure seems to suggest many many others
do the same: start by calling functionp.  Won't they _all_ be broken if
users
have been putting lists in those user vars by mistake?

> >> We could also consider an intermediate step where `functionp` returns
> >> t but emits a warning.
> > Indeed, though in that case I'd make the funcall warn. I think it's
there
> > that this proposed runtime warning  ultimately matters and is useful
> > to help users correct their elisp. Runtime warnings are a bit icky
though
> > :-| but better than nothing.
>
> We already have compile time warnings at those places where the compiler
> easily knows that the list should be a function, but for all those other
> cases (like `eglot-server-programs`), we don't have any tool currently
> other than run-time warnings.

Of course.  What I'm saying is basically.

* Do this patch, it's probably a good step.
* keep functionp returning t for those lists for a while (say one major
version)
  to avoid too much fallout.
* warn at runtime in the funcall.  Then users using these problematic
customizations will start fixing  their code. We'll probably catch some
library doing this too.

One thing it'd be nice to have for runtime warnings is some kind of
file location hint.  Maybe check the calling frame?

> >> - it can give a wrong impression to a beginner, encouraging confusion.
> >> - it can occasionally hide an error, making debugging a bit more
difficult.
> > Wouldn't you add "complicates type propagation, static analysis and
> > optimization" to that list?
>
> No.  All these can be blissfully unaware of what `funcall` does when
> presented with a list starting with `lambda`.  In theory, it's true that
> analysis&optimization could assume that *after* the `funcall` the
> argument was a valid function obeying `functionp`, but I suspect our
> analysis/optimization technology is pretty far from that level, and even
> if we were able to make that work, the potential for gains from such
> information seems vanishingly small.

Dunno.  SBCL's compiler is pretty good, it propagates types  and warns
say, when using generic+ instead of must faster fixum+.  You can really
extract  some good perf from it.  Our byte-compiler is not advanced enough?
Maybe.  But what about the native compiler?  And shouldn't we be making
way for these advances.

All in all, I can't see how these fishy funcalls don't amount to anything
other than 'eval' in disguise. And though I'm admittedly not as expert in
this field as you are, I've always learned  'eval' is known to be nono
because of these and other pitfalls.  See e.g. this very good Rainer Joswig
answer https://stackoverflow.com/a/2571549 (I'm aware you're probably
aware of those arguments, just mentioned for others who may be interested).

João

[-- Attachment #2: Type: text/html, Size: 4010 bytes --]

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> It sure seems to suggest many many others do the same: start by
> calling functionp.

Indeed, the current representation of interpreted functions tends to
encourage it.

> Won't they _all_ be broken if users have been
> putting lists in those user vars by mistake?

Could be.  That's one of the reasons why my patch doesn't change
`functionp` (yet).

> One thing it'd be nice to have for runtime warnings is some kind of
> file location hint.

Alan has plans to work on that, tho I strongly suspect that it won't
help for this problem: the functions for which we need this info are the
functions which Emacs thought were lists, so Alan's plan to try and keep
some source info along with function wouldn't help here (unless we keep
source info for cons cells, which I guess could be a possibility: it
would be expensive to do for all cons cells read from files, but if we
limit it to cons cells whose `car` is `lambda` is might even be doable).

> Maybe check the calling frame?

I suspect that most cases will look like your Eglot example, where by
the time you get to the `funcall/functionp` the backtrace doesn't
immediately tell you where that function comes from (it can still be
helpful info to trace it back, but it may require digging into someone
else's code).

> Dunno.  SBCL's compiler is pretty good, it propagates types  and warns
> say, when using generic+ instead of must faster fixum+.

Knowing "this is a function" isn't terribly better than "this is
either a function or a cons cell starting with `lambda`": it will rarely
let you turn a generic+ into a fixnum+.


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

On Wed, Jan 31, 2024 at 12:22 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> I suspect that most cases will look like your Eglot example, where by
> the time you get to the `funcall/functionp` the backtrace doesn't
> immediately tell you where that function comes from (

Of course, not where the interpreted function comes from.  But at least
I'll know where to look, where to edebug.

> Could be.  That's one of the reasons why my patch doesn't change
> `functionp` (yet).

Yes, my advice is to keep it like that but add the warning (in funcall).

> > Dunno.  SBCL's compiler is pretty good, it propagates types  and warns
> > say, when using generic+ instead of must faster fixum+.
>
> Knowing "this is a function" isn't terribly better than "this is
> either a function or a cons cell starting with `lambda`": it will rarely
> let you turn a generic+ into a fixnum+.

Isn't knowing "this is a function" a prerequisite for proving e.g.
"this is a function returning a fixnum"?

[-- Attachment #2: Type: text/html, Size: 1306 bytes --]

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

>> > Dunno.  SBCL's compiler is pretty good, it propagates types  and warns
>> > say, when using generic+ instead of must faster fixum+.
>> Knowing "this is a function" isn't terribly better than "this is
>> either a function or a cons cell starting with `lambda`": it will rarely
>> let you turn a generic+ into a fixnum+.
> Isn't knowing "this is a function" a prerequisite for proving e.g.
> "this is a function returning a fixnum"?

Maybe it's a prerequisite, but it's pretty damn far from sufficient: if
you're in a position to know that it returns a fixnum, then you
most likely won't need `funcall` to tell you that it's a function.

There are much lower hanging fruits.


        Stefan

Re: Distinguishing `consp` and `functionp`

[João Távora]

On Wed, Jan 31, 2024 at 3:37 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> > Dunno.  SBCL's compiler is pretty good, it propagates types  and warns
> >> > say, when using generic+ instead of must faster fixum+.
> >> Knowing "this is a function" isn't terribly better than "this is
> >> either a function or a cons cell starting with `lambda`": it will rarely
> >> let you turn a generic+ into a fixnum+.
> > Isn't knowing "this is a function" a prerequisite for proving e.g.
> > "this is a function returning a fixnum"?
>
> Maybe it's a prerequisite, but it's pretty damn far from sufficient: if
> you're in a position to know that it returns a fixnum, then you
> most likely won't need `funcall` to tell you that it's a function.

Not sure I follow, but fair enough.  Big or small, whatever benefit
it brings is certainly more than the benefits the the current
situation brings which is 0.

...apart from working with legacy code, of course.  So as I said I
really think you should leave functionp returning t for a good
while and warn about the funcall.  I think that'll make the numbers
of these things dwindle.  It's a fairly common error, not only
in user code [1,2], and the warning would have just told the users
what we end up telling them.

In fact, this is probably so common that the warning should have
some kind of anti-spam safeguard, like not warning twice for the
same object, or just limit the number of warnings per session to 5
or something.

> There are much lower hanging fruits.

By all means go grab them :-)

João

[1] https://github.com/joaotavora/eglot/pull/626#issuecomment-786606685
[2] https://github.com/joaotavora/eglot/discussions/691#discussioncomment-719357

Re: Distinguishing `consp` and `functionp`

[Stefan Monnier]

> ...apart from working with legacy code, of course.  So as I said I
> really think you should leave functionp returning t for a good
> while and warn about the funcall.

In any case, currently we can't know if a `(lambda ...)` list is the
result of "broken code" or is a perfectly valid function value returned
by `Ffunction` from a dynbound file.

So before we can contemplate such a warning, we need to either drop
dynbound interpreted code, or use a patch along the lines of what I sent :-)


        Stefan "whose local Emacs hacks don't support dynbound code any
                more and signal an error when `funcall` is passed a list :-)"

Re: Distinguishing `consp` and `functionp`

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Alright.  I wouldn't be surprised if I wasn't passing lists as lambdas
  > around in that extension.

The sentence refers to layers of previous context, including something
about a library called yasnippel.el which I don't recognize.  It
doesn't seem to be present in my checkout, either.  I can't figure out
what you are concerned you might be doing.

Could you please explain in a stand-alone way what practice you are
concerned about?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Distinguishing `consp` and `functionp`

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course, that's the point.  My point is that even

  > (let ((bla (list 'lambda nil '(+ 42 42))))
  >   (functionp bla))

  > will return t.

Yes, it will.  It is a valid function and would return 84 if you call it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Distinguishing `consp` and `functionp`

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Beside taste and philosophical differences, the main motivation is to
  > make `consp` and `functionp` mutually exclusive so as to eliminate the
  > risk that a list be considered mistakenly as a function or vice versa.

The fact that there are functions which are lists is not a minor
wrinkle of Lisp.  It is part of the foundational ideas of Lisp.
It is ok that there are functions which are atoms,
but the goal that ALL functions be atoms is perverse in Lisp terms.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)