Attaching context info to an error

Attaching context info to an error

[Stefan Monnier]

I'm playing with `handler-bind` and trying to see how we could make use
of such a functionality in Emacs.  So far I have encountered basically
two use case:

- Triggering the debugger.  I have patches which use `handler-bind` for
  that instead of `debug-on-error` in ERT, `eval-expression` and
  `--debug-init` which circumvent the usual problems linked to (ab)using
  `debug-on-error`.

- I see a potential use in Tramp where we currently use
  `signal-hook-function` instead.  I haven't looked closely enough yet
  to be sure that it's a good replacement there, tho.

- Collecting dynamic context info.

This third use case is for thing like
`macroexp--with-extended-form-stack` where we'd like to record the
`byte-compile-form-stack` that's current when the error is signaled so
we can use that info back where we actually catch that error.

Similarly when we load a file and that signals an error, we'd like to be
able to attach to the error the information that it occurred while
loading file FOO (and that could stack up if it occurred while file FOO
was loading file BAR).

The question is: where should we put this "context" info.

One possibility is to do something like the following:


    (defun load (file ...)
      (handler-bind ((error (lambda (err)
                              (signal 'error-during-load
                                      (cons file err)))))
        ...))

so an error that occurs during `load` is turned into an
`error-during-load` (and those errors contain the original error as
well as the relevant file name, the original error could be an
`error-during-load` itself, etc...).

Similarly the byte-compiler could do

    (defun byte-compile-blabla (...)
      ...
      (condition-case err
          (handler-bind ((error
                          (lambda (err)
                            (signal 'error-during-bytecomp
                                    (cons `byte-compile-form-stack` err)))))
            ...)
        (error
         ;; `err` should now be of the form
         ;; (error-during-bytecomp STACK ERROR)
         ...)))

but I don't like the idea of changing one kind of error into another.

I thought about adding the context info to an auxiliary hash table
indexed by the "error object", but that error object tends to be
decomposed into "error-name + error-data" and then recomposed fairly
liberally, so there's no guarantee that it stays `eq` to itself.

Ideally, I'd like to "append it" to the `cdr` of an error object
(i.e. the "error data"), but I can't think of a way to do it that's
reliable and doesn't introduce nasty backward compatibility issues
(and/or require changes in many places).

Any other idea how/where we could attach that info?


        Stefan

Re: Attaching context info to an error

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The question is: where should we put this "context" info.
>
> One possibility is to do something like the following:
>
>
>     (defun load (file ...)
>       (handler-bind ((error (lambda (err)
>                               (signal 'error-during-load
>                                       (cons file err)))))
>         ...))

I'm just guessing - so the argument ERR of the handler function is not a
`condition' object of some kind, in the CL sense? What is it?

Or can I look myself somewhere?

> ...
> Any other idea how/where we could attach that info?

I don't have a good idea either, I'm afraid.

Re: Attaching context info to an error

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Or can I look myself somewhere?

Found the scratch/handler-bind branch now, after fetching.

Re: Attaching context info to an error

[Stefan Monnier]

>> The question is: where should we put this "context" info.
>>
>> One possibility is to do something like the following:
>>
>>     (defun load (file ...)
>>       (handler-bind ((error (lambda (err)
>>                               (signal 'error-during-load
>>                                       (cons file err)))))
>>         ...))
>
> I'm just guessing - so the argument ERR of the handler function is not a
> `condition' object of some kind, in the CL sense? What is it?

No, it's the same kind of objects as are bound to `err` in
(condition-case err ...), i.e. a cons of (ERROR-NAME . ERROR-DATA).

I don't think it makes sense to have different kinds of objects for
`handler-bind` and for `condition-case`.


        Stefan

Re: Attaching context info to an error

[Jens Schmidt]

I might lack experience here, but I give it a try, anyway ...

On 2023-12-21  23:30, Stefan Monnier wrote:

> I'm playing with `handler-bind` and trying to see how we could make use
> of such a functionality in Emacs.

`handler-bind' seems to have the main advantage over `condition-case'
that you get the full information about the original error stack, while
in a `condition-case' that information is lost, correct?

> [...]
> 
> but I don't like the idea of changing one kind of error into another.

Why not?  Wrapping errors that way is what I know.

> Any other idea how/where we could attach that info?

Property list of the error-symbol?  Not beautiful, but probably more
stable than "external" storage as in a hash table.  Probably too stable
in the sense that the property list (without further provisions) would
survive the lifetime of the error.

Re: Attaching context info to an error

[Stefan Monnier]

>> I'm playing with `handler-bind` and trying to see how we could make use
>> of such a functionality in Emacs.
> `handler-bind' seems to have the main advantage over `condition-case'
> that you get the full information about the original error stack, while
> in a `condition-case' that information is lost, correct?

Yes, it's a kind of mix between `condition-case` and `signal-hook-function`.

>> but I don't like the idea of changing one kind of error into another.
> Why not?  Wrapping errors that way is what I know.

Because a (condition-case err (FOO) (wrong-number-of-arguments ...))
won't catch an `error-during-load` even if that error is wrapped around
a `wrong-number-of-arguments` error.

>> Any other idea how/where we could attach that info?
> Property list of the error-symbol?

That's storing the info in a global var, which is hackish: as you say it
will sometimes "survive the lifetime of the error" but it will also
sometimes live a too short life if some other error happens between the
moment the first error is signaled and that first error is caught.
[ This is likely to occur if we run the debugger between those two, for
  instance, or if an unwind form signal an error, ...  ]

FWIW, a global variable is what we use for the only such "error context
information" we currently have, which is stored in
`Vsignalling_function` (not exposed to ELisp, but visible sometimes in
*Messages*).


        Stefan

Re: Attaching context info to an error

[João Távora]

On Thu, Dec 21, 2023 at 10:31 PM Stefan Monnier
<monnier@iro.umontreal.ca> wrote:

> I'm playing with `handler-bind` and trying to see how we could make use
> of such a functionality in Emacs.

*loud cheers*

> So far I have encountered basically two use case:

(...proceeds to list three use cases...)

Amongst the killer weaponry this gives us is a restart system,
finally.

This completely changes interactive development.  Whereas before
you couldn't restart from an nearby stack frame after doing
some changes to the state (say recompiling a buggy function, or
setting a global variable),  with a handler-bind-powered restart
system you finally can.

So I had to try your branch...

;; -*- lexical-binding: t -*-
(require 'cl-lib)

(defvar pm/restarts nil)

(defun pm/call-with-retry-restart (outer inner msg fn)
  (catch outer
    (let ((saved-restarts pm/restarts))
      (unwind-protect
          (while t
            (catch inner
              (handler-bind ((error (lambda (_e)
                                      (push
                                       (list 'retry msg
                                             (lambda ()
                                               (throw inner nil)))
                                       pm/restarts))))
                (throw outer (funcall fn)))))
        (setq pm/restarts saved-restarts)))))

(cl-defmacro pm/with-retry-restart ((&key (msg "Retry.")) &body body)
  `(pm/call-with-retry-restart (gensym) (gensym) ,msg (lambda () ,@body)))

(defun pm/invoke-restart (restart)
  (interactive
   (some-read-from-minibuffer-or-somesuch))
  (funcall (caadr restart)))

(defun baz (sym)
  (error "whoops, it happened again for %s" sym))

(defun bar ()
  (pm/with-retry-restart (:msg "Retry") (baz (gensym))))

(defun foo ()
  (bar))

(foo)

;; Local Variables:
;; read-symbol-shorthands: (("pm/" . "poor-mans-restarts-"))
;; End:

This very code works fine in Common Lisp (just add a WHILE
macro to it) however when I try this in your Emacs branch and
evaluate (foo) I get:

   Re-entering top level after C stack overflow

and Emacs dies immediately.

João

PS: Of course Common Lisp has all of this built-in and with
much, much better primitives that cover a wealth of use cases.
So this was just a quick-n-dirty proof of concept.

Re: Attaching context info to an error

[João Távora]

On Sat, Dec 23, 2023 at 3:02 AM João Távora <joaotavora@gmail.com> wrote:
> (defun pm/invoke-restart (restart)
>   (interactive
>    (some-read-from-minibuffer-or-somesuch))
>   (funcall (caadr restart)))
>
> (defun baz (sym)
>   (error "whoops, it happened again for %s" sym))
>
> (defun bar ()
>   (pm/with-retry-restart (:msg "Retry") (baz (gensym))))
>
> (defun foo ()
>   (bar))
>
> (foo)
>
> ;; Local Variables:
> ;; read-symbol-shorthands: (("pm/" . "poor-mans-restarts-"))
> ;; End:
>
> This very code works fine in Common Lisp (just add a WHILE
> macro to it)

Just to be clear what I meant by this to those unfamiliar with
Common Lisp and interactive restarts.  It means I can invoke
the restart interactively from the Common Lisp debugger that
is invoked as the last step of the non-returning error.

'baz' will be called again and again with different arguments
and always errors until I recompile it to something else that
doesn't error, which I can do while the debugger is active.

I expected to be able to do the same from the Emacs Lisp
debugger-mode/backtrace-mode *Backtrace* buffer, but I never
get one such buffer.

João

Re: Attaching context info to an error

[Stefan Monnier]

> Just to be clear what I meant by this to those unfamiliar with
> Common Lisp and interactive restarts.  It means I can invoke
> the restart interactively from the Common Lisp debugger that
> is invoked as the last step of the non-returning error.
>
> 'baz' will be called again and again with different arguments
> and always errors until I recompile it to something else that
> doesn't error, which I can do while the debugger is active.
>
> I expected to be able to do the same from the Emacs Lisp
> debugger-mode/backtrace-mode *Backtrace* buffer, but I never
> get one such buffer.

FWIW, it works for me (with an Emacs compiled from the
`scratch/handler-bind` branch).
when I do `C-x C-e` at the end of `(foo)`, I get into the debugger, and
I do

    M-: (funcall (nth 2 (car pm/restarts))) RET

I get back into the debugger but with a different gensymd symbol.
And if I redefine `baz` to something that doesn't signal an error,

    M-: (funcall (nth 2 (car pm/restarts))) RET

then lets me return to the toplevel.


        Stefan

Re: Attaching context info to an error

[Stefan Monnier]

>> I'm playing with `handler-bind` and trying to see how we could make use
>> of such a functionality in Emacs.
> *loud cheers*
[...]
> Amongst the killer weaponry this gives us is a restart system,
> finally.

I fail to see the connection, to be honest.

> (defun pm/call-with-retry-restart (outer inner msg fn)
>   (catch outer
>     (let ((saved-restarts pm/restarts))
>       (unwind-protect
>           (while t
>             (catch inner
>               (handler-bind ((error (lambda (_e)
>                                       (push
>                                        (list 'retry msg
>                                              (lambda ()
>                                                (throw inner nil)))
>                                        pm/restarts))))
>                 (throw outer (funcall fn)))))
>         (setq pm/restarts saved-restarts)))))

Rather than `(let ((saved-restarts pm/restarts))` why not `(let
((pm/restarts pm/restarts))` so you don't need the `setq` in the
`unwind-protect`?

And why `handler-bind`.  IOW, why wait until the error happens before
pushing the retry onto the list?  Why not just:

    (defun pm/call-with-retry-restart (outer inner msg fn)
      (catch outer
        (let ((pm/restarts
               (cons (list 'retry msg
                           (lambda () (throw inner nil))))))
          (while t
            (catch inner
              (throw outer (funcall fn)))))))


-- Stefan

Re: Attaching context info to an error

[João Távora]

On Tue, Dec 26, 2023 at 8:47 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> I'm playing with `handler-bind` and trying to see how we could make use
> >> of such a functionality in Emacs.
> > *loud cheers*
> [...]
> > Amongst the killer weaponry this gives us is a restart system,
> > finally.
>
> I fail to see the connection, to be honest.

The restarts themselves can be made with simply try/catch,
you're right, but automatically invoking restarts with certain
parameters based on different each error object can only be done
with  handler-bind AFAIK.

Also, for restarts to become visible/invisible according to
some information in the condition themselves (the type
of error or some other cookie) we also need handler-bind to
attach that cookie to the condition object.

Typically, a restart gives you the option to try again with
something different (not just try again as I did in this
poc), like if a network connection fails, you're given
the option to try with new credentials, or a different
protocol, etc...

> And why `handler-bind`.  IOW, why wait until the error happens before
> pushing the retry onto the list?  Why not just:
>
>     (defun pm/call-with-retry-restart (outer inner msg fn)
>       (catch outer
>         (let ((pm/restarts
>                (cons (list 'retry msg
>                            (lambda () (throw inner nil))))))
>           (while t
>             (catch inner
>               (throw outer (funcall fn)))))))

Probably, this was a thinko, or I was trying to simulate visibility
of a restart like I described above, but there are much better
ways to do that.

Anyway, as Gerd suggested, I would suggest at least supporting
EIEIO in this handler-bind. This needn't necessarily clash with
our "error symbols" if the dispatching is done via generic functions
right? The usual performance setbacks of EIEIO object shouldn't
be a problem as conditions are meant to represent exceptional
situations by definition. The counterpart to handler-bind is
handler-case btw (which is more or less our condition-case).

Re: Attaching context info to an error

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Tue, Dec 26, 2023 at 8:47 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>> >> I'm playing with `handler-bind` and trying to see how we could make use
>> >> of such a functionality in Emacs.
>> > *loud cheers*
>> [...]
>> > Amongst the killer weaponry this gives us is a restart system,
>> > finally.
>>
>> I fail to see the connection, to be honest.
>
> The restarts themselves can be made with simply try/catch,
> you're right, but automatically invoking restarts with certain
> parameters based on different each error object can only be done
> with  handler-bind AFAIK.

BTW, and sorry if I'm cnfusing things here, maybe.

Do you & Stefan mean, with "restart", restarts like in CL, i.e.
restart-bind, invoke-restart, find-restart, ...? I guess not, because I
think, handler-bind handlers in CL would/could use invoke-restart. At
least that's what I have in the back of my mind. So, I guess my question
is what kind of restart is this?

Re: Attaching context info to an error

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Wed, Dec 27, 2023, 06:50 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

BTW, and sorry if I'm cnfusing things here, maybe.
>
> Do you & Stefan mean, with "restart", restarts like in CL, i.e.
> restart-bind, invoke-restart, find-restart, ...?


Yes, those are the ones I mean.

I guess not, because I
> think, handler-bind handlers in CL would/could use invoke-restart. At
> least that's what I have in the back of my mind. So, I guess my question
> is what kind of restart is this?
>

The same you had in the back of your mind. That's exactly how they
cooperate normally. It was I who was confusing and misremembering things
with my poc. Restarts normally aren't implemented in terms of handler-bind
as I clumsily suggested, but handler-bind is essential to invoke then or
even modify the condition as it travels up-stack.

João

>

[-- Attachment #2: Type: text/html, Size: 1703 bytes --]

Re: Attaching context info to an error

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Wed, Dec 27, 2023, 06:50 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
>  BTW, and sorry if I'm cnfusing things here, maybe.
>
>  Do you & Stefan mean, with "restart", restarts like in CL, i.e.
>  restart-bind, invoke-restart, find-restart, ...?
>
> Yes, those are the ones I mean.
>
>  I guess not, because I
>  think, handler-bind handlers in CL would/could use invoke-restart. At
>  least that's what I have in the back of my mind. So, I guess my question
>  is what kind of restart is this?
>
> The same you had in the back of your mind. That's exactly how they
> cooperate normally. It was I who was confusing and misremembering
> things with my poc. Restarts normally aren't implemented in terms of
> handler-bind as I clumsily suggested, but handler-bind is essential to
> invoke then or even modify the condition as it travels up-stack.

Ahhhh, very very good! Thanks for the clarification!

Re: Attaching context info to an error

[Stefan Monnier]

> Anyway, as Gerd suggested, I would suggest at least supporting
> EIEIO in this handler-bind.

Could you give some concrete examples of things you'd (be able to) do
with it?

Given that we start with (signal ERROR-SYMBOL ERROR-DATA) and the "other
end point" of an error is the `condition-case` variable bound to
a value of the form (ERROR-SYMBOL . ERROR-DATA), I find it a bit hard to
see how we could usefully use "objects" somewhere in-between, but maybe
some examples can help me out of the rut.


        Stefan

Re: Attaching context info to an error

[João Távora]

On Wed, Dec 27, 2023 at 5:50 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Anyway, as Gerd suggested, I would suggest at least supporting
> > EIEIO in this handler-bind.
>
> Could you give some concrete examples of things you'd (be able to) do
> with it?

Just look at anyCL program that uses conditions, there are
very many of them.  One of them is Snooze [1] which uses them
to represent HTTP error codes.

First, we'd have to be able to `signal` such objects.  And then,
CLOS is good :-) you get access to lots of existing useful protocols.
initialize-instance, print-object, etc for free.

> Given that we start with (signal ERROR-SYMBOL ERROR-DATA) and the "other
> end point" of an error is the `condition-case` variable bound to
> a value of the form (ERROR-SYMBOL . ERROR-DATA), I find it a bit hard to
> see how we could usefully use "objects" somewhere in-between, but maybe
> some examples can help me out of the rut.

From a CL perspective, and even other languages, using our
"error symbol with some 'error property" representation of
errors is almost self-evidently inferior to just making it
a first class object.

  (define-condition my/condition (error) docstring (slots...))

Which is a much more normal way to define a hierarchy of
exceptions like you find in C++, Python, and probably many
others.

Yes, I know there is `define-error` for a while, but we'd still
be reinventing all those protocols in subtly idiosyncratic and
buggy forms ad-hoc in Elisp.  Which admittedly, is what
we've been doing all along, but why stray gratuitously when
given a (presumably cheap) chance to rejoin?

João

[1]: https://github.com/joaotavora/snooze

Re: Attaching context info to an error

[João Távora]

On Wed, Dec 27, 2023 at 6:08 PM João Távora <joaotavora@gmail.com> wrote:

> > Could you give some concrete examples of things you'd (be able to) do
> > with it?

> Just look at anyCL program that uses conditions, there are
> very many of them.  One of them is Snooze [1] which uses them
> to represent HTTP error codes.

To elaborate a little more, Snooze is (I think) a good example
of how 'handler-bind' and 'restart-case' can cooperate to
allow different modes of operation for a HTTP server (or
HTTP REST router, which is what Snooze really is).  You
may want to catch or not catch error conditions, catch or
not catch HTTP conditions, debug these conditions in your
Lisp (or via Emacs with SLY/SLIME), explain these conditions
to the HTTP client verbosely (in developement environments)
, succinctly (in production environments), muffle them,
log them, etc.

The file

  https://github.com/joaotavora/snooze/blob/master/common.lisp

contains the meat and bones of this logic, and

  https://github.com/joaotavora/snooze/blob/master/api.lisp

contains a reasonably well docstring'ed description of the
interface to these facilities.  See the docstrings of
*catch-http-conditions* and *catch-errors*.

Restarts like 'explain-verbosely' and 'explain-succinctly'
are shown to the user when the debugger is popped.  During
development the user can thus interactively control what
gets sent to the client (usually a web-browser in another
window).  Unattended operation of such a server is achieved by
automatically invoking these restarts from handler-bind
handlers.

João

Re: Attaching context info to an error

[Stefan Monnier]

>> > Anyway, as Gerd suggested, I would suggest at least supporting
>> > EIEIO in this handler-bind.
>> Could you give some concrete examples of things you'd (be able to) do
>> with it?
> Just look at anyCL program that uses conditions, there are
> very many of them.  One of them is Snooze [1] which uses them
> to represent HTTP error codes.

But these use error objects throughout.
My question was about "at least supporting EIEIO in this handler-bind".

> First, we'd have to be able to `signal` such objects.  And then,
> CLOS is good :-) you get access to lots of existing useful protocols.
> initialize-instance, print-object, etc for free.

[ Not clear what `initialize-instance` would be used for.  I also don't
  think `print-object` would be super useful, but yes a generic
  function to compute the error message would be a natural use, tho
  currently we haven't even bothered to allow the `error-message` to be
  a function, so the non-object representation of errors doesn't seem
  to be what's holding us.  ]

More importantly, that doesn't tell me what new things we could do,
most importantly how to attach context info :-)

> From a CL perspective, and even other languages, using our
> "error symbol with some 'error property" representation of
> errors is almost self-evidently inferior to just making it
> a first class object.

You don't need to convince me of that, indeed.  But that's not what we
have right now, and it's not trivial to retro-fit it cleanly in the
current system.  Luckily, AFAICT it's orthogonal to `handler-bind`.


        Stefan

Re: Attaching context info to an error

[João Távora]

On Wed, Dec 27, 2023 at 7:08 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > Just look at anyCL program that uses conditions, there are
> > very many of them.  One of them is Snooze [1] which uses them
> > to represent HTTP error codes.
>
> But these use error objects throughout.
> My question was about "at least supporting EIEIO in this handler-bind".
>
> > First, we'd have to be able to `signal` such objects.  And then,
> > CLOS is good :-) you get access to lots of existing useful protocols.
> > initialize-instance, print-object, etc for free.
>
> [ Not clear what `initialize-instance` would be used for.

It is passed the keyword arguments that are passed to CL:ERROR
or CL:SIGNAL [1,2].

Though, I've noted before how our initialize-instance protocol
is inferior: It takes slots as its second argument, whereas is
should take initargs instead.  This makes changing the
representation without changing the interface hard, but that
can be worked around with the slot-missing protocol.

> I also don't
>   think `print-object` would be super useful,

It's useful when printing the human-readable error message isn't
appropriate say, because it takes multiple lines.

> More importantly, that doesn't tell me what new things we could do,
> most importantly how to attach context info :-)

You can modify a condition object like you can any other
object.

> > From a CL perspective, and even other languages, using our
> > "error symbol with some 'error property" representation of
> > errors is almost self-evidently inferior to just making it
> > a first class object.
>
> You don't need to convince me of that, indeed.  But that's not what we
> have right now, and it's not trivial to retro-fit it cleanly in the
> current system.  Luckily, AFAICT it's orthogonal to `handler-bind`.

AFAICT CL:HANDLER-BIND is aware of inheritance [3], i.e. the
things you put in "type" of each handler binding in CL:HANDLER-BIND
are interpreted as designators of subclasses of CL:CONDITION and
dispatch similarly to generics.  That shouldn't be too hard
to fit if we say `cl-condition` is the base class and
`cl-error` and `cl-warning` are its children, etc.

João

[1]: https://cl-community-spec.github.io/pages/signal.html
[2]: https://cl-community-spec.github.io/pages/error.html
[3]: https://cl-community-spec.github.io/pages/handler_002dbind.html

Re: Attaching context info to an error

[Stefan Monnier]

>> [ Not clear what `initialize-instance` would be used for.
> It is passed the keyword arguments that are passed to CL:ERROR
> or CL:SIGNAL [1,2].

Yes, I know, but that doesn't tell me what it lets us do that we can't
do right now.

> Though, I've noted before how our initialize-instance protocol
> is inferior: It takes slots as its second argument, whereas is
> should take initargs instead.  This makes changing the
> representation without changing the interface hard, but that
> can be worked around with the slot-missing protocol.

[ While Eric called it "slots" it holds the plist of initargs, just
  like the CLOS one.  The only difference I know is that CLOS passes it as
  an `&rest` arg.  You also told me that CLOS includes the "default" init
  args taken from the `defclass`, but I'm not familiar with
  those details (neither for CLOS nor for EIEIO).  ]

>> I also don't think `print-object` would be super useful,
> It's useful when printing the human-readable error message isn't
> appropriate say, because it takes multiple lines.

I'm not sure the difference qualifies as "super useful".
I don't mean to say we wouldn't be happy to be able to define ad-hoc
methods for `cl-print-object` of error objects, but it's much too minor
to motivate a change of representation.

>> More importantly, that doesn't tell me what new things we could do,
>> most importantly how to attach context info :-)
> You can modify a condition object like you can any other
> object.

I'm afraid that doesn't tell me how to attach context to an
error object.  The context I'm talking about are orthogonal to the error
objects themselves: any kind of context info could be attached to any
kind of error object.

I guess the only way this would help is if we had the foresight, when
designing the root class for error objects, to include a `context` slot
(even though it might go unused most of the time).
Is that what you had in mind?

>> > From a CL perspective, and even other languages, using our
>> > "error symbol with some 'error property" representation of
>> > errors is almost self-evidently inferior to just making it
>> > a first class object.
>>
>> You don't need to convince me of that, indeed.  But that's not what we
>> have right now, and it's not trivial to retro-fit it cleanly in the
>> current system.  Luckily, AFAICT it's orthogonal to `handler-bind`.
>
> AFAICT CL:HANDLER-BIND is aware of inheritance [3], i.e. the
> things you put in "type" of each handler binding in CL:HANDLER-BIND
> are interpreted as designators of subclasses of CL:CONDITION and
> dispatch similarly to generics.  That shouldn't be too hard
> to fit if we say `cl-condition` is the base class and
> `cl-error` and `cl-warning` are its children, etc.

What I mean is that if we start using something else than cons cells to
represent error objects, loads of code will break because of things like:

    (condition-case err
        ...
      (error ... (signal (car err) (cdr err))))


-- Stefan

Re: Attaching context info to an error

[João Távora]

On Wed, Dec 27, 2023 at 8:27 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> [ Not clear what `initialize-instance` would be used for.
> > It is passed the keyword arguments that are passed to CL:ERROR
> > or CL:SIGNAL [1,2].
>
> Yes, I know, but that doesn't tell me what it lets us do that we can't
> do right now.

I guess we can do everything with symbol and cons, it's just
more work.  I'd guess the cl-averse are still doing structs with vectors :-)

>
> > Though, I've noted before how our initialize-instance protocol
> > is inferior: It takes slots as its second argument, whereas is
> > should take initargs instead.  This makes changing the
> > representation without changing the interface hard, but that
> > can be worked around with the slot-missing protocol.
>
> [ While Eric called it "slots" it holds the plist of initargs, just
>   like the CLOS one.  The only difference I know is that CLOS passes it as
>   an `&rest` arg.

Yes, but it's a big one.

Out versionwill go looking for slots for that each initarg and if the
slot doesn't exist it barfs, whereas in Common Lisp you can make it
&accept-other-keys.

I think I also wrote about this off list.  In CL, make-instance can
be given arbitrary keyword args and they will show up in the
initialize-instance.  They may or may not line up with a slot's
initarg.  This enables us to change the data representation while
keeping the interface to the class.  See the current master's
lisp/jsonrpc.el for an example where I had to use slot-missing to
get around this.

> You also told me that CLOS includes the "default" init
>   args taken from the `defclass`, but I'm not familiar with
>   those details (neither for CLOS nor for EIEIO).  ]

Also that.

> >> I also don't think `print-object` would be super useful,
> > It's useful when printing the human-readable error message isn't
> > appropriate say, because it takes multiple lines.
>
> I'm not sure the difference qualifies as "super useful".
> I don't mean to say we wouldn't be happy to be able to define ad-hoc
> methods for `cl-print-object` of error objects, but it's much too minor
> to motivate a change of representation.

You don't even have to define anything.  The cl-print-object of
any EIEIO object is acceptable as a start.  A more advanced
version will do stuff with the objects 'format-control' and
'format-arguments', respecting print-escape and print-readably.

> >> More importantly, that doesn't tell me what new things we could do,
> >> most importantly how to attach context info :-)
> > You can modify a condition object like you can any other
> > object.
>
> I'm afraid that doesn't tell me how to attach context to an
> error object.  The context I'm talking about are orthogonal to the error
> objects themselves: any kind of context info could be attached to any
> kind of error object.

I don't know what "context" you are talking about (I guess it
was in the initial email, but I couldn't grok it).  A setf-able
condition-context generic sounds fair to me tho.  Maybe start
with an external representation in a weak-keyed hash table,
then move it to a slot in the top of the hierarchy?

> I guess the only way this would help is if we had the foresight, when
> designing the root class for error objects, to include a `context` slot
> (even though it might go unused most of the time).
> Is that what you had in mind?

Ah, yes!  Read above, great minds... as usual ;-)

> What I mean is that if we start using something else than cons cells to
> represent error objects, loads of code will break because of things like:
>
>     (condition-case err
>         ...
>       (error ... (signal (car err) (cdr err))))

Ugh indeed.  40 years of leaky abstractions, ¯\_(ツ)_/¯

So yes, "error" err would have that poor man's cons.  Maybe
condition-case would only catch those?  A new shiny
handler-case  on the other hand, would catch proper richie
rich errors (and have the better syntax).

Or maybe do some other syntax trick?

(condition-case err
   ...
  (error ((e)) ... ); 'err' is that strange cons, 'e' is a proper condition
  (warning ((w)) ... ); ditto for 'err' and w.
 )

I'm sure you can come up with better tho :-)

João

Re: Attaching context info to an error

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The question is: where should we put this "context" info.
>>>
>>> One possibility is to do something like the following:
>>>
>>>     (defun load (file ...)
>>>       (handler-bind ((error (lambda (err)
>>>                               (signal 'error-during-load
>>>                                       (cons file err)))))
>>>         ...))
>>
>> I'm just guessing - so the argument ERR of the handler function is not a
>> `condition' object of some kind, in the CL sense? What is it?
>
> No, it's the same kind of objects as are bound to `err` in
> (condition-case err ...), i.e. a cons of (ERROR-NAME . ERROR-DATA).
>
> I don't think it makes sense to have different kinds of objects for
> `handler-bind` and for `condition-case`.

WRT to "collecting dynamic context info". I don't know if the following
is useful, it depends on what the context info is, but maybe it can
serve as an inspiration?

I was reading CMUCL code a bit today. CMUCL (and SBCL) do something 
like the below for handler-bind (and analogously for restart-bind):

  (defvar *handler-clusters* nil)

  (defmacro handler-bind (bindings &body forms)
    "(HANDLER-BIND ( {(type handler)}* )  body)
     Executes body in a dynamic context where the given handler bindings are
     in effect.  Each handler must take the condition being signalled as an
     argument.  The bindings are searched first to last in the event of a
     signalled condition."
    (unless (every #'(lambda (x) (and (listp x) (= (length x) 2))) bindings)
      (simple-program-error (intl:gettext "Ill-formed handler bindings.")))
    `(let ((*handler-clusters*
            (cons (list ,@(mapcar #'(lambda (x) `(cons ',(car x) ,(cadr x)))
                                  bindings))
                  *handler-clusters*)))
       (multiple-value-prog1
        (progn ,@forms)
        ;; Wait for any float exceptions
        #+x87 (float-wait))))

Each handler-bind is associated with a binding of *handler-clusters*,
which contains two things: the handlers of the handler-bind, and a link
to the next such binding up the stack. (This chain is used to find
handlers etc.).

Maybe one could use such a construct in Emacs to additionally associate
context with dynamic occurrances of handler-bind?

Re: Attaching context info to an error

[Stefan Monnier]

>> >> [ Not clear what `initialize-instance` would be used for.
>> > It is passed the keyword arguments that are passed to CL:ERROR
>> > or CL:SIGNAL [1,2].
>> Yes, I know, but that doesn't tell me what it lets us do that we can't
>> do right now.
> I guess we can do everything with symbol and cons,

I don't mean "do" as in "Turing equivalent".
"less work" is a good thing to "do".  I just don't know in which way
`initialize-instance` lets one do less work.

> I'd guess the cl-averse are still doing structs with vectors :-)

I don't think I'm strongly "cl-averse", but I'm rather not familiar
enough with it to know its benefits.  Having modified some of its code
I'm familiar with its cost and what it does, but that doesn't
immediately translate into ideas for how I could make good use of it.

>> [ While Eric called it "slots" it holds the plist of initargs, just
>>   like the CLOS one.  The only difference I know is that CLOS passes it as
>>   an `&rest` arg.
>
> Yes, but it's a big one.

I completely fail to see why.

> Our version will go looking for slots for that each initarg and if the
> slot doesn't exist it barfs, whereas in Common Lisp you can make it
> &accept-other-keys.

You lost me here.  How does that relate to `&rest` vs `not &rest`?

> I think I also wrote about this off list.  In CL, make-instance can
> be given arbitrary keyword args and they will show up in the
> initialize-instance.  They may or may not line up with a slot's
> initarg.

AFAIK the same thing holds for EIEIO's `initialize-instance`.

> This enables us to change the data representation while
> keeping the interface to the class.  See the current master's
> lisp/jsonrpc.el for an example where I had to use slot-missing to
> get around this.

I don't think this has anything to do with `&rest` or not.  It sounds
like a bug in `shared-initialize` or `initialize-instance` (or
a disagreement between the two).  I suggest you try and write a short
recipe for it and make it a bug report.

>> I'm afraid that doesn't tell me how to attach context to an
>> error object.  The context I'm talking about are orthogonal to the error
>> objects themselves: any kind of context info could be attached to any
>> kind of error object.
> I don't know what "context" you are talking about (I guess it
> was in the initial email, but I couldn't grok it).

In different cases we may want to add contextual info about an error,
such as the time at which it was signaled, the backtrace, some subset of
the backtrace (e.g. just the name of the innermost function), or the
list of files we're in the middle of loading, or some node in
a datastructure within which we were operating when the error occurred,
...

Usually which info to collect will depend on what it will be used for,
so it's under the control of the outer code, and is thus orthogonal to
the error that's signaled.

>> What I mean is that if we start using something else than cons cells to
>> represent error objects, loads of code will break because of things like:
>>
>>     (condition-case err
>>         ...
>>       (error ... (signal (car err) (cdr err))))
>
> Ugh indeed.  40 years of leaky abstractions, ¯\_(ツ)_/¯
>
> So yes, "error" err would have that poor man's cons.  Maybe
> condition-case would only catch those?  A new shiny
> handler-case  on the other hand, would catch proper richie
> rich errors (and have the better syntax).

That entails a heavy amount of churn to change old code to use that new
form (while we'd want to support both for backward compatibility, we'd
also want to slowly get rid of the old form to keep the overall
complexity in check).

The upside doesn't seem worth the cost for now.


        Stefan

Re: Attaching context info to an error

[João Távora]

On Thu, Dec 28, 2023 at 7:05 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> >> [ Not clear what `initialize-instance` would be used for.
> >> > It is passed the keyword arguments that are passed to CL:ERROR
> >> > or CL:SIGNAL [1,2].
> >> Yes, I know, but that doesn't tell me what it lets us do that we can't
> >> do right now.
> > I guess we can do everything with symbol and cons,
>
> I don't mean "do" as in "Turing equivalent".
> "less work" is a good thing to "do".  I just don't know in which way
> `initialize-instance` lets one do less work.

It's not just initialize-instance, of course, it's CLOS or an
equivalent.  And what does that give us?  Much that we are reinventing.
And reinventing is "more work" (TM).  Top of my head, CLOS gives you
hierarchies easily, with generic functions doing the right thing
by default when called with an instance of the subclass.  That's the
"standard method combination" [1], easy to define readers, writers
or full accessors... all as generic functions.  A uniform data
representation for which you can someday (TM) write an interactive
object inspector.

> > I'd guess the cl-averse are still doing structs with vectors :-)
>
> I don't think I'm strongly "cl-averse", but I'm rather not familiar
> enough with it to know its benefits.  Having modified some of its code
> I'm familiar with its cost and what it does, but that doesn't
> immediately translate into ideas for how I could make good use of it.

Yes, I have a lot of experience with both Lisps so it's easy to
see the difference where one trumps the other, sometimes
spectacularly. Elisp does trump CL in some every specific areas,
Edebug is easily one of them.

I'm enjoying reading this new "community spec" of CL (links below),
it's really well done.

And if anyone can learn CL in a weekend, it's you ;-)

> I don't think this has anything to do with `&rest` or not.  It sounds
> like a bug in `shared-initialize` or `initialize-instance` (or
> a disagreement between the two).  I suggest you try and write a short
> recipe for it and make it a bug report.

I think I already did (well maybe not report-emacs-bug, but here's
the problem again)

; SLY 1.0.43 (#<MREPL mrepl-1-1>)
CL-USER> (defclass foo () ((bar :initarg :bar) (baz :initarg :baz)))
#<STANDARD-CLASS COMMON-LISP-USER::FOO>
CL-USER> (make-instance 'foo :bar 42 :baz 42) ; presumed client code
#<FOO {10043F2073}>
CL-USER> (defclass foo () ((bar :initarg :bar)))
#<STANDARD-CLASS COMMON-LISP-USER::FOO>
CL-USER> (make-instance 'foo :bar 42 :baz 42) ; whoops
; Debugger entered on #<SB-PCL::INITARG-ERROR {1004464F83}>
CL-USER> (defvar *bazzes* (make-hash-table :weakness :key))
*BAZZES*
CL-USER> (defmethod initialize-instance :after ((x foo) &key baz
&allow-other-keys)
           (setf (gethash x *bazzes*) baz))
#<STANDARD-METHOD COMMON-LISP:INITIALIZE-INSTANCE :AFTER (FOO) {10015066B3}>
CL-USER> (make-instance 'foo :bar 42 :baz 42) ; now works again
#<FOO {1001506EA3}>
CL-USER>

*** Welcome to IELM ***  Type (describe-mode) or press C-h m for help.
ELISP> (defclass foo () ((bar :initarg :bar) (baz :initarg :baz)))
foo
ELISP> (make-instance 'foo :bar 42 :baz 42) ; so far so good
#<foo foo-156aec4e27b2>
ELISP> (defclass foo () ((bar :initarg :bar))) ; redef
foo
ELISP> (make-instance 'foo :bar 42 :baz 42) ; will error, good, expected
*** Eval error ***  Invalid slot name: "#<foo foo-156aec62ba72>", :baz
ELISP> (cl-defmethod initialize-instance :after ((x foo) ((&key baz
&allow-other-keys) t))
         (setf (gethash x *bazzes*) baz))                  ^^^^ odd
specializer, but OK
initialize-instance
ELISP> (defvar bazzes (make-hash-table :weakness 'key))
bazzes
ELISP> (make-instance 'foo :bar 42 :baz 42) ;; will error unexpected
*** Eval error ***  Invalid slot name: "#<foo foo-156aec6034ce>", :baz

Yes, happens in shared-initialize, but can it be fixed while
also keeping the expected error above?  Doesn't seem easy.

In CLOS, the initialize-instance generic automagically becomes
"other-key" friendly if -- and only if -- one of its methods is.
From [2]:

  The set of valid initialization arguments for a class is the
  set of valid initialization arguments that either fill slots or
  supply arguments to methods, along with the predefined
  initialization argument :allow-other-keys. The default value for
  :allow-other-keys is nil.

In fact, this allows you to keep the tight initarg validation
behaviour even without &allow-other-keys in there.  So in CLOS, you
can

  (defmethod initialize-instance :after ((x foo) &key baz)
      (setf (gethash x *bazzes*) baz))

So that

   (make-instance 'foo :blergh 42)

still reassuringly errors.

> >> I'm afraid that doesn't tell me how to attach context to an
> >> error object.  The context I'm talking about are orthogonal to the error
> >> objects themselves: any kind of context info could be attached to any
> >> kind of error object.
> > I don't know what "context" you are talking about (I guess it
> > was in the initial email, but I couldn't grok it).
>
> In different cases we may want to add contextual info about an error,
> such as the time at which it was signaled, the backtrace, some subset of
> the backtrace (e.g. just the name of the innermost function), or the
> list of files we're in the middle of loading, or some node in
> a data structure within which we were operating when the error occurred,
> ...
>
> Usually which info to collect will depend on what it will be used for,
> so it's under the control of the outer code, and is thus orthogonal to
> the error that's signaled.

OK, I think I'm starting to see.  You're looking at this from a data
representation perspective, and CL solves (presumably) the same
problem you are trying to solve from more of a behavioral perspective.

What CL implementations do for the "conveniences "I presume you're
talking about is establish restarts and keep this information in
the stack.

Say you have this generic form

(with-foo-convenience ()
   (with-bar-convenience ()
      (the-great-unknown)))

So if a CALL-WITH-FOO-CONVENIENCE is dynamically wrapping some error
throwing code, it will typically let-bind *FOO-DATA*, some restart
that operates (reading or writing) on that *FOO-DATA* and a
handler-bind.

Note that restarts aren't only invocable, stack-unwinding things:
they're also self-reporting and can hide themselves.  This becomes
relevant as other handlers -- or the human operating the debugger --
tries to find them to fix the course of the program.

Anyway, when an error crosses the handler-bind in its upstack path,
*FOO-DATA* may be augmented with the "context" and nothing else
happens.  When ultimately the human-operated debugger is reached the
restarts that are available there can are nicely listed with
formatted details as their report functions are invoked and read
from the enriched *FOO-DATA*.

For example, *FOO-DATA* could keep track of the time it observed the
LITTLE-GREEN-MAN condition C pass by.  And the REPORT-TO-SETI restart's
human readable :REPORT would look something like
"Report suspicious sighting at XX:XX:XX UTC" or something.

Importantly, a HANDLER-BIND in the inner CALL-WITH-BAR-CONVENIENCE can
_also_ invoke restarts previously established by
CALL-WITH-FOO-CONVENIENCE, that are only "findable" if those
restarts' :TEST functions [3] say they are.  These functions also
have access to C and lexical access to *FOO-DATA*.

In summary, I think the "context" you are thinking about can just
be good old dynamic context, i.e. let-bindings of special
variables, with the new HANDLER-BIND-powered twist that these
variables can remain private to their users _and_ be augmented
after they are initially established.

Right?  Wrong?  It would at least seem to alleviate your own doubts
about the context "going unused most of the time".  It keeps the
"orthogonal" property and no general-purpose duck-typed "context"
needed in the base object or in a hash table or somesuch.

> > Ugh indeed.  40 years of leaky abstractions, ¯\_(ツ)_/¯
> >
> > So yes, "error" err would have that poor man's cons.  Maybe
> > condition-case would only catch those?  A new shiny
> > handler-case  on the other hand, would catch proper richie
> > rich errors (and have the better syntax).
>
> That entails a heavy amount of churn to change old code to use that new
> form (while we'd want to support both for backward compatibility, we'd

Not sure if you understand that what I proposed _was_ in principle
100% backward compatible.  So on day 0 that 'error' (or in fact
'signal) starts throwing objects and condition-case reconstructs
the cons nothing needs to change immediately.

> also want to slowly get rid of the old form to keep the overall
> complexity in check).

That's never stopped us from introducing better abstractions.
In fact there's talk right of introducing a cond* to replace
10 year old pcase, with very heavy question marks whether it is
really better.  Where I think these question marks simply are not
there for handler-bind and handler-case (restarts, CLOS and very
many CL things).

And -- I'm not kidding -- i think 90-95% of this particular
trivial code churn (turning condition-case to handler-case)
is easily handled by AI things these days. See [4] for a brilliant
presentation of something I plan to use soon.

> The upside doesn't seem worth the cost for now.

It'll be harder 40 years from now.

João

[1]: https://cl-community-spec.github.io/pages/Standard-Method-Combination.html
[2]: https://cl-community-spec.github.io/pages/Declaring-the-Validity-of-Initialization-Arguments.html#Declaring-the-Validity-of-Initialization-Arguments
[3]: https://cl-community-spec.github.io/pages/restart_002dcase.html
[4]: https://www.youtube.com/watch?v=bsRnh_brggM

Re: Attaching context info to an error

[Stefan Monnier]

> It's not just initialize-instance, of course, it's CLOS or
> an equivalent.

You wrote:

    [...] And then, CLOS is good :-) you get access to lots of existing
    useful protocols.  initialize-instance, print-object, etc for free.

Which implies that `initialize-instance` is a useful protocol.
I'd just be happy to see a concrete example of something useful you can
do with it that's easier than what we do with "old style Lisp".

> Top of my head, CLOS gives you hierarchies easily, with generic
> functions doing the right thing by default when called with an
> instance of the subclass.

You do realize that I wrote `cl-generic.el`, made changes to
`cl-defstruct` to better integrate with EIEIO, ...

> That's the "standard method combination" [1],

... and this as well, right?

> *** Welcome to IELM ***  Type (describe-mode) or press C-h m for help.
> ELISP> (defclass foo () ((bar :initarg :bar) (baz :initarg :baz)))
> foo
> ELISP> (make-instance 'foo :bar 42 :baz 42) ; so far so good
> #<foo foo-156aec4e27b2>
> ELISP> (defclass foo () ((bar :initarg :bar))) ; redef
> foo

Redefinition of a class is definitely not currently supported (with no
plan to support it in the foreseeable future).

But in the `jsonrpc.el` there is no redefinition, so it seems to be
another problem.  Can you open a bug with a small recipe that
reproduces the problem that you encountered while writing `jsonrpc.el`?

> In summary, I think the "context" you are thinking about can just
> be good old dynamic context, i.e. let-bindings of special
> variables, with the new HANDLER-BIND-powered twist that these
> variables can remain private to their users _and_ be augmented
> after they are initially established.
>
> Right?  Wrong?

That describes where the context is placed until the error is signaled.
The question is how/where to preserve (part of) this context information
when some code wants to use it *after* unwinding the stack.

E.g. in `bytecomp.el` on the handler-bind branch I solved it as follows:

	      (let ((form-stack nil))
		(condition-case error-info
		    (handler-bind
		        ((error (lambda (_err)
		                  (setq form-stack byte-compile-form-stack))))
		      (funcall body-fn))
		  (error (let ((byte-compile-form-stack form-stack))
		           (byte-compile-report-error error-info))))))))

where `byte-compile-form-stack` is basically your *FOO-DATA*, and
I preserve/propagate this info by stashing it in the
`form-stack` variable.

Maybe we could have gotten away with calling `byte-compile-report-error`
directly from the `handler-bind` handler, but that could be affected by
all kinds of dynamic let-bindings and whatnot.

Another example is in the C code: when we call `signal`, the C code
walks the backtrace to find the name of the inner function, stores it in
the global `Vsignaling_function` and then uses it in
`cmd_error(_internal)` (i.e. after unwinding the stack) to include it in
the error message.  The us of a global var here is not just ugly but
it's also wrong because other ELisp code can be executed between the
moment we store the info in the var and the moment we use it, so it can
be polluted by the info from some other error.
[ And yes, we could rewrite that code to do something like what I did
  in `bytecomp.el` above.  ]

>> The upside doesn't seem worth the cost for now.
> It'll be harder 40 years from now.

I guess we'll see :-)


        Stefan

Re: Attaching context info to an error

[João Távora]

On Thu, Dec 28, 2023 at 4:03 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > It's not just initialize-instance, of course, it's CLOS or
> > an equivalent.
>
> You wrote:
>
>     [...] And then, CLOS is good :-) you get access to lots of existing
>     useful protocols.  initialize-instance, print-object, etc for free.
>
> Which implies that `initialize-instance` is a useful protocol.
> I'd just be happy to see a concrete example of something useful you can
> do with it that's easier than what we do with "old style Lisp".

Just see eglot.el and jsonrpc.el for examples.  It's a constructor
in the C++ style.  Meaning the client of the library, when making
an object, won't get just a bunch of data.  They will either
get a fully functioning object _or_ an error.  You can't do that
in plain Elisp without "-init" custom constructors, just as
you can't do that in C without foo = (Foo*)malloc() + init_foo(foo).

Furthermore, just like in typical OO, derivation of such classes
are guaranteed to have the base class parts valid and constructed
in their `intialize :after` this is the bread and butter of CLOS,
and it's awkward and error-prone to program without it.

> > Top of my head, CLOS gives you hierarchies easily, with generic
> > functions doing the right thing by default when called with an
> > instance of the subclass.
>
> You do realize that I wrote `cl-generic.el`, made changes to
> `cl-defstruct` to better integrate with EIEIO, ...
>
> > That's the "standard method combination" [1],
>
> ... and this as well, right?

Right, it's because I _know_ that you wrote these things that
I find it odd you do now see how useful they are.  Maybe you
just did this as good samaritan work?  In any case, I am
extremely indebted.  It makes Eglot and other complex extensions
much easier to reason about.

> > *** Welcome to IELM ***  Type (describe-mode) or press C-h m for help.
> > ELISP> (defclass foo () ((bar :initarg :bar) (baz :initarg :baz)))
> > foo
> > ELISP> (make-instance 'foo :bar 42 :baz 42) ; so far so good
> > #<foo foo-156aec4e27b2>
> > ELISP> (defclass foo () ((bar :initarg :bar))) ; redef
> > foo
>
> Redefinition of a class is definitely not currently supported (with no
> plan to support it in the foreseeable future).

In this case, it was a new version of jsonrpc.el that has to
be backward compatible to clients programming against the older
version.  CLOS class redefinition is big business indeed,
but just removing a slot here in the same image worked nicely
here for demonstration purposes: it still errored as reassuringly
as it would have if in another Emacs session.

> But in the `jsonrpc.el` there is no redefinition, so it seems to be
> another problem.  Can you open a bug with a small recipe that
> reproduces the problem that you encountered while writing `jsonrpc.el`?

I was just making jsonrpc.el backward compatible to, say, older
eglot.el's.

> > In summary, I think the "context" you are thinking about can just
> > be good old dynamic context, i.e. let-bindings of special
> > variables, with the new HANDLER-BIND-powered twist that these
> > variables can remain private to their users _and_ be augmented
> > after they are initially established.
> >
> > Right?  Wrong?
>
> That describes where the context is placed until the error is signaled.

No.  _After_ the error is signalled, and before the error is _handled_.

> The question is how/where to preserve (part of) this context information
> when some code wants to use it *after* unwinding the stack.

If the stack is unwound all goes to heck indeed.  I think most problems
can (and should) be done precisely without unwinding the stack.  The
stack has all the good stuff.  Why unwind it before you need to?

> E.g. in `bytecomp.el` on the handler-bind branch I solved it as follows:
>
>               (let ((form-stack nil))
>                 (condition-case error-info
>                     (handler-bind
>                         ((error (lambda (_err)
>                                   (setq form-stack byte-compile-form-stack))))
>                       (funcall body-fn))
>                   (error (let ((byte-compile-form-stack form-stack))
>                            (byte-compile-report-error error-info))))))))
>
> where `byte-compile-form-stack` is basically your *FOO-DATA*, and
> I preserve/propagate this info by stashing it in the
> `form-stack` variable.

Why the condition-case?  Why not report it from the handler-bind and
let the error go about its business.  Or maybe don't let the error
go about its business and invoke a restart instead.

> Maybe we could have gotten away with calling `byte-compile-report-error`
> directly from the `handler-bind` handler, but that could be affected by
> all kinds of dynamic let-bindings and whatnot.

Exactly my idea above.

Are these let-bindings "legal"?  If they aren't, you can do a
WITH-STANDARD-BYTE-COMPILER-BINDINGS macro and report immediately
or you can establish a restart instead.  The restart transfers
control (and information, of course) to a point up-stack where
presumably sane bindings are in effect...

But the two options are very different.  I think the second
is better.  In general you want the error to travel further up
the stack where it can be analyzed by more clients, who add
more restarts.  You can have a much more educated pick much
in some outer part.

Then, if some *DEBUG-ON-BYTE-COMPILER-ERROR* was true, you'd get
the debugger again, with lots of interactive restarts, notably
retrying the compilation, reporting the error and aborting,
even interactive restart.  If it were false, one of those restarts
would be invoked automatically (I believe "report-and-abort")
and you'd have what you have now on your branch.

As long you invoke restarts from HANDLE-BIND clauses, you
know you have the full stack there, and this allows to have
higher level wrapping "monitoring" code that finds certain
standard restarts.  CL:ABORT the function, for example, calls
the most recently established restart named "ABORT".  See [1]
for more standard names.

> Another example is in the C code: when we call `signal`, the C code
> walks the backtrace to find the name of the inner function, stores it in
> the global `Vsignaling_function` and then uses it in
> `cmd_error(_internal)` (i.e. after unwinding the stack) to include it in
> the error message.  The us of a global var here is not just ugly but
> it's also wrong because other ELisp code can be executed between the
> moment we store the info in the var and the moment we use it, so it can
> be polluted by the info from some other error.
> [ And yes, we could rewrite that code to do something like what I did
>   in `bytecomp.el` above.  ]

Then that's what we should do :-)  Whatever this "include
in the error message" entails should be done while the Lisp
stack is still alive.  Immediately after, some Elisp version
of CL:ABORT should be called, I presume.

João

PS: Sorry for the CL::SHOUTING by the way, it's easier than
tiny quotes sometimes.

[1] https://cl-community-spec.github.io/pages/f_abort.html

Re: Attaching context info to an error

[Stefan Monnier]

> Just see eglot.el and jsonrpc.el for examples.  It's a constructor
> in the C++ style.  Meaning the client of the library, when making
> an object, won't get just a bunch of data.  They will either
> get a fully functioning object _or_ an error.  You can't do that
> in plain Elisp without "-init" custom constructors, just as
> you can't do that in C without foo = (Foo*)malloc() + init_foo(foo).

Not sure where using a "smart constructor" function (what you call
"-init" custom constructor) is less work than defining
a `initialize-instance` method.

> Furthermore, just like in typical OO, derivation of such classes
> are guaranteed to have the base class parts valid and constructed
> in their `intialize :after` this is the bread and butter of CLOS,
> and it's awkward and error-prone to program without it.

Ah... I think I'm beginning to see why I don't see the benefit: in my
world, objects are basically immutable, so an object that's partly
initialized (as in your "have the base class parts valid") is just
already very weird :-)

> Right, it's because I _know_ that you wrote these things that I find
> it odd you do now see how useful they are.

There's a difference between knowing how to build a car and being
a good driver, yes.

I can write a Prolog interpreter but my Prolog programming skills suck.
Similarly, I'm more comfortable using Haskell-style type classes (or
the closely related C++ templates) than "classic OOP" with subtyping.

>> Redefinition of a class is definitely not currently supported (with no
>> plan to support it in the foreseeable future).
> In this case, it was a new version of jsonrpc.el that has to
> be backward compatible to clients programming against the older
> version.

Then your bug report would be a feature request, I guess.
E.g. include a small patch to `jsonrpc.el` showing the code
you would *like* to use.

> CLOS class redefinition is big business indeed, but just removing
> a slot here in the same image worked nicely here for demonstration
> purposes: it still errored as reassuringly as it would have if in
> another Emacs session.

Yes, I'm aware of this.
But ELisp is a fairly limited "programming system" (as opposed to
programming language", see doi:10.1145/2384592.2384611).

I'm a PL guy and one from the functional programming side (i.e. averse
to mutation) to boot, so redefining the classes of existing objects
makes my head hurt.


        Stefan

Re: Attaching context info to an error

[João Távora]

On Thu, Dec 28, 2023 at 7:22 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Just see eglot.el and jsonrpc.el for examples.  It's a constructor
> > in the C++ style.  Meaning the client of the library, when making
> > an object, won't get just a bunch of data.  They will either
> > get a fully functioning object _or_ an error.  You can't do that
> > in plain Elisp without "-init" custom constructors, just as
> > you can't do that in C without foo = (Foo*)malloc() + init_foo(foo).
>
> Not sure where using a "smart constructor" function (what you call
> "-init" custom constructor) is less work than defining
> a `initialize-instance` method.

That's right!  It isn't, it's more work, and more dangerous because
you will forget when to call it.

> > Furthermore, just like in typical OO, derivation of such classes
> > are guaranteed to have the base class parts valid and constructed
> > in their `intialize :after` this is the bread and butter of CLOS,
> > and it's awkward and error-prone to program without it.
>
> Ah... I think I'm beginning to see why I don't see the benefit: in my
> world, objects are basically immutable, so an object that's partly
> initialized (as in your "have the base class parts valid") is just
> already very weird :-)

It's not weird.  If D derived from B then the object that is passed
to the derived mode's contructor is already a fully functioning B.
This is exactly as in C++ where at least that part is bullet proof.
Not so in C and plain Elisp where allocation is one thing and
initialization another thing.

> > Right, it's because I _know_ that you wrote these things that I find
> > it odd you do now see how useful they are.
>
> There's a difference between knowing how to build a car and being
> a good driver, yes.
>
> I can write a Prolog interpreter but my Prolog programming skills suck.
> Similarly, I'm more comfortable using Haskell-style type classes (or
> the closely related C++ templates) than "classic OOP" with subtyping.

Many modern C++ template tricks are done using inheritance,
std::tuple comes to mind.  "Classic OOP" is about runtime
polymophism/virtual dispatch and yes it's seen better days
in the systems programming world, because the indirection
usually trashes caches and double dispatch is silly hard.

But for sporadic exceptions/conditions with objects generic functions
that to multiple dispatch beautifully?  I'd be hard pressed to
see better than what CL has.

> >> Redefinition of a class is definitely not currently supported (with no
> >> plan to support it in the foreseeable future).
> > In this case, it was a new version of jsonrpc.el that has to
> > be backward compatible to clients programming against the older
> > version.
>
> Then your bug report would be a feature request, I guess.

It's a bug inasmuch as EIEIO purports to be a port of CLOS,
where it's easy to do.

I don't need any class redefinition support.  The recipes
I gave earlier would be easy to replicate even if both CL
didn't support such trivial things.

> E.g. include a small patch to `jsonrpc.el` showing the code
> you would *like* to use.

I'd just like to remove the horrible slot-missing hack, it's
clearly marked in jsonrpc.el, just search for "Yuck!".

> > CLOS class redefinition is big business indeed, but just removing
> > a slot here in the same image worked nicely here for demonstration
> > purposes: it still errored as reassuringly as it would have if in
> > another Emacs session.
>
> Yes, I'm aware of this.
> But ELisp is a fairly limited "programming system" (as opposed to
> programming language", see doi:10.1145/2384592.2384611).
>
> I'm a PL guy and one from the functional programming side (i.e. averse
> to mutation) to boot, so redefining the classes of existing objects
> makes my head hurt.

Mine too, in fact in many years of CL I've not seen a single
case of useful CHANGE-CLASS.  But Emacs and Lisp machines are
designed as object landscapes mutating all the way.

Anyway, this is all besides the point: that protocol has nothing
to do with the current limitations of EIEIO's initialize-instance,
and even those limitations are quite small in comparison to the
benefits.

I see you've snipped all the rest of the "where to keep
the context" discussion.  Does that mean that my answer of
"stack _is_ the context, unwind only when absolutely done with it"
is satisfactory?  Did you understand how a standard named
restart for reporting byte compiler errors could be invoked
from "up above"?

What object exactly goes into the handler bind handlers?  Is
it the funny cons?  Can we at least get some generic functions
to hide that representation so that we can change it later on?

João

Re: Attaching context info to an error

[Stefan Monnier]

> What object exactly goes into the handler bind handlers?

The same (even `eq`) as the one passed to the `condition-case` handler.

> Is it the funny cons?

Yup.  That's ELisp's current notion of an "error object".

> Can we at least get some generic functions to hide that representation
> so that we can change it later on?

Patch very welcome, yes (and this is orthogonal to `handler-bind`).


        Stefan

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 2:55 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > What object exactly goes into the handler bind handlers?
>
> The same (even `eq`) as the one passed to the `condition-case` handler.

Would it matter if it weren't?

> > Is it the funny cons?
>
> Yup.  That's ELisp's current notion of an "error object".
>
> > Can we at least get some generic functions to hide that representation
> > so that we can change it later on?
>
> Patch very welcome, yes (and this is orthogonal to `handler-bind`).

What operations are typically requested from it?

- Printing?  a cl-print-object would be sufficient, I think,
  along with a (orthogonal) directive to 'format' for using
  cl-prin1 (why doesn't cl-princ exist?)

- Getting the type for resignalling purposes?  Unfortunately,
  this returns 'cons'

     (condition-case err (error "bla") (error (type-of err)))

  Can it be made to return error?

- Can we import a typep (similar to cl-typep) that understands
  basic types EIEIO and also error hierarchy?

- simple-condition-format-control and
  simple-condition-format-arguments?

- change manual to explain that resignalling is discouraged now
  that handler-bind exists, but if resignalling _is_ done, then
  definitely avoid car and cdr and some form of

    (signal (error-symbol err) (error-data err)

  introducing these new functions?

- go through the hierarchy and add <error-name>-<data-slot>
  accessors? Or do we already consider these details of "data" to
  to be hidden?  If so, I think surely some violations exist,
  hopefully not many.

  They shouldn't be terribly hard to find.  From some greps I
  estimate there less than 700 condition-case that actually use
  the variable and the majority seems to be resignalling or
  printing.  The former will probably be fixed by
  handler-bind anyway, and printing is a generic operation
  already.  I checked about 20 occurances randomly, and they
  all fell into this basket.  This also tells me that switching
  to proper objects isn't that hard.

João

Re: Attaching context info to an error

[Stefan Monnier]

>> > What object exactly goes into the handler bind handlers?
>> The same (even `eq`) as the one passed to the `condition-case` handler.
> Would it matter if it weren't?

I think we want to preserve the identity of an error as it
passes through its various handlers, yes.
That makes it possible to use `eq` hash tables and to mutate them.

> - Printing?  a cl-print-object would be sufficient, I think,
>   along with a (orthogonal) directive to 'format' for using
>   cl-prin1 (why doesn't cl-princ exist?)

We cannot reliably distinguish an error object from random other cons
cells, so things like `cl-print-object` are simply not an option: the
caller needs to tell us that this is an error object.
We could have a `print-error-object`, OTOH.

> - Getting the type for resignalling purposes?  Unfortunately,
>   this returns 'cons'
>
>      (condition-case err (error "bla") (error (type-of err)))
>
>   Can it be made to return error?

If it's only for resignaling purposes, why not have
`error-resignal` instead which takes the error object itself?

> - Can we import a typep (similar to cl-typep) that understands
>   basic types EIEIO and also error hierarchy?

Nope for the reason mentioned above.
But we could have a `error-typep`.  AFAIK this kind of operation is very
rarely used in ELisp (it's only used internally in the C code which
looks for the appropriate handler).

> - simple-condition-format-control and
>   simple-condition-format-arguments?

I can't think of any ELisp code I've encountered that does something
related, so I think it's a YAGNI.

BTW, the best way to find what we need is probably to change the
`signal_or_quit` code which does the `Fcons (error_symbol, data)` and
make it build something else than a cons instead.

Then go and fix all the things that break, introducing new `error-FOO`
abstractions along the way.

> - change manual to explain that resignalling is discouraged now
>   that handler-bind exists, but if resignalling _is_ done, then
>   definitely avoid car and cdr and some form of
>
>     (signal (error-symbol err) (error-data err)
>
>   introducing these new functions?

We first want to help people make their existing code compatible with
other representations of error objects which minimal changes.
Changing the code to use `error-resignal` is trivial whereas making it
use `handler-bind` can be difficult since the semantics is subtly
different and it's not always easy to judge whether the difference
matters in this particular case.

> - go through the hierarchy and add <error-name>-<data-slot>
>   accessors?

I don't think we want to do that systematically, but yes (and I'm afraid
we'll discover along the way that the representations we use are messy
and don't always obey the subtyping suggested by the error hierarchy).

It presumably needs to come with a "similar" change on the constructor
side, so as long as we keep using (signal ERROR-TYPE ERROR-DATA) where
ERROR-DATA is a bare list it's not super pressing.

>   They shouldn't be terribly hard to find.  From some greps I
>   estimate there less than 700 condition-case that actually use
>   the variable and the majority seems to be resignalling or
>   printing.  The former will probably be fixed by
>   handler-bind anyway, and printing is a generic operation
>   already.  I checked about 20 occurances randomly, and they
>   all fell into this basket.  This also tells me that switching
>   to proper objects isn't that hard.

Even better,


        Stefan

Re: Attaching context info to an error

[Alan Mackenzie]

Hello, João.

I'll admit I've not been following this thread closely, but, ...

On Fri, Dec 29, 2023 at 03:43:54 +0000, João Távora wrote:
> On Fri, Dec 29, 2023 at 2:55 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > > What object exactly goes into the handler bind handlers?

> > The same (even `eq`) as the one passed to the `condition-case` handler.

> Would it matter if it weren't?

> > > Is it the funny cons?

> > Yup.  That's ELisp's current notion of an "error object".

> > > Can we at least get some generic functions to hide that representation
> > > so that we can change it later on?

No, no, no.  Please.  (See below.)

> > Patch very welcome, yes (and this is orthogonal to `handler-bind`).

> What operations are typically requested from it?

> - Printing?  a cl-print-object would be sufficient, I think,
>   along with a (orthogonal) directive to 'format' for using
>   cl-prin1 (why doesn't cl-princ exist?)

> - Getting the type for resignalling purposes?  Unfortunately,
>   this returns 'cons'

>      (condition-case err (error "bla") (error (type-of err)))

>   Can it be made to return error?

> - Can we import a typep (similar to cl-typep) that understands
>   basic types EIEIO and also error hierarchy?

> - simple-condition-format-control and
>   simple-condition-format-arguments?

> - change manual to explain that resignalling is discouraged now
>   that handler-bind exists, but if resignalling _is_ done, then
>   definitely avoid car and cdr and some form of

>     (signal (error-symbol err) (error-data err)

>   introducing these new functions?

> - go through the hierarchy and add <error-name>-<data-slot>
>   accessors? Or do we already consider these details of "data" to
>   to be hidden?  If so, I think surely some violations exist,
>   hopefully not many.

>   They shouldn't be terribly hard to find.  From some greps I
>   estimate there less than 700 condition-case that actually use
>   the variable and the majority seems to be resignalling or
>   printing.  The former will probably be fixed by
>   handler-bind anyway, and printing is a generic operation
>   already.  I checked about 20 occurances randomly, and they
>   all fell into this basket.  This also tells me that switching
>   to proper objects isn't that hard.

Talk about "generic functions" and "proper objects" makes me worried
indeed.  The Emacs error handling must work fully right from early
bootstrap, i.e. when no Lisp has yet been loaded.  The "proper object"
for this is a list, of defined structure.  This is approximately what we
have at the moment.  It would be useful to tighten up this definition.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 5:19 PM Alan Mackenzie <acm@muc.de> wrote:

> Talk about "generic functions" and "proper objects" makes me worried
> indeed.  The Emacs error handling must work fully right from early
> bootstrap, i.e. when no Lisp has yet been loaded.  The "proper object"
> for this is a list, of defined structure.

It's indistinguishable from a cons, which is bad for many reasons
explained at length.  Weren't you involved in "symbol-with-pos"?
It's a similar object with a unique type.  So it's going to need
more arguments than just being scared of generic functions.
Noone is advocating for calling generic functions from C or
preloading them, etc, so don't worry about it.

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 4:54 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> > What object exactly goes into the handler bind handlers?
> >> The same (even `eq`) as the one passed to the `condition-case` handler.
> > Would it matter if it weren't?
>
> I think we want to preserve the identity of an error as it
> passes through its various handlers, yes.
> That makes it possible to use `eq` hash tables and to mutate them.

First, I think that's a bad idea for the reasons I explained
at length.  You should "remember" errors via restarts.

Most importantly, and idea quality apart, who can be doing that
today, before h-b?  No-one or practically no-one, since it's useless
because currently the only place user code has access to
the signal also unwinds the stack, and resignalling will make a new
cons. Right?

So unless there's a specific use for it today, I recommend
specifically _not_ maintaining 'eq'ness.  This prevents
people from making the job to transition to a non-cons
representation of errors easier.

> > - Printing?  a cl-print-object would be sufficient, I think,
> >   along with a (orthogonal) directive to 'format' for using
> >   cl-prin1 (why doesn't cl-princ exist?)
>
> We cannot reliably distinguish an error object from random other cons
> cells, so things like `cl-print-object` are simply not an option: the
> caller needs to tell us that this is an error object.
> We could have a `print-error-object`, OTOH.

Indeed this representation is really bothersome.

> > - Getting the type for resignalling purposes?  Unfortunately,
> >   this returns 'cons'
> >
> >      (condition-case err (error "bla") (error (type-of err)))
> >
> >   Can it be made to return error?
>
> If it's only for resignaling purposes, why not have
> `error-resignal` instead which takes the error object itself?

This has to exist indeed.  Should be called "condition-resignal"
though (and the byte-compiler should warn you should use a
handler-bind instead).

> > - Can we import a typep (similar to cl-typep) that understands
> >   basic types EIEIO and also error hierarchy?
>
> Nope for the reason mentioned above.
> But we could have a `error-typep`.  AFAIK this kind of operation is very
> rarely used in ELisp (it's only used internally in the C code which
> looks for the appropriate handler).

Let's hope so.  Skipping that then.

> > - simple-condition-format-control and
> >   simple-condition-format-arguments?
>
> I can't think of any ELisp code I've encountered that does something
> related, so I think it's a YAGNI.

Hard to add with that random data bag'o'things anyway.

> BTW, the best way to find what we need is probably to change the
> `signal_or_quit` code which does the `Fcons (error_symbol, data)` and
> make it build something else than a cons instead.

True!  Let's just preload EIEIO and call make-instance.  Ahaha

> Then go and fix all the things that break, introducing new `error-FOO`
> abstractions along the way.
>
> > - change manual to explain that resignalling is discouraged now
> >   that handler-bind exists, but if resignalling _is_ done, then
> >   definitely avoid car and cdr and some form of
> >
> >     (signal (error-symbol err) (error-data err)
> >
> >   introducing these new functions?
>
> We first want to help people make their existing code compatible with
> other representations of error objects which minimal changes.
> Changing the code to use `error-resignal` is trivial whereas making it
> use `handler-bind` can be difficult since the semantics is subtly
> different and it's not always easy to judge whether the difference
> matters in this particular case.

Makes sense, though 100% of the cases I've seen for resignalling are
equivalent functionally to a handler-bind.

> > - go through the hierarchy and add <error-name>-<data-slot>
> >   accessors?
>
> I don't think we want to do that systematically, but yes (and I'm afraid
> we'll discover along the way that the representations we use are messy
> and don't always obey the subtyping suggested by the error hierarchy).
>
> It presumably needs to come with a "similar" change on the constructor
> side, so as long as we keep using (signal ERROR-TYPE ERROR-DATA) where
> ERROR-DATA is a bare list it's not super pressing.

I've also seen 0 examples so far where the DATA is searched inside
for actual properties.

João

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 5:29 PM João Távora <joaotavora@gmail.com> wrote:
>
> On Fri, Dec 29, 2023 at 4:54 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> >
> > >> > What object exactly goes into the handler bind handlers?
> > >> The same (even `eq`) as the one passed to the `condition-case` handler.
> > > Would it matter if it weren't?
> >
> > I think we want to preserve the identity of an error as it
> > passes through its various handlers, yes.
> > That makes it possible to use `eq` hash tables and to mutate them.
>
> First, I think that's a bad idea for the reasons I explained
> at length.  You should "remember" errors via restarts.

More precisely, even restarts are not necessary.  One should
remember errors by not forgetting about them!  I.e. use
handler-bind.

João

Re: Attaching context info to an error

[Alan Mackenzie]

Hello, João.

On Fri, Dec 29, 2023 at 17:24:29 +0000, João Távora wrote:
> On Fri, Dec 29, 2023 at 5:19 PM Alan Mackenzie <acm@muc.de> wrote:

> > Talk about "generic functions" and "proper objects" makes me worried
> > indeed.  The Emacs error handling must work fully right from early
> > bootstrap, i.e. when no Lisp has yet been loaded.  The "proper object"
> > for this is a list, of defined structure.

> It's indistinguishable from a cons, which is bad for many reasons
> explained at length.

But it's a good choice for early bootstrap.  How about dealing with
those reasons in other ways rather than using not-yet-existent data
structures.

> Weren't you involved in "symbol-with-pos"?

I developped them, yes.

> It's a similar object with a unique type.

Similar to what?

> So it's going to need more arguments than just being scared of generic
> functions.

There's no fear involved at all.  cl-lib is an exceptionally
controversial part of Emacs, as we established two or three weeks ago.
It's poorly documented, and likely buggy.  It's not something to be
moved to the centre of the Emacs Lisp machine.

> Noone is advocating for calling generic functions from C or preloading
> them, etc, so don't worry about it.

That's got no connection with what I wrote.  What I'm asking you to
confirm is that you're not contemplating introducing error handling
machinery which, during bootstrap, will only work after Lisp code has
been loaded.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 5:43 PM Alan Mackenzie <acm@muc.de> wrote:

> > It's a similar object with a unique type.
>
> Similar to what?

The object that signal() would craft and appear to Lisp is similar to how
symbol-with-pos is passed to, say, compiler macros (breaking some of
them, last I remember, but that's besides our discussion).

> > So it's going to need more arguments than just being scared of generic
> > functions.
>
> There's no fear involved at all.  cl-lib is an exceptionally
> controversial part of Emacs, as we established two or three weeks ago.
> It's poorly documented, and likely buggy.  It's not something to be
> moved to the centre of the Emacs Lisp machine.

generic functions aren't in cl-lib, and, once again, noone is proposing
doing anything of the sort, so why you bring cl-lib up is mystery to me.

> > Noone is advocating for calling generic functions from C or preloading
> > them, etc, so don't worry about it.
>
> That's got no connection with what I wrote.  What I'm asking you to
> confirm is that you're not contemplating introducing error handling
> machinery which, during bootstrap, will only work after Lisp code has
> been loaded.

I think I can confirm that yes.

João

Re: Attaching context info to an error

[Alan Mackenzie]

Hello, João.

On Fri, Dec 29, 2023 at 17:54:30 +0000, João Távora wrote:
> On Fri, Dec 29, 2023 at 5:43 PM Alan Mackenzie <acm@muc.de> wrote:

> > > It's a similar object with a unique type.

> > Similar to what?

> The object that signal() would craft and appear to Lisp is similar to how
> symbol-with-pos is passed to, say, compiler macros (breaking some of
> them, last I remember, but that's besides our discussion).

Ah, right!  So you're considering, possibly, a new type of pseudovector,
or something similar.

> > > So it's going to need more arguments than just being scared of generic
> > > functions.

> > There's no fear involved at all.  cl-lib is an exceptionally
> > controversial part of Emacs, as we established two or three weeks ago.
> > It's poorly documented, and likely buggy.  It's not something to be
> > moved to the centre of the Emacs Lisp machine.

> generic functions aren't in cl-lib, and, once again, noone is proposing
> doing anything of the sort, so why you bring cl-lib up is mystery to me.

The word "generic" was mentioned, which makes me think of cl-generic.el,
a file I'm struggling quite badly with, to debug.  By cl-lib, I meant
cl-*.el.  Sorry for not being more exact.

> > > Noone is advocating for calling generic functions from C or preloading
> > > them, etc, so don't worry about it.

> > That's got no connection with what I wrote.  What I'm asking you to
> > confirm is that you're not contemplating introducing error handling
> > machinery which, during bootstrap, will only work after Lisp code has
> > been loaded.

> I think I can confirm that yes.

Thanks!

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Attaching context info to an error

[Stefan Monnier]

> That's got no connection with what I wrote.  What I'm asking you to
> confirm is that you're not contemplating introducing error handling
> machinery which, during bootstrap, will only work after Lisp code has
> been loaded.

The idea is:

- In the short term, provide new functions that allow manipulating
  errors in a way that's independent from their representation
  (as cons cells).
  Hopefully this will make the code also more readable in many cases.
  E.g. (error-resignal ERR) instead of (signal (car ERR) (cdr ERR)).

- In the longer term once enough code has been adapted to use the new
  functions, we may be able to consider changing the representation of
  error objects.  Presumably error objects would then be changed from

      (list ERROR-TYPE DATA1 DATA2 ...)

  to something like

      (record ERROR-TYPE DATA1 DATA2 ...)

  This shouldn't require anything tricky during bootstrapping.
  But in any case we're pretty damn far from this.  I'd estimate that we
  have at least 10 years ahead of us to decide if and how we want to
  make that change.


-- Stefan

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 6:08 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Fri, Dec 29, 2023 at 17:54:30 +0000, João Távora wrote:
> > On Fri, Dec 29, 2023 at 5:43 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > > It's a similar object with a unique type.
>
> > > Similar to what?
>
> > The object that signal() would craft and appear to Lisp is similar to how
> > symbol-with-pos is passed to, say, compiler macros (breaking some of
> > them, last I remember, but that's besides our discussion).
>
> Ah, right!  So you're considering, possibly, a new type of pseudovector,
> or something similar.

Yes, exactly, something of the sort.

João

Re: Attaching context info to an error

[João Távora]

On Fri, Dec 29, 2023 at 6:35 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > That's got no connection with what I wrote.  What I'm asking you to
> > confirm is that you're not contemplating introducing error handling
> > machinery which, during bootstrap, will only work after Lisp code has
> > been loaded.
>
> The idea is:
>
> - In the short term, provide new functions that allow manipulating
>   errors in a way that's independent from their representation
>   (as cons cells).
>   Hopefully this will make the code also more readable in many cases.
>   E.g. (error-resignal ERR) instead of (signal (car ERR) (cdr ERR)).
>
> - In the longer term once enough code has been adapted to use the new
>   functions, we may be able to consider changing the representation of
>   error objects.  Presumably error objects would then be changed from
>
>       (list ERROR-TYPE DATA1 DATA2 ...)
>
>   to something like
>
>       (record ERROR-TYPE DATA1 DATA2 ...)
>
>   This shouldn't require anything tricky during bootstrapping.
>
>   But in any case we're pretty damn far from this.  I'd estimate that we
>   have at least 10 years ahead of us to decide if and how we want to
>   make that change.

I concur.  And maybe, to keep 100% backward compatibility to
current Lisp code, condition-case is AFAIK is the only current
to get at error objects from Lisp, would still get the
good-ol cons  representation for those 10 years, or even forever.

João

Re: Attaching context info to an error

[Stefan Monnier]

> Most importantly, and idea quality apart, who can be doing that
> today, before h-b?  No-one or practically no-one, since it's useless
> because currently the only place user code has access to
> the signal also unwinds the stack, and resignalling will make a new
> cons. Right?

Indeed, until now the identity of an error object can't be "noticed".
The only place other than the `condition-case` where an error can appear
is in `signal-hook-function` and this doesn't receive the error object
(it receives it in two pieces instead 🙁).

> So unless there's a specific use for it today,

Just because we can't think of a use case today doesn't mean there can't
be a good use case.  And yes, I do have some use cases.  Those use cases
may be able to use other approaches as well, but the various options all
have pros and cons.

> I recommend specifically _not_ maintaining 'eq'ness.
> This prevents people from making the job to transition to a non-cons
> representation of errors easier.

You lost me here.

>> We cannot reliably distinguish an error object from random other cons
>> cells, so things like `cl-print-object` are simply not an option: the
>> caller needs to tell us that this is an error object.
>> We could have a `print-error-object`, OTOH.
> Indeed this representation is really bothersome.

Actually, it's not.  We've lived with it quite happily for about 40
years now.  In practice, we really don't do much with error objects
other than re-signal them or pass them to `error-message-string`.

>> If it's only for resignaling purposes, why not have
>> `error-resignal` instead which takes the error object itself?
> This has to exist indeed.  Should be called "condition-resignal"
> though

Emacs's naming convention in this area is a bit murky, but I think it's
leaning towards `error` (e.g. the `error-message-string` function, the
`define-error` macro, the `error-conditions` and `error-message`
properties).
This said, as long as the new functions are nicely placed together in
their own namespace prefix, I'll be happy, so if you prefer
`condition-`, I can go along with that.

> (and the byte-compiler should warn you should use a
> handler-bind instead).

As I mentioned elsewhere, running code inside a `handler-bind` means
running in some arbitrary dynamic context, which can have surprising
effects because in ELisp dynvars influence the execution of a *lot* of
primitives (even `eq` is not immune).  So I don't think I can honestly
claim that re-signaling an error is almost always wrong.  Maybe with
more practice, I'll reach this conclusion, but I'm definitely not
there yet.  So, warning about resignaling seems premature.

>> I don't think we want to do that systematically, but yes (and I'm afraid
>> we'll discover along the way that the representations we use are messy
>> and don't always obey the subtyping suggested by the error hierarchy).
>> It presumably needs to come with a "similar" change on the constructor
>> side, so as long as we keep using (signal ERROR-TYPE ERROR-DATA) where
>> ERROR-DATA is a bare list it's not super pressing.
> I've also seen 0 examples so far where the DATA is searched inside
> for actual properties.

That's my impression as well.
The fact that the "data" part has no clearly defined shape (other than
being a list) means that it's used a bit "without purpose": I think that
in most cases, programmers put data in there only under the expectation
that it'll be useful when displayed in an error message or a backtrace.


        Stefan

Re: Attaching context info to an error

[João Távora]

On Sat, Dec 30, 2023 at 4:29 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Most importantly, and idea quality apart, who can be doing that
> > today, before h-b?  No-one or practically no-one, since it's useless
> > because currently the only place user code has access to
> > the signal also unwinds the stack, and resignalling will make a new
> > cons. Right?
>
> Indeed, until now the identity of an error object can't be "noticed".
> The only place other than the `condition-case` where an error can appear
> is in `signal-hook-function` and this doesn't receive the error object
> (it receives it in two pieces instead 🙁).

Indeed, but that's a good thing IMO.  It tells us something we don't
_have_ to do, which is always good from a language design evolution
point of view.

> Emacs's naming convention in this area is a bit murky, but I think it's
> leaning towards `error` (e.g. the `error-message-string` function, the
> `define-error` macro, the `error-conditions` and `error-message`
> properties).
> This said, as long as the new functions are nicely placed together in
> their own namespace prefix, I'll be happy, so if you prefer
> `condition-`, I can go along with that.

You're right, so I can go along with `error-` just fine.  We can
make a zillion aliases later :-) (or not).

> > (and the byte-compiler should warn you should use a
> > handler-bind instead).
>
> As I mentioned elsewhere, running code inside a `handler-bind` means
> running in some arbitrary dynamic context, which can have surprising
> effects because in ELisp dynvars influence the execution of a *lot* of
> primitives (even `eq` is not immune).  So I don't think I can honestly
> claim that re-signaling an error is almost always wrong.  Maybe with
> more practice, I'll reach this conclusion, but I'm definitely not
> there yet.  So, warning about resignaling seems premature.

Resignalling a different error is fine, it's sometimes needed.
Can be just a different error message.
Resignalling the very same error, without changes, is always a
fairly bad code smell (in handler-bind-capable languages,
in C++ and current Elisp you have no other option).

> The fact that the "data" part has no clearly defined shape (other than
> being a list) means that it's used a bit "without purpose": I think that
> in most cases, programmers put data in there only under the expectation
> that it'll be useful when displayed in an error message or a backtrace.

I'm the exception. :-D  I stashed some info in the jsonrpc-error data in
hopes they would become useful for things other than printing
when handler-bind arrives.  And then there's this <grimaces>

(condition-case err
            (electric-pair--with-syntax string-or-comment
              (scan-sexps (point) (if (> direction 0)
                                      (point-max)
                                    (- (point-max))))
              (funcall at-top-level-or-equivalent-fn))
          (scan-error
           (cond ((or
                   ;; some error happened and it is not of the "ended
                   ;; prematurely" kind...
                   (not (string-match "ends prematurely" (nth 1 err)))
                   ;; ... or we were in a comment and just came out of
                   ;; it.
                   (and string-or-comment
                        (not (nth 8 (syntax-ppss)))))
                  (funcall at-top-level-or-equivalent-fn))
                 (t
                  ;; exit the sexp
                  (goto-char (nth 3 err))
                  (funcall ended-prematurely-fn)))))

Also me :-)  but I really have doubts I'm the only one.