bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Emacs.

Supposing a variable has as a value a lambda function.  On doing C-h v
for this variable, there ensues a useless lack of information about the
lambda.  All one gets told is that it is a lambda function, not where the
source code for it is, or anything useful like that.

A typical example is C-h v search-default-mode, which prints something
like the following:

    Its value is
    #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
    Original value was nil
    Local in buffer *info*<3>; global value is nil

..  When native compilation is not enabled, the printout is only slightly
better.

Merely amongst the customisable options (i.e. not ordinary defvars and
defconsts) there are approximately 63 places where these options get set
to lambda functions.  The following list lists the file and line number
where it's done, and the customisable option it gets done to:

../autorevert.el:471 before-save-hook
../emacs-lisp/pp.el:320 temp-buffer-show-function
../emulation/viper.el:950 eshell-mode-hook
../erc/erc-pcomplete.el:99 pcomplete-default-completion-function
../eshell/em-cmpl.el:191 eshell-command-completion-function
../eshell/em-cmpl.el:202 eshell-default-completion-function
../eshell/em-cmpl.el:280 eshell-mode-hook
../eshell/em-prompt.el:50 eshell-prompt-function
../eshell/em-smart.el:187 eshell-post-command-hook
../gnus/gnus-art.el:3077 gnus-exit-gnus-hook
../gnus/gnus-delay.el:150 message-send-hook
../gnus/gnus-diary.el:213 gnus-summary-menu-hook
../gnus/gnus-diary.el:370 nndiary-request-accept-article-functions
../gnus/gnus-draft.el:274 message-post-method
../gnus/gnus-msg.el:1145 message-reply-to-function
../gnus/gnus-msg.el:387 message-header-setup-hook
../gnus/gnus-msg.el:392 message-mode-hook
../gnus/gnus-registry.el:1000 gnus-summary-menu-hook
../gnus/gnus-sum.el:1321 gnus-alter-articles-to-read-function
../gnus/gnus-util.el:1205 iswitchb-make-buflist-hook
../gnus/message.el:3157 facemenu-add-face-function
../gnus/message.el:701 message-cite-level-function
../gnus/nnmairix.el:286 nnmairix-widget-select-window-function
../help.el:1946 temp-buffer-max-height
../help.el:1963 temp-buffer-max-width
../isearch.el:4685 search-default-mode
../locate.el:670 locate-make-command-line
../mail/emacsbug.el:466 message-sent-hook
../mail/emacsbug.el:540 message-send-hook
../mail/feedmail.el:1400 feedmail-queue-runner-mode-setter
../mail/feedmail.el:1436 feedmail-queue-runner-message-sender
../mail/feedmail.el:1457 feedmail-queue-runner-cleaner-upper
../net/mairix.el:176 mairix-widget-select-window-function
../obsolete/gulp.el:103 mail-setup-hook
../org/ob-core.el:1524 org-mode-hook
../org/ob-latex.el:101 org-babel-latex-end-env
../org/ob-latex.el:79 org-babel-latex-preamble
../org/ob-latex.el:90 org-babel-latex-begin-env
../org/ob-python.el:208 python-shell-first-prompt-hook
../org/ol-bibtex.el:268 org-bibtex-headline-format-function
../org/org-compat.el:1272 imenu-after-jump-hook
../org/org-compat.el:1276 org-mode-hook
../org/org-compat.el:1339 speedbar-visiting-tag-hook
../org/org-crypt.el:317 org-mode-hook
../org/org-ctags.el:196 org-mode-hook
../org/org.el:15220 after-save-hook
../org/org.el:18651 occur-mode-find-occurrence-hook
../org/org-mouse.el:866 org-mode-hook
../org/ox-ascii.el:378 org-ascii-format-drawer-function
../org/ox-html.el:657 org-html-format-drawer-function
../org/ox-latex.el:888 org-latex-format-drawer-function
../org/ox-odt.el:635 org-odt-format-drawer-function
../org/ox-texinfo.el:334 org-texinfo-format-drawer-function
../pcomplete.el:305 pcomplete-command-completion-function
../pcomplete.el:315 pcomplete-default-completion-function
../play/gametree.el:118 gametree-make-heading-function
../progmodes/cperl-mode.el:1848 add-log-current-defun-function
../progmodes/octave.el:607 fill-nobreak-predicate
../progmodes/ps-mode.el:108 ps-mode-print-function
../vc/add-log.el:1129 fill-nobreak-predicate
../vc/emerge.el:2086 temp-buffer-show-function
../vc/smerge-mode.el:1358 ediff-quit-hook
../window.el:6662 pop-up-frame-function

..  Additionally there are about 29 instances in the doc strings where
users are urged to use lambda functions (rather than defuns) for this
purpose.  They are at:

../calendar/calendar.el:395 calendar-move-hook
../calendar/cal-tex.el:154 cal-tex-hook
../comint.el:189 comint-mode-hook
../dired-loaddefs.el:646 dired-mode-hook
../dired-x.el:152 dired-mode-hook
../erc/erc-log.el:218 erc-enable-logging
../erc/erc-page.el:55 erc-page-function
../files.el:466 auto-save-visited-predicate
../font-lock.el:704 c-mode-hook
../gnus/gnus-sum.el:1044 gnus-select-group-hook
../gnus/nnmail.el:179 nnmail-expiry-wait-function
../gnus/nnmail.el:286 nnmail-read-incoming-hook
../gnus/nnmail.el:299 nnmail-read-incoming-hook
../icomplete.el:154 icomplete-minibuffer-setup-hook
../ido.el:890 ido-minibuffer-setup-hook
../ielm.el:63 ielm-mode-hook
../ldefs-boot.el:29820 sql-mode-hook
../ldefs-boot.el:5069 c-mode-hook
../loaddefs.el:29950 sql-mode-hook
../loaddefs.el:5069 c-mode-hook
../obsolete/nnir.el:415 nnir-notmuch-filter-group-names-function
../org/org-keys.el:688 org-use-speed-commands
../org/ox-html.el:968 org-html-table-row-open-tag
../progmodes/compile.el:885 c-mode-hook
../progmodes/prog-mode.el:296 emacs-lisp-mode-hook
../progmodes/sql.el:4134 sql-mode-hook
../progmodes/sql.el:4256 sql-interactive-mode-hook
../textmodes/tildify.el:156 tildify-foreach-region-function
../transient.el:278 transient-substitute-key-function

..  Furthermore there are, perhaps, around 150 such settings of ordinary
defvars or defconsts to lambda functions.

#########################################################################

I see two ways to fix this bug:
(i) Systematically to replace each of these lambda functions by a named
  defun.
(ii) To adopt git branch feature/named-lambdas, which attaches to every
lambda function the name of the defun (or defvar) in which it was
defined.  This extra information could then be printed out for C-h v,
something like this:

    Its value is
    {isearch-fold-quotes-mode} #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
    Original value was nil
    Local in buffer *info*<3>; global value is nil

, thus enabling the user quickly to locate the source of the lambda form
in isearch.el.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Emacs.
>
> Supposing a variable has as a value a lambda function.  On doing C-h v
> for this variable, there ensues a useless lack of information about the
> lambda.  All one gets told is that it is a lambda function, not where the
> source code for it is, or anything useful like that.
>
> A typical example is C-h v search-default-mode, which prints something
> like the following:
>
>     Its value is
>     #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
>     Original value was nil
>     Local in buffer *info*<3>; global value is nil
>
> ..  When native compilation is not enabled, the printout is only slightly
> better.
>
> Merely amongst the customisable options (i.e. not ordinary defvars and
> defconsts) there are approximately 63 places where these options get set
> to lambda functions.  The following list lists the file and line number
> where it's done, and the customisable option it gets done to:
>
> ../autorevert.el:471 before-save-hook
> ../emacs-lisp/pp.el:320 temp-buffer-show-function
> ../emulation/viper.el:950 eshell-mode-hook
> ../erc/erc-pcomplete.el:99 pcomplete-default-completion-function
> ../eshell/em-cmpl.el:191 eshell-command-completion-function
> ../eshell/em-cmpl.el:202 eshell-default-completion-function
> ../eshell/em-cmpl.el:280 eshell-mode-hook
> ../eshell/em-prompt.el:50 eshell-prompt-function
> ../eshell/em-smart.el:187 eshell-post-command-hook
> ../gnus/gnus-art.el:3077 gnus-exit-gnus-hook
> ../gnus/gnus-delay.el:150 message-send-hook
> ../gnus/gnus-diary.el:213 gnus-summary-menu-hook
> ../gnus/gnus-diary.el:370 nndiary-request-accept-article-functions
> ../gnus/gnus-draft.el:274 message-post-method
> ../gnus/gnus-msg.el:1145 message-reply-to-function
> ../gnus/gnus-msg.el:387 message-header-setup-hook
> ../gnus/gnus-msg.el:392 message-mode-hook
> ../gnus/gnus-registry.el:1000 gnus-summary-menu-hook
> ../gnus/gnus-sum.el:1321 gnus-alter-articles-to-read-function
> ../gnus/gnus-util.el:1205 iswitchb-make-buflist-hook
> ../gnus/message.el:3157 facemenu-add-face-function
> ../gnus/message.el:701 message-cite-level-function
> ../gnus/nnmairix.el:286 nnmairix-widget-select-window-function
> ../help.el:1946 temp-buffer-max-height
> ../help.el:1963 temp-buffer-max-width
> ../isearch.el:4685 search-default-mode
> ../locate.el:670 locate-make-command-line
> ../mail/emacsbug.el:466 message-sent-hook
> ../mail/emacsbug.el:540 message-send-hook
> ../mail/feedmail.el:1400 feedmail-queue-runner-mode-setter
> ../mail/feedmail.el:1436 feedmail-queue-runner-message-sender
> ../mail/feedmail.el:1457 feedmail-queue-runner-cleaner-upper
> ../net/mairix.el:176 mairix-widget-select-window-function
> ../obsolete/gulp.el:103 mail-setup-hook
> ../org/ob-core.el:1524 org-mode-hook
> ../org/ob-latex.el:101 org-babel-latex-end-env
> ../org/ob-latex.el:79 org-babel-latex-preamble
> ../org/ob-latex.el:90 org-babel-latex-begin-env
> ../org/ob-python.el:208 python-shell-first-prompt-hook
> ../org/ol-bibtex.el:268 org-bibtex-headline-format-function
> ../org/org-compat.el:1272 imenu-after-jump-hook
> ../org/org-compat.el:1276 org-mode-hook
> ../org/org-compat.el:1339 speedbar-visiting-tag-hook
> ../org/org-crypt.el:317 org-mode-hook
> ../org/org-ctags.el:196 org-mode-hook
> ../org/org.el:15220 after-save-hook
> ../org/org.el:18651 occur-mode-find-occurrence-hook
> ../org/org-mouse.el:866 org-mode-hook
> ../org/ox-ascii.el:378 org-ascii-format-drawer-function
> ../org/ox-html.el:657 org-html-format-drawer-function
> ../org/ox-latex.el:888 org-latex-format-drawer-function
> ../org/ox-odt.el:635 org-odt-format-drawer-function
> ../org/ox-texinfo.el:334 org-texinfo-format-drawer-function
> ../pcomplete.el:305 pcomplete-command-completion-function
> ../pcomplete.el:315 pcomplete-default-completion-function
> ../play/gametree.el:118 gametree-make-heading-function
> ../progmodes/cperl-mode.el:1848 add-log-current-defun-function
> ../progmodes/octave.el:607 fill-nobreak-predicate
> ../progmodes/ps-mode.el:108 ps-mode-print-function
> ../vc/add-log.el:1129 fill-nobreak-predicate
> ../vc/emerge.el:2086 temp-buffer-show-function
> ../vc/smerge-mode.el:1358 ediff-quit-hook
> ../window.el:6662 pop-up-frame-function
>
> ..  Additionally there are about 29 instances in the doc strings where
> users are urged to use lambda functions (rather than defuns) for this
> purpose.  They are at:
>
> ../calendar/calendar.el:395 calendar-move-hook
> ../calendar/cal-tex.el:154 cal-tex-hook
> ../comint.el:189 comint-mode-hook
> ../dired-loaddefs.el:646 dired-mode-hook
> ../dired-x.el:152 dired-mode-hook
> ../erc/erc-log.el:218 erc-enable-logging
> ../erc/erc-page.el:55 erc-page-function
> ../files.el:466 auto-save-visited-predicate
> ../font-lock.el:704 c-mode-hook
> ../gnus/gnus-sum.el:1044 gnus-select-group-hook
> ../gnus/nnmail.el:179 nnmail-expiry-wait-function
> ../gnus/nnmail.el:286 nnmail-read-incoming-hook
> ../gnus/nnmail.el:299 nnmail-read-incoming-hook
> ../icomplete.el:154 icomplete-minibuffer-setup-hook
> ../ido.el:890 ido-minibuffer-setup-hook
> ../ielm.el:63 ielm-mode-hook
> ../ldefs-boot.el:29820 sql-mode-hook
> ../ldefs-boot.el:5069 c-mode-hook
> ../loaddefs.el:29950 sql-mode-hook
> ../loaddefs.el:5069 c-mode-hook
> ../obsolete/nnir.el:415 nnir-notmuch-filter-group-names-function
> ../org/org-keys.el:688 org-use-speed-commands
> ../org/ox-html.el:968 org-html-table-row-open-tag
> ../progmodes/compile.el:885 c-mode-hook
> ../progmodes/prog-mode.el:296 emacs-lisp-mode-hook
> ../progmodes/sql.el:4134 sql-mode-hook
> ../progmodes/sql.el:4256 sql-interactive-mode-hook
> ../textmodes/tildify.el:156 tildify-foreach-region-function
> ../transient.el:278 transient-substitute-key-function
>
> ..  Furthermore there are, perhaps, around 150 such settings of ordinary
> defvars or defconsts to lambda functions.
>
> #########################################################################
>
> I see two ways to fix this bug:
> (i) Systematically to replace each of these lambda functions by a named
>   defun.
> (ii) To adopt git branch feature/named-lambdas, which attaches to every
> lambda function the name of the defun (or defvar) in which it was
> defined.  This extra information could then be printed out for C-h v,
> something like this:
>
>     Its value is
>     {isearch-fold-quotes-mode} #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
>     Original value was nil
>     Local in buffer *info*<3>; global value is nil
>
> , thus enabling the user quickly to locate the source of the lambda form
> in isearch.el.

Hi Alan,

what (if any) do you think are the downsides of option ii?

Thanks

  Andrea

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Andrea.

On Wed, Oct 25, 2023 at 16:53:38 -0400, Andrea Corallo wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Emacs.

> > Supposing a variable has as a value a lambda function.  On doing C-h v
> > for this variable, there ensues a useless lack of information about the
> > lambda.  All one gets told is that it is a lambda function, not where the
> > source code for it is, or anything useful like that.

[ .... ]

> > #########################################################################

> > I see two ways to fix this bug:
> > (i) Systematically to replace each of these lambda functions by a named
> >   defun.
> > (ii) To adopt git branch feature/named-lambdas, which attaches to every
> > lambda function the name of the defun (or defvar) in which it was
> > defined.  This extra information could then be printed out for C-h v,
> > something like this:

> >     Its value is
> >     {isearch-fold-quotes-mode} #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
> >     Original value was nil
> >     Local in buffer *info*<3>; global value is nil

> > , thus enabling the user quickly to locate the source of the lambda form
> > in isearch.el.

> Hi Alan,

> what (if any) do you think are the downsides of option ii?

Currently feature/named-lambdas is mostly working, not fully working.
Probably the most pressing of these deficiencies is that native compiled
lambda functions don't yet get their defining symbol set.

There have been substantial amendments compared with the master branch,
and it has not been tested much by anybody but me.

Assuming it can be brought to a fully working state, there would be an
extra symbol in each function definition which would make Emacs
marginally bigger.

> Thanks

>   Andrea

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Andrea.

On Fri, Oct 27, 2023 at 11:35:16 +0000, Alan Mackenzie wrote:

> On Wed, Oct 25, 2023 at 16:53:38 -0400, Andrea Corallo wrote:
> > Alan Mackenzie <acm@muc.de> writes:

[ .... ]

> > > I see two ways to fix this bug:
> > > (i) Systematically to replace each of these lambda functions by a named
> > >   defun.
> > > (ii) To adopt git branch feature/named-lambdas, which attaches to every
> > > lambda function the name of the defun (or defvar) in which it was
> > > defined.  This extra information could then be printed out for C-h v,
> > > something like this:

> > >     Its value is
> > >     {isearch-fold-quotes-mode} #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_109>
> > >     Original value was nil
> > >     Local in buffer *info*<3>; global value is nil

> > > , thus enabling the user quickly to locate the source of the lambda form
> > > in isearch.el.

> > Hi Alan,

> > what (if any) do you think are the downsides of option ii?

> Currently feature/named-lambdas is mostly working, not fully working.
> Probably the most pressing of these deficiencies is that native compiled
> lambda functions don't yet get their defining symbol set.

They do now.  In feature/named-lambdas, I now see C-h v
search-default-mode giving the above display when invoked from an info
buffer.

> There have been substantial amendments compared with the master branch,
> and it has not been tested much by anybody but me.

> Assuming it can be brought to a fully working state, there would be an
> extra symbol in each function definition which would make Emacs
> marginally bigger.

> > Thanks

> >   Andrea

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Kangas]

Alan Mackenzie <acm@muc.de> writes:

> Currently feature/named-lambdas is mostly working, not fully working.
> Probably the most pressing of these deficiencies is that native compiled
> lambda functions don't yet get their defining symbol set.
>
> There have been substantial amendments compared with the master branch,
> and it has not been tested much by anybody but me.
>
> Assuming it can be brought to a fully working state, there would be an
> extra symbol in each function definition which would make Emacs
> marginally bigger.

Does it have any performance implications?

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sat, Oct 28, 2023 at 08:04:00 -0700, Stefan Kangas wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Currently feature/named-lambdas is mostly working, not fully working.
> > Probably the most pressing of these deficiencies is that native compiled
> > lambda functions don't yet get their defining symbol set.

As I've said, this is now fixed.

> > There have been substantial amendments compared with the master branch,
> > and it has not been tested much by anybody but me.

> > Assuming it can be brought to a fully working state, there would be an
> > extra symbol in each function definition which would make Emacs
> > marginally bigger.

> Does it have any performance implications?

Yes, but very little.

If I scroll through xdisp.c, displaying all the time, in C Mode:
(i) master:                  18.850s, 18.767s, 18.747s.
(ii) feature/named-lambdas:  18.630s, 18.738s, 18.782s.

If I likewise scroll through osprey_reg_map_macro.h (a 4 MB .h file
specifying some hardware registers), in c-ts-mode:
(i) master:                  10.157s, 10.203s, 10.169s
(ii) feature/named-lambdas:  10.207s, 10.219s, 10.257s

..  Not a very scientific timing, not done from emacs -Q, but a fair
test.  It would seem any slowdown is less than 1%.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>, acm@muc.de
> Date: Sat, 28 Oct 2023 15:59:26 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Stefan.
> 
> On Sat, Oct 28, 2023 at 08:04:00 -0700, Stefan Kangas wrote:
> > Alan Mackenzie <acm@muc.de> writes:
> 
> > > Currently feature/named-lambdas is mostly working, not fully working.
> > > Probably the most pressing of these deficiencies is that native compiled
> > > lambda functions don't yet get their defining symbol set.
> 
> As I've said, this is now fixed.

Above, you are talking about "the most pressing of these
deficiencies", which is now fixed.  What other deficiencies or issues
are you aware of?  Please provide a full list, so we could make an
informed decision whether the feature branch is ready to be landed.

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 28, 2023 at 19:26:25 +0300, Eli Zaretskii wrote:
> > Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
> >  Stefan Monnier <monnier@iro.umontreal.ca>, acm@muc.de
> > Date: Sat, 28 Oct 2023 15:59:26 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > Hello, Stefan.

> > On Sat, Oct 28, 2023 at 08:04:00 -0700, Stefan Kangas wrote:
> > > Alan Mackenzie <acm@muc.de> writes:

> > > > Currently feature/named-lambdas is mostly working, not fully working.
> > > > Probably the most pressing of these deficiencies is that native compiled
> > > > lambda functions don't yet get their defining symbol set.

> > As I've said, this is now fixed.

> Above, you are talking about "the most pressing of these
> deficiencies", which is now fixed.  What other deficiencies or issues
> are you aware of?

I'm not aware of anywhere else where the code doesn't work.  I'm not
sure whether or not there are any other defining macros (things like
define-minor-mode) which need amendment.  I don't think any user
documentation is needed.

> Please provide a full list, so we could make an informed decision
> whether the feature branch is ready to be landed.

There are currently no ert tests for the new feature.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > Currently feature/named-lambdas is mostly working, not fully working.

Didn't know about this feature.

I like the idea of keeping better track of the origin of lambda
expressions (tho, admittedly, the same problem occurs with other kinds
of data: I have several times been faced with a keymap or a char-table,
wondering where the heel it came from).

I took a look at

    git log -p main..origin/feature/named-lambdas

but that's 127kB, so ... could briefly describe the overall design
(IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
ELisp users)?

Also, what other approaches have you considered/tried and what were the
problems you've encountered, if any?


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sat, Oct 28, 2023 at 13:17:13 -0400, Stefan Monnier wrote:
> >> > Currently feature/named-lambdas is mostly working, not fully working.

> Didn't know about this feature.

It's been hanging around the savannah repository since July.

> I like the idea of keeping better track of the origin of lambda
> expressions (tho, admittedly, the same problem occurs with other kinds
> of data: I have several times been faced with a keymap or a char-table,
> wondering where the heel it came from).

Maybe something similar might be possible for those type of objects.

> I took a look at

>     git log -p main..origin/feature/named-lambdas

> but that's 127kB, so ... could [you] briefly describe the overall design
> (IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
> ELisp users)?

Certainly.  Each lambda expression has (usually) a defun within which it
is defined.  Sometimes it's in a defvar, or defcustom.  That
@dfn{defining symbol} is recorded in the lambda form in one of three
ways:
(i) For a cons form, it's (cadr form), a new field inserted between the
  symbol `lambda' and the argument list.
(ii) For a byte-compiled form, it's (aref form 5), this new field going
  after the doc string and before any interactive form in the compiled
  form.
(iii) For a native-compiled subr it's (subr-native-defining-symbol
subr), a function defined in data.c.  It accesses a new field in struct
Lisp_Subr called defining_symbol.

There are lots of detailed changes in eval.c and bytecomp.el (and
friends).  Also the macro `lambda' in subr.el has been amended to insert
the current global defining-symbol if there isn't already a non-nil
symbol in that position.  cl-print-object/compiled-function has been
amended to print the defining-symbol, and there is a new
cl-print-object/subr which does the same.

The intention is that compiled objects from earlier Emacsen can still be
loaded and run by feature/named-lambdas, just without the defining
symbols (which will appear to be nil).

> Also, what other approaches have you considered/tried and what were the
> problems you've encountered, if any?

feature/named-lambdas was originally intended for use in backtraces.

For the current bug, I've considered individually replacing each lambda
with a named defun, so that C-h v will show that name rather than an
unhelpful byte/native compiled anonymous function.  That would be a lot
of work - my scripting found 63 defcustoms set to lambdas, 29 uses in
doc strings, and 215 suspicious occurrences with ordinary variables
(quite a few of which will be harmless).  Amending all these (I guess
around 200 lambdas) would probably be too much work.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
>  Stefan Kangas <stefankangas@gmail.com>, acm@muc.de
> Date: Sat, 28 Oct 2023 18:17:04 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> > but that's 127kB, so ... could [you] briefly describe the overall design
> > (IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
> > ELisp users)?
> 
> Certainly.  Each lambda expression has (usually) a defun within which it
> is defined.  Sometimes it's in a defvar, or defcustom.  That
> @dfn{defining symbol} is recorded in the lambda form in one of three
> ways:
> (i) For a cons form, it's (cadr form), a new field inserted between the
>   symbol `lambda' and the argument list.
> (ii) For a byte-compiled form, it's (aref form 5), this new field going
>   after the doc string and before any interactive form in the compiled
>   form.
> (iii) For a native-compiled subr it's (subr-native-defining-symbol
> subr), a function defined in data.c.  It accesses a new field in struct
> Lisp_Subr called defining_symbol.
> 
> There are lots of detailed changes in eval.c and bytecomp.el (and
> friends).  Also the macro `lambda' in subr.el has been amended to insert
> the current global defining-symbol if there isn't already a non-nil
> symbol in that position.  cl-print-object/compiled-function has been
> amended to print the defining-symbol, and there is a new
> cl-print-object/subr which does the same.
> 
> The intention is that compiled objects from earlier Emacsen can still be
> loaded and run by feature/named-lambdas, just without the defining
> symbols (which will appear to be nil).
> 
> > Also, what other approaches have you considered/tried and what were the
> > problems you've encountered, if any?
> 
> feature/named-lambdas was originally intended for use in backtraces.
> 
> For the current bug, I've considered individually replacing each lambda
> with a named defun, so that C-h v will show that name rather than an
> unhelpful byte/native compiled anonymous function.  That would be a lot
> of work - my scripting found 63 defcustoms set to lambdas, 29 uses in
> doc strings, and 215 suspicious occurrences with ordinary variables
> (quite a few of which will be harmless).  Amending all these (I guess
> around 200 lambdas) would probably be too much work.

Thanks.  However, now I'm confused: what exactly does this feature
give us, if it doesn't handle all the lambdas in the Emacs tree and
(AFAIU) will not affect lambdas in third-party packages?  Which
lambdas will now have a defining symbol, and which will not?

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 28, 2023 at 21:38:36 +0300, Eli Zaretskii wrote:
> > Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
> >  Stefan Kangas <stefankangas@gmail.com>, acm@muc.de
> > Date: Sat, 28 Oct 2023 18:17:04 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > > but that's 127kB, so ... could [you] briefly describe the overall design
> > > (IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
> > > ELisp users)?

> > Certainly.  Each lambda expression has (usually) a defun within which it
> > is defined.  Sometimes it's in a defvar, or defcustom.  That
> > @dfn{defining symbol} is recorded in the lambda form in one of three
> > ways:
> > (i) For a cons form, it's (cadr form), a new field inserted between the
> >   symbol `lambda' and the argument list.
> > (ii) For a byte-compiled form, it's (aref form 5), this new field going
> >   after the doc string and before any interactive form in the compiled
> >   form.
> > (iii) For a native-compiled subr it's (subr-native-defining-symbol
> > subr), a function defined in data.c.  It accesses a new field in struct
> > Lisp_Subr called defining_symbol.

> > There are lots of detailed changes in eval.c and bytecomp.el (and
> > friends).  Also the macro `lambda' in subr.el has been amended to insert
> > the current global defining-symbol if there isn't already a non-nil
> > symbol in that position.  cl-print-object/compiled-function has been
> > amended to print the defining-symbol, and there is a new
> > cl-print-object/subr which does the same.

> > The intention is that compiled objects from earlier Emacsen can still be
> > loaded and run by feature/named-lambdas, just without the defining
> > symbols (which will appear to be nil).

> > > Also, what other approaches have you considered/tried and what were the
> > > problems you've encountered, if any?

> > feature/named-lambdas was originally intended for use in backtraces.

> > For the current bug, I've considered individually replacing each lambda
> > with a named defun, so that C-h v will show that name rather than an
> > unhelpful byte/native compiled anonymous function.  That would be a lot
> > of work - my scripting found 63 defcustoms set to lambdas, 29 uses in
> > doc strings, and 215 suspicious occurrences with ordinary variables
> > (quite a few of which will be harmless).  Amending all these (I guess
> > around 200 lambdas) would probably be too much work.

> Thanks.  However, now I'm confused: what exactly does this feature
> give us, if it doesn't handle all the lambdas in the Emacs tree and
> (AFAIU) will not affect lambdas in third-party packages?  Which
> lambdas will now have a defining symbol, and which will not?

It will handle all the lambdas in the Emacs tree.  Each of them will get
a defining-symbol set.  Also third-party packages with lambdas will also
get defining-symbols, provided they are compiled with
feature/named-lambdas.  It's just that .elc's/.eln's built with, say,
Emacs-29 will continue to be loadable and runnable, but obviously without
the defining symbol facility for those lambdas.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I like the idea of keeping better track of the origin of lambda
>> expressions (tho, admittedly, the same problem occurs with other kinds
>> of data: I have several times been faced with a keymap or a char-table,
>> wondering where the heel it came from).
> Maybe something similar might be possible for those type of objects.

[ Hmm... food for thought...  ]

>> I took a look at
>
>>     git log -p main..origin/feature/named-lambdas
>
>> but that's 127kB, so ... could [you] briefly describe the overall design
>> (IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
>> ELisp users)?
>
> Certainly.  Each lambda expression has (usually) a defun within which it
> is defined.  Sometimes it's in a defvar, or defcustom.  That
> @dfn{defining symbol} is recorded in the lambda form in one of three
> ways:
> (i) For a cons form, it's (cadr form), a new field inserted between the
>   symbol `lambda' and the argument list.
> (ii) For a byte-compiled form, it's (aref form 5), this new field going
>   after the doc string and before any interactive form in the compiled
>   form.

These two change the visible representation of functions, so I wouldn't
be surprised if they break some odd hack out there.

For (i), do we actually care enough to keep that info (IME the
functions are usually compiled, and if they're not the code itself
usually makes it fairly easy to find the code's origin)?

For (ii), why did you choose slot 5, thus moving the interactive-form slot?

> There are lots of detailed changes in eval.c and bytecomp.el (and
> friends).  Also the macro `lambda' in subr.el has been amended to insert
> the current global defining-symbol if there isn't already a non-nil
> symbol in that position.

So the source code can manually provide this extra symbol in `lambda`?
Where have you found that useful?

>> Also, what other approaches have you considered/tried and what were the
>> problems you've encountered, if any?
> feature/named-lambdas was originally intended for use in backtraces.
> For the current bug, I've considered individually replacing each lambda
> with a named defun, so that C-h v will show that name rather than an

Hmm... as a user, rather than the enclosing definition I think I'd
expect to get some kind of source information like FILE+LINE+COL.

If we give up on keeping that extra info in interpreted functions, the
FILE+LINE+COL information is readily available to the byte-compiler,
thanks to that famous Alan guy.

Actually, the bytecode functions can already keep the FILE info (in
the form of the (FILE . OFFSET) cons cell in the docstring slot of the
bytecode object), we just opt never to expose that FILE info to the user
(we only use it to fetch the actual docstring).
[ Tho, admittedly, it's also that the byte-compiler only uses such
  (FILE . OFFSET) for named functions, but it wouldn't be hard to change.  ]

As for how/where to store the LINE+COL, I guess (aref form 5) is still
an option, although, maybe we should store that info alongside the
docstring, like we already do for the arglist.

The way we do it for the arglist is hideous and a pain in the rear (I
blame the maintainers who accepted my patch back then), so it would be
an opportunity to "do it right" this time.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sat, Oct 28, 2023 at 15:13:16 -0400, Stefan Monnier wrote:
> >> I like the idea of keeping better track of the origin of lambda
> >> expressions (tho, admittedly, the same problem occurs with other kinds
> >> of data: I have several times been faced with a keymap or a char-table,
> >> wondering where the heel it came from).
> > Maybe something similar might be possible for those type of objects.

> [ Hmm... food for thought...  ]

> >> I took a look at

> >>     git log -p main..origin/feature/named-lambdas

> >> but that's 127kB, so ... could [you] briefly describe the overall design
> >> (IOW, how it's seen by ELisp programmers, byte-compiler hackers, and
> >> ELisp users)?

> > Certainly.  Each lambda expression has (usually) a defun within which it
> > is defined.  Sometimes it's in a defvar, or defcustom.  That
> > @dfn{defining symbol} is recorded in the lambda form in one of three
> > ways:
> > (i) For a cons form, it's (cadr form), a new field inserted between the
> >   symbol `lambda' and the argument list.
> > (ii) For a byte-compiled form, it's (aref form 5), this new field going
> >   after the doc string and before any interactive form in the compiled
> >   form.

> These two change the visible representation of functions, so I wouldn't
> be surprised if they break some odd hack out there.

There were a few in the test suite I had to amend.

> For (i), do we actually care enough to keep that info (IME the
> functions are usually compiled, and if they're not the code itself
> usually makes it fairly easy to find the code's origin)?

I considered that for some time, and in the end thought that on grounds
of consistency it would be better to put the defining symbol into the
list form as well.

> For (ii), why did you choose slot 5, thus moving the interactive-form
> slot?

Because an interactive form of nil means "(interactive)" with no
arguments.  Only an absent (aref form 6) means there isn't an interactive
form.  So it had to stay as the last slot.

> > There are lots of detailed changes in eval.c and bytecomp.el (and
> > friends).  Also the macro `lambda' in subr.el has been amended to insert
> > the current global defining-symbol if there isn't already a non-nil
> > symbol in that position.

> So the source code can manually provide this extra symbol in `lambda`?
> Where have you found that useful?

The first few functions in debug-early.el and byte-run.el, for a start.
There, there is not yet a defmacro macro.  Also in
comp-trampoline-compile, where there doesn't seem to be any other way to
get the name of the primitive into the defining symbol.  I think there'll
be quite a few other places, too.

> >> Also, what other approaches have you considered/tried and what were the
> >> problems you've encountered, if any?
> > feature/named-lambdas was originally intended for use in backtraces.
> > For the current bug, I've considered individually replacing each lambda
> > with a named defun, so that C-h v will show that name rather than an

> Hmm... as a user, rather than the enclosing definition I think I'd
> expect to get some kind of source information like FILE+LINE+COL.

Possibly: either is a good deal better than what we have at the moment.

> If we give up on keeping that extra info in interpreted functions, the
> FILE+LINE+COL information is readily available to the byte-compiler,
> thanks to that famous Alan guy.

:-)

> Actually, the bytecode functions can already keep the FILE info (in
> the form of the (FILE . OFFSET) cons cell in the docstring slot of the
> bytecode object), we just opt never to expose that FILE info to the user
> (we only use it to fetch the actual docstring).

That's the offset in the .elc file, though, not the source.

> [ Tho, admittedly, it's also that the byte-compiler only uses such
>   (FILE . OFFSET) for named functions, but it wouldn't be hard to change.  ]

> As for how/where to store the LINE+COL, I guess (aref form 5) is still
> an option, although, maybe we should store that info alongside the
> docstring, like we already do for the arglist.

Well, the doc string is at (aref form 4).  :-)

> The way we do it for the arglist is hideous and a pain in the rear (I
> blame the maintainers who accepted my patch back then), so it would be
> an opportunity to "do it right" this time.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > There are lots of detailed changes in eval.c and bytecomp.el (and
>> > friends).  Also the macro `lambda' in subr.el has been amended to insert
>> > the current global defining-symbol if there isn't already a non-nil
>> > symbol in that position.
>> So the source code can manually provide this extra symbol in `lambda`?
>> Where have you found that useful?
> The first few functions in debug-early.el and byte-run.el, for a start.
> There, there is not yet a defmacro macro.  Also in
> comp-trampoline-compile, where there doesn't seem to be any other way to
> get the name of the primitive into the defining symbol.  I think there'll
> be quite a few other places, too.

Hmm... it seems like keeping the LINE+COL info rather than
a surrounding definition would be better in this regard since it wouldn't
require any changes at all to the actual lambdas, regardless where
they are.

>> Hmm... as a user, rather than the enclosing definition I think I'd
>> expect to get some kind of source information like FILE+LINE+COL.
> Possibly: either is a good deal better than what we have at the moment.

The notion of "surrounding definition" is less precise (there can be
several lambdas inside a given definition, there can be lambdas outside
of any definition, ...).

Furthermore, that info is already available to the bytecompiler, so it's
actually easier to keep that info.

>> Actually, the bytecode functions can already keep the FILE info (in
>> the form of the (FILE . OFFSET) cons cell in the docstring slot of the
>> bytecode object), we just opt never to expose that FILE info to the user
>> (we only use it to fetch the actual docstring).
> That's the offset in the .elc file, though, not the source.

Yup.  [ Tho now that you make me think about it, if we want to be cheap,
we could go to that OFFSET position and look around for a `^(defalias`
and that might be sufficient to give us "the surrounding definition" in
many cases :-)  ]

>> [ Tho, admittedly, it's also that the byte-compiler only uses such
>>   (FILE . OFFSET) for named functions, but it wouldn't be hard to change.  ]
>> As for how/where to store the LINE+COL, I guess (aref form 5) is still
>> an option, although, maybe we should store that info alongside the
>> docstring, like we already do for the arglist.
> Well, the doc string is at (aref form 4).  :-)

What I meant is that the docstring is in the `.elc` file.  So the
bytecode object would be literally unchanged, it's just that the
bytes found at OFFSET inside FILE would be extended to hold:
- the docstring
- the human-readable arglist
- the LINE+COL info


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 29, 2023 at 00:14:24 -0400, Stefan Monnier wrote:
> >> > There are lots of detailed changes in eval.c and bytecomp.el (and
> >> > friends).  Also the macro `lambda' in subr.el has been amended to insert
> >> > the current global defining-symbol if there isn't already a non-nil
> >> > symbol in that position.
> >> So the source code can manually provide this extra symbol in `lambda`?
> >> Where have you found that useful?
> > The first few functions in debug-early.el and byte-run.el, for a start.
> > There, there is not yet a defmacro macro.  Also in
> > comp-trampoline-compile, where there doesn't seem to be any other way to
> > get the name of the primitive into the defining symbol.  I think there'll
> > be quite a few other places, too.

> Hmm... it seems like keeping the LINE+COL info rather than
> a surrounding definition would be better in this regard since it wouldn't
> require any changes at all to the actual lambdas, regardless where
> they are.

I don't agree with this.  Whether we add FILE+LINE+COL info is to a
large degree a separate question from that of adding defining symbol
info.

If you don't add information to the actual lambda, then that information
won't be available when that lambda turns up in a backtrace or as a
value in C-h v.  Surely you'd have to keep FILE+LINE+COL in the lambda
itself, somehow.

That also assumes that the source file will be available and unchanged
when it's needed.

It will also stress the user somewhat, since it will force her to look
up a file location to get any idea of what is failing.  A defining
symbol is much more immediately useful, even if it may not be 100%
dependable because of the issues you raise below.

> >> Hmm... as a user, rather than the enclosing definition I think I'd
> >> expect to get some kind of source information like FILE+LINE+COL.
> > Possibly: either is a good deal better than what we have at the moment.

> The notion of "surrounding definition" is less precise (there can be
> several lambdas inside a given definition, there can be lambdas outside
> of any definition, ...).

Yes, there can be several lambdas in a function, though I don't think
this is common.  Certainly it hasn't given me any problems in use, so
far.  But lambdas outside of definitions?  I doubt very much there are
any in the Emacs sources, or even in the packages.

Or are you thinking about something like an explicit (defalias foo
(lambda (..) ...))?

> Furthermore, that info is already available to the bytecompiler, so it's
> actually easier to keep that info.

How are you actually going to use this information?  What are you going
to print in a backtrace, in particular in a batch mode backtrace where
things like buttons are not useful?

For that matter, what are you going to print in that C-h v example?

> >> Actually, the bytecode functions can already keep the FILE info (in
> >> the form of the (FILE . OFFSET) cons cell in the docstring slot of the
> >> bytecode object), we just opt never to expose that FILE info to the user
> >> (we only use it to fetch the actual docstring).
> > That's the offset in the .elc file, though, not the source.

> Yup.  [ Tho now that you make me think about it, if we want to be cheap,
> we could go to that OFFSET position and look around for a `^(defalias`
> and that might be sufficient to give us "the surrounding definition" in
> many cases :-)  ]

That doesn't sound systematic, or at all attractive.  ;-)

> >> [ Tho, admittedly, it's also that the byte-compiler only uses such
> >>   (FILE . OFFSET) for named functions, but it wouldn't be hard to change.  ]
> >> As for how/where to store the LINE+COL, I guess (aref form 5) is still
> >> an option, although, maybe we should store that info alongside the
> >> docstring, like we already do for the arglist.
> > Well, the doc string is at (aref form 4).  :-)

> What I meant is that the docstring is in the `.elc` file.  So the
> bytecode object would be literally unchanged, it's just that the
> bytes found at OFFSET inside FILE would be extended to hold:
> - the docstring
> - the human-readable arglist
> - the LINE+COL info

Don't forget, all this info would have to go into .eln files too,
probably in the struct Lisp_Subr.  It would be better if it were also
somehow available in uncompiled lambdas, something which is needed for
debugging failures in early bootstrapping.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Hmm... it seems like keeping the LINE+COL info rather than
>> a surrounding definition would be better in this regard since it wouldn't
>> require any changes at all to the actual lambdas, regardless where
>> they are.
> I don't agree with this.  Whether we add FILE+LINE+COL info is to a
> large degree a separate question from that of adding defining symbol
> info.

It seems to me, if you have one of the two, the other becomes much less
important.

> If you don't add information to the actual lambda, then that information
> won't be available when that lambda turns up in a backtrace or as a
> value in C-h v.  Surely you'd have to keep FILE+LINE+COL in the lambda
> itself, somehow.

In the docstring.

> It will also stress the user somewhat, since it will force her to look
> up a file location to get any idea of what is failing.  A defining
> symbol is much more immediately useful, even if it may not be 100%
> dependable because of the issues you raise below.

??

>> The notion of "surrounding definition" is less precise (there can be
>> several lambdas inside a given definition, there can be lambdas outside
>> of any definition, ...).
> Yes, there can be several lambdas in a function, though I don't think
> this is common.

You'd be surprised, especially with lambdas introduced by macros.

> Certainly it hasn't given me any problems in use, so far.

In the past, the byte-compiler only provided the name of the surrounding
definition in its warnings.  Maybe we should go back to that?  :-)

> But lambdas outside of definitions?  I doubt very much there are
> any in the Emacs sources, or even in the packages.

    grep -C1 '^(add-hook' **/*.el | grep '(lambda'

begs to differ.

>> Furthermore, that info is already available to the bytecompiler, so it's
>> actually easier to keep that info.
> How are you actually going to use this information?  What are you going
> to print in a backtrace, in particular in a batch mode backtrace where
> things like buttons are not useful?

FILE:LINE ?

> For that matter, what are you going to print in that C-h v example?

#<subr FILE:LINE HEX>?

Tho we could skip the FILE:LINE and buttonize some part of the string
instead, for the `C-h v` case.

>> Yup.  [ Tho now that you make me think about it, if we want to be cheap,
>> we could go to that OFFSET position and look around for a `^(defalias`
>> and that might be sufficient to give us "the surrounding definition" in
>> many cases :-)  ]
> That doesn't sound systematic, or at all attractive.  ;-)

But so much fun!
Admittedly, I haven't jumped on this hack because my own local hacks
move the docstrings to the end of the `.elc` files, making this
additional hack impossible :-)

>> What I meant is that the docstring is in the `.elc` file.  So the
>> bytecode object would be literally unchanged, it's just that the
>> bytes found at OFFSET inside FILE would be extended to hold:
>> - the docstring
>> - the human-readable arglist
>> - the LINE+COL info
>
> Don't forget, all this info would have to go into .eln files too,
> probably in the struct Lisp_Subr.

That's the beauty of it: by storing the info inside the raw docstring,
we get to reuse all the existing code that takes care of storing the
docstring somewhere.

Also, the "performance profile" of docstring matches our needs here:
just like docstrings, we don't need FILE+LINE+COL info under normal use,
it's only needed when developing/debugging, so we want to store it "out
of line".

> It would be better if it were also somehow available in uncompiled
> lambdas, something which is needed for debugging failures in
> early bootstrapping.

Yeah, "WIBNI", but as I said, when you have an uncompiled lambda, the
content of the body is usually sufficient to manually find the
matching source.  So I think we should focus on the compiled case
first and then maybe consider how or if we can/should extend it to the
uncompiled case.
[ And, yes, if we really want to, I think we can add the FILE+COL info
  to uncompiled lambdas: it would force us to read `.el` files
  "with sympos" (thus slower) but it shouldn't be hard.  ]


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 29, 2023 at 12:32:32 -0400, Stefan Monnier wrote:
> >> Hmm... it seems like keeping the LINE+COL info rather than
> >> a surrounding definition would be better in this regard since it wouldn't
> >> require any changes at all to the actual lambdas, regardless where
> >> they are.
> > I don't agree with this.  Whether we add FILE+LINE+COL info is to a
> > large degree a separate question from that of adding defining symbol
> > info.

> It seems to me, if you have one of the two, the other becomes much less
> important.

True, but still important nevertheless.

> > If you don't add information to the actual lambda, then that information
> > won't be available when that lambda turns up in a backtrace or as a
> > value in C-h v.  Surely you'd have to keep FILE+LINE+COL in the lambda
> > itself, somehow.

> In the docstring.

But the docstring isn't in the lambda at the moment.  There's merely a
pointer to it there, which is no good for backtraces - it's too fragile.

> > It will also stress the user somewhat, since it will force her to look
> > up a file location to get any idea of what is failing.  A defining
> > symbol is much more immediately useful, even if it may not be 100%
> > dependable because of the issues you raise below.

> ??

Dealing with backtraces is a stressful business.  Forcing a user to look
somewhere else for pertinent information can only increase that level of
stress.

> >> The notion of "surrounding definition" is less precise (there can be
> >> several lambdas inside a given definition, there can be lambdas outside
> >> of any definition, ...).
> > Yes, there can be several lambdas in a function, though I don't think
> > this is common.

> You'd be surprised, especially with lambdas introduced by macros.

> > Certainly it hasn't given me any problems in use, so far.

> In the past, the byte-compiler only provided the name of the surrounding
> definition in its warnings.  Maybe we should go back to that?  :-)

> > But lambdas outside of definitions?  I doubt very much there are
> > any in the Emacs sources, or even in the packages.

>     grep -C1 '^(add-hook' **/*.el | grep '(lambda'

> begs to differ.

OK, good point!  Here, obviously, the hook symbol should become the
defining symbol, though admittedly this isn't coded up, yet.

> >> Furthermore, that info is already available to the bytecompiler, so it's
> >> actually easier to keep that info.
> > How are you actually going to use this information?  What are you going
> > to print in a backtrace, in particular in a batch mode backtrace where
> > things like buttons are not useful?

> FILE:LINE ?

That is anything but concise, in contrast to printing the defining
symbol in braces.  I've just had a look at a bytecomp.elc, and the
filename in the docstring field is 67 characters long.  In anything
short of a full screen Emacs window, this, together with the name of the
subr, is going to spill over onto the next line.  This is not good.  Or
you have to abbreviate the filename somehow, but this is also unlikely
to be satisfactory.

Besides which, you'd need to store all the pertinent information, the
filename + line + column in RAM, probably in the subr or the byte coded
function.  That's assuming there is a file from which the lambda was
compiled, which there might well not be.  There's another special case.

> > For that matter, what are you going to print in that C-h v example?

> #<subr FILE:LINE HEX>?

> Tho we could skip the FILE:LINE and buttonize some part of the string
> instead, for the `C-h v` case.

You'd have to make a user option to do this, since fragmenting the
information is unkind to those who don't like jumping backwards and
forwards.

> >> Yup.  [ Tho now that you make me think about it, if we want to be cheap,
> >> we could go to that OFFSET position and look around for a `^(defalias`
> >> and that might be sufficient to give us "the surrounding definition" in
> >> many cases :-)  ]
> > That doesn't sound systematic, or at all attractive.  ;-)

> But so much fun!
> Admittedly, I haven't jumped on this hack because my own local hacks
> move the docstrings to the end of the `.elc` files, making this
> additional hack impossible :-)

> >> What I meant is that the docstring is in the `.elc` file.  So the
> >> bytecode object would be literally unchanged, it's just that the
> >> bytes found at OFFSET inside FILE would be extended to hold:
> >> - the docstring
> >> - the human-readable arglist
> >> - the LINE+COL info

> > Don't forget, all this info would have to go into .eln files too,
> > probably in the struct Lisp_Subr.

> That's the beauty of it: by storing the info inside the raw docstring,
> we get to reuse all the existing code that takes care of storing the
> docstring somewhere.

Making another non-obvious structure to compete with the (lexical)
parameter specs.

As I keep on saying, a backtrace may not depend on reading files; the
error it's reporting on could well be in that very mechanism itself.  We
already had a problem once due to the use of cl-print.el in
debug-early.el, and cl-print is relatively simple compared with file
reading.

> Also, the "performance profile" of docstring matches our needs here:
> just like docstrings, we don't need FILE+LINE+COL info under normal use,
> it's only needed when developing/debugging, so we want to store it "out
> of line".

Disagree: anybody getting a backtrace, or looking up a variable with C-h
v needs this information, at the very least for including in a bug
report.  ;-)

> > It would be better if it were also somehow available in uncompiled
> > lambdas, something which is needed for debugging failures in
> > early bootstrapping.

> Yeah, "WIBNI", but as I said, when you have an uncompiled lambda, the
> content of the body is usually sufficient to manually find the
> matching source.

Usually, but not always.  There are some lambdas added to hooks which
are functionally the same as ignore.

> So I think we should focus on the compiled case first and then maybe
> consider how or if we can/should extend it to the uncompiled case.

No.  Uncompiled code is more difficult to instrument (I'm talking from
experience here), and how that gets to be done influences how it can be
done for compiled code.  We cannot ignore it, push it down the road, and
hope that somehow it can be tacked on later.

> [ And, yes, if we really want to, I think we can add the FILE+COL info
> to uncompiled lambdas: it would force us to read `.el` files "with
> sympos" (thus slower) but it shouldn't be hard.  ]

Again, there must be no file reading when constructing a backtrace.  All
the pertinent information must be in the list form of the lambda or
closure at load time.  If the location of this new info is early on in
the function, it must be distinguishable from what used to be there in
the current format.  I've been through all this already.  The use of a
non-nil symbol preceding the parameter list cannot get confused with the
parameter list itself, which is still there in functions we will still
load.

I still believe that the defining symbol scheme is the most practical
way of conveying the needed extra information to the user.  Otherwise I
would have hacked something different.  ;-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> It seems to me, if you have one of the two, the other becomes much less
>> important.
> True, but still important nevertheless.

Not in my opinion.  We've lived with neither for the last 40 years and
it hasn't been raised as a problem pretty much ever during those 40
years, so anything that covers the 90% cases will be already borderline
"overkill".

>> In the docstring.
> But the docstring isn't in the lambda at the moment.  There's merely a
> pointer to it there, which is no good for backtraces - it's too fragile.

Never had a problem with that.

>> ??
> Dealing with backtraces is a stressful business.

I think you're going to give more concrete details if you want to
convince me of that.

> Forcing a user to look somewhere else for pertinent information can
> only increase that level of stress.

Any concrete evidence to back this claim?

>>     grep -C1 '^(add-hook' **/*.el | grep '(lambda'
>> begs to differ.
> OK, good point!
> Here, obviously, the hook symbol should become the defining symbol,

So that when you look at the functions on SOMEHOOK wondering where they
came from, they'll tell you "oh, I come from SOMEHOOK"?

> though admittedly this isn't coded up, yet.

Not sure it would be very useful either.

>> >> Furthermore, that info is already available to the bytecompiler, so it's
>> >> actually easier to keep that info.
>> > How are you actually going to use this information?  What are you going
>> > to print in a backtrace, in particular in a batch mode backtrace where
>> > things like buttons are not useful?
>> FILE:LINE ?
>
> That is anything but concise, in contrast to printing the defining
> symbol in braces.  I've just had a look at a bytecomp.elc, and the
> filename in the docstring field is 67 characters long.

Hmm... have you ever seen the FILE info we print in `C-h v`?
Has it often reached 67 chars?
Then why do you presume we'd be so stupid as to include the full
absolute file name in there?

> In anything short of a full screen Emacs window, this, together with
> the name of the subr, is going to spill over onto the next line.

Yes, if we try hard enough, we can fuck things up badly, of course.

> Besides which, you'd need to store all the pertinent information, the
> filename + line + column in RAM, probably in the subr or the byte
> coded function.

I already told you where we'd store that info.  Not in RAM.

>> That's the beauty of it: by storing the info inside the raw docstring,
>> we get to reuse all the existing code that takes care of storing the
>> docstring somewhere.
> Making another non-obvious structure to compete with the (lexical)
> parameter specs.

Yes.  Hopefully fixing that arglist thingy along the way.

> As I keep on saying, a backtrace may not depend on reading files;

Saying so doesn't make it true.

>> Also, the "performance profile" of docstring matches our needs here:
>> just like docstrings, we don't need FILE+LINE+COL info under normal use,
>> it's only needed when developing/debugging, so we want to store it "out
>> of line".
> Disagree: anybody getting a backtrace, or looking up a variable with C-h
> v needs this information, at the very least for including in a bug
> report.  ;-)

IOW, you agree.

> We cannot ignore it, push it down the road, and hope that somehow it
> can be tacked on later.

Of course we can, because we don't *have* to instrument it.

No matter how hard we work at it, there will be cases where "the origin"
of an anonymous function will be anything but obvious.  Even if we can
pinpoint the source code where the `lambda` itself is found, it may be
of no use to the programmer trying to figure out where it really "comes
from".

So it'll always be a best effort, where we try and cover the important cases.

>> [ And, yes, if we really want to, I think we can add the FILE+COL info
>> to uncompiled lambdas: it would force us to read `.el` files "with
>> sympos" (thus slower) but it shouldn't be hard.  ]
> Again, there must be no file reading when constructing a backtrace.

I'm not talking about `read`ing when you try and display the lambda.
I'm talking about `read`ing where you `load` the code that will evaluate
that lambda.

> I still believe that the defining symbol scheme is the most practical
> way of conveying the needed extra information to the user.

Obviously.

> Otherwise I would have hacked something different.  ;-)

I don't think that's how it works, sadly.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 29, 2023 at 17:47:57 -0400, Stefan Monnier wrote:
> >> It seems to me, if you have one of the two, the other becomes much less
> >> important.
> > True, but still important nevertheless.

> Not in my opinion.  We've lived with neither for the last 40 years and
> it hasn't been raised as a problem pretty much ever during those 40
> years, so anything that covers the 90% cases will be already borderline
> "overkill".

Debugging Emacs Lisp is, in my experience, a ghastly task - incomplete
error messages, incomplete backtraces (largely due to unneeded
condition-cases), inability to run a debugger on redisplay hooks, ....
This latest amendment of mine is one small element of making it less
ghastly.

> >> In the docstring.
> > But the docstring isn't in the lambda at the moment.  There's merely a
> > pointer to it there, which is no good for backtraces - it's too fragile.

> Never had a problem with that.

Not yet.  Putting complicated stuff into backtrace code is a bad idea -
look what happened when cl-print got put into debug-early.el.  Bugs.

> > Dealing with backtraces is a stressful business.

> I think you're going to give more concrete details if you want to
> convince me of that.

Don't be silly.  Any time there's a backtrace, somthing Has Gone Wrong
and Isn't Working.  That's bound to raise stress levels.

> > Forcing a user to look somewhere else for pertinent information can
> > only increase that level of stress.

> Any concrete evidence to back this claim?

Of course not.  Just as you've got no evidence to back up the contrary.

> >>     grep -C1 '^(add-hook' **/*.el | grep '(lambda'
> >> begs to differ.
> > OK, good point!
> > Here, obviously, the hook symbol should become the defining symbol,

> So that when you look at the functions on SOMEHOOK wondering where they
> came from, they'll tell you "oh, I come from SOMEHOOK"?

> > though admittedly this isn't coded up, yet.

> Not sure it would be very useful either.

> >> >> Furthermore, that info is already available to the bytecompiler, so it's
> >> >> actually easier to keep that info.
> >> > How are you actually going to use this information?  What are you going
> >> > to print in a backtrace, in particular in a batch mode backtrace where
> >> > things like buttons are not useful?
> >> FILE:LINE ?

> > That is anything but concise, in contrast to printing the defining
> > symbol in braces.  I've just had a look at a bytecomp.elc, and the
> > filename in the docstring field is 67 characters long.

> Hmm... have you ever seen the FILE info we print in `C-h v`?
> Has it often reached 67 chars?
> Then why do you presume we'd be so stupid as to include the full
> absolute file name in there?

That falls outside the bounds of acceptable GNU mailing list language.

> > In anything short of a full screen Emacs window, this, together with
> > the name of the subr, is going to spill over onto the next line.

> Yes, if we try hard enough, we can fuck things up badly, of course.

So does that.

> > Besides which, you'd need to store all the pertinent information, the
> > filename + line + column in RAM, probably in the subr or the byte
> > coded function.

> I already told you where we'd store that info.  Not in RAM.

So with all the extra complexity you want to add, you'll risk getting a
recursive error and recursive backtrace.

> >> That's the beauty of it: by storing the info inside the raw docstring,
> >> we get to reuse all the existing code that takes care of storing the
> >> docstring somewhere.
> > Making another non-obvious structure to compete with the (lexical)
> > parameter specs.

> Yes.  Hopefully fixing that arglist thingy along the way.

> > As I keep on saying, a backtrace may not depend on reading files;

> Saying so doesn't make it true.

You haven't made any attempt to justify this, to my mind, questionable
design.

> >> Also, the "performance profile" of docstring matches our needs here:
> >> just like docstrings, we don't need FILE+LINE+COL info under normal use,
> >> it's only needed when developing/debugging, so we want to store it "out
> >> of line".
> > Disagree: anybody getting a backtrace, or looking up a variable with C-h
> > v needs this information, at the very least for including in a bug
> > report.  ;-)

> IOW, you agree.

> > We cannot ignore it, push it down the road, and hope that somehow it
> > can be tacked on later.

> Of course we can, because we don't *have* to instrument it.

Which looks like you intend to strip it out, leaving interpreted code as
a special case without the instrumentation.  I don't think that's a good
idea.

> No matter how hard we work at it, there will be cases where "the origin"
> of an anonymous function will be anything but obvious.  Even if we can
> pinpoint the source code where the `lambda` itself is found, it may be
> of no use to the programmer trying to figure out where it really "comes
> from".

> So it'll always be a best effort, where we try and cover the important cases.

Which is a good deal better than the current state of affairs.

> >> [ And, yes, if we really want to, I think we can add the FILE+COL info
> >> to uncompiled lambdas: it would force us to read `.el` files "with
> >> sympos" (thus slower) but it shouldn't be hard.  ]
> > Again, there must be no file reading when constructing a backtrace.

> I'm not talking about `read`ing when you try and display the lambda.
> I'm talking about `read`ing where you `load` the code that will evaluate
> that lambda.

You've been anything but clear in saying what you intend here.  You've
talked about "in the doc string" without saying whether you meant in the
..elc file or in the loaded program, for example.  That's a critical
difference.

> > I still believe that the defining symbol scheme is the most practical
> > way of conveying the needed extra information to the user.

> Obviously.

> > Otherwise I would have hacked something different.  ;-)

> I don't think that's how it works, sadly.

What are you talking about with "that"?

Thus far, all you've done is disparage every design decision I've made
with this code, saying how you would have done it differently, without
giving any justification for why your way would be better.  How am I
meant to react?

Thus far, you've given no sign that you've actually tried the code out,
or done anything else with it bar vaguely looking at a diff from master.
It would be nice if you were actually to evaluate this code.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alan Mackenzie [2023-10-30 09:44:54] wrote:
> Thus far, all you've done is disparage every design decision I've made
> with this code, saying how you would have done it differently, without
> giving any justification for why your way would be better.  How am I
> meant to react?

Thanks for putting into words how I feel :-)

> Not yet.  Putting complicated stuff into backtrace code is a bad idea -
> look what happened when cl-print got put into debug-early.el.  Bugs.

Bootstrap problems are rare, and difficult to predict, so we fix them in
a reactive way.  Debugging a bootstrap is generally more difficult.
That's also true when trying to debug GDB (the usual answer is to use
another GDB, and in Emacs you can do the same: debug from another
debugger, such as GDB, rather than from the built-in debugger).

We shouldn't design the featureset for the corner case of when the
bootstrap fails, because it is a very small fraction of the use-cases,
that 99.99% of Emacs users never face.

>> > Forcing a user to look somewhere else for pertinent information can
>> > only increase that level of stress.
>> Any concrete evidence to back this claim?
> Of course not.  Just as you've got no evidence to back up the contrary.

Fist, I did not make a claim to the contrary, so the burden of proof is
not on me.  This said, I believe there is ample evidence out there,
because all the programming tools I've ever used try first and foremost
to let you jump to the corresponding source code.  The source code is
what programmers are familiar with, so being able to jump from the
backtrace to the corresponding source code is generally the best known
way to help the user find the pertinent info.

From the FILE+LINE+COL info the user can recover the info your patch
provides, but the reverse is generally not true.  Of course, there is
the exception of when the source code has been changed or is missing or
somesuch, but I don't think we should consider those cases when
designing the featureset either, because here as well, this is expected
to be a fringe case.

>> Then why do you presume we'd be so stupid as to include the full
>> absolute file name in there?
> That falls outside the bounds of acceptable GNU mailing list language.

Yes, I was aware of it.  I just wasn't able to refrain from venting my
frustration: criticizing my suggestion based on the fact that the full
file name is too long falls squarely in the category of "strawman
argument".

>> > Besides which, you'd need to store all the pertinent information, the
>> > filename + line + column in RAM, probably in the subr or the byte
>> > coded function.
>> I already told you where we'd store that info.  Not in RAM.
> So with all the extra complexity you want to add, you'll risk getting a
> recursive error and recursive backtrace.

I want the better info, even if that means a slight increase in
complexity and the associated risk that I have to go and fix a bug in
that code next year, yes.

> You haven't made any attempt to justify this, to my mind, questionable
> design.

Hmm... I'd return the compliment :-)

> Which looks like you intend to strip it out, leaving interpreted code as
> a special case without the instrumentation.

Yes.

>> So it'll always be a best effort, where we try and cover the important cases.
> Which is a good deal better than the current state of affairs.

But as it currently stands, it comes at some costs:
- changing the syntax of `lambda`.
- changing the representation of functions (both bytecode and interpreted).
- added runtime costs even when the feature unused (no `C-h v`, no
  backtraces), which is the most likely use-case.

Storing the info in the docstring would fix the second and third point.
Using the FILE+LINE+COL info would fix the first.

>> > I still believe that the defining symbol scheme is the most practical
>> > way of conveying the needed extra information to the user.
>> Obviously.
>> > Otherwise I would have hacked something different.  ;-)
>> I don't think that's how it works, sadly.
> What are you talking about with "that"?

I have the impression that you're defending your design because that's
what you've coded.  :-(


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Wed, Nov 01, 2023 at 08:47:30 -0400, Stefan Monnier wrote:
> Alan Mackenzie [2023-10-30 09:44:54] wrote:

[ .... ]

> Thanks for putting into words how I feel :-)

:-)

> > Not yet.  Putting complicated stuff into backtrace code is a bad idea -
> > look what happened when cl-print got put into debug-early.el.  Bugs.

> Bootstrap problems are rare, and difficult to predict, so we fix them in
> a reactive way.  Debugging a bootstrap is generally more difficult.

I'm fully aware of this, having spent a significant part of the last few
years doing just that.  I aim to make it easier, as I have already done
to a certain degree.

> That's also true when trying to debug GDB (the usual answer is to use
> another GDB, and in Emacs you can do the same: debug from another
> debugger, such as GDB, rather than from the built-in debugger).

That doesn't seem relevant to bootstrapping bugs.  Using gdb on part of
the bootstrap is quite difficult.  Using edebug is not possible, as far
as I'm aware.

> We shouldn't design the featureset for the corner case of when the
> bootstrap fails, because it is a very small fraction of the use-cases,
> that 99.99% of Emacs users never face.

We should design the feature set to handle _ALL_ cases, including that
of bootstrapping.  That's why I wrote debug-early.el, for example; it's
proved its worth, certainly for me personally.

> >> > Forcing a user to look somewhere else for pertinent information can
> >> > only increase that level of stress.
> >> Any concrete evidence to back this claim?
> > Of course not.  Just as you've got no evidence to back up the contrary.

> Fist, I did not make a claim to the contrary, so the burden of proof is
> not on me.  This said, I believe there is ample evidence out there,
> because all the programming tools I've ever used try first and foremost
> to let you jump to the corresponding source code.  The source code is
> what programmers are familiar with, so being able to jump from the
> backtrace to the corresponding source code is generally the best known
> way to help the user find the pertinent info.

Even better is if the relevant information is present on the screen or
in the backtrace without having to jump to the source code.  This is the
point of the defining symbol.  The "jumping" is only straightforward
from a running Emacs session.  From a batch mode backtrace (e.g. from
bootstrapping or ERT), it is much more cumbersome.

> >From the FILE+LINE+COL info the user can recover the info your patch
> provides, but the reverse is generally not true.  Of course, there is
> the exception of when the source code has been changed or is missing or
> somesuch, but I don't think we should consider those cases when
> designing the featureset either, because here as well, this is expected
> to be a fringe case.

I would expect it to be quite common.  We clearly differ in our views of
what backtraces should do.  You seem to want them only to cover the most
usual cases, I want it to cover all cases, or as many as possible when
all is not possible.  Also, I think the backtrace code should be as
robust as possible, you don't seem to.

Anyhow, I now see you are correct in saying that we need file and
position information.  I've scanned the sources for uses of lambda at
the top level, and there are functions such as mapc, define-key, ...
which have no symbol associated with them that could be used as the
defining symbol.

I will be working on putting this file and position information into the
code.  It will be tedious rather than difficult.

[ .... ]

> > So with all the extra complexity you want to add, you'll risk getting a
> > recursive error and recursive backtrace.

> I want the better info, even if that means a slight increase in
> complexity and the associated risk that I have to go and fix a bug in
> that code next year, yes.

Debugging tools are a special case: if they stop working, the normal
means of diagnosing bugs don't work on them.  Thus they should be
robust.

[ .... ]

> > Which looks like you intend to strip it [debugging instrumentation]
> > out, leaving interpreted code as a special case without the
> > instrumentation.

> Yes.

OK, we have to disagree on this.  For example typing d in edebug to get
a backtrace should, in my view, show that info in the backtrace.  Your
way would make it unavailable.

> >> So it'll always be a best effort, where we try and cover the
> >> important cases.
> > Which is a good deal better than the current state of affairs.

> But as it currently stands, it comes at some costs:
> - changing the syntax of `lambda`.

lambda in the bulk of the sources remains unchanged.  Only for very
occasional cases has the syntax been enhanced.

> - changing the representation of functions (both bytecode and interpreted).

Don't forget the native compiled, too!  Only the change to interpreted
code is non-trivial.  For the compiled code, there has simply been an
extra field inserted at or near the end of the function structures.

> - added runtime costs even when the feature unused (no `C-h v`, no
>   backtraces), which is the most likely use-case.

There are no added runtime costs beyond the unmeasurably minute.  I've
compared before and after timings, and the differences, such as they
were, were much less than 1% and swamped by the random variation in
timings anyway.

> Storing the info in the docstring would fix the second and third point.
> Using the FILE+LINE+COL info would fix the first.

I don't know what you mean by "docstring".  You could mean either in the
..elc/.eln file or in the function structure or both.  I don't see
docstring storage as a very good idea.  It seems like a kludge, and
would complicate Emacs's handling, much as the existing new code
complicates it.

> >> > I still believe that the defining symbol scheme is the most practical
> >> > way of conveying the needed extra information to the user.
> >> Obviously.
> >> > Otherwise I would have hacked something different.  ;-)
> >> I don't think that's how it works, sadly.
> > What are you talking about with "that"?

> I have the impression that you're defending your design because that's
> what you've coded.  :-(

Only partly.  Mainly it was because I thought these things out at an
early stage of development.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Bootstrap problems are rare, and difficult to predict, so we fix them in
>> a reactive way.  Debugging a bootstrap is generally more difficult.
> I'm fully aware of this, having spent a significant part of the last few
> years doing just that.  I aim to make it easier, as I have already done
> to a certain degree.

Exactly.  That makes you put much more emphasis on this aspect than it
generally deserves.  *Very* few Emacs users ever do that.

> That doesn't seem relevant to bootstrapping bugs.  Using gdb on part of
> the bootstrap is quite difficult.

GDB is my "go to" tool when I have an error during bootstrap and I find
it fairly easy to use.

>> We shouldn't design the featureset for the corner case of when the
>> bootstrap fails, because it is a very small fraction of the use-cases,
>> that 99.99% of Emacs users never face.
> We should design the feature set to handle _ALL_ cases, including that
> of bootstrapping.  That's why I wrote debug-early.el, for example; it's
> proved its worth, certainly for me personally.

Note that `early-debug.el` doesn't handle "_ALL_" cases: it's only used
for those cases where the normal debugger can't be used.

Before catering to the same use-cases as early-debug, we should cater to
the use cases of the average user/developer who's never getting close to
the bootstrap.

>> Fist, I did not make a claim to the contrary, so the burden of proof is
>> not on me.  This said, I believe there is ample evidence out there,
>> because all the programming tools I've ever used try first and foremost
>> to let you jump to the corresponding source code.  The source code is
>> what programmers are familiar with, so being able to jump from the
>> backtrace to the corresponding source code is generally the best known
>> way to help the user find the pertinent info.
> Even better is if the relevant information is present on the screen or
> in the backtrace without having to jump to the source code.

How are you going to fix the problem without going to the source code?

> This is the point of the defining symbol.  The "jumping" is only
> straightforward from a running Emacs session.  From a batch mode
> backtrace (e.g. from bootstrapping or ERT), it is much
> more cumbersome.

If your batch session's output is piped to an auto-reverted file in
compilation-mode (mine usually is, FWIW), it's small matter of making
sure the format is recognized.

Still, I refuse to change the syntax of `lambda` just to make it
slightly easier to debug the odd bootstrap problem.

> >From the FILE+LINE+COL info the user can recover the info your patch
>> provides, but the reverse is generally not true.  Of course, there is
>> the exception of when the source code has been changed or is missing or
>> somesuch, but I don't think we should consider those cases when
>> designing the featureset either, because here as well, this is expected
>> to be a fringe case.
> I would expect it to be quite common.

From where I stand:

- If you don't have the source code any more, then you don't have anywhere
  to fix the problem.
- If the source code has changed, then maybe the problem has already been
  corrected (or modified) by that change, so there's not much point
  debugging it any further.

So, why would you consider it to be common?
What kind of scenario do you have in mind where this is sufficiently
common to be worth paying much attention to it?

> We clearly differ in our views of what backtraces should do.

I don't know why you insist so much on backtraces.
I've wished for that info in many more cases than backtraces.

> Also, I think the backtrace code should be as robust as possible, you
> don't seem to.

I want it to be featureful, with a fallback for those rare cases when
that's not an option.  You provided the fallback already in `debug-early`.

> I will be working on putting this file and position information into the
> code.  It will be tedious rather than difficult.

Why would it be tedious?  It should be easier than what you did because
you don't need an extra arg to `lambda` and you don't need to
`defining-symbol` any more.  It should make the change simpler.

>> > Which looks like you intend to strip it [debugging instrumentation]
>> > out, leaving interpreted code as a special case without the
>> > instrumentation.
>> Yes.
> OK, we have to disagree on this.  For example typing d in edebug to get
> a backtrace should, in my view, show that info in the backtrace.

I don't find this particularly important.

> Your way would make it unavailable.

You could tweak Edebug to add that info, thus keeping the change localized
to Edebug.

>> But as it currently stands, it comes at some costs:
>> - changing the syntax of `lambda`.
> lambda in the bulk of the sources remains unchanged.  Only for very
> occasional cases has the syntax been enhanced.

Still, it changes the syntax of a very fundamental building block.
That's a pretty high cost in my book compared to the benefit it brings.

If you want to do that, I'd rather you keep `lambda` as before and
macro-expand it to a new form that has the new syntax.

Part of the cost of your change here is that it prevents future changes
that might want to use a similar syntax for other purposes.
IOW it "eats up namespace", tho "namespace" is not quite the right word.

>> - changing the representation of functions (both bytecode and interpreted).
> Don't forget the native compiled, too!

That's more internal and it's currently not used for closures, so I find
this part of the cost negligible.

> Only the change to interpreted code is non-trivial.  For the compiled
> code, there has simply been an extra field inserted at or near the end
> of the function structures.

For bytecode it's still an extra field (which we have to pay for every
closure we create) thus increasing memory use.  Plus the incompatible
change of position of the interactive form (tho I find this less of an
problem).

Maybe in your tests, the increase is negligible, but remember: every
user will pay this tiny cost, and most of the users will never reap any
benefit from it, and those who do will only do very rarely.

It's a good change for us hard-core Emacs hackers, but it has to come
*very* cheap because the gain is really low for other users.

>> Storing the info in the docstring would fix the second and third point.
>> Using the FILE+LINE+COL info would fix the first.
> I don't know what you mean by "docstring".

Docstrings live in the `.elc` files.

> You could mean either in the ..elc/.eln file or in the function
> structure or both.  I don't see docstring storage as a very good idea.

Probably because you think of it as a docstring only.
Think of it as "out of line storage" where we can keep introspection
data for which we don't want to pay *any* cost (other than disk space
use) in the normal execution of code (i.e. when calling the closure or
when creating the closure) such as:
- documentation
- human-readable arglist
- source code location
- debug info
- a copy of the source code
- ...

> It seems like a kludge, and would complicate Emacs's handling, much as
> the existing new code complicates it.

I see it as an opportunity to clean up this kludge.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Wed, Nov 01, 2023 at 14:11:33 -0400, Stefan Monnier wrote:
> >> Bootstrap problems are rare, and difficult to predict, so we fix them in
> >> a reactive way.  Debugging a bootstrap is generally more difficult.
> > I'm fully aware of this, having spent a significant part of the last few
> > years doing just that.  I aim to make it easier, as I have already done
> > to a certain degree.

> Exactly.  That makes you put much more emphasis on this aspect than it
> generally deserves.  *Very* few Emacs users ever do that.

That's just your view.  You're wrong, here.  I've put little emphasis on
bootstrap problems.  That they are handled is a side effect of handling
everything.

> > That doesn't seem relevant to bootstrapping bugs.  Using gdb on part of
> > the bootstrap is quite difficult.

> GDB is my "go to" tool when I have an error during bootstrap and I find
> it fairly easy to use.

OK, I found it very difficult to get a working command line for this.
Maybe if you do it often enough it becomes second nature.

> >> We shouldn't design the featureset for the corner case of when the
> >> bootstrap fails, because it is a very small fraction of the use-cases,
> >> that 99.99% of Emacs users never face.
> > We should design the feature set to handle _ALL_ cases, including that
> > of bootstrapping.  That's why I wrote debug-early.el, for example; it's
> > proved its worth, certainly for me personally.

> Note that `early-debug.el` doesn't handle "_ALL_" cases: it's only used
> for those cases where the normal debugger can't be used.

That's twisted reasoning.  early-debug.el will handle anything that gets
thrown at it.

> Before catering to the same use-cases as early-debug, we should cater to
> the use cases of the average user/developer who's never getting close to
> the bootstrap.

I disagree.  I don't understand why you want to split off (some of) "the
use cases of the average user" and only handle those cases.  My
intention is to handle _ALL_ cases, and you find this objectionable for
some reason.

> >> Fist, I did not make a claim to the contrary, so the burden of proof is
> >> not on me.  This said, I believe there is ample evidence out there,
> >> because all the programming tools I've ever used try first and foremost
> >> to let you jump to the corresponding source code.  The source code is
> >> what programmers are familiar with, so being able to jump from the
> >> backtrace to the corresponding source code is generally the best known
> >> way to help the user find the pertinent info.
> > Even better is if the relevant information is present on the screen or
> > in the backtrace without having to jump to the source code.

> How are you going to fix the problem without going to the source code?

You're being contrary again.  My intention is that the problem should be
better understood before going to the source code - in fact better to
understand which source code is relevant.

> > This is the point of the defining symbol.  The "jumping" is only
> > straightforward from a running Emacs session.  From a batch mode
> > backtrace (e.g. from bootstrapping or ERT), it is much
> > more cumbersome.

> If your batch session's output is piped to an auto-reverted file in
> compilation-mode (mine usually is, FWIW), it's small matter of making
> sure the format is recognized.

In other words, assume the user is advanced enough to go through all
these motions, just to be able to work the way you do.  I suspect most
users will not be doing this.  I certainly don't (at least, not yet).

> Still, I refuse to change the syntax of `lambda` just to make it
> slightly easier to debug the odd bootstrap problem.

For crying out loud, I've explained to you often enough, the syntax of
lambda is _NOT_, I repeat _NOT_ being changed.

> > >From the FILE+LINE+COL info the user can recover the info your patch
> >> provides, but the reverse is generally not true.  Of course, there is
> >> the exception of when the source code has been changed or is missing or
> >> somesuch, but I don't think we should consider those cases when
> >> designing the featureset either, because here as well, this is expected
> >> to be a fringe case.
> > I would expect it to be quite common.

> >From where I stand:

> - If you don't have the source code any more, then you don't have anywhere
>   to fix the problem.
> - If the source code has changed, then maybe the problem has already been
>   corrected (or modified) by that change, so there's not much point
>   debugging it any further.

> So, why would you consider it to be common?

Because people change source code, without necessarily compiling it
straight away.  Maybe you don't.  Other people do.

> What kind of scenario do you have in mind where this is sufficiently
> common to be worth paying much attention to it?

Wrong question.  I've already explained enough, I believe backtraces and
things like C-h v should work well IN ALL CASES.  You don't.  I'm asking
you to accept this difference in view between us.

> > We clearly differ in our views of what backtraces should do.

> I don't know why you insist so much on backtraces.
> I've wished for that info in many more cases than backtraces.

That info will get displayed by anything using cl-prin1.

> > Also, I think the backtrace code should be as robust as possible, you
> > don't seem to.

> I want it to be featureful, with a fallback for those rare cases when
> that's not an option.  You provided the fallback already in `debug-early`.

But the information it currently prints is inadequate for lambda forms.
That's what I've fixed, and it will be fixed in the "normal" backtrace
code, too.  You don't like it, for some reason.

> > I will be working on putting this file and position information into the
> > code.  It will be tedious rather than difficult.

> Why would it be tedious?  It should be easier than what you did because
> you don't need an extra arg to `lambda` and you don't need to
> `defining-symbol` any more.  It should make the change simpler.

I'm not coding up your design decisions which I believe are misguided.
Perhaps you should code up these things yourself if you think it's going
to be so fascinating.  Then we'll be able to compare my patch with
yours.

> >> > Which looks like you intend to strip it [debugging instrumentation]
> >> > out, leaving interpreted code as a special case without the
> >> > instrumentation.
> >> Yes.
> > OK, we have to disagree on this.  For example typing d in edebug to get
> > a backtrace should, in my view, show that info in the backtrace.

> I don't find this particularly important.

Other people, including me, do.

> > Your way would make it unavailable.

> You could tweak Edebug to add that info, thus keeping the change localized
> to Edebug.

I doubt it.

> >> But as it currently stands, it comes at some costs:
> >> - changing the syntax of `lambda`.
> > lambda in the bulk of the sources remains unchanged.  Only for very
> > occasional cases has the syntax been enhanced.

> Still, it changes the syntax of a very fundamental building block.
> That's a pretty high cost in my book compared to the benefit it brings.

You don't like my patch for some other reason.  I think you're being
unreasonably inflexible.  As far as I can tell, you haven't tried it out
yet, or looked at the code, beyond a superficial glance at a large diff.

> If you want to do that, I'd rather you keep `lambda` as before and
> macro-expand it to a new form that has the new syntax.

There is no new syntax, as I keep telling you.

> Part of the cost of your change here is that it prevents future changes
> that might want to use a similar syntax for other purposes.
> IOW it "eats up namespace", tho "namespace" is not quite the right word.

I don't know what you're talking about here.

> >> - changing the representation of functions (both bytecode and interpreted).
> > Don't forget the native compiled, too!

> That's more internal and it's currently not used for closures, so I find
> this part of the cost negligible.

> > Only the change to interpreted code is non-trivial.  For the compiled
> > code, there has simply been an extra field inserted at or near the end
> > of the function structures.

> For bytecode it's still an extra field (which we have to pay for every
> closure we create) thus increasing memory use.  Plus the incompatible
> change of position of the interactive form (tho I find this less of an
> problem).

I think you're being thoroughly unrealistic here.  RAM in a modern
machine is measured in gigabytes, not kilobytes.  There are around 36k
DEFUNs/defuns/defmacros in Emacs.  Even if all were simultaneously
loaded into RAM, that would only be around a quarter of a megabyte of
storage used up by my patch.

If storage were really that critical, we would never have changed from
4-byte Lisp_Objects to 8-byte ones, something which will have hugely
increased the storage used.

The change in the interactive form in byte compiled functions isn't
incompatible with anything that uses the proper interfaces.

> Maybe in your tests, the increase is negligible, but remember: every
> user will pay this tiny cost, and most of the users will never reap any
> benefit from it, and those who do will only do very rarely.

Developers will benefit.

> It's a good change for us hard-core Emacs hackers, but it has to come
> *very* cheap because the gain is really low for other users.

It _is_ very cheap.

> >> Storing the info in the docstring would fix the second and third point.
> >> Using the FILE+LINE+COL info would fix the first.
> > I don't know what you mean by "docstring".

> Docstrings live in the `.elc` files.

OK, you've got your own vocabulary on this point.  To most people, a doc
string is something abstract, which is created in *scratch* or a .el
file, gets evaluated with, say C-x C-e, gets compiled by compile-defun,
and gets native compiled, too, all the while remaining that same doc
string.

> > You could mean either in the ..elc/.eln file or in the function
> > structure or both.  I don't see docstring storage as a very good idea.

> Probably because you think of it as a docstring only.
> Think of it as "out of line storage" where we can keep introspection
> data for which we don't want to pay *any* cost (other than disk space
> use) in the normal execution of code (i.e. when calling the closure or
> when creating the closure) such as:
> - documentation
> - human-readable arglist
> - source code location
> - debug info
> - a copy of the source code
> - ...

Yes, I think of the "docstring" as the doc string.  I don't think it a
good idea to cram other things into it which aren't the doc string.

> > It seems like a kludge, and would complicate Emacs's handling, much as
> > the existing new code complicates it.

> I see it as an opportunity to clean up this kludge.

Go ahead, then.  Then we can compare your version with mine.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Alan,

> That's just your view.  You're wrong, here.  I've put little emphasis on
> bootstrap problems.

Maybe you didn't think of it explicitly while designing your solution,
but you sure do mention it very frequently in this discussion.

> That they are handled is a side effect of handling everything.

Are you saying that you're using the argument that it works in the
bootstrap case as a post-hoc rationalization of your design choices?

>> GDB is my "go to" tool when I have an error during bootstrap and I find
>> it fairly easy to use.
> OK, I found it very difficult to get a working command line for this.
> Maybe if you do it often enough it becomes second nature.

Yes, it takes some practice at first to figure out how to get the
command line, indeed.  Since it's a problem that (as a rough
approximation) people either never face or on the contrary face
regularly, that seems acceptable.

>> Note that `early-debug.el` doesn't handle "_ALL_" cases: it's only used
>> for those cases where the normal debugger can't be used.
> That's twisted reasoning.  early-debug.el will handle anything that gets
> thrown at it.

There's "handle" and there's "handle".  You seem to use "handle" to mean
that it will so what it does without crashing or returning
incorrect info.  I'd use "handle" here to mean that it does what is
expected, which it doesn't because we expect more features, which is why
we use it only in special circumstances.

>> Before catering to the same use-cases as early-debug, we should cater to
>> the use cases of the average user/developer who's never getting close to
>> the bootstrap.
> I disagree.  I don't understand why you want to split off (some of) "the
> use cases of the average user" and only handle those cases.

I don't.  Instead I'm saying we should first and foremost design the
functionality for the most common case.  And *then* see what it takes
to extend it to other cases.

> My intention is to handle _ALL_ cases, and you find this objectionable
> for some reason.

Because your focus on the fringe cases has negative impacts on the
more common case.

>> > This is the point of the defining symbol.  The "jumping" is only
>> > straightforward from a running Emacs session.  From a batch mode
>> > backtrace (e.g. from bootstrapping or ERT), it is much
>> > more cumbersome.
>> If your batch session's output is piped to an auto-reverted file in
>> compilation-mode (mine usually is, FWIW), it's small matter of making
>> sure the format is recognized.
> In other words, assume the user is advanced enough to go through all
> these motions, just to be able to work the way you do.  I suspect most
> users will not be doing this.  I certainly don't (at least, not yet).

This is only needed when the error is emitted on the stdout/stderr.
An unusual case.  Users don't have to use the above setup: they can also
open the file by hand and jump to the line by hand.  I'm only pointing
out how to make it work conveniently if you're among those rare users
who face this situation often enough that the problem you mention is one
they actually care about.

>> Still, I refuse to change the syntax of `lambda` just to make it
>> slightly easier to debug the odd bootstrap problem.
> For crying out loud, I've explained to you often enough, the syntax of
> lambda is _NOT_, I repeat _NOT_ being changed.

That's not what I see in your branch.  Now if lambda's first arg is
a symbol it means something else than what it meant earlier.

So the syntax of lambda used to

    (lambda ARGSLIST [DOCSTRING] [DECLARATIONS] [INTERACTIVEFORM] BODY...)

and now it's

    (lambda [SYMBOL] ARGSLIST [DOCSTRING] [DECLARATIONS] [INTERACTIVEFORM] BODY...)

Crying out loud or not doesn't change the fact that the above is
a change of syntax.

E.g. I see:

    +  #'(lambda
    +      byte-run--strip-list
    +            (arg)

clearly, this is using a new syntax.

> Wrong question.  I've already explained enough, I believe backtraces and
> things like C-h v should work well IN ALL CASES.  You don't.  I'm asking
> you to accept this difference in view between us.

Fine.  But please accept as well my point of view which is that your
change should not cost an extra field in every closure and should not
change the syntax of `lambda` (I don't really care about changing the
representation of interpreted functions, to be honest: I see them as
a just crutches needed for bootstrap and Edebug).

> But the information it currently prints is inadequate for lambda forms.

Agreed.

> That's what I've fixed, and it will be fixed in the "normal" backtrace
> code, too.  You don't like it, for some reason.

You got me wrong: I like the idea.
I don't like some aspects of the execution, which is why I'm suggesting
changes to address those shortcomings.

>> You could tweak Edebug to add that info, thus keeping the change localized
>> to Edebug.
> I doubt it.

I'm pretty sure it's a straightforward change :-)
[ Tho, admittedly, you may not like the place where that defining-form
symbol would appear, since I'd put it somewhere at the beginning of the body of
the function.  ]

>> Still, it changes the syntax of a very fundamental building block.
>> That's a pretty high cost in my book compared to the benefit it brings.
> You don't like my patch for some other reason.

Ah, so you're presuming that I'm arguing in bad faith, which is why
you're being so defensive.  I'm not arguing in bad faith.

The two things that I object to are the two things above (the change in
`lambda` and the extra field in closures).  I proposed ways to fix them,
but if you find some other way to fix them, that's fine by me.

> I think you're being unreasonably inflexible.  As far as I can tell,
> you haven't tried it out yet, or looked at the code, beyond
> a superficial glance at a large diff.

Trying it out did not make me see anything I didn't know (except that
the print syntax you're using will/would need changing because a list
of functions ends up as:

    ({mysym} #f(compiled-function ...) {theirsym} #f(compiled-function ...))

which looks like the list contains 4 elements instead of 2, but that's
a completely irrelevant detail at this point).

>> If you want to do that, I'd rather you keep `lambda` as before and
>> macro-expand it to a new form that has the new syntax.
> There is no new syntax, as I keep telling you.

I don't think you know what "new syntax" means, then.

>> Part of the cost of your change here is that it prevents future changes
>> that might want to use a similar syntax for other purposes.
>> IOW it "eats up namespace", tho "namespace" is not quite the right word.
> I don't know what you're talking about here.

For example, let's say that I want to extend the language such that

    (lambda x <code>)

means the same as

    (lambda (&rest x) <code>)

like it does in Scheme.  After your patch is accepted, such a change
will not be possible anymore because it conflicts with your use of the
(lambda <SYMBOL> ...) syntax.

> I think you're being thoroughly unrealistic here.  RAM in a modern
> machine is measured in gigabytes, not kilobytes.  There are around 36k
> DEFUNs/defuns/defmacros in Emacs.  Even if all were simultaneously
> loaded into RAM, that would only be around a quarter of a megabyte of
> storage used up by my patch.

I'm talking about closures.  I don't care about that extra field in
those few thousand functions.  I care about that extra field in the
closures that are dynamically generated with code such as that output by
`generator.el`.
There can easily be thousands such closures generated during the
execution of a simple command, without loading any new code.

>> Maybe in your tests, the increase is negligible, but remember: every
>> user will pay this tiny cost, and most of the users will never reap any
>> benefit from it, and those who do will only do very rarely.
> Developers will benefit.

They'll benefit just as much if you put that info in the docstring/.elc
file (such that you don't need to change the size of your bytecode
objects).  For this reason, the extra cost in unacceptable no matter how
small you think it is: it's a pure waste.

> Yes, I think of the "docstring" as the doc string.  I don't think it a
> good idea to cram other things into it which aren't the doc string.

You have to distinguish the docstring itself, and the mechanism used to
store it.  I'm talking about the internal mechanism.  I don't want to
literally place the LINE+COL info in the docstring (contrary to what
I sadly did for arglists).  Instead I want to place it in the .elc file
next to the actual docstring such that we can recover it from the very
same (FILE . OFFSET) info that points to the docstring.

IOE, I suggest we rename the "docstring" found in `.elc` at (FILE
. OFFSET) into an "introspection data" which can contain the docstring
and a few other things.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
>  Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 01 Nov 2023 18:46:24 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> >> Before catering to the same use-cases as early-debug, we should cater to
> >> the use cases of the average user/developer who's never getting close to
> >> the bootstrap.
> > I disagree.  I don't understand why you want to split off (some of) "the
> > use cases of the average user" and only handle those cases.
> 
> I don't.  Instead I'm saying we should first and foremost design the
> functionality for the most common case.  And *then* see what it takes
> to extend it to other cases.

Agreed.

Alan, we are in the engineering business, where designing for the most
common case, before adjusting the design in small ways for less common
cases, is one of the important principles.

> > My intention is to handle _ALL_ cases, and you find this objectionable
> > for some reason.
> 
> Because your focus on the fringe cases has negative impacts on the
> more common case.

And this is the usual outcome of trying to solve _ALL_ cases from the
get-go, by focusing on the fringe ones.  Premature optimization and
all that.

Alan, I'd appreciate if you rethink your approach to the
implementation with the above principles in mind.

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Eli.

On Thu, Nov 02, 2023 at 08:13:30 +0200, Eli Zaretskii wrote:
> > Cc: 66750@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>,
> >  Stefan Kangas <stefankangas@gmail.com>
> > Date: Wed, 01 Nov 2023 18:46:24 -0400
> > From:  Stefan Monnier via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>

> > >> Before catering to the same use-cases as early-debug, we should cater to
> > >> the use cases of the average user/developer who's never getting close to
> > >> the bootstrap.
> > > I disagree.  I don't understand why you want to split off (some of) "the
> > > use cases of the average user" and only handle those cases.

> > I don't.  Instead I'm saying we should first and foremost design the
> > functionality for the most common case.  And *then* see what it takes
> > to extend it to other cases.

> Agreed.

What Stefan seems to be proposing does not accord with his previous
paragraph.  He is proposing an implementation which would preclude
extension to what he calls the "less common" cases - things like working
in edebug and interpreted code in general, amongst others.

> Alan, we are in the engineering business, where designing for the most
> common case, before adjusting the design in small ways for less common
> cases, is one of the important principles.

Another principle is not closing off future options.  Stefan wants partly
to implement this feature in a way which could not be extended to
interpreted code, I think.  He has not given any indication of having
thought about a mechanism to do this; just "we can think about that
later, perhaps".

We're at the stage where there's working code, which already works for
edebug, ERT backtraces, early-debug.el, and so on.  So why are we
debating abstract design principles?

> > > My intention is to handle _ALL_ cases, and you find this objectionable
> > > for some reason.

> > Because your focus on the fringe cases has negative impacts on the
> > more common case.

What are these negative impacts?

Stefan has alleged negative impacts, but has not done any measurements to
demonstrate them, as far as I can make out.

> And this is the usual outcome of trying to solve _ALL_ cases from the
> get-go, by focusing on the fringe ones.  Premature optimization and
> all that.

I haven't focussed on fringe cases.  I've focussed on getting a uniform
implementation without special cases.  It hasn't been easy, but I think
I've succeeded.

It would be nice if people actually tried out the code in practice before
criticising.  So far, there's no indication anybody has done this.

> Alan, I'd appreciate if you rethink your approach to the
> implementation with the above principles in mind.

I don't want to implement something which will only work most of the time
for the easy to implement cases, and which precludes extension to the
general case.  I'd welcome any suggestions for a mechanism which would be
capable of extension to the general case.

I think you've already decided not to merge feature/named-lambdas.  I'm
not surprised, but it's a shame.

Debugging Emacs Lisp is frequently a horrible experience, with partial
error messages which don't tell you exactly where an error occurred,
don't tell you which primitive triggered an error, and frequently give
truncated backtraces, if any at all, due to unwise condition-cases, and
so on.

One of these things is having meaningless anonymous entries in
backtraces and C-h v, with no connection back to the source code.
feature/named-lambdas fixes this.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Date: Thu, 2 Nov 2023 09:37:15 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 66750@debbugs.gnu.org,
>   acorallo@gnu.org, stefankangas@gmail.com, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> What Stefan seems to be proposing does not accord with his previous
> paragraph.  He is proposing an implementation which would preclude
> extension to what he calls the "less common" cases - things like working
> in edebug and interpreted code in general, amongst others.

I'm not sure your conclusion above is correct.  But even if it is,
whether we want such extensions should be the subject of a separate
dedicated discussion, after implementing what caters to the important
cases, if an implementation that caters to all of them is much more
complicated (which seems to be the case here).  There's no reason to
reject a much simpler implementation just because it doesn't support
110% of the cases.

> > Alan, we are in the engineering business, where designing for the most
> > common case, before adjusting the design in small ways for less common
> > cases, is one of the important principles.
> 
> Another principle is not closing off future options.

That is secondary to the above.  Future options might require a
complete re-implementation, so the fact that the current
implementation doesn't support those future extensions is not
necessarily a fatal flaw.  It is just one disadvantage, which should
be weighed against its advantages.

> We're at the stage where there's working code, which already works for
> edebug, ERT backtraces, early-debug.el, and so on.  So why are we
> debating abstract design principles?

Because having a working code doesn't necessarily mean we should be
happy about it.

> > And this is the usual outcome of trying to solve _ALL_ cases from the
> > get-go, by focusing on the fringe ones.  Premature optimization and
> > all that.
> 
> I haven't focussed on fringe cases.  I've focussed on getting a uniform
> implementation without special cases.

The point I'm trying to make is that in your attempt to find an
implementation that covers everything you entered the area of the
proverbial diminishing returns, and the result is problematic in
several aspects.  Your objections to alternative ideas seems to come
from the argument that those ideas will fail to support some rare
cases, but I consider rejection of ideas on these grounds as wrong and
even invalid, because it sacrifices too much for too little.

> It would be nice if people actually tried out the code in practice before
> criticising.  So far, there's no indication anybody has done this.

Why do we need to try the code before discussing its obvious
implications on Emacs?  The issues that Stefan is raising are crystal
clear from just reading the code.

> > Alan, I'd appreciate if you rethink your approach to the
> > implementation with the above principles in mind.
> 
> I don't want to implement something which will only work most of the time
> for the easy to implement cases, and which precludes extension to the
> general case.  I'd welcome any suggestions for a mechanism which would be
> capable of extension to the general case.

Once again, such ideas are definitely welcome, but they are not an
"over-my-dead-body" kind of requirement.

> I think you've already decided not to merge feature/named-lambdas.  I'm
> not surprised, but it's a shame.

I didn't yet make any decision, because I still hope you will agree
with at least some of the arguments.  Or at least agree to some kind
or compromise, even if you keep your opinions.

> Debugging Emacs Lisp is frequently a horrible experience, with partial
> error messages which don't tell you exactly where an error occurred,
> don't tell you which primitive triggered an error, and frequently give
> truncated backtraces, if any at all, due to unwise condition-cases, and
> so on.

I disagree with "frequently" and "horrible".  My experience is
different, although I do sometimes need to be "creative" to catch some
errors.  But the existing tools, including GDB (which you for some
reason seem to reject completely AFAU) and even printf-debugging
usually do the job for me.  So the current situation being described
in such apocalyptic terms is not something I can agree with.

> One of these things is having meaningless anonymous entries in
> backtraces and C-h v, with no connection back to the source code.
> feature/named-lambdas fixes this.

Won't jumping to the exact source line where the error happened fix
this better?  If I have a single significant gripe against Emacs Lisp
backtraces, it is that there's no way of jumping to the offending
source line in each stack frame, something that is very natural to
provide.  Anonymous lambdas come as a very distant second or third
problem, at least IME.

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Eli.

On Thu, Nov 02, 2023 at 12:09:28 +0200, Eli Zaretskii wrote:
> > Date: Thu, 2 Nov 2023 09:37:15 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 66750@debbugs.gnu.org,
> >   acorallo@gnu.org, stefankangas@gmail.com, acm@muc.de
> > From: Alan Mackenzie <acm@muc.de>

> > What Stefan seems to be proposing does not accord with his previous
> > paragraph.  He is proposing an implementation which would preclude
> > extension to what he calls the "less common" cases - things like working
> > in edebug and interpreted code in general, amongst others.

> I'm not sure your conclusion above is correct.  But even if it is,
> whether we want such extensions should be the subject of a separate
> dedicated discussion, after implementing what caters to the important
> cases, if an implementation that caters to all of them is much more
> complicated (which seems to be the case here).  There's no reason to
> reject a much simpler implementation just because it doesn't support
> 110% of the cases.

Implementing the "simpler" version which blocks extensions would be
wasted effort if we decide we want those extensions.  In practice, what
is likely to happen is that the extensions would never be implemented.
Surely these are central questions which should be answered before
embarking on a restricted implementation.  (OK, I can see the irony
here.)

> > > Alan, we are in the engineering business, where designing for the most
> > > common case, before adjusting the design in small ways for less common
> > > cases, is one of the important principles.

> > Another principle is not closing off future options.

> That is secondary to the above.  Future options might require a
> complete re-implementation, so the fact that the current
> implementation doesn't support those future extensions is not
> necessarily a fatal flaw.  It is just one disadvantage, which should
> be weighed against its advantages.

Perhaps a less heated comparison of feature/named-lambdas's advantages
with its disadvantages would be beneficial, here.

> > We're at the stage where there's working code, which already works for
> > edebug, ERT backtraces, early-debug.el, and so on.  So why are we
> > debating abstract design principles?

> Because having a working code doesn't necessarily mean we should be
> happy about it.

Really?  Would you be happier if this code didn't work, or didn't exist?

> > > And this is the usual outcome of trying to solve _ALL_ cases from the
> > > get-go, by focusing on the fringe ones.  Premature optimization and
> > > all that.

> > I haven't focussed on fringe cases.  I've focussed on getting a uniform
> > implementation without special cases.

> The point I'm trying to make is that in your attempt to find an
> implementation that covers everything you entered the area of the
> proverbial diminishing returns, and the result is problematic in
> several aspects.  Your objections to alternative ideas seems to come
> from the argument that those ideas will fail to support some rare
> cases, but I consider rejection of ideas on these grounds as wrong and
> even invalid, because it sacrifices too much for too little.

What would we be sacrificing if we were to merge this feature?

> > It would be nice if people actually tried out the code in practice before
> > criticising.  So far, there's no indication anybody has done this.

> Why do we need to try the code before discussing its obvious
> implications on Emacs?  The issues that Stefan is raising are crystal
> clear from just reading the code.

One of the issues that Stefan raised was that the defining symbol in
"thousands" of closures resulting from a single command would be an
unacceptable "waste" of storage.  I don't think this would be the case.
This is something capable of measurement, something I would expect Stefan
to do to back up his case.

> > > Alan, I'd appreciate if you rethink your approach to the
> > > implementation with the above principles in mind.

> > I don't want to implement something which will only work most of the time
> > for the easy to implement cases, and which precludes extension to the
> > general case.  I'd welcome any suggestions for a mechanism which would be
> > capable of extension to the general case.

> Once again, such ideas are definitely welcome, but they are not an
> "over-my-dead-body" kind of requirement.

Such ideas haven't been forthcoming in the last day or two.

> > I think you've already decided not to merge feature/named-lambdas.  I'm
> > not surprised, but it's a shame.

> I didn't yet make any decision, because I still hope you will agree
> with at least some of the arguments.  Or at least agree to some kind
> of compromise, even if you keep your opinions.

I don't think Stefan is talking about a compromise.  He's talking about
discarding my changes entirely, and starting again from scratch, working
to design principles and with design goals I disagree with.  How is that
a compromise?

Or had you in mind something less drastic?

> > Debugging Emacs Lisp is frequently a horrible experience, with partial
> > error messages which don't tell you exactly where an error occurred,
> > don't tell you which primitive triggered an error, and frequently give
> > truncated backtraces, if any at all, due to unwise condition-cases, and
> > so on.

> I disagree with "frequently" and "horrible".  My experience is
> different, although I do sometimes need to be "creative" to catch some
> errors.  But the existing tools, including GDB (which you for some
> reason seem to reject completely AFAU) and even printf-debugging
> usually do the job for me.  So the current situation being described
> in such apocalyptic terms is not something I can agree with.

I use gdb, not as fluently as you do, but I use it.  But debugging Lisp
with gdb doesn't seem natural and isn't easy.  And yes, I've put printf
(literally printf) into C code quite a bit, too.

A couple of days ago I got the error message:

    emacs-lisp/eieio.el:55:2: Error: Wrong type argument: listp, :autoload-end

At the indicated file position there was just a `require' form.  So there
was no information about where the error happened, what detected the
error, or what function or what variable gave or had the value
:autoload-end.  It says little more than "there was an error".  This is
what I mean by "horrible".

In this case I was able to sort it out relatively quickly from examining
my recent changes, but that's not always the case.  This is typical of
the quality of error messages which Emacs produces, in my experience.
I'd like to improve this.

> > One of these things is having meaningless anonymous entries in
> > backtraces and C-h v, with no connection back to the source code.
> > feature/named-lambdas fixes this.

> Won't jumping to the exact source line where the error happened fix
> this better?

It would be a different solution to a slightly different problem.  Just
seeing a file name and location is less informative than seeing the
defining symbol, and forces the user to take action to get more
information.  But I'm looking at putting this facility into the code too,
after comments from Mattias (in July) and Stefan.

> If I have a single significant gripe against Emacs Lisp backtraces, it
> is that there's no way of jumping to the offending source line in each
> stack frame, something that is very natural to provide.

This would be more difficult to implement.  We currently have in Emacs no 
connection between our execution point and our source code, beyond the
symbols with position used in compilation.

> Anonymous lambdas come as a very distant second or third problem, at
> least IME.

They've been a problem for me, at least.  That's why I set about solving
it.  And being of a lower priority is no reason to reject an existing
fix.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Date: Thu, 2 Nov 2023 11:52:45 +0000
> Cc: monnier@iro.umontreal.ca, 66750@debbugs.gnu.org, acorallo@gnu.org,
>   stefankangas@gmail.com, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I think you've already decided not to merge feature/named-lambdas.  I'm
> > > not surprised, but it's a shame.
> 
> > I didn't yet make any decision, because I still hope you will agree
> > with at least some of the arguments.  Or at least agree to some kind
> > of compromise, even if you keep your opinions.
> 
> I don't think Stefan is talking about a compromise.  He's talking about
> discarding my changes entirely, and starting again from scratch, working
> to design principles and with design goals I disagree with.  How is that
> a compromise?
> 
> Or had you in mind something less drastic?

How about if you propose a compromise with which you could live?

> A couple of days ago I got the error message:
> 
>     emacs-lisp/eieio.el:55:2: Error: Wrong type argument: listp, :autoload-end
> 
> At the indicated file position there was just a `require' form.  So there
> was no information about where the error happened, what detected the
> error, or what function or what variable gave or had the value
> :autoload-end.  It says little more than "there was an error".  This is
> what I mean by "horrible".

This is what I call "debugging".  By and off itself, such a situation
is not necessarily anywhere near "horrible".  For example, it could be
that in the 'require'd file you will easily find the reference to
:autoload-end.

I also don't necessarily see how this is relevant to the issue at
hand.

> > If I have a single significant gripe against Emacs Lisp backtraces, it
> > is that there's no way of jumping to the offending source line in each
> > stack frame, something that is very natural to provide.
> 
> This would be more difficult to implement.

Maybe so, but if your feature doesn't bring us closer to that goal,
then for me personally it is much less interesting.

> > Anonymous lambdas come as a very distant second or third problem, at
> > least IME.
> 
> They've been a problem for me, at least.  That's why I set about solving
> it.  And being of a lower priority is no reason to reject an existing
> fix.

You know very well that priority is not the reason we are arguing.
There's no need to bring up irrelevant side-tracking arguments, it
doesn't help in any discussion, and more so in this one.

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Eli.

On Thu, Nov 02, 2023 at 15:50:14 +0200, Eli Zaretskii wrote:
> > Date: Thu, 2 Nov 2023 11:52:45 +0000
> > Cc: monnier@iro.umontreal.ca, 66750@debbugs.gnu.org, acorallo@gnu.org,
> >   stefankangas@gmail.com, acm@muc.de
> > From: Alan Mackenzie <acm@muc.de>

> > > > I think you've already decided not to merge feature/named-lambdas.  I'm
> > > > not surprised, but it's a shame.

> > > I didn't yet make any decision, because I still hope you will agree
> > > with at least some of the arguments.  Or at least agree to some kind
> > > of compromise, even if you keep your opinions.

> > I don't think Stefan is talking about a compromise.  He's talking about
> > discarding my changes entirely, and starting again from scratch, working
> > to design principles and with design goals I disagree with.  How is that
> > a compromise?

> > Or had you in mind something less drastic?

> How about if you propose a compromise with which you could live?

That is difficult.  What you and Stefan are objecting to is not the minor
details of my patch, it's its very essence.  That essence is the
existence of the defining symbol, and the amendment of the three function
formats to accomodate it.  I could see myself replacing the defining
symbol with FILE:POSITION information, but I doubt that would make the
two of you happy enough.  Or, maybe put this defining symbol or F:P
information into the doc string somehow (including in the interpreted
format) like Stefan was suggesting recently.

> > A couple of days ago I got the error message:

> >     emacs-lisp/eieio.el:55:2: Error: Wrong type argument: listp, :autoload-end

> > At the indicated file position there was just a `require' form.  So there
> > was no information about where the error happened, what detected the
> > error, or what function or what variable gave or had the value
> > :autoload-end.  It says little more than "there was an error".  This is
> > what I mean by "horrible".

> This is what I call "debugging".  By and off itself, such a situation
> is not necessarily anywhere near "horrible".  For example, it could be
> that in the 'require'd file you will easily find the reference to
> :autoload-end.

Yes, but it takes an order of magnitude longer than if it had given the
position in the required file where the error happened and had told me
that it was a cadr operation which threw the error.  All these oders of
magnitude longer add up to hundreds of hours over the years.

> I also don't necessarily see how this is relevant to the issue at
> hand.

It's not.  Except it's all part of the same overriding topic, what I feel
to be the inadequacy of Emacs's Lisp debugging tools.

> > > If I have a single significant gripe against Emacs Lisp backtraces, it
> > > is that there's no way of jumping to the offending source line in each
> > > stack frame, something that is very natural to provide.

> > This would be more difficult to implement.

> Maybe so, but if your feature doesn't bring us closer to that goal,
> then for me personally it is much less interesting.

Maybe we could put no-ops into the byte compiled format, where each no-op
had a position argument.  But that would make Emacs a bit slower.  Or we
could add an extra "debugging" field to the structure which would contain
the position information.  As with lots of things, macros would
complicate such a scheme.  In the native compiled format, isn't there
already a standard way of writing this info?

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Eli Zaretskii]

> Date: Thu, 2 Nov 2023 15:55:22 +0000
> Cc: monnier@iro.umontreal.ca, 66750@debbugs.gnu.org, acorallo@gnu.org,
>   stefankangas@gmail.com, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> > How about if you propose a compromise with which you could live?
> 
> That is difficult.

Of course, it is.  But "digging in" your original position will never
allow us to make any progress here, when there are differences of
opinion.  So please try, however difficult it is.  You will need to
get creative.

> I could see myself replacing the defining symbol with FILE:POSITION
> information, but I doubt that would make the two of you happy
> enough.  Or, maybe put this defining symbol or F:P information into
> the doc string somehow (including in the interpreted format) like
> Stefan was suggesting recently.

My advice is to propose the alternatives which you could accept, and
then take it from there.  There's no need to try to decide up front
whether some compromise will or won't be acceptable by the other side.

> > Maybe so, but if your feature doesn't bring us closer to that goal,
> > then for me personally it is much less interesting.
> 
> Maybe we could put no-ops into the byte compiled format, where each no-op
> had a position argument.  But that would make Emacs a bit slower.  Or we
> could add an extra "debugging" field to the structure which would contain
> the position information.  As with lots of things, macros would
> complicate such a scheme.  In the native compiled format, isn't there
> already a standard way of writing this info?

I don't know, but Andrea probably will.

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> > Maybe so, but if your feature doesn't bring us closer to that goal,
>> > then for me personally it is much less interesting.
>> 
>> Maybe we could put no-ops into the byte compiled format, where each no-op
>> had a position argument.  But that would make Emacs a bit slower.  Or we
>> could add an extra "debugging" field to the structure which would contain
>> the position information.  As with lots of things, macros would
>> complicate such a scheme.  In the native compiled format, isn't there
>> already a standard way of writing this info?
>
> I don't know, but Andrea probably will.

In libgccjit one can append gcc_jit_locations [1] to store filename,
line and column of original code generating the statement/operator being
described.

These information are for native debuggers like gdb that parse object
files and the stack, so I think they will not help any internal Emacs
debugger.

Best Regards

  Andrea

[1]
<https://gcc.gnu.org/onlinedocs/jit/topics/locations.html#c.gcc_jit_location>

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Kangas]

Alan Mackenzie <acm@muc.de> writes:

> On Thu, Nov 02, 2023 at 15:50:14 +0200, Eli Zaretskii wrote:
>
>> How about if you propose a compromise with which you could live?
>
> That is difficult.

Thanks for persevering despite the difficulties.

I have just now reviewed this thread in full, and I reread also the
beginning of the discussion.  The contention here is not around the
general idea, which everyone seems to find more or less useful, but some
of the design decisions in the proposed implementation.

> I could see myself replacing the defining symbol with FILE:POSITION
> information, but I doubt that would make the two of you happy enough.
> Or, maybe put this defining symbol or F:P information into the doc
> string somehow (including in the interpreted format) like Stefan was
> suggesting recently.

If I understand you correctly here, you are willing to consider using
FILE+LINE+COL instead of defining symbol, and storing the information in
the docstring instead of the lambda form.  And if I understand Stefan
Monnier's argument correctly, it seems like he wouldn't object to that.

So why not proceed along those lines?

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan, Stefan and Eli.

On Thu, Nov 02, 2023 at 14:44:28 -0700, Stefan Kangas wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Thu, Nov 02, 2023 at 15:50:14 +0200, Eli Zaretskii wrote:

> >> How about if you propose a compromise with which you could live?

> > That is difficult.

> Thanks for persevering despite the difficulties.

:-)

> I have just now reviewed this thread in full, and I reread also the
> beginning of the discussion.  The contention here is not around the
> general idea, which everyone seems to find more or less useful, but some
> of the design decisions in the proposed implementation.

Or, more precisely, the design decisions in the actual implementation.

> > I could see myself replacing the defining symbol with FILE:POSITION
> > information, but I doubt that would make the two of you happy enough.
> > Or, maybe put this defining symbol or F:P information into the doc
> > string somehow (including in the interpreted format) like Stefan was
> > suggesting recently.

> If I understand you correctly here, you are willing to consider using
> FILE+LINE+COL instead of defining symbol, ....

or possibly as well as it.  Or possibly a character offset in the file.

> .... and storing the information in the docstring instead of the
> lambda form.

In the doc string _inside_ the lambda form.  Its first line could be,
for example, a space separated structured line containing a
characteristic introduction (like we have in .elc files), the source
file, line number, column number, maybe the buffer it came from
(optional), maybe a defining symbol (optional).  This line needn't be
displayed by the help system, and could be as long as needed.

> And if I understand Stefan Monnier's argument correctly, it seems like
> he wouldn't object to that.

I hope not.  It is more than a small compromise for me, since the info
will not be stored in RAM as I wanted, and also I'll have to start work
on it again from scratch (but with the benefit of experience from doing
it the first time).  It is a compromise for Stefan M and Eli, since it
would enable the mechanism to work for interpreted functions, too.

How do you feel about this, Stefan (M.)?

> So why not proceed along those lines?

I'm willing to do this.  I hope you (Eli) will find it OK, too.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> How do you feel about this, Stefan (M.)?

Seems like it would leave `lambda` as well as the closure
objects unchanged, so it looks good to me.

The one thing I'd point out is: try to pick a format for the "data in
docstring" that is easily/naturally extensible (contrary to what I did
for the arglists), so that when we get around to adding support for
things like debugging info we could add it there without having to
invent a new format.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Thu, Nov 02, 2023 at 23:20:36 -0400, Stefan Monnier wrote:
> > How do you feel about this, Stefan (M.)?

> Seems like it would leave `lambda` as well as the closure
> objects unchanged, so it looks good to me.

Thanks!

But it's not quite so simple as all that.  In order to get the doc
strings for lambdas into the .elc file, there'll have to be an
enhancement of the .elc format.  Currently, although doc strings for
defuns/demacros/etc. are stored as file name + offset, those for
lambdas (which are vanishingly rare at this point) are just stored
inline in the .elc, and would get loaded along with the lambdas.

> The one thing I'd point out is: try to pick a format for the "data in
> docstring" that is easily/naturally extensible (contrary to what I did
> for the arglists), so that when we get around to adding support for
> things like debugging info we could add it there without having to
> invent a new format.

I intend to go for simplicity here.  A signature at BOL (something like
what .elc files have) followed by space separated info fields in a fixed
order.  Empty fields would have adjacent spaces there.  Spaces inside
fields (and LFs) could be escaped with \.  The whole thing would be
terminated by the first (unescaped) LF.

As for debugging info in the doc string - well we can argue about that
when it comes up.  ;-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> But it's not quite so simple as all that.  In order to get the doc
> strings for lambdas into the .elc file, there'll have to be an
> enhancement of the .elc format.  Currently, although doc strings for
> defuns/demacros/etc. are stored as file name + offset, those for
> lambdas (which are vanishingly rare at this point) are just stored
> inline in the .elc, and would get loaded along with the lambdas.

Yes, I know.  I think it's an orthogonal issue.  It's OK.

Note: I have(had?) a local patch to use (FILE . OFFSET) for local
lambdas' docstrings as well.  I never pushed it to `master` because it
relied on a change in the `prin1` function (basically provide some way
for `prin1` to generate the "#$" string in its output) and I couldn't
think of a way to do that that wasn't too hackish to be worth the
trouble for the "vanishingly rare" local lambdas with docstrings.
[ And also, it broke the make-docfile code that scraped `.elc` files
  because it moved the docstrings a bit, but that's not an issue any more
  because we don't scrape `.elc` files any more.  ]

But if those become more common, the tradeoff would justify getting such
a change in `master` (especially since IIRC it simplifies `bytecomp.el`
a bit).

>> The one thing I'd point out is: try to pick a format for the "data in
>> docstring" that is easily/naturally extensible (contrary to what I did
>> for the arglists), so that when we get around to adding support for
>> things like debugging info we could add it there without having to
>> invent a new format.
>
> I intend to go for simplicity here.

+1

> A signature at BOL

+1

> followed by space separated info fields in a fixed order.

I'd have gone with "a `read`able sexp" so you don't need to write any
new parsing code and it naturally extends to "arbitrary" content.


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Fri, Nov 03, 2023 at 18:18:08 -0400, Stefan Monnier wrote:
> > But it's not quite so simple as all that.  In order to get the doc
> > strings for lambdas into the .elc file, there'll have to be an
> > enhancement of the .elc format.  Currently, although doc strings for
> > defuns/demacros/etc. are stored as file name + offset, those for
> > lambdas (which are vanishingly rare at this point) are just stored
> > inline in the .elc, and would get loaded along with the lambdas.

> Yes, I know.  I think it's an orthogonal issue.  It's OK.

> Note: I have(had?) a local patch to use (FILE . OFFSET) for local
> lambdas' docstrings as well.  I never pushed it to `master` because it
> relied on a change in the `prin1` function (basically provide some way
> for `prin1` to generate the "#$" string in its output) and I couldn't
> think of a way to do that that wasn't too hackish to be worth the
> trouble for the "vanishingly rare" local lambdas with docstrings.
> [ And also, it broke the make-docfile code that scraped `.elc` files
>   because it moved the docstrings a bit, but that's not an issue any more
>   because we don't scrape `.elc` files any more.  ]

> But if those become more common, the tradeoff would justify getting such
> a change in `master` (especially since IIRC it simplifies `bytecomp.el`
> a bit).

I've refactored and cleaned up that bit of bytecomp.el quite a bit.  It
should be obvious from the patch how I'm intending to proceed with the
internal lambdas.

In particular, I've separated byte-compile-output-docform into two
pieces, one of which will be recursively callable for the lambdas.
b-c-o-d is now called all the time, not just for forms with doc strings.
I've extracted the middle element out of the INFO parameter, and called
it DOCINDEX.  The trailing ")" is now inserted in b-c-o-d rather than
b-c-file-form-defmumble.  There are probably one or two other clean ups.

Please see the patch below.  On running make bootstrap and make check,
there are no successes.  I think it's suitable for master.

> >> The one thing I'd point out is: try to pick a format for the "data in
> >> docstring" that is easily/naturally extensible (contrary to what I did
> >> for the arglists), so that when we get around to adding support for
> >> things like debugging info we could add it there without having to
> >> invent a new format.

> > I intend to go for simplicity here.

> +1

> > A signature at BOL

> +1

> > followed by space separated info fields in a fixed order.

> I'd have gone with "a `read`able sexp" so you don't need to write any
> new parsing code and it naturally extends to "arbitrary" content.

That's a great idea!  Thanks!



diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index cc68db73c9f..cfa80eb355c 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2477,10 +2477,9 @@ byte-compile-output-file-form
         (print-quoted t)
         (print-gensym t)
         (print-circle t))               ; Handle circular data structures.
-    (if (and (memq (car-safe form) '(defvar defvaralias defconst
-                                      autoload custom-declare-variable))
-             (stringp (nth 3 form)))
-        (byte-compile-output-docform nil nil '("\n(" 3 ")") form nil
+    (if (memq (car-safe form) '(defvar defvaralias defconst
+                                 autoload custom-declare-variable))
+        (byte-compile-output-docform nil nil nil '("\n(" ")") form 3 nil
                                      (memq (car form)
                                            '(defvaralias autoload
                                               custom-declare-variable)))
@@ -2490,10 +2489,83 @@ byte-compile-output-file-form
 
 (defvar byte-compile--for-effect)
 
-(defun byte-compile-output-docform (preface name info form specindex quoted)
+(defun byte-compile--output-docform-recurse
+    (info position form docindex specindex quoted)
+  "Print a form with a doc string.  INFO is (prefix postfix).
+POSITION is where the next doc string is to be inserted.
+DOCINDEX is the index of the doc string (or nil) in the FORM.
+If SPECINDEX is non-nil, it is the index in FORM
+of the function bytecode string.  In that case,
+we output that argument and the following argument
+\(the constants vector) together, for lazy loading.
+QUOTED says that we have to put a quote before the
+list that represents a doc string reference.
+`defvaralias', `autoload' and `custom-declare-variable' need that.
+
+Return the position after any inserted docstrings as comments."
+  (let ((index 0)
+        doc-string-position)
+    ;; Insert the doc string, and make it a comment with #@LENGTH.
+    (when (and byte-compile-dynamic-docstrings
+               (stringp (nth docindex form)))
+      (goto-char position)
+      (setq doc-string-position
+            (byte-compile-output-as-comment
+             (nth docindex form) nil)
+            position (point))
+      (goto-char (point-max)))
+
+    (insert (car info))
+    (prin1 (car form) byte-compile--outbuffer)
+    (while (setq form (cdr form))
+      (setq index (1+ index))
+      (insert " ")
+      (cond ((and (numberp specindex) (= index specindex)
+                  ;; Don't handle the definition dynamically
+                  ;; if it refers (or might refer)
+                  ;; to objects already output
+                  ;; (for instance, gensyms in the arg list).
+                  (let (non-nil)
+                    (when (hash-table-p print-number-table)
+                      (maphash (lambda (_k v) (if v (setq non-nil t)))
+                               print-number-table))
+                    (not non-nil)))
+             ;; Output the byte code and constants specially
+             ;; for lazy dynamic loading.
+             (goto-char position)
+             (let ((lazy-position (byte-compile-output-as-comment
+                                   (cons (car form) (nth 1 form))
+                                   t)))
+               (setq position (point))
+               (goto-char (point-max))
+               (princ (format "(#$ . %d) nil" lazy-position)
+                      byte-compile--outbuffer)
+               (setq form (cdr form))
+               (setq index (1+ index))))
+            ((= index docindex)
+             (cond
+              (doc-string-position
+               (princ (format (if quoted "'(#$ . %d)"  "(#$ . %d)")
+                              doc-string-position)
+                      byte-compile--outbuffer))
+              ((nth docindex form)
+               (let ((print-escape-newlines nil))
+                 (goto-char (prog1 (1+ (point))
+                              (prin1 (car form)
+                                     byte-compile--outbuffer)))
+                 (insert "\\\n")
+                 (goto-char (point-max))))
+              (t (insert "nil"))))
+            (t (prin1 (car form) byte-compile--outbuffer))))
+    (insert (cadr info))
+    position))
+
+(defun byte-compile-output-docform (preface tailpiece name info form docindex
+                                            specindex quoted)
   "Print a form with a doc string.  INFO is (prefix doc-index postfix).
-If PREFACE and NAME are non-nil, print them too,
-before INFO and the FORM but after the doc string itself.
+If PREFACE, NAME, and TAILPIECE are non-nil, print them too,
+before/after INFO and the FORM but after the doc string itself.
+DOCINDEX is the index of the doc string (or nil) in the FORM.
 If SPECINDEX is non-nil, it is the index in FORM
 of the function bytecode string.  In that case,
 we output that argument and the following argument
@@ -2503,73 +2575,30 @@ byte-compile-output-docform
 `defvaralias', `autoload' and `custom-declare-variable' need that."
   ;; We need to examine byte-compile-dynamic-docstrings
   ;; in the input buffer (now current), not in the output buffer.
-  (let ((dynamic-docstrings byte-compile-dynamic-docstrings))
+  (let ((byte-compile-dynamic-docstrings byte-compile-dynamic-docstrings))
     (with-current-buffer byte-compile--outbuffer
-      (let (position)
-        ;; Insert the doc string, and make it a comment with #@LENGTH.
-        (when (and (>= (nth 1 info) 0) dynamic-docstrings)
-          (setq position (byte-compile-output-as-comment
-                          (nth (nth 1 info) form) nil)))
-
-        (let ((print-continuous-numbering t)
-              print-number-table
-              (index 0)
-              ;; FIXME: The bindings below are only needed for when we're
-              ;; called from ...-defmumble.
-              (print-escape-newlines t)
-              (print-length nil)
-              (print-level nil)
-              (print-quoted t)
-              (print-gensym t)
-              (print-circle t))         ; Handle circular data structures.
-          (if preface
-              (progn
-                ;; FIXME: We don't handle uninterned names correctly.
-                ;; E.g. if cl-define-compiler-macro uses uninterned name we get:
-                ;;    (defalias '#1=#:foo--cmacro #[514 ...])
-                ;;    (put 'foo 'compiler-macro '#:foo--cmacro)
-                (insert preface)
-                (prin1 name byte-compile--outbuffer)))
-          (insert (car info))
-          (prin1 (car form) byte-compile--outbuffer)
-          (while (setq form (cdr form))
-            (setq index (1+ index))
-            (insert " ")
-            (cond ((and (numberp specindex) (= index specindex)
-                        ;; Don't handle the definition dynamically
-                        ;; if it refers (or might refer)
-                        ;; to objects already output
-                        ;; (for instance, gensyms in the arg list).
-                        (let (non-nil)
-                          (when (hash-table-p print-number-table)
-                            (maphash (lambda (_k v) (if v (setq non-nil t)))
-                                     print-number-table))
-                          (not non-nil)))
-                   ;; Output the byte code and constants specially
-                   ;; for lazy dynamic loading.
-                   (let ((position
-                          (byte-compile-output-as-comment
-                           (cons (car form) (nth 1 form))
-                           t)))
-                     (princ (format "(#$ . %d) nil" position)
-                            byte-compile--outbuffer)
-                     (setq form (cdr form))
-                     (setq index (1+ index))))
-                  ((= index (nth 1 info))
-                   (if position
-                       (princ (format (if quoted "'(#$ . %d)"  "(#$ . %d)")
-                                      position)
-                              byte-compile--outbuffer)
-                     (let ((print-escape-newlines nil))
-                       (goto-char (prog1 (1+ (point))
-                                    (prin1 (car form)
-                                           byte-compile--outbuffer)))
-                       (insert "\\\n")
-                       (goto-char (point-max)))))
-                  (t
-                   (prin1 (car form) byte-compile--outbuffer)))))
-        (insert (nth 2 info)))))
-  nil)
+      (let ((position (point))
+            (print-continuous-numbering t)
+            print-number-table
+            ;; FIXME: The bindings below are only needed for when we're
+            ;; called from ...-defmumble.
+            (print-escape-newlines t)
+            (print-length nil)
+            (print-level nil)
+            (print-quoted t)
+            (print-gensym t)
+            (print-circle t))       ; Handle circular data structures.
+        (when preface
+          ;; FIXME: We don't handle uninterned names correctly.
+          ;; E.g. if cl-define-compiler-macro uses uninterned name we get:
+          ;;    (defalias '#1=#:foo--cmacro #[514 ...])
+          ;;    (put 'foo 'compiler-macro '#:foo--cmacro)
+          (insert preface)
+          (prin1 name byte-compile--outbuffer))
+        (byte-compile--output-docform-recurse
+         info position form docindex specindex quoted)
+        (when tailpiece
+          (insert tailpiece))))))
 
 (defun byte-compile-keep-pending (form &optional handler)
   (if (memq byte-optimize '(t source))
@@ -2897,60 +2926,58 @@ byte-compile-file-form-defmumble
           ;; Otherwise, we have a bona-fide defun/defmacro definition, and use
           ;; special code to allow dynamic docstrings and byte-code.
           (byte-compile-flush-pending)
-          (let ((index
-                 ;; If there's no doc string, provide -1 as the "doc string
-                 ;; index" so that no element will be treated as a doc string.
-                 (if (not (stringp (documentation code t))) -1 4)))
-            (when byte-native-compiling
-              ;; Spill output for the native compiler here.
-              (push
-                (if macro
-                    (make-byte-to-native-top-level
-                     :form `(defalias ',name '(macro . ,code) nil)
-                     :lexical lexical-binding)
-                  (make-byte-to-native-func-def :name name
-                                                :byte-func code))
-                byte-to-native-top-level-forms))
-            ;; Output the form by hand, that's much simpler than having
-            ;; b-c-output-file-form analyze the defalias.
-            (byte-compile-output-docform
-             "\n(defalias '"
-             bare-name
-             (if macro `(" '(macro . #[" ,index "])") `(" #[" ,index "]"))
-             (append code nil)          ; Turn byte-code-function-p into list.
-             (and (atom code) byte-compile-dynamic
-                  1)
-             nil))
-          (princ ")" byte-compile--outbuffer)
+          (when byte-native-compiling
+            ;; Spill output for the native compiler here.
+            (push
+             (if macro
+                 (make-byte-to-native-top-level
+                  :form `(defalias ',name '(macro . ,code) nil)
+                  :lexical lexical-binding)
+               (make-byte-to-native-func-def :name name
+                                             :byte-func code))
+             byte-to-native-top-level-forms))
+          ;; Output the form by hand, that's much simpler than having
+          ;; b-c-output-file-form analyze the defalias.
+          (byte-compile-output-docform
+           "\n(defalias '" ")"
+           bare-name
+           (if macro '(" '(macro . #[" "])") '(" #[" "]"))
+           (append code nil)          ; Turn byte-code-function-p into list.
+           4
+           (and (atom code) byte-compile-dynamic 1)
+           nil)
           t)))))
 
 (defun byte-compile-output-as-comment (exp quoted)
-  "Print Lisp object EXP in the output file, inside a comment.
-Return the file (byte) position it will have.
-If QUOTED is non-nil, print with quoting; otherwise, print without quoting."
+  "Print Lisp object EXP in the output file at point, inside a comment.
+Return the file (byte) position it will have.  Leave point after
+the inserted text.  If QUOTED is non-nil, print with quoting;
+otherwise, print without quoting."
   (with-current-buffer byte-compile--outbuffer
-    (let ((position (point)))
-
+    (let ((position (point)) end)
       ;; Insert EXP, and make it a comment with #@LENGTH.
       (insert " ")
       (if quoted
           (prin1 exp byte-compile--outbuffer)
         (princ exp byte-compile--outbuffer))
+      (setq end (point-marker))
+      (set-marker-insertion-type end t)
+
       (goto-char position)
       ;; Quote certain special characters as needed.
       ;; get_doc_string in doc.c does the unquoting.
-      (while (search-forward "\^A" nil t)
+      (while (search-forward "\^A" end t)
         (replace-match "\^A\^A" t t))
       (goto-char position)
-      (while (search-forward "\000" nil t)
+      (while (search-forward "\000" end t)
         (replace-match "\^A0" t t))
       (goto-char position)
-      (while (search-forward "\037" nil t)
+      (while (search-forward "\037" end t)
         (replace-match "\^A_" t t))
-      (goto-char (point-max))
+      (goto-char end)
       (insert "\037")
       (goto-char position)
-      (insert "#@" (format "%d" (- (position-bytes (point-max))
+      (insert "#@" (format "%d" (- (position-bytes end)
                                    (position-bytes position))))
 
       ;; Save the file position of the object.
@@ -2959,7 +2986,8 @@ byte-compile-output-as-comment
       ;; position to a file position.
       (prog1
           (- (position-bytes (point)) (point-min) -1)
-        (goto-char (point-max))))))
+        (goto-char end)
+        (set-marker end nil)))))
 
 (defun byte-compile--reify-function (fun)
   "Return an expression which will evaluate to a function value FUN.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, Stefan.

On Fri, Nov 03, 2023 at 18:18:08 -0400, Stefan Monnier wrote:
> > But it's not quite so simple as all that.  In order to get the doc
> > strings for lambdas into the .elc file, there'll have to be an
> > enhancement of the .elc format.  Currently, although doc strings for
> > defuns/demacros/etc. are stored as file name + offset, those for
> > lambdas (which are vanishingly rare at this point) are just stored
> > inline in the .elc, and would get loaded along with the lambdas.

> Yes, I know.  I think it's an orthogonal issue.  It's OK.

[ .... ]

> But if those become more common, the tradeoff would justify getting such
> a change in `master` (especially since IIRC it simplifies `bytecomp.el`
> a bit).

I've committed a change to master which writes lambdas' doc strings as
comments, like main functions' doc strings, rather than inline.

[ .... ]


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I've committed a change to master which writes lambdas' doc strings as
> comments, like main functions' doc strings, rather than inline.

Thanks, Alan,


        Stefan

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie]

Hello, everybody.

On Thu, Nov 02, 2023 at 23:20:36 -0400, Stefan Monnier wrote:
> > How do you feel about this, Stefan (M.)?

> Seems like it would leave `lambda` as well as the closure
> objects unchanged, so it looks good to me.

> The one thing I'd point out is: try to pick a format for the "data in
> docstring" that is easily/naturally extensible (contrary to what I did
> for the arglists), so that when we get around to adding support for
> things like debugging info we could add it there without having to
> invent a new format.

I've just committed a provisional fix to this bug in branch
feature/positioned-lambdas.

On starting Emacs and doing C-h v org-mode-hook RET, the following gets
displayed:


org-mode-hook is a variable defined in `org.el'.

Its value is shown below.

Not documented as a variable.

  This variable may be risky if used as a file-local variable.
  You can customize this variable.

Value:
({org-ctags.el 7605} #f(compiled-function () #<bytecode -0x15a0204bcec6752e>)
               {org-mouse.el 30108}
               #f(compiled-function () #<bytecode 0x1c77f77cde0ab8f0>) {org.el
               844741} #f(compiled-function () #<bytecode -0x2a221d361c9ad4a>)
               {ob-core.el 59418}
               #f(compiled-function () #<bytecode -0xa13ed1a788fcde6>)
               org-babel-result-hide-spec org-babel-hide-all-hashes)
Original value was nil


..  Each anonymous function is annotated with a buffer name and offset
which indicates the position of the source of the function.

Clearly there is room for improvement: The display is not well filled; a
file name may be better than a buffer name; line/column numbers may be
better than buffer offsets.  There is also a possibility for putting
buttons on the text fields to bring up the source code.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The way we do it for the arglist is hideous and a pain in the rear (I
> blame the maintainers who accepted my patch back then), so it would be
> an opportunity to "do it right" this time.

What is bad about it, and how do you propose it could be improved?

I'm asking because I've only run into minor issues with it, that have
been easy to work around.

bug#66750: help-split-fundoc (was: bug#66750: Unhelpful text in C-h v for variables with a lambda form as value)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The way we do it for the arglist is hideous and a pain in the rear (I
>> blame the maintainers who accepted my patch back then), so it would be
>> an opportunity to "do it right" this time.
> What is bad about it, and how do you propose it could be improved?

It shouldn't be hidden under a `help-` prefix.
`help-split-fundoc` should not return "either a cons or nil".
Things like that.
It's clunky.

Part of the problem is that arglists often need to be
manipulated/provided by macros whose output is a language that only
knows about docstrings and not arglists, so it necessarily exposes
this hack of storing the arglist inside the docstring.
It's somewhat difficult to make it much better without significant
changes elsewhere (like an extension to the syntax of `lambda` where we
can provide side info like arglists).

Maybe it would help to rename `function-documentation` to `function-info`
whose output is type shouldn't matter because we only access it via
other functions like `function-info-documentation`,
`function-info-arglist`, ...
And then provide a (function-info-join INFO TYPE VALUE) where type can
be `documentation` or `arglist` or ...

The function arglist thingy was originally designed so that function
arglists could be easily provided from the DEFUNs in the C code, so it
was important to use a syntax that was somewhat human-readable, but it's
not indispensable to use the same format in the DEFUNs as elsewhere, so
we could choose a less human-friendly encoding that's more robust and
easier to extend.


        Stefan