bug#72829: describe-function NEWS* scraper override

bug#72829: describe-function NEWS* scraper override

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

`describe-function` sometimes gives incorrect information about when certain functions were first introduced. NEWS.unknown can be used to fix some problems but it only works in one direction and is unable to help when a function name appears too early.

A robust solution would be to make NEWS* use a mark-up like @function{some-name} instead of just 'some-name' but meanwhile, here is a simple patch that replaces NEWS.unknown with a more structured file. This fixes the case for `always`.


[-- Attachment #2: 0001-Better-ad-hoc-Emacs-release-of-symbol-introduction-o.patch --]
[-- Type: application/octet-stream, Size: 5943 bytes --]

From bfa7f11acb8aeeeaa07af29ec18dd30f77e933b3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mattias=20Engdeg=C3=A5rd?= <mattiase@acm.org>
Date: Mon, 26 Aug 2024 17:18:25 +0200
Subject: [PATCH] Better ad-hoc Emacs release of symbol introduction override

The file etc/symbol-releases.eld now contains explicit version
information for selected symbols that our NEWS* scraper doesn't
resolve correctly.

* etc/NEWS.unknown: Remove this file, replaced with...
* etc/symbol-releases.eld: ...this new file.
* lisp/help-fns.el (help-fns--first-release-override)
(help-fns--mention-first-function-release)
(help-fns--mention-first-variable-release): New.
(help-fns--mention-first-release): Try the override information first
before scraping the NEWS* files.
---
 etc/NEWS.unknown        | 31 -------------------------------
 etc/symbol-releases.eld | 36 ++++++++++++++++++++++++++++++++++++
 lisp/help-fns.el        | 33 ++++++++++++++++++++++++++++-----
 3 files changed, 64 insertions(+), 36 deletions(-)
 delete mode 100644 etc/NEWS.unknown
 create mode 100644 etc/symbol-releases.eld

diff --git a/etc/NEWS.unknown b/etc/NEWS.unknown
deleted file mode 100644
index eafdc953cac..00000000000
--- a/etc/NEWS.unknown
+++ /dev/null
@@ -1,31 +0,0 @@
-This file contains mentions of functions and variables whose
-version of introduction would otherwise be guessed incorrectly
-by 'M-x describe-function'.
-
-Since much of early Emacs source history is lost, these versions are
-conservative estimates: the actual version of first appearance may very
-well be much earlier.
-
-* Changes in Emacs 19.7
-** 'defsubst'
-
-* Changes in Emacs 18.59
-** 'mark'
-
-* Changes in Emacs 13.8
-This may be the earliest surviving version with source code, although
-damaged.  See
-https://github.com/larsbrinkhoff/emacs-history/decuslib.com/decus/vax85b/gnuemax
-
-** 'nthcdr'
-** 'nreverse
-** 'let*'
-** 'rassq'
-** '>='
-** 'transpose-sexps'
-** 'buffer-modified-p'
-** 'current-column'
-** 'downcase'
-** 'previous-line'
-** 'catch', 'throw'
-** 'count-lines'
diff --git a/etc/symbol-releases.eld b/etc/symbol-releases.eld
new file mode 100644
index 00000000000..dc991ae5747
--- /dev/null
+++ b/etc/symbol-releases.eld
@@ -0,0 +1,36 @@
+;; Emacs versions when certain symbols and variables were first introduced,
+;; for use in `describe-function'.
+;;
+;; This file is used to explicitly override the heuristic scraping NEWS*
+;; files, when that would result in misleading information.
+;;
+;; It should contain a single list of (VERSION TYPE SYMBOL), where
+;; VERSION is the Emacs version when SYMBOL was introduced as a TYPE,
+;; TYPE being `fun' or `var'.
+
+(
+ ("28.1" fun always)
+
+;; Since much of early Emacs source history is lost, these versions are
+;; conservative estimates: the actual version of first appearance may very
+;; well be much earlier.
+;; 13.8 may be the earliest surviving version with source code, although
+;; damaged.  See
+;; https://github.com/larsbrinkhoff/emacs-history/decuslib.com/decus/vax85b/gnuemax
+
+ ("19.7" fun defsubst)
+ ("18.59" fun mark)
+ ("13.8" fun nthcdr)
+ ("13.8" fun nreverse)
+ ("13.8" fun let*)
+ ("13.8" fun rassq)
+ ("13.8" fun >=)
+ ("13.8" fun transpose-sexps)
+ ("13.8" fun buffer-modified-p)
+ ("13.8" fun current-column)
+ ("13.8" fun downcase)
+ ("13.8" fun previous-line)
+ ("13.8" fun catch)
+ ("13.8" fun throw)
+ ("13.8" fun count-lines)
+ )
diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index 8a2ae79736f..3d3b841d827 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -869,6 +869,21 @@ help-fns--first-release-regexp
     ))
 
 
+(defun help-fns--first-release-override (symbol type)
+  "The first release defining SYMBOL of TYPE, or nil.
+TYPE indicates the namespace and is `fun' or `var'."
+  (let* ((sym-rel-file (expand-file-name "symbol-releases.eld" data-directory))
+         (tuples
+          (with-temp-buffer
+            (ignore-errors
+              (insert-file-contents sym-rel-file)
+              (goto-char (point-min))
+              (read (current-buffer))))))
+    (unless (cl-every (lambda (x) (and (= (length x) 3) (stringp (car x))))
+                      tuples)
+      (error "Bad %s format" sym-rel-file))
+    (car (rassoc (list type symbol) tuples))))
+
 (defun help-fns--first-release (symbol)
   "Return the likely first release that defined SYMBOL, or nil."
   ;; Code below relies on the etc/NEWS* files.
@@ -949,16 +964,24 @@ help-fns--first-release
 ;;       (display-buffer (current-buffer)))))
 
 (add-hook 'help-fns-describe-function-functions
-          #'help-fns--mention-first-release)
+          #'help-fns--mention-first-function-release)
 (add-hook 'help-fns-describe-variable-functions
-          #'help-fns--mention-first-release)
-(defun help-fns--mention-first-release (object)
+          #'help-fns--mention-first-variable-release)
+
+(defun help-fns--mention-first-function-release (object)
+  (help-fns--mention-first-release object 'fun))
+
+(defun help-fns--mention-first-variable-release (object)
   ;; Don't output anything if we've already output the :version from
   ;; the `defcustom'.
   (unless (memq 'help-fns--customize-variable-version
                 help-fns--activated-functions)
-    (when-let ((first (and (symbolp object)
-                           (help-fns--first-release object))))
+    (help-fns--mention-first-release object 'var)))
+
+(defun help-fns--mention-first-release (object type)
+  (when (symbolp object)
+    (when-let ((first (or (help-fns--first-release-override object type)
+                          (help-fns--first-release object))))
       (with-current-buffer standard-output
         (insert (format "  Probably introduced at or before Emacs version %s.\n"
                         first))))))
-- 
2.32.0 (Apple Git-132)

bug#72829: describe-function NEWS* scraper override

[Eli Zaretskii]

> Cc: Stefan Monnier <monnier@iro.umontreal.ca>
> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Tue, 27 Aug 2024 13:35:41 +0200
> 
> `describe-function` sometimes gives incorrect information about when certain functions were first introduced. NEWS.unknown can be used to fix some problems but it only works in one direction and is unable to help when a function name appears too early.
> 
> A robust solution would be to make NEWS* use a mark-up like @function{some-name} instead of just 'some-name' but meanwhile, here is a simple patch that replaces NEWS.unknown with a more structured file. This fixes the case for `always`.

Thanks, I think I'm okay with this approach.

Should we perhaps mention this in some admin/notes/ files, like
perhaps the procedure described in make-tarball.txt?

Stefan, any comments?

bug#72829: describe-function NEWS* scraper override

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Stefan, any comments?

No, that looks fine to me.


        Stefan

bug#72829: describe-function NEWS* scraper override

[Mattias Engdegård]

31 aug. 2024 kl. 12.10 skrev Eli Zaretskii <eliz@gnu.org>:

> Should we perhaps mention this in some admin/notes/ files, like
> perhaps the procedure described in make-tarball.txt?

Not sure what to say and where so I'll leave that for others. This hack is a bit reactive in nature, but could become a little version database if we decided to enter new symbols here systematically.

Now pushed to master. Maybe it should be back-ported to emacs-30, maybe not.

bug#72829: describe-function NEWS* scraper override

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sat, 31 Aug 2024 19:13:48 +0200
> Cc: monnier@iro.umontreal.ca,
>  72829@debbugs.gnu.org
> 
> 31 aug. 2024 kl. 12.10 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> > Should we perhaps mention this in some admin/notes/ files, like
> > perhaps the procedure described in make-tarball.txt?
> 
> Not sure what to say and where so I'll leave that for others. This hack is a bit reactive in nature, but could become a little version database if we decided to enter new symbols here systematically.
> 
> Now pushed to master. Maybe it should be back-ported to emacs-30, maybe not.

No, it should stay on master.

bug#72829: describe-function NEWS* scraper override

[Stefan Kangas]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> `describe-function` sometimes gives incorrect information about when
> certain functions were first introduced. NEWS.unknown can be used to
> fix some problems but it only works in one direction and is unable to
> help when a function name appears too early.
>
> A robust solution would be to make NEWS* use a mark-up like
> @function{some-name} instead of just 'some-name' but meanwhile, here
> is a simple patch that replaces NEWS.unknown with a more structured
> file. This fixes the case for `always`.

This is a welcome change.

For symbols that are in symbol-releases.eld, which means we are actually
sure about the addition, shouldn't the help text be changed from

    Probably introduced at or before Emacs version XX.Y.

to something more like

    Added in Emacs version XX.Y.

?  I mean, we should be sure about what we put in that file, presumably.

Steve Purcell (in Cc) has been maintaining a relatively complete symbol
to version database here:

    https://github.com/purcell/package-lint/blob/master/data/stdlib-changes

Note that his version keeps track of also of `feature`s, and not just
additions but removals as well.  It would be nice if our version could
be extended to do the same.  Perhaps Steve has some code or ideas that
he would be willing to contribute.

bug#72829: describe-function NEWS* scraper override

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Note that his version keeps track of also of `feature`s, and not just
> additions but removals as well.  It would be nice if our version could
> be extended to do the same.  Perhaps Steve has some code or ideas that
> he would be willing to contribute.

Maybe we could offer some way to show all the matches in the various
NEWS files so the user can see all the changes to the calling
convention, the deprecations, etc...


        Stefan

bug#72829: describe-function NEWS* scraper override

[Eli Zaretskii]

> Cc: steve@sanityinc.com, Stefan Monnier <monnier@iro.umontreal.ca>
> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 31 Aug 2024 15:26:58 -0700
> 
> Steve Purcell (in Cc) has been maintaining a relatively complete symbol
> to version database here:
> 
>     https://github.com/purcell/package-lint/blob/master/data/stdlib-changes
> 
> Note that his version keeps track of also of `feature`s, and not just
> additions but removals as well.  It would be nice if our version could
> be extended to do the same.  Perhaps Steve has some code or ideas that
> he would be willing to contribute.

I'm not sure I understand how can removals help us in the Help
commands.  They could be used by a new command, which doesn't yet
exist, which would show when was a specified symbol removed and what
can be used in its stead, but that command needs to be written first,
I think (and I'm not sure it will be used enough to justify it, but
that's another matter).

bug#72829: describe-function NEWS* scraper override

[Mattias Engdegård]

1 sep. 2024 kl. 00.26 skrev Stefan Kangas <stefankangas@gmail.com>:

> For symbols that are in symbol-releases.eld, which means we are actually
> sure about the addition, shouldn't the help text be changed from
> 
>    Probably introduced at or before Emacs version XX.Y.
> 
> to something more like
> 
>    Added in Emacs version XX.Y.

That's how I did it first but then dropped it for simplicity, but it can certainly be added back in.
The file also contains information about some symbols whose first appearance is not exactly known but it hardly matters for non-historians.

bug#72829: describe-function NEWS* scraper override

[Steve Purcell]

Yep, happy to discuss. The case for removals is that some symbols were actually removed in one Emacs version and then re-added later.

My overall technique is to load everything that I can, dump all the symbols I found, and then diff those dumps between Emacs versions. Using my nix-emacs-ci builds makes this straightforward to automate.

Not all features shipped with an Emacs build can be loaded in that build, though, because some rely on native or optional features, e.g. Windows or Treesitter. As such, it’s hard to get an authoritative list of all the known symbols.

In practice, the lists I’ve generated have been good enough for package-lint, and I often use the `package-lint-describe-symbol-history` command when I’m curious about a given symbol.

-Steve