Obtaining a database of new functionality per Emacs version

Obtaining a database of new functionality per Emacs version

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

Hello everyone,

I'm looking into creating a tool helping me with ensuring that my
packages actually support the Emacs versions they claim to by
highlighting uses of unsupported functions/variables [1]. For example if
I were to commit myself on supporting Debian stable with its Emacs 26.1,
that tool would highlight uses of `with-suppressed-warnings`, a macro
introduced in Emacs 27.1. The first problem to tackle is that there
doesn't seem to be a ready-made database. I've identified the following
options for making my own:

- Relying on metadata from custom.el: Works on customizables only.
- Combing through NEWS files: Not machine-readable, tedious, prone to
  mistakes (not all new functionality is marked as such, for example
  there's renames like `with-connection-local-profiles` to
  `with-connection-local-variables`).
- Combing through CHANGELOG: Not machine-readable either, tedious, not
  cleanly separated by versions, seemingly the wrong data source to
  consult.
- Launching an Emacs process, loading up all functionality, dumping all
  symbols, diffing against the output of an Emacs process of an older
  version, filtering out symbols introduced by our own code: This might
  just work (loading up functionality has side effects), but surely
  there's a better way, right?

Anything obvious I'm overlooking? Should there perhaps be an effort
towards documenting versioned public API?

Vasilij

[1]: I realize that this is far from solving the whole problem as
incorrect uses might still slip through, for example APIs taking a
different amount of arguments. To detect these, CI is required. However
I believe there is still merit to a tool run before CI kicks in...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

> - Relying on metadata from custom.el: Works on customizables only.
> - Combing through NEWS files: Not machine-readable, tedious, prone to
>   mistakes (not all new functionality is marked as such, for example
>   there's renames like `with-connection-local-profiles` to
>   `with-connection-local-variables`).

The version info displayed in `C-h o` comes from the NEWS files.
It's indeed approximate but I've found it surprisingly usable so far.
For such uses (i.e. for human consumption), I've found the main problem
to be that it doesn't understand that vars&functions come from packages,
so when a new package gets added in Emacs-NN.MM it doesn't tell me that
the corresponding functions and vars appeared in that version :-(

Whichever solution you end up, I hope it can be used for `C-h o`
as well.


        Stefan

Re: Obtaining a database of new functionality per Emacs version

[Eli Zaretskii]

> Date: Mon, 7 Dec 2020 13:23:29 +0100
> From: Vasilij Schneidermann <mail@vasilij.de>
> 
> - Relying on metadata from custom.el: Works on customizables only.
> - Combing through NEWS files: Not machine-readable, tedious, prone to
>   mistakes (not all new functionality is marked as such, for example
>   there's renames like `with-connection-local-profiles` to
>   `with-connection-local-variables`).

Renames without leaving an alias behind can only be legitimately done
if the original variable was not yet released in any version;
otherwise it's a bug that should be reported and fixed ASAP.

> - Combing through CHANGELOG: Not machine-readable either, tedious, not
>   cleanly separated by versions, seemingly the wrong data source to
>   consult.
> - Launching an Emacs process, loading up all functionality, dumping all
>   symbols, diffing against the output of an Emacs process of an older
>   version, filtering out symbols introduced by our own code: This might
>   just work (loading up functionality has side effects), but surely
>   there's a better way, right?
> 
> Anything obvious I'm overlooking?

Did you try "make TAGS" in the top-level directory, followed by
searching the TAGS files?

> Should there perhaps be an effort towards documenting versioned
> public API?

Manually or automatically?  If manually, the result will be as
accurate and comprehensive as NEWS.  If automatically, please tell
what kind of implementation you have in mind.

Re: Obtaining a database of new functionality per Emacs version

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

> Renames without leaving an alias behind can only be legitimately done
> if the original variable was not yet released in any version;
> otherwise it's a bug that should be reported and fixed ASAP.

I've looked a bit more into it, my 26.3 checkout only has
`with-connection-local-profiles` and the 27.1 one only
`with-connection-local-variables`. On top of that it's not exactly a
rename as the function signature and behavior of the macro changed, so
that violates the assumption you can still use the 26.3 macro in 27.1.
I'll have to look deeper into this problem.

> Did you try "make TAGS" in the top-level directory, followed by
> searching the TAGS files?

Thanks, that's the kind of idea I'm looking for. I'd still need to diff
that output against that of an older Emacs version to find out what has
been removed/added between the releases, but that's fine. Another
problem is that it includes helper functions that aren't part of a
public API.

> Manually or automatically?  If manually, the result will be as
> accurate and comprehensive as NEWS.  If automatically, please tell
> what kind of implementation you have in mind.

Manually, similar to how version information is added to customizables.
The gain is having a machine-readable version you can consult from a
Lisp program. Who knows, maybe you could use it to make an Emacs Lisp
spec independent implementations could target...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Obtaining a database of new functionality per Emacs version

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

> The version info displayed in `C-h o` comes from the NEWS files.
> It's indeed approximate but I've found it surprisingly usable so far.

Hm, I couldn't find any evidence of it using it for functions, for
example `seq-first` which according to NEWS has been added in 27.1.

> Whichever solution you end up, I hope it can be used for `C-h o`
> as well.

I could imagine such a database to be included with Emacs as part of a
stable release.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Obtaining a database of new functionality per Emacs version

[Eli Zaretskii]

> Date: Mon, 7 Dec 2020 18:50:07 +0100
> From: Vasilij Schneidermann <mail@vasilij.de>
> Cc: emacs-devel@gnu.org
> 
> > Did you try "make TAGS" in the top-level directory, followed by
> > searching the TAGS files?
> 
> Thanks, that's the kind of idea I'm looking for. I'd still need to diff
> that output against that of an older Emacs version to find out what has
> been removed/added between the releases, but that's fine. Another
> problem is that it includes helper functions that aren't part of a
> public API.

I thought the latter problem will not be an issue in your case, since
my mental model of what you want was that you have a function in hand
and want to check whether it was available in Emacs XX.YY.  In which
case you will never need to look for internal functions.

> > Manually or automatically?  If manually, the result will be as
> > accurate and comprehensive as NEWS.  If automatically, please tell
> > what kind of implementation you have in mind.
> 
> Manually, similar to how version information is added to customizables.

That needs Someone™ to be extra vigilant and double-check any changes
that add functions to prod people to mark them with a version tag.  My
experience with being that cop in defcustoms case is that the
probability of some falling through the cracks is non-negligible.

> The gain is having a machine-readable version you can consult from a
> Lisp program. Who knows, maybe you could use it to make an Emacs Lisp
> spec independent implementations could target...

I know of libraries that have this as part of their SOPs, but Emacs is
so much larger than any library that I doubt if this scales well
enough.  So I don't believe such manual procedures will be reliable
enough in Emacs.  One possible idea is to have a Git commit hook that
would reject commits with new functions if they fail to include the
necessary tag.

Re: Obtaining a database of new functionality per Emacs version

[Eli Zaretskii]

> Date: Mon, 7 Dec 2020 19:11:13 +0100
> From: Vasilij Schneidermann <mail@vasilij.de>
> Cc: emacs-devel@gnu.org
> 
> > The version info displayed in `C-h o` comes from the NEWS files.
> > It's indeed approximate but I've found it surprisingly usable so far.
> 
> Hm, I couldn't find any evidence of it using it for functions, for
> example `seq-first` which according to NEWS has been added in 27.1.

Try some older function.  AFAICT, if you "C-h f" for a function that
was added in the release in which you do that, the version information
is not displayed, it is only displayed for functions introduced in
older versions.

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

>> The version info displayed in `C-h o` comes from the NEWS files.
>> It's indeed approximate but I've found it surprisingly usable so far.
> Hm, I couldn't find any evidence of it using it for functions, for
> example `seq-first` which according to NEWS has been added in 27.1.

I just tried it with Emacs's `master` and `C-h o seq-first RET` tells me:

    seq-first is a compiled Lisp function in ‘seq.el’.
    
    (seq-first SEQUENCE)
    
      Other relevant functions are documented in the sequence group.
      Probably introduced at or before Emacs version 27.1.
    
    Return the first element of SEQUENCE.


-- Stefan

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

>> > Manually or automatically?  If manually, the result will be as
>> > accurate and comprehensive as NEWS.  If automatically, please tell
>> > what kind of implementation you have in mind.
>> Manually, similar to how version information is added to customizables.
> That needs Someone™ to be extra vigilant and double-check any changes
> that add functions to prod people to mark them with a version tag.  My
> experience with being that cop in defcustoms case is that the
> probability of some falling through the cracks is non-negligible.

I also doubt we'll find someone willing to take on that job.
It's a big job and we've lived with it for so many years that it's hard
to convince oneself that it's worth the trouble.

I do think it'd be useful, but I think it can be made largely automatic,
which so it requires much less work and suffers from fewer errors.

>> The gain is having a machine-readable version you can consult from a
>> Lisp program. Who knows, maybe you could use it to make an Emacs Lisp
>> spec independent implementations could target...
> I know of libraries that have this as part of their SOPs, but Emacs is
> so much larger than any library that I doubt if this scales well
> enough.  So I don't believe such manual procedures will be reliable
> enough in Emacs.  One possible idea is to have a Git commit hook that
> would reject commits with new functions if they fail to include the
> necessary tag.

Also, if we want to have this as a "dependency checker", then it'll have
to deal not just with "core ELisp" but also with all the bundled
packages, which is quite different from the usual notion of "API version".

Maybe a script that scrapes the `DEFUN`s and `DEFVAR`s from the C code,
along with the `defun`s and `defvar`s from the Lisp code would be a good
start.  Of course, loading all the Elisp files and then dumping the
`obarray` would also be another valid alternative (but this one
requires building old versions of Emacs).

We could apply that to the source for various past Emacs versions to
build a prefix-tree indicating in which version each was introduced.
We could then compress this tree by "cutting the branches" as soon as
all the leaves below map to the same version.


        Stefan

Re: Obtaining a database of new functionality per Emacs version

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Vasilij Schneidermann <mail@vasilij.de>,  emacs-devel@gnu.org
> Date: Mon, 07 Dec 2020 15:08:42 -0500
> 
> Maybe a script that scrapes the `DEFUN`s and `DEFVAR`s from the C code,
> along with the `defun`s and `defvar`s from the Lisp code would be a good
> start.

That's "make TAGS", right?  Or, if you don't want the rest, you can
invoke etags to find only those constructs.

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

>> Maybe a script that scrapes the `DEFUN`s and `DEFVAR`s from the C code,
>> along with the `defun`s and `defvar`s from the Lisp code would be a good
>> start.
> That's "make TAGS", right?

Right, tho you'd still need to extract the corresponding info from the
TAGS files since we'd need it in a different form.


        Stefan

Re: Obtaining a database of new functionality per Emacs version

[Jean Louis]

* Vasilij Schneidermann <mail@vasilij.de> [2020-12-07 16:30]:
> Hello everyone,
> 
> I'm looking into creating a tool helping me with ensuring that my
> packages actually support the Emacs versions they claim to by
> highlighting uses of unsupported functions/variables [1]. 

If one could launch automatically various Emacs versions and obtain a
list of all functions and all variables, by also keeping those flags
if variable is obsolete or not, or function is obsolete or not, then
such lists could be exported for each Emacs version and their
differences easily found. Process could be automated and lists of
functions and variables could be included in Emacs package that could
help new packages to know if some function or variable is not
compatible with some versions.

Jean

Re: Obtaining a database of new functionality per Emacs version

[Eli Zaretskii]

> Date: Thu, 10 Dec 2020 10:31:18 +0300
> From: Jean Louis <bugs@gnu.support>
> 
> * Vasilij Schneidermann <mail@vasilij.de> [2020-12-07 16:30]:
> > Hello everyone,
> > 
> > I'm looking into creating a tool helping me with ensuring that my
> > packages actually support the Emacs versions they claim to by
> > highlighting uses of unsupported functions/variables [1]. 
> 
> If one could launch automatically various Emacs versions and obtain a
> list of all functions and all variables, by also keeping those flags
> if variable is obsolete or not, or function is obsolete or not, then
> such lists could be exported for each Emacs version and their
> differences easily found.

You do realize that Vasilij wanted to know ab out _all_ the functions,
not just those present in "emacs -Q"?  To do what he wanted, you'd
need to load all of the Emacs packages as well, one by one, taking
care to unload them or restart the session, to account for basic
incompatibilities between some of them.  I don't think this is a
practical solution.

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

> not just those present in "emacs -Q"?  To do what he wanted, you'd
> need to load all of the Emacs packages as well, one by one, taking
> care to unload them or restart the session, to account for basic
> incompatibilities between some of them.  I don't think this is a
> practical solution.

IIRC cus-dep.el does just that (and doesn't even bother to unload them
or use separate sessions), so maybe it's not that bad.


        Stefan

Re: Obtaining a database of new functionality per Emacs version

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On 12/10/20 at 11:47am, Stefan Monnier wrote:
> > not just those present in "emacs -Q"?  To do what he wanted, you'd
> > need to load all of the Emacs packages as well, one by one, taking
> > care to unload them or restart the session, to account for basic
> > incompatibilities between some of them.  I don't think this is a
> > practical solution.
> 
> IIRC cus-dep.el does just that (and doesn't even bother to unload them
> or use separate sessions), so maybe it's not that bad.

Last time I tried that, I ended up launching M-x dunnet as side effect
which kept my batch script from quitting, so I'd argue that it's not
good enough for programmatic use. But then, maybe cus-dep.el has some
secrets on how to avoid that interaction...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Obtaining a database of new functionality per Emacs version

[Stefan Monnier]

> Last time I tried that, I ended up launching M-x dunnet as side effect
> which kept my batch script from quitting, so I'd argue that it's not
> good enough for programmatic use.

Note that such behavior is considered as a bug.

> But then, maybe cus-dep.el has some
> secrets on how to avoid that interaction...

BTW, it looks like you're right: cus-dep.el doesn't actually load the
files, it only executes some of the forms it finds in the files
(`defcustom`, `defgroup`, ...).

In any case, I think for the job at hand, Eli's suggestion to take the
TAGs files as starting point sounds like the better idea since it should
make it much easier to collect information about old Emacsen (without
having to compile them successfully).


        Stefan