Proposal: Forwards-Compatibility Library for Emacs

Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 124 bytes --]


Hi,

over the last few days I have been working on a forwards-compatibility
library for Emacs, here is a first iteration:


[-- Attachment #2: compat.el --]
[-- Type: application/emacs-lisp, Size: 51650 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1384 bytes --]


The idea is to allow developers who don't want to break backwards
compatibility to use newer functionality that wasn't provided in older
versions of Emacs. This version tries to implement as much as possible
from Emacs 24.2 onwards.

By its very nature it is an intrusive package, as it defines functions,
macros and advice outside of the "namespace", but I don't see any way
around that if transparent compatibility is to be provided (anything
else would just replicate dash, s, f, ...).

As some of the functions are lisp reimplementations of core functions,
there exists the risk of a performance overhead. To minimize this, the
compatibility layer is only applied necessary: Ideally someone using
Emacs 28.1 should have a quasi-empty file, with all the definitions
byte-compiled away.

Eventually I would like to propose adding something like this to
ELPA. It would only makes sense, as more than a few functions were
copied verbatim or quasi-verbatim from Emacs itself.

There still is work to be done, before anything could be added to ELPA,
especially providing tests to ensure that the compatibility layer is
implemented correctly, and making sure that no functions are used that
break the compatibility promise.

So before I continue working on this, I would like to ask if there is
any interest/there are any objections to providing such a library?

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Adam Porter]

Philip Kaludercic <philipk@posteo.net> writes:

> So before I continue working on this, I would like to ask if there is
> any interest/there are any objections to providing such a library?

I'd appreciate having this available.  As it is, when new features are
added to Emacs, it generally takes a few years, at least, before one can
use them in packages outside of Emacs, since old Emacs versions are used
in the wild for so long.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> So before I continue working on this, I would like to ask if there is
> any interest/there are any objections to providing such a library?

In the past, it was frowned upon to create these shims because of
"pollution" and "cleanliness" and all that sort of stuff -- when
something breaks, the question becomes "which version of `when-let*' was
that, then?" which most users aren't able to answer.

On a practical level, I think it's likely that people are using the
presence of certain features as a proxy for Emacs versions, and defining
a function like this might break some people's code.  (I.e., they're
testing for the presence of `when-let*' to see whether they should do
something totally unrelated that was introduced in a specific Emacs
version, like :extend the face.)

That said, it's just so much more practical for package authors to have
these compat shim libraries.  I naughtily had a gnus-compat.el library
in the external Gnus distribution for a decade that did pretty much what
you're doing here, and I can't recall anything breaking.

So I'm for it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> So before I continue working on this, I would like to ask if there is
>> any interest/there are any objections to providing such a library?
>
> In the past, it was frowned upon to create these shims because of
> "pollution" and "cleanliness" and all that sort of stuff -- when
> something breaks, the question becomes "which version of `when-let*' was
> that, then?" which most users aren't able to answer.

To avoid this issue, I intentionally never overwrote the actual
function, so it should be possible to remove the compatibility layer,
and secondly the docstring for every compatibility function starts with
a line indicating that it is in fact a compatibility function:

  [Compatibility function for ‘cdddr’, defined in Emacs 26.1]

> That said, it's just so much more practical for package authors to have
> these compat shim libraries.  I naughtily had a gnus-compat.el library
> in the external Gnus distribution for a decade that did pretty much what
> you're doing here, and I can't recall anything breaking.
>
> So I'm for it.

Great, I'll ping this thread when the package is ready.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Monnier]

> By its very nature it is an intrusive package, as it defines functions,
> macros and advice outside of the "namespace", but I don't see any way
> around that if transparent compatibility is to be provided (anything
> else would just replicate dash, s, f, ...).

I think this uncleanliness is a bit risky.  Better put the definitions
in their own namespace.  I don't think "transparent compatibility" is
worth the trouble here, and I don't think `s`, `f`, ... solve the
same problem.


        Stefan

Re: Proposal: Forwards-Compatibility Library for Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 21 Sep 2021 08:31:39 -0400
> 
> > By its very nature it is an intrusive package, as it defines functions,
> > macros and advice outside of the "namespace", but I don't see any way
> > around that if transparent compatibility is to be provided (anything
> > else would just replicate dash, s, f, ...).
> 
> I think this uncleanliness is a bit risky.  Better put the definitions
> in their own namespace.  I don't think "transparent compatibility" is
> worth the trouble here, and I don't think `s`, `f`, ... solve the
> same problem.

Can symbol-renaming help here, perhaps?  I believe João is trying to
get it ready in time for Emacs 28.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Can symbol-renaming help here, perhaps?  I believe João is trying to
> get it ready in time for Emacs 28.

This sounds potentially exciting, especially with the two recent
posts, one from Richard and one from Eli, mentioning this work.  The
last post on emacs-devel that I can find about "symbol renaming"
before that is from May 2020, and AFAICT it was buried in a
sub-thread.

Perhaps this would be a good time for a new post that explains the
current status of this, with a quick re-cap and perhaps even some
code?  (If there exists a branch, I can't find it.)

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> By its very nature it is an intrusive package, as it defines functions,
>> macros and advice outside of the "namespace", but I don't see any way
>> around that if transparent compatibility is to be provided (anything
>> else would just replicate dash, s, f, ...).
>
> I think this uncleanliness is a bit risky.  Better put the definitions
> in their own namespace.

Just to be on the same page, what exactly is the risk? I understand the
issue on a gut-level, it goes against convention and recommendations,
but the idea is that this package would do that, so that others don't
have to.

Other than that, all the functions and macros are defined first using a
"compat--" prefix, then perhaps are aliased to a symbol without a
prefix.

>  I don't think "transparent compatibility" is
> worth the trouble here, and I don't think `s`, `f`, ... solve the
> same problem.

Of course not, s, f, and the rest of the dash-adjacent ecosystem have
different intentions, it is only as a side effect that they provide more
functionality than older versions of Emacs do.

Most of the examples I provide in the file I attached above are simple
or slower reimplementations that make programming more convenient, but
where "transparent compatibility" becomes interesting is when providing
noop compatibility that allows package developers to make use of newer
features that are not backwards compatible, such as the additional
interactive argument. The infrastructure doesn't exist to handle this
information, but package developers will hold back from using these new
features because they don't want to abandon all users that aren't on
28.1.

I don't see how examples like these could be handled without transparent
compatibility.

>         Stefan

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 21 Sep 2021 08:31:39 -0400
>> 
>> > By its very nature it is an intrusive package, as it defines functions,
>> > macros and advice outside of the "namespace", but I don't see any way
>> > around that if transparent compatibility is to be provided (anything
>> > else would just replicate dash, s, f, ...).
>> 
>> I think this uncleanliness is a bit risky.  Better put the definitions
>> in their own namespace.  I don't think "transparent compatibility" is
>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>> same problem.
>
> Can symbol-renaming help here, perhaps?  I believe João is trying to
> get it ready in time for Emacs 28.

Going by my proposal, the library does nothing when loaded in Emacs
28.1, because it is already compatible to itself. No need for an extra
layer. It is only of interest to older versions, and they won't be able
to make use of symbol-renaming.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> ;; Is this excusable?
> ;;
> ;; ;;;; Ensure NonGNU ELPA is in `package-archives'
> ;; (eval-after-load "package"
> ;;   (lambda ()
> ;;     (unless (or (rassoc "https://elpa.nongnu.org/nongnu/" package-archives)
> ;; 		(rassoc "http://elpa.nongnu.org/nongnu/" package-archives)
> ;; 		(rassoc "https://elpa.nongnu.org/nongnu-devel/" package-archives)
> ;; 		(rassoc "http://elpa.nongnu.org/nongnu-devel/" package-archives))
> ;;       (add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/")))))

I also wanted to bring attention to this snippet, which if uncommented
might be a lot more controversial than the actual content of the library
itself. More generally, if a compatibility library were to be used more
widely, it would offer a possible route to update old defaults, such as
in this case, adding NonGNU ELPA to package-archives, or perhaps update
the default values of the IRC servers used by rcirc and ERC from
freenode to libera. If this could even be excused (and I get why people
could have issues with this), it would have to be used very
conservatively to not interfere with any existing configurations.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Tue, 21 Sep 2021 13:00:16 +0000
> Cc: emacs-devel@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>  João Távora <joaotavora@gmail.com>
> 
> > Can symbol-renaming help here, perhaps?  I believe João is trying to
> > get it ready in time for Emacs 28.
> 
> Going by my proposal, the library does nothing when loaded in Emacs
> 28.1, because it is already compatible to itself. No need for an extra
> layer. It is only of interest to older versions, and they won't be able
> to make use of symbol-renaming.

Why it is a problem for older versions to load symbol-renaming as
well?

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think this uncleanliness is a bit risky.  Better put the definitions
>> in their own namespace.  I don't think "transparent compatibility" is
>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>> same problem.
>
> Can symbol-renaming help here, perhaps?  I believe João is trying to
> get it ready in time for Emacs 28.

Confirmed.  I am working on the documentation and hope to provide
"hygienic", drop-in versions of those very `s.el`, `f.el` libraries.  I
agree with Stefan that those libraries don't seem to be solving the same
problem of forward-compatibility, though (at least how I've understood
it, which may very well be "not at all").

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Philip Kaludercic <philipk@posteo.net> writes:


> The idea is to allow developers who don't want to break backwards
> compatibility to use newer functionality that wasn't provided in older
> versions of Emacs. This version tries to implement as much as possible
> from Emacs 24.2 onwards.
>
> By its very nature it is an intrusive package, as it defines functions,
> macros and advice outside of the "namespace", but I don't see any way
> around that if transparent compatibility is to be provided (anything
> else would just replicate dash, s, f, ...).

Is there a problem if something replicates something else? :) If it is a good
thing, we just call it a "design pattern".

> As some of the functions are lisp reimplementations of core functions,
> there exists the risk of a performance overhead. To minimize this, the
> compatibility layer is only applied necessary: Ideally someone using
> Emacs 28.1 should have a quasi-empty file, with all the definitions
> byte-compiled away.
>
> Eventually I would like to propose adding something like this to
> ELPA. It would only makes sense, as more than a few functions were
> copied verbatim or quasi-verbatim from Emacs itself.

I had a similar idea once, and I think it definitely belongs to an external
package.

If you had this as an external library, than people who use older Emacs:es could
install compat-libs as a package, and in cases where there is a compatibility
function provided, they would just load a compat library after a package is
loaded.

Say I use dired-empty-p which is introuded in Emacs 28, in "some-program". A user
of some-program could then:

(with-eval-after-load 'some-program
  (require 'compat-lib-28))

I am just a bit concerned about all-in approach. As I udnerstand your code, it is
loaded as a library, and all code is loaded at once. So there might be code that
affects files and naming, or they might be coad that affects displayign of
windows, frames or code of handling user input etc. User might not want to load
everything and change behaviour of his/her entire Emacs. This might also lead to
unnexpected problems when there are 3rd party packages that rely on the old
behaviour.

I would rather prefer if compat-lib-28.el (or whatver called), would check on
it's features it overloads.

Easy case is wiith named features. It could check if user has loaded dired, and
load it's compatibility functions for dired that were introuded in emacs 28, and
it could also put itself in with-eval-after-load statement, so that if some
feature is loaded again, it loads itself. Harder is case when library does not
export a named feature, like files.el. 

That would provide for transparent "namespace" handling as you describe in your
next paragraph.

Of course, newly introduced functions are not the problem; I am concerned with
already existing functions, which might have changed slightly how they
function. I don't know if such exists, but I am thinking that some code could be
broken or still not function, despite function being advised to '28 version'.

I think it is important to not load and advise too much. If a program uses only
function foo, than there is no need to twinkle with function bar.

Another thing I was thinking of is use of advice. Advice is global, it means it
affects all users of the function. Isn't it better to define a buffer-local
variable, stash old function in the variable, and than call this buffer-local
from the compat function. That would also require compat layer to be a minor
mode, which is why I call it a layer rather than a library. User would have to
do slightly more:

(with-eval-after-load 'some-program
  (require 'compat-lib-28)
  (compat-mode +1))

That way the risk of conflicts when function has slightly change from it's
previous version is minimized since it will run only in that buffer, affecting
(hepofully) just 'some-program.

Or something ... I don't know; those were just thoughts, maybe I am wrong about.

> There still is work to be done, before anything could be added to ELPA,
> especially providing tests to ensure that the compatibility layer is
> implemented correctly, and making sure that no functions are used that
> break the compatibility promise.

Ert tests provided in version it provides compatibility for can always be used,
when there are some? 

> So before I continue working on this, I would like to ask if there is
> any interest/there are any objections to providing such a library?

Why should there be objections? It is free to write a library and give it to the
world, right?

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Tue, 21 Sep 2021 13:00:16 +0000
>> Cc: emacs-devel@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>>  João Távora <joaotavora@gmail.com>
>> 
>> > Can symbol-renaming help here, perhaps?  I believe João is trying to
>> > get it ready in time for Emacs 28.
>> 
>> Going by my proposal, the library does nothing when loaded in Emacs
>> 28.1, because it is already compatible to itself. No need for an extra
>> layer. It is only of interest to older versions, and they won't be able
>> to make use of symbol-renaming.
>
> Why it is a problem for older versions to load symbol-renaming as
> well?

My understanding was that this would depend on Emacs 28, but if not then
I retract my comment.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> I think this uncleanliness is a bit risky.  Better put the definitions
>>> in their own namespace.  I don't think "transparent compatibility" is
>>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>>> same problem.
>>
>> Can symbol-renaming help here, perhaps?  I believe João is trying to
>> get it ready in time for Emacs 28.
>
> Confirmed.  I am working on the documentation and hope to provide
> "hygienic", drop-in versions of those very `s.el`, `f.el` libraries.  I
> agree with Stefan that those libraries don't seem to be solving the same
> problem of forward-compatibility, though (at least how I've understood
> it, which may very well be "not at all").

Where can I find the code/examples on how this would look like?

> João

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> By its very nature it is an intrusive package, as it defines functions,
>>> macros and advice outside of the "namespace", but I don't see any way
>>> around that if transparent compatibility is to be provided (anything
>>> else would just replicate dash, s, f, ...).
>>
>> I think this uncleanliness is a bit risky.  Better put the definitions
>> in their own namespace.
>
> Just to be on the same page, what exactly is the risk? I understand the
> issue on a gut-level, it goes against convention and recommendations,
> but the idea is that this package would do that, so that others don't
> have to.

It is not on a "gut-level" :). I was thinking about same, and wrote you a longer
answer I just sent. There were no replys when I started, I see now several
people replied.

There can be two functions defineing slightly different behaviour in different
version, there can also be 3rd party packages expecting certain behaviour, and
there is no need to advise too much, say windowing code if application need to
just check is directory-empty-p.

> Other than that, all the functions and macros are defined first using a
> "compat--" prefix, then perhaps are aliased to a symbol without a
> prefix.
>
>>  I don't think "transparent compatibility" is
>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>> same problem.
>
> Of course not, s, f, and the rest of the dash-adjacent ecosystem have
> different intentions, it is only as a side effect that they provide more
> functionality than older versions of Emacs do.

They define different API from Emacs, more used by people comming from different
Lisp(s), clojure if I am correct?

> Most of the examples I provide in the file I attached above are simple
> or slower reimplementations that make programming more convenient, but
> where "transparent compatibility" becomes interesting is when providing
> noop compatibility that allows package developers to make use of newer
> features that are not backwards compatible, such as the additional
> interactive argument. The infrastructure doesn't exist to handle this
> information, but package developers will hold back from using these new
> features because they don't want to abandon all users that aren't on
> 28.1.
>
> I don't see how examples like these could be handled without transparent
> compatibility.

When it comes to providing newly intoduced functionality, there shouldn't be a
problem, it should always be possible to just add a function, like
dired-empty-p:


(when (version< emacs-version "28")
  (defun directory-empty-p (file-name)
    "Check if a directory contains any other files then dot-files"
    (when (file-directory-p file-name)
      (null (directory-files file-name nil
                             directory-files-no-dot-files-regexp t)))))

You don't even need to advise directory-files for that.

But for already existing functions that might lead to problems. I think so at
least. In the answer I sent you, I was thinking of buffer-local function
variables instead of advices which are global.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Stefan Kangas <stefan@marxist.se> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can symbol-renaming help here, perhaps?  I believe João is trying to
>> get it ready in time for Emacs 28.
>
> This sounds potentially exciting, especially with the two recent
> posts, one from Richard and one from Eli, mentioning this work.  The
> last post on emacs-devel that I can find about "symbol renaming"
> before that is from May 2020, and AFAICT it was buried in a
> sub-thread.
>
> Perhaps this would be a good time for a new post that explains the
> current status of this, with a quick re-cap and perhaps even some
> code?  (If there exists a branch, I can't find it.)

Yes, please.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Arthur Miller <arthur.miller@live.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> The idea is to allow developers who don't want to break backwards
>> compatibility to use newer functionality that wasn't provided in older
>> versions of Emacs. This version tries to implement as much as possible
>> from Emacs 24.2 onwards.
>>
>> By its very nature it is an intrusive package, as it defines functions,
>> macros and advice outside of the "namespace", but I don't see any way
>> around that if transparent compatibility is to be provided (anything
>> else would just replicate dash, s, f, ...).
>
> Is there a problem if something replicates something else? :) If it is a good
> thing, we just call it a "design pattern".

Just to clarify: My point is that I *don't* want to just propose a
different dash/s/f/... library, that consists of only a few new useful
functions and macros.

>> As some of the functions are lisp reimplementations of core functions,
>> there exists the risk of a performance overhead. To minimize this, the
>> compatibility layer is only applied necessary: Ideally someone using
>> Emacs 28.1 should have a quasi-empty file, with all the definitions
>> byte-compiled away.
>>
>> Eventually I would like to propose adding something like this to
>> ELPA. It would only makes sense, as more than a few functions were
>> copied verbatim or quasi-verbatim from Emacs itself.
>
> I had a similar idea once, and I think it definitely belongs to an external
> package.
>
> If you had this as an external library, than people who use older Emacs:es could
> install compat-libs as a package, and in cases where there is a compatibility
> function provided, they would just load a compat library after a package is
> loaded.
>
> Say I use dired-empty-p which is introuded in Emacs 28, in "some-program". A user
> of some-program could then:
>
> (with-eval-after-load 'some-program
>   (require 'compat-lib-28))

Why would a user do this?

> I am just a bit concerned about all-in approach. As I udnerstand your code, it is
> loaded as a library, and all code is loaded at once. So there might be code that
> affects files and naming, or they might be coad that affects displayign of
> windows, frames or code of handling user input etc. User might not want to load
> everything and change behaviour of his/her entire Emacs. This might also lead to
> unnexpected problems when there are 3rd party packages that rely on the old
> behaviour.

The audience of this library are developers, not (non-developer)
users. The point is not to make older versions of Emacs work like 28 or
newer, but to make code that targets Emacs 28 work with older versions
(insofar this is possible).

> I would rather prefer if compat-lib-28.el (or whatver called), would check on
> it's features it overloads.
>
> Easy case is wiith named features. It could check if user has loaded dired, and
> load it's compatibility functions for dired that were introuded in emacs 28, and
> it could also put itself in with-eval-after-load statement, so that if some
> feature is loaded again, it loads itself. Harder is case when library does not
> export a named feature, like files.el. 
>
> That would provide for transparent "namespace" handling as you describe in your
> next paragraph.
>
> Of course, newly introduced functions are not the problem; I am concerned with
> already existing functions, which might have changed slightly how they
> function. I don't know if such exists, but I am thinking that some code could be
> broken or still not function, despite function being advised to '28 version'.
>
> I think it is important to not load and advise too much. If a program uses only
> function foo, than there is no need to twinkle with function bar.

I agree, it might make sense to remove some of the compatibility
definitions, that just supply an additional argument. I'd argue there
are cases where it makes sense, such as assoc and the testfn argument,
but they have to be evaluated.

> Another thing I was thinking of is use of advice. Advice is global, it means it
> affects all users of the function. Isn't it better to define a buffer-local
> variable, stash old function in the variable, and than call this buffer-local
> from the compat function. That would also require compat layer to be a minor
> mode, which is why I call it a layer rather than a library. User would have to
> do slightly more:
>
> (with-eval-after-load 'some-program
>   (require 'compat-lib-28)
>   (compat-mode +1))
>
> That way the risk of conflicts when function has slightly change from it's
> previous version is minimized since it will run only in that buffer, affecting
> (hepofully) just 'some-program.
>
> Or something ... I don't know; those were just thoughts, maybe I am wrong about.

A different idea might be to try an integrate the compatibility layer
into the byte-compilation process for specific files. But once again,
this wouldn't be done by the user, but a (package) developer would
request for it to be done.

>> There still is work to be done, before anything could be added to ELPA,
>> especially providing tests to ensure that the compatibility layer is
>> implemented correctly, and making sure that no functions are used that
>> break the compatibility promise.
>
> Ert tests provided in version it provides compatibility for can always be used,
> when there are some? 

I know, I am currently working on writing a test suite with ert.

>> So before I continue working on this, I would like to ask if there is
>> any interest/there are any objections to providing such a library?
>
> Why should there be objections? It is free to write a library and give it to the
> world, right?

Yes, but I respect the opinion of the list and would like to add this to
ELPA, so good objections could change my mind that it is worthwhile to
implement something like this to the end.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

Stefan Kangas <stefan@marxist.se> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can symbol-renaming help here, perhaps?  I believe João is trying to
>> get it ready in time for Emacs 28.
>
> This sounds potentially exciting, especially with the two recent
> posts, one from Richard and one from Eli, mentioning this work.  The
> last post on emacs-devel that I can find about "symbol renaming"
> before that is from May 2020, and AFAICT it was buried in a
> sub-thread.

Yes, it was discussed in a subthread about namespacing.  I think I
outlined the main approach there, and pointed to some code once or
twice.  I had originally an Elisp-only version, but then crafted a C
version.

Symbol renaming, also known as "shorthands", is a namespacing system for
Elisp.  Like all namespacing systems, it's about referring to the same
thing by different terms.  Much like in this thread, if I write "Eli"
it's obvious who I'm referring to: I don't have to write the full "Eli
Zaretskii".  But if I write "Stefan", there is ambiguity.

Elisp shorthands were designed to disturb Elisp's existing practice of
one global obarray as little as possible, and so their implementation is
fairly short, and they work at the reader level.  As always, only one
obarray exists, and symbols are always "interned" there under their full
names.  However, the developer may setup certain shorthands so that she
can write i.e. "s-concat" in her file, while still having the symbol
interned as the proper, more hygienic, and much harder to type
"the-magnificent-string-library-concat".  This can be customized
per-file using a file-local variable not unlike lexical-binding.  There
is no difference in what kind of Elisp entity (function, variable, plain
symbol) the symbols refer to: Elisp shorthands are agnostic to that.  

I'm currently working on the documentation, which I hope to be able to
clear the most obvious/burning questions.  I ask you to hold them off
for some days to give me time to clear things up there (yes,
xref-find-definitions and eldoc are working :-) )

> Perhaps this would be a good time for a new post that explains the
> current status of this, with a quick re-cap and perhaps even some
> code?  (If there exists a branch, I can't find it.)

If you're very curious, you can find some code (and tests) in the branch
scratch/shorthand-namespacing, but that _isn't_ the latest version.
(though it _should_ be working).  Indeed I thought I had published the
most recent version to our Git repo, but it seems I haven't (which means
I must get back to my other machine as push it asap).

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> I think this uncleanliness is a bit risky.  Better put the definitions
>>>> in their own namespace.  I don't think "transparent compatibility" is
>>>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>>>> same problem.
>>>
>>> Can symbol-renaming help here, perhaps?  I believe João is trying to
>>> get it ready in time for Emacs 28.
>>
>> Confirmed.  I am working on the documentation and hope to provide
>> "hygienic", drop-in versions of those very `s.el`, `f.el` libraries.  I
>> agree with Stefan that those libraries don't seem to be solving the same
>> problem of forward-compatibility, though (at least how I've understood
>> it, which may very well be "not at all").
>
> Where can I find the code/examples on how this would look like?

See my reply in the side thread.  Thanks.

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Arthur Miller <arthur.miller@live.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> By its very nature it is an intrusive package, as it defines functions,
>>>> macros and advice outside of the "namespace", but I don't see any way
>>>> around that if transparent compatibility is to be provided (anything
>>>> else would just replicate dash, s, f, ...).
>>>
>>> I think this uncleanliness is a bit risky.  Better put the definitions
>>> in their own namespace.
>>
>> Just to be on the same page, what exactly is the risk? I understand the
>> issue on a gut-level, it goes against convention and recommendations,
>> but the idea is that this package would do that, so that others don't
>> have to.
>
> It is not on a "gut-level" :). I was thinking about same, and wrote you a longer
> answer I just sent. There were no replys when I started, I see now several
> people replied.
>
> There can be two functions defineing slightly different behaviour in different
> version, there can also be 3rd party packages expecting certain behaviour, and
> there is no need to advise too much, say windowing code if application need to
> just check is directory-empty-p.

Yes, I think cases where different versions switch back and forth are
hard to make compatible, so they should be left out of any attempt like
this.

>> Other than that, all the functions and macros are defined first using a
>> "compat--" prefix, then perhaps are aliased to a symbol without a
>> prefix.
>>
>>>  I don't think "transparent compatibility" is
>>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>>> same problem.
>>
>> Of course not, s, f, and the rest of the dash-adjacent ecosystem have
>> different intentions, it is only as a side effect that they provide more
>> functionality than older versions of Emacs do.
>
> They define different API from Emacs, more used by people comming from different
> Lisp(s), clojure if I am correct?

From what I know they are inspired by Clojure and try to provide better
compatibility with the various macros that dash provides (such as
threading), but I am not sure if they are direct translations from
Clojure's standard library.

>> Most of the examples I provide in the file I attached above are simple
>> or slower reimplementations that make programming more convenient, but
>> where "transparent compatibility" becomes interesting is when providing
>> noop compatibility that allows package developers to make use of newer
>> features that are not backwards compatible, such as the additional
>> interactive argument. The infrastructure doesn't exist to handle this
>> information, but package developers will hold back from using these new
>> features because they don't want to abandon all users that aren't on
>> 28.1.
>>
>> I don't see how examples like these could be handled without transparent
>> compatibility.
>
> When it comes to providing newly intoduced functionality, there shouldn't be a
> problem, it should always be possible to just add a function, like
> dired-empty-p:
>
>
> (when (version< emacs-version "28")
>   (defun directory-empty-p (file-name)
>     "Check if a directory contains any other files then dot-files"
>     (when (file-directory-p file-name)
>       (null (directory-files file-name nil
>                              directory-files-no-dot-files-regexp t)))))
>
> You don't even need to advise directory-files for that.

That wouldn't even be advised. My definition

    (compat-defun directory-empty-p (dir)
      "Return t if DIR names an existing directory containing no other files.
    Return nil if DIR does not name a directory, or if there was
    trouble determining whether DIR is a directory or empty.

    Symbolic links to directories count as directories.
    See `file-symlink-p' to distinguish symlinks."
      :version "28.1"
      (and (file-directory-p dir)
           (null (directory-files dir nil directory-files-no-dot-files-regexp t))))

is expanded to

    (progn
      (declare-function compat--directory-empty-p "compat"
                        '(dir))
      (let nil
        (defun compat--directory-empty-p
            (dir)
          "[Compatibility function for `directory-empty-p', defined in Emacs 28.1]\n\nReturn t if DIR names an existing directory containing no other files.\nReturn nil if DIR does not name a directory, or if there was\ntrouble determining whether DIR is a directory or empty.\n\nSymbolic links to directories count as directories.\nSee `file-symlink-p' to distinguish symlinks." nil
          (and
           (file-directory-p dir)
           (null
            (directory-files dir nil directory-files-no-dot-files-regexp t))))
        (compat-ignore
         (defalias 'directory-empty-p #'compat--directory-empty-p))))

which is more or less the same as what you do (setting aside the fact
that on my system, the installation via defalias is ignored).

> But for already existing functions that might lead to problems. I think so at
> least. In the answer I sent you, I was thinking of buffer-local function
> variables instead of advices which are global.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Philip Kaludercic <philipk@posteo.net> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> The idea is to allow developers who don't want to break backwards
>>> compatibility to use newer functionality that wasn't provided in older
>>> versions of Emacs. This version tries to implement as much as possible
>>> from Emacs 24.2 onwards.
>>>
>>> By its very nature it is an intrusive package, as it defines functions,
>>> macros and advice outside of the "namespace", but I don't see any way
>>> around that if transparent compatibility is to be provided (anything
>>> else would just replicate dash, s, f, ...).
>>
>> Is there a problem if something replicates something else? :) If it is a good
>> thing, we just call it a "design pattern".
>
> Just to clarify: My point is that I *don't* want to just propose a
> different dash/s/f/... library, that consists of only a few new useful
> functions and macros.
Yes I understand that your point wasn't to create another s.el, but I didn't
understand from your mail why would that be a bad thing :). I was just refering
to your rhetorics: what is wrong with s, dash? They do well what they do. That
does not mean that I suggest you should do another "s.el"; I was just trying to
talk about it in general. I don't 

By the way, uour library would not be "another s, dash ..." per definition
:). You don't do what they do; and  Can we talk about things in general? So I
don't see why bringing in s, dash and other in this as they were a bad thing?

>>> As some of the functions are lisp reimplementations of core functions,
>>> there exists the risk of a performance overhead. To minimize this, the
>>> compatibility layer is only applied necessary: Ideally someone using
>>> Emacs 28.1 should have a quasi-empty file, with all the definitions
>>> byte-compiled away.
>>>
>>> Eventually I would like to propose adding something like this to
>>> ELPA. It would only makes sense, as more than a few functions were
>>> copied verbatim or quasi-verbatim from Emacs itself.
>>
>> I had a similar idea once, and I think it definitely belongs to an external
>> package.
>>
>> If you had this as an external library, than people who use older Emacs:es could
>> install compat-libs as a package, and in cases where there is a compatibility
>> function provided, they would just load a compat library after a package is
>> loaded.
>>
>> Say I use dired-empty-p which is introuded in Emacs 28, in "some-program". A user
>> of some-program could then:
>>
>> (with-eval-after-load 'some-program
>>   (require 'compat-lib-28))
>
> Why would a user do this?
Because the user runs older Emacs and all the 3rd party packages that might be
affected by an advised function with possibly changed behaviour. So user can
verify it works or not and developer does not need to burden itself with
different execution paths, which in long run makes code that is easir to
understand and maintain.

>> I am just a bit concerned about all-in approach. As I udnerstand your code, it is
>> loaded as a library, and all code is loaded at once. So there might be code that
>> affects files and naming, or they might be coad that affects displayign of
>> windows, frames or code of handling user input etc. User might not want to load
>> everything and change behaviour of his/her entire Emacs. This might also lead to
>> unnexpected problems when there are 3rd party packages that rely on the old
>> behaviour.
>
> The audience of this library are developers, not (non-developer)
> users. The point is not to make older versions of Emacs work like 28 or
> newer, but to make code that targets Emacs 28 work with older versions
> (insofar this is possible).

Yes, I understood that, and the problem is not when  you provide new functions,
but when existing functions are slightly changed. As a developer it can be very
hard to know how it will affect some 3rd party package in some older
version. Developers would have to test in old versions as well etc. I would
really dislike Lisp code to look like C code we seen in Emacs src; those if-defs
are really nasty sometimes.

>> I would rather prefer if compat-lib-28.el (or whatver called), would check on
>> it's features it overloads.
>>
>> Easy case is wiith named features. It could check if user has loaded dired, and
>> load it's compatibility functions for dired that were introuded in emacs 28, and
>> it could also put itself in with-eval-after-load statement, so that if some
>> feature is loaded again, it loads itself. Harder is case when library does not
>> export a named feature, like files.el. 
>>
>> That would provide for transparent "namespace" handling as you describe in your
>> next paragraph.
>>
>> Of course, newly introduced functions are not the problem; I am concerned with
>> already existing functions, which might have changed slightly how they
>> function. I don't know if such exists, but I am thinking that some code could be
>> broken or still not function, despite function being advised to '28 version'.
>>
>> I think it is important to not load and advise too much. If a program uses only
>> function foo, than there is no need to twinkle with function bar.
>
> I agree, it might make sense to remove some of the compatibility
> definitions, that just supply an additional argument. I'd argue there
> are cases where it makes sense, such as assoc and the testfn argument,
> but they have to be evaluated.
>
>> Another thing I was thinking of is use of advice. Advice is global, it means it
>> affects all users of the function. Isn't it better to define a buffer-local
>> variable, stash old function in the variable, and than call this buffer-local
>> from the compat function. That would also require compat layer to be a minor
>> mode, which is why I call it a layer rather than a library. User would have to
>> do slightly more:
>>
>> (with-eval-after-load 'some-program
>>   (require 'compat-lib-28)
>>   (compat-mode +1))
>>
>> That way the risk of conflicts when function has slightly change from it's
>> previous version is minimized since it will run only in that buffer, affecting
>> (hepofully) just 'some-program.
>>
>> Or something ... I don't know; those were just thoughts, maybe I am wrong about.
>
> A different idea might be to try an integrate the compatibility layer
> into the byte-compilation process for specific files. But once again,
> this wouldn't be done by the user, but a (package) developer would
> request for it to be done.
Indeed, but sounds like a labourous thing to implement. You could also do
something similar if you checked at package installation what a packages uses
and installed a compat layer, but it seems like a lot of work and an unreliable
approach. Both approaches would have to patch code in old Emacs (byte compiler
or package.el). Would be cool thing to see if you manager ot pull it off, but I
think it is a lot of work.

>>> There still is work to be done, before anything could be added to ELPA,
>>> especially providing tests to ensure that the compatibility layer is
>>> implemented correctly, and making sure that no functions are used that
>>> break the compatibility promise.
>>
>> Ert tests provided in version it provides compatibility for can always be used,
>> when there are some? 
>
> I know, I am currently working on writing a test suite with ert.
>
>>> So before I continue working on this, I would like to ask if there is
>>> any interest/there are any objections to providing such a library?
>>
>> Why should there be objections? It is free to write a library and give it to the
>> world, right?
>
> Yes, but I respect the opinion of the list and would like to add this to
> ELPA, so good objections could change my mind that it is worthwhile to
> implement something like this to the end.

Yes of course. I am not sure if mine are very good, but, good luck, I hope you
get something that works.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Philip Kaludercic <philipk@posteo.net> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>
>>>>> By its very nature it is an intrusive package, as it defines functions,
>>>>> macros and advice outside of the "namespace", but I don't see any way
>>>>> around that if transparent compatibility is to be provided (anything
>>>>> else would just replicate dash, s, f, ...).
>>>>
>>>> I think this uncleanliness is a bit risky.  Better put the definitions
>>>> in their own namespace.
>>>
>>> Just to be on the same page, what exactly is the risk? I understand the
>>> issue on a gut-level, it goes against convention and recommendations,
>>> but the idea is that this package would do that, so that others don't
>>> have to.
>>
>> It is not on a "gut-level" :). I was thinking about same, and wrote you a longer
>> answer I just sent. There were no replys when I started, I see now several
>> people replied.
>>
>> There can be two functions defineing slightly different behaviour in different
>> version, there can also be 3rd party packages expecting certain behaviour, and
>> there is no need to advise too much, say windowing code if application need to
>> just check is directory-empty-p.
>
> Yes, I think cases where different versions switch back and forth are
> hard to make compatible, so they should be left out of any attempt like
> this.
>
>>> Other than that, all the functions and macros are defined first using a
>>> "compat--" prefix, then perhaps are aliased to a symbol without a
>>> prefix.
>>>
>>>>  I don't think "transparent compatibility" is
>>>> worth the trouble here, and I don't think `s`, `f`, ... solve the
>>>> same problem.
>>>
>>> Of course not, s, f, and the rest of the dash-adjacent ecosystem have
>>> different intentions, it is only as a side effect that they provide more
>>> functionality than older versions of Emacs do.
>>
>> They define different API from Emacs, more used by people comming from different
>> Lisp(s), clojure if I am correct?
>
> From what I know they are inspired by Clojure and try to provide better
> compatibility with the various macros that dash provides (such as
> threading), but I am not sure if they are direct translations from
> Clojure's standard library.
>
>>> Most of the examples I provide in the file I attached above are simple
>>> or slower reimplementations that make programming more convenient, but
>>> where "transparent compatibility" becomes interesting is when providing
>>> noop compatibility that allows package developers to make use of newer
>>> features that are not backwards compatible, such as the additional
>>> interactive argument. The infrastructure doesn't exist to handle this
>>> information, but package developers will hold back from using these new
>>> features because they don't want to abandon all users that aren't on
>>> 28.1.
>>>
>>> I don't see how examples like these could be handled without transparent
>>> compatibility.
>>
>> When it comes to providing newly intoduced functionality, there shouldn't be a
>> problem, it should always be possible to just add a function, like
>> dired-empty-p:
>>
>>
>> (when (version< emacs-version "28")
>>   (defun directory-empty-p (file-name)
>>     "Check if a directory contains any other files then dot-files"
>>     (when (file-directory-p file-name)
>>       (null (directory-files file-name nil
>>                              directory-files-no-dot-files-regexp t)))))
>>
>> You don't even need to advise directory-files for that.
>
> That wouldn't even be advised. My definition
>
>     (compat-defun directory-empty-p (dir)
>       "Return t if DIR names an existing directory containing no other files.
>     Return nil if DIR does not name a directory, or if there was
>     trouble determining whether DIR is a directory or empty.
>
>     Symbolic links to directories count as directories.
>     See `file-symlink-p' to distinguish symlinks."
>       :version "28.1"
>       (and (file-directory-p dir)
>            (null (directory-files dir nil directory-files-no-dot-files-regexp t))))
>
> is expanded to
>
>     (progn
>       (declare-function compat--directory-empty-p "compat"
>                         '(dir))
>       (let nil
>         (defun compat--directory-empty-p
>             (dir)
>           "[Compatibility function for `directory-empty-p', defined in Emacs 28.1]\n\nReturn t if DIR names an existing directory containing no other files.\nReturn nil if DIR does not name a directory, or if there was\ntrouble determining whether DIR is a directory or empty.\n\nSymbolic links to directories count as directories.\nSee `file-symlink-p' to distinguish symlinks." nil
>           (and
>            (file-directory-p dir)
>            (null
>             (directory-files dir nil directory-files-no-dot-files-regexp t))))
>         (compat-ignore
>          (defalias 'directory-empty-p #'compat--directory-empty-p))))
>
> which is more or less the same as what you do (setting aside the fact
> that on my system, the installation via defalias is ignored).
Ah, ok, sorry I wasn't running any code; just so the advised directory-files and
that directory-empty-p used one with the counter. My bad.

>> But for already existing functions that might lead to problems. I think so at
>> least. In the answer I sent you, I was thinking of buffer-local function
>> variables instead of advices which are global.

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Can symbol-renaming help here, perhaps?  I believe João is trying to
> get it ready in time for Emacs 28.

I'm not sure how symbol-renaming would help here?  The issue is
providing an "official" forwards-compatibility library that would define
all the useful library functions/macros that are added to Emacs so that
ELPA packages can use these new functions/macros immediately.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> By its very nature it is an intrusive package, as it defines functions,
>> macros and advice outside of the "namespace", but I don't see any way
>> around that if transparent compatibility is to be provided (anything
>> else would just replicate dash, s, f, ...).
>
> I think this uncleanliness is a bit risky.  Better put the definitions
> in their own namespace.

The point here is to not to have any separate namespace, so that you can
just can put core packages into ELPA without doing any rewrites.

What's the concrete risk here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Tue, Sep 21, 2021 at 4:02 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> The point here is to not to have any separate namespace, so that you can
> just can put core packages into ELPA without doing any rewrites.
>
> What's the concrete risk here?

It was Philip who described as "intrusive", "by its very nature",
since it relies on advice and such.  This can be seen as "dirty" in itself.
But say that the contract/promise that a given function in Emacs 28 makes
is different from the promise that the same function in Emacs 24.2 makes.
The new ELPA code doesn't have a problem, but you have potential problem
to all the other 24.2 code that expects the "old promise".  Right?

Also macros, defstructs and compile-time/run-time pain,
something-something, hand-wavy-wavy, makes my brain hurt :-)

> I'm not sure how symbol-renaming would help here?  The issue is
> providing an "official" forwards-compatibility library that would define
> all the useful library functions/macros that are added to Emacs so that
> ELPA packages can use these new functions/macros immediately.

Symbol renaming could theoretically help by having the new name,
say "overlays-at" (I think that's an example of a function that
was changed in meaning, may be wrong) be interned as
"overlays-at-in-the-emacs-28-sense".  The ELPA package author
would only write "overlays-at".  In that file the redirection would occur
when needed.  Everywhere else it would be the symbol interned
"overlays-at".

Alas, symbol renaming for older Emacsen is not in scope for my work
(at least right now), so this point may be moot.

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Tue, Sep 21, 2021 at 4:02 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> The point here is to not to have any separate namespace, so that you can
>> just can put core packages into ELPA without doing any rewrites.
>>
>> What's the concrete risk here?
>
> It was Philip who described as "intrusive", "by its very nature",
> since it relies on advice and such.  This can be seen as "dirty" in itself.
> But say that the contract/promise that a given function in Emacs 28 makes
> is different from the promise that the same function in Emacs 24.2 makes.
> The new ELPA code doesn't have a problem, but you have potential problem
> to all the other 24.2 code that expects the "old promise".  Right?

Yes, that is why full compatibility isn't possible, but on the other
hand, these kinds of changes are not that common to begin with. After
all, you don't want to break existing code, with or without a
compatibility library.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

Tue, Sep 21, 2021 at 5:10 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > On Tue, Sep 21, 2021 at 4:02 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> >
> >> The point here is to not to have any separate namespace, so that you can
> >> just can put core packages into ELPA without doing any rewrites.
> >>
> >> What's the concrete risk here?
> >
> > It was Philip who described as "intrusive", "by its very nature",
> > since it relies on advice and such.  This can be seen as "dirty" in itself.
> > But say that the contract/promise that a given function in Emacs 28 makes
> > is different from the promise that the same function in Emacs 24.2 makes.
> > The new ELPA code doesn't have a problem, but you have potential problem
> > to all the other 24.2 code that expects the "old promise".  Right?
>
> Yes, that is why full compatibility isn't possible, but on the other
> hand, these kinds of changes are not that common to begin with.

That's true, but they do happen. Emacs is not fully 100% backward
compatible.

I don't have the biggest of pictures in my head, but for me the solution
lies in providing more and more packages as :core.  In the Eglot circle,
for example, most of the flymake.el, project.el, xref.el, jsonrpc.el
dependencies are themselves :core ELPA packages.  So Emacs 26.3
users can use Eglot with all the latest and greatest that is in trunk,

Just my 2 cents,
João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

> It was Philip who described as "intrusive", "by its very nature",
> since it relies on advice and such.  This can be seen as "dirty" in itself.
> But say that the contract/promise that a given function in Emacs 28 makes
> is different from the promise that the same function in Emacs 24.2 makes.
> The new ELPA code doesn't have a problem, but you have potential problem
> to all the other 24.2 code that expects the "old promise".  Right?

Yes, some compatibility shims may be problematic -- especially if they
aren't bug-compatible.

But I think the general idea is good, as long as we don't go overboard.
That is, if a new library function is introduced in Emacs core, then
having it in compat.el is always fine.  Extending function that gain
more parameters, on the other hand, may or may not be problematic, and
should be considered on a case by case basis.

The "NO NON-NAMESPACED COMPAT FUNCTION" thing that became a part of
Emacs culture happened because in the 90s there were several problems in
this area.  One thing that happened was that several packages defined
the same names for functions that did totally different things.  Another
thing that happened was that "compat" libraries defined their own
versions of functions even if Emacs core defined the functions in
question.

Neither issue is a problem with the proposed compat library.  It's a
single, "officially blessed" library, and it doesn't redefine anything
that's already defined.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Tue, Sep 21, 2021 at 6:22 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > It was Philip who described as "intrusive", "by its very nature",
> > since it relies on advice and such.  This can be seen as "dirty" in itself.
> > But say that the contract/promise that a given function in Emacs 28 makes
> > is different from the promise that the same function in Emacs 24.2 makes.
> > The new ELPA code doesn't have a problem, but you have potential problem
> > to all the other 24.2 code that expects the "old promise".  Right?
>
> Yes, some compatibility shims may be problematic -- especially if they
> aren't bug-compatible.

There's some recent prior art to this.  cl-lib.el vs cl.el.  It worked
in principle
but making cl.el-expecting things to function just right with the new cl-lib.el
made Stefan sweat a lot, I think.  I'm glad that's mostly over (at
least in SLY).

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Monnier]

> The point here is to not to have any separate namespace, so that you can
> just can put core packages into ELPA without doing any rewrites.
> What's the concrete risk here?

I think in practice it's usually not a problem, but I'd be surprised if
there aren't counter examples.  So for a library that wants to do it in
a somewhat predictable/systematic way, I think it's much simple to have
all the definitions in a specific namespace (like `compat28-`), and then
it's a small matter of have a trivial `compat28.el` library that comes
with Emacs-28 which has a bunch of `defalias`.

The main advantage is that we then don't need to assess the possible
risks at all, so we're free to think about the things that actually matter.


        Stefan

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The point here is to not to have any separate namespace, so that you can
>> just can put core packages into ELPA without doing any rewrites.
>> What's the concrete risk here?
>
> I think in practice it's usually not a problem, but I'd be surprised if
> there aren't counter examples.  So for a library that wants to do it in
> a somewhat predictable/systematic way, I think it's much simple to have
> all the definitions in a specific namespace (like `compat28-`), and then
> it's a small matter of have a trivial `compat28.el` library that comes
> with Emacs-28 which has a bunch of `defalias`.

I am not sure how this would help provide forwards compatibility for
older versions? Are you proposing that instead of using
file-name-concat, libraries use compat28-file-name-concat that is
defined in a library for older versions?

My intuition would be that this wouldn't be worth the effort, seeing as
most people would probably hesitate to use such long names.

> The main advantage is that we then don't need to assess the possible
> risks at all, so we're free to think about the things that actually matter.
>
>
>         Stefan

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Monnier]

> I am not sure how this would help provide forwards compatibility for
> older versions? Are you proposing that instead of using
> file-name-concat, libraries use compat28-file-name-concat that is
> defined in a library for older versions?

Yes.

> My intuition would be that this wouldn't be worth the effort, seeing as
> most people would probably hesitate to use such long names.

If the function is important enough that the author would otherwise make
its own local function, then I think they'd be happy to use that
slightly longer name rather than having their own local definition.

If not, then it's probably just as well if they simply don't use it
(and that should avoid having the compatibility library accrue
too many definitions of too little value).


        Stefan

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The point here is to not to have any separate namespace, so that you can
>> just can put core packages into ELPA without doing any rewrites.
>> What's the concrete risk here?
>
> I think in practice it's usually not a problem, but I'd be surprised if
> there aren't counter examples.

We can just start slowly and see whether any problems occur.

> So for a library that wants to do it in
> a somewhat predictable/systematic way, I think it's much simple to have
> all the definitions in a specific namespace (like `compat28-`), and then
> it's a small matter of have a trivial `compat28.el` library that comes
> with Emacs-28 which has a bunch of `defalias`.

The main point here is to be able to put things that are in core into
ELPA, too.  We don't want to have code in core that goes

(compat28-when-let (...)
  )

so I don't quite understand what you're suggesting here.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Tue, Sep 21, 2021 at 10:57 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> We can just start slowly and see whether any problems occur.

> The main point here is to be able to put things that are in core into
> ELPA, too.

Right.  And so if this is the goal (and given that I also agree to
"start slow"), why not have more :core ELPA packages?
Say subr-x.el, as an example.  It has no dependencies.  Why
not make it a :core package right now?  Then other ELPA packages
could just rely on it.  Or pp.el, or any other util-rich package which
has no dependencies.

This seems more in line with current Emacs development and would
it not basically solve the same problem, just sectioning it by domain
instead?  It would have the added (big) benefit that you only have
one copy of each definition, not two (in core and in compat.el).

João Távora

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

> Right.  And so if this is the goal (and given that I also agree to
> "start slow"), why not have more :core ELPA packages?
> Say subr-x.el, as an example.  It has no dependencies.  Why
> not make it a :core package right now?

If you overwrite the subr-x functions from, say, Emacs 25.1 with the
ones in Emacs 28, you'll likely end up with an Emacs that doesn't work.

(And we certainly don't want to take on the maintenance burden of
writing subr-x in such a way that it's backwards-compatible.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The idea is to allow developers who don't want to break backwards
  > compatibility to use newer functionality that wasn't provided in older
  > versions of Emacs. This version tries to implement as much as possible
  > from Emacs 24.2 onwards.

Would you please state concretely what this package would do?

  > By its very nature it is an intrusive package, as it defines functions,
  > macros and advice outside of the "namespace", but I don't see any way
  > around that if transparent compatibility is to be provided (anything
  > else would just replicate dash, s, f, ...).

I have a plan to put those names into optional namespaces (using
symbol renaming) so that the entry points of those packages will
be visible only from packages that require specific libraries.

This plan might affect whether your intended feature is desirable or not.
I don't know whether it will do so.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Tue, Sep 21, 2021 at 11:16 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Right.  And so if this is the goal (and given that I also agree to
> > "start slow"), why not have more :core ELPA packages?
> > Say subr-x.el, as an example.  It has no dependencies.  Why
> > not make it a :core package right now?
>
> If you overwrite the subr-x functions from, say, Emacs 25.1 with the
> ones in Emacs 28, you'll likely end up with an Emacs that doesn't work.

Curious. Which functions in particular are you thinking about? That's
the aforementioned forward-compatibility problem (or backward,
depending on the perspective) that the compat.e library would also
have, right?

> (And we certainly don't want to take on the maintenance burden of
> writing subr-x in such a way that it's backwards-compatible.)

Then the same problem will happen when you copy these functions
into compat.el. Unless you do what Stefan says, and rename them.

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > The idea is to allow developers who don't want to break backwards
>   > compatibility to use newer functionality that wasn't provided in older
>   > versions of Emacs. This version tries to implement as much as possible
>   > from Emacs 24.2 onwards.
>
> Would you please state concretely what this package would do?

The library defines versions of newer functions if Emacs doesn't already
have them.  For instance, Emacs 28 has a new function
'buffer-local-boundp'.  Philip's library would provide a definition of
functions like that for use in ELPA code that's written for newer Emacs
versions, so that they can be used in older Emacs versions without
introducing compatibility shims like `foo-package-buffer-local-boundp'.

This also allows us to put (more) core packages into GNU ELPA without
any code changes.

>   > By its very nature it is an intrusive package, as it defines functions,
>   > macros and advice outside of the "namespace", but I don't see any way
>   > around that if transparent compatibility is to be provided (anything
>   > else would just replicate dash, s, f, ...).
>
> I have a plan to put those names into optional namespaces (using
> symbol renaming) so that the entry points of those packages will
> be visible only from packages that require specific libraries.

Philip shouldn't have mentioned s and dash and the rest -- his proposal
has absolutely nothing to do with those, but it seems like many people
have lashed onto that part, somehow.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

>> (And we certainly don't want to take on the maintenance burden of
>> writing subr-x in such a way that it's backwards-compatible.)
>
> Then the same problem will happen when you copy these functions
> into compat.el. Unless you do what Stefan says, and rename them.

No, the same problem doesn't exist in compat.el.  Philip's package
doesn't redefine functions in that way.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Tue, Sep 21, 2021 at 11:26 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> >> (And we certainly don't want to take on the maintenance burden of
> >> writing subr-x in such a way that it's backwards-compatible.)
> >
> > Then the same problem will happen when you copy these functions
> > into compat.el. Unless you do what Stefan says, and rename them.
>
> No, the same problem doesn't exist in compat.el.  Philip's package
> doesn't redefine functions in that way.

I see (I think).  Then the :core packages using the subr-x functions
must deal with slightly different specifications (if the 25.1 version indeed
isn't the same as the 28.1 version).  You called this bug-compatibility a while
ago. So is this a lesser-of-two-evils situation? Is it better to break the ELPA
usage this way then to break the libraries in core Emacs 25.1 as in your
example?

And which functions are we talking about again? Or were you just
speculating about possibilities?

João

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I am not sure how this would help provide forwards compatibility for
>> older versions? Are you proposing that instead of using
>> file-name-concat, libraries use compat28-file-name-concat that is
>> defined in a library for older versions?
>
> Yes.
>
>> My intuition would be that this wouldn't be worth the effort, seeing as
>> most people would probably hesitate to use such long names.
>
> If the function is important enough that the author would otherwise make
> its own local function, then I think they'd be happy to use that
> slightly longer name rather than having their own local definition.
>
> If not, then it's probably just as well if they simply don't use it
> (and that should avoid having the compatibility library accrue
> too many definitions of too little value).

This I'd describe as compatibility for convenience, but there are also
examples where core ELPA implement general algorithms and functionality
that could be used elsewhere too (project--buffers-to-kill and
project--kill-buffer-check were mentioned as one example last year). But
they cannot be factored out, because that would raise the minimal
required version. The complementary example are external packages that
hesitate to use newer functionality, for the same reason (I already gave
the example of the second optional "interactive" argument). The
infrastructure may not exist, but for anyone before Emacs 28, this could
just be ignored away, while newer users get to keep it.

>         Stefan
>

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Tue, Sep 21, 2021 at 10:57 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>> We can just start slowly and see whether any problems occur.
>
>> The main point here is to be able to put things that are in core into
>> ELPA, too.
>
> Right.  And so if this is the goal (and given that I also agree to
> "start slow"), why not have more :core ELPA packages?
> Say subr-x.el, as an example.  It has no dependencies.  Why
> not make it a :core package right now?  Then other ELPA packages
> could just rely on it.  Or pp.el, or any other util-rich package which
> has no dependencies.

This might work for subr-x and other elisp packages (in which case these
could just be added as dependencies to the effectively virtual package
compat).

On the other hand: If the functions are implemented in C, then they have
to be replicated some other way. And this doesn't just apply to
length{<,=,>}, string-distance, etc. but any function that might end up
using them down the call stack.

In principle, these two approaches can be merged, but I have the
impression that this would be more complicated

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[João Távora]

On Wed, Sep 22, 2021 at 7:53 AM Philip Kaludercic <philipk@posteo.net> wrote:

> This might work for subr-x and other elisp packages (in which case these
> could just be added as dependencies to the effectively virtual package
> compat).

Yes, correct.  So my next ELPA package would just Package-Require: the
ELPA package compat, which would bring in "lots of compatibility" (though
not 100% right away).

> On the other hand: If the functions are implemented in C, then they have
> to be replicated some other way. And this doesn't just apply to
> length{<,=,>}, string-distance, etc. but any function that might end up
> using them down the call stack.

Those could be isolated in some "cshims.el" package.  I don't know
if they would effectively be dependencies of lots of other packages.
It depends.

> In principle, these two approaches can be merged,

Yes, correct, the two approaches should be merged, and my feeling
is that your approach is useful mostly to fill the gaps where the current
one doesn't cut it.

> but I have the impression that this would be more complicated

Funny, I have the opposite feeling.  I think it would be simpler, since
for some high percentage of cases just Package-Require-ing some
:core packages by domain  would be enough for the Emacs versions
that you wish to support.  Certainly with Eglot, a fairly complex package
it has been (mostly) [*] the case, but then again Eglot only goes back to
26.3.

João

*: "mostly", because we indeed have our shim for "executable-find" there.
"executable-find" comes from "files.el", so maybe we could just
Package-Require it?

Re: Proposal: Forwards-Compatibility Library for Emacs

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> The point here is to not to have any separate namespace, so that you can
>>> just can put core packages into ELPA without doing any rewrites.
>>> What's the concrete risk here?
>>
>> I think in practice it's usually not a problem, but I'd be surprised if
>> there aren't counter examples.
>
> We can just start slowly and see whether any problems occur.
>
>> So for a library that wants to do it in
>> a somewhat predictable/systematic way, I think it's much simple to have
>> all the definitions in a specific namespace (like `compat28-`), and then
>> it's a small matter of have a trivial `compat28.el` library that comes
>> with Emacs-28 which has a bunch of `defalias`.
>
> The main point here is to be able to put things that are in core into
> ELPA, too.  We don't want to have code in core that goes
>
> (compat28-when-let (...)
>   )

Yes please, don't do that! :)

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>   > The idea is to allow developers who don't want to break backwards
>>   > compatibility to use newer functionality that wasn't provided in older
>>   > versions of Emacs. This version tries to implement as much as possible
>>   > from Emacs 24.2 onwards.
>>
>> Would you please state concretely what this package would do?

Functionality-wise: The package uses custom macros to define
"compatibility functions". A compatibility function implements at least
a subset of the behaviour the "real function" would provide, if it were
defined in whatever release of Emacs.

The macros it provides are:

- compat-defun

  Defines a compatibility function that is only aliased to the "real"
  symbol, if this is unbound and the specified version is greater than
  the current version. Example:

        (compat-defun always (&rest _arguments)
          "Do nothing and return t.
        This function accepts any number of ARGUMENTS, but ignores them.
        Also see `ignore'."
          :version "28.1"
          t)

  On Emacs 28.1 an newer, this just expands to `nil', because the plist
  after the documentation sting specifies that this function was defined
  in Emacs 28.1. Other attributes can be given such as part of what
  library this function was defined, so that it is wrapped in an
  eval-after-load body, preventing the developer from using the
  function, without loading the right library beforehand.
  
  Otherwise it defines

        (defun compat--always (&rest _)
          "[Compatibility function for `always', defined in Emacs 28.1]

        Do nothing and return t.
        This function accepts any number of ARGUMENTS, but ignores them.
        Also see `ignore'."
          t)

  followed by

        (unless (fboundp 'always)
          (defalias 'always #'compat-always))

- compat-macro

  Analogous to compat-defun, just for macros.

- compat-advise

  Defines an advice function that is then applied to wrap (ie. :around)
  the actual function. Example:

        (compat-advise sort (seq predicate)
          "Handle SEQ of type vector."
          :version "25.1"
          (cond
           ((listp seq)
            (funcall oldfun seq predicate))
           ((vectorp seq)
            (let ((cseq (sort (append seq nil) predicate)))
              (dotimes (i (length cseq))
                (setf (aref seq i) (nth i cseq)))
              (apply #'vector cseq)))
           ((signal 'wrong-type-argument 'list-or-vector-p))))

   Again, it defines a compat--sort function with the same body as
   compat-advise, and an additional argument "oldfun".

- compat-defvar

  Defines a variable, constant, buffer-local or permanent-local
  variable, again using a version attribute and boundp to check if it
  should do so.

I plan to add at least compat-defface, for defining faces that older
versions of Emacs didn't provide.

> The library defines versions of newer functions if Emacs doesn't already
> have them.  For instance, Emacs 28 has a new function
> 'buffer-local-boundp'.  Philip's library would provide a definition of
> functions like that for use in ELPA code that's written for newer Emacs
> versions, so that they can be used in older Emacs versions without
> introducing compatibility shims like `foo-package-buffer-local-boundp'.
>
> This also allows us to put (more) core packages into GNU ELPA without
> any code changes.

I am not sure if no-code-changes are always possible, but that would be
the best-case scenario.

>>   > By its very nature it is an intrusive package, as it defines functions,
>>   > macros and advice outside of the "namespace", but I don't see any way
>>   > around that if transparent compatibility is to be provided (anything
>>   > else would just replicate dash, s, f, ...).
>>
>> I have a plan to put those names into optional namespaces (using
>> symbol renaming) so that the entry points of those packages will
>> be visible only from packages that require specific libraries.
>
> Philip shouldn't have mentioned s and dash and the rest -- his proposal
> has absolutely nothing to do with those, but it seems like many people
> have lashed onto that part, somehow.

The reason I mentioned these libraries at all, is that for more than a
few package developers, I have noticed that they use these libraries for
the sake of convenience *and* backwards compatibility. As these
libraries are updated externally, they provide more easily usable
functionality that wasn't given in say Emacs 24.x. If they are
interested in convince and less repetitive programming, as I am sure
many are, they would have to choose between raising the minimal version
required to use their package or to add these external libraries.

My point in mentioning these was not to give a direct comparison, but
just to say that if compat.el didn't use transparent compatibility, it
would just become another utility function library -- just instead of
using a Clojure style of programming, it would stick to the more
traditional Elisp conventions.

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Monnier]

My conclusion is that it all depends and it can't really be decided in
the abstract.  So, if you show a prototype things will be much
more clear.


        Stefan


Philip Kaludercic [2021-09-22 06:48:30] wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I am not sure how this would help provide forwards compatibility for
>>> older versions? Are you proposing that instead of using
>>> file-name-concat, libraries use compat28-file-name-concat that is
>>> defined in a library for older versions?
>>
>> Yes.
>>
>>> My intuition would be that this wouldn't be worth the effort, seeing as
>>> most people would probably hesitate to use such long names.
>>
>> If the function is important enough that the author would otherwise make
>> its own local function, then I think they'd be happy to use that
>> slightly longer name rather than having their own local definition.
>>
>> If not, then it's probably just as well if they simply don't use it
>> (and that should avoid having the compatibility library accrue
>> too many definitions of too little value).
>
> This I'd describe as compatibility for convenience, but there are also
> examples where core ELPA implement general algorithms and functionality
> that could be used elsewhere too (project--buffers-to-kill and
> project--kill-buffer-check were mentioned as one example last year). But
> they cannot be factored out, because that would raise the minimal
> required version. The complementary example are external packages that
> hesitate to use newer functionality, for the same reason (I already gave
> the example of the second optional "interactive" argument). The
> infrastructure may not exist, but for anyone before Emacs 28, this could
> just be ignored away, while newer users get to keep it.
>
>>         Stefan
>>

Re: Proposal: Forwards-Compatibility Library for Emacs

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> My conclusion is that it all depends and it can't really be decided in
> the abstract.  So, if you show a prototype things will be much
> more clear.

I attached a prototype to the first message?

>         Stefan
>
>
> Philip Kaludercic [2021-09-22 06:48:30] wrote:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> I am not sure how this would help provide forwards compatibility for
>>>> older versions? Are you proposing that instead of using
>>>> file-name-concat, libraries use compat28-file-name-concat that is
>>>> defined in a library for older versions?
>>>
>>> Yes.
>>>
>>>> My intuition would be that this wouldn't be worth the effort, seeing as
>>>> most people would probably hesitate to use such long names.
>>>
>>> If the function is important enough that the author would otherwise make
>>> its own local function, then I think they'd be happy to use that
>>> slightly longer name rather than having their own local definition.
>>>
>>> If not, then it's probably just as well if they simply don't use it
>>> (and that should avoid having the compatibility library accrue
>>> too many definitions of too little value).
>>
>> This I'd describe as compatibility for convenience, but there are also
>> examples where core ELPA implement general algorithms and functionality
>> that could be used elsewhere too (project--buffers-to-kill and
>> project--kill-buffer-check were mentioned as one example last year). But
>> they cannot be factored out, because that would raise the minimal
>> required version. The complementary example are external packages that
>> hesitate to use newer functionality, for the same reason (I already gave
>> the example of the second optional "interactive" argument). The
>> infrastructure may not exist, but for anyone before Emacs 28, this could
>> just be ignored away, while newer users get to keep it.
>>
>>>         Stefan
>>>
>

-- 
	Philip Kaludercic

Re: Proposal: Forwards-Compatibility Library for Emacs

[Stefan Monnier]

> I attached a prototype to the first message?

The first message is *far*!
I'll have to reserve a car to get there ;-)


        Stefan

[ELPA] Add compat.el

[Philip Kaludercic]

Hello again,

I have been working on compat.el over the last week, and have written a
number of tests. The source code has been uploaded here:

       https://git.sr.ht/~pkal/compat

The library should be usable in it's current state, but I'd be
interested to hear from other people and developers too.  There might be
missing functions or macros, edge cases that could be added to the
tests, etc. that could be improved.

Finally, compat.el now ends with these lines:

--8<---------------cut here---------------start------------->8---
;; ;;;; Ensure NonGNU ELPA is in `package-archives'
;; (with-eval-after-load 'package
;;   (unless (or (rassoc "https://elpa.nongnu.org/nongnu/" package-archives)
;;               (rassoc "http://elpa.nongnu.org/nongnu/" package-archives)
;;               (rassoc "https://elpa.nongnu.org/nongnu-devel/" package-archives)
;;               (rassoc "http://elpa.nongnu.org/nongnu-devel/" package-archives))
;;     (add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/"))))

;; To ensure that compat.el is loaded as soon as possible, a require
;; call is inserted directly into the autoload file:
;;;###autoload (require 'compat)
--8<---------------cut here---------------end--------------->8---

The first block ensures that NonGNU ELPA is added to package-archives,
in the hope that if ELPA packages were to use compat.el, older versions
would automatically be configured to use NonGNU ELPA.  The upside is
wider adoption, the downside is that this is pretty invasive (an
alternative would be to check if package-archives is unmodified, and
only update it in that case).

The second block would automatically require compat.el in the autoload
file, so as to avoid manual (require 'compat) statements in packages
that use the library. I haven't tested this part extensively yet, but
could there be something that this might break?

Philip Kaludercic <philipk@posteo.net> writes:

> Hi,
>
> over the last few days I have been working on a forwards-compatibility
> library for Emacs, here is a first iteration:
>
> The idea is to allow developers who don't want to break backwards
> compatibility to use newer functionality that wasn't provided in older
> versions of Emacs. This version tries to implement as much as possible
> from Emacs 24.2 onwards.
>
> [...]

-- 
	Philip Kaludercic

Old (static) builds of Emacs

[Philip Kaludercic]

Until now I have only been able to test the library using Emacs 27.2 and
Emacs 28.0.50/60. This doesn't seem like enough for a library that
actually targets older versions of Emacs.

My question is therefore: Does anyone know where to download old Emacs
builds? I have tried building 24.x and 25.x on my system, but it seems
(?) that changes in glibc prevent this from being done.

So before I have to set up virtual machines with older distributions, I
was wondering if anyone knew of an archive with pre-build, preferably
static, Emacs builds.

Philip Kaludercic <philipk@posteo.net> writes:

> Hello again,
>
> I have been working on compat.el over the last week, and have written a
> number of tests. The source code has been uploaded here:
>
>        https://git.sr.ht/~pkal/compat
>
> The library should be usable in it's current state, but I'd be
> interested to hear from other people and developers too.  There might be
> missing functions or macros, edge cases that could be added to the
> tests, etc. that could be improved.

-- 
	Philip Kaludercic

Re: Old (static) builds of Emacs

[Michael Albinus]

Philip Kaludercic <philipk@posteo.net> writes:

Hi Philip,

> Until now I have only been able to test the library using Emacs 27.2 and
> Emacs 28.0.50/60. This doesn't seem like enough for a library that
> actually targets older versions of Emacs.
>
> My question is therefore: Does anyone know where to download old Emacs
> builds? I have tried building 24.x and 25.x on my system, but it seems
> (?) that changes in glibc prevent this from being done.
>
> So before I have to set up virtual machines with older distributions, I
> was wondering if anyone knew of an archive with pre-build, preferably
> static, Emacs builds.

There's <https://hub.docker.com/r/silex/emacs>, which offers different
Emacs versions in a docker container. I've never tried it myself, but
maybe it helps you.

Best regards, Michael.

Re: Old (static) builds of Emacs

[Philip Kaludercic]

Michael Albinus <michael.albinus@gmx.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
> Hi Philip,
>
>> Until now I have only been able to test the library using Emacs 27.2 and
>> Emacs 28.0.50/60. This doesn't seem like enough for a library that
>> actually targets older versions of Emacs.
>>
>> My question is therefore: Does anyone know where to download old Emacs
>> builds? I have tried building 24.x and 25.x on my system, but it seems
>> (?) that changes in glibc prevent this from being done.
>>
>> So before I have to set up virtual machines with older distributions, I
>> was wondering if anyone knew of an archive with pre-build, preferably
>> static, Emacs builds.
>
> There's <https://hub.docker.com/r/silex/emacs>, which offers different
> Emacs versions in a docker container. I've never tried it myself, but
> maybe it helps you.

It might help, thank you!

> Best regards, Michael.

-- 
	Philip Kaludercic

Re: Old (static) builds of Emacs

[Stefan Monnier]

> So before I have to set up virtual machines with older distributions, I
> was wondering if anyone knew of an archive with pre-build, preferably
> static, Emacs builds.

FWIW, I keep older versions of Emacs in my Debian install by tweaking
the .deb files to eliminate the conflicts.


        Stefan

Re: Old (static) builds of Emacs

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So before I have to set up virtual machines with older distributions, I
>> was wondering if anyone knew of an archive with pre-build, preferably
>> static, Emacs builds.
>
> FWIW, I keep older versions of Emacs in my Debian install by tweaking
> the .deb files to eliminate the conflicts.

And they still run? I tried extracting a executable from an old package,
but it immediately failed.

>         Stefan

-- 
	Philip Kaludercic

Re: [ELPA] Add compat.el

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> Hello again,
>
> I have been working on compat.el over the last week, and have written a
> number of tests. The source code has been uploaded here:
>
>        https://git.sr.ht/~pkal/compat
>
> The library should be usable in it's current state, but I'd be
> interested to hear from other people and developers too.  There might be
> missing functions or macros, edge cases that could be added to the
> tests, etc. that could be improved.

An interesting issue I found when testing compat.el with older versions
is that string-replace and string-search fail, due to the same kind of
issue.

From Emacs 27 onwards, this works as expected:

        (string-replace "\377" "x" "a\377b")
        ;; => "axb"

internally, the compatibility function for string-replace uses
replace-regexp-in-string.

The same code, when evaluated by a version of Emacs older than 27 fails
to replace the character

        (string-replace "\377" "x" "a\377b")
        ;; => "a\377b"

For string-search, using string-match internally, the "unusual" byte
cannot be found:

       (string-search (string-to-multibyte "\377") "ab\377c")
       ;; => nil

whereas the expected answer is 2.

I couldn't find anything in NEWS.27 to indicate that something had
changed, so I wonder if there was a bug that was fixed?  Does anyone
know or remember why this behaviour changed?

-- 
	Philip Kaludercic

Re: Old (static) builds of Emacs

[Stefan Monnier]

Philip Kaludercic [2021-10-07 20:29:40] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> So before I have to set up virtual machines with older distributions, I
>>> was wondering if anyone knew of an archive with pre-build, preferably
>>> static, Emacs builds.
>>
>> FWIW, I keep older versions of Emacs in my Debian install by tweaking
>> the .deb files to eliminate the conflicts.
>
> And they still run?

Yup.

> I tried extracting a executable from an old package,
> but it immediately failed.

I can't remember having had such problems, FWIW.


        Stefan

Re: [ELPA] Add compat.el

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Thu, 07 Oct 2021 20:49:12 +0000
> 
> >From Emacs 27 onwards, this works as expected:
> 
>         (string-replace "\377" "x" "a\377b")
>         ;; => "axb"
> 
> internally, the compatibility function for string-replace uses
> replace-regexp-in-string.
> 
> The same code, when evaluated by a version of Emacs older than 27 fails
> to replace the character
> 
>         (string-replace "\377" "x" "a\377b")
>         ;; => "a\377b"

Why did you expect that to work?  What is the semantics of matching
unibyte strings to multibyte strings? do you compare bytes or do you
compare characters?  There are no "100% correct" answers for these
questions.

> For string-search, using string-match internally, the "unusual" byte
> cannot be found:
> 
>        (string-search (string-to-multibyte "\377") "ab\377c")
>        ;; => nil

Because it isn't a byte, internally.

> I couldn't find anything in NEWS.27 to indicate that something had
> changed, so I wonder if there was a bug that was fixed?  Does anyone
> know or remember why this behaviour changed?

This is a gray area of regexp matching, and it keeps constantly
changing in Emacs under the hood.  The changes are generally seen as
"bugfixes" (though IMNSHO they aren't), and so they aren't called out
in NEWS.  And what would you say there about such subtle changes,
anyway?  So no one should expect it to work the same in different
versions of Emacs.

Re: [ELPA] Add compat.el

[Mattias Engdegård]

7 okt. 2021 kl. 22.49 skrev Philip Kaludercic <philipk@posteo.net>:

> I couldn't find anything in NEWS.27 to indicate that something had
> changed, so I wonder if there was a bug that was fixed?

Yes, it was a bug, fixed in a1f76adfb03c23bb4242928e8efe6193c301f0c1.

Re: [ELPA] Add compat.el

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Thu, 07 Oct 2021 20:49:12 +0000
>> 
>> >From Emacs 27 onwards, this works as expected:
>> 
>>         (string-replace "\377" "x" "a\377b")
>>         ;; => "axb"
>> 
>> internally, the compatibility function for string-replace uses
>> replace-regexp-in-string.
>> 
>> The same code, when evaluated by a version of Emacs older than 27 fails
>> to replace the character
>> 
>>         (string-replace "\377" "x" "a\377b")
>>         ;; => "a\377b"
>
> Why did you expect that to work?  What is the semantics of matching
> unibyte strings to multibyte strings? do you compare bytes or do you
> compare characters?  There are no "100% correct" answers for these
> questions.

The only reason I expect it is that it does work for Emacs 27.1 and
newer.

>> I couldn't find anything in NEWS.27 to indicate that something had
>> changed, so I wonder if there was a bug that was fixed?  Does anyone
>> know or remember why this behaviour changed?
>
> This is a gray area of regexp matching, and it keeps constantly
> changing in Emacs under the hood.  The changes are generally seen as
> "bugfixes" (though IMNSHO they aren't), and so they aren't called out
> in NEWS.  And what would you say there about such subtle changes,
> anyway?  So no one should expect it to work the same in different
> versions of Emacs.

True, but my concern here is to provide older versions of Emacs with the
same behaviour (for these newer functions, like string-search and
string-replace), so I do have to take it into consideration.

-- 
	Philip Kaludercic

Re: [ELPA] Add compat.el

[Philip Kaludercic]

Mattias Engdegård <mattiase@acm.org> writes:

> 7 okt. 2021 kl. 22.49 skrev Philip Kaludercic <philipk@posteo.net>:
>
>> I couldn't find anything in NEWS.27 to indicate that something had
>> changed, so I wonder if there was a bug that was fixed?
>
> Yes, it was a bug, fixed in a1f76adfb03c23bb4242928e8efe6193c301f0c1.

Thank you, knowing what changed I can provide two different
implementations.

-- 
	Philip Kaludercic

Re: [ELPA] Add compat.el

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Oct 2021 09:59:45 +0000
> 
> > This is a gray area of regexp matching, and it keeps constantly
> > changing in Emacs under the hood.  The changes are generally seen as
> > "bugfixes" (though IMNSHO they aren't), and so they aren't called out
> > in NEWS.  And what would you say there about such subtle changes,
> > anyway?  So no one should expect it to work the same in different
> > versions of Emacs.
> 
> True, but my concern here is to provide older versions of Emacs with the
> same behaviour (for these newer functions, like string-search and
> string-replace), so I do have to take it into consideration.

That's fine, but you seem to be asking why the change and whether it's
"right".  I'm saying that you just need to accept that this is how
things are, and if the compatibility package needs to account for
that, so be it.

Re: [ELPA] Add compat.el

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 08 Oct 2021 09:59:45 +0000
>> 
>> > This is a gray area of regexp matching, and it keeps constantly
>> > changing in Emacs under the hood.  The changes are generally seen as
>> > "bugfixes" (though IMNSHO they aren't), and so they aren't called out
>> > in NEWS.  And what would you say there about such subtle changes,
>> > anyway?  So no one should expect it to work the same in different
>> > versions of Emacs.
>> 
>> True, but my concern here is to provide older versions of Emacs with the
>> same behaviour (for these newer functions, like string-search and
>> string-replace), so I do have to take it into consideration.
>
> That's fine, but you seem to be asking why the change and whether it's
> "right".  I'm saying that you just need to accept that this is how
> things are, and if the compatibility package needs to account for
> that, so be it.

Oh no, I wasn't questioning the current behaviour, I just wanted to know
what changed internally, to better understand what was going on.

-- 
	Philip Kaludercic

Re: [ELPA] Add compat.el

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> Hello again,

... and once more.

> I have been working on compat.el over the last week, and have written a
> number of tests. The source code has been uploaded here:
>
>        https://git.sr.ht/~pkal/compat
>
> The library should be usable in it's current state, but I'd be
> interested to hear from other people and developers too.  There might be
> missing functions or macros, edge cases that could be added to the
> tests, etc. that could be improved.

After debugging the code for a while, I can now say that the code is
functional and passes all the tests I prepared. Sadly this required me
to bump the minimal version up to 24.3. It would have been a lot easier
to bump it to 24.4, but 24.3 is the oldest release still in use on
CentOS 7 machines (I know this because my fathers workstation has this
version of Emacs).

For now I am using GitHub's CI to ensure that changes don't break the
test-suit. I plan to use SourceHut's service at some later point, but
for now the work of preparing old Emacs builds was already taken care of
by Steve Purcell (https://github.com/purcell/setup-emacs), so I took the
easier route for now.

With the infrastructure ready, this means the package could be added to
ELPA, and released along with Emacs 28.1, leaving my some time to
improve the documentation. I'd like to once again make sure there are no
fundamental objections to anything I am doing here. If so, I'd do my
best to fix the issues.

> Finally, compat.el now ends with these lines:
>
> ;; ;;;; Ensure NonGNU ELPA is in `package-archives'
> ;; (with-eval-after-load 'package
> ;;   (unless (or (rassoc "https://elpa.nongnu.org/nongnu/" package-archives)
> ;;               (rassoc "http://elpa.nongnu.org/nongnu/" package-archives)
> ;;               (rassoc "https://elpa.nongnu.org/nongnu-devel/" package-archives)
> ;;               (rassoc "http://elpa.nongnu.org/nongnu-devel/" package-archives))
> ;;     (add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/"))))

It seems better to steer away from controversy for now, so I have
removed this snippet.

-- 
	Philip Kaludercic

Re: Old (static) builds of Emacs

[Philip Kaludercic]

I have tried using GNU Guix to build old versions of Emacs, and it seems
to work well up until 24.5.  It seems something was changed in glibc (?)
that necessitated a change in Emacs, if I understand correctly.  Does
anyone remember what happened here, so that I can patch these versions
(or would it be easier to build an old version of Glibc)?

Philip Kaludercic <philipk@posteo.net> writes:

> Until now I have only been able to test the library using Emacs 27.2 and
> Emacs 28.0.50/60. This doesn't seem like enough for a library that
> actually targets older versions of Emacs.
>
> My question is therefore: Does anyone know where to download old Emacs
> builds? I have tried building 24.x and 25.x on my system, but it seems
> (?) that changes in glibc prevent this from being done.
>
> So before I have to set up virtual machines with older distributions, I
> was wondering if anyone knew of an archive with pre-build, preferably
> static, Emacs builds.
>
> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Hello again,
>>
>> I have been working on compat.el over the last week, and have written a
>> number of tests. The source code has been uploaded here:
>>
>>        https://git.sr.ht/~pkal/compat
>>
>> The library should be usable in it's current state, but I'd be
>> interested to hear from other people and developers too.  There might be
>> missing functions or macros, edge cases that could be added to the
>> tests, etc. that could be improved.

-- 
	Philip Kaludercic

Re: Old (static) builds of Emacs

[Stefan Monnier]

Philip Kaludercic [2021-11-30 21:21:51] wrote:
> I have tried using GNU Guix to build old versions of Emacs, and it seems
> to work well up until 24.5.  It seems something was changed in glibc (?)
> that necessitated a change in Emacs, if I understand correctly.  Does
> anyone remember what happened here, so that I can patch these versions
> (or would it be easier to build an old version of Glibc)?

Might have been when they dropped some malloc hooks we were using, or
something like that.  IIRC this problem was "really fixed" with the
pdumper and before that we circumvented it by using the src/gmalloc.c
code during the dump (and switch to the native malloc afterwards).

So, you might want to try and tell Emacs to use src/gmalloc.c instead of
the system's malloc.


        Stefan