Moving packages out of core to ELPA

Moving packages out of core to ELPA

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2630 bytes --]


There was a recent discussion about which lisp/progmodes packages belong in core.  A sentiment was expressed that useful languages with non-negligible user bases should probably go in, and others should be in ELPA.

I want to bring up a related point: it should be possible to retire packages from core, once their relevance drops below a critical threshold [1]. 

I am the former maintainer of the now mostly defunct IDLWAVE mode (lisp/progmodes/idl*.el).  IDLWAVE was put in core about 20 years ago, long before ELPA existed, by my predecessor Carsten Dominik (of org-mode fame).  I'd advocate moving IDLWAVE to ELPA, for the following reasons:

The proprietary IDL <https://en.wikipedia.org/wiki/IDL_(programming_language)> is a language in rapid decline.  In my field (astrophysics), its use is now relegated to older professionals who have not switched to Python (but most of whom wish they had the time to do so).  
IDL's holding company has changed approximately 5 times in the last 10 years, such that the link to its website on Wikipedia isn't even correct.
None of the free IDL alternatives have really caught on.  IDL's costs and licensing restrictions have gotten more onerous over the years. 
IDL is also the "interface description language", confusing users of that unrelated system.
IDL's file extension ".pro" is quite common, and in use for several other file types, including Qt's UI build system <https://doc.qt.io/qt-6/qmake-project-files.html> as well as Yarn2/Prolog (which org babel plugin ob-prolog <https://github.com/ljos/ob-prolog> supports).
Some of the best features in IDLWAVE (e.g. direct documentation linking) required maintenance support from IDL's owners, support which hasn't fully existed for almost a decade.
The version in core has diverged from the latest <https://github.com/jdtsmith/idlwave> (from 7 years ago).

Judging by my email traffic on the topic, there are very few users of IDLWAVE remaining.  Even I touch it only a few times a year.  No one has stepped up to maintain it in the 7 years since I stepped aside.  IDL is still a powerful language and IDLWAVE itself has a great number of features I sorely miss in other emacs programming modes.  It deserves support in Emacs.  Just not, IMO, in core.

[1] Such a threshold may be hard to define, but here is one idea: if the number of users who activate a mode on accident <https://www.reddit.com/r/emacs/comments/qefx8e/til_idl_is_a_thing_and_emacs_has_it/> is greater than the number who actively seek to do so, it may be time to retire a mode.  My experience indicates IDLWAVE has likely passed this threshold.

[-- Attachment #2: Type: text/html, Size: 3225 bytes --]

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 09:40:19 -0500
> 
> There was a recent discussion about which lisp/progmodes packages belong in core.  A sentiment was
> expressed that useful languages with non-negligible user bases should probably go in, and others should be
> in ELPA.
> 
> I want to bring up a related point: it should be possible to retire packages from core, once their relevance
> drops below a critical threshold [1]. 

We usually first retire such packages to lisp/obsolete/.  There, they
are still available, but need to be loaded manually.  Moving something
directly to ELPA might be too drastic, assuming someone somewhere
still want to use it.

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 10:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Sat, 17 Feb 2024 09:40:19 -0500
>> 
>> There was a recent discussion about which lisp/progmodes packages belong in core.  A sentiment was
>> expressed that useful languages with non-negligible user bases should probably go in, and others should be
>> in ELPA.
>> 
>> I want to bring up a related point: it should be possible to retire packages from core, once their relevance
>> drops below a critical threshold [1].
> 
> We usually first retire such packages to lisp/obsolete/.  There, they
> are still available, but need to be loaded manually.  Moving something
> directly to ELPA might be too drastic, assuming someone somewhere
> still want to use it.

Does that reduce the maintenance burden, for example because code in lisp/obsolete does not receive updates to documentation, stylistic code changes, etc.?  Do you typically also remove the associated entry in auto-mode-alist when obsoleting in this fashion?

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 11:22:35 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> 
> 
> > On Feb 17, 2024, at 10:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: JD Smith <jdtsmith@gmail.com>
> >> Date: Sat, 17 Feb 2024 09:40:19 -0500
> >> 
> >> There was a recent discussion about which lisp/progmodes packages belong in core.  A sentiment was
> >> expressed that useful languages with non-negligible user bases should probably go in, and others should be
> >> in ELPA.
> >> 
> >> I want to bring up a related point: it should be possible to retire packages from core, once their relevance
> >> drops below a critical threshold [1].
> > 
> > We usually first retire such packages to lisp/obsolete/.  There, they
> > are still available, but need to be loaded manually.  Moving something
> > directly to ELPA might be too drastic, assuming someone somewhere
> > still want to use it.
> 
> Does that reduce the maintenance burden, for example because code in lisp/obsolete does not receive updates to documentation, stylistic code changes, etc.?  Do you typically also remove the associated entry in auto-mode-alist when obsoleting in this fashion?

Yes, I think so (on both counts).

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 11:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> 
>>> On Feb 17, 2024, at 10:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> We usually first retire such packages to lisp/obsolete/.  There, they
>>> are still available, but need to be loaded manually.  Moving something
>>> directly to ELPA might be too drastic, assuming someone somewhere
>>> still want to use it.
>> 
>> Does that reduce the maintenance burden, for example because code in lisp/obsolete does not receive updates to documentation, stylistic code changes, etc.?  Do you typically also remove the associated entry in auto-mode-alist when obsoleting in this fashion?
> 
> Yes, I think so (on both counts).

OK, then I suppose staging in lisp/obsolete/ makes sense.  I do worry about the connotations of obsolete meaning "stop using", "does not work at all" or "soon to vanish".  In fact IDLWAVE does still work and is useful for a small set of users, it just does more harm than good in core, IMO.  

One idea would be to create another category for packages "on their way out" of core.  I'm sure there's a better name, but lisp/noncore comes to mind.  The intended difference being that obsolete packages will likely disappear entirely from the project, whereas "noncore" packages will be migrated to ELPA, where users can find them once they have left core for good.

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 12:08:37 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> >>> We usually first retire such packages to lisp/obsolete/.  There, they
> >>> are still available, but need to be loaded manually.  Moving something
> >>> directly to ELPA might be too drastic, assuming someone somewhere
> >>> still want to use it.
> >> 
> >> Does that reduce the maintenance burden, for example because code in lisp/obsolete does not receive updates to documentation, stylistic code changes, etc.?  Do you typically also remove the associated entry in auto-mode-alist when obsoleting in this fashion?
> > 
> > Yes, I think so (on both counts).
> 
> OK, then I suppose staging in lisp/obsolete/ makes sense.  I do worry about the connotations of obsolete meaning "stop using", "does not work at all" or "soon to vanish".  In fact IDLWAVE does still work and is useful for a small set of users, it just does more harm than good in core, IMO.  

We also call out the obsolescence in NEWS, so users will know, when
the package fails to load automatically.

> One idea would be to create another category for packages "on their way out" of core.  I'm sure there's a better name, but lisp/noncore comes to mind.  The intended difference being that obsolete packages will likely disappear entirely from the project, whereas "noncore" packages will be migrated to ELPA, where users can find them once they have left core for good.

We don't yet have examples of such packages, I think.  And in this
case, we intend to move IDLWAVE out precisely because it is obsolete,
so we don't need to consider a new category for this case, I think.

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 12:29 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> We don't yet have examples of such packages, I think. 

The idea would be to establish such a category.  Some user-facing note could be included — “This package will be removed from core Emacs in a future version, and is available on ELPA at XYZ”.  Could even leave a “stub” behind pointing there after final removal. 

> And in this case, we intend to move IDLWAVE out precisely because it is obsolete,

Not necessarily, depending on how one defines obsolete. Just “no longer worth keeping in core”.

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 13:52:44 -0500
> Cc: emacs-devel@gnu.org
> 
> > On Feb 17, 2024, at 12:29 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > And in this case, we intend to move IDLWAVE out precisely because it is obsolete,
> 
> Not necessarily, depending on how one defines obsolete. Just “no longer worth keeping in core”.

That's not my interpretation of your description of IDL state.  I
think "no longer worth keeping in core" is the conclusion, not the
reason.  The reason is that IDL is obsolete and rarely if ever used
nowadays.

RE: [External] : Re: Moving packages out of core to ELPA

[Drew Adams]

> obsolete and rarely if ever used nowadays.

Those are different predicates/categories.

The suggestion was to separate the bins we
send things to.

The two aren't exclusive, of course.  A thing
that's truly obsolete would likely be sent to
the obsolete bin, even when it may also be
rarely used.

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 2:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
>> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Sat, 17 Feb 2024 13:52:44 -0500
>> Cc: emacs-devel@gnu.org
>> 
>>>> On Feb 17, 2024, at 12:29 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> And in this case, we intend to move IDLWAVE out precisely because it is obsolete,
>> 
>> Not necessarily, depending on how one defines obsolete. Just “no longer worth keeping in core”.
> 
> That's not my interpretation of your description of IDL state.

Then I fear you may have misinterpreted me.  Many astronomers and earth scientists do still use it. A few with emacs.  Maybe it’s a bit like Perl that way.  There’s a lot of legacy code around. So it’s perfectly valuable to keep in ELPA, but IMO the cost/benefit no longer merits inclusion in core. 

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 16:16:31 -0500
> Cc: emacs-devel@gnu.org
> 
> >> Not necessarily, depending on how one defines obsolete. Just “no longer worth keeping in core”.
> > 
> > That's not my interpretation of your description of IDL state.
> 
> Then I fear you may have misinterpreted me.  Many astronomers and earth scientists do still use it. A few with emacs.  Maybe it’s a bit like Perl that way.  There’s a lot of legacy code around. So it’s perfectly valuable to keep in ELPA, but IMO the cost/benefit no longer merits inclusion in core. 

Why doesn't the cost/benefit merit inclusion in core?

How do you understand the considerations for keeping something in core
in general?

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 18, 2024, at 1:28 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Sat, 17 Feb 2024 16:16:31 -0500
>> Cc: emacs-devel@gnu.org
>> 
>>>> Not necessarily, depending on how one defines obsolete. Just “no longer worth keeping in core”.
>>> 
>>> That's not my interpretation of your description of IDL state.
>> 
>> Then I fear you may have misinterpreted me.  Many astronomers and earth scientists do still use it. A few with emacs.  Maybe it’s a bit like Perl that way.  There’s a lot of legacy code around. So it’s perfectly valuable to keep in ELPA, but IMO the cost/benefit no longer merits inclusion in core.
> 
> Why doesn't the cost/benefit merit inclusion in core?

Maybe I should be more specific here: I don't think the cost/benefit favors having it in the core ready to fire up when any .pro file is loaded (since there are other unrelated flavors with that file type in the wild, e.g. Qt config files, which confuses people), or having the IDLWAVE manual appear at the top level of M-x info (as nice as a manual it is, and as hard as I worked on it lo those many years ago).

> How do you understand the considerations for keeping something in core
> in general?

When the additional burden of extra information presented to users and the costs to maintain, update, seek out and correct outdated code is outweighed by the utility some users gain by having it readily at hand.  A small package that is effectively invisible and never loads unless the user summons it has fewer costs in this way of looking at it.  I do not suggest that judging this balance is trivial or even straightforward, but it's my opinion for this particular package, many of whose users I am well familiar with, the costs outweigh the benefits.  I am interested in how the maintainers proper see the costs of maintenance; maybe I overestimate those.  And I do note there are lots of things under M-x info that probably don't see much use, so I acknowledge these arguments could represent a "slippery slope".

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> Maybe I should be more specific here: I don't think the cost/benefit
> favors having it in the core ready to fire up when any .pro file is
> loaded (since there are other unrelated flavors with that file type in
> the wild, e.g. Qt config files, which confuses people)

That's easily resolved, if we judge that this is an issue.

> or having the IDLWAVE manual appear at the top level of M-x info (as
> nice as a manual it is, and as hard as I worked on it lo those many
> years ago).

Likewise.  But if I may digress a little, why should it be ever a
problem that an index or directory contain information of marginal
interest, when that is the rationale for maintaining such a directory to
begin with?  And why is IDLWAVE more of a problem than the remainder of
our eclectic corpus of manuals, many of which are to the average user no
more relevant than is IDLWAVE?  To be found in the card catalog of any
library are the answers to both these questions, as the Info directory's
role is quite comparable to theirs.

The purpose of the Info directory (or any directory, at that) is to
present users with references to all available sources of documentation,
which is rather irreconcilable with the idea of optimizing its
proportions by removing entries that do not interest a sufficient
segment of readers, for a definition of "sufficient" set by whoever
takes charge of the manual.  To this end, the GNU C library places
references to every documented Glibc function in the directory tree, the
scale of which list makes ours seem minuscule by comparison.

Against this background, removing IDLWAVE to clear up space in the Info
directory is an misinformed as well as ineffective gesture at best.

> hand.  A small package that is effectively invisible and never loads
> unless the user summons it has fewer costs in this way of looking at
> it.

This description could also apply to IDLWAVE as it stands in core.

Re: Moving packages out of core to ELPA

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]


> On Feb 18, 2024, at 8:46 AM, Po Lu <luangruo@yahoo.com> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> Maybe I should be more specific here: I don't think the cost/benefit
>> favors having it in the core ready to fire up when any .pro file is
>> loaded (since there are other unrelated flavors with that file type in
>> the wild, e.g. Qt config files, which confuses people)
> 
> That's easily resolved, if we judge that this is an issue.
> 
>> or having the IDLWAVE manual appear at the top level of M-x info (as
>> nice as a manual it is, and as hard as I worked on it lo those many
>> years ago).
> 
> Likewise.  But if I may digress a little, why should it be ever a
> problem that an index or directory contain information of marginal
> interest, when that is the rationale for maintaining such a directory to
> begin with?  And why is IDLWAVE more of a problem than the remainder of
> our eclectic corpus of manuals, many of which are to the average user no
> more relevant than is IDLWAVE?  To be found in the card catalog of any
> library are the answers to both these questions, as the Info directory's
> role is quite comparable to theirs.

This is the slippery slope I alluded to, and it is a credible argument.  To explore its boundaries, let's consider a scenario in which the IDL language were well and truly dead.  No existing licensed interpreter capable of running IDL code existed any longer on any system (licenses expire).  Would maintaining this package in the core be worth it at that point?  If not, this indicates that the costs are indeed non-zero.

>> hand.  A small package that is effectively invisible and never loads
>> unless the user summons it has fewer costs in this way of looking at
>> it.
> 
> This description could also apply to IDLWAVE as it stands in core.

I wonder if all agree on that point?  Consider:

git log --pretty="format:%an" lisp/progmodes/idlw*.el doc/misc/idlwave.texi | perl -ne 'chomp; $a{$_}++; END{foreach $n (sort {@a{$b} <=> @a{$a}} keys %a) {printf("%30s\t%s\n",$n,"+" x $a{$n})}}'  

This reveals that an impressive 39 individual maintainers have had their hand in the code over the last 20 years, some with more than 80 commits.  Yes, many of these are find/replace typos or other "no additional cost" commits, but many are not.

[-- Attachment #2: Type: text/html, Size: 3559 bytes --]

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> This is the slippery slope I alluded to, and it is a credible
> argument.  To explore its boundaries, let's consider a scenario in
> which the IDL language were well and truly dead.  No existing licensed
> interpreter capable of running IDL code existed any longer on any
> system (licenses expire).  Would maintaining this package in the core
> be worth it at that point?  If not, this indicates that the costs are
> indeed non-zero.

No, the question of cost to maintainence is simply moot if the software
should become irreversibly useless.  Please, let's be reasonable,
consider realistic scenarios, and save the extreme hypotheticals for
another day.

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 18, 2024, at 9:08 AM, Po Lu <luangruo@yahoo.com> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> This is the slippery slope I alluded to, and it is a credible
>> argument.  To explore its boundaries, let's consider a scenario in
>> which the IDL language were well and truly dead.  No existing licensed
>> interpreter capable of running IDL code existed any longer on any
>> system (licenses expire).  Would maintaining this package in the core
>> be worth it at that point?  If not, this indicates that the costs are
>> indeed non-zero.
> 
> No, the question of cost to maintainence is simply moot if the software
> should become irreversibly useless.  Please, let's be reasonable,
> consider realistic scenarios, and save the extreme hypotheticals for
> another day.

I don't judge this to be an unreasonable or extreme hypothetical.  I (sadly) give a 25% chance that this actually happens in the next 10 years — a demonstrable peril of non-free software.  

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sun, 18 Feb 2024 09:03:05 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> git log --pretty="format:%an" lisp/progmodes/idlw*.el doc/misc/idlwave.texi | perl -ne 'chomp; $a{$_}++; END
> {foreach $n (sort {@a{$b} <=> @a{$a}} keys %a) {printf("%30s\t%s\n",$n,"+" x $a{$n})}}'  
> 
> This reveals that an impressive 39 individual maintainers have had their hand in the code over the last 20
> years, some with more than 80 commits.  Yes, many of these are find/replace typos or other "no additional
> cost" commits, but many are not.

You consider 4 changes a year a significant maintenance cost?  Please
don't, because it's negligible.  I consider even 4 changes a day not
too many.

Look, why won't you two leave it to the maintainers to assess the
costs of the maintenance, okay?  You are both arguing about something
you know only in theory, at least when applied to Emacs maintenance.
I know this from practical everyday experience, and I'm telling you:
the costs are negligible.  Why isn't that enough? why do you insist on
telling me what are _my_ costs?

This discussion shouldn't have continued after the question was asked
and answered.  We are wasting each other's time by keeping it going.
Let's move on, okay?

My conclusion from this is that we can either move IDLWAVE to
lisp/obsolete (and remove the relevant association from
auto-mode-alist), or leave things as they are.  Which one is better is
up to the users of IDLWAVE (and I'm okay with either one), but either
way the maintenance costs are not a factor.

Whether to move IDLWAVE out of core to ELPA could be revisited later,
but the maintenance costs definitely don't justify moving it nor
leaving it; they are simply not relevant to that decision.

Re: Moving packages out of core to ELPA

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 163 bytes --]



> On Feb 18, 2024, at 9:17 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  leave it to the maintainers to assess the costs of the maintenance

Fair enough.

[-- Attachment #2: Type: text/html, Size: 1271 bytes --]

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> I wonder if all agree on that point?  Consider:
>
> git log --pretty="format:%an" lisp/progmodes/idlw*.el doc/misc/idlwave.texi | perl -ne 'chomp; $a{$_}++; END{foreach $n (sort {@a{$b}
> <=> @a{$a}} keys %a) {printf("%30s\t%s\n",$n,"+" x $a{$n})}}'  
>
> This reveals that an impressive 39 individual maintainers have had
> their hand in the code over the last 20 years, some with more than 80
> commits.  Yes, many of these are find/replace typos or other "no
> additional cost" commits, but many are not.

They're accomplished fact, and (with the exception of automated commits)
only exist because what little cost they embodied was consciously
evaluated by their authors and not deemed unacceptable.  In turn, an
unacceptable burden cannot ever arise, since none of us would ever
assume it.

As regards the impact of moving to ELPA, search the emacs-elpa-diffs
mailing list for changes by Emacs developers, notably Stefan Monnier.
You will discover that our attitude towards ELPA packages is not so
hands-off as you assume, and that they are on decent footing where
maintenance is concerned.

RE: [External] : Re: Moving packages out of core to ELPA

[Drew Adams]

> OK, then I suppose staging in lisp/obsolete/ makes sense.
>
> I do worry about the connotations of obsolete meaning
> "stop using", "does not work at all" or "soon to vanish".
> In fact IDLWAVE does still work and is useful for a small
> set of users, it just does more harm than good in core, IMO.

FWIW, I agree.  The label "obsolete" doesn't
correspond to the description.

I think of a longstanding, likely hardly used,
standard library such as `completion.el', for
example.  That code still works fine.

Following the criteria described, that could
well end up being sent to lisp/obsolete/.

But (IMO) it's food for thought.  The approach
could perhaps be picked up and revitalized,
maybe even combined with some other existing
in-buffer completion library - either within
Emacs or in some 3rd-party code.

IOW, it's an example (IMO) of code that could
be mined or thought about, to do something
new or in a new way.  And again, it still just
works.

I wouldn't object to it (& similar) libraries
being sent to a different directory, such as
"obsolete".  It's just the name "obsolete"
that I think is unfortunate for what's been
described here (no development support etc.). 

> One idea would be to create another category for packages "on their way
> out" of core.  I'm sure there's a better name, but lisp/noncore comes
> to mind.  The intended difference being that obsolete packages will
> likely disappear entirely from the project, whereas "noncore" packages
> will be migrated to ELPA, where users can find them once they have left
> core for good.

Something like that, yes, +1.  Some name that
doesn't have the connotation that the code's
been replaced by something better etc.  The
code's simply been removed from maintenance,
active development, and automatic loading.

 Eli> the package fails to load automatically

Failing to load automatically is _fine_ for
such a library.  Calling that out in NEWS is
also fine.

Calling the library "obsolete" and calling
the move "deprecation" or "obsoleting" is
misleading.  It pretty much guarantees that
folks won't try it or even look at it.

If we keep it around then a possible reason
for doing that should be that we think
there's some merit/interest in what it does
or tries to do, or in the code techniques
it uses.

This is the kind of thing that's sometimes
called "more" or "also interesting" or
"see also".  A name such as those can
invite serendipitous exploring and perusal,
as opposed to stigmatizing the library as
uninteresting, nonworking, or depassee -
"obsolete".

(Just one opinion.)

Re: [External] : Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 12:42 PM, Drew Adams <drew.adams@oracle.com> wrote:
> 
> The approach
> could perhaps be picked up and revitalized,
> maybe even combined with some other existing
> in-buffer completion library - either within
> Emacs or in some 3rd-party code.

Absolutely.  I’ve been mining nuggets from IDLWAVE myself for a mode I’ve been working on for a while.  I agree that obsolete connotes “never look here”.   

Re: [External] : Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 13:21:49 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel <emacs-devel@gnu.org>
> 
> I agree that obsolete connotes “never look here”.   

That's not how we treat it.

Re: [External] : Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 1:32 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
>> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Sat, 17 Feb 2024 13:21:49 -0500
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel <emacs-devel@gnu.org>
>> 
>> I agree that obsolete connotes “never look here”.   
> 
> That's not how we treat it.

I was not speaking about how emacs maintainers treat obsolete, rather how end users and external package developers (may) perceive it.  

If in 20 years I need to edit an IDL .pro file, I will use emacs and IDLWAVE to do it (which, side note, is amazing you can say about any piece of software!). So not really obsolete, just not actively maintained, and no longer part of core.  

RE: [External] : Re: Moving packages out of core to ELPA

[Drew Adams]

> > I agree that obsolete connotes “never look here”.
> 
> That's not how we treat it.

Understood.  But that's how many readers
of "obsolete" may understand it.  That's
the point raised here, I believe.  It's
the word, not the actual use/treatment
that is a bit unfortunate.

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> OK, then I suppose staging in lisp/obsolete/ makes sense.  I do worry
> about the connotations of obsolete meaning "stop using", "does not
> work at all" or "soon to vanish".

If its users have truly gone the way of the dodo, I see no reason this
should be a problem.  It won't be necessary to move it to ELPA either.

> In fact IDLWAVE does still work and is useful for a small set of
> users, it just does more harm than good in core, IMO.

Packages whose existence comes at the detriment of their host (Emacs) do
not develop this trait over time.  Either they suffer from deficiencies
that prevent the host from functioning, which become evident as soon as
they are merged, or their existence is an asset of the host's, however
minor it may be.

> One idea would be to create another category for packages "on their
> way out" of core.  I'm sure there's a better name, but lisp/noncore
> comes to mind.  The intended difference being that obsolete packages
> will likely disappear entirely from the project, whereas "noncore"
> packages will be migrated to ELPA, where users can find them once they
> have left core for good.

If it's not obsolete, it should remain in core, period.

Re: Moving packages out of core to ELPA

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]



> On Feb 17, 2024, at 8:55 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
>> In fact IDLWAVE does still work and is useful for a small set of
>> users, it just does more harm than good in core, IMO.
> 
> Packages whose existence comes at the detriment of their host (Emacs) do
> not develop this trait over time.

This strikes me as a very rigid perspective.  In my view, the costs and benefits of any capability of a system can ebb and flow over time, as the context in which they are evaluated changes.

[-- Attachment #2: Type: text/html, Size: 2620 bytes --]

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> This strikes me as a very rigid perspective.  In my view, the costs
> and benefits of any capability of a system can ebb and flow over time,
> as the context in which they are evaluated changes.

At least my outlook is applicable on the basis of objective criteria;
like the presumption of innocence, for example, which many a trial would
have better done without, if not for the arbitrariness that must have
figured in any decision to do so.

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> There was a recent discussion about which lisp/progmodes packages
> belong in core.  A sentiment was expressed that useful languages with
> non-negligible user bases should probably go in, and others should be
> in ELPA.
>
> I want to bring up a related point: it should be possible to retire
> packages from core, once their relevance drops below a critical
> threshold [1].

Every instance of this is a mistake.  Why is it so essential that such
packages be removed from core?  What practical advantage does that hold?

A Lisp file is considered part of Emacs, whether it be in core or in
ELPA.  They are expected to meet like standards, and bugs (in the
absence of a maintainer) are the responsibility of the same Emacs
developers who respond to bugs that concern Emacs in general, i.e., like
developers.  The only additional burden the existence of a file in core
implies over being hosted in ELPA is the copyright header update, an
automated process that consumes little time, and whose frequency is not
greater than once a year.

> Judging by my email traffic on the topic, there are very few users of
> IDLWAVE remaining.  Even I touch it only a few times a year.  No one
> has stepped up to maintain it in the 7 years since I stepped aside.
> IDL is still a powerful language and IDLWAVE itself has a great number
> of features I sorely miss in other emacs programming modes.  It
> deserves support in Emacs.  Just not, IMO, in core.

Why not?  Why does _anything_, to speak nothing of a package already in
core, not "deserve" support in core, while deserving support in ELPA?

Re: Moving packages out of core to ELPA

[JD Smith]

> On Feb 17, 2024, at 8:42 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> There was a recent discussion about which lisp/progmodes packages
>> belong in core.  A sentiment was expressed that useful languages with
>> non-negligible user bases should probably go in, and others should be
>> in ELPA.
>> 
>> I want to bring up a related point: it should be possible to retire
>> packages from core, once their relevance drops below a critical
>> threshold [1].
> 
> Every instance of this is a mistake.  

That would imply that every instance of bringing a package into core is not a mistake, which suggests superhuman forecasting and decision-making.  But even when including a package is beneficial, situations change over time (>20 yrs, in this case).  It seems only prudent to reevaluate whether a given package still provides sufficient benefit given the non-zero costs associated with it.

> Why is it so essential that such
> packages be removed from core?  What practical advantage does that hold?

Many which I mentioned in my initial message.  The most salient:

- Reduces maintenance burdens, freeing time for packages that have more pressing issues. I have heard from emacs maintainers who have spent significant time trying to understand and fix bugs in IDLWAVE code that is likely unused (even by me).
- Removes "tripping hazards" for users who inadvertently activate the mode for unrelated files and are confused (this is not hypothetical: I've had numerous reports).
- Cuts down on "extra noise" in, e.g., the top level Info help.

> A Lisp file is considered part of Emacs, whether it be in core or in
> ELPA.  They are expected to meet like standards, and bugs (in the
> absence of a maintainer) are the responsibility of the same Emacs
> developers who respond to bugs that concern Emacs in general, i.e., like
> developers.  

Is this really so, in practice?  I have packages in ELPA which are effectively untouched except by me, other than on first ingestion.  And they draw updates from a repo I maintain myself.  Maybe I've misconstrued the situation, but my understanding has been that core packages receive far more attention from maintainers.  And rightly so, IMO: everyone has them installed, after all.  

Also, if ELPA and core are truly equivalent, I cannot then understand the common strategy of "let it mature on ELPA for a few years, then potentially migrate to core."

>>   It deserves support in Emacs.  Just not, IMO, in core.
> 
> Why not?  Why does _anything_, to speak nothing of a package already in
> core, not "deserve" support in core, while deserving support in ELPA?

Because in ELPA, users must proactively opt-in to the use of the package.  For such users — those who have actively sought it out — in stark contrast to the vast majority of Emacs users, the benefits dramatically outweigh the costs.

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> That would imply that every instance of bringing a package into core
> is not a mistake

This is correct.

> , which suggests superhuman forecasting and decision-making.

Or a simple recognition of the principle that "more is better."

> But even when including a package is beneficial, situations change
> over time (>20 yrs, in this case).  It seems only prudent to
> reevaluate whether a given package still provides sufficient benefit
> given the non-zero costs associated with it.

Such costs are strictly zero, so long as the package has indeed fallen
out of interest.  The existence of maintainence work beyond the routine
copyright bump at each turn of the year suggests that there's life in
the old dog yet, and it would be vastly improper to declare it no longer
worthy of the same.

> Many which I mentioned in my initial message.  The most salient:
>
> - Reduces maintenance burdens, freeing time for packages that have
> more pressing issues. I have heard from emacs maintainers who have
> spent significant time trying to understand and fix bugs in IDLWAVE
> code that is likely unused (even by me).

Emacs developers also spend time scrutinizing and suggesting changes to
ELPA packages, which are reported through channels not unlike
emacs-diffs.  At any rate, there's nothing preventing its users from
demonstrating that IDLwave maintenance can be safely discontinued or
ignored--most of us would be more than glad to oblige.

> - Removes "tripping hazards" for users who inadvertently activate the
> mode for unrelated files and are confused (this is not hypothetical:
> I've had numerous reports).

Let's remove it from auto-mode-alist, then.

> - Cuts down on "extra noise" in, e.g., the top level Info help.

The IDLWAVE manual can be distributed separately, if that is important.

> Is this really so, in practice?  I have packages in ELPA which are
> effectively untouched except by me, other than on first ingestion.
> And they draw updates from a repo I maintain myself.  Maybe I've
> misconstrued the situation, but my understanding has been that core
> packages receive far more attention from maintainers.  And rightly so,
> IMO: everyone has them installed, after all.
>
> Also, if ELPA and core are truly equivalent, I cannot then understand
> the common strategy of "let it mature on ELPA for a few years, then
> potentially migrate to core."

I don't either.  It's a counterproductive practice that only serves to
give some of us the satisfaction of knowing that ELPA has made itself
useful.

> Because in ELPA, users must proactively opt-in to the use of the
> package.  For such users — those who have actively sought it out — in
> stark contrast to the vast majority of Emacs users, the benefits
> dramatically outweigh the costs.

The other side of this assertion is that users must opt-out of loading
packages in core, and therefore that the entire lisp directory (bar
obsolete) is loaded into every Emacs session at startup, which is
clearly untrue.

Re: Moving packages out of core to ELPA

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Such costs are strictly zero, so long as the package has indeed fallen
> out of interest.  The existence of maintainence work beyond the
                                     ............

Maintenance, sorry.

Re: Moving packages out of core to ELPA

[JD Smith]

> 
>> Because in ELPA, users must proactively opt-in to the use of the
>> package.  For such users — those who have actively sought it out — in
>> stark contrast to the vast majority of Emacs users, the benefits
>> dramatically outweigh the costs.
> 
> The other side of this assertion is that users must opt-out of loading
> packages in core, and therefore that the entire lisp directory (bar
> obsolete) is loaded into every Emacs session at startup, which is
> clearly untrue.

I know some inexperienced users have this concern, but I don't think anyone here believes that.  I also do not worry for one second about the 1,007,028 bytes **/idlw*.{el,texi,info} occupy, since downloading them takes <<1 second.  No, the costs I see are not related to unwanted bytes, neither bytes on disk nor bytes loaded into the running emacs process.  But reasonable people can disagree about the scope of these costs.  Disabling the auto-mode-alist and moving the manual from the top level would go a long way towards mitigating them, IMO.  

I am of course not myself an emacs maintainer, but I would be interested to hear whether everyone who is agrees that maintenance costs are "strictly zero" for seldom-used code. 

Thanks for your thoughts.

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sun, 18 Feb 2024 07:39:02 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> I am of course not myself an emacs maintainer, but I would be interested to hear whether everyone who is agrees that maintenance costs are "strictly zero" for seldom-used code. 

I already said that I agreed.  (And "seldom-used" is not the relevant
aspect; the relevant aspect is whether the package is still being
actively developed by someone who submits changes to us.)

Re: Moving packages out of core to ELPA

[Po Lu]

JD Smith <jdtsmith@gmail.com> writes:

> I know some inexperienced users have this concern, but I don't think
> anyone here believes that.  I also do not worry for one second about
> the 1,007,028 bytes **/idlw*.{el,texi,info} occupy, since downloading
> them takes <<1 second.  No, the costs I see are not related to
> unwanted bytes, neither bytes on disk nor bytes loaded into the
> running emacs process.  But reasonable people can disagree about the
> scope of these costs.  Disabling the auto-mode-alist and moving the
> manual from the top level would go a long way towards mitigating them,
> IMO.

Any costs that cannot be quantified in consumed disk space or bytes
loaded into the Emacs process are notional or perhaps even imaginary,
and plenty debatable.  No consensus has ever formed on the subject, and
any such attempt will be a wild goose chase, if you will.

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 17 Feb 2024 21:14:06 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> > Why is it so essential that such
> > packages be removed from core?  What practical advantage does that hold?
> 
> Many which I mentioned in my initial message.  The most salient:
> 
> - Reduces maintenance burdens, freeing time for packages that have more pressing issues. I have heard from emacs maintainers who have spent significant time trying to understand and fix bugs in IDLWAVE code that is likely unused (even by me).
> - Removes "tripping hazards" for users who inadvertently activate the mode for unrelated files and are confused (this is not hypothetical: I've had numerous reports).
> - Cuts down on "extra noise" in, e.g., the top level Info help.

These are very minor at best.  Some of them could be made even less
significant by simple measures like removing the associations from
auto-mode-alist.  Don't worry about those; if there are still users of
the package, leaving it in core and favoring those users completely
outweighs the above disadvantages.

> > A Lisp file is considered part of Emacs, whether it be in core or in
> > ELPA.  They are expected to meet like standards, and bugs (in the
> > absence of a maintainer) are the responsibility of the same Emacs
> > developers who respond to bugs that concern Emacs in general, i.e., like
> > developers.  
> 
> Is this really so, in practice?  I have packages in ELPA which are effectively untouched except by me, other than on first ingestion.  And they draw updates from a repo I maintain myself.  Maybe I've misconstrued the situation, but my understanding has been that core packages receive far more attention from maintainers.  And rightly so, IMO: everyone has them installed, after all.  

Please don't be bothered by the maintenance burden of dormant
packages: it is a non-issue.

> >>   It deserves support in Emacs.  Just not, IMO, in core.
> > 
> > Why not?  Why does _anything_, to speak nothing of a package already in
> > core, not "deserve" support in core, while deserving support in ELPA?
> 
> Because in ELPA, users must proactively opt-in to the use of the package.  For such users — those who have actively sought it out — in stark contrast to the vast majority of Emacs users, the benefits dramatically outweigh the costs.

The costs are minuscule, so it isn't worth considering them in such
cases.

Re: Moving packages out of core to ELPA

[Dmitry Gutov]

On 18/02/2024 09:14, Eli Zaretskii wrote:
>>>>    It deserves support in Emacs.  Just not, IMO, in core.
>>> Why not?  Why does_anything_, to speak nothing of a package already in
>>> core, not "deserve" support in core, while deserving support in ELPA?
>> Because in ELPA, users must proactively opt-in to the use of the package.  For such users — those who have actively sought it out — in stark contrast to the vast majority of Emacs users, the benefits dramatically outweigh the costs.
> The costs are minuscule, so it isn't worth considering them in such
> cases.

Those who say that the costs of maintenance is tiny, seem to have missed 
the simple statement that the built-in version has diverged from its 
upstream 7 years ago. The maintenance is clearly not being done.

Re: Moving packages out of core to ELPA

[Eli Zaretskii]

> Date: Sun, 18 Feb 2024 14:19:05 +0200
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 18/02/2024 09:14, Eli Zaretskii wrote:
> >>>>    It deserves support in Emacs.  Just not, IMO, in core.
> >>> Why not?  Why does_anything_, to speak nothing of a package already in
> >>> core, not "deserve" support in core, while deserving support in ELPA?
> >> Because in ELPA, users must proactively opt-in to the use of the package.  For such users — those who have actively sought it out — in stark contrast to the vast majority of Emacs users, the benefits dramatically outweigh the costs.
> > The costs are minuscule, so it isn't worth considering them in such
> > cases.
> 
> Those who say that the costs of maintenance is tiny, seem to have missed 
> the simple statement

They did?

Re: Moving packages out of core to ELPA

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]



> On Feb 18, 2024, at 8:06 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Sun, 18 Feb 2024 14:19:05 +0200
>> Cc: luangruo@yahoo.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> On 18/02/2024 09:14, Eli Zaretskii wrote:
>>>>>>   It deserves support in Emacs.  Just not, IMO, in core.
>>>>> Why not?  Why does_anything_, to speak nothing of a package already in
>>>>> core, not "deserve" support in core, while deserving support in ELPA?
>>>> Because in ELPA, users must proactively opt-in to the use of the package.  For such users — those who have actively sought it out — in stark contrast to the vast majority of Emacs users, the benefits dramatically outweigh the costs.
>>> The costs are minuscule, so it isn't worth considering them in such
>>> cases.
>> 
>> Those who say that the costs of maintenance is tiny, seem to have missed 
>> the simple statement
> 
> They did?

Indeed my repo diverged substantially from the version in core (>10 yrs ago), because the changes made in core were never communicated to me, and I didn't know to check for them, such that by the time I discovered them, the divergence was too great.  This all predated ELPA of course. 

I think on all sides this was inadvertent.

[-- Attachment #2: Type: text/html, Size: 2925 bytes --]

Re: Moving packages out of core to ELPA

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Those who say that the costs of maintenance is tiny, seem to have
                                                 ....
> missed the simple statement that the built-in version has diverged
> from its upstream 7 years ago. The maintenance is clearly not being
                                                            ...
> done.

We stand corrected.  The cost is _zero_, not tiny.

Re: Moving packages out of core to ELPA

[Dmitry Gutov]

On 18/02/2024 15:06, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> Those who say that the costs of maintenance is tiny, seem to have
>                                                   ....
>> missed the simple statement that the built-in version has diverged
>> from its upstream 7 years ago. The maintenance is clearly not being
>                                                              ...
>> done.
> We stand corrected.  The cost is_zero_, not tiny.

We seem to have a different understanding of the word "cost".

Re: Moving packages out of core to ELPA

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> We seem to have a different understanding of the word "cost".

Yours appears to revolve around the cost to "Emacs maintenance"--but
this is an abstract conception offering little in the way of information
that might contribute to a sound decision.  My definition is the
aggregate of the costs to each Emacs developer, which is zero, inasmuch
as work is not being done.

Re: Moving packages out of core to ELPA

[Dmitry Gutov]

On 18/02/2024 15:51, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> We seem to have a different understanding of the word "cost".
> Yours appears to revolve around the cost to "Emacs maintenance"--but
> this is an abstract conception offering little in the way of information
> that might contribute to a sound decision.  My definition is the
> aggregate of the costs to each Emacs developer, which is zero, inasmuch
> as work is not being done.

With "zero cost to each developer", the package now lies in a state 
which cannot be called "well-maintained".

So not only that means that the cost to have it in good state is 
non-zero, it also shows that the Emacs maintainers failed to identify 
the problem for a number of years. Because people weren't willing to 
expend the effort necessary to do so.

Re: Moving packages out of core to ELPA

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> With "zero cost to each developer", the package now lies in a state
> which cannot be called "well-maintained".
>
> So not only that means that the cost to have it in good state is

When few bug reports arrive and the package doesn't figure in our work
except when it attracts as to its apparent inactivity, the package is
either in a good state or does not require our attention by virtue of
disuse.  So this is not a problem at all.

Re: Moving packages out of core to ELPA

[Dmitry Gutov]

On 18/02/2024 16:26, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> With "zero cost to each developer", the package now lies in a state
>> which cannot be called "well-maintained".
>>
>> So not only that means that the cost to have it in good state is
> When few bug reports arrive and the package doesn't figure in our work
> except when it attracts as to its apparent inactivity, the package is
> either in a good state or does not require our attention by virtue of
> disuse.  So this is not a problem at all.

That implies that every problem encountered ends up in our bug tracker.

Re: Moving packages out of core to ELPA

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
>> Those who say that the costs of maintenance is tiny, seem to have
>                                                  ....
>> missed the simple statement that the built-in version has diverged
>> from its upstream 7 years ago. The maintenance is clearly not being
>                                                             ...
>> done.
>
> We stand corrected.  The cost is _zero_, not tiny.

It's not zero here.  See Bug#39992 and Bug#69171.

To be honest, I find the argument that maintaining code can somehow be
done for free surprising.  AFAICT, the only code that doesn't increase
the maintenance burden is the code that we don't have.  I'm therefore
not convinced by those that ask us to never delete anything.

If idlwave.el is not a sufficient example, I just spent some time on
private correspondence regarding another library in Emacs that I
personally have wanted to chuck into lisp/obsolete for years.  A few
weeks ago, I installed a patch someone had sent for another case much
like that.

Is that a good use of my time?  I'm not so sure.  But just ignoring
parts of Emacs and let them bitrot doesn't seem very attractive either.
I think that we do have to take some minimal amount of responsibility
for the things that we ship.

That said, we don't have too many such files -- in large part thanks to
obsoletions that have been done over the years -- but they do still
exist.  Given our highly limited resources, I think its useful to
occasionally ask ourselves if keeping this or that library around is
still worth it, and then obsolete it if it isn't.