Re: Request to add Package to GNU ELPA

[Philip Kaludercic <philipk@posteo.net>]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Jonas Bernoulli <jonas@bernoul.li> writes:
>>
>>> Hello,
>>
>> Hi, hope I can add a few useful comments here.
>>
>>> I would like to request that "package.el" be added to GNU Elpa as a
>>> "core" package.  This would make new features available to users who are
>>> stuck on older Emacs releases and it would even bring us one step closer
>>> to being able to make backward incompatible changes in the future.
>>>
>>> Some examples for why that would be desirable:
>>>
>>> - A recent addition that could be useful is the new "package-vc.el"
>>>   (which I think should be distributed as part of the `package' package,
>>>   but it could also be distributed as a separate core package).
>>
>> This will be difficult, since package-vc depends on the new "vc-clone"
>> function, which would also have to be provided on ELPA to work.
>
> The simplest way to deal with that, while still making the latest
> Package available to users of older Emacs releases, would be to *not*
> include package-vc.el in the Package package.
>
> Going one step further we could make package-vc available as a separate
> package.  That would not be of much use *now*, but future improvements
> would then be available to users of Emacs 29.

I would in principle be interested in supporting this.

> I had a look at vc-clone and a few vc-*-clone.  They seem trivial
> enough to backport, either as part of Compat or in package-vc.el.

[...]

>> All of this touches on the general topic of how Emacs should be
>> developed: Small and to be extended, therefore of little use OOTB or big
>> and feature-full, but slow to be updated.  IIUC a compromise could be
>> achieved by having two versions of Emacs that would respectively attempt
>> to satisfy the two aspirations.
>
> This doesn't sound like a good idea to me because I doubt that it would
> have the benefits you seem to expect.  A sizable portion of an external
> package would choose to use the slow-moving Emacs.  So the maintainer of
> the package would have to continue to support that, similar to how they
> now have to support older releases.  So even though there would be a
> fast-moving Emacs, they still would not be able to take full advantage
> of the new/improved features it provides.  The situation would be
> basically the same as now; but not supporting the slow-moving Emacs
> would be even harder to justify than dropping support for Emacs 25.1.

I am not really advocating for one or the other option, just wanted to
bring it up, if for nobody else then just for people lurking the mailing
list.

>> There was some work in that direction a
>> few years ago[0], but I don't know if anything has happened since.
>> Perhaps pushing for this more general solution would help solve this
>> particular problem?
>>
>> [0] https://git.savannah.gnu.org/cgit/emacs.git/log/?h=elparized-core
>
> I remember that there was some talk about moving in that direction, and
> I am in favor and have been patiently waiting for someone to make
> another push in that direction, but things indeed seem to have fizzled
> out.
>
> Meanwhile making certain packages available on a case by case basis does
> not seem like a bad idea to me.  It doesn't mean that we *commit* to
> making "most" packages available like that eventually, so even those you
> are opposed to that idea might be able to agree to doing it selectively.
> Doing it for a select few packages, allows us to gain some experience,
> whether that be bad or good.

True.

>>>   That isn't just useful for users who like to live on the edge.  For
>>>   example, if a package drops support for an old Emacs release, then
>>>   a user who is stuck on that release, may wish to keep using the last
>>>   release of the package that still supports that Emacs release, and
>>>   package-vc would allow them to do just that.
>>>
>>> - I don't use Package myself, so I am not fully aware of all the quality
>>>   of live enhancements that surely have accumulated over the years.  But
>>>   I am aware of some small missing features, that would be beneficial to
>>>   users of older Emacs releases.
>>>
>>> - For example, I think it would be nice if the columns displayed by
>>>   `list-packages' were customizable.  A user might want to increase the
>>>   width of the "Version" column, so that not every GNU-devel ELPA version
>>>   is truncated, 
>>
>> A more general solution to this problem would be if tabulated-list-mode
>> could dynamically resize a column, or at least shrink it possible.
>
> That would indeed be nice ... and would make tabulated-list-mode another
> candidate for GNU ELPA. ;P

It seems like this is possible, or at least I didn't see any difficult
dependencies from a brief skim.

>> Also on another topic, I am still uncertain as to the status of the
>> -devel archives.  I see them primarily as a means for developers to
>> check the status of a package pre-release or to make sure that the ELPA
>> build-system is working the way they want it to (e.g. when building
>> manuals, news or README files).  They are not advertised anywhere, and I
>> don't think they are made for general consumption -- or rather I don't
>> think they should be, especially with package-vc there is a much better
>> way to track development and make it possible to contribute changes
>> upstream.
>
> Most packages still get installed through Melpa these days and that
> means installing the latest commit from the development branch.  (The
> [Non]GNU-devel ELPA variants probably exist in part to satisfy users who
> are used to having access to the development version through "regular"
> package i.e., without using package-vc.)
>
> In my experience that seems to work surprisingly well; I can't really
> remember any regressions that were not fixed almost instantly, but that
> does of course depend on what packages one has installed.  I am sure
> little regressions happen all the time, but they get fixed so quickly,
> that any one user usually doesn't notice.  In my experience, using the
> development versions of some quality packages, is no different from
> using the development version of Emacs itself.

My experience was not that good, especially back before I got into Emacs
development (or understood what was going on at all) I constantly ran
into issues when updating packages from MELPA.  The result was that I'd
usually just not update packages at all for long periods of time.  The
next best thing for me was MELPA stable until NonGNU ELPA came around.

> But I agree that it would be better to release more often (and yes, I
> am guilty of not doing that myself), and tend to agree that it would be
> better if, in the long run, the community moved towards relying more on
> released versions.
>
> At the same time I don't think that the fact that we are not there yet
> is due to "mistakes".  Emacs/Elisp is different from other "languages",
> in that everything runs in the same process, and that is at least one
> of the causes of the status quo.
>
> We cannot really use multiple versions of the same library in different
> parts of Emacs, in the same way one Python or Rust application could
> use xyz-v1 and another, related but independent, application could use
> xyz-v2.  If one package depends on xyz-v2 but there were incompatible
> changes in that release and, another package has not been updated yet to
> support xyz-v2, then those two packages cannot be used at the same time.
>
> Again, that does not seem to happen very often, and I think that is
> partially the case because the hold out package can be quickly updated
> to support v2 on its development branch, *without* also having to do a
> release, which the maintainers might not wish to do due to where in its
> own release-cycle the package happens to be.
>
>>> - I should also mention that my main motivation for pushing for this
>>>   now, is that I would like to improve version handling.  That is a
>>>   whole other can of worms, so I do not wish to discuss it just yet,
>>>   to avoid distracting from the topic at hand, but I thought I should
>>>   at least mention it.  You might very well end up being against my
>>>   suggestions regarding versions strings, once I present them, but that
>>>   should not be cause to oppose the change requested here as well.  Even
>>>   if my suggestions regarding version strings are ultimately rejected, I
>>>   still think it would be a good idea to add `package' to GNU ELPA, for
>>>   the other reasons presented here.
>>
>> This implies that packages might themselves depend on newer versions of
>> package.el -- which I think one should put some thought into before
>> anything is done.
>
> How so? (Further down I suggest making some popular packages temporarily
> depend on Package from GNU Elpa as a means to get that installed as
> widely as possible, but that doesn't mean that those packages actually
> *need* a newer Package.)
>
>> How can you e.g. change the way versions are handled
>> in a way that people with older versions of package.el could still
>> handle the change without confusion?
>
> You can't, and that is exactly the point I was trying to make.  Certain
> things (including, but not limited to how version strings are handled)
> are effectively set in stone, unless certain packages/features are made
> available to users who, for whatever reason, stick to an old Emacs
> release.

Other than the version, it seems the other big one is the dependency
list.  Is there anything else that could cause issues?

>> (This will be an issue for at
>> least a decade to come, knowing the pace at which users get access to
>> newer versions of Emacs).  You mentioned updating the ELPA protocol a
>> few months back, and I believe a few points of interest were collected
>> in that thread.  It seems to me these issues are all intertwined.
>
> Yes indeed, they are all intertwined, and I am trying to make the point
> that we have to stop muddling along once in a while, and make a breaking
> change, even if the transition is rough and inconveniences users for a
> while.
>
>> With Emacs 29, `package-refresh-contents' now calls a Hook called
>> `package-refresh-contents-hook'.  This was initially introduced so that
>> package-vc can download the "elpa-packages.eld" file, but it could also
>> be of use here as well -- but it wouldn't help anyone with Emacs 28 or
>> older...  I could imagine a more hacky approach based on carefully
>> placed advice, but only as a last resort.
>
> We cannot travel back in time to fix past mistakes/omissions that are
> holding us back in the presence.  All we can do now is to pay the price
> and try harder to be forward compatible this time around.  Or live with
> the limitations forever.  Or wait some more until they become unbearable
> and a fix more urgent; but that would increase the odds of messing it up
> again.

2+

>>> Addressing (1) is harder, and I don't think it possible to do so in a
>>> way that guarantees that not a single user would end up seeing an error
>>> due to an incompatible change.  On the other hand "archive-contents"
>>> comes with a version field, and if we bump that, we at least get a
>>> meaningful warning: "Package archive version 2 is higher than 1".  This
>>> doesn't say that the solution is to install `package' from GNU ELPA, but
>>> it should be enough to feed into a search engine to get to that answer.
>>
>> The upgrading procedure for archive-contents has never appeared to me to
>> be very robust.  Perhaps it would be better to introduce a second file
>> (archive-contents.eld) with a more flexible format?
>
> An effort was made to provide an upgrade procedure -- the
> "archive-contents" were prefixed with a version number.  But then
> package.el was added to Emacs and we stopped to distribute it
> separately.  Apparently we didn't realize at the time that this only
> allowed us to tell the user "the archive and tool are not compatible,
> deal with it".

What could be done instead?  Should the updated version of package.el
try to install a newer version of itself in case the version string is
updated?

> We could continue to distribute "archive-contents", which continues to
> use version 1, and add "archive-contents.eld" which uses version 2.
> But that would do nothing to get users to install a forward-compatible
> version of Package (in the sense that once it becomes incompatible, it
> will provide a smooth upgrade path to the new version).

This seems like the better option to me.  Most people don't /need/ the
newest version right away, and even if they did there would be no
straightforward way of making sure all users would have it installed
within some fixed time-frame.

>>> Additionally, we could get many users to install `package' from GNU
>>> ELPA, by making a few popular packages explicitly depend on a `package'
>>> version that is available from GNU ELPA for a few months.
>>
>> Again, a few months wouldn't be enough to solve the issue.  But again as
>> well, Compat could help by adding package as a noop dependency.
>
> Much like your "archive-contents.eld" suggestion, doing this would not
> *solve* the issue.  Unlike that, it would however significantly reduce
> the number of users who would be negatively affected by the upgrade.
> (We could do it for a few years instead of months to further decrease
> the number of affected users.)

You are probably right.

> I love Compat, but I don't think it is the right hammer here (except
> maybe for the part where it could provide vc-clone et al.).
>
> Maybe it helps to make it explicit what is going to happen to users
> if/when all the *ELPA start to expect that package.el support
> "archive-contents" v2.
>
> - The user runs "M-x package-refresh-contents" and is told "Package
>   archive version 2 is higher than 1".
>
> - They ask a search engine what that means.  We could create a few blog
>   and forum posts before hand and give the search engines a few days to
>   index them, providing the following instructions:
>
>   (We should invest some time to investigate how to make this as smooth
>   as possible, but basically:)
>
> - 1. wget https://...package-2.0.0.tar
>   2. tar -xf package-2.2.0.tar -C ~/.config/emacs/elpa/
>   3. Restart Emacs and run package-refresh-contents again.

This is exactly what I would like to avoid.  What happens when the .tar
disappears and some user tries to fix this issue in a few years with
these outdated instructions?  What if they use .emacs.d or some other
directory?  For while there will be plenty of users that will figure
this out, I know many others will be confused (this also makes me think
that a v2 should have support for some kind of MOTD system to explain
issues like these).

So again, what would be the issue with an opt-in system to upgrading
package.el, that could also be pushed forward by a few popular packages,
without the need to break anything.

>      Cheers, 
>      Jonas