Re: Unboxed package manager

[Lynn Winebarger <owinebar@gmail.com>]

On Wed, Mar 22, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Tue, 21 Mar 2023 20:48:29 -0400
> > Cc: Gregory Heytings <gregory@heytings.org>, casouri@gmail.com, emacs-devel@gnu.org
> >
> > > A cache will not necessarily help, and you will pay the price of
> > > searching it at least once.  It should be clear that enlarging the
> > > length of load-path is not scalable, unless we make it a hash-table of
> > > sorts.  And I'm not sure we want this kind of complexity for such a
> > > hilariously marginal use case.
> >
> > Maybe 2000+ is marginal (today), but let's say someone just installs
> > the 300 packages available on gnu elpa.
>
> Why would someone want all 300 of them?  Some of them even contradict
> each other, in that they implement similar features in very different
> ways.

You're correct, many of them do address similar issues and should not
be "turned on" simultaneously.  As long as they are well-behaved and
able to be switched on and off reliably, though, there's no reason to
not have them simultaneously installed and loaded if the user is not
dedicated to minimizing resource consumption.
*  As a user, I don't want to be locked into one approach or the other
by the pain of having to uninstall one (set of) package(s) and install
another (set of) package(s).  That's usually one of the practical
benefits of free software.
*  It's easier if constructing a configuration that will be shared by
multiple users who either aren't inclined or aren't allowed to add
packages dynamically and/or write extensive elisp in an initialization
file, whether dynamically loaded or in a large dump file.

As I've noted in other responses, there is a substantial contingent of
users making use of customized package distributions whose organizers
appear to go to some length to keep load times reasonable.  Some of
them use extensive configuration to ensure only selected packages are
added to the load path, regardless of whether they are present on
disk.  Then there's the chemacs tool to completely switch
initialization directories.  I don't know the extent to which this
basic performance limitation in the design of package management is
causing this, but I have to think that if users found that they were
able to just load everything and simply turn features on and off at
will, some substantial contingent would find the customize system much
more useful and have less need of maintaining these alternative
configuration management systems.

One fellow using my shared setup on the sandboxed system regularly
uses doom emacs on his personal system.  He gave me a substantial set
of packages to add to the ones I was putting in a (byte-compiled) dump
file.  We use the same dump file (which I have arranged to remove all
traces of the build user in variables and replace with values
appropriate for the actual user at startup).   He's recreated doom
emacs (as far as I know) and I run a much more vanilla configuration.
He can use helm and I can use ivy or selectrum or vertico using
customize.  The dump file loads in a few seconds.  Even without
dumping, loading the world (which is over a 1000 packages + all
libraries included with core emacs) only takes ~30 seconds or so, as I
have manually unboxed almost all the packages.  The memory used by the
process is comparable to what VSCode would consume.

So, I don't find it hilariously marginal at all.  I think it's a
severe constraint that discourages non-experts from using emacs to its
fullest when they have alternatives like VSCode that come with a lot
of bells and whistles ready to go.  And a big chunk of the performance
pain appears to be associated with the design choice of putting
packages in their own directory, even when it's completely unnecessary
for their correct functioning.

And installing/deinstalling packages is a slow business.  The fact
that some of these projects go further and finely divide their files
into multiple packages so the user can tune the parts just aggravates
the situation - and makes switching between different approaches by
uninstalling one and installing the other even more painful.

>
> > Note I'm just installing
> > these packages, not actually loading any of them directly.
>
> Exactly.  So this is entirely theoretical use case, not a real one.

I was just noting that the performance hit comes from merely
installing the package (and enlarging the load-path), not from loading
it.

I will be loading all the packages and dumping them, once I have a
systematic unboxing facility.

> > Although if you look at something like doomemacs, people go to a lot
> > of trouble to make packages available on their system without paying
> > this overhead.  The simplest approach is probably just to avoid
> > increasing the size of the load path when it's not really necessary.
>
> As I already said: you can simply load each package by its explicit
> absolute name.  Poof! problem disappeared.

You're incorrect, unless you can ensure required system packages are
all explicitly loaded in order.  SImply specifying the full path of
one will not prevent it searching for its dependencies.

The analysis of the cost of pre-computing the load-path is also
misguided.  The fact that you would have to traverse the load-path at
least once to build the tables is hardly a criticism, since the
alternative is performing the search on every load.

That said, I don't advocate adding that complexity when rationalizing
the way packages are installed is an effective way to avoid enlarging
the load-path in the first place.  The number of packages whose design
really requires enlarging the load-path (just as a user of the
package, not in "development mode") is tiny in my experience.  realgud
and its extensions are in fact the only ones that come to mind,
because of the use of load-relative.  Others may be tricky to handle
automatically, but there's nothing in their design that fundamentally
requires enlarging the load path - a few manual modifications are
usually enough to handle those cases, which are fairly small in number
(magit being one of them due to the version function expecting to find
the package installed in its own directory or to be able to contact
the main git repository).

Lynn