Unboxed package manager

Unboxed package manager

[Lynn Winebarger]

Hi,
I want to write a small package "unboxed" that extends the built-in
package manager with packages whose elisp files are installed into a
common packages directory instead of one directory per package.  I've
done this manually on various systems I use with a significant
improvement in startup performance.  Of 2403 packages I've installed
on my current personal system, only 161 have a subdirectory.  Ignoring
"-pkg.el" and "-autoloads.el" files, 1971 of those 2403 packages have
a single elisp file.  I'm including a complete histogram table at the
end of this mail for reference.
I would like to just tweak the existing functionality so that, if a
predicate is satisfied, a given package will be locally administered
by "unboxed" rather than "package":
* mapping package contents to local paths for installation and removal
(e.g. info files go to a common info directory for unboxed packages,
themes to a separate theme directory, etc)
* creating autoloads for the common directory, preferably after an
entire set of packages has been installed/removed rather than
one-at-a-time
* providing package-desc objects to the package initialization routine
I can think of 3 predicates for determining when "unboxed" might be
assigned, depending on how risk-adverse the user is
1) A package contains a single elisp library
2) A package has no subdirectories
3) A package has subdirectories but it is referenced through some
variable set in one of the source files, which can be patched.
Most packages are amenable to the 3rd option - with the exception of a
small number of cases, of which none that I can recall have a
subdirectory anyway, the variable in question contains the only
reference to "load-file-name".
There are some packages that simply cannot be managed this way without
extraordinary effort, basically any package using "load-relative".
Packages that attempt to facilitate active development even after
installation can also do things that are problematic, like look for
git information in the installation directory.  An explicit list of
these cases (as a customization variable) can be maintained to ensure
they are handled the default way.
From my perspective, package.el already supports 2 package tracking
mechanisms - one for "built-in" packages, and the usual one for
external packages or updates to the built-in packages.  However, if
there is no appetite for facilitating a more general "wiring in" of
additional local package management (which might be useful if you
wanted to enable backends with an explicit database of installed
packages), then my best option would appear to be to derive from the
package-menu-mode as a replacement, and either define variants of the
functions in the package-menu and package namespaces or use advice.
The latter goes against my taste in code, but is probably the most
expedient.
Can any maintainers of package.el advise?

The following is the promised histogram of the number of packages from
my sample with a specified number of elisp libraries.  2403 package
directories in load-path absolutely devastates locate-file.  Even the
most conservative approach above (option 1) would drastically reduce
this effect.
#libs  #packs   Percent
 1         1971       82.022
 2          148       88.181
 3           57       90.553
 4           37       92.093
 5           17       92.801
 6           49        94.84
 8           11       95.298
 9           17       96.005
10            8       96.338
11            5       96.546
12            8       96.879
13            7        97.17
15            7       97.462
16            4       97.628
17            5       97.836
18            6       98.086
19            2       98.169
20            2       98.252
21            2       98.335
22            6       98.585
23            1       98.627
24            3       98.752
26            2       98.835
27            9       99.209
29            1       99.251
30            1       99.293
31            4       99.459
32            2       99.542
35            1       99.584
36            2       99.667
37            1       99.709
41            2       99.792
45            2       99.875
66            1       99.917
77            1       99.958
95            1          100

For the curious, the last three are jdee, doom-themes, and lsp-mode
respectively.  They can be "installed" by simply copying them into a
common directory, updating the directory autoloads, and byte-compiling
without any issue.
Thanks,
Lynn

Re: Unboxed package manager

[Yuan Fu]

> On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com> wrote:
> 
> I've
> done this manually on various systems I use with a significant
> improvement in startup performance.

It'll be interesting to see the numbers on the improvement. How much does it improve startup time if there is x packages and y are loaded at startup time?

Yuan

Re: Unboxed package manager

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2279 bytes --]

On Mon, Mar 20, 2023, 2:31 AM Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com> wrote:
> >
> > I've
> > done this manually on various systems I use with a significant
> > improvement in startup performance.
>
> It'll be interesting to see the numbers on the improvement. How much does
> it improve startup time if there is x packages and y are loaded at startup
> time?
>


Good question. The systems I've done this manually on are one-off builds of
28.x on sandboxed systems where the system emacs is 24.3.  So, I didn't
have to worry about managing package updates with any frequency.

I'm now working on unsandboxed systems where the system version is 27.2 and
I've installed 28.x, 29.x and 30.x builds into /use/local.  I don't want to
do a manual installation here because I (would like to) regularly update
packages with the package menu.   Additionally I discovered that 30.x (and
probably 29.x) has introduced a new bytecode instruction, after installing
packages under 30.x then trying to run 28.x.  So I've set up my
initialization files to use different custom-files, package archive
directories, and package-quickstart.el files tied to the major version of
emacs that's running.  Running the install through
package-install-selected-packages for each version is incredibly slow, but
seems the safest way for now.  Hence my interest in implementing a more
systematic approach now that doesn't rebuild a massive package-quickstart
file for every package installed.

I did file a bug report as a place to hang cpu and memory profiles of the
initialization when I loaded everything at startup under 27.2 - see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61004 .  If memory serves,
locate-file took something like 75% of the cpu time, and also did an
incredible (and dominant) amount of consing. Even if the autoloads in
package-quickstart specified a full path for the library (they don't),
every "require" of an unloaded system library still has to search the
entire 2403 (or however many were in that bug report) package directories
first, for every system library required, even if no packages are actually
loaded.

I'm using 28.x as my reference build because re-dumping has been broken
under 29.x.

Lynn

[-- Attachment #2: Type: text/html, Size: 3266 bytes --]

Re: Unboxed package manager

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1684 bytes --]

On Mon, Mar 20, 2023, 4:55 AM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Mon, Mar 20, 2023, 2:31 AM Yuan Fu <casouri@gmail.com> wrote:
>
>>
>>
>> > On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com>
>> wrote:
>> >
>> > I've
>> > done this manually on various systems I use with a significant
>> > improvement in startup performance.
>>
>> It'll be interesting to see the numbers on the improvement. How much does
>> it improve startup time if there is x packages and y are loaded at startup
>> time?
>>
>
>
> Good question. The systems I've done this manually on are one-off builds
> of 28.x on sandboxed systems where the system emacs is 24.3.  So, I didn't
> have to worry about managing package updates with any frequency.
>

I don't have any hard measurements, but my recollection is that installing
~1200 packages on those systems and "loading the world" took something like
5 minutes, while a similar set of packages installed in a unified directory
takes about 1.5 minutes.  Also, when set up with AOT native compiled
libraries, the startup time was more like 15 minutes, or at least long
enough for me to walk away, make coffee, drink the coffee slowly, and still
find the startup in progress.  The native libraries are in a cache
directory, but they have to locate the source file and compute a checksum
at load time, so the length of the load-path is still an issue....

Take those numbers with a container of salt though, since I did that last
summer and took a big detour to effectively enable dumping all the packages
as preloaded native-compiled libraries before going back and trying just
working with the byte-compiled version in December.

Lynn

[-- Attachment #2: Type: text/html, Size: 2995 bytes --]

Re: Unboxed package manager

[Philip Kaludercic]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Mon, Mar 20, 2023, 4:55 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
>> On Mon, Mar 20, 2023, 2:31 AM Yuan Fu <casouri@gmail.com> wrote:
>>
>>> > On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com>
>>> wrote:
>>> >
>>> > I've
>>> > done this manually on various systems I use with a significant
>>> > improvement in startup performance.
>>>
>>> It'll be interesting to see the numbers on the improvement. How much does
>>> it improve startup time if there is x packages and y are loaded at startup
>>> time?
>>
>> Good question. The systems I've done this manually on are one-off builds
>> of 28.x on sandboxed systems where the system emacs is 24.3.  So, I didn't
>> have to worry about managing package updates with any frequency.
>
> I don't have any hard measurements, but my recollection is that installing
> ~1200 packages on those systems and "loading the world" took something like
> 5 minutes, while a similar set of packages installed in a unified directory
> takes about 1.5 minutes.

I just skimmed through the initial message, and when reading this my
initial thought is does the the advantage justify effort/complexity,
considering that most users have an order of magnitude fewer packages
installed (of which a few are installed using package-vc which IIUC
would explicitly not want to make use of this feature).

-- 
Philip Kaludercic

Re: Unboxed package manager

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2010 bytes --]

On Mon, Mar 20, 2023, 11:25 AM Philip Kaludercic <philipk@posteo.net> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > On Mon, Mar 20, 2023, 4:55 AM Lynn Winebarger <owinebar@gmail.com>
> wrote:
> >
> >> On Mon, Mar 20, 2023, 2:31 AM Yuan Fu <casouri@gmail.com> wrote:
> >>
> >>> > On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com>
> >>> wrote:
> >>> >
> >>> > I've
> >>> > done this manually on various systems I use with a significant
> >>> > improvement in startup performance.
> >>>
> >>> It'll be interesting to see the numbers on the improvement. How much
> does
> >>> it improve startup time if there is x packages and y are loaded at
> startup
> >>> time?
> >>
> >> Good question. The systems I've done this manually on are one-off builds
> >> of 28.x on sandboxed systems where the system emacs is 24.3.  So, I
> didn't
> >> have to worry about managing package updates with any frequency.
> >
> > I don't have any hard measurements, but my recollection is that
> installing
> > ~1200 packages on those systems and "loading the world" took something
> like
> > 5 minutes, while a similar set of packages installed in a unified
> directory
> > takes about 1.5 minutes.
>
> I just skimmed through the initial message, and when reading this my
> initial thought is does the the advantage justify effort/complexity,
> considering that most users have an order of magnitude fewer packages
> installed (of which a few are installed using package-vc which IIUC
> would explicitly not want to make use of this feature).
>

I have no idea.  There's a bit of a chicken and egg problem when installing
packages is slow (even with a local mirror of the elpas) and having a lot
of packages loaded slows down startup.
I think I'm going to hack something together starting with advice on the
existing package management and taking some inspiration from the design of
Jonas Bernoulli's epkg and emir packages for tracking installed packages
and component files in a SQLite database.

Lynn

[-- Attachment #2: Type: text/html, Size: 3358 bytes --]

Re: Unboxed package manager

[Philip Kaludercic]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Mon, Mar 20, 2023, 11:25 AM Philip Kaludercic <philipk@posteo.net> wrote:
>
>> Lynn Winebarger <owinebar@gmail.com> writes:
>>
>> > On Mon, Mar 20, 2023, 4:55 AM Lynn Winebarger <owinebar@gmail.com>
>> wrote:
>> >
>> >> On Mon, Mar 20, 2023, 2:31 AM Yuan Fu <casouri@gmail.com> wrote:
>> >>
>> >>> > On Mar 19, 2023, at 6:18 PM, Lynn Winebarger <owinebar@gmail.com>
>> >>> wrote:
>> >>> >
>> >>> > I've
>> >>> > done this manually on various systems I use with a significant
>> >>> > improvement in startup performance.
>> >>>
>> >>> It'll be interesting to see the numbers on the improvement. How much
>> does
>> >>> it improve startup time if there is x packages and y are loaded at
>> startup
>> >>> time?
>> >>
>> >> Good question. The systems I've done this manually on are one-off builds
>> >> of 28.x on sandboxed systems where the system emacs is 24.3.  So, I
>> didn't
>> >> have to worry about managing package updates with any frequency.
>> >
>> > I don't have any hard measurements, but my recollection is that
>> installing
>> > ~1200 packages on those systems and "loading the world" took something
>> like
>> > 5 minutes, while a similar set of packages installed in a unified
>> directory
>> > takes about 1.5 minutes.
>>
>> I just skimmed through the initial message, and when reading this my
>> initial thought is does the the advantage justify effort/complexity,
>> considering that most users have an order of magnitude fewer packages
>> installed (of which a few are installed using package-vc which IIUC
>> would explicitly not want to make use of this feature).
>>
>
> I have no idea.  There's a bit of a chicken and egg problem when installing
> packages is slow (even with a local mirror of the elpas) and having a lot
> of packages loaded slows down startup.

What is the chicken and the egg in this situation?

> I think I'm going to hack something together starting with advice on the
> existing package management and taking some inspiration from the design of
> Jonas Bernoulli's epkg and emir packages for tracking installed packages
> and component files in a SQLite database.

I'd be interested to see how this looks like.  As I said, I am sceptical
of the need to optimise these extreme cases for everyone, but perhaps we
can also open package.el to be extended externally in ways that your
work could plug into.

> Lynn

(Also, could you leave an empty line between paragraphs in your
messages?  It would make it easier to read them)

-- 
Philip Kaludercic

Re: Unboxed package manager

[Jonas Bernoulli]

Lynn Winebarger <owinebar@gmail.com> writes:

> I think I'm going to hack something together starting with advice on the
> existing package management and taking some inspiration from the design of
> Jonas Bernoulli's epkg and emir packages for tracking installed packages
> and component files in a SQLite database.

If I were to start over now, I wouldn't write Closql.  At the time it
made a lot of sense because I knew nothing about databases and because
it allowed me to switch out the old data store ("everything is *its own*
file") without changing internal interfaces much.  Moving from files to
a database did wonders for performance, so at first I didn't mind the
performance impact of the OO interface on top of the database.
Meanwhile I have moved away from the OO interface for anything that
deals with more than one package at a time, turning many rows into
EIEIO objects is a bit costly.

When I switched to SQLite, Emacs had no built-in support (coming in
Emacs 29) and there also was no module, so EmacSQL was the natural
choice.  I am the maintainer of that now, so I definitely think it
serves a purpose, but I do have some reservations.

The next release will feature new backends that use the built-in support
and a module, but if I were to start now, I probably would go with the
built-in support directly.

EmacSQL allows writing SQL using vectors instead of concatenating
strings, which is nice, but for people just getting started with SQL, it
has the disadvantage that you now have to learn two things, SQL and the
almost SQL vector syntax, which isn't 100% complete and doesn't map 1:1.

The main limitation of EmacSQL is that it stores everything (except
NULL) as a string.  This is why I would probably avoid it now, because
it limits interoperability with anything that doesn't use EmacSQL.

Re: Unboxed package manager

[Lynn Winebarger]

On Mon, Mar 20, 2023 at 2:11 PM Jonas Bernoulli <jonas@bernoul.li> wrote:
>
> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > I think I'm going to hack something together starting with advice on the
> > existing package management and taking some inspiration from the design of
> > Jonas Bernoulli's epkg and emir packages for tracking installed packages
> > and component files in a SQLite database.
>
> If I were to start over now, I wouldn't write Closql.  At the time it
> made a lot of sense because I knew nothing about databases and because
> it allowed me to switch out the old data store ("everything is *its own*
> file") without changing internal interfaces much.  Moving from files to
> a database did wonders for performance, so at first I didn't mind the
> performance impact of the OO interface on top of the database.
> Meanwhile I have moved away from the OO interface for anything that
> deals with more than one package at a time, turning many rows into
> EIEIO objects is a bit costly.

Hmm - I had thought it might be an interesting exercise to get
acquainted with Emacs's version of CLOS, but maybe not.  However, just
reviewing the way you've organized the package data is probably going
to be useful.  For example, trying to understand the best way to
assess whether a particular version of a package is an upgrade.  I
don't know about the behavior of package.el in master, but
historically, if there is a version of a package on MELPA and the same
package is available on GNU or NONGNU ELPA, package.el will treat the
version on GNU/NONGNU as an upgrade even though the one on MELPA is
more recent.  I assume it has something to do with the comparison of
commit hashes or dates with more traditional version numbers, but
that's just a guess.

>
> When I switched to SQLite, Emacs had no built-in support (coming in
> Emacs 29) and there also was no module, so EmacSQL was the natural
> choice.  I am the maintainer of that now, so I definitely think it
> serves a purpose, but I do have some reservations.
>
> The next release will feature new backends that use the built-in support
> and a module, but if I were to start now, I probably would go with the
> built-in support directly.
>
> EmacSQL allows writing SQL using vectors instead of concatenating
> strings, which is nice, but for people just getting started with SQL, it
> has the disadvantage that you now have to learn two things, SQL and the
> almost SQL vector syntax, which isn't 100% complete and doesn't map 1:1.
>
> The main limitation of EmacSQL is that it stores everything (except
> NULL) as a string.  This is why I would probably avoid it now, because
> it limits interoperability with anything that doesn't use EmacSQL.

That's only because it was designed to interact with sqlite through a
pipe to the shell program, though, right?  It seems like a method for
compiling sexpr-type representations of sql queries into statements
usable with the builtin support would still be useful, and not limited
in the same way - since the returned values do not require serializing
as text by the sqlite shell then parsing them in Elisp.

[ Digression...]
Just looking at the src/sqlite.c in master, as there is no other
documentation of the sqlite support I can see, only a simplified form
of the core API is supported - not unreasonable for an initial
release.  The only noticeable absence I see, based on a cursory review
of the sqlite3 API spec, is that a select query cannot be reset.
Maybe because each db connection is associated with at most one
prepared statement at a time by the design of the Lisp_Sqlite
pseudovector?

Other than that, I note that rows are returned as lists rather than
arrays, which makes the semantics more incompatible with emacsql than
it really has to be.  Can that be changed before 29 is released?

The associated sqlite-mode looks interesting.  I only wonder why it
doesn't derive from tabulated-list mode instead of directly from
special.  Tabulated list mode would seem to be made for things like
database tables.

Lynn

Re: Unboxed package manager

[Jonas Bernoulli]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Mon, Mar 20, 2023 at 2:11 PM Jonas Bernoulli <jonas@bernoul.li> wrote:
>>
>> Lynn Winebarger <owinebar@gmail.com> writes:
>>
>> > I think I'm going to hack something together starting with advice on the
>> > existing package management and taking some inspiration from the design of
>> > Jonas Bernoulli's epkg and emir packages for tracking installed packages
>> > and component files in a SQLite database.
>>
>> If I were to start over now, I wouldn't write Closql.  At the time it
>> made a lot of sense because I knew nothing about databases and because
>> it allowed me to switch out the old data store ("everything is *its own*
>> file") without changing internal interfaces much.  Moving from files to
>> a database did wonders for performance, so at first I didn't mind the
>> performance impact of the OO interface on top of the database.
>> Meanwhile I have moved away from the OO interface for anything that
>> deals with more than one package at a time, turning many rows into
>> EIEIO objects is a bit costly.
>
> Hmm - I had thought it might be an interesting exercise to get
> acquainted with Emacs's version of CLOS, but maybe not.

That is well worth it.

And Closql is at least interesting; it's just that "turn every row into
an object" is too slow if you have thousands of rows and all you really
want to do is display a table of some of the columns.  (If you do use
Closql, then use the "next" branch, that would already be merged, if I
weren't waiting for two downstreams to act.)

> However, just
> reviewing the way you've organized the package data is probably going
> to be useful.  For example, trying to understand the best way to
> assess whether a particular version of a package is an upgrade.  I
> don't know about the behavior of package.el in master, but
> historically, if there is a version of a package on MELPA and the same
> package is available on GNU or NONGNU ELPA, package.el will treat the
> version on GNU/NONGNU as an upgrade even though the one on MELPA is
> more recent.  I assume it has something to do with the comparison of
> commit hashes or dates with more traditional version numbers, but
> that's just a guess.

I'm afraid I am not doing any of that.  I just track the HEAD of the
most authoritative repository.


>
>>
>> When I switched to SQLite, Emacs had no built-in support (coming in
>> Emacs 29) and there also was no module, so EmacSQL was the natural
>> choice.  I am the maintainer of that now, so I definitely think it
>> serves a purpose, but I do have some reservations.
>>
>> The next release will feature new backends that use the built-in support
>> and a module, but if I were to start now, I probably would go with the
>> built-in support directly.
>>
>> EmacSQL allows writing SQL using vectors instead of concatenating
>> strings, which is nice, but for people just getting started with SQL, it
>> has the disadvantage that you now have to learn two things, SQL and the
>> almost SQL vector syntax, which isn't 100% complete and doesn't map 1:1.
>>
>> The main limitation of EmacSQL is that it stores everything (except
>> NULL) as a string.  This is why I would probably avoid it now, because
>> it limits interoperability with anything that doesn't use EmacSQL.
>
> That's only because it was designed to interact with sqlite through a
> pipe to the shell program, though, right?

No, it was a design decision by the original author, in order to keep
things simple.

> It seems like a method for
> compiling sexpr-type representations of sql queries into statements
> usable with the builtin support would still be useful, and not limited
> in the same way - since the returned values do not require serializing
> as text by the sqlite shell then parsing them in Elisp.

Of course "SQL as vectors" and "store everything as a string" are not
tied to each other per se, but with EmacSQL you get both.

> [ Digression...]
> Just looking at the src/sqlite.c in master, as there is no other
> documentation of the sqlite support I can see, only a simplified form
> of the core API is supported - not unreasonable for an initial
> release.  The only noticeable absence I see, based on a cursory review
> of the sqlite3 API spec, is that a select query cannot be reset.
> Maybe because each db connection is associated with at most one
> prepared statement at a time by the design of the Lisp_Sqlite
> pseudovector?
>
> Other than that, I note that rows are returned as lists rather than
> arrays, which makes the semantics more incompatible with emacsql than
> it really has to be.  Can that be changed before 29 is released?

Why is that a problem?  What would we gain if the DSL and the output
both used vectors or both lists?

> The associated sqlite-mode looks interesting.  I only wonder why it
> doesn't derive from tabulated-list mode instead of directly from
> special.  Tabulated list mode would seem to be made for things like
> database tables.
>
> Lynn

     Jonas

Re: Unboxed package manager

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

On Wed, Mar 22, 2023, 7:17 AM Jonas Bernoulli <jonas@bernoul.li> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > It seems like a method for
> > compiling sexpr-type representations of sql queries into statements
> > usable with the builtin support would still be useful, and not limited
> > in the same way - since the returned values do not require serializing
> > as text by the sqlite shell then parsing them in Elisp.
>
> Of course "SQL as vectors" and "store everything as a string" are not
> tied to each other per se, but with EmacSQL you get both.
>

The whole SQL as vectors conceit isn't something I've given a lot of
thought to. As long as it's a syntax tree and not a pile of strings.
Vectors or conses - both can be analyzed by pcase, so I start at a point of
indifference.


> > [ Digression...]
> > Just looking at the src/sqlite.c in master, as there is no other
> > documentation of the sqlite support I can see, only a simplified form
> > of the core API is supported - not unreasonable for an initial
> > release.  The only noticeable absence I see, based on a cursory review
> > of the sqlite3 API spec, is that a select query cannot be reset.
> > Maybe because each db connection is associated with at most one
> > prepared statement at a time by the design of the Lisp_Sqlite
> > pseudovector?
> >
> > Other than that, I note that rows are returned as lists rather than
> > arrays, which makes the semantics more incompatible with emacsql than
> > it really has to be.  Can that be changed before 29 is released?
>
> Why is that a problem?  What would we gain if the DSL and the output
> both used vectors or both lists?
>

It's not the SQL expressions I'm concerned about.  I'm just thinking of the
existing sql-using code base expects rows to be represented with arrays,
why add the cost of remarshalling them from lists?

It just seems like a gratuitous incompatibility to me, unless there's a
strong reason to prefer lists. I have no other reason to prefer either
representation.  The die is cast now, anyway.

Lynn

[-- Attachment #2: Type: text/html, Size: 3001 bytes --]

Re: Unboxed package manager

[Lynn Winebarger]

On Wed, Mar 22, 2023 at 10:31 AM Lynn Winebarger <owinebar@gmail.com> wrote:
> It's not the SQL expressions I'm concerned about.  I'm just thinking of the existing sql-using code base expects rows to be represented with arrays, why add the cost of remarshalling them from lists?
>
> It just seems like a gratuitous incompatibility to me, unless there's a strong reason to prefer lists. I have no other reason to prefer either representation.  The die is cast now, anyway.

I did think of one other hackish (possibly kludgish) reason to return
arrays rather than lists.  It makes storing/returning collections of
structs (as currently implemented, anyway) to/from rows very low
overhead, at least if there is a way to store LispObjects
transparently.  There isn't a way to do so currently, of course.

Lynn

Re: Unboxed package manager

[Augusto Stoffel]

On Mon, 20 Mar 2023 at 15:25, Philip Kaludercic wrote:

> I just skimmed through the initial message, and when reading this my
> initial thought is does the the advantage justify effort/complexity,
> considering that most users have an order of magnitude fewer packages
> installed (of which a few are installed using package-vc which IIUC
> would explicitly not want to make use of this feature).

I do think there's at least one aspect of this idea that many many
people need, namely: a simple way to add a directory such as
~/.emacs.d/lisp to the load path, with automatic support for autoloads,
byte-compilation and what not.  Such a thing is also useful if you are
developing some package, say at ~/project/mypackage.

A package manager exploiting this form or organizing elisp libraries
would be nice for some, I guess, but certainly it's a more niche
necessity.

Re: Unboxed package manager

[Philip Kaludercic]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Mon, 20 Mar 2023 at 15:25, Philip Kaludercic wrote:
>
>> I just skimmed through the initial message, and when reading this my
>> initial thought is does the the advantage justify effort/complexity,
>> considering that most users have an order of magnitude fewer packages
>> installed (of which a few are installed using package-vc which IIUC
>> would explicitly not want to make use of this feature).
>
> I do think there's at least one aspect of this idea that many many
> people need, namely: a simple way to add a directory such as
> ~/.emacs.d/lisp to the load path, with automatic support for autoloads,
> byte-compilation and what not.  Such a thing is also useful if you are
> developing some package, say at ~/project/mypackage.
>
> A package manager exploiting this form or organizing elisp libraries
> would be nice for some, I guess, but certainly it's a more niche
> necessity.

There is no strict need for this to be a package manager.  I propose
something like that a year ago:

  https://wwwcip.cs.fau.de/~oj14ozun/src+etc/site-lisp.el

In principle I could bundle this into a package and add it to ELPA.

-- 
Philip Kaludercic

Re: Unboxed package manager

[Augusto Stoffel]

On Tue, 21 Mar 2023 at 19:10, Philip Kaludercic wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>> On Mon, 20 Mar 2023 at 15:25, Philip Kaludercic wrote:
>
> There is no strict need for this to be a package manager.  I propose
> something like that a year ago:
>
>   https://wwwcip.cs.fau.de/~oj14ozun/src+etc/site-lisp.el
>
> In principle I could bundle this into a package and add it to ELPA.

Yes, I remember that.  And if memory serves someone argued that this
could be greatly generalized and etc., which is probably true.

Anyway, I think it makes sense for basic things one needs to write a
sensible configuration to be built in, but if there's no consensus as to
how the ideal "site-lisp.el" should look like, then a package is better
than nothing.

Re: Unboxed package manager

[Philip Kaludercic]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Tue, 21 Mar 2023 at 19:10, Philip Kaludercic wrote:
>
>> Augusto Stoffel <arstoffel@gmail.com> writes:
>>
>>> On Mon, 20 Mar 2023 at 15:25, Philip Kaludercic wrote:
>>
>> There is no strict need for this to be a package manager.  I propose
>> something like that a year ago:
>>
>>   https://wwwcip.cs.fau.de/~oj14ozun/src+etc/site-lisp.el
>>
>> In principle I could bundle this into a package and add it to ELPA.
>
> Yes, I remember that.  And if memory serves someone argued that this
> could be greatly generalized and etc., which is probably true.

It was "generalised" into package-vc.

> Anyway, I think it makes sense for basic things one needs to write a
> sensible configuration to be built in, but if there's no consensus as to
> how the ideal "site-lisp.el" should look like, then a package is better
> than nothing.

OK, I'll try and prepare it then.

-- 
Philip Kaludercic

Re: Unboxed package manager

[Gregory Heytings]

> 
> my recollection is that installing ~1200 packages on those systems and 
> "loading the world" took something like 5 minutes
>

I'm not sure I understand what you mean here.  Do you mean that 
byte-compiling and loading ~1200 packages takes about 5 minutes (which 
would not be a problem per se AFAIU)?  Or that loading ~1200 already 
byte-compiled packages takes about 5 minutes?  I took the file you posted 
in bug#61004, modified it a bit to make it work with emacs -Q (without 
external packages), and on my computer emacs -Q -l loadall.el takes ~4.5 
seconds.

Re: Unboxed package manager

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]


>> my recollection is that installing ~1200 packages on those systems and 
>> "loading the world" took something like 5 minutes
>
> I'm not sure I understand what you mean here.  Do you mean that 
> byte-compiling and loading ~1200 packages takes about 5 minutes (which 
> would not be a problem per se AFAIU)?  Or that loading ~1200 already 
> byte-compiled packages takes about 5 minutes?  I took the file you 
> posted in bug#61004, modified it a bit to make it work with emacs -Q 
> (without external packages), and on my computer emacs -Q -l loadall.el 
> takes ~4.5 seconds.
>

(Of course I forgot to attach the file.  Here it is.)

[-- Attachment #2: loadall.el --]
[-- Type: text/plain, Size: 26748 bytes --]

(require 'charprop)
(require 'url-cookie)
(require 'url)
(require 'url-gw)
(require 'url-queue)
(require 'url-misc)
(require 'url-ldap)
(require 'url-proxy)
(require 'url-nfs)
(require 'url-privacy)
(require 'url-cache)
(require 'url-http)
(require 'url-tramp)
(require 'url-parse)
(require 'url-history)
(require 'url-dav)
(require 'url-expand)
(require 'url-file)
(require 'url-util)
(require 'url-domsuf)
(require 'url-handlers)
(require 'url-methods)
(require 'url-future)
(require 'url-auth)
(require 'url-irc)
(require 'url-mailto)
(require 'url-ftp)
(require 'url-news)
(require 'url-vars)
(require 'type-break)
(require 'calculator)
(require 'fileloop)
(require 'chistory)
(require 'winner)
(require 'ehelp)
(require 'cmuscheme)
(require 'ibuf-macs)
(require 'finder)
(require 'windmove)
(require 'srecode)
(require 'pulse)
(require 'data-debug)
(require 'mode-local)
(require 'ede)
(require 'semantic)
(require 'cedet-global)
(require 'cedet-idutils)
(require 'cedet-cscope)
(require 'cedet)
(require 'cedet-files)
(require 'dirtrack)
(require 'composite)
(require 'rect)
(require 'jka-compr)
(require 'imenu)
(require 'view)
(require 'ezimage)
(require 'recentf)
(require 'font-lock)
(require 'esh-proc)
(require 'em-smart)
(require 'em-cmpl)
(require 'em-banner)
(require 'em-dirs)
(require 'em-script)
(require 'em-ls)
(require 'em-unix)
(require 'em-term)
(require 'em-alias)
(require 'esh-mode)
(require 'esh-var)
(require 'em-tramp)
(require 'esh-util)
(require 'em-xtra)
(require 'em-basic)
(require 'em-prompt)
(require 'esh-opt)
(require 'esh-io)
(require 'em-glob)
(require 'em-rebind)
(require 'esh-ext)
(require 'eshell)
(require 'esh-cmd)
(require 'esh-module)
(require 'esh-arg)
(require 'em-pred)
(require 'em-hist)
(require 'hl-line)
(require 'epa-file)
(require 'bookmark)
(require 'flow-ctrl)
(require 'speedbar)
(require 'rfn-eshadow)
(require 'fringe)
(require 'ibuffer)
(require 'erc-track)
(require 'erc-autoaway)
(require 'erc-pcomplete)
(require 'erc-stamp)
(require 'erc-button)
(require 'erc-log)
(require 'erc-goodies)
(require 'erc-identd)
(require 'erc-xdcc)
(require 'erc-notify)
(require 'erc-list)
(require 'erc)
(require 'erc-fill)
(require 'erc-backend)
(require 'erc-spelling)
(require 'erc-services)
(require 'erc-desktop-notifications)
(require 'erc-menu)
(require 'erc-lang)
(require 'erc-ibuffer)
(require 'erc-dcc)
(require 'erc-match)
(require 'erc-truncate)
(require 'erc-ring)
(require 'erc-replace)
(require 'erc-networks)
(require 'erc-compat)
(require 'erc-imenu)
(require 'erc-sound)
(require 'erc-capab)
(require 'erc-page)
(require 'erc-ezbounce)
(require 'erc-speedbar)
(require 'erc-join)
(require 'erc-netsplit)
(require 'jka-cmpr-hook)
(require 'epa-mail)
(require 'disp-table)
(require 'kermit)
(require 'msb)
(require 'emacs-lock)
(require 'xml)
(require 'electric)
(require 'ruler-mode)
(require 'profiler)
(require 'filenotify)
(require 'image-file)
(require 'json)
(require 'man)
(require 'char-fold)
(require 'tabify)
(require 'thread)
(require 'array)
(require 'mouse-drag)
(require 'pcmpl-cvs)
(require 'cus-theme)
(require 'saveplace)
(require 'arc-mode)
(require 'xwidget)
(require 'cal-tex)
(require 'cal-mayan)
(require 'cal-french)
(require 'cal-islam)
(require 'cal-bahai)
(require 'solar)
(require 'cal-x)
(require 'cal-julian)
(require 'cal-coptic)
(require 'cal-dst)
(require 'timeclock)
(require 'cal-hebrew)
(require 'cal-iso)
(require 'cal-html)
(require 'icalendar)
(require 'cal-china)
(require 'todo-mode)
(require 'iso8601)
(require 'cal-move)
(require 'lunar)
(require 'holidays)
(require 'cal-menu)
(require 'parse-time)
(require 'diary-lib)
(require 'appt)
(require 'calendar)
(require 'cal-persia)
(require 'time-date)
(require 'double)
(require 'locate)
(require 'help-at-pt)
(require 'hex-util)
(require 'foldout)
(require 'doctor)
(require 'life)
(require 'spook)
(require 'tetris)
(require 'gomoku)
(require 'mpuz)
(require 'fortune)
(require 'animate)
(require 'studly)
(require 'dunnet)
(require 'cookie1)
(require 'snake)
(require 'dissociate)
(require 'pong)
(require 'hanoi)
(require 'decipher)
(require 'gametree)
(require 'gamegrid)
(require '5x5)
(require 'blackbox)
(require 'morse)
(require 'bubbles)
(require 'solitaire)
(require 'zone)
(require 'handwrite)
(require 'midnight)
(require 'hippie-exp)
(require 'reveal)
(require 'simple)
(require 'ps-samp)
(require 'x-dnd)
(require 'color)
(require 'tar-mode)
(require 'dos-w32)
(require 'tooltip)
(require 'xdg)
(require 'env)
(require 'epg-config)
(require 'tool-bar)
(require 'mailcap)
(require 'telnet)
(require 'goto-addr)
(require 'dbus)
(require 'newst-ticker)
(require 'trampver)
(require 'sasl-digest)
(require 'dns)
(require 'tramp-integration)
(require 'eudcb-mab)
(require 'webjump)
(require 'hmac-md5)
(require 'sasl-scram-rfc)
(require 'tramp-gvfs)
(require 'network-stream)
(require 'tramp-cmds)
(require 'tramp-rclone)
(require 'shr-color)
(require 'newst-plainview)
(require 'ange-ftp)
(require 'snmp-mode)
(require 'tramp-uu)
(require 'socks)
(require 'tramp-ftp)
(require 'sasl)
(require 'tramp-adb)
(require 'sieve)
(require 'newsticker)
(require 'gnutls)
(require 'tramp-smb)
(require 'soap-inspect)
(require 'net-utils)
(require 'newst-reader)
(require 'secrets)
(require 'sieve-mode)
(require 'ldap)
(require 'soap-client)
(require 'tramp-compat)
(require 'tramp-sudoedit)
(require 'tramp)
(require 'tramp-cache)
(require 'sieve-manage)
(require 'pop3)
(require 'imap)
(require 'sasl-ntlm)
(require 'eudcb-bbdb)
(require 'nsm)
(require 'shr)
(require 'ntlm)
(require 'newst-treeview)
(require 'dig)
(require 'eudc)
(require 'eudcb-ldap)
(require 'rfc2104)
(require 'puny)
(require 'newst-backend)
(require 'browse-url)
(require 'sasl-cram)
(require 'tramp-sh)
(require 'rcirc)
(require 'tramp-archive)
(require 'mairix)
(require 'hmac-def)
(require 'zeroconf)
(require 'eudc-vars)
(require 'eww)
(require 'nxml-parse)
(require 'rng-valid)
(require 'nxml-rap)
(require 'xmltok)
(require 'nxml-outln)
(require 'rng-nxml)
(require 'rng-xsd)
(require 'rng-pttrn)
(require 'xsd-regexp)
(require 'nxml-util)
(require 'rng-dt)
(require 'rng-uri)
(require 'nxml-maint)
(require 'nxml-ns)
(require 'rng-parse)
(require 'rng-util)
(require 'nxml-enc)
(require 'rng-cmpct)
(require 'rng-match)
(require 'rng-loc)
(require 'nxml-mode)
(require 'dabbrev)
(require 'frameset)
(require 'mb-depth)
(require 'forms)
(require 'lpr)
(require 'rot13)
(require 'notifications)
(require 'ls-lisp)
(require 'ps-print)
(require 'display-fill-column-indicator)
(require 'pcmpl-unix)
(require 'tab-bar)
(require 'image)
(require 'kmacro)
(require 'epa-hook)
(require 'strokes)
(require 'image-dired)
(require 'ielm)
(require 'elec-pair)
(require 'epa-dired)
(require 'image-converter)
(require 'compface)
(require 'gravatar)
(require 'exif)
(require 'two-column)
(require 'reftex-ref)
(require 'tildify)
(require 'nroff-mode)
(require 'po)
(require 'enriched)
(require 'picture)
(require 'reftex-parse)
(require 'less-css-mode)
(require 'refer)
(require 'reftex-vars)
(require 'dns-mode)
(require 'artist)
(require 'refill)
(require 'bibtex)
(require 'makeinfo)
(require 'texinfo)
(require 'text-mode)
(require 'texnfo-upd)
(require 'ispell)
(require 'reftex)
(require 'bib-mode)
(require 'reftex-cite)
(require 'table)
(require 'refbib)
(require 'rst)
(require 'texinfmt)
(require 'sgml-mode)
(require 'underline)
(require 'page)
(require 'remember)
(require 'css-mode)
(require 'reftex-sel)
(require 'reftex-dcr)
(require 'tex-mode)
(require 'reftex-toc)
(require 'bibtex-style)
(require 'reftex-index)
(require 'mhtml-mode)
(require 'reftex-global)
(require 'flyspell)
(require 'reftex-auc)
(require 'page-ext)
(require 'conf-mode)
(require 'tmm)
(require 'sendmail)
(require 'rfc822)
(require 'hashcash)
(require 'mail-hist)
(require 'rmailsort)
(require 'rmail)
(require 'mailabbrev)
(require 'rfc2231)
(require 'yenc)
(require 'mailclient)
(require 'flow-fill)
(require 'undigest)
(require 'unrmail)
(require 'rmail-spam-filter)
(require 'smtpmail)
(require 'mail-prsvr)
(require 'rfc2045)
(require 'binhex)
(require 'ietf-drums)
(require 'mail-parse)
(require 'rfc2047)
(require 'rmailkwd)
(require 'rmailedit)
(require 'rmailout)
(require 'rmailsum)
(require 'supercite)
(require 'mspools)
(require 'mail-extr)
(require 'mailalias)
(require 'footnote)
(require 'emacsbug)
(require 'feedmail)
(require 'qp)
(require 'mail-utils)
(require 'rmailmm)
(require 'uudecode)
(require 'mailheader)
(require 'reporter)
(require 'mh-buffers)
(require 'mh-alias)
(require 'mh-letter)
(require 'mh-folder)
(require 'mh-identity)
(require 'mh-scan)
(require 'mh-inc)
(require 'mh-thread)
(require 'mh-comp)
(require 'mh-mime)
(require 'mh-xface)
(require 'mh-tool-bar)
(require 'mh-junk)
(require 'mh-limit)
(require 'mh-funcs)
(require 'mh-e)
(require 'mh-utils)
(require 'mh-seq)
(require 'mh-show)
(require 'mh-gnus)
(require 'mh-speed)
(require 'mh-print)
(require 'mh-search)
(require 'menu-bar)
(require 'wid-browse)
(require 'htmlfontify)
(require 'allout)
(require 'follow)
(require 'cua-rect)
(require 'edt)
(require 'cua-gmrk)
(require 'keypad)
(require 'cua-base)
(require 'comint)
(require 'timezone)
(require 'savehist)
(require 'custom)
(require 'ecomplete)
(require 'scroll-all)
(require 'delim-col)
(require 'org-macs)
(require 'ob-css)
(require 'ox-org)
(require 'org-attach)
(require 'ob-awk)
(require 'ob-perl)
(require 'ob-lob)
(require 'org)
(require 'org-mouse)
(require 'ol)
(require 'ob-sass)
(require 'ox-md)
(require 'ox-latex)
(require 'org-clock)
(require 'org-ctags)
(require 'ob-ref)
(require 'org-entities)
(require 'ob-clojure)
(require 'ob)
(require 'org-refile)
(require 'org-agenda)
(require 'org-id)
(require 'ob-lisp)
(require 'ob-matlab)
(require 'ob-haskell)
(require 'org-colview)
(require 'ob-scheme)
(require 'ob-js)
(require 'ob-java)
(require 'ob-sqlite)
(require 'ol-bbdb)
(require 'ob-lilypond)
(require 'org-footnote)
(require 'ob-gnuplot)
(require 'ox-icalendar)
(require 'ox-odt)
(require 'ol-eshell)
(require 'ox)
(require 'ob-calc)
(require 'org-duration)
(require 'ox-html)
(require 'org-src)
(require 'ox-ascii)
(require 'ob-R)
(require 'org-protocol)
(require 'ox-texinfo)
(require 'org-list)
(require 'ol-irc)
(require 'ob-screen)
(require 'org-timer)
(require 'org-indent)
(require 'ol-rmail)
(require 'ox-publish)
(require 'org-mobile)
(require 'org-attach-git)
(require 'ob-dot)
(require 'ob-comint)
(require 'org-faces)
(require 'org-datetree)
(require 'ol-mhe)
(require 'ob-eval)
(require 'ob-fortran)
(require 'ol-w3m)
(require 'org-capture)
(require 'ob-org)
(require 'ol-info)
(require 'ob-shell)
(require 'ob-core)
(require 'ob-makefile)
(require 'ob-ruby)
(require 'ob-tangle)
(require 'ob-emacs-lisp)
(require 'org-archive)
(require 'ob-lua)
(require 'org-table)
(require 'ol-eww)
(require 'ob-ocaml)
(require 'ob-plantuml)
(require 'ob-eshell)
(require 'org-habit)
(require 'org-feed)
(require 'org-macro)
(require 'org-lint)
(require 'org-keys)
(require 'ol-bibtex)
(require 'org-compat)
(require 'ob-ditaa)
(require 'org-plot)
(require 'ob-exp)
(require 'ob-latex)
(require 'ob-sql)
(require 'org-goto)
(require 'ox-beamer)
(require 'ob-forth)
(require 'org-inlinetask)
(require 'org-tempo)
(require 'ob-C)
(require 'ob-python)
(require 'ob-sed)
(require 'org-num)
(require 'ob-maxima)
(require 'ob-processing)
(require 'ol-docview)
(require 'org-element)
(require 'ob-groovy)
(require 'ox-man)
(require 'ol-gnus)
(require 'ob-table)
(require 'org-crypt)
(require 'org-pcomplete)
(require 'ob-octave)
(require 'term)
(require 'uniquify)
(require 'minibuffer)
(require 'mouse-copy)
(require 'paren)
(require 'help-mode)
(require 'mouse)
(require 'server)
(require 'md4)
(require 'w32-vars)
(require 'filesets)
(require 'misc)
(require 'misearch)
(require 'battery)
(require 'pixel-scroll)
(require 'info-xref)
(require 'files-x)
(require 'ibuf-ext)
(require 'macros)
(require 'cus-face)
(require 'dframe)
(require 'help-fns)
(require 'pcomplete)
(require 'reposition)
(require 'password-cache)
(require 't-mouse)
(require 'find-dired)
(require 'tab-line)
(require 'find-lisp)
(require 'cus-edit)
(require 'epg)
(require 'case-table)
(require 'scroll-lock)
(require 'epa)
(require 'dnd)
(require 'format)
(require 'outline)
(require 'obarray)
(require 'descr-text)
(require 'calc-sel)
(require 'calc-fin)
(require 'calc-store)
(require 'calc-math)
(require 'calc-help)
(require 'calc)
(require 'calc-mode)
(require 'calc-incom)
(require 'calc-stat)
(require 'calc-ext)
(require 'calc-nlfit)
(require 'calc-forms)
(require 'calc-menu)
(require 'calc-graph)
(require 'calc-units)
(require 'calc-macs)
(require 'calcalg3)
(require 'calc-misc)
(require 'calc-trail)
(require 'calcsel2)
(require 'calc-mtx)
(require 'calc-poly)
(require 'calc-rewr)
(require 'calc-funcs)
(require 'calc-arith)
(require 'calc-embed)
(require 'calc-yank)
(require 'calc-map)
(require 'calc-frac)
(require 'calc-undo)
(require 'calc-comb)
(require 'calc-lang)
(require 'calc-prog)
(require 'calc-rules)
(require 'calc-aent)
(require 'calc-alg)
(require 'calc-cplx)
(require 'calc-bin)
(require 'calcalg2)
(require 'calccomp)
(require 'calc-stuff)
(require 'calc-keypd)
(require 'calc-vec)
(require 'soundex)
(require 'time)
(require 'gnus-uu)
(require 'nnmbox)
(require 'gnus-win)
(require 'mail-source)
(require 'legacy-gnus-agent)
(require 'nnoo)
(require 'gnus-util)
(require 'nnmaildir)
(require 'nnnil)
(require 'mm-bodies)
(require 'gnus-picon)
(require 'gnus-cache)
(require 'gnus-mh)
(require 'deuglify)
(require 'gnus-agent)
(require 'mml1991)
(require 'nnregistry)
(require 'gnus-score)
(require 'gnus-vm)
(require 'gnus-dired)
(require 'gnus-diary)
(require 'nnspool)
(require 'gnus-srvr)
(require 'gnus-bookmark)
(require 'nnbabyl)
(require 'gnus-art)
(require 'gnus-rfc1843)
(require 'gnus-html)
(require 'nnvirtual)
(require 'gnus-int)
(require 'gnus-start)
(require 'nnml)
(require 'mm-partial)
(require 'nndir)
(require 'gnus-sum)
(require 'mml-sec)
(require 'nnrss)
(require 'gnus-cloud)
(require 'gmm-utils)
(require 'gnus-demon)
(require 'mml)
(require 'gnus-salt)
(require 'gnus-gravatar)
(require 'gnus-draft)
(require 'mm-util)
(require 'nndiary)
(require 'mml-smime)
(require 'mml2015)
(require 'spam)
(require 'nneething)
(require 'mm-uu)
(require 'mm-url)
(require 'gssapi)
(require 'gnus-cus)
(require 'nntp)
(require 'nnmail)
(require 'gnus-spec)
(require 'gnus-range)
(require 'mm-archive)
(require 'nnweb)
(require 'nndraft)
(require 'nnmairix)
(require 'gnus-logic)
(require 'gnus-registry)
(require 'message)
(require 'gnus-undo)
(require 'gnus)
(require 'gnus-ml)
(require 'nngateway)
(require 'gnus-delay)
(require 'gnus-dup)
(require 'smiley)
(require 'mm-decode)
(require 'gnus-cite)
(require 'canlock)
(require 'mm-encode)
(require 'gnus-sieve)
(require 'nnfolder)
(require 'nndoc)
(require 'gnus-bcklg)
(require 'gnus-mlspl)
(require 'gnus-eform)
(require 'gnus-group)
(require 'spam-report)
(require 'gnus-msg)
(require 'nnheader)
(require 'gnus-kill)
(require 'mm-view)
(require 'gnus-notifications)
(require 'nnimap)
(require 'score-mode)
(require 'smime)
(require 'gnus-async)
(require 'nnmh)
(require 'spam-stat)
(require 'gnus-topic)
(require 'mm-extern)
(require 'nnagent)
(require 'spam-wash)
(require 'gnus-fun)
(require 'dired-aux)
(require 'scroll-bar)
(require 'pcmpl-rpm)
(require 'mpc)
(require 'talk)
(require 'vc-hg)
(require 'vc-dir)
(require 'ediff-vers)
(require 'vc-cvs)
(require 'vc-bzr)
(require 'diff-mode)
(require 'emerge)
(require 'vc-hooks)
(require 'vc-rcs)
(require 'pcvs-parse)
(require 'vc-sccs)
(require 'ediff-wind)
(require 'vc-filewise)
(require 'vc-git)
(require 'log-edit)
(require 'ediff-mult)
(require 'pcvs-util)
(require 'ediff-util)
(require 'add-log)
(require 'pcvs-info)
(require 'ediff-merg)
(require 'vc-svn)
(require 'vc-dispatcher)
(require 'vc-annotate)
(require 'ediff-help)
(require 'ediff)
(require 'ediff-init)
(require 'ediff-diff)
(require 'log-view)
(require 'vc-dav)
(require 'vc)
(require 'smerge-mode)
(require 'pcvs)
(require 'vc-src)
(require 'ediff-ptch)
(require 'ediff-hook)
(require 'pcvs-defs)
(require 'compare-w)
(require 'diff)
(require 'cvs-status)
(require 'shell)
(require 'bs)
(require 'loadhist)
(require 'minibuf-eldef)
(require 'informat)
(require 'xt-mouse)
(require 'sort)
(require 'printing)
(require 'allout-widgets)
(require 'ansi-color)
(require 'executable)
(require 'elisp-mode)
(require 'idlw-help)
(require 'flymake)
(require 'cc-guess)
(require 'idlw-shell)
(require 'cc-mode)
(require 'gud)
(require 'gdb-mi)
(require 'idlw-toolbar)
(require 'f90)
(require 'etags)
(require 'cc-engine)
(require 'opascal)
(require 'xscheme)
(require 'asm-mode)
(require 'idlwave)
(require 'cwarn)
(require 'pascal)
(require 'cfengine)
(require 'ebnf-yac)
(require 'ebnf-abn)
(require 'inf-lisp)
(require 'glasses)
(require 'ebnf-dtd)
(require 'make-mode)
(require 'ebnf-bnf)
(require 'autoconf)
(require 'octave)
(require 'xref)
(require 'prog-mode)
(require 'cc-defs)
(require 'cc-styles)
(require 'subword)
(require 'cc-align)
(require 'cc-fonts)
(require 'bug-reference)
(require 'cpp)
(require 'mixal-mode)
(require 'ebnf2ps)
(require 'prolog)
(require 'cc-langs)
(require 'cc-menus)
(require 'idlw-complete-structtag)
(require 'ebnf-otz)
(require 'ld-script)
(require 'cmacexp)
(require 'meta-mode)
(require 'cc-vars)
(require 'vera-mode)
(require 'bat-mode)
(require 'sh-script)
(require 'verilog-mode)
(require 'js)
(require 'simula)
(require 'ebnf-iso)
(require 'fortran)
(require 'dcl-mode)
(require 'cperl-mode)
(require 'project)
(require 'ruby-mode)
(require 'scheme)
(require 'flymake-proc)
(require 'modula2)
(require 'python)
(require 'grep)
(require 'vhdl-mode)
(require 'tcl)
(require 'ebnf-ebx)
(require 'm4-mode)
(require 'compile)
(require 'hideif)
(require 'cc-bytecomp)
(require 'which-func)
(require 'hideshow)
(require 'sql)
(require 'antlr-mode)
(require 'cc-awk)
(require 'perl-mode)
(require 'ebrowse)
(require 'flymake-cc)
(require 'cc-cmds)
(require 'icon)
(require 'ps-mode)
(require 'wdired)
(require 'isearch)
(require 'find-cmd)
(require 'whitespace)
(require 'doc-view)
(require 'facemenu)
(require 'shadowfile)
(require 'plstore)
(require 'ps-mule)
(require 'face-remap)
(require 'frame)
(require 'help-macro)
(require 'info-look)
(require 'files)
(require 'help)
(require 'dired)
(require 'pcmpl-linux)
(require 'register)
(require 'vcursor)
(require 'icomplete)
(require 'rtree)
(require 'time-stamp)
(require 'replace)
(require 'format-spec)
(require 'pcmpl-x)
(require 'ebuff-menu)
(require 'auth-source)
(require 'so-long)
(require 'proced)
(require 'faces)
(require 'apropos)
(require 'repeat)
(require 'svg)
(require 'display-line-numbers)
(require 'jit-lock)
(require 'button)
(require 'abbrev)
(require 'newcomment)
(require 'image-mode)
(require 'widget)
(require 'woman)
(require 'font-core)
(require 'filecache)
(require 'info)
(require 'cus-dep)
(require 'cdl)
(require 'novice)
(require 'ido)
(require 'hexl)
(require 'dos-vars)
(require 'master)
(require 'hfy-cmap)
(require 'dynamic-setting)
(require 'auth-source-pass)
(require 'delsel)
(require 'tree-widget)
(require 'find-file)
(require 'wid-edit)
(require 'thingatpt)
(require 'align)
(require 'hilit-chg)
(require 'edmacro)
(require 'autoinsert)
(require 'ses)
(require 'dos-fns)
(require 'ffap)
(require 'registry)
(require 'iimage)
(require 'tutorial)
(require 'hi-lock)
(require 'mwheel)
(require 'expand)
(require 'pcmpl-gnu)
(require 'jsonrpc)
(require 'echistory)
(require 'autorevert)
(require 'dired-x)
(require 'completion)
(require 'desktop)
(require 'mule-diag)
(require 'latexenc)
(require 'cp51932)
(require 'fontset)
(require 'ja-dic-utl)
(require 'ccl)
(require 'utf7)
(require 'utf-7)
(require 'ucs-normalize)
(require 'mule-util)
(require 'robin)
(require 'quail)
(require 'iso-transl)
(require 'mule)
(require 'kkc)
(require 'ogonek)
(require 'eucjp-ms)
(require 'rfc1843)
(require 'latin1-disp)
(require 'iso-ascii)
(require 'iso-cvt)
(require 'charscript)
(require 'elide-head)
(require 'dom)
(require 'ert)
(require 'checkdoc)
(require 'byte-opt)
(require 'eieio-datadebug)
(require 'unsafep)
(require 'eldoc)
(require 'lisp-mode)
(require 'ert-x)
(require 'warnings)
(require 'eieio-speedbar)
(require 'rx)
(require 'cl-extra)
(require 'eieio)
(require 'eieio-core)
(require 'easymenu)
(require 'bytecomp)
(require 'inline)
(require 'shadow)
(require 'generic)
(require 'rmc)
(require 'advice)
(require 'backtrace)
(require 'pcase)
(require 'cl-generic)
(require 'check-declare)
(require 'timer-list)
(require 're-builder)
(require 'package)
(require 'package-x)
(require 'backquote)
(require 'disass)
(require 'eieio-base)
(require 'elint)
(require 'edebug)
(require 'benchmark)
(require 'radix-tree)
(require 'lisp-mnt)
(require 'chart)
(require 'ring)
(require 'crm)
(require 'eieio-custom)
(require 'tq)
(require 'tabulated-list)
(require 'debug)
(require 'helper)
(require 'cl-seq)
(require 'nadvice)
(require 'elp)
(require 'text-property-search)
(require 'generator)
(require 'cl-indent)
(require 'faceup)
(require 'smie)
(require 'bindat)
(require 'cl-macs)
(require 'copyright)
(require 'let-alist)
(require 'testcover)
(require 'seq)
(require 'regexp-opt)
(require 'subr-x)
(require 'trace)
(require 'thunk)
(require 'cl-preloaded)
(require 'regi)
(require 'map)
(require 'syntax)
(require 'pp)
(require 'timer)
(require 'derived)
(require 'cl-lib)
(require 'find-func)
(require 'gv)
(require 'ewoc)
(require 'eieio-opt)
(require 'cconv)
(require 'cl-print)
(require 'macroexp)
(require 'easy-mmode)
(require 'cursor-sensor)
(require 'avl-tree)
(require 'ps-bdf)
(require 'avoid)
(require 'tempo)
(require 'select)
(require 'srecode/find)
(require 'srecode/fields)
(require 'srecode/args)
(require 'srecode/getset)
(require 'srecode/el)
(require 'srecode/texi)
(require 'srecode/mode)
(require 'srecode/ctxt)
(require 'srecode/java)
(require 'srecode/cpp)
(require 'srecode/expandproto)
(require 'srecode/table)
(require 'srecode/srt)
(require 'srecode/document)
(require 'srecode/insert)
(require 'srecode/dictionary)
(require 'srecode/semantic)
(require 'srecode/extract)
(require 'srecode/map)
(require 'srecode/template)
(require 'srecode/compile)
(require 'srecode/srt-mode)
(require 'srecode/srt-wy)
(require 'srecode/filters)
(require 'ede/linux)
(require 'ede/pmake)
(require 'ede/emacs)
(require 'ede/project-am)
(require 'ede/srecode)
(require 'ede/speedbar)
(require 'ede/locate)
(require 'ede/generic)
(require 'ede/proj-archive)
(require 'ede/simple)
(require 'ede/proj-aux)
(require 'ede/make)
(require 'ede/config)
(require 'ede/proj-misc)
(require 'ede/pconf)
(require 'ede/proj-scheme)
(require 'ede/proj-elisp)
(require 'ede/cpp-root)
(require 'ede/custom)
(require 'ede/system)
(require 'ede/base)
(require 'ede/proj-obj)
(require 'ede/shell)
(require 'ede/proj)
(require 'ede/makefile-edit)
(require 'ede/proj-comp)
(require 'ede/files)
(require 'ede/dired)
(require 'ede/source)
(require 'ede/detect)
(require 'ede/auto)
(require 'ede/autoconf-edit)
(require 'ede/proj-prog)
(require 'ede/util)
(require 'ede/proj-info)
(require 'ede/proj-shared)
(require 'semantic/find)
(require 'semantic/imenu)
(require 'semantic/complete)
(require 'semantic/db-ref)
(require 'semantic/db-javascript)
(require 'semantic/scope)
(require 'semantic/decorate/include)
(require 'semantic/decorate/mode)
(require 'semantic/decorate)
(require 'semantic/ia)
(require 'semantic/db-ebrowse)
(require 'semantic/db-find)
(require 'semantic/db-el)
(require 'semantic/html)
(require 'semantic/texi)
(require 'semantic/ede-grammar)
(require 'semantic/ctxt)
(require 'semantic/lex)
(require 'semantic/symref/global)
(require 'semantic/symref/filter)
(require 'semantic/symref/cscope)
(require 'semantic/symref/list)
(require 'semantic/symref/grep)
(require 'semantic/symref/idutils)
(require 'semantic/java)
(require 'semantic/ia-sb)
(require 'semantic/tag-ls)
(require 'semantic/lex-spp)
(require 'semantic/bovine)
(require 'semantic/util-modes)
(require 'semantic/grammar-wy)
(require 'semantic/chart)
(require 'semantic/dep)
(require 'semantic/debug)
(require 'semantic/wisent/js-wy)
(require 'semantic/wisent/python-wy)
(require 'semantic/wisent/wisent)
(require 'semantic/wisent/javat-wy)
(require 'semantic/wisent/python)
(require 'semantic/wisent/java-tags)
(require 'semantic/wisent/grammar)
(require 'semantic/wisent/comp)
(require 'semantic/doc)
(require 'semantic/tag-write)
(require 'semantic/wisent)
(require 'semantic/format)
(require 'semantic/fw)
(require 'semantic/db-file)
(require 'semantic/bovine/el)
(require 'semantic/bovine/c)
(require 'semantic/bovine/gcc)
(require 'semantic/bovine/scm-by)
(require 'semantic/bovine/make)
(require 'semantic/bovine/debug)
(require 'semantic/bovine/scm)
(require 'semantic/bovine/c-by)
(require 'semantic/bovine/grammar)
(require 'semantic/bovine/make-by)
(require 'semantic/sb)
(require 'semantic/tag-file)
(require 'semantic/sort)
(require 'semantic/db-typecache)
(require 'semantic/idle)
(require 'semantic/db-debug)
(require 'semantic/senator)
(require 'semantic/db)
(require 'semantic/edit)
(require 'semantic/db-global)
(require 'semantic/symref)
(require 'semantic/analyze/fcn)
(require 'semantic/analyze/complete)
(require 'semantic/analyze/debug)
(require 'semantic/analyze/refs)
(require 'semantic/grammar)
(require 'semantic/mru-bookmark)
(require 'semantic/db-mode)
(require 'semantic/util)
(require 'semantic/tag)
(require 'semantic/analyze)
(require 'transient)
(require 'external-completion)
(require 'eglot)
(require 'sha1)

Re: Unboxed package manager

[Lynn Winebarger]

On Mon, Mar 20, 2023 at 8:25 PM Gregory Heytings <gregory@heytings.org> wrote:
> >> my recollection is that installing ~1200 packages on those systems and
> >> "loading the world" took something like 5 minutes
> >
> > I'm not sure I understand what you mean here.  Do you mean that
> > byte-compiling and loading ~1200 packages takes about 5 minutes (which
> > would not be a problem per se AFAIU)?  Or that loading ~1200 already
> > byte-compiled packages takes about 5 minutes?  I took the file you
> > posted in bug#61004, modified it a bit to make it work with emacs -Q
> > (without external packages), and on my computer emacs -Q -l loadall.el
> > takes ~4.5 seconds.

You're right, my wording is very confusing.  The installation phase
took much, much longer than 5 minutes.  Loading the world, after
everything was byte-compiled, took about 5 minutes.  The 1200 packages
is just to give the number of directories prepended to the standard
load-path (it was more work to get those packages on those sandboxed
systems, hence "only" 1200, not the 2400+ I currently have on my
personal machine).  The number of libraries being loaded was closer to
4000, if I am recalling correctly (big if - this is ~9 months ago).
That was on fairly nice server hardware with SSDs, lots of RAM, and 24
cores.

I'm pretty sure the profiling report I filed for #61004 was generated
on a 2017-vintage laptop with a physically spinning disk for storage.
I don't think it would run emacs -Q -l loadall.el in 4.5 seconds on
that laptop, but with "-Q" you're taking out the main drag on the
startup time - having to search 1000-2000+ directories before getting
to the system directories where all the libraries in loadall.el will
actually be found.

Lynn

Re: Unboxed package manager

[Lynn Winebarger]

On Mon, Mar 20, 2023 at 9:55 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> On Mon, Mar 20, 2023 at 8:25 PM Gregory Heytings <gregory@heytings.org> wrote:
> > >> my recollection is that installing ~1200 packages on those systems and
> > >> "loading the world" took something like 5 minutes
> > >
> > > I'm not sure I understand what you mean here.  Do you mean that
> > > byte-compiling and loading ~1200 packages takes about 5 minutes (which
> > > would not be a problem per se AFAIU)?  Or that loading ~1200 already
> > > byte-compiled packages takes about 5 minutes?  I took the file you
> > > posted in bug#61004, modified it a bit to make it work with emacs -Q
> > > (without external packages), and on my computer emacs -Q -l loadall.el
> > > takes ~4.5 seconds.
>
> You're right, my wording is very confusing.  The installation phase
> took much, much longer than 5 minutes.  Loading the world, after
> everything was byte-compiled, took about 5 minutes.  The 1200 packages
> is just to give the number of directories prepended to the standard
> load-path (it was more work to get those packages on those sandboxed
> systems, hence "only" 1200, not the 2400+ I currently have on my
> personal machine).  The number of libraries being loaded was closer to
> 4000, if I am recalling correctly (big if - this is ~9 months ago).
> That was on fairly nice server hardware with SSDs, lots of RAM, and 24
> cores.
>
> I'm pretty sure the profiling report I filed for #61004 was generated
> on a 2017-vintage laptop with a physically spinning disk for storage.
> I don't think it would run emacs -Q -l loadall.el in 4.5 seconds on
> that laptop, but with "-Q" you're taking out the main drag on the
> startup time - having to search 1000-2000+ directories before getting
> to the system directories where all the libraries in loadall.el will
> actually be found.

Thanks for checking this out, BTW.

You could construct a test without external packages by just
generating N directories, prepending them to the load path, then doing
the same loadall to see if the degradation is purely due to the size
of the load path.

Lynn

Re: Unboxed package manager

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]


>
> You could construct a test without external packages by just generating 
> N directories, prepending them to the load path, then doing the same 
> loadall to see if the degradation is purely due to the size of the load 
> path.
>

Indeed, with the attached file, emacs -Q -l loadall.el takes 120 seconds. 
Hmmm...

[-- Attachment #2: loadall.el --]
[-- Type: text/plain, Size: 26883 bytes --]

(let ((i 0))
  (while (< i 2000)
    (setq load-path (append (list (make-temp-file "foobar-" t)) load-path))
    (setq i (1+ i))))
(require 'charprop)
(require 'url-cookie)
(require 'url)
(require 'url-gw)
(require 'url-queue)
(require 'url-misc)
(require 'url-ldap)
(require 'url-proxy)
(require 'url-nfs)
(require 'url-privacy)
(require 'url-cache)
(require 'url-http)
(require 'url-tramp)
(require 'url-parse)
(require 'url-history)
(require 'url-dav)
(require 'url-expand)
(require 'url-file)
(require 'url-util)
(require 'url-domsuf)
(require 'url-handlers)
(require 'url-methods)
(require 'url-future)
(require 'url-auth)
(require 'url-irc)
(require 'url-mailto)
(require 'url-ftp)
(require 'url-news)
(require 'url-vars)
(require 'type-break)
(require 'calculator)
(require 'fileloop)
(require 'chistory)
(require 'winner)
(require 'ehelp)
(require 'cmuscheme)
(require 'ibuf-macs)
(require 'finder)
(require 'windmove)
(require 'srecode)
(require 'pulse)
(require 'data-debug)
(require 'mode-local)
(require 'ede)
(require 'semantic)
(require 'cedet-global)
(require 'cedet-idutils)
(require 'cedet-cscope)
(require 'cedet)
(require 'cedet-files)
(require 'dirtrack)
(require 'composite)
(require 'rect)
(require 'jka-compr)
(require 'imenu)
(require 'view)
(require 'ezimage)
(require 'recentf)
(require 'font-lock)
(require 'esh-proc)
(require 'em-smart)
(require 'em-cmpl)
(require 'em-banner)
(require 'em-dirs)
(require 'em-script)
(require 'em-ls)
(require 'em-unix)
(require 'em-term)
(require 'em-alias)
(require 'esh-mode)
(require 'esh-var)
(require 'em-tramp)
(require 'esh-util)
(require 'em-xtra)
(require 'em-basic)
(require 'em-prompt)
(require 'esh-opt)
(require 'esh-io)
(require 'em-glob)
(require 'em-rebind)
(require 'esh-ext)
(require 'eshell)
(require 'esh-cmd)
(require 'esh-module)
(require 'esh-arg)
(require 'em-pred)
(require 'em-hist)
(require 'hl-line)
(require 'epa-file)
(require 'bookmark)
(require 'flow-ctrl)
(require 'speedbar)
(require 'rfn-eshadow)
(require 'fringe)
(require 'ibuffer)
(require 'erc-track)
(require 'erc-autoaway)
(require 'erc-pcomplete)
(require 'erc-stamp)
(require 'erc-button)
(require 'erc-log)
(require 'erc-goodies)
(require 'erc-identd)
(require 'erc-xdcc)
(require 'erc-notify)
(require 'erc-list)
(require 'erc)
(require 'erc-fill)
(require 'erc-backend)
(require 'erc-spelling)
(require 'erc-services)
(require 'erc-desktop-notifications)
(require 'erc-menu)
(require 'erc-lang)
(require 'erc-ibuffer)
(require 'erc-dcc)
(require 'erc-match)
(require 'erc-truncate)
(require 'erc-ring)
(require 'erc-replace)
(require 'erc-networks)
(require 'erc-compat)
(require 'erc-imenu)
(require 'erc-sound)
(require 'erc-capab)
(require 'erc-page)
(require 'erc-ezbounce)
(require 'erc-speedbar)
(require 'erc-join)
(require 'erc-netsplit)
(require 'jka-cmpr-hook)
(require 'epa-mail)
(require 'disp-table)
(require 'kermit)
(require 'msb)
(require 'emacs-lock)
(require 'xml)
(require 'electric)
(require 'ruler-mode)
(require 'profiler)
(require 'filenotify)
(require 'image-file)
(require 'json)
(require 'man)
(require 'char-fold)
(require 'tabify)
(require 'thread)
(require 'array)
(require 'mouse-drag)
(require 'pcmpl-cvs)
(require 'cus-theme)
(require 'saveplace)
(require 'arc-mode)
(require 'xwidget)
(require 'cal-tex)
(require 'cal-mayan)
(require 'cal-french)
(require 'cal-islam)
(require 'cal-bahai)
(require 'solar)
(require 'cal-x)
(require 'cal-julian)
(require 'cal-coptic)
(require 'cal-dst)
(require 'timeclock)
(require 'cal-hebrew)
(require 'cal-iso)
(require 'cal-html)
(require 'icalendar)
(require 'cal-china)
(require 'todo-mode)
(require 'iso8601)
(require 'cal-move)
(require 'lunar)
(require 'holidays)
(require 'cal-menu)
(require 'parse-time)
(require 'diary-lib)
(require 'appt)
(require 'calendar)
(require 'cal-persia)
(require 'time-date)
(require 'double)
(require 'locate)
(require 'help-at-pt)
(require 'hex-util)
(require 'foldout)
(require 'doctor)
(require 'life)
(require 'spook)
(require 'tetris)
(require 'gomoku)
(require 'mpuz)
(require 'fortune)
(require 'animate)
(require 'studly)
(require 'dunnet)
(require 'cookie1)
(require 'snake)
(require 'dissociate)
(require 'pong)
(require 'hanoi)
(require 'decipher)
(require 'gametree)
(require 'gamegrid)
(require '5x5)
(require 'blackbox)
(require 'morse)
(require 'bubbles)
(require 'solitaire)
(require 'zone)
(require 'handwrite)
(require 'midnight)
(require 'hippie-exp)
(require 'reveal)
(require 'simple)
(require 'ps-samp)
(require 'x-dnd)
(require 'color)
(require 'tar-mode)
(require 'dos-w32)
(require 'tooltip)
(require 'xdg)
(require 'env)
(require 'epg-config)
(require 'tool-bar)
(require 'mailcap)
(require 'telnet)
(require 'goto-addr)
(require 'dbus)
(require 'newst-ticker)
(require 'trampver)
(require 'sasl-digest)
(require 'dns)
(require 'tramp-integration)
(require 'eudcb-mab)
(require 'webjump)
(require 'hmac-md5)
(require 'sasl-scram-rfc)
(require 'tramp-gvfs)
(require 'network-stream)
(require 'tramp-cmds)
(require 'tramp-rclone)
(require 'shr-color)
(require 'newst-plainview)
(require 'ange-ftp)
(require 'snmp-mode)
(require 'tramp-uu)
(require 'socks)
(require 'tramp-ftp)
(require 'sasl)
(require 'tramp-adb)
(require 'sieve)
(require 'newsticker)
(require 'gnutls)
(require 'tramp-smb)
(require 'soap-inspect)
(require 'net-utils)
(require 'newst-reader)
(require 'secrets)
(require 'sieve-mode)
(require 'ldap)
(require 'soap-client)
(require 'tramp-compat)
(require 'tramp-sudoedit)
(require 'tramp)
(require 'tramp-cache)
(require 'sieve-manage)
(require 'pop3)
(require 'imap)
(require 'sasl-ntlm)
(require 'eudcb-bbdb)
(require 'nsm)
(require 'shr)
(require 'ntlm)
(require 'newst-treeview)
(require 'dig)
(require 'eudc)
(require 'eudcb-ldap)
(require 'rfc2104)
(require 'puny)
(require 'newst-backend)
(require 'browse-url)
(require 'sasl-cram)
(require 'tramp-sh)
(require 'rcirc)
(require 'tramp-archive)
(require 'mairix)
(require 'hmac-def)
(require 'zeroconf)
(require 'eudc-vars)
(require 'eww)
(require 'nxml-parse)
(require 'rng-valid)
(require 'nxml-rap)
(require 'xmltok)
(require 'nxml-outln)
(require 'rng-nxml)
(require 'rng-xsd)
(require 'rng-pttrn)
(require 'xsd-regexp)
(require 'nxml-util)
(require 'rng-dt)
(require 'rng-uri)
(require 'nxml-maint)
(require 'nxml-ns)
(require 'rng-parse)
(require 'rng-util)
(require 'nxml-enc)
(require 'rng-cmpct)
(require 'rng-match)
(require 'rng-loc)
(require 'nxml-mode)
(require 'dabbrev)
(require 'frameset)
(require 'mb-depth)
(require 'forms)
(require 'lpr)
(require 'rot13)
(require 'notifications)
(require 'ls-lisp)
(require 'ps-print)
(require 'display-fill-column-indicator)
(require 'pcmpl-unix)
(require 'tab-bar)
(require 'image)
(require 'kmacro)
(require 'epa-hook)
(require 'strokes)
(require 'image-dired)
(require 'ielm)
(require 'elec-pair)
(require 'epa-dired)
(require 'image-converter)
(require 'compface)
(require 'gravatar)
(require 'exif)
(require 'two-column)
(require 'reftex-ref)
(require 'tildify)
(require 'nroff-mode)
(require 'po)
(require 'enriched)
(require 'picture)
(require 'reftex-parse)
(require 'less-css-mode)
(require 'refer)
(require 'reftex-vars)
(require 'dns-mode)
(require 'artist)
(require 'refill)
(require 'bibtex)
(require 'makeinfo)
(require 'texinfo)
(require 'text-mode)
(require 'texnfo-upd)
(require 'ispell)
(require 'reftex)
(require 'bib-mode)
(require 'reftex-cite)
(require 'table)
(require 'refbib)
(require 'rst)
(require 'texinfmt)
(require 'sgml-mode)
(require 'underline)
(require 'page)
(require 'remember)
(require 'css-mode)
(require 'reftex-sel)
(require 'reftex-dcr)
(require 'tex-mode)
(require 'reftex-toc)
(require 'bibtex-style)
(require 'reftex-index)
(require 'mhtml-mode)
(require 'reftex-global)
(require 'flyspell)
(require 'reftex-auc)
(require 'page-ext)
(require 'conf-mode)
(require 'tmm)
(require 'sendmail)
(require 'rfc822)
(require 'hashcash)
(require 'mail-hist)
(require 'rmailsort)
(require 'rmail)
(require 'mailabbrev)
(require 'rfc2231)
(require 'yenc)
(require 'mailclient)
(require 'flow-fill)
(require 'undigest)
(require 'unrmail)
(require 'rmail-spam-filter)
(require 'smtpmail)
(require 'mail-prsvr)
(require 'rfc2045)
(require 'binhex)
(require 'ietf-drums)
(require 'mail-parse)
(require 'rfc2047)
(require 'rmailkwd)
(require 'rmailedit)
(require 'rmailout)
(require 'rmailsum)
(require 'supercite)
(require 'mspools)
(require 'mail-extr)
(require 'mailalias)
(require 'footnote)
(require 'emacsbug)
(require 'feedmail)
(require 'qp)
(require 'mail-utils)
(require 'rmailmm)
(require 'uudecode)
(require 'mailheader)
(require 'reporter)
(require 'mh-buffers)
(require 'mh-alias)
(require 'mh-letter)
(require 'mh-folder)
(require 'mh-identity)
(require 'mh-scan)
(require 'mh-inc)
(require 'mh-thread)
(require 'mh-comp)
(require 'mh-mime)
(require 'mh-xface)
(require 'mh-tool-bar)
(require 'mh-junk)
(require 'mh-limit)
(require 'mh-funcs)
(require 'mh-e)
(require 'mh-utils)
(require 'mh-seq)
(require 'mh-show)
(require 'mh-gnus)
(require 'mh-speed)
(require 'mh-print)
(require 'mh-search)
(require 'menu-bar)
(require 'wid-browse)
(require 'htmlfontify)
(require 'allout)
(require 'follow)
(require 'cua-rect)
(require 'edt)
(require 'cua-gmrk)
(require 'keypad)
(require 'cua-base)
(require 'comint)
(require 'timezone)
(require 'savehist)
(require 'custom)
(require 'ecomplete)
(require 'scroll-all)
(require 'delim-col)
(require 'org-macs)
(require 'ob-css)
(require 'ox-org)
(require 'org-attach)
(require 'ob-awk)
(require 'ob-perl)
(require 'ob-lob)
(require 'org)
(require 'org-mouse)
(require 'ol)
(require 'ob-sass)
(require 'ox-md)
(require 'ox-latex)
(require 'org-clock)
(require 'org-ctags)
(require 'ob-ref)
(require 'org-entities)
(require 'ob-clojure)
(require 'ob)
(require 'org-refile)
(require 'org-agenda)
(require 'org-id)
(require 'ob-lisp)
(require 'ob-matlab)
(require 'ob-haskell)
(require 'org-colview)
(require 'ob-scheme)
(require 'ob-js)
(require 'ob-java)
(require 'ob-sqlite)
(require 'ol-bbdb)
(require 'ob-lilypond)
(require 'org-footnote)
(require 'ob-gnuplot)
(require 'ox-icalendar)
(require 'ox-odt)
(require 'ol-eshell)
(require 'ox)
(require 'ob-calc)
(require 'org-duration)
(require 'ox-html)
(require 'org-src)
(require 'ox-ascii)
(require 'ob-R)
(require 'org-protocol)
(require 'ox-texinfo)
(require 'org-list)
(require 'ol-irc)
(require 'ob-screen)
(require 'org-timer)
(require 'org-indent)
(require 'ol-rmail)
(require 'ox-publish)
(require 'org-mobile)
(require 'org-attach-git)
(require 'ob-dot)
(require 'ob-comint)
(require 'org-faces)
(require 'org-datetree)
(require 'ol-mhe)
(require 'ob-eval)
(require 'ob-fortran)
(require 'ol-w3m)
(require 'org-capture)
(require 'ob-org)
(require 'ol-info)
(require 'ob-shell)
(require 'ob-core)
(require 'ob-makefile)
(require 'ob-ruby)
(require 'ob-tangle)
(require 'ob-emacs-lisp)
(require 'org-archive)
(require 'ob-lua)
(require 'org-table)
(require 'ol-eww)
(require 'ob-ocaml)
(require 'ob-plantuml)
(require 'ob-eshell)
(require 'org-habit)
(require 'org-feed)
(require 'org-macro)
(require 'org-lint)
(require 'org-keys)
(require 'ol-bibtex)
(require 'org-compat)
(require 'ob-ditaa)
(require 'org-plot)
(require 'ob-exp)
(require 'ob-latex)
(require 'ob-sql)
(require 'org-goto)
(require 'ox-beamer)
(require 'ob-forth)
(require 'org-inlinetask)
(require 'org-tempo)
(require 'ob-C)
(require 'ob-python)
(require 'ob-sed)
(require 'org-num)
(require 'ob-maxima)
(require 'ob-processing)
(require 'ol-docview)
(require 'org-element)
(require 'ob-groovy)
(require 'ox-man)
(require 'ol-gnus)
(require 'ob-table)
(require 'org-crypt)
(require 'org-pcomplete)
(require 'ob-octave)
(require 'term)
(require 'uniquify)
(require 'minibuffer)
(require 'mouse-copy)
(require 'paren)
(require 'help-mode)
(require 'mouse)
(require 'server)
(require 'md4)
(require 'w32-vars)
(require 'filesets)
(require 'misc)
(require 'misearch)
(require 'battery)
(require 'pixel-scroll)
(require 'info-xref)
(require 'files-x)
(require 'ibuf-ext)
(require 'macros)
(require 'cus-face)
(require 'dframe)
(require 'help-fns)
(require 'pcomplete)
(require 'reposition)
(require 'password-cache)
(require 't-mouse)
(require 'find-dired)
(require 'tab-line)
(require 'find-lisp)
(require 'cus-edit)
(require 'epg)
(require 'case-table)
(require 'scroll-lock)
(require 'epa)
(require 'dnd)
(require 'format)
(require 'outline)
(require 'obarray)
(require 'descr-text)
(require 'calc-sel)
(require 'calc-fin)
(require 'calc-store)
(require 'calc-math)
(require 'calc-help)
(require 'calc)
(require 'calc-mode)
(require 'calc-incom)
(require 'calc-stat)
(require 'calc-ext)
(require 'calc-nlfit)
(require 'calc-forms)
(require 'calc-menu)
(require 'calc-graph)
(require 'calc-units)
(require 'calc-macs)
(require 'calcalg3)
(require 'calc-misc)
(require 'calc-trail)
(require 'calcsel2)
(require 'calc-mtx)
(require 'calc-poly)
(require 'calc-rewr)
(require 'calc-funcs)
(require 'calc-arith)
(require 'calc-embed)
(require 'calc-yank)
(require 'calc-map)
(require 'calc-frac)
(require 'calc-undo)
(require 'calc-comb)
(require 'calc-lang)
(require 'calc-prog)
(require 'calc-rules)
(require 'calc-aent)
(require 'calc-alg)
(require 'calc-cplx)
(require 'calc-bin)
(require 'calcalg2)
(require 'calccomp)
(require 'calc-stuff)
(require 'calc-keypd)
(require 'calc-vec)
(require 'soundex)
(require 'time)
(require 'gnus-uu)
(require 'nnmbox)
(require 'gnus-win)
(require 'mail-source)
(require 'legacy-gnus-agent)
(require 'nnoo)
(require 'gnus-util)
(require 'nnmaildir)
(require 'nnnil)
(require 'mm-bodies)
(require 'gnus-picon)
(require 'gnus-cache)
(require 'gnus-mh)
(require 'deuglify)
(require 'gnus-agent)
(require 'mml1991)
(require 'nnregistry)
(require 'gnus-score)
(require 'gnus-vm)
(require 'gnus-dired)
(require 'gnus-diary)
(require 'nnspool)
(require 'gnus-srvr)
(require 'gnus-bookmark)
(require 'nnbabyl)
(require 'gnus-art)
(require 'gnus-rfc1843)
(require 'gnus-html)
(require 'nnvirtual)
(require 'gnus-int)
(require 'gnus-start)
(require 'nnml)
(require 'mm-partial)
(require 'nndir)
(require 'gnus-sum)
(require 'mml-sec)
(require 'nnrss)
(require 'gnus-cloud)
(require 'gmm-utils)
(require 'gnus-demon)
(require 'mml)
(require 'gnus-salt)
(require 'gnus-gravatar)
(require 'gnus-draft)
(require 'mm-util)
(require 'nndiary)
(require 'mml-smime)
(require 'mml2015)
(require 'spam)
(require 'nneething)
(require 'mm-uu)
(require 'mm-url)
(require 'gssapi)
(require 'gnus-cus)
(require 'nntp)
(require 'nnmail)
(require 'gnus-spec)
(require 'gnus-range)
(require 'mm-archive)
(require 'nnweb)
(require 'nndraft)
(require 'nnmairix)
(require 'gnus-logic)
(require 'gnus-registry)
(require 'message)
(require 'gnus-undo)
(require 'gnus)
(require 'gnus-ml)
(require 'nngateway)
(require 'gnus-delay)
(require 'gnus-dup)
(require 'smiley)
(require 'mm-decode)
(require 'gnus-cite)
(require 'canlock)
(require 'mm-encode)
(require 'gnus-sieve)
(require 'nnfolder)
(require 'nndoc)
(require 'gnus-bcklg)
(require 'gnus-mlspl)
(require 'gnus-eform)
(require 'gnus-group)
(require 'spam-report)
(require 'gnus-msg)
(require 'nnheader)
(require 'gnus-kill)
(require 'mm-view)
(require 'gnus-notifications)
(require 'nnimap)
(require 'score-mode)
(require 'smime)
(require 'gnus-async)
(require 'nnmh)
(require 'spam-stat)
(require 'gnus-topic)
(require 'mm-extern)
(require 'nnagent)
(require 'spam-wash)
(require 'gnus-fun)
(require 'dired-aux)
(require 'scroll-bar)
(require 'pcmpl-rpm)
(require 'mpc)
(require 'talk)
(require 'vc-hg)
(require 'vc-dir)
(require 'ediff-vers)
(require 'vc-cvs)
(require 'vc-bzr)
(require 'diff-mode)
(require 'emerge)
(require 'vc-hooks)
(require 'vc-rcs)
(require 'pcvs-parse)
(require 'vc-sccs)
(require 'ediff-wind)
(require 'vc-filewise)
(require 'vc-git)
(require 'log-edit)
(require 'ediff-mult)
(require 'pcvs-util)
(require 'ediff-util)
(require 'add-log)
(require 'pcvs-info)
(require 'ediff-merg)
(require 'vc-svn)
(require 'vc-dispatcher)
(require 'vc-annotate)
(require 'ediff-help)
(require 'ediff)
(require 'ediff-init)
(require 'ediff-diff)
(require 'log-view)
(require 'vc-dav)
(require 'vc)
(require 'smerge-mode)
(require 'pcvs)
(require 'vc-src)
(require 'ediff-ptch)
(require 'ediff-hook)
(require 'pcvs-defs)
(require 'compare-w)
(require 'diff)
(require 'cvs-status)
(require 'shell)
(require 'bs)
(require 'loadhist)
(require 'minibuf-eldef)
(require 'informat)
(require 'xt-mouse)
(require 'sort)
(require 'printing)
(require 'allout-widgets)
(require 'ansi-color)
(require 'executable)
(require 'elisp-mode)
(require 'idlw-help)
(require 'flymake)
(require 'cc-guess)
(require 'idlw-shell)
(require 'cc-mode)
(require 'gud)
(require 'gdb-mi)
(require 'idlw-toolbar)
(require 'f90)
(require 'etags)
(require 'cc-engine)
(require 'opascal)
(require 'xscheme)
(require 'asm-mode)
(require 'idlwave)
(require 'cwarn)
(require 'pascal)
(require 'cfengine)
(require 'ebnf-yac)
(require 'ebnf-abn)
(require 'inf-lisp)
(require 'glasses)
(require 'ebnf-dtd)
(require 'make-mode)
(require 'ebnf-bnf)
(require 'autoconf)
(require 'octave)
(require 'xref)
(require 'prog-mode)
(require 'cc-defs)
(require 'cc-styles)
(require 'subword)
(require 'cc-align)
(require 'cc-fonts)
(require 'bug-reference)
(require 'cpp)
(require 'mixal-mode)
(require 'ebnf2ps)
(require 'prolog)
(require 'cc-langs)
(require 'cc-menus)
(require 'idlw-complete-structtag)
(require 'ebnf-otz)
(require 'ld-script)
(require 'cmacexp)
(require 'meta-mode)
(require 'cc-vars)
(require 'vera-mode)
(require 'bat-mode)
(require 'sh-script)
(require 'verilog-mode)
(require 'js)
(require 'simula)
(require 'ebnf-iso)
(require 'fortran)
(require 'dcl-mode)
(require 'cperl-mode)
(require 'project)
(require 'ruby-mode)
(require 'scheme)
(require 'flymake-proc)
(require 'modula2)
(require 'python)
(require 'grep)
(require 'vhdl-mode)
(require 'tcl)
(require 'ebnf-ebx)
(require 'm4-mode)
(require 'compile)
(require 'hideif)
(require 'cc-bytecomp)
(require 'which-func)
(require 'hideshow)
(require 'sql)
(require 'antlr-mode)
(require 'cc-awk)
(require 'perl-mode)
(require 'ebrowse)
(require 'flymake-cc)
(require 'cc-cmds)
(require 'icon)
(require 'ps-mode)
(require 'wdired)
(require 'isearch)
(require 'find-cmd)
(require 'whitespace)
(require 'doc-view)
(require 'facemenu)
(require 'shadowfile)
(require 'plstore)
(require 'ps-mule)
(require 'face-remap)
(require 'frame)
(require 'help-macro)
(require 'info-look)
(require 'files)
(require 'help)
(require 'dired)
(require 'pcmpl-linux)
(require 'register)
(require 'vcursor)
(require 'icomplete)
(require 'rtree)
(require 'time-stamp)
(require 'replace)
(require 'format-spec)
(require 'pcmpl-x)
(require 'ebuff-menu)
(require 'auth-source)
(require 'so-long)
(require 'proced)
(require 'faces)
(require 'apropos)
(require 'repeat)
(require 'svg)
(require 'display-line-numbers)
(require 'jit-lock)
(require 'button)
(require 'abbrev)
(require 'newcomment)
(require 'image-mode)
(require 'widget)
(require 'woman)
(require 'font-core)
(require 'filecache)
(require 'info)
(require 'cus-dep)
(require 'cdl)
(require 'novice)
(require 'ido)
(require 'hexl)
(require 'dos-vars)
(require 'master)
(require 'hfy-cmap)
(require 'dynamic-setting)
(require 'auth-source-pass)
(require 'delsel)
(require 'tree-widget)
(require 'find-file)
(require 'wid-edit)
(require 'thingatpt)
(require 'align)
(require 'hilit-chg)
(require 'edmacro)
(require 'autoinsert)
(require 'ses)
(require 'dos-fns)
(require 'ffap)
(require 'registry)
(require 'iimage)
(require 'tutorial)
(require 'hi-lock)
(require 'mwheel)
(require 'expand)
(require 'pcmpl-gnu)
(require 'jsonrpc)
(require 'echistory)
(require 'autorevert)
(require 'dired-x)
(require 'completion)
(require 'desktop)
(require 'mule-diag)
(require 'latexenc)
(require 'cp51932)
(require 'fontset)
(require 'ja-dic-utl)
(require 'ccl)
(require 'utf7)
(require 'utf-7)
(require 'ucs-normalize)
(require 'mule-util)
(require 'robin)
(require 'quail)
(require 'iso-transl)
(require 'mule)
(require 'kkc)
(require 'ogonek)
(require 'eucjp-ms)
(require 'rfc1843)
(require 'latin1-disp)
(require 'iso-ascii)
(require 'iso-cvt)
(require 'charscript)
(require 'elide-head)
(require 'dom)
(require 'ert)
(require 'checkdoc)
(require 'byte-opt)
(require 'eieio-datadebug)
(require 'unsafep)
(require 'eldoc)
(require 'lisp-mode)
(require 'ert-x)
(require 'warnings)
(require 'eieio-speedbar)
(require 'rx)
(require 'cl-extra)
(require 'eieio)
(require 'eieio-core)
(require 'easymenu)
(require 'bytecomp)
(require 'inline)
(require 'shadow)
(require 'generic)
(require 'rmc)
(require 'advice)
(require 'backtrace)
(require 'pcase)
(require 'cl-generic)
(require 'check-declare)
(require 'timer-list)
(require 're-builder)
(require 'package)
(require 'package-x)
(require 'backquote)
(require 'disass)
(require 'eieio-base)
(require 'elint)
(require 'edebug)
(require 'benchmark)
(require 'radix-tree)
(require 'lisp-mnt)
(require 'chart)
(require 'ring)
(require 'crm)
(require 'eieio-custom)
(require 'tq)
(require 'tabulated-list)
(require 'debug)
(require 'helper)
(require 'cl-seq)
(require 'nadvice)
(require 'elp)
(require 'text-property-search)
(require 'generator)
(require 'cl-indent)
(require 'faceup)
(require 'smie)
(require 'bindat)
(require 'cl-macs)
(require 'copyright)
(require 'let-alist)
(require 'testcover)
(require 'seq)
(require 'regexp-opt)
(require 'subr-x)
(require 'trace)
(require 'thunk)
(require 'cl-preloaded)
(require 'regi)
(require 'map)
(require 'syntax)
(require 'pp)
(require 'timer)
(require 'derived)
(require 'cl-lib)
(require 'find-func)
(require 'gv)
(require 'ewoc)
(require 'eieio-opt)
(require 'cconv)
(require 'cl-print)
(require 'macroexp)
(require 'easy-mmode)
(require 'cursor-sensor)
(require 'avl-tree)
(require 'ps-bdf)
(require 'avoid)
(require 'tempo)
(require 'select)
(require 'srecode/find)
(require 'srecode/fields)
(require 'srecode/args)
(require 'srecode/getset)
(require 'srecode/el)
(require 'srecode/texi)
(require 'srecode/mode)
(require 'srecode/ctxt)
(require 'srecode/java)
(require 'srecode/cpp)
(require 'srecode/expandproto)
(require 'srecode/table)
(require 'srecode/srt)
(require 'srecode/document)
(require 'srecode/insert)
(require 'srecode/dictionary)
(require 'srecode/semantic)
(require 'srecode/extract)
(require 'srecode/map)
(require 'srecode/template)
(require 'srecode/compile)
(require 'srecode/srt-mode)
(require 'srecode/srt-wy)
(require 'srecode/filters)
(require 'ede/linux)
(require 'ede/pmake)
(require 'ede/emacs)
(require 'ede/project-am)
(require 'ede/srecode)
(require 'ede/speedbar)
(require 'ede/locate)
(require 'ede/generic)
(require 'ede/proj-archive)
(require 'ede/simple)
(require 'ede/proj-aux)
(require 'ede/make)
(require 'ede/config)
(require 'ede/proj-misc)
(require 'ede/pconf)
(require 'ede/proj-scheme)
(require 'ede/proj-elisp)
(require 'ede/cpp-root)
(require 'ede/custom)
(require 'ede/system)
(require 'ede/base)
(require 'ede/proj-obj)
(require 'ede/shell)
(require 'ede/proj)
(require 'ede/makefile-edit)
(require 'ede/proj-comp)
(require 'ede/files)
(require 'ede/dired)
(require 'ede/source)
(require 'ede/detect)
(require 'ede/auto)
(require 'ede/autoconf-edit)
(require 'ede/proj-prog)
(require 'ede/util)
(require 'ede/proj-info)
(require 'ede/proj-shared)
(require 'semantic/find)
(require 'semantic/imenu)
(require 'semantic/complete)
(require 'semantic/db-ref)
(require 'semantic/db-javascript)
(require 'semantic/scope)
(require 'semantic/decorate/include)
(require 'semantic/decorate/mode)
(require 'semantic/decorate)
(require 'semantic/ia)
(require 'semantic/db-ebrowse)
(require 'semantic/db-find)
(require 'semantic/db-el)
(require 'semantic/html)
(require 'semantic/texi)
(require 'semantic/ede-grammar)
(require 'semantic/ctxt)
(require 'semantic/lex)
(require 'semantic/symref/global)
(require 'semantic/symref/filter)
(require 'semantic/symref/cscope)
(require 'semantic/symref/list)
(require 'semantic/symref/grep)
(require 'semantic/symref/idutils)
(require 'semantic/java)
(require 'semantic/ia-sb)
(require 'semantic/tag-ls)
(require 'semantic/lex-spp)
(require 'semantic/bovine)
(require 'semantic/util-modes)
(require 'semantic/grammar-wy)
(require 'semantic/chart)
(require 'semantic/dep)
(require 'semantic/debug)
(require 'semantic/wisent/js-wy)
(require 'semantic/wisent/python-wy)
(require 'semantic/wisent/wisent)
(require 'semantic/wisent/javat-wy)
(require 'semantic/wisent/python)
(require 'semantic/wisent/java-tags)
(require 'semantic/wisent/grammar)
(require 'semantic/wisent/comp)
(require 'semantic/doc)
(require 'semantic/tag-write)
(require 'semantic/wisent)
(require 'semantic/format)
(require 'semantic/fw)
(require 'semantic/db-file)
(require 'semantic/bovine/el)
(require 'semantic/bovine/c)
(require 'semantic/bovine/gcc)
(require 'semantic/bovine/scm-by)
(require 'semantic/bovine/make)
(require 'semantic/bovine/debug)
(require 'semantic/bovine/scm)
(require 'semantic/bovine/c-by)
(require 'semantic/bovine/grammar)
(require 'semantic/bovine/make-by)
(require 'semantic/sb)
(require 'semantic/tag-file)
(require 'semantic/sort)
(require 'semantic/db-typecache)
(require 'semantic/idle)
(require 'semantic/db-debug)
(require 'semantic/senator)
(require 'semantic/db)
(require 'semantic/edit)
(require 'semantic/db-global)
(require 'semantic/symref)
(require 'semantic/analyze/fcn)
(require 'semantic/analyze/complete)
(require 'semantic/analyze/debug)
(require 'semantic/analyze/refs)
(require 'semantic/grammar)
(require 'semantic/mru-bookmark)
(require 'semantic/db-mode)
(require 'semantic/util)
(require 'semantic/tag)
(require 'semantic/analyze)
(require 'transient)
(require 'external-completion)
(require 'eglot)
(require 'sha1)

Re: Unboxed package manager

[Eli Zaretskii]

> Date: Tue, 21 Mar 2023 10:52:20 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Yuan Fu <casouri@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> 
> > You could construct a test without external packages by just generating 
> > N directories, prepending them to the load path, then doing the same 
> > loadall to see if the degradation is purely due to the size of the load 
> > path.
> >
> 
> Indeed, with the attached file, emacs -Q -l loadall.el takes 120 seconds. 
> Hmmm...

Why are you surprised?  Having 2000 directories in your load-path is a
bad idea regardless.

Re: Unboxed package manager

[Gregory Heytings]

>>> You could construct a test without external packages by just 
>>> generating N directories, prepending them to the load path, then doing 
>>> the same loadall to see if the degradation is purely due to the size 
>>> of the load path.
>>
>> Indeed, with the attached file, emacs -Q -l loadall.el takes 120 
>> seconds. Hmmm...
>
> Why are you surprised?  Having 2000 directories in your load-path is a 
> bad idea regardless.
>

I'm not surprised, but I wonder whether it's possible to do better.

Re: Unboxed package manager

[Eli Zaretskii]

> Date: Tue, 21 Mar 2023 13:33:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: owinebar@gmail.com, casouri@gmail.com, emacs-devel@gnu.org
> 
> >> Indeed, with the attached file, emacs -Q -l loadall.el takes 120 
> >> seconds. Hmmm...
> >
> > Why are you surprised?  Having 2000 directories in your load-path is a 
> > bad idea regardless.
> >
> 
> I'm not surprised, but I wonder whether it's possible to do better.

Yes: load each package with an explicit absolute file name, without
relying on load-path search.

Re: Unboxed package manager

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]


>> I'm not surprised, but I wonder whether it's possible to do better.
>
> Yes: load each package with an explicit absolute file name, without 
> relying on load-path search.
>

Of course 😉  But by "doing better" I meant "to improve the way Emacs uses 
load-path in such cases".  Likely with some kind of cache.

Re: Unboxed package manager

[Eli Zaretskii]

> Date: Tue, 21 Mar 2023 14:20:41 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: owinebar@gmail.com, casouri@gmail.com, emacs-devel@gnu.org
> 
> >> I'm not surprised, but I wonder whether it's possible to do better.
> >
> > Yes: load each package with an explicit absolute file name, without 
> > relying on load-path search.
> >
> 
> Of course 😉  But by "doing better" I meant "to improve the way Emacs uses 
> load-path in such cases".

Why try to solve a problem if it can be avoided altogether?

> Likely with some kind of cache.

A cache will not necessarily help, and you will pay the price of
searching it at least once.  It should be clear that enlarging the
length of load-path is not scalable, unless we make it a hash-table of
sorts.  And I'm not sure we want this kind of complexity for such a
hilariously marginal use case.

Re: Unboxed package manager

[Lynn Winebarger]

On Tue, Mar 21, 2023 at 1:29 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Tue, 21 Mar 2023 14:20:41 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: owinebar@gmail.com, casouri@gmail.com, emacs-devel@gnu.org
> >
> > >> I'm not surprised, but I wonder whether it's possible to do better.
> > >
> > > Yes: load each package with an explicit absolute file name, without
> > > relying on load-path search.

That is not guaranteed to work unless you are certain the required
files are loaded in the correct order and don't use any "load"
expressions themselves.

> > Of course 😉  But by "doing better" I meant "to improve the way Emacs uses
> > load-path in such cases".
>
> Why try to solve a problem if it can be avoided altogether?
>
> > Likely with some kind of cache.
>
> A cache will not necessarily help, and you will pay the price of
> searching it at least once.  It should be clear that enlarging the
> length of load-path is not scalable, unless we make it a hash-table of
> sorts.  And I'm not sure we want this kind of complexity for such a
> hilariously marginal use case.

Maybe 2000+ is marginal (today), but let's say someone just installs
the 300 packages available on gnu elpa.  Note I'm just installing
these packages, not actually loading any of them directly. Assuming
the time cost is linear in the number of prepended paths, that would
cause load time for system libraries, whether at startup or by
autoloading, to increase to ~18s, or increase by a factor of 4, even
if those packages are only available for autoloading and not actually
loaded.

Although if you look at something like doomemacs, people go to a lot
of trouble to make packages available on their system without paying
this overhead.  The simplest approach is probably just to avoid
increasing the size of the load path when it's not really necessary.

Lynn

Re: Unboxed package manager

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Tue, 21 Mar 2023 20:48:29 -0400
> Cc: Gregory Heytings <gregory@heytings.org>, casouri@gmail.com, emacs-devel@gnu.org
> 
> > A cache will not necessarily help, and you will pay the price of
> > searching it at least once.  It should be clear that enlarging the
> > length of load-path is not scalable, unless we make it a hash-table of
> > sorts.  And I'm not sure we want this kind of complexity for such a
> > hilariously marginal use case.
> 
> Maybe 2000+ is marginal (today), but let's say someone just installs
> the 300 packages available on gnu elpa.

Why would someone want all 300 of them?  Some of them even contradict
each other, in that they implement similar features in very different
ways.

> Note I'm just installing
> these packages, not actually loading any of them directly.

Exactly.  So this is entirely theoretical use case, not a real one.

> Although if you look at something like doomemacs, people go to a lot
> of trouble to make packages available on their system without paying
> this overhead.  The simplest approach is probably just to avoid
> increasing the size of the load path when it's not really necessary.

As I already said: you can simply load each package by its explicit
absolute name.  Poof! problem disappeared.

Re: Unboxed package manager

[Lynn Winebarger]

On Wed, Mar 22, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Tue, 21 Mar 2023 20:48:29 -0400
> > Cc: Gregory Heytings <gregory@heytings.org>, casouri@gmail.com, emacs-devel@gnu.org
> >
> > > A cache will not necessarily help, and you will pay the price of
> > > searching it at least once.  It should be clear that enlarging the
> > > length of load-path is not scalable, unless we make it a hash-table of
> > > sorts.  And I'm not sure we want this kind of complexity for such a
> > > hilariously marginal use case.
> >
> > Maybe 2000+ is marginal (today), but let's say someone just installs
> > the 300 packages available on gnu elpa.
>
> Why would someone want all 300 of them?  Some of them even contradict
> each other, in that they implement similar features in very different
> ways.

You're correct, many of them do address similar issues and should not
be "turned on" simultaneously.  As long as they are well-behaved and
able to be switched on and off reliably, though, there's no reason to
not have them simultaneously installed and loaded if the user is not
dedicated to minimizing resource consumption.
*  As a user, I don't want to be locked into one approach or the other
by the pain of having to uninstall one (set of) package(s) and install
another (set of) package(s).  That's usually one of the practical
benefits of free software.
*  It's easier if constructing a configuration that will be shared by
multiple users who either aren't inclined or aren't allowed to add
packages dynamically and/or write extensive elisp in an initialization
file, whether dynamically loaded or in a large dump file.

As I've noted in other responses, there is a substantial contingent of
users making use of customized package distributions whose organizers
appear to go to some length to keep load times reasonable.  Some of
them use extensive configuration to ensure only selected packages are
added to the load path, regardless of whether they are present on
disk.  Then there's the chemacs tool to completely switch
initialization directories.  I don't know the extent to which this
basic performance limitation in the design of package management is
causing this, but I have to think that if users found that they were
able to just load everything and simply turn features on and off at
will, some substantial contingent would find the customize system much
more useful and have less need of maintaining these alternative
configuration management systems.

One fellow using my shared setup on the sandboxed system regularly
uses doom emacs on his personal system.  He gave me a substantial set
of packages to add to the ones I was putting in a (byte-compiled) dump
file.  We use the same dump file (which I have arranged to remove all
traces of the build user in variables and replace with values
appropriate for the actual user at startup).   He's recreated doom
emacs (as far as I know) and I run a much more vanilla configuration.
He can use helm and I can use ivy or selectrum or vertico using
customize.  The dump file loads in a few seconds.  Even without
dumping, loading the world (which is over a 1000 packages + all
libraries included with core emacs) only takes ~30 seconds or so, as I
have manually unboxed almost all the packages.  The memory used by the
process is comparable to what VSCode would consume.

So, I don't find it hilariously marginal at all.  I think it's a
severe constraint that discourages non-experts from using emacs to its
fullest when they have alternatives like VSCode that come with a lot
of bells and whistles ready to go.  And a big chunk of the performance
pain appears to be associated with the design choice of putting
packages in their own directory, even when it's completely unnecessary
for their correct functioning.

And installing/deinstalling packages is a slow business.  The fact
that some of these projects go further and finely divide their files
into multiple packages so the user can tune the parts just aggravates
the situation - and makes switching between different approaches by
uninstalling one and installing the other even more painful.

>
> > Note I'm just installing
> > these packages, not actually loading any of them directly.
>
> Exactly.  So this is entirely theoretical use case, not a real one.

I was just noting that the performance hit comes from merely
installing the package (and enlarging the load-path), not from loading
it.

I will be loading all the packages and dumping them, once I have a
systematic unboxing facility.

> > Although if you look at something like doomemacs, people go to a lot
> > of trouble to make packages available on their system without paying
> > this overhead.  The simplest approach is probably just to avoid
> > increasing the size of the load path when it's not really necessary.
>
> As I already said: you can simply load each package by its explicit
> absolute name.  Poof! problem disappeared.

You're incorrect, unless you can ensure required system packages are
all explicitly loaded in order.  SImply specifying the full path of
one will not prevent it searching for its dependencies.

The analysis of the cost of pre-computing the load-path is also
misguided.  The fact that you would have to traverse the load-path at
least once to build the tables is hardly a criticism, since the
alternative is performing the search on every load.

That said, I don't advocate adding that complexity when rationalizing
the way packages are installed is an effective way to avoid enlarging
the load-path in the first place.  The number of packages whose design
really requires enlarging the load-path (just as a user of the
package, not in "development mode") is tiny in my experience.  realgud
and its extensions are in fact the only ones that come to mind,
because of the use of load-relative.  Others may be tricky to handle
automatically, but there's nothing in their design that fundamentally
requires enlarging the load path - a few manual modifications are
usually enough to handle those cases, which are fairly small in number
(magit being one of them due to the version function expecting to find
the package installed in its own directory or to be able to contact
the main git repository).

Lynn

Re: Unboxed package manager

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 22 Mar 2023 18:22:55 -0400
> Cc: gregory@heytings.org, casouri@gmail.com, emacs-devel@gnu.org
> 
> On Wed, Mar 22, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Why would someone want all 300 of them?  Some of them even contradict
> > each other, in that they implement similar features in very different
> > ways.
> 
> You're correct, many of them do address similar issues and should not
> be "turned on" simultaneously.  As long as they are well-behaved and
> able to be switched on and off reliably, though, there's no reason to
> not have them simultaneously installed and loaded if the user is not
> dedicated to minimizing resource consumption.

I didn't say people should not be able to do that.  They should, and
they are.  I just said it isn't a reasonable thing to do, and thus
doesn't justify our jumping through hoops to cater for it.

> > > Note I'm just installing
> > > these packages, not actually loading any of them directly.
> >
> > Exactly.  So this is entirely theoretical use case, not a real one.
> 
> I was just noting that the performance hit comes from merely
> installing the package (and enlarging the load-path), not from loading
> it.

And I was just noting that doing such a thing cannot be of any
practical interest to us, unless it causes severe bugs in Emacs, like
crashes etc.

> > As I already said: you can simply load each package by its explicit
> > absolute name.  Poof! problem disappeared.
> 
> You're incorrect, unless you can ensure required system packages are
> all explicitly loaded in order.  SImply specifying the full path of
> one will not prevent it searching for its dependencies.

I obviously assumed (and I think I even said that explicitly) that the
directories of those packages shouldn't be added to load-path;
_instead_, they should be loaded by their explicit file names,
including leading directories.  _Then_ the problem with load-path will
not happen.

As for the order of loading packages -- that problem exists anyway,
and I believe use-package, which is now part of Emacs, is capable of
making the solution a bit easier.  In any case, solving that is
basically a one-time issue, when you first install a new package.

> That said, I don't advocate adding that complexity when rationalizing
> the way packages are installed is an effective way to avoid enlarging
> the load-path in the first place.  The number of packages whose design
> really requires enlarging the load-path (just as a user of the
> package, not in "development mode") is tiny in my experience.  realgud
> and its extensions are in fact the only ones that come to mind,
> because of the use of load-relative.  Others may be tricky to handle
> automatically, but there's nothing in their design that fundamentally
> requires enlarging the load path - a few manual modifications are
> usually enough to handle those cases, which are fairly small in number
> (magit being one of them due to the version function expecting to find
> the package installed in its own directory or to be able to contact
> the main git repository).

So what exactly are you asking for?  Is it some change in package.el?
If so, what change?

Re: Unboxed package manager

[Lynn Winebarger]

On Thu, Mar 23, 2023 at 2:46 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 22 Mar 2023 18:22:55 -0400
> > Cc: gregory@heytings.org, casouri@gmail.com, emacs-devel@gnu.org
> >
> > On Wed, Mar 22, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > Why would someone want all 300 of them?  Some of them even contradict
> > > each other, in that they implement similar features in very different
> > > ways.
> >
> > You're correct, many of them do address similar issues and should not
> > be "turned on" simultaneously.  As long as they are well-behaved and
> > able to be switched on and off reliably, though, there's no reason to
> > not have them simultaneously installed and loaded if the user is not
> > dedicated to minimizing resource consumption.
>
> I didn't say people should not be able to do that.  They should, and
> they are.  I just said it isn't a reasonable thing to do, and thus
> doesn't justify our jumping through hoops to cater for it.

Who's asking anyone to jump through hoops to cater to anything?  All
I've done in this thread is ask for any insight on the best way to
derive from package.el, since I only want to extend the
installation/activation/deinstallation behavior, and then respond to
questions about why I would do such a thing.

In fact, I don't think I've asked for any concrete action from emacs
developers beyond not breaking functions that have worked in the past.
I've asked for insight into coding issues, and reported on the results
of my experiments.  I'm not sure how you define "a reasonable thing to
do" without considering evidence from experience when it's available.

> > > > Note I'm just installing
> > > > these packages, not actually loading any of them directly.
> > >
> > > Exactly.  So this is entirely theoretical use case, not a real one.
> >
> > I was just noting that the performance hit comes from merely
> > installing the package (and enlarging the load-path), not from loading
> > it.
>
> And I was just noting that doing such a thing cannot be of any
> practical interest to us, unless it causes severe bugs in Emacs, like
> crashes etc.

No, you were characterizing my efforts as "entirely theoretical",
which they are not.

I have no idea who you are speaking on behalf of when you say "to us".

> I obviously assumed (and I think I even said that explicitly) that the
> directories of those packages shouldn't be added to load-path;

But that is what is done by package.el.

> _instead_, they should be loaded by their explicit file names,
> including leading directories.  _Then_ the problem with load-path will
> not happen.

That wasn't clear at all - it appeared you were referring to the
"require" statements in the "loadall.el" file, since the packages were
not being loaded.

> As for the order of loading packages -- that problem exists anyway,
> and I believe use-package, which is now part of Emacs, is capable of
> making the solution a bit easier.  In any case, solving that is
> basically a one-time issue, when you first install a new package.

So, your definition of "reasonable" implies that packages are only
infrequently changed, or experimented with?

> So what exactly are you asking for?  Is it some change in package.el?
> If so, what change?

See above - I did not ask anyone to *do* anything in terms of coding.
I asked whether anyone more familiar with package.el and elisp than I
am had any better ideas for deriving functionality from package.el
than using "advice".

Lynn

Re: Unboxed package manager

[chad]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

I fear that this discussion might have veered too far into
theoretical territory for some of the conversants, and probably not the
_same_ theoretical territory for everyone. With that in mind, I *think*
that what Lynn Winebarger is asking for might be best answered by:

Try digging into the package-quickstart part of package, especially
> p-q-refresh and p-q-maybe-refresh.


On the other hand, I think that the best answer to "should emacs-devel
actually care about loading scads of packages" might be:

There seem to be a lot of (potential) emacs users, especially in the
> "interested but not already heavily enfranchised subset", that make use of
> emacs "distributions", and that model would probably benefit from
> improvements to "loading scads of packages".


This doesn't answer the "should" question, but I think it provides a
closer-to-real-practice basis for consideration.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 1360 bytes --]

Re: Unboxed package manager

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2054 bytes --]

On Fri, Mar 24, 2023, 1:55 PM chad <yandros@gmail.com> wrote:

> I fear that this discussion might have veered too far into
> theoretical territory for some of the conversants, and probably not the
> _same_ theoretical territory for everyone. With that in mind, I *think*
> that what Lynn Winebarger is asking for might be best answered by:
>
> Try digging into the package-quickstart part of package, especially
>> p-q-refresh and p-q-maybe-refresh.
>
>
I'm already well into writing the first draft of the code.  I'm writing it
so packages will get saved down as they are now, but will not be in the
"selected-packages" list used to construct the quickstart file.  Preferably
I will be able to circumvent the compilation step (of the "boxed"
libraries) as well.

I've had to write a little function for cleanly replacing text
corresponding to a sexpr according to a pcase matcher.  I'd prefer to use a
pre-existing implementation, but I'm not aware of one.  It might be handy
in emacs-lisp-mode.  The tree-sitter variant of emacs-lisp-mode might
already provide a similar facility for replacing text based on a syntactic
pattern.


> On the other hand, I think that the best answer to "should emacs-devel
> actually care about loading scads of packages" might be:
>
> There seem to be a lot of (potential) emacs users, especially in the
>> "interested but not already heavily enfranchised subset", that make use of
>> emacs "distributions", and that model would probably benefit from
>> improvements to "loading scads of packages".
>
>
That's a fair and concise summary.   When I preload the world, M-x
customize is very usable for configuration purposes.  I have a hard time
seeing many casual new users accustomed to GUI configuration spending time
writing extensive elisp init files just to try packages to get UI features
they're accustomed to from other editors.

>
> This doesn't answer the "should" question, but I think it provides a
> closer-to-real-practice basis for consideration.
>
> Hope that helps,
> ~Chad
>

I think so - thanks!

Lynn

[-- Attachment #2: Type: text/html, Size: 3614 bytes --]

Re: Unboxed package manager

[David Masterson]

Eli Zaretskii <eliz@gnu.org> writes:

> As I already said: you can simply load each package by its explicit
> absolute name.  Poof! problem disappeared.

So, the use case is managing packages in your .emacs over time.  As you
collect packages over time (job changes, etc.), the list of packages
grows (laziness might lead to keeping more than you need).  Also, packages
will get updated (perhaps, automatically via auto-package-update).
Constantly updating package absolute names is rife for mistake.

Perhaps (say) use-package could be adjusted to handle this via a
persistent index of '(pkgname . abs-pathname)' items. Some init function
could load the persistent index. Use-package would then use the index to
find the package directly. If the abs-pathname no longer exists,
use-package can try via load-path and update the persistent index
accordingly for next time.

Possible?

-- 
David Masterson

Re: Unboxed package manager

[Eli Zaretskii]

> From: David Masterson <dsmasterson@gmail.com>
> Cc: Lynn Winebarger <owinebar@gmail.com>,  gregory@heytings.org,
>   casouri@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 22 Mar 2023 18:44:32 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > As I already said: you can simply load each package by its explicit
> > absolute name.  Poof! problem disappeared.
> 
> So, the use case is managing packages in your .emacs over time.

Yes, it does.  And it should, IMNSHO.  I actually don't believe that
having hundreds of packages installed doesn't already cause
non-trivial management issues, even with the loading time and the
length of load-path put aside.  You will have compatibility problems
between packages, contradictions and clashes of names, key bindings,
etc.

So it isn't like what I suggest creates problems where previously
there were none.

Re: Unboxed package manager

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Tue, Mar 21, 2023 at 07:29:51PM +0200, Eli Zaretskii wrote:
> > Date: Tue, 21 Mar 2023 14:20:41 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: owinebar@gmail.com, casouri@gmail.com, emacs-devel@gnu.org
> > 
> > >> I'm not surprised, but I wonder whether it's possible to do better.
> > >
> > > Yes: load each package with an explicit absolute file name, without 
> > > relying on load-path search.
> > >
> > 
> > Of course 😉  But by "doing better" I meant "to improve the way Emacs uses 
> > load-path in such cases".
> 
> Why try to solve a problem if it can be avoided altogether?
> 
> > Likely with some kind of cache.
> 
> A cache will not necessarily help, and you will pay the price of
> searching it at least once.  It should be clear that enlarging the
> length of load-path is not scalable, unless we make it a hash-table of
> sorts.  And I'm not sure we want this kind of complexity for such a
> hilariously marginal use case.

I agree 100%. A cache is not free (performance wise, but also, more
importantly bug wise). You'd make the overwhelming majority of users
pay that price for a corner case which, again, in its vast majority
would translate to "well, don't do that, then".

This might change if our landscape changed radically, but: are we
there yet?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]