ELPA and EmacsWiki Updates

ELPA and EmacsWiki Updates

[Nordlöw]

I am using a lot of Emacs Extensions from EmacsWiki for example the
brilliant Icicles. Can ELPA or some other package automatically
synchronize/update my copies of these extensions with some nice Emacs
Package? When I do elpa-refresh-contents() and then elpa-list-
packages() I "only" get the extensions given below. Can I add
repositories to elpa as I can I can with apt?

Thanks in advance,
Nordlöw


  Package	    Version   Status
  -------	    -------   ------
  abacus	    1.0.2     available
  archive-downloader 1.1      available
  blank-mode	    6.6	      available
  bubbles	    0.3	      available
  cal-china-x	    0.6	      available
  caps-mode	    1.0	      available
  changelog-url	    0.1	      available
  chess		    1.96      available
  compilation-recenter-end 2  available
  css-mode	    1.0	      available
  dictionary	    1.8.7     available
  dired-isearch	    0.3	      available
  emacs		    23.0
  emms		    3.0	      available
  erc		    5.2	      available
  etags-select	    1.1	      available
  gdb-shell	    0.2	      available
  gtk-look	    5	      available
  highlight-parentheses	0.9.1 available
  highline	    4.2	      available
  iresize	    0.2	      available
  javascript	    1.99.8    available
  jimb-patch	    1.1	      available
  kill-ring-search  1.1	      available
  lambdacalc	    1.0	      available
  less		    0.2	      available
  light-symbol	    0.1	      available
  lisppaste	    1.2	      available
  lua-mode	    20070608  available
  muse		    3.11      available
  newsticker	    1.10      available
  package	    0.5	      available
  sgftree	    0.1	      available
  smart-operator    0.9	      available
  sudoku	    0.3	      available
  tex-math-preview  4	      available
  url		    1.16
  wajig		    0.53      available
  weblogger	    1.2	      available
  worklog	    2.4.2     available
  wtf		    2.0	      available
  xml-rpc	    1.6.4     available
  xtide		    4	      available

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Nordlöw" == Nordlöw  <per.nordlow@gmail.com> writes:

Nordlöw> I am using a lot of Emacs Extensions from EmacsWiki for example the
Nordlöw> brilliant Icicles. Can ELPA or some other package automatically
Nordlöw> synchronize/update my copies of these extensions with some nice Emacs
Nordlöw> Package?

There is some other installer on the wiki that might help here.  I
don't know, I haven't used it (since it doesn't have the features I
want).

FWIW, the reason that Icicles is not in ELPA is that the Icicles
author did not want it there.

As for other things not being in ELPA, the general reason for this is
that I probably haven't gotten around to it yet.  Smaller (single
file) packages tend to require some comment fixes first, and I'm not
always clear on what people actually use (and what therefore would be
worthwhile to upload).  Larger packages often require some extra
preparation.

And, finally, I like to coordinate with maintainers so that:

* Any needed patches go upstream
* The package has a "nice" activation approach (autoloads comments in
  place, whatever), and
* so that maintainers know to send me new versions when released

I'm happy to have help with any or all of this :-).  If there's enough
demand I'll move the ELPA repository to a site like savannah so that
other folks can do uploads as well.

Tom

RE: ELPA and EmacsWiki Updates

[Drew Adams]

> Nordlöw> I am using a lot of Emacs Extensions from EmacsWiki for 
> Nordlöw> example the brilliant Icicles. Can ELPA or some other
> Nordlöw> package automatically synchronize/update my copies of
> Nordlow> these extensions with some nice Emacs Package?
> 
> There is some other installer on the wiki that might help here.  I
> don't know, I haven't used it (since it doesn't have the features I
> want).
> 
> FWIW, the reason that Icicles is not in ELPA is that the Icicles
> author did not want it there.

Hmm. That's not quite right, Tom. What I said at the time was that it seemed (then) that some package system (possibly ELPA) would soon be added to Emacs, and I was intending to wait to see what Icicles changes might be needed to adapt it for use with the (future) standard package system.

Now, it looks like there will be no such standard Emacs package system. Why don't you email me off list to talk again about what I might need to do to make Icicles Elpable?

However, it's also true that I said that (1) I appreciate the simplicity of uploading to Emacs Wiki, and (2) there are already several ways to download all of Icicles at once from the wiki (see http://www.emacswiki.org/cgi-bin/wiki/Icicles_-_Libraries#BulkIciclesDownload). I'm not sure what Nordlow needs beyond that, or whether he has even tried those ways.

Anyway, my mind is not closed against ELPA for Icicles (or Icicles for ELPA). I'm not too clear on what I would need to do - feel free to contact me about it. Especially if it's just a one-time change, and not too difficult, it's likely that I will do it.

Is there a way for ELPA to get stuff automatically from Emacs Wiki - that is, stuff that has been made Elpable? That would be a big help.

There are already several sites that provide Emacs-Lisp libraries (including Emacs-Lisp List, Emacs Wiki, and ELPA, not to mention sites of individual authors and group projects), and there is gnu-emacs-sources@gnu.org. 

It would be great if library authors could simply upload to one site and have the others feed off of that. (Via RSS? I don't know anything about RSS, so don't laugh if it's irrelevant here.) Emacs Wiki already mirrors the Emacs-Lisp List, but it would, IMO, be much better the other way around: it would be good if other sites that are more like simple repositories fed off of the code posted to the wiki.

I prefer Emacs Wiki for its simplicity, including the ability - by anyone - to easily post or update code and associated documentation. In particular, the hyperlinking makes it a tremendous resource for everyone - much better, IMO, than just a check-in/out code repository. The wiki has no notion of packages, however, so it would be helpful if a package system such as ELPA could feed off of the wiki somehow.

> As for other things not being in ELPA, the general reason for this is
> that I probably haven't gotten around to it yet.  Smaller (single
> file) packages tend to require some comment fixes first, and I'm not
> always clear on what people actually use (and what therefore would be
> worthwhile to upload).  Larger packages often require some extra
> preparation.
> 
> And, finally, I like to coordinate with maintainers so that:
> 
> * Any needed patches go upstream
> * The package has a "nice" activation approach (autoloads comments in
>   place, whatever), and
> * so that maintainers know to send me new versions when released

See my comments above. In my case, I don't make package "releases". I enhance or fix one library (or more) of the "package" and upload it immediately to the wiki, without changing anything else in the "package". I don't repackage, zip, tar, or anything else. I just diff the old and the new file, write a comment in the new file, and upload it - without the unchanged rest of the package. On the wiki I can easily compare previous revisions of a file and roll the current revision back if necessary. Users can easily see what has changed and download just a single library change. (They can also download the entire package, if they want.)

I'm not against a package system, depending on what that means, but it would be ideal, I think, if packages were more or less virtual, updated automatically whenever a member file is updated on the wiki. Perhaps a wiki-based package system of some kind would be appropriate? If not, then it would be good if a standalone package system such as ELPA could feed itself off of the wiki.

> I'm happy to have help with any or all of this :-).  If there's enough
> demand I'll move the ELPA repository to a site like savannah so that
> other folks can do uploads as well.

FWIW, I haven't noticed Savannah being anywhere near as simple to use as Emacs Wiki. No flames from Savannah-ites, please.

Re: ELPA and EmacsWiki Updates

[Edward O'Connor]

> Smaller (single file) packages tend to require some comment fixes
> first[...]

What sort of fixes? Could ELPA be enhanced to require fewer changes, or
no changes at all, for the common case?


Ted

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Ted" == Edward O'Connor <hober0@gmail.com> writes:

>> Smaller (single file) packages tend to require some comment fixes
>> first[...]

Ted> What sort of fixes? Could ELPA be enhanced to require fewer changes, or
Ted> no changes at all, for the common case?

Usually it is just header comments.  In particular package.el needs 2
things:

* The header and trailer comments have to be correct according to the
  Emacs commenting guidelines.  The header comment holds the package's
  name and also a comment describing the package; also the header and
  trailer are used by package-upload-file to find the boundaries of
  the file.

* package.el needs a "Version:" header comment whose value is a
  "dotted numeric" version number.  package.el uses this to handle
  upgrading packages, only activating a single version, etc.
  (There are other header comments that package.el can use, but this
  is the only required one.)

I suppose we could dispense with these somehow, at the cost of reduced
functionality.  It does make it simpler for uploading if a file has
these -- I don't have to enter anything by hand.  And versioning, I
think, is good for users.

I also like to make sure that a file has proper ;;;###autoload
comments.  I think it is important that packages come ready for action
-- a big part of the goal of ELPA is to make it simple for users to
install and use Emacs packages.

Tom

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:

>> FWIW, the reason that Icicles is not in ELPA is that the Icicles
>> author did not want it there.

Drew> Hmm. That's not quite right, Tom. What I said at the time was
Drew> that it seemed (then) that some package system (possibly ELPA)
Drew> would soon be added to Emacs, and I was intending to wait to see
Drew> what Icicles changes might be needed to adapt it for use with
Drew> the (future) standard package system.

Ok.  I must have misremembered, thanks for clearing that up.

Drew> However, it's also true that I said that (1) I appreciate the
Drew> simplicity of uploading to Emacs Wiki

Yeah.  That is simpler, for sure.  I considered having ELPA download
straight from the wiki, but I was not comfortable with the
uncontrolled aspect of it.  My concern was that if ELPA became
commonplace, someone would upload a trojan.

Drew> I'm not too clear on what I would need to do - feel free to
Drew> contact me about it. Especially if it's just a one-time change,
Drew> and not too difficult, it's likely that I will do it.

I documented what to do for single-file packages on the web site.  For
a multi-file package, you must:

* Put all the files into a .tar which has the package name and
  version.  The files must be in a directory of the same name.  E.g.,
  the package "emms-3.0.tar" the files must all appear beneath the
  top-level directory "emms-3.0/"

* Make a -pkg.el file in the package.  This holds the call to
  define-package.  E.g., "emms-3.0/emms-pkg.el".  The define-package
  call sets the version number, description, dependencies, etc.

* If you have a texinfo manual, make a .info file and a dir file and
  put those in the package directory in the tar.

In other packages I've ELPA-ized, I made a 'make elpa' Makefile target
to run info, run install-info, set the version number in the -pkg
file, and make the tar.

Drew> Is there a way for ELPA to get stuff automatically from Emacs
Drew> Wiki - that is, stuff that has been made Elpable? That would be
Drew> a big help.

There isn't.  For a multi-file package it would have to be pretty
complicated.

Drew> It would be great if library authors could simply upload to one
Drew> site and have the others feed off of that. (Via RSS? I don't
Drew> know anything about RSS, so don't laugh if it's irrelevant
Drew> here.) Emacs Wiki already mirrors the Emacs-Lisp List, but it
Drew> would, IMO, be much better the other way around: it would be
Drew> good if other sites that are more like simple repositories fed
Drew> off of the code posted to the wiki.

Yeah.  For the Wiki the security issue concerns me quite a bit, but
maybe this could be fixed somehow.  My plan for ELPA was to move to a
hosting site that would more easily allow multiple maintainers, but I
haven't done that yet.

Drew> The wiki has no notion of packages, however, so it would be
Drew> helpful if a package system such as ELPA could feed off of the
Drew> wiki somehow.

My experience based on looking at a *lot* of Emacs packages is that
the solution will never be as convenient for authors as "dump it on
the wiki".  A nicely (IMO) functioning system will always require
meta-data; the typical Emacs Lisp program handles this by a comment
and dropping the problem on the user.

Drew> See my comments above. In my case, I don't make package
Drew> "releases"
[...]
Drew> I'm not against a package system, depending on what that means,
Drew> but it would be ideal, I think, if packages were more or less
Drew> virtual, updated automatically whenever a member file is updated
Drew> on the wiki.

I see.  ELPA is not a good fit for what you want then.  Most
non-trivial Emacs packages are developed along more traditional lines,
with releases and (non-wiki) version control, etc.

ELPA was designed to solve a few problems that I encounter:

* Download, install, and compile a package
* Automatically handle package dependencies
* Handle upgrading a package
* Handle the case where an external package is incorporated into Emacs
  core but also has ongoing external development.  This is the
  requirement behind much of the versioning support -- the idea is to
  always choose the newer package regardless of where it appears.

Tom

RE: ELPA and EmacsWiki Updates

[Drew Adams]

Hi Tom,

> Drew> However, it's also true that I said that (1) I appreciate the
> Drew> simplicity of uploading to Emacs Wiki
>
> Yeah.  That is simpler, for sure.  I considered having ELPA download
> straight from the wiki, but I was not comfortable with the
> uncontrolled aspect of it.  My concern was that if ELPA became
> commonplace, someone would upload a trojan.

I still think it might be worth pursuing. I don't see why the risks would be
greater than what is already true for the wiki itself.

If your concern is also about your site becoming infected, then perhaps this
could be done on a different site - perhaps the wiki itself.

> For a multi-file package, you must:

<snip>

OK, thanks; I will think about it. I certainly have no problem with defining
an overall package-description file ("-pkg.el") that says what files are
included or whatever other such info is needed.

I don't use (or like much) a notion of a package "version" or "release" for
my little offerings. I prefer to treat changes ("releases", "revisions" or
whatever) at the level of individual files - each file in a package has its
own update (version) number. A file's update number is incremented
automatically in the file header each time I save the file - there's nothing
to do.

I think that such file-level version numbers can be more flexible for both
authors and users, at least for single-author projects. In any case, it's
what I prefer. So, if I were to upload a package to your site, it's unlikely
that I would often declare a new release and update the package as a whole.
The ELPA version is likely to be stale, due to my laziness.

That you need some overall description of a package makes sense. That that
description includes a list of files or other dependencies and a verbal
explanation (how to install etc.) also makes sense. This is what your
"package" file is, I guess.

But why the constraint that a package have an overall version number?
Unless, for instance, you offer multiple versions of the same package. In my
case, I would like the package file to simply point to the requisite files
in the package, and that's it. The same package file would then serve for
any component file versions. The package file would only need to be changed
if the set of component files changed.

The second thing that I'm not crazy about, besides the notion of package
version, is my needing to package everything up and deliver it (in a
directory structure that fits your site etc.). I can do that, but, again,
it's not likely that I'll do it often. Users wanting the most recent version
will always find it on the wiki.

Why not separate this physical packaging from the logical packaging that is
simply declaring the set of needed files? Why couldn't the physical
packaging for downloading etc. be done automatically by your site (or by
whatever sites make packages available)? You might even consider delaying
the physical packaging until some user actually requests the package (and
then caching it for a given period).

It would be great if an author could simply register a package description
with your site, and have the site automatically periodically retrieve the
requisite files from a specified source (which in my case would be Emacs
Wiki).

If such a system were in place:

. It would require nothing from authors beyond a file specifying the package
structure and component source file locations and a description. (An HTML
form for the metadata could be an alternative to the file.)

. It would require nothing from you beyond periodically (automatically)
retrieving the latest source files from the specified source. You would
control the sync tempo, but authors could also presumably upload an entire
package to you on demand.

. It would require nothing extra from users. At your site they could obtain
info about available packages (descriptions, dependencies, sizes, dates
etc.), and they could download an entire package in compressed form for
their platform (tar, zip, whatever).

. If such a system were in place on the wiki itself, then it would be just
an interface for downloading files that belong together: the latest version
of a package would be assembled and downloaded on the fly when needed.

> Drew> Is there a way for ELPA to get stuff automatically from Emacs
> Drew> Wiki - that is, stuff that has been made Elpable? That would be
> Drew> a big help.
>
> There isn't.  For a multi-file package it would have to be pretty
> complicated.

I take your word for it - I know nothing about these things. It just seems
to me that it should be possible, even if not easy, and it would be a lot
easier for everyone once it were in place.

Single-site easy updating, multiple-site downloading. If someone wrote an
on-the-fly package system, it could be used on multiple sites. They would
each pull packages (their component files) from the wiki (or wherever) when
needed or periodically, compress them, and package them (tar). Because of
multiple-site caching, the wiki would be hit less for downloads. The
download sites would be, well, just download sites - nothing more. The same
software could run on the wiki itself, so it would also be a package
download site.

> Drew> It would be great if library authors could simply upload to one
> Drew> site and have the others feed off of that. (Via RSS? I don't
> Drew> know anything about RSS, so don't laugh if it's irrelevant
> Drew> here.) Emacs Wiki already mirrors the Emacs-Lisp List, but it
> Drew> would, IMO, be much better the other way around: it would be
> Drew> good if other sites that are more like simple repositories fed
> Drew> off of the code posted to the wiki.
>
> Yeah.  For the Wiki the security issue concerns me quite a bit, but
> maybe this could be fixed somehow.  My plan for ELPA was to move to a
> hosting site that would more easily allow multiple maintainers, but I
> haven't done that yet.

Perhaps others here could help address some of the security issues - I don't
know anything about that. I believe that the main problem Emacs Wiki has is
occasional spam, but it doesn't seem to be a great problem. Perhaps Alex
Schroeder (Emacs Wiki) has a suggestion.

> Drew> The wiki has no notion of packages, however, so it would be
> Drew> helpful if a package system such as ELPA could feed off of the
> Drew> wiki somehow.
>
> My experience based on looking at a *lot* of Emacs packages is that
> the solution will never be as convenient for authors as "dump it on
> the wiki".  A nicely (IMO) functioning system will always require
> meta-data; the typical Emacs Lisp program handles this by a comment
> and dropping the problem on the user.

Speaking for myself, I don't mind "registering" metadata with some package
system. I imagine that that data is mainly a list of the component files
(perhaps URLs) and any other dependencies that might exist. Is there more to
it than that?

Maybe I'm missing the boat here. The package system doesn't actually install
packages for the user, does it? It just makes them available for download,
right? And it controls (checks) dependencies and maybe other potential
consistency gotchas, right?

> Drew> See my comments above. In my case, I don't make package
> Drew> "releases"
> [...]
> Drew> I'm not against a package system, depending on what that means,
> Drew> but it would be ideal, I think, if packages were more or less
> Drew> virtual, updated automatically whenever a member file is updated
> Drew> on the wiki.
>
> I see.  ELPA is not a good fit for what you want then.  Most
> non-trivial Emacs packages are developed along more traditional lines,
> with releases and (non-wiki) version control, etc.

I don't see how that approach (generally for large packages, it sounds like)
necessarily conflicts with the low-key, trivial-package approach I use. IOW,
if a project does have well-defined package versions and uses version
control, so much the better for it. But why would not following that
approach for some packages be an obstacle for ELPA? That ELPA needs to
accomodate such needs when they arise I understand, but why would that mean
that ELPA would treat every little package as if it too had such needs and
imposed such constraints? I can see why ELPA allows package versions, for
example, but why would it require them for all packages it handles?

ELPA is also for non-non-trivial packages, no? You even handle single-file
packages, so why would ELPA's package requirements be one-size-fits-all,
modeled on large, multi-developer projects with version control, QA, and the
whole nine yards?

> ELPA was designed to solve a few problems that I encounter:
>
> * Download, install, and compile a package

I figured it was only downloading. How does ELPA help a user install and
compile? Do you provide make files etc. that one can use on various
platforms, or how does it work?

And is that a requirement for ELPA packages, or just a plus? That is, are
all parts of that required, or could someone use ELPA only for downloading
the most recent files in a package (or the files corresponding to some
specific package version, if preferred)?

> * Automatically handle package dependencies

Dependencies between packages or just among the files of a package?

> * Handle upgrading a package

Is it also OK if there is no notion of upgrading, for some package?

> * Handle the case where an external package is incorporated into Emacs
>   core but also has ongoing external development.  This is the
>   requirement behind much of the versioning support -- the idea is to
>   always choose the newer package regardless of where it appears.

I think I see. You mean that I can pull a package from the system and
install it, even if I already have an older (or a newer?) version of the
same package installed. Is that right?

You say that that need motivates much of the ELPA versioning support. But
what if that is not needed for a given package? Why not provide that for
packages that need it, but let other packages be lightweight (no notion of
version) if they don't need it?

Again, as is no doubt obvious, I don't know much about this area. Wrt
Icicles files: if it is easy to do so (just supply a package description and
tar up the files), then I'll submit the files to ELPA. But I probably won't
update the "package" very often, for the reasons I gave (mainly laziness).

You might be right that ELPA is not a good fit for my homebrew "packages". I
still don't know what Nordlow was asking for beyond the ability (already
there on the wiki) of downloading all of the files at once. IOW, if there is
a need for it then I can try ELPA, but I would like to hear what the need
is.

If you find any of the above worth discussing further, we can do so off
list - or perhaps on the wiki. - Drew

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:

Drew> If your concern is also about your site becoming infected, then
Drew> perhaps this could be done on a different site - perhaps the
Drew> wiki itself.

My concern is that, if an automated approach becomes popular, then it
is more worthwhile for people to try to slip in trojans.

Of course there are no guarantees.  But with ELPA I'm usually in touch
with the authors, or I'm uploading an established package.  With the
Wiki there is little oversight.

Even better would be some Debian-like web-of-trust thing, and package
signing.  But, that could come later I suppose, if needed.

Drew> The ELPA version is likely to be stale, due to my laziness.

Then that's another reason that ELPA is not a good match for your
purposes.  I don't think it is too valuable to users to host stale
packages.

Drew> But why the constraint that a package have an overall version number?

Here is a (semi-) real example.  ELPA includes EMMS.  But, I've heard
that someday EMMS will be included in Emacs.  And, I suppose that EMMS
will continue to be developed outside Emacs as well -- this is common
for larger packages (gnus, erc, etc), especially because Emacs has a
very slow release cycle.

Suppose I install EMMS 3.0 now, then sometime later upgrade to Emacs
23 which (let's suppose) includes EMMS 3.1.  How does package.el know
which EMMS to use?

This is the scenario I wanted to solve -- one that has bit me numerous
times over the course of my Emacs life.  Version numbers are a simple,
conventional way to solve it.

To continue the scenario, if EMMS 3.2 comes out, I can upload it to
ELPA and folks can use it; and when they upgrade to Emacs 24 they
won't have to do anything special.

This isn't the only thing ELPA is good for, but it was one of the use
cases I designed for.

Drew> Why not separate this physical packaging from the logical
Drew> packaging that is simply declaring the set of needed files?

I thought about things like this.  But, in the end I decided to try
KISS as much as possible.  And it turns out that most packages fit
nicely into this approach.

Drew> It would be great if an author could simply register a package
Drew> description with your site, and have the site automatically
Drew> periodically retrieve the requisite files from a specified
Drew> source (which in my case would be Emacs Wiki).

This opens the trojan door again.

Also, IME it is rare for packages to be upload-ready.  They typically
don't follow the comment guidelines.  Packages with dependencies on
other packages need ELPA-specific comments as well.

Your ideas are interesting though.  I'm just not sure they would work
that well in practice.  E.g., I've looked through ELL numerous times.
It is, unfortunately, full of broken links.  That is something that is
inevitable with a multi-site approach.  Instead I was trying for
something like "Ohio State Elisp Archive meets CPAN-ish trivial
install".

Drew> Maybe I'm missing the boat here. The package system doesn't
Drew> actually install packages for the user, does it? It just makes
Drew> them available for download, right? And it controls (checks)
Drew> dependencies and maybe other potential consistency gotchas,
Drew> right?

Nope, it does download, dependency resolution (both at activation time
and download time), and install.  And deletion, too, whenever I get
around to it.

FWIW you can try out ELPA in just a couple minutes.  There's an
auto-installer on the web page.  Run that, then:

* M-x package-list-packages
* "r" (re-reads the package list from ELPA)
* "i" to mark for install, "x" to execute the installs

I suggest 'bubbles' as a starter package.  After it is installed,
M-x bubbles.

Drew> I can see why ELPA allows package versions, for example, but why
Drew> would it require them for all packages it handles?

Simplicity.  Honestly I didn't consider that there would be a package
without a version.  Versioning is just, well, conventional and normal
:).

Drew> I figured it was only downloading. How does ELPA help a user install and
Drew> compile? Do you provide make files etc. that one can use on various
Drew> platforms, or how does it work?

ELPA takes a very simple approach.  It downloads the package and
installs it in ~/.emacs.d/elpa/, then byte-compiles the .el files.  It
also extracts autoloads.  If those things don't work, I simply don't
upload the package.

The auto-installer adds a line to your .emacs.  When Emacs starts,
package.el will do a dependency check and activate whatever packages
it can.  (So, e.g., you can change Emacs versions and things mostly
still work.)

Activation means modifying load-path and evaluating the autoloads.  If
the package contains a 'dir' file, package.el also modifies the info
path.

>> * Automatically handle package dependencies

Drew> Dependencies between packages or just among the files of a package?

Between packages.  E.g., lisppaste or weblogger require xml-rpc.  If
you install one of the former, the latter will be installed.  Likewise
for activation.

One other big goal for ELPA was to take the pain out of installing
Emacs Lisp packages -- to focus on the user experience.  We certainly
could write something "dumb" (not meaning that in a bad way) that
simply downloads a .el and drops it somewhere.  In my view this only
solves part of the problem... dependencies matter too.  So do some
things not easily enforceable in a tool: does the package have
autoload comments in place?  Does it respect Emacs namespace rules?
Key-binding rules?  Etc.  I do try to review packages for things like
this before uploading (e.g., that is why quilt.el isn't there).

So, part of the mission here is raising the standards for Emacs
add-ons.

Drew> Is it also OK if there is no notion of upgrading, for some package?

Since version numbers are assumed, all upgrading means is, install a
newer version.


Anyway, I hope I've answered your questions.  I didn't answer them all
since I thought some of the answers would be redundant given other
answers.  If I missed something important I'm happy to talk and talk
about ELPA :-)

Tom

RE: ELPA and EmacsWiki Updates

[Drew Adams]

> Drew> The ELPA version is likely to be stale, due to my laziness.
>
> Then that's another reason that ELPA is not a good match for your
> purposes.  I don't think it is too valuable to users to host stale
> packages.

I agree. On the other hand, some people will hear of ELPA and go to ELPA
looking for packages. They won't necessarily know, for instance, to look for
Icicles on Emacs Wiki. In that case, it could be helpful to make sure that
ELPA has Icicles, even if it is not necessarily the latest and greatest.

You might also consider hosting a list of "packages" or libraries that are
not Elpable, with links to where to obtain them. Yes, those links will
become outdated, but such a wide-ranging list could still help users. They
might go first to ELPA looking for things, and it would then serve also as a
directory of what is available. I'm guessing that you've considered this and
decided against it, but I'll still make the suggestion ;-).

> Drew> But why the constraint that a package have an overall
> version number?
>
> Here is a (semi-) real example.  ELPA includes EMMS.  But, I've heard
> that someday EMMS will be included in Emacs.  And, I suppose that EMMS
> will continue to be developed outside Emacs as well -- this is common
> for larger packages (gnus, erc, etc), especially because Emacs has a
> very slow release cycle.
>
> Suppose I install EMMS 3.0 now, then sometime later upgrade to Emacs
> 23 which (let's suppose) includes EMMS 3.1.  How does package.el know
> which EMMS to use?
>
> This is the scenario I wanted to solve -- one that has bit me numerous
> times over the course of my Emacs life.  Version numbers are a simple,
> conventional way to solve it.
>
> To continue the scenario, if EMMS 3.2 comes out, I can upload it to
> ELPA and folks can use it; and when they upgrade to Emacs 24 they
> won't have to do anything special.
>
> This isn't the only thing ELPA is good for, but it was one of the use
> cases I designed for.

That's clearly expressed, and I understand all that. My question is why
impose version numbers on _all_ "packages" just to make them available at
ELPA. IOW, the need for package versions for some packages does not imply
such a need for all packages.

You have a high standard for packages. You might consider also having a
lower standard with correspondingly less ELPA support, thus allowing lighter
weight packages to benefit from some of what ELPA offers. I'm thinking of
both users and authors here - not every library needs to be a buttoned-down
package at the level you've defined it, IMO. What's appropriate for a large,
communal project is not necessarily also appropriate for a simple but
perhaps multi-file library.

> Drew> Why not separate this physical packaging from the logical
> Drew> packaging that is simply declaring the set of needed files?
>
> I thought about things like this.  But, in the end I decided to try
> KISS as much as possible.  And it turns out that most packages fit
> nicely into this approach.

I understand. I have no idea how much work would be involved in an approach
such as I suggested. It's just something that seems good to me from the
point of view of users and authors - for at least some (e.g. simple)
packages.

> Drew> It would be great if an author could simply register a package
> Drew> description with your site, and have the site automatically
> Drew> periodically retrieve the requisite files from a specified
> Drew> source (which in my case would be Emacs Wiki).
>
> This opens the trojan door again.
>
> Also, IME it is rare for packages to be upload-ready.  They typically
> don't follow the comment guidelines.  Packages with dependencies on
> other packages need ELPA-specific comments as well.

Again, I have nothing against adding "package-level" comments that are
specific for ELPA.

It would be preferable, of course, to do that in a form that could be used
by multiple package systems, but perhaps it's not yet possible to
standardize that. I personally don't want to get into one format for ELPA,
another for GNU Emacs, another for ELL, and so on. But that's life, I guess.

> Your ideas are interesting though.  I'm just not sure they would work
> that well in practice.  E.g., I've looked through ELL numerous times.
> It is, unfortunately, full of broken links.  That is something that is
> inevitable with a multi-site approach.  Instead I was trying for
> something like "Ohio State Elisp Archive meets CPAN-ish trivial
> install".

I too used to use OSELA, until it fell off the edge of the Earth.

> Drew> Maybe I'm missing the boat here. The package system doesn't
> Drew> actually install packages for the user, does it? It just makes
> Drew> them available for download, right? And it controls (checks)
> Drew> dependencies and maybe other potential consistency gotchas,
> Drew> right?
>
> Nope, it does download, dependency resolution (both at activation time
> and download time), and install.  And deletion, too, whenever I get
> around to it.
>
> FWIW you can try out ELPA in just a couple minutes.  There's an
> auto-installer on the web page.  Run that, then:
>
> * M-x package-list-packages
> * "r" (re-reads the package list from ELPA)
> * "i" to mark for install, "x" to execute the installs
>
> I suggest 'bubbles' as a starter package.  After it is installed,
> M-x bubbles.

OK, will do.

> Drew> I can see why ELPA allows package versions, for example, but why
> Drew> would it require them for all packages it handles?
>
> Simplicity.  Honestly I didn't consider that there would be a package
> without a version.  Versioning is just, well, conventional and normal
> :).

So call me unconventional and abnormal ;-).

> Drew> I figured it was only downloading. How does ELPA help a
> user install and
> Drew> compile? Do you provide make files etc. that one can use on various
> Drew> platforms, or how does it work?
>
> ELPA takes a very simple approach.  It downloads the package and
> installs it in ~/.emacs.d/elpa/, then byte-compiles the .el files.  It
> also extracts autoloads.  If those things don't work, I simply don't
> upload the package.
>
> The auto-installer adds a line to your .emacs.  When Emacs starts,
> package.el will do a dependency check and activate whatever packages
> it can.  (So, e.g., you can change Emacs versions and things mostly
> still work.)

Ouch! I don't want to tell you what to do or not do, but I don't much like
the idea of something (even something I initiate) mucking with my .emacs.
Anyway, that's another story, and I'm sure you have good reasons for doing
it the way you do.

> Activation means modifying load-path and evaluating the autoloads.  If
> the package contains a 'dir' file, package.el also modifies the info
> path.

Ouch again!

I sure hope there is a foolproof way to uninstall - a way that backs out all
of the automatic changes that you make to a user's environment. Been bit by
this kind of "helper" tool too much to welcome it blindly.

> >> * Automatically handle package dependencies
>
> Drew> Dependencies between packages or just among the files of a package?
>
> Between packages.  E.g., lisppaste or weblogger require xml-rpc.  If
> you install one of the former, the latter will be installed.  Likewise
> for activation.
>
> One other big goal for ELPA was to take the pain out of installing
> Emacs Lisp packages -- to focus on the user experience.  We certainly
> could write something "dumb" (not meaning that in a bad way) that
> simply downloads a .el and drops it somewhere.  In my view this only
> solves part of the problem... dependencies matter too.  So do some
> things not easily enforceable in a tool: does the package have
> autoload comments in place?  Does it respect Emacs namespace rules?
> Key-binding rules?  Etc.  I do try to review packages for things like
> this before uploading (e.g., that is why quilt.el isn't there).
>
> So, part of the mission here is raising the standards for Emacs
> add-ons.

Please consider not necessarily raising the standards for all Emacs add-ons
to the same level. Not every contribution needs to be at your level of
"packageness", IMO.

> Drew> Is it also OK if there is no notion of upgrading, for some package?
>
> Since version numbers are assumed, all upgrading means is, install a
> newer version.

Right. I meant is it OK if there is no notion of version number. The answer
seems to be no.

> Anyway, I hope I've answered your questions.  I didn't answer them all
> since I thought some of the answers would be redundant given other
> answers.  If I missed something important I'm happy to talk and talk
> about ELPA :-)

Yes, thanks.

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:

Drew> You might also consider hosting a list of "packages" or
Drew> libraries that are not Elpable, with links to where to obtain
Drew> them.
[...]
Drew> I'm guessing that you've considered this and decided against it,
Drew> but I'll still make the suggestion ;-).

:-).  But yeah, we've got the wiki and ELL for this.  Those do a
pretty good job.

Drew> That's clearly expressed, and I understand all that. My question
Drew> is why impose version numbers on _all_ "packages" just to make
Drew> them available at ELPA. IOW, the need for package versions for
Drew> some packages does not imply such a need for all packages.

Yeah, I suppose that's true.  I will think about it some more.

Drew> Again, I have nothing against adding "package-level" comments
Drew> that are specific for ELPA.

Drew> It would be preferable, of course, to do that in a form that
Drew> could be used by multiple package systems, but perhaps it's not
Drew> yet possible to standardize that.

ELPA uses a few things already defined by the Emacs Lisp reference
guide -- the Version header (though ELPA needs a specific form for the
value) and the start- and end-comments.  The start comment is used to
extract the package description.

ELPA also defines a couple new headers: Package-Version (in case
Version has the wrong format) and Package-Requires.  The latter is
used to express dependencies.

>> The auto-installer adds a line to your .emacs.  When Emacs starts,
>> package.el will do a dependency check and activate whatever packages
>> it can.  (So, e.g., you can change Emacs versions and things mostly
>> still work.)

Drew> Ouch! I don't want to tell you what to do or not do, but I don't
Drew> much like the idea of something (even something I initiate)
Drew> mucking with my .emacs.  Anyway, that's another story, and I'm
Drew> sure you have good reasons for doing it the way you do.

Yeah.  The goal of the auto-installer was to make it dead simple to
start using ELPA.  It is easy not to do this -- just don't run it :-).
You can still use ELPA if you install package.el by hand and edit your
.emacs, or whatever, yourself.

Basically I thought it would be fun and worthwhile to make it so
people could be running package.el in a few seconds without needing to
know anything... since after all one of my goals for ELPA is to make
it possible for folks who aren't emacs lisp wizards to run stuff --
not to mention removing the drudgery for those of us who can do it,
but would just rather not :-)

>> Activation means modifying load-path and evaluating the autoloads.  If
>> the package contains a 'dir' file, package.el also modifies the info
>> path.

Drew> Ouch again!

Drew> I sure hope there is a foolproof way to uninstall - a way that
Drew> backs out all of the automatic changes that you make to a user's
Drew> environment. Been bit by this kind of "helper" tool too much to
Drew> welcome it blindly.

The load-path stuff is all done dynamically, when package-initialize
is called at startup.  To remove a package you can just "rm -rf" the
directory in ~/.emacs.d/elpa/.

Eventually I'll write package deletion support for the package menu
mode.  I've put it off.

Unloading something after Emacs has started and package.el has
initialized can't really work.  Emacs just isn't good at this kind of
thing.  There's unload-feature these days, but even that won't undo
the additions of autoloads and stuff.  So, if you want to delete
something you have to restart Emacs.  That sucks from a UI point of
view.

I keep going back and forth on the question of package activation.
Right now, if a package is installed, package.el tries to activate it
at startup.  But, we could store a preference so you can choose not to
have some activated.

One way this would be nice is if package.el was really widely used,
including to install 3rd party elisp in a distro.  Distros, in my
view, have a problem where for most users they want to enable
autoloads for all additional packages, but some power users want to
turn off certain things.  In this situation package.el would allow
deactivating the uninteresting things.

Anyway I'm not sure whether this is worth implementing.  On the minus
side it adds more complexity and UI.

Tom

RE: ELPA and EmacsWiki Updates

[Drew Adams]

> >> The auto-installer adds a line to your .emacs.  When Emacs starts,
> >> package.el will do a dependency check and activate whatever packages
> >> it can.  (So, e.g., you can change Emacs versions and things mostly
> >> still work.)
>
> Drew> Ouch! I don't want to tell you what to do or not do, but I don't
> Drew> much like the idea of something (even something I initiate)
> Drew> mucking with my .emacs.  Anyway, that's another story, and I'm
> Drew> sure you have good reasons for doing it the way you do.
>
> Yeah.  The goal of the auto-installer was to make it dead simple to
> start using ELPA.  It is easy not to do this -- just don't run it :-).
> You can still use ELPA if you install package.el by hand and edit your
> .emacs, or whatever, yourself.

But that means that a user must know how to do that manual tinkering. A
novice will just choose the automatic behavior that does lots of stuff
invisibly. That's helpful (s?he normally won't care about those details,
presumably), but what to do when something goes wrong (of different from
what was expected)? The problem is that lack of knowledge can make things
easier when things are going as expected, and it can make things much harder
when they are not.

It's a classic help-the-user bind. To really help the user, it is important
to go beyond a monolithic, totally automatic, push-button approach. To
really help the user, tools need to also deal with trouble-shooting,
preferences, and so on. Too often, there are only two widely separated
levels (choices): (1) totally push-button: you push the button and
relinquish all control from then on, or (2) you are completely on your own,
from scratch. For #2, there is too often little doc etc., and the attitude
of developers is that if you choose #2, then you should be able to figure
things out on your own. What's really needed much of the time is a way to
move smoothly along the continuum between #1 and #2.

Emacs, in general, is helpful when something goes wrong - you can easily and
interactively poke around to understand and learn about what's happening.
This is mainly because much of it is implemented in Lisp and because of the
integrated help system.

> Basically I thought it would be fun and worthwhile to make it so
> people could be running package.el in a few seconds without needing to
> know anything... since after all one of my goals for ELPA is to make
> it possible for folks who aren't emacs lisp wizards to run stuff --
> not to mention removing the drudgery for those of us who can do it,
> but would just rather not :-)

OK, but do you need to modify a user's .emacs to do that? Couldn't you use
Customize? That would limit any modification of a user's .emacs to the
standard, easily recognized, Customize section. And if the user has a
`custom-file', then it would appear there instead. (FWIW, I always recommend
that users have a `custom-file', and keep their .emacs for their own code.)

IOW, can't you save the data you need without changing a user's .emacs in a
non-standard way?

> >> Activation means modifying load-path and evaluating the autoloads.  If
> >> the package contains a 'dir' file, package.el also modifies the info
> >> path.
>
> Drew> Ouch again!
>
> Drew> I sure hope there is a foolproof way to uninstall - a way that
> Drew> backs out all of the automatic changes that you make to a user's
> Drew> environment. Been bit by this kind of "helper" tool too much to
> Drew> welcome it blindly.
>
> The load-path stuff is all done dynamically, when package-initialize
> is called at startup.  To remove a package you can just "rm -rf" the
> directory in ~/.emacs.d/elpa/.

And edit your .emacs to remove the changes package.el made to it, including
to the load-path? Anything else? ;-)

> Eventually I'll write package deletion support for the package menu
> mode.  I've put it off.

I'd think that would be the first thing to write ;-). Making it easy for
users to install some new software is great, but it is even more important
to make it easy for them to get back to exactly the way things were before
they tried to install it. This is a lesson that Windows users learned on Day
2 - the hard way, unfortunately.

I don't know. I'm all in favor of making things easy for users. But in
practice that too often means imposing a bunch of stuff that a user might
not really want (if fully aware) and that s?he finds it difficult to get rid
off or bypass thereafter. (I'm not saying that this is the case for
package.el.) I'm in favor of conscious opt-in as opposed to opt-out, and
when I uninstall something I want it to be totally gone - (1) no vestigial
files, directories, registry entries (Windows), or text added to any of my
files, and (2) everything that was removed to install or run the program is
put back in place.

Again, I don't say there is any particular problem with package.el - I'm
speaking generally. I'm leary of the push-button getting-started approach.
It makes things easy, but too often installing is easy and removing is not
so easy.

There's a tradeoff when a program does things on behalf of a user; it's
important to somehow let users still control things if they want to, in
particular, to let them easily opt out of some of the automatically imposed
behavior. And to let them (and help them) regain control later if they
initially chose the automatic route. The tradeoff is that making things easy
often means making details invisible - doing things behind the user's back.
That's convenient, but it can have its disadvantages, as we all know.

Here's the thing: As long as an Emacs tool or library does _not_ hold you by
the hand, it can expect you to be ready and able to tackle Emacs Lisp to
some extent. If, however, it takes you by the hand initially and purports to
do some heavy lifting on your behalf to get you started, it should also
accept the responsibility of staying by your side throughout the journey.
IOW, if a tool gives you a push-button to get started, it should also
provide help to you beyond that point. Anything less is unfair and
unhelpful.

> Unloading something after Emacs has started and package.el has
> initialized can't really work.  Emacs just isn't good at this kind of
> thing.  There's unload-feature these days, but even that won't undo
> the additions of autoloads and stuff.  So, if you want to delete
> something you have to restart Emacs.  That sucks from a UI point of
> view.

It's not a big problem for a user to restart Emacs. If that takes care of
removing everything that was added and reverts all changes that installing
effected, then there is no problem.

> I keep going back and forth on the question of package activation.
> Right now, if a package is installed, package.el tries to activate it
> at startup.  But, we could store a preference so you can choose not to
> have some activated.

I have lots of Emacs stuff installed (available) that I don't start up
automatically. I have lots of stuff that I don't even autoload - I just want
it to be there (in my load-path) so I can use it when I want.

I would think that installing and activating (depending on what is meant by
the latter) would be two different choices for a user to make. Why couple
them?

> One way this would be nice is if package.el was really widely used,
> including to install 3rd party elisp in a distro.  Distros, in my
> view, have a problem where for most users they want to enable
> autoloads for all additional packages, but some power users want to
> turn off certain things.  In this situation package.el would allow
> deactivating the uninteresting things.

Yes, it would be good if package.el let you select stuff that you wanted to
activate.

Similarly, I think that a simple GUI for managing one's load-path might be
useful. I think that Emacs users need to be aware of their load-path. I
don't think it can be hidden from them without courting danger. But a good
interface for viewing and managing the load-path and library dependencies
could perhaps be developed.

IIUC, package.el helps you manage package conflicts - for example, multiple
versions of the same package. That feature could perhaps be integrated with
a general load-path browser that would let you query, auto-detect, and
manage other possible conflicts (whether from "packages" or anything else).

> Anyway I'm not sure whether this is worth implementing.  On the minus
> side it adds more complexity and UI.

One Tromey can only do so much! We are all grateful for your contributions.
Even though package.el might not be a good fit for some of my own code, as
you pointed out, I recognize that it can be helpful.

Sorry for the generalizations, most of which probably aren't relevant to
package.el. Something must have pushed my button ;-).

Re: ELPA and EmacsWiki Updates

[Tom Tromey]

>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:

Drew> What's really needed much of the time is a way to move smoothly
Drew> along the continuum between #1 and #2.

It depends.  Not everybody wants to learn how to hack elisp, or has
the time, or whatever.  But this doesn't mean they should not use
Emacs or install an external package or whatever... sure, we know it
was better for us to learn all this stuff, but we probably live in
Emacs (I do) -- but there are plenty of casual users as well.

Anyway package.el can't disguise the nature of the Emacs on which it
rests.  If you hit a bug, elisp hacking skills are needed, no question
there.

Drew> OK, but do you need to modify a user's .emacs to do that?
Drew> Couldn't you use Customize?

I don't see how it could use customize.  It needs to run code at
startup.  Is that possible with customize?

Drew> And edit your .emacs to remove the changes package.el made to
Drew> it, including to the load-path? Anything else? ;-)

load-path is hacked at runtime, during package activation.  The result
isn't written to any file.

I thought you were asking about deleting a single package.  If you
want to really uninstall package.el, then you must remove the bits
from your .emacs, and then "rm -rf ~/.emacs.d/elpa/".

Nothing else is modified... well, unless you customize some variable
for use by a package you just deleted.  I don't know how to handle
that case.

>> Eventually I'll write package deletion support for the package menu
>> mode.  I've put it off.

Drew> I'd think that would be the first thing to write ;-).

*Shrug*.  There's a reason package.el is only version 0.5.

Drew> I don't know. I'm all in favor of making things easy for
Drew> users. But in practice that too often means imposing a bunch of
Drew> stuff that a user might not really want (if fully aware) and
Drew> that s?he finds it difficult to get rid off or bypass
Drew> thereafter.

In my view, package.el is just doing what an experienced Emacs user
would do -- only, better than most folks bother, and regularized.  The
way this scratches my own itch -- since I'm not really a beginning
Emacs user -- is that it eliminates the drudgery of package install.

Of course there are always folks who want complete control, who (e.g.)
detest the file name ~/.emacs.d/elpa/, or whatever.  Not to be too
harsh or anything, but they should probably roll their own.  We'll all
be happier that way :-)

Drew> I would think that installing and activating (depending on what
Drew> is meant by the latter) would be two different choices for a
Drew> user to make. Why couple them?

Convenience.  Also, my thought experiment is: what external package is
there that is useful enough for me to install but which, if it were
incorporated into Emacs, should not be autoloaded?

Tom

RE: ELPA and EmacsWiki Updates

[Drew Adams]

> Drew> What's really needed much of the time is a way to move smoothly
> Drew> along the continuum between #1 and #2.
>
> It depends.  Not everybody wants to learn how to hack elisp, or has
> the time, or whatever.  But this doesn't mean they should not use
> Emacs or install an external package or whatever... sure, we know it
> was better for us to learn all this stuff, but we probably live in
> Emacs (I do) -- but there are plenty of casual users as well.

Casual users are the ones I'm worried about here. #1 gives them the
impression that they can count on help when they get into trouble. "Would
that it were..."

> Anyway package.el can't disguise the nature of the Emacs on which it
> rests.  If you hit a bug, elisp hacking skills are needed, no question
> there.
>
> Drew> OK, but do you need to modify a user's .emacs to do that?
> Drew> Couldn't you use Customize?
>
> I don't see how it could use customize.  It needs to run code at
> startup.  Is that possible with customize?

When you save customizations, they are saved in your `custom-file', if that
variable is defined. If not, they are saved in your .emacs. At startup, both
.emacs and `custom-file' are loaded. So yes, customizations are read at
startup.

> Drew> And edit your .emacs to remove the changes package.el made to
> Drew> it, including to the load-path? Anything else? ;-)
>
> load-path is hacked at runtime, during package activation.  The result
> isn't written to any file.

OK, I misunderstood that.

> I thought you were asking about deleting a single package.  If you
> want to really uninstall package.el, then you must remove the bits
> from your .emacs, and then "rm -rf ~/.emacs.d/elpa/".

That's what will bite a casual user. It's error prone, and it might require
some knowledge of Emacs Lisp or OS commands. IMO, it should be just as easy
to uninstall something as to install it. No - it should be easier. This is
not as important for experienced users as it is for novices, but it helps
everyone.

> Nothing else is modified... well, unless you customize some variable
> for use by a package you just deleted.  I don't know how to handle
> that case.
>
> >> Eventually I'll write package deletion support for the package menu
> >> mode.  I've put it off.
>
> Drew> I'd think that would be the first thing to write ;-).
>
> *Shrug*.  There's a reason package.el is only version 0.5.

;-)

> Drew> I don't know. I'm all in favor of making things easy for
> Drew> users. But in practice that too often means imposing a bunch of
> Drew> stuff that a user might not really want (if fully aware) and
> Drew> that s?he finds it difficult to get rid off or bypass
> Drew> thereafter.
>
> In my view, package.el is just doing what an experienced Emacs user
> would do -- only, better than most folks bother, and regularized.  The
> way this scratches my own itch -- since I'm not really a beginning
> Emacs user -- is that it eliminates the drudgery of package install.

And that is a good thing.

> Of course there are always folks who want complete control, who (e.g.)
> detest the file name ~/.emacs.d/elpa/, or whatever.  Not to be too
> harsh or anything, but they should probably roll their own.  We'll all
> be happier that way :-)

Agreed.

> Drew> I would think that installing and activating (depending on what
> Drew> is meant by the latter) would be two different choices for a
> Drew> user to make. Why couple them?
>
> Convenience.

That argues for being _able_ to do them at the same time, but not for being
_obliged_ to do them together.

> Also, my thought experiment is: what external package is
> there that is useful enough for me to install but which, if it were
> incorporated into Emacs, should not be autoloaded?

You might want to try a package, without committing yourself to integrating
it with your regular Emacs startup. It's not so much a question about "what
package" but a question about what a user might want to do in different
contexts. I have some libraries that I use all of the time, and others that
I use only occasionally, on demand.