RE: ELPA and EmacsWiki Updates

["Drew Adams" <drew.adams@oracle.com>]

Hi Tom,

> Drew> However, it's also true that I said that (1) I appreciate the
> Drew> simplicity of uploading to Emacs Wiki
>
> Yeah.  That is simpler, for sure.  I considered having ELPA download
> straight from the wiki, but I was not comfortable with the
> uncontrolled aspect of it.  My concern was that if ELPA became
> commonplace, someone would upload a trojan.

I still think it might be worth pursuing. I don't see why the risks would be
greater than what is already true for the wiki itself.

If your concern is also about your site becoming infected, then perhaps this
could be done on a different site - perhaps the wiki itself.

> For a multi-file package, you must:

<snip>

OK, thanks; I will think about it. I certainly have no problem with defining
an overall package-description file ("-pkg.el") that says what files are
included or whatever other such info is needed.

I don't use (or like much) a notion of a package "version" or "release" for
my little offerings. I prefer to treat changes ("releases", "revisions" or
whatever) at the level of individual files - each file in a package has its
own update (version) number. A file's update number is incremented
automatically in the file header each time I save the file - there's nothing
to do.

I think that such file-level version numbers can be more flexible for both
authors and users, at least for single-author projects. In any case, it's
what I prefer. So, if I were to upload a package to your site, it's unlikely
that I would often declare a new release and update the package as a whole.
The ELPA version is likely to be stale, due to my laziness.

That you need some overall description of a package makes sense. That that
description includes a list of files or other dependencies and a verbal
explanation (how to install etc.) also makes sense. This is what your
"package" file is, I guess.

But why the constraint that a package have an overall version number?
Unless, for instance, you offer multiple versions of the same package. In my
case, I would like the package file to simply point to the requisite files
in the package, and that's it. The same package file would then serve for
any component file versions. The package file would only need to be changed
if the set of component files changed.

The second thing that I'm not crazy about, besides the notion of package
version, is my needing to package everything up and deliver it (in a
directory structure that fits your site etc.). I can do that, but, again,
it's not likely that I'll do it often. Users wanting the most recent version
will always find it on the wiki.

Why not separate this physical packaging from the logical packaging that is
simply declaring the set of needed files? Why couldn't the physical
packaging for downloading etc. be done automatically by your site (or by
whatever sites make packages available)? You might even consider delaying
the physical packaging until some user actually requests the package (and
then caching it for a given period).

It would be great if an author could simply register a package description
with your site, and have the site automatically periodically retrieve the
requisite files from a specified source (which in my case would be Emacs
Wiki).

If such a system were in place:

. It would require nothing from authors beyond a file specifying the package
structure and component source file locations and a description. (An HTML
form for the metadata could be an alternative to the file.)

. It would require nothing from you beyond periodically (automatically)
retrieving the latest source files from the specified source. You would
control the sync tempo, but authors could also presumably upload an entire
package to you on demand.

. It would require nothing extra from users. At your site they could obtain
info about available packages (descriptions, dependencies, sizes, dates
etc.), and they could download an entire package in compressed form for
their platform (tar, zip, whatever).

. If such a system were in place on the wiki itself, then it would be just
an interface for downloading files that belong together: the latest version
of a package would be assembled and downloaded on the fly when needed.

> Drew> Is there a way for ELPA to get stuff automatically from Emacs
> Drew> Wiki - that is, stuff that has been made Elpable? That would be
> Drew> a big help.
>
> There isn't.  For a multi-file package it would have to be pretty
> complicated.

I take your word for it - I know nothing about these things. It just seems
to me that it should be possible, even if not easy, and it would be a lot
easier for everyone once it were in place.

Single-site easy updating, multiple-site downloading. If someone wrote an
on-the-fly package system, it could be used on multiple sites. They would
each pull packages (their component files) from the wiki (or wherever) when
needed or periodically, compress them, and package them (tar). Because of
multiple-site caching, the wiki would be hit less for downloads. The
download sites would be, well, just download sites - nothing more. The same
software could run on the wiki itself, so it would also be a package
download site.

> Drew> It would be great if library authors could simply upload to one
> Drew> site and have the others feed off of that. (Via RSS? I don't
> Drew> know anything about RSS, so don't laugh if it's irrelevant
> Drew> here.) Emacs Wiki already mirrors the Emacs-Lisp List, but it
> Drew> would, IMO, be much better the other way around: it would be
> Drew> good if other sites that are more like simple repositories fed
> Drew> off of the code posted to the wiki.
>
> Yeah.  For the Wiki the security issue concerns me quite a bit, but
> maybe this could be fixed somehow.  My plan for ELPA was to move to a
> hosting site that would more easily allow multiple maintainers, but I
> haven't done that yet.

Perhaps others here could help address some of the security issues - I don't
know anything about that. I believe that the main problem Emacs Wiki has is
occasional spam, but it doesn't seem to be a great problem. Perhaps Alex
Schroeder (Emacs Wiki) has a suggestion.

> Drew> The wiki has no notion of packages, however, so it would be
> Drew> helpful if a package system such as ELPA could feed off of the
> Drew> wiki somehow.
>
> My experience based on looking at a *lot* of Emacs packages is that
> the solution will never be as convenient for authors as "dump it on
> the wiki".  A nicely (IMO) functioning system will always require
> meta-data; the typical Emacs Lisp program handles this by a comment
> and dropping the problem on the user.

Speaking for myself, I don't mind "registering" metadata with some package
system. I imagine that that data is mainly a list of the component files
(perhaps URLs) and any other dependencies that might exist. Is there more to
it than that?

Maybe I'm missing the boat here. The package system doesn't actually install
packages for the user, does it? It just makes them available for download,
right? And it controls (checks) dependencies and maybe other potential
consistency gotchas, right?

> Drew> See my comments above. In my case, I don't make package
> Drew> "releases"
> [...]
> Drew> I'm not against a package system, depending on what that means,
> Drew> but it would be ideal, I think, if packages were more or less
> Drew> virtual, updated automatically whenever a member file is updated
> Drew> on the wiki.
>
> I see.  ELPA is not a good fit for what you want then.  Most
> non-trivial Emacs packages are developed along more traditional lines,
> with releases and (non-wiki) version control, etc.

I don't see how that approach (generally for large packages, it sounds like)
necessarily conflicts with the low-key, trivial-package approach I use. IOW,
if a project does have well-defined package versions and uses version
control, so much the better for it. But why would not following that
approach for some packages be an obstacle for ELPA? That ELPA needs to
accomodate such needs when they arise I understand, but why would that mean
that ELPA would treat every little package as if it too had such needs and
imposed such constraints? I can see why ELPA allows package versions, for
example, but why would it require them for all packages it handles?

ELPA is also for non-non-trivial packages, no? You even handle single-file
packages, so why would ELPA's package requirements be one-size-fits-all,
modeled on large, multi-developer projects with version control, QA, and the
whole nine yards?

> ELPA was designed to solve a few problems that I encounter:
>
> * Download, install, and compile a package

I figured it was only downloading. How does ELPA help a user install and
compile? Do you provide make files etc. that one can use on various
platforms, or how does it work?

And is that a requirement for ELPA packages, or just a plus? That is, are
all parts of that required, or could someone use ELPA only for downloading
the most recent files in a package (or the files corresponding to some
specific package version, if preferred)?

> * Automatically handle package dependencies

Dependencies between packages or just among the files of a package?

> * Handle upgrading a package

Is it also OK if there is no notion of upgrading, for some package?

> * Handle the case where an external package is incorporated into Emacs
>   core but also has ongoing external development.  This is the
>   requirement behind much of the versioning support -- the idea is to
>   always choose the newer package regardless of where it appears.

I think I see. You mean that I can pull a package from the system and
install it, even if I already have an older (or a newer?) version of the
same package installed. Is that right?

You say that that need motivates much of the ELPA versioning support. But
what if that is not needed for a given package? Why not provide that for
packages that need it, but let other packages be lightweight (no notion of
version) if they don't need it?

Again, as is no doubt obvious, I don't know much about this area. Wrt
Icicles files: if it is easy to do so (just supply a package description and
tar up the files), then I'll submit the files to ELPA. But I probably won't
update the "package" very often, for the reasons I gave (mainly laziness).

You might be right that ELPA is not a good fit for my homebrew "packages". I
still don't know what Nordlow was asking for beyond the ability (already
there on the wiki) of downloading all of the files at once. IOW, if there is
a need for it then I can try ELPA, but I would like to hear what the need
is.

If you find any of the above worth discussing further, we can do so off
list - or perhaps on the wiki. - Drew