Re: Interoperation between package managers

[Radon Rosborough <radon.neon@gmail.com>]

> I'd probably want to also unify the two files into one (which would
> likely hold the concatenation of <pkg>-pkg.el and
> <pkg>-autoloads.el).

I'm a little wary of this idea. It seems to me like <pkg>-autoloads.el
is purely an implementation detail (or "build artifact") of the
package manager, whereas <pkg>-pkg.el contains author-maintained
metadata that should be checked into version control in the upstream
repository.

To give some perspective on why this might be important, consider how
source-based package managers work:

* clone upstream repository
* inspect <pkg>-pkg.el (or in-file headers) to determine dependencies,
  etc.
* generate autoloads and byte-compile on the client-side

In this workflow it makes no sense to combine the package metadata
with the autoloads.

Now, straight.el doesn't make any demands on how the upstream formats
things, which I consider one of its strengths. So as long as the
metadata is available in some kind of standard format, I don't think
there will be a problem. Most likely the situation will be similar for
other source-based package managers.

> I think it's very important to be able to have several versions of a
> given package installed at the same time.

This is another concept which makes a lot of sense in the
tarball-from-ELPA workflow, but not much sense in the source-based
workflow (where you don't need to save old versions because you can
revert to any version at any time).

I don't know what the best solution is for package.el to accommodate
both of these workflows (presuming that this is desired).

> > I think the biggest problem is the documentation.
>
> To me, all of that is pretty "obvious" because I've spent enough
> time both in the design and in the code, which makes it difficult to
> figure out what people might need to know.

Yes, that's the problem. I wrote a package manager too, and I have
exactly the same experience of everything seeming obvious to me. I
think the best solution is to just sit down and explain everything.
Have you seen the README for straight.el [1]? That is the kind of
documentation that I would like to see from every library I use. (I
know it's long; splitting into multiple documents is on my to-do
list.)

> Specific requests (especially patches) are very welcome here.

Unfortunately, I can't contribute patches since I am already too busy
maintaining my own package manager. However, I can give some
suggestions for things that need to be explained.

* What is the format of a package on the server? What is the format on
  the client? Be specific.
* What does package.el do to a package? I know that it generates
  autoloads and byte-compiles, but most people don't know that.
* What does activating a package mean? Again, I know that it entails
  evaluating the autoloads file and adding a directory to the load
  path, but most people don't.
* Why is package-initialize so slow? What are the bottlenecks when you
  activate lots of packages?
* What exactly is package-refresh-contents doing? Where is it saving
  things? What are the error conditions if you forget to run it? What
  commands is it automatically run by?
* Why does package-initialize keep getting inserted into my init-file?
  What's up with package-selected-packages?
* How am I supposed to load a single package and its dependencies for
  a bug report, without loading everything else?
* What happens when I need to run a package directly from its Git
  repository? How do I make local changes to a package installed from
  package.el? How do I contribute changes upstream?
* How do I sync my configuration between machines? How can I make sure
  that package.el creates a reproducible configuration that won't
  break if I move my configuration to a new machine?
* What's this business with package-install-file? How does it differ
  from other ways of loading a file into my Emacs configuration?
* How would I host my own ELPA server? Would I want to do such a
  thing? How do I deal with needing to install packages that aren't
  available in the default archives?
* How do I install old versions of packages? Is this even possible?
* What's the deal with built-in packages? What does that mean? Are all
  the lisp/ files built-in packages? None of them? An arbitrary
  subset?
* How do I install a package if there's also a built-in version of it?
* What are the known limitations of package.el? Clearly there are a
  lot of them, judging by the number of FIXME notices in the code.
* What are the interactions between installing different versions of
  packages with different dependencies from different archives? What
  rules are used to resolve all of this?
* What are the various states the package management system can be in?
  How does calling package-activate and/or package-initialize in
  various places during init or later behave, especially if the
  backing data on disk has changed in the interim?
* How strict are the version requirement rules? Are they always
  guaranteed to find a working set of versions if such a set exists?
  (No, as I found out while looking at FIXME notices in the source
  code.)

This is just a few simple things I thought of. I'm sure if I had a
better understanding of package.el, I would have more questions, and
more specific ones. Especially given all the features you told me
about in the last few emails, which I had no idea even existed, and
therefore didn't even consider asking questions about.

Maybe package.el is simple, but unless there's a clear statement that
"this documentation covers exactly what package.el does", there's
always the concern that more magic is going on, given how opaque the
source code is.

> What (w|c)ould be such a "more comprehensible source of
> information"?

The problem is that no such source exists at the moment. We need to
create one. What I mean by "comprehensible" is that any random Emacs
user can visit a nicely formatted modern web page and understand in
just a few minutes:

* what package.el does
* where packages live (on the server, in ~/.emacs.d, etc.)
* what package-initialize does
* where they should put code to configure packages
* commonly-asked questions and troubleshooting information

Then after the short summary, there should be comprehensive
documentation on everything. Again, see my documentation [1] which I
think is a pretty comprehensive description of both the concepts and
usage of straight.el.

I want my source code to be as readable as prose, but nevertheless try
to write my documentation to be such that my users never *have* to
look at the source code. I don't think it would be a bad thing if
package.el did the same.

Best,
Radon

[1]: https://github.com/raxod502/straight.el