XEmacs package code

XEmacs package code

[Michael Sperber [Mr. Preprocessor]]

I should note that I am not the author of much of the package code per
se; Steve Baur is.  I have written all of the code dealing with path
management.

More importantly, the current design of the package system is
generally perceived as a failure in the XEmacs community, even by
Steve.  (I'll be happy to explain why if anyone's interested.)  Thus,
it may not be desirable to adapt this code for Emacs.

I have designed a new one, but have not been able to spend much time
on implementing it.  Once I do, I'll try my best to make the code
modular and sharable between the Emacsen.

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: XEmacs package code

[Miles Bader]

sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.  Preprocessor]) writes:
> the current design of the package system is generally perceived as a
> failure in the XEmacs community, even by Steve.  (I'll be happy to
> explain why if anyone's interested.)

Please do...

> I have designed a new one, but have not been able to spend much time
> on implementing it.  Once I do, I'll try my best to make the code
> modular and sharable between the Emacsen.

Do you have documents or notes about the new design too?

Thanks,

-Miles

-- 
I'd rather be consing.

Re: XEmacs package code

[Michael Sperber [Mr. Preprocessor]]

>>>>> "Miles" == Miles Bader <miles@gnu.org> writes:

Miles> sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.  Preprocessor]) writes:
>> the current design of the package system is generally perceived as a
>> failure in the XEmacs community, even by Steve.  (I'll be happy to
>> explain why if anyone's interested.)

Miles> Please do...

The main problem is that our package system identifies each package
only by its name, and this is the only identifier you get for
resolving dependencies.  However, packages change over time, and with
change, incompatibilities arise, both with the user and with other
packages.  (Out here, local users often request that a certain version
of some package be version-frozen because they don't want to deal with
bugs or changes in new versions.)

Unfortunately, the system of searching Lisp files in a global
load-path is not equipped to deal with this problem, since it
identifies only a single file foo.el (the first in a list of
directories) where there may be several different ones.  Note that
this is also tied to the current API for `provide' and `require.'

So we really need an infrastructure that deals with versioning and
allows you to express requirements like "version x of package z" or
"any version after x of package z" or "any version of package z
between x and y" and so on.   This extends to the underlying Emacs for
which you also want to express dependencies.  Currently, this kind of
stuff is done inside the packages with a bunch of `boundp' calls or
other clever ESP hackery to figure out what other packages you're
actually dealing with.  Instead, there should be a declarative way
which allows you (without forcing you) to specify what you want or
what you have.

Moreover, you would like to be able to have a single package tree
which allows you to keep several versions of a single package
accessible to be more flexible fulfilling those requirements.  (Of
course, not all requirements are fulfillable---you usually can't
simultaneously load two versions of the same package, for example.
because Emacs Lisp lacks a proper module system.  It used to be you
couldn't load vc and vc-cc at the same time, and so on.  But you'd
like to be able to try and find out if it's possible.)

Therefore, the design draft allows you to annotate your existing
package with requirements, and there's a little compositional language
for specifying them.  If you don't do this, you'll still essentially
get the old functionality.  It also allows the user to say: When, at
some point in the future, I'll request package x to be loaded, I want
version y.  (Or express some more complicated requirement.)

>> I have designed a new one, but have not been able to spend much time
>> on implementing it.  Once I do, I'll try my best to make the code
>> modular and sharable between the Emacsen.

Miles> Do you have documents or notes about the new design too?

There's a thread at

http://list-archive.xemacs.org/xemacs-beta/199907/msg00143.html

whose root has some notes.  I'll be happy to flesh out particular
issues if anyone's interested.

Interested readers might also want to look at SRFI 7

http://srfi.schemers.org/srfi-7/

which uses similar ideas (which I formulated around the same time) for
a related purpose.

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: XEmacs package code

[Kai Großjohann]

sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.  Preprocessor]) writes:

> So we really need an infrastructure that deals with versioning and
> allows you to express requirements like "version x of package z" or
> "any version after x of package z" or "any version of package z
> between x and y" and so on.

There are Emacs developers who feel that this is not the right
answer.  Instead of requiring a specific version, one should look for
the features supported by that version and require those, instead.

My humble opinion is that one should look for the features whenever
feasible, but sometimes it might not be feasible to do that.  In those
cases, it is better to be able to require some version of a package.
(All of the operations suggested by Michael.)  Also, when we are
talking about packages that are external to the Emacs core, version
numbers become much more meaningful.  After all, if I require a
feature which was added in AUC-TeX 11.10, then I can just require
that version number and be sure that the feature has been added.
There is only one AUC-TeX package.  For the Emacs core, the situation
is different.  There, it makes sense to ask questions like "can Emacs
do faces now" instead of questions like "is Emacs running under X11".

So how about adding the features that Michael is looking for and how
about educating Lisp programmers about the problems of just checking
for versions?

kai
-- 
Silence is foo!

Re: XEmacs package code

[Michael Sperber [Mr. Preprocessor]]

>>>>> "Kai" == Kai Großjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

Kai> sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.  Preprocessor]) writes:

>> So we really need an infrastructure that deals with versioning and
>> allows you to express requirements like "version x of package z" or
>> "any version after x of package z" or "any version of package z
>> between x and y" and so on.

Kai> There are Emacs developers who feel that this is not the right
Kai> answer.  Instead of requiring a specific version, one should look for
Kai> the features supported by that version and require those, instead.

Kai> My humble opinion is that one should look for the features whenever
Kai> feasible, but sometimes it might not be feasible to do that.  

In fact, it is quite often not feasible to do that: package evoluation
very often does not happen in terms of features provided.  Instead, it
often happens in terms of bug fixing or an evolving API.  Moreover,
you learn about features only *after* you've loaded the package.  This
is usually too late in an environment where multiple versions of a
single package can coexist.  Also, this is much too hard for users to
deal with who really want to say "This version and no others."

Of course, some of that can be fixed.  But the present provide/require
machinery is way too coarse for it.  And are you really willing to
provide features like "recompile-wrong-buffer-bug-fixed"?  Or
"find-file-takes-2-arguments"?

Of course there are many configuration questions you *can* attach to
features, and my design draft specifically accounts for those.  But
there's no way it can solve all or even most of the problems
associated with configuration management in the face of the situation
I envision.

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: XEmacs package code

[Stefan Monnier]

> The main problem is that our package system identifies each package
> only by its name, and this is the only identifier you get for
> resolving dependencies.  However, packages change over time, and with
> change, incompatibilities arise, both with the user and with other
> packages.  (Out here, local users often request that a certain version
> of some package be version-frozen because they don't want to deal with
> bugs or changes in new versions.)
> 
> Unfortunately, the system of searching Lisp files in a global
> load-path is not equipped to deal with this problem, since it
> identifies only a single file foo.el (the first in a list of
> directories) where there may be several different ones.  Note that
> this is also tied to the current API for `provide' and `require.'

You obviously have a lot of experience with such a package system, so
I'd love to know more about the details and the reasons for those details.

I was imagining a system as follows:

Packages distributed as a directory (like url, w3, gnus, x-symbol, ...)
get installed into <prefix>/w3-3.0.45.  Packages distributed as a single
file get installed into <prefix>/<subdir>/<filename>.el(c)

The user then puts in his .emacs (and/or site-lisp/site-start.el) a set
of versioning options:

   (load "<prefix>/w3-3.0.44/autoloads")
   (load "<prefix>/subdir-beta/autoloads")

These "autoloads" file add their directory to the load-path and
setup the usual autoloads and auto-mode-alist kind of things.

So you can install several versions of packages and the user chooses
which one she wants.  For single-file-packages, she can only choose
on a group-of-packages basis, where the grouping is up to the installer.

Now this does not automatically handle the issue of "package X wants
version Y of package Z", but it lets the user take care of it.
And of course, if we assume that the (load "foobar/autoloads") commands
are added by the package-manager, the dependencies could still be
obeyed without the user having to do anything, as long as some kind
of dependency-description language like the one you propose is available
(but that would be a separate layer).

Alright, I haven't really asked too many questions, have I ?

I'm not sure what question I want to ask apart from
- is the above hopelessly flawed ?
- if not, does it look anything like what you have in mind ?
- what are the layer/decomposition you're considering ?
  there are obviously several issues:
  - package format when installing.
  - package format once installed.
  - representation of dependency.
  - who/when do we use the dependency info.
  - ...


	Stefan

Re: XEmacs package code

[Michael Sperber [Mr. Preprocessor]]

>>>>> "Stefan" == Stefan Monnier <monnier+gnu/emacs@RUM.cs.yale.edu> writes:

Stefan> You obviously have a lot of experience with such a package system, so
Stefan> I'd love to know more about the details and the reasons for those details.

Stefan> I was imagining a system as follows:

Stefan> Packages distributed as a directory (like url, w3, gnus, x-symbol, ...)
Stefan> get installed into <prefix>/w3-3.0.45.  Packages distributed as a single
Stefan> file get installed into <prefix>/<subdir>/<filename>.el(c)

Stefan> The user then puts in his .emacs (and/or site-lisp/site-start.el) a set
Stefan> of versioning options:

Stefan>    (load "<prefix>/w3-3.0.44/autoloads")
Stefan>    (load "<prefix>/subdir-beta/autoloads")

Stefan> These "autoloads" file add their directory to the load-path and
Stefan> setup the usual autoloads and auto-mode-alist kind of things.

Stefan> So you can install several versions of packages and the user chooses
Stefan> which one she wants.  For single-file-packages, she can only choose
Stefan> on a group-of-packages basis, where the grouping is up to the installer.

Stefan> Now this does not automatically handle the issue of "package X wants
Stefan> version Y of package Z", but it lets the user take care of it.
Stefan> And of course, if we assume that the (load "foobar/autoloads") commands
Stefan> are added by the package-manager, the dependencies could still be
Stefan> obeyed without the user having to do anything, as long as some kind
Stefan> of dependency-description language like the one you propose is available
Stefan> (but that would be a separate layer).

Stefan> Alright, I haven't really asked too many questions, have I ?

Stefan> I'm not sure what question I want to ask apart from
Stefan> - is the above hopelessly flawed ?
Stefan> - if not, does it look anything like what you have in mind ?

It looks almost exactly like what I have in mind, except my has
provisions for doing automatically what you have the user "take care"
of.  I do think, however, that the latter is crucial.  In cases where
this becomes relevant, the symptoms of bad package interaction are way
too bad for the user to figure out.

Stefan> - what are the layer/decomposition you're considering ?
Stefan>   there are obviously several issues:
Stefan>   - package format when installing.

I'm all for .tar.gz.

Stefan>   - package format once installed.

I intend to keep the specification of that as loose as possible.

Stefan>   - representation of dependency.
Stefan>   - who/when do we use the dependency info.
Stefan>   - ...

That stuff is in the draft, I believe.  It's the most elaborately
formulated part of all that's there.

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: XEmacs package code

[Richard Stallman]

    More importantly, the current design of the package system is
    generally perceived as a failure in the XEmacs community, even by
    Steve.  (I'll be happy to explain why if anyone's interested.)  Thus,
    it may not be desirable to adapt this code for Emacs.

    I have designed a new one, but have not been able to spend much time
    on implementing it.

It would be useful to explain why, and post your design--perhaps
someone here could help work on new package code.

XEmacs package code

[Richard Stallman]

Sperber, the author of the package code in XEmacs,
signed papers for it but does not have time to adapt it for Emacs.
It is in files packages.el and some of startup.el.
Would someone like to adapt that?