Emacs Package Management

Emacs Package Management

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]

Disclaimer: I have no idea if I am flogging a dead horse. If so, please
disregard.

Compared to most people here, I am a pretty young Emacs user, barely a year
and a half since I've "converted". However, my .emacs is already growing
huge.

Part of this is due to Ruby on Rails development. I had to gather quite a
lot of scripts to do what I want (rails-mode, nxhtml-mode, rinari [for
find-file-in-project], color-theme, rdebug) and so on. This setup *almost*
works, as some of the scripts do not play well with each other.

Since there appears to be work under way to get some IDE-like features into
Emacs, I suppose some kind of "packaging system" wouldbe helpful. I have
tried ELPA (http://tromey.com/elpa/) and loved its simplicity. It's an order
of magnitude more convenient than seaching the web for a package, finding
the appropriate download site, getting the latest revision, studying the
README to figure out how to install it, copying it to .emacs.d and adding to
.emacs... I am sure everyone here has done that, countless times.

With a slightly improved system, we could have dependencies. This could make
easier to solve the aforementioned problem of gathering multiple,
independent packages from different sources.

Also, some packages have built-in bug reporting, but not all of them do.
Some of them are maintained in the Emacs Wiki, some are not maintained at
all, some have changed places more than once. Getting a package system
inside Emacs *could* allow for simpler updating and a simpler way to notify
bugs. I am aware of emacsbug, but it does require the ability to send
e-mails from inside Emacs and is not aware of packages (obviously).

Has this already been tried before? My searches point to XEmacs, but I
haven't installed it to see what its package manager looks like.

Does anyone see a major flaw in a system like that? Or is it a matter of
"show me the code and I'll comment"? ELPA could be the starting point.

--Stephen

programmer, n:
A red eyed, mumbling mammal capable of conversing with inanimate monsters.

[-- Attachment #2: Type: text/html, Size: 2258 bytes --]

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Stephen" == Stephen Eilert <spedrosa@gmail.com> writes:

Stephen> With a slightly improved system, we could have
Stephen> dependencies. This could make easier to solve the
Stephen> aforementioned problem of gathering multiple, independent
Stephen> packages from different sources.

Just FYI -- package.el (the elisp side of ELPA) does handle dependencies.

:-)

Stephen> Does anyone see a major flaw in a system like that? Or is it
Stephen> a matter of "show me the code and I'll comment"? ELPA could
Stephen> be the starting point.

There was a discussion a while ago on this list.  RMS wanted to
restrict the available packages to those which had been assigned to
the FSF, but I did not agree with that.

I would reconsider my position if the Emacs maintainers were
interested -- I think it would be useful to Emacs users if there were
a simple, standard way to install and activate packages.

However, this would still not help you directly, because I think some
of the packages you want are not assigned.  So, you would have to
solve that problem as well.

Tom

Re: Emacs Package Management

[Phil Hagelberg]

Tom Tromey <tromey@redhat.com> writes:

> Stephen> Does anyone see a major flaw in a system like that? Or is it
> Stephen> a matter of "show me the code and I'll comment"? ELPA could
> Stephen> be the starting point.
>
> There was a discussion a while ago on this list.  RMS wanted to
> restrict the available packages to those which had been assigned to
> the FSF, but I did not agree with that.
>
> I would reconsider my position if the Emacs maintainers were
> interested -- I think it would be useful to Emacs users if there were
> a simple, standard way to install and activate packages.
>
> However, this would still not help you directly, because I think some
> of the packages you want are not assigned.  So, you would have to
> solve that problem as well.

As the original author of rinari, I can see a place for packages in such
a system that are not part of Emacs, but still have their copyright
assigned to the FSF. I have a number of elisp packages that are not
candidates for inclusion in Emacs (for a number of reasons, including
rapid change rate, usage of the CL library, or just limited appeal), and
I would be glad to assign copyright over if it meant that they could be
distributed via a built-in package manager.

This would make them much, much easier to install and try out, which I
think is a big win for the overall Elisp ecosystem. People are more
likely to get interested and contribute if you lower the barrier to
trying new things.

It seems the debate so far has been held in terms of "packaging system"
vs "don't make it too easy to install non-FSF-copyrighted code", but I
don't think the two need to be mutually exclusive.

-Phil

Re: Emacs Package Management

[Lennart Borgman (gmail)]

Phil Hagelberg wrote:
> This would make them much, much easier to install and try out, which I
> think is a big win for the overall Elisp ecosystem. People are more
> likely to get interested and contribute if you lower the barrier to
> trying new things.


And the barrier would be even lower if ELPA had an uninstall command. 
Or, maybe I should say that it would be lower for me if I knew the name 
of the uninstall command in ELPA?

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Lennart" == Lennart Borgman (gmail) <lennart.borgman@gmail.com> writes:

Lennart> And the barrier would be even lower if ELPA had an uninstall
Lennart> command. Or, maybe I should say that it would be lower for me if I
Lennart> knew the name of the uninstall command in ELPA?

You can delete a package from the package menu using 'd'.
I don't think package.el can uninstall itself atm, though I forget.

Tom

Re: Emacs Package Management

[Richard M. Stallman]

    As the original author of rinari, I can see a place for packages in such
    a system that are not part of Emacs, but still have their copyright
    assigned to the FSF.

That might be a good idea; I have nothing against it.

However, even these packages should not load CL at run time.
Loading it encroaches on the user's namespace.

Re: Emacs Package Management

[Stefan Monnier]

>     As the original author of rinari, I can see a place for packages in such
>     a system that are not part of Emacs, but still have their copyright
>     assigned to the FSF.

> That might be a good idea; I have nothing against it.

Yes, it sounds like a very good idea.

> However, even these packages should not load CL at run time.

I don't see any benefit in trying to enforce such a restriction.

> Loading it encroaches on the user's namespace.

If so, the user is free to not use that package.


        Stefan

Re: Emacs Package Management

[Richard M. Stallman]

    > Loading it encroaches on the user's namespace.

    If so, the user is free to not use that package.

Excusing a flaw like that isn't the way to get smooth operation.

Re: Emacs Package Management

[Stefan Monnier]

>> Loading it encroaches on the user's namespace.
>     If so, the user is free to not use that package.
> Excusing a flaw like that isn't the way to get smooth operation.

Some unbundled Elisp packages (and some bundled ones as well, sadly)
have all kinds of nasty flaws.  Using CL is the least of my worries in
this arena,


        Stefan

Re: Emacs Package Management

[Richard M. Stallman]

    Some unbundled Elisp packages (and some bundled ones as well, sadly)
    have all kinds of nasty flaws.  Using CL is the least of my worries in
    this arena,

Perhaps we can't do anything about flaws in the Lisp programs that are
distributed independently of us.  Their developers never made any
commitment to follow our standards.  But FSF-copyrighted programs
are GNU packages, and have maintainers who have undertaken to do so.

Re: Emacs Package Management

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]

On Tue, Aug 5, 2008 at 5:04 AM, Richard M. Stallman <rms@gnu.org> wrote:

>    Some unbundled Elisp packages (and some bundled ones as well, sadly)
>    have all kinds of nasty flaws.  Using CL is the least of my worries in
>    this arena,
>
> Perhaps we can't do anything about flaws in the Lisp programs that are
> distributed independently of us.  Their developers never made any
> commitment to follow our standards.  But FSF-copyrighted programs
> are GNU packages, and have maintainers who have undertaken to do so.
>

In the context of package management, who's going to enforce those package
restrictions? Should packages be approved manually before being added to
whatever repository we devise?

There are many possible uses for a packaging system. One of them is making
it easier to install packages that are *not* part of Emacs. In this context,
making it harder to submit packages is probably not a good idea. AFAIK,
there's no M-x send-paperwork yet...

There's another possible use: to enable the distribution of a "lightweight"
Emacs, removing a lot of packages that are currently bundled but providing
an easy way to get and install them. This was not my initial motivation, but
I can see some utility in that approach. In this case, it's obvious that the
packages should conform to whatever standards the FSF wants to enforce.


--Stephen

programmer, n:
       A red eyed, mumbling mammal capable of conversing with inanimate
monsters.

[-- Attachment #2: Type: text/html, Size: 1914 bytes --]

Re: Emacs Package Management

[Paul R]

Hello Stephen,

Stephen> There's another possible use: to enable the distribution of
Stephen> a "lightweight" Emacs, removing a lot of packages that are
Stephen> currently bundled but providing an easy way to get and
Stephen> install them. This was not my initial motivation, but I can
Stephen> see some utility in that approach. In this case, it's obvious
Stephen> that the packages should conform to whatever standards the
Stephen> FSF wants to enforce.

We have a lot to learn from existing systems. The first I can think of
is the Debian distribution, because it has a strong policy about what
criteria packages must meet to enter in a repository or an other.
Also, only some of Debian repositories are activated in default
distribution. Please note I'm not judging the nature of this policy,
I'm only observing that Debian has very similar needs and seems to
handle them well, since a long time.
Although I'm not sure, I think I read here some posts from a person
involved in the Debian GNUemacs (and emacs extensions) packaging. He
probably has a lot to say in this area.


-- 
  Paul

Re: Emacs Package Management

[Richard M. Stallman]

    In the context of package management, who's going to enforce those package
    restrictions? Should packages be approved manually before being added to
    whatever repository we devise?

We could treat them the same way as if they were included in the
usual Emacs distribution, in effect spitting that distribution into parts.

Re: Emacs Package Management

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

On Wed, Aug 6, 2008 at 12:35 AM, Richard M. Stallman <rms@gnu.org> wrote:

>     In the context of package management, who's going to enforce those
> package
>    restrictions? Should packages be approved manually before being added to
>    whatever repository we devise?
>
> We could treat them the same way as if they were included in the
> usual Emacs distribution, in effect spitting that distribution into parts.
>

To sum it up: there *is* interest in a packaging feature, and the issues
seem to be mostly in the area of copyright assignment. If that's correct,
this is good.

ELPA is a nice starting point. But I think it is disproportionally easy to
download packages, as opposed to submitting them. For single files, I would
very much like to have a M-x submit-package, and have it do anything
required to upload it, adding licenses and so on. I don't mind if packages
have to be approved for inclusion in official repositories, but it should be
easy to upload them. For FSF packages, it could sign them with the user's
key.

Dependencies management is also nice, but we'd have to have a way to
identify packages.

And, most importantly, failing packages should not interrupt emacs startup.
I've implemented a simple system(for my own use) where it creates a buffer
with package load status. Failing packages are skipped, but obviously there
is no way to undo side-effects.


--Stephen

programmer, n:
A red eyed, mumbling mammal capable of conversing with inanimate monsters.

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Stephen" == Stephen Eilert <spedrosa@gmail.com> writes:

Stephen> To sum it up: there *is* interest in a packaging feature, and
Stephen> the issues seem to be mostly in the area of copyright
Stephen> assignment.

I suppose copyright assignment of packages.  There is no issue with
package.el itself, I will donate it any time.

Stephen> ELPA is a nice starting point. But I think it is
Stephen> disproportionally easy to download packages, as opposed to
Stephen> submitting them. For single files, I would very much like to
Stephen> have a M-x submit-package, and have it do anything required to
Stephen> upload it, adding licenses and so on.

Yeah.  I would like this too, but that would mean finding time to write
the web app and everything else :-)

Stephen> Dependencies management is also nice, but we'd have to have a
Stephen> way to identify packages.

Yes, this is something that package.el provides.  One thing that would
be nice is to have Emacs also advertise the packages it provides -- that
is, the ones that are also on separate release schedules.  Right now I
track this information by hand, but that is somewhat error prone.

Stephen> And, most importantly, failing packages should not interrupt
Stephen> emacs startup.  I've implemented a simple system(for my own
Stephen> use) where it creates a buffer with package load
Stephen> status. Failing packages are skipped, but obviously there is no
Stephen> way to undo side-effects.

If this is a problem with package.el or some package in ELPA, please
send me some email (preferably to the ELPA address).  I will try to fix
any problems soon.

In general my rule with ELPA has been to try to avoid this problem by
only uploading packages that will at least properly "activate".  Testing
this could be automated and made more robust, but again, I have little
time.

Tom

Re: Emacs Package Management

[Stefan Monnier]

Stephen> ELPA is a nice starting point. But I think it is
Stephen> disproportionally easy to download packages, as opposed to
Stephen> submitting them. For single files, I would very much like to
Stephen> have a M-x submit-package, and have it do anything required to
Stephen> upload it, adding licenses and so on.

> Yeah.  I would like this too, but that would mean finding time to write
> the web app and everything else :-)

For the FSF-hosted repository of packages, I'd want all the packages
to be version controlled, so the "submit-package" would be nothing else
than "bzr commit" (but yes, some extra code would need to be written to
automatically build a tarball out of the newly committed code, ...).

> Yes, this is something that package.el provides.  One thing that would
> be nice is to have Emacs also advertise the packages it provides -- that

Indeed, we'd of course want that, if/once we provide such
a package system.  But that should be easy.


        Stefan

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yes, this is something that package.el provides.  One thing that would
>> be nice is to have Emacs also advertise the packages it provides -- that

Stefan> Indeed, we'd of course want that, if/once we provide such
Stefan> a package system.  But that should be easy.

Yeah, it is.  It amounts to either some new autoload comments, or
running some elisp to extract package declarations.  Come to think of
it, I don't know why I didn't implement this yet.

Tom

Re: Emacs Package Management

[Eric M. Ludlam]

On Thu, 2009-09-17 at 09:43 -0400, Stefan Monnier wrote:
> Stephen> ELPA is a nice starting point. But I think it is
> Stephen> disproportionally easy to download packages, as opposed to
> Stephen> submitting them. For single files, I would very much like to
> Stephen> have a M-x submit-package, and have it do anything required to
> Stephen> upload it, adding licenses and so on.
> 
> > Yeah.  I would like this too, but that would mean finding time to write
> > the web app and everything else :-)
> 
> For the FSF-hosted repository of packages, I'd want all the packages
> to be version controlled, so the "submit-package" would be nothing else
> than "bzr commit" (but yes, some extra code would need to be written to
> automatically build a tarball out of the newly committed code, ...).

EDE builds makefiles that builds dist files for projects.  CEDET's
distribution files are fabricated by EDE's code generator.  If there
were to be a "preferred" Emacs Lisp project style, EDE would be able to
handle it easily as a wrapper on top of whatever the preferred
implementation is, or you can use the same pattern that CEDET uses now.

Eric

Re: Emacs Package Management

[Phil Hagelberg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> For the FSF-hosted repository of packages, I'd want all the packages
> to be version controlled, so the "submit-package" would be nothing else
> than "bzr commit" (but yes, some extra code would need to be written to
> automatically build a tarball out of the newly committed code, ...).
>
>> Yes, this is something that package.el provides.  One thing that would
>> be nice is to have Emacs also advertise the packages it provides -- that
>
> Indeed, we'd of course want that, if/once we provide such
> a package system.  But that should be easy.

It seems like though there are a few questions remaining about policies
for submissions, there's a general consensus that Emacs should get a
packaging system, probably based on package.el with some enhancements
for submitting projects.

I'd like to start brainstorming as to what this would look like.

One way to do it would be to have every package register as a Savannah
project, and have some kind of operation (probably implemented as an
elisp function) that could produce the static files that package.el
consumes and place them in a publicly accessible http-served
directory. Making package releases correspond with VCS tags would
simplify things, so with some kind of post-commit hooks Savannah could
run this function whenever a tag is updated in the project.

This would require very few changes to package.el. Writing such a
function should not be difficult. The only question is how easy it would
be to get Savannah to run it at the right time.

Thoughts?

-Phil

Re: Emacs Package Management

[Lennart Borgman]

On Mon, Sep 28, 2009 at 11:13 PM, Phil Hagelberg <phil@hagelb.org> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> For the FSF-hosted repository of packages, I'd want all the packages
>> to be version controlled, so the "submit-package" would be nothing else
>> than "bzr commit" (but yes, some extra code would need to be written to
>> automatically build a tarball out of the newly committed code, ...).
>>
>>> Yes, this is something that package.el provides.  One thing that would
>>> be nice is to have Emacs also advertise the packages it provides -- that
>>
>> Indeed, we'd of course want that, if/once we provide such
>> a package system.  But that should be easy.
>
> It seems like though there are a few questions remaining about policies
> for submissions, there's a general consensus that Emacs should get a
> packaging system, probably based on package.el with some enhancements
> for submitting projects.
>
> I'd like to start brainstorming as to what this would look like.
>
> One way to do it would be to have every package register as a Savannah
> project, and have some kind of operation (probably implemented as an
> elisp function) that could produce the static files that package.el
> consumes and place them in a publicly accessible http-served
> directory. Making package releases correspond with VCS tags would
> simplify things, so with some kind of post-commit hooks Savannah could
> run this function whenever a tag is updated in the project.
>
> This would require very few changes to package.el. Writing such a
> function should not be difficult. The only question is how easy it would
> be to get Savannah to run it at the right time.
>
> Thoughts?

I think the way to go is to use a source code management system.
However I wonder if it would not be better with one project. That
would allow some structure to it.

The drawback is that some management have to be done and maybe it
looses the "everyone" can put something there thing then.

Perhaps structure can be combined with this somewhere else? Maybe as a
separate project that is a layer between the package system and the
package projects?

Re: Emacs Package Management

[Chong Yidong]

Phil Hagelberg <phil@hagelb.org> writes:

> One way to do it would be to have every package register as a Savannah
> project, and have some kind of operation (probably implemented as an
> elisp function) that could produce the static files that package.el
> consumes and place them in a publicly accessible http-served
> directory. Making package releases correspond with VCS tags would
> simplify things, so with some kind of post-commit hooks Savannah could
> run this function whenever a tag is updated in the project.
>
> This would require very few changes to package.el. Writing such a
> function should not be difficult. The only question is how easy it would
> be to get Savannah to run it at the right time.
>
> Thoughts?

We're not necessarily restricted to Savannah; if necessary, we might be
able to use the gnu.org ftp servers, and their mirrors (which we
currently already use for distributing the monolithic tarball, after
all).

Re: Emacs Package Management

[Phil Hagelberg]

Chong Yidong <cyd@stupidchicken.com> writes:

>> One way to do it would be to have every package register as a Savannah
>> project, and have some kind of operation (probably implemented as an
>> elisp function) that could produce the static files that package.el
>> consumes and place them in a publicly accessible http-served
>> directory. Making package releases correspond with VCS tags would
>> simplify things, so with some kind of post-commit hooks Savannah could
>> run this function whenever a tag is updated in the project.
>>
>> This would require very few changes to package.el. Writing such a
>> function should not be difficult. The only question is how easy it would
>> be to get Savannah to run it at the right time.
>
> We're not necessarily restricted to Savannah; if necessary, we might be
> able to use the gnu.org ftp servers, and their mirrors (which we
> currently already use for distributing the monolithic tarball, after
> all).

I think Stefan had a great point that a lot of complexity involving
user, project, and release management can be sidestepped by using the
existing infrastructure of a version-control system. Savannah seems to
be a good fit for this, but if we can host an Emacs-package-specific VCS
somewhere else, that would work too.

-Phil

Re: Emacs Package Management

[Richard Stallman]

I like the idea of making Emacs packages work with Savannah.
Keep in mind that we are only going to advertise a specific
list of packages that we have copyright assignments on.
We intend to avoid letting all and sundry stick packages into
this system and bypassing us.

Re: Emacs Package Management

[Stefan Monnier]

In order to move forward on this thing, here's what I'd like to have as
a first step: the equivalent of `dpkg', i.e. an elisp package that can
take a package (a tarball, most likely) and "install it", list the set
of installed packages, uninstall a package.

An important feature for me is that this should be able to "install
without making available", so you can install various conflicting
packages at the same time (e.g. different versions of Gnus).

So basically, the installation process would distinguish the following
steps:
- install: may include byte-compiling and things like that
- activate: eval the autoload declarations, adjust the load-path, ...
- make available: setup the .emacs so that the package gets activated
  at startup.

Activation should be done by loading a single file.  I.e. "make
available" is nothing more than add a (load "/foo/bar/baz") in the
.emacs.

This "dpkg"-equivalent may also want to do dependency checking, although
this is not absolutely necessary for the first step.

As you can see, this first step is independent from the place and the
manner we upload&download the packages, so we can work on it, while we
figure out the repository part of the problem.


        Stefan

Re: Emacs Package Management

[Tom Tromey]

Stefan> In order to move forward on this thing, here's what I'd like to have as
Stefan> a first step: the equivalent of `dpkg', i.e. an elisp package that can
Stefan> take a package (a tarball, most likely) and "install it", list the set
Stefan> of installed packages, uninstall a package.

package.el does all of these...

Stefan> An important feature for me is that this should be able to "install
Stefan> without making available"

... but not this.

Stefan> So basically, the installation process would distinguish the following
Stefan> steps:
Stefan> - install: may include byte-compiling and things like that
Stefan> - activate: eval the autoload declarations, adjust the load-path, ...
Stefan> - make available: setup the .emacs so that the package gets activated
Stefan>   at startup.

package.el does this.

Stefan> Activation should be done by loading a single file.  I.e. "make
Stefan> available" is nothing more than add a (load "/foo/bar/baz") in the
Stefan> .emacs.

It does this too, though only a single (package-initialize) is needed --
package.el handles activating all the packages that can be activated.

Stefan> This "dpkg"-equivalent may also want to do dependency checking, although
Stefan> this is not absolutely necessary for the first step.

Also done.

Also, package.el sets the info path when a package ships a dir file.

Tom

Re: Emacs Package Management

[Stefan Monnier]

Stefan> An important feature for me is that this should be able to "install
Stefan> without making available"
> ... but not this.

That's what I remembered, indeed.  Could you change it?

Stefan> So basically, the installation process would distinguish the following
Stefan> steps:
Stefan> - install: may include byte-compiling and things like that
Stefan> - activate: eval the autoload declarations, adjust the load-path, ...
Stefan> - make available: setup the .emacs so that the package gets activated
Stefan> at startup.

> package.el does this.

You mean that it "does" all those steps, right?
Not that it "distinguishes" them, right?  IOW, it collapses "activate"
and "make available" into a single entity.


        Stefan

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> So basically, the installation process would distinguish the following
Stefan> steps:
Stefan> - install: may include byte-compiling and things like that
Stefan> - activate: eval the autoload declarations, adjust the load-path, ...
Stefan> - make available: setup the .emacs so that the package gets activated
Stefan> at startup.

>> package.el does this.

Stefan> You mean that it "does" all those steps, right?
Stefan> Not that it "distinguishes" them, right?  IOW, it collapses "activate"
Stefan> and "make available" into a single entity.

Yeah, sorry about that.

The reason it is this way is just that I thought the UI was better.
Few people have asked for a separate "deactivated" state.

Naturally, it is implementable.  I still think the default on
installation should be to activate; requiring two steps seems
user-unfriendly.

Tom

Re: Emacs Package Management

[Stefan Monnier]

Stefan> You mean that it "does" all those steps, right?
Stefan> Not that it "distinguishes" them, right?  IOW, it collapses "activate"
Stefan> and "make available" into a single entity.

> Yeah, sorry about that.

> The reason it is this way is just that I thought the UI was better.
> Few people have asked for a separate "deactivated" state.

> Naturally, it is implementable.  I still think the default on
> installation should be to activate; requiring two steps seems
> user-unfriendly.

Yes, I agree on the UI front.  But I think it's important to let the
user have more control, when she needs it.  So the default will be to
auto-activate.


        Stefan

Re: Emacs Package Management

[Tom Tromey]

Stefan> In order to move forward on this thing, here's what I'd like to have as
Stefan> a first step: the equivalent of `dpkg', i.e. an elisp package that can
Stefan> take a package (a tarball, most likely) and "install it", list the set
Stefan> of installed packages, uninstall a package.

I was thinking more about integrating package.el and I remembered one
problem.

Right now, package.el byte-compiles at install time.  However, this
yields incorrect results if the .elc format changes incompatibly, and
the user upgrades Emacs.  I believe this has happened in the past.

We discussed a solution to this long ago: putting the .elc files into a
shadow directory hierarchy.  Is this still a good idea?

I will start a branch in the emacs-mt git repository to test out
integration.

Tom

Re: Emacs Package Management

[Stefan Monnier]

Stefan> In order to move forward on this thing, here's what I'd like to
Stefan> have as a first step: the equivalent of `dpkg', i.e. an elisp
Stefan> package that can take a package (a tarball, most likely) and
Stefan> "install it", list the set of installed packages, uninstall
Stefan> a package.

> I was thinking more about integrating package.el and I remembered one
> problem.

> Right now, package.el byte-compiles at install time.  However, this
> yields incorrect results if the .elc format changes incompatibly, and
> the user upgrades Emacs.  I believe this has happened in the past.

We normally try to preserve forward compatibility, so old .elcs should
work fine.  IOW, the problem can only show up if the user byte-compiles
with the new Emacs and then uses the file with an older Emacs.
If that's the problem, then I wouldn't worry about it (after all, the
same happens already for the normal manual installation procedure or
external packages).

> We discussed a solution to this long ago: putting the .elc files into a
> shadow directory hierarchy.  Is this still a good idea?

If possible, I'd rather avoid the complexity.


        Stefan

Re: Emacs Package Management

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 1722 bytes --]

On Fri, Aug 1, 2008 at 7:58 PM, Tom Tromey <tromey@redhat.com> wrote:

> >>>>> "Stephen" == Stephen Eilert <spedrosa@gmail.com> writes:
>
> Stephen> With a slightly improved system, we could have
> Stephen> dependencies. This could make easier to solve the
> Stephen> aforementioned problem of gathering multiple, independent
> Stephen> packages from different sources.
>
> Just FYI -- package.el (the elisp side of ELPA) does handle dependencies.
>
> :-)


It appears I have overlooked that. Nice to know.

>
>
> Stephen> Does anyone see a major flaw in a system like that? Or is it
> Stephen> a matter of "show me the code and I'll comment"? ELPA could
> Stephen> be the starting point.
>
> There was a discussion a while ago on this list.  RMS wanted to
> restrict the available packages to those which had been assigned to
> the FSF, but I did not agree with that.


Meh. Just set up a repository maintained by the FSF by default and leave the
ability to add others, at the user's discretion. It won't prevent anything
that isn't already being done, as it is just for convenience.

>
>
> I would reconsider my position if the Emacs maintainers were
> interested -- I think it would be useful to Emacs users if there were
> a simple, standard way to install and activate packages.


Definitely. Grouping modules together (instead of all over the web at some
guy's personal homepage) would be just an added bonus.


>
>
> However, this would still not help you directly, because I think some
> of the packages you want are not assigned.  So, you would have to
> solve that problem as well.


One problem at a time ;)


--Stephen

programmer, n:
A red eyed, mumbling mammal capable of conversing with inanimate monsters.

[-- Attachment #2: Type: text/html, Size: 2638 bytes --]

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Stephen" == Stephen Eilert <spedrosa@gmail.com> writes:

Tom> There was a discussion a while ago on this list.  RMS wanted to
Tom> restrict the available packages to those which had been assigned
Tom> to the FSF, but I did not agree with that.

Stephen> Meh. Just set up a repository maintained by the FSF by
Stephen> default and leave the ability to add others, at the user's
Stephen> discretion.

My recollection of the previous discussion is that this idea was
rejected.

Stephen> One problem at a time ;)

Yes :)

Tom

Re: Emacs Package Management

[Richard M Stallman]

Currently there is a big visible difference between the Lisp programs
that are part of Emacs and the ones that are not.  Thus, programs
which are not FSF-copyrighted do not appear to the user as part of
Emacs.

That is very important, and we should not install changes that would
make the difference less striking.

Re: Emacs Package Management

[Thomas Lord]

Tom Tromey wrote:
> There was a discussion a while ago on this list.  RMS wanted to
> restrict the available packages to those which had been assigned to
> the FSF, but I did not agree with that.
>   

Please consider making the package system such that packages are
signed and users consciously pick and choose among authorities (i.e.,
which package signatures to trust).

To an extent, that makes the problem harder:  a key management
system has to be part of the package system if "average" users are
expected to be able to use it.

On the other hand, it creates an economic market  -- especially if
you keep that in mind while designing the package system.   That is,
a new possible commercial service (entirely free-software friendly)
is to let people subscribe to a source of trusted emacs packages.
Providers of this service can differentiate and compete in lots of ways.
I think a worthy goal is to design a package system that helps
such a market flourish.

Firefox has a feature that illustrates a nice and relevant paradigm:
It frequently "calls home" to find out if a new upgrade has been
published and then automates the process of installing upgrades.
Emacs and the possible marketplace for emacs package providers
could benefit from a similar infrastructure.   It is hard to get right,
of course:  very easy to introduce vulnerabilities through design or
coding mistakes.   Nevertheless, if it *is* gotten right, it helps to create
that marketplace.

In other threads about DOS and Windows support, people were
talking about the ideal of never having to earn a living in the proprietary
software world.   It seems to me that the most direct way to make that
possible is to collectively concentrate on software systems that create
markets for free software developer talent (by virtue of the architecture
of those systems).

Package systems are exactly the right place to hack markets to create
free software jobs.  RHAT and Canonical both discovered this a ways
back.  An industry standard here, a really solid one, would crank up 
competition
and generally "lubricate" the free software economy (by lowering the
barrier to entering the market as a package-by-subscription provider while
still preserving the opportunity for package providers to differentiate and
add value between upstream and the user).   It is not something that those
companies are likely to be eager for until it becomes inevitable.   
Therefore
it might well be a good strategic investment for volunteers.

One final technical note:   In designing the package system it is
desirable not only to allow competing package provider services,
but also (reflecting the reality of software development) to afford
composing providers into "pipelines".   That is, the package system
should model the concept of an upstream developer giving code to
downstream service providers who in turn give code (possibly patched)
to end users.   An end-user package download transaction should be able
to report the "heritage" of the package to that level.   The reason for
this is to create the possibility of a market for "royalties" paid to 
upstream
developers -- that is, to create a funding pipeline for free software R&D.


-t

Re: Emacs Package Management

[Phil Hagelberg]

Tom Tromey <tromey@redhat.com> writes:

> Just FYI -- package.el (the elisp side of ELPA) does handle
> dependencies.
>
> There was a discussion a while ago on this list.  RMS wanted to
> restrict the available packages to those which had been assigned to
> the FSF, but I did not agree with that.
>
> I would reconsider my position if the Emacs maintainers were
> interested -- I think it would be useful to Emacs users if there were
> a simple, standard way to install and activate packages.

It seems like the dust has settled a bit since the 23 release and we're
out of feature freeze; I wonder if this would be a good time to discuss
the inclusion of package.el again. I've been using it extensively as
well as distributing all of my own elisp through it, and it has greatly
simplified things for me and the users of my software.

Some people have asked what the use would be; why not just include them
in Emacs itself? There are several reasons:

* Much of my elisp is still undergoing a rapid rate of change. Often
  versions from two months ago are quite out of date. Using package.el
  lets me use my own release cycle rather than relying on Emacs', which
  is much improved, but still not appropriate for many projects.

* Emacs is already too big. If we made it easy to install third-party
  packages, we could spin many packages already included in Emacs
  (perhaps gnus, erc, and org) out into their own independent
  projects. We would avoid problems like "the big gnus merge" that
  happens every so often as well as allowing them to follow their own
  release cycles. In addition, a lot of packages are simply fringe;
  they're only useful to a small subset of Emacs users.

* I make use of the cl library, disqualifying much of my code for
  inclusion in Emacs proper. I'd venture to say close to a majority of
  the nontrivial packages out there do as well.

It sounds like the main objection to its inclusion is that it would
lessen the incentive to assign copyright to the FSF. I don't think this
is necessarily the case; we can hook it up to a repository that only
accepts FSF-copyrighted code.  I would be happy to contribute copyright
assignment for the libraries I maintain if it meant that they could be
offered via a package manager built-in to Emacs. I suspect many others
would too.

There would be some work involved in making package.el acceptable for
Emacs. Currently uploading of new packages is done by hand; this process
will need to be automated. Perhaps it could be integrated with Savannah
project hosting; I'm sure something could be worked out.

But I think this is a big opportunity to make it easier for developers
and users of Emacs to share code.

-Phil

Re: Emacs Package Management

[Eric M. Ludlam]

On Sat, 2009-09-12 at 15:38 -0700, Phil Hagelberg wrote:
> Tom Tromey <tromey@redhat.com> writes:
> 
[...]
> It seems like the dust has settled a bit since the 23 release and we're
> out of feature freeze; I wonder if this would be a good time to discuss
> the inclusion of package.el again. I've been using it extensively as
> well as distributing all of my own elisp through it, and it has greatly
> simplified things for me and the users of my software.
> 
> Some people have asked what the use would be; why not just include them
> in Emacs itself? There are several reasons:
[...]
> 
> There would be some work involved in making package.el acceptable for
> Emacs. Currently uploading of new packages is done by hand; this process
> will need to be automated. Perhaps it could be integrated with Savannah
> project hosting; I'm sure something could be worked out.

[..]

The EDE part or CEDET is all about developing projects.  I'd be willing
to work with you to have it "create" projects/packages appropriate for
an Emacs package system, since CEDET itself would want to take advantage
of such a thing if it were made a part of Emacs.

Of course, this implies that Emacs packages are built w/ EDE's project
system, which some consider too heavy-handed.

Eric

Re: Emacs Package Management

[Richard Stallman]

If we do set up a package system, we should insist that any packages
to be recommended from Emacs follow the same rules as code to be
included in Emacs.  This includes copyright assignments, and all the
technical conventions for Emacs code.

    * Emacs is already too big. If we made it easy to install third-party
      packages, we could spin many packages already included in Emacs
      (perhaps gnus, erc, and org) out into their own independent
      projects. We would avoid problems like "the big gnus merge" that
      happens every so often as well as allowing them to follow their own
      release cycles.

Splitting things up can be good in reducing the size of the main
Emacs distro.  But watch out for a possible problem.

Supporting the latest Gnus on old versions of Emacs already
complicates things.  If Emacs does not come with a copy of Gnus, we
will also be asked to support running an old Gnus in a new Emacs,
which will square the difficulty.

    * I make use of the cl library, disqualifying much of my code for
      inclusion in Emacs proper. I'd venture to say close to a majority of
      the nontrivial packages out there do as well.

The rule against using CL functions at run time is for the sake of the
user.  The CL functions are not a standard part of the Emacs Lisp
namespace.  Thus, loading CL can conflict with the user's function
definitions.

Even if we separate out parts of Emacs, this policy will still apply.
It has nothing to do with whether Emacs si distributed as one package
or several packages.

Instead of this policy, we could declare those functions standard, but
I think some of them are not well designed and not good to include.
Also, to do it right we would need to document these functions in the
Emacs Lisp manual, which is a big job.

Re: Emacs Package Management

[joakim]

Richard Stallman <rms@gnu.org> writes:
> The rule against using CL functions at run time is for the sake of the
> user.  The CL functions are not a standard part of the Emacs Lisp
> namespace.  Thus, loading CL can conflict with the user's function
> definitions.

Could we then privide aliases like "cl-loop" for "loop" ?

> Instead of this policy, we could declare those functions standard, but
> I think some of them are not well designed and not good to include.
> Also, to do it right we would need to document these functions in the
> Emacs Lisp manual, which is a big job.

Maybe we could include Cl functions in the Emacs core incrementaly then?
Each CL function moved to the core would require an agreement on the
functions inclusion and a documentation patch.

-- 
Joakim Verona

Re: Emacs Package Management

[David Kastrup]

joakim@verona.se writes:

> Richard Stallman <rms@gnu.org> writes:
>> The rule against using CL functions at run time is for the sake of the
>> user.  The CL functions are not a standard part of the Emacs Lisp
>> namespace.  Thus, loading CL can conflict with the user's function
>> definitions.
>
> Could we then privide aliases like "cl-loop" for "loop" ?

I would prefer cl:loop here.

-- 
David Kastrup

Re: Emacs Package Management

[Richard Stallman]

    Could we then privide aliases like "cl-loop" for "loop" ?

I am not against it.

    Maybe we could include Cl functions in the Emacs core incrementaly then?
    Each CL function moved to the core would require an agreement on the
    functions inclusion and a documentation patch.

That's more or less what we've been doing.  Originally I strove very
hard to keep Emacs itself small.  Many basic and obviously useful
functions were not standardly available, but they were in CL.  Since
then we have made a number of them standard, and we could certainly
do this for more of them in the future when it seems best.

But some of the CL facilities are overly complex.  And some,
specifically setf and friends, are not implemented quite right in the
Emacs context, which makes them ugly to include.

David Kastrup wrote

    I would prefer cl:loop here.

cl-loop fits better with Emacs Lisp, in which we do not
have Common Lisp packages.

Re: Emacs Package Management

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
> That's more or less what we've been doing.  Originally I strove very
> hard to keep Emacs itself small.  Many basic and obviously useful
> functions were not standardly available, but they were in CL.  Since
> then we have made a number of them standard, and we could certainly
> do this for more of them in the future when it seems best.
>
> But some of the CL facilities are overly complex.  And some,
> specifically setf and friends, are not implemented quite right in the
> Emacs context, which makes them ugly to include.

Some suggestions for core:

  * remove-if / delete-if / remove-if-not / delete-if-not
     (_without_ the CL keyword arguments)

    [These are very handy for filtering lists of values.  The keyword
     arguments are not very important, and keywordless core versions
     would be compatible with keyword-supporting cl.el versions in the
     same way that core `dolist' is compatible with cl.el `dolist']
    

  * mapcan

    [Useful for mapping when there isn't a 1-1 relationship between
    argument list elements and result list elements]


  * multiple-argument mapping (mapcar, mapc, etc) functions

    [Useful and obvious generalization of existing mapping functions;
     Gnus already contains its own implementation of a multiple-argument
     mapcar]


[I think all of these are simple and practical.]

-Miles

-- 
Sabbath, n. A weekly festival having its origin in the fact that God made the
world in six days and was arrested on the seventh.

Re: Emacs Package Management

[Richard Stallman]

Your suggestions seem fine to me.

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Miles" == Miles Bader <miles@gnu.org> writes:

Miles>   * remove-if / delete-if / remove-if-not / delete-if-not
Miles>      (_without_ the CL keyword arguments)

This is a side question, but I have been wondering for a while why Emacs
doesn't have CL keyword arguments.  Is there a reason?  I looked in the
Elisp manual, but didn't see anything.

It seems to me that plenty of elisp actually uses them in practice, even
functions in the Emacs C core.  Direct support for them would probably
simplify some code.

Tom

Re: Emacs Package Management

[Miles Bader]

Tom Tromey <tromey@redhat.com> writes:
> This is a side question, but I have been wondering for a while why Emacs
> doesn't have CL keyword arguments.  Is there a reason?  I looked in the
> Elisp manual, but didn't see anything.

They're complicated and inefficient?

Certainly there are cases where they're the best thing, but the CL core
seems to use them _way_ too much... forcing users to parse their own
keywords makes the complexity more obvious, and hopefully avoids
overuse.

-Miles

-- 
Joy, n. An emotion variously excited, but in its highest degree arising from
the contemplation of grief in another.

Re: Emacs Package Management

[Richard Stallman]

    This is a side question, but I have been wondering for a while why Emacs
    doesn't have CL keyword arguments.

I would not oppose adding that facility, but I think it would be a very
bad idea to follow that by following Common Lisp in the set of
keyword arguments that functions define.

Re: Emacs Package Management

[Stefan Monnier]

>   * remove-if / delete-if / remove-if-not / delete-if-not
>      (_without_ the CL keyword arguments)

>     [These are very handy for filtering lists of values.  The keyword
>      arguments are not very important, and keywordless core versions
>      would be compatible with keyword-supporting cl.el versions in the
>      same way that core `dolist' is compatible with cl.el `dolist']

Clearly we do not need both the *-if and the *-if-not, but I agree that
such a filtering function would come in handy at times.
OTOH the (delq nil (mapcar ...)) idiom isn't a bad replacement.

>   * mapcan
>     [Useful for mapping when there isn't a 1-1 relationship between
>     argument list elements and result list elements]

I find that (apply 'nconc (mapcar ...)) works just as well, so I don't
find it very useful.

>   * multiple-argument mapping (mapcar, mapc, etc) functions
>     [Useful and obvious generalization of existing mapping functions;
>      Gnus already contains its own implementation of a multiple-argument
>      mapcar]

I wonder what kind of impact it does/would have on performance.

In any case my main concern right now is that I do not want to introduce
any more of the "duplicate definition", like the ones of `push'
and `dolist'.

I just found out that my "clever" idea of reloading stale preloaded
files in lisp/Makefile's `compile_one' rule bumps into this problem
pretty badly: if both subr.el and vc-hooks.el are stale, then when
recompiling vc-hooks.el, we end up first reloading subr.el which
overrides the push definition previously installed by cl.el, leading to
a vc-hooks.elc that is corrupt (because it requires CL's version of
push).


        Stefan

Re: Emacs Package Management

[Geoff Gole]

>   [These are very handy for filtering lists of values.  The keyword
>    arguments are not very important, and keywordless core versions
>    would be compatible with keyword-supporting cl.el versions in the
>    same way that core `dolist' is compatible with cl.el `dolist']

One question. Common Lisp's sequence functions operate on, um,
sequences (lists or vectors or strings). Should emacs' versions do the
same or be restricted to the simpler list case?

Re: Emacs Package Management

[Richard Stallman]

    One question. Common Lisp's sequence functions operate on, um,
    sequences (lists or vectors or strings). Should emacs' versions do the
    same or be restricted to the simpler list case?

There is no point in making them handle anything but lists.

Re: Emacs Package Management

[Tom Tromey]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS> Supporting the latest Gnus on old versions of Emacs already
RMS> complicates things.  If Emacs does not come with a copy of Gnus, we
RMS> will also be asked to support running an old Gnus in a new Emacs,
RMS> which will square the difficulty.

package.el only lets users install the latest version of a package that
is available in the package archive.  Older versions are not displayed
in the package menu.

Of course, a user might upgrade Emacs but not upgrade existing packages.
And, this may cause something to stop working.  I did not see a way to
solve this problem, and anyhow it is not any worse than the currently
existing situation with 3rd party .el libraries.

Tom

Re: Emacs Package Management

[Richard Stallman]

    RMS> Supporting the latest Gnus on old versions of Emacs already
    RMS> complicates things.  If Emacs does not come with a copy of Gnus, we
    RMS> will also be asked to support running an old Gnus in a new Emacs,
    RMS> which will square the difficulty.

    package.el only lets users install the latest version of a package that
    is available in the package archive.  Older versions are not displayed
    in the package menu.

I agree that will avoid that problem, if people install things like
Gnus only through package.el.

Re: Emacs Package Management

[Tassilo Horn]

Richard Stallman <rms@gnu.org> writes:

>     RMS> Supporting the latest Gnus on old versions of Emacs already
>     RMS> complicates things.  If Emacs does not come with a copy of
>     RMS> Gnus, we will also be asked to support running an old Gnus in
>     RMS> a new Emacs, which will square the difficulty.
>
>     package.el only lets users install the latest version of a package
>     that is available in the package archive.  Older versions are not
>     displayed in the package menu.
>
> I agree that will avoid that problem, if people install things like
> Gnus only through package.el.

But the newest package might not run on an older emacs version.  So it
would be good if package.el would show the latest version of a package
that is supposed to work with my emacs version, and if it requires other
packages, then their versions have to be taken into account, too.  Does
it do something like that?

Bye,
Tassilo

Re: Emacs Package Management

[joakim]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     RMS> Supporting the latest Gnus on old versions of Emacs already
>>     RMS> complicates things.  If Emacs does not come with a copy of
>>     RMS> Gnus, we will also be asked to support running an old Gnus in
>>     RMS> a new Emacs, which will square the difficulty.
>>
>>     package.el only lets users install the latest version of a package
>>     that is available in the package archive.  Older versions are not
>>     displayed in the package menu.
>>
>> I agree that will avoid that problem, if people install things like
>> Gnus only through package.el.
>
> But the newest package might not run on an older emacs version.  So it
> would be good if package.el would show the latest version of a package
> that is supposed to work with my emacs version, and if it requires other
> packages, then their versions have to be taken into account, too.  Does
> it do something like that?

The way most distros seems to solve this is to simply have different
repos for different distro versions. That should work for package.el
also. We could even adopt the stable/testing/bleeding scheme, so 3 repos
for every supported major emacs version.

>
> Bye,
> Tassilo
>
-- 
Joakim Verona

Re: Emacs Package Management

[Lennart Borgman]

On Thu, Sep 17, 2009 at 10:37 AM,  <joakim@verona.se> wrote:
>
> The way most distros seems to solve this is to simply have different
> repos for different distro versions. That should work for package.el
> also. We could even adopt the stable/testing/bleeding scheme, so 3 repos
> for every supported major emacs version.


That seems to be much more work for a program that is in development
all the time.

Re: Emacs Package Management

[Tassilo Horn]

joakim@verona.se writes:

>> But the newest package might not run on an older emacs version.  So
>> it would be good if package.el would show the latest version of a
>> package that is supposed to work with my emacs version, and if it
>> requires other packages, then their versions have to be taken into
>> account, too.  Does it do something like that?
>
> The way most distros seems to solve this is to simply have different
> repos for different distro versions. That should work for package.el
> also. We could even adopt the stable/testing/bleeding scheme, so 3
> repos for every supported major emacs version.

I don't think that's a good approach, because it requires a lot of
maintenance and testing on the package repository side.  It would be
better if a package could specify something like

  (package "my-package"
    (version "1.0")
    (need ">=emacs-23"))

  (package "my-package"
    (version "0.8")
    (need ">=emacs-22.1" "<emacs-23"))

and package.el does the resolving.  So when I run emacs 22, package.el
would show version 0.8 of my-package, with emacs 23 it would show
version 1.0, and with emacs 21 it wouldn't be shown as not installable.

In the latter case, it would be good if a user could figure out why it's
not installable.

Of course, in the `need' dependency list, there could also be other
external packages.  For example, there could be

  (package "my-package"
    (version "1.0")
    (need ">=emacs-23" ">=foo-mode-0.7"))

so package.el on emacs 23 should show my-package 1.0 to be installable
only if some version >= 0.7 of foo-mode is installable, too.  When I
select it to be installed, then the newest foo-mode should be installed,
too.

So basically for each package version starting with the newest, the
packages in `need' have to be checked for availability/installability
recursively.  The first version where all needs can be satisfied, is the
one displayed.

Another issue is updating.  There the question is: Can I update package
A without breaking any other package that needs it?

Bye,
Tassilo

Re: Emacs Package Management

[Lennart Borgman]

On Thu, Sep 17, 2009 at 11:31 AM, Tassilo Horn <tassilo@member.fsf.org> wrote:
>>
>> The way most distros seems to solve this is to simply have different
>> repos for different distro versions. That should work for package.el
>> also. We could even adopt the stable/testing/bleeding scheme, so 3
>> repos for every supported major emacs version.
>
> I don't think that's a good approach, because it requires a lot of
> maintenance and testing on the package repository side.  It would be
> better if a package could specify something like
>
>  (package "my-package"
>    (version "1.0")
>    (need ">=emacs-23"))
>
>  (package "my-package"
>    (version "0.8")
>    (need ">=emacs-22.1" "<emacs-23"))
>
> and package.el does the resolving.


I agree, but I also think it should match against development sources
checkout date.

Re: Emacs Package Management

[Rupert Swarbrick]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Another issue is updating.  There the question is: Can I update package
> A without breaking any other package that needs it?

The package management system in Debian has two things to deal with this

1) Packages can explicitly depend on versions of their
   dependencies. This is mostly used to make libfoo-dev version 1.2.3
   only get installed with libfoo version 1.2.3

2) Sonames. When there is an ABI change in a library, there's supposed
   to be a "soname bump", which in Debian means actually changing the
   name of the package. It seems to me that one could simplify this
   slightly by having a "soname-version" field to perform the same
   function.

One thing to bear in mind is that shared libraries on unix are probably
a simpler problem than emacs libraries: I can install multiple versions
of libfoo and the dynamic linker will select the correct one for an
application. Trying to install multiple versions of an emacs package
would result in a godawful mess, because it's not just a list of
functions that can be called.

I don't know how one could solve that, so I suspect that the only
solution is to only allow one version of a package at once. I notice
that (for debian, at least) transitions between these incompatible
packages are non-trivial to organise at best.

Rupert

[-- Attachment #2: Type: application/pgp-signature, Size: 315 bytes --]

Re: Emacs Package Management

[Bob Rogers]

   From: Rupert Swarbrick <rswarbrick@gmail.com>
   Date: Thu, 17 Sep 2009 12:50:04 +0100

   . . .

   One thing to bear in mind is that shared libraries on unix are probably
   a simpler problem than emacs libraries: I can install multiple versions
   of libfoo and the dynamic linker will select the correct one for an
   application. Trying to install multiple versions of an emacs package
   would result in a godawful mess, because it's not just a list of
   functions that can be called.

   I don't know how one could solve that, so I suspect that the only
   solution is to only allow one version of a package at once . . .

   Rupert

This does not seem so hard.  If the "installed" location depended on the
Emacs version, e.g. by having the version number as part of the
directory name, then you could easily have one version of a package per
Emacs version.  The downside is that if you want to have all X versions
of Emacs on a given system find the "same" package, you would have to
install that package X times.  But this seems like a small price to pay
to be certain that each installed package version is known to work with
that version of Emacs -- and that it has been byte-compiled by the right
version of the byte compiler.

   FWIW, I don't usually have more than one version of Emacs per system.
So, for packages installed by the sysadmin, I think version-dependence
would rarely be an issue.  But if I'm not the sysadmin, and have to
install packages in my shared home directory, version-dependence could
be quite important if I need to use a variety of Emacsen on different
systems.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Emacs Package Management

[Rupert Swarbrick]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]

Bob Rogers <rogers-emacs@rgrjr.dyndns.org> writes:

>    From: Rupert Swarbrick <rswarbrick@gmail.com>
>    Date: Thu, 17 Sep 2009 12:50:04 +0100
>
>    . . .
>
>    One thing to bear in mind is that shared libraries on unix are probably
>    a simpler problem than emacs libraries: I can install multiple versions
>    of libfoo and the dynamic linker will select the correct one for an
>    application. Trying to install multiple versions of an emacs package
>    would result in a godawful mess, because it's not just a list of
>    functions that can be called.
>
>    I don't know how one could solve that, so I suspect that the only
>    solution is to only allow one version of a package at once . . .
>
>    Rupert
>
> This does not seem so hard.  If the "installed" location depended on the
> Emacs version, e.g. by having the version number as part of the
> directory name, then you could easily have one version of a package per
> Emacs version.  The downside is that if you want to have all X versions
> of Emacs on a given system find the "same" package, you would have to
> install that package X times.  But this seems like a small price to pay
> to be certain that each installed package version is known to work with
> that version of Emacs -- and that it has been byte-compiled by the right
> version of the byte compiler.

Ah, sorry, I don't think I was clear. When I said "package" above, I
meant elisp libraries. So, for example, maybe foo.el needs nxml version
<= n to work, but bar.el needs nxml version >= n+1. Is it possible to
install foo.el on my system?

Rupert

[-- Attachment #2: Type: application/pgp-signature, Size: 315 bytes --]

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Tassilo" == Tassilo Horn <tassilo@member.fsf.org> writes:

Tassilo> I don't think that's a good approach, because it requires a lot of
Tassilo> maintenance and testing on the package repository side.  It would be
Tassilo> better if a package could specify something like

Tassilo>   (package "my-package"
Tassilo>     (version "1.0")
Tassilo>     (need ">=emacs-23"))

It does do this.
Take a look at package.el.

Tassilo>   (package "my-package"
Tassilo>     (version "0.8")
Tassilo>     (need ">=emacs-22.1" "<emacs-23"))

I didn't implement "<" dependencies.
I didn't think this was worth the effort.

Tassilo> So when I run emacs 22, package.el would show version 0.8 of
Tassilo> my-package, with emacs 23 it would show version 1.0, and with
Tassilo> emacs 21 it wouldn't be shown as not installable.

I didn't do this, either.  I think few people bounce between Emacs
versions this way.

Tassilo> Another issue is updating.  There the question is: Can I update
Tassilo> package A without breaking any other package that needs it?

Yes.  Assuming package A is backward compatible.  package.el doesn't
check this, it has no way to.

Tom

Re: Emacs Package Management

[Tassilo Horn]

Tom Tromey <tromey@redhat.com> writes:

Hi Tom,

> Tassilo> So when I run emacs 22, package.el would show version 0.8 of
> Tassilo> my-package, with emacs 23 it would show version 1.0, and with
> Tassilo> emacs 21 it wouldn't be shown as not installable.
>
> I didn't do this, either.  I think few people bounce between Emacs
> versions this way.

The problem is not that people frequently switch versions, but that some
distros like RedHat Enterprise Linux still ship with Emacs 21.  For
example org-mode dropped emacs 21 support at 6.27, so version 6.26
should be displayed in package.el running in emacs 21, although the
newest version is 6.30.

Bye,
Tassilo

Re: Emacs Package Management

[Eric M. Ludlam]

Hi,

  The inversion tool I wrote a while back:
http://www.emacswiki.org/emacs/InVersion

  is part of CEDET which will eventually be part of Emacs.  I wrote it
due to the many version dependencies my users were dealing with.  It
supports the basic concepts displayed below.  It is used to validate
versions of save files, and versions of packages required by other
packages.  It does so based on a version stored in some Lisp file as a
defvar instead of some new declaration because that was something that
was prevalent in miscellaneous other sources I didn't write.

  Perhaps it would prove useful here too.

Eric

On Thu, 2009-09-17 at 11:31 +0200, Tassilo Horn wrote:
> joakim@verona.se writes:
> 
> >> But the newest package might not run on an older emacs version.  So
> >> it would be good if package.el would show the latest version of a
> >> package that is supposed to work with my emacs version, and if it
> >> requires other packages, then their versions have to be taken into
> >> account, too.  Does it do something like that?
> >
> > The way most distros seems to solve this is to simply have different
> > repos for different distro versions. That should work for package.el
> > also. We could even adopt the stable/testing/bleeding scheme, so 3
> > repos for every supported major emacs version.
> 
> I don't think that's a good approach, because it requires a lot of
> maintenance and testing on the package repository side.  It would be
> better if a package could specify something like
> 
>   (package "my-package"
>     (version "1.0")
>     (need ">=emacs-23"))
> 
>   (package "my-package"
>     (version "0.8")
>     (need ">=emacs-22.1" "<emacs-23"))
> 
> and package.el does the resolving.  So when I run emacs 22, package.el
> would show version 0.8 of my-package, with emacs 23 it would show
> version 1.0, and with emacs 21 it wouldn't be shown as not installable.
> 
> In the latter case, it would be good if a user could figure out why it's
> not installable.
> 
> Of course, in the `need' dependency list, there could also be other
> external packages.  For example, there could be
> 
>   (package "my-package"
>     (version "1.0")
>     (need ">=emacs-23" ">=foo-mode-0.7"))
> 
> so package.el on emacs 23 should show my-package 1.0 to be installable
> only if some version >= 0.7 of foo-mode is installable, too.  When I
> select it to be installed, then the newest foo-mode should be installed,
> too.
> 
> So basically for each package version starting with the newest, the
> packages in `need' have to be checked for availability/installability
> recursively.  The first version where all needs can be satisfied, is the
> one displayed.
> 
> Another issue is updating.  There the question is: Can I update package
> A without breaking any other package that needs it?
> 
> Bye,
> Tassilo
> 
> 

Re: Emacs Package Management

[Stefan Monnier]

>> But the newest package might not run on an older emacs version.  So it
>> would be good if package.el would show the latest version of a package
>> that is supposed to work with my emacs version, and if it requires other
>> packages, then their versions have to be taken into account, too.  Does
>> it do something like that?

> The way most distros seems to solve this is to simply have different
> repos for different distro versions.  That should work for package.el
> also.  We could even adopt the stable/testing/bleeding scheme, so 3 repos
> for every supported major emacs version.

FWIW, Debian does not solve this problem by using 3 different
repositories.  It uses "precise" version dependencies between
packages instead.


        Stefan

Re: Emacs Package Management

[Tom Tromey]

>>>>> "Tassilo" == Tassilo Horn <tassilo@member.fsf.org> writes:

Tassilo> But the newest package might not run on an older emacs version.
Tassilo> So it would be good if package.el would show the latest version
Tassilo> of a package that is supposed to work with my emacs version,
Tassilo> and if it requires other packages, then their versions have to
Tassilo> be taken into account, too.  Does it do something like that?

Right now, package.el defines a dummy package named "emacs" with a
version computed in the obvious way.  A package can then declare a
dependency on a minimum version of Emacs.

I forget offhand what exactly package.el does here.  I think it won't
let you install a package whose emacs dependency is not met (it has been
a while since I hacked on this so I don't remember its exact behavior).

In any case, it definitely won't activate such a package (you could get
one installed, for instance, by running Emacs 23, installing something,
and then running Emacs 21).

Tom

Re: Emacs Package Management

[Eric Schulte]

Phil Hagelberg <phil@hagelb.org> writes:
[...]
> * Emacs is already too big. If we made it easy to install third-party
>   packages, we could spin many packages already included in Emacs
>   (perhaps gnus, erc, and org) out into their own independent
>   projects. We would avoid problems like "the big gnus merge" that
>   happens every so often as well as allowing them to follow their own
>   release cycles. In addition, a lot of packages are simply fringe;
>   they're only useful to a small subset of Emacs users.

I think this point is increasingly important as people try to install
Emacs on mobile devices.  The use of a package system which tracks
dependencies between packages would be a great help to users who are
currently lopping off large parts of Emacs to fit it onto devices with
limited memory.

Re: Emacs Package Management

[Paul R]

Hi,

Stephen> Does anyone see a major flaw in a system like that? Or is it
Stephen> a matter of "show me the code and I'll comment"? ELPA could
Stephen> be the starting point.

Tom is the right person to ask, but I have the feeling that 2 points
at least need some special care :
 - autoloads, or more generally "package activation"
 - documentation, or "how to find in the Info system the entry to the
   documentation of the installed package"

Other than that, a package management system would be of course
incredibly useful to emacs. We may even, at some point, use it to
lower the minimum size of the emacs base distribution. I often hear
that emacs distribution bundles way too many packages. And I sometime
experience this feeling too :)

-- 
  Paul