Re: Integrating package.el

Re: Integrating package.el

[Ted Zlatanov]

On Sat, 02 Jan 2010 21:38:58 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> I've actually been thinking in even more incremental steps. Installing
PH> package.el in Emacs without altering any of the existing Emacs code
PH> would be an easy first step and would give some immediate benefit in
PH> terms of packages that are not included in Emacs.

Agreed.  I think it's mature enough and would benefit from the wider
exposure.  But at that point I think it becomes important to separate
the repository into "Emacs supported," "Emacs unsupported," "ELPA," and
any others deemed necessary.  Then the Emacs-related repositories can be
managed by Emacs maintainers and developers and hosted within the Emacs
repo while ELPA can remain independent.  

Can package.el support that?  It looks to me, from reading the current
source, that it's limited to a single repository.

PH> The next step would be to work on package submission. If the centralized
PH> system has a list of packages mapped to a list of DVCS repositories,
PH> they could be polled periodically and all tags matching a certain
PH> convention (say, starting with "v" and followed by a dotted number
PH> series) would be treated as package. That version would then be
PH> processed and published to a downloadable location for clients to pull
PH> in.

This should probably be a per-repository policy and process.

PH> I wasn't thinking about integrating packages that Emacs already contains
PH> until after these steps were complete. One thing that may be infeasible
PH> but would certainly simplify things a lot would be if we spun off
PH> packages like org-mode into their own separate DVCS repository and
PH> removed them from the Emacs source tree before making package.el treat
PH> them as packages. However, this may cause some unwanted chaos; I don't
PH> want to barge in and create a lot of work for people. It might also
PH> imply that network access would be necessary to perform a full build of
PH> Emacs since it would have to download bundled packages at compilation
PH> time. Not sure if that is a serious problem.

I don't know about the other packages that come with Emacs, but Gnus
could probably benefit from package.el-style installation (I don't speak
for the Gnus project, though, so I can bring it up on the mailing list
if necessary).  A better setup process for Gnus would be really nice,
though.  The usual pre-install and post-install scripts you find in RPM
or DEB would help.  I suspect many other packages would benefit from a
better setup process through package.el.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> Can package.el support that?  It looks to me, from reading the current
Ted> source, that it's limited to a single repository.

Yes, it is currently limited to a single repository.
This should not be too difficult to fix.

It also needs to be changed to be able to manage multiple install
locations, if it is to be used to manage site-lisp.

Ted> I don't know about the other packages that come with Emacs, but Gnus
Ted> could probably benefit from package.el-style installation (I don't speak
Ted> for the Gnus project, though, so I can bring it up on the mailing list
Ted> if necessary).  A better setup process for Gnus would be really nice,
Ted> though.  The usual pre-install and post-install scripts you find in RPM
Ted> or DEB would help.  I suspect many other packages would benefit from a
Ted> better setup process through package.el.

If you want to try packaging it, I can explain what you need to do.
It is usually quite easy.

Tom

Re: Integrating package.el

[Phil Hagelberg]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> Can package.el support that?  It looks to me, from reading the current
> Ted> source, that it's limited to a single repository.
>
> Yes, it is currently limited to a single repository.
> This should not be too difficult to fix.

I'm not sure why I left it out of my list of goals in my last email;
this is definitely high on my priority list.

I've also got a quick prototype of a DVCS-centric archive management
tool (for gathering packages from library authors and publishing them)
that I will post soon once I've fleshed it out a bit more.

> Ted> I don't know about the other packages that come with Emacs, but Gnus
> Ted> could probably benefit from package.el-style installation (I don't speak
> Ted> for the Gnus project, though, so I can bring it up on the mailing list
> Ted> if necessary).  A better setup process for Gnus would be really nice,
> Ted> though.  The usual pre-install and post-install scripts you find in RPM
> Ted> or DEB would help.  I suspect many other packages would benefit from a
> Ted> better setup process through package.el.
>
> If you want to try packaging it, I can explain what you need to do.
> It is usually quite easy.

I think in general this might be best done on an opt-in basis where the
maintainer of a library indicates that they'd like to convert to a
package and perhaps someone more familiar with package.el can help them
integrate.

-Phil

Re: Integrating package.el

[Lennart Borgman]

On Tue, Jan 5, 2010 at 6:02 AM, Phil Hagelberg <phil@hagelb.org> wrote:
> Tom Tromey <tromey@redhat.com> writes:
>
> I've also got a quick prototype of a DVCS-centric archive management
> tool (for gathering packages from library authors and publishing them)
> that I will post soon once I've fleshed it out a bit more.


This sounds good to me. I believe that ELPA should not hold the files
but rather have pointers to them. Otherwise the turnaround time will
be longer for changes made to the files.

Re: Integrating package.el

[Stefan Monnier]

>> Tom Tromey <tromey@redhat.com> writes:
>> I've also got a quick prototype of a DVCS-centric archive management
>> tool (for gathering packages from library authors and publishing them)
>> that I will post soon once I've fleshed it out a bit more.
> This sounds good to me. I believe that ELPA should not hold the files
> but rather have pointers to them. Otherwise the turnaround time will
> be longer for changes made to the files.

The default repositor(y|ies) will want to be under FSF control (not just
the copyright, but we also want to have a copy of the files, etc...) so
we can assume that the FSF will usually also hold the "master
repository" for those packages.

I.e. we probably don't want to only hold pointers to files.  Tho of
course, such an indirection might still be useful somewhere.


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 04 Jan 2010 12:51:06 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.

Good point, but do you mean multiple search locations for existing
installs?  I would expect a single install location, otherwise it gets
complicated...

Ted> I don't know about the other packages that come with Emacs, but Gnus
Ted> could probably benefit from package.el-style installation (I don't speak
Ted> for the Gnus project, though, so I can bring it up on the mailing list
Ted> if necessary).  A better setup process for Gnus would be really nice,
Ted> though.  The usual pre-install and post-install scripts you find in RPM
Ted> or DEB would help.  I suspect many other packages would benefit from a
Ted> better setup process through package.el.

Tom> If you want to try packaging it, I can explain what you need to do.
Tom> It is usually quite easy.

I think the pre-install and post-install steps are pretty important.
Without them, packaging Gnus doesn't do much.  I want the post-install
to actually set up the user's Gnus configuration.  This has been a very
common complaint about Gnus.  Installing it is the easy part, since it
comes with Emacs.  Can package.el support that?  I can't tell if this is
the "activate" or the "load" stage (are they states or state
transitions?  English can be so ambiguous...) or something new; also it
seems like this is something external to Gnus, a function of the ELPA
wrapper (although it may be bundled within Gnus) rather than something
Gnus will always run for new users.  This is the major question I have
before I propose this packaging on the Gnus mailing list.

I also want to know if you and Phil want to make Gnus your first
Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.
I'd actually like both: the Emacs-hosted version is "Emacs-supported
Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
"latest Gnus release."  The last one is usually but not always
synchronized with the Emacs release; I'll need to ask the other Gnus
developers what they think.  At least two Gnus versions in package.el
make sense in any case.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 10:06:56 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> Tom Tromey <tromey@redhat.com> writes:
>>> I've also got a quick prototype of a DVCS-centric archive management
>>> tool (for gathering packages from library authors and publishing them)
>>> that I will post soon once I've fleshed it out a bit more.
>> This sounds good to me. I believe that ELPA should not hold the files
>> but rather have pointers to them. Otherwise the turnaround time will
>> be longer for changes made to the files.

SM> The default repositor(y|ies) will want to be under FSF control (not just
SM> the copyright, but we also want to have a copy of the files, etc...) so
SM> we can assume that the FSF will usually also hold the "master
SM> repository" for those packages.

SM> I.e. we probably don't want to only hold pointers to files.  Tho of
SM> course, such an indirection might still be useful somewhere.

The "package file" (analogous to a RPM/DEB package fike) should contain
the real, final version of all the files.  The package repository may
accumulate metadata about all the package files it contains, but I
should be able to copy a single package file to another system and
install it.  As a sysadmin I don't want package files to be
indeterminate.  For instance, how can I set up a local mirror if some of
the files inside some of the package files are possibly remote?  There's
all the other security risks I listed in my previous note, too,
concerning remote network access.

This is a huge sysadmin problem with Perl for instance, where the
liberal packaging standard and complicated build process make it hard to
synchronize packages across multiple installations.  I've suffered
through that many times and hope it doesn't recur with Emacs packages.
Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
as a possibility.

Ted

Re: Integrating package.el

[Stefan Monnier]

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.
> Good point, but do you mean multiple search locations for existing
> installs?  I would expect a single install location, otherwise it gets
> complicated...

You need at least 2 install locations: one for user-installed packages
and one for admin-installed packages (the "site-lisp" mentioned above).

> I think the pre-install and post-install steps are pretty important.
> Without them, packaging Gnus doesn't do much.  I want the post-install
> to actually set up the user's Gnus configuration.  This has been a very

This is not part of an install, it's a separate step (a "configuration"
step).  Consider the case where the package is installed in site-lisp,
where it needs to work for all users.

"packaging Gnus" is only supposed to solve the problem of replacing
a version of Gnus with another independently from the version of Emacs
you use.

> seems like this is something external to Gnus, a function of the ELPA
> wrapper (although it may be bundled within Gnus) rather than something
> Gnus will always run for new users.  This is the major question I have
> before I propose this packaging on the Gnus mailing list.

I don't think it needs to be external to Gnus.  You're basically asking
for a Gnus wizard, I think.


        Stefan

Re: Integrating package.el

[Stefan Monnier]

> The "package file" (analogous to a RPM/DEB package fike) should contain
> the real, final version of all the files.  The package repository may
> accumulate metadata about all the package files it contains, but I
> should be able to copy a single package file to another system and
> install it.  As a sysadmin I don't want package files to be
> indeterminate.  For instance, how can I set up a local mirror if some of
> the files inside some of the package files are possibly remote?  There's
> all the other security risks I listed in my previous note, too,
> concerning remote network access.

> This is a huge sysadmin problem with Perl for instance, where the
> liberal packaging standard and complicated build process make it hard to
> synchronize packages across multiple installations.  I've suffered
> through that many times and hope it doesn't recur with Emacs packages.
> Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
> as a possibility.

OK, I'm not sure whether we're talking about the same things.  The way
I see it, there will be the following elements:
- a (set of) Bzr repository holding the package sources.
- a tool that will build tarballs from those sources.
- an area where those tarballs are stored, along with some metadata
  (think Debian repository).
- a tool that can scan such repositories and downlooad packages from
  them, obeying dependencies (think APT).
- a tool that can install/activate/uninstall a given package, ... (think
  DPKG).  This last one is what I tried to write when I wrote install.el
  many moons ago.


        Stefan

Re: Integrating package.el

[Phil Hagelberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I also want to know if you and Phil want to make Gnus your first
> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

I'm actually somewhat intimidated by the thought of packaging gnus. I'd
much rather start with supporting smaller packages first, but if someone
more familiar with hacking gnus wants to start work on packaging it I
would be happy to provide support.

-Phil

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 10:03:18 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I also want to know if you and Phil want to make Gnus your first
>> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

PH> I'm actually somewhat intimidated by the thought of packaging gnus. I'd
PH> much rather start with supporting smaller packages first, but if someone
PH> more familiar with hacking gnus wants to start work on packaging it I
PH> would be happy to provide support.

I'll go to the Gnus mailing list and discuss this further, but at least
for now let's assume Reiner or I will make the package.  I don't mind
doing the work, though I can't promise I'll get it done quickly.  I
still need to know about the 2+ versions of Gnus that I mentioned
earlier--how do you want to deal with that?  Also, how do you want to
package an application that's already part of Emacs for the
"Emacs-hosted Gnus" version?

Thanks
Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.

Ted> Good point, but do you mean multiple search locations for existing
Ted> installs?  I would expect a single install location, otherwise it gets
Ted> complicated...

Basically we need to handle 3 install locations: the Emacs install tree,
the global site-lisp, and the user's install location.

These only need to be handled during the activation step.  When the user
installs a new package, it should always go to his personal install
location.

Right now package.el handles the Emacs install tree in a hacky way and
ignores site-lisp entirely.

Ted> This has been a very
Ted> common complaint about Gnus.  Installing it is the easy part, since it
Ted> comes with Emacs.  Can package.el support that?  I can't tell if this is
Ted> the "activate" or the "load" stage (are they states or state
Ted> transitions?  English can be so ambiguous...) or something new; also it
Ted> seems like this is something external to Gnus, a function of the ELPA
Ted> wrapper (although it may be bundled within Gnus) rather than something
Ted> Gnus will always run for new users.  This is the major question I have
Ted> before I propose this packaging on the Gnus mailing list.

This would be a new thing.

I agree with the other posters who recommend that this be done in Gnus,
not as something related to package.el.

Ted> I also want to know if you and Phil want to make Gnus your first
Ted> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

FWIW, ELPA already includes some packages that are also included in
Emacs.  I maintain the necessary metadata for this by hand (and, I'm
afraid, not very well at the moment due to lack of time -- I think I am
missing some updates).

Ted> I'd actually like both: the Emacs-hosted version is "Emacs-supported
Ted> Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
Ted> "latest Gnus release."  The last one is usually but not always
Ted> synchronized with the Emacs release; I'll need to ask the other Gnus
Ted> developers what they think.  At least two Gnus versions in package.el
Ted> make sense in any case.

package.el doesn't support this at the moment.

Thanks for bringing this up, though.  I think it is pretty important to
flush out all these use cases.

I can think of a couple solutions to this problem.

One would be to somehow let users select different package versions.
Normally, though, users don't actually want this -- when a new version
is released, ordinarily all the old ones are obsolete.

Another solution would be to have two repositories, one for stable
packages and one for experimental.  This wouldn't require any changes to
package.el.

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 12:14:14 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> Right now package.el handles the Emacs install tree in a hacky way
Tom> and ignores site-lisp entirely.

OK, I hope you and Phil have some ideas.  I don't know much about this.

Tom> I agree with the other posters who recommend that [post-install
Tom> configuration] be done in Gnus, not as something related to
Tom> package.el.

I would like to at least show a message or offer a prompt (answering 'y'
launches the assistant), is that possible?  If not that's OK.

Ted> I'd actually like both: the Emacs-hosted version is "Emacs-supported
Ted> Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
Ted> "latest Gnus release."  The last one is usually but not always
Ted> synchronized with the Emacs release; I'll need to ask the other Gnus
Ted> developers what they think.  At least two Gnus versions in package.el
Ted> make sense in any case.

Tom> package.el doesn't support this at the moment.

Tom> Thanks for bringing this up, though.  I think it is pretty important to
Tom> flush out all these use cases.

Tom> I can think of a couple solutions to this problem.

Tom> One would be to somehow let users select different package versions.
Tom> Normally, though, users don't actually want this -- when a new version
Tom> is released, ordinarily all the old ones are obsolete.

This is probably too confusing for most users.

Tom> Another solution would be to have two repositories, one for stable
Tom> packages and one for experimental.  This wouldn't require any changes to
Tom> package.el.

This works for me.  We're talking now about the proposed FSF supported
repository (part of Emacs), not about an ELPA "stable" repository,
right?  The bleeding-edge Gnus can live in the external ELPA archive
that's currently live, and perhaps can move to the FSF unsupported
repository we discussed earlier (you and Phil may want different names
for the repositories).

I think RMS, Stefan, Reiner, and you and Phil need to agree on the setup
and where things will live.  Jonas will probably have an opinion as
well.  I can do the mechanical work for packaging Gnus but can't make
this kind of policy decision.  Whatever the decision, Gnus is a good
test case because of its complexity and baggage[1].

Ted

[1] by "baggage" I mean that Gnus has a handbag, two purses, six
wallets, a rolling suitcase, a laundry basket, and also stuffs things in
its socks "just in case." :)

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 11:47:46 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The "package file" (analogous to a RPM/DEB package fike) should contain
>> the real, final version of all the files.  The package repository may
>> accumulate metadata about all the package files it contains, but I
>> should be able to copy a single package file to another system and
>> install it.  As a sysadmin I don't want package files to be
>> indeterminate.  For instance, how can I set up a local mirror if some of
>> the files inside some of the package files are possibly remote?  There's
>> all the other security risks I listed in my previous note, too,
>> concerning remote network access.

>> This is a huge sysadmin problem with Perl for instance, where the
>> liberal packaging standard and complicated build process make it hard to
>> synchronize packages across multiple installations.  I've suffered
>> through that many times and hope it doesn't recur with Emacs packages.
>> Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
>> as a possibility.

SM> OK, I'm not sure whether we're talking about the same things.

I digressed too much, sorry.

SM> The way I see it, there will be the following elements:

SM> - a (set of) Bzr repository holding the package sources.

Yes, with initial members "FSF supported," "FSF unsupported" (both
hosted inside the Emacs Bzr repo) and "ELPA" (hosted externally, may not
be a Bazaar repository at all).  This is the storage backend.  The "FSF
supported" storage may be the lisp/ directory inside Emacs, for
instance.

SM> - a tool that will build tarballs from those sources.

Yes, including the necessary package metadata inside the file.  This
tool will probably come from ELPA.

SM> - an area where those tarballs are stored, along with some metadata
SM>   (think Debian repository).

All of this is MHO: there should be three areas to parallel the storage
backends above.  They may be stored inside the backend or externally.
The package repositories should be identified by a single URL; the ones
that come with Emacs should point to a secure Savannah URL.  That may
address RMS' concerns about loading software over the network.  Jonas'
emacsmirror.org would be a fourth package repository, probably not
enabled by default but easy to enable.

SM> - a tool that can scan such repositories and downlooad packages from
SM>   them, obeying dependencies (think APT).

This is package.el IIUC.

SM> - a tool that can install/activate/uninstall a given package, ... (think
SM>   DPKG).  This last one is what I tried to write when I wrote install.el
SM>   many moons ago.

This is also package.el IIUC.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I would like to at least show a message or offer a prompt (answering 'y'
Ted> launches the assistant), is that possible?  If not that's OK.

It is sort of possible.

You can do anything you like in an autoload.  These will be evaluated
when the package is activated, which does happen when the package is
installed.

I think this would be pretty unfriendly to users, though.  For example,
what if the user installs several packages at once?

Tom

Re: Integrating package.el

[Jonas Bernoulli]

2010/1/5 Ted Zlatanov <tzz@lifelogs.com>:
> All of this is MHO: there should be three areas to parallel the storage
> backends above.  They may be stored inside the backend or externally.
> The package repositories should be identified by a single URL; the ones
> that come with Emacs should point to a secure Savannah URL.  That may
> address RMS' concerns about loading software over the network.  Jonas'
> emacsmirror.org would be a fourth package repository, probably not
> enabled by default but easy to enable.

Sounds good to me. I by now means expect my mirror to be enabled by default,
but it would certainly be nice if that were possible.

-- Jonas

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 16:19:56 -0700 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> I would like to at least show a message or offer a prompt (answering 'y'
Ted> launches the assistant), is that possible?  If not that's OK.

Tom> It is sort of possible.

Tom> You can do anything you like in an autoload.  These will be evaluated
Tom> when the package is activated, which does happen when the package is
Tom> installed.

Tom> I think this would be pretty unfriendly to users, though.  For example,
Tom> what if the user installs several packages at once?

OK, if it doesn't fit with the package.el flow, let's drop it.  Stefan
and Reiner are also not crazy about the idea so it's probably a bad one
:)

I'll make the assistant run on the first Gnus startup.  There's several
other issues I raised, but at least this one can be safely ignored.

Ted

Re: Integrating package.el

[Richard Stallman]

    > The package repositories should be identified by a single URL; the ones
    > that come with Emacs should point to a secure Savannah URL.  That may
    > address RMS' concerns about loading software over the network.  

Alas that doesn't affect the issue.  Running software directly from
the network is bad because it is a process that invites users to give
up control.

Who wrote the program makes no difference, not for this.
Where the program comes from makes no difference.
Suppose I wrote the program and people fetch it from gnu.org:
that makes no difference.  It is still bad for people to
adopt a practice that gives them less control.

So we won't lead people in that direction.

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 06 Jan 2010 15:45:37 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> The package repositories should be identified by a single URL; the ones
>> that come with Emacs should point to a secure Savannah URL.  That may
>> address RMS' concerns about loading software over the network.  

RS> Alas that doesn't affect the issue.  Running software directly from
RS> the network is bad because it is a process that invites users to give
RS> up control.

RS> Who wrote the program makes no difference, not for this.
RS> Where the program comes from makes no difference.
RS> Suppose I wrote the program and people fetch it from gnu.org:
RS> that makes no difference.  It is still bad for people to
RS> adopt a practice that gives them less control.

RS> So we won't lead people in that direction.

Sorry for confusing the issue between the package.el discussion and
Lennart's discussion.  I was talking about installing software, not
running it directly from the repository.  I would expect packages to be
signed by the package repository maintainer as well, ensuring integrity.

Ted

Re: Integrating package.el

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The way I see it, there will be the following elements:
> - a (set of) Bzr repository holding the package sources.
> - a tool that will build tarballs from those sources.
> - an area where those tarballs are stored, along with some metadata
>   (think Debian repository).
> - a tool that can scan such repositories and downlooad packages from
>   them, obeying dependencies (think APT).
> - a tool that can install/activate/uninstall a given package, ... (think
>   DPKG).  This last one is what I tried to write when I wrote install.el
>   many moons ago.

Here are the pieces to the puzzle from my initial brainstorming:

* An index file contains a alist mapping projects to DVCS
  repositories. This would be manually curated. Adding a project would
  just be done by adding an entry to this file.

* A function that builds packages for every version in the repository of
  each project. A version is defined by a tag matching a given pattern.

* The packages are deployed to an archive source, which is publicly
  available over HTTP.

* Currently package.el can perform the last two points from your list
  (scan/download/dependencies and install/activate/uninstall).

The attached file implements the server-side maintenance, albeit in a
rough manner. It's limited to single-file packages kept in git
repositories, simply because that's what I'm personally familiar with,
though I don't think supporting other DVCSes would be tricky at all.

I welcome comments on this. Once the server-side maintenance tool is
ready, I want to add support for multiple sources to the client-side
package.el, and once that's done supporting multiple install locations
on disk (as Tom has explained) would be the next step.

Right now the code is at http://github.com/technomancy/package.el,
though I've attached a copy of the relevant file.

-Phil


[-- Attachment #2: package-maint.el --]
[-- Type: application/emacs-lisp, Size: 7092 bytes --]

Re: Integrating package.el

[Richard Stallman]

    * An index file contains a alist mapping projects to DVCS
      repositories. This would be manually curated. Adding a project would
      just be done by adding an entry to this file.

This could be ok, but that shouldn't be misunderstood as a decision
to include libraries that we don't have papers for.

Re: Integrating package.el

[Phil Hagelberg]

Richard Stallman <rms@gnu.org> writes:

>     * An index file contains a alist mapping projects to DVCS
>       repositories. This would be manually curated. Adding a project would
>       just be done by adding an entry to this file.
>
> This could be ok, but that shouldn't be misunderstood as a decision
> to include libraries that we don't have papers for.

Naturally, it's up to whoever manually curates the index to ensure the
package follows the guidelines for that archive. For the FSF archive(s)
it would mean ensuring copyright assignment is in place; for folks who
want to maintain their own archives it would probably be something
simpler like making sure it's licensed as free software.

-Phil

Re: Integrating package.el

[Richard Stallman]

    Naturally, it's up to whoever manually curates the index to ensure the
    package follows the guidelines for that archive. For the FSF archive(s)
    it would mean ensuring copyright assignment is in place; for folks who
    want to maintain their own archives it would probably be something
    simpler like making sure it's licensed as free software.

I am worried that this will lead to a problem.
Namely, there may be increasing numbers of packages
that we can't get adequate papers for and can't put into Emacs.

Re: Integrating package.el

[Stephen J. Turnbull]

Richard Stallman writes:
 >     Naturally, it's up to whoever manually curates the index to ensure the
 >     package follows the guidelines for that archive. For the FSF archive(s)
 >     it would mean ensuring copyright assignment is in place; for folks who
 >     want to maintain their own archives it would probably be something
 >     simpler like making sure it's licensed as free software.
 > 
 > I am worried that this will lead to a problem.
 > Namely, there may be increasing numbers of packages
 > that we can't get adequate papers for and can't put into Emacs.

This is a problem as opposed to increasing numbers of packages that
are available only to their authors and you can't put into Emacs?  At
least this way you'll have a contact address for authors.

Re: Integrating package.el

[Chong Yidong]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>  >     Naturally, it's up to whoever manually curates the index to ensure the
>  >     package follows the guidelines for that archive. For the FSF archive(s)
>  >     it would mean ensuring copyright assignment is in place; for folks who
>  >     want to maintain their own archives it would probably be something
>  >     simpler like making sure it's licensed as free software.
>  > 
>  > I am worried that this will lead to a problem.
>  > Namely, there may be increasing numbers of packages
>  > that we can't get adequate papers for and can't put into Emacs.
>
> This is a problem as opposed to increasing numbers of packages that
> are available only to their authors and you can't put into Emacs?  At
> least this way you'll have a contact address for authors.

No, it's a valid concern.  It's hard work hunting down papers and
integrating packages.  We must make sure a package system does not tempt
us to just "dump and forget".

Re: Integrating package.el

[joakim]

Chong Yidong <cyd@stupidchicken.com> writes:

> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
>> Richard Stallman writes:
>>  >     Naturally, it's up to whoever manually curates the index to ensure the
>>  >     package follows the guidelines for that archive. For the FSF archive(s)
>>  >     it would mean ensuring copyright assignment is in place; for folks who
>>  >     want to maintain their own archives it would probably be something
>>  >     simpler like making sure it's licensed as free software.
>>  > 
>>  > I am worried that this will lead to a problem.
>>  > Namely, there may be increasing numbers of packages
>>  > that we can't get adequate papers for and can't put into Emacs.
>>
>> This is a problem as opposed to increasing numbers of packages that
>> are available only to their authors and you can't put into Emacs?  At
>> least this way you'll have a contact address for authors.
>
> No, it's a valid concern.  It's hard work hunting down papers and
> integrating packages.  We must make sure a package system does not tempt
> us to just "dump and forget".

A well made package system might also conceivably make it easier to
acquire papers right?

For instance, it might be more prestigious to have ones package in the
FSF repos.

Also, digital signatures might be legally binding at some point. (I think
they are in Swedenalready, we have a national system for it anyway)

>
-- 
Joakim Verona

Re: Integrating package.el

[Phil Hagelberg]

Chong Yidong <cyd@stupidchicken.com> writes:

>>> I am worried that this will lead to a problem.
>>> Namely, there may be increasing numbers of packages
>>> that we can't get adequate papers for and can't put into Emacs.
>>
>> This is a problem as opposed to increasing numbers of packages that
>> are available only to their authors and you can't put into Emacs?  At
>> least this way you'll have a contact address for authors.
>
> No, it's a valid concern.  It's hard work hunting down papers and
> integrating packages.  We must make sure a package system does not tempt
> us to just "dump and forget".

Can you explain what you mean by this? Are you saying Emacs perhaps
should not have a package manager, or just that we need to be careful
about what packages we allow into the repository?

I'm going to continue my work on package.el whether or not it is
accepted into Emacs, but it would be a shame to make users go through
extra installation steps to use it.

If package.el is included, I will make every effort to assign copyright
for the projects I maintain for which I can, but there are some for
which it simply won't be possible due to my accepting patches by email
and then losing some of my old email archives in a hard drive crash. I
would hate for this situation to impede adding useful features to Emacs.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

joakim@verona.se writes:

>> No, it's a valid concern.  It's hard work hunting down papers and
>> integrating packages.  We must make sure a package system does not tempt
>> us to just "dump and forget".
>
> A well made package system might also conceivably make it easier to
> acquire papers right?
>
> For instance, it might be more prestigious to have ones package in the
> FSF repos.

In my mind it has little to do with prestige and more to do with
streamlining the installation process for users of my projects. Being
able to tell users "install $FOO through package.el" rather than giving
a many-step install process means it will be easier for users to
discover and try out my projects, not to mention keeping them up to date
with the latest version. For this benefit I (and surely many other
package authors) would be willing to put up with the hassle of paperwork
where possible, thereby increasing the number of packages for which the
FSF owns copyright.

-Phil

Re: Integrating package.el

[Stefan Monnier]

> I'm going to continue my work on package.el whether or not it is
> accepted into Emacs, but it would be a shame to make users go through
> extra installation steps to use it.

Having myself written a package manager (install.el), I'm clearly fully
in favor of integrating something like package.el.
I don't think there's necessarily much work to be done to integrate it.

E.g. It is not necessary to set up a VCS area to keep the package
sources (and other such things) before integrating package.el.

Basically, all I want is for it to be able to distinguish between
"packages available locally, either installed by the user or the
sysadmin", and "packages that will be autoloaded in a given session", so
as to be able to install different versions of the same package at the
same time.


        Stefan

Re: Integrating package.el

[MON KEY]

Joakim Verona writes,
> A well made package system might also conceivably make it easier to
> acquire papers right?
>
> For instance, it might be more prestigious to have ones package in the
> FSF repos.
>
> Also, digital signatures might be legally binding at some point. (I think
> they are in Swedenalready, we have a national system for it anyway)

I don't get why the licensing/assignment need necessarily be an issue.
For example, I was issued a `.pdf' file for my assignment.

The file name has format: `<LASTNAME>.<NUMBER>.EMACS.pdf'

Following the letterhead on pg 1 are the lines:

,----
|                                       SURNAME-UID  <--BLOCK-1
|                                         RT NNNNNN  <--BLOCK-2
| ASSIGNMENT - GNU EMACS                             <--BLOCK-3
`----

 Following that, the opening paragraph reads:

,----
| For $1 and other good { boilerplate elided }
| hereinafter "FSF", <MY-FULL-NAME>, hereinafter      <--BLOCK-4
| "DEVELOPER", hereby agrees as follows:
`----

The bottom of page two has two signature blocks:

,----
|      Thank you for the contribution!
|
| Signed:
|
| __<MY_SIGNATURE>__
|                                                    <--BLOCK-5
| __<DATE_SIGNED>___
|
|
| Accepted by the Free Software Foundation
|
| __<EXECUTIVE_DIR_SIG__
|                                                    <--BLOCK-6
| __<DATE_SIGNED>_______
|
`----

The scan also has an inked date stamp which appears in the manner of a Bates
number.

:NOTE The above template is recent, obv. older such docs may differ.

While I'm quite sure the FSF has facilities to OCR/digitize this type of content
in bulk it is surely outside their purview to extend such facility (and time
required to do so) to the Emacs project or any other GNU project FTM.

That said, Joakim's has developed dragbox.el which using GOCR should already be
capable of the following:

o Extracting digital images key portions of the document Blocks 1-6,

o Extracting OCR'd conversion of Blocks 1-4,

The `Bates Date stamp' (BLOCK-7) would prob. need to be manually identified and
extracted as these most likely appear skewed, incompletely inked, and
arbitrarily placed for the majority of assignment papers. Regardless, Joakim's
dragbox.el already provides adequate facilities for accommodating a manual crops
of BLOCK-7 as well.

With such extractions, the digital image portions of Blocks 1-7, and the OCR'd
conversion of Blocks 1-4, a `sanctioned repository maintainer' would
key-sign/encrypt
each block (or the entire bundle) of extracted content for:

o The package author(s);

o The Emacs-devels;

This encrypted bundle could be and distributed with the package or retained
separately.  In either case, the bundle would receive a Unique ID which would be
included in the header commentary of each file distributed with the package.

Assuming all the elements of this regime are technically possible what are the
benefits it would provide that other approaches wouldn't:

o It promotes user awareness of, and re-assures the user benefits of, using a
  suite of congruent and consistently licensed packages.

o It promotes and encourages package authors to acquire assignment _papers_ in
  addition to incorporating boilerplate license terms in package/library
  headers.

o Once an authors assignment papers have been processed per above it provides a
  mechanism whereby package authors can readily integrate new libraries with a
  sanctioned repository without additional overhead. IOW assuming no change of
  employment, surname, etc. an author simply notifies the repository maintainer
  that a new package exists and that said new package should fall under the
  existing Unique ID umbrella.

o Changes in assignment status e.g. withdrawal of assignment, change of
  employment, contact address, surname, etc. can be more readily propagates
  through the system. The repository maintainer is notified and alters the scope
  of the Unique ID umbrella accordingly; this alteration is made known to users
  and devels upon next synchronization.

o It can be integrated with existing assignment papers dating back to ???.

o It reinforces the existing practice that assignment papers need be signed on
  physical media (e.g. _paper_).

o It allows upstream Emacs-devels to verify assignment should they elect to
  incorporate a package or portions of code therein with less burdensome
  overhead.

o It may remove some of the initial immediate overhead Emacs-devels face with
  regards incorporation of new features in the absence of existing assignment
  papers. IOW going forward new packages/authors will presumably already have
  acquired papers and these papers will already have been converted processed.

o It may ease management of existing/future assignment papers w/re upstream
  Emacs-devels, i.e. by breaking up the assignment document into discrete
  elements Blocks 1-7 could be databased in any number of ways including xrefs
  across multiple authors/packages, verification of date and timestamp
  incongruities etc.

Assuming the above are points are indeed beneficial what are the detractors:

o It places a burden on package authors that may not have been there before;
  namely, acquisition of assignment papers.

o For new authors there is a time-delay between the initial creation of
  package/source and any sequencing of assignment papers, digitization,
  propagation etc. required before a package can become part of a
  repository. IOW An author can publish code with a GPL header to github,
  bit-bucket, launchpad, etc. in seconds.

o It places a significant burden on `sanctioned repository maintainers' to
  accomplish the initial digital conversion and subsequent propagation to
  appropriate parties.

o It may place a burden on FSF if it has to process more assignment papers.

o It doesn't (currently) address the issue of older assignment papers that are
  in a different format, are lost, out of date, etc.

o It doesn't accommodate purely digital code signing models.

o It doesn't accommodate alternative licenses (philosophical negative).

o It places a burden on users to grok the benefits/rationale of what in other
  situations would be considered an esoteric approach, i.e. One can _fax_ a
  contract that is binding but authors/contributors of a premier piece of
  software must make available hand signed media.

o None of this exists yet.

Personally, I believe there is a net gain for package authors when there is a
cost of entry to a project.  AFAIK Right now there isn't a cost of entry because
there isn't really an entry point and what little there is places the burden on
CYD, Stefan, et al to manage the assignment but that this generally takes a
backseat to other more pressing concerns... So, some neat stuff lingers because
it is a chore to manage the paperwork (real or metaphorical).

Likewise, I believe there is a net gain for community package repository
maintainers when there is some cost of entry to accept/process a package. There
are many examples of projects which maintain `unofficial repositories' of third
party tools. Those that provide a degree of oversight seem to be more
trustworthy and propagate more reliable code than those that to quote Dennis
Hopper in the movie Blue Velvet, "F**k anything that moves". When package
authors understand that some legwork is required of them in order to play they
may be more apt to contribute more robust and tested code (as contributions
aren't worth the trouble otherwise) where this is the case repository
maintainers will benefit from less need to troubleshoot, and less need for
frequent integration of code changes.

Most likely there will continue to be more than one model of Emacs community
developed and distributed packages. These will either compete for
attention/resources or each will offer appropriate paths to increased
integration with Emacs.  In a competitive environment (read forking) environment
the repo with the _most_ packages will thrive i.e. "Worse is Better" even
if/when it isn't.  In a cooperative/integrated/tired community repository
environment the repo with the best, cleanest, most open, code will achieve a
higher "status" than her fast and loose neighbor.

IOW repository maintainers have no incentive to adopt the above assignment
clearing regime where a competing repo environment is afforded an equal status
by the upstream Emacs-devels. However, where there is some upstream endorsement
preference for a particular _regime_ then disparate repo maintainers and
package authors can participate with, promote, and adopt this regime (or not)
according to their particular philosophy, time restraints, personalities, etc.

/s_P\

Re: Integrating package.el

[Phil Hagelberg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm going to continue my work on package.el whether or not it is
>> accepted into Emacs, but it would be a shame to make users go through
>> extra installation steps to use it.
>
> Having myself written a package manager (install.el), I'm clearly fully
> in favor of integrating something like package.el.
> I don't think there's necessarily much work to be done to integrate it.
>
> E.g. It is not necessary to set up a VCS area to keep the package
> sources (and other such things) before integrating package.el.

True, this is not of the highest importance since the steps it takes can
still be done manually without too much tedium.

> Basically, all I want is for it to be able to distinguish between
> "packages available locally, either installed by the user or the
> sysadmin", and "packages that will be autoloaded in a given session", so
> as to be able to install different versions of the same package at the
> same time.

OK, I will prioritize this once I finish adding support for multiple
archive sources. I think we'll need this in order to support separating
out fully-supported packages vs unsupported packages for which we simply
have copyright assignment as Ted has suggested.

thanks,
Phil

Re: Integrating package.el

[Stephen J. Turnbull]

Chong Yidong writes:

 > > This is a problem as opposed to increasing numbers of packages that
 > > are available only to their authors and you can't put into Emacs?  At
 > > least this way you'll have a contact address for authors.
 > 
 > No, it's a valid concern.  It's hard work hunting down papers and
 > integrating packages.

Nobody said otherwise.  I am painfully aware of the costs of both.

 > We must make sure a package system does not tempt us to just "dump
 > and forget".

Too late.  By refusing to handle registration of 3rd-party packages
itself, Emacs has been "dumping and forgetting" for decades.  That
hasn't hindered growth of repositories of unassigned code in
any way, it has just created a class system and contributed to
integration problems for maintainers of packages like AUCTeX who do
exercise due diligence ... but they arrived on the scene too late.

The temptation is a *human* problem, not the system's, and refusing to
list unassigned packages is just as much succumbing to the temptation
as "forgetting" to chase papers for listed packages.  At least if you
list the packages, you have the goad of shame to go chasing the papers.

Re: Integrating package.el

[Stefan Monnier]

> OK, I will prioritize this once I finish adding support for multiple
> archive sources. I think we'll need this in order to support separating
> out fully-supported packages vs unsupported packages for which we simply
> have copyright assignment as Ted has suggested.

Feel free to set your own priorities, but from where I stand, a single
repository would already be useful (used for the packages we'd distribute
separately from Emacs).


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Fri, 08 Jan 2010 21:40:31 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> The attached file implements the server-side maintenance, albeit in a
PH> rough manner. It's limited to single-file packages kept in git
PH> repositories, simply because that's what I'm personally familiar with,
PH> though I don't think supporting other DVCSes would be tricky at all.

PH> I welcome comments on this. Once the server-side maintenance tool is
PH> ready, I want to add support for multiple sources to the client-side
PH> package.el, and once that's done supporting multiple install locations
PH> on disk (as Tom has explained) would be the next step.

Technically it seems fine.  I wouldn't mind working with it.  The git
calls could perhaps be abstracted to the VC equivalents, though I don't
know the Emacs VC well at all.

Ted

Re: Integrating package.el

[Phil Hagelberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 08 Jan 2010 21:40:31 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 
>
> PH> The attached file implements the server-side maintenance, albeit in a
> PH> rough manner. It's limited to single-file packages kept in git
> PH> repositories, simply because that's what I'm personally familiar with,
> PH> though I don't think supporting other DVCSes would be tricky at all.
>
> PH> I welcome comments on this. Once the server-side maintenance tool is
> PH> ready, I want to add support for multiple sources to the client-side
> PH> package.el, and once that's done supporting multiple install locations
> PH> on disk (as Tom has explained) would be the next step.
>
> Technically it seems fine.  I wouldn't mind working with it.  The git
> calls could perhaps be abstracted to the VC equivalents, though I don't
> know the Emacs VC well at all.

Unfortunately the abstracted VC functionality doesn't currently support
creating new checkouts/cloning, which is why I didn't investigate that
much further. (Plus I know of 20+ elisp libs in git and only 1 or 2 in
bzr.) It may not be a common enough operation to justify adding to VC,
or maybe it's a long-standing TODO that nobody's tackled yet.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.
>
> Feel free to set your own priorities, but from where I stand, a single
> repository would already be useful (used for the packages we'd distribute
> separately from Emacs).

Yeah, I've already started this work, so I think it won't take too long
to wrap up. Plus it will be useful to users of package.el even before it
gets merged into Emacs.

-Phil

[trimmed cc list]

Re: Integrating package.el

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Phil Hagelberg <phil@hagelb.org> writes:

>> Basically, all I want is for it to be able to distinguish between
>> "packages available locally, either installed by the user or the
>> sysadmin", and "packages that will be autoloaded in a given session", so
>> as to be able to install different versions of the same package at the
>> same time.
>
> OK, I will prioritize this once I finish adding support for multiple
> archive sources. I think we'll need this in order to support separating
> out fully-supported packages vs unsupported packages for which we simply
> have copyright assignment as Ted has suggested.

I've just implemented support for multiple package sources in
package.el. Right now it is preconfigured to use ELPA, but the following
snippet will let it pull in packages from my own archive source as well:

(add-to-list 'package-archives
             '("technomancy" . "http://repo.technomancy.us/emacs/") t)

Hit M-x package-list-packages to see all available packages. The "bork"
and "bingle" packages are dummies that are only available from my
repository, and should be available for installation at the same time as
the standard ELPA-provided packages.

I'd love to see some other elisp library authors try to set up their own
package archive sources using package-maint.el and test the
multiple-archive support.

I hope that a new version of package.el incorporating my changes could
be pushed out to ELPA and then possibly included in Emacs once a little
more work has gone into integration with the Emacs load process.

-Phil


[-- Attachment #2: package.el --]
[-- Type: application/emacs-lisp, Size: 55248 bytes --]

[-- Attachment #3: package-maint.el --]
[-- Type: application/emacs-lisp, Size: 7552 bytes --]

Re: Integrating package.el

[Dan Nicolaescu]

Please add a menu to package.el.
This patch vs an older version of package.el does that.  It should still apply.


--- package.el	2009-09-19 11:33:11.000000000 -0700
+++ package.el.with-menu	2009-09-19 11:31:00.000000000 -0700
@@ -1129,28 +1129,71 @@
 
 ;;;; Package menu mode.
 
-(defvar package-menu-mode-map nil
+(defvar package-menu-mode-map
+  (let ((map (make-keymap))
+	(menu-map (make-sparse-keymap "Package")))
+    (suppress-keymap map)
+    (define-key map "q" 'quit-window)
+    (define-key map "n" 'next-line)
+    (define-key map "p" 'previous-line)
+    (define-key map "u" 'package-menu-mark-unmark)
+    (define-key map "\177" 'package-menu-backup-unmark)
+    (define-key map "d" 'package-menu-mark-delete)
+    (define-key map "i" 'package-menu-mark-install)
+    (define-key map "g" 'package-menu-revert)
+    (define-key map "r" 'package-menu-refresh)
+    (define-key map "~" 'package-menu-mark-obsolete-for-deletion)
+    (define-key map "x" 'package-menu-execute)
+    (define-key map "h" 'package-menu-quick-help)
+    (define-key map "?" 'package-menu-view-commentary)
+    (define-key map [menu-bar package-menu] (cons "Package" menu-map))
+    (define-key menu-map [mq]
+      '(menu-item "Quit" quit-window
+		  :help "Quit package selection"))
+    (define-key menu-map [s1] '("--"))
+    (define-key menu-map [mn]
+      '(menu-item "Next" next-line
+		  :help "Next Line"))
+    (define-key menu-map [mp]
+      '(menu-item "Previous" previous-line
+		  :help "Previous Line"))
+    (define-key menu-map [s2] '("--"))
+    (define-key menu-map [mu]
+      '(menu-item "Unmark" package-menu-mark-unmark
+		  :help "Clear any marks on a package and move to the next line"))
+    (define-key menu-map [munm]
+      '(menu-item "Unmark backwards" package-menu-backup-unmark
+		  :help "Back up one line and clear any marks on that package"))
+    (define-key menu-map [md]
+      '(menu-item "Mark for deletion" package-menu-mark-delete
+		  :help "Mark a package for deletion and move to the next line"))
+    (define-key menu-map [mi]
+      '(menu-item "Mark for install" package-menu-mark-install
+		  :help "Mark a package for installation and move to the next line"))
+    (define-key menu-map [s3] '("--"))
+    (define-key menu-map [mg]
+      '(menu-item "Update package list" package-menu-revert
+		  :help "Update the list of packages"))
+    (define-key menu-map [mr]
+      '(menu-item "Refresh package list" package-menu-refresh
+		  :help "Download the ELPA archive"))
+    (define-key menu-map [s4] '("--"))
+    (define-key menu-map [mt]
+      '(menu-item "Mark obsolete packages" package-menu-mark-obsolete-for-deletion
+		  :help "Mark all obsolete packages for deletion"))
+    (define-key menu-map [mx]
+      '(menu-item "Execute actions" package-menu-execute
+		  :help "Perform all the marked actions"))
+    (define-key menu-map [s5] '("--"))
+    (define-key menu-map [mh]
+      '(menu-item "Help" package-menu-quick-help
+		  :help "Show short key binding help for package-menu-mode"))
+    (define-key menu-map [mc]
+      '(menu-item "View Commentary" package-menu-view-commentary
+		  :help "Display information about this package"))
+    map)
   "Local keymap for `package-menu-mode' buffers.")
 
-(unless package-menu-mode-map
-  (setq package-menu-mode-map (make-keymap))
-  (suppress-keymap package-menu-mode-map)
-  (define-key package-menu-mode-map "q" 'quit-window)
-  (define-key package-menu-mode-map "n" 'next-line)
-  (define-key package-menu-mode-map "p" 'previous-line)
-  (define-key package-menu-mode-map "u" 'package-menu-mark-unmark)
-  (define-key package-menu-mode-map "\177" 'package-menu-backup-unmark)
-  (define-key package-menu-mode-map "d" 'package-menu-mark-delete)
-  (define-key package-menu-mode-map "i" 'package-menu-mark-install)
-  (define-key package-menu-mode-map "g" 'package-menu-revert)
-  (define-key package-menu-mode-map "r" 'package-menu-refresh)
-  (define-key package-menu-mode-map "~"
-    'package-menu-mark-obsolete-for-deletion)
-  (define-key package-menu-mode-map "x" 'package-menu-execute)
-  (define-key package-menu-mode-map "h" 'package-menu-quick-help)
-  (define-key package-menu-mode-map "?" 'package-menu-view-commentary)
-  )
-
 (defvar package-menu-sort-button-map
   (let ((map (make-sparse-keymap)))
     (define-key map [header-line mouse-1] 'package-menu-sort-by-column)

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Phil Hagelberg <phil@hagelb.org> writes:
>>> Basically, all I want is for it to be able to distinguish between
>>> "packages available locally, either installed by the user or the
>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>> as to be able to install different versions of the same package at the
>>> same time.
>> 
>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.

PH> I've just implemented support for multiple package sources in
PH> package.el. Right now it is preconfigured to use ELPA, but the following
PH> snippet will let it pull in packages from my own archive source as well:

PH> (add-to-list 'package-archives
PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)

PH> Hit M-x package-list-packages to see all available packages. The "bork"
PH> and "bingle" packages are dummies that are only available from my
PH> repository, and should be available for installation at the same time as
PH> the standard ELPA-provided packages.

PH> I'd love to see some other elisp library authors try to set up their own
PH> package archive sources using package-maint.el and test the
PH> multiple-archive support.

PH> I hope that a new version of package.el incorporating my changes could
PH> be pushed out to ELPA and then possibly included in Emacs once a little
PH> more work has gone into integration with the Emacs load process.

Hi Phil,

that's wonderful news.  I was just reading (and commented) at
http://monkey.org/~marius/self-contained-emacs.html about how package.el
with custom repositories could solve a very common need to have a
"personalized" self-contained Emacs.

Have you considered support for non-HTTP sources?  I'm OK with HTTP only
for now but I think a Bazaar source would make sense if there's going to
be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
supported?

What remains to be done to make package.el a part of Emacs?  I think
Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
else?

Once it's in the core we can start work on making more Emacs packages
installable directly.  Again, I think this is great progress and I
appreciate your work.

Thanks
Ted

Re: Integrating package.el

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 
>
> PH> Phil Hagelberg <phil@hagelb.org> writes:
>>>> Basically, all I want is for it to be able to distinguish between
>>>> "packages available locally, either installed by the user or the
>>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>>> as to be able to install different versions of the same package at the
>>>> same time.
>>> 
>>> OK, I will prioritize this once I finish adding support for multiple
>>> archive sources. I think we'll need this in order to support separating
>>> out fully-supported packages vs unsupported packages for which we simply
>>> have copyright assignment as Ted has suggested.
>
> PH> I've just implemented support for multiple package sources in
> PH> package.el. Right now it is preconfigured to use ELPA, but the following
> PH> snippet will let it pull in packages from my own archive source as well:
>
> PH> (add-to-list 'package-archives
> PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)
>
> PH> Hit M-x package-list-packages to see all available packages. The "bork"
> PH> and "bingle" packages are dummies that are only available from my
> PH> repository, and should be available for installation at the same time as
> PH> the standard ELPA-provided packages.
>
> PH> I'd love to see some other elisp library authors try to set up their own
> PH> package archive sources using package-maint.el and test the
> PH> multiple-archive support.
>
> PH> I hope that a new version of package.el incorporating my changes could
> PH> be pushed out to ELPA and then possibly included in Emacs once a little
> PH> more work has gone into integration with the Emacs load process.
>
> Hi Phil,
>
> that's wonderful news.  I was just reading (and commented) at
> http://monkey.org/~marius/self-contained-emacs.html about how package.el
> with custom repositories could solve a very common need to have a
> "personalized" self-contained Emacs.

Thats a brilliant idea!

> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
> for now but I think a Bazaar source would make sense if there's going to
> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
> supported?
>
> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
> else?
>
> Once it's in the core we can start work on making more Emacs packages
> installable directly.  Again, I think this is great progress and I
> appreciate your work.
>
> Thanks
> Ted
>
>
-- 
Joakim Verona

Re: Integrating package.el

[Stefan Monnier]

> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
> else?

For me, the only thing holding it back is the ability to concurrently
install different versions of the same package.

E.g. the sysadmin may install a whole bunch of packages and even
different versions of those packages, and then each user gets to choose
which ones to use in her ~/.emacs.


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 23 Feb 2010 17:25:17 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

Phil and other ELPA folks, please comment, here are my opinions.

Emacs could recognize the Version header and always track it internally
when it loads a library file, maybe with a (version . "1.2") in the
load-history.  That would help a lot, I think.  There's already good
logic in package.el for tracking its own installations and they are
required to have a version.  They get installed as 

$prefix/NAME-VERSION/NAME.el

which is a good way to resolve path conflicts.

I think we should ensure this works in common scenarios; the users who
break package.el in ways we haven't anticipated should either stop using
it, submit patches, or suffer.  I think that's reasonable, especially if
it happens as a major upgrade (in Emacs 24 perhaps).  The fact that
there are many happy users of package.el today shows that it's all right
even in its current form.

I also think that once package.el is available from within Emacs, many
of the custom installations will simply become redundant or be reduced
to ELPA-style repositories local to the site.  So the need for handling
edge cases will shrink as word spreads.

Ted

Re: Integrating package.el

[Stefan Monnier]

>>> What remains to be done to make package.el a part of Emacs?  I think
>>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

> Phil and other ELPA folks, please comment, here are my opinions.

> Emacs could recognize the Version header and always track it internally
> when it loads a library file, maybe with a (version . "1.2") in the
> load-history.  That would help a lot, I think.

I personnally don't care about the behavior if the user tries to
activate two different versions of the same package in a given session.
So package.el doesn't even need to do anything special about it AFAIC.

> They get installed as 
> $prefix/NAME-VERSION/NAME.el
> which is a good way to resolve path conflicts.

Yes, it's a good choice for installation location.  It still remains to
distinguish between "installed in the file system" and "active in
a particular session" (where "active" doesn't mean loaded but rather
just that autoloads are in place).


        Stefan

Re: Integrating package.el

[David De La Harpe Golden]

Stefan Monnier wrote:
>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>> else?
> 
> For me, the only thing holding it back is the ability to concurrently
> install different versions of the same package.
> 

I find _calling_ the facility "package" of (minor) concern.
Clearly it's not package as in Common Lisp package::symbol

Obviously, Emacs Lisp is not Common Lisp and there's more than enough 
precedent for the other "package" usage from elsewhere, I expect most 
Emacs users would in fact be more used to it from GNU+Linux distros for 
starters.

But could maybe just add something like this to the ELPA FAQ? Though it 
should be fairly obvious, newbies who don't know some core differences 
between Emacs Lisp and Common Lisp may be briefly caught out:

Q. Does this implement Common Lisp style "packages" (of symbols) for 
Emacs Lisp?

A. No, it does not, neither as its primary goal nor as an implementation 
detail.  Think "package" purely as in "program code distribution 
bundle".  Note that the fact that a Common Lisp "program code 
distribution bundle"* is typically a bundle of some program code that 
defines and then defines itself in a particular Common Lisp "package of 
symbols" is an aspect of Common Lisp language and conventions completely 
irrelevant for and alien to Emacs Lisp.

* sometimes referred to as a "package" even in the Common Lisp world.

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Phil Hagelberg <phil@hagelb.org> writes:
>>> Basically, all I want is for it to be able to distinguish between
>>> "packages available locally, either installed by the user or the
>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>> as to be able to install different versions of the same package at the
>>> same time.
>> 
>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.

PH> I've just implemented support for multiple package sources in
PH> package.el. Right now it is preconfigured to use ELPA, but the following
PH> snippet will let it pull in packages from my own archive source as well:

PH> (add-to-list 'package-archives
PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)

PH> Hit M-x package-list-packages to see all available packages. The "bork"
PH> and "bingle" packages are dummies that are only available from my
PH> repository, and should be available for installation at the same time as
PH> the standard ELPA-provided packages.

PH> I'd love to see some other elisp library authors try to set up their own
PH> package archive sources using package-maint.el and test the
PH> multiple-archive support.

PH> I hope that a new version of package.el incorporating my changes could
PH> be pushed out to ELPA and then possibly included in Emacs once a little
PH> more work has gone into integration with the Emacs load process.

Jonas, have you looked at this update to package.el?  Any chance you can
make your Emacsmirror accessible as a package.el repository, over HTTP
or through patched-in Git/SVN support in package.el?

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Thu, 25 Feb 2010 15:05:47 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> What remains to be done to make package.el a part of Emacs?  I think
>>>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>>>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

>> Phil and other ELPA folks, please comment, here are my opinions.

>> Emacs could recognize the Version header and always track it internally
>> when it loads a library file, maybe with a (version . "1.2") in the
>> load-history.  That would help a lot, I think.

SM> I personnally don't care about the behavior if the user tries to
SM> activate two different versions of the same package in a given session.
SM> So package.el doesn't even need to do anything special about it AFAIC.

OK.  But I still think Emacs should record the version as I suggested
whenever it finds it in a .el/.elc file.  It would help resolve many
annoying user-level bugs by showing exactly what version of the library
was loaded, not implied from the directory but directly from the version
header.  Does that require lots of changes?

>> They get installed as 
>> $prefix/NAME-VERSION/NAME.el
>> which is a good way to resolve path conflicts.

SM> Yes, it's a good choice for installation location.  It still remains to
SM> distinguish between "installed in the file system" and "active in
SM> a particular session" (where "active" doesn't mean loaded but rather
SM> just that autoloads are in place).

I think Phil or Tom should comment, I don't know how they plan to
address your concerns.

Ted

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote:
> PH> I've just implemented support for multiple package sources in
> PH> package.el. Right now it is preconfigured to use ELPA, but the following
> PH> snippet will let it pull in packages from my own archive source as well:
>
> PH> I'd love to see some other elisp library authors try to set up their own
> PH> package archive sources using package-maint.el and test the
> PH> multiple-archive support.
>
> PH> I hope that a new version of package.el incorporating my changes could
> PH> be pushed out to ELPA and then possibly included in Emacs once a little
> PH> more work has gone into integration with the Emacs load process.
>
> Jonas, have you looked at this update to package.el?

No, I have not found the time yet but I am aware that Phil has added
this feature [0].

> Any chance you can make your Emacsmirror accessible as a package.el repository,

Not immediately. Of course I do want the Emacsmirror to support (a)
package manager(s)
but due to lack of time I have concentrated on mirroring (adding and
*keeping up-to-date*)
packages.

I also haven't updated the metadata which I generate using elm.el in a
long time. But this
is next on my todo list - I might actually do this today so after that
I will hopefully find some
time to look into package-maint.el. (While I haven't pushed any
updates to the metadata I
have worked on improving the code used to generate it though.)

Without actually having looked at package-maint.el I believe that
elm.el is more advanced -
so I have heard and do in part believe... At least the metadata that I
generate is *different*
from that generated by package-maint.el or manually; and used by
package.el. I think the
data I generate has some advantages - mainly I do extract more information.

Also I do store it in one file per package instead of just one file
(which would be huge given
that now more than 2100 packages are being mirrored). In the future I
would like to extend
this to one file per package version. (I used to have support for this
but dropped it because
it is not clear yet how manual fixes would be merged when new versions
are released.)

Of course anyone is free to generate the package list as required by
package.el himself.
This could either be done by using the metadata I have generated and
converting it [1] [2]
or by getting a recent tarball [3] of all the git directories of all
mirrored packages and
generating it directly using package-maint.el.

Do not misunderstand me - I do plan to support package.el in the
future. But right now I
am working on other things.  Also I am more than willing to support
anyone who would like
to experiment with this.  It might actually be a good idea to test
package-maint.el using
the 2100 packages in the Emacsmirror tarball. Right now it probably
has only been tested
with some selected packages which closely follow the conventions - but
I have seen things
you people would not dream of - attack ships on fire...

> over HTTP or through patched-in Git/SVN support in package.el?

I am working on git support as I do think it has some considerable
advantages - generally
people seam to disagree.  Since I think using a dvcs instead of
tarballs should be preferred
I have absolutely no interest in working on generating tarballs myself.

Again, like for package.el-style-metadata, that does not mean that I
won't add support for
tarballs if someone else writes the code.  I just do not have an
interest in doing it myself.

I will quietly work on git support and once it is ready people can
test it and decide whether
it does have advantages that out weight the cost of requiring users to
install git or not.

Also I do not much feel like discussing this anymore.  At this point I
simply have nothing
that would demonstrate the benefits and at the same time I doubt that
anyone can come
up with an argument that would convince me that it's not even worth
writing a prove-of-concept.

So in summary: I do wish package.el and the Emacsmirror to support one
another but due
to lack of time, different priorities and because it is unclear which
of the two has to go how
far to meet the other it is not a short term priority.

-- Jonas

[0] And I do not know if anyone has found the time to look at my
alternative: elm.el.
[1] Though while generally more informative it also lacks some
information required by package.el.
[2] But please wait until I have updated it. As of now it contains
some severe bugs.
[3] http://github.com/emacsmirror/elm-backup-pkgs

Re: Integrating package.el

[Jonas Bernoulli]

(I am rather confused by the mixture of quoting styles.)

>>> Emacs could recognize the Version header and always track it internally
>>> when it loads a library file, maybe with a (version . "1.2") in the
>>> load-history.  That would help a lot, I think.

Absolutely. However this information is missing from a lot of
libraries. Also even
if library contain version information it is still uncommon that
libraries state what
version of their dependencies are actually required.  Cedet contains something
like this - has this portion of cedet been merged into 23.2?

If such an extended `require' form (or similar standardized information in the
library header) are not adopted there is no way to automatically extract this
information.

This means that maintainers of package repositories have to figure it
out manually
or just leave this information out completely.  I am mirroring 2100
packages - I won't
do it manually.

> SM> I personnally don't care about the behavior if the user tries to
> SM> activate two different versions of the same package in a given session.

If I remember correctly what he meant to say was that loading multiple versions
is a user error. Having multiple versions installed is still an option
as is choosing
a particular version by the user (as opposed to the sysadmin) at runtime.

> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.  Does that require lots of changes?

I agree. But as explained above this still doesn't help the package manager to
determine which version of a dependency is required.

-- Jonas

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 17:32:21 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> (I am rather confused by the mixture of quoting styles.)
>>>> Emacs could recognize the Version header and always track it internally
>>>> when it loads a library file, maybe with a (version . "1.2") in the
>>>> load-history.  That would help a lot, I think.

JB> Absolutely. However this information is missing from a lot of
JB> libraries. Also even if library contain version information it is
JB> still uncommon that libraries state what version of their
JB> dependencies are actually required.  Cedet contains something like
JB> this - has this portion of cedet been merged into 23.2?

We have to start somewhere.  Tracking dependencies is not possible until
versions are consistent so I think it's OK to use (version . nil) in the
load-history for the libraries that don't have a version header.

JB> If such an extended `require' form (or similar standardized
JB> information in the library header) are not adopted there is no way
JB> to automatically extract this information.

JB> This means that maintainers of package repositories have to figure
JB> it out manually or just leave this information out completely.  I am
JB> mirroring 2100 packages - I won't do it manually.

I think it's reasonable to also require unit and integration tests for
packages, at least for those that are considered "reliable" by the
packaging system, to accomplish these goals.  But these are long-term
goals and I think they merit a separate discussion on emacs-devel.  I'll
start the discussion with my (as usual) naive questions :)

SM> I personnally don't care about the behavior if the user tries to
SM> activate two different versions of the same package in a given session.

JB> If I remember correctly what he meant to say was that loading
JB> multiple versions is a user error. Having multiple versions
JB> installed is still an option as is choosing a particular version by
JB> the user (as opposed to the sysadmin) at runtime.

Does elm.el support that scheme as package.el does, distinguishing
between package installation (with a versioned install dir) and package
activation?

>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.  Does that require lots of changes?

JB> I agree. But as explained above this still doesn't help the package
JB> manager to determine which version of a dependency is required.

Yes, in this context I only meant to say it would save time when
debugging user errors.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 17:26:46 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> Without actually having looked at package-maint.el I believe that
JB> elm.el is more advanced - so I have heard and do in part
JB> believe... At least the metadata that I generate is *different* from
JB> that generated by package-maint.el or manually; and used by
JB> package.el. I think the data I generate has some advantages - mainly
JB> I do extract more information.

Can you or Phil/Tom explain the metadata capabilities of elm.el and
package-maint.el respectively, please?  It would help to know what the
authors think and do a side-by-side comparison.

JB> Of course anyone is free to generate the package list as required by
JB> package.el himself.  This could either be done by using the metadata
JB> I have generated and converting it [1] [2] or by getting a recent
JB> tarball [3] of all the git directories of all mirrored packages and
JB> generating it directly using package-maint.el.

That may be necessary if package.el becomes the default package manager
in Emacs.  It doesn't seem too hard since, as you say, the ELPA-style
metadata is a subset of the Emacsmirror metadata.  It would also be
possible to make a special package.el setup for Emacsmirror which would
pull elm.el and manage elm.el packages through it, if that was required.

JB> I am working on git support as I do think it has some considerable
JB> advantages - generally people seam to disagree.  Since I think using
JB> a dvcs instead of tarballs should be preferred I have absolutely no
JB> interest in working on generating tarballs myself.

I agree.  2100 packages over HTTP is crazy, an update would be very
costly whether you use a single database or one per package.  OTOH for a
few packages or specific situations HTTP makes sense so it's a matter of
providing a uniform frontend over various backend protocols.

JB> Also I do not much feel like discussing this anymore.  At this point
JB> I simply have nothing that would demonstrate the benefits and at the
JB> same time I doubt that anyone can come up with an argument that
JB> would convince me that it's not even worth writing a
JB> prove-of-concept.

Your input is valuable at any stage in the discussions, so thank you for
contributing.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> Can you or Phil/Tom explain the metadata capabilities of elm.el and
Ted> package-maint.el respectively, please?  It would help to know what the
Ted> authors think and do a side-by-side comparison.

ELPA keeps a file, called archive-contents, that tracks metadata about
all the uploaded packages.  This metadata is extracted from the package
when I do an upload.

The information tracked are package name, the version, the requirements
list (see my other note), a docstring (for the package menu mode), and
whether a package is a single file or a .tar.  The latter is just so we
can construct the right URL at download time.

There's actually a second set of metadata about packages that are
included in Emacs itself.  This is (poorly) hand-maintained.  Ideally
this would be generated during the Emacs build.

I still haven't had time to look at Phil's changes, so I don't know what
he might have added or changed.

Tom

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> On Mon, 1 Mar 2010 17:32:21 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote:
>
> JB> (I am rather confused by the mixture of quoting styles.)
>>>>> Emacs could recognize the Version header and always track it internally
>>>>> when it loads a library file, maybe with a (version . "1.2") in the
>>>>> load-history.  That would help a lot, I think.
>
> JB> Absolutely. However this information is missing from a lot of
> JB> libraries. Also even if library contain version information it is
> JB> still uncommon that libraries state what version of their
> JB> dependencies are actually required.  Cedet contains something like
> JB> this - has this portion of cedet been merged into 23.2?
>
> We have to start somewhere.  Tracking dependencies is not possible until
> versions are consistent so I think it's OK to use (version . nil) in the
> load-history for the libraries that don't have a version header.

Yes.

> Does elm.el support that scheme as package.el does, distinguishing
> between package installation (with a versioned install dir) and package
> activation?

No, it is not a package manager it doesn't care if a package is
installed it just
needs to know where the sources are and then extracts information from it.
elx.el and lisp-mnt.el are used to extract information from individual packages.
elm.el is used to collect, check and store that information.

If I write a package manager it will be called epkg until (and if) it
can be merged
with package.el.

-- Jonas

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> Can you or Phil/Tom explain the metadata capabilities of elm.el and
> package-maint.el respectively, please?  It would help to know what the
> authors think and do a side-by-side comparison.

$ [ -e .git ] && echo "yes please"
yes please
$ ls -1 | wc --lines
2121
$ cat elm.epkg
(:summary    "Utilities for maintaining the Emacsmirror"
 :created    "20081202"
 :updated    "20100226"
 :license    "GPL-3"
 :authors    (("Jonas Bernoulli" . "jonas@bernoul.li"))
 :maintainer ("Jonas Bernoulli" . "jonas@bernoul.li")
 :provided   (elm elm-org)
 :required   ((("elx" elx)
               ("emacs" assoc cl finder)))
 :keywords   ("libraries")
 :homepage   "https://github.com/tarsius/elm")
$ cat ../package-commentaries/elm.txt
This package is used to maintain the Emacsmirror which can be found at
http://www.emacsmirror.org.

This library is mostly used to extracts metadata from packages and save
it for future use by e.g. a package manager.

Most of the code used to generate the mirrors webpage can be found in
the accompanying libary `elm-org.el' which might eventually be replaced
by a more dynamic webpage.



That's what it looks like. Thoughts follow later.

-- Jonas

Re: Integrating package.el

[Stefan Monnier]

> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.  Does that require lots of changes?

I don't see much evidence that "it would help resolve many annoying
user-level bugs".  I'm not opposed to it, but when it comes to making
external Elisp packages easier to manage for the user, this comes very
far down the list of priorities.


        Stefan

Re: Integrating package.el

[Jonas Bernoulli]

> I still haven't had time to look at Phil's changes, so I don't know what
> he might have added or changed.

Daniel Hackney has also made additions:
http://github.com/technomancy/package.el/network

Tom, do you use a vcs to develop package.el? If so it might be a good
idea to make it
publicly available or start using vcs if you are not doing so already.
Might make it easier
to contribute. What vcs doesn't really matter to me with git it's
rather easy to work with
foreign repositories, though git (since Phil, Daniel and I use it) or
bzr (since it is the
political correct solution:) would probably be preferable.

-- Jonas

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.

The information you get like this at runtime is not reliable. Some people
bump right after making a new release other develop for months keeping
the version from the latest release.

Rather this information should be extracted by the repository maintainers,
They have the full history of the package (if available) and cat determine
from that which of the many revision containing a particular version string,
actually IS that version... (I have some slightly buggy code for this
somewhere).

Speaking of version strings, are there any conventions how an author
should version his packages? Currently when I make edits after a
release and make them public while not wanting to release yet another
version I usually just add a "+" after the version.

0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2

Not really happy with it. But what should I be doing instead?

-- Jonas

Re: Integrating package.el

[Štěpán Němec]

On Mon, Mar 01, 2010 at 10:37:43PM +0100, Jonas Bernoulli wrote:
> Speaking of version strings, are there any conventions how an author
> should version his packages? Currently when I make edits after a
> release and make them public while not wanting to release yet another
> version I usually just add a "+" after the version.
> 
> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2

[just a thought:]
Maybe using something compatible with what `version-to-list' and friends
can understand?


    Štěpán

Re: Integrating package.el

[Štěpán Němec]

On Mon, Mar 01, 2010 at 11:18:32PM +0100, Štěpán Němec wrote:
> On Mon, Mar 01, 2010 at 10:37:43PM +0100, Jonas Bernoulli wrote:
> > Speaking of version strings, are there any conventions how an author
> > should version his packages? Currently when I make edits after a
> > release and make them public while not wanting to release yet another
> > version I usually just add a "+" after the version.
> > 
> > 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
> 
> [just a thought:]
> Maybe using something compatible with what `version-to-list' and friends
> can understand?
... which is actually the case even with the above:

(version-to-list "0.1+")
=> (0 1 -3)

so it's treated like an alpha release by default.

Re: Integrating package.el

[Jonas Bernoulli]

>> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
>
> [just a thought:]
> Maybe using something compatible with what `version-to-list' and friends
> can understand?

Hm, yes even my own such library does not understand it.

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 01 Mar 2010 11:06:15 -0700 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> Can you or Phil/Tom explain the metadata capabilities of elm.el and
Ted> package-maint.el respectively, please?  It would help to know what the
Ted> authors think and do a side-by-side comparison.

Tom> ELPA keeps a file, called archive-contents, that tracks metadata about
Tom> all the uploaded packages.  This metadata is extracted from the package
Tom> when I do an upload.

Tom> The information tracked are package name, the version, the requirements
Tom> list (see my other note), a docstring (for the package menu mode), and
Tom> whether a package is a single file or a .tar.  The latter is just so we
Tom> can construct the right URL at download time.

Tom> There's actually a second set of metadata about packages that are
Tom> included in Emacs itself.  This is (poorly) hand-maintained.  Ideally
Tom> this would be generated during the Emacs build.

On Mon, 1 Mar 2010 22:09:59 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> $ cat elm.epkg
JB> (:summary    "Utilities for maintaining the Emacsmirror"
JB>  :created    "20081202"
JB>  :updated    "20100226"
JB>  :license    "GPL-3"
JB>  :authors    (("Jonas Bernoulli" . "jonas@bernoul.li"))
JB>  :maintainer ("Jonas Bernoulli" . "jonas@bernoul.li")
JB>  :provided   (elm elm-org)
JB>  :required   ((("elx" elx)
JB>                ("emacs" assoc cl finder)))
JB>  :keywords   ("libraries")
JB>  :homepage   "https://github.com/tarsius/elm")
JB> $ cat ../package-commentaries/elm.txt
JB> This package is used to maintain the Emacsmirror which can be found at
JB> http://www.emacsmirror.org.

Thanks!

The Emacs package metadata will, ideally, be replaced with a package
repository that resides inside Emacs itself.  Thus users can explore and
activate Emacs packages without the usual drudgery, and some default
packages can even be activated for them.  This is my plan at least and I
hope it's accepted into Emacs.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 01 Mar 2010 16:19:47 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.  Does that require lots of changes?

SM> I don't see much evidence that "it would help resolve many annoying
SM> user-level bugs".

There was one just a few days ago on the Gnus newsgroup, which Katsumi
Yamaoka resolved by prior experience but otherwise would have been
really puzzling.  I've seen many like it.  The general problem of
shadowed paths is pretty pervasive and frustrating, especially since
users know many ways to add to the load-path.

SM> I'm not opposed to it, but when it comes to making external Elisp
SM> packages easier to manage for the user, this comes very far down the
SM> list of priorities.

All right, thanks.  I will keep it in mind.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 22:37:43 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> 2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.

JB> The information you get like this at runtime is not reliable. Some people
JB> bump right after making a new release other develop for months keeping
JB> the version from the latest release.

JB> Rather this information should be extracted by the repository maintainers,
JB> They have the full history of the package (if available) and cat determine
JB> from that which of the many revision containing a particular version string,
JB> actually IS that version... (I have some slightly buggy code for this
JB> somewhere).

I don't think that's ideal either.  I'd rather use the authors' version
and then allow an override by the repository maintainer.  Otherwise
you'll certainly end up with different version levels between repos and
that's not a good scenario for the user.

The repo could associate a content hash with the version to generate a
unique identifier (and some insurance against corruption) for a
particular library snapshot.  I wonder if Emacs has a standard way to do
this (ignoring comments and whitespace).

Ted

Re: Integrating package.el

[Richard Stallman]

It looks like some people are trying to turn this into a package
system with all the features of Debian.  Please don't do that.  The
reason it is ok to install package.el is that it doesn't try to be a
full-blown package system.

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 02 Mar 2010 12:39:34 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> It looks like some people are trying to turn this into a package
RS> system with all the features of Debian.  Please don't do that.  The
RS> reason it is ok to install package.el is that it doesn't try to be a
RS> full-blown package system.

The only major feature that has been added (by Phil Hagelberg) is
support for multiple package repositories.  I don't think any others are
under review; the only outstanding issues I know of are Stefan's
concerns about overlapping installs and a request for menus.

All the discussion about testing, versions, and dependency tracking is
purely theoretical and I expect it to benefit the package repository
contributors and maintainers in the long run.  I don't think that
functionality should necessarily be pushed into package.el.

Ted

Re: Integrating package.el

[Davis Herring]

> Speaking of version strings, are there any conventions how an author
> should version his packages? Currently when I make edits after a
> release and make them public while not wanting to release yet another
> version I usually just add a "+" after the version.
>
> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
>
> Not really happy with it. But what should I be doing instead?

Why not just use another layer of versions?

  0.1, 0.1.1, 0.1.2, ..., 0.1.74, 0.2.

The rule I (try to) follow is that the major version number should be
incremented on an incompatible change, the minor version number on a
feature addition, the revision on any (public) bug fix, and (if one uses
it) the build number on any test.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I don't think that's ideal either.  I'd rather use the authors' version
Ted> and then allow an override by the repository maintainer.

Yes.  My policy for ELPA has just been to require sanity from the
maintainers.  In general I don't upload a package that is broken, and I
require changes to go upstream before uploading.  My reasoning was that
a little sanity is not very hard to achieve (though in real life there
are a some recalcitrant types out there) and it was better to try to
distribute the work instead of taking it all on myself.

Tom

Re: Integrating package.el

[Tom Tromey]

>>>>> "Jonas" == Jonas Bernoulli <jonas@bernoulli.cc> writes:

>> I still haven't had time to look at Phil's changes, so I don't know what
>> he might have added or changed.

Jonas> Daniel Hackney has also made additions:
Jonas> http://github.com/technomancy/package.el/network

Interesting, thanks.

Jonas> Tom, do you use a vcs to develop package.el?

Just locally, nothing public.

Jonas> If so it might be a good idea to make it publicly available or
Jonas> start using vcs if you are not doing so already.

I wouldn't mind but I have basically zero time for ELPA right now, and
for the foreseeable future.  I've been hoping Phil will drive the
project of adding the missing features and then integrating package.el
into Emacs.

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 03 Mar 2010 13:15:53 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I wouldn't mind but I have basically zero time for ELPA right now, and
Tom> for the foreseeable future.  I've been hoping Phil will drive the
Tom> project of adding the missing features and then integrating package.el
Tom> into Emacs.

Phil hasn't replied to questions in this thread for a bit so he must be
busy as well.  Would it be OK if I worked with the Emacs maintainers to
patch up Phil's latest package.el so it can go into Emacs?  Or should I
wait a bit longer?  I really don't mind waiting, but if I can help...

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> On Wed, 03 Mar 2010 13:15:53 -0700 Tom Tromey <tromey@redhat.com> wrote: 
Tom> I wouldn't mind but I have basically zero time for ELPA right now, and
Tom> for the foreseeable future.  I've been hoping Phil will drive the
Tom> project of adding the missing features and then integrating package.el
Tom> into Emacs.

Ted> Phil hasn't replied to questions in this thread for a bit so he must be
Ted> busy as well.  Would it be OK if I worked with the Emacs maintainers to
Ted> patch up Phil's latest package.el so it can go into Emacs?  Or should I
Ted> wait a bit longer?  I really don't mind waiting, but if I can help...

Yes, it is fine by me.

Tom

Re: Integrating package.el

[Phil Hagelberg]

> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
> for now but I think a Bazaar source would make sense if there's going to
> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
> supported?

Rather than supporting installing straight out of a VC repository, I
have opted to make it easy to create an archive from a list of VC
repositories. (package-maint.el) The idea is that only releases
would be included in the archive rather than any-old-version.
This also reduces the dependencies on the client-side.

If you want to add support for installing directly from a VC repository
don't let me stop you, but I think there are more important things to
do first.

> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there
> anything else?

I want to look at the menus patch soon, but I haven't found the time
yet. Stefan also mentioned supporting having multiple versions of the
same project installed at once. Rather than supporting this by
allowing multiple versions to coexist in a single install location, I
think it makes more sense to allow multiple install locations. I
haven't thought through the implications of this though. Perhaps it
would be as easy as simply activating the newest version available,
but perhaps we need something more explicit. Probably some mechanism
would be necessary for allowing users to opt-out of system packages.

I think supporting the distinction between system-level installs vs
user-level installs is the biggest blocker (besides more thorough
testing) to including package.el in Emacs. I would like to implement
this, but it probably won't happen within the next month unless
someone else steps up to start work on it.

> Phil hasn't replied to questions in this thread for a bit so he must
> be busy as well.  Would it be OK if I worked with the Emacs
> maintainers to patch up Phil's latest package.el so it can go into
> Emacs?  Or should I wait a bit longer?  I really don't mind waiting,
> but if I can help...

Don't wait on account of me. I definitely want to see this through,
but if you're able to put work into it now then go for it.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

On Wed, Mar 3, 2010 at 12:13 PM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> I don't think that's ideal either.  I'd rather use the authors' version
> Ted> and then allow an override by the repository maintainer.
>
> Yes.  My policy for ELPA has just been to require sanity from the
> maintainers.  In general I don't upload a package that is broken, and I
> require changes to go upstream before uploading.  My reasoning was that
> a little sanity is not very hard to achieve (though in real life there
> are a some recalcitrant types out there) and it was better to try to
> distribute the work instead of taking it all on myself.

It should be noted that if package.el is included in Emacs, it will
provide a lot more incentive for package authors to provide proper
metadata and ensure that their code works with package.el themselves.
So I don't think this needs to be worried about.

-Phil

Re: Integrating package.el

[Stephen J. Turnbull]

Phil Hagelberg writes:

 > I think supporting the distinction between system-level installs vs
 > user-level installs is the biggest blocker (besides more thorough
 > testing) to including package.el in Emacs. I would like to implement
 > this, but it probably won't happen within the next month unless
 > someone else steps up to start work on it.

Warning: I'm not really sure what the problems are[1], but the user
install vs. system install distinction has never really worked right
in XEmacs's package system.  It probably just requires a little bit of
thought[2], but it will be well worth it in reducing user frustration
if you take care in designing this feature.


Footnotes: 
[1]  Probably partly an incomplete design and partly PEBKACs.

[2]  My apologies to whoever implemented that part of our system, but
that's my opinion. ;-)

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 3 Mar 2010 21:39:17 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

>> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
>> for now but I think a Bazaar source would make sense if there's going to
>> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
>> supported?

PH> Rather than supporting installing straight out of a VC repository, I
PH> have opted to make it easy to create an archive from a list of VC
PH> repositories. (package-maint.el) The idea is that only releases
PH> would be included in the archive rather than any-old-version.
PH> This also reduces the dependencies on the client-side.

OK.  See below about file:/// URLs.

PH> If you want to add support for installing directly from a VC repository
PH> don't let me stop you, but I think there are more important things to
PH> do first.

Understood.  The TODO list has more important items, I agree.

>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there
>> anything else?

PH> I want to look at the menus patch soon, but I haven't found the time
PH> yet. 

Great, thanks!

PH> Stefan also mentioned supporting having multiple versions of the
PH> same project installed at once. Rather than supporting this by
PH> allowing multiple versions to coexist in a single install location,
PH> I think it makes more sense to allow multiple install locations. I
PH> haven't thought through the implications of this though. Perhaps it
PH> would be as easy as simply activating the newest version available,
PH> but perhaps we need something more explicit. Probably some mechanism
PH> would be necessary for allowing users to opt-out of system packages.

PH> I think supporting the distinction between system-level installs vs
PH> user-level installs is the biggest blocker (besides more thorough
PH> testing) to including package.el in Emacs. I would like to implement
PH> this, but it probably won't happen within the next month unless
PH> someone else steps up to start work on it.

How about associating a single install location with each repository?
Logically that would make sense and there would be a default (nil)
repository for things not managed by package.el.  Then the user, by
managing the repository priority, would also manage the load-path
priority.

>> Phil hasn't replied to questions in this thread for a bit so he must
>> be busy as well.  Would it be OK if I worked with the Emacs
>> maintainers to patch up Phil's latest package.el so it can go into
>> Emacs?  Or should I wait a bit longer?  I really don't mind waiting,
>> but if I can help...

PH> Don't wait on account of me. I definitely want to see this through,
PH> but if you're able to put work into it now then go for it.

Can you review Dan Haxney's (CC here and to you) patches as well at
http://github.com/haxney/package?  He supports file:/// URLs and has
made many other improvements.  I don't know if he has rebased against
your support for multiple repositories.  If the two of you could decide
what pieces are worth keeping and merge, it would make it easier for me
to start work on the merged version.  Otherwise I may end up wasting
effort.

Thanks
Ted

Re: Integrating package.el

[Stefan Monnier]

> Warning: I'm not really sure what the problems are[1], but the user
> install vs. system install distinction has never really worked right
> in XEmacs's package system.

It would help if you could describe the problems encountered,


        Stefan

Re: Integrating package.el

[Stefan Monnier]

> yet. Stefan also mentioned supporting having multiple versions of the
> same project installed at once. Rather than supporting this by
> allowing multiple versions to coexist in a single install location, I
> think it makes more sense to allow multiple install locations. I

I strongly disagree.  I want to be able to install all kinds of versions
in a single spot so I can quickly see which versions are available.
This said, I agree that allowing several install locations is desirable.

Actually the "install" part of the code should be able to install
a package *anywhere*.  Similarly, the activation of a package
(i.e. setting up the autoloads) should be doable from a package located
anywhere.

It doesn't need to be part of the visible UI, but there should be
a command like M-x package-install that takes a file name (a single .el
file or a tarball) and a target directory (defaults to the main "install
location") and installs it there.

The only part that really needs to know about "install locations" is the
part that lets you list all the installed packages, deinstall some of
them, setup/remove activation for some of them, etc...

> I think supporting the distinction between system-level installs vs
> user-level installs is the biggest blocker (besides more thorough
> testing) to including package.el in Emacs.

No, I think the only blocker is the support for multiple
simultaneous versions.  Tho I haven't kept track closely enough to know
if there might be copyright-paperwork issues left.


        Stefan

Re: Integrating package.el

[Tom Tromey]

>>>>> "Phil" == Phil Hagelberg <phil@hagelb.org> writes:

Phil> Stefan also mentioned supporting having multiple versions of the
Phil> same project installed at once. Rather than supporting this by
Phil> allowing multiple versions to coexist in a single install
Phil> location, I think it makes more sense to allow multiple install
Phil> locations.

I consider these to be orthogonal.

You need multiple install locations to let package.el manage additions
in site-lisp, and also (maybe -- there are some questions here) the
load-path for packages which are included with Emacs but also released
separately.

So suppose Emacs starts up and sees, say, 3 versions of gnus: one
included in Emacs, one in site-lisp, and one in ~/.emacs.d/elpa/.
Which one should be activated?

Right now the obvious answer would be the newest one -- but Stefan wants
to let the user pick a specific one.  So, I think we'll need a bit of
additional metadata about this.

Tom

Re: Integrating package.el

[Tom Tromey]

Stefan> No, I think the only blocker is the support for multiple
Stefan> simultaneous versions.  Tho I haven't kept track closely enough
Stefan> to know if there might be copyright-paperwork issues left.

My version was written by me, and all other contributions were under the
copyright threshold.  So, it is clear here.  I don't know about the
status of other contributions (Phil's branch, the other github branch,
or the menu patch).

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Thu, 04 Mar 2010 13:29:49 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I want to be able to install all kinds of versions in a single spot
SM> so I can quickly see which versions are available.  This said, I
SM> agree that allowing several install locations is desirable.

SM> Actually the "install" part of the code should be able to install
SM> a package *anywhere*.  Similarly, the activation of a package
SM> (i.e. setting up the autoloads) should be doable from a package located
SM> anywhere.

SM> It doesn't need to be part of the visible UI, but there should be
SM> a command like M-x package-install that takes a file name (a single .el
SM> file or a tarball) and a target directory (defaults to the main "install
SM> location") and installs it there.

SM> The only part that really needs to know about "install locations" is the
SM> part that lets you list all the installed packages, deinstall some of
SM> them, setup/remove activation for some of them, etc...

On Thu, 04 Mar 2010 11:30:26 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> You need multiple install locations to let package.el manage additions
Tom> in site-lisp, and also (maybe -- there are some questions here) the
Tom> load-path for packages which are included with Emacs but also released
Tom> separately.

Tom> So suppose Emacs starts up and sees, say, 3 versions of gnus: one
Tom> included in Emacs, one in site-lisp, and one in ~/.emacs.d/elpa/.
Tom> Which one should be activated?

Tom> Right now the obvious answer would be the newest one -- but Stefan wants
Tom> to let the user pick a specific one.  So, I think we'll need a bit of
Tom> additional metadata about this.

How about letting the user define his own install location for each
repository, but with a sensible default?  The location could also be
tagged with a "is it versioned?" boolean.  While installing may not be
possible into every location, the user will be able to activate whatever
is installed in any location.

For example:

ELPA's Gnus defaults to ~/.emacs.d/elpa
(from repository "ELPA", versioned to $location/gnus-x.yz )

Emacs' Gnus defaults to /usr/local/share/emacs/23.1/lisp
(from repository "emacs 23.1", unversioned to $location/gnus)

site's Gnus is set by user to /usr/local/share/oursite/lisp
(from user-defined repository "oursite", unversioned to $location/gnus)

So then "package-install" without a specific repository will pick the
first repository in the package-archives list and try to write to it.
If it fails, that's an error.  Similarly for package removal.

With a specific repository, both install and remove go to the specific
location set by the user.  Yes, unversioned install locations will cause
conflicts, but they are necessary to make this backwards compatible.

So then resolving the question of multiple installs will actually be

"Do you want to pick the ELPA Gnus [in ...], the Emacs 23.1 Gnus 
[in ...], the oursite Gnus [in ...], or this CVS checkout we found in
~/gnus/lisp because it was in your load-path?"

IOW, the user would choose between repository install locations
(represented by the name of the repository) and unmanaged paths to
resolve the conflict.  Would that work?

Ted

Re: Integrating package.el

[Stefan Monnier]

> Right now the obvious answer would be the newest one -- but Stefan wants
> to let the user pick a specific one.  So, I think we'll need a bit of
> additional metadata about this.

The simplest solution is to not decide which at runtime but at some
earlier time.  This of course means that this choice can be out-of-date
(e.g. the choice may say "activate /usr/local/elisp/foo-2.3" but that
directory doesn't even exist any more), but if the changes are made
through the package manager, than those out-of-date problems can be
caught at that point.  And otherwise we can just signal a warning and
let the user invoke the package manager to fix it.


        Stefan

Re: Integrating package.el

[Stephen J. Turnbull]

Stefan Monnier writes:
 > > Warning: I'm not really sure what the problems are[1], but the user
 > > install vs. system install distinction has never really worked right
 > > in XEmacs's package system.
 > 
 > It would help if you could describe the problems encountered,

At my consulting rates, that would be worth about $500. ;-) As I said,
it's not clear what is design and what is PEBKAC, and clarifying that
would require a lot of effort.  We've just had a lot of user
complaints over the years related to specified and/or default
installation locations, permissions issues, etc.  Packages get
installed in places the running XEmacs isn't looking for them,
insufficient privilege to install in the default place, that kind of
thing.  This often seems to be related to system vs. user issues.

At this point I just want to offer the advice that Here Be Dragons (or
at least some cute little lizards) to watch out for.

Re: Integrating package.el

[Phil Hagelberg]

On Thu, Mar 4, 2010 at 10:33 AM, Tom Tromey <tromey@redhat.com> wrote:
> Stefan> No, I think the only blocker is the support for multiple
> Stefan> simultaneous versions.  Tho I haven't kept track closely enough
> Stefan> to know if there might be copyright-paperwork issues left.
>
> My version was written by me, and all other contributions were under the
> copyright threshold.  So, it is clear here.  I don't know about the
> status of other contributions (Phil's branch, the other github branch,
> or the menu patch).

I haven't gotten any contributions apart from Daniel's patches, and I
have my copyright assignment in. So I think Daniel's is the only one
missing.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

On Thu, Mar 4, 2010 at 10:29 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> yet. Stefan also mentioned supporting having multiple versions of the
>> same project installed at once. Rather than supporting this by
>> allowing multiple versions to coexist in a single install location, I
>> think it makes more sense to allow multiple install locations. I
>
> I strongly disagree.  I want to be able to install all kinds of versions
> in a single spot so I can quickly see which versions are available.

OK, if that's the most important to you then we should focus on that
next. I imagine it might be a fairly invasive modification though;
perhaps Tom could offer some hints on how it could be implemented
since he's the only one really familiar with the package activation
logic?

-Phil

Re: Integrating package.el

[Tom Tromey]

>>>>> "Phil" == Phil Hagelberg <phil@hagelb.org> writes:

Phil> OK, if that's the most important to you then we should focus on that
Phil> next. I imagine it might be a fairly invasive modification though;
Phil> perhaps Tom could offer some hints on how it could be implemented
Phil> since he's the only one really familiar with the package activation
Phil> logic?

I think the default should be to activate the most recent package, and
to activate a package at install time.  So, if the user picks a specific
older version to activate (or equivalently deactivates the most recent
version), record that, and take it into account during activation.

This would also need some changes in package-activate and the package
menu mode, and also we'd have to store some more info somewhere in
~/.emacs.d/elpa/.

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Sun, 07 Mar 2010 20:17:23 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

So it's a version override alist, where a missing entry implies the user
wants the latest?  Do you mean something like this:

((package1 (repository1 path-to-specific-version1)) 
 (package2 (repository2 path-to-specific-version2)))

or am I misunderstanding something?

Tom> This would also need some changes in package-activate and the package
Tom> menu mode, and also we'd have to store some more info somewhere in
Tom> ~/.emacs.d/elpa/.

If this is just an alist it could be stored with Customize and not as
part of the repository itself.  Conceptually it seems to be at the
global level, not per repository.

If the repository list is customizable and global (as I think Phil's
changes assume) then the overrides could simply be part of the
repository list:

(setq package-archives '((repository1 :overrides (package1 path-to-specific-version1))
                         (repository2 :overrides (package2 path-to-specific-version2))))

Ted

Re: Integrating package.el

[Stefan Monnier]

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

> So it's a version override alist, where a missing entry implies the user
> wants the latest?  Do you mean something like this:

> ((package1 (repository1 path-to-specific-version1)) 
>  (package2 (repository2 path-to-specific-version2)))

> or am I misunderstanding something?

Don't know about you, but I seem to be missing something: why is the
word "repository" used here?  AFAIK the "repository" for package.el are
(typically remote) locations from where to download packages, and they
should have no interaction whatsoever with the part that worries about
installation and activation of packages (just like Debian's dpkg has no
idea about repositories).

BTW, my take on package activation and selection of versions (which IIRC
I already discussed with Tom a while back and we agreed to disagree) is
that activation of a package is controlled by lines in .emacs that take
a form similar to:

   (load "/some/where/package/init/file.el" 'package)

so there is no search or selection of package version at startup-time.
This means that when a new version of a package is installed, Emacs has
various ways to react to it:
- don't do anything, and at the next startup warn the user that the
  package he requested doesn't exist any more because it was replaced by
  a new version.
- update the "load" line right when the package gets upgraded (tho only
  for the .emacs of the user that performs the upgrade).
- use symlinks like "/some/where/elisp/sml-mode" pointing to
  "sml-mode-4.1" and use (load "/some/where/elisp/sml-mode/startup.el")
  so the symlink can be adjusted during the upgrade.
- use a different load, like

   (package-activate-latest "/some/where/elisp/sml-mode")

  which will look for the latest sml-mode package installed in
  /some/where/elisp/, or

   (package-activate-latest "sml-mode")

  which will look for the latest sml-mode package installed in
  any of the installation targets.


-- Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 08 Mar 2010 12:01:52 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

>> So it's a version override alist, where a missing entry implies the user
>> wants the latest?  Do you mean something like this:

>> ((package1 (repository1 path-to-specific-version1)) 
>> (package2 (repository2 path-to-specific-version2)))

>> or am I misunderstanding something?

SM> Don't know about you, but I seem to be missing something: why is the
SM> word "repository" used here?  AFAIK the "repository" for package.el are
SM> (typically remote) locations from where to download packages, and they
SM> should have no interaction whatsoever with the part that worries about
SM> installation and activation of packages (just like Debian's dpkg has no
SM> idea about repositories).

I thought it made sense to remember the repository that a particular
version came from, so selection (for activation) and warning messages
are more helpful.  But this can be guessed based on the path prefix so
it's not strictly necessary, and you seem to be looking for package
management that's less tied to repositories anyhow.  So forget the above
and see if the following makes sense.

SM> BTW, my take on package activation and selection of versions (which IIRC
SM> I already discussed with Tom a while back and we agreed to disagree) is
SM> that activation of a package is controlled by lines in .emacs that take
SM> a form similar to:

SM>    (load "/some/where/package/init/file.el" 'package)

SM> so there is no search or selection of package version at startup-time.

SM> This means that when a new version of a package is installed, Emacs has
SM> various ways to react to it:
SM> - don't do anything, and at the next startup warn the user that the
SM>   package he requested doesn't exist any more because it was replaced by
SM>   a new version.

This warning is always necessary because packages can get removed by
external agents as well (so the user doesn't know the load path is
incorrect).  So there could be no new version at all.

SM> - update the "load" line right when the package gets upgraded (tho only
SM>   for the .emacs of the user that performs the upgrade).

Too many potential problems here, I'd stay away from this.

SM> - use symlinks like "/some/where/elisp/sml-mode" pointing to
SM>   "sml-mode-4.1" and use (load "/some/where/elisp/sml-mode/startup.el")
SM>   so the symlink can be adjusted during the upgrade.

I did this for 7 years in a lab I was managing and it's not fun.  You
can automate some of the tedium but inevitably it becomes unwieldy.
If the user or admin wants to do it this way, let them manage it
externally, there are decent tools to automate it or they can write
ELisp wrappers to fit their environment.

Also symlinks don't work reliably in some environments.

SM> - use a different load, like

SM>    (package-activate-latest "/some/where/elisp/sml-mode")

SM>   which will look for the latest sml-mode package installed in
SM>   /some/where/elisp/, or

SM>    (package-activate-latest "sml-mode")

SM>   which will look for the latest sml-mode package installed in
SM>   any of the installation targets.

For the sake of backward compatibility this seems like the best option.
(load) and (require) can keep working normally.  Users can then
transition to the package.el management gradually.  Emacs can start
using these options gradually too.

Version overrides can be then specified as

(package-activate-specific "sml-mode" "specific-version")
(package-activate-specific "/path/to/sml-mode" "specific-version")

Together with the startup warnings (which can be issued by
package-activate-{specific,latest}) I think this approach makes sense.

Ted