Autoload from a web page?

Autoload from a web page?

[Lennart Borgman]

Just a wild idea:

Would it be possible to setup a function to download an elisp file
from a web page when called and then evaluate that file and call
itself again (with the old parameters), now with the new definition
from the web page?

Would it be possible to redefine `require' during this eval so that it
downloads required files from the same web site (if they are missing)?

Re: Autoload from a web page?

[Richard Stallman]

    Would it be possible to setup a function to download an elisp file
    from a web page when called and then evaluate that file and call
    itself again (with the old parameters), now with the new definition
    from the web page?

It must be possible, but we should warn people not to take the risk
of doing this.

Re: Autoload from a web page?

[Lennart Borgman]

On Sun, Dec 27, 2009 at 4:13 AM, Richard Stallman <rms@gnu.org> wrote:
>    Would it be possible to setup a function to download an elisp file
>    from a web page when called and then evaluate that file and call
>    itself again (with the old parameters), now with the new definition
>    from the web page?
>
> It must be possible, but we should warn people not to take the risk
> of doing this.


Hm, yes. I have tested a bit. Technically it can be done this way in
Emacs. You can, with some tweaks, setup to autoload a function from
the web.

It would perhaps be quite easy to distribute (and/or update) parts of
Emacs and/or external packages this way.

But the risk is about doing this in an uncontrolled way. A lot of sw
is updated in a similar way today (in a way that tries to control the
risks).

Re: Autoload from a web page?

[Richard Stallman]

    But the risk is about doing this in an uncontrolled way. A lot of sw
    is updated in a similar way today (in a way that tries to control the
    risks).

Just because it is common does not make it acceptable.

We do not want software to be autoloaded from anywhere outside
the user's machine.  The user should always have control over
installing it in the machine.

(See http://www.gnu.org/philosophy/javascript-trap.html
for a similar issue.)

Re: Autoload from a web page?

[Lennart Borgman]

On Mon, Dec 28, 2009 at 7:48 PM, Richard Stallman <rms@gnu.org> wrote:
>    But the risk is about doing this in an uncontrolled way. A lot of sw
>    is updated in a similar way today (in a way that tries to control the
>    risks).
>
> Just because it is common does not make it acceptable.
>
> We do not want software to be autoloaded from anywhere outside
> the user's machine.  The user should always have control over
> installing it in the machine.
>
> (See http://www.gnu.org/philosophy/javascript-trap.html
> for a similar issue.)


That is a totally different context than the one I am  targetting.


My scenario is this:

- You have group of related elisp files in a common place on the web.
- You download one of them, but it turns out that it does `require' or
autoload that supposes you have downloaded the other files too.

In a situation like that it is possible to set up for autoloading
things by downloading them from the web without compromising security
(or at least not more than when you download a file yourself) since
you know which files will be downloaded.


Note: Of course this first assumes that you download one or two files
with the necessary information.

Re: Autoload from a web page?

[Stephen J. Turnbull]

Lennart Borgman writes:

 > In a situation like that it is possible to set up for autoloading
 > things by downloading them from the web without compromising security
 > (or at least not more than when you download a file yourself) since
 > you know which files will be downloaded.

Why would the user know which files would be downloaded, and from
where?  The whole point is to do it automatically, without troubling
the user.  If the user is going to go to the trouble of *carefully*
verifying sources and so on, little is saved.  On the other hand, it
would be easy enough to register a domain like "savannah.gnu.org.to",
and a lot of folks might miss that, if they even looked at all.

Think of this like rules against talking on your cellphone while
driving.  Cellphone use is not an absolute guarantee you'll kill
somebody, and abstinence from cellphone use is no guarantee you
won't.  Some people do use cellphones responsibly while driving, and
so on.  But for the typical driver it does vastly increase the risks.

And so does any automated downloading and execution scheme.

Re: Autoload from a web page?

[Lennart Borgman]

On Tue, Dec 29, 2009 at 2:32 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> Lennart Borgman writes:
>
>  > In a situation like that it is possible to set up for autoloading
>  > things by downloading them from the web without compromising security
>  > (or at least not more than when you download a file yourself) since
>  > you know which files will be downloaded.
>
> Why would the user know which files would be downloaded, and from
> where?  The whole point is to do it automatically, without troubling
> the user.  If the user is going to go to the trouble of *carefully*
> verifying sources and so on, little is saved.  On the other hand, it
> would be easy enough to register a domain like "savannah.gnu.org.to",
> and a lot of folks might miss that, if they even looked at all.
>
> Think of this like rules against talking on your cellphone while
> driving.  Cellphone use is not an absolute guarantee you'll kill
> somebody, and abstinence from cellphone use is no guarantee you
> won't.  Some people do use cellphones responsibly while driving, and
> so on.  But for the typical driver it does vastly increase the risks.
>
> And so does any automated downloading and execution scheme.


While the comparision is funny it is not quite fair ;-)

To answer your questions:

* The user should not have to know much, only the starting location
for the download.
* It is thereafter the already downloaded files that have information
about what to download.
* This will hopefully ensure that only trusted files are downloaded.

* The information about what to download is basically in autoload and require
* ... because during collection of autoloads the relative file paths
are also recorded.
* ... and this information have been downloaded first.
* The downloaded files will be in a mirrored tree structure to where
they came from.

* The price carries of course a defadviced require that does this magic.
* It would be possible to download many of the Emacs core files this
way, actually
* ... but of course it is more meant for external packages.


Unfortunately the whole thing fails at the moment because of bugs in
url-copy-file... - some timeout or something. I have never cared to
report this bugs before because they are very complicated... ;-)

Re: Autoload from a web page?

[joakim]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Mon, Dec 28, 2009 at 7:48 PM, Richard Stallman <rms@gnu.org> wrote:
>>    But the risk is about doing this in an uncontrolled way. A lot of sw
>>    is updated in a similar way today (in a way that tries to control the
>>    risks).
>>
>> Just because it is common does not make it acceptable.
>>
>> We do not want software to be autoloaded from anywhere outside
>> the user's machine.  The user should always have control over
>> installing it in the machine.
>>
>> (See http://www.gnu.org/philosophy/javascript-trap.html> for a similar issue.)
>
>
> That is a totally different context than the one I am  targetting.
>
>
> My scenario is this:
>
> - You have group of related elisp files in a common place on the web.
> - You download one of them, but it turns out that it does `require' or
> autoload that supposes you have downloaded the other files too.

Is this situation not handled well enough by Tom Tromeys package.el and
ELPA? Especially if we move ahead with the plan to have ELPA:s backend
be something like Savannah.


>
> In a situation like that it is possible to set up for autoloading
> things by downloading them from the web without compromising security
> (or at least not more than when you download a file yourself) since
> you know which files will be downloaded.
>
>
> Note: Of course this first assumes that you download one or two files
> with the necessary information.
>
>
-- 
Joakim Verona

Re: Autoload from a web page?

[Lennart Borgman]

On Tue, Dec 29, 2009 at 3:45 AM,  <joakim@verona.se> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> My scenario is this:
>>
>> - You have group of related elisp files in a common place on the web.
>> - You download one of them, but it turns out that it does `require' or
>> autoload that supposes you have downloaded the other files too.
>
> Is this situation not handled well enough by Tom Tromeys package.el and
> ELPA? Especially if we move ahead with the plan to have ELPA:s backend
> be something like Savannah.


I am not aware that ELPA is downloading a file on require or autoload. Is it?

Re: Autoload from a web page?

[Tom Tromey]

>>>>> "Lennart" == Lennart Borgman <lennart.borgman@gmail.com> writes:

Lennart> I am not aware that ELPA is downloading a file on require or
Lennart> autoload. Is it?

Nope.  It works in a different way, where users download and install
entire packages, including their dependencies.

Tom

Re: Autoload from a web page?

[Richard Stallman]

    - You have group of related elisp files in a common place on the web.
    - You download one of them, but it turns out that it does `require' or
    autoload that supposes you have downloaded the other files too.

I understand that is the case you're talking about.
It is wrong to cut the user out of the process of deciding what
software to install.

    In a situation like that it is possible to set up for autoloading
    things by downloading them from the web without compromising security

This issue is not about security in the usual sense.
It is about preserving the user's control of his computing.
This is a fundamental goal for the GNU Project

Please see http://www.gnu.org/philosophy/javascript-trap.html.  That
issue is not related to Emacs, but it is basically the same issue.

Re: Autoload from a web page?

[Lennart Borgman]

On Tue, Dec 29, 2009 at 6:22 AM, Richard Stallman <rms@gnu.org> wrote:
>    - You have group of related elisp files in a common place on the web.
>    - You download one of them, but it turns out that it does `require' or
>    autoload that supposes you have downloaded the other files too.
>
> I understand that is the case you're talking about.
> It is wrong to cut the user out of the process of deciding what
> software to install.


But in this situation he is not cut out of the decision. At least that
is not my intention.

In this scenario the user wants to download a package, but he may not
want all of it at once. It is just like autoloading locally, but with
the difference that the files are loaded from the web (in a well
defined manner).


>    In a situation like that it is possible to set up for autoloading
>    things by downloading them from the web without compromising security
>
> This issue is not about security in the usual sense.
> It is about preserving the user's control of his computing.
> This is a fundamental goal for the GNU Project


Yes, I understand that and I read about the problem with javascript.
The same problems can not arise here since the user will be in control
of where to download files from. Unlike the situation with javascript
a single site (or a list of well defined sites) will be where the
download comes from.


> Please see http://www.gnu.org/philosophy/javascript-trap.html.  That
> issue is not related to Emacs, but it is basically the same issue.
>
>

Re: Autoload from a web page?

[Ted Zlatanov]

On Tue, 29 Dec 2009 03:45:38 +0100 joakim@verona.se wrote: 

j> Is this situation not handled well enough by Tom Tromeys package.el and
j> ELPA? Especially if we move ahead with the plan to have ELPA:s backend
j> be something like Savannah.

There's also emacsmirror.org, just recently announced but nothing usable
yet (see http://www.emacsmirror.org/ and I'm cc-ing Jonas Bernoulli).

This is just a suggestion: considering that Emacs has moved to a
powerful DVCS, maybe it makes sense to make ELPA and emacsmirror.org
hook into the Emacs repo.  Give commit bits and some structure to a few
people within an isolated "unsupported packages" area, and then let them
organize that area and the interfaces to it, including tracking of
foreign packages and repos, any way they see fit.  As long as it's clear
that it's an area for unsupported packages, this has the benefit of
integrating Emacs with the packages people find useful.  When people
pull Emacs, they'll also pull the unsupported packages, which IMHO is a
good thing.  Developers can still work outside of Emacs, but those who
choose to host their unsupported packages within Emacs don't have to
rely on external hosting (GitHub, SourceForge, Google Projects, etc.).

Ted

Re: Autoload from a web page?

[Stefan Monnier]

> Would it be possible to setup a function to download an elisp file
> from a web page when called and then evaluate that file and call
> itself again (with the old parameters), now with the new definition
> from the web page?

(url-handler-mode 1)
(autoload 'foo-function "http://toto.bar/baz/foo.el")

> Would it be possible to redefine `require' during this eval so that it
> downloads required files from the same web site (if they are missing)?

I think that using `autoload' this would only work if that URL is in
your global load-path (probably not something you want) or if your
remote "foo.el" is careful to temporarily add its (file-name-directory
load-file-name) to the load-path (c.f. thread about loading files
relative to the current file).

Otherwise you might be able to do something like

(defun foo-function (&rest args)
  (let ((load-path (append load-path '("http://toto.bar/baz/"))))
    (load "foo.el")
    ;; Here you might additionally want to check that foo-function
    ;; was redefined.
    (apply 'foo-function args)))

Do it at your own risk.  Using SSH (via Tramp) rather than HTTP (via
url-handler-mode) might be preferable.


        Stefan


PS: Of course this is completely untested and may bump into lacking
features of url-handler-mode.

Re: Autoload from a web page?

[Richard Stallman]

    This is just a suggestion: considering that Emacs has moved to a
    powerful DVCS, maybe it makes sense to make ELPA and emacsmirror.org
    hook into the Emacs repo.  Give commit bits and some structure to a few
    people within an isolated "unsupported packages" area, and then let them
    organize that area and the interfaces to it, including tracking of
    foreign packages and repos, any way they see fit.  As long as it's clear
    that it's an area for unsupported packages, this has the benefit of
    integrating Emacs with the packages people find useful.

This might be a good idea, but we want to get copyright assignments for
these packages just as we do for the "supported" parts of Emacs.
That way we can move Lisp files between the two categories based on
practical motives.

unsupported packages area in the Emacs repo (was: Autoload from a web page?)

[Ted Zlatanov]

On Tue, 29 Dec 2009 13:46:15 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS>     This is just a suggestion: considering that Emacs has moved to a
RS>     powerful DVCS, maybe it makes sense to make ELPA and emacsmirror.org
RS>     hook into the Emacs repo.  Give commit bits and some structure to a few
RS>     people within an isolated "unsupported packages" area, and then let them
RS>     organize that area and the interfaces to it, including tracking of
RS>     foreign packages and repos, any way they see fit.  As long as it's clear
RS>     that it's an area for unsupported packages, this has the benefit of
RS>     integrating Emacs with the packages people find useful.

RS> This might be a good idea, but we want to get copyright assignments for
RS> these packages just as we do for the "supported" parts of Emacs.
RS> That way we can move Lisp files between the two categories based on
RS> practical motives.

Yes, that makes a lot of sense.  Promotion could work like Apache's
incubation process or like Haskell's hackage system, or in another way
more suited to Emacs' model.  I think we should also consider demotion
for packages that should not be supported but are still marginally
useful.

I hope Tom Tromey and Jonas Bernoulli are interested in participating
too, and that the Emacs maintainers are willing to give this a try.
FWIW I will help any way I can and I think it would really improve the
Emacs user experience to install unsupported packages easily.

Ted

Re: unsupported packages area in the Emacs repo (was: Autoload from a web page?)

[Deniz Dogan]

2009/12/29 Ted Zlatanov <tzz@lifelogs.com>:
> On Tue, 29 Dec 2009 13:46:15 -0500 Richard Stallman <rms@gnu.org> wrote:
>
> RS>     This is just a suggestion: considering that Emacs has moved to a
> RS>     powerful DVCS, maybe it makes sense to make ELPA and emacsmirror.org
> RS>     hook into the Emacs repo.  Give commit bits and some structure to a few
> RS>     people within an isolated "unsupported packages" area, and then let them
> RS>     organize that area and the interfaces to it, including tracking of
> RS>     foreign packages and repos, any way they see fit.  As long as it's clear
> RS>     that it's an area for unsupported packages, this has the benefit of
> RS>     integrating Emacs with the packages people find useful.
>
> RS> This might be a good idea, but we want to get copyright assignments for
> RS> these packages just as we do for the "supported" parts of Emacs.
> RS> That way we can move Lisp files between the two categories based on
> RS> practical motives.
>
> Yes, that makes a lot of sense.  Promotion could work like Apache's
> incubation process or like Haskell's hackage system, or in another way
> more suited to Emacs' model.
>
>

What do you mean by "like Haskell's hackage system"? As far as I know,
anyone can upload their package to Hackage and it will be made
available to everyone. I don't think there are any
"supported/unsupported" categories as such.

-- 
Deniz Dogan

Re: unsupported packages area in the Emacs repo

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I hope Tom Tromey and Jonas Bernoulli are interested in participating
Ted> too, and that the Emacs maintainers are willing to give this a try.
Ted> FWIW I will help any way I can and I think it would really improve the
Ted> Emacs user experience to install unsupported packages easily.

I am interested but I don't have time to work on it.

Recently, Phil Hagelberg <technomancy@gmail.com> has been looking into
integrating package.el into Emacs.  I don't know how much progress he
has made.

There are a couple problems to be solved in the Emacs source tree, and
then a more sane server side must be written.  The latter may just be
some version control system plus some post-commit hooks to regenerate
the metadata.

Tom

Re: unsupported packages area in the Emacs repo

[Ted Zlatanov]

On Tue, 29 Dec 2009 22:21:14 +0100 Deniz Dogan <deniz.a.m.dogan@gmail.com> wrote: 

DD> 2009/12/29 Ted Zlatanov <tzz@lifelogs.com>:

>> Yes, that makes a lot of sense.  Promotion could work like Apache's
>> incubation process or like Haskell's hackage system, or in another way
>> more suited to Emacs' model.

DD> What do you mean by "like Haskell's hackage system"? As far as I know,
DD> anyone can upload their package to Hackage and it will be made
DD> available to everyone. I don't think there are any
DD> "supported/unsupported" categories as such.

Apache is one way to promote packages, with a tightly controlled
incubation and voting process.  Hackage is another way, with everything
"unsupported"--an unlikely model but I was trying to show extremes in
the way packages are promoted.  Sorry I wasn't clear.

Ted

Re: Autoload from a web page?

[Lennart Borgman]

On Tue, Dec 29, 2009 at 5:52 AM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Lennart" == Lennart Borgman <lennart.borgman@gmail.com> writes:
>
> Lennart> I am not aware that ELPA is downloading a file on require or
> Lennart> autoload. Is it?
>
> Nope.  It works in a different way, where users download and install
> entire packages, including their dependencies.


My intention is not to replace ELPA, but rather to create a backend
(and partly a frontend) for installing/updating through the web from a
repository (like bazaar on Launchpad).

Currently I have written to different ways to do this:

- Download a whole package of elisp files.
- Download a package part by part as the elisp files are needed (kind
of autoloading and installing locally through the web).

Those work currently in the only conventient way I could find out at
the moment. They download files directly from the web interface of the
repository (currently only Launchpad is supported). This has the
benefit that no sw except for Emacs is needed. From the user's point
of view it is easy.

There is no support for this use of the repository (bazaar) web
interface so this use is a bit fragile. I imagine ELPA could be
something in between giving this support (for example by simply taking
a snapshot from the repository and placing that on the web as just
directly accessible files).

ELPA could be the trusted source of where to begin, a kind of
directory for resources to download this way. An end Emacs user should
be able to trust ELPA that the sources are correct and virus/trojan
free (at least).


The second form, ie downloading part by part, depends simply on a file
with autoload information for the package of elisp files. ELPA could
produce such a file (or the package maintainer of course). (I have the
tools for that.)

Downloading part by part naturally puts some more rigid restrictions
on the elisp files in the package than normally, but it looks
solveable to me. (I am struggling with the elisp package nXhtml to
make it fit this right now.)

Downloading part by part has some advantages and disadvantages:

+ You download (automatically) just what you need and use.
+ To update you simply delete the local file.

- Files could become out of phase. Some version control should be good.
- You have to be connected to the internet when you want to use a
feature you have not used before.

Re: Autoload from a web page?

[Richard Stallman]

    My intention is not to replace ELPA, but rather to create a backend
    (and partly a frontend) for installing/updating through the web from a
    repository (like bazaar on Launchpad).

It is ok if it installs files on disk
which the user can then edit.

Re: unsupported packages area in the Emacs repo

[Ted Zlatanov]

On Tue, 29 Dec 2009 14:36:45 -0700 Tom Tromey <elpa@tromey.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> I hope Tom Tromey and Jonas Bernoulli are interested in participating
Ted> too, and that the Emacs maintainers are willing to give this a try.
Ted> FWIW I will help any way I can and I think it would really improve the
Ted> Emacs user experience to install unsupported packages easily.

Tom> I am interested but I don't have time to work on it.

Tom> Recently, Phil Hagelberg <technomancy@gmail.com> has been looking into
Tom> integrating package.el into Emacs.  I don't know how much progress he
Tom> has made.

Can you and Phil give us your view?  How would you integrate ELPA (package.el)
with Emacs, specifically using the DVCS features?  Your experience with
ELPA so far is very valuable since you've gone through all the pain of
setting up the packaging, dependencies, installer, and user support.

Tom> There are a couple problems to be solved in the Emacs source tree, and
Tom> then a more sane server side must be written.  The latter may just be
Tom> some version control system plus some post-commit hooks to regenerate
Tom> the metadata.

That's the technical side, but what about the supported/unsupported
division?  Have you considered making package.el integrated into the
Emacs core to work like the Cygwin installer, so unsupported or
experimental packages can be installed and loaded easily?  It may even
make sense to use package.el to manage supported packages, e.g. to set
up Gnus.  I wouldn't mind letting package.el manage my libraries, doing
it by hand is a waste of time 90% of the time.

There's a flexible assistant framework (analogous to the standard
"wizard" UI) in Gnus itself, originally intended for setting up Gnus.
Maybe it can be useful for package.el.

Ted

Re: Autoload from a web page?

[Ted Zlatanov]

On Tue, 29 Dec 2009 15:05:28 -0200 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Would it be possible to setup a function to download an elisp file
>> from a web page when called and then evaluate that file and call
>> itself again (with the old parameters), now with the new definition
>> from the web page?

SM> (url-handler-mode 1)
SM> (autoload 'foo-function "http://toto.bar/baz/foo.el")

There are so many security risks with this, and better ways to manage
packages have already been devised!  Ubuntu, for instance, signs the
packages and there's a whole key infrastructure set up in the distro.

Downloading and evaluating a file is the easy part, technically.
Listing Lisp files on a web page requires no security.  Managing
software, though, is hard.  For instance:

- your worst enemy is trying to break/infect/DDOS your distribution,
  what are your defenses?  What if your webserver dies?  What if your
  domain name expires or gets hijacked?

- how do you issue an emergency patch and notify your users?

- how do you back out of a change?  How do you version releases?

- how do you deal with circular dependencies?

- how do you minimize bandwidth, disk, and CPU costs?

- who will manage the repository when you can't?

- who will support the users when they have the inevitable problems?

Ted

Re: Autoload from a web page?

[Stefan Monnier]

SM> (url-handler-mode 1)
SM> (autoload 'foo-function "http://toto.bar/baz/foo.el")
> There are so many security risks with this, and better ways to manage

100% agreement.  The above is just an example of how to shoot oneself in
the foot, as requested.


        Stefan

Re: Autoload from a web page?

[Lennart Borgman]

On Wed, Dec 30, 2009 at 5:58 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> SM> (url-handler-mode 1)
> SM> (autoload 'foo-function "http://toto.bar/baz/foo.el")
>> There are so many security risks with this, and better ways to manage
>
> 100% agreement.  The above is just an example of how to shoot oneself in
> the foot, as requested.

;-)

I did not request that, but the hint actually made me thought about
other ways to structure my code (which was how I interpreted the
intention of your reply) so it was useful.

Re: unsupported packages area in the Emacs repo

[Jonas Bernoulli]

This did not go to the list in the first try (only Tom). So again:

On Tue, Dec 29, 2009 at 22:36, Tom Tromey <elpa@tromey.com> wrote:
>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> I hope Tom Tromey and Jonas Bernoulli are interested in participating
> Ted> too, and that the Emacs maintainers are willing to give this a try.
> Ted> FWIW I will help any way I can and I think it would really improve the
> Ted> Emacs user experience to install unsupported packages easily.
>
> I am interested but I don't have time to work on it.

I am also interested and will have time to work on it starting sometime next
month.

> Recently, Phil Hagelberg <technomancy@gmail.com> has been looking into
> integrating package.el into Emacs.  I don't know how much progress he
> has made.

Phil has contacted my privately uttered the worry that I might be duplicating
efforts. As I have explained to him this is not my intention and I plan to work
with anyone who is interested in the general idea of "package managment"
for Emacs, which of course could be implemented in many different ways.

I have given this a lot of thought some of which are mentioned on the mirrors
homepage [1]. Phil indicated that he agrees with many of them at least in
spirit.

He also said that he has found a few like minded people on #elpa.

He also mentioned that the maintainers have agreed to (possibly) include
`package.el' in the 24 (I think [2] is the relevant thread). As this idea seams
to gain some momentum I think the best way to proceed from here is this:

A few dedicated people - Phil, me, hopefully Tom and others who have
shown support and interest in the past (and of course those that haven't in
the past but do now) - work together more closely to create a working
implementation. (see below, and elsewhere in this thread).

Then in time for the 24 release it has to be decided whether things are
ready and if not we should admit it and not include it. There is no need
to hurry and add anything prematurely.

> There are a couple problems to be solved in the Emacs source tree, and
> then a more sane server side must be written.  The latter may just be
> some version control system plus some post-commit hooks to regenerate
> the metadata.

I and Phil also think that the use of a _distributed_ version control system is
the way to go.

About the metadata: I have created `elx.el' [3] an extension to `lisp-mnt.el'
which has more features (can extract additional data) and can deal with library
header that don't fully comply with the header conventions (as well as aspects
that should be defined there but haven't).

I have written to this list before [4] about `elx.el' and Stefan has
agreed that some
of it should probably be merged into `lisp-mnt.el'.  I have since made
quite a few
extensions and others will follow this is why I did not prepare any patches for
`lisp-mnt.el' yet.

The current implementation is able to generate what I call "epkg" a sexp like
this:

(:summary    "Extract information from Emacs Lisp libraries"
 :created    "20081202"
 :updated    "20091208"
 :license    "GPL-3"
 :authors    (("Jonas Bernoulli" . "jonas@bernoul.li"))
 :maintainer ("Jonas Bernoulli" . "jonas@bernoul.li")
 :provided   (elx)
 :required   ((("dconv" dconv)
              ("emacs" cl lisp-mnt)
              ("vcomp" vcomp)))
 :keywords   ("libraries")
 :homepage   "https://github.com/tarsius/elx")

[Tom: I also have some patches for `package.el' and will send them to you
once I find the time to work on it. I would have contacted you some time in
January but since this discussion is talking place now: Hi! I hope we will
work together well!]

I will comment on some other suggestions at the appropriate places in this
thread. (In particular about my thought on how the repository(ies) should be
implemented, what additions to package.el are necessary, and some more
on the use of dvcs).

And you might also want to read [1] and the relevant pages on ELPA [5].

-- Jonas

[1] http://www.emacsmirror.org
[2] http://thread.gmane.org/gmane.emacs.devel/101927
[3] http://github.com/tarsius/elx
[4] http://thread.gmane.org/gmane.emacs.devel/117127
[5] http://tromey.com/elpa/

Re: Autoload from a web page?

[Jonas Bernoulli]

(what's up with me today. This also did not go to the list in the first try)

On Tue, Dec 29, 2009 at 23:25, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Tue, Dec 29, 2009 at 5:52 AM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>>> "Lennart" == Lennart Borgman <lennart.borgman@gmail.com> writes:
> My intention is not to replace ELPA, but rather to create a backend
> (and partly a frontend) for installing/updating through the web from a
> repository (like bazaar on Launchpad).

The package manager should support multiple package repositories.

In theory this could mean that we end up with many competing repositories
and this is clearly something that should be avoided. Instead we should have
a few repositories with well defined purposes:

1. A "complete" repository.

This is what I am currently trying to do with the Emacsmirror [1]. This should
contain most packages that are available, regardless of quality and general
usefulness. It does not mean that packages could absolutely not be removed
or that it should not impose a minimal set of restrictions but the focus would
be on quantity not quality. (more on this below)

This would include packages that are broken and this would have to be pointed
out to users. I do think that it does make sense to include broken (or
very basic)
packages, because this allows potential new maintainers to easily find existing
implementations and go from there.

2. A "tested" repository.

ELPA could probably take this role. This repository should only contain packages
known to work with recent versions of Emacs and that are otherwise sound. QA
should happen here.

3. The FSF repository

This would contain only packages that are of could quality and generally useful
AND for which copyright assignments have been filled out.

I do not think that authors who release their packages under incompatible
licenses or do not want (or can't) assign the copyright to the FSF have bad
intentions.  Especially I do not think that because a package is still released
under the GPL-2 instead of GPL-3 is because they are against the move but
that they simply do not see much benefit in it or have forgotten to do it.

I have released all my Emacs extensions under the GPL-3 (at least I think so)
and do not think that it does not make sense to use any other license.  However
authors of packages that are only updated when bugs are reported might just not
see the need (or just forget about it) to update the license.

Having a "complete" repository is in my eyes not against the notion that we
want authors to move to the GPL-3.  Instead they might actually relize that
their packages are only included in the "complete" repository a long with many
clearly inferior code, not because of it's quality but because if
licensing issues.

I do not know what ELPA's position on license issues are. So if they include
packages e.g. releases as-is (and especially it includes package still released
under the GPL-2 (which I expect)) so it might be necessary to add an addtional
repository inbetween 2 and 3.

However I do strongly believe that it would be very unproductive to have more
than four repositories. So e.g. I do not think that it would make any sense to
create a repository for packages where the license is good but that are rubbish.

> Currently I have written to different ways to do this:
>
> - Download a whole package of elisp files.

This could be done in two ways.

- Download an archive
- Clone a dvcs repository

I favor the model where the package manager does clone the repository from
the repository - ehm ... Okay I have used the terminology used by other in
this thread so far. But actually I think we should talk of the repository as
a dvcs (git, bazaar) repository containing the history of a particular package
which is included in a mirror (Emacsmirror, ELPA) do avoid confusion.

And while we are talking about terms. I have used the term package to mean
the thing installed by the package manager.  Of course many packages
consist of only one library (the thing providing a feature) but
nevertheless these
are to different things.

So again:

I favor the model where the package manager does clone the repository from
the mirror as one of the steps required to install a package.

This would mean that the dvcs used by the mirror does have to be installed
on the users machine. So if this is not desirable archives could be created on
the mirror from each tagged release. For packages where upstream does not
use a vcs an archive could be created for each version where the library of
the "mainfile" has changed.

Creating the archives on the mirror would not be to hard. But adding support
in the package manager for installing either from a repository OR from an
archive would complicate things. So unless there are strong arguments
against it I would favor the vcs approach a lot, as it adds at least
the following
benefits:

- When a user installs a package for the first time from an untrusted source
e.g. the Emacsmirror he should read the packages source which is a
considerable amount of work. (I will comment elsewhere on how the installation
process would have to be adjusted to allow this.)

But provided the user verified the integrity of a package when he
first installed
it, the use of a vcs would make it much easier to only verify the difference to
the installed package once it is updated.

- The user can use the same interface (the package manager) used to find and
install working packages in order to clone broken packages, fix them, then
install them, and push his own fixed branch (and of course this should be
done in a way that makes it easy to also inform upstream of the changes,
either to have him merge the changes or to take over as a maintainer.

> - Download a package part by part as the elisp files are needed (kind
> of autoloading and installing locally through the web).

I do not like this idea at all. As I have said (and as you are well aware) there
is a difference between package and libraries (and supporting files like README,
or images).

The package manager should only operate on complete packages.

> Those work currently in the only conventient way I could find out at
> the moment. They download files directly from the web interface of the
> repository (currently only Launchpad is supported). This has the
> benefit that no sw except for Emacs is needed. From the user's point
> of view it is easy.

As I have said above I favor the vcs approach. It it is not feasible to clone
repositories then archives should be offered for download (which isn't hard).
Fetching seperate files seam backward and it actually complicates things
and as far as I can tell it has no benefits whatsoever.

Also while the Emacsmirror hasn't been integrated with a package manager
yet I do think that it already now makes the process of installing packages
(and especially updating) much easier than to fetch packages from all over
the internet.

And this is precisely because it uses a dvcs. You simply clone the desired
package and from that on you simply have to run "git pull" to bring your local
copy up-to-date. You do not have to remember where you got the package.

This is especially useful for packages that are distributed upstream as
versioned files (that is with a version suffix) dumped on a webpage.  The
task of finding the most recent version is already handled by the mirror -
otherwise you would have to go to the homepage, locate the most recent
link and the download it. And that for all installed packages.

(And yes of course while the Emacsmirror has not been integrated with
a package manager yet you will have have to make the package known
to Emacs manually. There also is not automated way for your local QA
yet, but "git diff OLD_HEAD.." make this much easier.

> There is no support for this use of the repository (bazaar) web
> interface so this use is a bit fragile. I imagine ELPA could be
> something in between giving this support

I agree as described above.

> (for example by simply taking
> a snapshot from the repository and placing that on the web as just
> directly accessible files).

I also agree but only if the snapshot is a archive. (I also think that it is
possible to download individual raw files from lp but haven't checked as
I am not interested in that.)

> ELPA could be the trusted source of where to begin, a kind of
> directory for resources to download this way. An end Emacs user should
> be able to trust ELPA that the sources are correct and virus/trojan
> free (at least).

Yes. But there should also be the Emacsmirror which isn't about trust
but completeness.

> The second form, ie downloading part by part, depends simply on a file
> with autoload information for the package of elisp files. ELPA could
> produce such a file (or the package maintainer of course). (I have the
> tools for that.)

I also have a tool for this (`elx.el' [2]). I would like to see yours - maybe we
can merge the two.

> Downloading part by part naturally puts some more rigid restrictions
> on the elisp files in the package than normally, but it looks
> solveable to me. (I am struggling with the elisp package nXhtml to
> make it fit this right now.)

I do not understand what nxhtml has to do with it, after all the headers
in emacs lisp libraries are not in html/xml.

> Downloading part by part has some advantages and disadvantages:
>
> + You download (automatically) just what you need and use.

And when you have no connection to the internet you are stuck with
what you have.  Again the atom a package manager should operate on
are packages. (Then again it might make sense to create seperate
packages for libraries found in the contrib directory of larger packages.)

> + To update you simply delete the local file.

Deleting a directory containing the complete package isn't much harder.
You just have to add "-r" to your "rm" - in fact it is easier, as "rm -r foo" is
saver than "rm foo*" which might also remove "foo-bar" ;-)

> - Files could become out of phase. Some version control should be good.

Yes exactly.

> - You have to be connected to the internet when you want to use a
> feature you have not used before.

Yes.

[1] http://www.emacsmirror.org
[2] http://github.com/tarsius/elx

Re: Autoload from a web page?

[Stefan Monnier]

> The package manager should support multiple package repositories.
[...]
> 3. The FSF repository

> This would contain only packages that are of could quality and
> generally useful AND for which copyright assignments have been
> filled out.

For political reasons, we only want to support repositories where
the copyright is owned by the FSF.  We also may want to support or
notion of package quality, tho.  Also I find it important that multiple
versions of a package can coexist at the same time (within reasonable
feasability constraints: we don't need to handle multiple versions in
a single Emacs sessions).


        Stefan

Re: Autoload from a web page?

[Jonas Bernoulli]

On Wed, Dec 30, 2009 at 19:30, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> The package manager should support multiple package repositories.
> [...]
>> 3. The FSF repository
>
>> This would contain only packages that are of could quality and
>> generally useful AND for which copyright assignments have been
>> filled out.
>
> For political reasons, we only want to support repositories where
> the copyright is owned by the FSF.

I do understand. But that hopefully does not imply that you want to
discourage other people from creating other repositories, or does it?

Even when the other repositories are not supported by the FSF they
can still be of use for users and can also be used by the FSF to find
libraries that are worth to be included (e.g. where the author should
be contacted fill out copyright assignments).

> We also may want to support or
> notion of package quality, tho.  Also I find it important that multiple
> versions of a package can coexist at the same time

Yes, I also think that this is desirable.

> (within reasonable
> feasability constraints: we don't need to handle multiple versions in
> a single Emacs sessions).

Anything else wouldn't make any sense.

-- Jonas

Re: unsupported packages area in the Emacs repo

[Phil Hagelberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 29 Dec 2009 14:36:45 -0700 Tom Tromey <elpa@tromey.com> wrote: 
> Tom> Recently, Phil Hagelberg <technomancy@gmail.com> has been looking into
> Tom> integrating package.el into Emacs.  I don't know how much progress he
> Tom> has made.
>
> Can you and Phil give us your view?  How would you integrate ELPA
> (package.el) with Emacs, specifically using the DVCS features?  Your
> experience with ELPA so far is very valuable since you've gone through
> all the pain of setting up the packaging, dependencies, installer, and
> user support.

I haven't given DVCS integration much thought at this point, though I've
recently contacted Jonas Bernoulli who seems to have done some
experiments with it. The initial idea was to just add "Emacs package
source" capabilities to Savannah, but it seems like the Savannah hackers
are already extremely overworked, and nobody I've talked to is much
interested in hacking in PHP, which this would entail.

But after talking with Jonas I'm starting to think it might be better to
allow packages to be used from DVCS hosted anywhere and just maintain a
centrally curated list of repositories that contain packages. This
definitely needs more brainstorming and discussion.

Integration with Emacs mostly consists of deciding what parts of Emacs
should be treated as packages and marking them as such so they may be
upgraded separately.

Hopefully in the future this will also allow things like gnus and
org-mode not to have to be merged into Emacs' own VCS periodically, a
process which I understand is rather tedious. These packages should be
able to be installed straight from the upstream repositories.

> That's the technical side, but what about the supported/unsupported
> division?  Have you considered making package.el integrated into the
> Emacs core to work like the Cygwin installer, so unsupported or
> experimental packages can be installed and loaded easily?

I hope to get it working that way, yes.

-Phil

Re: Autoload from a web page?

[Phil Hagelberg]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The package manager should support multiple package repositories.
> [...]
>> 3. The FSF repository
>
>> This would contain only packages that are of could quality and
>> generally useful AND for which copyright assignments have been
>> filled out.
>
> For political reasons, we only want to support repositories where
> the copyright is owned by the FSF.  We also may want to support or
> notion of package quality, tho.  Also I find it important that multiple
> versions of a package can coexist at the same time (within reasonable
> feasability constraints: we don't need to handle multiple versions in
> a single Emacs sessions).

I am fine (but not thrilled) with Emacs only including an FSF-owned
repository or repositories out of the box. But I'd like to make sure
that this *doesn't* mean making it difficult for the user to add
third-party repositories for other software they may be interested
in. I'm sure this kind of antifeature wouldn't make it in to Emacs
anyway; I just thought it would be good to make that clear.

-Phil

Re: unsupported packages area in the Emacs repo

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Tom> Recently, Phil Hagelberg <technomancy@gmail.com> has been looking into
Tom> integrating package.el into Emacs.  I don't know how much progress he
Tom> has made.

Ted> Can you and Phil give us your view?  How would you integrate ELPA
Ted> (package.el) with Emacs, specifically using the DVCS features?

I don't think DVCS matters very much.

Basically what is needed is to integrate package.el into Emacs' startup
sequence.  This requires some build-time changes, so that metadata for
packages is extracted from the source tree; and it requires some source
changes, so that such packages are not automatically added to load-path,
but instead are added by package activation.  Only packages that might
be shipped separately will need this treatment.

package.el requires some metadata to operate properly.  Right now this
metadata is generated by some elisp when I upload a package -- see
package-upload-buffer.  This doesn't scale nicely, though, and I think
the future plan should be to generate this metadata in a commit hook on
the server side.  That is, when a package maintainer pushes a new
version, the hook is run and the various server-side files are rebuilt.

Just doing this would be pretty good.  There are a few feature requests
from ELPA users that would be nice to implement, that require similar
treatment, but that could comfortably be done later.

Ted> That's the technical side, but what about the supported/unsupported
Ted> division?  Have you considered making package.el integrated into the
Ted> Emacs core to work like the Cygwin installer, so unsupported or
Ted> experimental packages can be installed and loaded easily?

package.el really has a few different parts.  The stuff for resolving
dependencies and downloading from ELPA is separate from the
install-and-byte-compile bits.  So, you can use package.el to install a
random .el file, assuming it follows the comment guidelines.

One thing I would like for the long term is to see distros use
package.el's mechanism for installing elisp into site-lisp.  That way
users can see the same activation benefits for things not distributed
with Emacs.  This should be pretty easy to implement.

Tom

Re: unsupported packages area in the Emacs repo

[Tom Tromey]

>>>>> "Jonas" == Jonas Bernoulli <jonas@emacsmirror.org> writes:

Jonas> The current implementation is able to generate what I call "epkg"
Jonas> a sexp like this:

Jonas>  :provided   (elx)
Jonas>  :required   ((("dconv" dconv)
Jonas>               ("emacs" cl lisp-mnt)
Jonas>               ("vcomp" vcomp)))

You really need versions, not just package names.
It is common for a package to require a relatively new version of some
other package.

package.el already does this.  What it does is sufficient for many
purposes, but still not completely ideal... there have been cases of
packages being obsoleted by some other package, and package.el has no
notion of this.

Jonas> [Tom: I also have some patches for `package.el' and will send
Jonas> them to you once I find the time to work on it. I would have
Jonas> contacted you some time in January but since this discussion is
Jonas> talking place now: Hi! I hope we will work together well!]

Sure thing.

Tom

Re: Autoload from a web page?

[Tom Tromey]

>>>>> "Jonas" == Jonas Bernoulli <jonas@bernoulli.cc> writes:

Jonas> I do not know what ELPA's position on license issues are.

From http://tromey.com/elpa/upload.html

    ELPA accepts only free software. If in doubt check the DFSG to see if
    your license qualifies. We encourage all authors to assign their code to
    the FSF and to try to get their new packages into the official Emacs
    release.

Jonas> I favor the model where the package manager does clone the
Jonas> repository from the repository

To me this seems to have several negatives for end users.  It takes more
space and requires them to have some particular version control software
installed.

The ELPA approach does not require anything outside of Emacs.

>> - Download a package part by part as the elisp files are needed (kind
>> of autoloading and installing locally through the web).

Jonas> I do not like this idea at all.
[...]
Jonas> The package manager should only operate on complete packages.

Yeah, I agree.  I couldn't think of when I would ever want to just
install some part of a package.

Jonas> And this is precisely because it uses a dvcs. You simply clone
Jonas> the desired package and from that on you simply have to run "git
Jonas> pull" to bring your local copy up-to-date. You do not have to
Jonas> remember where you got the package.

That is not sufficient.  The user also has to make sure the dependencies
are correct, and has to re-byte-compile the files.

Tom

Re: Autoload from a web page?

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

Stefan> Also I find it important that multiple versions of a package can
Stefan> coexist at the same time (within reasonable feasability
Stefan> constraints: we don't need to handle multiple versions in a
Stefan> single Emacs sessions).

Thanks for mentioning this again.

Phil, if you're reading this, this is the one missing required feature
that I keep forgetting about.  We need to have a way to disable a
package to prevent it from being activated.  (But by default, installing
should activate.)

Tom

Re: Autoload from a web page?

[Richard Stallman]

    In this scenario the user wants to download a package, but he may not
    want all of it at once. It is just like autoloading locally, but with
    the difference that the files are loaded from the web (in a well
    defined manner).

This difference is precisely the problem.
Autoloading from files on disk gives you the chance
to edit those files.
Autoloading from a server means you can't do so.

    Yes, I understand that and I read about the problem with javascript.
    The same problems can not arise here since the user will be in control
    of where to download files from. Unlike the situation with javascript
    a single site (or a list of well defined sites) will be where the
    download comes from.

I see your argument, that it is the user who decides to load from
those files.  But I'm not convinced this makes it ok, because it would
be easy for programs to do this without the user's knowing about it.
If we want to be safe, we need to tell users to watch out for this
as a danger sign.

Integrating package.el (was Re: unsupported packages area in the Emacs repo)

[Phil Hagelberg]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
> Ted> Can you and Phil give us your view?  How would you integrate ELPA
> Ted> (package.el) with Emacs, specifically using the DVCS features?
>
> I don't think DVCS matters very much.

I agree that users should not need a DVCS installed on their system to
use packages. However, leveraging a DVCS to get packages from their
authors to a centralized listing seems like a really good idea.

> Basically what is needed is to integrate package.el into Emacs' startup
> sequence.  This requires some build-time changes, so that metadata for
> packages is extracted from the source tree; and it requires some source
> changes, so that such packages are not automatically added to load-path,
> but instead are added by package activation.  Only packages that might
> be shipped separately will need this treatment.
>
> Just doing this would be pretty good.  There are a few feature requests
> from ELPA users that would be nice to implement, that require similar
> treatment, but that could comfortably be done later.

I've actually been thinking in even more incremental steps. Installing
package.el in Emacs without altering any of the existing Emacs code
would be an easy first step and would give some immediate benefit in
terms of packages that are not included in Emacs.

The next step would be to work on package submission. If the centralized
system has a list of packages mapped to a list of DVCS repositories,
they could be polled periodically and all tags matching a certain
convention (say, starting with "v" and followed by a dotted number
series) would be treated as package. That version would then be
processed and published to a downloadable location for clients to pull
in.

I wasn't thinking about integrating packages that Emacs already contains
until after these steps were complete. One thing that may be infeasible
but would certainly simplify things a lot would be if we spun off
packages like org-mode into their own separate DVCS repository and
removed them from the Emacs source tree before making package.el treat
them as packages. However, this may cause some unwanted chaos; I don't
want to barge in and create a lot of work for people. It might also
imply that network access would be necessary to perform a full build of
Emacs since it would have to download bundled packages at compilation
time. Not sure if that is a serious problem.

Anyway, I welcome thoughts on this.

-Phil

Re: Autoload from a web page?

[Lennart Borgman]

On Sat, Jan 2, 2010 at 4:45 PM, Richard Stallman <rms@gnu.org> wrote:
>    In this scenario the user wants to download a package, but he may not
>    want all of it at once. It is just like autoloading locally, but with
>    the difference that the files are loaded from the web (in a well
>    defined manner).
>
> This difference is precisely the problem.
> Autoloading from files on disk gives you the chance
> to edit those files.
> Autoloading from a server means you can't do so.


I have added an optional pause in the download where the user may
check the file before it is byte compiled and used.


>    Yes, I understand that and I read about the problem with javascript.
>    The same problems can not arise here since the user will be in control
>    of where to download files from. Unlike the situation with javascript
>    a single site (or a list of well defined sites) will be where the
>    download comes from.
>
> I see your argument, that it is the user who decides to load from
> those files.  But I'm not convinced this makes it ok, because it would
> be easy for programs to do this without the user's knowing about it.
> If we want to be safe, we need to tell users to watch out for this
> as a danger sign.


Yes, of course. The user must trust the sources. And be able to have a
look at them before using them.


I have implemented this incremental installing now for nXhtml. The
sources are in Launchpad so the user can have a look at them before
downloading. And during the download is another chance.

As I said before there is no contract, no API, for how two access
files from Launchpad this way. A bit unfortunate, but it is still
possible to do use the web interface this way. To make it more
reliable would not take much (but requires cooperation).

Something like ELPA might stand for the necessary API (beside
cooperation, version dependence etc).


Some people argued that downloading files as you need them is not
useful. I think it can be. If something like this was used for ELPA
then different people could depend on each others files. Those who
provides the elisp files would not have to tell others to download
this and that. It would be downloaded automatically. And the sources
could be anywhere (as long as they are trusted).

But as RMS pointed out this has security implications so it could only
be used for reviewed and trusted files/sources. So this would be (IMO)
a special groups of files in ELPA.


If anyone wants to test then please go to
http://www.emacswiki.org/cgi-bin/wiki/NxhtmlMode and follow the
instructions for web-vcs.el there.

If you do you could for example try commands like

  M-x resize-windows
  open and html file
  M-x n-back-game

or anything from the nXhtml menu (do M-x nxhtml-minor-mode if it is not shown).

And please tell me who it works and how you want it to work, it is a
bit beta still ... ;-)

Re: Autoload from a web page?

[Richard Stallman]

    I have added an optional pause in the download where the user may
    check the file before it is byte compiled and used.

That is rather vague, so I don't know exactly what you mean.  Judging
from what you've said, a "chance to check the file" is insufficient to
make it easy for users to install and run edited versions.  So this
autoloading is still harmful.

    Some people argued that downloading files as you need them is not
    useful. I think it can be. If something like this was used for ELPA

The idea of an ELPA is just theoretical at this point, but if we have
one, "autoloading from the net" won't be allowed in it unless you
convince me to approve it.  You won't convince me unless you show me
you understand and care about the issue I am concerned about.

    then different people could depend on each others files.

If that is desirable technically (which is far from clear; it could be
unmodular), there are other ways to do it.  Packages in Debian depend
on other packages and they don't autoload from the net.

But even if it turned out that autoloading gave some advantage, that
isn't a major factor in a decision this important.  A minor
convenience is insignificant compared with the community harm that
this could do.

You are trying to rush us into a major policy change which is
potentially dangerous.  We will not be rushed.  Please slow down!

Please take this change out of your distribution sources, pause for a
week, and then post a clear explanation of what you are proposing now,
so that I can have a chance to see what its consequences could be.

Re: Integrating package.el

[Ted Zlatanov]

On Sat, 02 Jan 2010 21:38:58 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> I've actually been thinking in even more incremental steps. Installing
PH> package.el in Emacs without altering any of the existing Emacs code
PH> would be an easy first step and would give some immediate benefit in
PH> terms of packages that are not included in Emacs.

Agreed.  I think it's mature enough and would benefit from the wider
exposure.  But at that point I think it becomes important to separate
the repository into "Emacs supported," "Emacs unsupported," "ELPA," and
any others deemed necessary.  Then the Emacs-related repositories can be
managed by Emacs maintainers and developers and hosted within the Emacs
repo while ELPA can remain independent.  

Can package.el support that?  It looks to me, from reading the current
source, that it's limited to a single repository.

PH> The next step would be to work on package submission. If the centralized
PH> system has a list of packages mapped to a list of DVCS repositories,
PH> they could be polled periodically and all tags matching a certain
PH> convention (say, starting with "v" and followed by a dotted number
PH> series) would be treated as package. That version would then be
PH> processed and published to a downloadable location for clients to pull
PH> in.

This should probably be a per-repository policy and process.

PH> I wasn't thinking about integrating packages that Emacs already contains
PH> until after these steps were complete. One thing that may be infeasible
PH> but would certainly simplify things a lot would be if we spun off
PH> packages like org-mode into their own separate DVCS repository and
PH> removed them from the Emacs source tree before making package.el treat
PH> them as packages. However, this may cause some unwanted chaos; I don't
PH> want to barge in and create a lot of work for people. It might also
PH> imply that network access would be necessary to perform a full build of
PH> Emacs since it would have to download bundled packages at compilation
PH> time. Not sure if that is a serious problem.

I don't know about the other packages that come with Emacs, but Gnus
could probably benefit from package.el-style installation (I don't speak
for the Gnus project, though, so I can bring it up on the mailing list
if necessary).  A better setup process for Gnus would be really nice,
though.  The usual pre-install and post-install scripts you find in RPM
or DEB would help.  I suspect many other packages would benefit from a
better setup process through package.el.

Ted

Re: Autoload from a web page?

[Ted Zlatanov]

On Wed, 30 Dec 2009 19:06:27 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> I favor the model where the package manager does clone the repository from
JB> the mirror as one of the steps required to install a package.

I think the delivery method should not require DVCS or network access.
The repository should simply be a URL which can point to a file or a
directory or a DVCS location.  Thus a repository snapshot can be
delivered in a single file or over the network (e.g. see Ubuntu's
repositories for a more static version of this).  It should be a single
URL, though, not a set of URLs or some other list of information.
Adding a repository and listing the known repositories should be really
easy.

DVCS only benefits repository maintainers.  End users should be shielded
from DVCS operations IMHO.  They can still benefit from delivery via
DVCS but it should all be hidden.

JB> The package manager should only operate on complete packages.

Agreed.  Piecemeal installation of .el files is unnecessary in at least
90% of the cases I've seen.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> Can package.el support that?  It looks to me, from reading the current
Ted> source, that it's limited to a single repository.

Yes, it is currently limited to a single repository.
This should not be too difficult to fix.

It also needs to be changed to be able to manage multiple install
locations, if it is to be used to manage site-lisp.

Ted> I don't know about the other packages that come with Emacs, but Gnus
Ted> could probably benefit from package.el-style installation (I don't speak
Ted> for the Gnus project, though, so I can bring it up on the mailing list
Ted> if necessary).  A better setup process for Gnus would be really nice,
Ted> though.  The usual pre-install and post-install scripts you find in RPM
Ted> or DEB would help.  I suspect many other packages would benefit from a
Ted> better setup process through package.el.

If you want to try packaging it, I can explain what you need to do.
It is usually quite easy.

Tom

Re: Integrating package.el

[Phil Hagelberg]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> Can package.el support that?  It looks to me, from reading the current
> Ted> source, that it's limited to a single repository.
>
> Yes, it is currently limited to a single repository.
> This should not be too difficult to fix.

I'm not sure why I left it out of my list of goals in my last email;
this is definitely high on my priority list.

I've also got a quick prototype of a DVCS-centric archive management
tool (for gathering packages from library authors and publishing them)
that I will post soon once I've fleshed it out a bit more.

> Ted> I don't know about the other packages that come with Emacs, but Gnus
> Ted> could probably benefit from package.el-style installation (I don't speak
> Ted> for the Gnus project, though, so I can bring it up on the mailing list
> Ted> if necessary).  A better setup process for Gnus would be really nice,
> Ted> though.  The usual pre-install and post-install scripts you find in RPM
> Ted> or DEB would help.  I suspect many other packages would benefit from a
> Ted> better setup process through package.el.
>
> If you want to try packaging it, I can explain what you need to do.
> It is usually quite easy.

I think in general this might be best done on an opt-in basis where the
maintainer of a library indicates that they'd like to convert to a
package and perhaps someone more familiar with package.el can help them
integrate.

-Phil

Re: Integrating package.el

[Lennart Borgman]

On Tue, Jan 5, 2010 at 6:02 AM, Phil Hagelberg <phil@hagelb.org> wrote:
> Tom Tromey <tromey@redhat.com> writes:
>
> I've also got a quick prototype of a DVCS-centric archive management
> tool (for gathering packages from library authors and publishing them)
> that I will post soon once I've fleshed it out a bit more.


This sounds good to me. I believe that ELPA should not hold the files
but rather have pointers to them. Otherwise the turnaround time will
be longer for changes made to the files.

Re: Integrating package.el

[Stefan Monnier]

>> Tom Tromey <tromey@redhat.com> writes:
>> I've also got a quick prototype of a DVCS-centric archive management
>> tool (for gathering packages from library authors and publishing them)
>> that I will post soon once I've fleshed it out a bit more.
> This sounds good to me. I believe that ELPA should not hold the files
> but rather have pointers to them. Otherwise the turnaround time will
> be longer for changes made to the files.

The default repositor(y|ies) will want to be under FSF control (not just
the copyright, but we also want to have a copy of the files, etc...) so
we can assume that the FSF will usually also hold the "master
repository" for those packages.

I.e. we probably don't want to only hold pointers to files.  Tho of
course, such an indirection might still be useful somewhere.


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 04 Jan 2010 12:51:06 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.

Good point, but do you mean multiple search locations for existing
installs?  I would expect a single install location, otherwise it gets
complicated...

Ted> I don't know about the other packages that come with Emacs, but Gnus
Ted> could probably benefit from package.el-style installation (I don't speak
Ted> for the Gnus project, though, so I can bring it up on the mailing list
Ted> if necessary).  A better setup process for Gnus would be really nice,
Ted> though.  The usual pre-install and post-install scripts you find in RPM
Ted> or DEB would help.  I suspect many other packages would benefit from a
Ted> better setup process through package.el.

Tom> If you want to try packaging it, I can explain what you need to do.
Tom> It is usually quite easy.

I think the pre-install and post-install steps are pretty important.
Without them, packaging Gnus doesn't do much.  I want the post-install
to actually set up the user's Gnus configuration.  This has been a very
common complaint about Gnus.  Installing it is the easy part, since it
comes with Emacs.  Can package.el support that?  I can't tell if this is
the "activate" or the "load" stage (are they states or state
transitions?  English can be so ambiguous...) or something new; also it
seems like this is something external to Gnus, a function of the ELPA
wrapper (although it may be bundled within Gnus) rather than something
Gnus will always run for new users.  This is the major question I have
before I propose this packaging on the Gnus mailing list.

I also want to know if you and Phil want to make Gnus your first
Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.
I'd actually like both: the Emacs-hosted version is "Emacs-supported
Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
"latest Gnus release."  The last one is usually but not always
synchronized with the Emacs release; I'll need to ask the other Gnus
developers what they think.  At least two Gnus versions in package.el
make sense in any case.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 10:06:56 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> Tom Tromey <tromey@redhat.com> writes:
>>> I've also got a quick prototype of a DVCS-centric archive management
>>> tool (for gathering packages from library authors and publishing them)
>>> that I will post soon once I've fleshed it out a bit more.
>> This sounds good to me. I believe that ELPA should not hold the files
>> but rather have pointers to them. Otherwise the turnaround time will
>> be longer for changes made to the files.

SM> The default repositor(y|ies) will want to be under FSF control (not just
SM> the copyright, but we also want to have a copy of the files, etc...) so
SM> we can assume that the FSF will usually also hold the "master
SM> repository" for those packages.

SM> I.e. we probably don't want to only hold pointers to files.  Tho of
SM> course, such an indirection might still be useful somewhere.

The "package file" (analogous to a RPM/DEB package fike) should contain
the real, final version of all the files.  The package repository may
accumulate metadata about all the package files it contains, but I
should be able to copy a single package file to another system and
install it.  As a sysadmin I don't want package files to be
indeterminate.  For instance, how can I set up a local mirror if some of
the files inside some of the package files are possibly remote?  There's
all the other security risks I listed in my previous note, too,
concerning remote network access.

This is a huge sysadmin problem with Perl for instance, where the
liberal packaging standard and complicated build process make it hard to
synchronize packages across multiple installations.  I've suffered
through that many times and hope it doesn't recur with Emacs packages.
Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
as a possibility.

Ted

Re: Integrating package.el

[Stefan Monnier]

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.
> Good point, but do you mean multiple search locations for existing
> installs?  I would expect a single install location, otherwise it gets
> complicated...

You need at least 2 install locations: one for user-installed packages
and one for admin-installed packages (the "site-lisp" mentioned above).

> I think the pre-install and post-install steps are pretty important.
> Without them, packaging Gnus doesn't do much.  I want the post-install
> to actually set up the user's Gnus configuration.  This has been a very

This is not part of an install, it's a separate step (a "configuration"
step).  Consider the case where the package is installed in site-lisp,
where it needs to work for all users.

"packaging Gnus" is only supposed to solve the problem of replacing
a version of Gnus with another independently from the version of Emacs
you use.

> seems like this is something external to Gnus, a function of the ELPA
> wrapper (although it may be bundled within Gnus) rather than something
> Gnus will always run for new users.  This is the major question I have
> before I propose this packaging on the Gnus mailing list.

I don't think it needs to be external to Gnus.  You're basically asking
for a Gnus wizard, I think.


        Stefan

Re: Integrating package.el

[Stefan Monnier]

> The "package file" (analogous to a RPM/DEB package fike) should contain
> the real, final version of all the files.  The package repository may
> accumulate metadata about all the package files it contains, but I
> should be able to copy a single package file to another system and
> install it.  As a sysadmin I don't want package files to be
> indeterminate.  For instance, how can I set up a local mirror if some of
> the files inside some of the package files are possibly remote?  There's
> all the other security risks I listed in my previous note, too,
> concerning remote network access.

> This is a huge sysadmin problem with Perl for instance, where the
> liberal packaging standard and complicated build process make it hard to
> synchronize packages across multiple installations.  I've suffered
> through that many times and hope it doesn't recur with Emacs packages.
> Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
> as a possibility.

OK, I'm not sure whether we're talking about the same things.  The way
I see it, there will be the following elements:
- a (set of) Bzr repository holding the package sources.
- a tool that will build tarballs from those sources.
- an area where those tarballs are stored, along with some metadata
  (think Debian repository).
- a tool that can scan such repositories and downlooad packages from
  them, obeying dependencies (think APT).
- a tool that can install/activate/uninstall a given package, ... (think
  DPKG).  This last one is what I tried to write when I wrote install.el
  many moons ago.


        Stefan

Installation, activation and setup of packages (e.g. Gnus) (was: Integrating package.el)

[Reiner Steib]

On Tue, Jan 05 2010, Ted Zlatanov wrote:

> On Mon, 04 Jan 2010 12:51:06 -0700 Tom Tromey <tromey@redhat.com> wrote: 
> Ted> A better setup process for Gnus would be really nice, though.

ACK

> Ted> The usual pre-install and post-install scripts you find in RPM
> Ted> or DEB would help.

Those don't start per-user setup.  The (missing) setup process (using
Gnus half-done assistant.el or anything else) is per-user: configuring
news server, mail sources, IMAP, SMTP.

> Ted> I suspect many other packages would benefit from a better setup
> Ted> process through package.el.
>
> Tom> If you want to try packaging it, I can explain what you need to do.
> Tom> It is usually quite easy.
>
> I think the pre-install and post-install steps are pretty important.
> Without them, packaging Gnus doesn't do much.  I want the post-install
> to actually set up the user's Gnus configuration.  This has been a very
> common complaint about Gnus.  

I don't think the purpose of package.el is to do or start the user
setup.  The Gnus assistant is supposed to be run when a *user* fires
up Gnus for the first time, which is after the package is already
installed and "activated" (by the *administrator*).  I would oppose to
mix admistrative tasks (installation) with user setup.

> Installing it is the easy part, since it comes with Emacs.  Can
> package.el support that?  I can't tell if this is the "activate" or
> the "load" stage (are they states or state transitions?  English can
> be so ambiguous...) or something new; also it seems like this is
> something external to Gnus, a function of the ELPA wrapper (although
> it may be bundled within Gnus) rather than something Gnus will
> always run for new users.  This is the major question I have before
> I propose this packaging on the Gnus mailing list.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Installation, activation and setup of packages (e.g. Gnus) (was: Integrating package.el)

[Ted Zlatanov]

On Tue, 05 Jan 2010 11:42:47 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> seems like this is something external to Gnus, a function of the ELPA
>> wrapper (although it may be bundled within Gnus) rather than something
>> Gnus will always run for new users.  This is the major question I have
>> before I propose this packaging on the Gnus mailing list.

SM> I don't think it needs to be external to Gnus.  You're basically asking
SM> for a Gnus wizard, I think.

Yeah, "gnus/assistant.el" is capable enough to do it.  It's been on my
TODO list for a while to write some basic configuration assistants.

On Tue, 05 Jan 2010 18:27:31 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Tue, Jan 05 2010, Ted Zlatanov wrote:

Ted> The usual pre-install and post-install scripts you find in RPM
Ted> or DEB would help.

RS> Those don't start per-user setup.  The (missing) setup process (using
RS> Gnus half-done assistant.el or anything else) is per-user: configuring
RS> news server, mail sources, IMAP, SMTP.

>> I think the pre-install and post-install steps are pretty important.
>> Without them, packaging Gnus doesn't do much.  I want the post-install
>> to actually set up the user's Gnus configuration.  This has been a very
>> common complaint about Gnus.  

RS> I don't think the purpose of package.el is to do or start the user
RS> setup.  The Gnus assistant is supposed to be run when a *user* fires
RS> up Gnus for the first time, which is after the package is already
RS> installed and "activated" (by the *administrator*).  I would oppose to
RS> mix admistrative tasks (installation) with user setup.

Well, this can be debated for a while.  I think users expect an
installed package to work (Gnus doesn't do much if just installed) and
it's convenient to set it up when installed instead of later.  System
packages these days are commonly set up like that, e.g. Postfix on a
Debian system asks several basic setup questions IIRC.

OTOH it's often painful to have to answer setup questions when you're
installing packages, especially when you don't want or don't have an
interactive session.

My proposal is:

1) at least show a message when installing Gnus that the first-time
assistant can be run with commands X and Y.  If the session is
interactive, ask the user if he wants to do it now.  This requires some
package.el support.

2) make Gnus smarter about starting up without any configuration.  The
assistant should run automatically in this case.  As a start, provide
links to the documentation, but work on making the assistant interactive
and capable of setting up news server, mail sources (IMAP or POP), an
IMAP server, a Maildir, SMTP, etc.  GMail should have a specific setup
path since it's so popular.  This is entirely on the Gnus side.

Ted

Re: Integrating package.el

[Phil Hagelberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I also want to know if you and Phil want to make Gnus your first
> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

I'm actually somewhat intimidated by the thought of packaging gnus. I'd
much rather start with supporting smaller packages first, but if someone
more familiar with hacking gnus wants to start work on packaging it I
would be happy to provide support.

-Phil

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 10:03:18 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I also want to know if you and Phil want to make Gnus your first
>> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

PH> I'm actually somewhat intimidated by the thought of packaging gnus. I'd
PH> much rather start with supporting smaller packages first, but if someone
PH> more familiar with hacking gnus wants to start work on packaging it I
PH> would be happy to provide support.

I'll go to the Gnus mailing list and discuss this further, but at least
for now let's assume Reiner or I will make the package.  I don't mind
doing the work, though I can't promise I'll get it done quickly.  I
still need to know about the 2+ versions of Gnus that I mentioned
earlier--how do you want to deal with that?  Also, how do you want to
package an application that's already part of Emacs for the
"Emacs-hosted Gnus" version?

Thanks
Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Tom> It also needs to be changed to be able to manage multiple install
Tom> locations, if it is to be used to manage site-lisp.

Ted> Good point, but do you mean multiple search locations for existing
Ted> installs?  I would expect a single install location, otherwise it gets
Ted> complicated...

Basically we need to handle 3 install locations: the Emacs install tree,
the global site-lisp, and the user's install location.

These only need to be handled during the activation step.  When the user
installs a new package, it should always go to his personal install
location.

Right now package.el handles the Emacs install tree in a hacky way and
ignores site-lisp entirely.

Ted> This has been a very
Ted> common complaint about Gnus.  Installing it is the easy part, since it
Ted> comes with Emacs.  Can package.el support that?  I can't tell if this is
Ted> the "activate" or the "load" stage (are they states or state
Ted> transitions?  English can be so ambiguous...) or something new; also it
Ted> seems like this is something external to Gnus, a function of the ELPA
Ted> wrapper (although it may be bundled within Gnus) rather than something
Ted> Gnus will always run for new users.  This is the major question I have
Ted> before I propose this packaging on the Gnus mailing list.

This would be a new thing.

I agree with the other posters who recommend that this be done in Gnus,
not as something related to package.el.

Ted> I also want to know if you and Phil want to make Gnus your first
Ted> Emacs-hosted package, or if there's going to be a ELPA clone of Gnus.

FWIW, ELPA already includes some packages that are also included in
Emacs.  I maintain the necessary metadata for this by hand (and, I'm
afraid, not very well at the moment due to lack of time -- I think I am
missing some updates).

Ted> I'd actually like both: the Emacs-hosted version is "Emacs-supported
Ted> Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
Ted> "latest Gnus release."  The last one is usually but not always
Ted> synchronized with the Emacs release; I'll need to ask the other Gnus
Ted> developers what they think.  At least two Gnus versions in package.el
Ted> make sense in any case.

package.el doesn't support this at the moment.

Thanks for bringing this up, though.  I think it is pretty important to
flush out all these use cases.

I can think of a couple solutions to this problem.

One would be to somehow let users select different package versions.
Normally, though, users don't actually want this -- when a new version
is released, ordinarily all the old ones are obsolete.

Another solution would be to have two repositories, one for stable
packages and one for experimental.  This wouldn't require any changes to
package.el.

Tom

Re: Installation, activation and setup of packages (e.g. Gnus)

[Stefan Monnier]

> Well, this can be debated for a while.

I don't think there's much to debate: when the admin installs a user
package, it can't automatically configure it for all the users
(especially not for the users that haven't yet been created).
Sometimes, some part of the package depends on site data, so that part
might be configured when the admin installs the package, but usually
this is very limited (e.g. in the case of Gnus, the admin might set the
DNS name of the local NNTP server [tho I'd sadly expect nowadays that
for most admins there's simply no such local NNTP server to specify]).

> I think users expect an installed package to work (Gnus doesn't do
> much if just installed) and it's convenient to set it up when
> installed instead of later.

When I start `evolution', it doesn't "just work": it starts by running
the wizard.  How could my sysadmin know that I read my email from
imap.iro.umontreal.ca rather than pop.teksavvy.com?

> System packages these days are commonly set up like that, e.g. Postfix
> on a Debian system asks several basic setup questions IIRC.

The key word above is "System": Postfix is a *system* package, Gnus isn't.

> 1) at least show a message when installing Gnus that the first-time
> assistant can be run with commands X and Y.  If the session is
> interactive, ask the user if he wants to do it now.  This requires some
> package.el support.

It might be a good idea to let the package management run a wizard when
a user package is installed by a user (or when a system package is
installed, tho there are very few Emacs system packages).

> 2) make Gnus smarter about starting up without any configuration.  The
> assistant should run automatically in this case.

Sounds fine.  But that's a Gnus issue, unrelated to packaging.

> GMail should have a specific setup path since it's so popular.

I definitely do not want Emacs to encourage the use of GMail over any
other service.


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 12:14:14 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> Right now package.el handles the Emacs install tree in a hacky way
Tom> and ignores site-lisp entirely.

OK, I hope you and Phil have some ideas.  I don't know much about this.

Tom> I agree with the other posters who recommend that [post-install
Tom> configuration] be done in Gnus, not as something related to
Tom> package.el.

I would like to at least show a message or offer a prompt (answering 'y'
launches the assistant), is that possible?  If not that's OK.

Ted> I'd actually like both: the Emacs-hosted version is "Emacs-supported
Ted> Gnus" while the ELPA versions are "bleeding-edge Gnus from CVS" and
Ted> "latest Gnus release."  The last one is usually but not always
Ted> synchronized with the Emacs release; I'll need to ask the other Gnus
Ted> developers what they think.  At least two Gnus versions in package.el
Ted> make sense in any case.

Tom> package.el doesn't support this at the moment.

Tom> Thanks for bringing this up, though.  I think it is pretty important to
Tom> flush out all these use cases.

Tom> I can think of a couple solutions to this problem.

Tom> One would be to somehow let users select different package versions.
Tom> Normally, though, users don't actually want this -- when a new version
Tom> is released, ordinarily all the old ones are obsolete.

This is probably too confusing for most users.

Tom> Another solution would be to have two repositories, one for stable
Tom> packages and one for experimental.  This wouldn't require any changes to
Tom> package.el.

This works for me.  We're talking now about the proposed FSF supported
repository (part of Emacs), not about an ELPA "stable" repository,
right?  The bleeding-edge Gnus can live in the external ELPA archive
that's currently live, and perhaps can move to the FSF unsupported
repository we discussed earlier (you and Phil may want different names
for the repositories).

I think RMS, Stefan, Reiner, and you and Phil need to agree on the setup
and where things will live.  Jonas will probably have an opinion as
well.  I can do the mechanical work for packaging Gnus but can't make
this kind of policy decision.  Whatever the decision, Gnus is a good
test case because of its complexity and baggage[1].

Ted

[1] by "baggage" I mean that Gnus has a handbag, two purses, six
wallets, a rolling suitcase, a laundry basket, and also stuffs things in
its socks "just in case." :)

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 11:47:46 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The "package file" (analogous to a RPM/DEB package fike) should contain
>> the real, final version of all the files.  The package repository may
>> accumulate metadata about all the package files it contains, but I
>> should be able to copy a single package file to another system and
>> install it.  As a sysadmin I don't want package files to be
>> indeterminate.  For instance, how can I set up a local mirror if some of
>> the files inside some of the package files are possibly remote?  There's
>> all the other security risks I listed in my previous note, too,
>> concerning remote network access.

>> This is a huge sysadmin problem with Perl for instance, where the
>> liberal packaging standard and complicated build process make it hard to
>> synchronize packages across multiple installations.  I've suffered
>> through that many times and hope it doesn't recur with Emacs packages.
>> Some OS integration (DEB, RPM, MacOS X, etc.) would be useful, at least
>> as a possibility.

SM> OK, I'm not sure whether we're talking about the same things.

I digressed too much, sorry.

SM> The way I see it, there will be the following elements:

SM> - a (set of) Bzr repository holding the package sources.

Yes, with initial members "FSF supported," "FSF unsupported" (both
hosted inside the Emacs Bzr repo) and "ELPA" (hosted externally, may not
be a Bazaar repository at all).  This is the storage backend.  The "FSF
supported" storage may be the lisp/ directory inside Emacs, for
instance.

SM> - a tool that will build tarballs from those sources.

Yes, including the necessary package metadata inside the file.  This
tool will probably come from ELPA.

SM> - an area where those tarballs are stored, along with some metadata
SM>   (think Debian repository).

All of this is MHO: there should be three areas to parallel the storage
backends above.  They may be stored inside the backend or externally.
The package repositories should be identified by a single URL; the ones
that come with Emacs should point to a secure Savannah URL.  That may
address RMS' concerns about loading software over the network.  Jonas'
emacsmirror.org would be a fourth package repository, probably not
enabled by default but easy to enable.

SM> - a tool that can scan such repositories and downlooad packages from
SM>   them, obeying dependencies (think APT).

This is package.el IIUC.

SM> - a tool that can install/activate/uninstall a given package, ... (think
SM>   DPKG).  This last one is what I tried to write when I wrote install.el
SM>   many moons ago.

This is also package.el IIUC.

Ted

Gnus encouragement of GMail (was: Installation, activation and setup of packages (e.g. Gnus))

[Ted Zlatanov]

On Tue, 05 Jan 2010 14:43:33 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

> Ted Z wrote:
>> GMail should have a specific setup path since it's so popular.

SM> I definitely do not want Emacs to encourage the use of GMail over
SM> any other service.

Well, users are constantly pointed to the relevant EmacsWiki page for
GMail support in Gnus.  The settings required are not intuitive.  So we
either accept the support burden and frustrated users, or we give them
the sensible settings as I suggested, or we point them to the
EmacsWiki.

I don't think the second and third options encourage GMail use in
particular.  They are just tweaks for those who use GMail IMAP.  GMail
is the only major IMAP provider to require well-known tweaks; when
others arise we'll handle them.

Similarly, there's code in Gnus to address Exchange IMAP issues; does
that encourage Exchange use?  I don't think so.  IMHO it actually
broadens the appeal of free software to support Exchange IMAP better.

I guess a fourth option is to write gnus-gmail.el, host it externally,
and make it clear that it's not endorsed by the FSF.  But that's no
better--how will the users know they need gnus-gmail.el?  I'd rather
include the support and make it clear it's not an endorsement.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I would like to at least show a message or offer a prompt (answering 'y'
Ted> launches the assistant), is that possible?  If not that's OK.

It is sort of possible.

You can do anything you like in an autoload.  These will be evaluated
when the package is activated, which does happen when the package is
installed.

I think this would be pretty unfriendly to users, though.  For example,
what if the user installs several packages at once?

Tom

Re: Integrating package.el

[Jonas Bernoulli]

2010/1/5 Ted Zlatanov <tzz@lifelogs.com>:
> All of this is MHO: there should be three areas to parallel the storage
> backends above.  They may be stored inside the backend or externally.
> The package repositories should be identified by a single URL; the ones
> that come with Emacs should point to a secure Savannah URL.  That may
> address RMS' concerns about loading software over the network.  Jonas'
> emacsmirror.org would be a fourth package repository, probably not
> enabled by default but easy to enable.

Sounds good to me. I by now means expect my mirror to be enabled by default,
but it would certainly be nice if that were possible.

-- Jonas

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 05 Jan 2010 16:19:56 -0700 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> I would like to at least show a message or offer a prompt (answering 'y'
Ted> launches the assistant), is that possible?  If not that's OK.

Tom> It is sort of possible.

Tom> You can do anything you like in an autoload.  These will be evaluated
Tom> when the package is activated, which does happen when the package is
Tom> installed.

Tom> I think this would be pretty unfriendly to users, though.  For example,
Tom> what if the user installs several packages at once?

OK, if it doesn't fit with the package.el flow, let's drop it.  Stefan
and Reiner are also not crazy about the idea so it's probably a bad one
:)

I'll make the assistant run on the first Gnus startup.  There's several
other issues I raised, but at least this one can be safely ignored.

Ted

Re: Integrating package.el

[Richard Stallman]

    > The package repositories should be identified by a single URL; the ones
    > that come with Emacs should point to a secure Savannah URL.  That may
    > address RMS' concerns about loading software over the network.  

Alas that doesn't affect the issue.  Running software directly from
the network is bad because it is a process that invites users to give
up control.

Who wrote the program makes no difference, not for this.
Where the program comes from makes no difference.
Suppose I wrote the program and people fetch it from gnu.org:
that makes no difference.  It is still bad for people to
adopt a practice that gives them less control.

So we won't lead people in that direction.

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 06 Jan 2010 15:45:37 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> The package repositories should be identified by a single URL; the ones
>> that come with Emacs should point to a secure Savannah URL.  That may
>> address RMS' concerns about loading software over the network.  

RS> Alas that doesn't affect the issue.  Running software directly from
RS> the network is bad because it is a process that invites users to give
RS> up control.

RS> Who wrote the program makes no difference, not for this.
RS> Where the program comes from makes no difference.
RS> Suppose I wrote the program and people fetch it from gnu.org:
RS> that makes no difference.  It is still bad for people to
RS> adopt a practice that gives them less control.

RS> So we won't lead people in that direction.

Sorry for confusing the issue between the package.el discussion and
Lennart's discussion.  I was talking about installing software, not
running it directly from the repository.  I would expect packages to be
signed by the package repository maintainer as well, ensuring integrity.

Ted

Re: Integrating package.el

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The way I see it, there will be the following elements:
> - a (set of) Bzr repository holding the package sources.
> - a tool that will build tarballs from those sources.
> - an area where those tarballs are stored, along with some metadata
>   (think Debian repository).
> - a tool that can scan such repositories and downlooad packages from
>   them, obeying dependencies (think APT).
> - a tool that can install/activate/uninstall a given package, ... (think
>   DPKG).  This last one is what I tried to write when I wrote install.el
>   many moons ago.

Here are the pieces to the puzzle from my initial brainstorming:

* An index file contains a alist mapping projects to DVCS
  repositories. This would be manually curated. Adding a project would
  just be done by adding an entry to this file.

* A function that builds packages for every version in the repository of
  each project. A version is defined by a tag matching a given pattern.

* The packages are deployed to an archive source, which is publicly
  available over HTTP.

* Currently package.el can perform the last two points from your list
  (scan/download/dependencies and install/activate/uninstall).

The attached file implements the server-side maintenance, albeit in a
rough manner. It's limited to single-file packages kept in git
repositories, simply because that's what I'm personally familiar with,
though I don't think supporting other DVCSes would be tricky at all.

I welcome comments on this. Once the server-side maintenance tool is
ready, I want to add support for multiple sources to the client-side
package.el, and once that's done supporting multiple install locations
on disk (as Tom has explained) would be the next step.

Right now the code is at http://github.com/technomancy/package.el,
though I've attached a copy of the relevant file.

-Phil


[-- Attachment #2: package-maint.el --]
[-- Type: application/emacs-lisp, Size: 7092 bytes --]

Re: Integrating package.el

[Richard Stallman]

    * An index file contains a alist mapping projects to DVCS
      repositories. This would be manually curated. Adding a project would
      just be done by adding an entry to this file.

This could be ok, but that shouldn't be misunderstood as a decision
to include libraries that we don't have papers for.

Re: Integrating package.el

[Phil Hagelberg]

Richard Stallman <rms@gnu.org> writes:

>     * An index file contains a alist mapping projects to DVCS
>       repositories. This would be manually curated. Adding a project would
>       just be done by adding an entry to this file.
>
> This could be ok, but that shouldn't be misunderstood as a decision
> to include libraries that we don't have papers for.

Naturally, it's up to whoever manually curates the index to ensure the
package follows the guidelines for that archive. For the FSF archive(s)
it would mean ensuring copyright assignment is in place; for folks who
want to maintain their own archives it would probably be something
simpler like making sure it's licensed as free software.

-Phil

Re: Integrating package.el

[Richard Stallman]

    Naturally, it's up to whoever manually curates the index to ensure the
    package follows the guidelines for that archive. For the FSF archive(s)
    it would mean ensuring copyright assignment is in place; for folks who
    want to maintain their own archives it would probably be something
    simpler like making sure it's licensed as free software.

I am worried that this will lead to a problem.
Namely, there may be increasing numbers of packages
that we can't get adequate papers for and can't put into Emacs.

Re: Integrating package.el

[Stephen J. Turnbull]

Richard Stallman writes:
 >     Naturally, it's up to whoever manually curates the index to ensure the
 >     package follows the guidelines for that archive. For the FSF archive(s)
 >     it would mean ensuring copyright assignment is in place; for folks who
 >     want to maintain their own archives it would probably be something
 >     simpler like making sure it's licensed as free software.
 > 
 > I am worried that this will lead to a problem.
 > Namely, there may be increasing numbers of packages
 > that we can't get adequate papers for and can't put into Emacs.

This is a problem as opposed to increasing numbers of packages that
are available only to their authors and you can't put into Emacs?  At
least this way you'll have a contact address for authors.

Re: Integrating package.el

[Chong Yidong]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>  >     Naturally, it's up to whoever manually curates the index to ensure the
>  >     package follows the guidelines for that archive. For the FSF archive(s)
>  >     it would mean ensuring copyright assignment is in place; for folks who
>  >     want to maintain their own archives it would probably be something
>  >     simpler like making sure it's licensed as free software.
>  > 
>  > I am worried that this will lead to a problem.
>  > Namely, there may be increasing numbers of packages
>  > that we can't get adequate papers for and can't put into Emacs.
>
> This is a problem as opposed to increasing numbers of packages that
> are available only to their authors and you can't put into Emacs?  At
> least this way you'll have a contact address for authors.

No, it's a valid concern.  It's hard work hunting down papers and
integrating packages.  We must make sure a package system does not tempt
us to just "dump and forget".

Re: Integrating package.el

[joakim]

Chong Yidong <cyd@stupidchicken.com> writes:

> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
>> Richard Stallman writes:
>>  >     Naturally, it's up to whoever manually curates the index to ensure the
>>  >     package follows the guidelines for that archive. For the FSF archive(s)
>>  >     it would mean ensuring copyright assignment is in place; for folks who
>>  >     want to maintain their own archives it would probably be something
>>  >     simpler like making sure it's licensed as free software.
>>  > 
>>  > I am worried that this will lead to a problem.
>>  > Namely, there may be increasing numbers of packages
>>  > that we can't get adequate papers for and can't put into Emacs.
>>
>> This is a problem as opposed to increasing numbers of packages that
>> are available only to their authors and you can't put into Emacs?  At
>> least this way you'll have a contact address for authors.
>
> No, it's a valid concern.  It's hard work hunting down papers and
> integrating packages.  We must make sure a package system does not tempt
> us to just "dump and forget".

A well made package system might also conceivably make it easier to
acquire papers right?

For instance, it might be more prestigious to have ones package in the
FSF repos.

Also, digital signatures might be legally binding at some point. (I think
they are in Swedenalready, we have a national system for it anyway)

>
-- 
Joakim Verona

Re: Integrating package.el

[Phil Hagelberg]

Chong Yidong <cyd@stupidchicken.com> writes:

>>> I am worried that this will lead to a problem.
>>> Namely, there may be increasing numbers of packages
>>> that we can't get adequate papers for and can't put into Emacs.
>>
>> This is a problem as opposed to increasing numbers of packages that
>> are available only to their authors and you can't put into Emacs?  At
>> least this way you'll have a contact address for authors.
>
> No, it's a valid concern.  It's hard work hunting down papers and
> integrating packages.  We must make sure a package system does not tempt
> us to just "dump and forget".

Can you explain what you mean by this? Are you saying Emacs perhaps
should not have a package manager, or just that we need to be careful
about what packages we allow into the repository?

I'm going to continue my work on package.el whether or not it is
accepted into Emacs, but it would be a shame to make users go through
extra installation steps to use it.

If package.el is included, I will make every effort to assign copyright
for the projects I maintain for which I can, but there are some for
which it simply won't be possible due to my accepting patches by email
and then losing some of my old email archives in a hard drive crash. I
would hate for this situation to impede adding useful features to Emacs.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

joakim@verona.se writes:

>> No, it's a valid concern.  It's hard work hunting down papers and
>> integrating packages.  We must make sure a package system does not tempt
>> us to just "dump and forget".
>
> A well made package system might also conceivably make it easier to
> acquire papers right?
>
> For instance, it might be more prestigious to have ones package in the
> FSF repos.

In my mind it has little to do with prestige and more to do with
streamlining the installation process for users of my projects. Being
able to tell users "install $FOO through package.el" rather than giving
a many-step install process means it will be easier for users to
discover and try out my projects, not to mention keeping them up to date
with the latest version. For this benefit I (and surely many other
package authors) would be willing to put up with the hassle of paperwork
where possible, thereby increasing the number of packages for which the
FSF owns copyright.

-Phil

Re: Integrating package.el

[Stefan Monnier]

> I'm going to continue my work on package.el whether or not it is
> accepted into Emacs, but it would be a shame to make users go through
> extra installation steps to use it.

Having myself written a package manager (install.el), I'm clearly fully
in favor of integrating something like package.el.
I don't think there's necessarily much work to be done to integrate it.

E.g. It is not necessary to set up a VCS area to keep the package
sources (and other such things) before integrating package.el.

Basically, all I want is for it to be able to distinguish between
"packages available locally, either installed by the user or the
sysadmin", and "packages that will be autoloaded in a given session", so
as to be able to install different versions of the same package at the
same time.


        Stefan

Re: Integrating package.el

[MON KEY]

Joakim Verona writes,
> A well made package system might also conceivably make it easier to
> acquire papers right?
>
> For instance, it might be more prestigious to have ones package in the
> FSF repos.
>
> Also, digital signatures might be legally binding at some point. (I think
> they are in Swedenalready, we have a national system for it anyway)

I don't get why the licensing/assignment need necessarily be an issue.
For example, I was issued a `.pdf' file for my assignment.

The file name has format: `<LASTNAME>.<NUMBER>.EMACS.pdf'

Following the letterhead on pg 1 are the lines:

,----
|                                       SURNAME-UID  <--BLOCK-1
|                                         RT NNNNNN  <--BLOCK-2
| ASSIGNMENT - GNU EMACS                             <--BLOCK-3
`----

 Following that, the opening paragraph reads:

,----
| For $1 and other good { boilerplate elided }
| hereinafter "FSF", <MY-FULL-NAME>, hereinafter      <--BLOCK-4
| "DEVELOPER", hereby agrees as follows:
`----

The bottom of page two has two signature blocks:

,----
|      Thank you for the contribution!
|
| Signed:
|
| __<MY_SIGNATURE>__
|                                                    <--BLOCK-5
| __<DATE_SIGNED>___
|
|
| Accepted by the Free Software Foundation
|
| __<EXECUTIVE_DIR_SIG__
|                                                    <--BLOCK-6
| __<DATE_SIGNED>_______
|
`----

The scan also has an inked date stamp which appears in the manner of a Bates
number.

:NOTE The above template is recent, obv. older such docs may differ.

While I'm quite sure the FSF has facilities to OCR/digitize this type of content
in bulk it is surely outside their purview to extend such facility (and time
required to do so) to the Emacs project or any other GNU project FTM.

That said, Joakim's has developed dragbox.el which using GOCR should already be
capable of the following:

o Extracting digital images key portions of the document Blocks 1-6,

o Extracting OCR'd conversion of Blocks 1-4,

The `Bates Date stamp' (BLOCK-7) would prob. need to be manually identified and
extracted as these most likely appear skewed, incompletely inked, and
arbitrarily placed for the majority of assignment papers. Regardless, Joakim's
dragbox.el already provides adequate facilities for accommodating a manual crops
of BLOCK-7 as well.

With such extractions, the digital image portions of Blocks 1-7, and the OCR'd
conversion of Blocks 1-4, a `sanctioned repository maintainer' would
key-sign/encrypt
each block (or the entire bundle) of extracted content for:

o The package author(s);

o The Emacs-devels;

This encrypted bundle could be and distributed with the package or retained
separately.  In either case, the bundle would receive a Unique ID which would be
included in the header commentary of each file distributed with the package.

Assuming all the elements of this regime are technically possible what are the
benefits it would provide that other approaches wouldn't:

o It promotes user awareness of, and re-assures the user benefits of, using a
  suite of congruent and consistently licensed packages.

o It promotes and encourages package authors to acquire assignment _papers_ in
  addition to incorporating boilerplate license terms in package/library
  headers.

o Once an authors assignment papers have been processed per above it provides a
  mechanism whereby package authors can readily integrate new libraries with a
  sanctioned repository without additional overhead. IOW assuming no change of
  employment, surname, etc. an author simply notifies the repository maintainer
  that a new package exists and that said new package should fall under the
  existing Unique ID umbrella.

o Changes in assignment status e.g. withdrawal of assignment, change of
  employment, contact address, surname, etc. can be more readily propagates
  through the system. The repository maintainer is notified and alters the scope
  of the Unique ID umbrella accordingly; this alteration is made known to users
  and devels upon next synchronization.

o It can be integrated with existing assignment papers dating back to ???.

o It reinforces the existing practice that assignment papers need be signed on
  physical media (e.g. _paper_).

o It allows upstream Emacs-devels to verify assignment should they elect to
  incorporate a package or portions of code therein with less burdensome
  overhead.

o It may remove some of the initial immediate overhead Emacs-devels face with
  regards incorporation of new features in the absence of existing assignment
  papers. IOW going forward new packages/authors will presumably already have
  acquired papers and these papers will already have been converted processed.

o It may ease management of existing/future assignment papers w/re upstream
  Emacs-devels, i.e. by breaking up the assignment document into discrete
  elements Blocks 1-7 could be databased in any number of ways including xrefs
  across multiple authors/packages, verification of date and timestamp
  incongruities etc.

Assuming the above are points are indeed beneficial what are the detractors:

o It places a burden on package authors that may not have been there before;
  namely, acquisition of assignment papers.

o For new authors there is a time-delay between the initial creation of
  package/source and any sequencing of assignment papers, digitization,
  propagation etc. required before a package can become part of a
  repository. IOW An author can publish code with a GPL header to github,
  bit-bucket, launchpad, etc. in seconds.

o It places a significant burden on `sanctioned repository maintainers' to
  accomplish the initial digital conversion and subsequent propagation to
  appropriate parties.

o It may place a burden on FSF if it has to process more assignment papers.

o It doesn't (currently) address the issue of older assignment papers that are
  in a different format, are lost, out of date, etc.

o It doesn't accommodate purely digital code signing models.

o It doesn't accommodate alternative licenses (philosophical negative).

o It places a burden on users to grok the benefits/rationale of what in other
  situations would be considered an esoteric approach, i.e. One can _fax_ a
  contract that is binding but authors/contributors of a premier piece of
  software must make available hand signed media.

o None of this exists yet.

Personally, I believe there is a net gain for package authors when there is a
cost of entry to a project.  AFAIK Right now there isn't a cost of entry because
there isn't really an entry point and what little there is places the burden on
CYD, Stefan, et al to manage the assignment but that this generally takes a
backseat to other more pressing concerns... So, some neat stuff lingers because
it is a chore to manage the paperwork (real or metaphorical).

Likewise, I believe there is a net gain for community package repository
maintainers when there is some cost of entry to accept/process a package. There
are many examples of projects which maintain `unofficial repositories' of third
party tools. Those that provide a degree of oversight seem to be more
trustworthy and propagate more reliable code than those that to quote Dennis
Hopper in the movie Blue Velvet, "F**k anything that moves". When package
authors understand that some legwork is required of them in order to play they
may be more apt to contribute more robust and tested code (as contributions
aren't worth the trouble otherwise) where this is the case repository
maintainers will benefit from less need to troubleshoot, and less need for
frequent integration of code changes.

Most likely there will continue to be more than one model of Emacs community
developed and distributed packages. These will either compete for
attention/resources or each will offer appropriate paths to increased
integration with Emacs.  In a competitive environment (read forking) environment
the repo with the _most_ packages will thrive i.e. "Worse is Better" even
if/when it isn't.  In a cooperative/integrated/tired community repository
environment the repo with the best, cleanest, most open, code will achieve a
higher "status" than her fast and loose neighbor.

IOW repository maintainers have no incentive to adopt the above assignment
clearing regime where a competing repo environment is afforded an equal status
by the upstream Emacs-devels. However, where there is some upstream endorsement
preference for a particular _regime_ then disparate repo maintainers and
package authors can participate with, promote, and adopt this regime (or not)
according to their particular philosophy, time restraints, personalities, etc.

/s_P\

Re: Integrating package.el

[Phil Hagelberg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm going to continue my work on package.el whether or not it is
>> accepted into Emacs, but it would be a shame to make users go through
>> extra installation steps to use it.
>
> Having myself written a package manager (install.el), I'm clearly fully
> in favor of integrating something like package.el.
> I don't think there's necessarily much work to be done to integrate it.
>
> E.g. It is not necessary to set up a VCS area to keep the package
> sources (and other such things) before integrating package.el.

True, this is not of the highest importance since the steps it takes can
still be done manually without too much tedium.

> Basically, all I want is for it to be able to distinguish between
> "packages available locally, either installed by the user or the
> sysadmin", and "packages that will be autoloaded in a given session", so
> as to be able to install different versions of the same package at the
> same time.

OK, I will prioritize this once I finish adding support for multiple
archive sources. I think we'll need this in order to support separating
out fully-supported packages vs unsupported packages for which we simply
have copyright assignment as Ted has suggested.

thanks,
Phil

Re: Integrating package.el

[Stephen J. Turnbull]

Chong Yidong writes:

 > > This is a problem as opposed to increasing numbers of packages that
 > > are available only to their authors and you can't put into Emacs?  At
 > > least this way you'll have a contact address for authors.
 > 
 > No, it's a valid concern.  It's hard work hunting down papers and
 > integrating packages.

Nobody said otherwise.  I am painfully aware of the costs of both.

 > We must make sure a package system does not tempt us to just "dump
 > and forget".

Too late.  By refusing to handle registration of 3rd-party packages
itself, Emacs has been "dumping and forgetting" for decades.  That
hasn't hindered growth of repositories of unassigned code in
any way, it has just created a class system and contributed to
integration problems for maintainers of packages like AUCTeX who do
exercise due diligence ... but they arrived on the scene too late.

The temptation is a *human* problem, not the system's, and refusing to
list unassigned packages is just as much succumbing to the temptation
as "forgetting" to chase papers for listed packages.  At least if you
list the packages, you have the goad of shame to go chasing the papers.

Re: Integrating package.el

[Stefan Monnier]

> OK, I will prioritize this once I finish adding support for multiple
> archive sources. I think we'll need this in order to support separating
> out fully-supported packages vs unsupported packages for which we simply
> have copyright assignment as Ted has suggested.

Feel free to set your own priorities, but from where I stand, a single
repository would already be useful (used for the packages we'd distribute
separately from Emacs).


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Fri, 08 Jan 2010 21:40:31 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> The attached file implements the server-side maintenance, albeit in a
PH> rough manner. It's limited to single-file packages kept in git
PH> repositories, simply because that's what I'm personally familiar with,
PH> though I don't think supporting other DVCSes would be tricky at all.

PH> I welcome comments on this. Once the server-side maintenance tool is
PH> ready, I want to add support for multiple sources to the client-side
PH> package.el, and once that's done supporting multiple install locations
PH> on disk (as Tom has explained) would be the next step.

Technically it seems fine.  I wouldn't mind working with it.  The git
calls could perhaps be abstracted to the VC equivalents, though I don't
know the Emacs VC well at all.

Ted

Re: Integrating package.el

[Phil Hagelberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 08 Jan 2010 21:40:31 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 
>
> PH> The attached file implements the server-side maintenance, albeit in a
> PH> rough manner. It's limited to single-file packages kept in git
> PH> repositories, simply because that's what I'm personally familiar with,
> PH> though I don't think supporting other DVCSes would be tricky at all.
>
> PH> I welcome comments on this. Once the server-side maintenance tool is
> PH> ready, I want to add support for multiple sources to the client-side
> PH> package.el, and once that's done supporting multiple install locations
> PH> on disk (as Tom has explained) would be the next step.
>
> Technically it seems fine.  I wouldn't mind working with it.  The git
> calls could perhaps be abstracted to the VC equivalents, though I don't
> know the Emacs VC well at all.

Unfortunately the abstracted VC functionality doesn't currently support
creating new checkouts/cloning, which is why I didn't investigate that
much further. (Plus I know of 20+ elisp libs in git and only 1 or 2 in
bzr.) It may not be a common enough operation to justify adding to VC,
or maybe it's a long-standing TODO that nobody's tackled yet.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.
>
> Feel free to set your own priorities, but from where I stand, a single
> repository would already be useful (used for the packages we'd distribute
> separately from Emacs).

Yeah, I've already started this work, so I think it won't take too long
to wrap up. Plus it will be useful to users of package.el even before it
gets merged into Emacs.

-Phil

[trimmed cc list]

Re: Integrating package.el

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Phil Hagelberg <phil@hagelb.org> writes:

>> Basically, all I want is for it to be able to distinguish between
>> "packages available locally, either installed by the user or the
>> sysadmin", and "packages that will be autoloaded in a given session", so
>> as to be able to install different versions of the same package at the
>> same time.
>
> OK, I will prioritize this once I finish adding support for multiple
> archive sources. I think we'll need this in order to support separating
> out fully-supported packages vs unsupported packages for which we simply
> have copyright assignment as Ted has suggested.

I've just implemented support for multiple package sources in
package.el. Right now it is preconfigured to use ELPA, but the following
snippet will let it pull in packages from my own archive source as well:

(add-to-list 'package-archives
             '("technomancy" . "http://repo.technomancy.us/emacs/") t)

Hit M-x package-list-packages to see all available packages. The "bork"
and "bingle" packages are dummies that are only available from my
repository, and should be available for installation at the same time as
the standard ELPA-provided packages.

I'd love to see some other elisp library authors try to set up their own
package archive sources using package-maint.el and test the
multiple-archive support.

I hope that a new version of package.el incorporating my changes could
be pushed out to ELPA and then possibly included in Emacs once a little
more work has gone into integration with the Emacs load process.

-Phil


[-- Attachment #2: package.el --]
[-- Type: application/emacs-lisp, Size: 55248 bytes --]

[-- Attachment #3: package-maint.el --]
[-- Type: application/emacs-lisp, Size: 7552 bytes --]

Re: Integrating package.el

[Dan Nicolaescu]

Please add a menu to package.el.
This patch vs an older version of package.el does that.  It should still apply.


--- package.el	2009-09-19 11:33:11.000000000 -0700
+++ package.el.with-menu	2009-09-19 11:31:00.000000000 -0700
@@ -1129,28 +1129,71 @@
 
 ;;;; Package menu mode.
 
-(defvar package-menu-mode-map nil
+(defvar package-menu-mode-map
+  (let ((map (make-keymap))
+	(menu-map (make-sparse-keymap "Package")))
+    (suppress-keymap map)
+    (define-key map "q" 'quit-window)
+    (define-key map "n" 'next-line)
+    (define-key map "p" 'previous-line)
+    (define-key map "u" 'package-menu-mark-unmark)
+    (define-key map "\177" 'package-menu-backup-unmark)
+    (define-key map "d" 'package-menu-mark-delete)
+    (define-key map "i" 'package-menu-mark-install)
+    (define-key map "g" 'package-menu-revert)
+    (define-key map "r" 'package-menu-refresh)
+    (define-key map "~" 'package-menu-mark-obsolete-for-deletion)
+    (define-key map "x" 'package-menu-execute)
+    (define-key map "h" 'package-menu-quick-help)
+    (define-key map "?" 'package-menu-view-commentary)
+    (define-key map [menu-bar package-menu] (cons "Package" menu-map))
+    (define-key menu-map [mq]
+      '(menu-item "Quit" quit-window
+		  :help "Quit package selection"))
+    (define-key menu-map [s1] '("--"))
+    (define-key menu-map [mn]
+      '(menu-item "Next" next-line
+		  :help "Next Line"))
+    (define-key menu-map [mp]
+      '(menu-item "Previous" previous-line
+		  :help "Previous Line"))
+    (define-key menu-map [s2] '("--"))
+    (define-key menu-map [mu]
+      '(menu-item "Unmark" package-menu-mark-unmark
+		  :help "Clear any marks on a package and move to the next line"))
+    (define-key menu-map [munm]
+      '(menu-item "Unmark backwards" package-menu-backup-unmark
+		  :help "Back up one line and clear any marks on that package"))
+    (define-key menu-map [md]
+      '(menu-item "Mark for deletion" package-menu-mark-delete
+		  :help "Mark a package for deletion and move to the next line"))
+    (define-key menu-map [mi]
+      '(menu-item "Mark for install" package-menu-mark-install
+		  :help "Mark a package for installation and move to the next line"))
+    (define-key menu-map [s3] '("--"))
+    (define-key menu-map [mg]
+      '(menu-item "Update package list" package-menu-revert
+		  :help "Update the list of packages"))
+    (define-key menu-map [mr]
+      '(menu-item "Refresh package list" package-menu-refresh
+		  :help "Download the ELPA archive"))
+    (define-key menu-map [s4] '("--"))
+    (define-key menu-map [mt]
+      '(menu-item "Mark obsolete packages" package-menu-mark-obsolete-for-deletion
+		  :help "Mark all obsolete packages for deletion"))
+    (define-key menu-map [mx]
+      '(menu-item "Execute actions" package-menu-execute
+		  :help "Perform all the marked actions"))
+    (define-key menu-map [s5] '("--"))
+    (define-key menu-map [mh]
+      '(menu-item "Help" package-menu-quick-help
+		  :help "Show short key binding help for package-menu-mode"))
+    (define-key menu-map [mc]
+      '(menu-item "View Commentary" package-menu-view-commentary
+		  :help "Display information about this package"))
+    map)
   "Local keymap for `package-menu-mode' buffers.")
 
-(unless package-menu-mode-map
-  (setq package-menu-mode-map (make-keymap))
-  (suppress-keymap package-menu-mode-map)
-  (define-key package-menu-mode-map "q" 'quit-window)
-  (define-key package-menu-mode-map "n" 'next-line)
-  (define-key package-menu-mode-map "p" 'previous-line)
-  (define-key package-menu-mode-map "u" 'package-menu-mark-unmark)
-  (define-key package-menu-mode-map "\177" 'package-menu-backup-unmark)
-  (define-key package-menu-mode-map "d" 'package-menu-mark-delete)
-  (define-key package-menu-mode-map "i" 'package-menu-mark-install)
-  (define-key package-menu-mode-map "g" 'package-menu-revert)
-  (define-key package-menu-mode-map "r" 'package-menu-refresh)
-  (define-key package-menu-mode-map "~"
-    'package-menu-mark-obsolete-for-deletion)
-  (define-key package-menu-mode-map "x" 'package-menu-execute)
-  (define-key package-menu-mode-map "h" 'package-menu-quick-help)
-  (define-key package-menu-mode-map "?" 'package-menu-view-commentary)
-  )
-
 (defvar package-menu-sort-button-map
   (let ((map (make-sparse-keymap)))
     (define-key map [header-line mouse-1] 'package-menu-sort-by-column)

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Phil Hagelberg <phil@hagelb.org> writes:
>>> Basically, all I want is for it to be able to distinguish between
>>> "packages available locally, either installed by the user or the
>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>> as to be able to install different versions of the same package at the
>>> same time.
>> 
>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.

PH> I've just implemented support for multiple package sources in
PH> package.el. Right now it is preconfigured to use ELPA, but the following
PH> snippet will let it pull in packages from my own archive source as well:

PH> (add-to-list 'package-archives
PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)

PH> Hit M-x package-list-packages to see all available packages. The "bork"
PH> and "bingle" packages are dummies that are only available from my
PH> repository, and should be available for installation at the same time as
PH> the standard ELPA-provided packages.

PH> I'd love to see some other elisp library authors try to set up their own
PH> package archive sources using package-maint.el and test the
PH> multiple-archive support.

PH> I hope that a new version of package.el incorporating my changes could
PH> be pushed out to ELPA and then possibly included in Emacs once a little
PH> more work has gone into integration with the Emacs load process.

Hi Phil,

that's wonderful news.  I was just reading (and commented) at
http://monkey.org/~marius/self-contained-emacs.html about how package.el
with custom repositories could solve a very common need to have a
"personalized" self-contained Emacs.

Have you considered support for non-HTTP sources?  I'm OK with HTTP only
for now but I think a Bazaar source would make sense if there's going to
be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
supported?

What remains to be done to make package.el a part of Emacs?  I think
Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
else?

Once it's in the core we can start work on making more Emacs packages
installable directly.  Again, I think this is great progress and I
appreciate your work.

Thanks
Ted

Re: Integrating package.el

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 
>
> PH> Phil Hagelberg <phil@hagelb.org> writes:
>>>> Basically, all I want is for it to be able to distinguish between
>>>> "packages available locally, either installed by the user or the
>>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>>> as to be able to install different versions of the same package at the
>>>> same time.
>>> 
>>> OK, I will prioritize this once I finish adding support for multiple
>>> archive sources. I think we'll need this in order to support separating
>>> out fully-supported packages vs unsupported packages for which we simply
>>> have copyright assignment as Ted has suggested.
>
> PH> I've just implemented support for multiple package sources in
> PH> package.el. Right now it is preconfigured to use ELPA, but the following
> PH> snippet will let it pull in packages from my own archive source as well:
>
> PH> (add-to-list 'package-archives
> PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)
>
> PH> Hit M-x package-list-packages to see all available packages. The "bork"
> PH> and "bingle" packages are dummies that are only available from my
> PH> repository, and should be available for installation at the same time as
> PH> the standard ELPA-provided packages.
>
> PH> I'd love to see some other elisp library authors try to set up their own
> PH> package archive sources using package-maint.el and test the
> PH> multiple-archive support.
>
> PH> I hope that a new version of package.el incorporating my changes could
> PH> be pushed out to ELPA and then possibly included in Emacs once a little
> PH> more work has gone into integration with the Emacs load process.
>
> Hi Phil,
>
> that's wonderful news.  I was just reading (and commented) at
> http://monkey.org/~marius/self-contained-emacs.html about how package.el
> with custom repositories could solve a very common need to have a
> "personalized" self-contained Emacs.

Thats a brilliant idea!

> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
> for now but I think a Bazaar source would make sense if there's going to
> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
> supported?
>
> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
> else?
>
> Once it's in the core we can start work on making more Emacs packages
> installable directly.  Again, I think this is great progress and I
> appreciate your work.
>
> Thanks
> Ted
>
>
-- 
Joakim Verona

Re: Integrating package.el

[Stefan Monnier]

> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
> else?

For me, the only thing holding it back is the ability to concurrently
install different versions of the same package.

E.g. the sysadmin may install a whole bunch of packages and even
different versions of those packages, and then each user gets to choose
which ones to use in her ~/.emacs.


        Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 23 Feb 2010 17:25:17 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

Phil and other ELPA folks, please comment, here are my opinions.

Emacs could recognize the Version header and always track it internally
when it loads a library file, maybe with a (version . "1.2") in the
load-history.  That would help a lot, I think.  There's already good
logic in package.el for tracking its own installations and they are
required to have a version.  They get installed as 

$prefix/NAME-VERSION/NAME.el

which is a good way to resolve path conflicts.

I think we should ensure this works in common scenarios; the users who
break package.el in ways we haven't anticipated should either stop using
it, submit patches, or suffer.  I think that's reasonable, especially if
it happens as a major upgrade (in Emacs 24 perhaps).  The fact that
there are many happy users of package.el today shows that it's all right
even in its current form.

I also think that once package.el is available from within Emacs, many
of the custom installations will simply become redundant or be reduced
to ELPA-style repositories local to the site.  So the need for handling
edge cases will shrink as word spreads.

Ted

Re: Integrating package.el

[Stefan Monnier]

>>> What remains to be done to make package.el a part of Emacs?  I think
>>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

> Phil and other ELPA folks, please comment, here are my opinions.

> Emacs could recognize the Version header and always track it internally
> when it loads a library file, maybe with a (version . "1.2") in the
> load-history.  That would help a lot, I think.

I personnally don't care about the behavior if the user tries to
activate two different versions of the same package in a given session.
So package.el doesn't even need to do anything special about it AFAIC.

> They get installed as 
> $prefix/NAME-VERSION/NAME.el
> which is a good way to resolve path conflicts.

Yes, it's a good choice for installation location.  It still remains to
distinguish between "installed in the file system" and "active in
a particular session" (where "active" doesn't mean loaded but rather
just that autoloads are in place).


        Stefan

Re: Integrating package.el

[David De La Harpe Golden]

Stefan Monnier wrote:
>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>> else?
> 
> For me, the only thing holding it back is the ability to concurrently
> install different versions of the same package.
> 

I find _calling_ the facility "package" of (minor) concern.
Clearly it's not package as in Common Lisp package::symbol

Obviously, Emacs Lisp is not Common Lisp and there's more than enough 
precedent for the other "package" usage from elsewhere, I expect most 
Emacs users would in fact be more used to it from GNU+Linux distros for 
starters.

But could maybe just add something like this to the ELPA FAQ? Though it 
should be fairly obvious, newbies who don't know some core differences 
between Emacs Lisp and Common Lisp may be briefly caught out:

Q. Does this implement Common Lisp style "packages" (of symbols) for 
Emacs Lisp?

A. No, it does not, neither as its primary goal nor as an implementation 
detail.  Think "package" purely as in "program code distribution 
bundle".  Note that the fact that a Common Lisp "program code 
distribution bundle"* is typically a bundle of some program code that 
defines and then defines itself in a particular Common Lisp "package of 
symbols" is an aspect of Common Lisp language and conventions completely 
irrelevant for and alien to Emacs Lisp.

* sometimes referred to as a "package" even in the Common Lisp world.

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

PH> Phil Hagelberg <phil@hagelb.org> writes:
>>> Basically, all I want is for it to be able to distinguish between
>>> "packages available locally, either installed by the user or the
>>> sysadmin", and "packages that will be autoloaded in a given session", so
>>> as to be able to install different versions of the same package at the
>>> same time.
>> 
>> OK, I will prioritize this once I finish adding support for multiple
>> archive sources. I think we'll need this in order to support separating
>> out fully-supported packages vs unsupported packages for which we simply
>> have copyright assignment as Ted has suggested.

PH> I've just implemented support for multiple package sources in
PH> package.el. Right now it is preconfigured to use ELPA, but the following
PH> snippet will let it pull in packages from my own archive source as well:

PH> (add-to-list 'package-archives
PH>              '("technomancy" . "http://repo.technomancy.us/emacs/") t)

PH> Hit M-x package-list-packages to see all available packages. The "bork"
PH> and "bingle" packages are dummies that are only available from my
PH> repository, and should be available for installation at the same time as
PH> the standard ELPA-provided packages.

PH> I'd love to see some other elisp library authors try to set up their own
PH> package archive sources using package-maint.el and test the
PH> multiple-archive support.

PH> I hope that a new version of package.el incorporating my changes could
PH> be pushed out to ELPA and then possibly included in Emacs once a little
PH> more work has gone into integration with the Emacs load process.

Jonas, have you looked at this update to package.el?  Any chance you can
make your Emacsmirror accessible as a package.el repository, over HTTP
or through patched-in Git/SVN support in package.el?

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Thu, 25 Feb 2010 15:05:47 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> What remains to be done to make package.el a part of Emacs?  I think
>>>> Dan Nicolaescu's suggestion about menus makes sense.  Is there anything
>>>> else?

SM> For me, the only thing holding it back is the ability to concurrently
SM> install different versions of the same package.

SM> E.g. the sysadmin may install a whole bunch of packages and even
SM> different versions of those packages, and then each user gets to choose
SM> which ones to use in her ~/.emacs.

>> Phil and other ELPA folks, please comment, here are my opinions.

>> Emacs could recognize the Version header and always track it internally
>> when it loads a library file, maybe with a (version . "1.2") in the
>> load-history.  That would help a lot, I think.

SM> I personnally don't care about the behavior if the user tries to
SM> activate two different versions of the same package in a given session.
SM> So package.el doesn't even need to do anything special about it AFAIC.

OK.  But I still think Emacs should record the version as I suggested
whenever it finds it in a .el/.elc file.  It would help resolve many
annoying user-level bugs by showing exactly what version of the library
was loaded, not implied from the directory but directly from the version
header.  Does that require lots of changes?

>> They get installed as 
>> $prefix/NAME-VERSION/NAME.el
>> which is a good way to resolve path conflicts.

SM> Yes, it's a good choice for installation location.  It still remains to
SM> distinguish between "installed in the file system" and "active in
SM> a particular session" (where "active" doesn't mean loaded but rather
SM> just that autoloads are in place).

I think Phil or Tom should comment, I don't know how they plan to
address your concerns.

Ted

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> On Tue, 19 Jan 2010 03:40:11 -0800 Phil Hagelberg <phil@hagelb.org> wrote:
> PH> I've just implemented support for multiple package sources in
> PH> package.el. Right now it is preconfigured to use ELPA, but the following
> PH> snippet will let it pull in packages from my own archive source as well:
>
> PH> I'd love to see some other elisp library authors try to set up their own
> PH> package archive sources using package-maint.el and test the
> PH> multiple-archive support.
>
> PH> I hope that a new version of package.el incorporating my changes could
> PH> be pushed out to ELPA and then possibly included in Emacs once a little
> PH> more work has gone into integration with the Emacs load process.
>
> Jonas, have you looked at this update to package.el?

No, I have not found the time yet but I am aware that Phil has added
this feature [0].

> Any chance you can make your Emacsmirror accessible as a package.el repository,

Not immediately. Of course I do want the Emacsmirror to support (a)
package manager(s)
but due to lack of time I have concentrated on mirroring (adding and
*keeping up-to-date*)
packages.

I also haven't updated the metadata which I generate using elm.el in a
long time. But this
is next on my todo list - I might actually do this today so after that
I will hopefully find some
time to look into package-maint.el. (While I haven't pushed any
updates to the metadata I
have worked on improving the code used to generate it though.)

Without actually having looked at package-maint.el I believe that
elm.el is more advanced -
so I have heard and do in part believe... At least the metadata that I
generate is *different*
from that generated by package-maint.el or manually; and used by
package.el. I think the
data I generate has some advantages - mainly I do extract more information.

Also I do store it in one file per package instead of just one file
(which would be huge given
that now more than 2100 packages are being mirrored). In the future I
would like to extend
this to one file per package version. (I used to have support for this
but dropped it because
it is not clear yet how manual fixes would be merged when new versions
are released.)

Of course anyone is free to generate the package list as required by
package.el himself.
This could either be done by using the metadata I have generated and
converting it [1] [2]
or by getting a recent tarball [3] of all the git directories of all
mirrored packages and
generating it directly using package-maint.el.

Do not misunderstand me - I do plan to support package.el in the
future. But right now I
am working on other things.  Also I am more than willing to support
anyone who would like
to experiment with this.  It might actually be a good idea to test
package-maint.el using
the 2100 packages in the Emacsmirror tarball. Right now it probably
has only been tested
with some selected packages which closely follow the conventions - but
I have seen things
you people would not dream of - attack ships on fire...

> over HTTP or through patched-in Git/SVN support in package.el?

I am working on git support as I do think it has some considerable
advantages - generally
people seam to disagree.  Since I think using a dvcs instead of
tarballs should be preferred
I have absolutely no interest in working on generating tarballs myself.

Again, like for package.el-style-metadata, that does not mean that I
won't add support for
tarballs if someone else writes the code.  I just do not have an
interest in doing it myself.

I will quietly work on git support and once it is ready people can
test it and decide whether
it does have advantages that out weight the cost of requiring users to
install git or not.

Also I do not much feel like discussing this anymore.  At this point I
simply have nothing
that would demonstrate the benefits and at the same time I doubt that
anyone can come
up with an argument that would convince me that it's not even worth
writing a prove-of-concept.

So in summary: I do wish package.el and the Emacsmirror to support one
another but due
to lack of time, different priorities and because it is unclear which
of the two has to go how
far to meet the other it is not a short term priority.

-- Jonas

[0] And I do not know if anyone has found the time to look at my
alternative: elm.el.
[1] Though while generally more informative it also lacks some
information required by package.el.
[2] But please wait until I have updated it. As of now it contains
some severe bugs.
[3] http://github.com/emacsmirror/elm-backup-pkgs

Re: Integrating package.el

[Jonas Bernoulli]

(I am rather confused by the mixture of quoting styles.)

>>> Emacs could recognize the Version header and always track it internally
>>> when it loads a library file, maybe with a (version . "1.2") in the
>>> load-history.  That would help a lot, I think.

Absolutely. However this information is missing from a lot of
libraries. Also even
if library contain version information it is still uncommon that
libraries state what
version of their dependencies are actually required.  Cedet contains something
like this - has this portion of cedet been merged into 23.2?

If such an extended `require' form (or similar standardized information in the
library header) are not adopted there is no way to automatically extract this
information.

This means that maintainers of package repositories have to figure it
out manually
or just leave this information out completely.  I am mirroring 2100
packages - I won't
do it manually.

> SM> I personnally don't care about the behavior if the user tries to
> SM> activate two different versions of the same package in a given session.

If I remember correctly what he meant to say was that loading multiple versions
is a user error. Having multiple versions installed is still an option
as is choosing
a particular version by the user (as opposed to the sysadmin) at runtime.

> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.  Does that require lots of changes?

I agree. But as explained above this still doesn't help the package manager to
determine which version of a dependency is required.

-- Jonas

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 17:32:21 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> (I am rather confused by the mixture of quoting styles.)
>>>> Emacs could recognize the Version header and always track it internally
>>>> when it loads a library file, maybe with a (version . "1.2") in the
>>>> load-history.  That would help a lot, I think.

JB> Absolutely. However this information is missing from a lot of
JB> libraries. Also even if library contain version information it is
JB> still uncommon that libraries state what version of their
JB> dependencies are actually required.  Cedet contains something like
JB> this - has this portion of cedet been merged into 23.2?

We have to start somewhere.  Tracking dependencies is not possible until
versions are consistent so I think it's OK to use (version . nil) in the
load-history for the libraries that don't have a version header.

JB> If such an extended `require' form (or similar standardized
JB> information in the library header) are not adopted there is no way
JB> to automatically extract this information.

JB> This means that maintainers of package repositories have to figure
JB> it out manually or just leave this information out completely.  I am
JB> mirroring 2100 packages - I won't do it manually.

I think it's reasonable to also require unit and integration tests for
packages, at least for those that are considered "reliable" by the
packaging system, to accomplish these goals.  But these are long-term
goals and I think they merit a separate discussion on emacs-devel.  I'll
start the discussion with my (as usual) naive questions :)

SM> I personnally don't care about the behavior if the user tries to
SM> activate two different versions of the same package in a given session.

JB> If I remember correctly what he meant to say was that loading
JB> multiple versions is a user error. Having multiple versions
JB> installed is still an option as is choosing a particular version by
JB> the user (as opposed to the sysadmin) at runtime.

Does elm.el support that scheme as package.el does, distinguishing
between package installation (with a versioned install dir) and package
activation?

>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.  Does that require lots of changes?

JB> I agree. But as explained above this still doesn't help the package
JB> manager to determine which version of a dependency is required.

Yes, in this context I only meant to say it would save time when
debugging user errors.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 17:26:46 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> Without actually having looked at package-maint.el I believe that
JB> elm.el is more advanced - so I have heard and do in part
JB> believe... At least the metadata that I generate is *different* from
JB> that generated by package-maint.el or manually; and used by
JB> package.el. I think the data I generate has some advantages - mainly
JB> I do extract more information.

Can you or Phil/Tom explain the metadata capabilities of elm.el and
package-maint.el respectively, please?  It would help to know what the
authors think and do a side-by-side comparison.

JB> Of course anyone is free to generate the package list as required by
JB> package.el himself.  This could either be done by using the metadata
JB> I have generated and converting it [1] [2] or by getting a recent
JB> tarball [3] of all the git directories of all mirrored packages and
JB> generating it directly using package-maint.el.

That may be necessary if package.el becomes the default package manager
in Emacs.  It doesn't seem too hard since, as you say, the ELPA-style
metadata is a subset of the Emacsmirror metadata.  It would also be
possible to make a special package.el setup for Emacsmirror which would
pull elm.el and manage elm.el packages through it, if that was required.

JB> I am working on git support as I do think it has some considerable
JB> advantages - generally people seam to disagree.  Since I think using
JB> a dvcs instead of tarballs should be preferred I have absolutely no
JB> interest in working on generating tarballs myself.

I agree.  2100 packages over HTTP is crazy, an update would be very
costly whether you use a single database or one per package.  OTOH for a
few packages or specific situations HTTP makes sense so it's a matter of
providing a uniform frontend over various backend protocols.

JB> Also I do not much feel like discussing this anymore.  At this point
JB> I simply have nothing that would demonstrate the benefits and at the
JB> same time I doubt that anyone can come up with an argument that
JB> would convince me that it's not even worth writing a
JB> prove-of-concept.

Your input is valuable at any stage in the discussions, so thank you for
contributing.

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> Can you or Phil/Tom explain the metadata capabilities of elm.el and
Ted> package-maint.el respectively, please?  It would help to know what the
Ted> authors think and do a side-by-side comparison.

ELPA keeps a file, called archive-contents, that tracks metadata about
all the uploaded packages.  This metadata is extracted from the package
when I do an upload.

The information tracked are package name, the version, the requirements
list (see my other note), a docstring (for the package menu mode), and
whether a package is a single file or a .tar.  The latter is just so we
can construct the right URL at download time.

There's actually a second set of metadata about packages that are
included in Emacs itself.  This is (poorly) hand-maintained.  Ideally
this would be generated during the Emacs build.

I still haven't had time to look at Phil's changes, so I don't know what
he might have added or changed.

Tom

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> On Mon, 1 Mar 2010 17:32:21 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote:
>
> JB> (I am rather confused by the mixture of quoting styles.)
>>>>> Emacs could recognize the Version header and always track it internally
>>>>> when it loads a library file, maybe with a (version . "1.2") in the
>>>>> load-history.  That would help a lot, I think.
>
> JB> Absolutely. However this information is missing from a lot of
> JB> libraries. Also even if library contain version information it is
> JB> still uncommon that libraries state what version of their
> JB> dependencies are actually required.  Cedet contains something like
> JB> this - has this portion of cedet been merged into 23.2?
>
> We have to start somewhere.  Tracking dependencies is not possible until
> versions are consistent so I think it's OK to use (version . nil) in the
> load-history for the libraries that don't have a version header.

Yes.

> Does elm.el support that scheme as package.el does, distinguishing
> between package installation (with a versioned install dir) and package
> activation?

No, it is not a package manager it doesn't care if a package is
installed it just
needs to know where the sources are and then extracts information from it.
elx.el and lisp-mnt.el are used to extract information from individual packages.
elm.el is used to collect, check and store that information.

If I write a package manager it will be called epkg until (and if) it
can be merged
with package.el.

-- Jonas

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> Can you or Phil/Tom explain the metadata capabilities of elm.el and
> package-maint.el respectively, please?  It would help to know what the
> authors think and do a side-by-side comparison.

$ [ -e .git ] && echo "yes please"
yes please
$ ls -1 | wc --lines
2121
$ cat elm.epkg
(:summary    "Utilities for maintaining the Emacsmirror"
 :created    "20081202"
 :updated    "20100226"
 :license    "GPL-3"
 :authors    (("Jonas Bernoulli" . "jonas@bernoul.li"))
 :maintainer ("Jonas Bernoulli" . "jonas@bernoul.li")
 :provided   (elm elm-org)
 :required   ((("elx" elx)
               ("emacs" assoc cl finder)))
 :keywords   ("libraries")
 :homepage   "https://github.com/tarsius/elm")
$ cat ../package-commentaries/elm.txt
This package is used to maintain the Emacsmirror which can be found at
http://www.emacsmirror.org.

This library is mostly used to extracts metadata from packages and save
it for future use by e.g. a package manager.

Most of the code used to generate the mirrors webpage can be found in
the accompanying libary `elm-org.el' which might eventually be replaced
by a more dynamic webpage.



That's what it looks like. Thoughts follow later.

-- Jonas

Re: Integrating package.el

[Stefan Monnier]

> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.  Does that require lots of changes?

I don't see much evidence that "it would help resolve many annoying
user-level bugs".  I'm not opposed to it, but when it comes to making
external Elisp packages easier to manage for the user, this comes very
far down the list of priorities.


        Stefan

Re: Integrating package.el

[Jonas Bernoulli]

> I still haven't had time to look at Phil's changes, so I don't know what
> he might have added or changed.

Daniel Hackney has also made additions:
http://github.com/technomancy/package.el/network

Tom, do you use a vcs to develop package.el? If so it might be a good
idea to make it
publicly available or start using vcs if you are not doing so already.
Might make it easier
to contribute. What vcs doesn't really matter to me with git it's
rather easy to work with
foreign repositories, though git (since Phil, Daniel and I use it) or
bzr (since it is the
political correct solution:) would probably be preferable.

-- Jonas

Re: Integrating package.el

[Jonas Bernoulli]

2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
> OK.  But I still think Emacs should record the version as I suggested
> whenever it finds it in a .el/.elc file.  It would help resolve many
> annoying user-level bugs by showing exactly what version of the library
> was loaded, not implied from the directory but directly from the version
> header.

The information you get like this at runtime is not reliable. Some people
bump right after making a new release other develop for months keeping
the version from the latest release.

Rather this information should be extracted by the repository maintainers,
They have the full history of the package (if available) and cat determine
from that which of the many revision containing a particular version string,
actually IS that version... (I have some slightly buggy code for this
somewhere).

Speaking of version strings, are there any conventions how an author
should version his packages? Currently when I make edits after a
release and make them public while not wanting to release yet another
version I usually just add a "+" after the version.

0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2

Not really happy with it. But what should I be doing instead?

-- Jonas

Re: Integrating package.el

[Štěpán Němec]

On Mon, Mar 01, 2010 at 10:37:43PM +0100, Jonas Bernoulli wrote:
> Speaking of version strings, are there any conventions how an author
> should version his packages? Currently when I make edits after a
> release and make them public while not wanting to release yet another
> version I usually just add a "+" after the version.
> 
> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2

[just a thought:]
Maybe using something compatible with what `version-to-list' and friends
can understand?


    Štěpán

Re: Integrating package.el

[Štěpán Němec]

On Mon, Mar 01, 2010 at 11:18:32PM +0100, Štěpán Němec wrote:
> On Mon, Mar 01, 2010 at 10:37:43PM +0100, Jonas Bernoulli wrote:
> > Speaking of version strings, are there any conventions how an author
> > should version his packages? Currently when I make edits after a
> > release and make them public while not wanting to release yet another
> > version I usually just add a "+" after the version.
> > 
> > 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
> 
> [just a thought:]
> Maybe using something compatible with what `version-to-list' and friends
> can understand?
... which is actually the case even with the above:

(version-to-list "0.1+")
=> (0 1 -3)

so it's treated like an alpha release by default.

Re: Integrating package.el

[Jonas Bernoulli]

>> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
>
> [just a thought:]
> Maybe using something compatible with what `version-to-list' and friends
> can understand?

Hm, yes even my own such library does not understand it.

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 01 Mar 2010 11:06:15 -0700 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> Can you or Phil/Tom explain the metadata capabilities of elm.el and
Ted> package-maint.el respectively, please?  It would help to know what the
Ted> authors think and do a side-by-side comparison.

Tom> ELPA keeps a file, called archive-contents, that tracks metadata about
Tom> all the uploaded packages.  This metadata is extracted from the package
Tom> when I do an upload.

Tom> The information tracked are package name, the version, the requirements
Tom> list (see my other note), a docstring (for the package menu mode), and
Tom> whether a package is a single file or a .tar.  The latter is just so we
Tom> can construct the right URL at download time.

Tom> There's actually a second set of metadata about packages that are
Tom> included in Emacs itself.  This is (poorly) hand-maintained.  Ideally
Tom> this would be generated during the Emacs build.

On Mon, 1 Mar 2010 22:09:59 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> $ cat elm.epkg
JB> (:summary    "Utilities for maintaining the Emacsmirror"
JB>  :created    "20081202"
JB>  :updated    "20100226"
JB>  :license    "GPL-3"
JB>  :authors    (("Jonas Bernoulli" . "jonas@bernoul.li"))
JB>  :maintainer ("Jonas Bernoulli" . "jonas@bernoul.li")
JB>  :provided   (elm elm-org)
JB>  :required   ((("elx" elx)
JB>                ("emacs" assoc cl finder)))
JB>  :keywords   ("libraries")
JB>  :homepage   "https://github.com/tarsius/elm")
JB> $ cat ../package-commentaries/elm.txt
JB> This package is used to maintain the Emacsmirror which can be found at
JB> http://www.emacsmirror.org.

Thanks!

The Emacs package metadata will, ideally, be replaced with a package
repository that resides inside Emacs itself.  Thus users can explore and
activate Emacs packages without the usual drudgery, and some default
packages can even be activated for them.  This is my plan at least and I
hope it's accepted into Emacs.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 01 Mar 2010 16:19:47 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.  Does that require lots of changes?

SM> I don't see much evidence that "it would help resolve many annoying
SM> user-level bugs".

There was one just a few days ago on the Gnus newsgroup, which Katsumi
Yamaoka resolved by prior experience but otherwise would have been
really puzzling.  I've seen many like it.  The general problem of
shadowed paths is pretty pervasive and frustrating, especially since
users know many ways to add to the load-path.

SM> I'm not opposed to it, but when it comes to making external Elisp
SM> packages easier to manage for the user, this comes very far down the
SM> list of priorities.

All right, thanks.  I will keep it in mind.

Ted

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 1 Mar 2010 22:37:43 +0100 Jonas Bernoulli <jonas@bernoulli.cc> wrote: 

JB> 2010/3/1 Ted Zlatanov <tzz@lifelogs.com>:
>> OK.  But I still think Emacs should record the version as I suggested
>> whenever it finds it in a .el/.elc file.  It would help resolve many
>> annoying user-level bugs by showing exactly what version of the library
>> was loaded, not implied from the directory but directly from the version
>> header.

JB> The information you get like this at runtime is not reliable. Some people
JB> bump right after making a new release other develop for months keeping
JB> the version from the latest release.

JB> Rather this information should be extracted by the repository maintainers,
JB> They have the full history of the package (if available) and cat determine
JB> from that which of the many revision containing a particular version string,
JB> actually IS that version... (I have some slightly buggy code for this
JB> somewhere).

I don't think that's ideal either.  I'd rather use the authors' version
and then allow an override by the repository maintainer.  Otherwise
you'll certainly end up with different version levels between repos and
that's not a good scenario for the user.

The repo could associate a content hash with the version to generate a
unique identifier (and some insurance against corruption) for a
particular library snapshot.  I wonder if Emacs has a standard way to do
this (ignoring comments and whitespace).

Ted

Re: Integrating package.el

[Richard Stallman]

It looks like some people are trying to turn this into a package
system with all the features of Debian.  Please don't do that.  The
reason it is ok to install package.el is that it doesn't try to be a
full-blown package system.

Re: Integrating package.el

[Ted Zlatanov]

On Tue, 02 Mar 2010 12:39:34 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> It looks like some people are trying to turn this into a package
RS> system with all the features of Debian.  Please don't do that.  The
RS> reason it is ok to install package.el is that it doesn't try to be a
RS> full-blown package system.

The only major feature that has been added (by Phil Hagelberg) is
support for multiple package repositories.  I don't think any others are
under review; the only outstanding issues I know of are Stefan's
concerns about overlapping installs and a request for menus.

All the discussion about testing, versions, and dependency tracking is
purely theoretical and I expect it to benefit the package repository
contributors and maintainers in the long run.  I don't think that
functionality should necessarily be pushed into package.el.

Ted

Re: Integrating package.el

[Davis Herring]

> Speaking of version strings, are there any conventions how an author
> should version his packages? Currently when I make edits after a
> release and make them public while not wanting to release yet another
> version I usually just add a "+" after the version.
>
> 0.1 -> 0.1+ -> .... -> 0.1+ -> 0.2
>
> Not really happy with it. But what should I be doing instead?

Why not just use another layer of versions?

  0.1, 0.1.1, 0.1.2, ..., 0.1.74, 0.2.

The rule I (try to) follow is that the major version number should be
incremented on an incompatible change, the minor version number on a
feature addition, the revision on any (public) bug fix, and (if one uses
it) the build number on any test.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I don't think that's ideal either.  I'd rather use the authors' version
Ted> and then allow an override by the repository maintainer.

Yes.  My policy for ELPA has just been to require sanity from the
maintainers.  In general I don't upload a package that is broken, and I
require changes to go upstream before uploading.  My reasoning was that
a little sanity is not very hard to achieve (though in real life there
are a some recalcitrant types out there) and it was better to try to
distribute the work instead of taking it all on myself.

Tom

Re: Integrating package.el

[Tom Tromey]

>>>>> "Jonas" == Jonas Bernoulli <jonas@bernoulli.cc> writes:

>> I still haven't had time to look at Phil's changes, so I don't know what
>> he might have added or changed.

Jonas> Daniel Hackney has also made additions:
Jonas> http://github.com/technomancy/package.el/network

Interesting, thanks.

Jonas> Tom, do you use a vcs to develop package.el?

Just locally, nothing public.

Jonas> If so it might be a good idea to make it publicly available or
Jonas> start using vcs if you are not doing so already.

I wouldn't mind but I have basically zero time for ELPA right now, and
for the foreseeable future.  I've been hoping Phil will drive the
project of adding the missing features and then integrating package.el
into Emacs.

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 03 Mar 2010 13:15:53 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I wouldn't mind but I have basically zero time for ELPA right now, and
Tom> for the foreseeable future.  I've been hoping Phil will drive the
Tom> project of adding the missing features and then integrating package.el
Tom> into Emacs.

Phil hasn't replied to questions in this thread for a bit so he must be
busy as well.  Would it be OK if I worked with the Emacs maintainers to
patch up Phil's latest package.el so it can go into Emacs?  Or should I
wait a bit longer?  I really don't mind waiting, but if I can help...

Ted

Re: Integrating package.el

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> On Wed, 03 Mar 2010 13:15:53 -0700 Tom Tromey <tromey@redhat.com> wrote: 
Tom> I wouldn't mind but I have basically zero time for ELPA right now, and
Tom> for the foreseeable future.  I've been hoping Phil will drive the
Tom> project of adding the missing features and then integrating package.el
Tom> into Emacs.

Ted> Phil hasn't replied to questions in this thread for a bit so he must be
Ted> busy as well.  Would it be OK if I worked with the Emacs maintainers to
Ted> patch up Phil's latest package.el so it can go into Emacs?  Or should I
Ted> wait a bit longer?  I really don't mind waiting, but if I can help...

Yes, it is fine by me.

Tom

Re: Integrating package.el

[Phil Hagelberg]

> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
> for now but I think a Bazaar source would make sense if there's going to
> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
> supported?

Rather than supporting installing straight out of a VC repository, I
have opted to make it easy to create an archive from a list of VC
repositories. (package-maint.el) The idea is that only releases
would be included in the archive rather than any-old-version.
This also reduces the dependencies on the client-side.

If you want to add support for installing directly from a VC repository
don't let me stop you, but I think there are more important things to
do first.

> What remains to be done to make package.el a part of Emacs?  I think
> Dan Nicolaescu's suggestion about menus makes sense.  Is there
> anything else?

I want to look at the menus patch soon, but I haven't found the time
yet. Stefan also mentioned supporting having multiple versions of the
same project installed at once. Rather than supporting this by
allowing multiple versions to coexist in a single install location, I
think it makes more sense to allow multiple install locations. I
haven't thought through the implications of this though. Perhaps it
would be as easy as simply activating the newest version available,
but perhaps we need something more explicit. Probably some mechanism
would be necessary for allowing users to opt-out of system packages.

I think supporting the distinction between system-level installs vs
user-level installs is the biggest blocker (besides more thorough
testing) to including package.el in Emacs. I would like to implement
this, but it probably won't happen within the next month unless
someone else steps up to start work on it.

> Phil hasn't replied to questions in this thread for a bit so he must
> be busy as well.  Would it be OK if I worked with the Emacs
> maintainers to patch up Phil's latest package.el so it can go into
> Emacs?  Or should I wait a bit longer?  I really don't mind waiting,
> but if I can help...

Don't wait on account of me. I definitely want to see this through,
but if you're able to put work into it now then go for it.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

On Wed, Mar 3, 2010 at 12:13 PM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> I don't think that's ideal either.  I'd rather use the authors' version
> Ted> and then allow an override by the repository maintainer.
>
> Yes.  My policy for ELPA has just been to require sanity from the
> maintainers.  In general I don't upload a package that is broken, and I
> require changes to go upstream before uploading.  My reasoning was that
> a little sanity is not very hard to achieve (though in real life there
> are a some recalcitrant types out there) and it was better to try to
> distribute the work instead of taking it all on myself.

It should be noted that if package.el is included in Emacs, it will
provide a lot more incentive for package authors to provide proper
metadata and ensure that their code works with package.el themselves.
So I don't think this needs to be worried about.

-Phil

Re: Integrating package.el

[Stephen J. Turnbull]

Phil Hagelberg writes:

 > I think supporting the distinction between system-level installs vs
 > user-level installs is the biggest blocker (besides more thorough
 > testing) to including package.el in Emacs. I would like to implement
 > this, but it probably won't happen within the next month unless
 > someone else steps up to start work on it.

Warning: I'm not really sure what the problems are[1], but the user
install vs. system install distinction has never really worked right
in XEmacs's package system.  It probably just requires a little bit of
thought[2], but it will be well worth it in reducing user frustration
if you take care in designing this feature.


Footnotes: 
[1]  Probably partly an incomplete design and partly PEBKACs.

[2]  My apologies to whoever implemented that part of our system, but
that's my opinion. ;-)

Re: Integrating package.el

[Ted Zlatanov]

On Wed, 3 Mar 2010 21:39:17 -0800 Phil Hagelberg <phil@hagelb.org> wrote: 

>> Have you considered support for non-HTTP sources?  I'm OK with HTTP only
>> for now but I think a Bazaar source would make sense if there's going to
>> be a ELPA-style repository inside Emacs itself.  Also are file:/// URLs
>> supported?

PH> Rather than supporting installing straight out of a VC repository, I
PH> have opted to make it easy to create an archive from a list of VC
PH> repositories. (package-maint.el) The idea is that only releases
PH> would be included in the archive rather than any-old-version.
PH> This also reduces the dependencies on the client-side.

OK.  See below about file:/// URLs.

PH> If you want to add support for installing directly from a VC repository
PH> don't let me stop you, but I think there are more important things to
PH> do first.

Understood.  The TODO list has more important items, I agree.

>> What remains to be done to make package.el a part of Emacs?  I think
>> Dan Nicolaescu's suggestion about menus makes sense.  Is there
>> anything else?

PH> I want to look at the menus patch soon, but I haven't found the time
PH> yet. 

Great, thanks!

PH> Stefan also mentioned supporting having multiple versions of the
PH> same project installed at once. Rather than supporting this by
PH> allowing multiple versions to coexist in a single install location,
PH> I think it makes more sense to allow multiple install locations. I
PH> haven't thought through the implications of this though. Perhaps it
PH> would be as easy as simply activating the newest version available,
PH> but perhaps we need something more explicit. Probably some mechanism
PH> would be necessary for allowing users to opt-out of system packages.

PH> I think supporting the distinction between system-level installs vs
PH> user-level installs is the biggest blocker (besides more thorough
PH> testing) to including package.el in Emacs. I would like to implement
PH> this, but it probably won't happen within the next month unless
PH> someone else steps up to start work on it.

How about associating a single install location with each repository?
Logically that would make sense and there would be a default (nil)
repository for things not managed by package.el.  Then the user, by
managing the repository priority, would also manage the load-path
priority.

>> Phil hasn't replied to questions in this thread for a bit so he must
>> be busy as well.  Would it be OK if I worked with the Emacs
>> maintainers to patch up Phil's latest package.el so it can go into
>> Emacs?  Or should I wait a bit longer?  I really don't mind waiting,
>> but if I can help...

PH> Don't wait on account of me. I definitely want to see this through,
PH> but if you're able to put work into it now then go for it.

Can you review Dan Haxney's (CC here and to you) patches as well at
http://github.com/haxney/package?  He supports file:/// URLs and has
made many other improvements.  I don't know if he has rebased against
your support for multiple repositories.  If the two of you could decide
what pieces are worth keeping and merge, it would make it easier for me
to start work on the merged version.  Otherwise I may end up wasting
effort.

Thanks
Ted

Re: Integrating package.el

[Stefan Monnier]

> Warning: I'm not really sure what the problems are[1], but the user
> install vs. system install distinction has never really worked right
> in XEmacs's package system.

It would help if you could describe the problems encountered,


        Stefan

Re: Integrating package.el

[Stefan Monnier]

> yet. Stefan also mentioned supporting having multiple versions of the
> same project installed at once. Rather than supporting this by
> allowing multiple versions to coexist in a single install location, I
> think it makes more sense to allow multiple install locations. I

I strongly disagree.  I want to be able to install all kinds of versions
in a single spot so I can quickly see which versions are available.
This said, I agree that allowing several install locations is desirable.

Actually the "install" part of the code should be able to install
a package *anywhere*.  Similarly, the activation of a package
(i.e. setting up the autoloads) should be doable from a package located
anywhere.

It doesn't need to be part of the visible UI, but there should be
a command like M-x package-install that takes a file name (a single .el
file or a tarball) and a target directory (defaults to the main "install
location") and installs it there.

The only part that really needs to know about "install locations" is the
part that lets you list all the installed packages, deinstall some of
them, setup/remove activation for some of them, etc...

> I think supporting the distinction between system-level installs vs
> user-level installs is the biggest blocker (besides more thorough
> testing) to including package.el in Emacs.

No, I think the only blocker is the support for multiple
simultaneous versions.  Tho I haven't kept track closely enough to know
if there might be copyright-paperwork issues left.


        Stefan

Re: Integrating package.el

[Tom Tromey]

>>>>> "Phil" == Phil Hagelberg <phil@hagelb.org> writes:

Phil> Stefan also mentioned supporting having multiple versions of the
Phil> same project installed at once. Rather than supporting this by
Phil> allowing multiple versions to coexist in a single install
Phil> location, I think it makes more sense to allow multiple install
Phil> locations.

I consider these to be orthogonal.

You need multiple install locations to let package.el manage additions
in site-lisp, and also (maybe -- there are some questions here) the
load-path for packages which are included with Emacs but also released
separately.

So suppose Emacs starts up and sees, say, 3 versions of gnus: one
included in Emacs, one in site-lisp, and one in ~/.emacs.d/elpa/.
Which one should be activated?

Right now the obvious answer would be the newest one -- but Stefan wants
to let the user pick a specific one.  So, I think we'll need a bit of
additional metadata about this.

Tom

Re: Integrating package.el

[Tom Tromey]

Stefan> No, I think the only blocker is the support for multiple
Stefan> simultaneous versions.  Tho I haven't kept track closely enough
Stefan> to know if there might be copyright-paperwork issues left.

My version was written by me, and all other contributions were under the
copyright threshold.  So, it is clear here.  I don't know about the
status of other contributions (Phil's branch, the other github branch,
or the menu patch).

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Thu, 04 Mar 2010 13:29:49 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I want to be able to install all kinds of versions in a single spot
SM> so I can quickly see which versions are available.  This said, I
SM> agree that allowing several install locations is desirable.

SM> Actually the "install" part of the code should be able to install
SM> a package *anywhere*.  Similarly, the activation of a package
SM> (i.e. setting up the autoloads) should be doable from a package located
SM> anywhere.

SM> It doesn't need to be part of the visible UI, but there should be
SM> a command like M-x package-install that takes a file name (a single .el
SM> file or a tarball) and a target directory (defaults to the main "install
SM> location") and installs it there.

SM> The only part that really needs to know about "install locations" is the
SM> part that lets you list all the installed packages, deinstall some of
SM> them, setup/remove activation for some of them, etc...

On Thu, 04 Mar 2010 11:30:26 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> You need multiple install locations to let package.el manage additions
Tom> in site-lisp, and also (maybe -- there are some questions here) the
Tom> load-path for packages which are included with Emacs but also released
Tom> separately.

Tom> So suppose Emacs starts up and sees, say, 3 versions of gnus: one
Tom> included in Emacs, one in site-lisp, and one in ~/.emacs.d/elpa/.
Tom> Which one should be activated?

Tom> Right now the obvious answer would be the newest one -- but Stefan wants
Tom> to let the user pick a specific one.  So, I think we'll need a bit of
Tom> additional metadata about this.

How about letting the user define his own install location for each
repository, but with a sensible default?  The location could also be
tagged with a "is it versioned?" boolean.  While installing may not be
possible into every location, the user will be able to activate whatever
is installed in any location.

For example:

ELPA's Gnus defaults to ~/.emacs.d/elpa
(from repository "ELPA", versioned to $location/gnus-x.yz )

Emacs' Gnus defaults to /usr/local/share/emacs/23.1/lisp
(from repository "emacs 23.1", unversioned to $location/gnus)

site's Gnus is set by user to /usr/local/share/oursite/lisp
(from user-defined repository "oursite", unversioned to $location/gnus)

So then "package-install" without a specific repository will pick the
first repository in the package-archives list and try to write to it.
If it fails, that's an error.  Similarly for package removal.

With a specific repository, both install and remove go to the specific
location set by the user.  Yes, unversioned install locations will cause
conflicts, but they are necessary to make this backwards compatible.

So then resolving the question of multiple installs will actually be

"Do you want to pick the ELPA Gnus [in ...], the Emacs 23.1 Gnus 
[in ...], the oursite Gnus [in ...], or this CVS checkout we found in
~/gnus/lisp because it was in your load-path?"

IOW, the user would choose between repository install locations
(represented by the name of the repository) and unmanaged paths to
resolve the conflict.  Would that work?

Ted

Re: Integrating package.el

[Stefan Monnier]

> Right now the obvious answer would be the newest one -- but Stefan wants
> to let the user pick a specific one.  So, I think we'll need a bit of
> additional metadata about this.

The simplest solution is to not decide which at runtime but at some
earlier time.  This of course means that this choice can be out-of-date
(e.g. the choice may say "activate /usr/local/elisp/foo-2.3" but that
directory doesn't even exist any more), but if the changes are made
through the package manager, than those out-of-date problems can be
caught at that point.  And otherwise we can just signal a warning and
let the user invoke the package manager to fix it.


        Stefan

Re: Integrating package.el

[Stephen J. Turnbull]

Stefan Monnier writes:
 > > Warning: I'm not really sure what the problems are[1], but the user
 > > install vs. system install distinction has never really worked right
 > > in XEmacs's package system.
 > 
 > It would help if you could describe the problems encountered,

At my consulting rates, that would be worth about $500. ;-) As I said,
it's not clear what is design and what is PEBKAC, and clarifying that
would require a lot of effort.  We've just had a lot of user
complaints over the years related to specified and/or default
installation locations, permissions issues, etc.  Packages get
installed in places the running XEmacs isn't looking for them,
insufficient privilege to install in the default place, that kind of
thing.  This often seems to be related to system vs. user issues.

At this point I just want to offer the advice that Here Be Dragons (or
at least some cute little lizards) to watch out for.

Re: Integrating package.el

[Phil Hagelberg]

On Thu, Mar 4, 2010 at 10:33 AM, Tom Tromey <tromey@redhat.com> wrote:
> Stefan> No, I think the only blocker is the support for multiple
> Stefan> simultaneous versions.  Tho I haven't kept track closely enough
> Stefan> to know if there might be copyright-paperwork issues left.
>
> My version was written by me, and all other contributions were under the
> copyright threshold.  So, it is clear here.  I don't know about the
> status of other contributions (Phil's branch, the other github branch,
> or the menu patch).

I haven't gotten any contributions apart from Daniel's patches, and I
have my copyright assignment in. So I think Daniel's is the only one
missing.

-Phil

Re: Integrating package.el

[Phil Hagelberg]

On Thu, Mar 4, 2010 at 10:29 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> yet. Stefan also mentioned supporting having multiple versions of the
>> same project installed at once. Rather than supporting this by
>> allowing multiple versions to coexist in a single install location, I
>> think it makes more sense to allow multiple install locations. I
>
> I strongly disagree.  I want to be able to install all kinds of versions
> in a single spot so I can quickly see which versions are available.

OK, if that's the most important to you then we should focus on that
next. I imagine it might be a fairly invasive modification though;
perhaps Tom could offer some hints on how it could be implemented
since he's the only one really familiar with the package activation
logic?

-Phil

Re: Integrating package.el

[Tom Tromey]

>>>>> "Phil" == Phil Hagelberg <phil@hagelb.org> writes:

Phil> OK, if that's the most important to you then we should focus on that
Phil> next. I imagine it might be a fairly invasive modification though;
Phil> perhaps Tom could offer some hints on how it could be implemented
Phil> since he's the only one really familiar with the package activation
Phil> logic?

I think the default should be to activate the most recent package, and
to activate a package at install time.  So, if the user picks a specific
older version to activate (or equivalently deactivates the most recent
version), record that, and take it into account during activation.

This would also need some changes in package-activate and the package
menu mode, and also we'd have to store some more info somewhere in
~/.emacs.d/elpa/.

Tom

Re: Integrating package.el

[Ted Zlatanov]

On Sun, 07 Mar 2010 20:17:23 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

So it's a version override alist, where a missing entry implies the user
wants the latest?  Do you mean something like this:

((package1 (repository1 path-to-specific-version1)) 
 (package2 (repository2 path-to-specific-version2)))

or am I misunderstanding something?

Tom> This would also need some changes in package-activate and the package
Tom> menu mode, and also we'd have to store some more info somewhere in
Tom> ~/.emacs.d/elpa/.

If this is just an alist it could be stored with Customize and not as
part of the repository itself.  Conceptually it seems to be at the
global level, not per repository.

If the repository list is customizable and global (as I think Phil's
changes assume) then the overrides could simply be part of the
repository list:

(setq package-archives '((repository1 :overrides (package1 path-to-specific-version1))
                         (repository2 :overrides (package2 path-to-specific-version2))))

Ted

Re: Integrating package.el

[Stefan Monnier]

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

> So it's a version override alist, where a missing entry implies the user
> wants the latest?  Do you mean something like this:

> ((package1 (repository1 path-to-specific-version1)) 
>  (package2 (repository2 path-to-specific-version2)))

> or am I misunderstanding something?

Don't know about you, but I seem to be missing something: why is the
word "repository" used here?  AFAIK the "repository" for package.el are
(typically remote) locations from where to download packages, and they
should have no interaction whatsoever with the part that worries about
installation and activation of packages (just like Debian's dpkg has no
idea about repositories).

BTW, my take on package activation and selection of versions (which IIRC
I already discussed with Tom a while back and we agreed to disagree) is
that activation of a package is controlled by lines in .emacs that take
a form similar to:

   (load "/some/where/package/init/file.el" 'package)

so there is no search or selection of package version at startup-time.
This means that when a new version of a package is installed, Emacs has
various ways to react to it:
- don't do anything, and at the next startup warn the user that the
  package he requested doesn't exist any more because it was replaced by
  a new version.
- update the "load" line right when the package gets upgraded (tho only
  for the .emacs of the user that performs the upgrade).
- use symlinks like "/some/where/elisp/sml-mode" pointing to
  "sml-mode-4.1" and use (load "/some/where/elisp/sml-mode/startup.el")
  so the symlink can be adjusted during the upgrade.
- use a different load, like

   (package-activate-latest "/some/where/elisp/sml-mode")

  which will look for the latest sml-mode package installed in
  /some/where/elisp/, or

   (package-activate-latest "sml-mode")

  which will look for the latest sml-mode package installed in
  any of the installation targets.


-- Stefan

Re: Integrating package.el

[Ted Zlatanov]

On Mon, 08 Mar 2010 12:01:52 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

Tom> I think the default should be to activate the most recent package, and
Tom> to activate a package at install time.  So, if the user picks a specific
Tom> older version to activate (or equivalently deactivates the most recent
Tom> version), record that, and take it into account during activation.

>> So it's a version override alist, where a missing entry implies the user
>> wants the latest?  Do you mean something like this:

>> ((package1 (repository1 path-to-specific-version1)) 
>> (package2 (repository2 path-to-specific-version2)))

>> or am I misunderstanding something?

SM> Don't know about you, but I seem to be missing something: why is the
SM> word "repository" used here?  AFAIK the "repository" for package.el are
SM> (typically remote) locations from where to download packages, and they
SM> should have no interaction whatsoever with the part that worries about
SM> installation and activation of packages (just like Debian's dpkg has no
SM> idea about repositories).

I thought it made sense to remember the repository that a particular
version came from, so selection (for activation) and warning messages
are more helpful.  But this can be guessed based on the path prefix so
it's not strictly necessary, and you seem to be looking for package
management that's less tied to repositories anyhow.  So forget the above
and see if the following makes sense.

SM> BTW, my take on package activation and selection of versions (which IIRC
SM> I already discussed with Tom a while back and we agreed to disagree) is
SM> that activation of a package is controlled by lines in .emacs that take
SM> a form similar to:

SM>    (load "/some/where/package/init/file.el" 'package)

SM> so there is no search or selection of package version at startup-time.

SM> This means that when a new version of a package is installed, Emacs has
SM> various ways to react to it:
SM> - don't do anything, and at the next startup warn the user that the
SM>   package he requested doesn't exist any more because it was replaced by
SM>   a new version.

This warning is always necessary because packages can get removed by
external agents as well (so the user doesn't know the load path is
incorrect).  So there could be no new version at all.

SM> - update the "load" line right when the package gets upgraded (tho only
SM>   for the .emacs of the user that performs the upgrade).

Too many potential problems here, I'd stay away from this.

SM> - use symlinks like "/some/where/elisp/sml-mode" pointing to
SM>   "sml-mode-4.1" and use (load "/some/where/elisp/sml-mode/startup.el")
SM>   so the symlink can be adjusted during the upgrade.

I did this for 7 years in a lab I was managing and it's not fun.  You
can automate some of the tedium but inevitably it becomes unwieldy.
If the user or admin wants to do it this way, let them manage it
externally, there are decent tools to automate it or they can write
ELisp wrappers to fit their environment.

Also symlinks don't work reliably in some environments.

SM> - use a different load, like

SM>    (package-activate-latest "/some/where/elisp/sml-mode")

SM>   which will look for the latest sml-mode package installed in
SM>   /some/where/elisp/, or

SM>    (package-activate-latest "sml-mode")

SM>   which will look for the latest sml-mode package installed in
SM>   any of the installation targets.

For the sake of backward compatibility this seems like the best option.
(load) and (require) can keep working normally.  Users can then
transition to the package.el management gradually.  Emacs can start
using these options gradually too.

Version overrides can be then specified as

(package-activate-specific "sml-mode" "specific-version")
(package-activate-specific "/path/to/sml-mode" "specific-version")

Together with the startup warnings (which can be issued by
package-activate-{specific,latest}) I think this approach makes sense.

Ted