ELPA update

ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 171 bytes --]

Hi,

When GNU ELPA is being updated?

I though it was at least daily and automatic, but it seems… it's at
least not daily. Which is a shame.

-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Chong Yidong]

Julien Danjou <julien@danjou.info> writes:

> When GNU ELPA is being updated?
>
> I though it was at least daily and automatic, but it seems… it's at
> least not daily. Which is a shame.

It's done manually, a couple of times a week (if I remember).  If you
want it updated, shoot me an email and I will do it; you'll also need to
bump the version number so that the package manager sees the change as
an update.

It seemed preferable to have some human component in the procedure of
rolling out packages to users.  For instance, I try to scan the bzr logs
before doing each update.

But I'm open to arguments for simply setting it up as a cron job.

For Emacs developers, I'd recommend deploying the elpa branch as a "site
installation" (see the README file in that branch), so that you get
updates to the elpa repository directly rather than the http package
archive.

Re: ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On Tue, Sep 27 2011, Chong Yidong wrote:

> It's done manually, a couple of times a week (if I remember).  If you
> want it updated, shoot me an email and I will do it; you'll also need to
> bump the version number so that the package manager sees the change as
> an update.

Yeah, I did that.

> It seemed preferable to have some human component in the procedure of
> rolling out packages to users.  For instance, I try to scan the bzr logs
> before doing each update.
>
> But I'm open to arguments for simply setting it up as a cron job.

I hope that the checks are done before commiting. :-) So I'd rather like
a daily cronjob rathen than disturbing you each time I commit a bugfix
that I want to give to users.
And as the number of package will increase, I'm not sure you'll be able
to do this review manually so, it might be best to trust us on short
term. :)

-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Tassilo Horn]

Julien Danjou <julien@danjou.info> writes:

>> But I'm open to arguments for simply setting it up as a cron job.
>
> I hope that the checks are done before commiting. :-) So I'd rather
> like a daily cronjob rathen than disturbing you each time I commit a
> bugfix that I want to give to users.

I second that.  Currently, I don't use GNU's ELPA but have a checkout of
the emacs elpa branch that I update and "make site" daily, and then I
add the "elpa/packages/<foo>/" dirs of the packages I'm interested into
my load-path manually.  That completely defeats the purpose of a package
manager...

Just found out: I guess, I could simply replace the entry

  ("gnu" . "http://elpa.gnu.org/packages/")

with

 ("gnu" . "/home/horn/repos/el/emacs/elpa/packages/")

in `package-archives' to always have the most current packages in
`package-list-packages'.

But as long as the people committing in the elpa branch pretend to know
what they do and nobody prooves the opposite, I'd still vote for an
automated process.  As the number of changes grows, a manual review
process won't catch all errors anyway, and then it's good to be able to
quickly commit a bugfix and know that that will hit the ELPA archive
short time later.

Bye,
Tassilo

Re: ELPA update

[Ted Zlatanov]

On Wed, 28 Sep 2011 09:48:23 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Tue, Sep 27 2011, Chong Yidong wrote:

>> It seemed preferable to have some human component in the procedure of
>> rolling out packages to users.  For instance, I try to scan the bzr logs
>> before doing each update.
>> 
>> But I'm open to arguments for simply setting it up as a cron job.

JD> I hope that the checks are done before commiting. :-) So I'd rather like
JD> a daily cronjob rathen than disturbing you each time I commit a bugfix
JD> that I want to give to users.
JD> And as the number of package will increase, I'm not sure you'll be able
JD> to do this review manually so, it might be best to trust us on short
JD> term. :)

I think the GNU ELPA is much more like a package repository than a
source code repository, so it makes sense to have some human overview,
especially considering the large number of committers.  We don't want
rogue code sneaking in and compromising our users.

To that end it would also be nice if we asked committers to sign their
contributions with their private GPG key, but I don't know if Bazaar
supports that.  If they did, we could have a list of approved public GPG
keys for any given package and contributions signed with those could be
automatically approved.  This is just a proposal though, I don't know
the best way to do it.

Most of us don't know how to run a package repository, so maybe we
should look at the Debian maintainers' process or ask them if we don't
have the local expertise.

Ted

Re: ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

On Wed, Sep 28 2011, Ted Zlatanov wrote:

> To that end it would also be nice if we asked committers to sign their
> contributions with their private GPG key, but I don't know if Bazaar
> supports that.  If they did, we could have a list of approved public GPG
> keys for any given package and contributions signed with those could be
> automatically approved.  This is just a proposal though, I don't know
> the best way to do it.

I though people having commit access to ELPA were already trusted, since
they got their write access SSH authentified?

> Most of us don't know how to run a package repository, so maybe we
> should look at the Debian maintainers' process or ask them if we don't
> have the local expertise.

Well, there's no manual review of packages already present in the
archive at Debian. Only new packages got reviewed (for licensing issue
mainly).

-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Stefan Monnier]

> I second that.  Currently, I don't use GNU's ELPA but have a checkout of
> the emacs elpa branch that I update and "make site" daily, and then I
> add the "elpa/packages/<foo>/" dirs of the packages I'm interested into
> my load-path manually.  That completely defeats the purpose of a package
> manager...

FWIW I use "make site" and then add the following to my .emacs:

   (eval-after-load 'package
     '(add-to-list 'package-directory-list "~/src/emacs/elpa/site"))
   (setq package-load-list '((auctex nil) all))
   (if (fboundp 'package-initialize) (package-initialize))

Whether it defeats the purpose of a package manager, I cannot tell, but
it does let me use the GNU ELPA packages transparently without having to
install them one by one and upgrade them, etc... and it forces me to
test the harmlessness of those packages.

BTW here's a few things about GNU ELPA, some of which are calls for help:
- We need someone to write some script that can split a stream of "elpa
  commit email messages" and dispatch them to the respective
  package maintainers.
  I consider this very important and urgent and would really welcome
  help on this.
- We need to improve the "make site" so that the compile warnings aren't
  drowned with everything else.
- We need to improve the "make archive" so it includes a ChangeLog in
  each package.
- I'd like to push GNU ELPA as a kind of "Savannah for Elisp".
  That means I encourage package maintainers to use elpa's Bzr as their
  official VCS repository.  This currently means that any package
  maintainer needs to be a member of the `emacs' group and gets write
  access not just to his package but also to other packages and to
  Emacs.  I'm not sure how well this is going to scale, but let's
  be optimistic.
- If the `elpa' repository is used for active development, that means it
  may contain packages in "unreleasable state", so we'll want to come up
  with some convention to indicate when a new release is made (and
  "make archive" should obey that).
- I'm a bit concerned about authorship (we keep a pretty close eye on
  it within Emacs, but I don't know/think we can do the same for all
  packages in GNU ELPA since not all patches will have gone through
  emacs-devel), so I'm thinking that the *VC-Log* should not
  just start with "Summary: \n\n" but with "Summary: \nAuthor: \n\n".


        Stefan

Re: ELPA update

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 28 Sep 2011 08:52:04 -0500
> 
> To that end it would also be nice if we asked committers to sign their
> contributions with their private GPG key, but I don't know if Bazaar
> supports that.

Evidently, it does, see this page:

  http://blogs.gnome.org/jamesh/2007/10/04/signed-revisions-with-bazaar/

Re: ELPA update

[Ted Zlatanov]

On Wed, 28 Sep 2011 16:00:04 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Wed, Sep 28 2011, Ted Zlatanov wrote:
>> To that end it would also be nice if we asked committers to sign their
>> contributions with their private GPG key, but I don't know if Bazaar
>> supports that.  If they did, we could have a list of approved public GPG
>> keys for any given package and contributions signed with those could be
>> automatically approved.  This is just a proposal though, I don't know
>> the best way to do it.

JD> I though people having commit access to ELPA were already trusted, since
JD> they got their write access SSH authentified?

Yes, but my proposal would give them automatic approval to their own
packages.  So if you decide to modify someone else's package you will be
able to commit but it won't be automatically approved.  The end result
would be that when you commit a bug fix, it goes out immediately, but if
you commit a new feature to my package, it won't.

>> Most of us don't know how to run a package repository, so maybe we
>> should look at the Debian maintainers' process or ask them if we don't
>> have the local expertise.

JD> Well, there's no manual review of packages already present in the
JD> archive at Debian. Only new packages got reviewed (for licensing issue
JD> mainly).

Yes, but you can't commit changes to someone else's package, can you?
And they don't roll out commits immediately, there's a release process?

Ted

Re: ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Wed, Sep 28 2011, Ted Zlatanov wrote:

Well, in our case I'll take commit = package upload.

> Yes, but you can't commit changes to someone else's package, can you?

Yes, you can on exceptional circumstances (critical bugs), it's called a
NMU (Non-Maintainer Upload).

> And they don't roll out commits immediately, there's a release process?

Every packages goes in the unstable distribution, and then the release
process takes 2 years, so it's not a valid model to copy. ;-)

-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Ted Zlatanov]

On Wed, 28 Sep 2011 10:05:30 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I second that.  Currently, I don't use GNU's ELPA but have a checkout of
>> the emacs elpa branch that I update and "make site" daily, and then I
>> add the "elpa/packages/<foo>/" dirs of the packages I'm interested into
>> my load-path manually.  That completely defeats the purpose of a package
>> manager...

SM> BTW here's a few things about GNU ELPA, some of which are calls for help:
SM> - We need someone to write some script that can split a stream of "elpa
SM>   commit email messages" and dispatch them to the respective
SM>   package maintainers.
SM>   I consider this very important and urgent and would really welcome
SM>   help on this.

Please post here or send me directly a sample of the input and I'll
write this splitter.

SM> - We need to improve the "make site" so that the compile warnings aren't
SM>   drowned with everything else.
SM> - We need to improve the "make archive" so it includes a ChangeLog in
SM>   each package.

I don't know enough, especially considering your recent notes about
removing ChangeLogs, to help here.

SM> - I'd like to push GNU ELPA as a kind of "Savannah for Elisp".
SM>   That means I encourage package maintainers to use elpa's Bzr as their
SM>   official VCS repository.  This currently means that any package
SM>   maintainer needs to be a member of the `emacs' group and gets write
SM>   access not just to his package but also to other packages and to
SM>   Emacs.  I'm not sure how well this is going to scale, but let's
SM>   be optimistic.

See my proposal in this thread about signing commits with a private key,
which should alleviate this permissions problem.

Ted

Re: ELPA update

[Ted Zlatanov]

On Wed, 28 Sep 2011 19:28:44 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Wed, Sep 28 2011, Ted Zlatanov wrote:
JD> Well, in our case I'll take commit = package upload.

Right, I see the utility of an immediate deployment, but I don't think
your commit should deploy my package.  Let's see what Stefan and Chong
think.  See below...

>> Yes, but you can't commit changes to someone else's package, can you?

JD> Yes, you can on exceptional circumstances (critical bugs), it's called a
JD> NMU (Non-Maintainer Upload).

We can have that too, with Stefan and Chong (and possibly me and anyone
else with shell access to the GNU ELPA webserver) enabled to do it.  But
not for everyone, I hope!  And see below for a more flexible approach...

>> And they don't roll out commits immediately, there's a release process?

JD> Every packages goes in the unstable distribution, and then the release
JD> process takes 2 years, so it's not a valid model to copy. ;-)

I think a 2-stage release process would be a very good thing.  For our
purposes it's enough to have an unstable stage, which could simply be
the "elpa" branch.  So developers would commit to "elpa", test it all
they want, then merge into "elpa-dev".  That merge, if the commit is
signed and the commit targets the right packages for the private keys,
would kick off the deployment; otherwise it would start the approval
queue process.  Eventually a maintainer would push to "elpa-stable" with
a release cycle of at least 3 months (maybe synchronized to Emacs
releases).  Regular committers can't push to "elpa-stable".

Users can point to the "elpa" or "elpa-dev" branched directly with 
`make site' if they want to use them, but otherwise they would use
"elpa-stable".  The GNU ELPA web site would offer the dev and stable
branches.

The 2-year cycle is too long, but I think 3 months is reasonable.  We
have a responsibility to our users to treat GNU ELPA package releases as
seriously as we treat Emacs itself, so this proposal is IMO a good
compromise between fast releases for the developers and stability for
the users.

Ted

Re: ELPA update

[Stefan Monnier]

> Well, in our case I'll take commit = package upload.

We could go this way, but I'd rather we encourage maintainers to use it
for active development.  Some of the benefit, is that Emacs developers
can then directly fix bugs in those packages.


        Stefan

Re: ELPA update

[PJ Weisberg]

On Wed, Sep 28, 2011 at 1:58 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Well, in our case I'll take commit = package upload.
>
> We could go this way, but I'd rather we encourage maintainers to use it
> for active development.  Some of the benefit, is that Emacs developers
> can then directly fix bugs in those packages.

You could set up your cron job to publish new versions only of
packages that have had their version number changed.  Updating the
version number could be analogous to "tagging" a release in a normal
one-project repository.

-PJ

Re: ELPA update

[Stephen J. Turnbull]

PJ Weisberg writes:

 > You could set up your cron job to publish new versions only of
 > packages that have had their version number changed.  Updating the
 > version number could be analogous to "tagging" a release in a normal
 > one-project repository.

That's a little constraining.  I don't do it for any personal projects
since I started using git, but I used to bump versions every time I
got a new working version.  In a multi-developer project, that kind of
custom might be harder to terminate.

How about triggering based on a tag name convention, like "a tag of
the form 'release-FOO-N-NN' means to update the downloadable package"?

Re: ELPA update

[Jambunathan K]

This could be slightly off topic but not totally unrelated.

1. Under what conditions will an ELPA package land up in core Emacs
   distribution?

2. If a user raises a bug or a feature request against a package, how
   and where should she report them.

3. Allowing for the possibility that GNU ELPA packages or even popular
   Emacs packages might get orphaned either by the author or the new
   owner (in case of ownership transitions), will the Emacs project take
   care of continued sustenance of the package.

Please note that 3 is not a hypothetical question.

ps: In another few days time, I will make yet another proposal to have
my OpenDocument exporter included in ELPA.

Jambunathan K.
-- 

Re: ELPA update

[Ted Zlatanov]

On Thu, 29 Sep 2011 10:16:10 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> PJ Weisberg writes:
>> You could set up your cron job to publish new versions only of
>> packages that have had their version number changed.  Updating the
>> version number could be analogous to "tagging" a release in a normal
>> one-project repository.

SJT> That's a little constraining.  I don't do it for any personal projects
SJT> since I started using git, but I used to bump versions every time I
SJT> got a new working version.  In a multi-developer project, that kind of
SJT> custom might be harder to terminate.

SJT> How about triggering based on a tag name convention, like "a tag of
SJT> the form 'release-FOO-N-NN' means to update the downloadable package"?

I don't see what's wrong with my proposal to have a `elpa-dev' branch
for unstable and `elpa-stable' for stable packages; merging `elpa' into
`elpa-dev' will be a much clearer sign to publish a new version than
incrementing the version number (which, as we know, is often
unparseable).

Ted

Re: ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

On Thu, Sep 29 2011, Ted Zlatanov wrote:

> I don't see what's wrong with my proposal to have a `elpa-dev' branch
> for unstable and `elpa-stable' for stable packages; merging `elpa' into
> `elpa-dev' will be a much clearer sign to publish a new version than
> incrementing the version number (which, as we know, is often
> unparseable).

For the record, I am ok to do something more cmplex than the
commit->publish work flow, as far as it is (mostly) automatic and I
don't have to ping anyone to see my package online. :)

The problem I see with branch merging, is that you can't select a
precise package to merge in a flow of commit, unless you just do a merge
without keeping the history, which kind of sucks too.
(I'm more a git expert than a bzr one, but I don't think I'm breaking
assumption here).

-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I don't see what's wrong with my proposal to have a `elpa-dev' branch
 > for unstable and `elpa-stable' for stable packages; merging `elpa' into
 > `elpa-dev' will be a much clearer sign to publish a new version than
 > incrementing the version number (which, as we know, is often
 > unparseable).

Nothing.  I was responding to P.J.'s suggestion, which wasn't bad
either.

Re: ELPA update

[Ted Zlatanov]

On Thu, 29 Sep 2011 12:14:22 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Thu, Sep 29 2011, Ted Zlatanov wrote:
>> I don't see what's wrong with my proposal to have a `elpa-dev' branch
>> for unstable and `elpa-stable' for stable packages; merging `elpa' into
>> `elpa-dev' will be a much clearer sign to publish a new version than
>> incrementing the version number (which, as we know, is often
>> unparseable).

JD> The problem I see with branch merging, is that you can't select a
JD> precise package to merge in a flow of commit, unless you just do a merge
JD> without keeping the history, which kind of sucks too.
JD> (I'm more a git expert than a bzr one, but I don't think I'm breaking
JD> assumption here).

In Git you can cherry-pick between branches, which makes a new commit ID
but the patch contents (including the commit message) are the same.  So
that lets you take just one change from one branch to another.  It's not
ideal but it's very easy to use and automate.

If that's too complicated or undesirable in Bazaar and there's no other
way to merge just a few commits from one branch to another, I'm open to
suggestions.  I still feel, no matter the merge mechanism, that we
should use the VCS branch and not the package version as the deciding
factor whether to publish a package.

Ted

Re: ELPA update

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

On Thu, Sep 29 2011, Ted Zlatanov wrote:

> In Git you can cherry-pick between branches, which makes a new commit ID
> but the patch contents (including the commit message) are the same.  So
> that lets you take just one change from one branch to another.  It's not
> ideal but it's very easy to use and automate.

Let's hope nobody will ever modify more than one file/package in a
commit. Or your strategy will begin to be much more complicated. :)

> If that's too complicated or undesirable in Bazaar and there's no other
> way to merge just a few commits from one branch to another, I'm open to
> suggestions.  I still feel, no matter the merge mechanism, that we
> should use the VCS branch and not the package version as the deciding
> factor whether to publish a package.

I do not disagree, this is actually kind of how the Linux kernel is
handled if you look at it that way.

But since there's only 2 branches (stable/dev) I'm just not sure it's
the best approach here:
- merging parts of -dev branch into a -stable branch is complicated
- tagging a branch when it's ready to be publish with so many packages
  is impossible since developement is happening there on multiple
  packages at the same time

At least, the solution to have a set of tags named "{$package}-ready"
which would move on commits will assure that each package is put into
ELPA at the right stage. You can even rollback by moving the tag.
(I'm just not sure bzr can handle that correctly, and that the bzr users
would not make mistakes)


-- 
Julien Danjou

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: ELPA update

[Stefan Monnier]

> 1. Under what conditions will an ELPA package land up in core Emacs
>    distribution?

Good question.  I think it's largely the same conditions as including
any other package in Emacs.  The difference being that a GNU ELPA
package is more "ready for inclusion" (w.r.t things like copyright or
autoload cookies).

Admittedly, the existence of GNU ELPA makes it less important to include
a package directly in Emacs, since ELPA packages can be so
easily installed.  So I think that the main candidates for inclusion
will be packages that provide core infrastructure, such as libraries.

> 2. If a user raises a bug or a feature request against a package, how
>    and where should she report them.

I think it should be to the package maintainer, but currently, we don't
do anything special for it.

> 3. Allowing for the possibility that GNU ELPA packages or even popular
>    Emacs packages might get orphaned either by the author or the new
>    owner (in case of ownership transitions), will the Emacs project take
>    care of continued sustenance of the package.
> Please note that 3 is not a hypothetical question.

Most bundled Emacs packages are orphaned in some sense, so I definitely
don't take that as hypothetical.  I don't think we need a policy on
this, but the idea is that GNU ELPA packages will end up maintained
similarly to bundled packages: so if a maintainer drops the ball,
anybody else can pick it up.  And the general "Emacs maintenance team"
may install fixes at any time and provide minimal maintenance.
And we'll probably want to structure the set of packages into
categories, with one of the categories being maybe
"obsolete/old/unmaintained".

> ps: In another few days time, I will make yet another proposal to have
> my OpenDocument exporter included in ELPA.

Please do,


        Stefan

Re: ELPA update

[Ted Zlatanov]

On Thu, 29 Sep 2011 16:09:07 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Thu, Sep 29 2011, Ted Zlatanov wrote:
>> In Git you can cherry-pick between branches, which makes a new commit ID
>> but the patch contents (including the commit message) are the same.  So
>> that lets you take just one change from one branch to another.  It's not
>> ideal but it's very easy to use and automate.

JD> Let's hope nobody will ever modify more than one file/package in a
JD> commit. Or your strategy will begin to be much more complicated. :)

So far it hasn't been a problem, most non-maintainer commits cover one
thing.

>> If that's too complicated or undesirable in Bazaar and there's no other
>> way to merge just a few commits from one branch to another, I'm open to
>> suggestions.  I still feel, no matter the merge mechanism, that we
>> should use the VCS branch and not the package version as the deciding
>> factor whether to publish a package.

JD> I do not disagree, this is actually kind of how the Linux kernel is
JD> handled if you look at it that way.

Yes, it's a common strategy.  It works well.

JD> But since there's only 2 branches (stable/dev) I'm just not sure it's
JD> the best approach here:
JD> - merging parts of -dev branch into a -stable branch is complicated

No more than managing 20+ package versions, and you have the benefit
that you can publish the branch directly from a cron job.

JD> - tagging a branch when it's ready to be publish with so many packages
JD>   is impossible since developement is happening there on multiple
JD>   packages at the same time

Right, tags are not so good with many moving parts.  Cherry-picking
is better to select only the "ready" pieces for promotion.

JD> At least, the solution to have a set of tags named "{$package}-ready"
JD> which would move on commits will assure that each package is put into
JD> ELPA at the right stage. You can even rollback by moving the tag.
JD> (I'm just not sure bzr can handle that correctly, and that the bzr users
JD> would not make mistakes)

I'm not sure tags would help as much as separate branches, but certainly
it could be beaten into a working state :)

Ted

Re: ELPA update

[Stefan Monnier]

> SM> - We need someone to write some script that can split a stream of "elpa
> SM>   commit email messages" and dispatch them to the respective
> Please post here or send me directly a sample of the input and I'll
> write this splitter.

You can subscribe to emacs-elpa-diffs to get the messages.
It's OK for the script to fail when the diff applies to more than a single
package (especially if it reports the failure somewhere), and the script should
also report a failure to find the needed maintainer's email address.


        Stefan