Emacs release and bundling GNU Elpa

Emacs release and bundling GNU Elpa

[Artur Malabarba]

I think the ability to specify Elpa packages that should be bundled
with Emacs would be a big improvement.

Stefan, you've mentioned a few times here you'd like some
build-scripts integration between GNU Elpa and the Emacs source at
build time. Could you clarify on what it would take to get that done?

- Is it just a matter of pulling in some extra specified packages when
building the tarball?
- What/where are the files involved in that process?
- How would this work with developers who just clone emacs.git and run
`make bootstrap`? Should the script ask the user to clone GElpa too?


Thanks and cheer
Artur

Re: Emacs release and bundling GNU Elpa

[Stefan Monnier]

> Stefan, you've mentioned a few times here you'd like some
> build-scripts integration between GNU Elpa and the Emacs source at
> build time. Could you clarify on what it would take to get that done?

Good question.  It's mostly open.
I see it happening in steps, mostly for historical reasons:

step-1: we change the tarball-building script so as to pull some
  packages from elpa.git and include them somewhere under the `lisp'
  directory (for example) in the tarballs we distribute.
step-2: we change the build scripts used from a Git checkout so they can
  also use those packages from elpa.git.
step-3: we change the build scripts again so they always use those
  packages from elpa.git.

At step 3, we'd have the following novelties:
- the end user has to checkout both emacs.git and elpa.git before she
  can build Emacs (I suspect there will be some resistance, here).
- packages in emacs/lisp can start to depend on packages from elpa.git.
- we could even include preload some elpa.git packages (i.e. from loadup.el).

I'm not exactly sure we'll ever get to step 3.

> - Is it just a matter of pulling in some extra specified packages when
> building the tarball?

At first, yes.  Tho the "just" has to be adjusted based on the need to
specify exactly which version of the package to include.

> - What/where are the files involved in that process?

admin/make-tarball.txt should give you all the answers.  But the main
step is clearly to run the `make-dist' script.


        Stefan

Re: Emacs release and bundling GNU Elpa

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> At step 3, we'd have the following novelties:
> - the end user has to checkout both emacs.git and elpa.git before she
>   can build Emacs (I suspect there will be some resistance, here).
> - packages in emacs/lisp can start to depend on packages from elpa.git.
> - we could even include preload some elpa.git packages (i.e. from loadup.el).
>
> I'm not exactly sure we'll ever get to step 3.


Would it not be possible to put elpa.get into emacs.git as some kind of
subtree?

The reason I ask, is that I try to keep Emacs running somewhere close to
trunk for my daily emacs, but I find that I get mysterious errors in
unrelated packages. These are normally solved by deleting my
.emacs.d/elpa directory and starting from fresh (my config, like many I
am sure, makes this easy).

It would be nice if running make clean on emacs.git checkout would also
do a make clean on its related elpa.git.

Just a thought.

Phil

Re: Emacs release and bundling GNU Elpa

[Artur Malabarba]

2015-06-22 12:20 GMT+01:00 Phillip Lord <phillip.lord@newcastle.ac.uk>:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> At step 3, we'd have the following novelties:
>> - the end user has to checkout both emacs.git and elpa.git before she
>>   can build Emacs (I suspect there will be some resistance, here).
>> - packages in emacs/lisp can start to depend on packages from elpa.git.
>> - we could even include preload some elpa.git packages (i.e. from loadup.el).
>>
>> I'm not exactly sure we'll ever get to step 3.
>
>
> Would it not be possible to put elpa.get into emacs.git as some kind of
> subtree?

Or a submodule?

> It would be nice if running make clean on emacs.git checkout would also
> do a make clean on its related elpa.git.

I thinThe elpa.git would not be built. I

Re: Emacs release and bundling GNU Elpa

[Artur Malabarba]

>> It would be nice if running make clean on emacs.git checkout would also
>> do a make clean on its related elpa.git.
>
> I thinThe elpa.git would not be built. I

Stupid trackpad... Sent by accident.

I think the elpa.git would be built. Unless we're planning of bundling
the entire GNU Elpa into Emacs, it would only be better to copy those
few desired packages into emacs.git somewhere, and then these packages
would be compiled as part of building emacs.

Re: Emacs release and bundling GNU Elpa

[Artur Malabarba]

2015-06-22 14:30 GMT+01:00 Artur Malabarba <bruce.connor.am@gmail.com>:
>>> It would be nice if running make clean on emacs.git checkout would also
>>> do a make clean on its related elpa.git.
>>
>> I thinThe elpa.git would not be built. I
>
> Stupid trackpad... Sent by accident.
>
> I think the elpa.git would be built.

Gaah!! Sorry again. I meant: Would NOT be build.

Re: Emacs release and bundling GNU Elpa

[Phillip Lord]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-06-22 14:30 GMT+01:00 Artur Malabarba <bruce.connor.am@gmail.com>:
>>>> It would be nice if running make clean on emacs.git checkout would also
>>>> do a make clean on its related elpa.git.
>>>
>>> I thinThe elpa.git would not be built. I
>>
>> Stupid trackpad... Sent by accident.
>>
>> I think the elpa.git would be built.
>
> Gaah!! Sorry again. I meant: Would NOT be build.



You are having a rough time of it!

So, I think either way could work. Or setting up paths so that only the
desired packages are there, and byte-compiling the relevant bits).

One problem that leaps to mind, BTW, with the current ELPA set up is
that you don't actually get all of ELPA packages at once with a single
clone, since many packages are on branches. So, you'd need to do
multiple clones (depending on how many externals you want). Or something
clever with git-new-workdir. I have little experience with either of
these, I am afraid.

Phil

Re: Emacs release and bundling GNU Elpa

[Eli Zaretskii]

Fair disclosure: I don't like this "move to ELPA" attitude.  I think
the net result will be more bugs because of unsynchronized development
and less exposure of packages to people who track development on
master, and more hassle due to the need to work with more than one Git
repository, multiple development philosophies, etc.

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 21 Jun 2015 20:45:36 -0400
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> step-1: we change the tarball-building script so as to pull some
>   packages from elpa.git and include them somewhere under the `lisp'
>   directory (for example) in the tarballs we distribute.
> step-2: we change the build scripts used from a Git checkout so they can
>   also use those packages from elpa.git.
> step-3: we change the build scripts again so they always use those
>   packages from elpa.git.
> 
> At step 3, we'd have the following novelties:
> - the end user has to checkout both emacs.git and elpa.git before she
>   can build Emacs (I suspect there will be some resistance, here).
> - packages in emacs/lisp can start to depend on packages from elpa.git.
> - we could even include preload some elpa.git packages (i.e. from loadup.el).
> 
> I'm not exactly sure we'll ever get to step 3.

At step 1, we'd have the following novelties:

 . Package developers will have to abide by some of the core's
   development methodology, like freezing development when core does
   so, perhaps using release branches, timely fixing of critical or
   blocking bugs during pretests, etc., let alone abiding by style and
   documentation guidelines.

 . Core maintainers will probably start pushing more changes to the
   packages, something I'm not sure package developers will like.
   Alternatively, we will have to develop procedures and protocols we
   don't currently have for having the package developers themselves
   expedite fixing of certain problems deemed urgent.

 . We'd need to find a way of providing ChangeLogs for the packages,
   either by merging their Git logs somehow, or by keeping their
   ChangeLogs in separate directories (which would mean each package
   will have its own directory, making load-path longer).

 . We'd need to produce NEWS entries for the packages, which will
   probably mean the packages will have to maintain their own NEWS
   files, using the same methodology and style as in core development.

 . If any of the packages have manuals, or are mentioned in the Emacs
   manuals, changes there will have to be merged as well, and we will
   have to track those updates, e.g. like we do in NEWS.

 . Our defcustom's have a ':version' tag, which is useful for quickly
   examining new options since some release -- how will this work in
   packages whose release cycle is not synchronized with Emacs?  At
   the very least, some changes to support that in
   customize-changed-options will be needed.  Similarly with
   make-obsolete: we will need at least some standardized wording for
   the WHEN argument, to avoid confusion between versions of Emacs and
   the packages.

(I'm sure I'm missing some more fun.)

Re: Emacs release and bundling GNU Elpa

[Stefan Monnier]

> Would it not be possible to put elpa.get into emacs.git as some kind of
> subtree?

"git subtree", no, I don't think that'll fly.

But some of the suggested steps might indeed want to use "git submodule".
This said, that would be just a detail of implementation of those steps.


        Stefan

Re: Emacs release and bundling GNU Elpa

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> Fair disclosure: I don't like this "move to ELPA" attitude.  I think
> the net result will be more bugs because of unsynchronized development
> and less exposure of packages to people who track development on
> master, and more hassle due to the need to work with more than one Git
> repository, multiple development philosophies, etc.
>
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Sun, 21 Jun 2015 20:45:36 -0400
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> 
>> step-1: we change the tarball-building script so as to pull some
>>   packages from elpa.git and include them somewhere under the `lisp'
>>   directory (for example) in the tarballs we distribute.
>> step-2: we change the build scripts used from a Git checkout so they can
>>   also use those packages from elpa.git.
>> step-3: we change the build scripts again so they always use those
>>   packages from elpa.git.
>> 
>> At step 3, we'd have the following novelties:
>> - the end user has to checkout both emacs.git and elpa.git before she
>>   can build Emacs (I suspect there will be some resistance, here).
>> - packages in emacs/lisp can start to depend on packages from elpa.git.
>> - we could even include preload some elpa.git packages (i.e. from loadup.el).
>> 
>> I'm not exactly sure we'll ever get to step 3.
>
> At step 1, we'd have the following novelties:
>
>  . Package developers will have to abide by some of the core's
>    development methodology, like freezing development when core does
>    so, perhaps using release branches, timely fixing of critical or
>    blocking bugs during pretests, etc., let alone abiding by style and
>    documentation guidelines.

Only for ELPA pacakges that will be included in the Emacs tarball. These
are the same requirements as on packages that are in emacs git, so it's
nothing new.

>  . Core maintainers will probably start pushing more changes to the
>    packages, something I'm not sure package developers will like.

That is a requirement of including a package in the tarball. Again,
nothing new.

>  . We'd need to find a way of providing ChangeLogs for the packages,
>    either by merging their Git logs somehow, or by keeping their
>    ChangeLogs in separate directories (which would mean each package
>    will have its own directory, making load-path longer).

Good points.

>  . We'd need to produce NEWS entries for the packages, which will
>    probably mean the packages will have to maintain their own NEWS
>    files, using the same methodology and style as in core development.

That should be the case for ELPA packages anyway.

>  . If any of the packages have manuals, or are mentioned in the Emacs
>    manuals, changes there will have to be merged as well, and we will
>    have to track those updates, e.g. like we do in NEWS.

yes.

>  . Our defcustom's have a ':version' tag, which is useful for quickly
>    examining new options since some release -- how will this work in
>    packages whose release cycle is not synchronized with Emacs?  At
>    the very least, some changes to support that in
>    customize-changed-options will be needed.  Similarly with
>    make-obsolete: we will need at least some standardized wording for
>    the WHEN argument, to avoid confusion between versions of Emacs and
>    the packages.

I suggest :version contain the package version. To correlate that with
an Emacs version, the NEWS entry for the package release date should be
sufficient. 

-- 
-- Stephe

Re: Emacs release and bundling GNU Elpa

[Stefan Monnier]

>  . Package developers will have to abide by some of the core's
>    development methodology, like freezing development when core does
>    so, perhaps using release branches, timely fixing of critical or
>    blocking bugs during pretests, etc., let alone abiding by style and
>    documentation guidelines.

Yup, no worse than what happens for current bundled packages.
I'm not expecting major issues in this respect.  Furthermore, the
pressure to have the latest&greatest version bundled is less severe
since users can easily install a newer version later on.

>  . Core maintainers will probably start pushing more changes to the
>    packages, something I'm not sure package developers will like.

That's part of deciding which packages get bundled.  Again, that's no
worse than what happens with current bundled packages.

>  . We'd need to find a way of providing ChangeLogs for the packages,
>    either by merging their Git logs somehow, or by keeping their
>    ChangeLogs in separate directories (which would mean each package
>    will have its own directory, making load-path longer).

When installed via package.el, they currently each get their own
directory indeed.  Maybe we could combine some of those directories, but
this decision will probably not be based on minor issues such as where
to put the ChangeLog entries.

>  . We'd need to produce NEWS entries for the packages, which will
>    probably mean the packages will have to maintain their own NEWS
>    files, using the same methodology and style as in core development.

I was thinking of putting the NEWS entries of those packages in the
normal NEWS file.  They'd be updated by hand when we change the version
of this elpa.git package that the emacs.git tree refers to
(remember emacs.git wouldn't just say "bundle company", but "bundle
version <foo> of company").

>  . If any of the packages have manuals, or are mentioned in the Emacs
>    manuals, changes there will have to be merged as well, and we will
>    have to track those updates, e.g. like we do in NEWS.

package.el already knows how to handle manuals, and while we may decide
to handle it slightly differently, I don't foresee any major issues
there (at least none that are really important to solve).

>  . Our defcustom's have a ':version' tag, which is useful for quickly
>    examining new options since some release -- how will this work in
>    packages whose release cycle is not synchronized with Emacs?

Good question.  This one has existed for a long time and is not directly
related to whether ELPA packages get included in Emacs's tarball.

>    At the very least, some changes to support that in
>    customize-changed-options will be needed.
>    Similarly with make-obsolete: we will need at least some
>    standardized wording for the WHEN argument, to avoid confusion
>    between versions of Emacs and the packages.

No.  We can keep living with these problems unsolved.  Of course, I'd be
happy to see them solved, but bundling ELPA packages doesn't make them
significantly more pressing.


        Stefan

Re: Emacs release and bundling GNU Elpa

[Artur Malabarba]

I was thinking that keep a list somewhere of packages that we want to
pull from elpa. At build time, these will be copied from "./elpa.git/"
(which will probably be a submodule containing the elpa repo) to
"./lisp/from-elpa/" (which will be a .gitignored directory containing
nothing initially). This should also involve copying info manuals from
"./elpa.git/" to the right place.
I see only two differences from the current situation:

1. Instead of the usual `git pull && make bootstrap` the user wil be
expected to do something like `git pull && git submodule update &&
make bootstrap` which should be a no-brainer (maybe there's a hook or
something that can automatically do the update when `pull'ing).

2. To push a change, instead of editing the file inside
lisp/from-elpa/ you have to edit the file in the elpa.git submodule.
This is a relevant difference, but it shouldn't lead to issues. Since
the from-elpa dir is gitignored, there's no way for people to
accidentally commit changes to it. And having to edit on elpa.git is
certainly an improvement over having to edit on both emacs and
elpa.git.

2015-06-22 16:33 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
> Fair disclosure: I don't like this "move to ELPA" attitude.  I think
> the net result will be more bugs because of unsynchronized development

Why would it unsynchronise? Everyone who `git pulls` today at 12pm has
the same Emacs. Similarly, under this alternative, everyone who pulls
&& updates today at 12pm will have the same Emacs.

> and less exposure of packages to people who track development on
> master,

These packages will effectively be part of master. Pulling them in
will be a part of the build process, why would there be less exposure?

> and more hassle due to the need to work with more than one Git
> repository, multiple development philosophies, etc.

This is a fine point against moving out of emacs packages that are
only in emacs.
OTOH, we can start with these other two options (a) pull into emacs
packages that are only on Elpa (like company), and (b) move out of
emacs packages that are on both repositories (these are already a
hassle, and the hassle would only be reduced).


> At step 1, we'd have the following novelties:
>
>  . Package developers will have to abide by some of the core's
>    development methodology, like freezing development when core does
>    so, perhaps using release branches, timely fixing of critical or
>    blocking bugs during pretests, etc., let alone abiding by style and
>    documentation guidelines.

Yes. This wouldn't be applied to all of elpa.git, only to those
packages that are bundled in.

>  . Core maintainers will probably start pushing more changes to the
>    packages, something I'm not sure package developers will like.
>    Alternatively, we will have to develop procedures and protocols we
>    don't currently have for having the package developers themselves
>    expedite fixing of certain problems deemed urgent.

Yes, this is something the package developer would have to be aware,
in order for a package to be bundled in. Nobody is forced to bundle a
package in Emacs.
OTOH, for packages that are currently on both repos this already
happens, and it will be much better if for the developer if these
changes are all made to the same place.

>  . We'd need to find a way of providing ChangeLogs for the packages,
>    either by merging their Git logs somehow, or by keeping their
>    ChangeLogs in separate directories (which would mean each package
>    will have its own directory, making load-path longer).

Keep their ChangeLogs in the same place as their source (in the
"./elpa.git/packages/package-name/" directory). This has nothing to do
with the load-path. The load-path will point to "./lisp/from-elpa"
(which is where we copy the sources before compiling).

>  . We'd need to produce NEWS entries for the packages, which will
>    probably mean the packages will have to maintain their own NEWS
>    files, using the same methodology and style as in core development.

Yes. That can sit alongside the ChangeLog.

>  . If any of the packages have manuals, or are mentioned in the Emacs
>    manuals, changes there will have to be merged as well, and we will
>    have to track those updates, e.g. like we do in NEWS.

As I said at the start. Just like the sources are pulled in at
compiled time, so are their manuals. Changelog entries for edits to
the manual go in the package's ChangeLog.

>  . Our defcustom's have a ':version' tag, which is useful for quickly
>    examining new options since some release -- how will this work in
>    packages whose release cycle is not synchronized with Emacs?

They will use the :package-version tag. Like packages have always done.

Re: Emacs release and bundling GNU Elpa

[Eli Zaretskii]

> Date: Mon, 22 Jun 2015 18:45:50 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel <emacs-devel@gnu.org>
> 
> > Fair disclosure: I don't like this "move to ELPA" attitude.  I think
> > the net result will be more bugs because of unsynchronized development
> 
> Why would it unsynchronise?

Because Emacs development adds new features that the packages may or
may not want to use, and Emacs deprecates obsolete features that the
packages may or may not want to remove.  When a single team handles
that, the policy is always consistent; not so when independent teams
with independent agendas do the same.

> > and less exposure of packages to people who track development on
> > master,
> 
> These packages will effectively be part of master.

Only if one decides to pull them.  Since that's at least a bit of an
additional hassle, one could decide not to bother.

> >  . We'd need to find a way of providing ChangeLogs for the packages,
> >    either by merging their Git logs somehow, or by keeping their
> >    ChangeLogs in separate directories (which would mean each package
> >    will have its own directory, making load-path longer).
> 
> Keep their ChangeLogs in the same place as their source (in the
> "./elpa.git/packages/package-name/" directory). This has nothing to do
> with the load-path. The load-path will point to "./lisp/from-elpa"
> (which is where we copy the sources before compiling).

Aren't you forgetting that ChangeLog files need to be part of the
release tarball?  Or do you mean we will have each package twice in a
tarball, in 2 different directories?

Re: Emacs release and bundling GNU Elpa

[Artur Malabarba]

>> > and less exposure of packages to people who track development on
>> > master,
>>
>> These packages will effectively be part of master.
>
> Only if one decides to pull them.  Since that's at least a bit of an
> additional hassle, one could decide not to bother.

If this is taken forward, copying these packages sources from elpa.git
to emacs not be optional, i.e., it will be a mandatory step in the
make command. This is the only way for other packages in “./lisp/” to
be able to depend on these imported packages.
Updating the submodule will only be necessary once in a while (when
the configuration file specifies a newer version for a package than
the version you have locally, and the makefile will tell you about
that), but it will be necessary.

>> >  . We'd need to find a way of providing ChangeLogs for the packages,
>> >    either by merging their Git logs somehow, or by keeping their
>> >    ChangeLogs in separate directories (which would mean each package
>> >    will have its own directory, making load-path longer).
>>
>> Keep their ChangeLogs in the same place as their source (in the
>> "./elpa.git/packages/package-name/" directory). This has nothing to do
>> with the load-path. The load-path will point to "./lisp/from-elpa"
>> (which is where we copy the sources before compiling).
>
> Aren't you forgetting that ChangeLog files need to be part of the
> release tarball?  Or do you mean we will have each package twice in a
> tarball, in 2 different directories?

Good point. How about, whenever we copy the source to
“./lisp/from-elpa/” we also copy the ChangeLog to the same place, but
rename it to something like ChangeLog-packagename? So they can all
coexist in that directory.

Re: Emacs release and bundling GNU Elpa

[Stefan Monnier]

> If this is taken forward, copying these packages sources from elpa.git
> to emacs not be optional, i.e., it will be a mandatory step in the
> make command. This is the only way for other packages in “./lisp/” to
> be able to depend on these imported packages.
> Updating the submodule will only be necessary once in a while (when
> the configuration file specifies a newer version for a package than
> the version you have locally, and the makefile will tell you about
> that), but it will be necessary.

I think that for 25.1, we could limit this to step 1 or 2.  I.e. it
should mandatorily be included in the tarball, but when building from
Git it should still be optional (or not supported at all).


        Stefan

Re: Emacs release and bundling GNU Elpa

[Barry]

Stefan:
> step-3: we change the build scripts again so they always use those
> packages from elpa.git.

What are the essential reasons for having two Git repos rather than one?

Stefan:
> - the end user has to checkout both emacs.git and elpa.git before
>   she can build Emacs (I suspect there will be some resistance,
>   here).

Is the vision that core Emacs could eventually be reduced to the Elisp engine and other infrastructure, and ELPA constitutes the features? If so, then shouldn't Emacs in principle be buildable and runnable without ELPA packages, with the caveat that commands would be missing?

Eli:
> I think the net result will be more bugs because of unsynchronized
> development and less exposure of packages to people who track
> development on master

Right, tracking emacs.git master would hopefully entail tracking elpa.git master.

Eli:
> Core maintainers will probably start pushing more changes to the
> packages, something I'm not sure package developers will like.

This is a virtue. Package maintainers would be involved in the same way as when their package is in emacs.git.

Eli:
> We'd need to find a way of providing ChangeLogs for the packages

Duplicates 'git log PACKAGE-DIR'.

Re: Emacs release and bundling GNU Elpa

[Stefan Monnier]

>> step-3: we change the build scripts again so they always use those
>> packages from elpa.git.
> What are the essential reasons for having two Git repos rather than one?

If you look beyond actual repositories (which are just arbitrary
containers of data), there are more than 2 branches: there's the
emacs.git branch "master", the elpa.git branch "master", and the various
other elpa.git branches of the form "externals/<pkg>".

We want ELPA packages to live their own life, to be released at their
own pace.

> Is the vision that core Emacs could eventually be reduced to the Elisp
> engine and other infrastructure, and ELPA constitutes the features?

No.

>> I think the net result will be more bugs because of unsynchronized
>> development and less exposure of packages to people who track
>> development on master

Yes, there will be more total bugs, simply because the Emacs tarball will
be bigger.  But as long as those "bundled ELPA packages" (BEPs) don't
affect other bundled packages, it's not worse than if the user
had to install those packages by hand.

So all it takes to keep those extra bugs under control is to check the
autoloads of those BEPs.


        Stefan

Re: Emacs release and bundling GNU Elpa

[Eli Zaretskii]

> From: Barry <gundaetiapo@yahoo.com>
> Date: Wed, 24 Jun 2015 01:12:16 -0700
> 
> Eli:
> > We'd need to find a way of providing ChangeLogs for the packages
> 
> Duplicates 'git log PACKAGE-DIR'.

No: I meant provide them in the release tarball.

Re: Emacs release and bundling GNU Elpa

[Phillip Lord]

Barry <gundaetiapo@yahoo.com> writes:
>> I think the net result will be more bugs because of unsynchronized
>> development and less exposure of packages to people who track
>> development on master
>
> Right, tracking emacs.git master would hopefully entail tracking
> elpa.git master.


"masters". elpa.git is a multi-headed beast, unfortunately. Many
packages are on branches.

Phil