decision on moving core packages to ELPA; also move to obsolete?

decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

I'd like to close https://debbugs.gnu.org/39553 .

One way to resolve it is to restore the old emacs core ada-mode, but in
lisp/obsolete, and with comments indicatiung that it is unmaintained and
superceded by the ELPA package.

Another way to resolve it is to accept the status quo.

Another way is to include ada-mode in the distribution tarball, but I
believe the infrastructure for that is not yet implemented. In
addition, since ELPA ada-mode requires Ada code compilation, it is not a
direct replacement for the elisp-only old core ada-mode.

This would establish a precedent/policy for other packages that move to
ELPA but prefer not to keep a copy in core.

Personally, I prefer the status quo, but I'm ok with putting the old
code in lisp/obsolete.

Eli, Lars; I'd like an official ruling on this.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Mon, 14 Dec 2020 11:59:34 -0800
> 
> I'd like to close https://debbugs.gnu.org/39553 .
> 
> One way to resolve it is to restore the old emacs core ada-mode, but in
> lisp/obsolete, and with comments indicatiung that it is unmaintained and
> superceded by the ELPA package.
> 
> Another way to resolve it is to accept the status quo.
> 
> Another way is to include ada-mode in the distribution tarball, but I
> believe the infrastructure for that is not yet implemented. In
> addition, since ELPA ada-mode requires Ada code compilation, it is not a
> direct replacement for the elisp-only old core ada-mode.
> 
> This would establish a precedent/policy for other packages that move to
> ELPA but prefer not to keep a copy in core.
> 
> Personally, I prefer the status quo, but I'm ok with putting the old
> code in lisp/obsolete.
> 
> Eli, Lars; I'd like an official ruling on this.

I think having it in lisp/obsolete would be a good compromise, until
we figure out how to bundle ELPA packages with a release tarball.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> I think having it in lisp/obsolete would be a good compromise, until
> we figure out how to bundle ELPA packages with a release tarball.

For the record, we know how to do that.
There was a working patch for that some years back and I can't see why
they wouldn't work any more.

IOW the difficulty is not a question of implementing it but of deciding
of the resulting tradeoffs (mostly around whether or not building from
emacs.git should fetch the extra bundled GNU ELPA packages and if so
when and how).


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 14 Dec 2020 18:40:49 -0500
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> IOW the difficulty is not a question of implementing it but of deciding
> of the resulting tradeoffs (mostly around whether or not building from
> emacs.git should fetch the extra bundled GNU ELPA packages and if so
> when and how).

AFAIR, the main difficulty that remains to be resolved is how to allow
users to update a bundled package from ELPA.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Daniele Nicolodi]

On 15/12/2020 06:06, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 14 Dec 2020 18:40:49 -0500
>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
>>
>> IOW the difficulty is not a question of implementing it but of deciding
>> of the resulting tradeoffs (mostly around whether or not building from
>> emacs.git should fetch the extra bundled GNU ELPA packages and if so
>> when and how).
> 
> AFAIR, the main difficulty that remains to be resolved is how to allow
> users to update a bundled package from ELPA.

Do you mean this in a technical way or in an user experience way? Org is
distributed as part of Emacs but it is routinely also installed and
updated from ELPA and AFAIK it does not implement any special trick to
make this work smotly.

Cheers,
Dan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Tue, 15 Dec 2020 10:32:46 +0100
> 
> > AFAIR, the main difficulty that remains to be resolved is how to allow
> > users to update a bundled package from ELPA.
> 
> Do you mean this in a technical way or in an user experience way?

Is there a difference?

> Org is distributed as part of Emacs but it is routinely also
> installed and updated from ELPA and AFAIK it does not implement any
> special trick to make this work smotly.

The idea is that users should be able to install, upgrade, and
downgrade such packages exactly like they do with unbundled ones.
One can always install in another location that shadows the default
one, and even overwrite the files distributed with the tarball, but
that is hardly something we want to tell users as the method of using
ELPA packages.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> Org is distributed as part of Emacs but it is routinely also
>> installed and updated from ELPA and AFAIK it does not implement any
>> special trick to make this work smotly.
> The idea is that users should be able to install, upgrade, and
> downgrade such packages exactly like they do with unbundled ones.

I still don't see what's the problem.

With current Emacs, users can upgrade/downgrade their version of Org,
python.el (etc...) via `list-packages`.  Clearly the same would apply to
bundled ELPA packages.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Glenn Morris]

Stefan Monnier wrote:

> I still don't see what's the problem.
>
> With current Emacs, users can upgrade/downgrade their version of Org,
> python.el (etc...) via `list-packages`.  Clearly the same would apply to
> bundled ELPA packages.

Here's one problem: https://debbugs.gnu.org/44830

PS dealing with the merge conflicts between Org-in-emacs-27 and
Org-in-master prompts me to say yet again that keeping a big codebase in
two repositories is a PITA. So from that perspective a solution for
elpa-packages-distributed-with-emacs-releases would be most welcome.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Glenn Morris]

Glenn Morris wrote:

> Here's one problem: https://debbugs.gnu.org/44830

And here's another. :)
https://debbugs.gnu.org/15763#16

Re: decision on moving core packages to ELPA; also move to obsolete?

[Glenn Morris]

Glenn Morris wrote:

> And here's another. :)
> https://debbugs.gnu.org/15763#16

Sorry, that's probably a bug in org.el, not a package issue.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> I still don't see what's the problem.
>> With current Emacs, users can upgrade/downgrade their version of Org,
>> python.el (etc...) via `list-packages`.  Clearly the same would apply to
>> bundled ELPA packages.
> Here's one problem: https://debbugs.gnu.org/44830

AFAIK this problem applies to "normal" packages just as well.
E.g. if you have the Tuareg package installed, `package-install` won't
accept `tuareg` as argument.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniele Nicolodi <daniele@grinta.net>,  stephen_leake@stephe-leake.org,
>   emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 12:03:57 -0500
> 
> >> Org is distributed as part of Emacs but it is routinely also
> >> installed and updated from ELPA and AFAIK it does not implement any
> >> special trick to make this work smotly.
> > The idea is that users should be able to install, upgrade, and
> > downgrade such packages exactly like they do with unbundled ones.
> 
> I still don't see what's the problem.
> 
> With current Emacs, users can upgrade/downgrade their version of Org,
> python.el (etc...) via `list-packages`.  Clearly the same would apply to
> bundled ELPA packages.

Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
If so, what happens with installed Lisp files under /usr/share/ ?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Daniele Nicolodi <daniele@grinta.net>,  stephen_leake@stephe-leake.org,
>>   emacs-devel@gnu.org
>> Date: Tue, 15 Dec 2020 12:03:57 -0500
>> 
>> >> Org is distributed as part of Emacs but it is routinely also
>> >> installed and updated from ELPA and AFAIK it does not implement any
>> >> special trick to make this work smotly.
>> > The idea is that users should be able to install, upgrade, and
>> > downgrade such packages exactly like they do with unbundled ones.
>> 
>> I still don't see what's the problem.
>> 
>> With current Emacs, users can upgrade/downgrade their version of Org,
>> python.el (etc...) via `list-packages`.  Clearly the same would apply to
>> bundled ELPA packages.
>
> Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?

yes.

> > If so, what happens with installed Lisp files under /usr/share/ ?

They stay there, but ~/.emacs.d/elpa is earlier in load-path.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>>> Cc: Daniele Nicolodi <daniele@grinta.net>,  stephen_leake@stephe-leake.org,
>>>   emacs-devel@gnu.org
>>> Date: Tue, 15 Dec 2020 12:03:57 -0500
>>> 
>>> >> Org is distributed as part of Emacs but it is routinely also
>>> >> installed and updated from ELPA and AFAIK it does not implement any
>>> >> special trick to make this work smotly.
>>> > The idea is that users should be able to install, upgrade, and
>>> > downgrade such packages exactly like they do with unbundled ones.
>>> 
>>> I still don't see what's the problem.
>>> 
>>> With current Emacs, users can upgrade/downgrade their version of Org,
>>> python.el (etc...) via `list-packages`.  Clearly the same would apply to
>>> bundled ELPA packages.
>>
>> Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
>
> yes.
>
>> > If so, what happens with installed Lisp files under /usr/share/ ?
>
> They stay there, but ~/.emacs.d/elpa is earlier in load-path.

And also, once bundling ELPA packages in the tarball works, the package
can be removed from Emacs core, simplifying the pakcage maintenance
process.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: daniele@grinta.net,  Stefan Monnier <monnier@iro.umontreal.ca>,
>   emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 10:53:38 -0800
> 
> >> > If so, what happens with installed Lisp files under /usr/share/ ?
> >
> > They stay there, but ~/.emacs.d/elpa is earlier in load-path.
> 
> And also, once bundling ELPA packages in the tarball works, the package
> can be removed from Emacs core, simplifying the pakcage maintenance
> process.

So please describe how you envision the process of building a release
tarball under this assumption.  E.g., how do I know which version of
package A I want to bundle is stable enough to go to a bugfix elease
of Emacs?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> So please describe how you envision the process of building a release
> tarball under this assumption.  E.g., how do I know which version of
> package A I want to bundle is stable enough to go to a bugfix elease
> of Emacs?

I'd expect it to work the same as for Org, MH-E, Gnus, you name it.

But your question is getting to the more meaty stuff indeed: the problem
of bundling is one of making decisions/tradeoffs.  The actual code
needed to do the bundling is quite simple and most of it is
already written.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,  daniele@grinta.net,
>   emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 14:54:52 -0500
> 
> > So please describe how you envision the process of building a release
> > tarball under this assumption.  E.g., how do I know which version of
> > package A I want to bundle is stable enough to go to a bugfix elease
> > of Emacs?
> 
> I'd expect it to work the same as for Org, MH-E, Gnus, you name it.

What do you mean by "the same as"?  Currently, it is not our decision
which Org/MH-E/etc. version will be in what Emacs branch.  The
respective developers make that decision and simply push the version
they decided into our repository.

Once we separate the repositories, the decision will have to be made
by whoever prepares the tarball, and I don't see how he or she could
be equipped to make that decision, nor how the packages are organized
for this modus operandi.

> But your question is getting to the more meaty stuff indeed: the problem
> of bundling is one of making decisions/tradeoffs.  The actual code
> needed to do the bundling is quite simple and most of it is
> already written.

I'm not sure most of it is done.  Over the years, we had several long
discussions here about this stuff, and AFAIR none of them ended with
solutions.  All those issues raised in those discussions are still
with us, waiting to bite us.

We need to go over those issues and solve them before we can seriously
talk about unbundling ada-mode or any other package.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Dmitry Gutov]

On 15.12.2020 22:11, Eli Zaretskii wrote:

>>> So please describe how you envision the process of building a release
>>> tarball under this assumption.  E.g., how do I know which version of
>>> package A I want to bundle is stable enough to go to a bugfix elease
>>> of Emacs?
>>
>> I'd expect it to work the same as for Org, MH-E, Gnus, you name it.
> 
> What do you mean by "the same as"?  Currently, it is not our decision
> which Org/MH-E/etc. version will be in what Emacs branch.  The
> respective developers make that decision and simply push the version
> they decided into our repository.
> 
> Once we separate the repositories, the decision will have to be made
> by whoever prepares the tarball, and I don't see how he or she could
> be equipped to make that decision, nor how the packages are organized
> for this modus operandi.

Forgive me if that has already been suggested, but how about either:

- When making an Emacs X.YZ release, we require that every "external" 
package has a tag emacs-xyz in its repository, and we pull in and 
package that version.

- In the very same file in Emacs' tree that the repositories with 
external packages will be listed, we also list the corresponding 
revisions that will go into the upcoming release.

The latter is essentially the "git modules" model, which we can also 
use. One advantage it has is that the package author forgotten to push a 
tag for some Emacs release, we're still covered.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> Cc: daniele@grinta.net, stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 15 Dec 2020 22:55:21 +0200
> 
> Forgive me if that has already been suggested, but how about either:
> 
> - When making an Emacs X.YZ release, we require that every "external" 
> package has a tag emacs-xyz in its repository, and we pull in and 
> package that version.
> 
> - In the very same file in Emacs' tree that the repositories with 
> external packages will be listed, we also list the corresponding 
> revisions that will go into the upcoming release.
> 
> The latter is essentially the "git modules" model, which we can also 
> use.

I think, if the above suits us (see my other message in this thread),
we should indeed use "git submodule".

> One advantage it has is that the package author forgotten to push a 
> tag for some Emacs release, we're still covered.

You mean, advantage compared to using submodules?  If so, I don't
think I see how submodules are different in this regard.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Dmitry Gutov]

On 16.12.2020 17:33, Eli Zaretskii wrote:

>> - When making an Emacs X.YZ release, we require that every "external"
>> package has a tag emacs-xyz in its repository, and we pull in and
>> package that version.
>>
>> - In the very same file in Emacs' tree that the repositories with
>> external packages will be listed, we also list the corresponding
>> revisions that will go into the upcoming release.
>>
>> The latter is essentially the "git modules" model, which we can also
>> use.
> 
> I think, if the above suits us (see my other message in this thread),
> we should indeed use "git submodule".

Perhaps. And since it's a standard model with a checkout inside the 
worktree, it should be easy to integrate in the build script, and it 
might even solve the future problem of how to depend on the code in a 
bundled package.

>> One advantage it has is that the package author forgotten to push a
>> tag for some Emacs release, we're still covered.
> 
> You mean, advantage compared to using submodules?  If so, I don't
> think I see how submodules are different in this regard.

No, compared to the first option (external repos having special tags).

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> I'd expect it to work the same as for Org, MH-E, Gnus, you name it.
>
> What do you mean by "the same as"?  Currently, it is not our decision
> which Org/MH-E/etc. version will be in what Emacs branch.  The
> respective developers make that decision and simply push the version
> they decided into our repository.
>
> Once we separate the repositories, the decision will have to be made
> by whoever prepares the tarball, and I don't see how he or she could
> be equipped to make that decision, nor how the packages are organized
> for this modus operandi.

You tell me how you want it to work, and we'll write the support
code accordingly.  E.g. one way this could work is as follows:

Currently every GNU ELPA package has a corresponding branch
`externals/[PKGNAME]` in `elpa.git` where the source code is kept.
We could add to elpa.git a bunch of branches `emacs/[PKGNAME]` which
keep the version of the code we want to include in the next release
of Emacs.  Those branches can be updated whenever we or the package's
maintainer deems appropriate.

Our tarball-builder-script would then grab the bundled packages's code
from those branches.

> I'm not sure most of it is done.  Over the years, we had several long
> discussions here about this stuff, and AFAIR none of them ended with
> solutions.  All those issues raised in those discussions are still
> with us, waiting to bite us.

Most of the harder discussions have to do with the unsolvable problems:
E.g. we'll have code in the tarball that's not present in a Git
checkout, which means that people who work from the Git may
be disappointed.  We could "fix" that by making our Git script fetch the
extra code as well, but what about those that don't want it (or what
about the extra complexity of having to pull/update those branches)?
Also, the proposed bundling doesn't let Emacs's own code depend on GNU
ELPA code, so it still wouldn't allow us to use `dash.el` inside Emacs.

I don't see any need to "solve" or discuss those issues before we can
bundle packages.

> We need to go over those issues and solve them before we can seriously
> talk about unbundling ada-mode or any other package.

I'm not talking about unbundling anything here.  I'm talking about
bundling *additional* packages currently not distributed with Emacs.
[ BTW, ada-mode is already "unbundled", AFAIK.  ]

Obviously, the ability to bundle GNU ELPA packages might allow/encourage
moving some packages from Emacs to GNU ELPA.  It's indeed one of the
longer term benefits that makes bundling interesting, but in the short
term (i.e. Emacs-28), this is not on the table AFAIK.


        Stefan

Tests in core depending on GNU ELPA packages (was: decision on moving core packages to ELPA; also move to obsolete?)

[Thomas Fitzsimmons]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[...]

> Also, the proposed bundling doesn't let Emacs's own code depend on GNU
> ELPA code, so it still wouldn't allow us to use `dash.el` inside Emacs.
>
> I don't see any need to "solve" or discuss those issues before we can
> bundle packages.

Nice work reorganizing GNU ELPA.

As part of my work on bug#43566, I'd like to add a test to
test/lisp/net/ntlm-tests.el that depends on the GNU ELPA packages
ws-server and url-http-ntlm.  I'd prefer the test to be in core rather
than GNU ELPA, since it's specifically to prevent ntlm.el regressions
introduced by core changes (e.g., url).

Can you suggest a way to accomplish this?

Thanks,
Thomas

Re: Tests in core depending on GNU ELPA packages

[Stefan Monnier]

>> Also, the proposed bundling doesn't let Emacs's own code depend on GNU
>> ELPA code, so it still wouldn't allow us to use `dash.el` inside Emacs.
>>
>> I don't see any need to "solve" or discuss those issues before we can
>> bundle packages.
>
> Nice work reorganizing GNU ELPA.
>
> As part of my work on bug#43566, I'd like to add a test to
> test/lisp/net/ntlm-tests.el that depends on the GNU ELPA packages
> ws-server and url-http-ntlm.  I'd prefer the test to be in core rather
> than GNU ELPA, since it's specifically to prevent ntlm.el regressions
> introduced by core changes (e.g., url).
>
> Can you suggest a way to accomplish this?

You could add the tests and mark them (skip-unless (fboundp '...)).
This means they won't be tested in the "normal" regression tests where
those extra packages aren't installed, but at least anyone could run them
by installing the corresponding packages.


        Stefan

Re: Tests in core depending on GNU ELPA packages

[Thomas Fitzsimmons]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Also, the proposed bundling doesn't let Emacs's own code depend on GNU
>>> ELPA code, so it still wouldn't allow us to use `dash.el` inside Emacs.
>>>
>>> I don't see any need to "solve" or discuss those issues before we can
>>> bundle packages.
>>
>> Nice work reorganizing GNU ELPA.
>>
>> As part of my work on bug#43566, I'd like to add a test to
>> test/lisp/net/ntlm-tests.el that depends on the GNU ELPA packages
>> ws-server and url-http-ntlm.  I'd prefer the test to be in core rather
>> than GNU ELPA, since it's specifically to prevent ntlm.el regressions
>> introduced by core changes (e.g., url).
>>
>> Can you suggest a way to accomplish this?
>
> You could add the tests and mark them (skip-unless (fboundp '...)).
> This means they won't be tested in the "normal" regression tests where
> those extra packages aren't installed, but at least anyone could run them
> by installing the corresponding packages.

OK, that approach works.  For loading the files, how about the attached?
It doesn't change the "make check" results on master tip, and I
confirmed that my new NTLM tests are properly skipped (via skip-unless)
when anything goes wrong locating the GNU ELPA dependencies.

Thanks,
Thomas

diff --git a/test/Makefile.in b/test/Makefile.in
index f907602a62..ff228d1261 100644
--- a/test/Makefile.in
+++ b/test/Makefile.in
@@ -71,6 +71,15 @@ am__v_at_0 =
 am__v_at_1 =
 
 
+# Load any GNU ELPA dependencies that are present, for optional tests.
+GNU_ELPA_DIRECTORY ?= $(srcdir)/../../elpa
+# Keep elpa_dependencies dependency-ordered.
+elpa_dependencies = \
+	url-http-ntlm/url-http-ntlm.el \
+	web-server/web-server.el
+elpa_els = $(addprefix $(GNU_ELPA_DIRECTORY)/packages/,$(elpa_dependencies))
+elpa_opts = $(foreach el,$(elpa_els),$(and $(wildcard $(el)),-L $(dir $(el)) -l $(el)))
+
 # We never change directory before running Emacs, so a relative file
 # name is fine, and makes life easier.  If we need to change
 # directory, we can use emacs --chdir.
@@ -81,7 +90,7 @@ EMACS_EXTRAOPT=
 # Command line flags for Emacs.
 # Apparently MSYS bash would convert "-L :" to "-L ;" anyway,
 # but we might as well be explicit.
-EMACSOPT = --no-init-file --no-site-file --no-site-lisp -L "$(SEPCHAR)$(srcdir)" $(EMACS_EXTRAOPT)
+EMACSOPT = --no-init-file --no-site-file --no-site-lisp -L "$(SEPCHAR)$(srcdir)" $(elpa_opts) $(EMACS_EXTRAOPT)
 
 # Prevent any settings in the user environment causing problems.
 unexport EMACSDATA EMACSDOC EMACSPATH GREP_OPTIONS
@@ -105,7 +114,7 @@ TEST_RUN_ERT =
     $(if $(findstring $(MAKECMDGOALS), all check check-maybe),no,yes)
 
 # Additional settings for ert.
-ert_opts =
+ert_opts += $(elpa_opts)
 
 # Maximum length of lines in ert backtraces; nil for no limit.
 # (if empty, use the default ert-batch-backtrace-right-margin).

Re: Tests in core depending on GNU ELPA packages

[Stefan Monnier]

>> You could add the tests and mark them (skip-unless (fboundp '...)).
>> This means they won't be tested in the "normal" regression tests where
>> those extra packages aren't installed, but at least anyone could run them
>> by installing the corresponding packages.
>
> OK, that approach works.  For loading the files, how about the attached?
> It doesn't change the "make check" results on master tip, and I
> confirmed that my new NTLM tests are properly skipped (via skip-unless)
> when anything goes wrong locating the GNU ELPA dependencies.

Looks good to me,


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 17:01:42 -0500
> 
> You tell me how you want it to work, and we'll write the support
> code accordingly.

I'm not sure mine is the only voice we should listen to, but I try to
outline something below.

> E.g. one way this could work is as follows:
> 
> Currently every GNU ELPA package has a corresponding branch
> `externals/[PKGNAME]` in `elpa.git` where the source code is kept.
> We could add to elpa.git a bunch of branches `emacs/[PKGNAME]` which
> keep the version of the code we want to include in the next release
> of Emacs.

That would have to be emacs-NN/PKGNAME, where NN is the major Emacs
version.

> Those branches can be updated whenever we or the package's
> maintainer deems appropriate.

Which would mean having procedures in place for those maintainers to
make such branches as part of preparing a release and the pretest.

> Our tarball-builder-script would then grab the bundled packages's code
> from those branches.

I hope this could be much more seamless, see below.

> E.g. we'll have code in the tarball that's not present in a Git
> checkout, which means that people who work from the Git may
> be disappointed.

That's something we should resolve as part of the ELPA integration.

> I don't see any need to "solve" or discuss those issues before we can
> bundle packages.

I don't agree.  We should have a reasonably comprehensive solution in
place, and it cannot just include building the tarball.  The solution
doesn't have to be perfect, but it should exist for all of the aspects
I mention below, and maybe some more.

> > We need to go over those issues and solve them before we can seriously
> > talk about unbundling ada-mode or any other package.
> 
> I'm not talking about unbundling anything here.  I'm talking about
> bundling *additional* packages currently not distributed with Emacs.
> [ BTW, ada-mode is already "unbundled", AFAIK.  ]

That was a mistake, and this thread started as an attempt to fix it.

> Obviously, the ability to bundle GNU ELPA packages might allow/encourage
> moving some packages from Emacs to GNU ELPA.  It's indeed one of the
> longer term benefits that makes bundling interesting, but in the short
> term (i.e. Emacs-28), this is not on the table AFAIK.

AFAICT, it depends whom you are asking.

Here's what I think we should have working to start bundling ELPA
packages:

 1. Git

    We need to have the bundled ELPA packages in the Emacs Git tree,
    and we should have an easy way of cloning/updating from upstream
    in a way that will also update the packages from ELPA.

    This should be set up in a way that both master and the release
    branch get updates from designated branches of the ELPA package's
    repository.  This way, working with the Emacs Git repository will
    remain almost unchanged.  Building Emacs from Git, something I at
    least do all the time, will also remain unchanged.

    (Yes, "git submodule" commands can do most or all of the above,
    and I think we should use them.)

    Package maintainers will have to provide designated branches with
    uniform names that will serve for fetching the package into each
    branch of the Emacs repository.

    One thing that will constitute a change is that each bundled
    package will need to have its own subdirectory of lisp/, even if
    the package is just one .el file.  Not a catastrophe, I think.

    Another aspect to consider is the auxiliary files -- README,
    documentation, etc. -- which come with the package.  We will need
    some changes in the Makefile's to have the products in the right
    places.

 2. Tarball

    Given that the Git repository will be ready due to the above, the
    procedures for creating a tarball will also remain almost
    unchanged.

 3. Users

    We need a clear picture of how this Emacs will be installed and
    used, both when installed from a distro and when built locally
    from the source tarball.  Maybe there's nothing to discuss here,
    but I at least don't yet have such a clear picture.  It should be
    possible to install, upgrade, and downgrade such packages, as much
    as possible using the same package.el commands as for unbundled
    packages.  As a minimum, AFAIU we will need to provide a directory
    in the tarball/installation that will have the same role as
    ~/.emacs.d/elpa/, because a tarball cannot possibly install files
    in the users' home directories.

I hope we can discuss each of these aspects (and others, if I missed
something), and this time actually come to a consensus and implement
that.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> Here's what I think we should have working to start bundling ELPA
> packages:

Then I think the "git submodule" approach proposed earlier by Andrea
is the better solution: every ELPA package we want to include will have
a branch in `emacs.git` (e.g. with name `elpa/[PKGNAME]`) and in Emacs's
`master` branch we will add git submodules which refer to (specific
revisions on) those branches.

This way a Git checkout can also contain those ELPA packages.  We can
easily control which version of a package is included into any given
version of Emacs because git submodules refer to specific revisions.
And those `elpa/[PKGNAME]` branches can easily kept in sync with
the corresponding branches in `elpa.git` (and upstream if applicable).

>  3. Users
>
>     We need a clear picture of how this Emacs will be installed and
>     used, both when installed from a distro and when built locally
>     from the source tarball.  Maybe there's nothing to discuss here,
>     but I at least don't yet have such a clear picture.  It should be
>     possible to install, upgrade, and downgrade such packages, as much
>     as possible using the same package.el commands as for unbundled
>     packages.  As a minimum, AFAIU we will need to provide a directory
>     in the tarball/installation that will have the same role as
>     ~/.emacs.d/elpa/, because a tarball cannot possibly install files
>     in the users' home directories.

We will place the submodules under some `elpa` subdirectory of our
choice after which we simply have to add that directory to the default
value of `package-directory-list`.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 13:35:01 -0500
> 
> > Here's what I think we should have working to start bundling ELPA
> > packages:
> 
> Then I think the "git submodule" approach proposed earlier by Andrea
> is the better solution

(I also proposed to use "git submodule"...)

> every ELPA package we want to include will have
> a branch in `emacs.git` (e.g. with name `elpa/[PKGNAME]`)

It's the other way around: each package in elpa.git will have branch
named emacs/PKGNAME.  Right?

> and in Emacs's `master` branch we will add git submodules which
> refer to (specific revisions on) those branches.

Yes.

> >  3. Users
> >
> >     We need a clear picture of how this Emacs will be installed and
> >     used, both when installed from a distro and when built locally
> >     from the source tarball.  Maybe there's nothing to discuss here,
> >     but I at least don't yet have such a clear picture.  It should be
> >     possible to install, upgrade, and downgrade such packages, as much
> >     as possible using the same package.el commands as for unbundled
> >     packages.  As a minimum, AFAIU we will need to provide a directory
> >     in the tarball/installation that will have the same role as
> >     ~/.emacs.d/elpa/, because a tarball cannot possibly install files
> >     in the users' home directories.
> 
> We will place the submodules under some `elpa` subdirectory of our
> choice

Why not under lisp/ ?

And what directory will be the parent of that elpa/ subdirectory you
propose, when Emacs is installed?

> after which we simply have to add that directory to the default
> value of `package-directory-list`.

Note that this is not a single directory, but at least 2 different
ones: we need to support both running Emacs uninstalled, from the
source/build tree, and running an installed Emacs.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> every ELPA package we want to include will have
>> a branch in `emacs.git` (e.g. with name `elpa/[PKGNAME]`)
> It's the other way around: each package in elpa.git will have branch
> named emacs/PKGNAME.  Right?

If we want to use them in all Git checkouts, then I assumed it'd make
sense to host them in `emacs.git`.  But either way is fine by me.

>> We will place the submodules under some `elpa` subdirectory of our
>> choice
> Why not under lisp/ ?

Because we don't want it to interfere with the subdirs.el and other
such things.

> And what directory will be the parent of that elpa/ subdirectory you
> propose, when Emacs is installed?

I think it'd makes sense to put use /usr/share/emacs/NN.MM/elpa
(i.e. keep it as a sibling of `lisp` both in the source code and in the
install).

>> after which we simply have to add that directory to the default
>> value of `package-directory-list`.
> Note that this is not a single directory, but at least 2 different
> ones: we need to support both running Emacs uninstalled, from the
> source/build tree, and running an installed Emacs.

It's only one directory, but indeed just like `etc`, `lisp`, and others,
its location will depend on whether Emacs is run uninstalled or not.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 15:01:44 -0500
> 
> >> We will place the submodules under some `elpa` subdirectory of our
> >> choice
> > Why not under lisp/ ?
> 
> Because we don't want it to interfere with the subdirs.el and other
> such things.

What kind of interference are we talking about here?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> >> We will place the submodules under some `elpa` subdirectory of our
>> >> choice
>> > Why not under lisp/ ?
>> Because we don't want it to interfere with the subdirs.el and other
>> such things.
> What kind of interference are we talking about here?

I don't know, I haven't thought about it, but you were worried about it
earlier in the thread (where you also mentioned
normal-top-level-add-subdirs-* IIRC), so I think you know more than
I about that ;-)


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So please describe how you envision the process of building a release
>> tarball under this assumption.  E.g., how do I know which version of
>> package A I want to bundle is stable enough to go to a bugfix elease
>> of Emacs?
>
> I'd expect it to work the same as for Org, MH-E, Gnus, you name it.

But those all currently have code in emacs.git; later you say that makes
them not candidates for ELPA bundling. So this is inconsistent.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>>> So please describe how you envision the process of building a release
>>> tarball under this assumption.  E.g., how do I know which version of
>>> package A I want to bundle is stable enough to go to a bugfix elease
>>> of Emacs?
>>
>> I'd expect it to work the same as for Org, MH-E, Gnus, you name it.
>
> But those all currently have code in emacs.git; later you say that makes
> them not candidates for ELPA bundling. So this is inconsistent.

Because the above sentence of mine is talking about who decides (and
how) which particular version of which package will be bundled.
This same process works regardless *how* the package is bundled, so
whatever we do for packages currently copied into emacs.git, we can do
the same for bundled ELPA packages.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: daniele@grinta.net,  Stefan Monnier <monnier@iro.umontreal.ca>,
>>   emacs-devel@gnu.org
>> Date: Tue, 15 Dec 2020 10:53:38 -0800
>> 
>> >> > If so, what happens with installed Lisp files under /usr/share/ ?
>> >
>> > They stay there, but ~/.emacs.d/elpa is earlier in load-path.
>> 
>> And also, once bundling ELPA packages in the tarball works, the package
>> can be removed from Emacs core, simplifying the pakcage maintenance
>> process.
>
> So please describe how you envision the process of building a release
> tarball under this assumption.  E.g., how do I know which version of
> package A I want to bundle is stable enough to go to a bugfix elease
> of Emacs?

The simplest choice is the ELPA version that is current at the time the
tarball is built.

Nominally, ELPA versions are defined to be stable. But given the ease of
releasing a new version via ELPA, some package developers may have a
policy of releasing without significant testing. In that case, it might
make sense to ask developers to put more effort into validating a
particular version for a release. Or simply say such a package is not a
good candidate for bundling.

In my case, I always run a large test suite on ada-mode before I do an
ELPA release.

If we want to freeze some earlier version at the start of release
testing (as opposed to final tarball build time), we'd need some
mechanism to record the versions of all the bundled packages, and then
only use those versions in testing. And a way to change that frozen
version number for a bug fix.

Or ask ELPA packages that are bundled to also freeze for the duration of
release testing; that would be reasonable, and simplify handling package
bug fixes.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: daniele@grinta.net,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 11:21:31 -0800
> 
> > So please describe how you envision the process of building a release
> > tarball under this assumption.  E.g., how do I know which version of
> > package A I want to bundle is stable enough to go to a bugfix elease
> > of Emacs?
> 
> The simplest choice is the ELPA version that is current at the time the
> tarball is built.

I don't think this is a good idea: bundling a package means that we as
the project assume the responsibility for its quality, which means we
need to be reasonably sure the version we bundle is stable.  This
requires more from the package maintainers, but it's the other side of
having the package bundled.

> If we want to freeze some earlier version at the start of release
> testing (as opposed to final tarball build time), we'd need some
> mechanism to record the versions of all the bundled packages, and then
> only use those versions in testing. And a way to change that frozen
> version number for a bug fix.
> 
> Or ask ELPA packages that are bundled to also freeze for the duration of
> release testing; that would be reasonable, and simplify handling package
> bug fixes.

Yes, we will need something like that, and we will need a mechanism to
make a checkout of the Emacs Git repository included the bundled
packages more or less automatically, so that builds from the Git
repository still work as they do today.  See the other discussions in
this thread for the details.

We need to figure all that out, and have this set up for each package
we want to bundle, before we bundle it.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
> yes.
>> > If so, what happens with installed Lisp files under /usr/share/ ?
> They stay there, but ~/.emacs.d/elpa is earlier in load-path.

Not only that: we could very easily setup the bundled packages such that
they are only added to `load-path` by `package-activate`, so if a newer
version is in ~/.emacs.d/elpa the bundled version won't even make it to
the `load-path` (contrary to what happens with the packages hosted in
Core like `python.el` or Org).


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 14:57:41 -0500
> 
> >> Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
> > yes.
> >> > If so, what happens with installed Lisp files under /usr/share/ ?
> > They stay there, but ~/.emacs.d/elpa is earlier in load-path.
> 
> Not only that: we could very easily setup the bundled packages such that
> they are only added to `load-path` by `package-activate`, so if a newer
> version is in ~/.emacs.d/elpa the bundled version won't even make it to
> the `load-path` (contrary to what happens with the packages hosted in
> Core like `python.el` or Org).

So now we are talking about changing the basic logic in
normal-top-level-add-subdirs-to-load-path and
normal-top-level-add-to-load-path?  And maybe in subdirs.el as well?
We can, of course, do any or all of those, but we do need a
comprehensive picture of what needs to be done and how this stuff will
work when done, in all the relevant use cases.  What we have now is a
long list of issues and a longer list of ideas how to solve them.
Like I said, a lot of loose ends that needs to be tied.

We are not ready.  Not yet.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> So now we are talking about changing the basic logic in
> normal-top-level-add-subdirs-to-load-path and
> normal-top-level-add-to-load-path?

Not at all.

What I'm suggesting is the following:

- the tarball we build will include the same file as before in
  `emacs/lisp`.
- it will additionally contain a new directory `emacs/elpa` in which
  each bundled package has its own directory (all in the normal format
  of installed packages in ~/.emacs.d/elpa).
- the only change to the code of Emacs will be that at startup
  `package-directory-list` will include this `emacs/elpa` directory.

So until `package-activate-all` is called, the bundled packages will
just sit there on your file system but Emacs won't "see" them.

We could also place some or all of the bundled packages directly inside
`lisp` and have them be activated in the same way all other Emacs's code
is activated (i.e. basically by loading `loaddef.el` at dump time), so
they'll behave a bit less like normal packages and a bit more like Org
and Tramp do now (i.e. you can't "not have them"), but I think the above
suggestion is more conservative and flexible for the user (the downside
is that it's less efficient at startup).


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> We could also place some or all of the bundled packages directly inside
> `lisp` and have them be activated in the same way all other Emacs's code
> is activated (i.e. basically by loading `loaddef.el` at dump time), so
> they'll behave a bit less like normal packages and a bit more like Org
> and Tramp do now (i.e. you can't "not have them"), but I think the above
> suggestion is more conservative and flexible for the user (the downside
> is that it's less efficient at startup).

There would be a version conflict then for builtin packages, like
Tramp. Usually, I freeze the code in the Emacs tree months before the
real release. For Emacs 27.1, this freeze was done in December 2019
(Tramp 2.4.3.27.1); Emacs 27.1 was released in August 2020, when ELPA
did offer Tramp 2.4.5 already.

So you would have two different versions of Tramp bundled with the Emacs
tarball. I believe, builtin packages like Tramp or Org shouldn't be
bundled in the tarball as ELPA packages. People can always upgrade to
the ELPA version, if needed.

>         Stefan

Best regards, Michael.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> So you would have two different versions of Tramp bundled with the Emacs
> tarball.

Of course not:  We're not discussing bundling every single GNU
ELPA package, but only bundling *some* GNU ELPA packages.  We haven't
even started to discuss which ones we'd choose, but obviously we
wouldn't include those packages which are already present in
Emacs itself.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So you would have two different versions of Tramp bundled with the Emacs
>> tarball.
>
> Of course not:  We're not discussing bundling every single GNU
> ELPA package, but only bundling *some* GNU ELPA packages.  We haven't
> even started to discuss which ones we'd choose, but obviously we
> wouldn't include those packages which are already present in
> Emacs itself.

Thanks. I'll sleep next night better :-)

>         Stefan

Best regards, Michael.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So now we are talking about changing the basic logic in
>> normal-top-level-add-subdirs-to-load-path and
>> normal-top-level-add-to-load-path?
>
> Not at all.
>
> What I'm suggesting is the following:
>
> - the tarball we build will include the same file as before in
>   `emacs/lisp`.
> - it will additionally contain a new directory `emacs/elpa` in which
>   each bundled package has its own directory (all in the normal format
>   of installed packages in ~/.emacs.d/elpa).

We might want to use git submodule to have the selected packages (and
the precise sha1 we want for their release) there.

This gives the advantage that everybody could obtain and test the
packages bundled with the Emacs release just checking-out all the git
submodules of emacs.git.

Developers with this system can also develop directly in the folder of
the bundled package given this is a git repo pointing to the original
branch in elpa.git.

Every time we decide to update a bundled package this would translates
into a regular commit in emacs.git updating the sha1 of the submodule,
so is very easy to keep track of these.

Yes thinking about I'm quite convinced submodule fits pretty naturally
what we are trying to do here, and would be a solution with no
additional dependencies and that does not require custom scripting.

  Andrea

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 17:09:11 -0500
> 
> What I'm suggesting is the following:
> 
> - the tarball we build will include the same file as before in
>   `emacs/lisp`.
> - it will additionally contain a new directory `emacs/elpa` in which
>   each bundled package has its own directory (all in the normal format
>   of installed packages in ~/.emacs.d/elpa).

So we will have 2 copies of each package's Lisp files in the tarball?

> So until `package-activate-all` is called, the bundled packages will
> just sit there on your file system but Emacs won't "see" them.

This already happens, right? we already call package-activate-all at
startup, right?

> We could also place some or all of the bundled packages directly inside
> `lisp`

Now I'm confused: how 'lisp/' is different from 'emacs/lisp' you
mentioned above?

> and have them be activated in the same way all other Emacs's code
> is activated (i.e. basically by loading `loaddef.el` at dump time), so
> they'll behave a bit less like normal packages and a bit more like Org
> and Tramp do now (i.e. you can't "not have them"), but I think the above
> suggestion is more conservative and flexible for the user (the downside
> is that it's less efficient at startup).

What are the pros and cons of each of these 2 alternatives?  I think
we should carefully consider them before deciding which one we prefer.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> What I'm suggesting is the following:
>> - the tarball we build will include the same file as before in
>>   `emacs/lisp`.
>> - it will additionally contain a new directory `emacs/elpa` in which
>>   each bundled package has its own directory (all in the normal format
>>   of installed packages in ~/.emacs.d/elpa).
> So we will have 2 copies of each package's Lisp files in the tarball?

No, just one, the one in the `emacs/elpa` directory.

>> So until `package-activate-all` is called, the bundled packages will
>> just sit there on your file system but Emacs won't "see" them.
> This already happens, right? we already call package-activate-all at
> startup, right?

Yes, between `early-init.el` and `init.el`.

>> We could also place some or all of the bundled packages directly inside
>> `lisp`
> Now I'm confused: how 'lisp/' is different from 'emacs/lisp' you
> mentioned above?

It's the same thing.  I just omitted to write the "emacs/" prefix that time.

> What are the pros and cons of each of these 2 alternatives?  I think
> we should carefully consider them before deciding which one we prefer.

Basically, the question is whether the autoloads of those ELPA packages
are processed once and for all when we dump Emacs (like we do for all
the packages that come with Emacs), or whether that's done during
`package-activate-all` (i.e. between `early-init.el` and `init.el`).

Doing it at dump time gives better startup times, at the cost of making
it impossible for the end-user to prevent activation of a package (they
can still undo the activation after the fact, of course, but that needs
to be done in ad-hoc ways).

I think as a first step we should keep those bundled ELPA packages more
like normal ELPA packages (i.e. activate them from
`package-activate-all` rather than when dumping Emacs).  We can later
revise this (even on a per-package basis) once we have more experience.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 13:46:37 -0500
> 
> >> What I'm suggesting is the following:
> >> - the tarball we build will include the same file as before in
> >>   `emacs/lisp`.
> >> - it will additionally contain a new directory `emacs/elpa` in which
> >>   each bundled package has its own directory (all in the normal format
> >>   of installed packages in ~/.emacs.d/elpa).
> > So we will have 2 copies of each package's Lisp files in the tarball?
> 
> No, just one, the one in the `emacs/elpa` directory.

So what did you mean by "the tarball will include the same file as
before in emacs/lisp"? which file will that be, and how is it
different from what will be in emacs/elpa?

> >> We could also place some or all of the bundled packages directly inside
> >> `lisp`
> > Now I'm confused: how 'lisp/' is different from 'emacs/lisp' you
> > mentioned above?
> 
> It's the same thing.  I just omitted to write the "emacs/" prefix that time.

Still confused: how is that different from the first alternative,
where you said the file will be in emacs/lisp?

> > What are the pros and cons of each of these 2 alternatives?  I think
> > we should carefully consider them before deciding which one we prefer.
> 
> Basically, the question is whether the autoloads of those ELPA packages
> are processed once and for all when we dump Emacs (like we do for all
> the packages that come with Emacs), or whether that's done during
> `package-activate-all` (i.e. between `early-init.el` and `init.el`).
> 
> Doing it at dump time gives better startup times, at the cost of making
> it impossible for the end-user to prevent activation of a package (they
> can still undo the activation after the fact, of course, but that needs
> to be done in ad-hoc ways).
> 
> I think as a first step we should keep those bundled ELPA packages more
> like normal ELPA packages (i.e. activate them from
> `package-activate-all` rather than when dumping Emacs).  We can later
> revise this (even on a per-package basis) once we have more experience.

I actually like the other alternative, because people will want faster
startup, and because it makes no sense to let users disable bundled
packages.  If we think people will want to disable a package, we
shouldn't bundle it.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> So what did you mean by "the tarball will include the same file as
> before in emacs/lisp"?

I meant that the bundling of ELPA packages won't touch the `lisp`
subdirectory at all because the added files are placed elsewhere (in
the sibling `elpa` directory).

> I actually like the other alternative, because people will want faster
> startup, and because it makes no sense to let users disable bundled
> packages.  If we think people will want to disable a package, we
> shouldn't bundle it.

OK, we can do that.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniele Nicolodi <daniele@grinta.net>,  stephen_leake@stephe-leake.org,
>   emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 12:03:57 -0500
> 
> >> Org is distributed as part of Emacs but it is routinely also
> >> installed and updated from ELPA and AFAIK it does not implement any
> >> special trick to make this work smotly.
> > The idea is that users should be able to install, upgrade, and
> > downgrade such packages exactly like they do with unbundled ones.
> 
> I still don't see what's the problem.
> 
> With current Emacs, users can upgrade/downgrade their version of Org,
> python.el (etc...) via `list-packages`.  Clearly the same would apply to
> bundled ELPA packages.

Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
If so, what happens with installed Lisp files under /usr/share/ ?

(Sorry, sent the previous response too soon.)

And what about the relation between the version in ELPA and the
branches/versions of Emacs in the Emacs repository?  IOW, how will a
package that needs Emacs version N+1 work with Emacs version N?

Bottom line, I feel that there's some kind of "trust us, it will be
fine" attitude here, whereas I would expect careful investigation of
all these aspects and some description of the procedures.  We had
discussions about this a year or two ago, and my impression from them
was that there are still loose ends that no one has bothered to
resolve.  I don't think dismissing these aspects is a good way of
making progress here.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
> If so, what happens with installed Lisp files under /usr/share/ ?

The way package.el works, is that it collects all the packages it can
find installed locally under `package-directory-list`.  This list can
contain many different versions of the same package.  `package-activate`
then choose which ones of those packages to activate, under the
constraint that only one version of each package can be activated in
a given session (and under the additional constraints setup by the user
in `package-load-list`).

This was designed so that a sysadmin can install a bunch of packages and
users can then make use of them without being stuck using all those the
sysadmin has chosen to install, not stuck with the version that the
sysadmin installed.

If the Emacs tarball bundles packages, it would basically act as "a
sysadmin" in this regard.  Users could still override that set of
bundled packages with older/newer versions or even choose not to
activate some of the bundled packages (tho I'd hope that the packages we
choose to bundle are clean enough that this would never be useful, just
like it's not considered useful for the user to be able to remove stuff
from lisp/loaddefs.el).

> And what about the relation between the version in ELPA and the
> branches/versions of Emacs in the Emacs repository?  IOW, how will a
> package that needs Emacs version N+1 work with Emacs version N?

What about it?  The packages bundled with Emacs-N+1 would *not* be in
the `package-directory-list` of Emacs-N (and vice-versa), so there'd be
no particular issue at stake here.

> Bottom line, I feel that there's some kind of "trust us, it will be
> fine" attitude here, whereas I would expect careful investigation of
> all these aspects and some description of the procedures.

AFAIK all the investigation that can be done has been done.  I don't
doubt that there will be issues to resolve, but I don't think we will
find them before we actually start doing it.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: daniele@grinta.net,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 15:11:47 -0500
> 
> > Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
> > If so, what happens with installed Lisp files under /usr/share/ ?
> 
> The way package.el works, is that it collects all the packages it can
> find installed locally under `package-directory-list`.  This list can
> contain many different versions of the same package.  `package-activate`
> then choose which ones of those packages to activate, under the
> constraint that only one version of each package can be activated in
> a given session (and under the additional constraints setup by the user
> in `package-load-list`).
> 
> This was designed so that a sysadmin can install a bunch of packages and
> users can then make use of them without being stuck using all those the
> sysadmin has chosen to install, not stuck with the version that the
> sysadmin installed.
> 
> If the Emacs tarball bundles packages, it would basically act as "a
> sysadmin" in this regard.  Users could still override that set of
> bundled packages with older/newer versions or even choose not to
> activate some of the bundled packages (tho I'd hope that the packages we
> choose to bundle are clean enough that this would never be useful, just
> like it's not considered useful for the user to be able to remove stuff
> from lisp/loaddefs.el).

Are you talking about something that already works, or about something
that _could_ work, after some changes?  If the former, then where's
the code to support it?

> > And what about the relation between the version in ELPA and the
> > branches/versions of Emacs in the Emacs repository?  IOW, how will a
> > package that needs Emacs version N+1 work with Emacs version N?
> 
> What about it?  The packages bundled with Emacs-N+1 would *not* be in
> the `package-directory-list` of Emacs-N (and vice-versa), so there'd be
> no particular issue at stake here.

That won't fly, because usually some previous version of that same
package did work with Emacs-N.  And since packages don't usually have
a stable branch and a development branch, how to bundle them and how
to upgrade them later becomes a mess.  Some of that was discussed in
this thread, for example:

  https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg01000.html

> > Bottom line, I feel that there's some kind of "trust us, it will be
> > fine" attitude here, whereas I would expect careful investigation of
> > all these aspects and some description of the procedures.
> 
> AFAIK all the investigation that can be done has been done.  I don't
> doubt that there will be issues to resolve, but I don't think we will
> find them before we actually start doing it.

We are being asked to "start doing it" right now in emacs-27.  I'm
sorry, but this is not a serious suggestion, given the state of the
affairs and the number of issues for which we just think we know how
to solve them.  Like that pianist that knows how to play the piano,
but has never tried.

And what does it mean "start doing it", really?  We prepare a tarball
only once we start a pretest, by which time it's too late to
experiment with untested machinery.  And doing that on master makes no
sense, because there are no tarballs.  So here's one more issue for
us: how can we ever get this stuff to maturity if we cannot test-drive
it until it is too late?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> Are you talking about something that already works, or about something
> that _could_ work, after some changes?

The sysadmin scenario I described is one that has worked pretty much
from the beginning of package.el.

> If the former, then where's the code to support it?

`package-activate` is the function which activates the
"preferred" version of a package among those that have been found.

>> > And what about the relation between the version in ELPA and the
>> > branches/versions of Emacs in the Emacs repository?  IOW, how will a
>> > package that needs Emacs version N+1 work with Emacs version N?
>> 
>> What about it?  The packages bundled with Emacs-N+1 would *not* be in
>> the `package-directory-list` of Emacs-N (and vice-versa), so there'd be
>> no particular issue at stake here.
> That won't fly, because usually some previous version of that same
> package did work with Emacs-N.

I don't see how this statement relates to what I said or to what you had
said before.

> And since packages don't usually have a stable branch and
> a development branch, how to bundle them and how to upgrade them later
> becomes a mess.

That's a very vague statement, so I'm not sure what to do here.

> Some of that was discussed in this thread, for example:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg01000.html

This thread talks about issues that happen if you install a package
Foo-N.M from an Emacs session where Foo-P.Q is in active use.
This applies to bundled and unbundled packages equally and is completely
orthogonal to the discussion at hand.
[ Yes, the discussion was around Org, because it's a large and popular
  package, so it's one where the problem appeared with more frequency,
  and it also happens to be distributed with Emacs, but that's just an
  accidental aspect: the same problem affected (and to some extent
  still affects) other non-bundled packages.  ]

> how can we ever get this stuff to maturity if we cannot test-drive it
> until it is too late?

I guess at this point the question for me is: are you interested in
finding a positive answer to the question?


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: daniele@grinta.net,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 17:25:46 -0500
> 
> > Are you talking about something that already works, or about something
> > that _could_ work, after some changes?
> 
> The sysadmin scenario I described is one that has worked pretty much
> from the beginning of package.el.

That's not what I asked.

> I guess at this point the question for me is: are you interested in
> finding a positive answer to the question?

Of course, I don't!  I actually think that this silly idea shouldn't
have seen the light of day.  I just need to present that in a sneaky
way so that no one makes me.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> > Are you talking about something that already works, or about something
>> > that _could_ work, after some changes?
>> The sysadmin scenario I described is one that has worked pretty much
>> from the beginning of package.el.
> That's not what I asked.

Then I don't know what you asked :-(
Could you clarify your question?

>> I guess at this point the question for me is: are you interested in
>> finding a positive answer to the question?
> Of course, I don't!  I actually think that this silly idea shouldn't
> have seen the light of day.  I just need to present that in a sneaky
> way so that no one makes me.

You almost managed to fool me ;-)


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: daniele@grinta.net,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 13:50:10 -0500
> 
> >> > Are you talking about something that already works, or about something
> >> > that _could_ work, after some changes?
> >> The sysadmin scenario I described is one that has worked pretty much
> >> from the beginning of package.el.
> > That's not what I asked.
> 
> Then I don't know what you asked :-(
> Could you clarify your question?

Here's how we got to that:

> > Doesn't package.el install stuff under ~/.emacs.d/elpa/ ?
> > If so, what happens with installed Lisp files under /usr/share/ ?
> 
> The way package.el works, is that it collects all the packages it can
> find installed locally under `package-directory-list`.  This list can
> contain many different versions of the same package.  `package-activate`
> then choose which ones of those packages to activate, under the
> constraint that only one version of each package can be activated in
> a given session (and under the additional constraints setup by the user
> in `package-load-list`).
  > 
  > This was designed so that a sysadmin can install a bunch of packages and
  > users can then make use of them without being stuck using all those the
  > sysadmin has chosen to install, not stuck with the version that the
  > sysadmin installed.
  > 
  > If the Emacs tarball bundles packages, it would basically act as "a
  > sysadmin" in this regard.  Users could still override that set of
  > bundled packages with older/newer versions or even choose not to
  > activate some of the bundled packages (tho I'd hope that the packages we
  > choose to bundle are clean enough that this would never be useful, just
  > like it's not considered useful for the user to be able to remove stuff
  > from lisp/loaddefs.el).

  Are you talking about something that already works, or about something
  that _could_ work, after some changes?  If the former, then where's
  the code to support it?

My question was about the last part: it seemed to describe how things
_should_ work, but I'm not sure we already have that implemented.  Do
we?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>   Are you talking about something that already works, or about something
>   that _could_ work, after some changes?  If the former, then where's
>   the code to support it?
>
> My question was about the last part: it seemed to describe how things
> _should_ work, but I'm not sure we already have that implemented.  Do
> we?

What I'm describing is the use of a facility that was implemented for
another use-case scenario.  There is no code to write to use that
facility: just add the relevant directory to `package-directory-list`
and you're done.  So yes, it's implemented and it works.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: daniele@grinta.net,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 15:06:59 -0500
> 
> >   Are you talking about something that already works, or about something
> >   that _could_ work, after some changes?  If the former, then where's
> >   the code to support it?
> >
> > My question was about the last part: it seemed to describe how things
> > _should_ work, but I'm not sure we already have that implemented.  Do
> > we?
> 
> What I'm describing is the use of a facility that was implemented for
> another use-case scenario.  There is no code to write to use that
> facility: just add the relevant directory to `package-directory-list`
> and you're done.  So yes, it's implemented and it works.

But that addition of a directory (or maybe more than one) to
package-directory-list -- that bit still needs to be done, for this to
really work, right?

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> But that addition of a directory (or maybe more than one) to
> package-directory-list -- that bit still needs to be done, for this to
> really work, right?

Of course, as in the patch below, for example.


        Stefan


diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index b7c48dfd3f..64a2acac3e 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -305,6 +305,7 @@ package-directory-list
       (and (stringp f)
            (equal (file-name-nondirectory f) "site-lisp")
            (push (expand-file-name "elpa" f) result)))
+    (push (expand-file-name "../elpa" data-directory) result)
     (nreverse result))
   "List of additional directories containing Emacs Lisp packages.
 Each directory name should be absolute.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> AFAIK all the investigation that can be done has been done.  I don't
>> doubt that there will be issues to resolve, but I don't think we will
>> find them before we actually start doing it.
>
> We are being asked to "start doing it" right now in emacs-27.  

No, in master, for the 28 release ("it" meaning bundling ada-mode and
maybe other ELPA packages in the release tarball). 

> I'm sorry, but this is not a serious suggestion, given the state of
> the affairs and the number of issues for which we just think we know
> how to solve them. Like that pianist that knows how to play the piano,
> but has never tried.
>
> And what does it mean "start doing it", really?  We prepare a tarball
> only once we start a pretest, by which time it's too late to
> experiment with untested machinery.  

Clearly we need to test this; I suggest we create a branch
feature/bundle-elpa, and do a practice tarball release on that branch.

We also need a place to document decisions, rationales, and process; I
suggest admin/bundle-elpa to start. We may migrate parts of that into
other documents later.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  daniele@grinta.net,
>   emacs-devel@gnu.org
> Date: Wed, 16 Dec 2020 11:44:48 -0800
> 
> > And what does it mean "start doing it", really?  We prepare a tarball
> > only once we start a pretest, by which time it's too late to
> > experiment with untested machinery.  
> 
> Clearly we need to test this; I suggest we create a branch
> feature/bundle-elpa, and do a practice tarball release on that branch.

Fine with me, assuming we can convince enough people to try that
tarball.

> We also need a place to document decisions, rationales, and process; I
> suggest admin/bundle-elpa to start. We may migrate parts of that into
> other documents later.

SGTM, thanks.

ELPA notes, README

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

I'm starting to work on documenting how to bundle Gnu ELPA packages in
the Emacs distribution tarball.

As a first step, I checked out the current documentation, and I have
some suggested improvements even without the bundling option.

In Emacs admin/notes/elpa: see attached elpa_notes.diff. This reflects
recent changes to elpa.

In ELPA README: see attached elpa_readme.diff. This moves 'make
packages/<pkgname>' to the "getting the source" section, and documents
the need for 'make setup'.

Ok to commit?

Next step: I'll go thru the recent thread and compile the policy and
process info into emacs/admin/notes/elpa (without committing anything).

-- 
-- Stephe

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: notes_elpa.diff --]
[-- Type: text/x-patch, Size: 1827 bytes --]

diff --git a/admin/notes/elpa b/admin/notes/elpa
index ea6c132fe1..1e9e7a9f52 100644
--- a/admin/notes/elpa
+++ b/admin/notes/elpa
@@ -5,17 +5,31 @@ repository named "elpa", hosted on Savannah.  To check it out:
 
   git clone git://git.sv.gnu.org/emacs/elpa
   cd elpa
-  git remote set-url --push origin git+ssh://git.sv.gnu.org/srv/git/emacs/elpa
-  [create task branch for edits, etc.]
+  make setup
 
-Changes to this branch propagate to elpa.gnu.org via a "deployment" script run
-daily.  This script (which is kept in elpa/admin/update-archive.sh) generates
-the content visible at https://elpa.gnu.org/packages.
+That leaves the elpa/packages directory empty; you must check out the
+ones you want.
 
-A new package is released as soon as the "version number" of that package is
-changed.  So you can use 'elpa' to work on a package without fear of releasing
-those changes prematurely.  And once the code is ready, just bump the
-version number to make a new release of the package.
+If you wish to check out all the packages into the packages directory,
+you can run the command:
+
+   make worktrees
+
+You can check out a specific package <pkgname> into the packages
+directory with:
+
+   make packages/<pkgname>
+
+
+Changes to this repository propagate to elpa.gnu.org via a
+"deployment" script run daily.  This script generates the content
+visible at https://elpa.gnu.org/packages.
+
+A new package is released as soon as the "version number" of that
+package is changed.  So you can use 'elpa' to work on a package
+without fear of releasing those changes prematurely.  And once the
+code is ready, just bump the version number to make a new release of
+the package.
 
 It is easy to use the elpa branch to deploy a "local" copy of the
 package archive.  For details, see the README file in the elpa branch.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: elpa_readme.diff --]
[-- Type: text/x-patch, Size: 1862 bytes --]

diff --git a/README b/README
index d6114a852c..9e77d6c1f3 100644
--- a/README
+++ b/README
@@ -23,6 +23,36 @@ for testing purposes).
 Start with source that is cloned directly from Savannah.  See [[https://savannah.gnu.org/git/?group=emacs][the Savannah page]]
 and look for "ELPA".  Using a clone of a clone does not work.
 
+You must then do some setup:
+#+begin_src shell
+   make setup
+#+end_src
+
+That leaves the =packages= directory empty; you must check out the
+ones you want.
+
+If you wish to check out all the packages into the =packages=
+directory, you can run the command:
+
+#+begin_src shell
+   make worktrees
+#+end_src
+
+You can check out a specific package =<pkgname>= into the =packages=
+directory with this command:
+
+#+begin_src
+   make packages/<pkgname>
+#+end_src
+
+If you already have a =packages/<pkgname>= directory with a previous
+checkout, you can update it like this:
+
+#+begin_src
+   cd packages/PACKAGE
+   git pull
+#+end_src
+
 * Directory layout
 
 ** =admin/=    -- scripts for administering the package archive.
@@ -221,28 +251,6 @@ and push that change to the master branch of =elpa=.  After it's added to
 the =elpa-packages= file, the package can be maintained just by
 pushing changes to the =externals/<pkgname>= branch.
 
-If you wish to check out all the packages into the =packages=
-directory, you can run the command:
-
-#+begin_src shell
-   make worktrees
-#+end_src
-
-You can check out a specific package =<pkgname>= into the =packages=
-directory with these commands:
-
-#+begin_src
-   make packages/<pkgname>
-#+end_src
-
-If you already have a =packages/<pkgname>= directory with a previous
-checkout, you can update it like this:
-
-#+begin_src 
-   cd packages/PACKAGE
-   git pull
-#+end_src
-
 ** Public incubation
 
 If you want to develop a package publicly prior to its first release (to

Re: ELPA notes, README

[Stefan Monnier]

> Ok to commit?

LGTM, tho I'd rather try and avoid duplicating into
emacs/admin/notes/elpa the info available in elpa/README.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

>> IOW the difficulty is not a question of implementing it but of deciding
>> of the resulting tradeoffs (mostly around whether or not building from
>> emacs.git should fetch the extra bundled GNU ELPA packages and if so
>> when and how).
> AFAIR, the main difficulty that remains to be resolved is how to allow
> users to update a bundled package from ELPA.

I can't remember any such difficulty.  `package.el` is fully equipped to
deal with having to choose between various versions of packages.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Tue, 15 Dec 2020 09:05:18 -0500
> 
> >> IOW the difficulty is not a question of implementing it but of deciding
> >> of the resulting tradeoffs (mostly around whether or not building from
> >> emacs.git should fetch the extra bundled GNU ELPA packages and if so
> >> when and how).
> > AFAIR, the main difficulty that remains to be resolved is how to allow
> > users to update a bundled package from ELPA.
> 
> I can't remember any such difficulty.  `package.el` is fully equipped to
> deal with having to choose between various versions of packages.

Not what I remember.  But maybe if you describe how to go about
upgrading/downgrading a bundled package, I will see where I am
confused.

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> Not what I remember.  But maybe if you describe how to go about
> upgrading/downgrading a bundled package, I will see where I am
> confused.

You `M-x list-packages`, then select the package you want to
upgrade/downgrade, like for any non-bundled package.

I must be missing something.


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Eli, Lars; I'd like an official ruling on this.
>
> I think having it in lisp/obsolete would be a good compromise, until
> we figure out how to bundle ELPA packages with a release tarball.

Yup.  But figuring out the latter would be really nice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Eli, Lars; I'd like an official ruling on this.
>>
>> I think having it in lisp/obsolete would be a good compromise, until
>> we figure out how to bundle ELPA packages with a release tarball.
>
> Yup.  But figuring out the latter would be really nice.

I volunteer ada-mode to be the first such package. I suspect there's
nothing I need to edit in ada-mode, but let me know.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> I'd like to close https://debbugs.gnu.org/39553 .

This provoked a discussion on how to distribute an Gnu ELPA package in
the Emacs tarball, with ada-mode as an example (see earlier messages in
this thread).

But we already have working examples of such packages; project, eldoc,
jsonrpc, etc. They are ":core" packages; apparently the Gnu ELPA
repository build code retrieves them from emacs.git instead of elpa.git.

So the simplest solution for ada-mode is to move it to emacs.git, and
make it a :core package. I would still maintain a separate upstream
repository in git.savannah.nongnu.org/git/ada-mode.git, and only update
emacs.git with releases (as I do currently for elpa.git).

This would have no effect on my current ada-mode release process, other
than the precise name of the publishing repository. It would also have
no effect on the current Emacs development process.

One possible problem with this; ada-mode contains one huge file
ada_lr1_parse_table.txt.gz; this is a full LR1 parse table for the Ada
language; it is about 5 MB compressed in ada-mode 7.1.4, and will grow
to about 23 MB in the next release (Ada 2020 has several new features).
The current emacs master .git is 804 MB, so that's not a huge increase,
but it is significant.

Another problem: currently :core only supports single files, so we
would need to extend it to a directory, and decide where that directory
lives in emacs.git (lisp/progmodes/ada-mode/ seems the obvious choice).

Another problem: ada-mode requires two external executables; the parser
and xref tools. If those executables are not present, every significant
ada-mode function results in an error message requesting that the user
install them. The current ada-mode ELPA package distributes a shell
script build.sh that builds and installs those executables, assuming the
user has an Ada compiler and the required libraries installed. That is
ok for an ELPA package that the user decides to install; it seems less
ok for a package included in the emacs tarball distribution. This
objection applies no matter where ada-mode code resides.

-- 
-- Stephe

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stefan Monnier]

> So the simplest solution for ada-mode is to move it to emacs.git, and
> make it a :core package. I would still maintain a separate upstream
> repository in git.savannah.nongnu.org/git/ada-mode.git, and only update
> emacs.git with releases (as I do currently for elpa.git).

I think this is a very suboptimal solution since synchronization between
the two copies becomes painful.

> This would have no effect on my current ada-mode release process, other
> than the precise name of the publishing repository. It would also have
> no effect on the current Emacs development process.
>
> One possible problem with this; ada-mode contains one huge file
> ada_lr1_parse_table.txt.gz; this is a full LR1 parse table for the Ada
> language; it is about 5 MB compressed in ada-mode 7.1.4, and will grow
> to about 23 MB in the next release (Ada 2020 has several new features).

[ IIUC this is a generated file, so it should ideally not be stored in
  Git, tho I understand it might be impractical, like requiring an Ada
  compiler to (re)generate.  ]

> The current emacs master .git is 804 MB, so that's not a huge increase,
> but it is significant.

[ FWIW, mine says 470MB.  It probably depends on how recently Git ran
  its GC.  ]


        Stefan

Re: decision on moving core packages to ELPA; also move to obsolete?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So the simplest solution for ada-mode is to move it to emacs.git, and
>> make it a :core package. I would still maintain a separate upstream
>> repository in git.savannah.nongnu.org/git/ada-mode.git, and only update
>> emacs.git with releases (as I do currently for elpa.git).
>
> I think this is a very suboptimal solution since synchronization between
> the two copies becomes painful.

Because it is not a simple 'git push" to publish. ok.

>> One possible problem with this; ada-mode contains one huge file
>> ada_lr1_parse_table.txt.gz; this is a full LR1 parse table for the Ada
>> language; it is about 5 MB compressed in ada-mode 7.1.4, and will grow
>> to about 23 MB in the next release (Ada 2020 has several new features).
>
> [ IIUC this is a generated file, so it should ideally not be stored in
>   Git, tho I understand it might be impractical, like requiring an Ada
>   compiler to (re)generate.  ]

Yes, this could be moved into build.sh to be done after elpa install.
Currently it takes ~ 200 seconds to run on my fast machine, but it's
only once at build time. I'll do that for the next release.

-- 
-- Stephe