policy discussion on bundling ELPA packages in the emacs tarball

policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 4363 bytes --]

Attached is the file 'elpa', containing the current
emacs/admin/notes/elpa, plus a baseline policy for bundling GNU ELPA
packages in the emacs tarball, and some unresolved issues.

This is _not_ a final plan; it is a starting point for discussion.

(I tried to post it in the savannah ada-mode download area, but trying
to retrieve it gives me 404. And now I can't delete it. Sigh).

The biggest open issue is whether the bundled packages live only in the
elpa.git worktree, or if they are git submodules that are checked out into
the emacs.git worktree.

The baseline policy described in the attached file states that the
bundled packages live only in the elpa git worktree.

Eli has said he wants the bundled packages in the emacs worktree; I'd
like to explore the specific rationale for that.

For the specific example of ada-mode, I'm guessing nobody much noticed
when its files were removed from emacs/lisp/progmodes. Other packages
that are not used by many Emacs developers will be similarly not much
missed.

At the moment, ada-mode is the _only_ package we are proposing to
bundle, as a test case. But we need to plan for the future.

Currently, org and others are "duplicated" packages; they have files in
emacs.git master/lisp/[PKGNAME]/* and in elpa.git externals/[PKGNAME].
At some point, they could become "bundled" packages.

Consider what this would look like for org under the baseline policy.
The files in master/lisp/org/* are deleted, and "org elpa.git
externals/org" is added to the list of bundled packages used by 'make
dist' (I need to add explicit detail to that mechanism; Stefan Monnier
mentioned this might already be written somewhere).

Any Emacs developer that wants to continue using org has two choices;
install it via M-x list-packages, or, if they want the development head,
checkout ELPA, run 'make packages/org', and either install it from there
or add elpa/packages/org to their load-path. This is exactly what Emacs
and ELPA developers do now for non-bundled packages.

The baseline policy describes a shell script `checkout-bundled-elpa' and
an emacs command `add-bundled-elpa-packages' (for ~/.emacs) to simplify
using all bundled packages.

Now consider what bundling org looks like if we use some variant of the
"git submodules" policy. (for an introduction to git submodules, see
https://git-scm.com/book/en/v2/Git-Tools-Submodules). The files in
master/lisp/org/* are deleted, and "org elpa.git externals/org" is added
to the emacs/master/.gitmodules file (via 'git submodule add ...').
'make dist' includes emacs/master/elpa, so nothing in 'make dist' is
changed for 'org'.

In addition, Emacs developers who want the bundled ELPA packages would
run 'git submodule update --init'; this would checkout all the bundled
ELPA packages, into emacs/master/elpa/[PKGNAME]. Emacs startup code
would put these in the default load-path.

A fresh checkout of Emacs with the bundled submodules would be done by
'git clone .../emacs.git --recurse-submodules'. Ideally, the savannah
page describing how to get Emacs from git would say that. but many
people will forget the --recurse-submodules.


In both cases, the bundled packages are in load-path.

Finding them via C-x C-f (find-file) is slightly different; in the
baseline case it requires navigating to <root>/elpa/packages/... for
bundled packages. In the submodules case is requires navigating to
<root>/emacs/master/elpa/... Both are different from the current
<root>/emacs/master/lisp/..., so in either case you have to remember
which packages are bundled, and that can change over time. I suggest we
standardize on an emacs elisp project that allows using C-x p f
(project-find-file) to complete on files in load-path for elisp files;
that removes the issue of remembering where things are.

Another difference is testing for a release; that should always be done
with the identified release version of bundled packages. That is easier
to accomplish via git submodules, since they can identify a specific
version (I think?), or a specific release branch. Otherwise people doing
release testing would have to use `checkout-bundled-elpa' to get the
correct versions in elpa.

There will probably be a similar number of posts to emacs.devel about
"where is org?"; the answer is either "cd elpa; make packages/org" or
"git submodule update --init".

Comments?

--
-- Stephe

[-- Attachment #2: elpa --]
[-- Type: application/octet-stream, Size: 7219 bytes --]

NOTES ON THE EMACS PACKAGE ARCHIVE

The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git
repository named "elpa", hosted on Savannah.  To check it out:

  git clone git://git.sv.gnu.org/emacs/elpa
  cd elpa
  make setup

That leaves the elpa/packages directory empty; you must check out the
ones you want.

If you wish to check out all the packages into the packages directory,
you can run the command:

   make worktrees

You can check out a specific package <pkgname> into the packages
directory with:

   make packages/<pkgname>


Changes to this repository propagate to elpa.gnu.org via a
"deployment" script run daily.  This script generates the content
visible at https://elpa.gnu.org/packages.

A new package is released as soon as the "version number" of that
package is changed.  So you can use 'elpa' to work on a package
without fear of releasing those changes prematurely.  And once the
code is ready, just bump the version number to make a new release of
the package.

It is easy to use the elpa branch to deploy a "local" copy of the
package archive.  For details, see the README file in the elpa branch.


ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL

Why Bundle?

    - To provide backward compatibility for users while allowing more
      flexible releases, and more convenient maintenance.

      - ada-mode was originally included in the emacs tarball, but was
        removed when the ELPA version was capable enough. Some users
        complained; bundling ada-mode would (almost) restore the
        original behavior.

      - org and other packages currently maintain two copies of their
        code, in emacs.git master and elpa.git. Synchronizing them is
        difficult; it is not a simple `git push/pull'. Bundling would
        eliminate the copy in emacs.git.
    
Users do not need to do anything special update a bundled ELPA
package; the normal package.el mechanisms recognize all the versions
properly. 

Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
packages unless they are ":core" packages (ie, they have code in both
emacs.git master and elpa.git); if that functionality is required, the
package must be moved to Emacs.

   Rationale:
   
   - A simple git clone/checkout of emacs.git master creates a
     workspace will all code required to build and run Emacs.

   - Any ELPA package can be changed without affecting Emacs core.

For single file packages, the file can simply be in both elpa.git and
emacs.git master. The package is marked ":core" in the ELPA
"elpa-packages" file. For example, eldoc and flymake are :core
packages. This mechanism is not supported for multi-file packages,
because maintaining sync between the files in emacs.git and elpa.git
is more problematic - with the mechanism described below, you can use
git push/pull to maintain sync.


Every GNU ELPA package has a corresponding branch
`externals/[PKGNAME]` in elpa.git. To bundle a package in Emacs, add
to elpa.git a branch `emacs-NN/[PKGNAME]` which holds the version to
be included in the next Emacs release of version NN. `make dist' would
then grab the bundled packages's code from those branches, and include
them in /usr/share/emacs/NN.MM/elpa/* in the distribution. 

At Emacs startup, the bundled `elpa' directory is added to
package-directory-list, and is treated the same as other entries in
that list.

For Emacs developer convenience, there is a script
`checkout-bundled-elpa' that checks out the correct versions of the
bundled ELPA packages into an elpa worktree. In addition, there is an
emacs function `add-bundled-elpa-packages' that adds those package
directories to `load-path', so they work as if they were simply in the
emacs.git worktree. Both of these features are optional. 

UNRESOLVED ISSUES:

- There are currently (jan 2020) some multi-file packages with code in
  both elpa.git and emacs.git; org, gnus for example. Once the ELPA
  bundling process is worked out, it is expected that those packages
  will use the bundling process instead.

- In the future, for packages like org that used to have code in
  emacs.git master and elpa.git, the emacs.git checkout no longer has
  the package, which is not backward compatible for developers. Emacs
  developers who want those packages now have to install them via
  `list-packages', or check them out from elpa.git.

- Should fetching a worktree from emacs.git also fetch the bundled GNU
  ELPA packages (into the Emacs worktree or somewhere else)?

     - Not fetching ELPA packages into the emacs.git worktree helps
       enforce the rule that Emacs core code may not depend on ELPA
       code.

     - Not fetching ELPA packages means all packages are treated
       equally; Emacs developers do not have to be aware of whether
       some packages are bundled or not. Only the package developer
       has to know.

     - Fetching them makes it clearer what's bundled and what isn't. 
     
     - With the baseline mechanism, developers who want a bundled ELPA
       package checkout that package from elpa.git and either install
       the package from there or include elpa/packages/[PKGNAME] in
       their `load-path', as for any (non-bundled) ELPA package.
     
     - An alternative would be to declare a git submodule for each
       bundled package, with a corresponding branch `elpa/[PKGNAME]`
       (instead of the branches 'emacs-nn/[PKGNAME]'). The submodule
       declaration would place the packages in emacs/elpa (parallel to
       emacs/lisp), so Emacs subdirs.el does not see them. emacs/elpa
       is in package-directory-list by default when running from the
       source directory.

     - If we have submodules, new developers or a fresh checkout must
       use:

          git clone .../emacs.git --recurse-submodules

       for the initial clone. Ideally, the Savannah web page
       https://savannah.gnu.org/git/?group=emacs would say that.

     - Other variations on where to place the submodules in the emacs
       worktree offer more backward compatibility.

       - One issue is whether the autoloads of bundled ELPA packages
         are processed when we dump Emacs (like we do for all the
         packages that are in Emacs core), or whether that's done
         during `package-activate-all` (i.e. between `early-init.el`
         and `init.el`).

         Doing it at dump time gives better startup times, at the cost
         of making it impossible for the end-user to prevent
         activation of a package (they can still undo the activation
         after the fact, of course, but that needs to be done in
         ad-hoc ways).

         Users who care about startup time can do their own dump.


     - Using submodules would help automate cleaning and updating the
       bundled packages; with the baseline mechanism, developers must
       separately manage the elpa worktree.

- Automating compiling C, Ada, Rust or other language code required by
  the package, to executables or modules. This includes defining
  standard places to put executables, modules, and other
  aritifacts. This is more important for bundled packages; users
  expect Emacs to "just work" out of the box.

-- updated 20 Jan 2021

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Wed, 20 Jan 2021 17:41:39 -0800
> 
> Eli has said he wants the bundled packages in the emacs worktree; I'd
> like to explore the specific rationale for that.

I explained that in the discussion we had back in December.  See

  https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00917.html

> Comments?

We had this discussion in December, and I think at least some aspects
were agreed upon; one of them is that using Git submodules is probably
the best alternative.  Why don't we pick up where that discussion has
left off, instead of starting it all anew?

Thanks.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Wed, 20 Jan 2021 17:41:39 -0800
>> 
>> Eli has said he wants the bundled packages in the emacs worktree; I'd
>> like to explore the specific rationale for that.
>
> I explained that in the discussion we had back in December.  See
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00917.html
>
>> Comments?
>
> We had this discussion in December, and I think at least some aspects
> were agreed upon; one of them is that using Git submodules is probably
> the best alternative.  Why don't we pick up where that discussion has
> left off, instead of starting it all anew?

I did not realize that was a decision. I'll change the baseline to use submodules.


-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jan 2021 02:56:35 -0800
> 
> > We had this discussion in December, and I think at least some aspects
> > were agreed upon; one of them is that using Git submodules is probably
> > the best alternative.  Why don't we pick up where that discussion has
> > left off, instead of starting it all anew?
> 
> I did not realize that was a decision.

It wasn't.  But several people agreed and no one objected.  So I think
we have a consensus there.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> It wasn't.  But several people agreed and no one objected.  So I think
> we have a consensus there.

FWIW that was my understanding as well ;-)


        Stefan

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> There will probably be a similar number of posts to emacs.devel about
> "where is org?"; the answer is either "cd elpa; make packages/org" or
> "git submodule update --init".
>
> Comments?


A few questions, rather.

One, both for myself and when building for Windows I use worktrees. So I have

emacs-git/feature/native-comp
emacs-git/master
emacs-git/emacs-27

For personal use I build in source, for Windows building I use these for
out of source builds.

How does that work with ELPA sub-modules? Git worktrees I think hard
links, so the files sizes are much smaller, but with submodules I am
going to get some of ELPA and it's .git multiple times?

My own abortive attempt to achieve ELPA bundled packages
(https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy),
solved one of these two problems -- a single ELPA git repo, but still
duplicated bundled packages in each worktree.



Two, I can live with the submodule update if I have to, but this will
have to be done on each Worktree I guess. Why not add `configure
--with-elpa`
https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy

Three, out of source builds would I presume include the submodule in the
main git tree, not in the out of source location?

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> One, both for myself and when building for Windows I use worktrees. So I have
>
> emacs-git/feature/native-comp
> emacs-git/master
> emacs-git/emacs-27
>
> For personal use I build in source, for Windows building I use these for
> out of source builds.
>
> How does that work with ELPA sub-modules? Git worktrees I think hard
> links, so the files sizes are much smaller, but with submodules I am
> going to get some of ELPA and it's .git multiple times?

Hmm. Perusing https://git-scm.com/docs/gitsubmodules hints that the
submodules could be worktrees if the "Git directory located under the
$GIT_DIR/modules/" can be a worktree link, but there's no option for
'git submodule add' to specify that.

Doing a web search found:

https://stackoverflow.com/questions/31871888/what-goes-wrong-when-using-git-worktree-with-git-submodules

which says using worktrees that contain submodules is not a good idea.

and
https://github.com/git/git/commit/1a248cf21d450eb911d01a89c84412c2da365e66

which is 4 years old, but indicates that mixing worktrees and submodules
was an issue then.

I'll have to test some stuff. If the submodules cannot be worktrees,
then I think we have to abandon this approach. Although maybe using
--depth=1 (or some small number) would be acceptable.

> My own abortive attempt to achieve ELPA bundled packages
> (https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy),
> solved one of these two problems -- a single ELPA git repo, but still
> duplicated bundled packages in each worktree.

I'll look at that.

> Two, I can live with the submodule update if I have to, but this will
> have to be done on each Worktree I guess. Why not add `configure
> --with-elpa`
>
https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy

I'll have to look at that.

But "copy" implies that the git information is lost, so you can't edit
the copy and push the changes. That won't do.

> Three, out of source builds would I presume include the submodule in the
> main git tree, not in the out of source location?

Yes; with submodules, 'make build' (out of source or in source) doesn't use
git to retrieve files, just the filesystem.

With the alternative "only in elpa" approach, 'make build' will have to
know where the local elpa source directory is; it should still not use
git to retrieve files. For building a tarball, step one is get emacs and
elpa checked out to the right versions, then invoke 'make build'.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 22 Jan 2021 18:50:02 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> > How does that work with ELPA sub-modules? Git worktrees I think hard
> > links, so the files sizes are much smaller, but with submodules I am
> > going to get some of ELPA and it's .git multiple times?
> 
> Hmm. Perusing https://git-scm.com/docs/gitsubmodules hints that the
> submodules could be worktrees if the "Git directory located under the
> $GIT_DIR/modules/" can be a worktree link, but there's no option for
> 'git submodule add' to specify that.
> 
> Doing a web search found:
> 
> https://stackoverflow.com/questions/31871888/what-goes-wrong-when-using-git-worktree-with-git-submodules
> 
> which says using worktrees that contain submodules is not a good idea.
> 
> and
> https://github.com/git/git/commit/1a248cf21d450eb911d01a89c84412c2da365e66
> 
> which is 4 years old, but indicates that mixing worktrees and submodules
> was an issue then.
> 
> I'll have to test some stuff. 

My impression is that some of the problems were fixed, but it will be
important to know which versions of Git fixed them, so that those who
use worktrees could decide whether they need to upgrade.

> If the submodules cannot be worktrees, then I think we have to
> abandon this approach.

I'm not sure.  There are alternatives to using worktrees, so the fact
that some of us use worktrees in some workflows should not necessarily
veto the submodule solution to integrating ELPA into Emacs, because
this integration is more important than personal workflows.

But maybe I misunderstand what you mean by "submodule cannot be
worktrees", because AFAIU Phillip raised an issue that is a different
one: whether submodules live well with different branches being
checked out in different worktrees -- there's nothing there to suggest
that submodules themselves should be worktrees.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> But maybe I misunderstand what you mean by "submodule cannot be
> worktrees", 

That was a poor way to phrase what I meant.

I meant that the submodule .git is just a reference to the local
elpa.git, as they are in git worktrees. 

This is in response to Phillip's point that 'git add submodule' copies
the entire ELPA repository.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> If the submodules cannot be worktrees, then I think we have to
>> abandon this approach.
>
> I'm not sure.  There are alternatives to using worktrees, so the fact
> that some of us use worktrees in some workflows should not necessarily
> veto the submodule solution to integrating ELPA into Emacs, because
> this integration is more important than personal workflows.
>
> But maybe I misunderstand what you mean by "submodule cannot be
> worktrees", because AFAIU Phillip raised an issue that is a different
> one: whether submodules live well with different branches being
> checked out in different worktrees -- there's nothing there to suggest
> that submodules themselves should be worktrees.

Just so.

The version of git (2.25.1) on my laptop reports worktree and submodules
as being broken.

And the 2.30 also the same https://git-scm.com/docs/git-worktree

There are ways around all of this, of course, but this use of submodules
will break my workflow at least.

Phil

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> If the submodules cannot be worktrees, then I think we have to
>>> abandon this approach.
>>
>> I'm not sure.  There are alternatives to using worktrees, so the fact
>> that some of us use worktrees in some workflows should not necessarily
>> veto the submodule solution to integrating ELPA into Emacs, because
>> this integration is more important than personal workflows.
>>
>> But maybe I misunderstand what you mean by "submodule cannot be
>> worktrees", because AFAIU Phillip raised an issue that is a different
>> one: whether submodules live well with different branches being
>> checked out in different worktrees -- there's nothing there to suggest
>> that submodules themselves should be worktrees.
>
> Just so.
>
> The version of git (2.25.1) on my laptop reports worktree and submodules
> as being broken.
>
> And the 2.30 also the same https://git-scm.com/docs/git-worktree

Ah. I'm using 2.17 in mingw64, but I haven't updated in a long time.
I'll update.

In particular, my tests so far have been creating new submodules in
worktrees. I have not committed a submodule (ie .gitmodules), then made
a worktree from that.

But we should not rely on something that the git maintainers say doesn't
work.

So we are back to using worktrees from a local elpa.git checkout. Sigh.

-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 25 Jan 2021 11:38:20 -0800
> 
> So we are back to using worktrees from a local elpa.git checkout. Sigh.

I don't see why.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

I don't have a strong opinion over using worktrees or submodules or
what, but I think we should first clarify whether we intend to bundle
*all* GNU ELPA packages or only some of them.

And then whether we want to keep the code (for those GNU ELPA packages
we want to bundle) exclusively in elpa.git.

I think we'll only want to bundle a few specific packages, and I think
it might be simpler to keep a copy of those specific packages in
emacs.git (in the form of branches, of course: we want to be able to
easily sync those copies with the ones in elpa.git).

One advantage is that cloning emacs.git would get us all the data we
need (no need to refer to some other Git repository).
Another is that this will only get the subset of GNU ELPA which we
intend to bundle, so its size won't grow quite like that of `elpa.git`.
Also it naturally gives us separate release branches that can evolve
at a slightly different rate than the code in elpa.git, without having
to touch elpa.git itself.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Andy Moreton]

On Mon 25 Jan 2021, Stefan Monnier wrote:

> I don't have a strong opinion over using worktrees or submodules or
> what, but I think we should first clarify whether we intend to bundle
> *all* GNU ELPA packages or only some of them.
>
> And then whether we want to keep the code (for those GNU ELPA packages
> we want to bundle) exclusively in elpa.git.
>
> I think we'll only want to bundle a few specific packages, and I think
> it might be simpler to keep a copy of those specific packages in
> emacs.git (in the form of branches, of course: we want to be able to
> easily sync those copies with the ones in elpa.git).

I agree that only keeping copies of packages that are actually bundled
with emacs is preferable.

> One advantage is that cloning emacs.git would get us all the data we
> need (no need to refer to some other Git repository).
> Another is that this will only get the subset of GNU ELPA which we
> intend to bundle, so its size won't grow quite like that of `elpa.git`.
> Also it naturally gives us separate release branches that can evolve
> at a slightly different rate than the code in elpa.git, without having
> to touch elpa.git itself.

Another possibility is git subtrees:

https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt

This keeps a copy of the files from a project plus enough metadata to
allow normal commits in the subtree and merging changes back into their
upstream origin.

However a downside I have observed with a repo using git subtree is that
'git log' is not yet clever enough to follow the history of files in the
subtree properly (although 'git blame' works).

    AndyM

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> Another possibility is git subtrees:

This makes it much too hard to sync back&forth.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 25 Jan 2021 15:39:27 -0500
> 
> I think we'll only want to bundle a few specific packages, and I think
> it might be simpler to keep a copy of those specific packages in
> emacs.git

Does this mean you are backing up from the consensus of using
submodules?

> (in the form of branches, of course: we want to be able to
> easily sync those copies with the ones in elpa.git).

What does it mean "to keep a copy of a package in the form of
branches"?  Can you describe the series of Git commands needed to
import such a copy into the Emacs Git repository, and what needs to be
done to routinely update such a copy from ELPA?

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>> I think we'll only want to bundle a few specific packages, and I think
>> it might be simpler to keep a copy of those specific packages in
>> emacs.git
> Does this mean you are backing up from the consensus of using
> submodules?

I don't think it does, no.

For example, ada-mode would be kept inside emacs.git in the branch
`elpa/ada-mode`, and then you could do

   git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it


-- Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Tue, 26 Jan 2021 10:25:20 -0500
> 
> >> I think we'll only want to bundle a few specific packages, and I think
> >> it might be simpler to keep a copy of those specific packages in
> >> emacs.git
> > Does this mean you are backing up from the consensus of using
> > submodules?
> 
> I don't think it does, no.

I guess I'm confused.

> For example, ada-mode would be kept inside emacs.git in the branch
> `elpa/ada-mode`, and then you could do
> 
>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it

AFAIU, "add -b" names a branch in the repository _from_which_ we add
(i.e. the upstream), which in this case is elpa.git.  So I don't think
I understand how your previous sentence:

   ada-mode would be kept inside emacs.git in the branch
   `elpa/ada-mode`

can be reconciled with the Git command you show.  What did I miss?

In any case, if elpa/ada-mode is a branch within emacs.git, then how
can we build some branch of Emacs in a way that includes ada-mode, or
produce a tarball from that Emacs branch?  For example, emacs-NN
release branch will be different from the elpa/ada-mode branch, and we
can never have them together unless we merge them in Git, which is
probably not what we want?

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it
>
> AFAIU, "add -b" names a branch in the repository _from_which_ we add
> (i.e. the upstream), which in this case is elpa.git.

Read again: there's no "elpa.git" in the command above.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Tue, 26 Jan 2021 11:14:26 -0500
> 
> >>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it
> >
> > AFAIU, "add -b" names a branch in the repository _from_which_ we add
> > (i.e. the upstream), which in this case is elpa.git.
> 
> Read again: there's no "elpa.git" in the command above.

Please humor me with a more detailed response.  This is confusing
enough without extra riddles.

If elpa.git is not in the frame, then how did elpa/ada-mode wind up in
emacs.git?

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> If elpa.git is not in the frame, then how did elpa/ada-mode wind up in
> emacs.git?

Someone put it there manually.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Tue, 26 Jan 2021 12:27:58 -0500
> 
> > If elpa.git is not in the frame, then how did elpa/ada-mode wind up in
> > emacs.git?
> 
> Someone put it there manually.

So when the upstream repository in elpa.git gets new commits, that
someone will have to do that again?

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> So when the upstream repository in elpa.git gets new commits, that
> someone will have to do that again?

Yes, we'll need to sync them every once in a while.

The same would be true if we wanted to use the `elpa.git` repository
directly: we'd need to use a separate "ada-mode for Emacs" branch (call
it `emacs/ada-mode`, placed alongside `externals/ada-mode`) in order to
have control over when it gets updated, independently from the main
development of ada-mode.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen_leake@stephe-leake.org,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Tue, 26 Jan 2021 12:57:17 -0500
> 
> > So when the upstream repository in elpa.git gets new commits, that
> > someone will have to do that again?
> 
> Yes, we'll need to sync them every once in a while.

Then I think this is only workable if such updates are done
automatically by some cron job.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: stephen_leake@stephe-leake.org,  phillip.lord@russet.org.uk,
>>   emacs-devel@gnu.org
>> Date: Tue, 26 Jan 2021 12:57:17 -0500
>> 
>> > So when the upstream repository in elpa.git gets new commits, that
>> > someone will have to do that again?
>> 
>> Yes, we'll need to sync them every once in a while.
>
> Then I think this is only workable if such updates are done
> automatically by some cron job.

One way to avoid the need for syncing the emacs release branches is to
keep _all_ branches of bundled packages in emacs.git; when a package
becomes bundled, it moves from elpa.git to emacs.git.

I think the current elpa 'make package/[PKNAME]' could handle that?

-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> One way to avoid the need for syncing the emacs release branches is to
> keep _all_ branches of bundled packages in emacs.git;

We can also have a cron job doing the two-way sync.

> when a package becomes bundled, it moves from elpa.git to emacs.git.

Another option is to use another repository dedicated to those packages
bundled with Emacs.

> I think the current elpa 'make package/[PKNAME]' could handle that?

A small matter of programming,


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 21:05:46 -0500
> 
> Another option is to use another repository dedicated to those packages
> bundled with Emacs.

You mean, to create elpa-bundled.git with those packages?  If so, how
will they be handled from emacs.git, e.g. when building Emacs with all
the bundled packages?

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 16:15:10 -0800
> 
> One way to avoid the need for syncing the emacs release branches is to
> keep _all_ branches of bundled packages in emacs.git; when a package
> becomes bundled, it moves from elpa.git to emacs.git.

What does it mean in practice "to move from elpa.git to emacs.git"?
Does it mean emacs.git becomes the upstream for that package?  If so,
how will the separate development of those package be handled, to
support updates via package.el without waiting for Emacs releases?

(I wish people would explain their proposals in terms of Git commands
needed to support each proposal, instead of talking about general
concepts without the details necessary to understand the essence of
each proposal and alternative.  I believe there's a lot of confusion
and misunderstanding caused by too general description of the
alternatives.)

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Dmitry Gutov]

On 26.01.2021 19:27, Stefan Monnier wrote:
>> If elpa.git is not in the frame, then how did elpa/ada-mode wind up in
>> emacs.git?
> Someone put it there manually.

One (moderate) downside of this is those of us who are already 
subscribed to elpa-diffs will receive the notification emails for the 
same commits again from emacs-diffs.

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>>> If elpa.git is not in the frame, then how did elpa/ada-mode wind up in
>>> emacs.git?
>> Someone put it there manually.
> One (moderate) downside of this is those of us who are already subscribed to
> elpa-diffs will receive the notification emails for the same commits again
> from emacs-diffs.

Good point, I had not considered that.
It's somewhat minor, but it would definitely annoy me.


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it

This seems not to work the way I want it: it re-downloads the whole
emacs.git repository :-(


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it
> This seems not to work the way I want it: it re-downloads the whole
> emacs.git repository :-(

OK, I can make it behave somewhat sanely with:

    git submodule add --reference . -b elpa/ada-mode ./ where/I/want/to/put/it


-- Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it
>
> This seems not to work the way I want it: it re-downloads the whole
> emacs.git repository :-(

But that doesn't actually increase disk space, because git realizes it
alreayd has the files, and uses links. So the end result is what we
want; the local repository for emacs with bundled elpa packages contains
only one copy of elpa.git.

It would be nice if git were even smarter, but we can work with this.

-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

>>>    git submodule add -b elpa/ada-mode .../emacs.git where/I/want/to/put/it
>> This seems not to work the way I want it: it re-downloads the whole
>> emacs.git repository :-(
> But that doesn't actually increase disk space, because git realizes it
> alreayd has the files, and uses links.

In my test (admittedly using another repository since downloading 300MB
is not my idea of fun), that did not happen :-(


        Stefan

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  Eli Zaretskii
>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Mon, 25 Jan 2021 15:39:27 -0500
>> 
>> I think we'll only want to bundle a few specific packages, and I think
>> it might be simpler to keep a copy of those specific packages in
>> emacs.git
>
> Does this mean you are backing up from the consensus of using
> submodules?
>
>> (in the form of branches, of course: we want to be able to
>> easily sync those copies with the ones in elpa.git).
>
> What does it mean "to keep a copy of a package in the form of
> branches"?  Can you describe the series of Git commands needed to
> import such a copy into the Emacs Git repository, and what needs to be
> done to routinely update such a copy from ELPA?

Normally 'git push' in a checkout from a local copy of elpa.git pushes
to savannah elpa.git, but that's just the default.

'git push <remote repository>' pushes to a different remote repository,
which in this case would be savannah emacs.git.

I'll write up the details and post a new baseline. That will take a
while.

-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't have a strong opinion over using worktrees or submodules or
> what, but I think we should first clarify whether we intend to bundle
> *all* GNU ELPA packages or only some of them.

The process should support bundling any GNU ELPA package, but we don't
expect to bundle more than a few.

The process must also support converting a package that is currently in
emacs/lisp/... into a bundled GNU ELPA package.

> And then whether we want to keep the code (for those GNU ELPA packages
> we want to bundle) exclusively in elpa.git.
>
> I think we'll only want to bundle a few specific packages, and I think
> it might be simpler to keep a copy of those specific packages in
> emacs.git (in the form of branches, of course: we want to be able to
> easily sync those copies with the ones in elpa.git).

Yes. For the ELPA developer, it makes very little different whether the
release branch is in elpa.git or emacs.git. For the emacs developer,
emacs.git is better.

-- 
-- Stephe

Re: [SPAM UNSURE] Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 06:42:00 -0800
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > I don't have a strong opinion over using worktrees or submodules or
> > what, but I think we should first clarify whether we intend to bundle
> > *all* GNU ELPA packages or only some of them.
> 
> The process should support bundling any GNU ELPA package, but we don't
> expect to bundle more than a few.

Yes.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> I'll have to test some stuff. If the submodules cannot be worktrees,
> then I think we have to abandon this approach. Although maybe using
> --depth=1 (or some small number) would be acceptable.

I've played with 'git submodules' on Windows mingw64. You can pass a
local ELPA repository to 'git submodules add', but it copies the whole
thing into the emacs repository:

$git submodule add /c/Projects/elpa/.git packages/ada-mode
Cloning into '/c/Projects/emacs/feature/core-elpa-by-copy/elpa/packages/ada-mode'...
done.

(I should have left out the 'packages/')

The copied repostory is in
/c/Projects/emacs/master/.git/worktrees/core-elpa-by-copy/modules/elpa/packages/ada-mode
and is 237 MB (all of ELPA).

There is not a way to specify the branch in the 'git submodule add'
command, but we can do a 'git checkout external/packages/ada-mode' in
the submodule root to get the right files.

However, this will copy all of elpa.git for _each_ bundled package;
clearly not acceptable.


An alternative is to use worktrees from a local elpa.git. This is what
ELPA does now for packages/*. There is elisp code to manage the
worktrees in elpa/admin/elpa-admin.el - see elpaa--worktree-sync for
example. We could reuse that code; we just have to add an arg for the
root directory name (emacs/elpa instead of elpa/packages).

With worktrees, we don't have git commands that operate on all the
bundled packages at once, as 'git submodules ...' does. However, we'll
be using an emacs front-end for edit/push/pull anyway. We may want to
customize our favorite front-end to do them all at once; at least report
changes in those directories so we know to visit them. At the very
least, we'll want a shell script that runs 'git status' in each bundled
package.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 23 Jan 2021 01:27:15 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Stephen Leake <stephen_leake@stephe-leake.org> writes:
> 
> > I'll have to test some stuff. If the submodules cannot be worktrees,
> > then I think we have to abandon this approach. Although maybe using
> > --depth=1 (or some small number) would be acceptable.
> 
> I've played with 'git submodules' on Windows mingw64. You can pass a
> local ELPA repository to 'git submodules add', but it copies the whole
> thing into the emacs repository:

It is quite clear that ELPA will need some changes on its side to
support this integration.  One such change is to have branches that
roughly correspond to Emacs's 'master' and 'release' branches, because
we would want to have only the stable branches of the ELPA packages to
be visible on the Emacs's release branch.

> There is not a way to specify the branch in the 'git submodule add'
> command

AFAIU, you can do that in the .gitmodules file.

> With worktrees, we don't have git commands that operate on all the
> bundled packages at once, as 'git submodules ...' does. However, we'll
> be using an emacs front-end for edit/push/pull anyway.

Not necessarily: we need Git commands to support that in some way in
order to be able to build a release or pretest tarball.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Dmitry Gutov]

On 23.01.2021 11:57, Eli Zaretskii wrote:
>> I've played with 'git submodules' on Windows mingw64. You can pass a
>> local ELPA repository to 'git submodules add', but it copies the whole
>> thing into the emacs repository:
> It is quite clear that ELPA will need some changes on its side to
> support this integration.  One such change is to have branches that
> roughly correspond to Emacs's 'master' and 'release' branches, because
> we would want to have only the stable branches of the ELPA packages to
> be visible on the Emacs's release branch.

Or the package maintainers could bump the submodule refs manually from 
time to time.

>> There is not a way to specify the branch in the 'git submodule add'
>> command
> AFAIU, you can do that in the .gitmodules file.

'man git submodule' mentions the -b option:

add [-b <branch>] [-f|--force] [--name <name>] [--reference 
<repository>] [--depth <depth>] [--] <repository> [<path>]

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>> There is not a way to specify the branch in the 'git submodule add'
>>> command
>> AFAIU, you can do that in the .gitmodules file.
>
> 'man git submodule' mentions the -b option:
>
> add [-b <branch>] [-f|--force] [--name <name>] [--reference
> <repository>] [--depth <depth>] [--] <repository> [<path>]

Ah. Somehow I missed that.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 23 Jan 2021 01:27:15 -0800
>> Cc: emacs-devel <emacs-devel@gnu.org>
>>
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>>
>> > I'll have to test some stuff. If the submodules cannot be worktrees,
>> > then I think we have to abandon this approach. Although maybe using
>> > --depth=1 (or some small number) would be acceptable.
>>
>> I've played with 'git submodules' on Windows mingw64. You can pass a
>> local ELPA repository to 'git submodules add', but it copies the whole
>> thing into the emacs repository:
>
> It is quite clear that ELPA will need some changes on its side to
> support this integration.  One such change is to have branches that
> roughly correspond to Emacs's 'master' and 'release' branches, because
> we would want to have only the stable branches of the ELPA packages to
> be visible on the Emacs's release branch.

Yes.

However, I don't see how that affects my point, which was that 'git add
submodule' appears to copy the entire ELPA repository for each bundled
package.

This is on Windows, using mingw64 git.

However, after doing more investigating, it seems git recogizes that the
submodules are from the same repository, and uses hard links to avoid
file duplication. The Windows File Explorer Properties dialog
double-counts hard links, so it reports a bogus size
(https://docs.microsoft.com/en-us/troubleshoot/windows-client/deployment/address-disk-space-issues-caused-by-winsxs).
mingw64 'du' reports the correct size.

We can live with one extra copy of the ELPA repository.

--
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sun, 24 Jan 2021 09:30:35 -0800
> 
> > It is quite clear that ELPA will need some changes on its side to
> > support this integration.  One such change is to have branches that
> > roughly correspond to Emacs's 'master' and 'release' branches, because
> > we would want to have only the stable branches of the ELPA packages to
> > be visible on the Emacs's release branch.
> 
> Yes.
> 
> However, I don't see how that affects my point, which was that 'git add
> submodule' appears to copy the entire ELPA repository for each bundled
> package.

I think it affects your point, because if ELPA will not present itself
as a single monolith repository, but instead will allow us to
submodule it at package granularity, the problem you mention will go
away.

> This is on Windows, using mingw64 git.
> 
> However, after doing more investigating, it seems git recogizes that the
> submodules are from the same repository, and uses hard links to avoid
> file duplication. The Windows File Explorer Properties dialog
> double-counts hard links, so it reports a bogus size
> (https://docs.microsoft.com/en-us/troubleshoot/windows-client/deployment/address-disk-space-issues-caused-by-winsxs).
> mingw64 'du' reports the correct size.

So should 'ls'.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Sun, 24 Jan 2021 09:30:35 -0800
>> 
>> > It is quite clear that ELPA will need some changes on its side to
>> > support this integration.  One such change is to have branches that
>> > roughly correspond to Emacs's 'master' and 'release' branches, because
>> > we would want to have only the stable branches of the ELPA packages to
>> > be visible on the Emacs's release branch.
>> 
>> Yes.
>> 
>> However, I don't see how that affects my point, which was that 'git add
>> submodule' appears to copy the entire ELPA repository for each bundled
>> package.
>
> I think it affects your point, because if ELPA will not present itself
> as a single monolith repository, but instead will allow us to
> submodule it at package granularity, the problem you mention will go
> away.

Currently, each package in ELPA has a dedicated branch;
external/[PKGNAME]. I think the only way to get better than that is to
split the repository itself into one repository for each package. I
suspect that's possible, but I don't think it's necessary.

I agree that we need to add release branches in elpa.git for bundled
packages.

As I said later, there's only one copy of the ELPA repository, not one
per bundled package.

Do you agree that having one full copy of the ELPA repository in your
local emacs .git repository is acceptable?

For developers who also have elpa.git cloned separately, that's a
duplicate copy.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sun, 24 Jan 2021 12:10:02 -0800
> 
> Do you agree that having one full copy of the ELPA repository in your
> local emacs .git repository is acceptable?

IMO, people who are interested only in packages bundled with Emacs
should not need to have ELPA on their local machines.  Not even one
checkout of ELPA should be needed.  They should just need to clone the
Emacs Git repository (modulo the submodules-related options), and
that's all.  Exactly like they do today: there's no need to clone ELPA
to have a fully functional clone of the Emacs Git repository.

> For developers who also have elpa.git cloned separately, that's a
> duplicate copy.

Which it shouldn't be.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Dmitry Gutov]

On 24.01.2021 22:15, Eli Zaretskii wrote:
> IMO, people who are interested only in packages bundled with Emacs
> should not need to have ELPA on their local machines.  Not even one
> checkout of ELPA should be needed.  They should just need to clone the
> Emacs Git repository (modulo the submodules-related options), and
> that's all.  Exactly like they do today: there's no need to clone ELPA
> to have a fully functional clone of the Emacs Git repository.

Git allows one to check out one branch's contents, without the rest of 
the repository.

So said people wouldn't need to download the whole of ELPA, if you were 
worried about that.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 24.01.2021 22:15, Eli Zaretskii wrote:
>> IMO, people who are interested only in packages bundled with Emacs
>> should not need to have ELPA on their local machines.  Not even one
>> checkout of ELPA should be needed.  They should just need to clone the
>> Emacs Git repository (modulo the submodules-related options), and
>> that's all.  Exactly like they do today: there's no need to clone ELPA
>> to have a fully functional clone of the Emacs Git repository.
>
> Git allows one to check out one branch's contents, without the rest of
> the repository.

Just to be clear, "checkout" means retrieve from a local git repository
into a directory tree; that is always a single branch.

> So said people wouldn't need to download the whole of ELPA, if you
> were worried about that.

"clone" and "add submodule" download from the remote repository.

"add submodule -b <branch>" specifies what branch to checkout, but still
downloads the whole repository.

Ah; "clone ... --single-branch --branch <branch>" allows downloading
only that branch into the local repository. But you can't specify more
than one branch, so that would be inconvenient for more than one bundled
package.

And "git add submodule" does not accept --single-branch.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Dmitry Gutov]

On 26.01.2021 00:00, Stephen Leake wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 24.01.2021 22:15, Eli Zaretskii wrote:
>>> IMO, people who are interested only in packages bundled with Emacs
>>> should not need to have ELPA on their local machines.  Not even one
>>> checkout of ELPA should be needed.  They should just need to clone the
>>> Emacs Git repository (modulo the submodules-related options), and
>>> that's all.  Exactly like they do today: there's no need to clone ELPA
>>> to have a fully functional clone of the Emacs Git repository.
>>
>> Git allows one to check out one branch's contents, without the rest of
>> the repository.
> 
> Just to be clear, "checkout" means retrieve from a local git repository
> into a directory tree; that is always a single branch.

I referred to doing only a partial download, of course.

>> So said people wouldn't need to download the whole of ELPA, if you
>> were worried about that.
> 
> "clone" and "add submodule" download from the remote repository.
> 
> "add submodule -b <branch>" specifies what branch to checkout, but still
> downloads the whole repository.
> 
> Ah; "clone ... --single-branch --branch <branch>" allows downloading
> only that branch into the local repository. But you can't specify more
> than one branch, so that would be inconvenient for more than one bundled
> package.
> 
> And "git add submodule" does not accept --single-branch.

There is a pretty long description here: 
https://stackoverflow.com/questions/2144406/how-to-make-shallow-git-submodules, 
which I didn't have time to read whole.

But from a small experiment, here's what you can do:

git submodule add --depth 1 <repo> <new-module-path>
cd <new-module-path>
git fetch --depth 512 # or however more you need

There are also options like --shallow-since or --deepen.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Mon, 25 Jan 2021 14:00:07 -0800
> 
> Ah; "clone ... --single-branch --branch <branch>" allows downloading
> only that branch into the local repository. But you can't specify more
> than one branch, so that would be inconvenient for more than one bundled
> package.

By "branch" here do you mean an elpa.git branch?  If so, we can have a
single elpa.git branch where each package is represented by a suitable
commit, right?  For example, we can have a "stable" branch in
elpa.git, where the last commit of each package corresponds to the
latest stable version of that package.  Is that possible?

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Andreas Schwab]

On Jan 26 2021, Eli Zaretskii wrote:

> By "branch" here do you mean an elpa.git branch?  If so, we can have a
> single elpa.git branch where each package is represented by a suitable
> commit, right?  For example, we can have a "stable" branch in
> elpa.git, where the last commit of each package corresponds to the
> latest stable version of that package.  Is that possible?

That must be a merge commit.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  phillip.lord@russet.org.uk,
>>   emacs-devel@gnu.org
>> Date: Mon, 25 Jan 2021 14:00:07 -0800
>> 
>> Ah; "clone ... --single-branch --branch <branch>" allows downloading
>> only that branch into the local repository. But you can't specify more
>> than one branch, so that would be inconvenient for more than one bundled
>> package.
>
> By "branch" here do you mean an elpa.git branch?  If so, we can have a
> single elpa.git branch where each package is represented by a suitable
> commit, right?  For example, we can have a "stable" branch in
> elpa.git, where the last commit of each package corresponds to the
> latest stable version of that package.  Is that possible?

Yes, you could create a branch that merges all the bundled packages
(from their separate branches) together,

However, if you then make changes in that merged branch (which we
definitely want to support), it is not simple to propagate them back to
the separate branches; you have to cherry-pick.

Keeping the release branches of bundled ELPA packages in emacs.git is
simpler.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: dgutov@yandex.ru,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 06:37:41 -0800
> 
> > By "branch" here do you mean an elpa.git branch?  If so, we can have a
> > single elpa.git branch where each package is represented by a suitable
> > commit, right?  For example, we can have a "stable" branch in
> > elpa.git, where the last commit of each package corresponds to the
> > latest stable version of that package.  Is that possible?
> 
> Yes, you could create a branch that merges all the bundled packages
> (from their separate branches) together,
> 
> However, if you then make changes in that merged branch (which we
> definitely want to support), it is not simple to propagate them back to
> the separate branches; you have to cherry-pick.

I don't think we want to maintain the bundled packages inside the
Emacs repository.  IOW, the flow of changes should be unidirectional:
from elpa.git to emacs.git, but not in the other direction.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: dgutov@yandex.ru,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Wed, 27 Jan 2021 06:37:41 -0800
>> 
>> > By "branch" here do you mean an elpa.git branch?  If so, we can have a
>> > single elpa.git branch where each package is represented by a suitable
>> > commit, right?  For example, we can have a "stable" branch in
>> > elpa.git, where the last commit of each package corresponds to the
>> > latest stable version of that package.  Is that possible?
>> 
>> Yes, you could create a branch that merges all the bundled packages
>> (from their separate branches) together,
>> 
>> However, if you then make changes in that merged branch (which we
>> definitely want to support), it is not simple to propagate them back to
>> the separate branches; you have to cherry-pick.
>
> I don't think we want to maintain the bundled packages inside the
> Emacs repository.  IOW, the flow of changes should be unidirectional:
> from elpa.git to emacs.git, but not in the other direction.

As a rule, yes. But part of the point of doing this via git is allow
occasional edits in the emacs source tree, and have those edits
propagate back to the upstream.

For emacs developers that do not also have a local elpa repository,
that's the only way to fix a bug in a bundled package for a release.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: dgutov@yandex.ru,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 16:10:07 -0800
> 
> >> However, if you then make changes in that merged branch (which we
> >> definitely want to support), it is not simple to propagate them back to
> >> the separate branches; you have to cherry-pick.
> >
> > I don't think we want to maintain the bundled packages inside the
> > Emacs repository.  IOW, the flow of changes should be unidirectional:
> > from elpa.git to emacs.git, but not in the other direction.
> 
> As a rule, yes. But part of the point of doing this via git is allow
> occasional edits in the emacs source tree, and have those edits
> propagate back to the upstream.

If that can be easily supported without sacrificing anything
important, fine.  But it isn't a requirement, at least not from my
POV.

> For emacs developers that do not also have a local elpa repository,
> that's the only way to fix a bug in a bundled package for a release.

I think any Emacs developer who wants to contribute to ELPA should
have an elpa.git checkout locally.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Sun, 24 Jan 2021 12:10:02 -0800
>> 
>> Do you agree that having one full copy of the ELPA repository in your
>> local emacs .git repository is acceptable?
>
> IMO, people who are interested only in packages bundled with Emacs
> should not need to have ELPA on their local machines.  Not even one
> checkout of ELPA should be needed.  They should just need to clone the
> Emacs Git repository (modulo the submodules-related options), and
> that's all.  Exactly like they do today: there's no need to clone ELPA
> to have a fully functional clone of the Emacs Git repository.

That is exactly what using submodules accomplishes.

But I'm not sure you've answered my question. Let me phrase it
differently:

Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git'
results in disk space usage of around 300 MB in emacs/.git, and does not
populate emacs/elpa.

Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git
--recurse-submodules' results in disk space usage of around 500 MB in
emacs/.git, and populates emacs/elpa.

If we implemented the suggestion to split the ELPA repository into one
repository per bundled package, that extra 200 MB would be smaller.

Is 200 MB acceptable?

>> For developers who also have elpa.git cloned separately, that's a
>> duplicate copy.
>
> Which it shouldn't be.

I agree it would be nice, but that's up to the git maintainers. Or we
could look for another CM tool ...

I can see some reasons for the current git design; _all_ of the info needed
to update the code for project foo is in foo/.git. Worktrees stretch
that; allowing submodules to be worktree-like references to yet another
repository somewhere else would probably break many things in git.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Mon, 25 Jan 2021 11:30:47 -0800
> 
> Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git'
> results in disk space usage of around 300 MB in emacs/.git, and does not
> populate emacs/elpa.

Where did emacs/elpa come from? I don't think we ever discussed such a
beast, and I don't understand what it is and what it will contain.

> Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git
> --recurse-submodules' results in disk space usage of around 500 MB in
> emacs/.git, and populates emacs/elpa.
> 
> If we implemented the suggestion to split the ELPA repository into one
> repository per bundled package, that extra 200 MB would be smaller.
> 
> Is 200 MB acceptable?

That's a the question I can answer, because I have no clear idea what
is included in those 200MB, and in particular how much of that is
ballast we don't need for the purposes of maintaining Emacs and
building its development versions.  Maybe if you explained why
splitting ELPA into several repositories would save some of that disk
space, I could then try answering the question.

> I can see some reasons for the current git design; _all_ of the info needed
> to update the code for project foo is in foo/.git. Worktrees stretch
> that; allowing submodules to be worktree-like references to yet another
> repository somewhere else would probably break many things in git.

I think we should first find a way to have a single worktree with all
the bundled packages that come from ELPA.  How to have several
worktrees from that is something we should consider later.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Mon, 25 Jan 2021 11:30:47 -0800
>> 
>> Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git'
>> results in disk space usage of around 300 MB in emacs/.git, and does not
>> populate emacs/elpa.
>
> Where did emacs/elpa come from? I don't think we ever discussed such a
> beast, and I don't understand what it is and what it will contain.

emacs/elpa is the directory in the emacs directory tree where bundled
elpa packages are kept.

I agree this is confusing; I need to write up a new baseline description.

>> Doing 'git clone <membername>@git.savannah.gnu.org:/srv/git/emacs.git
>> --recurse-submodules' results in disk space usage of around 500 MB in
>> emacs/.git, and populates emacs/elpa.
>> 
>> If we implemented the suggestion to split the ELPA repository into one
>> repository per bundled package, that extra 200 MB would be smaller.
>> 
>> Is 200 MB acceptable?
>
> That's a the question I can answer, because I have no clear idea what
> is included in those 200MB, and in particular how much of that is
> ballast we don't need for the purposes of maintaining Emacs and
> building its development versions.  

Some of it is ballast, yes; all of the ELPA packages that are not
bundled with emacs.

> Maybe if you explained why splitting ELPA into several repositories
> would save some of that disk space, I could then try answering the
> question.

Gnu ELPA is currently a single repository with lots of branches, one
branch per package.

git submodule downloads the entire repository, even if you only want one
branch.

So one way to eliminate the ballast is to split Gnu ELPA into one
repository per package; then git submodule will only download the
repository for that package.

Stefan's suggestion eliminates the ballast in a different way; by keeping
the release branches of bundled packages in emacs.git, there is no
ballast in the local emacs repository.

>> I can see some reasons for the current git design; _all_ of the info needed
>> to update the code for project foo is in foo/.git. Worktrees stretch
>> that; allowing submodules to be worktree-like references to yet another
>> repository somewhere else would probably break many things in git.
>
> I think we should first find a way to have a single worktree with all
> the bundled packages that come from ELPA.  How to have several
> worktrees from that is something we should consider later.

I don't think we can leave it until later; if we choose a design that
explicitly prohibits worktrees, there is nothing that can be done later.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> So one way to eliminate the ballast is to split Gnu ELPA into one
> repository per package; then git submodule will only download the
> repository for that package.

There's a strong technical reason why `elpa.git` and `nongnu.git` are
single repositories: Savannah does not let us create new repositories
nor change their default branch, nor setup/change the commit-diffs for
them.  Every time such a thing needs to happen, we have to ask
a sysadmin to do it for us.

So for the foreseeable future, "one repository per package" is
a complete non-starter.  [ The "all in one repository" approach also
has other advantages, but there's no point weighing their importance
since there's currently no credible alternative anyway.  ]


        Stefan

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Andy Moreton]

On Wed 27 Jan 2021, Stefan Monnier wrote:

>> So one way to eliminate the ballast is to split Gnu ELPA into one
>> repository per package; then git submodule will only download the
>> repository for that package.
>
> There's a strong technical reason why `elpa.git` and `nongnu.git` are
> single repositories: Savannah does not let us create new repositories
> nor change their default branch, nor setup/change the commit-diffs for
> them.  Every time such a thing needs to happen, we have to ask
> a sysadmin to do it for us.

Using technical means to work around a social/political problem usually
results in later difficulties.

    AndyM

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Wed, 27 Jan 2021 06:31:13 -0800
> 
> > Maybe if you explained why splitting ELPA into several repositories
> > would save some of that disk space, I could then try answering the
> > question.
> 
> Gnu ELPA is currently a single repository with lots of branches, one
> branch per package.
> 
> git submodule downloads the entire repository, even if you only want one
> branch.
> 
> So one way to eliminate the ballast is to split Gnu ELPA into one
> repository per package; then git submodule will only download the
> repository for that package.

Since Stefan says this is a non-starter, we can forget about this
alternative.

> Stefan's suggestion eliminates the ballast in a different way; by keeping
> the release branches of bundled packages in emacs.git, there is no
> ballast in the local emacs repository.

This could work, but we will have to arrange for some periodic job to
update the copies of the bundled packages frequently enough.

> >> I can see some reasons for the current git design; _all_ of the info needed
> >> to update the code for project foo is in foo/.git. Worktrees stretch
> >> that; allowing submodules to be worktree-like references to yet another
> >> repository somewhere else would probably break many things in git.
> >
> > I think we should first find a way to have a single worktree with all
> > the bundled packages that come from ELPA.  How to have several
> > worktrees from that is something we should consider later.
> 
> I don't think we can leave it until later; if we choose a design that
> explicitly prohibits worktrees, there is nothing that can be done later.

It will mean that people who use worktrees will have to find some way
of doing that, or give up worktrees.  But IMO the convenience of
bundling a package and handling such a bundled package trumps the
convenience of people who use worktrees a lot.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> I can see some reasons for the current git design; _all_ of the info needed
>> >> to update the code for project foo is in foo/.git. Worktrees stretch
>> >> that; allowing submodules to be worktree-like references to yet another
>> >> repository somewhere else would probably break many things in git.
>> >
>> > I think we should first find a way to have a single worktree with all
>> > the bundled packages that come from ELPA.  How to have several
>> > worktrees from that is something we should consider later.
>> 
>> I don't think we can leave it until later; if we choose a design that
>> explicitly prohibits worktrees, there is nothing that can be done later.
>
> It will mean that people who use worktrees will have to find some way
> of doing that, or give up worktrees.  But IMO the convenience of
> bundling a package and handling such a bundled package trumps the
> convenience of people who use worktrees a lot.

To give you an idea of why I use worktrees, it is because Emacs does not
really do an out of source build. So, on the windows build machine I
currently have:

emacs-build/master
emacs-build/emacs-27
emacs-build/emacs-27.1.91
emacs-build/feature/native-comp

For emacs-27 I only ever do full builds; creating a new worktree means
that I start from clean, but is clearly not the only way to achieve
this.

For emacs-28, though, when I release snapshots, I do not (always) do
clean builds. I use the .elc files from the previous build. The same
will be true for native-comp once I have that working enough to build
snapshots.

It's not a disaster, but I will have to switch the windows packaging
scripts away from this schema if worktrees do not work; I guess I will
use mulitple clones instead. This may mean I run out of disk space on my
windows build machine, though which only has a tiny disc, I don't
know. It depends on how well git works, and how clever Windows file
system compression is.

I do the same on my working machines, so I can keep multiple branches
open. So I can test master, but jump back to a release branch if it
breaks and I am busy. I guess I will use multiple clones here again.

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Robert Pluim]

>>>>> On Fri, 29 Jan 2021 17:47:39 +0000, Phillip Lord <phillip.lord@russet.org.uk> said:
    Phillip> It's not a disaster, but I will have to switch the windows packaging
    Phillip> scripts away from this schema if worktrees do not work; I guess I will
    Phillip> use mulitple clones instead. This may mean I run out of disk space on my
    Phillip> windows build machine, though which only has a tiny disc, I don't
    Phillip> know. It depends on how well git works, and how clever Windows file
    Phillip> system compression is.

Does 'git clone --reference' not work on MS-Windows?

Robert

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stephen Leake
>  <stephen_leake@stephe-leake.org>,  emacs-devel@gnu.org
> Date: Mon, 01 Feb 2021 09:45:15 +0100
> 
> Does 'git clone --reference' not work on MS-Windows?

It's in the docs, so I don't see why it wouldn't work.  But I never
tried it.

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> which is 4 years old, but indicates that mixing worktrees and submodules
> was an issue then.

It's marked as a bug on the git worktrees manual (right at the end).

>
> I'll have to test some stuff. If the submodules cannot be worktrees,
> then I think we have to abandon this approach. Although maybe using
> --depth=1 (or some small number) would be acceptable.
>
>> My own abortive attempt to achieve ELPA bundled packages
>> (https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy),
>> solved one of these two problems -- a single ELPA git repo, but still
>> duplicated bundled packages in each worktree.
>
> I'll look at that.

To caveat, I wouldn't necessarily put this forward as working
solution. It felt kludgey then and it feels so now. I stopped working on
it because of time, though not because it didn't work.


>> Two, I can live with the submodule update if I have to, but this will
>> have to be done on each Worktree I guess. Why not add `configure
>> --with-elpa`
>>
> https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/core-elpa-by-copy
>
> I'll have to look at that.
>
> But "copy" implies that the git information is lost, so you can't edit
> the copy and push the changes. That won't do.

Yes, that's the flaw, and it's a fairly big one. If you wanted to edit,
and commit that would require checking out from the ELPA repo, updating
the pointer on the Emacs tree and then checking back in again. This
sounds clunky, although in practice, the number of times you need to
edit a package in tree would be small, I think. I mean, it's the
workflow you have at the moment for ELPA packages right? If you have a
test Emacs and install an ELPA package, you don't want to edit the
source of that ELPA package.

I solve this problem for my own packages by not installing them from
ELPA, but from their git repos. This means normal users and me as
developer have a different set up.

straight.el solves this problem in a more principled way; the package is
the git repo, but then the package manager is completely dependent on
git.

>> Three, out of source builds would I presume include the submodule in the
>> main git tree, not in the out of source location?
>
> Yes; with submodules, 'make build' (out of source or in source) doesn't use
> git to retrieve files, just the filesystem.

Yes. My system uses git archive to retrieve files during the build. I
seem to remember putting an option to have make clone ELPA also,
although that was configurable for exactly the reason we are discussing;
I wanted one copy of ELPA not lots. But, a full git repo build of Emacs
would require git on the build machine. I figured that wasn't a problem
because, well, if you have a git checkout on that machine, then you
probably have git.


> With the alternative "only in elpa" approach, 'make build' will have to
> know where the local elpa source directory is; it should still not use
> git to retrieve files. For building a tarball, step one is get emacs and
> elpa checked out to the right versions, then invoke 'make build'.

Yes. My version doesn't use a check out of ELPA, so it can take source
for different packages from different versions of the same branch.

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> But "copy" implies that the git information is lost, so you can't edit
>> the copy and push the changes. That won't do.
>
> Yes, that's the flaw, and it's a fairly big one. If you wanted to edit,
> and commit that would require checking out from the ELPA repo, updating
> the pointer on the Emacs tree and then checking back in again. 

> I solve this problem for my own packages by not installing them from
> ELPA, but from their git repos. This means normal users and me as
> developer have a different set up.

Yes; package developers edit package code in an ELPA checkout; typically
they will also run the code from there as well for testing, by putting
that directory in load-path.

But with bundled packages as submodules, they could switch to editing
them in the emacs workspace instead.

The main point is that other emacs developers can also edit the package
files, to fix bugs or make changes consistent with some core emacs change.

> straight.el solves this problem in a more principled way; the package is
> the git repo, but then the package manager is completely dependent on
> git.

From https://github.com/raxod502/straight.el. That sounds useful.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> I solve this problem for my own packages by not installing them from
>> ELPA, but from their git repos. This means normal users and me as
>> developer have a different set up.
>
> Yes; package developers edit package code in an ELPA checkout; typically
> they will also run the code from there as well for testing, by putting
> that directory in load-path.
>
> But with bundled packages as submodules, they could switch to editing
> them in the emacs workspace instead.
>
> The main point is that other emacs developers can also edit the package
> files, to fix bugs or make changes consistent with some core emacs change.

Just so, and this would be a substantial win. It is something that we
get at the moment by having a monolithic repo with all its
packages. Having said that, most Emacs packages do not get this. There
are 3000 packages on MELPA and they just update themselves as new Emacs
versions come out.

An alternative solution would be strength and automate the links between
a package in a repo and package.el. So, package.el would be able to main
a local package that was stored in a git repo, would like to the source,
but would do all the autoload generation, putting into the path and so
forth.

>> straight.el solves this problem in a more principled way; the package is
>> the git repo, but then the package manager is completely dependent on
>> git.
>
> From https://github.com/raxod502/straight.el. That sounds useful.

The nice thing about is that it unifies package management. It worries
me slightly that Emacs will be saying "Emacs uses package.el to manage
packages and their dependencies". But, then when looking at bundled
packages, we are saying "Emacs uses submodules". Two different systems
to achieve the same thing.

But, moving to the straight.el model would be a big change, of course
and would require git to be available for Emacs to do any package
management. This isn't true for the Windows bundle, for instance.

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stefan Monnier]

> An alternative solution would be strength and automate the links between
> a package in a repo and package.el. So, package.el would be able to main
> a local package that was stored in a git repo, would like to the source,
> but would do all the autoload generation, putting into the path and so
> forth.

The `elpa-admin.el` file in the `elpa-admin` branch of `elpa.git`, which
is used to build the GNU ELPA and NonGNU ELPA tarballs also has support
for using those packages "in place":

    git clone .../elpa.git; cd elpa; make   # Generic setup, done once.
    make packages/FOO                       # Checkout package "FOO"
    make packages/BAR                       # Checkout package "BAR"
    make

gives you a directory `.../elpa/packages` which can be added to
`packages-directory-list` so that the packages are installed (from
`package.el`s point of view) straight from their Git worktree.


        Stefan

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> The nice thing about is that it unifies package management. It worries
> me slightly that Emacs will be saying "Emacs uses package.el to manage
> packages and their dependencies". But, then when looking at bundled
> packages, we are saying "Emacs uses submodules". Two different systems
> to achieve the same thing.

It's not really "the same thing"; in the case of bundled packages, we
also need to support testing emacs releases with the bundled packages
via git branches (and tarballs), making the release tarball,
building from the unpacked tarball, and running from the unpacked and
built tarball.

Using submodules makes testing via git branches easy. However, as you
have pointed out, using submodules breaks git worktrees. So we have to
choose. I'll add some rationale for both choices in my next writeup.

Making the tarball, and building and running from the unpacked tarball
require that the bundled ELPA packages appear in the emacs source
directory tree. It's better for emacs developers who do both if that is
consistent with an emacs directory tree created via checkout from git;
the bundled packages should be in the same place.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> One, both for myself and when building for Windows I use worktrees. So I have
>
> emacs-git/feature/native-comp
> emacs-git/master
> emacs-git/emacs-27
>
> How does that work with ELPA sub-modules? Git worktrees I think hard
> links, so the files sizes are much smaller, but with submodules I am
> going to get some of ELPA and it's .git multiple times?

I've now tested this; I added an elpa submodule in a straight emacs
clone checkout, and in a worktree created from that clone. There is only
one copy of ELPA in the emacs clone repository.

> Two, I can live with the submodule update if I have to, but this will
> have to be done on each Worktree I guess. 

You have to update each worktree now; updating the submodules should be
done by the same command that updates the worktree. If there is no git
command that does that directly, we need to write a shell script; that's
certainly possible.

Changing each submodule to the next release branch will be done in
emacs/master/.gitmodules by the package developer; I assume that will
propagate everywhere.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball

[Dmitry Gutov]

On 21.01.2021 03:41, Stephen Leake wrote:
> I suggest we
> standardize on an emacs elisp project that allows using C-x p f
> (project-find-file) to complete on files in load-path for elisp files;
> that removes the issue of remembering where things are.

While visiting an elisp file, try 'C-x p F' (project-or-external-find-file).