policy discussion on bundling ELPA packages in the emacs tarball - take 2

policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Attached is an updated version of emacs/admin/notes/elpa, describing a
proposal for bundling ELPA packages in the emacs distribution.

I believe I have included all the recent discussions.

I think I figured out a way to resolve the submodles vs worktrees
dilemma; each emacs developer can choose. If they don't execute any
submodule commands, they can use worktrees.

If we can reach consensus on this, the next step would be to create a
feature branch and edit the code to implement this, then do a practice
release with that branch.

-- 
-- Stephe

[-- Attachment #2: elpa --]
[-- Type: application/octet-stream, Size: 8372 bytes --]

NOTES ON THE EMACS PACKAGE ARCHIVE

The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git
repository named "elpa", hosted on Savannah.  To check it out:

  git clone git://git.sv.gnu.org/emacs/elpa
  cd elpa
  make setup

That leaves the elpa/packages directory empty; you must check out the
ones you want.

If you wish to check out all the packages into the packages directory,
you can run the command:

   make worktrees

You can check out a specific package <pkgname> into the packages
directory with:

   make packages/<pkgname>


Changes to this repository propagate to elpa.gnu.org via a
"deployment" script run daily.  This script generates the content
visible at https://elpa.gnu.org/packages.

A new package is released as soon as the "version number" of that
package is changed.  So you can use 'elpa' to work on a package
without fear of releasing those changes prematurely.  And once the
code is ready, just bump the version number to make a new release of
the package.

It is easy to use the elpa branch to deploy a "local" copy of the
package archive.  For details, see the README file in the elpa branch.


ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL
-- updated 29 Jan 2021

Why Bundle?

    - To provide backward compatibility for users while allowing more
      flexible releases, and more convenient maintenance.

      - ada-mode was originally included in the emacs tarball, but was
        removed when the ELPA version was capable enough. Some users
        complained; bundling ada-mode would (almost) restore the
        original behavior.

      - org and other packages currently maintain two copies of their
        code, in emacs.git master and elpa.git. Synchronizing them is
        difficult; it is not a simple `git push/pull'. Bundling would
        allow using 'git push/pull' to synchonize between emacs.git
        and elpa.git.

   - To provide a better "out-of-the-box" Emacs experience for new
     Emacs users, by bundling new useful packages.
   
After the initial install of Emacs, users do not need to do anything
special to update a bundled ELPA package; the normal package.el
mechanisms recognize all the versions properly.

Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
packages unless they are ":core" packages; if that functionality is
required, the package must be moved to Emacs.
   
   - A simple git clone/checkout of emacs.git master creates a
     workspace will all code required to build and run Emacs.

   - Any ELPA package can be changed without affecting Emacs core.

For single file packages, the file can simply be in both elpa.git and
emacs.git master. The package is marked ":core" in the ELPA
"elpa-packages" file. For example, eldoc and flymake are :core
packages. This mechanism is not supported for multi-file packages,
because maintaining sync between the files in emacs.git and elpa.git
is more problematic.

To bundle a multi-file package in Emacs, add to elpa.git a branch
`emacs-NN/[PKGNAME]` which holds the version to be included in the
next Emacs release of version NN:

    git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]

Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
elpa.git.

Also push the release branch to emacs.git:

    git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME]

    This allows emacs developers to checkout the bundled package
    without requiring a separate clone of elpa.git.
    
In emacs/master, add a git submodule for that branch:

    git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]

Emacs developers should add '--recurse-submodules' to the 'git clone'
command when cloning Emacs; that will checkout the submodules with the
rest of Emacs. To checkout the submodules in an existing Emacs
directory tree:

    git submodule update --reference . --init

  However, the git worktree manual (https://git-scm.com/docs/git-worktree), in
  the Bugs section at the end, says:

    Multiple checkout in general is still experimental, and the
    support for submodules is incomplete. It is NOT recommended to
    make multiple checkouts of a superproject.

  git worktrees allow keeping several checked out directory trees
  while having only one local copy of the emacs.git repository. Some
  emacs developers use worktrees, for example to keep emacs master,
  emacs-27, and some feature branches checked out and built.

  Emacs developers that wish to use worktrees must not execute any
  submodule commands; do not include --recurse-submodules' on 'git
  clone', do not execute 'git submodule update'. Then the local git
  repository is not a "superproject", and worktrees can be safely
  used.

  We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
  worktree' to checkout each bundled ELPA branch into emac/elpa,
  instead of 'git submodule update'.

There are currently (jan 2020) some multi-file packages with code in
both elpa.git and emacs.git; org, gnus for example. To convert those
packages into bundled packages, they are deleted from emacs.git, and
then follow the process above. The net result in the emacs directory
tree is that the code moves from emacs/lisp/[PKGNAME] to
emacs/elpa/[PKGNAME].

'make' treats emacs/elpa the same as it treats emacs/lisp and
emacs/doc; .el files are byte compiled, .texi files are compiled to
info. Note that documentation source .texi files for bundled packages
reside in emacs/elpa/[PKGNAME], not in emacs/doc.

'make dist' treats emacs/elpa the same as it treats emacs/lisp; the
files are included in the distribution tarball.

'make install' places the installed emacs/elpa directory next to the
installed emacs/lisp directory.

At Emacs startup, the (installed or source) `emacs/elpa' directory is
added to `package-directory-list', and is treated the same as other
entries in that list. Note that this is different from files in
emacs/lisp; those are added to `load-path'. Using
`package-directory-list' allows users to disable bundled packages via
`package-load-list'.

Emacs developers use the various 'git submodule' commands to maintain
packages in emacs/elpa. It is expected that very few edits will be
made there; the primary development of ELPA packages is done in a
checkout from elpa.git. Changes can be made in emacs/elpa to fix a bug
for a release, or to change code to be consistent with some emacs core
change. Such changes can be pushed to elpa.git and merged into the
package development branch.

UNRESOLVED ISSUES/TODO:

- One issue is whether the autoloads of bundled ELPA packages are
  processed when we dump Emacs (like we do for all the packages that
  are in Emacs core), or whether that's done during
  `package-activate-all`. The baseline design does it during
  `package-activate-all'.

  Doing it at dump time gives better startup times, at the cost of
  making it impossible for the end-user to prevent activation of a
  package.

  Users who care about startup time can do their own dump.

- emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME]

  The baseline design keeps all bundled ELPA packages in
  emacs/elpa/[PKNAME], both in the source and installed directory
  trees. This makes it very easy to distinguish an emacs directory
  tree with bundled packages from one without.

  An alternative is to keep the bundled ELPA packages in
  emacs/lisp/[PKGNAME]; that minimizes the change when a package that
  is current in emacs.git converts to a bundled package. This would
  also mean that bundled packages are put in `load-path' at Emacs
  startup, not in `package-directory-list'.

- Does vc support submodules and nested worktrees?

- Define a useful emacs lisp project (or something simpler),
  So `C-x p f' searches `load-path' via `project-find-file'

  That reduces the need to remember what packages are bundled, and
  where you have them checked out.

- Update the mechanism that populates
  https://www.gnu.org/software/emacs/manual/html_mono/*

  For example, ada-mode.html there still reflects the ancient version
  that was in emacs core.

- Automating compiling C, Ada, Rust or other language code required by
  the package, to executables or modules. This includes defining
  standard places to put executables, modules, and other
  aritifacts. This is more important for bundled packages; users
  expect Emacs to "just work" out of the box.

Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Attached is an updated version of emacs/admin/notes/elpa, describing a
> proposal for bundling ELPA packages in the emacs distribution.
>
> I believe I have included all the recent discussions.
>
> I think I figured out a way to resolve the submodles vs worktrees
> dilemma; each emacs developer can choose. If they don't execute any
> submodule commands, they can use worktrees.

That will be a significant help, actually. I presume that any build
would be missing bundled packages? For me, this would not be a hassle
because I'll just set it up so they install from ELPA on first use.

It would still break my windows packaging scripts which use worktrees,
(or as it stands the windows distribution will miss ada-mode) but I
guess that only affects me. We are some way of Emacs-28, though, so now
is a good time to break things. I would hope that big packages (org!!!)
is not going to move for a while.

Also, some research suggests that the git developers are slowing
working out how to fix the module/worktree breakage, so who knows they
might have fixed it by then.


> If we can reach consensus on this, the next step would be to create a
> feature branch and edit the code to implement this, then do a practice
> release with that branch.

It's fine with me!

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> Attached is an updated version of emacs/admin/notes/elpa, describing a
>> proposal for bundling ELPA packages in the emacs distribution.
>>
>> I believe I have included all the recent discussions.
>>
>> I think I figured out a way to resolve the submodles vs worktrees
>> dilemma; each emacs developer can choose. If they don't execute any
>> submodule commands, they can use worktrees.
>
> That will be a significant help, actually. I presume that any build
> would be missing bundled packages? 

I don't follow.

If you mean "a build from an emacs directory tree", then it will have
whatever the developer checked out into that tree.

If they did "git clone <emacs> --recurse-submodules", then the build
will have the bundled packages.

If they left --recurse-submodules off the clone, but later did either:

    git submodule update --reference . --init

or:

    ./checkout_git_elpa_worktrees.sh

then the build will have the bundled packages.

The last option is indented for your Windows packaging workflow; it uses
git worktrees instead of submodules.

If they do neither of these options, the build will not have the bundled
packages.

> For me, this would not be a hassle because I'll just set it up so they
> install from ELPA on first use.

This is with your emacs user hat on, I guess. Yes, using a build from an
emacs directory tree without bundled packages would require getting
those packages via M-x list-packages (or equivalent).

> It would still break my windows packaging scripts which use worktrees,

It should not, as described above.

-- 
-- Stephe

Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>>
>> That will be a significant help, actually. I presume that any build
>> would be missing bundled packages? 
>
> I don't follow.
>
> If you mean "a build from an emacs directory tree", then it will have
> whatever the developer checked out into that tree.
>
>
>     ./checkout_git_elpa_worktrees.sh
>
> then the build will have the bundled packages.
>
> The last option is indented for your Windows packaging workflow; it uses
> git worktrees instead of submodules.

Apologies that was a very badly worded question, but yes, you have
answered it. I had missed the "checkout_git_elpa_worktrees.sh" option
apologies. If that works it would require only a small update to my
scripts so sounds excellent.

Are you going to get this onto a branch first; I'll be happy to test the
Windows packaging when you have.


> If they do neither of these options, the build will not have the bundled
> packages.
>
>> For me, this would not be a hassle because I'll just set it up so they
>> install from ELPA on first use.
>
> This is with your emacs user hat on, I guess. Yes, using a build from an
> emacs directory tree without bundled packages would require getting
> those packages via M-x list-packages (or equivalent).

Yes, that's as a normal user.

>> It would still break my windows packaging scripts which use worktrees,
>
> It should not, as described above.

Thank your taking the time and effort to think about this, it's really
greatly appreciated!

Phil

Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 29 Jan 2021 11:32:34 -0800
> 
> Attached is an updated version of emacs/admin/notes/elpa, describing a
> proposal for bundling ELPA packages in the emacs distribution.
> 
> I believe I have included all the recent discussions.

Thanks.  The text confused me a couple of times, so I think it "needs
some work"; see the questions and comments below.

> I think I figured out a way to resolve the submodles vs worktrees
> dilemma; each emacs developer can choose. If they don't execute any
> submodule commands, they can use worktrees.

I think the consequences of each of the two choices shall be spelled
out, because at least to me they are not clear.  What are the
advantages and disadvantages of each choice?

> If we can reach consensus on this, the next step would be to create a
> feature branch and edit the code to implement this, then do a practice
> release with that branch.

I'm not sure we need a public branch for that.  What would be the
purpose of such a branch?

> After the initial install of Emacs, users do not need to do anything
> special to update a bundled ELPA package; the normal package.el
> mechanisms recognize all the versions properly.

This should tell more about "not having to do anything special": I'm
guessing (or rather hoping) you allude here to upgrading/downgrading
the bundled packages from ELPA, *not* to how the bundled packages are
used without upgrading.  IOW, to just use the version shipped with the
Emacs release tarball, users don't need to use package.el, nor will
they need to add any package-related code into their init files or
create early-init.el if they didn't have it before.

That is, what does this mean from the POV of a user who didn't until
now use package.el at all, but just used what came with the release
tarball?

> Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
> packages unless they are ":core" packages; if that functionality is
> required, the package must be moved to Emacs.

Another unclear part.  The following questions arise, and I don't see
any answers to them:

  . how to know whether a package is ":core", and how to indicate that
    it's ":core"? is the elpa-packages file the only place?

  . "if that functionality is required, the package must be moved to
    Emacs" -- what functionality is alluded to here, and what does it
    mean "moved to Emacs"? does any part of the following description
    apply to such "moved to Emacs" packages?

>    - A simple git clone/checkout of emacs.git master creates a
>      workspace will all code required to build and run Emacs.

What does this mean in practical terms?  Is this any change from the
current situation, and if so, how?  And what is "workspace" in this
context?

>    - Any ELPA package can be changed without affecting Emacs core.

How is this related?  What changes to ELPA packages and in what
location (on Savannah, on my local clone of elpa.git, in my clone of
emacs.git, something else?) will not "affect Emacs core", and what
does "affect" mean in this context?

> For single file packages, the file can simply be in both elpa.git and
> emacs.git master. The package is marked ":core" in the ELPA
> "elpa-packages" file. For example, eldoc and flymake are :core
> packages.

So to know whether a package is ":core", one must look in the local
clone of elpa.git?  What if there's no local clone of elpa.git: does
it mean Emacs (and package.el specifically) won't know whether a
package is ":core"? does that matter?

> This mechanism is not supported for multi-file packages, because
> maintaining sync between the files in emacs.git and elpa.git is more
> problematic.
> 
> To bundle a multi-file package in Emacs, add to elpa.git a branch
> `emacs-NN/[PKGNAME]` which holds the version to be included in the
> next Emacs release of version NN:
> 
>     git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]
> 
> Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
> elpa.git.

My Git documentation doesn't mention the -c switch of "git branch".
What did I miss?

More importantly, why do we need more than one protocol for bundling a
package?  Wouldn't it be simpler to have just one?

> Also push the release branch to emacs.git:
> 
>     git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME]
> 
>     This allows emacs developers to checkout the bundled package
>     without requiring a separate clone of elpa.git.
>     
> In emacs/master, add a git submodule for that branch:
> 
>     git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]

You say "emacs/master", and I'm guessing this means the master branch
of emacs.git (but please be explicit about this, and don't let us
guess).  If that is correct, then the master branch is not the most
important place to do this: the most important place is the emacs-NN
release branch, because that's where we care the most about having the
stable version of each package.

And finally, what does the above "git push" command do?  You are
pushing from one Git repository (elpa.git) to another (emacs.git),
right?  What will that do in emacs.git, and what would be the effect
on users who will next "git pull" from emacs.git?

> Emacs developers should add '--recurse-submodules' to the 'git clone'
> command when cloning Emacs; that will checkout the submodules with the
> rest of Emacs. To checkout the submodules in an existing Emacs
> directory tree:
> 
>     git submodule update --reference . --init

To which branch is this applicable?  E.g., I have a separate clone
where I work on the current release branch (emacs-27 as of this
writing); do I need to do the above in that clone as well, and should
the command change to tell Git that I want the versions of the bundled
packages that are appropriate for the branch I'm tracking there?  What
if I switch the branch inside a repository -- do I need to do anything
special for the submodules?

>   We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
>   worktree' to checkout each bundled ELPA branch into emac/elpa,
>   instead of 'git submodule update'.

What does this script do? does it allow developers to use "git
worktrees" safely without losing the access to bundled packages?  IOW,
more details about the script need to be spelled out.

> There are currently (jan 2020) some multi-file packages with code in
> both elpa.git and emacs.git; org, gnus for example. To convert those
> packages into bundled packages, they are deleted from emacs.git, and
> then follow the process above. The net result in the emacs directory
> tree is that the code moves from emacs/lisp/[PKGNAME] to
> emacs/elpa/[PKGNAME].

How will this move affect Git history and the related "git log" and
"git blame" commands? what about "git bisect" across the move?

> At Emacs startup, the (installed or source) `emacs/elpa' directory is
> added to `package-directory-list', and is treated the same as other
> entries in that list. Note that this is different from files in
> emacs/lisp; those are added to `load-path'.

What does this mean from the POV of autoloading?  E.g., when I run the
latest Emacs from the current master branch, package-directory-list is
nil; does this mean Emacs will be unable to load packages in
emacs/elpa without my setting package-directory-list to something
non-nil?  Or will package-directory-list be set up automatically at
startup?  If the latter what will be the order of searching packages:
first load-path, then package-directory-list, or vice versa?

> Using `package-directory-list' allows users to disable bundled
> packages via `package-load-list'.

What does "disable" mean in this context?  Why would users want to
"disable" a bundled package?

> - One issue is whether the autoloads of bundled ELPA packages are
>   processed when we dump Emacs (like we do for all the packages that
>   are in Emacs core), or whether that's done during
>   `package-activate-all`. The baseline design does it during
>   `package-activate-all'.
> 
>   Doing it at dump time gives better startup times, at the cost of
>   making it impossible for the end-user to prevent activation of a
>   package.

That goes back to the question why would users want to prevent
activation of a bundled package.

>   Users who care about startup time can do their own dump.

If this is an acceptable solution (given that re-dumping is still not
reliable enough and has known bugs), why not turn the table and say
that users who want to prevent activation of a package can do their
own dump?

> - emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME]
> 
>   The baseline design keeps all bundled ELPA packages in
>   emacs/elpa/[PKNAME], both in the source and installed directory
>   trees. This makes it very easy to distinguish an emacs directory
>   tree with bundled packages from one without.
> 
>   An alternative is to keep the bundled ELPA packages in
>   emacs/lisp/[PKGNAME]; that minimizes the change when a package that
>   is current in emacs.git converts to a bundled package. This would
>   also mean that bundled packages are put in `load-path' at Emacs
>   startup, not in `package-directory-list'.

I prefer the latter, unless it's really impractical.

> - Define a useful emacs lisp project (or something simpler),
>   So `C-x p f' searches `load-path' via `project-find-file'
> 
>   That reduces the need to remember what packages are bundled, and
>   where you have them checked out.

I couldn't follow this item.  Can you elaborate?

> - Automating compiling C, Ada, Rust or other language code required by
>   the package, to executables or modules. This includes defining
>   standard places to put executables, modules, and other
>   aritifacts. This is more important for bundled packages; users
>   expect Emacs to "just work" out of the box.

I'm not sure I understand the issue: the standard places for all of
this are already defined.  Why would they need to change under this
proposal?

Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Fri, 29 Jan 2021 11:32:34 -0800
>> 
>> Attached is an updated version of emacs/admin/notes/elpa, describing a
>> proposal for bundling ELPA packages in the emacs distribution.
>> 
>> I believe I have included all the recent discussions.
>
> Thanks.  The text confused me a couple of times, so I think it "needs
> some work"; see the questions and comments below.

Thanks for the detailed review. 

>> I think I figured out a way to resolve the submodles vs worktrees
>> dilemma; each emacs developer can choose. If they don't execute any
>> submodule commands, they can use worktrees.
>
> I think the consequences of each of the two choices shall be spelled
> out, because at least to me they are not clear.  What are the
> advantages and disadvantages of each choice?

With submodules:

    - a single command 'git clone <emacs> --recurse-submodules'
    populates the entire Emacs directory tree.

    - vc understands submodules
    
    - cannot use any git worktrees.
    
With worktrees:

    - developer must remember to not execute any submodule commands

    - a second command ./checkout_git_elpa_worktrees.sh must be used to
    finish populating the Emacs directory tree

    - developer can use git worktrees to maintain multiple checkouts
    from the local emacs.git repository
    
I'll add this to the text.

>> If we can reach consensus on this, the next step would be to create a
>> feature branch and edit the code to implement this, then do a practice
>> release with that branch.
>
> I'm not sure we need a public branch for that.  What would be the
> purpose of such a branch?

I'd like Phillip Lord to test packaging windows, at least. But I'll
start with a private branch.

>> After the initial install of Emacs, users do not need to do anything
>> special to update a bundled ELPA package; the normal package.el
>> mechanisms recognize all the versions properly.
>
> This should tell more about "not having to do anything special": I'm
> guessing (or rather hoping) you allude here to upgrading/downgrading
> the bundled packages from ELPA, *not* to how the bundled packages are
> used without upgrading.  IOW, to just use the version shipped with the
> Emacs release tarball, users don't need to use package.el, nor will
> they need to add any package-related code into their init files or
> create early-init.el if they didn't have it before.

Perhaps it would be best to say:

    For users, bundled packages are treated the same as any other
package, except that they are pre-installed.

> That is, what does this mean from the POV of a user who didn't until
> now use package.el at all, but just used what came with the release
> tarball?

They don't have to do anything.

>> Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
>> packages unless they are ":core" packages; if that functionality is
>> required, the package must be moved to Emacs.
>
> Another unclear part.  The following questions arise, and I don't see
> any answers to them:
>
>   . how to know whether a package is ":core", and how to indicate that
>     it's ":core"? is the elpa-packages file the only place?

Yes, but it actually doesn't matter. A better way to say this is:

    Emacs core code may not depend on any code in the emacs/elpa
directory.

>   . "if that functionality is required, the package must be moved to
>     Emacs" -- what functionality is alluded to here, 

the functionality of some package.

>   and what does it mean "moved to Emacs"? 
>   does any part of the following description apply to such "moved to
>   Emacs" packages?

I don't describe that process. I'll add it.

>>    - A simple git clone/checkout of emacs.git master creates a
>>      workspace will all code required to build and run Emacs.
>
> What does this mean in practical terms?  

The command 'git clone <emacs.gi> --recurse-submodules' creates an Emacs
directory tree.

It's probably best to just delete this paragraph; that command is given
later.

> Is this any change from the current situation, and if so, how? And
> what is "workspace" in this context?

The only change is the addition of --recurse-submodules

By "workspace" I mean "emacs diretory tree".

>>    - Any ELPA package can be changed without affecting Emacs core.
>
> How is this related?  

It's a consequence of the rule "Emacs core code cannot depend on ELPA packages"

> What changes to ELPA packages and in what location (on Savannah, on my
> local clone of elpa.git, in my clone of emacs.git, something else?)

Any; that's the point.

> will not "affect Emacs core", and what does "affect" mean in this
> context?

The normal English meaning; no byte compile errors, no change in behavior.

>> For single file packages, the file can simply be in both elpa.git and
>> emacs.git master. 

Stefan Monnier: I think this is incorrect; there is no copy in elpa.git?
For example, 'cl-print' is a :core package, but there is no
externals/cl-print branch, and no file 'cl-print' in the master branch.
It appears that elpaa--core-package-sync links the file to a checkout
(on the ELPA web server) of the emacs.git repository. And that it can
handle directories.

>> The package is marked ":core" in the ELPA >> "elpa-packages" file.
>> For example, eldoc and flymake are :core >> packages.
>
> So to know whether a package is ":core", one must look in the local
> clone of elpa.git?  

Yes.

> What if there's no local clone of elpa.git: does it mean Emacs (and
> package.el specifically) won't know whether a package is ":core"? 

Yes.

> does that matter?

No.

The Emacs directory tree contains a copy of the package file (in
emacs/lisp/...). The ELPA admin code makes that available as a package
that can be installed via package.el. The package maintainer can release
a new version of the package by changing the version number and
committing to emacs.git.

>> This mechanism is not supported for multi-file packages, because
>> maintaining sync between the files in emacs.git and elpa.git is more
>> problematic.

This should say "sync between the files in emacs.git and upstream".

>> To bundle a multi-file package in Emacs, add to elpa.git a branch
>> `emacs-NN/[PKGNAME]` which holds the version to be included in the
>> next Emacs release of version NN:
>> 
>>     git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]
>> 
>> Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
>> elpa.git.
>
> My Git documentation doesn't mention the -c switch of "git branch".
> What did I miss?

https://git-scm.com/docs/git-branch

git branch (-c | -C) [<oldbranch>] <newbranch>

I may have the wrong option; that page does not say "create a new
branch from an existing branch", but I think that's what -c does.

> More importantly, why do we need more than one protocol for bundling a
> package?  Wouldn't it be simpler to have just one?

I guess you mean :core is one protocol, and create emacs-NN/[PKGNAME]
branch is another? I was not using the term "bundled" to refer to :core,
but you are correct that they are similar.

The main rationale for having :core is that it requires no change in
emacs.git to make a package available via ELPA.

The main rationale for having emacs-NN/[PKGNAME] is that it allows using
'git push/pull' to sync with an upstream development repository.

:core requires manual sync with an upstream repository; the developer
must use ediff or equivalent, not 'git push/pull'. 

>> Also push the release branch to emacs.git:
>> 
>>     git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME]
>> 
>>     This allows emacs developers to checkout the bundled package
>>     without requiring a separate clone of elpa.git.
>>     
>> In emacs/master, add a git submodule for that branch:
>> 
>>     git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]
>
> You say "emacs/master", and I'm guessing this means the master branch
> of emacs.git (but please be explicit about this, and don't let us
> guess).  

Yes.

> If that is correct, then the master branch is not the most important
> place to do this: the most important place is the emacs-NN release
> branch, because that's where we care the most about having the stable
> version of each package.

Yes. 

> And finally, what does the above "git push" command do?  You are
> pushing from one Git repository (elpa.git) to another (emacs.git),
> right?  

Right. It copies all required files and related meta info from the
elpa.git repository to the emacs.git repository.

> What will that do in emacs.git, 

Similar to any other branch push; the branch is available for checkout.

> and what would be the effect on users who will next "git pull" from
> emacs.git?

None, until they checkout a local copy of the new branch, either
directly or via 'git submodule update'.

>> Emacs developers should add '--recurse-submodules' to the 'git clone'
>> command when cloning Emacs; that will checkout the submodules with the
>> rest of Emacs. To checkout the submodules in an existing Emacs
>> directory tree:
>> 
>>     git submodule update --reference . --init
>
> To which branch is this applicable?  

master or emacs-nn where nn >= 28.

> E.g., I have a separate clone where I work on the current release
> branch (emacs-27 as of this writing); do I need to do the above in
> that clone as well, 

No, because we are only proposing to introduce this feature in emacs 28
(or later).

> and should the command change to tell Git that I want the versions of
> the bundled packages that are appropriate for the branch I'm tracking
> there? 

Assuming we have reached the point where emacs > 28 are available, the
file .gitmodules contains the submodule info, including the branch name;
'git submodule update' uses that file to determine what to do.

.gitmodules is committed to emacs.git.

I think there may be an issue here; if .gitmodules records the branch
name for a submodule, but not a specific commit, and there are multiple
commits on that submodule branch, then attempting to checkout anything
other than HEAD from emacs-nn might get the wrong version of the
submodule. That could be fixed by manually checking out the correct
version in the submodule directory.

> What if I switch the branch inside a repository -- do I need to do
> anything special for the submodules?

You need to use 'git checkout --recurse-submodules'; that updates the
submodules properly.

>>   We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
>>   worktree' to checkout each bundled ELPA branch into emac/elpa,
>>   instead of 'git submodule update'.
>
> What does this script do? 

For each bundled package:

    git worktree add -b emacs-NN/[PKGNAME] elpa/[PKGNAME] origin/[PKGNAME]

although the details may vary depending on whether it's been done before
or not; git gets confusing sometimes.

> does it allow developers to use "git worktrees" safely without losing
> the access to bundled packages? 

Yes.

>> There are currently (jan 2020) some multi-file packages with code in
>> both elpa.git and emacs.git; org, gnus for example. To convert those
>> packages into bundled packages, they are deleted from emacs.git, and
>> then follow the process above. The net result in the emacs directory
>> tree is that the code moves from emacs/lisp/[PKGNAME] to
>> emacs/elpa/[PKGNAME].
>
> How will this move affect Git history and the related "git log" and
> "git blame" commands? what about "git bisect" across the move?

There is already separate history in emacs.git and elpa.git. The history
in emacs.git will simply end; the history in elpa.git will simply
continue.

>> At Emacs startup, the (installed or source) `emacs/elpa' directory is
>> added to `package-directory-list', and is treated the same as other
>> entries in that list. Note that this is different from files in
>> emacs/lisp; those are added to `load-path'.
>
> What does this mean from the POV of autoloading?  E.g., when I run the
> latest Emacs from the current master branch, package-directory-list is
> nil; does this mean Emacs will be unable to load packages in
> emacs/elpa without my setting package-directory-list to something
> non-nil?  

The startup code will add emacs/elpa to `package-directory-list', and
`package-activate' will load the autoloads file, so you don't have to do
anything.

> Or will package-directory-list be set up automatically at startup? If
> the latter what will be the order of searching packages: first
> load-path, then package-directory-list, or vice versa?

package-directory-list is not used to find files for 'load'; it is used
to find packages for `package-activate'. When `package-activate' runs,
it prepends the activated package directories to `load-path'.

>> Using `package-directory-list' allows users to disable bundled
>> packages via `package-load-list'.
>
> What does "disable" mean in this context?  Why would users want to
> "disable" a bundled package?

I'm quoting the doc string for package-load-list; if `package-load-list'
contains ("foo" nil), then package "foo" is not available; it is
"disabled".

One reason to do this is if you are a package developer, and want to use
the development version of a package, not the installed version (I do
that for ada-mode).

Or there's something in the auto-load for an installed package that
interferes with something you are doing; perhaps you are working on an
alternative package for a similar feature.

For non-bundled package an alternative to "disable" is to uninstall the
package, but that's not an option with bundled packages.

The point is that package.el provides this mechanism to manage installed
packages; to be consistent, it should be available for bundled (ie
"pre-installed") packages as well.

>> - Define a useful emacs lisp project (or something simpler),
>>   So `C-x p f' searches `load-path' via `project-find-file'
>> 
>>   That reduces the need to remember what packages are bundled, and
>>   where you have them checked out.
>
> I couldn't follow this item.  Can you elaborate?

If you use C-x f (find-file), you need to know what top-level directory
to look in to find a package file; for example ada-mode.el will be in
emacs/elpa/ada-mode, not emacs/lisp/progmodes/ada-mode

If you use C-x p f (project-find-file), with an appropriate elisp
project active, then you have file-name completion on all elisp files in
load-path, so you don't have to know what directory to look in. Directory
information is only needed to disambiguate files with identical names
under emacs/lisp/cedet.

>> - Automating compiling C, Ada, Rust or other language code required by
>>   the package, to executables or modules. This includes defining
>>   standard places to put executables, modules, and other
>>   aritifacts. This is more important for bundled packages; users
>>   expect Emacs to "just work" out of the box.
>
> I'm not sure I understand the issue: the standard places for all of
> this are already defined.  

Ok, I was not aware of that.

-- 
-- Stephe