policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Stephen Leake <stephen_leake@stephe-leake.org>]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Attached is an updated version of emacs/admin/notes/elpa, describing a
proposal for bundling ELPA packages in the emacs distribution.

I believe I have included all the recent discussions.

I think I figured out a way to resolve the submodles vs worktrees
dilemma; each emacs developer can choose. If they don't execute any
submodule commands, they can use worktrees.

If we can reach consensus on this, the next step would be to create a
feature branch and edit the code to implement this, then do a practice
release with that branch.

-- 
-- Stephe

[-- Attachment #2: elpa --]
[-- Type: application/octet-stream, Size: 8372 bytes --]

NOTES ON THE EMACS PACKAGE ARCHIVE

The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git
repository named "elpa", hosted on Savannah.  To check it out:

  git clone git://git.sv.gnu.org/emacs/elpa
  cd elpa
  make setup

That leaves the elpa/packages directory empty; you must check out the
ones you want.

If you wish to check out all the packages into the packages directory,
you can run the command:

   make worktrees

You can check out a specific package <pkgname> into the packages
directory with:

   make packages/<pkgname>


Changes to this repository propagate to elpa.gnu.org via a
"deployment" script run daily.  This script generates the content
visible at https://elpa.gnu.org/packages.

A new package is released as soon as the "version number" of that
package is changed.  So you can use 'elpa' to work on a package
without fear of releasing those changes prematurely.  And once the
code is ready, just bump the version number to make a new release of
the package.

It is easy to use the elpa branch to deploy a "local" copy of the
package archive.  For details, see the README file in the elpa branch.


ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL
-- updated 29 Jan 2021

Why Bundle?

    - To provide backward compatibility for users while allowing more
      flexible releases, and more convenient maintenance.

      - ada-mode was originally included in the emacs tarball, but was
        removed when the ELPA version was capable enough. Some users
        complained; bundling ada-mode would (almost) restore the
        original behavior.

      - org and other packages currently maintain two copies of their
        code, in emacs.git master and elpa.git. Synchronizing them is
        difficult; it is not a simple `git push/pull'. Bundling would
        allow using 'git push/pull' to synchonize between emacs.git
        and elpa.git.

   - To provide a better "out-of-the-box" Emacs experience for new
     Emacs users, by bundling new useful packages.
   
After the initial install of Emacs, users do not need to do anything
special to update a bundled ELPA package; the normal package.el
mechanisms recognize all the versions properly.

Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
packages unless they are ":core" packages; if that functionality is
required, the package must be moved to Emacs.
   
   - A simple git clone/checkout of emacs.git master creates a
     workspace will all code required to build and run Emacs.

   - Any ELPA package can be changed without affecting Emacs core.

For single file packages, the file can simply be in both elpa.git and
emacs.git master. The package is marked ":core" in the ELPA
"elpa-packages" file. For example, eldoc and flymake are :core
packages. This mechanism is not supported for multi-file packages,
because maintaining sync between the files in emacs.git and elpa.git
is more problematic.

To bundle a multi-file package in Emacs, add to elpa.git a branch
`emacs-NN/[PKGNAME]` which holds the version to be included in the
next Emacs release of version NN:

    git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]

Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
elpa.git.

Also push the release branch to emacs.git:

    git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME]

    This allows emacs developers to checkout the bundled package
    without requiring a separate clone of elpa.git.
    
In emacs/master, add a git submodule for that branch:

    git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]

Emacs developers should add '--recurse-submodules' to the 'git clone'
command when cloning Emacs; that will checkout the submodules with the
rest of Emacs. To checkout the submodules in an existing Emacs
directory tree:

    git submodule update --reference . --init

  However, the git worktree manual (https://git-scm.com/docs/git-worktree), in
  the Bugs section at the end, says:

    Multiple checkout in general is still experimental, and the
    support for submodules is incomplete. It is NOT recommended to
    make multiple checkouts of a superproject.

  git worktrees allow keeping several checked out directory trees
  while having only one local copy of the emacs.git repository. Some
  emacs developers use worktrees, for example to keep emacs master,
  emacs-27, and some feature branches checked out and built.

  Emacs developers that wish to use worktrees must not execute any
  submodule commands; do not include --recurse-submodules' on 'git
  clone', do not execute 'git submodule update'. Then the local git
  repository is not a "superproject", and worktrees can be safely
  used.

  We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
  worktree' to checkout each bundled ELPA branch into emac/elpa,
  instead of 'git submodule update'.

There are currently (jan 2020) some multi-file packages with code in
both elpa.git and emacs.git; org, gnus for example. To convert those
packages into bundled packages, they are deleted from emacs.git, and
then follow the process above. The net result in the emacs directory
tree is that the code moves from emacs/lisp/[PKGNAME] to
emacs/elpa/[PKGNAME].

'make' treats emacs/elpa the same as it treats emacs/lisp and
emacs/doc; .el files are byte compiled, .texi files are compiled to
info. Note that documentation source .texi files for bundled packages
reside in emacs/elpa/[PKGNAME], not in emacs/doc.

'make dist' treats emacs/elpa the same as it treats emacs/lisp; the
files are included in the distribution tarball.

'make install' places the installed emacs/elpa directory next to the
installed emacs/lisp directory.

At Emacs startup, the (installed or source) `emacs/elpa' directory is
added to `package-directory-list', and is treated the same as other
entries in that list. Note that this is different from files in
emacs/lisp; those are added to `load-path'. Using
`package-directory-list' allows users to disable bundled packages via
`package-load-list'.

Emacs developers use the various 'git submodule' commands to maintain
packages in emacs/elpa. It is expected that very few edits will be
made there; the primary development of ELPA packages is done in a
checkout from elpa.git. Changes can be made in emacs/elpa to fix a bug
for a release, or to change code to be consistent with some emacs core
change. Such changes can be pushed to elpa.git and merged into the
package development branch.

UNRESOLVED ISSUES/TODO:

- One issue is whether the autoloads of bundled ELPA packages are
  processed when we dump Emacs (like we do for all the packages that
  are in Emacs core), or whether that's done during
  `package-activate-all`. The baseline design does it during
  `package-activate-all'.

  Doing it at dump time gives better startup times, at the cost of
  making it impossible for the end-user to prevent activation of a
  package.

  Users who care about startup time can do their own dump.

- emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME]

  The baseline design keeps all bundled ELPA packages in
  emacs/elpa/[PKNAME], both in the source and installed directory
  trees. This makes it very easy to distinguish an emacs directory
  tree with bundled packages from one without.

  An alternative is to keep the bundled ELPA packages in
  emacs/lisp/[PKGNAME]; that minimizes the change when a package that
  is current in emacs.git converts to a bundled package. This would
  also mean that bundled packages are put in `load-path' at Emacs
  startup, not in `package-directory-list'.

- Does vc support submodules and nested worktrees?

- Define a useful emacs lisp project (or something simpler),
  So `C-x p f' searches `load-path' via `project-find-file'

  That reduces the need to remember what packages are bundled, and
  where you have them checked out.

- Update the mechanism that populates
  https://www.gnu.org/software/emacs/manual/html_mono/*

  For example, ada-mode.html there still reflects the ancient version
  that was in emacs core.

- Automating compiling C, Ada, Rust or other language code required by
  the package, to executables or modules. This includes defining
  standard places to put executables, modules, and other
  aritifacts. This is more important for bundled packages; users
  expect Emacs to "just work" out of the box.