unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* policy discussion on bundling ELPA packages in the emacs tarball
@ 2021-01-21  1:41 Stephen Leake
  2021-01-21 14:17 ` Eli Zaretskii
                   ` (2 more replies)
  0 siblings, 3 replies; 70+ messages in thread
From: Stephen Leake @ 2021-01-21  1:41 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 4363 bytes --]

Attached is the file 'elpa', containing the current
emacs/admin/notes/elpa, plus a baseline policy for bundling GNU ELPA
packages in the emacs tarball, and some unresolved issues.

This is _not_ a final plan; it is a starting point for discussion.

(I tried to post it in the savannah ada-mode download area, but trying
to retrieve it gives me 404. And now I can't delete it. Sigh).

The biggest open issue is whether the bundled packages live only in the
elpa.git worktree, or if they are git submodules that are checked out into
the emacs.git worktree.

The baseline policy described in the attached file states that the
bundled packages live only in the elpa git worktree.

Eli has said he wants the bundled packages in the emacs worktree; I'd
like to explore the specific rationale for that.

For the specific example of ada-mode, I'm guessing nobody much noticed
when its files were removed from emacs/lisp/progmodes. Other packages
that are not used by many Emacs developers will be similarly not much
missed.

At the moment, ada-mode is the _only_ package we are proposing to
bundle, as a test case. But we need to plan for the future.

Currently, org and others are "duplicated" packages; they have files in
emacs.git master/lisp/[PKGNAME]/* and in elpa.git externals/[PKGNAME].
At some point, they could become "bundled" packages.

Consider what this would look like for org under the baseline policy.
The files in master/lisp/org/* are deleted, and "org elpa.git
externals/org" is added to the list of bundled packages used by 'make
dist' (I need to add explicit detail to that mechanism; Stefan Monnier
mentioned this might already be written somewhere).

Any Emacs developer that wants to continue using org has two choices;
install it via M-x list-packages, or, if they want the development head,
checkout ELPA, run 'make packages/org', and either install it from there
or add elpa/packages/org to their load-path. This is exactly what Emacs
and ELPA developers do now for non-bundled packages.

The baseline policy describes a shell script `checkout-bundled-elpa' and
an emacs command `add-bundled-elpa-packages' (for ~/.emacs) to simplify
using all bundled packages.

Now consider what bundling org looks like if we use some variant of the
"git submodules" policy. (for an introduction to git submodules, see
https://git-scm.com/book/en/v2/Git-Tools-Submodules). The files in
master/lisp/org/* are deleted, and "org elpa.git externals/org" is added
to the emacs/master/.gitmodules file (via 'git submodule add ...').
'make dist' includes emacs/master/elpa, so nothing in 'make dist' is
changed for 'org'.

In addition, Emacs developers who want the bundled ELPA packages would
run 'git submodule update --init'; this would checkout all the bundled
ELPA packages, into emacs/master/elpa/[PKGNAME]. Emacs startup code
would put these in the default load-path.

A fresh checkout of Emacs with the bundled submodules would be done by
'git clone .../emacs.git --recurse-submodules'. Ideally, the savannah
page describing how to get Emacs from git would say that. but many
people will forget the --recurse-submodules.


In both cases, the bundled packages are in load-path.

Finding them via C-x C-f (find-file) is slightly different; in the
baseline case it requires navigating to <root>/elpa/packages/... for
bundled packages. In the submodules case is requires navigating to
<root>/emacs/master/elpa/... Both are different from the current
<root>/emacs/master/lisp/..., so in either case you have to remember
which packages are bundled, and that can change over time. I suggest we
standardize on an emacs elisp project that allows using C-x p f
(project-find-file) to complete on files in load-path for elisp files;
that removes the issue of remembering where things are.

Another difference is testing for a release; that should always be done
with the identified release version of bundled packages. That is easier
to accomplish via git submodules, since they can identify a specific
version (I think?), or a specific release branch. Otherwise people doing
release testing would have to use `checkout-bundled-elpa' to get the
correct versions in elpa.

There will probably be a similar number of posts to emacs.devel about
"where is org?"; the answer is either "cd elpa; make packages/org" or
"git submodule update --init".

Comments?

--
-- Stephe

[-- Attachment #2: elpa --]
[-- Type: application/octet-stream, Size: 7219 bytes --]

NOTES ON THE EMACS PACKAGE ARCHIVE

The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git
repository named "elpa", hosted on Savannah.  To check it out:

  git clone git://git.sv.gnu.org/emacs/elpa
  cd elpa
  make setup

That leaves the elpa/packages directory empty; you must check out the
ones you want.

If you wish to check out all the packages into the packages directory,
you can run the command:

   make worktrees

You can check out a specific package <pkgname> into the packages
directory with:

   make packages/<pkgname>


Changes to this repository propagate to elpa.gnu.org via a
"deployment" script run daily.  This script generates the content
visible at https://elpa.gnu.org/packages.

A new package is released as soon as the "version number" of that
package is changed.  So you can use 'elpa' to work on a package
without fear of releasing those changes prematurely.  And once the
code is ready, just bump the version number to make a new release of
the package.

It is easy to use the elpa branch to deploy a "local" copy of the
package archive.  For details, see the README file in the elpa branch.


ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL

Why Bundle?

    - To provide backward compatibility for users while allowing more
      flexible releases, and more convenient maintenance.

      - ada-mode was originally included in the emacs tarball, but was
        removed when the ELPA version was capable enough. Some users
        complained; bundling ada-mode would (almost) restore the
        original behavior.

      - org and other packages currently maintain two copies of their
        code, in emacs.git master and elpa.git. Synchronizing them is
        difficult; it is not a simple `git push/pull'. Bundling would
        eliminate the copy in emacs.git.
    
Users do not need to do anything special update a bundled ELPA
package; the normal package.el mechanisms recognize all the versions
properly. 

Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
packages unless they are ":core" packages (ie, they have code in both
emacs.git master and elpa.git); if that functionality is required, the
package must be moved to Emacs.

   Rationale:
   
   - A simple git clone/checkout of emacs.git master creates a
     workspace will all code required to build and run Emacs.

   - Any ELPA package can be changed without affecting Emacs core.

For single file packages, the file can simply be in both elpa.git and
emacs.git master. The package is marked ":core" in the ELPA
"elpa-packages" file. For example, eldoc and flymake are :core
packages. This mechanism is not supported for multi-file packages,
because maintaining sync between the files in emacs.git and elpa.git
is more problematic - with the mechanism described below, you can use
git push/pull to maintain sync.


Every GNU ELPA package has a corresponding branch
`externals/[PKGNAME]` in elpa.git. To bundle a package in Emacs, add
to elpa.git a branch `emacs-NN/[PKGNAME]` which holds the version to
be included in the next Emacs release of version NN. `make dist' would
then grab the bundled packages's code from those branches, and include
them in /usr/share/emacs/NN.MM/elpa/* in the distribution. 

At Emacs startup, the bundled `elpa' directory is added to
package-directory-list, and is treated the same as other entries in
that list.

For Emacs developer convenience, there is a script
`checkout-bundled-elpa' that checks out the correct versions of the
bundled ELPA packages into an elpa worktree. In addition, there is an
emacs function `add-bundled-elpa-packages' that adds those package
directories to `load-path', so they work as if they were simply in the
emacs.git worktree. Both of these features are optional. 

UNRESOLVED ISSUES:

- There are currently (jan 2020) some multi-file packages with code in
  both elpa.git and emacs.git; org, gnus for example. Once the ELPA
  bundling process is worked out, it is expected that those packages
  will use the bundling process instead.

- In the future, for packages like org that used to have code in
  emacs.git master and elpa.git, the emacs.git checkout no longer has
  the package, which is not backward compatible for developers. Emacs
  developers who want those packages now have to install them via
  `list-packages', or check them out from elpa.git.

- Should fetching a worktree from emacs.git also fetch the bundled GNU
  ELPA packages (into the Emacs worktree or somewhere else)?

     - Not fetching ELPA packages into the emacs.git worktree helps
       enforce the rule that Emacs core code may not depend on ELPA
       code.

     - Not fetching ELPA packages means all packages are treated
       equally; Emacs developers do not have to be aware of whether
       some packages are bundled or not. Only the package developer
       has to know.

     - Fetching them makes it clearer what's bundled and what isn't. 
     
     - With the baseline mechanism, developers who want a bundled ELPA
       package checkout that package from elpa.git and either install
       the package from there or include elpa/packages/[PKGNAME] in
       their `load-path', as for any (non-bundled) ELPA package.
     
     - An alternative would be to declare a git submodule for each
       bundled package, with a corresponding branch `elpa/[PKGNAME]`
       (instead of the branches 'emacs-nn/[PKGNAME]'). The submodule
       declaration would place the packages in emacs/elpa (parallel to
       emacs/lisp), so Emacs subdirs.el does not see them. emacs/elpa
       is in package-directory-list by default when running from the
       source directory.

     - If we have submodules, new developers or a fresh checkout must
       use:

          git clone .../emacs.git --recurse-submodules

       for the initial clone. Ideally, the Savannah web page
       https://savannah.gnu.org/git/?group=emacs would say that.

     - Other variations on where to place the submodules in the emacs
       worktree offer more backward compatibility.

       - One issue is whether the autoloads of bundled ELPA packages
         are processed when we dump Emacs (like we do for all the
         packages that are in Emacs core), or whether that's done
         during `package-activate-all` (i.e. between `early-init.el`
         and `init.el`).

         Doing it at dump time gives better startup times, at the cost
         of making it impossible for the end-user to prevent
         activation of a package (they can still undo the activation
         after the fact, of course, but that needs to be done in
         ad-hoc ways).

         Users who care about startup time can do their own dump.


     - Using submodules would help automate cleaning and updating the
       bundled packages; with the baseline mechanism, developers must
       separately manage the elpa worktree.

- Automating compiling C, Ada, Rust or other language code required by
  the package, to executables or modules. This includes defining
  standard places to put executables, modules, and other
  aritifacts. This is more important for bundled packages; users
  expect Emacs to "just work" out of the box.

-- updated 20 Jan 2021

^ permalink raw reply	[flat|nested] 70+ messages in thread

end of thread, other threads:[~2021-02-01 15:09 UTC | newest]

Thread overview: 70+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-01-21  1:41 policy discussion on bundling ELPA packages in the emacs tarball Stephen Leake
2021-01-21 14:17 ` Eli Zaretskii
2021-01-22 10:56   ` [SPAM UNSURE] " Stephen Leake
2021-01-22 11:50     ` Eli Zaretskii
2021-01-22 14:05       ` Stefan Monnier
2021-01-21 21:54 ` Phillip Lord
2021-01-23  2:50   ` Stephen Leake
2021-01-23  8:17     ` Eli Zaretskii
2021-01-24 16:26       ` [SPAM UNSURE] " Stephen Leake
2021-01-24 21:49       ` Phillip Lord
2021-01-25 19:38         ` [SPAM UNSURE] " Stephen Leake
2021-01-25 19:54           ` Eli Zaretskii
2021-01-25 20:39           ` Stefan Monnier
2021-01-26  1:04             ` Andy Moreton
2021-01-26  2:38               ` Stefan Monnier
2021-01-26 14:44             ` Eli Zaretskii
2021-01-26 15:25               ` Stefan Monnier
2021-01-26 15:34                 ` Eli Zaretskii
2021-01-26 16:14                   ` Stefan Monnier
2021-01-26 16:30                     ` Eli Zaretskii
2021-01-26 17:27                       ` Stefan Monnier
2021-01-26 17:42                         ` Eli Zaretskii
2021-01-26 17:57                           ` Stefan Monnier
2021-01-26 19:12                             ` Eli Zaretskii
2021-01-28  0:15                               ` Stephen Leake
2021-01-28  2:05                                 ` Stefan Monnier
2021-01-28 13:47                                   ` Eli Zaretskii
2021-01-28 13:43                                 ` Eli Zaretskii
2021-01-27 14:52                         ` Dmitry Gutov
2021-01-27 14:57                           ` Stefan Monnier
2021-01-26 18:02                 ` Stefan Monnier
2021-01-26 18:12                   ` Stefan Monnier
2021-01-27 14:21                   ` Stephen Leake
2021-01-27 14:35                     ` Stefan Monnier
2021-01-27 14:48               ` Stephen Leake
2021-01-27 14:42             ` Stephen Leake
2021-01-27 16:22               ` Eli Zaretskii
2021-01-23  9:27     ` Stephen Leake
2021-01-23  9:57       ` Eli Zaretskii
2021-01-24  2:28         ` Dmitry Gutov
2021-01-24 16:27           ` Stephen Leake
2021-01-24 17:30         ` Stephen Leake
2021-01-24 18:10           ` Eli Zaretskii
2021-01-24 20:10             ` Stephen Leake
2021-01-24 20:15               ` Eli Zaretskii
2021-01-24 20:42                 ` Dmitry Gutov
2021-01-25 22:00                   ` Stephen Leake
2021-01-26  1:57                     ` Dmitry Gutov
2021-01-26 14:48                     ` Eli Zaretskii
2021-01-26 15:24                       ` Andreas Schwab
2021-01-27 14:37                       ` Stephen Leake
2021-01-27 16:22                         ` Eli Zaretskii
2021-01-28  0:10                           ` Stephen Leake
2021-01-28 13:38                             ` Eli Zaretskii
2021-01-25 19:30                 ` Stephen Leake
2021-01-25 19:53                   ` Eli Zaretskii
2021-01-27 14:31                     ` Stephen Leake
2021-01-27 14:44                       ` Stefan Monnier
2021-01-28  3:34                         ` Andy Moreton
2021-01-27 16:20                       ` Eli Zaretskii
2021-01-29 17:47                         ` Phillip Lord
2021-02-01  8:45                           ` Robert Pluim
2021-02-01 15:09                             ` Eli Zaretskii
2021-01-24 22:09     ` Phillip Lord
2021-01-25 19:14       ` Stephen Leake
2021-01-27 11:10         ` Phillip Lord
2021-01-27 14:31           ` Stefan Monnier
2021-01-28  0:05           ` Stephen Leake
2021-01-24 20:19   ` Stephen Leake
2021-01-24  2:59 ` Dmitry Gutov

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).