* policy discussion on bundling ELPA packages in the emacs tarball - take 2 @ 2021-01-29 19:32 Stephen Leake 2021-01-29 23:01 ` Phillip Lord 2021-01-30 10:40 ` Eli Zaretskii 0 siblings, 2 replies; 6+ messages in thread From: Stephen Leake @ 2021-01-29 19:32 UTC (permalink / raw) To: emacs-devel [-- Attachment #1: Type: text/plain, Size: 551 bytes --] Attached is an updated version of emacs/admin/notes/elpa, describing a proposal for bundling ELPA packages in the emacs distribution. I believe I have included all the recent discussions. I think I figured out a way to resolve the submodles vs worktrees dilemma; each emacs developer can choose. If they don't execute any submodule commands, they can use worktrees. If we can reach consensus on this, the next step would be to create a feature branch and edit the code to implement this, then do a practice release with that branch. -- -- Stephe [-- Attachment #2: elpa --] [-- Type: application/octet-stream, Size: 8372 bytes --] NOTES ON THE EMACS PACKAGE ARCHIVE The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git repository named "elpa", hosted on Savannah. To check it out: git clone git://git.sv.gnu.org/emacs/elpa cd elpa make setup That leaves the elpa/packages directory empty; you must check out the ones you want. If you wish to check out all the packages into the packages directory, you can run the command: make worktrees You can check out a specific package <pkgname> into the packages directory with: make packages/<pkgname> Changes to this repository propagate to elpa.gnu.org via a "deployment" script run daily. This script generates the content visible at https://elpa.gnu.org/packages. A new package is released as soon as the "version number" of that package is changed. So you can use 'elpa' to work on a package without fear of releasing those changes prematurely. And once the code is ready, just bump the version number to make a new release of the package. It is easy to use the elpa branch to deploy a "local" copy of the package archive. For details, see the README file in the elpa branch. ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL -- updated 29 Jan 2021 Why Bundle? - To provide backward compatibility for users while allowing more flexible releases, and more convenient maintenance. - ada-mode was originally included in the emacs tarball, but was removed when the ELPA version was capable enough. Some users complained; bundling ada-mode would (almost) restore the original behavior. - org and other packages currently maintain two copies of their code, in emacs.git master and elpa.git. Synchronizing them is difficult; it is not a simple `git push/pull'. Bundling would allow using 'git push/pull' to synchonize between emacs.git and elpa.git. - To provide a better "out-of-the-box" Emacs experience for new Emacs users, by bundling new useful packages. After the initial install of Emacs, users do not need to do anything special to update a bundled ELPA package; the normal package.el mechanisms recognize all the versions properly. Emacs core code may not depend on (bundled or non-bundled) GNU ELPA packages unless they are ":core" packages; if that functionality is required, the package must be moved to Emacs. - A simple git clone/checkout of emacs.git master creates a workspace will all code required to build and run Emacs. - Any ELPA package can be changed without affecting Emacs core. For single file packages, the file can simply be in both elpa.git and emacs.git master. The package is marked ":core" in the ELPA "elpa-packages" file. For example, eldoc and flymake are :core packages. This mechanism is not supported for multi-file packages, because maintaining sync between the files in emacs.git and elpa.git is more problematic. To bundle a multi-file package in Emacs, add to elpa.git a branch `emacs-NN/[PKGNAME]` which holds the version to be included in the next Emacs release of version NN: git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME] Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in elpa.git. Also push the release branch to emacs.git: git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME] This allows emacs developers to checkout the bundled package without requiring a separate clone of elpa.git. In emacs/master, add a git submodule for that branch: git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME] Emacs developers should add '--recurse-submodules' to the 'git clone' command when cloning Emacs; that will checkout the submodules with the rest of Emacs. To checkout the submodules in an existing Emacs directory tree: git submodule update --reference . --init However, the git worktree manual (https://git-scm.com/docs/git-worktree), in the Bugs section at the end, says: Multiple checkout in general is still experimental, and the support for submodules is incomplete. It is NOT recommended to make multiple checkouts of a superproject. git worktrees allow keeping several checked out directory trees while having only one local copy of the emacs.git repository. Some emacs developers use worktrees, for example to keep emacs master, emacs-27, and some feature branches checked out and built. Emacs developers that wish to use worktrees must not execute any submodule commands; do not include --recurse-submodules' on 'git clone', do not execute 'git submodule update'. Then the local git repository is not a "superproject", and worktrees can be safely used. We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git worktree' to checkout each bundled ELPA branch into emac/elpa, instead of 'git submodule update'. There are currently (jan 2020) some multi-file packages with code in both elpa.git and emacs.git; org, gnus for example. To convert those packages into bundled packages, they are deleted from emacs.git, and then follow the process above. The net result in the emacs directory tree is that the code moves from emacs/lisp/[PKGNAME] to emacs/elpa/[PKGNAME]. 'make' treats emacs/elpa the same as it treats emacs/lisp and emacs/doc; .el files are byte compiled, .texi files are compiled to info. Note that documentation source .texi files for bundled packages reside in emacs/elpa/[PKGNAME], not in emacs/doc. 'make dist' treats emacs/elpa the same as it treats emacs/lisp; the files are included in the distribution tarball. 'make install' places the installed emacs/elpa directory next to the installed emacs/lisp directory. At Emacs startup, the (installed or source) `emacs/elpa' directory is added to `package-directory-list', and is treated the same as other entries in that list. Note that this is different from files in emacs/lisp; those are added to `load-path'. Using `package-directory-list' allows users to disable bundled packages via `package-load-list'. Emacs developers use the various 'git submodule' commands to maintain packages in emacs/elpa. It is expected that very few edits will be made there; the primary development of ELPA packages is done in a checkout from elpa.git. Changes can be made in emacs/elpa to fix a bug for a release, or to change code to be consistent with some emacs core change. Such changes can be pushed to elpa.git and merged into the package development branch. UNRESOLVED ISSUES/TODO: - One issue is whether the autoloads of bundled ELPA packages are processed when we dump Emacs (like we do for all the packages that are in Emacs core), or whether that's done during `package-activate-all`. The baseline design does it during `package-activate-all'. Doing it at dump time gives better startup times, at the cost of making it impossible for the end-user to prevent activation of a package. Users who care about startup time can do their own dump. - emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME] The baseline design keeps all bundled ELPA packages in emacs/elpa/[PKNAME], both in the source and installed directory trees. This makes it very easy to distinguish an emacs directory tree with bundled packages from one without. An alternative is to keep the bundled ELPA packages in emacs/lisp/[PKGNAME]; that minimizes the change when a package that is current in emacs.git converts to a bundled package. This would also mean that bundled packages are put in `load-path' at Emacs startup, not in `package-directory-list'. - Does vc support submodules and nested worktrees? - Define a useful emacs lisp project (or something simpler), So `C-x p f' searches `load-path' via `project-find-file' That reduces the need to remember what packages are bundled, and where you have them checked out. - Update the mechanism that populates https://www.gnu.org/software/emacs/manual/html_mono/* For example, ada-mode.html there still reflects the ancient version that was in emacs core. - Automating compiling C, Ada, Rust or other language code required by the package, to executables or modules. This includes defining standard places to put executables, modules, and other aritifacts. This is more important for bundled packages; users expect Emacs to "just work" out of the box. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2 2021-01-29 19:32 policy discussion on bundling ELPA packages in the emacs tarball - take 2 Stephen Leake @ 2021-01-29 23:01 ` Phillip Lord 2021-01-30 0:07 ` Stephen Leake 2021-01-30 10:40 ` Eli Zaretskii 1 sibling, 1 reply; 6+ messages in thread From: Phillip Lord @ 2021-01-29 23:01 UTC (permalink / raw) To: Stephen Leake; +Cc: emacs-devel Stephen Leake <stephen_leake@stephe-leake.org> writes: > Attached is an updated version of emacs/admin/notes/elpa, describing a > proposal for bundling ELPA packages in the emacs distribution. > > I believe I have included all the recent discussions. > > I think I figured out a way to resolve the submodles vs worktrees > dilemma; each emacs developer can choose. If they don't execute any > submodule commands, they can use worktrees. That will be a significant help, actually. I presume that any build would be missing bundled packages? For me, this would not be a hassle because I'll just set it up so they install from ELPA on first use. It would still break my windows packaging scripts which use worktrees, (or as it stands the windows distribution will miss ada-mode) but I guess that only affects me. We are some way of Emacs-28, though, so now is a good time to break things. I would hope that big packages (org!!!) is not going to move for a while. Also, some research suggests that the git developers are slowing working out how to fix the module/worktree breakage, so who knows they might have fixed it by then. > If we can reach consensus on this, the next step would be to create a > feature branch and edit the code to implement this, then do a practice > release with that branch. It's fine with me! Phil ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2 2021-01-29 23:01 ` Phillip Lord @ 2021-01-30 0:07 ` Stephen Leake 2021-01-30 10:08 ` Phillip Lord 0 siblings, 1 reply; 6+ messages in thread From: Stephen Leake @ 2021-01-30 0:07 UTC (permalink / raw) To: Phillip Lord; +Cc: emacs-devel Phillip Lord <phillip.lord@russet.org.uk> writes: > Stephen Leake <stephen_leake@stephe-leake.org> writes: > >> Attached is an updated version of emacs/admin/notes/elpa, describing a >> proposal for bundling ELPA packages in the emacs distribution. >> >> I believe I have included all the recent discussions. >> >> I think I figured out a way to resolve the submodles vs worktrees >> dilemma; each emacs developer can choose. If they don't execute any >> submodule commands, they can use worktrees. > > That will be a significant help, actually. I presume that any build > would be missing bundled packages? I don't follow. If you mean "a build from an emacs directory tree", then it will have whatever the developer checked out into that tree. If they did "git clone <emacs> --recurse-submodules", then the build will have the bundled packages. If they left --recurse-submodules off the clone, but later did either: git submodule update --reference . --init or: ./checkout_git_elpa_worktrees.sh then the build will have the bundled packages. The last option is indented for your Windows packaging workflow; it uses git worktrees instead of submodules. If they do neither of these options, the build will not have the bundled packages. > For me, this would not be a hassle because I'll just set it up so they > install from ELPA on first use. This is with your emacs user hat on, I guess. Yes, using a build from an emacs directory tree without bundled packages would require getting those packages via M-x list-packages (or equivalent). > It would still break my windows packaging scripts which use worktrees, It should not, as described above. -- -- Stephe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2 2021-01-30 0:07 ` Stephen Leake @ 2021-01-30 10:08 ` Phillip Lord 0 siblings, 0 replies; 6+ messages in thread From: Phillip Lord @ 2021-01-30 10:08 UTC (permalink / raw) To: Stephen Leake; +Cc: emacs-devel Stephen Leake <stephen_leake@stephe-leake.org> writes: > Phillip Lord <phillip.lord@russet.org.uk> writes: > >> Stephen Leake <stephen_leake@stephe-leake.org> writes: >> >> That will be a significant help, actually. I presume that any build >> would be missing bundled packages? > > I don't follow. > > If you mean "a build from an emacs directory tree", then it will have > whatever the developer checked out into that tree. > > > ./checkout_git_elpa_worktrees.sh > > then the build will have the bundled packages. > > The last option is indented for your Windows packaging workflow; it uses > git worktrees instead of submodules. Apologies that was a very badly worded question, but yes, you have answered it. I had missed the "checkout_git_elpa_worktrees.sh" option apologies. If that works it would require only a small update to my scripts so sounds excellent. Are you going to get this onto a branch first; I'll be happy to test the Windows packaging when you have. > If they do neither of these options, the build will not have the bundled > packages. > >> For me, this would not be a hassle because I'll just set it up so they >> install from ELPA on first use. > > This is with your emacs user hat on, I guess. Yes, using a build from an > emacs directory tree without bundled packages would require getting > those packages via M-x list-packages (or equivalent). Yes, that's as a normal user. >> It would still break my windows packaging scripts which use worktrees, > > It should not, as described above. Thank your taking the time and effort to think about this, it's really greatly appreciated! Phil ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2 2021-01-29 19:32 policy discussion on bundling ELPA packages in the emacs tarball - take 2 Stephen Leake 2021-01-29 23:01 ` Phillip Lord @ 2021-01-30 10:40 ` Eli Zaretskii 2021-01-30 19:25 ` Stephen Leake 1 sibling, 1 reply; 6+ messages in thread From: Eli Zaretskii @ 2021-01-30 10:40 UTC (permalink / raw) To: Stephen Leake; +Cc: emacs-devel > From: Stephen Leake <stephen_leake@stephe-leake.org> > Date: Fri, 29 Jan 2021 11:32:34 -0800 > > Attached is an updated version of emacs/admin/notes/elpa, describing a > proposal for bundling ELPA packages in the emacs distribution. > > I believe I have included all the recent discussions. Thanks. The text confused me a couple of times, so I think it "needs some work"; see the questions and comments below. > I think I figured out a way to resolve the submodles vs worktrees > dilemma; each emacs developer can choose. If they don't execute any > submodule commands, they can use worktrees. I think the consequences of each of the two choices shall be spelled out, because at least to me they are not clear. What are the advantages and disadvantages of each choice? > If we can reach consensus on this, the next step would be to create a > feature branch and edit the code to implement this, then do a practice > release with that branch. I'm not sure we need a public branch for that. What would be the purpose of such a branch? > After the initial install of Emacs, users do not need to do anything > special to update a bundled ELPA package; the normal package.el > mechanisms recognize all the versions properly. This should tell more about "not having to do anything special": I'm guessing (or rather hoping) you allude here to upgrading/downgrading the bundled packages from ELPA, *not* to how the bundled packages are used without upgrading. IOW, to just use the version shipped with the Emacs release tarball, users don't need to use package.el, nor will they need to add any package-related code into their init files or create early-init.el if they didn't have it before. That is, what does this mean from the POV of a user who didn't until now use package.el at all, but just used what came with the release tarball? > Emacs core code may not depend on (bundled or non-bundled) GNU ELPA > packages unless they are ":core" packages; if that functionality is > required, the package must be moved to Emacs. Another unclear part. The following questions arise, and I don't see any answers to them: . how to know whether a package is ":core", and how to indicate that it's ":core"? is the elpa-packages file the only place? . "if that functionality is required, the package must be moved to Emacs" -- what functionality is alluded to here, and what does it mean "moved to Emacs"? does any part of the following description apply to such "moved to Emacs" packages? > - A simple git clone/checkout of emacs.git master creates a > workspace will all code required to build and run Emacs. What does this mean in practical terms? Is this any change from the current situation, and if so, how? And what is "workspace" in this context? > - Any ELPA package can be changed without affecting Emacs core. How is this related? What changes to ELPA packages and in what location (on Savannah, on my local clone of elpa.git, in my clone of emacs.git, something else?) will not "affect Emacs core", and what does "affect" mean in this context? > For single file packages, the file can simply be in both elpa.git and > emacs.git master. The package is marked ":core" in the ELPA > "elpa-packages" file. For example, eldoc and flymake are :core > packages. So to know whether a package is ":core", one must look in the local clone of elpa.git? What if there's no local clone of elpa.git: does it mean Emacs (and package.el specifically) won't know whether a package is ":core"? does that matter? > This mechanism is not supported for multi-file packages, because > maintaining sync between the files in emacs.git and elpa.git is more > problematic. > > To bundle a multi-file package in Emacs, add to elpa.git a branch > `emacs-NN/[PKGNAME]` which holds the version to be included in the > next Emacs release of version NN: > > git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME] > > Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in > elpa.git. My Git documentation doesn't mention the -c switch of "git branch". What did I miss? More importantly, why do we need more than one protocol for bundling a package? Wouldn't it be simpler to have just one? > Also push the release branch to emacs.git: > > git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME] > > This allows emacs developers to checkout the bundled package > without requiring a separate clone of elpa.git. > > In emacs/master, add a git submodule for that branch: > > git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME] You say "emacs/master", and I'm guessing this means the master branch of emacs.git (but please be explicit about this, and don't let us guess). If that is correct, then the master branch is not the most important place to do this: the most important place is the emacs-NN release branch, because that's where we care the most about having the stable version of each package. And finally, what does the above "git push" command do? You are pushing from one Git repository (elpa.git) to another (emacs.git), right? What will that do in emacs.git, and what would be the effect on users who will next "git pull" from emacs.git? > Emacs developers should add '--recurse-submodules' to the 'git clone' > command when cloning Emacs; that will checkout the submodules with the > rest of Emacs. To checkout the submodules in an existing Emacs > directory tree: > > git submodule update --reference . --init To which branch is this applicable? E.g., I have a separate clone where I work on the current release branch (emacs-27 as of this writing); do I need to do the above in that clone as well, and should the command change to tell Git that I want the versions of the bundled packages that are appropriate for the branch I'm tracking there? What if I switch the branch inside a repository -- do I need to do anything special for the submodules? > We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git > worktree' to checkout each bundled ELPA branch into emac/elpa, > instead of 'git submodule update'. What does this script do? does it allow developers to use "git worktrees" safely without losing the access to bundled packages? IOW, more details about the script need to be spelled out. > There are currently (jan 2020) some multi-file packages with code in > both elpa.git and emacs.git; org, gnus for example. To convert those > packages into bundled packages, they are deleted from emacs.git, and > then follow the process above. The net result in the emacs directory > tree is that the code moves from emacs/lisp/[PKGNAME] to > emacs/elpa/[PKGNAME]. How will this move affect Git history and the related "git log" and "git blame" commands? what about "git bisect" across the move? > At Emacs startup, the (installed or source) `emacs/elpa' directory is > added to `package-directory-list', and is treated the same as other > entries in that list. Note that this is different from files in > emacs/lisp; those are added to `load-path'. What does this mean from the POV of autoloading? E.g., when I run the latest Emacs from the current master branch, package-directory-list is nil; does this mean Emacs will be unable to load packages in emacs/elpa without my setting package-directory-list to something non-nil? Or will package-directory-list be set up automatically at startup? If the latter what will be the order of searching packages: first load-path, then package-directory-list, or vice versa? > Using `package-directory-list' allows users to disable bundled > packages via `package-load-list'. What does "disable" mean in this context? Why would users want to "disable" a bundled package? > - One issue is whether the autoloads of bundled ELPA packages are > processed when we dump Emacs (like we do for all the packages that > are in Emacs core), or whether that's done during > `package-activate-all`. The baseline design does it during > `package-activate-all'. > > Doing it at dump time gives better startup times, at the cost of > making it impossible for the end-user to prevent activation of a > package. That goes back to the question why would users want to prevent activation of a bundled package. > Users who care about startup time can do their own dump. If this is an acceptable solution (given that re-dumping is still not reliable enough and has known bugs), why not turn the table and say that users who want to prevent activation of a package can do their own dump? > - emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME] > > The baseline design keeps all bundled ELPA packages in > emacs/elpa/[PKNAME], both in the source and installed directory > trees. This makes it very easy to distinguish an emacs directory > tree with bundled packages from one without. > > An alternative is to keep the bundled ELPA packages in > emacs/lisp/[PKGNAME]; that minimizes the change when a package that > is current in emacs.git converts to a bundled package. This would > also mean that bundled packages are put in `load-path' at Emacs > startup, not in `package-directory-list'. I prefer the latter, unless it's really impractical. > - Define a useful emacs lisp project (or something simpler), > So `C-x p f' searches `load-path' via `project-find-file' > > That reduces the need to remember what packages are bundled, and > where you have them checked out. I couldn't follow this item. Can you elaborate? > - Automating compiling C, Ada, Rust or other language code required by > the package, to executables or modules. This includes defining > standard places to put executables, modules, and other > aritifacts. This is more important for bundled packages; users > expect Emacs to "just work" out of the box. I'm not sure I understand the issue: the standard places for all of this are already defined. Why would they need to change under this proposal? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2 2021-01-30 10:40 ` Eli Zaretskii @ 2021-01-30 19:25 ` Stephen Leake 0 siblings, 0 replies; 6+ messages in thread From: Stephen Leake @ 2021-01-30 19:25 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> From: Stephen Leake <stephen_leake@stephe-leake.org> >> Date: Fri, 29 Jan 2021 11:32:34 -0800 >> >> Attached is an updated version of emacs/admin/notes/elpa, describing a >> proposal for bundling ELPA packages in the emacs distribution. >> >> I believe I have included all the recent discussions. > > Thanks. The text confused me a couple of times, so I think it "needs > some work"; see the questions and comments below. Thanks for the detailed review. >> I think I figured out a way to resolve the submodles vs worktrees >> dilemma; each emacs developer can choose. If they don't execute any >> submodule commands, they can use worktrees. > > I think the consequences of each of the two choices shall be spelled > out, because at least to me they are not clear. What are the > advantages and disadvantages of each choice? With submodules: - a single command 'git clone <emacs> --recurse-submodules' populates the entire Emacs directory tree. - vc understands submodules - cannot use any git worktrees. With worktrees: - developer must remember to not execute any submodule commands - a second command ./checkout_git_elpa_worktrees.sh must be used to finish populating the Emacs directory tree - developer can use git worktrees to maintain multiple checkouts from the local emacs.git repository I'll add this to the text. >> If we can reach consensus on this, the next step would be to create a >> feature branch and edit the code to implement this, then do a practice >> release with that branch. > > I'm not sure we need a public branch for that. What would be the > purpose of such a branch? I'd like Phillip Lord to test packaging windows, at least. But I'll start with a private branch. >> After the initial install of Emacs, users do not need to do anything >> special to update a bundled ELPA package; the normal package.el >> mechanisms recognize all the versions properly. > > This should tell more about "not having to do anything special": I'm > guessing (or rather hoping) you allude here to upgrading/downgrading > the bundled packages from ELPA, *not* to how the bundled packages are > used without upgrading. IOW, to just use the version shipped with the > Emacs release tarball, users don't need to use package.el, nor will > they need to add any package-related code into their init files or > create early-init.el if they didn't have it before. Perhaps it would be best to say: For users, bundled packages are treated the same as any other package, except that they are pre-installed. > That is, what does this mean from the POV of a user who didn't until > now use package.el at all, but just used what came with the release > tarball? They don't have to do anything. >> Emacs core code may not depend on (bundled or non-bundled) GNU ELPA >> packages unless they are ":core" packages; if that functionality is >> required, the package must be moved to Emacs. > > Another unclear part. The following questions arise, and I don't see > any answers to them: > > . how to know whether a package is ":core", and how to indicate that > it's ":core"? is the elpa-packages file the only place? Yes, but it actually doesn't matter. A better way to say this is: Emacs core code may not depend on any code in the emacs/elpa directory. > . "if that functionality is required, the package must be moved to > Emacs" -- what functionality is alluded to here, the functionality of some package. > and what does it mean "moved to Emacs"? > does any part of the following description apply to such "moved to > Emacs" packages? I don't describe that process. I'll add it. >> - A simple git clone/checkout of emacs.git master creates a >> workspace will all code required to build and run Emacs. > > What does this mean in practical terms? The command 'git clone <emacs.gi> --recurse-submodules' creates an Emacs directory tree. It's probably best to just delete this paragraph; that command is given later. > Is this any change from the current situation, and if so, how? And > what is "workspace" in this context? The only change is the addition of --recurse-submodules By "workspace" I mean "emacs diretory tree". >> - Any ELPA package can be changed without affecting Emacs core. > > How is this related? It's a consequence of the rule "Emacs core code cannot depend on ELPA packages" > What changes to ELPA packages and in what location (on Savannah, on my > local clone of elpa.git, in my clone of emacs.git, something else?) Any; that's the point. > will not "affect Emacs core", and what does "affect" mean in this > context? The normal English meaning; no byte compile errors, no change in behavior. >> For single file packages, the file can simply be in both elpa.git and >> emacs.git master. Stefan Monnier: I think this is incorrect; there is no copy in elpa.git? For example, 'cl-print' is a :core package, but there is no externals/cl-print branch, and no file 'cl-print' in the master branch. It appears that elpaa--core-package-sync links the file to a checkout (on the ELPA web server) of the emacs.git repository. And that it can handle directories. >> The package is marked ":core" in the ELPA >> "elpa-packages" file. >> For example, eldoc and flymake are :core >> packages. > > So to know whether a package is ":core", one must look in the local > clone of elpa.git? Yes. > What if there's no local clone of elpa.git: does it mean Emacs (and > package.el specifically) won't know whether a package is ":core"? Yes. > does that matter? No. The Emacs directory tree contains a copy of the package file (in emacs/lisp/...). The ELPA admin code makes that available as a package that can be installed via package.el. The package maintainer can release a new version of the package by changing the version number and committing to emacs.git. >> This mechanism is not supported for multi-file packages, because >> maintaining sync between the files in emacs.git and elpa.git is more >> problematic. This should say "sync between the files in emacs.git and upstream". >> To bundle a multi-file package in Emacs, add to elpa.git a branch >> `emacs-NN/[PKGNAME]` which holds the version to be included in the >> next Emacs release of version NN: >> >> git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME] >> >> Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in >> elpa.git. > > My Git documentation doesn't mention the -c switch of "git branch". > What did I miss? https://git-scm.com/docs/git-branch git branch (-c | -C) [<oldbranch>] <newbranch> I may have the wrong option; that page does not say "create a new branch from an existing branch", but I think that's what -c does. > More importantly, why do we need more than one protocol for bundling a > package? Wouldn't it be simpler to have just one? I guess you mean :core is one protocol, and create emacs-NN/[PKGNAME] branch is another? I was not using the term "bundled" to refer to :core, but you are correct that they are similar. The main rationale for having :core is that it requires no change in emacs.git to make a package available via ELPA. The main rationale for having emacs-NN/[PKGNAME] is that it allows using 'git push/pull' to sync with an upstream development repository. :core requires manual sync with an upstream repository; the developer must use ediff or equivalent, not 'git push/pull'. >> Also push the release branch to emacs.git: >> >> git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git emacs-NN/[PKGNAME] >> >> This allows emacs developers to checkout the bundled package >> without requiring a separate clone of elpa.git. >> >> In emacs/master, add a git submodule for that branch: >> >> git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME] > > You say "emacs/master", and I'm guessing this means the master branch > of emacs.git (but please be explicit about this, and don't let us > guess). Yes. > If that is correct, then the master branch is not the most important > place to do this: the most important place is the emacs-NN release > branch, because that's where we care the most about having the stable > version of each package. Yes. > And finally, what does the above "git push" command do? You are > pushing from one Git repository (elpa.git) to another (emacs.git), > right? Right. It copies all required files and related meta info from the elpa.git repository to the emacs.git repository. > What will that do in emacs.git, Similar to any other branch push; the branch is available for checkout. > and what would be the effect on users who will next "git pull" from > emacs.git? None, until they checkout a local copy of the new branch, either directly or via 'git submodule update'. >> Emacs developers should add '--recurse-submodules' to the 'git clone' >> command when cloning Emacs; that will checkout the submodules with the >> rest of Emacs. To checkout the submodules in an existing Emacs >> directory tree: >> >> git submodule update --reference . --init > > To which branch is this applicable? master or emacs-nn where nn >= 28. > E.g., I have a separate clone where I work on the current release > branch (emacs-27 as of this writing); do I need to do the above in > that clone as well, No, because we are only proposing to introduce this feature in emacs 28 (or later). > and should the command change to tell Git that I want the versions of > the bundled packages that are appropriate for the branch I'm tracking > there? Assuming we have reached the point where emacs > 28 are available, the file .gitmodules contains the submodule info, including the branch name; 'git submodule update' uses that file to determine what to do. .gitmodules is committed to emacs.git. I think there may be an issue here; if .gitmodules records the branch name for a submodule, but not a specific commit, and there are multiple commits on that submodule branch, then attempting to checkout anything other than HEAD from emacs-nn might get the wrong version of the submodule. That could be fixed by manually checking out the correct version in the submodule directory. > What if I switch the branch inside a repository -- do I need to do > anything special for the submodules? You need to use 'git checkout --recurse-submodules'; that updates the submodules properly. >> We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git >> worktree' to checkout each bundled ELPA branch into emac/elpa, >> instead of 'git submodule update'. > > What does this script do? For each bundled package: git worktree add -b emacs-NN/[PKGNAME] elpa/[PKGNAME] origin/[PKGNAME] although the details may vary depending on whether it's been done before or not; git gets confusing sometimes. > does it allow developers to use "git worktrees" safely without losing > the access to bundled packages? Yes. >> There are currently (jan 2020) some multi-file packages with code in >> both elpa.git and emacs.git; org, gnus for example. To convert those >> packages into bundled packages, they are deleted from emacs.git, and >> then follow the process above. The net result in the emacs directory >> tree is that the code moves from emacs/lisp/[PKGNAME] to >> emacs/elpa/[PKGNAME]. > > How will this move affect Git history and the related "git log" and > "git blame" commands? what about "git bisect" across the move? There is already separate history in emacs.git and elpa.git. The history in emacs.git will simply end; the history in elpa.git will simply continue. >> At Emacs startup, the (installed or source) `emacs/elpa' directory is >> added to `package-directory-list', and is treated the same as other >> entries in that list. Note that this is different from files in >> emacs/lisp; those are added to `load-path'. > > What does this mean from the POV of autoloading? E.g., when I run the > latest Emacs from the current master branch, package-directory-list is > nil; does this mean Emacs will be unable to load packages in > emacs/elpa without my setting package-directory-list to something > non-nil? The startup code will add emacs/elpa to `package-directory-list', and `package-activate' will load the autoloads file, so you don't have to do anything. > Or will package-directory-list be set up automatically at startup? If > the latter what will be the order of searching packages: first > load-path, then package-directory-list, or vice versa? package-directory-list is not used to find files for 'load'; it is used to find packages for `package-activate'. When `package-activate' runs, it prepends the activated package directories to `load-path'. >> Using `package-directory-list' allows users to disable bundled >> packages via `package-load-list'. > > What does "disable" mean in this context? Why would users want to > "disable" a bundled package? I'm quoting the doc string for package-load-list; if `package-load-list' contains ("foo" nil), then package "foo" is not available; it is "disabled". One reason to do this is if you are a package developer, and want to use the development version of a package, not the installed version (I do that for ada-mode). Or there's something in the auto-load for an installed package that interferes with something you are doing; perhaps you are working on an alternative package for a similar feature. For non-bundled package an alternative to "disable" is to uninstall the package, but that's not an option with bundled packages. The point is that package.el provides this mechanism to manage installed packages; to be consistent, it should be available for bundled (ie "pre-installed") packages as well. >> - Define a useful emacs lisp project (or something simpler), >> So `C-x p f' searches `load-path' via `project-find-file' >> >> That reduces the need to remember what packages are bundled, and >> where you have them checked out. > > I couldn't follow this item. Can you elaborate? If you use C-x f (find-file), you need to know what top-level directory to look in to find a package file; for example ada-mode.el will be in emacs/elpa/ada-mode, not emacs/lisp/progmodes/ada-mode If you use C-x p f (project-find-file), with an appropriate elisp project active, then you have file-name completion on all elisp files in load-path, so you don't have to know what directory to look in. Directory information is only needed to disambiguate files with identical names under emacs/lisp/cedet. >> - Automating compiling C, Ada, Rust or other language code required by >> the package, to executables or modules. This includes defining >> standard places to put executables, modules, and other >> aritifacts. This is more important for bundled packages; users >> expect Emacs to "just work" out of the box. > > I'm not sure I understand the issue: the standard places for all of > this are already defined. Ok, I was not aware of that. -- -- Stephe ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-01-30 19:25 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-01-29 19:32 policy discussion on bundling ELPA packages in the emacs tarball - take 2 Stephen Leake 2021-01-29 23:01 ` Phillip Lord 2021-01-30 0:07 ` Stephen Leake 2021-01-30 10:08 ` Phillip Lord 2021-01-30 10:40 ` Eli Zaretskii 2021-01-30 19:25 ` Stephen Leake
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).