Re: Improving documentation of Org Mode integration into Emacs.

[Karl Fogel <kfogel@red-bean.com>]

[-- Attachment #1: Type: text/plain, Size: 7782 bytes --]

Replying to several emails together here -- from Stefan, Michael, 
Protesilaos, Rudolf, and Eli -- as they're related.   TL;DR: Just 
see the attached revised patch, which incorporates information and 
suggestions from those folks.

On 03 Jan 2022, Stefan Monnier wrote: 
>"Externally maintained" might not be quite right for the 
>following packages, but they do fit the description to some 
>extent in the sense that the maintainer also distributes the code 
>elsewhere (e.g. for XEmacs): - CC-mode - VHDL-mode - Verilog-mode 
 
Well, the question isn't about whether the code is also 
distributed elsewhere.  Rather, the question is "Where should a 
developer go first with bug reports, bug patches, enhancement 
suggestions, etc?"   If she should go to Emacs Devel (or our 
debbugs tracker) first, then no special note is needed.   If she 
should go to some other upstream origin first, or should at least 
consider doing that, then a note is needed.   (For the above three 
packages specifically, I don't know the answer to that question, 
as I haven't looked at them in depth.)   On 03 Jan 2022, Michael 
Albinus wrote: 
>> +Each externally maintained package lives in its own 
>> subdirectory in +the Emacs tree, and that subdirectory should 
>> have a "README" file that +describes the upstream origin.  For 
>> example, "lisp/org/README" says +where the Org Mode project 
>> lives and how it's synchronized into Emacs. 
> That's not the case now. Tramp, for example, does not live in an 
>own subdirectory. Should it? 
 
Ah, thank you for pointing this out.  Okay, there are two 
different ways we could handle that:  1) Document the current 
messy reality (described in detail below).   2) Clean up reality, 
then document that new reality :-).   I think (1) is better right 
now, as it's a smaller change.  The "messy reality" I'm referring 
to is this:  *Some* multi-file externally maintained packages, 
like Tramp, are not contained in a dedicated subdirectory.  All 
the Tramp files live in "lisp/net/", alongside other non-Tramp 
file (some of which are also multi-file packages, like EUDC, 
although unlike Tramp they are not externally maintained packages 
as far as I know).   Meanwhile, *other* multi-file externally 
maintained packages, such as Org Mode, do have their own dedicated 
subdirectory (e.g., "lisp/org/").  In these cases, there is as 
easy solution for documenting the fact of external maintenance: 
put a README in that subdir.  The attached revised patch shows 
this.   Doing (2) would mean putting every multi-file package into 
its own dedicated subdirectory (e.g., "lisp/net/tramp/") -- for 
both externally and internally maintained packages.  I think this 
would be a good organizational improvement for us to make anyway, 
not only because it would simplify our strategy for documenting 
external maintenance for multi-file packages.  But (1) seems like 
the right option here, as it's a smaller step.  (2) would be a 
separate decision that goes beyond the scope of my original 
proposal. 

On 03 Jan 2022, Protesilaos Stavrou wrote: 
>The modus-themes are maintained externally.  I sync them with 
>emacs.git every month or so when I release a new version.  Their 
>.el files are in etc/themes/ while the manual is in doc/misc/. 
>I am happy to provide a README, though neither of those two 
>directories is specific to the modus-themes.   Maybe I should 
>include the relevant information in the themes' manual and/or the 
>"Commentary:" of each .el file? 
 
Thanks for mentioning these.  Please see the attached patch, which 
I think might answer your question (in the diff to CONTRIBUTE). 
On 03 Jan 2022, Rudolf Adamkovič wrote:
>I have a small comment. :)
>I think we can drop the word "actually" from the first sentence:

I agree -- thank you.  Your suggestion is incorporated into the 
new patch attached here.

On 03 Jan 2022, Eli Zaretskii wrote: 
>Thanks.   However, IMO the text as written doesn't belong to 
>CONTRIBUTE.  That file includes actionable instructions to 
>contributors regarding our development procedures, conventions, 
>and requirements.  The purpose of those instructions is to make 
>the contributed changes acceptable and matching our practices. 
>By contrast, the text that you propose is just information that 
>is not actionable.  So if this text is to stay in its present 
>form, it should be somewhere else, perhaps in README.   If you do 
>want it to be in CONTRIBUTE, it mostly be comprised of some 
>specific instructions what to do or what not to do.  The purely 
>informational part should ideally be shorter and more focused on 
>what contributors need to know in order to, well, contribute. 
 
The attached (revised) patch gives some specific instructions as 
examples.  But CONTRIBUTE's job is just to explain the general 
situation about externally maintained packages and give a few 
examples, so that developers understand why they need to be watch 
out for this situation.  That's what the revised text in the 
attached patch does.   Let me explain more clearly why I think 
CONTRIBUTE needs this section:  The CONTRIBUTE file is where 
developers go to learn how to contribute to Emacs.  But right now, 
when a developer finds a bug in (or wants to make an improvement 
to) a package that is distributed with Emacs but is externally 
maintained, she could easily miss the fact that the package is 
externally maintained!  That means she'll often bring her bug 
report or enhancement suggestion to the wrong place first: Emacs's 
own forums, instead of to the true upstream forums.   In other 
words, CONTRIBUTE needs to document this phenomenon because 
CONTRIBUTE is where we point developers to to learn how to 
contribute to Emacs -- and this information is about routing 
contributions correctly.   But CONTRIBUTE shouldn't give the 
specific instructions for each package.  It should just show the 
developer how to figure out if a package is externally maintained, 
and then that package can point the developer in the right 
direction in whatever way is most appropriate.

>Please note that none of what you say here is in the text, not 
>even as a hint.  If the idea is to tell contributors to direct 
>bug reports for those packages elsewhere, why not tell that 
>explicitly, and why not include the relevant URLs where those 
>bugs should be reported? 
 
This is made more explicit now in CONTRIBUTE.

Again, I don't include all the specific upstream URLs in 
CONTRIBUTE because we're not listing all the externally maintained 
packages in CONTRIBUTE.  CONTRIBUTE just explains how to detect 
such a package, and then lets the package's files give the proper 
pointers. 
 
>> +See https://orgmode.org/ for more information, and see 
>> specifically +https://orgmode.org/worg/org-maintenance.html for 
>> information about +the process of synchronization between 
>> upstream Org Mode and Emacs. 
> The beginning of this is promising, but then it strays: instead 
>of telling people to send patches and bug reports about Org to 
>those addresses and not to Emacs's tracker, it invites them to 
>learn about the process of synchronization between Org and Emacs, 
>something that I think is of secondary importance (to say the 
>least) for causal contributors. 
 
Thanks; good idea.  I've improved the proposed lisp/org/README as 
you indicate above.  The new language also makes clearer why -- in 
the specific case of Org Mode -- one might want to learn about the 
process of synchronization, since with Org Mode changes flow 
bidirectionally (which may not be the case for all externally 
maintained packages).

Best regards,
-Karl


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: [PATCH] Document external maintenance of some packages --]
[-- Type: text/x-diff, Size: 5104 bytes --]

From 79527535f74b8299fa2d1612d0458d4972e05ac1 Mon Sep 17 00:00:00 2001
From: Karl Fogel <kfogel@red-bean.com>
Date: Sun, 2 Jan 2022 22:49:35 -0600
Subject: [PATCH] Document external maintenance of some packages

Document the fact that some packages in Emacs are externally
maintained and document how to discover their upstream origin.

For more context, see the thread that starts here:

  https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00366.html
  From: Karl Fogel
  To: Emacs Devel
  Subject: Improving documentation of Org Mode integration into Emacs.
  Date: Sat, 04 Dec 2021 15:14:59 -0600
  Message-ID: <87zgpgax7w.fsf@red-bean.com>
---
 CONTRIBUTE      | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 lisp/org/README | 19 +++++++++++++++++++
 2 files changed, 68 insertions(+)
 create mode 100644 lisp/org/README

diff --git CONTRIBUTE CONTRIBUTE
index 7c3421ed75..527ad1ed35 100644
--- CONTRIBUTE
+++ CONTRIBUTE
@@ -366,6 +366,55 @@ reasons.  These should be marked by including something like "Do not
 merge to master" or anything that matches gitmerge-skip-regexp (see
 admin/gitmerge.el) in the commit message.
 
+** Some packages in Emacs are maintained externally
+
+Sometimes a package that ships as part of GNU Emacs is maintained as a
+separate project, with its own upstream repository, its own maintainer
+group, its own development conventions, etc.  The upstream project's
+code is periodically merged into Emacs (exactly when and how such
+merges happen depends on the package).
+
+So when you are making a contribution -- such as fixing a bug or
+proposing an enhancement -- to one of these externally maintained
+packages, you often need to deal with that package at its upstream
+source.  For example, if you're trying to fix a bug, you should obtain
+the latest version of the packages's code from that upstream source,
+since that code will likely be ahead of whatever version is in Emacs
+and you'll want to test the latest code to see if the bug is still
+present.  Likewise, if you're starting a development discussion, you
+should generally do so in the package's own forums rather than in
+Emacs's forums; the upstream developers can provide guidance about
+whether Emacs's developers need to be brought in to the discussion.
+
+*** How to tell which packages are externally maintained
+
+When an externally maintained package is just one file, then a comment
+near the top of the file should indicate the upstream origin.
+
+When an externally maintained package involves multiple files, then
+there are a couple of possibilities:
+
+1. The files all live in one dedicated subdirectory that is specific
+   to that package.  In this case, there should be a README in that
+   subdirectory indicating the upstream origin.  For example,
+   "lisp/org/README" indicates where the Org Mode project lives and
+   what that provenance implies for contributors.
+
+2. The package's files all live in the same directory, and share a
+   common filename prefix, but there are also other files in that
+   directory that are unrelated to the package in question.
+
+   In this case, there should be a comment near the top of the
+   package's main entry-point file -- i.e., the file that one loads to
+   `provide' the package -- giving the provenance indication.  For
+   example, the Tramp package is made up of multiple files
+   ("lisp/net/tramp*") and the package's upstream origin is described
+   in a comment near the top of "lisp/net/tramp.el".
+
+Note that there may be some independently-maintained packages in Emacs
+that still lack a clear provenance indicator.  If you find one, please
+send in a patch or just let us know.
+
 ** GNU ELPA
 
 This repository does not contain the Emacs Lisp package archive
diff --git lisp/org/README lisp/org/README
new file mode 100644
index 0000000000..6c8022f56a
--- /dev/null
+++ lisp/org/README
@@ -0,0 +1,19 @@
+Org Mode is maintained as a separate project, and is periodically
+merged into Emacs.  To view or participate in Org Mode development,
+please go to https://orgmode.org/ and follow the instructions there.
+
+The source code from the upstream Org Mode project is usually not
+identical to the version of Org Mode in Emacs.  The upstream project
+often has recent changes that have not yet been merged into Emacs, and
+Emacs sometimes has local changes to Org Mode that have not yet been
+backported to upstream (https://orgmode.org/worg/org-maintenance.html
+documents how the Org Mode project synchronizes changes with Emacs).
+
+Thus, if you're investigating a bug you encountered in Org Mode in
+Emacs, you should obtain the latest upstream code and see if the bug
+is present there.  If the bug is present, then the upstream Org Mode
+project is the proper place to fix it.  If the bug is not present
+there, that could be because it has already been fixed upstream, or it
+could be because the bug was only introduced on the Emacs side and has
+not yet been backported upstream.  Either way, this is something you
+will need to know in order to know where to contribute your fix.
-- 
2.34.1