Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross <theophilusx@gmail.com>]

Ihor Radchenko <yantar92@posteo.net> writes:

>> It might be worthwhile defining what is meant by 3rd party packages.
>>
>> For example, ob-scheme relying on geiser as a 3rd party package is one
>> thing. Org roam is another type of 3rd party package. I think they need
>> different approaches. The first is about our (org) interface to them and
>> the latter is about their (org roam) interface with org.
>
> Sorry for not being clear.
> I was referring to third-party packages Org optionally depends on.
> Like geiser or citeproc or engrave-faces.
>
>> For the former (e.g. geiser type), I don't think backwards compatibility
>> is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
>> fairly trivial to update to the current version. We just need to support
>> the current version.
>
> Yes, but what if, say, the newest geiser version no longer supports the
> Emacs version Org still supports?
>

I guess we are limited by what the packages we rely on support. For
example, if geiser doesn't support Emacs 26 but org is supposed to,
there isn't much we can do. We cannot afford to fork geiser and modify
it to add the support and even if we provided a patch to add the support
to the geiser project, they may not merge it.

Best we can do is best effort and reflect this limitation in the manual
where we stipulate what our backwards compatibility policy is. 

>> For the latter (e.g. org-roam), backwards compatibility is much more
>> important. Org needs to provide as stable a public API for such packages
>> as possible. It may even be worthwhile separating the API into a
>> public/external API and an internal/private API. The former would be as
>> stable as possible and when changes are made, all efforts to provide
>> backwards compatibility are made. The latter is also stable, but more
>> subject to change. It is the API intended for internal org use. If
>> external packages use it, it is at their own risk. We would still try to
>> keep it stable, but with less of a guarantee and we may not provide
>> backwards compatibility if doing so was going to cause too much
>> complexity or maintenance burden etc.
>
> That's what we already do.
> I mean, https://bzg.fr/en/the-software-maintainers-pledge/ :)
>
> Public/private is the usual org-* vs. org--*.
> We never break org-*. Even if we do, we first deprecate things.
>

Yes, I know. I included it for completeness. However, I'm not convinced
worg is an effective communication channel for this information. We may
need to either add it to the manual or reference it from the
manual. (I'm reminded of the Douglas Adams quote from the Hitch Hikers
Guide [1]).

>> The big thing here is IMO avoiding surprise. We probably should add a
>> section to the 'Hacking' section of the manual which outlines our
>> position wrt backwards compatibility and API (public/private) and change
>> etc. I think most people expect change (even if they might not like
>> it). What they don't like is unexpected change. As long as we are
>> conservative and communicate change when necessary, we will likely be
>> OK.
>
> I hope that we never had unexpected changes. Isn't it for granted? I
> felt like it was always the case in Org and Emacs forever.
>
> It feels so obvious that I cannot even figure out how we could clarify
> it in the manual.

The one thing we can expect is unexpected change!

What is expected by one can be unexpected by others and what may seem
obvious to you or me may not be as obvious to others. We should be
extremely careful about assumptions regarding what can be taken for
granted. Org mode has a very broad user base. We have users with varying
familiarity with Emacs, software development practices and basic
software lifecycle management. For example, I know that symbol names
with '--' in them tend to indicate private API elements. However, that
is because I've been using Emacs for a long time. Someone new or someone
who is not a programmer may not be familiar with this convention.

There has been multiple occasions where I've seen posts on this list
from users complaining about unexpected and breaking changes. Such posts
usually end up in a thread about all the things org developers do to try
and mitigate such things. This would indicate it isn't necessarily taken
for granted by all. Often, those complaining were unaware of changes
being documented in the NEWS file or failed to check it before upgrading
or were unaware of backwards compatibility policy or didn't understand
the extent to which org relies on other external packages/services etc.

Then there is the situation where a change has unforeseen
impact. Consider the change a while ago with electric-indent.

I agree that in general, the org development process is pretty good in
its approach to handling change and maintenance of compatibility. Where
we are perhaps a little weak is in ensuring we communicate this
effectively. 

Perhaps we don't need to do anything extra. However, I feel it would be
beneficial to be very explicit regarding how we manage change and what
developers will promise and provide some guidance on how to use the org
APIs in the manual. Start from the position of users who have no
familiarity with what we do or even common development/maintenance
processes. Take little for granted and be explicit. I would consider

- Adding a section regarding pubic/private API and naming conventions to
  the Hacking section of the manual. This section could outline what the
  processes are for adding/changing APIs. 

- Add the maintainers pledge to the manual as well. It is useful for
  users to know what the maintainers pledge to try and do. I doubt many
  users will find the page on worg which already exists. At the very
  least, add a link to the wrog page.

- Briefly document the org maintenance and release process or add links
  to relevant worg pages. . 

One possibility would be to add/extend the 'Hacking' section to be
'Maintenance & Hacking' and add new sections which explicitly outline
the maintenance approach, the API conventions, the developer/maintainer
pledge, filing bug reports, requesting API change/enhancements, etc.

I realise some of this information is already in the manual (for
example, the feedback section covers submitting bug reports) and we
could either move it or reference that information. I also know much of
this is covered in worg, but I'm not sure that is a terribly effective
communication channel in itself. It may be sufficient to just add links
to worg content. However, for content which is very static, I think it
would be good to add it into the manual as I suspect that is the first
port of call for most users.

---
[1] “But the plans were on display…”
    “On display? I eventually had to go down to the cellar to find them.”
    “That’s the display department.”
    “With a flashlight.”
    “Ah, well, the lights had probably gone.”
    “So had the stairs.”
    “But look, you found the notice, didn’t you?”
    “Yes,” said Arthur,
    “yes I did. It was on display in the bottom of a locked filing
     cabinet stuck in a disused lavatory with a sign on the door saying
     ‘Beware of the Leopard.”

- Douglas Adams, The Hitchhike's Guide to the Galaxy