changing function signatures and no library version # => must use too-general test

changing function signatures and no library version # => must use too-general test

[Drew Adams]

You've heard this before, so those who might in fact be able to do something
about it will no doubt skip reading this, but...

Claim: A priori, it's good to avoid changing Emacs function signatures
(especially gratuitous changes), even if you think of the function as being
internal-only. Why? Because external Emacs libraries might try to use the
function across multiple Emacs versions. In Emacs, there is nothing that is
really "internal-only" in the sense that people will not see it and use it;
there are only different degrees of support by Emacs developers for external
use.

Yes, I know the mantra: You don't care about backward compatibility or
supporting multiple Emacs versions. But I do, and so do other
user-developers. Why send this to emacs-devel? Because in some cases there
is no reason not to be courteous to external users; that is, in some cases
it doesn't cost you anything to treat Emacs source code as if it will be
used by users.

Here's a simple problem: I have a library that uses function
`help-insert-xref-button'. Is it internal-only? I don't know, and I don't
really care. It has a nice doc string anyway, and it does just what I want,
so I use it.

Well, it turns out that the signature of this function changed fundamentally
from Emacs 21 to Emacs 22. Instead of just creating a new, different
function (whose existence could be tested with `fboundp'), which would have
been clean, some Emacs developer decided to keep the same function name and
completely change the parameter list. That's my complaint, in a nutshell -
that's NOT TRT, IMO.

It turns out that I can live without Emacs 21 support in this particular
case, so I need to test for the Emacs version in order to not use the wrong
function definition. Test for the Emacs version? Why don't I test for
something more specific? Good question.

There is no good way to test for the function with the right signature,
AFAIK. In Emacs 22 I could use `subr-arity' to test the number of args,
which would work in this case but not in cases where the number was the same
but the parameter types were different or their order changed - and, anyway,
that would require testing whether `subr-arity' is defined, and, if not, it
would require a different test (for Emacs other than 22).

There is no way to test for the particular version of the library that
defines the function. In this case, that is not a problem, because this and
related functions in fact moved from help.el to the new library
help-mode.el, so I can just test for (featurep 'help-mode). But in the more
general case, this is a problem: most Emacs libraries have no library
version.

In sum, it would be great if the following were true (in decreasing order of
desirability):

1. Emacs developers avoid changing function signatures or try to make such
changes compatible (e.g. via &optional). At least when other things are
equal. At the very least, avoid gratuitous changes. Rename a function if it
is not a user command and the signature is incompatibly different. Create an
alias to the old name and behavior, where feasible.

2. There is some way to test the signature of a function: number of args and
their types. The latter is no doubt unfeasible, given that Emacs Lisp is
generally untyped. The only notion of types we have in many cases is what
the doc string tells us. I mention this only as something that would be
desirable, not feasible.

3. Individual Emacs libraries have version numbers that can be tested.

Yes, I try to always use a test such as `fboundp' or `boundp', instead of an
Emacs version test or a `featurep' test. But in some cases, such as this
one, a too-general test seems unavoidable.

Re: changing function signatures and no library version # => must use too-general test

[Stefan Monnier]

> There is no good way to test for the function with the right signature,
> AFAIK. In Emacs 22 I could use `subr-arity' to test the number of args,
> which would work in this case but not in cases where the number was the same
> but the parameter types were different or their order changed - and, anyway,
> that would require testing whether `subr-arity' is defined, and, if not, it
> would require a different test (for Emacs other than 22).

Testing with subr-arity won't work because it's not a subroutine, and even
if it works it's difficult to use and inconvenient.  A better check is:

   (condition-case nil
       (help-insert-xref-button arg1 arg2 arg3)
     (wrong-number-of-arguments
      (help-insert-xref-button arg6 arg7))

and interestingly, this approach can also be used when the number of args
hasn't changed, as long as you can arrange for the first call to fail
immediately (e.g. because you use some new type of argument that wasn't
supported before).

Of course, this is not 100% errorproof since the error you catch may
actually come from some other piece of code within help-insert-xref-button.
In many cases this is not a problem because you can be reasonably sure that
having the same error signalled some other way is extremely unlikely
(e.g. this is the case for wrong-number-of-arguments).


        Stefan

RE: changing function signatures and no library version # => must use too-general test

[Drew Adams]

    > There is no good way to test for the function with the right
    signature,
    > AFAIK. In Emacs 22 I could use `subr-arity' to test the
    number of args,
    > which would work in this case but not in cases where the
    number was the same
    > but the parameter types were different or their order changed
    - and, anyway,
    > that would require testing whether `subr-arity' is defined,
    and, if not, it
    > would require a different test (for Emacs other than 22).

(As Davis Herrington pointed out off-list, I should have written "pre-Emacs
21" instead of "Emacs other than 22".)

    Testing with subr-arity won't work because it's not a
    subroutine, and even
    if it works it's difficult to use and inconvenient.  A better check is:

       (condition-case nil
           (help-insert-xref-button arg1 arg2 arg3)
         (wrong-number-of-arguments
          (help-insert-xref-button arg6 arg7))

Yes, I considered that.

    and interestingly, this approach can also be used when the
    number of args
    hasn't changed, as long as you can arrange for the first call to fail
    immediately (e.g. because you use some new type of argument that wasn't
    supported before).

Only if the different type in fact raises an error (as opposed to producing
incorrect results or leading to an error farther down the road).

    Of course, this is not 100% errorproof since the error you catch may
    actually come from some other piece of code within
    help-insert-xref-button.
    In many cases this is not a problem because you can be
    reasonably sure that
    having the same error signalled some other way is extremely unlikely
    (e.g. this is the case for wrong-number-of-arguments).

Yes, I agree with everything you said. It doesn't change the general
problem, however. Rather than users needing to define their own function
that does what you wrote above (if case they want to do that in multiple
places, for instance), Emacs developers should just define a new, different
function themselves (not pour new wine into old bottles).

Re: changing function signatures and no library version # => must use too-general test

[Stefan Monnier]

> Yes, I agree with everything you said. It doesn't change the general
> problem, however. Rather than users needing to define their own function
> that does what you wrote above (if case they want to do that in multiple
> places, for instance), Emacs developers should just define a new, different
> function themselves (not pour new wine into old bottles).

I don't know specifically what happened with help-insert-xref-button, but
it's prety common to change a function's list of arguments without
subsantially changing its body and/or its conceptual meaning, in which case
it will usually makes more sense to keep the same name.

Some functions are known to be "exported" in which case we have to be
careful, which is already a pain in the _|_ at times, but if we start to
have to be careful like that with each and every function, it's going to be
*really* annoying.

Next time around you'll come and complain that Emacs-22.1 defined foo-titi
and when its interface was changed in Emacs-22.4 the name was changed to
foo-toto, and when the interface was changed yet again in Emacs-23.1 the name
was changed to foo-titi ('cause the coder didn't know that name had already
been in use a few years back), so now you again have no way to tell whether
foo-titi uses the Emacs-22.1 or the Emacs-23.1 convention.


        Stefan

RE: changing function signatures and no library version # => must use too-general test

[Drew Adams]

    > Yes, I agree with everything you said. It doesn't change the general
    > problem, however. Rather than users needing to define their
    own function
    > that does what you wrote above (if case they want to do that
    in multiple
    > places, for instance), Emacs developers should just define a
    new, different
    > function themselves (not pour new wine into old bottles).

    I don't know specifically what happened with
    help-insert-xref-button, but
    it's prety common to change a function's list of arguments without
    subsantially changing its body and/or its conceptual meaning,
    in which case
    it will usually makes more sense to keep the same name.

Too bad for Emacs users. No, I don't see how that usually makes more sense,
whether it is common or not. Especially in the case of a non-interactive
function that has not particularly been advertised (e.g. in the Emacs-Lisp
manual), to me it makes more sense to change the function name when the
signature changes that much. (Simply adding parameters is of course not a
problem, if &optional or &rest is used to maintain backward compatibility.)

    Some functions are known to be "exported"

I don't know what you mean here. Do you mean documented in the manual?

    in which case we have to be
    careful, which is already a pain in the _|_ at times, but if we start to
    have to be careful like that with each and every function, it's
    going to be *really* annoying.

Annoyance to developers is important, but it is not as important as clean
software and clear software maintenance. Some developers have always felt
that the effort required to be user-friendly was simply a pain in the _|_
and a waste of their time and creativity. Most, fortunately, out-grow that,
or they learn better from being also users or maintainers of code written by
others.

Take me back a few decades, when writing clear code and commenting it was
considered, well wimpy. I knew a Lisp programmer who never, ever used any
whitespace that was not syntactically significant - tons of code in one long
line (no exaggeration). His personal shortcuts weren't limited to saving
whitespace (which could have been gotten around by pretty-printing). He was
brilliant and Lisp was about his only mode of thought, but his code was
considered strictly personal - no one wanted to try to modify, reuse, or
understand it.

To him it was "pretty common" to save time and space in his own way and it
would have been "*really* annoying" to write code that something other than
a Lisp machine or byte-compiler could read. He was, yes, very fast and very
good. He and his code just didn't play well with others.

    Next time around you'll come and complain that Emacs-22.1
    defined foo-titi
    and when its interface was changed in Emacs-22.4 the name was changed to
    foo-toto, and when the interface was changed yet again in
    Emacs-23.1 the name
    was changed to foo-titi ('cause the coder didn't know that name
    had already
    been in use a few years back), so now you again have no way to
    tell whether
    foo-titi uses the Emacs-22.1 or the Emacs-23.1 convention.

That would still be one problem less. That is, the same problem might
conceivably resurface, as you point out, but only in a much more limited and
less likely situation.

As far as I can see, you've simply pointed out an additional (unlikely)
problem to be tackled, not an argument for the status quo.

My general point is that Emacs users, more than users of other applications
(non-free and perhaps even other free applications), also read and reuse
Emacs source code. Keeping an eye out for how users might be affected by
source-code changes would be a *good thing*. I know the stock answer is that
Emacs developers don't care about backward compatibility, but some Emacs
users do: users who develop external libraries.

Do those users matter to Emacs development? Should you care about them? Put
it another way: How many useful Emacs features were imported from code
written originally by users as external libraries?

Re: changing function signatures and no library version # => must use too-general test

[Miles Bader]

"Drew Adams" <drew.adams@oracle.com> writes:
>     Some functions are known to be "exported"
>
> I don't know what you mean here. Do you mean documented in the manual?
>
>     in which case we have to be
>     careful, which is already a pain in the _|_ at times, but if we start to
>     have to be careful like that with each and every function, it's
>     going to be *really* annoying.
>
> Annoyance to developers is important, but it is not as important as clean
> software and clear software maintenance.

Elisp doesn't have a well-defined notion of interfaces, so it's
_possible_ that just about any random "internal" function will get
called by an external package.  However it would be completely absurd to
try and maintain absolute compatibility for _every_ function.

So it ends up being a judgement call:  If I judge a function to be
"internal" (and no evidence that people are using it despite that), then
I can decide to put the burden on external packages (to fix their code)
rather than on myself (to do the additional work and add the additional
cruft required for maintaining compatibility).  If I judge a function to
be "an interface", I will likely put the burden on myself.  I think this
is fair tradeoff.

If one is writing an external package, there are some vague clues one
can use to avoid problems -- as you suggest, if it's documented in the
manual, it's probably thought of as some sort of official interface, and
is less likely to be changed incompatibly; if it doesn't even have a
doc-string, it's likely thought of as an internal function.

-Miles
-- 
A zen-buddhist walked into a pizza shop and
said, "Make me one with everything."

Re: changing function signatures and no library version # => must use too-general test

[Stefan Monnier]

>     I don't know specifically what happened with help-insert-xref-button,
>     but it's prety common to change a function's list of arguments without
>     subsantially changing its body and/or its conceptual meaning, in which
>     case it will usually makes more sense to keep the same name.

> Too bad for Emacs users. No, I don't see how that usually makes more sense,
> whether it is common or not.

In most coding practices, there is this thing about choosing your function
names based on what they do.

> Annoyance to developers is important, but it is not as important as clean
> software and clear software maintenance.

It is *specifically* for clarity of the code and clear software maintenance
that function names should be chosen correctly, without having to worry
about previously used names.


        Stefan

RE: changing function signatures and no library version # => must use too-general test

[Drew Adams]

    >     I don't know specifically what happened with
    >     help-insert-xref-button,
    >     but it's prety common to change a function's list of
    >     arguments without
    >     subsantially changing its body and/or its conceptual
    >     meaning, in which
    >     case it will usually makes more sense to keep the same name.

    > Too bad for Emacs users. No, I don't see how that usually
    > makes more sense, whether it is common or not.

    In most coding practices, there is this thing about choosing
    your function names based on what they do.

A good, general rule.

    > Annoyance to developers is important, but it is not as
    > important as clean
    > software and clear software maintenance.

    It is *specifically* for clarity of the code and clear software
    maintenance that function names should be chosen correctly,
    without having to worry about previously used names.

Do I detect a little mauvais foi, or do you really believe that is what this
is about?

1. You won't find an argument against that from me, at that level of
abstraction. As you know, I've argued long and hard for getting function
names to accurately reflect what the functions do. Think back to the
discussion of the myriad different names for exiting view mode, none of
which were very "parlant". Likewise, menu names...

2. However, your argument is at too high an abstraction level. There is not
one, single, acceptable name for a given function. For the example given,
`help-insert-xref-button', there are plenty of acceptable alternatives, even
given the constraints of starting with "help" and including "xref-button":
`help-xref-button-insert', `help-insert-an-xref-button',
`help-add-xref-button', and so on.

3. Your abstract argument is also presented in stark isolation. It is
inappropriate to consider only one (good, general) software-engineering rule
(heuristic) and elevate it to the status of something inviolable. Such rules
need to be applied, and their application judged appropriate or not,
together, in combination, in a particular concrete context. The guidelines I
discussed are also important.

In a concrete, practical context there are tradeoffs to consider - for
example, is it more important that the name be exactly
`href-insert-xref-button' (because it accurately describes the function) or
is it more important to use a different name because that name is already
used for something different (in another Emacs version that is likely to
still be in use). Emacs developers might never get confused or bothered by
the existence of two different `href-insert-xref-button' functions because
they generally work with only one Emacs version, but others who work with
multiple versions will.

When you add "without having to worry about previously used names", you
violate the utility of the good guideline you promote. It's about balancing
such general rules, in context, together. It's not about promoting one's
favorite rule du jour and trying to sneak in an addendum that it is
inviolate even in the face of another rule (in all contexts, no less).

Of course names should be as accurate as possible, but yes, we do have to
worry also about previously used names much of the time. Legacy is something
that software engineering rules need to deal with. Only one-off school
projects can ignore legacy (there is none). It's ugly and mundane, no doubt,
but it's also part of what makes software development interesting.

4. Also to be measured in such a combined judgement are mitigating factors:
A function name is one way to document a function's meaning or behavior, but
it is not the only way; a function name that does not communicate
sufficiently accurately what the function is about can be supplemented by
documentation (the manual), a doc string, and source-code comments.