Re: declare function/macro private

Re: declare function/macro private

[Boruch Baum]

On Sun, 06 Jun 2021 22:54:41 -0400, Stefan Monnier wrote:

> My current guess is that you fear that "--" has currently been used
> carelessly

Using 'ivy/counsel' and performing M-x -- gives me 56 completion
candidates. 23 are from magit, Here are the rest:

    company--select-next-and-warn
    company--select-previous-and-warn
    company--warn-changed-binding
    counsel--info-lookup-symbol
    doc-view--text-view-mode
    elisp-flymake--batch-compile-for-flymake
    footnote--narrow-to-footnotes
    js--end-of-do-while-loop-p
    js--flush-caches
    js--show-cache-at-point
    ls-lisp--dired
    menu-bar--display-line-numbers-mode-absolute
    menu-bar--display-line-numbers-mode-none
    menu-bar--display-line-numbers-mode-relative
    menu-bar--display-line-numbers-mode-visual
    menu-bar--toggle-truncate-long-lines
    menu-bar--visual-line-mode-enable
    menu-bar--wrap-long-lines-window-edge
    org--backward-paragraph-once
    org--forward-paragraph-once
    python-skeleton--else
    python-skeleton--except
    python-skeleton--finally
    smie--next-indent-change
    term--xterm-paste
    vr--minibuffer-help
    vr--shortcut-toggle-limit
    vr--shortcut-toggle-preview
    w3m-page-nav--duckduckgo-style-next
    w3m-page-nav--duckduckgo-style-previous
    xref--mouse-2
    xref--transient-buffer-mode
    xref--xref-buffer-mode

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On Sun, 06 Jun 2021 22:54:41 -0400, Stefan Monnier wrote:
> 
> No, I'm still wondering what it is you find to be different.
> 
> My current guess is that you fear that "--" has currently been used
> carelessly and imposing a more "structured" meaning to it after the 
> fact
> will hence introduce problems, whereas your declaration would come 
> right
> away with an associated "precise" meaning.

I was *so sure* I had made this clear having said it four times, but 
okay: I do not wish to impose or change anything.

Now, I *know* that someone on emacs-devel will read that and decide it 
means "he wants to impose or change something" so again, for that 
reader: please put down your pitchfork, re-read the initial email, 
understand that it does not suggest imposing or changing anything.

What I am suggesting is that if a programmer wishes to make it 
abundantly and extraordinarily clear that a function/macro is internal 
and should never be used outside of this program, they would have this 
extra declaration which they could explicitly set that would provide a 
simple and discoverable warning to other elisp programmers.

An optional addition. That's it. No one forces anyone to use it, just 
like we have the interactive-only declaration and no one's house burnt 
down.

> If so, I agree to some extent, but:
> - As mentioned your declaration would suffer from the difficulty of
> clarifying "internal to what" since sometimes a function is internal
> to a file, sometimes to a multifile package and sometimes to a subset
> of a file.  Of course, that can be solved by adding an extra argument
> to your annotation, but:

I think internal to the file is easiest but if there's some easy way to 
make it "internal to the package/library" then cool. But as you've 
already said, does Elisp have a definition of what is and isn't part of 
a package/library? Not really. It does have plenty of things that work 
file-local though.

> - your annotation is placed on those functions that are internal, which
> (as a coder) are those functions which I'd rather not burden with too
> much extra annotations, otherwise I'll just not bother declaring them
> as internal (which is what we had before the "--" convention).

For you nothing would change.

> - As long as the effect is just a few font-lock-warning faces here and
> there, I think the problem is harmless enough (and it does point to
> real misfeatures in many cases, so it would help improve the code).

Cool.

> Yes, currently it's a bit of a wild west because the "--" has no 
> effect,
> but maybe it's time to impose some order on this, indeed.

I am not suggesting imposing anything. Trying to retroactively impose 
some definitive meaning upon people's use of "--" is, as I said, the 
path to ruin.

Others do not necessarily know what I know, i.e. while I may know that 
"--" is a convention that means "internal" in Elisp, other people may 
not (or likely do not). I suspect many programmers use it just because 
they've seen it used in other packages. And given that Elisp does not 
have any explicit definition of what is "internal" it would make little 
sense to impose one now and say "oh well that's what we meant all 
along".

So could someone prove how clever they are by making some sort of 
heuristic code to treat anything with "--" as internal? Of course. Would 
this have the net effect of raising false positives, confusing people, 
sparking endless angry emacs-devel threads, kicking your dog, and 
increasing bug reports? Absolutely.

This is not and was never part of my suggestion.

>> Also I just dislike computer heuristics.
> 
> It's not a heuristic.

Okay.

> But "file" is the wrong granularity for many Gnus-internal definitions
> (and same applies to all multifile packages).  So what would use for
> those if your declaration are only for file-local internal definitions?

This is outside the scope of this suggestion. For them, nothing changes.

So the objection I see is: if this explicit internal declaration were to 
be introduced, and say its scope is limited to the file in which it is 
declared, and then a programmer explicitly declares a function as 
internal knowing full well that this declaration means file-local, and 
that programmer then uses this internal function external to that file, 
they are rightfully triggering whatever ramifications the warnings are 
warning them against. See my alligator analogy.

Re: declare function/macro private

[Stefan Monnier]

>> No, I'm still wondering what it is you find to be different.
>> My current guess is that you fear that "--" has currently been used
>> carelessly and imposing a more "structured" meaning to it after the fact
>> will hence introduce problems, whereas your declaration would come right
>> away with an associated "precise" meaning.
> I was *so sure* I had made this clear having said it four times, but okay:
> I do not wish to impose or change anything.

I'm not sure which part of what you quoted made you think that I think
you with to impose or change anything (other than add a new
declaration, obviously).

> An optional addition. That's it. No one forces anyone to use it, just like
> we have the interactive-only declaration and no one's house burnt down.

Of course.  My participation here is just based on the fact that your
suggestion made me think about what we should do with "--".

From that stand point the two discussions are orthogonal.  They only
interact to the extent that they provide similar features so they are
somewhat redundant.

> Trying to retroactively impose some definitive meaning upon people's
> use of "--" is, as I said, the path to ruin.

I disagree.  I've been in the business of slowly changing ELisp coding
style for more than 20 years now, and while I'm not sure that what I've
proposed here to do with "--" would work well, I'm pretty sure it would
not be a path to ruin.

> Others do not necessarily know what I know, i.e. while I may know that "--"
> is a convention that means "internal" in Elisp, other people may not (or
> likely do not). I suspect many programmers use it just because they've seen
> it used in other packages. And given that Elisp does not have any explicit
> definition of what is "internal" it would make little sense to impose one
> now and say "oh well that's what we meant all along".

I think most people who don't know better won't see any difference
either after we'd introduce the new rule.  Or if they do, they'd then
learn about it and either adjust their code or ignore the warning.

> This is not and was never part of my suggestion.

Of course not.  That was my suggestion.


        Stefan

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

On 2021-06-07 15:59, Stefan Monnier wrote:
>>> No, I'm still wondering what it is you find to be different.
>>> My current guess is that you fear that "--" has currently been used
>>> carelessly and imposing a more "structured" meaning to it after the 
>>> fact
>>> will hence introduce problems, whereas your declaration would come 
>>> right
>>> away with an associated "precise" meaning.
>> I was *so sure* I had made this clear having said it four times, but 
>> okay:
>> I do not wish to impose or change anything.
> 
> I'm not sure which part of what you quoted made you think that I think
> you with to impose or change anything (other than add a new
> declaration, obviously).

Cool. This was not necessarily directed at you individually. I just want 
it be be crystal clear the limited extent of what I'm prepared to 
suggest/defend. Wording-wise, I wouldn't consider something like this a 
"change" in the sense of something existing that gets replaced with 
something else, but yes it's a "change" in the sense of something added.

> From that stand point the two discussions are orthogonal.  They only
> interact to the extent that they provide similar features so they are
> somewhat redundant.

In a way yes. From (info "(elisp) Tips for Defining"):

`PREFIX--...'
      The variable is intended for internal use and is defined in the
      file `PREFIX.el'.  (Emacs code contributed before 2018 may follow
      other conventions, which are being phased out.)

So right now it's just a tip that is supposed to communicate the 
author's intent. This is why I don't buy the idea that anyone could have 
used internal functions in multi-file packages -- because all they've 
done is followed a tip to communicate intent.

>> Trying to retroactively impose some definitive meaning upon people's
>> use of "--" is, as I said, the path to ruin.
> 
> I disagree.  I've been in the business of slowly changing ELisp coding
> style for more than 20 years now, and while I'm not sure that what I've
> proposed here to do with "--" would work well, I'm pretty sure it would
> not be a path to ruin.

Okay perhaps not.

>> Others do not necessarily know what I know, i.e. while I may know that 
>> "--"
>> is a convention that means "internal" in Elisp, other people may not 
>> (or
>> likely do not). I suspect many programmers use it just because they've 
>> seen
>> it used in other packages. And given that Elisp does not have any 
>> explicit
>> definition of what is "internal" it would make little sense to impose 
>> one
>> now and say "oh well that's what we meant all along".
> 
> I think most people who don't know better won't see any difference
> either after we'd introduce the new rule.  Or if they do, they'd then
> learn about it and either adjust their code or ignore the warning.
> 
>> This is not and was never part of my suggestion.
> 
> Of course not.  That was my suggestion.

I just really do not want to be caught defending the "--" idea against 
the hordes. I hope the rest of the thread can please consider that a 
wholly separate idea.

The idea of a `(declare (internal ARG))' seems an easier sell because it 
won't affect anyone who doesn't explicitly use it. It seems to open more 
possibilities for font-locking as Lars mentioned, and for generating 
info in the help buffer. Further to that, you could provide "levels" of 
how internal something should be treated, e.g.

(declare (internal t))  <- basic warning
(declare (internal "use `foo-public' instead."))  <- warning with hint
(declare (internal 'strict))  <- compiler signals an error, computer 
catches fire

Re: declare function/macro private

[Eli Zaretskii]

> Date: Sun, 6 Jun 2021 23:35:26 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: "Paul W. Rankin" <0278C47F-42CE-45C4-B789-83C57DF1A191@bydasein.com>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> 
> On Sun, 06 Jun 2021 22:54:41 -0400, Stefan Monnier wrote:
> 
> > My current guess is that you fear that "--" has currently been used
> > carelessly
> 
> Using 'ivy/counsel' and performing M-x -- gives me 56 completion
> candidates. 23 are from magit, Here are the rest:

You need to remove from this list those which are intended for
invocation via the mouse.  They aren't relevant to the issue at hand,
because the "--" was used there "for other purposes".

Re: declare function/macro private

[Boruch Baum]

On 2021-06-07 15:39, Eli Zaretskii wrote:
> > Date: Sun, 6 Jun 2021 23:35:26 -0400
> > From: Boruch Baum <boruch_baum@gmx.com>
> > Cc: "Paul W. Rankin" <0278C47F-42CE-45C4-B789-83C57DF1A191@bydasein.com>,
> >  Stefan Monnier <monnier@iro.umontreal.ca>
> >
> > On Sun, 06 Jun 2021 22:54:41 -0400, Stefan Monnier wrote:
> >
> > > My current guess is that you fear that "--" has currently been used
> > > carelessly
> >
> > Using 'ivy/counsel' and performing M-x -- gives me 56 completion
> > candidates. 23 are from magit, Here are the rest:
>
> You need to remove from this list those which are intended for
> invocation via the mouse.  They aren't relevant to the issue at hand,
> because the "--" was used there "for other purposes".

The point was just to support Paul Rankin's position that a '--'
substring isn't a reliable determinant of a function's public/private
nature, so can't be used as an alternative to an explicit (optional)
declaration.

My concern isn't the declaration, but the scope of its consequences. For
example, an emacs user who does no elisp programming would be happy to
have no private variable/function results appear among completion
candidates when performing M-x describe-variable/function. Elisp
programmers would want those completion candidates exposed, and subject
to being extended/advised.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: declare function/macro private

[Arthur Miller]

Boruch Baum <boruch_baum@gmx.com> writes:

> On 2021-06-07 15:39, Eli Zaretskii wrote:
>> > Date: Sun, 6 Jun 2021 23:35:26 -0400
>> > From: Boruch Baum <boruch_baum@gmx.com>
>> > Cc: "Paul W. Rankin" <0278C47F-42CE-45C4-B789-83C57DF1A191@bydasein.com>,
>> >  Stefan Monnier <monnier@iro.umontreal.ca>
>> >
>> > On Sun, 06 Jun 2021 22:54:41 -0400, Stefan Monnier wrote:
>> >
>> > > My current guess is that you fear that "--" has currently been used
>> > > carelessly
>> >
>> > Using 'ivy/counsel' and performing M-x -- gives me 56 completion
>> > candidates. 23 are from magit, Here are the rest:
>>
>> You need to remove from this list those which are intended for
>> invocation via the mouse.  They aren't relevant to the issue at hand,
>> because the "--" was used there "for other purposes".
>
> The point was just to support Paul Rankin's position that a '--'
> substring isn't a reliable determinant of a function's public/private
> nature, so can't be used as an alternative to an explicit (optional)
> declaration.
>
> My concern isn't the declaration, but the scope of its consequences. For
> example, an emacs user who does no elisp programming would be happy to
> have no private variable/function results appear among completion
> candidates when performing M-x describe-variable/function. Elisp
> programmers would want those completion candidates exposed, and subject
> to being extended/advised.

How would Emacs now if it was a programmer asking or not programmer
asking? :-)

Re: declare function/macro private

[tomas]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Tue, Jun 08, 2021 at 06:20:46AM +0200, Arthur Miller wrote:
> Boruch Baum <boruch_baum@gmx.com> writes:

[...]

> > My concern isn't the declaration, but the scope of its consequences. For
> > example, an emacs user who does no elisp programming would be happy to
> > have no private variable/function results appear among completion
> > candidates [...]

> How would Emacs now if it was a programmer asking or not programmer
> asking? :-)

That's exactly my take. Enforcing things because "I think they should
be this way" is not the kind of software I like to interact with.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: declare function/macro private

[Stefan Monnier]

> The point was just to support Paul Rankin's position that a '--'
> substring isn't a reliable determinant of a function's public/private
> nature, so can't be used as an alternative to an explicit (optional)
> declaration.

FWIW, I do not think a function should be considered non-internal just
because it has to be declared as interactive.

Similarly, the default value of a `foo-function` or `foo-hook` variable
can very much be an internal function, IMO.

The point of "internal" is that other code shouldn't use this directly
because it is subject to change, removal, or unexpected behavior if
called in a different context, ...

More concretely it means that end users are recommended not to do:

   (define-key company-active-map [my-key-sequence]
     #'company--select-next-and-warn)

If they want to use another key than `M-n` they should presumably do:

   (define-key company-active-map [my-key-sequence]
     (lookup-key company-active-map (kbd "M-n")))

[ I don't know if this use of "--" in company.el was a good idea,
  OTOH. ;-)  ]


        Stefan

declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

Given we have a function/macro declare interactive-only property, would it be worthwhile to consider a `private' property?

Of course there's already the convention of prefix--my-private-function, but my thinking here is that a program could declare a function/macro as private, then the compiler could signal a warning/error if that function appeared in a library outside the library it was defined and declared private.

e.g. in foo.el:

(defun foo-private ()
  (declare (private "use `foo-public' instead."))
  ...)

(provide 'foo)

..in bar.el:

(defun bar-function ()
  (foo-private))

Compiling bar.el would signal a warning/error with hint to use `foo-public' instead.

-- 
Paul W. Rankin
https://bydasein.com

The single best thing you can do for the world is to delete your social media accounts.

Re: declare function/macro private

[Omar Polo]

Paul W. Rankin via Emacs development discussions. <emacs-devel@gnu.org> writes:

> Given we have a function/macro declare interactive-only property, would it be worthwhile to consider a `private' property?
>
> Of course there's already the convention of prefix--my-private-function, but my thinking here is that a program could declare a function/macro as private, then the compiler could signal a warning/error if that function appeared in a library outside the library it was defined and declared private.
>
> e.g. in foo.el:
>
> (defun foo-private ()
>   (declare (private "use `foo-public' instead."))
>   ...)
>
> (provide 'foo)
>
> ..in bar.el:
>
> (defun bar-function ()
>   (foo-private))
>
> Compiling bar.el would signal a warning/error with hint to use `foo-public' instead.

but if foo.el and bar.el are part of the same package, wouldn't be fine
for bar.el to use internal functions from bar.el?

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On 6 Jun 2021, at 5:09 pm, Omar Polo <op@omarpolo.com> wrote:
> 
> but if foo.el and bar.el are part of the same package, wouldn't be fine
> for bar.el to use internal functions from bar.el?

Yep, in which case it would not be a good idea to declare them as private.

Re: declare function/macro private

[Lars Ingebrigtsen]

"Paul W. Rankin" via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Given we have a function/macro declare interactive-only property,
> would it be worthwhile to consider a `private' property?
>
> Of course there's already the convention of
> prefix--my-private-function, but my thinking here is that a program
> could declare a function/macro as private, then the compiler could
> signal a warning/error if that function appeared in a library outside
> the library it was defined and declared private.

The compiler could do this with "--" symbols if we wanted to, I guess.  

> e.g. in foo.el:
>
> (defun foo-private ()
>   (declare (private "use `foo-public' instead."))
>   ...)

I'm not necessarily against this or anything, but I do like the "--"
convention, because it makes it very explicit what's internal and what's
not.  On the other hand, if we had tagging like this, we could also
make (say) font-lock react to this information and make it as obvious as
the "--" convention.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On 6 Jun 2021, at 7:53 pm, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> "Paul W. Rankin" via "Emacs development discussions."
> <emacs-devel@gnu.org> writes:
> 
>> Given we have a function/macro declare interactive-only property,
>> would it be worthwhile to consider a `private' property?
>> 
>> Of course there's already the convention of
>> prefix--my-private-function, but my thinking here is that a program
>> could declare a function/macro as private, then the compiler could
>> signal a warning/error if that function appeared in a library outside
>> the library it was defined and declared private.
> 
> The compiler could do this with "--" symbols if we wanted to, I guess.  
> 
>> e.g. in foo.el:
>> 
>> (defun foo-private ()
>>  (declare (private "use `foo-public' instead."))
>>  ...)
> 
> I'm not necessarily against this or anything, but I do like the "--"
> convention, because it makes it very explicit what's internal and what's
> not.  On the other hand, if we had tagging like this, we could also
> make (say) font-lock react to this information and make it as obvious as
> the "--" convention.

Ah cool, I didn't think of font-lock, but that would be great if a function declared as internal appears outside of its program then it gets highlighted in font-lock-warning-face. Then the curious programmer calls `C-h f' on it and the help buffer could show e.g.

  This function is internal to <library> and should not be used
  in other lisp programs. Use <other-function> instead.

The "--" convention is fine and I wouldn't suggest to change/replace this; it's clear to people what it means once they've learnt what the convention means.

I think requiring a program to explicitly declare something as internal will cause less trouble than adding a kind of compiler heuristic for "--" symbol names because there are likely plenty of those where people have used the name with the perspective that it's just a hint and doesn't actually do anything. e.g. `imenu--index-alist' is supposedly internal but any elisp program hooking into Imenu needs to use this variable.

Also I just dislike computer heuristics.

I'll revise the initial suggestion from `private' to `internal' since that's more the Emacs lexicon, i.e.

  (declare (internal t))
  (declare (internal "use `other-function' instead."))

Re: declare function/macro private

[Lars Ingebrigtsen]

"Paul W. Rankin" <pwr@bydasein.com> writes:

> The "--" convention is fine and I wouldn't suggest to change/replace
> this; it's clear to people what it means once they've learnt what the
> convention means.

[...]

> Also I just dislike computer heuristics.

The "--" convention is heuristic-ey...  But it is very reminiscent of
the Common Lisp foo:bar vs foo::private-bar thing (which is not a
heuristic, but a language thing).

That is, I think it's very nice to have the code be quite explicit about
what's internal and what's external, so if you're using
foo--private-thing in Emacs outside of a file called foo*.el, you know
you're doing something that may break.

This being a convention instead of a language feature does have
problems.  For instance, if somebody decides to make a private function
public, all the usages have to be renamed.  (declare (private ...))
avoids that.

But...  I think I agree with people that this isn't really the way to
go -- the current convention provides more information than the declare
would do (most of all, it says what the prefix is).

It'd be nice if Emacs grew a real package system -- then this
private/public thing would happen naturally.  But I don't think adding
`private' in and of itself is the way forward.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: declare function/macro private

[Robin Tarsiger]

Lars Ingebrigtsen wrote:
> But...  I think I agree with people that this isn't really the way to
> go -- the current convention provides more information than the declare
> would do (most of all, it says what the prefix is).

FWIW, when I saw this thread, what I imagined was something like
(put 'foo--check-baz 'warn-if-used-outside '("foo-")) underneath,
probably with a function for applying that to a lot of symbols at
once. But the lack of a real package system could make that slow
if nothing else---AFAIK obarrays don't do any prefix indexing,
since they're described as hash tables?

-RTT

Re: declare function/macro private

[Stefan Monnier]

> Of course there's already the convention of prefix--my-private-function, but

What would be the difference between this convention and your proposed declaration?

> my thinking here is that a program could declare a function/macro as
> private, then the compiler could signal a warning/error if that function
> appeared in a library outside the library it was defined and
> declared private.

We don't have a definition for "library", sadly.

I think the "--" convention actually includes an extra information
compared to yours which could be useful here: we could warn for uses of
"foo--" if and only if the use appears in something whose name doesn't
start with "foo-".

IOW, use the string before the "--" as the definition of the "library"
(so that would support cases where a library is spread over several
files, as well as the case where a single file contains "internal
libraries").

Of course, if we enable such a warning now, we'll get a fairly large
number of warnings, I think ;-)

Another advantage of the "--" convention is that you don't need to load
the definition of that function in order to discover that it's private.


        Stefan

Re: declare function/macro private

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the "--" convention actually includes an extra information
> compared to yours which could be useful here: we could warn for uses
> of "foo--" if and only if the use appears in something whose name
> doesn't start with "foo-".
>
> IOW, use the string before the "--" as the definition of the "library"
> (so that would support cases where a library is spread over several
> files, as well as the case where a single file contains "internal
> libraries").

If my package was structured like

  foo.el
  foo-search.el
  foo-parse.el

I'd find it quite natural to name a private function in foo-search.el
`foo-search--do-stuff'.  Of course, it's ok if my `foo-base-do-stuff'
function from foo.el calls it but it would trigger a warning with your
approach because the "library" would be foo-search, no?

Bye,
Tassilo

Re: declare function/macro private

[Stefan Monnier]

Tassilo Horn [2021-06-06 20:12:49] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I think the "--" convention actually includes an extra information
>> compared to yours which could be useful here: we could warn for uses
>> of "foo--" if and only if the use appears in something whose name
>> doesn't start with "foo-".
>>
>> IOW, use the string before the "--" as the definition of the "library"
>> (so that would support cases where a library is spread over several
>> files, as well as the case where a single file contains "internal
>> libraries").
>
> If my package was structured like
>
>   foo.el
>   foo-search.el
>   foo-parse.el
>
> I'd find it quite natural to name a private function in foo-search.el
> `foo-search--do-stuff'.  Of course, it's ok if my `foo-base-do-stuff'
> function from foo.el calls it but it would trigger a warning with your
> approach because the "library" would be foo-search, no?

Indeed, the idea is that you'd have to use foo--search-do-stuff for
those functions which are private to your foo package but not to
foo-search.el, whereas you'd use foo-search--do-stuff for functions
which are not only private to your package but even private to
foo-search.el.


        Stefan

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On 7 Jun 2021, at 6:12 am, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> Tassilo Horn [2021-06-06 20:12:49] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I think the "--" convention actually includes an extra information
>>> compared to yours which could be useful here: we could warn for uses
>>> of "foo--" if and only if the use appears in something whose name
>>> doesn't start with "foo-".
>>> 
>>> IOW, use the string before the "--" as the definition of the "library"
>>> (so that would support cases where a library is spread over several
>>> files, as well as the case where a single file contains "internal
>>> libraries").
>> 
>> If my package was structured like
>> 
>>  foo.el
>>  foo-search.el
>>  foo-parse.el
>> 
>> I'd find it quite natural to name a private function in foo-search.el
>> `foo-search--do-stuff'.  Of course, it's ok if my `foo-base-do-stuff'
>> function from foo.el calls it but it would trigger a warning with your
>> approach because the "library" would be foo-search, no?
> 
> Indeed, the idea is that you'd have to use foo--search-do-stuff for
> those functions which are private to your foo package but not to
> foo-search.el, whereas you'd use foo-search--do-stuff for functions
> which are not only private to your package but even private to
> foo-search.el.

If there is a lake with a bunch of alligators in it, it would be a good thing to put up a sign that says "Danger: No swimming, Alligators".

Such a sign's sole purpose is to provide warning to people considering swimming in the alligator-infested lake, it does not actually prevent them from being eaten by alligators should they ignore the warning.

What I'm suggesting is to put up the sign, not to figure out a way to prevent alligators from eating swimmers who ignore the sign.

Re: declare function/macro private

[Arthur Miller]

"Paul W. Rankin" via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

>> On 7 Jun 2021, at 6:12 am, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>> Tassilo Horn [2021-06-06 20:12:49] wrote:
>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I think the "--" convention actually includes an extra information
>>>> compared to yours which could be useful here: we could warn for uses
>>>> of "foo--" if and only if the use appears in something whose name
>>>> doesn't start with "foo-".
>>>> 
>>>> IOW, use the string before the "--" as the definition of the "library"
>>>> (so that would support cases where a library is spread over several
>>>> files, as well as the case where a single file contains "internal
>>>> libraries").
>>> 
>>> If my package was structured like
>>> 
>>>  foo.el
>>>  foo-search.el
>>>  foo-parse.el
>>> 
>>> I'd find it quite natural to name a private function in foo-search.el
>>> `foo-search--do-stuff'.  Of course, it's ok if my `foo-base-do-stuff'
>>> function from foo.el calls it but it would trigger a warning with your
>>> approach because the "library" would be foo-search, no?
>> 
>> Indeed, the idea is that you'd have to use foo--search-do-stuff for
>> those functions which are private to your foo package but not to
>> foo-search.el, whereas you'd use foo-search--do-stuff for functions
>> which are not only private to your package but even private to
>> foo-search.el.
>
> If there is a lake with a bunch of alligators in it, it would be a good thing to put up a sign that says "Danger: No swimming, Alligators".
>
> Such a sign's sole purpose is to provide warning to people considering swimming in the alligator-infested lake, it does not actually prevent them from being eaten by alligators should they ignore the warning.
>
> What I'm suggesting is to put up the sign, not to figure out a way to prevent alligators from eating swimmers who ignore the sign.

Isn't a '--' that sign?

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On 7 Jun 2021, at 11:37 am, Arthur Miller <arthur.miller@live.com> wrote:
> 
> 
> Isn't a '--' that sign?

Not is this analogy no.

Re: declare function/macro private

[Arthur Miller]

"Paul W. Rankin" <pwr@bydasein.com> writes:

>> On 7 Jun 2021, at 11:37 am, Arthur Miller <arthur.miller@live.com> wrote:
>> 
>> 
>> Isn't a '--' that sign?
>
> Not is this analogy no.

Well, when I see a '--' in Emacs API I interpret that as internal
functions, not for use by me in my projects, unless I am hacking Emacs
itself. So for me it is a sign :). But maybe I don't understand what you
are trying to achieve.

Re: declare function/macro private

[Paul W. Rankin via Emacs development discussions.]

> On 7 Jun 2021, at 4:05 am, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> Of course there's already the convention of prefix--my-private-function, but
> 
> What would be the difference between this convention and your proposed declaration?

Stefan, you must have missed my followup reply:

..that would be great if a function declared as internal appears outside of its program then it gets highlighted in font-lock-warning-face. Then the curious programmer calls `C-h f' on it and the help buffer could show e.g.

 This function is internal to <library> and should not be used
 in other lisp programs. Use <other-function> instead.

The "--" convention is fine and I wouldn't suggest to change/replace this; it's clear to people what it means once they've learnt what the convention means.

I think requiring a program to explicitly declare something as internal will cause less trouble than adding a kind of compiler heuristic for "--" symbol names because there are likely plenty of those where people have used the name with the perspective that it's just a hint and doesn't actually do anything. e.g. `imenu--index-alist' is supposedly internal but any elisp program hooking into Imenu needs to use this variable.

Also I just dislike computer heuristics.

I'll revise the initial suggestion from `private' to `internal' since that's more the Emacs lexicon, i.e.

 (declare (internal t))
 (declare (internal "use `other-function' instead."))


>> my thinking here is that a program could declare a function/macro as
>> private, then the compiler could signal a warning/error if that function
>> appeared in a library outside the library it was defined and
>> declared private.
> 
> We don't have a definition for "library", sadly.

M-x find library says otherwise. But the definition of a "library" is inconsequential. Using "file" might be more helpful.

> I think the "--" convention actually includes an extra information
> compared to yours which could be useful here: we could warn for uses of
> "foo--" if and only if the use appears in something whose name doesn't
> start with "foo-".
> 

> IOW, use the string before the "--" as the definition of the "library"
> (so that would support cases where a library is spread over several
> files, as well as the case where a single file contains "internal
> libraries").
> 
> Of course, if we enable such a warning now, we'll get a fairly large
> number of warnings, I think ;-)
> 
> Another advantage of the "--" convention is that you don't need to load
> the definition of that function in order to discover that it's private.

I'm not suggesting to do away with using "--" as a handy visual marker. I use that too.

I would caution that trying to introduce a heuristic approach to interpret human intent from just "--" in a symbol name is the path to ruin.

As for the case of "libraries" over separate files, again it's probably best to use the word "file" instead of "library" to define the limits of the suggestion; consider declare-function as corollary:

  Tell the byte-compiler that function FN is defined, in FILE.

And `(declare (internal t))' would be:

  Tell the byte-compile that this function is internal to this file.

Re: declare function/macro private

[Stefan Monnier]

>>> Of course there's already the convention of prefix--my-private-function, but
>> What would be the difference between this convention and your proposed declaration?
> Stefan, you must have missed my followup reply:

No, I'm still wondering what it is you find to be different.

My current guess is that you fear that "--" has currently been used
carelessly and imposing a more "structured" meaning to it after the fact
will hence introduce problems, whereas your declaration would come right
away with an associated "precise" meaning.

If so, I agree to some extent, but:
- As mentioned your declaration would suffer from the difficulty of
  clarifying "internal to what" since sometimes a function is internal
  to a file, sometimes to a multifile package and sometimes to a subset
  of a file.  Of course, that can be solved by adding an extra argument
  to your annotation, but:
- your annotation is placed on those functions that are internal, which
  (as a coder) are those functions which I'd rather not burden with too
  much extra annotations, otherwise I'll just not bother declaring them
  as internal (which is what we had before the "--" convention).
- As long as the effect is just a few font-lock-warning faces here and
  there, I think the problem is harmless enough (and it does point to
  real misfeatures in many cases, so it would help improve the code).

> I think requiring a program to explicitly declare something as internal will
> cause less trouble than adding a kind of compiler heuristic for "--" symbol
> names because there are likely plenty of those where people have used the
> name with the perspective that it's just a hint and doesn't actually do
> anything. e.g. `imenu--index-alist' is supposedly internal but any elisp
> program hooking into Imenu needs to use this variable.

Yes, currently it's a bit of a wild west because the "--" has no effect,
but maybe it's time to impose some order on this, indeed.

> Also I just dislike computer heuristics.

It's not a heuristic.

>>> my thinking here is that a program could declare a function/macro as
>>> private, then the compiler could signal a warning/error if that function
>>> appeared in a library outside the library it was defined and
>>> declared private.
>> We don't have a definition for "library", sadly.
> M-x find library says otherwise. But the definition of a "library" is
> inconsequential. Using "file" might be more helpful.

But "file" is the wrong granularity for many Gnus-internal definitions
(and same applies to all multifile packages).  So what would use for
those if your declaration are only for file-local internal definitions?


        Stefan

Re: declare function/macro private

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Of course there's already the convention of prefix--my-private-function, but
>>> What would be the difference between this convention and your proposed declaration?
>> Stefan, you must have missed my followup reply:
>
> No, I'm still wondering what it is you find to be different.
>
> My current guess is that you fear that "--" has currently been used
> carelessly and imposing a more "structured" meaning to it after the fact
> will hence introduce problems, whereas your declaration would come right
> away with an associated "precise" meaning.
>
> If so, I agree to some extent, but:
> - As mentioned your declaration would suffer from the difficulty of
>   clarifying "internal to what" since sometimes a function is internal
>   to a file, sometimes to a multifile package and sometimes to a subset
>   of a file.  Of course, that can be solved by adding an extra argument
>   to your annotation, but:
> - your annotation is placed on those functions that are internal, which
>   (as a coder) are those functions which I'd rather not burden with too
>   much extra annotations, otherwise I'll just not bother declaring them
>   as internal (which is what we had before the "--" convention).
> - As long as the effect is just a few font-lock-warning faces here and
>   there, I think the problem is harmless enough (and it does point to
>   real misfeatures in many cases, so it would help improve the code).
>
>> I think requiring a program to explicitly declare something as internal will
>> cause less trouble than adding a kind of compiler heuristic for "--" symbol
>> names because there are likely plenty of those where people have used the
>> name with the perspective that it's just a hint and doesn't actually do
>> anything. e.g. `imenu--index-alist' is supposedly internal but any elisp
>> program hooking into Imenu needs to use this variable.
>
> Yes, currently it's a bit of a wild west because the "--" has no effect,
> but maybe it's time to impose some order on this, indeed.
>
>> Also I just dislike computer heuristics.
>
> It's not a heuristic.
>
>>>> my thinking here is that a program could declare a function/macro as
>>>> private, then the compiler could signal a warning/error if that function
>>>> appeared in a library outside the library it was defined and
>>>> declared private.
>>> We don't have a definition for "library", sadly.
>> M-x find library says otherwise. But the definition of a "library" is
>> inconsequential. Using "file" might be more helpful.
>
> But "file" is the wrong granularity for many Gnus-internal definitions
> (and same applies to all multifile packages).  So what would use for
> those if your declaration are only for file-local internal definitions?
>
>

I think one of problems with '--' is that is *relatively* new
convention. Lot's of old code in Emacs sources does not follow it, and
lot's of 3rd party packages does not follow it. Some older 3rd party
packages use different conventions like 'prefix/function-name' or
'prefix:function-name', where prefix is package name. As Ingebritsen
wrote in some other mail yesterday you could have Emacs highlight
syntax, but then there should be some way to tell Emacs what prefix is.

What OP propose is to have programmatic control over what is
"public/private", but in my opinion, as he suggestes, it would be quite
tedious to type for each and every "private" function an explicit
declaration as he proposes. People would probably not do it, just like
they don't care to write proper javadocs, doxygen docs and other
"proper" things that require explicit work.

But if Emacs can auto detect prefixes as "private", than it is
completely different game.

Also there should be some way to distinguish between code in same
package which shouldn't produce warnings and client code that uses
library where warnings should be emited if a "private" function is used.