use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Drew Adams]

The question you (Eli) raise is not specific to isearch or to delsel.  The
proper place to discuss it is emacs-devel - especially if you propose to change
things.  So I'm ccing emacs-devel - but bug #11381 should be dropped from the cc
list from now on.

> Actually, I very much dislike this "feature" of putting properties on
> function symbols, or at least the way we use it.  The problem is
> documentation:

We agree that that is the problem.  It is not something else.

> there's no good place visible to users where to
> document the possible values of these properties and their meanings,
> so they are only documented in the code.

Yes.  Documentation could be improved.

But the use of symbol properties, including for function symbols, is nothing
special.  It is similar to using attributes or methods with objects in OOP.
What's sometimes missing perhaps is better documentation/discoverability.

Emacs Lisp symbols are rudimentary objects, and, yes, there is not a lot of
built-in support for discovering or navigating them - how particular properties
are used in various contexts etc.

> This makes maintenance much harder and error-prone than
> it needs to be.

If Emacs maintainers substitute basic Emacs keys/commands here and there then
they need to be aware of `delete-selection-mode' - yes.  The same is true for
other modes that users might use - whether visual lines or whatever.  Changing
one thing can affect another.

The consequence for delsel if that advice is ignored is to risk making it a
no-op in some cases: you can neuter `delete-selection-mode' if you do not pay
attention when you replace commands with new ones (e.g. in some mode).


Wrt the user, and visibility of this information:

In some cases, the user (yes, user) is an Elisp programmer, and s?he would
sometimes need to consult the code anyway, depending on what s?he wants to do.

A user of delsel can be someone who never changes such a property on any
function symbol (most users), or it can be someone who does.  For the latter, it
would probably not be necessary to consult the delsel code if the documentation
included some information that is currently only in the delsel.el Commentary.

[This is not unique to delsel.el, BTW - there are other libraries for which a
programming user must consult the Commentary in order to understand - even when
it might not be necessary to read the code itself.  IOW, there are other cases
where some of the Commentary info might fruitfully be added to the doc.

Library thingatpt.el comes to mind - e.g., the doc strings where we say "See the
file `thingatpt.el' for documentation on how to define a symbol as a valid
THING."  That's almost as long as the info in the Commentary.]

Typically, someone minimally familiar with using delsel as a programmer can
easily answer a question from a user about what's happening or how to change the
behavior.  This info could be added to the `delete-selection-mode' doc string.
The doc could include info about programming/customizing the behavior.  That
info is not a big deal IMO, either to present or to understand.

Some libraries have whole Info manuals that delve into such
programming/customization aspects.  Library delsel.el is very simple, so that
info could just be added to the doc string for the mode.  The entire Commentary
text that explains it is only a few lines long (labeled "Interface"), and that
is all that is needed, I think.

> Look at delsel.el, for one such example.  The various uses of the
> related properties, scattered through half a dozen other Lisp
> packages, are completely obfuscated, and the only place you can really
> understand what these properties mean is by looking at the code in
> delsel.el that handles them, because even the commentary in delsel.el
> leaves a lot of place for improvement.

Not sure I agree, though it is hard to disagree that the application of
properties to function symbols is scattered.  It is of course the case that the
place to look for understanding delete-selection mode is function
`delete-selection-mode'.  And it is true that today that means looking also at
the delsel.el Commentary.  Moving that info to the doc string would help.

Can you be more specific about what is missing from the Commentary in delsel.el?
It seems to me that how delsel works and how to customize the behavior is pretty
well explained there, if a bit succinctly.  What else would you like to see
documented in this regard?

> I say, either we start documenting these properties in the doc strings
> and in the manual, or we should deprecate and discourage the practice
> of using them.

OK for the former.  I disagree with the latter.  Let's not throw out the baby
with the bathwater.

I really don't think this is a big deal.  I've seen a few user questions about
delsel behavior in help-gnu-emacs over the years, and IIRC all of them had an
obvious explanation to someone familiar with the behavior of delsel (which is
quite simple).  Putting the info about that behavior into the doc string would
be helpful - and sufficient, I think.

But of course that won't stop a few "end" users from having the occasional
question because some library they picked up effectively customizes delsel
behavior without saying so.

A typical question comes from someone who already uses delsel (and who is used
to using it), who then picks up a library that changes some key bindings,
effectively replacing some commands that are handled by delsel with some
commands that are not.

This case is not a big deal.  Such a user can tell right away that something has
changed the delsel behavior, and a quick question to help-gnu-emacs - or a quick
check of the new doc string - would likely turn on the light.

But a user who has never used delsel before, and who already uses some library
that substitutes keys/commands so that delsel becomes a no-op for them, will
perhaps not understand why delsel does not seem to behave as advertised.  IOW,
trying to understand the expected vanilla behavior of delsel (having never used
it), and trying to understand a modified behavior at the same time, might be
more difficult.

But even that user, if s?he consults the new doc string, would have a good
chance of understanding the situation.

Attaching properties to Lisp function symbols in order to affect behavior is as
old as Lisp symbols themselves.  Please do not try to discourage this practice.
But it is fine to try to improve the documentation or discoverability.

[Belongs in a separate thread, at another time, but if `delete-selection-mode'
were the default (yes) then: (a) users new to Emacs would find behavior much
closer to what they are used to, and (b) Emacs developers (including 3rd-party)
would be more conscious of delsel, so would likely DTRT wrt it more often.]

(Just one opinion.)

Re: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <11381@debbugs.gnu.org>, <emacs-devel@gnu.org>
> Date: Mon, 28 May 2012 10:34:34 -0700
> 
> The question you (Eli) raise is not specific to isearch or to
> delsel.

It was relevant to proliferation of the technique discussed there.

> But the use of symbol properties, including for function symbols, is nothing
> special.  It is similar to using attributes or methods with objects in OOP.
> What's sometimes missing perhaps is better documentation/discoverability.

That's what I was trying to say.  "Discoverability" means that someone
who calls a function from Org mode (say) that invokes some magic via
delsel should be able to understand how that magic happened.  I don't
see how this can be done today without a lot of digging.  If you are
hard-pressed for time, this could be a maintenance nightmare.

> Can you be more specific about what is missing from the Commentary in delsel.el?

It basically doesn't explain anything apart of the fact that the
property can have several values.  The exact effect of each value is
left unclear to the degree of obfuscation.  And the names of the
values are similarly non-specific.

What is needed is a clear description what each value does, and it
should be part of the doc string.  Right now, this info is only
available by reading the code.

RE: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Drew Adams]

> > Can you be more specific about what is missing from the 
> > Commentary in delsel.el?
> 
> It basically doesn't explain anything apart of the fact that the
> property can have several values.  The exact effect of each value is
> left unclear to the degree of obfuscation.

The effect of each value is described, though perhaps imperfectly and not to
your liking.

Again, what specifically do you feel is missing?  Just repeating "obfuscation"
does not help.  Please specify the information that you would add.

> And the names of the values are similarly non-specific.

The names do not stand alone without their descriptions.  Even so, I think that
`yank' and `kill' are pretty clear even without reading their descriptions.
`supersede' is not so obvious, but its description seems clear.

"non-nil" should be "anything else non-nil".  And we should say that this is the
usual case.

FWIW, in my version of delsel.el I changed the wording a tiny bit, to improve it
slightly:

;; The property can have one of these values:
;;
;;  `yank' - For commands that do a yank. Ensures that the region
;;           about to be deleted is not yanked.
;;
;;  `supersede' - Delete the active region and ignore the current
;;           command: the command just deletes the region.
;;
;;  `kill' - `kill-region' is used on the selection, rather than
;;           `delete-region'.  Text selected with the mouse is
;;           typically yankable anyway.
;;
;;  anything else non-nil - Deletes the active region prior to
;;           executing the command, which inserts replacement
;;           text. This is the usual case.

I'm not saying the existing descriptions should not be changed.  I'm asking you
what changes you have in mind.

> What is needed is a clear description what each value does,

For instance?  What is missing, in your opinion?

> and it should be part of the doc string.  Right now, this info is only
> available by reading the code.

Agreed.

Re: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <monnier@iro.umontreal.ca>, <emacs-devel@gnu.org>
> Date: Mon, 28 May 2012 11:26:35 -0700
> 
> > And the names of the values are similarly non-specific.
> 
> The names do not stand alone without their descriptions.

I'm saying that the names don't help to understand the descriptions,
that's all.  If the descriptions were clear, I would care much less
for the names.  As things are, I'm grasping at every straw in sight.

>  Even so, I think that `yank' and `kill' are pretty clear even
> without reading their descriptions.

Not to me.  "For commands that do a yank" is not a description, it's
tautology at best.  "Ensures that the region about to be deleted is
not yanked"? why would it be yanked otherwise? and why the value is
'yank' if it "ensures the region will NOT be yanked"?  That just
doesn't make any sense!

'kill'? let's see... "`kill-region' is used on the selection, rather
than `delete-region'".  So? why would that matter, and when would I
want the former rather than the latter?  And what is "Text selected
with the mouse is typically yankable anyway" about? why "anyway"?

Etc., etc.  These are riddles, or maybe notes for someone who once
knew everything about this but forgot.  This is not documentation
suitable for first-time reading.  It leaves too much to previous
knowledge.

> I'm not saying the existing descriptions should not be changed.  I'm asking you
> what changes you have in mind.

How can I explain what I don't understand?  I _really_ have no clear
idea about what each value does, can you believe that?  If I'd study
the code long enough, I'd eventually understand, but I didn't.

> > What is needed is a clear description what each value does,
> 
> For instance?  What is missing, in your opinion?

I don't know.  How about if you try explaining each value in full,
from scratch?  Assume that only primitives like kill-region,
delete-region, 'yank', etc. are known, all the rest is not.  Explain
the effect of each value, both on the text that was selected and on
any other text that's involved.  Give at least one example for a use
case where each value will be useful.  That kind of stuff.

RE: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Drew Adams]

> > Even so, I think that `yank' and `kill' are pretty clear even
> > without reading their descriptions.
> 
> Not to me.  "For commands that do a yank" is not a description, it's
> tautology at best.  "Ensures that the region about to be deleted is
> not yanked"? why would it be yanked otherwise? and why the value is
> 'yank' if it "ensures the region will NOT be yanked"?  That just
> doesn't make any sense!

It does makes sense, I think, but perhaps only after you understand what it
does, not necessarily as a lesson. ;-)

Let me try to do a little better.

Whenever the value of property `delete-selection' for a command symbol is
non-nil, and you are in `delete-selection-mode', the text of the active region
is removed.  Always.  That is the first thing to understand.

This removal happens before - or instead of - the action normally performed by
the command.  That is the second thing to understand.

So in the above text, "the region about to be deleted" refers to the deletion of
the active region, which takes place before the yank command does its own thing
(yanks).

And "ensures the region will NOT be yanked" means that the property value `yank'
ensures that when the yank command does its yanking it does not just yank the
exact same text that was just deleted - the active region text - even if that
was the text that the kill ring offered.

That's an attempt to explain the existing commentary text.  Some more help
below.

> 'kill'? let's see... "`kill-region' is used on the selection, rather
> than `delete-region'".  So? why would that matter, and when would I
> want the former rather than the latter?

You would use `kill' when you want the active region text killed instead of
being simply deleted.  That's all that that is trying to say.

> And what is "Text selected with the mouse is typically
> yankable anyway" about?  why "anyway"?

No idea, sorry.

> > I'm not saying the existing descriptions should not be 
> > changed.  I'm asking you what changes you have in mind.
> 
> How can I explain what I don't understand?  I _really_ have no clear
> idea about what each value does, can you believe that?  If I'd study
> the code long enough, I'd eventually understand, but I didn't.

I see.  I thought you were saying before that you understood the code, and that
it is necessary to study the code in order to understand.  I misunderstood
probably from your saying this:

  the only place you can really understand what these properties
  mean is by looking at the code in delsel.el that handles them,
  because even the commentary in delsel.el leaves a lot of place
  for improvement.

I took that to mean that you "really understand" the code, and that the
commentary description does not live up to such an understanding.  My bad.

> > > What is needed is a clear description what each value does,
> > For instance?  What is missing, in your opinion?
> 
> I don't know.  How about if you try explaining each value in full,
> from scratch?  Assume that only primitives like kill-region,
> delete-region, 'yank', etc. are known, all the rest is not.  Explain
> the effect of each value, both on the text that was selected and on
> any other text that's involved.  Give at least one example for a use
> case where each value will be useful.  That kind of stuff.

See if this helps at all -

In all cases when `delete-selection-mode' is turned on and the
`delete-selection' value of a command symbol is non-nil, the text of
the active region is removed.  It is either deleted or killed (i.e.,
deleted and moved to the `kill-ring').  It is killed only if the value
of property `delete-selection' is `kill'.

The action of the invoked command itself is either carried out (after
deletion of the active region), which is the case for
`delete-selection' values `yank' and `t', or not (it is ignored),
which is the case for values `supersede' and `kill'.

`delete-selection' value `yank' ensures that the invoked yanking
command does not yank the same text as that in the region.  The
kill ring is rotated if necessary, so that the yank operation is not a
no-op.

Here is a summary of the `delete-selection' property values.  You can apply a
value to a command symbol to make the command delete the active region text and
perhaps do something more:

`kill'      - Use it if you want the command to kill the region text,
              so you can yank it later.

              The command's own operation does NOT take place - it is
              superseded by the region killing.

              Typically, the command itself is a kill command: its
              own killing is replaced by killing the active region.

`supersede' - Use it for a command that deletes text that you do not
              want to yank later.

              The command's own operation does NOT take place - it is
              superseded by the region deletion.

              Typically, the command is a deletion command: its own
              deletion is replaced by deleting the active region.

`t'         - Use it for a command that inserts text.

              The command's insertion operation takes place after
              deleting the active region.

`yank'      - Use it for a command that yanks text.

              The command's yank operation takes place after deleting
              the active region.

              If the kill-ring entry to be yanked has the same text as
              the region, then first rotate the kill ring so that the
              yank will not be a no-op.

`nil'       - Invoke the command normally.  Delete-selection mode does
              nothing for a command whose `delete-selection' property 
              value is nil.

That is a summary, but it also outlines the important use cases.  In fact, there
is nothing that requires you to use only a deletion command with `supersede' or
only an insertion command with `t', etc.

Example use cases:

* If the command action is to kill some text, then you might want it
  to kill the region instead, when it is active.  In this case, use 
  `kill' as the `delete-selection' value.

* If the command action is to delete some text without killing it, 
  then you might want it to delete the active region instead (without
  killing it).  In this case, use `supersede' as the
  `delete-selection' value.

* If the command action is to insert some text, you typically want 
  that text to replace the active region.  In this case, use `t' as
  the `delete-selection' value.

* If the command action is to yank some text, then use `yank' as
  the `delete-selection' value.

Concrete examples from Org mode:

 `org-self-insert-command' gets value `t' (i.e., replace)
 `org-delete-char' gets value `supersede' (i.e., just delete)
 `org-yank' gets `value yank' (i.e., replace, but prevent a no-op)

FWIW, Org mode does EXACTLY the same thing for handling `flyspell-mode': It puts
property `flyspell-delayed' on commands `org-self-insert-command' and
`org-delete-char'.  And it does EXACTLY the same thing for handling
`pabbrev-mode': It puts property `pabbrev-expand-after-command' on command
`org-self-insert-command'.

The attachment of properties to function symbols in order to affect behavior in
some mode is not a new idiom.  And it is not limited to Emacs Lisp.



In any case, this is all apparently moot, since Stefan appears to want to do
away with this (simple & elegant) mechanism and instead do something like what
was done recently for shift-select or even his completion tables.
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11381#62

If you had trouble understanding the delsel approach and code, well, then ...
bonjour les degats when it is replaced by a concoction a la completion tables.
;-)  And good luck with the relevant doc strings and Commentary.

HTH.

Re: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <monnier@iro.umontreal.ca>, <emacs-devel@gnu.org>
> Date: Tue, 29 May 2012 17:10:33 -0700
> 
> > > Even so, I think that `yank' and `kill' are pretty clear even
> > > without reading their descriptions.
> > 
> > Not to me.  "For commands that do a yank" is not a description, it's
> > tautology at best.  "Ensures that the region about to be deleted is
> > not yanked"? why would it be yanked otherwise? and why the value is
> > 'yank' if it "ensures the region will NOT be yanked"?  That just
> > doesn't make any sense!
> 
> It does makes sense, I think, but perhaps only after you understand what it
> does, not necessarily as a lesson. ;-)
> 
> Let me try to do a little better.

Thanks.  It would be nice to have all that in a doc string and/or a
manual near me, some time soon.

>               If the kill-ring entry to be yanked has the same text as
>               the region, then first rotate the kill ring so that the
>               yank will not be a no-op.

This could use some more clarification.  E.g., does it rotate the kill
ring until there's some entry that is different from the text in the
region, or just once?  The "so the yank will not be a no-op" part is
just the rationale, so it should be worded as such (e.g., in parens).

RE: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Drew Adams]

> Thanks.  It would be nice to have all that in a doc string and/or a
> manual near me, some time soon.

+1 - that information/text or similar

> > If the kill-ring entry to be yanked has the same text as
> > the region, then first rotate the kill ring so that the
> > yank will not be a no-op.
> 
> This could use some more clarification.  E.g., does it rotate the kill
> ring until there's some entry that is different from the text in the
> region, or just once?

Just once.  But it would no doubt be better to instead rotate through the list
until we reach a non-identical entry (if there is one).

> The "so the yank will not be a no-op" part is just the rationale,
> so it should be worded as such (e.g., in parens).

Agreed.  But if improved as suggested then it would become the stop-rotating
condition: "rotate the kill ring until the first kill is different" etc.  In
that case it need not be in parens.

If there is no non-identical ring entry then punt (the yank operation becomes a
no-op).

RE: use and doc of function symbol properties [was: bug#11381: 23.3; isearch-search-and-update issue?]

[Drew Adams]

> But it would no doubt be better to instead rotate through
> the list until we reach a non-identical entry (if there is one).

I.e.,
(cond ...

 ((eq type 'yank)
  ;; Before a yank command.  Make sure we do not yank
  ;; the same region that we are going to delete.
  ;; That would make yank a no-op.
  (let ((tail  kill-ring))
    (while (and tail  (string= (buffer-substring-no-properties
                                (point) (mark))
                               (car tail)))
      (current-kill 1)
      (setq tail  (cdr tail))))
  (delete-active-region))

 ...)