bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Drew Adams]

See https://emacs.stackexchange.com/q/47302/105, as one possible
motivaton.

The only doc I can find about making Isearch and `perform-replace' (all
of its uses) ignore/exclude certain matches is the doc string of
variable `isearch-filter-predicate'.

And that doc string isn't very precise about the args of the predicate.
It says only: "The function has two arguments: the positions of start
and end of text matched by the search."

It would help to add that these positions are `(match-beginning 0)' and
`(match-end 0)', respectively, and to say that the match start position
is the first of the two args.  (Sure, start coming first is not
surprising, but it also doesn't follow from the description.)

I suggest adding a short topic about filtering with this predicate,
perhaps with a simple example.  At least mention that this is used in
the predefined search commands (including Isearch) and the predefined
replacement commands.

It would also be good to state whether predefined search functions such
as `re-search-forward' respect it.  (I imagine that they do not, but I
haven't checked, and there's no doc about this AFAIK.)  You could guess
no, based on the `isearch' part of the variable name.  But if you guess
like that then you likely won't also guess that the variable applies to
`perform-replace' - it's not just about Isearch.

One thing that it would also be good to make extra clear is that
filtering takes place _after_ input matching; it is not part of
matching.  Not getting this can be a gotcha with greedy regexp matching.
For example, a filter predicate that excludes matches that extend past
column 70 does not keep the part of a match before column 70, even if
that part also matches the same regexp.

In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-30
Repository revision: 07f8f9bc5a51f5aa94eb099f3e15fbe0c20ea1ea
Windowing system distributor `Microsoft Corp.', version 10.0.17134
Configured using:
 `configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Eli Zaretskii]

> Date: Sun, 20 Jan 2019 16:17:41 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> 
> See https://emacs.stackexchange.com/q/47302/105, as one possible
> motivaton.
> 
> The only doc I can find about making Isearch and `perform-replace' (all
> of its uses) ignore/exclude certain matches is the doc string of
> variable `isearch-filter-predicate'.

I don't see why the doc string shouldn't be enough.  This is a quite
obscure feature, so I don't think it warrants to be described in the
manual.

> And that doc string isn't very precise about the args of the predicate.
> It says only: "The function has two arguments: the positions of start
> and end of text matched by the search."
> 
> It would help to add that these positions are `(match-beginning 0)' and
> `(match-end 0)', respectively, and to say that the match start position
> is the first of the two args.

To me, "the positions of start and end of the matched text" says
precisely that.  I don't see what can references to match-beginning
and match-end add; if anything, they might confuse, because at least
some readers will be sent down the rabbit hole to the descriptions of
those two, something that IMO is entirely unnecessary for writing a
filter.

> It would also be good to state whether predefined search functions such
> as `re-search-forward' respect it.  (I imagine that they do not, but I
> haven't checked, and there's no doc about this AFAIK.)  You could guess
> no, based on the `isearch' part of the variable name.  But if you guess
> like that then you likely won't also guess that the variable applies to
> `perform-replace' - it's not just about Isearch.

I modified the doc string to mention Isearch and replace commands.

> One thing that it would also be good to make extra clear is that
> filtering takes place _after_ input matching; it is not part of
> matching.

How can it be part of matching, if the filter needs to be passed the
limits of the matched text?

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Drew Adams]

> > The only doc I can find about making Isearch and `perform-replace'
> (all
> > of its uses) ignore/exclude certain matches is the doc string of
> > variable `isearch-filter-predicate'.
> 
> I don't see why the doc string shouldn't be enough.  This is a quite
> obscure feature, so I don't think it warrants to be described in the
> manual.

I disagree that it is obscure - or that it should be,
at least.  But of course if it is not well documented
and it is (still) hardly ever used by Emacs itself
then it will not necessarily be noticed, understood,
and used.  Its current obscurity is a self-fulfilling
prophecy.

> > It would also be good to state whether predefined search functions
> > such as `re-search-forward' respect it.  (I imagine that they do
> > not, but I haven't checked, and there's no doc about this AFAIK.)
>
> I modified the doc string to mention Isearch and replace commands.

Thanks.  And non-command functions such as `re-search-forward'?

> > One thing that it would also be good to make extra clear is that
> > filtering takes place _after_ input matching; it is not part of
> > matching.
> 
> How can it be part of matching, if the filter needs to be passed the
> limits of the matched text?

No one contests that impossibility.

But it and its consequences are not necessarily
obvious - especially to a user searching, as opposed
to a programmer writing a filter predicate.

Isearch's handling of filter limits is very different
from its handling of buffer limits, for example.  A
filter doesn't constrain search to be inside its
limits, in the sense that the search matches take no
account of such limits.  This is not necessarily
obvious to a _user_.  (It might or might not be clear
to some programmer who defines the filter.)

You can easily notice this as a user if you try to
regexp-search when a filter excludes text outside a
rectangle or a set of columns, for instance.

A user could easily, and incorrectly, expect the
rectangle to "contain" search, similarly to how a
buffer restriction contains it.  She could ask herself
how it is that a regexp with `.*' doesn't "match" some
particular text within the rectangle, the answer being
that it instead matched longer text that extended
outside the rectangle, and that match was filtered out.

If this user-level description makes no sense to you
I expect it's because you haven't played with filters
enough or, more likely, because you start from an
understanding of the code - which a user does not.

Just emphasizing explicitly that filtering takes
place _after_ input matching can help, I think.  As
you know, filtering can in general be done before or
during querying/searching/matching, as well as after
it.  IMO, it's worth emphasizing that this is only
post-match filtering.

If you ask why a _user_ needs to know about filter
predicates and filtering then my answer is longer.
I'll leave that out, unless you ask for it.

Using `isearch-filter-predicate' can be powerful.
But it is also somewhat limited/weak because
filtering cannot be taken into account as part of
matching.

Imagine being able to contain search within a
rectangle, for instance.  That's something you
cannot do with only `isearch-filter-predicate'.

Whether or not such limitation is obvious to a
particular filter writer, it certainly is not
obvious to a filter user, I think.

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Eli Zaretskii]

> Date: Mon, 21 Jan 2019 10:18:23 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 34150-done@debbugs.gnu.org
> 
> > I don't see why the doc string shouldn't be enough.  This is a quite
> > obscure feature, so I don't think it warrants to be described in the
> > manual.
> 
> I disagree that it is obscure - or that it should be,
> at least.

??? Most searches don't need any filtering at all.

> > I modified the doc string to mention Isearch and replace commands.
> 
> Thanks.  And non-command functions such as `re-search-forward'?

Not primitives, no.  This is a Lisp-only (application-level) feature.

> > > One thing that it would also be good to make extra clear is that
> > > filtering takes place _after_ input matching; it is not part of
> > > matching.
> > 
> > How can it be part of matching, if the filter needs to be passed the
> > limits of the matched text?
> 
> No one contests that impossibility.
> 
> But it and its consequences are not necessarily
> obvious - especially to a user searching, as opposed
> to a programmer writing a filter predicate.

This is not a user-level facility, so the user perspective is not
relevant.

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Drew Adams]

> > But it and its consequences are not necessarily
> > obvious - especially to a user searching, as opposed
> > to a programmer writing a filter predicate.
> 
> This is not a user-level facility, so the user perspective is not
> relevant.

The "user perspective" is always relevant for Emacs.
We are all users.

Searching is a user-level facility.  If a search
imposes/provides filtering, that certainly affects
user-visible behavior.  Users should understand
that behavior.

Anyone imposing such filtering on users should
consider its effects on users, including the
considerations raised by this bug report.  That
starts with making the behavior and consequences
clear to filter implementors.

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Eli Zaretskii]

> Date: Mon, 21 Jan 2019 10:40:36 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 34150@debbugs.gnu.org
> 
> > > But it and its consequences are not necessarily
> > > obvious - especially to a user searching, as opposed
> > > to a programmer writing a filter predicate.
> > 
> > This is not a user-level facility, so the user perspective is not
> > relevant.
> 
> The "user perspective" is always relevant for Emacs.
> We are all users.
> 
> Searching is a user-level facility.  If a search
> imposes/provides filtering, that certainly affects
> user-visible behavior.  Users should understand
> that behavior.
> 
> Anyone imposing such filtering on users should
> consider its effects on users, including the
> considerations raised by this bug report.  That
> starts with making the behavior and consequences
> clear to filter implementors.

All true, but the filtering itself is not on the user level: the
_results_ of the filtering are.  So the behavior on the user level
should be described and understood by users in higher-level terms.
For example, with the default filtering, the behavior should be
described in terms of searching inside invisible text, not in terms of
filtering out some of the hits; the latter is just the implementation,
not the user-level behavior.

Therefore, the details of how filtering of matches works, and how to
write a filter, is not user-level information.  If you re-read what
you wrote above, you will see that your arguments are all consistent
with what I said, they don;'t contradict that in any way.

bug#34150: 26.1; Document filtering with `isearch-filter-predicate' in Elisp manual

[Drew Adams]

> > > This is not a user-level facility, so the user perspective is not
> > > relevant.
> >
> > The "user perspective" is always relevant for Emacs.
> > We are all users.
> >
> > Searching is a user-level facility.  If a search
> > imposes/provides filtering, that certainly affects
> > user-visible behavior.  Users should understand
> > that behavior.
> >
> > Anyone imposing such filtering on users should
> > consider its effects on users, including the
> > considerations raised by this bug report.  That
> > starts with making the behavior and consequences
> > clear to filter implementors.
> 
> All true, but the filtering itself is not on the user level: the
> _results_ of the filtering are.  So the behavior on the user level
> should be described and understood by users in higher-level terms.

Correct.  It should be.

But those who implement filter predicates also need
to be aware of the behavior, and to think in user
terms, in order to think of documenting it for their
users.

I know, speaking as one such implementor.  When the
resulting filtering behavior does not have an obvious
explanation without understanding something about
filtering, an explanation for users is called for.

> For example, with the default filtering, the behavior should be
> described in terms of searching inside invisible text, not in terms of
> filtering out some of the hits; the latter is just the implementation,
> not the user-level behavior.

Uh, no, aside from Isearch being able to "open" hidden
text containing matches - which is something else again,
use of such filtering by Isearch is not about searching
inside INvisible text.  It's about filtering out some
of what would otherwise be considered search hits.

Aside from the exceptional behavior of being able to
"open" invisible text, search with filtering is about
search visible, not invisible, text.

Not to mention that users can, themselves, add filters
to exclude searchable text.  (With my Isearch+ code
they can even do that on the fly, interactively.)

Thinking in terms of filtering, excluding possible
matches is entirely appropriate at the user level.

It's akin to narrowing a set of completion candidates
by progressively imposing additional match constraints.

And even for completion there are two possibilities
of filtering candidates: before matching user input
or afterward.  There are specific advantages to each.
And yes, when one or the other is employed (or both)
it can be important for users to know this, as it can
affect not only what they see but how they choose to
interact with the completion UI.

The grain of truth in what you say is that in some
cases users need not be aware of _some_ filtering
that goes on.

And even in those cases the possibility of their
ignorance can depend on the kind of matching
employed.  See my example of regexp matching within
a rectangle - no such problem for simple string
matching.

For simple, non-regexp searching users generally
need not think in terms of (1) searching the whole
buffer for matches and then (2) excluding matches
that extend beyond the visible text.

In that case a user-level description wouldn't
need to mention filtering at all.  Not so for the
regexp-search case, however.  Users need some
description/explanation to provide them a conceptual
model that explains the behavior they'll run into. 

> Therefore, the details of how filtering of matches works, and how to
> write a filter, is not user-level information.

Certainly user-visible behavior should be described
to users in user terms, not implementation terms.
The implementation is irrelevant to them (unless
they dig into Lisp during interaction).

But a user description here must cover what I said,
one way or another.  They need to know why their
regexp of .* does not find a match in the visible
text, when they can see a match for it.

Their conceptual model of what's happening, what's
going on, needs to make this clear to them.  It need
_not_ be in terms that mirror the implementation.
But it needs to describe/explain the behavior they
see, one way or another.

> If you re-read what
> you wrote above, you will see that your arguments are all consistent
> with what I said, they don;'t contradict that in any way.

Dunno what expected contradictions you want me to
look for.  I would like to see doc that makes users
aware of the behavior they run into.  Today that's
not the case.

And a separation of programmer-level from user-level
doc does not preclude making programmers themselves
aware of what behavior users need to expect or know
about.  If they don't themselves understand the
behavior then they can't manage it well or explain
it to their users.