how to control isearch for invisible text

how to control isearch for invisible text

[Drew Adams]

It seems like there should be some way to control whether isearch matches
text that is invisible or ignores it. I couldn't find anything on this in
either the Emacs manual or the Elisp manual. Is there no such option?

I came across `isearch-open-invisible-temporary' in the Elisp manual, but I
coudn't tell if that was related. Its explanation speaks of "overlays", even
though the rest of the same Info node speaks of text properties (?). Plus,
it seems to be a property of the overlay itself, rather than an option for
how isearch should treat invisible text generally.

If there is no option to control isearch's sensitivity to invisible text,
then how about adding one? And adding an isearch keymap binding for toggling
the option while searching. It would be a plus if the same option could act
similarly for query-replace and occur. For query-replace, perhaps the same
toggle binding could be used.

Please let me know if something like this is already available. In that
case, it probably needs better doc (e.g. Emacs manual).

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

> It seems like there should be some way to control whether isearch matches
> text that is invisible or ignores it. I couldn't find anything on this in
> either the Emacs manual or the Elisp manual. Is there no such option?
>
> I came across `isearch-open-invisible-temporary' in the Elisp manual, but I
> coudn't tell if that was related. Its explanation speaks of "overlays", even
> though the rest of the same Info node speaks of text properties (?). Plus,
> it seems to be a property of the overlay itself, rather than an option for
> how isearch should treat invisible text generally.
>
> If there is no option to control isearch's sensitivity to invisible text,
> then how about adding one? And adding an isearch keymap binding for toggling
> the option while searching. It would be a plus if the same option could act
> similarly for query-replace and occur. For query-replace, perhaps the same
> toggle binding could be used.
>
> Please let me know if something like this is already available. In that
> case, it probably needs better doc (e.g. Emacs manual).

M-x apropos-documentation RET isearch.*invisible RET

If moving the mouse over text changes the text's color, you can click
mouse-2 (second button from right) on that text to get more information.
In this buffer, go to the name of the command, or function, or variable,
and type RET to get full documentation.


----------------
search-invisible
  Variable: If t incremental search can match hidden text.
A nil value means don't match invisible text.
When the value is `open', if the text matched is made invisible by
an overlay having an `invisible' property and that overlay has a property
`isearch-open-invisible', then incremental search will show the contents.
(This applies when using `outline.el' and `hideshow.el'.)
See also `reveal-mode' if you want overlays to automatically be opened
whenever point is in one of them.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    > It seems like there should be some way to control whether
    > isearch matches text that is invisible or ignores it.

    > If there is no option..., then how about adding one?
    > And adding an isearch keymap binding for toggling the
    > option while searching.

    > It would be a plus if the same option could act
    > similarly for query-replace and occur. For query-replace,
    > perhaps the same toggle binding could be used.

    M-x apropos-documentation RET isearch.*invisible RET

Thanks.

So, now, how about:

- documenting `isearch-invisible' in the Emacs manual?
- and the Elisp manual?
- creating a toggle for it?
- having `occur' and `query-replace' respect the option?

Also, I'm curious why the treatment of an `invisible' text-property value of
`isearch-open-invisible' is limited to overlays. Why shouldn't the same
behavior hold for the property if applied directly to text (that is, without
an overlay)?

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     > It seems like there should be some way to control whether
>     > isearch matches text that is invisible or ignores it.
>
>     > If there is no option..., then how about adding one?
>     > And adding an isearch keymap binding for toggling the
>     > option while searching.
>
>     > It would be a plus if the same option could act
>     > similarly for query-replace and occur. For query-replace,
>     > perhaps the same toggle binding could be used.
>
>     M-x apropos-documentation RET isearch.*invisible RET
>
> Thanks.
>
> So, now, how about:
>
> - documenting `isearch-invisible' in the Emacs manual?

It is `search-invisible'.  I consider it far too special to be
documented in the Emacs manual: this is a variable that should, if at
all, be used by the modes or packages making stuff invisible.

> - and the Elisp manual?

Not sure whether it is important enough for that.

> - creating a toggle for it?

Why?  It is not a user-level feature.  Providing a toggle would be the
task of any mode that makes stuff invisible and that would require to
have it accessible.

> - having `occur' and `query-replace' respect the option?

Sounds sort of reasonable, at least with query-replace.  With occur, I
am less sure, since it is sort of an internal grep.  But since occur
could not reasonably display something useful without making it
visible, it might be an idea.

> Also, I'm curious why the treatment of an `invisible' text-property
> value of `isearch-open-invisible' is limited to overlays. Why
> shouldn't the same behavior hold for the property if applied
> directly to text (that is, without an overlay)?

Because you can make some text visible without modification when there
is an invisible overlay over it.  If there are invisible text
properties, you can't remove them without changing the buffer, and
searching should not change the buffer.

Fiddling with invisibility-spec would affect all invisible regions,
not just the one at point.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    > So, now, how about:
    >
    > - documenting `isearch-invisible' in the Emacs manual?

    It is `search-invisible'.

Oops; sorry.

    I consider it far too special to be
    documented in the Emacs manual: this is a variable that should, if at
    all, be used by the modes or packages making stuff invisible.

Then why is it a defcustom? Why can you customize it?

I don't think it's special, or I wouldn't, as a user, have asked for such a
feature. I don't see why this shouldn't be useful for users. When I'm
searching, and there is some invisible text (which I may or may not know),
I'd like to be able to toggle isearch, to make it (in)sensitive to that
text.

    > - and the Elisp manual?

    Not sure whether it is important enough for that.

I think it's especially important for the Emacs manual.

    > - creating a toggle for it?

    Why?  It is not a user-level feature.

It's a user option - customizable. And so it should be, IMO. That's why I
was asking for such an option - as a user. I want to interactively control
whether isearch hits hidden text. One use is to find whether something
particular is perhaps present but hidden(!).

    Providing a toggle would be the
    task of any mode that makes stuff invisible and that would require to
    have it accessible.

Why would each mode do that? Why not have a general toggle for isearch - the
same binding for all modes? If I have a mode that hides dired details, that
mode will provide a toggle to show/hide the details, but it shouldn't have
to also provide a toggle for isearch sensitivity to hidden text. If each
mode does that, then users will not have a single, consistent binding to
rely on - and the toggle should be available while isearching. To me, this
is no different from toggling case-sensitivity or regexp sensitivity.

    > - having `occur' and `query-replace' respect the option?

    Sounds sort of reasonable, at least with query-replace.  With occur, I
    am less sure, since it is sort of an internal grep.  But since occur
    could not reasonably display something useful without making it
    visible, it might be an idea.

I think the use case is basically the same, for isearch, occur, and
query-replace. I can imagine a mode that might temporarily or selectively
hide things in an *Occur* buffer, for various reasons. Query-replace is
general, just like isearch.

I can imagine that one might be able to get into trouble using query-replace
with invisible hits, but I also think it could be useful.

    > Also, I'm curious why the treatment of an `invisible' text-property
    > value of `isearch-open-invisible' is limited to overlays. Why
    > shouldn't the same behavior hold for the property if applied
    > directly to text (that is, without an overlay)?

    Because you can make some text visible without modification when there
    is an invisible overlay over it.  If there are invisible text
    properties, you can't remove them without changing the buffer, and
    searching should not change the buffer.

Ah, I guess you mean that the parallel use case would exist for text without
overlays, but it would be difficult or impossible to implement.

    Fiddling with invisibility-spec would affect all invisible regions,
    not just the one at point.

Not sure I understand what you mean, there. Is that part of the explanation
of the implementation difficulty?

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     I consider it far too special to be
>     documented in the Emacs manual: this is a variable that should, if at
>     all, be used by the modes or packages making stuff invisible.
>
> Then why is it a defcustom? Why can you customize it?

Not much of a reason I see there.

> I don't think it's special, or I wouldn't, as a user, have asked for
> such a feature. I don't see why this shouldn't be useful for
> users. When I'm searching, and there is some invisible text (which I
> may or may not know), I'd like to be able to toggle isearch, to make
> it (in)sensitive to that text.
>
>     > - and the Elisp manual?
>
>     Not sure whether it is important enough for that.
>
> I think it's especially important for the Emacs manual.

I don't, and I explained why.

>     > - creating a toggle for it?
>
>     Why?  It is not a user-level feature.
>
> It's a user option - customizable. And so it should be, IMO. That's
> why I was asking for such an option - as a user. I want to
> interactively control whether isearch hits hidden text. One use is
> to find whether something particular is perhaps present but
> hidden(!).

But that depends on the mode in question.

>     Providing a toggle would be the task of any mode that makes
>     stuff invisible and that would require to have it accessible.
>
> Why would each mode do that?

Because invisibility does not serve a unique purpose.

> Why not have a general toggle for isearch - the same binding for all
> modes? If I have a mode that hides dired details, that mode will
> provide a toggle to show/hide the details, but it shouldn't have to
> also provide a toggle for isearch sensitivity to hidden text. If
> each mode does that, then users will not have a single, consistent
> binding to rely on - and the toggle should be available while
> isearching. To me, this is no different from toggling
> case-sensitivity or regexp sensitivity.

To me it is.  Case is not something a mode applies to a buffer, nor
are regexps.

>     > - having `occur' and `query-replace' respect the option?
>
>     Sounds sort of reasonable, at least with query-replace.  With
>     occur, I am less sure, since it is sort of an internal grep.
>     But since occur could not reasonably display something useful
>     without making it visible, it might be an idea.
>
> I think the use case is basically the same, for isearch, occur, and
> query-replace. I can imagine a mode that might temporarily or
> selectively hide things in an *Occur* buffer,

Who is talking about the occur buffer?  We are talking about the
source buffer.  The occur buffer would not copy overlays, anyway.

> for various reasons. Query-replace is general, just like isearch.

I said it sounded reasonable for query-replace.

> I can imagine that one might be able to get into trouble using
> query-replace with invisible hits, but I also think it could be
> useful.

Invisible hits are made visible when searching: no trouble here.

>     > Also, I'm curious why the treatment of an `invisible'
>     > text-property value of `isearch-open-invisible' is limited to
>     > overlays. Why shouldn't the same behavior hold for the
>     > property if applied directly to text (that is, without an
>     > overlay)?
>
>     Because you can make some text visible without modification when
>     there is an invisible overlay over it.  If there are invisible
>     text properties, you can't remove them without changing the
>     buffer, and searching should not change the buffer.
>
> Ah, I guess you mean that the parallel use case would exist for text
> without overlays, but it would be difficult or impossible to
> implement.

No, the parallel use case does not exist.  Text with invisible
property is intended to never be a visible part of the buffer.
Overlays, in contrast, can be removed and put back without a buffer
change.

>     Fiddling with invisibility-spec would affect all invisible regions,
>     not just the one at point.
>
> Not sure I understand what you mean, there. Is that part of the
> explanation of the implementation difficulty?

There is no implementation difficulty.  It is not useful to search for
text that is not part of the visible buffer and can't become so
without changing its properties.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    >     this is a variable that should, if at
    >     all, be used by the modes or packages making stuff invisible.
    >
    > Then why is it a defcustom? Why can you customize it?

    Not much of a reason I see there.

So do you consider the defcustom a bug, rather than a feature? You say this
is "not a user-level feature", but, in fact, it is.

    > I don't see why this shouldn't be useful for users. When I'm
    > searching, and there is some invisible text (which I may or
    > may not know), I'd like to be able to toggle isearch, to make
    > it (in)sensitive to that text.

    >     > - creating a toggle for it?
    >
    >     Why?  It is not a user-level feature.
    >
    > It's a user option - customizable. And so it should be, IMO. That's
    > why I was asking for such an option - as a user. I want to
    > interactively control whether isearch hits hidden text.

    > One use is to find whether something particular is perhaps
    > present but hidden(!).

    But that depends on the mode in question.

My wish to search for invisible text or not does not necessarily depend on
the mode. What depends on the mode? Just which text is hidden. This is about
being able to turn on/off search of invisible text, not about the choice of
which text is to be invisible.

    >     Providing a toggle would be the task of any mode that makes
    >     stuff invisible and that would require to have it accessible.
    >
    > Why would each mode do that?

    Because invisibility does not serve a unique purpose.

So what? Different modes can decide to capitalize words differently too, and
some modes (e.g. SQL) are generally case-insensitive while others (e.g.
Emacs Lisp) are case-sensitive, but isearch case sensitivity can still be
toggled. By your argument, people should not be able to toggle
case-sensitivity: if the mode is case-sensitive, then you should search only
that way; if the mode is case-insensitive, then you should search only that
way.

    > Why not have a general toggle for isearch - the same binding for all
    > modes? If I have a mode that hides dired details, that mode will
    > provide a toggle to show/hide the details, but it shouldn't have to
    > also provide a toggle for isearch sensitivity to hidden text. If
    > each mode does that, then users will not have a single, consistent
    > binding to rely on - and the toggle should be available while
    > isearching. To me, this is no different from toggling
    > case-sensitivity or regexp sensitivity.

    To me it is.  Case is not something a mode applies to a buffer, nor
    are regexps.

Invisible text doesn't necessarily "apply to a buffer". Text can be
invisible for different reasons.

And case can very well be something that "applies to a buffer" - play with
SQL mode and see if its buffers are generally sensitive to case.

    >     > - having `occur' and `query-replace' respect the option?
    >
    >     Sounds sort of reasonable, at least with query-replace.  With
    >     occur, I am less sure, since it is sort of an internal grep.
    >     But since occur could not reasonably display something useful
    >     without making it visible, it might be an idea.
    >
    > I think the use case is basically the same, for isearch, occur, and
    > query-replace. I can imagine a mode that might temporarily or
    > selectively hide things in an *Occur* buffer,

    Who is talking about the occur buffer?  We are talking about the
    source buffer.  The occur buffer would not copy overlays, anyway.

Sorry, I meant the source buffer that is searched to create the *Occur*
buffer.

    >     > Also, I'm curious why the treatment of an `invisible'
    >     > text-property value of `isearch-open-invisible' is limited to
    >     > overlays. Why shouldn't the same behavior hold for the
    >     > property if applied directly to text (that is, without an
    >     > overlay)?
    >
    >     Because you can make some text visible without modification when
    >     there is an invisible overlay over it.  If there are invisible
    >     text properties, you can't remove them without changing the
    >     buffer, and searching should not change the buffer.
    >
    > Ah, I guess you mean that the parallel use case would exist for text
    > without overlays, but it would be difficult or impossible to
    > implement.

    No, the parallel use case does not exist.  Text with invisible
    property is intended to never be a visible part of the buffer.

How can you make such a blanket statement? Someone or some application could
easily make some text in a buffer invisible for a particular use, without
using an overlay. And someone might well want to be able to search that text
in either mode: show/ignore invisible text. Nothing is verboten or
predetermined about how someone or some application might use invisible
text.

You say that "searching should not change the buffer". But the change can be
temporary. (Plus, query-replace is likely to change the buffer anyway.)

    Overlays, in contrast, can be removed and put back without a buffer
    change.

There are lots of temporary "buffer changes" in Emacs. Adding and removing
an overlay is also a kind of "buffer change", even if it doesn't change the
text.

I'm not saying people shouldn't use overlays in the way you describe - far
from it. I'm only asking why isearch shouldn't behave the same for invisible
text, with or without an overlay.

    >     Fiddling with invisibility-spec would affect all
    >     invisible regions, not just the one at point.
    >
    > Not sure I understand what you mean, there. Is that part of the
    > explanation of the implementation difficulty?

    There is no implementation difficulty.  It is not useful to search for
    text that is not part of the visible buffer and can't become so
    without changing its properties.

You're just repeating here that you don't see the usefulness. I ask why it
wouldn't be useful, and you answer only "It is not useful."

Can you explain what you meant about "fiddling...not just the one at point"?

I'm guessing perhaps you meant something like this: Toggling isearch's
showing of invisible hits would, in the case of invisible text (no
overlays), require toggling the behavior for the whole buffer. If that's
your point, so what? What's wrong with toggling the behavior for the whole
buffer?

And why would that be the case necessarily? Why could't isearch temporarily
show only the current hit, and then make it invisible again when you move on
to another hit?

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     There is no implementation difficulty.  It is not useful to search for
>     text that is not part of the visible buffer and can't become so
>     without changing its properties.
>
> You're just repeating here that you don't see the usefulness. I ask
> why it wouldn't be useful, and you answer only "It is not useful."

You are arguing for an invasive change.  Please describe a concrete
actual situation you have encountered where it would actually be
useful.  Up to now, you just provided generic handwaving.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    >     There is no implementation difficulty.  It is not useful
    >     to search for text that is not part of the visible buffer
    >     and can't become so without changing its properties.
    >
    > You're just repeating here that you don't see the usefulness. I ask
    > why it wouldn't be useful, and you answer only "It is not useful."

    You are arguing for an invasive change.  Please describe a concrete
    actual situation you have encountered where it would actually be
    useful.  Up to now, you just provided generic handwaving.

OK, so the argument is, after all, that this is not easy to implement.

I have no "actual situation". I was searching through a buffer that had some
invisible text (as an overlay), and I wanted to know if there was an isearch
option for finding it - that's all. You answered "yes" - thank you.

I suggested documenting that option better and making it a toggle in
`isearch'. You said it shouldn't be documented better, it shouldn't be a
user option after all, and, a fortiori, it shouldn't be toggled in
`isearch'.

I wondered, and asked, why the "open" behavior was limited to overlays. Your
answers were not clear to me, but it seems now that the answer is that that
would be an invasive change (although "there is no implementation
difficulty"). Thanks for clarifying.

I do hope the user option `search-invisible' 1) remains a user option, 2)
gets better documented, as it seems useful, and 3) gets an associated toggle
in `isearch'.

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     >     There is no implementation difficulty.  It is not useful
>     >     to search for text that is not part of the visible buffer
>     >     and can't become so without changing its properties.
>     >
>     > You're just repeating here that you don't see the
>     > usefulness. I ask why it wouldn't be useful, and you answer
>     > only "It is not useful."
>
>     You are arguing for an invasive change.  Please describe a
>     concrete actual situation you have encountered where it would
>     actually be useful.  Up to now, you just provided generic
>     handwaving.
>
> OK, so the argument is, after all, that this is not easy to
> implement.

We don't implement things because they are easy.  We implement them
when they are useful.

> I have no "actual situation". I was searching through a buffer that
> had some invisible text (as an overlay), and I wanted to know if
> there was an isearch option for finding it - that's all. You
> answered "yes" - thank you.
>
> I suggested documenting that option better and making it a toggle in
> `isearch'. You said it shouldn't be documented better, it shouldn't
> be a user option after all, and, a fortiori, it shouldn't be toggled
> in `isearch'.

What was it that rendered the text invisible in the first place?  I
know search-invisible since preview-latex has to deal with it
(actually, it is just the XEmacs port, since the Emacs port works with
display properties instead).  The mode was responsible for hiding
stuff away, and consequently it was also responsible for letting them
show up.  Since the behavior of isearch needed fixing, I had to check
in the source file, anyway.  The details were complicated enough that
it would have needed quite long and complicated explanations in the
Elisp manual, for material that would get checked in the code, anyway.

This is my contact with search-invisible.  Now could you please
explain what mode rendered parts of your text invisible so that one
can get an idea whether a global user option makes sense in connection
with that?

> I do hope the user option `search-invisible' 1) remains a user
> option, 2) gets better documented, as it seems useful, and 3) gets
> an associated toggle in `isearch'.

Would you please present an actual example where this would be useful?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    > I have no "actual situation". I was searching through a buffer that
    > had some invisible text (as an overlay), and I wanted to know if
    > there was an isearch option for finding it - that's all. You
    > answered "yes" - thank you.
    >
    > I suggested documenting that option better and making it a toggle in
    > `isearch'. You said it shouldn't be documented better, it shouldn't
    > be a user option after all, and, a fortiori, it shouldn't be toggled
    > in `isearch'.

    What was it that rendered the text invisible in the first place?

I used `dired-details-toggle' (a command in library dired-details.el) to
hide details in Dired (and, as I said, it uses an overlay). It happened that
I wanted to search for something invisible in that context, and I wondered
if there was a search option. That's all - there is nothing special about
that context. A single keystroke can make everything visible anyway, so
there is no special advantage in that context to searching for invisible
text.

    I know search-invisible since preview-latex has to deal with it
    (actually, it is just the XEmacs port, since the Emacs port works with
    display properties instead).  The mode was responsible for hiding
    stuff away, and consequently it was also responsible for letting them
    show up.  Since the behavior of isearch needed fixing, I had to check
    in the source file, anyway.  The details were complicated enough that
    it would have needed quite long and complicated explanations in the
    Elisp manual, for material that would get checked in the code, anyway.

    This is my contact with search-invisible.  Now could you please
    explain what mode rendered parts of your text invisible so that one
    can get an idea whether a global user option makes sense in connection
    with that?

As I said, the context that made me wonder about a possible search option is
irrelevant; there is no special need for searching invisible text in that
context. It occurred to me that it might be useful, generally, to be able to
search invisible text; that's all.

    > I do hope the user option `search-invisible' 1) remains a user
    > option, 2) gets better documented, as it seems useful, and 3) gets
    > an associated toggle in `isearch'.

    Would you please present an actual example where this would be useful?

Sorry, I don't have one, as I already stated in my last message. My crystal
ball tells me it will be useful, but my crystal ball has been wrong before
too. I like being able to toggle isearch options on the fly, and this is
apparently an isearch option (until/unless you kill it).

More importantly, I think the option should be documented - perhaps in the
section on invisible text, if there is to be no isearch toggle for it. If
you want to remove the user option and make it a plain `defvar', that's your
choice; I have no actual example that will dissuade you. End of story.

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

> More importantly, I think the option should be documented - perhaps
> in the section on invisible text, if there is to be no isearch
> toggle for it. If you want to remove the user option and make it a
> plain `defvar', that's your choice; I have no actual example that
> will dissuade you. End of story.

The doc string mentions outline.el and hideshow.el.  Both of those
don't have invisible text as much as hidden text, text that clearly is
part of the buffer, with the invisibility being a display aspect and
not a text aspect.  The overlays will also get an
`isearch-open-invisible' property allowing them to open.  For this
sort of thing, the default setting of search-invisible, namely `open',
makes sense.  I don't see where a different setting would make sense
in connection with those modes which cater with appropriate properties
for opening in searches. `open' might be less necessary when someone
uses reveal-mode, but I have not checked it.

In short: I fail to see where a global setting of search-invisible to
anything other than `open' would make even remotely sense with the
mentioned outline.el and hideshow.el.

For dired-mode, this is not as clearcut in my case.  When I hide
aspects of the buffer away with an explicit command, it is less likely
that I actually would want to find them.  And those overlays don't use
`isearch-open-invisible' seemingly, which again seems to match what is
useful and likely the intent of the user.

So in short, I see nothing worth changing here.  If your needs as a
programmer are sufficiently strange that you'll need to meddle with
those variables, you'll be reading the source file for searches,
anyway.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    > More importantly, I think the option should be documented - perhaps
    > in the section on invisible text, if there is to be no isearch
    > toggle for it. If you want to remove the user option and make it a
    > plain `defvar', that's your choice; I have no actual example that
    > will dissuade you. End of story.

    The doc string mentions outline.el and hideshow.el.  Both of those
    don't have invisible text as much as hidden text, text that clearly is
    part of the buffer, with the invisibility being a display aspect and
    not a text aspect.  The overlays will also get an
    `isearch-open-invisible' property allowing them to open.  For this
    sort of thing, the default setting of search-invisible, namely `open',
    makes sense.  I don't see where a different setting would make sense
    in connection with those modes which cater with appropriate properties
    for opening in searches. `open' might be less necessary when someone
    uses reveal-mode, but I have not checked it.

    In short: I fail to see where a global setting of search-invisible to
    anything other than `open' would make even remotely sense with the
    mentioned outline.el and hideshow.el.

    For dired-mode, this is not as clearcut in my case.  When I hide
    aspects of the buffer away with an explicit command, it is less likely
    that I actually would want to find them.  And those overlays don't use
    `isearch-open-invisible' seemingly, which again seems to match what is
    useful and likely the intent of the user.

    So in short, I see nothing worth changing here.  If your needs as a
    programmer are sufficiently strange that you'll need to meddle with
    those variables, you'll be reading the source file for searches,
    anyway.

I don't know what you're arguing about. I never asked you to change the
default value of the option. I never argued for special uses in different
modes. I never suggested changing anything about the option's behavior. I
never indicated any "sufficiently strange" "needs as a programmer" or any
interest in "meddling with those variables". We spoke of only one variable,
and I assure you that I have no desire to meddle with it.

I simply said that I'm glad there is such a user option, and I hope it will
remain an option. And I thank you again for pointing it out.

I didn't find such an option when I looked for it in the doc, and I added
that I thought it would help to mention it in the Emacs manual, so people
can be aware of it. You don't agree, so it will remain findable via `M-x
apropos-documentation RET isearch.*invisible RET' or `M-x apropos-variable
RET search.*invisible RET' ("search", not "isearch"). Fair enough.

You also indicated that you think it shouldn't be a user option. If you
change it to a `defvar', I'll still be able to use it, so I don't have a
problem, for my own use. Please leave the doc string, at least, to help
others find the variable using `apropos-*' ;-).

I added that I thought it might be useful to toggle it during `isearch'. You
don't think so. End of story.

Re: how to control isearch for invisible text

[Stefan Monnier]

> - documenting `isearch-invisible' in the Emacs manual?
> - and the Elisp manual?
> - creating a toggle for it?
> - having `occur' and `query-replace' respect the option?

I think `occur' always makes things visible because it copies text and
overlays are not copied along, so the invisibility gets lost.
I do think query-replace should be improved to open invisible text just like
isearch does.

> Also, I'm curious why the treatment of an `invisible' text-property value of
> `isearch-open-invisible' is limited to overlays. Why shouldn't the same
> behavior hold for the property if applied directly to text (that is, without
> an overlay)?

I'm not exactly sure why isearch does it like that, but reveal-mode (which
does something similar, tho not restricted to isearch) does the same for the
reason that it's simpler and that text-properties tend to be used for
different circumstances.  E.g. overlays are used for things like outline.el
or folding.el whereas invisible text-properties seem to be limited to
smaller extents which are never-ever meant to be shown to the user.


        Stefan

Re: how to control isearch for invisible text

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> - documenting `isearch-invisible' in the Emacs manual?
>> - and the Elisp manual?
>> - creating a toggle for it?
>> - having `occur' and `query-replace' respect the option?
>
> I think `occur' always makes things visible because it copies text
> and overlays are not copied along, so the invisibility gets lost.  I
> do think query-replace should be improved to open invisible text
> just like isearch does.

For what it's worth, preview-latex contains the following code:

(defadvice replace-highlight (before preview)
  "Make `query-replace' open preview text about to be replaced."
  (preview-open-overlays
   (overlays-in (ad-get-arg 0) (ad-get-arg 1))))

(defcustom preview-query-replace-reveal t
  "*Make `query-replace' autoreveal previews."
  :group 'preview-appearance
  :type 'boolean
  :require 'preview
  :set (lambda (symbol value)
	 (set-default symbol value)
	 (if value
	     (ad-enable-advice 'replace-highlight 'before 'preview)
	   (ad-disable-advice 'replace-highlight 'before 'preview))
	 (ad-activate 'replace-highlight))
  :initialize #'custom-initialize-reset)

Text opened by preview-open-overlays gets closed again when
post-command-hook finds it has left the region.


It turns out that for XEmacs we do things differently for some reason:
probably it does not have the invisible open properties.  There we
have

(defun preview-open-overlay (ovr ignored)
  "Open the active preview OVR, IGNORED gets ignored.
NIL is returned: this is for `map-extents'."
  (preview-toggle ovr)
  (push ovr preview-temporary-opened)
  nil)

(defadvice isearch-highlight (before preview protect disable)
  "Make isearch open preview text that's a search hit.
Also make `query-replace' open preview text about to be replaced."
  (map-extents #'preview-open-overlay nil
	       (ad-get-arg 0) (ad-get-arg 1)
	       nil nil 'preview-state 'active))


I remember it took quite a long time to find this least invasive place
for the advice, so it is likely that the call places of
replace-highlight might be the best location for implementing this
functionality within Emacs.

Bonus points if it does not clash with preview-latex's current
implementation...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Richard Stallman]

    > I think `occur' always makes things visible because it copies text
    > and overlays are not copied along, so the invisibility gets lost.  I
    > do think query-replace should be improved to open invisible text
    > just like isearch does.

I agree that query-replace should show invisible text
when a match occurs inside it.

Would someone like to implement that?

What is `preview-open-overlays'?  It is not part of Emacs.

Re: how to control isearch for invisible text

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > I think `occur' always makes things visible because it copies text
>     > and overlays are not copied along, so the invisibility gets lost.  I
>     > do think query-replace should be improved to open invisible text
>     > just like isearch does.
>
> I agree that query-replace should show invisible text
> when a match occurs inside it.
>
> Would someone like to implement that?
>
> What is `preview-open-overlays'?  It is not part of Emacs.

Part of AUCTeX's preview-latex which replaces LaTeX formulas by their
typeset images.  Not exactly invisibility, but still requires similar
action on search and search&replace so that the user can see what
source text he is actually replacing.

Its code would not be useful for isearch, but I thought it helpful to
point out where I had anchored the advice for this similar task.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Stefan Monnier]

>> I think `occur' always makes things visible because it copies text
>> and overlays are not copied along, so the invisibility gets lost.  I
>> do think query-replace should be improved to open invisible text
>> just like isearch does.

> I agree that query-replace should show invisible text
> when a match occurs inside it.

For reveal-mode, we should just provide a "pre-redisplay-hook" run, as the
name indicates, just before doing redisplay.  That would allow reveal-mode
to do the "show invisible text" even in query-replace and this without even
changing query-replace's code.


        Stefan

Re: how to control isearch for invisible text

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I think `occur' always makes things visible because it copies text
>>> and overlays are not copied along, so the invisibility gets lost.  I
>>> do think query-replace should be improved to open invisible text
>>> just like isearch does.
>
>> I agree that query-replace should show invisible text
>> when a match occurs inside it.
>
> For reveal-mode, we should just provide a "pre-redisplay-hook" run,
> as the name indicates, just before doing redisplay.  That would
> allow reveal-mode to do the "show invisible text" even in
> query-replace and this without even changing query-replace's code.

That is a bit heavy-handed for my taste, at least for this purpose.
Such a hook would have to be global, as redisplay is not a
buffer-centric activity, and thus every function with a possible need
would have to register and unregister there, and check the state
across buffers for every redisplay.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Stefan Monnier]

>> For reveal-mode, we should just provide a "pre-redisplay-hook" run,
>> as the name indicates, just before doing redisplay.  That would
>> allow reveal-mode to do the "show invisible text" even in
>> query-replace and this without even changing query-replace's code.

> That is a bit heavy-handed for my taste, at least for this purpose.
> Such a hook would have to be global, as redisplay is not a
> buffer-centric activity, and thus every function with a possible need
> would have to register and unregister there, and check the state
> across buffers for every redisplay.

I doubt it's that hard to implement a proper pre-redisplay-hook which can be
set buffer-locally.


        Stefan

Re: how to control isearch for invisible text

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> For reveal-mode, we should just provide a "pre-redisplay-hook" run,
>>> as the name indicates, just before doing redisplay.  That would
>>> allow reveal-mode to do the "show invisible text" even in
>>> query-replace and this without even changing query-replace's code.
>
>> That is a bit heavy-handed for my taste, at least for this purpose.
>> Such a hook would have to be global, as redisplay is not a
>> buffer-centric activity, and thus every function with a possible need
>> would have to register and unregister there, and check the state
>> across buffers for every redisplay.
>
> I doubt it's that hard to implement a proper pre-redisplay-hook which can be
> set buffer-locally.

This is pretty useless, since in your kind of application, it would be
important to know window-point rather than point.  The documentation
of window-point already states:

    window-point is a built-in function in `C source code'.
    (window-point &optional WINDOW)

    Return current value of point in WINDOW.
    For a nonselected window, this is the value point would have
    if that window were selected.

    Note that, when WINDOW is the selected window and its buffer
    is also currently selected, the value returned is the same as (point).
    It would be more strictly correct to return the `top-level' value
    of point, outside of any save-excursion forms.
    But that is hard to define.

    [back]

And that is precisely the same problem with reveal-mode: sit-for can
be called inside of save-excursions, and the current window-point need
not correspond with the point relevant at top-level.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Stefan Monnier]

> This is pretty useless, since in your kind of application, it would be
> important to know window-point rather than point.  The documentation
> of window-point already states:

So it's called multiple times on the same buffer for each window that
displays the buffer.  And you pass the window as argument (so you rename it
to pre-redisplay-functions).  Big deal.  Next "problem"?

> And that is precisely the same problem with reveal-mode: sit-for can
> be called inside of save-excursions, and the current window-point need
> not correspond with the point relevant at top-level.

What matters is what is displayed.  Which is why a pre-redisplay-hoook is
what is needed.


        Stefan

Re: how to control isearch for invisible text

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> This is pretty useless, since in your kind of application, it would
>> be important to know window-point rather than point.  The
>> documentation of window-point already states:
>
> So it's called multiple times on the same buffer for each window
> that displays the buffer.  And you pass the window as argument (so
> you rename it to pre-redisplay-functions).  Big deal.  Next
> "problem"?
>
>> And that is precisely the same problem with reveal-mode: sit-for can
>> be called inside of save-excursions, and the current window-point need
>> not correspond with the point relevant at top-level.
>
> What matters is what is displayed.  Which is why a
> pre-redisplay-hoook is what is needed.

But how would reveal-mode know what point position it should act on?
For example, when outline-mode is active, and I type `)', should
show-paren-mode open an overlay on `(', even if this causes
recentering?  After all, `pos-visible-in-window-p' is consulted by
show-paren-mode before redisplay.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Stefan Monnier]

>>>>> "David" == David Kastrup <dak@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> This is pretty useless, since in your kind of application, it would
>>> be important to know window-point rather than point.  The
>>> documentation of window-point already states:
>> 
>> So it's called multiple times on the same buffer for each window
>> that displays the buffer.  And you pass the window as argument (so
>> you rename it to pre-redisplay-functions).  Big deal.  Next
>> "problem"?
>> 
>>> And that is precisely the same problem with reveal-mode: sit-for can
>>> be called inside of save-excursions, and the current window-point need
>>> not correspond with the point relevant at top-level.
>> 
>> What matters is what is displayed.  Which is why a
>> pre-redisplay-hoook is what is needed.

> But how would reveal-mode know what point position it should act on?
> For example, when outline-mode is active, and I type `)', should
> show-paren-mode open an overlay on `(', even if this causes
> recentering?  After all, `pos-visible-in-window-p' is consulted by
> show-paren-mode before redisplay.

Of course, a pre-redisplay-hook is not the only thing needed.
In some cases it's enough, in others it's more tricky.  But it's a damn good
starting point.


        Stefan

Re: how to control isearch for invisible text

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>>> "David" == David Kastrup <dak@gnu.org> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> Of course, a pre-redisplay-hook is not the only thing needed.
>
> In some cases it's enough, in others it's more tricky.  But it's a
> damn good starting point.

That's an imprudent thing to tell a paranoiac in feature freeze, you
know...  At the moment it might be more appropriate thinking about
finishing rather than starting things.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: how to control isearch for invisible text

[Drew Adams]

    At the moment it might be more appropriate thinking about
    finishing rather than starting things.

I started this thread, with more of a question (how to?) than a concrete
feature suggestion. In any case, I don't think anyone has suggested
implementing such a feature before the release.

My only request for the immediate was that the user option be documented in
the manual.

Re: how to control isearch for invisible text

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     At the moment it might be more appropriate thinking about
>     finishing rather than starting things.
>
> I started this thread, with more of a question (how to?) than a
> concrete feature suggestion. In any case, I don't think anyone has
> suggested implementing such a feature before the release.

I don't think it would be wrong to make query-replace work analogously
to isearch-forward with respect to invisibility properties, even in
light of the upcoming release.  And I pointed out where, in a previous
analysis of the situation, I had found a suitable vicinity for adding
the required mechanism with a least invasive change.  A change I would
think tolerable to make yet, as it has pretty local consequences.

In contrast, I think that a hook such as the one Stefan is thinking of
would be opening a can of worms I don't particularly enjoy to see at
this point of time.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: how to control isearch for invisible text

[Richard Stallman]

    For reveal-mode, we should just provide a "pre-redisplay-hook" run, as the
    name indicates, just before doing redisplay.

I get a bad feeling about making redisplay itself do this.
That is certainly too radical to do now, and may be simply a bad idea.

I would rather have a subroutine that can be called by things
like isearch and query-replace.

Re: how to control isearch for invisible text

[Richard Stallman]

    I started this thread, with more of a question (how to?) than a concrete
    feature suggestion. In any case, I don't think anyone has suggested
    implementing such a feature before the release.

It makes sense to fix query-replace to handle invisible text more or
less the same way isearch does, and that can be done now.