auto-composition-mode and display properties

auto-composition-mode and display properties

[Stephen Berman]

Since the unicode merge, composite.el contains auto-composition-mode,
which is enabled globally by default.  This adds the auto-composed
property to every character.  The auto-composed property also gets added
to display properties.  For example:

1. emacs -Q
2. Type the string "ab".
3. Put the cursor after "b" and type 
   `M-: (put-text-property (1- (point)) (point) 'display "test")'.
Now the buffer display "atest".
4. Put the cursor after "a", i.e. on the first "t" and type `C-u C-x
='.  The *Help* buffer says:
     There are text properties here:
       auto-composed        t
       display              [Show]
By clicking on `[Show]' we get the value of the display property:
   #("test" 0 4
     (auto-composed t))

I assume the display property is supposed to be like this, i.e. this is
not a bug?  I ask because it is wreaking havoc with isearch in a mode I
have that uses display properties (e.g., isearch stops when it gets to
the bottom of the window, even though there are further occurrences out
of view, and the cursor cycles over lines above the last visible
occurrence).  There may be a bug in the mode that is triggered by
isearch with auto-composition-mode enabled, but I don't know how to
debug it: when I instrument isearch-mode for edebug and step through it,
after isearch-update is called the focus for entering isearch strings
becomes the buffer of isearch.el and switching back to the buffer with
my mode interrupts isearch.  So I cannot trigger the bug under edebug.
All I know is that when I disable auto-composition-mode, isearch works
fine in my mode (and it works fine in CVS Emacs prior to the unicode
merge).  So I would like to know, again, if auto-composition-mode is
really supposed to add the auto-composed property to the display string.
And does anyone have any suggestions for how I can debug isearch?

Steve Berman

Re: auto-composition-mode and display properties

[Stefan Monnier]

> merge).  So I would like to know, again, if auto-composition-mode is
> really supposed to add the auto-composed property to the display string.

Yes.  As a matter of fact, it goes through a fair bit of extra trouble
to be able to do that.

> And does anyone have any suggestions for how I can debug isearch?

When Edebug is uncooperative, `message' is your friend,


        Stefan

isearch, auto-composition-mode and display properties

[Stephen Berman]

[-- Attachment #1: Type: text/plain, Size: 2808 bytes --]

On Thu, 21 Feb 2008 11:23:45 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>> merge).  So I would like to know, again, if auto-composition-mode is
>> really supposed to add the auto-composed property to the display string.
>
> Yes.  As a matter of fact, it goes through a fair bit of extra trouble
> to be able to do that.

Ok.  Too bad it's also causing me trouble, but maybe there's hope ...

>> And does anyone have any suggestions for how I can debug isearch?
>
> When Edebug is uncooperative, `message' is your friend,

I don't see where in the isearch code it would be helpful to invoke
`message' for the problem at hand.  But maybe someone else will: I have
in the mean time come up with a fairly minimal recipe for reproducing
the isearch problem that does not depend on my mode.  Here it is:

1. emacs -Q --eval "(add-hook 'post-command-hook 'ignore)" isearch-test[1]

2. M-: (setq header-line-format "test")

3. Evaluate the following sexp (this assumes frame-height is 40 and
frame-width is 80):

	(progn
	  (put-text-property 70 71 'display " \n")
	  (put-text-property 137 138 'display " \n")
	  (copy-region-as-kill 1 188)
	  (goto-char 188)
	  (dotimes (_ 20) (yank))
	  (goto-char (point-min))
	  (scroll-up 1))

Now point should be at window-start, point-min should be above
window-start, and the last visible line in the window should contain an
occurrence of the string "search".

4. Type `C-s s e C-w' to make "search" the isearch target.

5. Keeping type `C-s' until it reaches the "search" in the last visible
line (line 13).

6. Typing `C-s' again makes the cursor jump to the beginning of line 7.
Continuing to type `C-s' cycles over lines 7 to 13 ad infinitum.

7. Type `M-x auto-composition-mode' to disable auto-composition-mode.
Now typing `C-s' makes isearch find all "search" strings to the end of
the buffer, i.e., now isearch functions as it should.

I note that the problematic effect in step 6 only happens if
post-command-hook is non-nil (but AFAICT the value is otherwise
irrelevant), header-line-format is non-nil and the text is situated as
described at the end of step 3.  (With the mode where I first observed
this problem, I saw somewhat different effects, possibly depending on
frame-height: in one test isearch simply stopped at the bottom of the
window, i.e. continuing to type `C-s' had no effect; in another test the
cursor jumped only a couple of lines higher, not to the middle of the
window; in yet another test the cursor jumped to the beginning of the
line.  Also there were configurations where the last occurrence of the
isearch target string was not on the last visible line but several
above.)

I hope others can reproduce this problem and someone can figure out
what's going on.

Steve Berman

Footnotes: 
[1]  Here is the file isearch-test:

[-- Attachment #2: isearch-test --]
[-- Type: application/octet-stream, Size: 187 bytes --]

Test test test test test test test test test test test test test test test test test test test test test search test test test test test test test test test test test test test test test

Re: isearch, auto-composition-mode and display properties

[Juri Linkov]

> I hope others can reproduce this problem and someone can figure out
> what's going on.

Thank you for the detailed test case.  I was able to reproduce it.

To debug isearch, you can instrument one of the top isearch functions
(e.g. `isearch-repeat-forward'), open the isearch.el file in another
frame/buffer (another frame is more preferable when debugging display
problems like in your case), and start typing C-s in the original
buffer.  When it stops on an uninteresting place, just type `g' to
continue execution.  Use `i' to go to a deeper function.

The hardest part in debugging isearch is not forget not to use isearch
to navigate in the source buffer while edebug is active (e.g. to find
the next function to instrument, etc :)

Using this technique, it is possible to find that your trouble is cased
by the following part of `isearch-update':

	  ;; Keep same hscrolling as at the start of the search when possible
	  (let ((current-scroll (window-hscroll)))
	    (set-window-hscroll (selected-window) isearch-start-hscroll)
	    (unless (pos-visible-in-window-p)
	      (set-window-hscroll (selected-window) current-scroll)))

Removing this part eliminates the bug you reported.  Unfortunately,
I have no idea how to fix this since this depends on the details
how the display engine works.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: isearch, auto-composition-mode and display properties

[Stephen Berman]

On Sun, 24 Feb 2008 21:09:07 +0200 Juri Linkov <juri@jurta.org> wrote:

>> I hope others can reproduce this problem and someone can figure out
>> what's going on.
>
> Thank you for the detailed test case.  I was able to reproduce it.
>
> To debug isearch, you can instrument one of the top isearch functions
> (e.g. `isearch-repeat-forward'), open the isearch.el file in another
> frame/buffer (another frame is more preferable when debugging display
> problems like in your case), and start typing C-s in the original
> buffer.  When it stops on an uninteresting place, just type `g' to
> continue execution.  Use `i' to go to a deeper function.
>
> The hardest part in debugging isearch is not forget not to use isearch
> to navigate in the source buffer while edebug is active (e.g. to find
> the next function to instrument, etc :)
>
> Using this technique, it is possible to find that your trouble is cased
> by the following part of `isearch-update':
>
> 	  ;; Keep same hscrolling as at the start of the search when possible
> 	  (let ((current-scroll (window-hscroll)))
> 	    (set-window-hscroll (selected-window) isearch-start-hscroll)
> 	    (unless (pos-visible-in-window-p)
> 	      (set-window-hscroll (selected-window) current-scroll)))
>
> Removing this part eliminates the bug you reported.  Unfortunately,
> I have no idea how to fix this since this depends on the details
> how the display engine works.

I'd like to return to this.  Assuming a fix at the display level is too
difficult or laborious, I wonder whether the above code really does what
the comment says.  In fact, I'm not sure what the comment is intending
to achieve.  I tested the code by starting isearch with different
horizontal scrolling positions set, both with and without the above
code, and saw no difference in behavior or resulting hscroll position.
What difference is expected and what do you have to do to see it?
Moreover, unless I misunderstand the manual, it looks like the above
will always leave the hscroll position at (window-hscroll) (which is
what my tests resulted in):

(elisp) Window Start:

     The `pos-visible-in-window-p' function considers only vertical
     scrolling.  If POSITION is out of view only because WINDOW has
     been scrolled horizontally, `pos-visible-in-window-p' returns
     non-`nil' anyway.

Is it possible for (pos-visible-in-window-p) to return nil after an
isearch?

The above code was added by this change:

2001-12-15  Richard M. Stallman  <rms@gnu.org>

	* isearch.el (isearch-start-hscroll): New variable.
	(isearch-mode): Set isearch-start-hscroll.
	(isearch-update): Restore original hscrolling if possible.

I searched the emacs-devel, bug, and help archives prior to this change
but found no relevant posts.  If the above code either doesn't do what
it was intended for, or if it doesn't make a difference in current
Emacs, could it be eliminated?  That would at least work around the
still existing problem raised by my OP.  Or if the code is still needed,
can someone explain it to me and provide a concrete example where it
DTRT?

Steve Berman

Re: isearch, auto-composition-mode and display properties

[Stephen Berman]

On Fri, 30 May 2008 10:49:10 +0200 Stephen Berman <Stephen.Berman@gmx.net> wrote:

> On Sun, 24 Feb 2008 21:09:07 +0200 Juri Linkov <juri@jurta.org> wrote:
>
>>> I hope others can reproduce this problem and someone can figure out
>>> what's going on.
>>
>> Thank you for the detailed test case.  I was able to reproduce it.
>>
>> To debug isearch, you can instrument one of the top isearch functions
>> (e.g. `isearch-repeat-forward'), open the isearch.el file in another
>> frame/buffer (another frame is more preferable when debugging display
>> problems like in your case), and start typing C-s in the original
>> buffer.  When it stops on an uninteresting place, just type `g' to
>> continue execution.  Use `i' to go to a deeper function.
>>
>> The hardest part in debugging isearch is not forget not to use isearch
>> to navigate in the source buffer while edebug is active (e.g. to find
>> the next function to instrument, etc :)
>>
>> Using this technique, it is possible to find that your trouble is cased
>> by the following part of `isearch-update':
>>
>> 	  ;; Keep same hscrolling as at the start of the search when possible
>> 	  (let ((current-scroll (window-hscroll)))
>> 	    (set-window-hscroll (selected-window) isearch-start-hscroll)
>> 	    (unless (pos-visible-in-window-p)
>> 	      (set-window-hscroll (selected-window) current-scroll)))
>>
>> Removing this part eliminates the bug you reported.  Unfortunately,
>> I have no idea how to fix this since this depends on the details
>> how the display engine works.
>
> I'd like to return to this.  Assuming a fix at the display level is too
> difficult or laborious, I wonder whether the above code really does what
> the comment says.  In fact, I'm not sure what the comment is intending
> to achieve.  I tested the code by starting isearch with different
> horizontal scrolling positions set, both with and without the above
> code, and saw no difference in behavior or resulting hscroll position.
> What difference is expected and what do you have to do to see it?
> Moreover, unless I misunderstand the manual, it looks like the above
> will always leave the hscroll position at (window-hscroll) (which is
> what my tests resulted in):
>
> (elisp) Window Start:
>
>      The `pos-visible-in-window-p' function considers only vertical
>      scrolling.  If POSITION is out of view only because WINDOW has
>      been scrolled horizontally, `pos-visible-in-window-p' returns
>      non-`nil' anyway.
>
> Is it possible for (pos-visible-in-window-p) to return nil after an
> isearch?

Sorry, much of what I wrote above is gibberish, due to insufficient
testing and not paying enough attention.  As should have been clear to
me, (pos-visible-in-window-p) returns nil during isearch precisely when
the next occurrence of the search term is below (window-end) (d'oh!).
And it is exactly at this point in the code that the problem I reported
arises (but it doesn't happen every time p-v-i-w-p returns nil; I'll try
to use gdb to find out more).  If the unless-sexp is eliminated, isearch
works as expected in the configuration given in my OP.

What still confuses me, however, is whether the unless-clause has a
noticeable effect.  When isearch-mode is invoked, isearch-start-hscroll
gets the current hscroll position, as returned by (window-hscroll).
IIUC, if, in the course of the isearch, the hscroll position changes,
the above code resets it to the initial value isearch-start-hscroll,
unless the next search term occurrence is beyond the visible window, in
which case the hscroll position becomes the new return value of
(window-hscroll).  I tried to test for this, but I could see no
difference: whether or not the next occurrence is in the visible window,
and whether or not the unless-sexp (or indeed the entire let-sexp) is
present, the window scrolls as required by isearch, and stays in the
current hscroll position upon exiting isearch, or returns to the initial
hscroll position upon cancelling isearch.  Can anyone else confirm this,
or have I again made a mistake in my tests or misunderstood the code?
And again, if I haven't made a mistake and the unless-sexp (or the
entire let-sexp) has no effect, can it be removed?

Steve Berman