Text Properties And Buffer Modification

Text Properties And Buffer Modification

[T.V Raman]

At present, modifying text properties (adding, removing or changing
properties) markes the buffer as  modified.
Since property changes are mostly cosmetic, is there a good reason why
this is so?

Given the present state of buffer-modified status with respect to
text-properties, one  has to be fairly careful e.g.
naively writing 
(with-silent-modifications (put-text-property ...))
will work *most of the time* except when you say something like:
(with-silent-modifications 
  (put-text-property beg end 'face 'bold other-buffer))

you actually need to write this:

(with-silent-modifications 
  (with-current-buffer other-buffer 
    (put-text-property beg end 'face 'bold other-buffer)))

-- 
Id: kg:/m/0285kf1 

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Tue, 04 Dec 2018 15:39:31 -0800
> 
> At present, modifying text properties (adding, removing or changing
> properties) markes the buffer as  modified.
> Since property changes are mostly cosmetic, is there a good reason why
> this is so?

Because text properties are part of the buffer text: if you copy some
of the text into another place, the properties go there as well.

> Given the present state of buffer-modified status with respect to
> text-properties, one  has to be fairly careful e.g.
> naively writing 
> (with-silent-modifications (put-text-property ...))
> will work *most of the time* except when you say something like:
> (with-silent-modifications 
>   (put-text-property beg end 'face 'bold other-buffer))
> 
> you actually need to write this:
> 
> (with-silent-modifications 
>   (with-current-buffer other-buffer 
>     (put-text-property beg end 'face 'bold other-buffer)))

You could instead simply make the target buffer not modified after you
are done changing the properties, e.g. by using set-buffer-modified-p.

Re: Text Properties And Buffer Modification

[Joost Kremers]

On Wed, Dec 05 2018, Eli Zaretskii wrote:
>> From: "T.V Raman" <raman@google.com>
>> Date: Tue, 04 Dec 2018 15:39:31 -0800
>> 
>> At present, modifying text properties (adding, removing or 
>> changing
>> properties) markes the buffer as  modified.
>> Since property changes are mostly cosmetic, is there a good 
>> reason why
>> this is so?
>
> Because text properties are part of the buffer text: if you copy 
> some
> of the text into another place, the properties go there as well.

But isn't that at least somewhat counter-intuitive, given that 
they are usually not saved to the file the buffer is visiting? Up 
until now I've always understood the modified status of a buffer 
to mean that there is a discrepancy between what is on disk and 
what would be on disk if the buffer were saved in its current 
from. The Emacs Manual indeed seems to say so: "If a buffer 
contains changes that have not been saved, we say the buffer is 
'modified'" (info "(emacs) Visiting").

Granted, buffers that are not visiting a file can also be 
modified, but they are essentially *always* modified, so their 
modification status is largely irrelevant. (Isn't it?)



-- 
Joost Kremers
Life has its moments

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Date: Wed, 05 Dec 2018 08:15:36 +0100
> Cc: "T.V Raman" <raman@google.com>
> 
> > Because text properties are part of the buffer text: if you copy
> > some of the text into another place, the properties go there as
> > well.
> 
> But isn't that at least somewhat counter-intuitive, given that 
> they are usually not saved to the file the buffer is visiting?

Whether or not the properties are saved depends on the properties and
on the major mode.  E.g., modes that use markup are expected to save
at least some text properties.

> Up until now I've always understood the modified status of a buffer
> to mean that there is a discrepancy between what is on disk and what
> would be on disk if the buffer were saved in its current from. The
> Emacs Manual indeed seems to say so: "If a buffer contains changes
> that have not been saved, we say the buffer is 'modified'" (info
> "(emacs) Visiting").

There's nothing wrong with what the manual says, unless you interpret
"save" too literally, by actually examining what's on the disk file.

As a counter-example, consider "C-x RET f": it doesn't change the
buffer text, but still marks the buffer modified.

So yes, it might take some getting used to, but I see no conceptual
problem with how Emacs treats changes in text properties.  Granted,
I've got used to that many years ago.

> Granted, buffers that are not visiting a file can also be 
> modified, but they are essentially *always* modified, so their 
> modification status is largely irrelevant. (Isn't it?)

No, I don't agree with that.  Don't forget that the modified status is
important for more than just saving to a file: it is important for
redisplaying the buffer, for example.  And some buffers that don't
visit files are "written" in other ways, like email message being
sent.

Re: Text Properties And Buffer Modification

[Stefan Monnier]

> Whether or not the properties are saved depends on the properties and
> on the major mode.  E.g., modes that use markup are expected to save
> at least some text properties.

Right: in 99% of the cases, when text-properties are modified you want
to wrap it within a with-silent-modifications.

I think when text-properties were added it would have been perfectly
fine to make them not touch the modified-ticks so the buffer is not
marked as modified.  Making this change now is a good bit
more delicate, which is why we haven't made it in the last 10 years,
even though the situation was already the same 10 years ago.

> No, I don't agree with that.  Don't forget that the modified status is
> important for more than just saving to a file: it is important for
> redisplaying the buffer, for example.

Really?


        Stefan

Re: Text Properties And Buffer Modification

[T.V Raman]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
Hi Stefan --

One way forward might be to add an additional optional arg to
put-text-property and friends  which if presents doesn't  change the
buffer-modified flag:

so today:

(put-text-property START END PROPERTY VALUE &optional OBJECT)
Future:
(put-text-property START END PROPERTY VALUE &optional OBJECT
leave-unmodified)

Thus, all existing code would continue to behave as before; new code
could explicitly pass leave-unmodified as an additional arg:

1. If object is a string,, leave-unmodified is a no-op and is ignored 
2. if object is a buffer, and leave-unmodified is  non-nil, then the
buffer is not marked as modified after that call.
>> Whether or not the properties are saved depends on the properties and
>> on the major mode.  E.g., modes that use markup are expected to save
>> at least some text properties.
>
> Right: in 99% of the cases, when text-properties are modified you want
> to wrap it within a with-silent-modifications.
>
> I think when text-properties were added it would have been perfectly
> fine to make them not touch the modified-ticks so the buffer is not
> marked as modified.  Making this change now is a good bit
> more delicate, which is why we haven't made it in the last 10 years,
> even though the situation was already the same 10 years ago.
>
>> No, I don't agree with that.  Don't forget that the modified status is
>> important for more than just saving to a file: it is important for
>> redisplaying the buffer, for example.
>
> Really?
>
>
>         Stefan
>
>

-- 
Id: kg:/m/0285kf1 

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 05 Dec 2018 10:32:28 -0500
> 
> > Don't forget that the modified status is important for more than
> > just saving to a file: it is important for redisplaying the
> > buffer, for example.
> 
> Really?

Is that a rhetorical question?

Re: Text Properties And Buffer Modification

[martin rudalics]

 >>> Don't forget that the modified status is important for more than
 >>> just saving to a file: it is important for redisplaying the
 >>> buffer, for example.
 >>
 >> Really?
 >
 > Is that a rhetorical question?

I would have thought that buffer redisplay is triggered only by
bset_redisplay and BUF_COMPUTE_UNCHANGED which IIUC are called the
same way for overlays and text properties.  Am I wrong?

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Wed, 05 Dec 2018 20:04:25 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org
> 
>  >>> Don't forget that the modified status is important for more than
>  >>> just saving to a file: it is important for redisplaying the
>  >>> buffer, for example.
>  >>
>  >> Really?
>  >
>  > Is that a rhetorical question?
> 
> I would have thought that buffer redisplay is triggered only by
> bset_redisplay and BUF_COMPUTE_UNCHANGED which IIUC are called the
> same way for overlays and text properties.  Am I wrong?

Yes.  The display engine sometimes needs to know not only whether the
buffer needs to be redisplayed, but also _why_.  Search xdisp.c for
MODIFF, and you will see.

Re: Text Properties And Buffer Modification

[martin rudalics]

 >> I would have thought that buffer redisplay is triggered only by
 >> bset_redisplay and BUF_COMPUTE_UNCHANGED which IIUC are called the
 >> same way for overlays and text properties.  Am I wrong?
 >
 > Yes.  The display engine sometimes needs to know not only whether the
 > buffer needs to be redisplayed, but also _why_.  Search xdisp.c for
 > MODIFF, and you will see.

I have tried to but still don't see the difference (but for setting
last_had_star).  I'm interested because I'd like to know whether the
'restore-buffer-modified-p' mechanism could be problematic, somehow.

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Thu, 06 Dec 2018 10:07:46 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>  >> I would have thought that buffer redisplay is triggered only by
>  >> bset_redisplay and BUF_COMPUTE_UNCHANGED which IIUC are called the
>  >> same way for overlays and text properties.  Am I wrong?
>  >
>  > Yes.  The display engine sometimes needs to know not only whether the
>  > buffer needs to be redisplayed, but also _why_.  Search xdisp.c for
>  > MODIFF, and you will see.
> 
> I have tried to but still don't see the difference (but for setting
> last_had_star).

What did you try, exactly?  And what kind of difference did you expect
to see?

Re: Text Properties And Buffer Modification

[martin rudalics]

 >>   > Yes.  The display engine sometimes needs to know not only whether the
 >>   > buffer needs to be redisplayed, but also _why_.  Search xdisp.c for
 >>   > MODIFF, and you will see.
 >>
 >> I have tried to but still don't see the difference (but for setting
 >> last_had_star).
 >
 > What did you try, exactly?  And what kind of difference did you expect
 > to see?

I tried to find all occurrences of the string MODIFF in xdisp.c and
look at the context of every occurrence I found.  I then tried to find
out whether that context provided any clue that the display engine
would handle a text property differently from an overlay change.  In
particular I tried to find out whether there were any redisplay
optimizations where this was the case.  Probably I'm missing something
but the "you will see" hinted at something quite obvious.  Maybe you
can name 2 or 3 such occurrences and I will then try to look harder.

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Thu, 06 Dec 2018 11:10:23 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> I tried to find all occurrences of the string MODIFF in xdisp.c and
> look at the context of every occurrence I found.  I then tried to find
> out whether that context provided any clue that the display engine
> would handle a text property differently from an overlay change.  In
> particular I tried to find out whether there were any redisplay
> optimizations where this was the case.  Probably I'm missing something
> but the "you will see" hinted at something quite obvious.  Maybe you
> can name 2 or 3 such occurrences and I will then try to look harder.

The optimization I had in mind is try_window_id.  You will see there
that this optimization is disabled if overlays were changed, but not
if the buffer (including text properties) was changed.  you will also
see there that buffer-modification tick is used there to update the
unchanged text at beginning and end of the buffer when necessary.

The other place where the buffer modification tick is used is in
mark_window_display_accurate_1.  If we somehow disrupt that, we could
have excess re-entries into redisplay, because Emacs might think some
windows are not yet up to date.

There's also a place in redisplay_internal where we set the
update_mode_line flag of the selected window when that window's buffer
was modified, and a similar use in window_buffer_changed.

Re: Text Properties And Buffer Modification

[martin rudalics]

 > The optimization I had in mind is try_window_id.  You will see there
 > that this optimization is disabled if overlays were changed, but not
 > if the buffer (including text properties) was changed.  you will also
 > see there that buffer-modification tick is used there to update the
 > unchanged text at beginning and end of the buffer when necessary.

This must be the

   if (w->last_overlay_modified != OVERLAY_MODIFF)
     GIVE_UP (200);

form.  I clearly missed that.

IIUC try_window_id is based on the idea that typing should be fast
(that is, cause no excessive redisplay) as long as it does not happen
within an overlay.  But why do we optimize changes of text properties
as well?  For historical reasons, I suppose.

 > The other place where the buffer modification tick is used is in
 > mark_window_display_accurate_1.  If we somehow disrupt that, we could
 > have excess re-entries into redisplay, because Emacs might think some
 > windows ae not yet up to date.

This is one of those I looked at and found no difference between
overlay and text changes.  Now the former of these

   w->last_modified = accurate_p ? BUF_MODIFF (b) : 0;
   w->last_overlay_modified = accurate_p ? BUF_OVERLAY_MODIFF (b) : 0;

is checked by try_window_id as

   if (w->last_modified == 0)
     GIVE_UP (6);

and the latter is needed by the check I cited above.  But is there an
additioal twist?

 > There's also a place in redisplay_internal where we set the
 > update_mode_line flag of the selected window when that window's buffer
 > was modified, and a similar use in window_buffer_changed.

These are to be expected.

Many thanks for the explanations, martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Thu, 06 Dec 2018 19:46:42 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>  > The optimization I had in mind is try_window_id.  You will see there
>  > that this optimization is disabled if overlays were changed, but not
>  > if the buffer (including text properties) was changed.  you will also
>  > see there that buffer-modification tick is used there to update the
>  > unchanged text at beginning and end of the buffer when necessary.
> 
> This must be the
> 
>    if (w->last_overlay_modified != OVERLAY_MODIFF)
>      GIVE_UP (200);
> 
> form.  I clearly missed that.

That's one of them.  The other is the snippet that begins with

  if (MODIFF > SAVE_MODIFF


> IIUC try_window_id is based on the idea that typing should be fast
> (that is, cause no excessive redisplay) as long as it does not happen
> within an overlay.

Not sure I agree with this interpretation.  My mental model of
try_window_id was that it attempts to reuse what is already on display
by scrolling parts of the displayed stuff, and then inserting/deleting
a few lines that have changed.  It determines which parts of the
display could be reused by analyzing where are the changes in the
buffer relative to the displayed portion.  How did we arrive at the
situation where only some lines need to be changed is immaterial, and
in general is not by typing, quickly or otherwise.

try_window_id gives up if overlay changed because it finds the
portions of the displayed text to be reused by looking at buffer text,
and overlays invalidate the assumption that only buffer text
determines what's on screen.

> But why do we optimize changes of text properties as well?

I'm not sure I understand the question.  What do you mean by "optimize
changes of text properties"?

> This is one of those I looked at and found no difference between
> overlay and text changes.  Now the former of these
> 
>    w->last_modified = accurate_p ? BUF_MODIFF (b) : 0;
>    w->last_overlay_modified = accurate_p ? BUF_OVERLAY_MODIFF (b) : 0;
> 
> is checked by try_window_id as
> 
>    if (w->last_modified == 0)
>      GIVE_UP (6);
> 
> and the latter is needed by the check I cited above.  But is there an
> additioal twist?

Since we rejected the case where overlays changed, we don't need to
check it again.

In general, the above condition means that the window's glyph matri
might not be up to date, so we cannot safely reuse it.

Re: Text Properties And Buffer Modification

[martin rudalics]

 >> IIUC try_window_id is based on the idea that typing should be fast
 >> (that is, cause no excessive redisplay) as long as it does not happen
 >> within an overlay.
 >
 > Not sure I agree with this interpretation.  My mental model of
 > try_window_id was that it attempts to reuse what is already on display
 > by scrolling parts of the displayed stuff, and then inserting/deleting
 > a few lines that have changed.  It determines which parts of the
 > display could be reused by analyzing where are the changes in the
 > buffer relative to the displayed portion.

The current description of try_window_id in xdisp.c is

       This function attempts to redisplay a window by reusing parts of
       its existing display.  It finds and reuses the part that was not
       changed, and redraws the rest.  (The "id" part in the function's
       name stands for "insert/delete", not for "identification" or
       somesuch.)

Could you kindly add that "scrolling parts of the displayed stuff"
there somehow.  For a simple minded reader like me, it's intuitively
not clear that it can do such virtual scrolling of unchanged contents.
Maybe also tell how many such unaltered chunks can be reused and/or
how large they must be.

 > How did we arrive at the
 > situation where only some lines need to be changed is immaterial, and
 > in general is not by typing, quickly or otherwise.

OK.  I was looking for the rationale of try_window_id and that's what
I came up with.  Till now I naively imagined try_window_id to reuse
parts of the old glyph matrix as long as a user did _not add or remove
a newline_ thus causing the part below the line where the change
occurred to move within the glyph matrix and consequently invalidate
the entire matrix and trigger a redisplay of the entire window.  The
fact that the code is smarter than that has eluded me so far.

 > try_window_id gives up if overlay changed because it finds the
 > portions of the displayed text to be reused by looking at buffer text,
 > and overlays invalidate the assumption that only buffer text
 > determines what's on screen.

It wouldn't matter much but I think that redisplay could handle an
overlay change just like any text property change.

 >> But why do we optimize changes of text properties as well?
 >
 > I'm not sure I understand the question.  What do you mean by "optimize
 > changes of text properties"?

I was asking myself why it's important to treat text property changes
in an optimized way in try_window_id because I thought they wouldn't
be affected much by typing.  And as far as font-locking is concerned,
it invalidates the rest of the buffer parts shown in a window anyway
whenever a text change occurs.

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Sat, 08 Dec 2018 10:42:31 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>  > Not sure I agree with this interpretation.  My mental model of
>  > try_window_id was that it attempts to reuse what is already on display
>  > by scrolling parts of the displayed stuff, and then inserting/deleting
>  > a few lines that have changed.  It determines which parts of the
>  > display could be reused by analyzing where are the changes in the
>  > buffer relative to the displayed portion.
> 
> The current description of try_window_id in xdisp.c is
> 
>        This function attempts to redisplay a window by reusing parts of
>        its existing display.  It finds and reuses the part that was not
>        changed, and redraws the rest.  (The "id" part in the function's
>        name stands for "insert/delete", not for "identification" or
>        somesuch.)
> 
> Could you kindly add that "scrolling parts of the displayed stuff"
> there somehow.

There's a much more detailed description in the commentary before the
function itself, did you read that?

> For a simple minded reader like me, it's intuitively
> not clear that it can do such virtual scrolling of unchanged contents.

The scrolling is not virtual, it's a real scrolling of what's on the
glass (reflected in the glyph matrix, of course).

> Maybe also tell how many such unaltered chunks can be reused and/or
> how large they must be.

AFAIU, there are no limitations.

>  > try_window_id gives up if overlay changed because it finds the
>  > portions of the displayed text to be reused by looking at buffer text,
>  > and overlays invalidate the assumption that only buffer text
>  > determines what's on screen.
> 
> It wouldn't matter much but I think that redisplay could handle an
> overlay change just like any text property change.
> 
>  >> But why do we optimize changes of text properties as well?
>  >
>  > I'm not sure I understand the question.  What do you mean by "optimize
>  > changes of text properties"?
> 
> I was asking myself why it's important to treat text property changes
> in an optimized way in try_window_id because I thought they wouldn't
> be affected much by typing.  And as far as font-locking is concerned,
> it invalidates the rest of the buffer parts shown in a window anyway
> whenever a text change occurs.

Text properties have the advantage that it's very easy to know where
they start and end in the buffer, and thus this function can know
where in the window the changes begin by just comparing buffer
positions.  By contrast, to know what portions are affected by an
overlay needs a costly access to the overlay data, and the information
needed to deduce which parts of the window are affected by the overlay
changes is not readily available.  I believe this is why the function
gives up if overlays have changed.

Re: Text Properties And Buffer Modification

[Stefan Monnier]

>> > Don't forget that the modified status is important for more than
>> > just saving to a file: it is important for redisplaying the
>> > buffer, for example.
>> Really?
> Is that a rhetorical question?

Not at all.  I know that set-buffer-modified-p has historically been
linked to force-mode-line-update, but other than that I had no idea that
the ticks would affect redisplay in any significant way (beside the
obvious stars in the mode-line, obviously).

I'm still not clear why either.


        Stefan

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Wed, 05 Dec 2018 15:54:08 -0500
> 
> >> > Don't forget that the modified status is important for more than
> >> > just saving to a file: it is important for redisplaying the
> >> > buffer, for example.
> >> Really?
> > Is that a rhetorical question?
> 
> Not at all.  I know that set-buffer-modified-p has historically been
> linked to force-mode-line-update, but other than that I had no idea that
> the ticks would affect redisplay in any significant way (beside the
> obvious stars in the mode-line, obviously).
> 
> I'm still not clear why either.

See my response to Martin.  Some redisplay optimizations need to know
the reason why a buffer should be redisplayed, so they look at the
buffer's modification tick count (and on a similar count for
overlays).  Maybe that's not "significant" for you, but IMO redisplay
optimizations are very important, and disabling them could sometimes
mean severe degradation in responsiveness.

Re: Text Properties And Buffer Modification

[martin rudalics]

 >> (with-silent-modifications
 >>    (with-current-buffer other-buffer
 >>      (put-text-property beg end 'face 'bold other-buffer)))

I suppose this should be

(with-current-buffer other-buffer
   (with-silent-modifications
     (put-text-property beg end 'face 'bold other-buffer)))

since otherwise the modified status and 'buffer-undo-list' are taken
from the current buffer instead from 'other-buffer'.

 > You could instead simply make the target buffer not modified after you
 > are done changing the properties, e.g. by using set-buffer-modified-p.

Which might break in the case where text has been changed in the other
buffer before.

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Wed, 05 Dec 2018 10:16:28 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org
> 
>  > You could instead simply make the target buffer not modified after you
>  > are done changing the properties, e.g. by using set-buffer-modified-p.
> 
> Which might break in the case where text has been changed in the other
> buffer before.

No, because set-buffer-modified-p accepts an argument.

Re: Text Properties And Buffer Modification

[martin rudalics]

 >> Which might break in the case where text has been changed in the other
 >> buffer before.
 >
 > No, because set-buffer-modified-p accepts an argument.

Would we recommend that instead of 'with-silent-modifications'?

martin

Re: Text Properties And Buffer Modification

[Eli Zaretskii]

> Date: Wed, 05 Dec 2018 20:04:14 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org, raman@google.com
> 
>  >> Which might break in the case where text has been changed in the other
>  >> buffer before.
>  >
>  > No, because set-buffer-modified-p accepts an argument.
> 
> Would we recommend that instead of 'with-silent-modifications'?

I didn't recommend anything, I just enumerated the possible solutions.