Thinking about changed buffers

Thinking about changed buffers

[Lars Magne Ingebrigtsen]

In conjunction with the wishlist item "`M-q' shouldn't say that the
buffer hasn't changed when it hasn't", we started talking a bit about
further issues about what it means that a buffer has changed or not.

If you load a file, and then hit "a", and then delete the "a", then
Emacs will say that the buffer has changed.  If you hit "a" and then
`undo', Emacs will say that it hasn't.

If there was a way to deal with this discrepancy, that would be very
nice, I think.

One idea that popped up is that whenever we mark a buffer as unchanged
(that is, `(set-buffer-modified-p nil)', we save the byte size of the
buffer and a cryptographic hash of the buffer.  Then `buffer-modified-p'
would simply see whether either the size had changed, and if not,
whether the hash had changed.  If both are identical, then the buffer
hasn't changed.

This would basically allow us to really tell the users "yes, your buffer
is now back to the state it was when you loaded it".  I think that would
be very nice.

However, there are two problems:

1) Speed.  When editing files normally, `buffer-modified-p' would be
very fast, because buffers would change size, and we'd just be comparing
the sizes and say "yup, changed".  If, however, you're somehow altering
the buffer a lot but always returning to the same size, you'd have to
compute the hash.  (On my five year old, the current implementation
takes 2.7s on a 1GB buffer.)

2) Text properties.  If you call `add-text-properties' on a buffer, the
buffer becomes marked as changed.  The hashing function could look at
the intervals, too, so that's not a problem, but many (most?) of the
text properties are added by font locking mores with
`with-silent-modifications', which means "no, no, these text properties
here don't change the buffer".  But there's nothing in the text
properties themselves that will reveal this after the fact, unless I'm
reading the code incorrectly.

Óscar suggested that to deal with 2), Emacs should simply not regard
text properties as changing the buffer at all, but I think there are
various "rich text" modes that use text properties to generate the
output file (i.e., you put "bold" on some text and it gets written out
as <bold>).  I may be wrong about that.  Anybody know?

Anybody have any thoughts on this issue?

---------
I feel the need to add this, given the way the discussion went in the
`M-q' bug report, but let's hope it's unnecessary:

(Let's take it as a given that, yes, you can create hash collisions, but
that's irrelevant.  In normal, non-cryptographically-constructed text,
the likelihood of two texts having the same MD5 hash is 10^-29 and for
SHA1 it's 10^-39 (if I remember correctly), so it's Not Going To Happen
and we don't need to have that discussions.  (And yes, you can construct
MD5 collisions as fast as you want, but it. is. irrelevant.)  Sheesh.
There's something about cryptography that brings out the most irrelevant
stuff in some people.  If you want to discuss that part, please take it
to emacs-tangents.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Andreas Schwab]

You cannot just look at the decoded contents.  If the coding system is
changed the encoded contents will change even if the buffer contents
didn't.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> You cannot just look at the decoded contents.  If the coding system is
> changed the encoded contents will change even if the buffer contents
> didn't.

I don't think that's correct.  Loading a file does (basically)

1) put the bytes from the file into memory
2) decode them into internal representation
3) (set-buffer-modified-p nil)

It's from that point on we care whether the contents of the buffer has
changed, not from when we loaded the bytes from the file into memory.
So the coding system is irrelevant.

Or do you mean if the user explicitly changes the coding system... in
the buffer?  How does that happen?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Or do you mean if the user explicitly changes the coding system... in
> the buffer?  How does that happen?

C-x C-m f

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Or do you mean if the user explicitly changes the coding system... in
>> the buffer?  How does that happen?
>
> C-x C-m f

Ok, so `buffer-file-coding-system' is one more thing to include in the
hash.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 28 Mar 2016 20:19:13 +0200
> Cc: emacs-devel@gnu.org
> 
> > C-x C-m f
> 
> Ok, so `buffer-file-coding-system' is one more thing to include in the
> hash.

What about annotations?

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Ok, so `buffer-file-coding-system' is one more thing to include in the
>> hash.
>
> What about annotations?

I'm not sure what that means in this context?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Mon, 28 Mar 2016 20:53:39 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Ok, so `buffer-file-coding-system' is one more thing to include in the
> >> hash.
> >
> > What about annotations?
> 
> I'm not sure what that means in this context?

I'm not sure what you mean by "what that means".  Did you know about
write-region-annotate-functions?

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm not sure what you mean by "what that means".  Did you know about
> write-region-annotate-functions?

Nope.  Now that I know about it, I still have no idea what it does.  :-)
And there's nothing in the lispref concept index about "annotation".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Mon, 28 Mar 2016 21:06:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm not sure what you mean by "what that means".  Did you know about
> > write-region-annotate-functions?
> 
> Nope.  Now that I know about it, I still have no idea what it does.  :-)
> And there's nothing in the lispref concept index about "annotation".

Look in fileio.c.

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Nope.  Now that I know about it, I still have no idea what it does.  :-)
>> And there's nothing in the lispref concept index about "annotation".
>
> Look in fileio.c.

I am, but I still don't understand what this has to do with whether we
consider a buffer to be modified or not.  You'll have to spell it out.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Mon, 28 Mar 2016 21:23:50 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Look in fileio.c.
> 
> I am, but I still don't understand what this has to do with whether we
> consider a buffer to be modified or not.  You'll have to spell it out.

It affects how buffer text is written to a disk file.  Like
buffer-file-coding-system.

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> It affects how buffer text is written to a disk file.  Like
> buffer-file-coding-system.

Yes.  But the `set-buffer-file-coding-system' explicitly marks the
buffer as changed when you set it.  Setting, say,
`write-region-annotate-functions' does not.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yes.  But the `set-buffer-file-coding-system' explicitly marks the
> buffer as changed when you set it.  Setting, say,
> `write-region-annotate-functions' does not.

Hm.  Or perhaps including the coding system in the buffer hash is
unnecessary.

I'm just perusing the `set-buffer-modified-p' code and happened onto
this:


  /* Here we have a problem.  SAVE_MODIFF is used here to encode
     buffer-modified-p (as SAVE_MODIFF<MODIFF) as well as
     recent-auto-save-p (as SAVE_MODIFF<auto_save_modified).  So if we
     modify SAVE_MODIFF to affect one, we may affect the other
     as well.
     E.g. if FLAG is nil we need to set SAVE_MODIFF to MODIFF, but
     if SAVE_MODIFF<auto_save_modified that means we risk changing
     recent-auto-save-p from t to nil.
     Vice versa, if FLAG is non-nil and SAVE_MODIFF>=auto_save_modified
     we risk changing recent-auto-save-p from nil to t.  */
  SAVE_MODIFF = (NILP (flag)
		 /* FIXME: This unavoidably sets recent-auto-save-p to nil.  */
		 ? MODIFF
		 /* Let's try to preserve recent-auto-save-p.  */
		 : SAVE_MODIFF < MODIFF ? SAVE_MODIFF
		 /* If SAVE_MODIFF == auto_save_modified == MODIFF,
		    we can either decrease SAVE_MODIFF and auto_save_modified
		    or increase MODIFF.  */
		 : MODIFF++);

:-)

I think instead of all this, we could just introduce a new field in the
buffer struct...  er...  explicit_modified_flag_p.  Calling
`(set-buffer-modified-p t)' would just set that, and `buffer-modified-p'
would just return that.

(`(set-buffer-modified-p nil)' would clear it, of course, in addition to
doing everything else it's doing.)

So then we're back to just hashing the buffer contents and (some) text
properties, I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Mon, 28 Mar 2016 22:21:36 +0200
> 
>   /* Here we have a problem.  SAVE_MODIFF is used here to encode
>      buffer-modified-p (as SAVE_MODIFF<MODIFF) as well as
>      recent-auto-save-p (as SAVE_MODIFF<auto_save_modified).  So if we
>      modify SAVE_MODIFF to affect one, we may affect the other
>      as well.
>      E.g. if FLAG is nil we need to set SAVE_MODIFF to MODIFF, but
>      if SAVE_MODIFF<auto_save_modified that means we risk changing
>      recent-auto-save-p from t to nil.
>      Vice versa, if FLAG is non-nil and SAVE_MODIFF>=auto_save_modified
>      we risk changing recent-auto-save-p from nil to t.  */
>   SAVE_MODIFF = (NILP (flag)
> 		 /* FIXME: This unavoidably sets recent-auto-save-p to nil.  */
> 		 ? MODIFF
> 		 /* Let's try to preserve recent-auto-save-p.  */
> 		 : SAVE_MODIFF < MODIFF ? SAVE_MODIFF
> 		 /* If SAVE_MODIFF == auto_save_modified == MODIFF,
> 		    we can either decrease SAVE_MODIFF and auto_save_modified
> 		    or increase MODIFF.  */
> 		 : MODIFF++);
> 
> :-)
> 
> I think instead of all this, we could just introduce a new field in the
> buffer struct...  er...  explicit_modified_flag_p.  Calling
> `(set-buffer-modified-p t)' would just set that, and `buffer-modified-p'
> would just return that.
> 
> (`(set-buffer-modified-p nil)' would clear it, of course, in addition to
> doing everything else it's doing.)
> 
> So then we're back to just hashing the buffer contents and (some) text
> properties, I think?

If you do that, you'll need to modify all the places that look at
MODIFF, SAVE_MODIFF, and their buffer-specific variants.  In
particular, the display engine.

Re: Thinking about changed buffers

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Date: Mon, 28 Mar 2016 20:19:13 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> > C-x C-m f
>> 
>> Ok, so `buffer-file-coding-system' is one more thing to include in the
>> hash.
>
> What about annotations?

write-file-functions, write-contents-functions, before-save-hook

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Thinking about changed buffers

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 28 Mar 2016 20:00:30 +0200
> Cc: emacs-devel@gnu.org
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> > You cannot just look at the decoded contents.  If the coding system is
> > changed the encoded contents will change even if the buffer contents
> > didn't.
> 
> I don't think that's correct.  Loading a file does (basically)
> 
> 1) put the bytes from the file into memory
> 2) decode them into internal representation

Decoding can change the bytes in memory.  For example, when a Latin-1
text is decoded, you end up with (2-byte) UTF-8 sequences for every
non-ASCII character in memory that was only one byte on disk.

> 3) (set-buffer-modified-p nil)

Yes, and if the user changes the encoding, then saves the buffer, the
file on disk will have different bytes.

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> 2) Text properties.  If you call `add-text-properties' on a buffer, the
> buffer becomes marked as changed.  The hashing function could look at
> the intervals, too, so that's not a problem, but many (most?) of the
> text properties are added by font locking mores with
> `with-silent-modifications'

Ok, I've been grepping through the code a bit.  It also just occurred to
me that people might be using `with-silent-modifications' to make
textual changes in a buffer that "shouldn't" be noticed.  I have not
read all the matches in detail, of course, but I could find only two
instances where text was obviously being modified.  And they both had
this comment:

    (with-silent-modifications
      ;; FIXME: Why not use a temporary buffer and avoid this
      ;; "insert&delete" business?  --Stef
      (insert-file-contents file)

That is, it's code that inserts something in the buffer and then deletes
it.  Which would work just fine if we'd use a hash to keep track of file
modification-ness.   :-)

Another thing that occurred to me is that `with-silent-modification'
could make `add-text-properties' add a special flag to the interval
saying "this interval should not be counted when computing the hash".  I
think that would be a kinda trivial thing to do...

Hm...  I'm starting to think that this might be feasible.  I didn't
think so when I wrote the original email.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

RE: Thinking about changed buffers

[Stephan Mueller]

On Monday, March 28, 2016 10:31 AM Lars writes:

" If you load a file, and then hit "a", and then delete the "a", then
" Emacs will say that the buffer has changed.  If you hit "a" and then
" `undo', Emacs will say that it hasn't.

Personally, I have relied on the a-then-delete behavior when I
_want_ the file to be considered modified*.  That is, my mental
model of 'modified' matches the simplistic approach currently
implemented.  (And I use undo when I don't want to have the
buffer marked modified).

I'm not convinced that this aspect (text change-and-happens-to
-get-restored) is worth correcting.  Or even, a flaw at all; I suppose
It may depend on whether one reads 'modified' as meaning "has
been touched" or as "is no longer identical".

I've not come across any other editor with the proposed behaviour;
perhaps it's worth considering "compatibility with typical expectations"
here as well as "can Emacs be better in this regard than other editors".

stephan();

* that said, I don't necessarily have a compelling reason for wanting
a modified buffer.  I've used it to ensure that revert-buffer will
reload the file to trigger a re-colourization of the buffer, fixing
cases where cperl-mode got confused.  There's lots wrong with this
scenario: I should pursue the cperl issue, and I've since found that
the recolourization happens on revert-buffer without it being
modified, but the point remains that char-then-backspace is a
concise, intuitive, idiom that works with the current simple model.

Re: Thinking about changed buffers

[Stefan Monnier]

> Personally, I have relied on the a-then-delete behavior when I
> _want_ the file to be considered modified*.

Same here (tho I use SPC DEL instead).

> * that said, I don't necessarily have a compelling reason for wanting
> a modified buffer.

For me, the full idiom is "SPC DEL C-x C-s" which I use to force Emacs
to save the file.  I'm not sure exactly where/when I need it (it's too
ingrained in muscle memory), but I think it's mostly when I edit a LaTeX
document while it's being compiled, because running "pdflatex" takes
enough time that I may end up saving a file before the end of the
previous compilation is done, so the file's timstamps don't faithfully
record the "freshness" of the output.

> I should pursue the cperl issue, and I've since found that the
> recolourization happens on revert-buffer without it being modified,
> but the point remains that char-then-backspace is a concise,
> intuitive, idiom that works with the current simple model.

For this, `SPC C-x u' at point-min should be sufficient to cause
cperl-mode (and perl-mode as well) to flush all the existing
highlighting data.


        Stefan

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> For me, the full idiom is "SPC DEL C-x C-s" which I use to force Emacs
> to save the file.  I'm not sure exactly where/when I need it (it's too
> ingrained in muscle memory), but I think it's mostly when I edit a LaTeX
> document while it's being compiled, because running "pdflatex" takes
> enough time that I may end up saving a file before the end of the
> previous compilation is done, so the file's timstamps don't faithfully
> record the "freshness" of the output.

Now that you mention it, I remember doing that, too.  :-)

Does this perhaps mean that the interactive `C-x C-s' command should
always save the file, even if it's unchanged?  We've apparently all
taken to doing pointless modifications because we haven't found an easy
way to make Emacs re-save the file...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 367 bytes --]

On 03/28/2016 09:20 PM, Lars Magne Ingebrigtsen wrote:
> Does this perhaps mean that the interactive `C-x C-s' command should
> always save the file, even if it's unchanged?  We've apparently all
> taken to doing pointless modifications because we haven't found an easy
> way to make Emacs re-save the file...

Indeed, I was going to suggest exactly that :)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Thinking about changed buffers

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> For me, the full idiom is "SPC DEL C-x C-s" which I use to force Emacs
>> to save the file. [snip]
>
> Now that you mention it, I remember doing that, too.  :-)

Me too... until I started to use all sorts of electric modes which
gurantee that a mindless char&delete will create a mess.

> Does this perhaps mean that the interactive `C-x C-s' command should
> always save the file, even if it's unchanged?

Yes. With an option to turn it off, of course :-)

[snip]

RE: Thinking about changed buffers

[Stephan Mueller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

" For this, `SPC C-x u' at point-min should be sufficient to cause
" cperl-mode (and perl-mode as well) to flush all the existing
" highlighting data.

Thanks, I'll try that. Moving to point-min and back may make
that more awkward than revert-buffer.  Regardless, at the very
least, I should hide one or the other approach in a function and
bind it to a key.

Lars writes:

" Does this perhaps mean that the interactive `C-x C-s' command should
" always save the file, even if it's unchanged?  We've apparently all
" taken to doing pointless modifications because we haven't found an easy
" way to make Emacs re-save the file...

I follow your logic, but I'm not sure about this simplification; I
find that I hit `C-x C-s' all the time (whenever I enter my mental
idle loop, perhaps), and am pleased that it doesn't do anything
unnecessarily on those extremely numerous occasions.  Perhaps
a new,  distinct command really-save-buffer, bound to C-x C-S-s ?
 Or prefix save-buffer with four `C-u's (1 through 3 are already
taken)?

stephan();

Re: Thinking about changed buffers

[Marcin Borkowski]

On 2016-03-28, at 21:13, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> Personally, I have relied on the a-then-delete behavior when I
>> _want_ the file to be considered modified*.
>
> Same here (tho I use SPC DEL instead).

And here.

>> * that said, I don't necessarily have a compelling reason for wanting
>> a modified buffer.
>
> For me, the full idiom is "SPC DEL C-x C-s" which I use to force Emacs
> to save the file.  [...]

Same here.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> 1) Speed.

And I had an idea here, too.  :-)  `buffer-modified-p' is today just
this simple integer comparison:

DEFUN ("buffer-modified-p", Fbuffer_modified_p, Sbuffer_modified_p,
[...]
  return BUF_SAVE_MODIFF (buf) < BUF_MODIFF (buf) ? Qt : Qnil;
}

How could it look under a hashey regime?  Pseudoish code coming up:

{
  if (BUF_SAVE_MODIFF (buf) == BUF_MODIFF (buf))
    return Qnil;
  if (BUF_SAVE_HASHED_SIZE (buf) != BUF_HASHED_SIZE (buf))
    return Qt;
  same = string_equal(buffer_hash (buf), buf->save_hash);
  if (! NILP (same))
    buf->text->save_modiff = buf->text->modiff;
  return same;
}

Uhm...  is that correct?  *squints*  Perhaps.

Anyway, if we do it this way (that is, keep the modiff counter around),
we can short circuit the hash calculation like, almost always.  It would
only be necessary (the first time per ... time) the buffer reaches the
same size that it used to be.

That is, it would basically happen in the "insert an 'a' and then delete
the 'a'" case, but very seldom otherwise, so `buffer-modified-p' would
be almost as fast as it usually is.

Now I'm kinda starting to think that this isn't just feasible, but
possible in practice, too.  :-)

I might be totally wrong, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Stefan Monnier]

>   if (BUF_SAVE_MODIFF (buf) == BUF_MODIFF (buf))
>     return Qnil;

This test basically never succeeds except when we already know the file
in not modified.

> we can short circuit the hash calculation like, almost always.  It would
> only be necessary (the first time per ... time) the buffer reaches the
> same size that it used to be.

Still seems problematic if your 5 year old takes 2.7s to compute it on
a 1GB file.  You don't want to freeze for 2s in the normal course of
editing just because you happen to cross the "original size" threshold.


        Stefan

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>   if (BUF_SAVE_MODIFF (buf) == BUF_MODIFF (buf))
>>     return Qnil;
>
> This test basically never succeeds except when we already know the file
> in not modified.

It's to avoid having the hashing function being called several times in
a row if you happen to have the "original size" in the buffer.  It's
what makes this hash only being called the first time per ... time.  :-)

>> we can short circuit the hash calculation like, almost always.  It would
>> only be necessary (the first time per ... time) the buffer reaches the
>> same size that it used to be.
>
> Still seems problematic if your 5 year old takes 2.7s to compute it on
> a 1GB file.  You don't want to freeze for 2s in the normal course of
> editing just because you happen to cross the "original size" threshold.

Yeah, I don't see any way around that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Dmitry Gutov]

On 03/28/2016 10:27 PM, Lars Magne Ingebrigtsen wrote:

>> Still seems problematic if your 5 year old takes 2.7s to compute it on
>> a 1GB file.  You don't want to freeze for 2s in the normal course of
>> editing just because you happen to cross the "original size" threshold.
>
> Yeah, I don't see any way around that.

Don't use hashing. Use e.g. buffer-undo-list. We save enough data to 
return the buffer contents to the previous state, right? It should be 
possible to detect whether a given sequence of undo-s is a no-op.

Re: Thinking about changed buffers

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 761 bytes --]



On 03/28/2016 09:32 PM, Dmitry Gutov wrote:
> On 03/28/2016 10:27 PM, Lars Magne Ingebrigtsen wrote:
> 
>>> Still seems problematic if your 5 year old takes 2.7s to compute it on
>>> a 1GB file.  You don't want to freeze for 2s in the normal course of
>>> editing just because you happen to cross the "original size" threshold.
>>
>> Yeah, I don't see any way around that.
> 
> Don't use hashing. Use e.g. buffer-undo-list. We save enough data to return the buffer contents to the previous state, right? It should be possible to detect whether a given sequence of undo-s is a no-op.

Alternatively, introduce a threshold above which that hash-based check does not happen, and instead fall back to the old, less complex behaviour in that case.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> Alternatively, introduce a threshold above which that hash-based check
> does not happen, and instead fall back to the old, less complex
> behaviour in that case.

That's indeed a possibility.  We already disable other CPU intensive
operations in huge buffers...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[Stefan Monnier]

>> Still seems problematic if your 5 year old takes 2.7s to compute it on
>> a 1GB file.  You don't want to freeze for 2s in the normal course of
>> editing just because you happen to cross the "original size" threshold.
> Yeah, I don't see any way around that.

I guess to do better, you'd have to keep track of some bounds of
changes, and to only hash the part of the buffer within those bounds.
E.g. instead of a single hash, keep a list of hash-per-megabyte, plus
a bitmap of touched-megabytes (one bit per megabyte), so you can limit
your check to just those megabytes that have been touched.

This said, it doesn't strike me as a worthwhile exercise.  The benefit
seems very small, and the extra work is out of proportion to that
benefit (and then you have to add the backward incompatibilities this
will inevitably introduce).


        Stefan

Re: Thinking about changed buffers

[Florian Weimer]

On 03/28/2016 07:31 PM, Lars Magne Ingebrigtsen wrote:
> In conjunction with the wishlist item "`M-q' shouldn't say that the
> buffer hasn't changed when it hasn't", we started talking a bit about
> further issues about what it means that a buffer has changed or not.
> 
> If you load a file, and then hit "a", and then delete the "a", then
> Emacs will say that the buffer has changed.  If you hit "a" and then
> `undo', Emacs will say that it hasn't.
> 
> If there was a way to deal with this discrepancy, that would be very
> nice, I think.

I feel a bit silly, but I rely on this discrepancy (SPC DEL marks the
buffer as changed) so that saving the buffer is an actual operation
which updates the file timestamp, to trigger a rebuild of its dependencies.

> One idea that popped up is that whenever we mark a buffer as unchanged
> (that is, `(set-buffer-modified-p nil)', we save the byte size of the
> buffer and a cryptographic hash of the buffer.  Then `buffer-modified-p'
> would simply see whether either the size had changed, and if not,
> whether the hash had changed.  If both are identical, then the buffer
> hasn't changed.

For me, the major nuisance is that Emacs claims that the file has
changed on disk, when it fact it hasn't—the mtime changed, but the
contents is the same.  At least it's easier to detect this case with the
hashing approach you propose.

Florian

Re: Thinking about changed buffers

[Phillip Lord]

On Mon, March 28, 2016 6:31 pm, Lars Magne Ingebrigtsen wrote:
> One idea that popped up is that whenever we mark a buffer as unchanged
> (that is, `(set-buffer-modified-p nil)', we save the byte size of the
> buffer and a cryptographic hash of the buffer.  Then `buffer-modified-p'
> would simply see whether either the size had changed, and if not, whether
> the hash had changed.  If both are identical, then the buffer hasn't
> changed.
>


Rather than doing this in general, why not have a
"with-potentially-unmodified" macro, which calculates this hash and
restores buffer-modified-p if no changes have happened. This would solve
the "fill-paragraph" problem.

Although, I would worry about the implications for the before and
after-change hooks. fill-paragraph is already problematic here because
before and after changes hooks are not necessarily consistent. In the case
where no changes happen, would we expect before and after change hook to
run? Because they would have to, as we do not know whether the buffer will
change before we change.

The only solution I can see for fill-paragraph is to copy the paragraph to
a temp buffer, fill that, check whether it has changed, then if it has,
signal before-change, copy the changed paragraph back, signal after
change. If it has not changed, then fill-paragraph becomes a no-op.

Phil

Re: Thinking about changed buffers

[Stefan Monnier]

> The only solution I can see for fill-paragraph is to copy the paragraph to
> a temp buffer, fill that, check whether it has changed, then if it has,
> signal before-change, copy the changed paragraph back, signal after
> change. If it has not changed, then fill-paragraph becomes a no-op.

BTW, another solution for fill-paragraph is to reimplement it from
scratch such that it doesn't "unfill+refill".  I.e. it could instead
loop go to each LF in the paragraph, such if that LF is found to be "at
the right spot", it moves on, if it's "too soon", then it removes it,
and if it's "too far" then it adds another somewhere earlier.

This would naturally lead to the result that the buffer would stay
unmodified if there was no change.


        Stefan

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

I think the conclusion here is that this would be feasible to
implement.  There's a lot of details you'd have to cover, but it's
doable.

And it wouldn't have to be enabled (for those that enable it) in very
large buffers.  And if so, it'd be fast enough that you wouldn't notice
it.

However, it seems like it would be a rather unusual feature for an
editor to have, and many people might not like it, so it's perhaps not
worth doing.

But one take away from this discussion is that Emacs needs a convenient
command to save an unchanged buffer.  I think I'll file a wishlist bug
report for that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[John Wiegley]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But one take away from this discussion is that Emacs needs a convenient
> command to save an unchanged buffer. I think I'll file a wishlist bug report
> for that.

C-x C-w RET will do that, and I use it often for that effect.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Thinking about changed buffers

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 485 bytes --]

On 04/04/2016 12:05 AM, John Wiegley wrote:
>>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
>> But one take away from this discussion is that Emacs needs a convenient
>> command to save an unchanged buffer. I think I'll file a wishlist bug report
>> for that.
> 
> C-x C-w RET will do that, and I use it often for that effect.

You need two `RET's, actually, don't you? And it shows a scary overwrite message. Also, it doesn't work with ido-mode, unfortunately.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Thinking about changed buffers

[John Wiegley]

>>>>> Clément Pit--Claudel <clement.pit@gmail.com> writes:

>> C-x C-w RET will do that, and I use it often for that effect.

> You need two `RET's, actually, don't you? And it shows a scary overwrite
> message. Also, it doesn't work with ido-mode, unfortunately.

Ah, maybe. And for ido, I think you just need to use C-j in place of RET.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Thinking about changed buffers

[Óscar Fuentes]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

>>> But one take away from this discussion is that Emacs needs a
>>> convenient command to save an unchanged buffer. I think I'll file a
>>> wishlist bug report for that.
>> 
>> C-x C-w RET will do that, and I use it often for that effect.
>
> You need two `RET's, actually, don't you?

To be precise:

C-x C-w RET y RET

> And it shows a scary
> overwrite message. Also, it doesn't work with ido-mode, unfortunately.

For avoiding the scary overwrite confirmation you use the prefix
argument:

C-u C-x C-w RET

Those of us who use ido-mode:

C-u C-x C-w C-j

IMO, those sequences are above of the threshold of what can be
considered convenient (definitely not recommended if you are prone to
suffering from RSI) and I expressed my preference for a new shorcut.
However, others were quick at dismissing that possibility.

Re: Thinking about changed buffers

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 455 bytes --]

On 04/04/2016 12:44 AM, Óscar Fuentes wrote:
> C-u C-x C-w C-j
> 
> IMO, those sequences are above of the threshold of what can be
> considered convenient

Agreed :) What about a setting that would make C-x C-s always save, regardless of whether saving is actually needed?

Also, I'd argue that it's almost a bug that you get a overwriting warning using C-x C-w when you're writing to a patch equal to buffer-file-name. Isn't it?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Thinking about changed buffers

[Óscar Fuentes]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> Agreed :) What about a setting that would make C-x C-s always save,
> regardless of whether saving is actually needed?

An option for `save-buffer' saving inconditionally when called
interactively would be useful.

> Also, I'd argue that it's almost a bug that you get a overwriting
> warning using C-x C-w when you're writing to a patch equal to
> buffer-file-name. Isn't it?

Yea, a QoI issue.

Re: Thinking about changed buffers

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Those of us who use ido-mode:
>
> C-u C-x C-w C-j
>
> IMO, those sequences are above of the threshold of what can be
> considered convenient (definitely not recommended if you are prone to
> suffering from RSI) and I expressed my preference for a new shorcut.
> However, others were quick at dismissing that possibility.

I think it's a psychological issue.  If you've accepted doing convoluted
things for years because your favourite tool lacks obvious
functionality, then it's quite common to feel that that should never be
fixed.  Because fixing the problem would mean accepting that you've done
silly things for years as a work around, and the obvious fix becomes a
personal insult, almost.

Or perhaps it's just that it's Monday and I'm grouchy.  Ier.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Thinking about changed buffers

[John Wiegley]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think it's a psychological issue. If you've accepted doing convoluted
> things for years because your favourite tool lacks obvious functionality,
> then it's quite common to feel that that should never be fixed. Because
> fixing the problem would mean accepting that you've done silly things for
> years as a work around, and the obvious fix becomes a personal insult,
> almost.

I think the first part is certainly true. I've become so used to a silly
command sequence that I see nothing wrong it; but if I had to explain it to
someone else, there would be some shame. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Thinking about changed buffers

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The only solution I can see for fill-paragraph is to copy the paragraph to
>> a temp buffer, fill that, check whether it has changed, then if it has,
>> signal before-change, copy the changed paragraph back, signal after
>> change. If it has not changed, then fill-paragraph becomes a no-op.
>
> BTW, another solution for fill-paragraph is to reimplement it from
> scratch such that it doesn't "unfill+refill".  I.e. it could instead
> loop go to each LF in the paragraph, such if that LF is found to be "at
> the right spot", it moves on, if it's "too soon", then it removes it,
> and if it's "too far" then it adds another somewhere earlier.
>
> This would naturally lead to the result that the buffer would stay
> unmodified if there was no change.


Yes, that would seem a clean solution, as it would avoid changing a
buffer which might not need to be changed. It would also mean that a
fill-paragraph which doesn't change anything would leave no undo events.

Phil

Re: Thinking about changed buffers

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, another solution for fill-paragraph is to reimplement it from scratch
> such that it doesn't "unfill+refill". I.e. it could instead loop go to each
> LF in the paragraph, such if that LF is found to be "at the right spot", it
> moves on, if it's "too soon", then it removes it, and if it's "too far" then
> it adds another somewhere earlier.

I too think this is the best solution, and I believe it was mentioned in the
bug thread on this subject.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2