bug#15797: 24.3.50; Info: Mention cache-long-scans

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

NEWS file seems to be lying wrt `cache-long-line-scans' or
`cache-long-scans'.  I see no entry for these in the manual.

Temporary note:
+++ indicates that all necessary updates to the manuals in doc/ are complete.


+++
** `cache-long-line-scans' has been renamed to `cache-long-scans'
because it affects caching of paragraph scanning results as well.

----------------------------------------------------------------

In GNU Emacs 24.3.50.4 (i686-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2013-10-30 on debian-6.05
Bzr revision: 114868 rgm@gnu.org-20131030102316-8vif7u6ecyo3yieg
Windowing system distributor `The X.Org Foundation', version 11.0.10707000
System Description:	Debian GNU/Linux 6.0.5 (squeeze)

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Glenn Morris]

cd doc
find . -name '*.texi' -exec grep 'cache-long.*scans' {} +

./lispref/display.texi:@code{cache-long-scans} to @code{t}.
./lispref/display.texi:@defvar cache-long-scans
./lispref/positions.texi:performance of your code.  @xref{Truncation, cache-long-scans}.

So: cache-long-scans is defined, and there are no references to the old
name cache-long-line-scans.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Why is this variable in lispref.  The variable deserves atleast a
mention in the main manual.

Glenn Morris <rgm@gnu.org> writes:

> cd doc
> find . -name '*.texi' -exec grep 'cache-long.*scans' {} +
>
> ./lispref/display.texi:@code{cache-long-scans} to @code{t}.
> ./lispref/display.texi:@defvar cache-long-scans
> ./lispref/positions.texi:performance of your code.  @xref{Truncation,
> cache-long-scans}.
>
> So: cache-long-scans is defined, and there are no references to the old
> name cache-long-line-scans.

bug#15797: closed (Re: bug#15797: 24.3.50; Info: Mention cache-long-scans)

[Jambunathan K]

Glenn,

I will not agree that this bug is closed unless (atleast) a reference to
this variable is made available in the Emacs manual.

Please talk to the OP before closing the bug.  Anyways ...

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Michael Heerdegen]

Jambunathan K <kjambunathan@gmail.com> writes:

> Why is this variable in lispref.  The variable deserves atleast a
> mention in the main manual.

A small note by a user: In my experience, line caching is a highly
experimental feature.

The first time I tried it - this was 2 or 3 years ago, Emacs crashed
reproducably few seconds after setting the variable to non-nil in any
buffer (bug 12196).  Presumably nobody using Emacs used it that time.

After this had been fixed, I tried to work with cache-long-line-scans
non-nil, and after some minutes I found that it breaks dired (bug 15148,
still not fixed).  Not sure how much other problems are to be
discovered.

So, it's probably better not to advertise it until it works ok.  I would
love to use it when it worked, btw.


Regards,

Michael.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> So, it's probably better not to advertise it until it works ok.

What comes first: Hen or the Egg?  Difficult to answer.

I happened to suggest this variable on Org-mode list.  Having long lines
(i.e., avoiding wrapped lines) has it's advantages.  It helps produce
sensible diffs.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Mon, 04 Nov 2013 14:10:10 +0100
> Cc: 15797@debbugs.gnu.org
> 
> A small note by a user: In my experience, line caching is a highly
> experimental feature.

I wonder if we should simply turn it on by default, and leave it
there.  That way, any bugs that it exposes will be flushed out very
quickly.  Stefan?

> After this had been fixed, I tried to work with cache-long-line-scans
> non-nil, and after some minutes I found that it breaks dired (bug 15148,
> still not fixed).

That bug is now fixed.  Sorry, it just seemed to fall through the
cracks.  The actual problem was very simple, and the fix is a
one-liner.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Stefan Monnier]

> I wonder if we should simply turn it on by default, and leave it
> there.  That way, any bugs that it exposes will be flushed out very
> quickly.  Stefan?

Let's try it.


        Stefan

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Let's try it.

Please don't hide the variable in lispref.  

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > After this had been fixed, I tried to work with cache-long-line-scans
> > non-nil, and after some minutes I found that it breaks dired (bug 15148,
> > still not fixed).
>
> That bug is now fixed.  Sorry, it just seemed to fall through the
> cracks.  The actual problem was very simple, and the fix is a
> one-liner.

Thanks for fixing, Eli.  I'm using cache-long-scans t for 10 hours now
and didn't see any more problems.

BTW, when cache-long-scans t works now, is there any benefit in setting
it nil?  The manual says that the only downside is that it makes
processing of short lines slightly slower - which is presumably
negligible on modern hardware.


Regards,

Michael.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Wed, 06 Nov 2013 19:02:11 +0100
> 
> BTW, when cache-long-scans t works now, is there any benefit in setting
> it nil?

We will shortly turn it on by default, as you see from the rest of
this discussion.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Michael Heerdegen <michael_heerdegen@web.de>
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> > 15797@debbugs.gnu.org, kjambunathan@gmail.com
> > Date: Wed, 06 Nov 2013 19:02:11 +0100
> > 
> > BTW, when cache-long-scans t works now, is there any benefit in setting
> > it nil?
>
> We will shortly turn it on by default, as you see from the rest of
> this discussion.

Sorry, you misunderstood.  What I meant was: do we (perspectively) need
a variable at all, when a nil value only has downsides?  Why should
anybody want to configure this variable to nil (the default value is t
now) when it has only disadvantages?  That's what I meant.

Michael.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Drew Adams]

> > BTW, when cache-long-scans t works now, is there any benefit in
> > setting it nil?
> 
> We will shortly turn it on by default, as you see from the rest of
> this discussion.

Only for visual-line-mode, or in general?  Only when there are
actually long lines in the buffer, or in general?

And the question was not just about the default behavior, but
whether there is (ever) any benefit in setting it to nil.

The variable is always buffer-local.  If it will be on by default,
will it ever be turned off?  If so, just what turns it off?  (I
assume I can turn it off explicitly in a given buffer, but what
else might turn it off?)

Also, I wonder about this part of the doc (I don't have the C
source code to check what it really does):

  "If `cache-long-scans' is non-nil, these motion functions cache
   the results of their scans"

That does not say that they cache only the result of scanning long
lines.  Is that correct?  Do they cache the result of scanning even
short lines?

[BTW, the doc speaks of both "the cache" (singular) and "the
caches", which is confusing.  Please pick one, or if there is some
reason for using both then please make it clear: why the difference,
etc.]

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: monnier@iro.umontreal.ca,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Wed, 06 Nov 2013 19:50:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > From: Michael Heerdegen <michael_heerdegen@web.de>
> > > Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> > > 15797@debbugs.gnu.org, kjambunathan@gmail.com
> > > Date: Wed, 06 Nov 2013 19:02:11 +0100
> > > 
> > > BTW, when cache-long-scans t works now, is there any benefit in setting
> > > it nil?
> >
> > We will shortly turn it on by default, as you see from the rest of
> > this discussion.
> 
> Sorry, you misunderstood.  What I meant was: do we (perspectively) need
> a variable at all, when a nil value only has downsides?

You are rushing forward too fast, IMO.  Just yesterday you said the
feature was highly experimental.  Let's have it on by default for a
while, before we decide; meanwhile people will at least have a fire
escape.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> Date: Wed, 6 Nov 2013 10:58:55 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 15797@debbugs.gnu.org, kjambunathan@gmail.com
> 
> > > BTW, when cache-long-scans t works now, is there any benefit in
> > > setting it nil?
> > 
> > We will shortly turn it on by default, as you see from the rest of
> > this discussion.
> 
> Only for visual-line-mode, or in general?  Only when there are
> actually long lines in the buffer, or in general?

Always.

> And the question was not just about the default behavior, but
> whether there is (ever) any benefit in setting it to nil.

There's overhead of having it non-nil, but it is small, or so we
think.

> The variable is always buffer-local.  If it will be on by default,
> will it ever be turned off?

When the user does that.

> I assume I can turn it off explicitly in a given buffer, but what
> else might turn it off?

Nothing.

> Also, I wonder about this part of the doc (I don't have the C
> source code to check what it really does):
> 
>   "If `cache-long-scans' is non-nil, these motion functions cache
>    the results of their scans"
> 
> That does not say that they cache only the result of scanning long
> lines.  Is that correct?

Yes.

> Do they cache the result of scanning even short lines?

Yes.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Michael Heerdegen <michael_heerdegen@web.de>,  kjambunathan@gmail.com,  15797@debbugs.gnu.org
> Date: Tue, 05 Nov 2013 14:24:28 -0500
> 
> > I wonder if we should simply turn it on by default, and leave it
> > there.  That way, any bugs that it exposes will be flushed out very
> > quickly.  Stefan?
> 
> Let's try it.

Done as trunk revision 115033.  Let's see what becomes broken by this.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Jambunathan K <kjambunathan@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen <michael_heerdegen@web.de>,  15797@debbugs.gnu.org
> Date: Wed, 06 Nov 2013 10:14:37 +0530
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > Let's try it.
> 
> Please don't hide the variable in lispref.  

Given that the variable is now on by default, and I see no reason why
anyone would might want to turn it off, except fro debugging, does it
still make sense to have this advertised in the user manual?  If so,
why?

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Eli Zaretskii <eliz@gnu.org> writes:

> Given that the variable is now on by default, and I see no reason why
> anyone would might want to turn it off, except fro debugging, does it
> still make sense to have this advertised in the user manual?  If so,
> why?

If you are super-confident, remove the variable :-).

Otherwise, re-open the bug and mark it as "wait-and-watch" for the
current release and "must-test" feature for the beta builds.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Stefan Monnier]

> Otherwise, re-open the bug and mark it as "wait-and-watch" for the
> current release and "must-test" feature for the beta builds.

I think the variable belongs neither in the Emacs manual nor the
Lisp manual.  It's only useful for debugging performance problems.


        Stefan

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the variable belongs neither in the Emacs manual nor the
> Lisp manual.

or NEWS. 

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Jambunathan K <kjambunathan@gmail.com>
> Cc: monnier@iro.umontreal.ca,  michael_heerdegen@web.de,  15797@debbugs.gnu.org
> Date: Fri, 08 Nov 2013 18:43:21 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Given that the variable is now on by default, and I see no reason why
> > anyone would might want to turn it off, except fro debugging, does it
> > still make sense to have this advertised in the user manual?  If so,
> > why?
> 
> If you are super-confident, remove the variable :-).

I asked why it would make sense to have it in user manual if it is
_not_ removed, but is ON by default.

> Otherwise, re-open the bug

I didn't close it.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Jambunathan K]

Eli Zaretskii <eliz@gnu.org> writes:

> I asked why it would make sense to have it in user manual if it is
> _not_ removed, but is ON by default.

Both of us are argumentative.  We should stop before it gets worse :-).

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Jambunathan K <kjambunathan@gmail.com>
> Cc: 15797@debbugs.gnu.org
> Date: Fri, 08 Nov 2013 20:03:07 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I asked why it would make sense to have it in user manual if it is
> > _not_ removed, but is ON by default.
> 
> Both of us are argumentative.

I'm not.  My question was real and serious.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Nathan Trapuzzano]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I wonder if we should simply turn it on by default, and leave it
>> there.  That way, any bugs that it exposes will be flushed out very
>> quickly.  Stefan?
>
> Let's try it.

Sorry I'm late to the party, but this broke linum and nlinum for me,
using the defaults.  Any time I insert a new line, the line numbers get
totally messed up--most of them don't even display any more.  And then
using motion commands sometimes results in really bizarre behavior
(apparently only with linum/nlinum so far), such as when I do
forwad-sexp and point goes to the end of some sexp other than the one at
point.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Stefan Monnier]

> Sorry I'm late to the party, but this broke linum and nlinum for me,
> using the defaults.

I tried:

  src/emacs -Q src/xdisp.c -l .../nlinum.el -f nlinum-mode

and then moved about in the buffer, but I didn't notice any problem.


        Stefan

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen <michael_heerdegen@web.de>,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Fri, 08 Nov 2013 14:07:44 -0500
> 
> Sorry I'm late to the party, but this broke linum and nlinum for me,
> using the defaults.

Any breakage that this does means bugs somewhere in Emacs, because the
same function that invalidates the scan cache when a buffer is
modified also adjusts overlays and markers.  So turning this option on
does us a favor by exposing hidden bugs that are otherwise extremely
hard to reproduce.

> Any time I insert a new line, the line numbers get
> totally messed up--most of them don't even display any more.  And then
> using motion commands sometimes results in really bizarre behavior
> (apparently only with linum/nlinum so far), such as when I do
> forwad-sexp and point goes to the end of some sexp other than the one at
> point.

I cannot reproduce this, and neither can Stefan, so it seems.  Please
provide a recipe for reproducing these problems.

Thanks.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Glenn Morris]

Eli Zaretskii wrote:

> I cannot reproduce this, and neither can Stefan, so it seems.  Please
> provide a recipe for reproducing these problems.

I suggest opening a new report for those issues.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Nathan Trapuzzano]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> I tried:
>
>   src/emacs -Q src/xdisp.c -l .../nlinum.el -f nlinum-mode
>
> and then moved about in the buffer, but I didn't notice any problem.

Try this with the attached file:

src/emacs -nw -Q foo.el

M-x linum-mode    (you may notice the problem already here)
M-x goto-line 10
C-o

The numbers in the margin get messed up.  I haven't figured out a
consistent way to screw up the motion commands yet.


[-- Attachment #2: foo.el --]
[-- Type: application/emacs-lisp, Size: 1401 bytes --]

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Nathan Trapuzzano]

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> I haven't figured out a consistent way to screw up the motion commands
> yet.

Here we go.  Visit the same file and do:

M-x linum-mode
C-o
C-n
C-e

Cursor moves almost to end of entire file.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Michael Heerdegen]

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> Sorry I'm late to the party, but this broke linum and nlinum for me,
> using the defaults.  Any time I insert a new line, the line numbers get
> totally messed up--most of them don't even display any more.  And then
> using motion commands sometimes results in really bizarre behavior
> (apparently only with linum/nlinum so far), such as when I do
> forwad-sexp and point goes to the end of some sexp other than the one at
> point.

I see something probably related.  Sometimes, evaluating

  (line-number-at-pos)

at the end of my .emacs (10000 lines) needs over a second.  Doing

  (setq cache-long-scans nil)

immediately fixes this.  Dunno yet how to provoke the problem.


Regards,

Michael.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

Redirected to a new bug report, please use this one in the future.

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen <michael_heerdegen@web.de>,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Fri, 08 Nov 2013 14:07:44 -0500
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> I wonder if we should simply turn it on by default, and leave it
> >> there.  That way, any bugs that it exposes will be flushed out very
> >> quickly.  Stefan?
> >
> > Let's try it.
> 
> Sorry I'm late to the party, but this broke linum and nlinum for me,
> using the defaults.  Any time I insert a new line, the line numbers get
> totally messed up--most of them don't even display any more.  And then
> using motion commands sometimes results in really bizarre behavior
> (apparently only with linum/nlinum so far), such as when I do
> forwad-sexp and point goes to the end of some sexp other than the one at
> point.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 09 Nov 2013 03:37:49 +0100
> Cc: 15797@debbugs.gnu.org, kjambunathan@gmail.com
> 
> I see something probably related.  Sometimes, evaluating
> 
>   (line-number-at-pos)
> 
> at the end of my .emacs (10000 lines) needs over a second.  Doing
> 
>   (setq cache-long-scans nil)
> 
> immediately fixes this.  Dunno yet how to provoke the problem.

Please do try to find a recipe.  I just did that with xdisp.c (almost
30000 lines), and couldn't reproduce this.  line-number-at-pos is
instantaneous, whether I try it at the beginning, the end, or the
middle of the file.

bug#15797: 24.3.50; Info: Mention cache-long-scans

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen <michael_heerdegen@web.de>,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Fri, 08 Nov 2013 16:36:50 -0500
> 
> Try this with the attached file:
> 
> src/emacs -nw -Q foo.el
> 
> M-x linum-mode    (you may notice the problem already here)
> M-x goto-line 10
> C-o
> 
> The numbers in the margin get messed up.  I haven't figured out a
> consistent way to screw up the motion commands yet.


[-- Attachment #2: foo.el --]
[-- Type: application/emacs-lisp, Size: 1401 bytes --]

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen <michael_heerdegen@web.de>,  15797@debbugs.gnu.org,  kjambunathan@gmail.com
> Date: Fri, 08 Nov 2013 16:36:50 -0500
> 
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
> > I tried:
> >
> >   src/emacs -Q src/xdisp.c -l .../nlinum.el -f nlinum-mode
> >
> > and then moved about in the buffer, but I didn't notice any problem.
> 
> Try this with the attached file:
> 
> src/emacs -nw -Q foo.el
> 
> M-x linum-mode    (you may notice the problem already here)
> M-x goto-line 10
> C-o
> 
> The numbers in the margin get messed up.  I haven't figured out a
> consistent way to screw up the motion commands yet.


[-- Attachment #2: foo.el --]
[-- Type: application/emacs-lisp, Size: 1401 bytes --]

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Date: Fri, 08 Nov 2013 18:11:44 -0500
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, 15797@debbugs.gnu.org,
> 	kjambunathan@gmail.com
> 
> Nathan Trapuzzano <nbtrap@nbtrap.com> writes:
> 
> > I haven't figured out a consistent way to screw up the motion commands
> > yet.
> 
> Here we go.  Visit the same file and do:
> 
> M-x linum-mode
> C-o
> C-n
> C-e
> 
> Cursor moves almost to end of entire file.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap@nbtrap.com>
> Date: Fri, 08 Nov 2013 18:11:44 -0500
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, 15797@debbugs.gnu.org,
> 	kjambunathan@gmail.com
> 
> Nathan Trapuzzano <nbtrap@nbtrap.com> writes:
> 
> > I haven't figured out a consistent way to screw up the motion commands
> > yet.
> 
> Here we go.  Visit the same file and do:
> 
> M-x linum-mode
> C-o
> C-n
> C-e
> 
> Cursor moves almost to end of entire file.

Should be fixed in trunk revision 115050.

However, there are still problems which can be seen with this recipe:

 emacs -Q --eval "(setq-default cache-long-scans nil)"
 C-x C-f src/xdisp.c
 M-: (setq cache-long-scans t) RET
 M-x linum-mode RET
 M->

Many line numbers near the end of the buffer are not shown in the
margin.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> Date: Sat, 09 Nov 2013 13:18:29 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: michael_heerdegen@web.de, 15841@debbugs.gnu.org
> 
>  emacs -Q --eval "(setq-default cache-long-scans nil)"
>  C-x C-f src/xdisp.c
>  M-: (setq cache-long-scans t) RET
>  M-x linum-mode RET
>  M->
> 
> Many line numbers near the end of the buffer are not shown in the
> margin.

This seems to be related to font-lock: turning off
global-font-lock-mode makes the problems go away.  Manually invoking
font-lock-fontify-buffer before turning on linum-mode also eliminates
the problems, so it looks like JIT Font Lock is the prime suspect.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> Date: Sat, 09 Nov 2013 16:02:21 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: michael_heerdegen@web.de, 15841@debbugs.gnu.org
> 
> > Date: Sat, 09 Nov 2013 13:18:29 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: michael_heerdegen@web.de, 15841@debbugs.gnu.org
> > 
> >  emacs -Q --eval "(setq-default cache-long-scans nil)"
> >  C-x C-f src/xdisp.c
> >  M-: (setq cache-long-scans t) RET
> >  M-x linum-mode RET
> >  M->
> > 
> > Many line numbers near the end of the buffer are not shown in the
> > margin.
> 
> This seems to be related to font-lock: turning off
> global-font-lock-mode makes the problems go away.  Manually invoking
> font-lock-fontify-buffer before turning on linum-mode also eliminates
> the problems, so it looks like JIT Font Lock is the prime suspect.

The problem was that while the "dumb loop" in find_newline was running,
memory allocation in know_region_cache could have caused relocation of
buffer text, so C pointers needed to be adjusted, but weren't.

Fixed in trunk revision 115052.

bug#15841: Display bugs with cache-long-lines non-nil

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > I see something probably related.  Sometimes, evaluating
> > 
> >   (line-number-at-pos)
> > 
> > at the end of my .emacs (10000 lines) needs over a second.  Doing
> > 
> >   (setq cache-long-scans nil)
> > 
> > immediately fixes this.  Dunno yet how to provoke the problem.
>
> Please do try to find a recipe.  I just did that with xdisp.c (almost
> 30000 lines), and couldn't reproduce this.  line-number-at-pos is
> instantaneous, whether I try it at the beginning, the end, or the
> middle of the file.

Sorry, I was misguided.  What was slow wasn't `line-number-at-pos' but
redisplay (so only the result of evaluation showed up delayed).  This
long time for redisplay does only happen with some code I'm currently
developing, but also only when cache-long-scans is non-nil.  Anyway,
I'll tell you when I have a recipe.

Thanks,

Michael.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 15841@debbugs.gnu.org,  nbtrap@nbtrap.com
> Date: Sun, 10 Nov 2013 19:20:13 +0100
> 
> Sorry, I was misguided.  What was slow wasn't `line-number-at-pos' but
> redisplay (so only the result of evaluation showed up delayed).  This
> long time for redisplay does only happen with some code I'm currently
> developing, but also only when cache-long-scans is non-nil.

Thanks for clarifying.

> Anyway, I'll tell you when I have a recipe.

Please do, and thanks.

bug#15841: Display bugs with cache-long-lines non-nil

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > Anyway, I'll tell you when I have a recipe.

Some elaborations, feel free to ignore.

The culprit was my own code: it placed myriads of invisible overlays
with no properties into the buffer.  Under these extreme circumstances,
`line-number-at-pos' indeed gets extremely slow at the end of my 10000
lines buffer: one invocation needs over a second.  I saw that with elp
as well as with profiler.  Setting `cache-long-scans' to nil (or
removing the overlays) cures this.

Although this is a corner case, I wonder why overlays slow down
`line-number-at-pos' so much for `cache-long-scans' non-nil - is that
expected?  Or can the profiler times I saw span redisplay times?

Because, when I use this:

(defmacro my-measure-time (expr)
  "Eval EXPR, display how much time it took."
  (with-gensyms (time)
    `(let ((,time (current-time)))
       ,expr
       (message "%s secs"
          (float-time (time-subtract (current-time) ,time))))))

and evaluate (my-measure-time (line-number-at-pos)) manually with M-:
(in the same situation), it shows a very tiny value.  But I'm sure that
from code, `line-number-at-pos' really lasts over a second.  Strange, I
don't understand it.



Regards,

Michael.

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

This change broke dired-maybe-insert-subdir.  Perhaps this should be a
new bug number, but since part of the breakage is display-related, I
decided to report it here.  To reproduce:

0. emacs -Q (built from trunk bzr 115033 or later)
1. Open a directory in Dired, e.g. the Emacs source tree root.
2. Put point on an entry listing a directory, e.g. admin.
3. Type `i'.
=> Emacs hangs.

If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
nil)', then at step 3, `i' works as expected.

Here are more details of the breakage:

Typing `C-g' releases the hang and shows the admin subdirectory, but its
display is corrupted: (i) the "available" blocks information, which
should appear at the end of the "total" line, instead appears to the
right of the first entry `.' and is fontified with dired-directory face;
(ii) while all the entries of admin are listed below the `..' entry, the
entire line of each entry is fontified with dired-directory face, and
these lines cannot be accessed by vertical motion commands like `C-n' or
`C-p', though they can be with horizontal motion commands like `C-f' and
`C-p'.  In fact, all of these entries appear to Dired to be part of the
line of the `..' entry, and AFACT this is what causes the hang: Emacs
infloops in dired-move-to-filename because, when point appears to be at
eof, beginning-of-line moves it back to the start of the `..' entry.

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 15841@debbugs.gnu.org,  nbtrap@nbtrap.com
> Date: Mon, 11 Nov 2013 04:39:56 +0100
> 
> The culprit was my own code: it placed myriads of invisible overlays
> with no properties into the buffer.  Under these extreme circumstances,
> `line-number-at-pos' indeed gets extremely slow at the end of my 10000
> lines buffer: one invocation needs over a second.  I saw that with elp
> as well as with profiler.  Setting `cache-long-scans' to nil (or
> removing the overlays) cures this.

Can you show some simple enough code that puts so many overlays, and
has this effect?  It sounds like some optimization is in order.

> Although this is a corner case, I wonder why overlays slow down
> `line-number-at-pos' so much for `cache-long-scans' non-nil - is that
> expected?

I don't know.  Armed with a simple test case, I will look into this
and see what I find.

> Because, when I use this:
> 
> (defmacro my-measure-time (expr)
>   "Eval EXPR, display how much time it took."
>   (with-gensyms (time)
>     `(let ((,time (current-time)))
>        ,expr
>        (message "%s secs"
>           (float-time (time-subtract (current-time) ,time))))))
> 
> and evaluate (my-measure-time (line-number-at-pos)) manually with M-:
> (in the same situation), it shows a very tiny value.  But I'm sure that
> from code, `line-number-at-pos' really lasts over a second.  Strange, I
> don't understand it.

Redisplay is indeed a likely suspect, but still cache-long-scans
somehow comes into play.

Thanks.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: 15841@debbugs.gnu.org
> Date: Mon, 11 Nov 2013 15:12:15 +0100
> 
> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
> new bug number, but since part of the breakage is display-related, I
> decided to report it here.  To reproduce:
> 
> 0. emacs -Q (built from trunk bzr 115033 or later)
> 1. Open a directory in Dired, e.g. the Emacs source tree root.
> 2. Put point on an entry listing a directory, e.g. admin.
> 3. Type `i'.
> => Emacs hangs.

Actually, if you rebuild with assertions (--enable-checking), Emacs
hits an assertion violation before it starts inflooping, and aborts.

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

On Mon, 11 Nov 2013 22:13:36 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: 15841@debbugs.gnu.org
>> Date: Mon, 11 Nov 2013 15:12:15 +0100
>> 
>> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
>> new bug number, but since part of the breakage is display-related, I
>> decided to report it here.  To reproduce:
>> 
>> 0. emacs -Q (built from trunk bzr 115033 or later)
>> 1. Open a directory in Dired, e.g. the Emacs source tree root.
>> 2. Put point on an entry listing a directory, e.g. admin.
>> 3. Type `i'.
>> => Emacs hangs.
>
> Actually, if you rebuild with assertions (--enable-checking), Emacs
> hits an assertion violation before it starts inflooping, and aborts.

As a matter of fact, I've also gotten an abort a couple of times while
trying to track down this bug, though I did not build with
--enable-checking; unfortunately, I don't remember exactly what I what
doing, nor was I running under gdb, so no backtrace.  I also have
another observation, which seems to be reproducible, though I haven't
tested it extensively: if I type `i' on any directory entry in Dired,
which makes Emacs hang, then type `C-g' to unhang Emacs, then type `i'
on a *higher* directory entry in the same Dired buffer, whose name
begins with `.' (e.g. `.bzr'), then `i' works as expected.  This appears
to hold for all higher dotted (and only dotted) directory entries; but
if I type `i' on a *lower* dotted directory entry, then Emacs again
hangs.

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

On Mon, 11 Nov 2013 15:12:15 +0100 Stephen Berman <stephen.berman@gmx.net> wrote:

> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
> new bug number, but since part of the breakage is display-related, I
> decided to report it here.  To reproduce:
>
> 0. emacs -Q (built from trunk bzr 115033 or later)
> 1. Open a directory in Dired, e.g. the Emacs source tree root.
> 2. Put point on an entry listing a directory, e.g. admin.
> 3. Type `i'.
> => Emacs hangs.
>
> If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
> nil)', then at step 3, `i' works as expected.
>
> Here are more details of the breakage:
>
> Typing `C-g' releases the hang and shows the admin subdirectory, but its
> display is corrupted: (i) the "available" blocks information, which
> should appear at the end of the "total" line, instead appears to the
> right of the first entry `.' and is fontified with dired-directory face;
> (ii) while all the entries of admin are listed below the `..' entry, the
> entire line of each entry is fontified with dired-directory face, and
> these lines cannot be accessed by vertical motion commands like `C-n' or
> `C-p', though they can be with horizontal motion commands like `C-f' and
> `C-p'.  In fact, all of these entries appear to Dired to be part of the
> line of the `..' entry, and AFACT this is what causes the hang: Emacs
> infloops in dired-move-to-filename because, when point appears to be at
> eof, beginning-of-line moves it back to the start of the `..' entry.

I tracked the problematic fontification and motion behavior to
insert-directory in files.el: it happens during the loop when
decode-coding-region is called on the file names of the subdirectory
entries.  I stepped through the code with Edebug but could not tell why
it goes wrong here, and I'm too tired to pursue it further now.  Also,
when I step through this code, the "available" information is added at
the end of the entire subdirectory listing, unlike what I observed above
when just invoking `i' and then `C-g'.  I guess this is due to the
interaction of redisplay with stepping through the code; it's still
clear that the subdirectory listing is being treated as part of the line
containing the `..' entry.

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

On Tue, 12 Nov 2013 01:38:34 +0100 Stephen Berman <stephen.berman@gmx.net> wrote:

> On Mon, 11 Nov 2013 15:12:15 +0100 Stephen Berman <stephen.berman@gmx.net> wrote:
>
>> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
>> new bug number, but since part of the breakage is display-related, I
>> decided to report it here.  To reproduce:
>>
>> 0. emacs -Q (built from trunk bzr 115033 or later)
>> 1. Open a directory in Dired, e.g. the Emacs source tree root.
>> 2. Put point on an entry listing a directory, e.g. admin.
>> 3. Type `i'.
>> => Emacs hangs.
>>
>> If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
>> nil)', then at step 3, `i' works as expected.
>>
>> Here are more details of the breakage:
>>
>> Typing `C-g' releases the hang and shows the admin subdirectory, but its
>> display is corrupted: (i) the "available" blocks information, which
>> should appear at the end of the "total" line, instead appears to the
>> right of the first entry `.' and is fontified with dired-directory face;
>> (ii) while all the entries of admin are listed below the `..' entry, the
>> entire line of each entry is fontified with dired-directory face, and
>> these lines cannot be accessed by vertical motion commands like `C-n' or
>> `C-p', though they can be with horizontal motion commands like `C-f' and
>> `C-p'.  In fact, all of these entries appear to Dired to be part of the
>> line of the `..' entry, and AFACT this is what causes the hang: Emacs
>> infloops in dired-move-to-filename because, when point appears to be at
>> eof, beginning-of-line moves it back to the start of the `..' entry.
>
> I tracked the problematic fontification and motion behavior to
> insert-directory in files.el: it happens during the loop when
> decode-coding-region is called on the file names of the subdirectory
> entries.  I stepped through the code with Edebug but could not tell why
> it goes wrong here, and I'm too tired to pursue it further now.  Also,
> when I step through this code, the "available" information is added at
> the end of the entire subdirectory listing, unlike what I observed above
> when just invoking `i' and then `C-g'.  I guess this is due to the
> interaction of redisplay with stepping through the code; it's still
> clear that the subdirectory listing is being treated as part of the line
> containing the `..' entry.

I set a breakpoint on decode_coding_object and stepped through it after
invoking `i', but the Dired display changed to show the problematic
fontification only upon exiting decode_coding_object.  I don't know how
to find out when cache_long_scans is checked other than manually tracing
the call chains of each subroutine of decode_coding_object, which is too
laborious.  Is it possible to have execution halt when cache_long_scans
is checked, and if so, how?  Or is there a better way to try to track
down this bug?

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: 15841@debbugs.gnu.org
> Date: Tue, 12 Nov 2013 11:03:47 +0100
> 
> > I tracked the problematic fontification and motion behavior to
> > insert-directory in files.el: it happens during the loop when
> > decode-coding-region is called on the file names of the subdirectory
> > entries.  I stepped through the code with Edebug but could not tell why
> > it goes wrong here, and I'm too tired to pursue it further now.  Also,
> > when I step through this code, the "available" information is added at
> > the end of the entire subdirectory listing, unlike what I observed above
> > when just invoking `i' and then `C-g'.  I guess this is due to the
> > interaction of redisplay with stepping through the code; it's still
> > clear that the subdirectory listing is being treated as part of the line
> > containing the `..' entry.
> 
> I set a breakpoint on decode_coding_object and stepped through it after
> invoking `i', but the Dired display changed to show the problematic
> fontification only upon exiting decode_coding_object.  I don't know how
> to find out when cache_long_scans is checked other than manually tracing
> the call chains of each subroutine of decode_coding_object, which is too
> laborious.

There's no need: I already established that, as you point out, the
changes to the buffer that cause the problem are indeed made by
decode-coding-region.  The problem becomes visible when redisplay,
entered after decode-coding-region finishes its job, re-fontifies
portions of the buffer that were affected by the changes.  The way
this affects redisplay is through forward-line and
line-beginning-position, of which JIT Font Lock is a heavy user.
That's why you only see the effect after decode-coding-region returns.

> Is it possible to have execution halt when cache_long_scans is
> checked, and if so, how?

Watchpoints are the answer.  But in this case, there's only one place
in the whole Emacs where this variable is consulted: in search.c,
around line 610, so you could just put a breakpoint there.

In any case, I already traced through the code that is involved, and
the immediate reason for the assertion violation is that the cache
isn't being updated wrt changes in buffer size (which are caused by
decoding the stuff brought in by 'ls').  However, a naive attempt to
force such updates didn't solve the whole problem: the aborts are
gone, but the infloop is still there, and also other minor display
issues.  So I guess there's another factor at work there...

I also need to figure out how to keep the cache up to date without
penalizing performance, which would render the cache worthless.

> Or is there a better way to try to track down this bug?

The cache has only 3 public interfaces (see region-cache.c), so it is
easy to put breakpoints in all of them and see what happens.  That's
what I did.

Thanks.

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

On Tue, 12 Nov 2013 18:31:32 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

> I already established that, as you point out, the
> changes to the buffer that cause the problem are indeed made by
> decode-coding-region.  The problem becomes visible when redisplay,
> entered after decode-coding-region finishes its job, re-fontifies
> portions of the buffer that were affected by the changes.  The way
> this affects redisplay is through forward-line and
> line-beginning-position, of which JIT Font Lock is a heavy user.
> That's why you only see the effect after decode-coding-region returns.

Thanks for the explanation.

>> Is it possible to have execution halt when cache_long_scans is
>> checked, and if so, how?
>
> Watchpoints are the answer.  But in this case, there's only one place
> in the whole Emacs where this variable is consulted: in search.c,
> around line 610, so you could just put a breakpoint there.
>
> In any case, I already traced through the code that is involved, and
> the immediate reason for the assertion violation is that the cache
> isn't being updated wrt changes in buffer size (which are caused by
> decoding the stuff brought in by 'ls').  However, a naive attempt to
> force such updates didn't solve the whole problem: the aborts are
> gone, but the infloop is still there, and also other minor display
> issues.  So I guess there's another factor at work there...

Fascinating, what unsuspected and apparently unrelated effects can be
brought to the surface by toggling the value of a variable!

> I also need to figure out how to keep the cache up to date without
> penalizing performance, which would render the cache worthless.
>
>> Or is there a better way to try to track down this bug?
>
> The cache has only 3 public interfaces (see region-cache.c), so it is
> easy to put breakpoints in all of them and see what happens.  That's
> what I did.

Thanks for the advice and for working on this bug.

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Michael Heerdegen <michael_heerdegen@web.de>
> > Cc: 15841@debbugs.gnu.org,  nbtrap@nbtrap.com
> > Date: Mon, 11 Nov 2013 04:39:56 +0100
> > 
> > The culprit was my own code: it placed myriads of invisible overlays
> > with no properties into the buffer.  Under these extreme circumstances,
> > `line-number-at-pos' indeed gets extremely slow at the end of my 10000
> > lines buffer: one invocation needs over a second.  I saw that with elp
> > as well as with profiler.  Setting `cache-long-scans' to nil (or
> > removing the overlays) cures this.
>
> Can you show some simple enough code that puts so many overlays, and
> has this effect?  It sounds like some optimization is in order.

Sorry, I did not find an easy test case.  I tried to simplify the
essence of what my code does, but the effect didn't occur.  Seems that
lots of different things must come together to provoke this symptom.
Given the fact that my code was really broken, I don't think it's worth
the time to follow this up.  It would cost me many hours to compile some
test code for emacs -Q.  Or do you think it would be worth it?  If you
think it could be very important, I would do it, but it would be very
time intensive.  My code works well now with `cache-long-scans' t after
the right fixes, btw.


Regards,

Michael.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 15841@debbugs.gnu.org,  nbtrap@nbtrap.com
> Date: Thu, 14 Nov 2013 00:45:17 +0100
> 
> > Can you show some simple enough code that puts so many overlays, and
> > has this effect?  It sounds like some optimization is in order.
> 
> Sorry, I did not find an easy test case.  I tried to simplify the
> essence of what my code does, but the effect didn't occur.  Seems that
> lots of different things must come together to provoke this symptom.
> Given the fact that my code was really broken, I don't think it's worth
> the time to follow this up.  It would cost me many hours to compile some
> test code for emacs -Q.  Or do you think it would be worth it?  If you
> think it could be very important, I would do it, but it would be very
> time intensive.  My code works well now with `cache-long-scans' t after
> the right fixes, btw.

If the bug isn't reproducible, it isn't worth working on.

Thanks.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> Date: Tue, 12 Nov 2013 18:31:32 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 15841@debbugs.gnu.org
> 
> In any case, I already traced through the code that is involved, and
> the immediate reason for the assertion violation is that the cache
> isn't being updated wrt changes in buffer size (which are caused by
> decoding the stuff brought in by 'ls').  However, a naive attempt to
> force such updates didn't solve the whole problem: the aborts are
> gone, but the infloop is still there, and also other minor display
> issues.  So I guess there's another factor at work there...

I think I might have found a solution for this.  Could you please run
with the patch below for a while, and see if it gives good results?

=== modified file 'src/coding.c'
--- src/coding.c	2013-10-08 06:40:09 +0000
+++ src/coding.c	2013-11-15 16:27:56 +0000
@@ -9358,6 +9358,14 @@ code_convert_region (Lisp_Object start, 
   setup_coding_system (coding_system, &coding);
   coding.mode |= CODING_MODE_LAST_BLOCK;
 
+  if (BUFFERP (dst_object) && !EQ (dst_object, src_object))
+    {
+      struct buffer *buf = XBUFFER (dst_object);
+      ptrdiff_t buf_pt = BUF_PT (buf);
+
+      invalidate_buffer_caches (buf, buf_pt, buf_pt);
+    }
+
   if (encodep)
     encode_coding_object (&coding, src_object, from, from_byte, to, to_byte,
 			  dst_object);
@@ -9447,6 +9455,15 @@ code_convert_string (Lisp_Object string,
   coding.mode |= CODING_MODE_LAST_BLOCK;
   chars = SCHARS (string);
   bytes = SBYTES (string);
+
+  if (BUFFERP (dst_object))
+    {
+      struct buffer *buf = XBUFFER (dst_object);
+      ptrdiff_t buf_pt = BUF_PT (buf);
+
+      invalidate_buffer_caches (buf, buf_pt, buf_pt);
+    }
+
   if (encodep)
     encode_coding_object (&coding, string, 0, 0, chars, bytes, dst_object);
   else

=== modified file 'src/insdel.c'
--- src/insdel.c	2013-11-11 05:18:53 +0000
+++ src/insdel.c	2013-11-15 16:27:56 +0000
@@ -993,6 +993,11 @@ insert_from_gap (ptrdiff_t nchars, ptrdi
   if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
     nchars = nbytes;
 
+  /* No need to call prepare_to_modify_buffer, since this is called
+     from places that replace some region with a different text, so
+     prepare_to_modify_buffer was already called by the deletion part
+     of this dance.  */
+  invalidate_buffer_caches (current_buffer, GPT, GPT);
   record_insert (GPT, nchars);
   MODIFF++;
 
@@ -1869,19 +1874,26 @@ prepare_to_modify_buffer (ptrdiff_t star
 			  ptrdiff_t *preserve_ptr)
 {
   prepare_to_modify_buffer_1 (start, end, preserve_ptr);
+  invalidate_buffer_caches (current_buffer, start, end);
+}
 
-  if (current_buffer->newline_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->newline_cache,
-                             start - BEG, Z - end);
-  if (current_buffer->width_run_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->width_run_cache,
-                             start - BEG, Z - end);
-  if (current_buffer->bidi_paragraph_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->bidi_paragraph_cache,
-                             start - BEG, Z - end);
+/* Invalidate the caches maintained by the buffer BUF, if any, for the
+   region between buffer positions START and END.  */
+void
+invalidate_buffer_caches (struct buffer *buf, ptrdiff_t start, ptrdiff_t end)
+{
+  if (buf->newline_cache)
+    invalidate_region_cache (buf,
+                             buf->newline_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
+  if (buf->width_run_cache)
+    invalidate_region_cache (buf,
+                             buf->width_run_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
+  if (buf->bidi_paragraph_cache)
+    invalidate_region_cache (buf,
+                             buf->bidi_paragraph_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
 }
 
 /* These macros work with an argument named `preserve_ptr'

=== modified file 'src/lisp.h'
--- src/lisp.h	2013-11-15 08:18:37 +0000
+++ src/lisp.h	2013-11-15 16:27:56 +0000
@@ -3486,6 +3486,7 @@ extern Lisp_Object del_range_2 (ptrdiff_
 extern void modify_text (ptrdiff_t, ptrdiff_t);
 extern void prepare_to_modify_buffer (ptrdiff_t, ptrdiff_t, ptrdiff_t *);
 extern void prepare_to_modify_buffer_1 (ptrdiff_t, ptrdiff_t, ptrdiff_t *);
+extern void invalidate_buffer_caches (struct buffer *, ptrdiff_t, ptrdiff_t);
 extern void signal_after_change (ptrdiff_t, ptrdiff_t, ptrdiff_t);
 extern void adjust_after_insert (ptrdiff_t, ptrdiff_t, ptrdiff_t,
 				 ptrdiff_t, ptrdiff_t);

bug#15841: Display bugs with cache-long-lines non-nil

[Stephen Berman]

On Fri, 15 Nov 2013 18:34:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Tue, 12 Nov 2013 18:31:32 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 15841@debbugs.gnu.org
>> 
>> In any case, I already traced through the code that is involved, and
>> the immediate reason for the assertion violation is that the cache
>> isn't being updated wrt changes in buffer size (which are caused by
>> decoding the stuff brought in by 'ls').  However, a naive attempt to
>> force such updates didn't solve the whole problem: the aborts are
>> gone, but the infloop is still there, and also other minor display
>> issues.  So I guess there's another factor at work there...
>
> I think I might have found a solution for this.  Could you please run
> with the patch below for a while, and see if it gives good results?

Initial tests succeeded: `i' in Dired works as expected and there is no
infloop or display oddities.  I'll report back if any problems crop up
on further use.  Otherwise, thanks for fixing this!

Steve Berman

bug#15841: Display bugs with cache-long-lines non-nil

[Andy Moreton]

On Fri 15 Nov 2013, Eli Zaretskii wrote:

>> Date: Tue, 12 Nov 2013 18:31:32 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 15841@debbugs.gnu.org
>> 
>> In any case, I already traced through the code that is involved, and
>> the immediate reason for the assertion violation is that the cache
>> isn't being updated wrt changes in buffer size (which are caused by
>> decoding the stuff brought in by 'ls').  However, a naive attempt to
>> force such updates didn't solve the whole problem: the aborts are
>> gone, but the infloop is still there, and also other minor display
>> issues.  So I guess there's another factor at work there...
>
> I think I might have found a solution for this.  Could you please run
> with the patch below for a while, and see if it gives good results?

Thanks Eli - I've seen the same problem with inserting a subdirectory in
a dired buffer. Applying your patch to r115122 fixed dired for me.

    AndyM

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 16 Nov 2013 18:53:31 +0000
> 
> Thanks Eli - I've seen the same problem with inserting a subdirectory in
> a dired buffer. Applying your patch to r115122 fixed dired for me.

Thanks for trying.  I will wait till Monday, to see if no adverse
effects pop up, then commit those changes.

bug#15841: Display bugs with cache-long-lines non-nil

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: 15841@debbugs.gnu.org
> Date: Fri, 15 Nov 2013 19:05:42 +0100
> 
> Initial tests succeeded: `i' in Dired works as expected and there is no
> infloop or display oddities.  I'll report back if any problems crop up
> on further use.  Otherwise, thanks for fixing this!

No further problems, so I committed the changes as trunk revision
115138, and I'm closing this bug.