unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* How the long-lines "optimisation" breaks font locking.
@ 2022-08-03 18:39 Alan Mackenzie
  2022-08-03 19:03 ` Lars Ingebrigtsen
  2022-08-03 19:15 ` Eli Zaretskii
  0 siblings, 2 replies; 32+ messages in thread
From: Alan Mackenzie @ 2022-08-03 18:39 UTC (permalink / raw)
  To: emacs-devel, Gregory Heytings

Hello Gregory and Emacs.

In the current master,
(i) start emacs -Q
(ii) C-x C-f ~/long-line.cc
(iii) Enter the following into the buffer (3 lines):

char long_line[] = R"foo(

)foo"

(iv) Create a long line (> 10,000 characters) somehow and get it into
  the kill ring.
(v) On the blank line 2 in long-line.cc, C-y inserting the long line.

(vi) Note that there is now a long valid raw string in the buffer,
  correctly fontified.
(vii) Put point on a "o" in the foo on L3.  Delete the o
(viii) Note that the opening delimiter of the raw string on L1 now has
  warning-face.  This is correct.
(ix) Undo this last change with C-_.
(x) Note that the opening delimiter falsely retains the warning-face,
  although it is now the opener of a validly terminated raw string.
  This is a bug.

(xi) Insert a LF (or several, if needed) into the long line, separating
  it into two or more lines, none of which is "long".
(xii) Note that the raw string opener still has warning-face.  A bug.
(xiii) Do M-x c++-mode.  This fails to initialise the fontification
  correctly, leaving the warning-face where it no longer belongs.  This
  is a bug.
(xiv) Note that in the foregoing operations, sometimes an error message
  appears in the echo area about the lack of an expected syntax-table
  text property.  This is a bug.

I don't think it is improbable that a C++ hacker will create a long line
in a raw string.  That this can cause permanent damage to the buffer's
fontification doesn't seem a good thing.  That it is the recent
"optimisation" for long lines which causes it can be seen by repeating
the above steps, but with a line less than 10,000 characters long.  The
fontification then behaves correctly.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-03 18:39 How the long-lines "optimisation" breaks font locking Alan Mackenzie
@ 2022-08-03 19:03 ` Lars Ingebrigtsen
  2022-08-04 10:44   ` Alan Mackenzie
  2022-08-03 19:15 ` Eli Zaretskii
  1 sibling, 1 reply; 32+ messages in thread
From: Lars Ingebrigtsen @ 2022-08-03 19:03 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: emacs-devel, Gregory Heytings

Alan Mackenzie <acm@muc.de> writes:

> I don't think it is improbable that a C++ hacker will create a long line
> in a raw string.

If you don't think that's improbable, then you should like the new
optimisations a lot -- because if you, instead of a 10K line in that C++
file, inserted a 1M line, then Emacs would previously hang indefinitely,
but with the optimisation, it doesn't.

Which is the point here.




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-03 18:39 How the long-lines "optimisation" breaks font locking Alan Mackenzie
  2022-08-03 19:03 ` Lars Ingebrigtsen
@ 2022-08-03 19:15 ` Eli Zaretskii
  1 sibling, 0 replies; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-03 19:15 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: emacs-devel, gregory

> Date: Wed, 3 Aug 2022 18:39:16 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello Gregory and Emacs.
> 
> In the current master,
> (i) start emacs -Q
> (ii) C-x C-f ~/long-line.cc
> (iii) Enter the following into the buffer (3 lines):
> 
> char long_line[] = R"foo(
> 
> )foo"
> 
> (iv) Create a long line (> 10,000 characters) somehow and get it into
>   the kill ring.
> (v) On the blank line 2 in long-line.cc, C-y inserting the long line.
> 
> (vi) Note that there is now a long valid raw string in the buffer,
>   correctly fontified.
> (vii) Put point on a "o" in the foo on L3.  Delete the o
> (viii) Note that the opening delimiter of the raw string on L1 now has
>   warning-face.  This is correct.
> (ix) Undo this last change with C-_.
> (x) Note that the opening delimiter falsely retains the warning-face,
>   although it is now the opener of a validly terminated raw string.
>   This is a bug.
> 
> (xi) Insert a LF (or several, if needed) into the long line, separating
>   it into two or more lines, none of which is "long".
> (xii) Note that the raw string opener still has warning-face.  A bug.
> (xiii) Do M-x c++-mode.  This fails to initialise the fontification
>   correctly, leaving the warning-face where it no longer belongs.  This
>   is a bug.
> (xiv) Note that in the foregoing operations, sometimes an error message
>   appears in the echo area about the lack of an expected syntax-table
>   text property.  This is a bug.

I cannot reproduce this.  I don't see any warning-face in step viii,
and no problem in step x.

> I don't think it is improbable that a C++ hacker will create a long line
> in a raw string.  That this can cause permanent damage to the buffer's
> fontification doesn't seem a good thing.  That it is the recent
> "optimisation" for long lines which causes it can be seen by repeating
> the above steps, but with a line less than 10,000 characters long.  The
> fontification then behaves correctly.

Alan, please stop fighting this with impractical specially-concocted
scenarios.  No one in their right minds will need such long strings,
and if they do, and don't like whatever problems that causes (which I
cannot reproduce in the first place), they can enlarge the value of
long-line-threshold, or even make it nil.  Puff! problem solved.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-03 19:03 ` Lars Ingebrigtsen
@ 2022-08-04 10:44   ` Alan Mackenzie
  2022-08-04 12:54     ` Eli Zaretskii
  2022-08-04 13:11     ` Eric S Fraga
  0 siblings, 2 replies; 32+ messages in thread
From: Alan Mackenzie @ 2022-08-04 10:44 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel, Gregory Heytings

Hello, Lars.

On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I don't think it is improbable that a C++ hacker will create a long line
> > in a raw string.

> If you don't think that's improbable, then you should like the new
> optimisations a lot -- ....

These "optimisations" break CC Mode.

> .... because if you, instead of a 10K line in that C++ file, inserted
> a 1M line, then Emacs would previously hang indefinitely, but with the
> optimisation, it doesn't.

Well I tried CC Mode with a 1,000,000 character raw string.  It was
indeed a bit sluggish but "hang indefinitely" is an exaggeration.

Having loaded the file in C++ Mode (without the spiking of
narrow-to-region and widen), it took 90 seconds for M-> (first time).
Moving point was sluggish, taking perhaps 1 to 3 seconds, as did
scrolling by a screenful.  Inserting text in the middle of the line
caused an overflow in the regexp engine stack.  Interfering with the raw
string delimiters (specifically, changing the identifier in the
delimiter at either end of the string) again took 90 seconds.

Doing these things in the current master branch indeed appeared to be
fast, but at one point an error in an after-change-function caused
after-change-functions to get set to nil, crashing CC Mode.

> Which is the point here.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 10:44   ` Alan Mackenzie
@ 2022-08-04 12:54     ` Eli Zaretskii
  2022-08-04 14:44       ` Alan Mackenzie
  2022-08-05 10:56       ` Alan Mackenzie
  2022-08-04 13:11     ` Eric S Fraga
  1 sibling, 2 replies; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-04 12:54 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: larsi, emacs-devel, gregory

> Date: Thu, 4 Aug 2022 10:44:05 +0000
> Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Lars.
> 
> On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> > Alan Mackenzie <acm@muc.de> writes:
> 
> > > I don't think it is improbable that a C++ hacker will create a long line
> > > in a raw string.
> 
> > If you don't think that's improbable, then you should like the new
> > optimisations a lot -- ....
> 
> These "optimisations" break CC Mode.

They don't "break" it, they cause it occasionally produce inaccurate
fontifications.  And only when otherwise Emacs becomes unusable.

> > .... because if you, instead of a 10K line in that C++ file, inserted
> > a 1M line, then Emacs would previously hang indefinitely, but with the
> > optimisation, it doesn't.
> 
> Well I tried CC Mode with a 1,000,000 character raw string.  It was
> indeed a bit sluggish but "hang indefinitely" is an exaggeration.

Try it with a 20MB raw string, then.  And, for good measure, in an
unoptimized build.  These are the cases we are trying to make
workable.

If all you are saying is that the default value of long-line-threshold
is too low, we can definitely discuss a better value.

> Having loaded the file in C++ Mode (without the spiking of
> narrow-to-region and widen), it took 90 seconds for M-> (first time).

And you consider that reasonable?

> Moving point was sluggish, taking perhaps 1 to 3 seconds, as did
> scrolling by a screenful.

Did you try moving and the beginning of the buffer or at its end?  The
speed is very different.

> Inserting text in the middle of the line caused an overflow in the
> regexp engine stack.  Interfering with the raw string delimiters
> (specifically, changing the identifier in the delimiter at either
> end of the string) again took 90 seconds.

And this is in your opinion reasonable performance?

> Doing these things in the current master branch indeed appeared to be
> fast, but at one point an error in an after-change-function caused
> after-change-functions to get set to nil, crashing CC Mode.

If you can show a recipe for this problem, we will fix it.  This code
is WIP, so some problems definitely remain and should be reported and
fixed.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 10:44   ` Alan Mackenzie
  2022-08-04 12:54     ` Eli Zaretskii
@ 2022-08-04 13:11     ` Eric S Fraga
  2022-08-04 13:34       ` Po Lu
  1 sibling, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-04 13:11 UTC (permalink / raw)
  To: emacs-devel

I don't know if this is related or not but I have just built emacs from
current git and gnus buffers have lost all fontification.  All other
modes, including message-mode, seem to be just fine.

I've not investigated beyond this but thought I'd highlight the issue.
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 13:11     ` Eric S Fraga
@ 2022-08-04 13:34       ` Po Lu
  2022-08-04 14:38         ` Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Po Lu @ 2022-08-04 13:34 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: emacs-devel

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I don't know if this is related or not but I have just built emacs from
> current git and gnus buffers have lost all fontification.  All other
> modes, including message-mode, seem to be just fine.

Did you try "make bootstrap"?  It was enough to fix that problem when I
experienced it several weeks ago (around one of the loaddefs-related
changes).



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 13:34       ` Po Lu
@ 2022-08-04 14:38         ` Eric S Fraga
  2022-08-04 16:16           ` SOLVED: " Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-04 14:38 UTC (permalink / raw)
  To: emacs-devel

On Thursday,  4 Aug 2022 at 21:34, Po Lu wrote:
> Did you try "make bootstrap"?  

Yes, I always build using bootstrap.

Interestingly, in one post I tried to read (but not in any others), I
did get an error message:

gnus-summary-highlight-line: Wrong type argument: listp, font-lock-string-face

but it doesn't help me figure anything out.  The face itself is fine as
is being used as it should in other buffers/modes.

I need to check if anything has changed in gnus recently.

thank you,
eric

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 12:54     ` Eli Zaretskii
@ 2022-08-04 14:44       ` Alan Mackenzie
  2022-08-04 14:52         ` Lars Ingebrigtsen
                           ` (2 more replies)
  2022-08-05 10:56       ` Alan Mackenzie
  1 sibling, 3 replies; 32+ messages in thread
From: Alan Mackenzie @ 2022-08-04 14:44 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: larsi, emacs-devel, gregory

Hello, Eli.

On Thu, Aug 04, 2022 at 15:54:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Aug 2022 10:44:05 +0000
> > Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> > From: Alan Mackenzie <acm@muc.de>

> > Hello, Lars.

> > On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> > > Alan Mackenzie <acm@muc.de> writes:

> > > > I don't think it is improbable that a C++ hacker will create a long line
> > > > in a raw string.

> > > If you don't think that's improbable, then you should like the new
> > > optimisations a lot -- ....

> > These "optimisations" break CC Mode.

> They don't "break" it, they cause it occasionally produce inaccurate
> fontifications.  And only when otherwise Emacs becomes unusable.

> > > .... because if you, instead of a 10K line in that C++ file, inserted
> > > a 1M line, then Emacs would previously hang indefinitely, but with the
> > > optimisation, it doesn't.

> > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > indeed a bit sluggish but "hang indefinitely" is an exaggeration.

> Try it with a 20MB raw string, then.  And, for good measure, in an
> unoptimized build.  These are the cases we are trying to make
> workable.

No matter what criterion one deems "workable", there will be some size of
buffer, or line, which renders Emacs "unworkable".

> If all you are saying is that the default value of long-line-threshold
> is too low, we can definitely discuss a better value.

> > Having loaded the file in C++ Mode (without the spiking of
> > narrow-to-region and widen), it took 90 seconds for M-> (first time).

> And you consider that reasonable?

No, I don't.  I was just correcting Lars's mistaken assertion that such a
line would make Emacs "hang indefinitely".

[ .... ]

> > Doing these things in the current master branch indeed appeared to be
> > fast, but at one point an error in an after-change-function caused
> > after-change-functions to get set to nil, crashing CC Mode.

> If you can show a recipe for this problem, we will fix it.  This code
> is WIP, so some problems definitely remain and should be reported and
> fixed.

I'll see if I can recreate it.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 14:44       ` Alan Mackenzie
@ 2022-08-04 14:52         ` Lars Ingebrigtsen
  2022-08-04 15:22         ` Gregory Heytings
  2022-08-04 16:09         ` Eli Zaretskii
  2 siblings, 0 replies; 32+ messages in thread
From: Lars Ingebrigtsen @ 2022-08-04 14:52 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Eli Zaretskii, emacs-devel, gregory

Alan Mackenzie <acm@muc.de> writes:

> No, I don't.  I was just correcting Lars's mistaken assertion that such a
> line would make Emacs "hang indefinitely".

90s for a point movement command is "indefinitely" in my book.




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 14:44       ` Alan Mackenzie
  2022-08-04 14:52         ` Lars Ingebrigtsen
@ 2022-08-04 15:22         ` Gregory Heytings
  2022-08-05  1:23           ` Ihor Radchenko
  2022-08-04 16:09         ` Eli Zaretskii
  2 siblings, 1 reply; 32+ messages in thread
From: Gregory Heytings @ 2022-08-04 15:22 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Eli Zaretskii, larsi, emacs-devel


>
> No matter what criterion one deems "workable", there will be some size 
> of buffer, or line, which renders Emacs "unworkable".
>

Wrong.  You can now edit files with 1 GB or 5 GB on a single line almost 
fine.  The limit is now the amount of memory on your computer.  I keep 
investigating to see if it's possible to overcome that limit.  Stay tuned!



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 14:44       ` Alan Mackenzie
  2022-08-04 14:52         ` Lars Ingebrigtsen
  2022-08-04 15:22         ` Gregory Heytings
@ 2022-08-04 16:09         ` Eli Zaretskii
  2 siblings, 0 replies; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-04 16:09 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: larsi, emacs-devel, gregory

> Date: Thu, 4 Aug 2022 14:44:05 +0000
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > > indeed a bit sluggish but "hang indefinitely" is an exaggeration.
> 
> > Try it with a 20MB raw string, then.  And, for good measure, in an
> > unoptimized build.  These are the cases we are trying to make
> > workable.
> 
> No matter what criterion one deems "workable", there will be some size of
> buffer, or line, which renders Emacs "unworkable".

Not with narrowing, no.  Because the size of the narrowed region is
independent of the buffer size.  That's the main idea of these
changes, and that's why the idea is IMO a good one.

> > > Doing these things in the current master branch indeed appeared to be
> > > fast, but at one point an error in an after-change-function caused
> > > after-change-functions to get set to nil, crashing CC Mode.
> 
> > If you can show a recipe for this problem, we will fix it.  This code
> > is WIP, so some problems definitely remain and should be reported and
> > fixed.
> 
> I'll see if I can recreate it.

TIA.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 14:38         ` Eric S Fraga
@ 2022-08-04 16:16           ` Eric S Fraga
  2022-08-04 16:21             ` Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-04 16:16 UTC (permalink / raw)
  To: emacs-devel

Hello all again,

having done some investigating, I found that an old setting in my
configuration caused all kinds of problems.  It was using
quail-define-package to define emojis and some special accented
characters for email, a configuration which I have not needed for quite
some time.

Thank you all,
eric

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 16:16           ` SOLVED: " Eric S Fraga
@ 2022-08-04 16:21             ` Eric S Fraga
  2022-08-04 17:36               ` Stefan Kangas
  0 siblings, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-04 16:21 UTC (permalink / raw)
  To: emacs-devel

... but I have no input methods at all.  If I M-x list-input-methods,
Emacs tells me:

,----
| No input method is available, perhaps because you have not
| installed LEIM (Libraries of Emacs Input Methods).
`----

One step forward, two steps back...  Hey hum.

I'm off to have a beer.  Long day.  ;-)

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 16:21             ` Eric S Fraga
@ 2022-08-04 17:36               ` Stefan Kangas
  2022-08-05  9:36                 ` Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Stefan Kangas @ 2022-08-04 17:36 UTC (permalink / raw)
  To: Eric S Fraga, emacs-devel

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> ... but I have no input methods at all.  If I M-x list-input-methods,
> Emacs tells me:
>
> ,----
> | No input method is available, perhaps because you have not
> | installed LEIM (Libraries of Emacs Input Methods).
> `----

That's an odd message.  I don't think any user should ever need to
install LEIM these days?



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 15:22         ` Gregory Heytings
@ 2022-08-05  1:23           ` Ihor Radchenko
  0 siblings, 0 replies; 32+ messages in thread
From: Ihor Radchenko @ 2022-08-05  1:23 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: Alan Mackenzie, Eli Zaretskii, larsi, emacs-devel

Gregory Heytings <gregory@heytings.org> writes:

>> No matter what criterion one deems "workable", there will be some size 
>> of buffer, or line, which renders Emacs "unworkable".
>>
>
> Wrong.  You can now edit files with 1 GB or 5 GB on a single line almost 
> fine.  The limit is now the amount of memory on your computer.  I keep 
> investigating to see if it's possible to overcome that limit.  Stay tuned!

Mmm... https://elpa.gnu.org/packages/vlf.html ?



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 17:36               ` Stefan Kangas
@ 2022-08-05  9:36                 ` Eric S Fraga
  2022-08-05 11:13                   ` Eli Zaretskii
  0 siblings, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-05  9:36 UTC (permalink / raw)
  To: emacs-devel

On Thursday,  4 Aug 2022 at 17:36, Stefan Kangas wrote:
> That's an odd message.  I don't think any user should ever need to
> install LEIM these days?

Takes me back...

I have built Emacs from git just now and I'm getting the same behaviour.
I have no input methods.  Not sure what is happening.  This happens with
"emacs -Q" as well, by the way.

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-04 12:54     ` Eli Zaretskii
  2022-08-04 14:44       ` Alan Mackenzie
@ 2022-08-05 10:56       ` Alan Mackenzie
  2022-08-05 11:20         ` Eli Zaretskii
  1 sibling, 1 reply; 32+ messages in thread
From: Alan Mackenzie @ 2022-08-05 10:56 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: larsi, emacs-devel, gregory

Hello, Eli.

On Thu, Aug 04, 2022 at 15:54:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Aug 2022 10:44:05 +0000
> > Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > > .... because if you, instead of a 10K line in that C++ file, inserted
> > > a 1M line, then Emacs would previously hang indefinitely, but with the
> > > optimisation, it doesn't.

> > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > indeed a bit sluggish but "hang indefinitely" is an exaggeration.

> Try it with a 20MB raw string, then.  And, for good measure, in an
> unoptimized build.  These are the cases we are trying to make
> workable.

> If all you are saying is that the default value of long-line-threshold
> is too low, we can definitely discuss a better value.

> > Having loaded the file in C++ Mode (without the spiking of
> > narrow-to-region and widen), it took 90 seconds for M-> (first time).

> And you consider that reasonable?

No, it wasn't reasonable, but neither was it "hang indefinitely".

The problem was a single font-lock clause, after whose removal, the M->
took about 1 second (first time) in the file with a 1,000,000 long line.
This clause can be put back into CC Mode and optimised, probably by
checking for being inside a literal.

All the other sluggishness has also vanished with that change.  I still
get the overflow in the regexp engine stack on inserting text.  That,
again, is a bug that surely can be fixed.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05  9:36                 ` Eric S Fraga
@ 2022-08-05 11:13                   ` Eli Zaretskii
  2022-08-05 12:28                     ` Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-05 11:13 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: emacs-devel

> From: Eric S Fraga <e.fraga@ucl.ac.uk>
> Date: Fri, 05 Aug 2022 10:36:47 +0100
> 
> On Thursday,  4 Aug 2022 at 17:36, Stefan Kangas wrote:
> > That's an odd message.  I don't think any user should ever need to
> > install LEIM these days?
> 
> Takes me back...
> 
> I have built Emacs from git just now and I'm getting the same behaviour.
> I have no input methods.  Not sure what is happening.  This happens with
> "emacs -Q" as well, by the way.

Try "make bootstrap".



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 10:56       ` Alan Mackenzie
@ 2022-08-05 11:20         ` Eli Zaretskii
  2022-08-05 13:04           ` Dmitry Gutov
  0 siblings, 1 reply; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-05 11:20 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: larsi, emacs-devel, gregory

> Date: Fri, 5 Aug 2022 10:56:20 +0000
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Having loaded the file in C++ Mode (without the spiking of
> > > narrow-to-region and widen), it took 90 seconds for M-> (first time).
> 
> > And you consider that reasonable?
> 
> No, it wasn't reasonable, but neither was it "hang indefinitely".

Many/most users would never wait for 90 seconds for such a simple
command.  So yes, it's "indefinitely" for all practical purposes.
(And imagine how long it would take in an unoptimized build that I'm
running every day.)

> The problem was a single font-lock clause, after whose removal, the M->
> took about 1 second (first time) in the file with a 1,000,000 long line.
> This clause can be put back into CC Mode and optimised, probably by
> checking for being inside a literal.
> 
> All the other sluggishness has also vanished with that change.  I still
> get the overflow in the regexp engine stack on inserting text.  That,
> again, is a bug that surely can be fixed.

Thanks.  Any improvements in font-lock of any major mode is welcome.
If/when enough of them get their act together, we might revisit the
default value of long-line-threshold, as I already said many times.

I also encourage the interested parties to submit changes to extend
'widen' so that it could optionally break the lock.  Then we could
allow using that option in modes whose font-lock is not misbehaving
with long lines.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 11:13                   ` Eli Zaretskii
@ 2022-08-05 12:28                     ` Eric S Fraga
  2022-08-05 13:40                       ` Eric S Fraga
  0 siblings, 1 reply; 32+ messages in thread
From: Eric S Fraga @ 2022-08-05 12:28 UTC (permalink / raw)
  To: emacs-devel

On Friday,  5 Aug 2022 at 14:13, Eli Zaretskii wrote:
> Try "make bootstrap".

As a matter of course, I always build Emacs with 'make -j 20 bootstrap'.

Any other suggestions please?  I am having a few problems with the
current Emacs including input methods (none) and various undefined
font-lock faces, having changed *nothing* in my configuration.

Thank you,
eric
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 11:20         ` Eli Zaretskii
@ 2022-08-05 13:04           ` Dmitry Gutov
  2022-08-05 14:22             ` Eli Zaretskii
  0 siblings, 1 reply; 32+ messages in thread
From: Dmitry Gutov @ 2022-08-05 13:04 UTC (permalink / raw)
  To: Eli Zaretskii, Alan Mackenzie; +Cc: larsi, emacs-devel, gregory

On 05.08.2022 14:20, Eli Zaretskii wrote:
> Thanks.  Any improvements in font-lock of any major mode is welcome.
> If/when enough of them get their act together, we might revisit the
> default value of long-line-threshold, as I already said many times.

As I said, changing long-line-threshold's default is not the solution 
because this variable also determines when the fixes for the other 
redisplay logic come into play.

And those cause sluggishness more extensively and much earlier than 
font-lock becomes a problem.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 12:28                     ` Eric S Fraga
@ 2022-08-05 13:40                       ` Eric S Fraga
  2022-08-05 23:11                         ` Tim Cross
  2022-08-05 23:23                         ` Lars Ingebrigtsen
  0 siblings, 2 replies; 32+ messages in thread
From: Eric S Fraga @ 2022-08-05 13:40 UTC (permalink / raw)
  To: emacs-devel

On Friday,  5 Aug 2022 at 13:28, Eric S Fraga wrote:
> I am having a few problems with the current Emacs including input
> methods (none) and various undefined font-lock faces, having changed
> *nothing* in my configuration.

Okay, I have at least tracked down the font lock faces problem:
font-lock-reference-face has been deleted as it had been marked obsolete
for some time.  Unfortunately, there are several packages out there in
the wild still using it including ess, gnuplot-mode, and
graphviz-dot-mode, both of which are crucial to my work.  Hey hum.  Time
to edit and post issues where relevant.

But I still have no input methods...  is there an autoload missing in
the build or init process maybe, as the relevant .el files are all
there?

Thank you,
eric

PS -I may have hijacked this thread: apologies if so and happy to start
a new one.
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 13:04           ` Dmitry Gutov
@ 2022-08-05 14:22             ` Eli Zaretskii
  2022-08-05 14:29               ` Dmitry Gutov
  0 siblings, 1 reply; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-05 14:22 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: acm, larsi, emacs-devel, gregory

> Date: Fri, 5 Aug 2022 16:04:08 +0300
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 14:20, Eli Zaretskii wrote:
> > Thanks.  Any improvements in font-lock of any major mode is welcome.
> > If/when enough of them get their act together, we might revisit the
> > default value of long-line-threshold, as I already said many times.
> 
> As I said, changing long-line-threshold's default is not the solution 
> because this variable also determines when the fixes for the other 
> redisplay logic come into play.
> 
> And those cause sluggishness more extensively and much earlier than 
> font-lock becomes a problem.

That's incorrect, or at least inaccurate: part of slow redisplay is
due to the fact that we fontify as part of displaying or as part of
layout calculations.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 14:22             ` Eli Zaretskii
@ 2022-08-05 14:29               ` Dmitry Gutov
  2022-08-05 15:30                 ` Eli Zaretskii
  0 siblings, 1 reply; 32+ messages in thread
From: Dmitry Gutov @ 2022-08-05 14:29 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: acm, larsi, emacs-devel, gregory

On 05.08.2022 17:22, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 16:04:08 +0300
>> Cc:larsi@gnus.org,emacs-devel@gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 05.08.2022 14:20, Eli Zaretskii wrote:
>>> Thanks.  Any improvements in font-lock of any major mode is welcome.
>>> If/when enough of them get their act together, we might revisit the
>>> default value of long-line-threshold, as I already said many times.
>> As I said, changing long-line-threshold's default is not the solution
>> because this variable also determines when the fixes for the other
>> redisplay logic come into play.
>>
>> And those cause sluggishness more extensively and much earlier than
>> font-lock becomes a problem.
> That's incorrect, or at least inaccurate: part of slow redisplay is
> due to the fact that we fontify as part of displaying or as part of
> layout calculations.

The slow part I'm referring to is not in font-lock's code, nor in CC 
Mode's font-lock rules. So it can be "fixed" independently.

Because while it's still there, it makes debugging and optimizing 
font-lock code and major modes's font-lock rules harder.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 14:29               ` Dmitry Gutov
@ 2022-08-05 15:30                 ` Eli Zaretskii
  2022-08-05 19:50                   ` Dmitry Gutov
  0 siblings, 1 reply; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-05 15:30 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: acm, larsi, emacs-devel, gregory

> Date: Fri, 5 Aug 2022 17:29:46 +0300
> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 17:22, Eli Zaretskii wrote:
> >> Date: Fri, 5 Aug 2022 16:04:08 +0300
> >> Cc:larsi@gnus.org,emacs-devel@gnu.org,gregory@heytings.org
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >> On 05.08.2022 14:20, Eli Zaretskii wrote:
> >>> Thanks.  Any improvements in font-lock of any major mode is welcome.
> >>> If/when enough of them get their act together, we might revisit the
> >>> default value of long-line-threshold, as I already said many times.
> >> As I said, changing long-line-threshold's default is not the solution
> >> because this variable also determines when the fixes for the other
> >> redisplay logic come into play.
> >>
> >> And those cause sluggishness more extensively and much earlier than
> >> font-lock becomes a problem.
> > That's incorrect, or at least inaccurate: part of slow redisplay is
> > due to the fact that we fontify as part of displaying or as part of
> > layout calculations.
> 
> The slow part I'm referring to is not in font-lock's code, nor in CC 
> Mode's font-lock rules. So it can be "fixed" independently.

Then where is it?



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 15:30                 ` Eli Zaretskii
@ 2022-08-05 19:50                   ` Dmitry Gutov
  2022-08-06  5:36                     ` Eli Zaretskii
  0 siblings, 1 reply; 32+ messages in thread
From: Dmitry Gutov @ 2022-08-05 19:50 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: acm, larsi, emacs-devel, gregory

On 05.08.2022 18:30, Eli Zaretskii wrote:
>> The slow part I'm referring to is not in font-lock's code, nor in CC
>> Mode's font-lock rules. So it can be "fixed" independently.
> Then where is it?

Probably somewhere in the recent xdisp.c changes. Not in 
'handle_fontified_prop', though.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 13:40                       ` Eric S Fraga
@ 2022-08-05 23:11                         ` Tim Cross
  2022-08-05 23:23                         ` Lars Ingebrigtsen
  1 sibling, 0 replies; 32+ messages in thread
From: Tim Cross @ 2022-08-05 23:11 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: emacs-devel


Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday,  5 Aug 2022 at 13:28, Eric S Fraga wrote:
>> I am having a few problems with the current Emacs including input
>> methods (none) and various undefined font-lock faces, having changed
>> *nothing* in my configuration.
>
> Okay, I have at least tracked down the font lock faces problem:
> font-lock-reference-face has been deleted as it had been marked obsolete
> for some time.  Unfortunately, there are several packages out there in
> the wild still using it including ess, gnuplot-mode, and
> graphviz-dot-mode, both of which are crucial to my work.  Hey hum.  Time
> to edit and post issues where relevant.
>
> But I still have no input methods...  is there an autoload missing in
> the build or init process maybe, as the relevant .el files are all
> there?
>
>

What (if any) arguments do you pass to configure? 



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 13:40                       ` Eric S Fraga
  2022-08-05 23:11                         ` Tim Cross
@ 2022-08-05 23:23                         ` Lars Ingebrigtsen
  2022-08-08 11:51                           ` Fraga, Eric
  1 sibling, 1 reply; 32+ messages in thread
From: Lars Ingebrigtsen @ 2022-08-05 23:23 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: emacs-devel

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> But I still have no input methods...  is there an autoload missing in
> the build or init process maybe, as the relevant .el files are all
> there?

Should be fixed now.




^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 19:50                   ` Dmitry Gutov
@ 2022-08-06  5:36                     ` Eli Zaretskii
  2022-08-06 10:54                       ` Dmitry Gutov
  0 siblings, 1 reply; 32+ messages in thread
From: Eli Zaretskii @ 2022-08-06  5:36 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: acm, larsi, emacs-devel, gregory

> Date: Fri, 5 Aug 2022 22:50:07 +0300
> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 18:30, Eli Zaretskii wrote:
> >> The slow part I'm referring to is not in font-lock's code, nor in CC
> >> Mode's font-lock rules. So it can be "fixed" independently.
> > Then where is it?
> 
> Probably somewhere in the recent xdisp.c changes.

Very unlikely.  The recent changes in xdisp.c could not have possibly
slowed down the code there.

> Not in 'handle_fontified_prop', though.

That still leaves a lot of possibilities, so is not useful for making
any progress in this matter.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: How the long-lines "optimisation" breaks font locking.
  2022-08-06  5:36                     ` Eli Zaretskii
@ 2022-08-06 10:54                       ` Dmitry Gutov
  0 siblings, 0 replies; 32+ messages in thread
From: Dmitry Gutov @ 2022-08-06 10:54 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: acm, larsi, emacs-devel, gregory

On 06.08.2022 08:36, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 22:50:07 +0300
>> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 05.08.2022 18:30, Eli Zaretskii wrote:
>>>> The slow part I'm referring to is not in font-lock's code, nor in CC
>>>> Mode's font-lock rules. So it can be "fixed" independently.
>>> Then where is it?
>>
>> Probably somewhere in the recent xdisp.c changes.
> 
> Very unlikely.  The recent changes in xdisp.c could not have possibly
> slowed down the code there.

No, no. They fixed it.

That's why the advice to set long-line-threshold to nil (if one wants 
correct fontification) is not so great: it brings those issues back.



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.
  2022-08-05 23:23                         ` Lars Ingebrigtsen
@ 2022-08-08 11:51                           ` Fraga, Eric
  0 siblings, 0 replies; 32+ messages in thread
From: Fraga, Eric @ 2022-08-08 11:51 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel@gnu.org

On Saturday,  6 Aug 2022 at 01:23, Lars Ingebrigtsen wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> But I still have no input methods...  is there an autoload missing in
>> the build or init process maybe, as the relevant .el files are all
>> there?
>
> Should be fixed now.

Thank you.  Working fine now!

-- 
Eric S Fraga, @ericsfraga:matrix.org, GnuPG: 0xc89193d8fffcf67d


^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2022-08-08 11:51 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-03 18:39 How the long-lines "optimisation" breaks font locking Alan Mackenzie
2022-08-03 19:03 ` Lars Ingebrigtsen
2022-08-04 10:44   ` Alan Mackenzie
2022-08-04 12:54     ` Eli Zaretskii
2022-08-04 14:44       ` Alan Mackenzie
2022-08-04 14:52         ` Lars Ingebrigtsen
2022-08-04 15:22         ` Gregory Heytings
2022-08-05  1:23           ` Ihor Radchenko
2022-08-04 16:09         ` Eli Zaretskii
2022-08-05 10:56       ` Alan Mackenzie
2022-08-05 11:20         ` Eli Zaretskii
2022-08-05 13:04           ` Dmitry Gutov
2022-08-05 14:22             ` Eli Zaretskii
2022-08-05 14:29               ` Dmitry Gutov
2022-08-05 15:30                 ` Eli Zaretskii
2022-08-05 19:50                   ` Dmitry Gutov
2022-08-06  5:36                     ` Eli Zaretskii
2022-08-06 10:54                       ` Dmitry Gutov
2022-08-04 13:11     ` Eric S Fraga
2022-08-04 13:34       ` Po Lu
2022-08-04 14:38         ` Eric S Fraga
2022-08-04 16:16           ` SOLVED: " Eric S Fraga
2022-08-04 16:21             ` Eric S Fraga
2022-08-04 17:36               ` Stefan Kangas
2022-08-05  9:36                 ` Eric S Fraga
2022-08-05 11:13                   ` Eli Zaretskii
2022-08-05 12:28                     ` Eric S Fraga
2022-08-05 13:40                       ` Eric S Fraga
2022-08-05 23:11                         ` Tim Cross
2022-08-05 23:23                         ` Lars Ingebrigtsen
2022-08-08 11:51                           ` Fraga, Eric
2022-08-03 19:15 ` Eli Zaretskii

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).