How the long-lines "optimisation" breaks font locking.

How the long-lines "optimisation" breaks font locking.

[Alan Mackenzie]

Hello Gregory and Emacs.

In the current master,
(i) start emacs -Q
(ii) C-x C-f ~/long-line.cc
(iii) Enter the following into the buffer (3 lines):

char long_line[] = R"foo(

)foo"

(iv) Create a long line (> 10,000 characters) somehow and get it into
  the kill ring.
(v) On the blank line 2 in long-line.cc, C-y inserting the long line.

(vi) Note that there is now a long valid raw string in the buffer,
  correctly fontified.
(vii) Put point on a "o" in the foo on L3.  Delete the o
(viii) Note that the opening delimiter of the raw string on L1 now has
  warning-face.  This is correct.
(ix) Undo this last change with C-_.
(x) Note that the opening delimiter falsely retains the warning-face,
  although it is now the opener of a validly terminated raw string.
  This is a bug.

(xi) Insert a LF (or several, if needed) into the long line, separating
  it into two or more lines, none of which is "long".
(xii) Note that the raw string opener still has warning-face.  A bug.
(xiii) Do M-x c++-mode.  This fails to initialise the fontification
  correctly, leaving the warning-face where it no longer belongs.  This
  is a bug.
(xiv) Note that in the foregoing operations, sometimes an error message
  appears in the echo area about the lack of an expected syntax-table
  text property.  This is a bug.

I don't think it is improbable that a C++ hacker will create a long line
in a raw string.  That this can cause permanent damage to the buffer's
fontification doesn't seem a good thing.  That it is the recent
"optimisation" for long lines which causes it can be seen by repeating
the above steps, but with a line less than 10,000 characters long.  The
fontification then behaves correctly.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How the long-lines "optimisation" breaks font locking.

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> I don't think it is improbable that a C++ hacker will create a long line
> in a raw string.

If you don't think that's improbable, then you should like the new
optimisations a lot -- because if you, instead of a 10K line in that C++
file, inserted a 1M line, then Emacs would previously hang indefinitely,
but with the optimisation, it doesn't.

Which is the point here.

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Wed, 3 Aug 2022 18:39:16 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello Gregory and Emacs.
> 
> In the current master,
> (i) start emacs -Q
> (ii) C-x C-f ~/long-line.cc
> (iii) Enter the following into the buffer (3 lines):
> 
> char long_line[] = R"foo(
> 
> )foo"
> 
> (iv) Create a long line (> 10,000 characters) somehow and get it into
>   the kill ring.
> (v) On the blank line 2 in long-line.cc, C-y inserting the long line.
> 
> (vi) Note that there is now a long valid raw string in the buffer,
>   correctly fontified.
> (vii) Put point on a "o" in the foo on L3.  Delete the o
> (viii) Note that the opening delimiter of the raw string on L1 now has
>   warning-face.  This is correct.
> (ix) Undo this last change with C-_.
> (x) Note that the opening delimiter falsely retains the warning-face,
>   although it is now the opener of a validly terminated raw string.
>   This is a bug.
> 
> (xi) Insert a LF (or several, if needed) into the long line, separating
>   it into two or more lines, none of which is "long".
> (xii) Note that the raw string opener still has warning-face.  A bug.
> (xiii) Do M-x c++-mode.  This fails to initialise the fontification
>   correctly, leaving the warning-face where it no longer belongs.  This
>   is a bug.
> (xiv) Note that in the foregoing operations, sometimes an error message
>   appears in the echo area about the lack of an expected syntax-table
>   text property.  This is a bug.

I cannot reproduce this.  I don't see any warning-face in step viii,
and no problem in step x.

> I don't think it is improbable that a C++ hacker will create a long line
> in a raw string.  That this can cause permanent damage to the buffer's
> fontification doesn't seem a good thing.  That it is the recent
> "optimisation" for long lines which causes it can be seen by repeating
> the above steps, but with a line less than 10,000 characters long.  The
> fontification then behaves correctly.

Alan, please stop fighting this with impractical specially-concocted
scenarios.  No one in their right minds will need such long strings,
and if they do, and don't like whatever problems that causes (which I
cannot reproduce in the first place), they can enlarge the value of
long-line-threshold, or even make it nil.  Puff! problem solved.

Re: How the long-lines "optimisation" breaks font locking.

[Alan Mackenzie]

Hello, Lars.

On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I don't think it is improbable that a C++ hacker will create a long line
> > in a raw string.

> If you don't think that's improbable, then you should like the new
> optimisations a lot -- ....

These "optimisations" break CC Mode.

> .... because if you, instead of a 10K line in that C++ file, inserted
> a 1M line, then Emacs would previously hang indefinitely, but with the
> optimisation, it doesn't.

Well I tried CC Mode with a 1,000,000 character raw string.  It was
indeed a bit sluggish but "hang indefinitely" is an exaggeration.

Having loaded the file in C++ Mode (without the spiking of
narrow-to-region and widen), it took 90 seconds for M-> (first time).
Moving point was sluggish, taking perhaps 1 to 3 seconds, as did
scrolling by a screenful.  Inserting text in the middle of the line
caused an overflow in the regexp engine stack.  Interfering with the raw
string delimiters (specifically, changing the identifier in the
delimiter at either end of the string) again took 90 seconds.

Doing these things in the current master branch indeed appeared to be
fast, but at one point an error in an after-change-function caused
after-change-functions to get set to nil, crashing CC Mode.

> Which is the point here.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 10:44:05 +0000
> Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Lars.
> 
> On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> > Alan Mackenzie <acm@muc.de> writes:
> 
> > > I don't think it is improbable that a C++ hacker will create a long line
> > > in a raw string.
> 
> > If you don't think that's improbable, then you should like the new
> > optimisations a lot -- ....
> 
> These "optimisations" break CC Mode.

They don't "break" it, they cause it occasionally produce inaccurate
fontifications.  And only when otherwise Emacs becomes unusable.

> > .... because if you, instead of a 10K line in that C++ file, inserted
> > a 1M line, then Emacs would previously hang indefinitely, but with the
> > optimisation, it doesn't.
> 
> Well I tried CC Mode with a 1,000,000 character raw string.  It was
> indeed a bit sluggish but "hang indefinitely" is an exaggeration.

Try it with a 20MB raw string, then.  And, for good measure, in an
unoptimized build.  These are the cases we are trying to make
workable.

If all you are saying is that the default value of long-line-threshold
is too low, we can definitely discuss a better value.

> Having loaded the file in C++ Mode (without the spiking of
> narrow-to-region and widen), it took 90 seconds for M-> (first time).

And you consider that reasonable?

> Moving point was sluggish, taking perhaps 1 to 3 seconds, as did
> scrolling by a screenful.

Did you try moving and the beginning of the buffer or at its end?  The
speed is very different.

> Inserting text in the middle of the line caused an overflow in the
> regexp engine stack.  Interfering with the raw string delimiters
> (specifically, changing the identifier in the delimiter at either
> end of the string) again took 90 seconds.

And this is in your opinion reasonable performance?

> Doing these things in the current master branch indeed appeared to be
> fast, but at one point an error in an after-change-function caused
> after-change-functions to get set to nil, crashing CC Mode.

If you can show a recipe for this problem, we will fix it.  This code
is WIP, so some problems definitely remain and should be reported and
fixed.

Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

I don't know if this is related or not but I have just built emacs from
current git and gnus buffers have lost all fontification.  All other
modes, including message-mode, seem to be just fine.

I've not investigated beyond this but thought I'd highlight the issue.
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3

Re: How the long-lines "optimisation" breaks font locking.

[Po Lu]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I don't know if this is related or not but I have just built emacs from
> current git and gnus buffers have lost all fontification.  All other
> modes, including message-mode, seem to be just fine.

Did you try "make bootstrap"?  It was enough to fix that problem when I
experienced it several weeks ago (around one of the loaddefs-related
changes).

Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

On Thursday,  4 Aug 2022 at 21:34, Po Lu wrote:
> Did you try "make bootstrap"?  

Yes, I always build using bootstrap.

Interestingly, in one post I tried to read (but not in any others), I
did get an error message:

gnus-summary-highlight-line: Wrong type argument: listp, font-lock-string-face

but it doesn't help me figure anything out.  The face itself is fine as
is being used as it should in other buffers/modes.

I need to check if anything has changed in gnus recently.

thank you,
eric

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3

Re: How the long-lines "optimisation" breaks font locking.

[Alan Mackenzie]

Hello, Eli.

On Thu, Aug 04, 2022 at 15:54:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Aug 2022 10:44:05 +0000
> > Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> > From: Alan Mackenzie <acm@muc.de>

> > Hello, Lars.

> > On Wed, Aug 03, 2022 at 21:03:56 +0200, Lars Ingebrigtsen wrote:
> > > Alan Mackenzie <acm@muc.de> writes:

> > > > I don't think it is improbable that a C++ hacker will create a long line
> > > > in a raw string.

> > > If you don't think that's improbable, then you should like the new
> > > optimisations a lot -- ....

> > These "optimisations" break CC Mode.

> They don't "break" it, they cause it occasionally produce inaccurate
> fontifications.  And only when otherwise Emacs becomes unusable.

> > > .... because if you, instead of a 10K line in that C++ file, inserted
> > > a 1M line, then Emacs would previously hang indefinitely, but with the
> > > optimisation, it doesn't.

> > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > indeed a bit sluggish but "hang indefinitely" is an exaggeration.

> Try it with a 20MB raw string, then.  And, for good measure, in an
> unoptimized build.  These are the cases we are trying to make
> workable.

No matter what criterion one deems "workable", there will be some size of
buffer, or line, which renders Emacs "unworkable".

> If all you are saying is that the default value of long-line-threshold
> is too low, we can definitely discuss a better value.

> > Having loaded the file in C++ Mode (without the spiking of
> > narrow-to-region and widen), it took 90 seconds for M-> (first time).

> And you consider that reasonable?

No, I don't.  I was just correcting Lars's mistaken assertion that such a
line would make Emacs "hang indefinitely".

[ .... ]

> > Doing these things in the current master branch indeed appeared to be
> > fast, but at one point an error in an after-change-function caused
> > after-change-functions to get set to nil, crashing CC Mode.

> If you can show a recipe for this problem, we will fix it.  This code
> is WIP, so some problems definitely remain and should be reported and
> fixed.

I'll see if I can recreate it.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How the long-lines "optimisation" breaks font locking.

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> No, I don't.  I was just correcting Lars's mistaken assertion that such a
> line would make Emacs "hang indefinitely".

90s for a point movement command is "indefinitely" in my book.

Re: How the long-lines "optimisation" breaks font locking.

[Gregory Heytings]

>
> No matter what criterion one deems "workable", there will be some size 
> of buffer, or line, which renders Emacs "unworkable".
>

Wrong.  You can now edit files with 1 GB or 5 GB on a single line almost 
fine.  The limit is now the amount of memory on your computer.  I keep 
investigating to see if it's possible to overcome that limit.  Stay tuned!

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 14:44:05 +0000
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > > indeed a bit sluggish but "hang indefinitely" is an exaggeration.
> 
> > Try it with a 20MB raw string, then.  And, for good measure, in an
> > unoptimized build.  These are the cases we are trying to make
> > workable.
> 
> No matter what criterion one deems "workable", there will be some size of
> buffer, or line, which renders Emacs "unworkable".

Not with narrowing, no.  Because the size of the narrowed region is
independent of the buffer size.  That's the main idea of these
changes, and that's why the idea is IMO a good one.

> > > Doing these things in the current master branch indeed appeared to be
> > > fast, but at one point an error in an after-change-function caused
> > > after-change-functions to get set to nil, crashing CC Mode.
> 
> > If you can show a recipe for this problem, we will fix it.  This code
> > is WIP, so some problems definitely remain and should be reported and
> > fixed.
> 
> I'll see if I can recreate it.

TIA.

SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

Hello all again,

having done some investigating, I found that an old setting in my
configuration caused all kinds of problems.  It was using
quail-define-package to define emojis and some special accented
characters for email, a configuration which I have not needed for quite
some time.

Thank you all,
eric

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

... but I have no input methods at all.  If I M-x list-input-methods,
Emacs tells me:

,----
| No input method is available, perhaps because you have not
| installed LEIM (Libraries of Emacs Input Methods).
`----

One step forward, two steps back...  Hey hum.

I'm off to have a beer.  Long day.  ;-)

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-04) on Debian 11.3

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Stefan Kangas]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> ... but I have no input methods at all.  If I M-x list-input-methods,
> Emacs tells me:
>
> ,----
> | No input method is available, perhaps because you have not
> | installed LEIM (Libraries of Emacs Input Methods).
> `----

That's an odd message.  I don't think any user should ever need to
install LEIM these days?

Re: How the long-lines "optimisation" breaks font locking.

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> No matter what criterion one deems "workable", there will be some size 
>> of buffer, or line, which renders Emacs "unworkable".
>>
>
> Wrong.  You can now edit files with 1 GB or 5 GB on a single line almost 
> fine.  The limit is now the amount of memory on your computer.  I keep 
> investigating to see if it's possible to overcome that limit.  Stay tuned!

Mmm... https://elpa.gnu.org/packages/vlf.html ?

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

On Thursday,  4 Aug 2022 at 17:36, Stefan Kangas wrote:
> That's an odd message.  I don't think any user should ever need to
> install LEIM these days?

Takes me back...

I have built Emacs from git just now and I'm getting the same behaviour.
I have no input methods.  Not sure what is happening.  This happens with
"emacs -Q" as well, by the way.

-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3

Re: How the long-lines "optimisation" breaks font locking.

[Alan Mackenzie]

Hello, Eli.

On Thu, Aug 04, 2022 at 15:54:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Aug 2022 10:44:05 +0000
> > Cc: emacs-devel@gnu.org, Gregory Heytings <gregory@heytings.org>
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > > .... because if you, instead of a 10K line in that C++ file, inserted
> > > a 1M line, then Emacs would previously hang indefinitely, but with the
> > > optimisation, it doesn't.

> > Well I tried CC Mode with a 1,000,000 character raw string.  It was
> > indeed a bit sluggish but "hang indefinitely" is an exaggeration.

> Try it with a 20MB raw string, then.  And, for good measure, in an
> unoptimized build.  These are the cases we are trying to make
> workable.

> If all you are saying is that the default value of long-line-threshold
> is too low, we can definitely discuss a better value.

> > Having loaded the file in C++ Mode (without the spiking of
> > narrow-to-region and widen), it took 90 seconds for M-> (first time).

> And you consider that reasonable?

No, it wasn't reasonable, but neither was it "hang indefinitely".

The problem was a single font-lock clause, after whose removal, the M->
took about 1 second (first time) in the file with a 1,000,000 long line.
This clause can be put back into CC Mode and optimised, probably by
checking for being inside a literal.

All the other sluggishness has also vanished with that change.  I still
get the overflow in the regexp engine stack on inserting text.  That,
again, is a bug that surely can be fixed.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> From: Eric S Fraga <e.fraga@ucl.ac.uk>
> Date: Fri, 05 Aug 2022 10:36:47 +0100
> 
> On Thursday,  4 Aug 2022 at 17:36, Stefan Kangas wrote:
> > That's an odd message.  I don't think any user should ever need to
> > install LEIM these days?
> 
> Takes me back...
> 
> I have built Emacs from git just now and I'm getting the same behaviour.
> I have no input methods.  Not sure what is happening.  This happens with
> "emacs -Q" as well, by the way.

Try "make bootstrap".

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 10:56:20 +0000
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Having loaded the file in C++ Mode (without the spiking of
> > > narrow-to-region and widen), it took 90 seconds for M-> (first time).
> 
> > And you consider that reasonable?
> 
> No, it wasn't reasonable, but neither was it "hang indefinitely".

Many/most users would never wait for 90 seconds for such a simple
command.  So yes, it's "indefinitely" for all practical purposes.
(And imagine how long it would take in an unoptimized build that I'm
running every day.)

> The problem was a single font-lock clause, after whose removal, the M->
> took about 1 second (first time) in the file with a 1,000,000 long line.
> This clause can be put back into CC Mode and optimised, probably by
> checking for being inside a literal.
> 
> All the other sluggishness has also vanished with that change.  I still
> get the overflow in the regexp engine stack on inserting text.  That,
> again, is a bug that surely can be fixed.

Thanks.  Any improvements in font-lock of any major mode is welcome.
If/when enough of them get their act together, we might revisit the
default value of long-line-threshold, as I already said many times.

I also encourage the interested parties to submit changes to extend
'widen' so that it could optionally break the lock.  Then we could
allow using that option in modes whose font-lock is not misbehaving
with long lines.

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

On Friday,  5 Aug 2022 at 14:13, Eli Zaretskii wrote:
> Try "make bootstrap".

As a matter of course, I always build Emacs with 'make -j 20 bootstrap'.

Any other suggestions please?  I am having a few problems with the
current Emacs including input methods (none) and various undefined
font-lock faces, having changed *nothing* in my configuration.

Thank you,
eric
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3

Re: How the long-lines "optimisation" breaks font locking.

[Dmitry Gutov]

On 05.08.2022 14:20, Eli Zaretskii wrote:
> Thanks.  Any improvements in font-lock of any major mode is welcome.
> If/when enough of them get their act together, we might revisit the
> default value of long-line-threshold, as I already said many times.

As I said, changing long-line-threshold's default is not the solution 
because this variable also determines when the fixes for the other 
redisplay logic come into play.

And those cause sluggishness more extensively and much earlier than 
font-lock becomes a problem.

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Eric S Fraga]

On Friday,  5 Aug 2022 at 13:28, Eric S Fraga wrote:
> I am having a few problems with the current Emacs including input
> methods (none) and various undefined font-lock faces, having changed
> *nothing* in my configuration.

Okay, I have at least tracked down the font lock faces problem:
font-lock-reference-face has been deleted as it had been marked obsolete
for some time.  Unfortunately, there are several packages out there in
the wild still using it including ess, gnuplot-mode, and
graphviz-dot-mode, both of which are crucial to my work.  Hey hum.  Time
to edit and post issues where relevant.

But I still have no input methods...  is there an autoload missing in
the build or init process maybe, as the relevant .el files are all
there?

Thank you,
eric

PS -I may have hijacked this thread: apologies if so and happy to start
a new one.
-- 
Eric S Fraga via gnus (Emacs 29.0.50 2022-08-05) on Debian 11.3

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 16:04:08 +0300
> Cc: larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 14:20, Eli Zaretskii wrote:
> > Thanks.  Any improvements in font-lock of any major mode is welcome.
> > If/when enough of them get their act together, we might revisit the
> > default value of long-line-threshold, as I already said many times.
> 
> As I said, changing long-line-threshold's default is not the solution 
> because this variable also determines when the fixes for the other 
> redisplay logic come into play.
> 
> And those cause sluggishness more extensively and much earlier than 
> font-lock becomes a problem.

That's incorrect, or at least inaccurate: part of slow redisplay is
due to the fact that we fontify as part of displaying or as part of
layout calculations.

Re: How the long-lines "optimisation" breaks font locking.

[Dmitry Gutov]

On 05.08.2022 17:22, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 16:04:08 +0300
>> Cc:larsi@gnus.org,emacs-devel@gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 05.08.2022 14:20, Eli Zaretskii wrote:
>>> Thanks.  Any improvements in font-lock of any major mode is welcome.
>>> If/when enough of them get their act together, we might revisit the
>>> default value of long-line-threshold, as I already said many times.
>> As I said, changing long-line-threshold's default is not the solution
>> because this variable also determines when the fixes for the other
>> redisplay logic come into play.
>>
>> And those cause sluggishness more extensively and much earlier than
>> font-lock becomes a problem.
> That's incorrect, or at least inaccurate: part of slow redisplay is
> due to the fact that we fontify as part of displaying or as part of
> layout calculations.

The slow part I'm referring to is not in font-lock's code, nor in CC 
Mode's font-lock rules. So it can be "fixed" independently.

Because while it's still there, it makes debugging and optimizing 
font-lock code and major modes's font-lock rules harder.

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 17:29:46 +0300
> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 17:22, Eli Zaretskii wrote:
> >> Date: Fri, 5 Aug 2022 16:04:08 +0300
> >> Cc:larsi@gnus.org,emacs-devel@gnu.org,gregory@heytings.org
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >> On 05.08.2022 14:20, Eli Zaretskii wrote:
> >>> Thanks.  Any improvements in font-lock of any major mode is welcome.
> >>> If/when enough of them get their act together, we might revisit the
> >>> default value of long-line-threshold, as I already said many times.
> >> As I said, changing long-line-threshold's default is not the solution
> >> because this variable also determines when the fixes for the other
> >> redisplay logic come into play.
> >>
> >> And those cause sluggishness more extensively and much earlier than
> >> font-lock becomes a problem.
> > That's incorrect, or at least inaccurate: part of slow redisplay is
> > due to the fact that we fontify as part of displaying or as part of
> > layout calculations.
> 
> The slow part I'm referring to is not in font-lock's code, nor in CC 
> Mode's font-lock rules. So it can be "fixed" independently.

Then where is it?

Re: How the long-lines "optimisation" breaks font locking.

[Dmitry Gutov]

On 05.08.2022 18:30, Eli Zaretskii wrote:
>> The slow part I'm referring to is not in font-lock's code, nor in CC
>> Mode's font-lock rules. So it can be "fixed" independently.
> Then where is it?

Probably somewhere in the recent xdisp.c changes. Not in 
'handle_fontified_prop', though.

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday,  5 Aug 2022 at 13:28, Eric S Fraga wrote:
>> I am having a few problems with the current Emacs including input
>> methods (none) and various undefined font-lock faces, having changed
>> *nothing* in my configuration.
>
> Okay, I have at least tracked down the font lock faces problem:
> font-lock-reference-face has been deleted as it had been marked obsolete
> for some time.  Unfortunately, there are several packages out there in
> the wild still using it including ess, gnuplot-mode, and
> graphviz-dot-mode, both of which are crucial to my work.  Hey hum.  Time
> to edit and post issues where relevant.
>
> But I still have no input methods...  is there an autoload missing in
> the build or init process maybe, as the relevant .el files are all
> there?
>
>

What (if any) arguments do you pass to configure? 

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Lars Ingebrigtsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> But I still have no input methods...  is there an autoload missing in
> the build or init process maybe, as the relevant .el files are all
> there?

Should be fixed now.

Re: How the long-lines "optimisation" breaks font locking.

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 22:50:07 +0300
> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 18:30, Eli Zaretskii wrote:
> >> The slow part I'm referring to is not in font-lock's code, nor in CC
> >> Mode's font-lock rules. So it can be "fixed" independently.
> > Then where is it?
> 
> Probably somewhere in the recent xdisp.c changes.

Very unlikely.  The recent changes in xdisp.c could not have possibly
slowed down the code there.

> Not in 'handle_fontified_prop', though.

That still leaves a lot of possibilities, so is not useful for making
any progress in this matter.

Re: How the long-lines "optimisation" breaks font locking.

[Dmitry Gutov]

On 06.08.2022 08:36, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 22:50:07 +0300
>> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 05.08.2022 18:30, Eli Zaretskii wrote:
>>>> The slow part I'm referring to is not in font-lock's code, nor in CC
>>>> Mode's font-lock rules. So it can be "fixed" independently.
>>> Then where is it?
>>
>> Probably somewhere in the recent xdisp.c changes.
> 
> Very unlikely.  The recent changes in xdisp.c could not have possibly
> slowed down the code there.

No, no. They fixed it.

That's why the advice to set long-line-threshold to nil (if one wants 
correct fontification) is not so great: it brings those issues back.

Re: SOLVED: Re: How the long-lines "optimisation" breaks font locking.

[Fraga, Eric]

On Saturday,  6 Aug 2022 at 01:23, Lars Ingebrigtsen wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> But I still have no input methods...  is there an autoload missing in
>> the build or init process maybe, as the relevant .el files are all
>> there?
>
> Should be fixed now.

Thank you.  Working fine now!

-- 
Eric S Fraga, @ericsfraga:matrix.org, GnuPG: 0xc89193d8fffcf67d