bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1.1: Type: text/plain, Size: 976 bytes --]

To reproduce, save the attachment as `font-lock-hangs.el' and execute:

    $ emacs -Q -l font-lock-hangs.el

C-g doesn't help anymore. The only thing you can do is to kill and start
Emacs anew.

Git commit fd1ee05977.

To quote a discussion from another bug, the reply is from Eli Zaretski:

> > By the way, it would really be nice if Emacs could do something about
hangs irrespective of what causes
> > that. Even if Elisp code is buggy, Emacs itself should never allow it
to fall into an infinite loop and stop
> > responding to C-g, leaving full restart as the only way out.
>
> I think that's impossible in general, unless we restrict what Lisp
> programs can do.  Every programming language can be used to write a
> buggy program.
>
> However, it should be possible to prevent some cases of such
> problematic behavior, certainly so when the infloop is caused by our
> bug.  But for that we need to know the details of the specific case in
> order to investigate.

Paul

[-- Attachment #1.2: Type: text/html, Size: 1243 bytes --]

[-- Attachment #2: font-lock-hangs.el --]
[-- Type: text/x-emacs-lisp, Size: 843 bytes --]

(define-derived-mode buggy-mode nil "buggy"
  (setf font-lock-defaults '(nil nil t nil (font-lock-fontify-region-function . buggy-fontifier)))
  (font-lock-mode 1))

(defun buggy-fontifier (start end loudly)
  (add-face-text-property start (min (+ start 15) end) 'bold)
  (while t
    "whoopsie")
  nil)

(switch-to-buffer "*scratch*")
(dotimes (k 50)
  (insert "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n\n"))
(goto-char 1)
(buggy-mode)

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[dick]

That you think embedding the `(while t "whoopsie")` in a font-lock
callback somehow make emacs more culpable is indicative of a young
person's rush to judgment and inclination to conflate.

I challenge you to find another single-threaded program in which "while
(true)" doesn't require a kernel-level interrupt.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

> indicative of a young
> person's rush to judgment and inclination to conflate

Oh god, how much self-righteousness. Just fuck off if you don't understand
anything about creating reproducers.

Paul

[-- Attachment #2: Type: text/html, Size: 290 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[dick]

Oh what a horse's ass am I.

I see what you're saying.  A simple evaluation of (while t) is C-g'able.

jit-lock callbacks enjoy special status in that they're invoked
out-of-band in redisplay C code.  That's probably why they're not
interruptible from the interpreter loop.  But as you can tell from my
earlier outburst, I'm constantly talking out of school.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

dick <dick.r.chiang@gmail.com> writes:

> jit-lock callbacks enjoy special status in that they're invoked
> out-of-band in redisplay C code. That's probably why they're not
> interruptible from the interpreter loop.

If I remember correctly, the problem isn't really that `C-g' isn't able
to break, but that we then call the redisplay immediately again, which
then calls the font-locking code.

Let's see...  yes, with this slightly modified version of Paul's code,
after hitting `C-g' eight times, I get a redisplay finally and it says
"Called 8 times".

So I think there's scope for us to do something practical here with this
annoying problem.  It's hard enough to develop font locking code without
Emacs suddenly (and unbreakably) hanging on you when you've typed in
some buggy code.


[-- Attachment #2: font-lock-hangs.el --]
[-- Type: application/emacs-lisp, Size: 986 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Cc: 57804@debbugs.gnu.org, Paul Pogonyshev <pogonyshev@gmail.com>
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 14 Sep 2022 19:52:45 +0200
> 
> So I think there's scope for us to do something practical here with this
> annoying problem.  It's hard enough to develop font locking code without
> Emacs suddenly (and unbreakably) hanging on you when you've typed in
> some buggy code.

Just run your development version under a debugger, and you will
always be able to interrupt any loop, anywhere.

And I think you greatly underestimate the conceptual difficulty here.
Your conclusions are specific to this toy program.  But in real-life
use cases, it is entirely unclear how to handle this:

  . the fact that redisplay is called several times in a row doesn't
    necessarily mean we have a bug, there are packages out there which
    cause that due to stuff they do in various hooks (like linum.el,
    which moves overlays etc.)
  . the buggy code could be in some function registered with jit-lock
    or run from some hook invoked by redisplay that have nothing to do
    with font-lock, in which case turning off font-lock will not solve
    anything

IOW, the general solution to this is not apparent, and I'm not sure it
exists, the only solution I'm aware of is to run unstable code under a
debugger.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> C-g doesn't help anymore. The only thing you can do is to kill and start Emacs anew.

[...]

> (define-derived-mode buggy-mode nil "buggy"
>   (setf font-lock-defaults '(nil nil t nil (font-lock-fontify-region-function . buggy-fontifier)))
>   (font-lock-mode 1))
>
> (defun buggy-fontifier (start end loudly)
>   (add-face-text-property start (min (+ start 15) end) 'bold)
>   (while t
>     "whoopsie")
>   nil)

Yes, it's really unfortunate that you can't get out of errors like this
without killing Emacs.

We've previously discussed making a certain key sequence disable font
locking in a buffer -- or blacklist the particular functions that's
inflooping somehow.

For instance, if the user hits `C-g' three times (and Emacs doesn't idle
in between), then we disable font-lock in that buffer.

There's also `max-redisplay-ticks' -- but I'm not sure that would have
helped here?

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

> We've previously discussed making a certain key sequence disable font
> locking in a buffer -- or blacklist the particular functions that's
> inflooping somehow.

I think from a user point of view a _special_ sequence is a bad idea,
as most users won't know about it.

Blacklisting I'd say is even worse, because that would mean a user
first have to expirience a hang of Emacs, find out _which_ function
caused it (remember, that Emacs is either unresponsive or is killed by
now), find out that it is possible to blacklist it and do it. Way too
much to expect from a user.

Besides, a function may not be outright malicious, only buggy in
certain corner-cases, so it would be much nicer to be able to
force-abort it when that happens. E.g. in my case this happened in
Logview mode, because it expected `widen' to, well, widen, but Emacs
`master' introduced half-cooked narrowed locking that broke that
expectation and then Logview fell into an infinite loop, because the
buffer was not in the state the mode expected it to be.

> For instance, if the user hits `C-g' three times (and Emacs doesn't idle
> in between), then we disable font-lock in that buffer.

That sounds much better. I know that `C-g' is what to press when Emacs
is stuck.

> There's also `max-redisplay-ticks' -- but I'm not sure that would have
> helped here?

If I add `(setf max-redisplay-ticks 100)' right after entering
`buggy-mode', it seems to have some effect, but not exactly what I'd
hope on. I suggest you try and see for yourself, hard to describe
them.

Paul

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

>> For instance, if the user hits `C-g' three times (and Emacs doesn't idle
>> in between), then we disable font-lock in that buffer.
>
> That sounds much better. I know that `C-g' is what to press when Emacs
> is stuck.

A single `C-g' wouldn't be quite the right thing, because the user may
hit `C-g' by accident while font-lock happens to run (and it's not
hanging), which is why I though a special key sequence like `C-g C-g
C-g' might make sense.

>> There's also `max-redisplay-ticks' -- but I'm not sure that would have
>> helped here?
>
> If I add `(setf max-redisplay-ticks 100)' right after entering
> `buggy-mode', it seems to have some effect, but not exactly what I'd
> hope on. I suggest you try and see for yourself, hard to describe
> them.

Doesn't seem to have any effect here...

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

>
> > A single `C-g' wouldn't be quite the right thing, because the user may
> > hit `C-g' by accident while font-lock happens to run (and it's not
> > hanging), which is why I though a special key sequence like `C-g C-g
> > C-g' might make sense.
>
> That one is too fragile I agree. If "special" means `C-g C-g C-g' then
> that is fine in my book. If something gets stuck and doesn't respond
> to one "abort this" command, I usually tend to try several times
> anyway, and suspect this is the same for a lot of people.
>
> > Doesn't seem to have any effect here...
>
> It seems to have some effect with `C-g' after some time.  But still
> doesn't abort it, only shows some warning, yet doesn't make Emacs
> responsive.
>
> Paul
>

[-- Attachment #2: Type: text/html, Size: 1019 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Cc: 57804@debbugs.gnu.org
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 14 Sep 2022 19:25:49 +0200
> 
> Paul Pogonyshev <pogonyshev@gmail.com> writes:
> 
> >> For instance, if the user hits `C-g' three times (and Emacs doesn't idle
> >> in between), then we disable font-lock in that buffer.
> >
> > That sounds much better. I know that `C-g' is what to press when Emacs
> > is stuck.
> 
> A single `C-g' wouldn't be quite the right thing, because the user may
> hit `C-g' by accident while font-lock happens to run (and it's not
> hanging), which is why I though a special key sequence like `C-g C-g
> C-g' might make sense.

I frequently find myself hitting C-g several times for reasons
unrelated to any infloop.

> > If I add `(setf max-redisplay-ticks 100)' right after entering
> > `buggy-mode', it seems to have some effect, but not exactly what I'd
> > hope on. I suggest you try and see for yourself, hard to describe
> > them.
> 
> Doesn't seem to have any effect here...

It shouldn't, because an infloop doesn't consume any "redisplay
ticks".

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> A single `C-g' wouldn't be quite the right thing, because the user may
>> hit `C-g' by accident while font-lock happens to run (and it's not
>> hanging), which is why I though a special key sequence like `C-g C-g
>> C-g' might make sense.
>
> I frequently find myself hitting C-g several times for reasons
> unrelated to any infloop.

But like I said, three `C-g's without any idle time in between would be
the magical command to switch off font locking.  I think that's hard to
do accidentally in normal circumstances.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: pogonyshev@gmail.com,  57804@debbugs.gnu.org
> Date: Wed, 14 Sep 2022 19:45:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> A single `C-g' wouldn't be quite the right thing, because the user may
> >> hit `C-g' by accident while font-lock happens to run (and it's not
> >> hanging), which is why I though a special key sequence like `C-g C-g
> >> C-g' might make sense.
> >
> > I frequently find myself hitting C-g several times for reasons
> > unrelated to any infloop.
> 
> But like I said, three `C-g's without any idle time in between would be
> the magical command to switch off font locking.  I think that's hard to
> do accidentally in normal circumstances.

There's nothing accidental about my typing several C-g in a row.  It
just isn't related to any font-lock.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> But like I said, three `C-g's without any idle time in between would be
>> the magical command to switch off font locking.  I think that's hard to
>> do accidentally in normal circumstances.
>
> There's nothing accidental about my typing several C-g in a row.  It
> just isn't related to any font-lock.

Org mode uses a similar method to catch rare edge cases and make users
report them. We make the user aware about possible reaction to multiple
C-g's using the following message:

`org-element--parse-buffer': Suppressed `\\[keyboard-quit]'. Press
`\\[keyboard-quit]' %d more times to force interruption.

The code goes like

(when (and inhibit-quit org-element--cache-interrupt-C-g quit-flag)
               (when quit-flag
	         (cl-incf org-element--cache-interrupt-C-g-count)
                 (setq quit-flag nil))
               (when (>= org-element--cache-interrupt-C-g-count
                        org-element--cache-interrupt-C-g-max-count)
                 (setq quit-flag t)
                 (setq org-element--cache-interrupt-C-g-count 0)
                 (org-element-cache-reset)
                 (error "org-element: Parsing aborted by user.  Cache has been cleared.
If you observe Emacs hangs frequently, please report this to Org mode mailing list (M-x org-submit-bug-report)."))
               (message (substitute-command-keys
                         "`org-element--parse-buffer': Suppressed `\\[keyboard-quit]'.  Press `\\[keyboard-quit]' %d more times to force interruption.")
                        (- org-element--cache-interrupt-C-g-max-count
                           org-element--cache-interrupt-C-g-count)))

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

> 
> E.g. in my case this happened in Logview mode, because it expected 
> `widen' to, well, widen, but Emacs `master' introduced half-cooked 
> narrowed locking that broke that expectation
>

Yes, master contains work in progress.  If you want stable code, you 
should use the release branch.

Note that the "fully cooked" narrowing will not magically solve that 
problem, though.  Logview will have to be adapted to deal with the 
possibility of a locked narrowing.  What you should most probably do in 
your case is to increase the value of long-line-threshold (or disable 
whatever causes Logview to infloop when locked narrowing is in effect, if 
that's feasible).

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

> Logview will have to be adapted to deal with the
> possibility of a locked narrowing.

Is it technically impossible to lift narrowing restrictions
in Emacs 29 (as in, something would break), or is it "you
have to rewrite, because we have decided so"?

Paul

On Wed, 14 Sept 2022 at 19:34, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > E.g. in my case this happened in Logview mode, because it expected
> > `widen' to, well, widen, but Emacs `master' introduced half-cooked
> > narrowed locking that broke that expectation
> >
>
> Yes, master contains work in progress.  If you want stable code, you
> should use the release branch.
>
> Note that the "fully cooked" narrowing will not magically solve that
> problem, though.  Logview will have to be adapted to deal with the
> possibility of a locked narrowing.  What you should most probably do in
> your case is to increase the value of long-line-threshold (or disable
> whatever causes Logview to infloop when locked narrowing is in effect, if
> that's feasible).
>

[-- Attachment #2: Type: text/html, Size: 1501 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

> 
> Is it technically impossible to lift narrowing restrictions in Emacs 29
>

It is not, you can choose a radical approach and long-line-threshold to 
nil.  Or you can choose a less radical approach and set 
long-line-threshold to a larger value (as I already told you twice IIRC).

>
> (as in, something would break),
>

Locked narrowing is one part of the solution to the long lines bug in 
Emacs, so something could indeed break if you turn that solution off.

>
> or is it "you have to rewrite, because we have decided so"?
>

That's not how I would present things, no.  Why does Logview infloop when 
locked narrowing is in effect?  This might be a bug in Logview: 
fontification-functions / jit-lock-function is not supposed to access and 
modify a large portion of the buffer.  I don't use Logview, and you did 
not give a detailed recipe to reproduce the issue, so it's difficult to 
give more precise advice.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3463 bytes --]

> > or is it "you have to rewrite, because we have decided so"?
>
> That's not how I would present things, no.

It doesn't matter how you would present it, it's the result that
matters.

> Why does Logview infloop when
> locked narrowing is in effect?  This might be a bug in Logview

Because it assumes that after `widen' all restrictions are gone.  If I
do

    x = 10 + 20
    if (x == 25)
      throw new Error ();

and this results in an error, then the bug is not _in my code_.

If I need to treat every function as "pretty please, do this, but if
you don't, it's fine, I'll do something else", my code will inflate
tenfolds and will still not achieve its goals.

> fontification-functions / jit-lock-function is not supposed to access and
> modify a large portion of the buffer.

"Is not supposed" is not the same as "must not do, else everything
crashes".

> I don't use Logview, and you did
> not give a detailed recipe to reproduce the issue

Recipe is obvious:

    (widen)
    (if (/= (point-min) 1) (error "WTF is going on?"))

Why Logview needs to access the full buffer at any time? Because it
operates lazily. It expects that log files are several megabytes and
therefore doesn't parse them fully. There are at least two passes:
first splits the file into entries, second fontifies. The first can be
used for other reasons too. In other words, when fontification comes
around, the relevant buffer part may not be even split into entries
yet, so fontification code calls the splitting code first. Logview
also almost never moves the point internally (because that might be
slow in huge buffers), instead it needs to be able to query text
properties anywhere. And with narrowing this is impossible.

Frankly, I don't see why I need to tell you _why_ I need this. Even if
I write five pages explaining every single detail, all things I have
considered in the years this mode had been written (with one major
refactoring for performance reason), I know the answer: "Yeah, but you
could do X or Y instead". Of course, I have never thought about that,
yeah. Of course I have just chosen this approach randomly.

You either accept that there _might_ be something you haven't thought
about, you have never discovered or simply never needed. Or you
don't. Then you just say: "Do it this way! For me it's enough,
therefore it _must_ be enough for everyone. Or if it's not it's their
problem and I don't care.".

Paul

On Thu, 15 Sept 2022 at 17:10, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > Is it technically impossible to lift narrowing restrictions in Emacs 29
> >
>
> It is not, you can choose a radical approach and long-line-threshold to
> nil.  Or you can choose a less radical approach and set
> long-line-threshold to a larger value (as I already told you twice IIRC).
>
> >
> > (as in, something would break),
> >
>
> Locked narrowing is one part of the solution to the long lines bug in
> Emacs, so something could indeed break if you turn that solution off.
>
> >
> > or is it "you have to rewrite, because we have decided so"?
> >
>
> That's not how I would present things, no.  Why does Logview infloop when
> locked narrowing is in effect?  This might be a bug in Logview:
> fontification-functions / jit-lock-function is not supposed to access and
> modify a large portion of the buffer.  I don't use Logview, and you did
> not give a detailed recipe to reproduce the issue, so it's difficult to
> give more precise advice.
>

[-- Attachment #2: Type: text/html, Size: 4252 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>
> You either accept that there _might_ be something you haven't thought 
> about, you have never discovered or simply never needed. Or you don't. 
> Then you just say: "Do it this way! For me it's enough, therefore it 
> _must_ be enough for everyone. Or if it's not it's their problem and I 
> don't care.".
>

May I perhaps tell you that your attitude isn't exactly encouraging anyone 
to help you?  I tried to give you some advice in reply to your previous 
paragraphs, and deleted everything after reading that one.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

My attitude is explained by my general experience with Emacs itself and
reporting any problems I have had with it. Sometimes everything went out
fine, but far less often than I would have hoped. You may see me as a jerk,
I see most Emacs developers as unhelpful and not willing to take into
account real problems when those don't affect them personally. Doesn't
matter "who started first", that's just how it is.

I suspect your "help" was to the tune of "instead, you should just...".
That I won't miss, I have had a lot of "help" of this kind on StackOverflow.

So there.

Paul

On Thu, 15 Sept 2022 at 18:08, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > You either accept that there _might_ be something you haven't thought
> > about, you have never discovered or simply never needed. Or you don't.
> > Then you just say: "Do it this way! For me it's enough, therefore it
> > _must_ be enough for everyone. Or if it's not it's their problem and I
> > don't care.".
> >
>
> May I perhaps tell you that your attitude isn't exactly encouraging anyone
> to help you?  I tried to give you some advice in reply to your previous
> paragraphs, and deleted everything after reading that one.
>

[-- Attachment #2: Type: text/html, Size: 1711 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>
> My attitude is explained by my general experience with Emacs itself and 
> reporting any problems I have had with it. Sometimes everything went out 
> fine, but far less often than I would have hoped.
>

We're all doing our best to improve Emacs in our free time.  It is 
unreasonable to expect that everything would go fine (with your own 
definition of "fine") in all cases.  It would not be reasonable to expect 
that from a private company, either.

>
> You may see me as a jerk,
>

I do not.

>
> I suspect your "help" was to the tune of "instead, you should just...".
>

That's a rather strange conclusion.  I already told you thrice that a 
simple way to fix your problem is to set long-line-threshold to a larger 
value (or to nil).

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

> already told you thrice that a
> simple way to fix your problem is to set long-line-threshold to a larger
> value (or to nil).

Thanks, I added  `(setf long-line-threshold nil)' to Emacs startup
configuration
file a couple of days before. But unless I'm missing something, this is not
a
fix, only a workaround. The variable is not buffer-local, so I cannot
change it
in the mode, only as my personal setting.

Paul


On Thu, 15 Sept 2022 at 18:44, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > My attitude is explained by my general experience with Emacs itself and
> > reporting any problems I have had with it. Sometimes everything went out
> > fine, but far less often than I would have hoped.
> >
>
> We're all doing our best to improve Emacs in our free time.  It is
> unreasonable to expect that everything would go fine (with your own
> definition of "fine") in all cases.  It would not be reasonable to expect
> that from a private company, either.
>
> >
> > You may see me as a jerk,
> >
>
> I do not.
>
> >
> > I suspect your "help" was to the tune of "instead, you should just...".
> >
>
> That's a rather strange conclusion.  I already told you thrice that a
> simple way to fix your problem is to set long-line-threshold to a larger
> value (or to nil).
>

[-- Attachment #2: Type: text/html, Size: 1843 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 57804@debbugs.gnu.org
> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Thu, 15 Sep 2022 20:49:17 +0200
> 
> > already told you thrice that a
> > simple way to fix your problem is to set long-line-threshold to a larger
> > value (or to nil).
> 
> Thanks, I added  `(setf long-line-threshold nil)' to Emacs startup configuration
> file a couple of days before. But unless I'm missing something, this is not a
> fix, only a workaround.

What do you mean by ":workaround"?  Doing that disables the feature
which you don't want, so it fits the definition of "solution" in my
book.

> The variable is not buffer-local, so I cannot change it
> in the mode, only as my personal setting.

Every variable can be given a local value by using setq-local.  Did
you try that?  If you did, and it didn't work, please show a recipe to
reproduce this problem.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2508 bytes --]

> What do you mean by ":workaround"?

In this case something that can be done for me personally, but
not in the mode itself.

> Every variable can be given a local value by using setq-local.  Did
> you try that?

No. Tried now, it works.

Several questions though.

* I see that manually evaluating `(setq-local long-line-threshold
nil)' in a buffer where the optimization is already in effect (i.e.
where `(long-line-optimizations-p)' evaluates to t) doesn't disable
the optimization. Do you have a solution for that? Depending on the
mode being activated before Emacs decides to enable the optimization
(e.g. because on of the first lines is very long, I don't know how
exactly this is determined) seems very shaky. Also, what if someone
opens file `my-log-with-funny-extension.records' and then manually
activates Logview mode?

* I briefly looked at the branch `feature/improved-narrowed-locking'
and saw that locking grew "tags". This probably implies that this is
going to be used more in the future, maybe already in Emacs 29.1. Is
there going to be some way to disable each and every new tag? Should I
monitor Emacs sources for new cases of narrowed locking with a tag
previously not used? What if one day this becomes available to Elisp
and a submode that decides to narrow-lock for whatever reason?  How
could I prevent that?

* Wouldn't something like

    (let ((disable-locked-narrowing-yes-i-know-this-is-bad-but-still t))
      (widen)
      ...
      )

not be much more robust?

Paul

On Thu, 15 Sept 2022 at 21:16, Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, 57804@debbugs.gnu.org
> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Thu, 15 Sep 2022 20:49:17 +0200
> >
> > > already told you thrice that a
> > > simple way to fix your problem is to set long-line-threshold to a
> larger
> > > value (or to nil).
> >
> > Thanks, I added  `(setf long-line-threshold nil)' to Emacs startup
> configuration
> > file a couple of days before. But unless I'm missing something, this is
> not a
> > fix, only a workaround.
>
> What do you mean by ":workaround"?  Doing that disables the feature
> which you don't want, so it fits the definition of "solution" in my
> book.
>
> > The variable is not buffer-local, so I cannot change it
> > in the mode, only as my personal setting.
>
> Every variable can be given a local value by using setq-local.  Did
> you try that?  If you did, and it didn't work, please show a recipe to
> reproduce this problem.
>

[-- Attachment #2: Type: text/html, Size: 3424 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Thu, 15 Sep 2022 21:36:27 +0200
> Cc: Gregory Heytings <gregory@heytings.org>, Lars Ingebrigtsen <larsi@gnus.org>, 57804@debbugs.gnu.org
> 
> * I see that manually evaluating `(setq-local long-line-threshold
> nil)' in a buffer where the optimization is already in effect (i.e.
> where `(long-line-optimizations-p)' evaluates to t) doesn't disable
> the optimization. Do you have a solution for that?

Not at present, I believe (but Gregory may know better).  I don't
think we saw a need for that, yet.

> Depending on the
> mode being activated before Emacs decides to enable the optimization
> (e.g. because on of the first lines is very long, I don't know how
> exactly this is determined) seems very shaky.

I think you can rely on this: the decision is made the first time the
buffer is displayed, which is after the mode is turned on in it.

Of course, the danger is that if the file really has very long lines,
Emacs might become unusable.  That's what this feature is about:
making Emacs usable when editing files with very long lines.

> Also, what if someone
> opens file `my-log-with-funny-extension.records' and then manually
> activates Logview mode?

No solution for now.

> * I briefly looked at the branch `feature/improved-narrowed-locking'
> and saw that locking grew "tags". This probably implies that this is
> going to be used more in the future, maybe already in Emacs 29.1. Is
> there going to be some way to disable each and every new tag? Should I
> monitor Emacs sources for new cases of narrowed locking with a tag
> previously not used? What if one day this becomes available to Elisp
> and a submode that decides to narrow-lock for whatever reason?  How
> could I prevent that?

I'll leave it for Gregory to answer, since I don't yet know his plans
for that branch.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]


>
> I see that manually evaluating `(setq-local long-line-threshold nil)' in 
> a buffer where the optimization is already in effect (i.e. where 
> `(long-line-optimizations-p)' evaluates to t) doesn't disable the 
> optimization. Do you have a solution for that?
>

No, and that will not be supported.

>
> Depending on the mode being activated before Emacs decides to enable the 
> optimization (e.g. because one of the first lines is very long, I don't 
> know how exactly this is determined) seems very shaky.
>

Indeed.  As I told you the proper fix is not to disable these 
optimizations, but to adapt the code to handle locked narrowing, by 
explicitly unlocking the locked narrowing when, and only when, it needs to 
access a larger portion of the buffer.

>
> I briefly looked at the branch `feature/improved-narrowed-locking' and 
> saw that locking grew "tags". This probably implies that this is going 
> to be used more in the future, maybe already in Emacs 29.1. Is there 
> going to be some way to disable each and every new tag? Should I monitor 
> Emacs sources for new cases of narrowed locking with a tag previously 
> not used?
>

No, if your function is called inside fontification-functions, you will 
not have to monitor Emacs sources, your code will use a single tag, namely 
'fontification-functions.

>
> What if one day this becomes available to Elisp and a submode that 
> decides to narrow-lock for whatever reason?
>

Don't worry, that won't happen.

>
> Wouldn't something like
> 
> (let ((disable-locked-narrowing-yes-i-know-this-is-bad-but-still t))
>    (widen)
>    ...
> )
>
> not be much more robust?
>

Definitely not.  It is more important to take measures to ensure that 
Emacs remains responsive for its users than to minimize the effort of 
Elisp programmers.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

>> I see that manually evaluating `(setq-local long-line-threshold
>> nil)' in a buffer where the optimization is already in effect
>> (i.e. where `(long-line-optimizations-p)' evaluates to t) doesn't
>> disable the optimization. Do you have a solution for that?
>
> No, and that will not be supported.

Well, that's when you need it -- you've discovered that the current
buffer is all wonky because of this, so you want to switch it off in
that particular buffer.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2905 bytes --]

> No, if your function is called inside fontification-functions, you will
> not have to monitor Emacs sources, your code will use a single tag, namely
> 'fontification-functions.

So, I will be able to unlock?

> > Wouldn't something like [...] not be much more robust?
>
> Definitely not.

Or not? I don't understand how these two answers from you combine.

It is impossible to correctly fontify log buffers if you are
restricted to semi-arbitrary parts of it: you need to at least see
where the current entry starts, even if it for whatever reason
includes a line with 100 K characters.

> It isn't for me personally, no.  It is important for Emacs users.

How about letting Emacs users decide for themselves? Let Logview
behave against your recommendations. And then, when the users discover
how crappy and unresponsive it is, as a result, compared to everything
else, they will just uninstall the package and use something better.

Paul

On Thu, 15 Sept 2022 at 22:18, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > I see that manually evaluating `(setq-local long-line-threshold nil)' in
> > a buffer where the optimization is already in effect (i.e. where
> > `(long-line-optimizations-p)' evaluates to t) doesn't disable the
> > optimization. Do you have a solution for that?
> >
>
> No, and that will not be supported.
>
> >
> > Depending on the mode being activated before Emacs decides to enable the
> > optimization (e.g. because one of the first lines is very long, I don't
> > know how exactly this is determined) seems very shaky.
> >
>
> Indeed.  As I told you the proper fix is not to disable these
> optimizations, but to adapt the code to handle locked narrowing, by
> explicitly unlocking the locked narrowing when, and only when, it needs to
> access a larger portion of the buffer.
>
> >
> > I briefly looked at the branch `feature/improved-narrowed-locking' and
> > saw that locking grew "tags". This probably implies that this is going
> > to be used more in the future, maybe already in Emacs 29.1. Is there
> > going to be some way to disable each and every new tag? Should I monitor
> > Emacs sources for new cases of narrowed locking with a tag previously
> > not used?
> >
>
> No, if your function is called inside fontification-functions, you will
> not have to monitor Emacs sources, your code will use a single tag, namely
> 'fontification-functions.
>
> >
> > What if one day this becomes available to Elisp and a submode that
> > decides to narrow-lock for whatever reason?
> >
>
> Don't worry, that won't happen.
>
> >
> > Wouldn't something like
> >
> > (let ((disable-locked-narrowing-yes-i-know-this-is-bad-but-still t))
> >    (widen)
> >    ...
> > )
> >
> > not be much more robust?
> >
>
> Definitely not.  It is more important to take measures to ensure that
> Emacs remains responsive for its users than to minimize the effort of
> Elisp programmers.

[-- Attachment #2: Type: text/html, Size: 3641 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>> No, if your function is called inside fontification-functions, you will 
>> not have to monitor Emacs sources, your code will use a single tag, 
>> namely 'fontification-functions.
> 
> So, I will be able to unlock?
>

Yes.

>
> Let Logview behave against your recommendations. And then, when the 
> users discover how crappy and unresponsive it is, as a result, compared 
> to everything else, they will just uninstall the package and use 
> something better.
>

Given that you will be able to unlock, that will be possible.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2588 bytes --]

>>> No, if your function is called inside fontification-functions, you will
>>> not have to monitor Emacs sources, your code will use a single tag,
>>> namely 'fontification-functions.
>>
>> So, I will be able to unlock?
>
> Yes.

Good.

> > Wouldn't something like [...] not be much more robust?
>
> Definitely not.

But why? If you allow to lift restrictions, why not make it easy to
do? Call the variable/function that achieves this with a 100-character
name, brand the programmer an idiot in the name or documentation -
whatever. Don't advertise it in the manual, maybe add two lines
somewhere in a dark corner, repeating that only morons would use
it. But please, add something generic and simple to use.

Logview needs to temporarily cancel restrictions in _practically all_
its function. In the branch I see code like this:

    narrow_to_region_locked (make_fixnum (get_narrowed_begv (w, PT)),
                             make_fixnum (get_narrowed_zv (w, PT)),
                             hook);
                             ^^^^

So, suppose something in that hook (maybe even not installed by
Logview itself, e.g. by a minor mode or by the user: it is common to
have `add-hook' in Emacs initialization code) calls a function from
Logview. Now, that function from Logview does

    (please-widen-i-cant-work-with-locked-narrowing ...)

It, presumably, cannot know the tag used to install locked narrowing.
Will it be able to temporarily lift _any and all_ locked narrowing or
not? It probably won't freeze Emacs even if falling into an infinite
loop, likely there are not so many non-bugs like the one in this
thread, where misbehaving code can make Emacs hang irrepairably. But
still, if restrictions are not lifted, Logview function will likely
produce some incorrect result, require C-g to abort, or die with an
error.

Will the knowledge of the tag be required, as a sort of a "password",
to lift narrowing restrictions, or not?

Paul

On Thu, 15 Sept 2022 at 22:44, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >> No, if your function is called inside fontification-functions, you will
> >> not have to monitor Emacs sources, your code will use a single tag,
> >> namely 'fontification-functions.
> >
> > So, I will be able to unlock?
> >
>
> Yes.
>
> >
> > Let Logview behave against your recommendations. And then, when the
> > users discover how crappy and unresponsive it is, as a result, compared
> > to everything else, they will just uninstall the package and use
> > something better.
> >
>
> Given that you will be able to unlock, that will be possible.
>

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>
> Logview needs to temporarily cancel restrictions in _practically all_ 
> its function.
>

It can do so (or more precisely will be able to do so) by calling 
narrowing-unlock with the appropriate tag, which IIUC correctly is, in 
your case, 'fontification-functions.  I don't see why you would need 
anything else, especially given that a general mechanism to disable locked 
narrowing everywhere is already available for users who, like you, don't 
care about long lines.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]

> It can do so (or more precisely will be able to do so) by calling
> narrowing-unlock with the appropriate tag, which IIUC correctly is, in
> your case, 'fontification-functions.

OK, let me rewrite this in a bit more details. Here are some quotes
from the C code in the branch:

    safe_run_hooks_maybe_narrowed (Qpre_command_hook,
                                   XWINDOW (selected_window));

    ...

    safe_run_hooks_maybe_narrowed (Lisp_Object hook, struct window *w)
    {
      ...
      if (current_buffer->long_line_optimizations_p)
        narrow_to_region_locked (make_fixnum (get_narrowed_begv (w, PT)),
                                 make_fixnum (get_narrowed_zv (w, PT)),
                                 hook);

Note how `hook' becomes `tag', i.e. narrowing is locked with tag
`pre-command-hook' _in this case_. According to the code, it can also
be locked with a few others, e.g. `post-command-hook'.

Now, user writes in his `init.el' sth. like:

    (add-hook 'pre-command-hook (lambda ()
                                  (when (eq major-mode 'logview-mode)
                                    (logview-do-bla-bla-bla))))

I don't know why, this is a hypothetical, but fairly realistic
situation, right?

Now, let's say function `logview-do-bla-bla-bla' cannot work with
narrowed buffer (half of functions in Logview cannot).  So, as a first
step it does something like this:

    (save-restriction
      (widen)
      ...
      )

For this to keep working in Emacs 29, I will need to replace `(widen)'
with I-don't-yet-know-what.  But _I don't know the tag_.  That's the
problem.  The function is called from somewhere I have no control, it
can be bound to any hook that does install locked narrowing with some
tag or maybe does not - I have no way to know that.

Will I be able to lift locked narrowing restrictions without knowing
the tag?

Paul

On Thu, 15 Sept 2022 at 23:32, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > Logview needs to temporarily cancel restrictions in _practically all_
> > its function.
> >
>
> It can do so (or more precisely will be able to do so) by calling
> narrowing-unlock with the appropriate tag, which IIUC correctly is, in
> your case, 'fontification-functions.  I don't see why you would need
> anything else, especially given that a general mechanism to disable locked
> narrowing everywhere is already available for users who, like you, don't
> care about long lines.
>

[-- Attachment #2: Type: text/html, Size: 3149 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>
> I don't know why, this is a hypothetical, but fairly realistic 
> situation, right?
>

Discussing hypothetical issues leads nowhere, according to my experience. 
Let's focus on actual issues.

>
> Now, let's say function `logview-do-bla-bla-bla' cannot work with 
> narrowed buffer (half of functions in Logview cannot).
>

You said I'm not allowed to tell you that your code could do things 
differently, but that doesn't mean it isn't true.  It is for example 
possible to parse the buffer outside of fontification-functions and to use 
the result of that parsing inside fontification-functions.  Yet another 
method would be to use a simpler fontification method in buffers with too 
long lines.  Yet another method would be to turn font locking off in 
buffers with too long lines.  Yet another method would be to truncate too 
long lines.  I also cannot understand why it is necessary, in log files in 
which all lines are independent, to move beyond the beginning and end of a 
line to decide how it must be fontified.

> 
> Will I be able to lift locked narrowing restrictions without knowing the 
> tag?
>

This is akin to a security mechanism, there is no reason to make it 
possible to turn it off "too easily".

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3949 bytes --]

> > I don't know why, this is a hypothetical, but fairly realistic
> > situation, right?
>
> Discussing hypothetical issues leads nowhere, according to my experience.
> Let's focus on actual issues.

I have shown you a simple way how your supposed can break things. Yes,
it's not an everyday occurence, but if I could think of one easily,
this or something else to this effect will be used in reality. As a
reminder, Emacs self-advertises itself as "advanced, extensible"
editor. You are making some extensions illegal, because maybe
something somewhere might become slow. You are sacrificing
functionality in the name of performance.

> It is for example possible to parse the buffer outside of
> fontification-functions

You are ignoring that this is not only about fontification. It can
happen anywhere, because I may need full access to the buffer at any
time.

> and to use the result of that parsing inside
> fontification-functions.

Parsing is done lazily. You are saying I need to prepare parsing
results everywhere in the whole buffer at once (which easily can be 10
MB - I have seen logs 250 MB in size, but not ones with lines longer
than ~50 K characters), because I don't know in advance where
fontification (or whatever else) might be needed.

> Yet another method would be to use a simpler fontification method in
> buffers with too long lines.  Yet another method would be to turn
> font locking off in buffers with too long lines.

I don't want to lose fontification in a 10 MB buffer because one line
somewhere there is over 10000 characters.

> Yet another method would be to truncate too long lines.

Eh? And corrupt the log? Otherwise, even if it is invisible it stays
in the buffer.

> I also cannot understand why it is necessary, in log files in
> which all lines are independent

No they are not. Typically Java exceptions are logged like this:

    2022-09-15 23:11:00.014 ERROR [some.Class] (some-thread) bla-bla-bla
    exception.Class: bla-bla-bla
          at some.Class

and so on. Exceptions are not the only multiline entries.

I have already complained about this often-seen-here "I also cannot
understand", which implies "therefore no-one will ever need", but you
said it was bad attitude from me.

> > Will I be able to lift locked narrowing restrictions without knowing the
> > tag?
>
> This is akin to a security mechanism, there is no reason to make it
> possible to turn it off "too easily".

Security against what, for fuck's sake? By trying to prevent "making
it too easily", you are preventing this at all in general case. And
what are the gains?

Paul

On Fri, 16 Sept 2022 at 00:16, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >
> > I don't know why, this is a hypothetical, but fairly realistic
> > situation, right?
> >
>
> Discussing hypothetical issues leads nowhere, according to my experience.
> Let's focus on actual issues.
>
> >
> > Now, let's say function `logview-do-bla-bla-bla' cannot work with
> > narrowed buffer (half of functions in Logview cannot).
> >
>
> You said I'm not allowed to tell you that your code could do things
> differently, but that doesn't mean it isn't true.  It is for example
> possible to parse the buffer outside of fontification-functions and to use
> the result of that parsing inside fontification-functions.  Yet another
> method would be to use a simpler fontification method in buffers with too
> long lines.  Yet another method would be to turn font locking off in
> buffers with too long lines.  Yet another method would be to truncate too
> long lines.  I also cannot understand why it is necessary, in log files in
> which all lines are independent, to move beyond the beginning and end of a
> line to decide how it must be fontified.
>
> >
> > Will I be able to lift locked narrowing restrictions without knowing the
> > tag?
> >
>
> This is akin to a security mechanism, there is no reason to make it
> possible to turn it off "too easily".
>

[-- Attachment #2: Type: text/html, Size: 4797 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]


>> I also cannot understand why it is necessary, in log files in which all 
>> lines are independent
> 
> No they are not. Typically Java exceptions are logged like this:
> 
>     2022-09-15 23:11:00.014 ERROR [some.Class] (some-thread) bla-bla-bla
>     exception.Class: bla-bla-bla
>           at some.Class
> 
> and so on. Exceptions are not the only multiline entries.
>

Okay, but that still doesn't mean that you need to access the whole buffer 
at any time, does it?  You do not need to go further than the beginning of 
the "2022-09-15 23:11:00.014 ERROR..." line, or am I missing something?

>
> I have already complained about this often-seen-here "I also cannot 
> understand", which implies "therefore no-one will ever need"
>

No, that's not what it implies.  It is a request for clarification.

>
> By trying to prevent "making it too easily", you are preventing this at 
> all in general case.
>

If it is possible to unlock a locked narrowing, and if it is possible to 
disable locked narrowing completely, nothing is prevented.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Fri, 16 Sep 2022 00:53:37 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, 57804@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
> 
> > It is for example possible to parse the buffer outside of
> > fontification-functions
> 
> You are ignoring that this is not only about fontification. It can
> happen anywhere, because I may need full access to the buffer at any
> time.

Your mode is requested to be able to run in some kind of "simplified
mode" when long-line-threshold is exceeded, in a way that doesn't
require such full access.

> > Yet another method would be to use a simpler fontification method in
> > buffers with too long lines.  Yet another method would be to turn
> > font locking off in buffers with too long lines.
> 
> I don't want to lose fontification in a 10 MB buffer because one line
> somewhere there is over 10000 characters.

The locked restriction leaves accessible a region of the buffer that
is quite large.  So if you are fontifying a line that is not very long
in such a buffer, you should still be able to access enough text
before and after the window to be able to fontify.  It is expected
that modes which need to access more than that will be amended to
perform, perhaps sub-optimally, in such cases.  In extreme cases, you
can simply not fontify if there's no reasonable way of fontifying even
approximately.  (I don't believe such extreme measures should be
necessary, but they are a possibility.)

IOW, we fully expect that in some cases and with some features, there
will be partial loss of functionality in these cases.  We consider
that still much better than a locked-up Emacs that can only be killed.

> Security against what, for fuck's sake?

Language!

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

I give up. Do as you like.

Paul

On Fri, 16 Sept 2022 at 08:40, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Fri, 16 Sep 2022 00:53:37 +0200
> > Cc: Eli Zaretskii <eliz@gnu.org>, 57804@debbugs.gnu.org, Lars
> Ingebrigtsen <larsi@gnus.org>
> >
> > > It is for example possible to parse the buffer outside of
> > > fontification-functions
> >
> > You are ignoring that this is not only about fontification. It can
> > happen anywhere, because I may need full access to the buffer at any
> > time.
>
> Your mode is requested to be able to run in some kind of "simplified
> mode" when long-line-threshold is exceeded, in a way that doesn't
> require such full access.
>
> > > Yet another method would be to use a simpler fontification method in
> > > buffers with too long lines.  Yet another method would be to turn
> > > font locking off in buffers with too long lines.
> >
> > I don't want to lose fontification in a 10 MB buffer because one line
> > somewhere there is over 10000 characters.
>
> The locked restriction leaves accessible a region of the buffer that
> is quite large.  So if you are fontifying a line that is not very long
> in such a buffer, you should still be able to access enough text
> before and after the window to be able to fontify.  It is expected
> that modes which need to access more than that will be amended to
> perform, perhaps sub-optimally, in such cases.  In extreme cases, you
> can simply not fontify if there's no reasonable way of fontifying even
> approximately.  (I don't believe such extreme measures should be
> necessary, but they are a possibility.)
>
> IOW, we fully expect that in some cases and with some features, there
> will be partial loss of functionality in these cases.  We consider
> that still much better than a locked-up Emacs that can only be killed.
>
> > Security against what, for fuck's sake?
>
> Language!
>

[-- Attachment #2: Type: text/html, Size: 2683 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Fri, 16 Sep 2022 12:08:49 +0200
> Cc: Gregory Heytings <gregory@heytings.org>, 57804@debbugs.gnu.org, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> I give up. Do as you like.

That's unfortunate, because we hope to see maintainers of 3rd-party
modes sympathetic to this effort of making Emacs capable of coping
with very long lines, and cooperate with us in adapting their modes to
the core mechanisms that deal with these situations.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Thu, 15 Sep 2022 22:40:24 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, 57804@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
> 
> It is impossible to correctly fontify log buffers if you are
> restricted to semi-arbitrary parts of it: you need to at least see
> where the current entry starts, even if it for whatever reason
> includes a line with 100 K characters.

In buffers with very long lines, fontifications are supposed and
requested to simplify their act so as not to need access to arbitrary
portions of the buffer outside the restriction.  If such
simplification is for some reason impossible or impractical, then the
code called from fontification-functions should at least be amended
not to produce infloops when its calls to 'widen' don't widen.

This is all part of solving a long-standing problem in Emacs with
becoming completely unresponsive when displaying buffers with very
long lines.  It is understood and accepted that in some cases this
will come at a price of less accurate fontifications or minor glitches
in other features.

> > It isn't for me personally, no.  It is important for Emacs users.
> 
> How about letting Emacs users decide for themselves?

The users already can decide for themselves: they can set the
long-line-threshold variable to larger values or even to nil.

> Let Logview
> behave against your recommendations. And then, when the users discover
> how crappy and unresponsive it is, as a result, compared to everything
> else, they will just uninstall the package and use something better.

We expect package developers to cooperate with our effort of making
Emacs responsive and reasonably functional in buffers with very long
lines.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Date: Mon, 28 Nov 2022 17:35:35 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: pogonyshev@gmail.com, dgutov@yandex.ru, 57804@debbugs.gnu.org, 
>     larsi@gnus.org
> 
> By the way, can you tell me what "in due time" means?  It means "before 
> Emacs 29 is released", which will happen only in a couple of months, 
> right?

If I need to do this, I planned on doing it in the next week or two.  I
don't think we should wait much longer, because these instructions need to
be subject to the release-cycle testing together with Emacs itself.

> The background of that question is that I'd like to explore a few 
> things in that area before writing the NEWS entry.

If you have considerable doubt what to write, it is fine to wait a bit
longer than a week or two.  But please also consider writing soon what we
know now and later augmenting it given the experience and the feedback.  I
would definitely like to see this out in the open as soon as we practically
can.

TIA

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> Depending on the mode being activated before Emacs decides to enable the 
>> optimization (e.g. because one of the first lines is very long, I don't 
>> know how exactly this is determined) seems very shaky.
>
> Indeed.  As I told you the proper fix is not to disable these 
> optimizations, but to adapt the code to handle locked narrowing, by 
> explicitly unlocking the locked narrowing when, and only when, it needs to 
> access a larger portion of the buffer.

I would like to point out that some fontification code may use functions
not specifically designed for the fontification. Moreover, the functions
used by fontification may depend on user-defined hooks that can potentially
contain unsuspecting (widen) calls.

While I understand why locked narrowing is needed in buffers with large
files, I would find it helpful if Emacs could throw some kind of warning
when widen is called from inside locked narrowing. It will, at least,
help with adapting the existing fontification code in the new Emacs
versions.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Date: Thu, 15 Sep 2022 20:18:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 57804@debbugs.gnu.org, 
>     Lars Ingebrigtsen <larsi@gnus.org>
> 
> > I see that manually evaluating `(setq-local long-line-threshold nil)' in 
> > a buffer where the optimization is already in effect (i.e. where 
> > `(long-line-optimizations-p)' evaluates to t) doesn't disable the 
> > optimization. Do you have a solution for that?
> 
> No, and that will not be supported.

I don't necessarily agree.  We didn't see the need for this yet, but
if we do find significant use cases where that could be important, we
will consider adding something, like perhaps re-evaluation of the
optimization flag under some circumstances.  For example, changing the
value of long-line-threshold could be one such trigger.

But for now we don't see a need for that, and prefer that Lisp
programs that run via fontification-functions will be adapted to this
new protocol.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]


>> already told you thrice that a simple way to fix your problem is to set 
>> long-line-threshold to a larger value (or to nil).
> 
> Thanks, I added  `(setf long-line-threshold nil)' to Emacs startup 
> configuration file a couple of days before. But unless I'm missing 
> something, this is not a fix, only a workaround.
>

Why did you immediately set it to nil instead of trying to making it 
sufficiently large?  You said that some lines in the Logview buffer are 
long, but I bet they are never longer than, say, 100000 characters.

As I told you, the general fix will be to adapt Logview to handle locked 
narrowing, by explicitly unlocking the locked narrowing when it needs to 
access a larger portion of the buffer.  You were somewhat reluctant at 
that idea.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3128 bytes --]

> Why did you immediately set it to nil instead of trying to making it
> sufficiently large?  You said that some lines in the Logview buffer
> are long, but I bet they are never longer than, say, 100000
> characters.

How do I decide what is sufficiently large? With this optimization
Logview can fall into that infloop in fontification code, at which
point Emacs becomes completely frozen, with the only way out to
restart it. As have been determined in this thread, this is not a
bug. I still kinda want to avoid this non-bug, because it is a bit
disruptive to my work.

With long lines I can suffer slow and not-quite-responsive (but still
not hung to death) Emacs, which is much more bearable. Usually that
only happens when I open or grep something like a minified JS file by
mistake anyway.

For you this long-line optimization is probably very important, maybe
you often work with files that trigger this and make using Emacs a
pain without it. But for me it's a minor feature that I have barely
noticed, but that incidentally completely broke my normal workflow by
making Emacs seemingly randomly freeze (until I found time to debug
it, which was not easy as Emacs would not respond to anything).

> As I told you, the general fix will be to adapt Logview to handle locked
> narrowing, by explicitly unlocking the locked narrowing when it needs to
> access a larger portion of the buffer.  You were somewhat reluctant at
> that idea.

Maybe I misunderstood you. If I have to add a one-time adjustment to
the code to the effect of "unlock the narrowing it inside this block",
then it is fine for me. Now that I reread:

    > the "fully cooked" narrowing will not magically solve that
    > problem, though.  Logview will have to be adapted to deal with
    > the possibility of a locked narrowing.

I don't see implications that unlocking would be impossible. If I only
would have to use sth. like sketched (I don't insist on it looking
like this):

    (let ((disable-locked-narrowing-yes-i-know-this-is-bad-but-still t))
      (widen)
      ...
      )

once the branch is finished and merged (and that it would work for all
kinds of tags or whatever at once), then sorry for misunderstanding
and starting a heated discussion for nothing.

Paul

On Thu, 15 Sept 2022 at 21:44, Gregory Heytings <gregory@heytings.org>
wrote:

>
> >> already told you thrice that a simple way to fix your problem is to set
> >> long-line-threshold to a larger value (or to nil).
> >
> > Thanks, I added  `(setf long-line-threshold nil)' to Emacs startup
> > configuration file a couple of days before. But unless I'm missing
> > something, this is not a fix, only a workaround.
> >
>
> Why did you immediately set it to nil instead of trying to making it
> sufficiently large?  You said that some lines in the Logview buffer are
> long, but I bet they are never longer than, say, 100000 characters.
>
> As I told you, the general fix will be to adapt Logview to handle locked
> narrowing, by explicitly unlocking the locked narrowing when it needs to
> access a larger portion of the buffer.  You were somewhat reluctant at
> that idea.

[-- Attachment #2: Type: text/html, Size: 3775 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

> 
> How do I decide what is sufficiently large? With this optimization 
> Logview can fall into that infloop in fontification code, at which point 
> Emacs becomes completely frozen, with the only way out to restart it.
>

You just described what would happen without this optimization with a 
sufficiently long line.

>
> With long lines I can suffer slow and not-quite-responsive (but still 
> not hung to death) Emacs, which is much more bearable.
>

Obviously you did not have to deal with long enough lines.

> 
> For you this long-line optimization is probably very important, maybe 
> you often work with files that trigger this and make using Emacs a pain 
> without it.
>

It isn't for me personally, no.  It is important for Emacs users.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 57804@debbugs.gnu.org
> Date: Thu, 15 Sep 2022 20:26:54 +0000
> From: Gregory Heytings <gregory@heytings.org>
> 
> > For you this long-line optimization is probably very important, maybe 
> > you often work with files that trigger this and make using Emacs a pain 
> > without it.
> 
> It isn't for me personally, no.  It is important for Emacs users.

Indeed, the complaints about Emacs behavior with very long lines are
abundantly seen on the Internet and in our bug database.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[dick]

GH> the general fix will be to adapt Logview to handle locked narrowing

I wonder what kind of person is so idle, recalcitrant, and obnoxious
that he has all day to work on a private fork of emacs, refuses to
upstream his changes, all the while insulting the community.  Ah, that
would be I.

I also wonder however, what kind of person becomes so triggered by a
YouTube video about long lines, dedicates innumerable hours to
implementing and shepherding a fix, amplifies the exceptions to a
competing heuristic, but blithely dismisses exceptions to his own
heuristic ("I bet we'll never get a bug report about that," "Don't
worry, that won't happen"), who also dismisses hard performance metrics
by claiming unrealistic usage patterns, who then when presented with a
real-world failure suggests the aggrieved module ought to
accommodate his long lines fix, and goes so far as to characterize that
accommodation as "a general fix."

Generally, I wonder what it is about emacs that brings out not just the
worst developers, but also the worst people.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gregory Heytings]

>
> who then when presented with a real-world failure suggests the aggrieved 
> module ought to accommodate his long lines fix
>

It was clear from the very beginning that the long line fixes would break 
some code in unusual circumstances, and that such code would have to be 
adapted.  That is unavoidable, except of course with frozen programs such 
as the TeX engine.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> Cc: 57804@debbugs.gnu.org, Paul Pogonyshev <pogonyshev@gmail.com>
> From: dick <dick.r.chiang@gmail.com>
> Date: Thu, 15 Sep 2022 18:20:49 -0400
> 
> Generally, I wonder what it is about emacs that brings out not just the
> worst developers, but also the worst people.

This message has exactly zero content related to this bug report, it
is all about offending other people.

You are dangerously close to being banned from this forum as well.
Consider this your last warning.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Gerd Möllmann]

dick <dick.r.chiang@gmail.com> writes:

> Generally, I wonder what it is about emacs that brings out not just the
> worst developers, but also the worst people.

I can only talk about me, but for me it is that I'm a masochist.  I
really like to be vistimized, that makes things so much easier.  But
that's just me, of course :-).

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

tags 57804 wontfix
close 57804
thanks

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Wed, 14 Sep 2022 17:05:54 +0200
> 
> To reproduce, save the attachment as `font-lock-hangs.el' and execute:
> 
>     $ emacs -Q -l font-lock-hangs.el
> 
> C-g doesn't help anymore. The only thing you can do is to kill and start Emacs anew.
> 
> Git commit fd1ee05977.
> 
> To quote a discussion from another bug, the reply is from Eli Zaretski:
> 
> > > By the way, it would really be nice if Emacs could do something about hangs irrespective of what
> causes
> > > that. Even if Elisp code is buggy, Emacs itself should never allow it to fall into an infinite loop and stop
> > > responding to C-g, leaving full restart as the only way out.
> > 
> > I think that's impossible in general, unless we restrict what Lisp
> > programs can do.  Every programming language can be used to write a
> > buggy program.
> > 
> > However, it should be possible to prevent some cases of such
> > problematic behavior, certainly so when the infloop is caused by our
> > bug.  But for that we need to know the details of the specific case in
> > order to investigate.

This case is exactly one of those which I think we shouldn't try to
fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
that".  There's no reason for any useful Lisp program to have an
infloop like this:

>   (while t
>     "whoopsie")

So I'm closing this bug as "wontfix".

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

> This case is exactly one of those which I think we shouldn't try to
> fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
> that".  There's no reason for any useful Lisp program to have an
> infloop like this:
>
> >   (while t
> >     "whoopsie")

It is no wonder that Emacs is in such a poor state with 2 out 3
responding developers failing to make _one_ mental step from "while t"
to "this could be a 100-line loop that accidentally falls into
infinite recursion".  And even because of incompatible change in Emacs
itself.

Just wish I didn't get accustomed to this pile of crap 20 years ago
and just used a normal IDE like every smart person.

Paul

[-- Attachment #2: Type: text/html, Size: 847 bytes --]

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Wed, 14 Sep 2022 19:25:09 +0200
> Cc: 57804@debbugs.gnu.org
> 
> > >   (while t
> > >     "whoopsie")
> 
> It is no wonder that Emacs is in such a poor state with 2 out 3
> responding developers failing to make _one_ mental step from "while t"
> to "this could be a 100-line loop that accidentally falls into
> infinite recursion".  And even because of incompatible change in Emacs
> itself.

Emacs gives Lisp programmers enough rope to hang themselves, and
expects them to be wise enough not to do so.  Our own sources should
not use so much rope, so if there are such loops in our code, please
point them out, or show a recipe that uncovers them, and we will
certainly fix them.

But preventing programmers from writing infinite loops for the benefit
of writing infinite loops, and in font-lock functions on top of that,
is not my idea of good investment of our resources.  It is much easier
to fix such infinite loops so they aren't there in the first place.

> Just wish I didn't get accustomed to this pile of crap 20 years ago
> and just used a normal IDE like every smart person.

You are welcome.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> This case is exactly one of those which I think we shouldn't try to
> fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
> that".  There's no reason for any useful Lisp program to have an
> infloop like this:
>
>>   (while t
>>     "whoopsie")

You've never accidentally put an infloop into a font locking function?
Then you've been very lucky.

We should definitely fix this is we can.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Pogonyshev <pogonyshev@gmail.com>,  57804@debbugs.gnu.org
> Date: Wed, 14 Sep 2022 19:38:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This case is exactly one of those which I think we shouldn't try to
> > fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
> > that".  There's no reason for any useful Lisp program to have an
> > infloop like this:
> >
> >>   (while t
> >>     "whoopsie")
> 
> You've never accidentally put an infloop into a font locking function?

While testing my code? sure.  But after testing, no, the loops are
generally gone.

> We should definitely fix this is we can.

I disagree.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> While testing my code? sure.  But after testing, no, the loops are
> generally gone.

Of course.  This is a real problem while developing font locking methods
(and usually not afterwards).

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: pogonyshev@gmail.com,  57804@debbugs.gnu.org
> Date: Wed, 14 Sep 2022 19:46:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > While testing my code? sure.  But after testing, no, the loops are
> > generally gone.
> 
> Of course.  This is a real problem while developing font locking methods
> (and usually not afterwards).

While testing, I usually run Emacs under a debugger, where I always
can interrupt any loop, including in font-lock.

I think it is also possible without a debugger -- if you bind
'sigusr2' to some useful function.

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> This case is exactly one of those which I think we shouldn't try to
>> fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
>> that".  There's no reason for any useful Lisp program to have an
>> infloop like this:
>>
>>>   (while t
>>>     "whoopsie")
>
> You've never accidentally put an infloop into a font locking function?
> Then you've been very lucky.
>
> We should definitely fix this is we can.

Why isn't jit-lock-debug-mode an option?

bug#57804: An infinite loop in a `fontify-region' function causes Emacs to hang indefinitely

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  57804@debbugs.gnu.org,  Paul Pogonyshev
>  <pogonyshev@gmail.com>
> Date: Thu, 15 Sep 2022 13:20:51 +0800
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> This case is exactly one of those which I think we shouldn't try to
> >> fix, because it's the case of "buggy Lisp program", a.k.a. "don't do
> >> that".  There's no reason for any useful Lisp program to have an
> >> infloop like this:
> >>
> >>>   (while t
> >>>     "whoopsie")
> >
> > You've never accidentally put an infloop into a font locking function?
> > Then you've been very lucky.
> >
> > We should definitely fix this is we can.
> 
> Why isn't jit-lock-debug-mode an option?

It definitely is, for the particular use case of debugging font-lock.

So is running font-lock-fontify-buffer by hand.

So is attaching a debugger when such an infloop is bumped into
accidentally without expecting it.

There's also the recently-added backtrace-on-redisplay-error feature,
which could help with diagnosing such problems, perhaps combined with
debug-on-entry or somesuch.

IOW, I think we have already several useful tools available for the
particular problem of debugging loops in font-lock, if we want to
interpret this bug report in that narrow interpretation.