locked narrowing in ELisp

locked narrowing in ELisp

[Stefan Monnier]

The new "fast long-lines" code uses a "locked narrowing" to try and
avoid being defeated by naive font-lock (and other) code.

This is a good heuristic to try and make sure Emacs stays responsive
even in the presence of naive packages not designed for such long lines.

Beside the need to override this in those rare packages where we can
provide better behavior with careful coding, the new code also lacks
a way to install such a locked narrowing from ELisp, even though it
seems only natural to use similar heuristics from ELisp code (e.g. when
jit-lock is run from a timer).

But as pointed out by Lars, this is related to the old discussion about
"classifying" narrowing, so as to distinguish narrowings installed by
the user via `C-x n n` from those installed by packages like MMM-mode or
Info-mode (or nowadays those installed by the redisplay).
These distinctions are important because various code may need to look
"outside" of those narrowings in some circumstances, but which "outside"
to use depends on the particular situation.

We've discussed it in the past and I still don't know what would be
a good API, but one particular class of `widen` which is known to be
used within narrowings that should *not* be overridden is those `widen`
calls used in "parsers" that try to understand the context of
an operation.

Those `widen` will sometimes be used in circumstances where they may
actually want to *narrow* rather than widen (e.g. the narrowing might be
needed to make sure the parser only looks at the part of the buffer that
it can understand (e.g. in MMM-mode), or it might be needed in order to
avoid the risk of spending an eternity parsing GBs of text (at the cost
of misparsing in too-large buffers)).

Last time we discussed these issues, we ended up deciding that
indentation (like font-lock before it) should "eagerly widen" (i.e. undo
any user-setup narrowing) before calling the mode-specific code.
This way, the indentation and font-lock code of a major mode should
simply never need to call `widen` and things like the MMM-mode (or the
long-lines code in redisplay) can just setup the narrowing before
calling that major mode code without any need to "label" the narrowing.

This was an elegant solution, but it seems to be too limited, because
the parsing code that want(ed) to `widen` is typically also called from
other parts of the major mode than the indentation or font-lock code.

In the past, we played with having a new widening function (there was
a `prog-widen` briefly in `prog-mode.el`).  Maybe we should
(re)introduce such a thing (or maybe an optional arg to `widen` which
would indicate what "kind" of widening we want to do), coupled with some
kind of `widen-function` which MMM-mode and the redisplay could
setup to filter/tweak those widening.

Within an MMM-mode or "redisplay locked narrowing", we *generally* want
`widen` not to widen past the limits imposed by MMM or by the redisplay,
but there will be cases where we do want to allow it (e.g. in
`nlinum-mode` the widening will usually want to be allowed to override
those narrowings in order to compute the right line number), so we
clearly not all calls to `widen` are the same.


        Stefan

Re: locked narrowing in ELisp

[Dmitry Gutov]

Hi Stefan,

On 16.08.2022 23:18, Stefan Monnier wrote:
> Last time we discussed these issues, we ended up deciding that
> indentation (like font-lock before it) should "eagerly widen" (i.e. undo
> any user-setup narrowing) before calling the mode-specific code.
> This way, the indentation and font-lock code of a major mode should
> simply never need to call `widen` and things like the MMM-mode (or the
> long-lines code in redisplay) can just setup the narrowing before
> calling that major mode code without any need to "label" the narrowing.
> 
> This was an elegant solution, but it seems to be too limited, because
> the parsing code that want(ed) to `widen` is typically also called from
> other parts of the major mode than the indentation or font-lock code.

Have you given any thought to the "soft widen" alternative I voiced 
recently?

https://lists.gnu.org/archive/html/emacs-devel/2022-08/msg00291.html

If the user-level narrow/widen commands didn't use _actual_ narrowing, 
but display engine tricks or whatever (example: 
https://github.com/Malabarba/fancy-narrow/), "other parts of the major 
mode" wouldn't have to do the usual (save-restriction (widen) ...) 
dance, which a lot of code is littered with.

Then there would be no need for "hard" or "locked" narrowing to restrict 
those calls to 'widen', because there wouldn't be any. The 
'narrow-to-region' and 'widen' would be restricted to lower-level uses, 
like mmm-mode, Info-mode, and the display engine long-line wrangling magic.

The migration path seems difficult, but the result might be worth it.

> In the past, we played with having a new widening function (there was
> a `prog-widen` briefly in `prog-mode.el`).  Maybe we should
> (re)introduce such a thing (or maybe an optional arg to `widen` which
> would indicate what "kind" of widening we want to do), coupled with some
> kind of `widen-function` which MMM-mode and the redisplay could
> setup to filter/tweak those widening.

'widen-function' can work too. IIRC various 'multi-mode' packages have 
tried to advice 'widen', without much of a reliable success. Hopefully 
the extra indirection doesn't cause too much overhead. But it's a more 
complicated solution either way.

Re: locked narrowing in ELisp

[Stefan Monnier]

> Have you given any thought to the "soft widen" alternative
> I voiced recently?
>
> https://lists.gnu.org/archive/html/emacs-devel/2022-08/msg00291.html
>
> If the user-level narrow/widen commands didn't use _actual_ narrowing, but
> display engine tricks or whatever (example:
> https://github.com/Malabarba/fancy-narrow/), "other parts of the major mode"
> wouldn't have to do the usual (save-restriction (widen) ...) dance, which
> a lot of code is littered with.
>
> Then there would be no need for "hard" or "locked" narrowing to restrict
> those calls to 'widen', because there wouldn't be any. The
> 'narrow-to-region' and 'widen' would be restricted to lower-level uses, like
> mmm-mode, Info-mode, and the display engine long-line wrangling magic.
>
> The migration path seems difficult, but the result might be worth it.

Such a display-only narrowing might be a good alternative for many uses
of narrowing, but narrowing is also used quite commonly (either by the
end-user or in ELisp code) in order to restrict the effect of an
operation (like search&replace) to a particular region.

> 'widen-function' can work too. IIRC various 'multi-mode' packages have tried
> to advice 'widen', without much of a reliable success.

`widen` is one of those functions that has its own bytecode (I'm far
from convinced that it's used frequently enough to justify it, but that's
another issue), so indeed advising it is unlikely to give good results.


        Stefan

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 17.08.2022 03:55, Stefan Monnier wrote:
>> Have you given any thought to the "soft widen" alternative
>> I voiced recently?
>>
>> https://lists.gnu.org/archive/html/emacs-devel/2022-08/msg00291.html
>>
>> If the user-level narrow/widen commands didn't use _actual_ narrowing, but
>> display engine tricks or whatever (example:
>> https://github.com/Malabarba/fancy-narrow/), "other parts of the major mode"
>> wouldn't have to do the usual (save-restriction (widen) ...) dance, which
>> a lot of code is littered with.
>>
>> Then there would be no need for "hard" or "locked" narrowing to restrict
>> those calls to 'widen', because there wouldn't be any. The
>> 'narrow-to-region' and 'widen' would be restricted to lower-level uses, like
>> mmm-mode, Info-mode, and the display engine long-line wrangling magic.
>>
>> The migration path seems difficult, but the result might be worth it.
> 
> Such a display-only narrowing might be a good alternative for many uses
> of narrowing, but narrowing is also used quite commonly (either by the
> end-user or in ELisp code) in order to restrict the effect of an
> operation (like search&replace) to a particular region.

Certain operations could look up the "soft narrowing" bounds and act 
accordingly, if it's implemented in the core and appropriately documented.

To my understanding, there are more commands and facilities that want to 
ignore user-level narrowing, rather than the ones that want to obey it. 
Also a lot of "undecided" ones, waiting for someone to report that they 
should, in fact, ignore narrowing.

Re: locked narrowing in ELisp

[Stefan Monnier]

>> Such a display-only narrowing might be a good alternative for many uses
>> of narrowing, but narrowing is also used quite commonly (either by the
>> end-user or in ELisp code) in order to restrict the effect of an
>> operation (like search&replace) to a particular region.
>
> Certain operations could look up the "soft narrowing" bounds and act
> accordingly, if it's implemented in the core and appropriately documented.

Could be, but unless we go through the whole C code looking for checks
of BEGV/ZV and updating the code to also check the "soft bounds" this
won't work reliably enough to replace existing uses of narrowing.

> To my understanding, there are more commands and facilities that want to
> ignore user-level narrowing, rather than the ones that want to obey it. Also
> a lot of "undecided" ones, waiting for someone to report that they should,
> in fact, ignore narrowing.

But for backward compatibility reasons, we can't break the cases that
need the bounds to be hard, even if it could fix some cases.


        Stefan

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 17.08.2022 16:03, Stefan Monnier wrote:
>>> Such a display-only narrowing might be a good alternative for many uses
>>> of narrowing, but narrowing is also used quite commonly (either by the
>>> end-user or in ELisp code) in order to restrict the effect of an
>>> operation (like search&replace) to a particular region.
>>
>> Certain operations could look up the "soft narrowing" bounds and act
>> accordingly, if it's implemented in the core and appropriately documented.
> 
> Could be, but unless we go through the whole C code looking for checks
> of BEGV/ZV and updating the code to also check the "soft bounds" this
> won't work reliably enough to replace existing uses of narrowing.

The point is to avoid doing that. If we can.

For instance, use two overlays in the current buffer with `invisible' 
property rather than have the display engine refer to the new kind of 
narrowing bounds.

>> To my understanding, there are more commands and facilities that want to
>> ignore user-level narrowing, rather than the ones that want to obey it. Also
>> a lot of "undecided" ones, waiting for someone to report that they should,
>> in fact, ignore narrowing.
> 
> But for backward compatibility reasons, we can't break the cases that
> need the bounds to be hard, even if it could fix some cases.

The "regular" narrowing will play the role of the hard one because there 
will be little need to protect against 'widen' calls all around.

I agree the migration path is non-obvious, though.

Re: locked narrowing in ELisp

[Eli Zaretskii]

> Date: Wed, 17 Aug 2022 16:40:01 +0300
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Could be, but unless we go through the whole C code looking for checks
> > of BEGV/ZV and updating the code to also check the "soft bounds" this
> > won't work reliably enough to replace existing uses of narrowing.
> 
> The point is to avoid doing that. If we can.
> 
> For instance, use two overlays in the current buffer with `invisible' 
> property rather than have the display engine refer to the new kind of 
> narrowing bounds.

That's a time bomb waiting to go off, because invisible text is
handled at a relatively high level in the display engine, and
otherwise the invisible property is largely ignored in Emacs.

Moreover, it will make redisplay slower.  Skipping invisible text is
faster than iterating through it, but it still takes time, whereas not
going beyond BEGV..ZV is instantaneous.

And finally, I don't think I see the benefit of this, even if it'd
work: you want to get rid of (save-restriction (widen)), but you are
willing to have to replace that with tests of overlays and invisible
text all over the place?

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 17.08.2022 16:55, Eli Zaretskii wrote:
>> Date: Wed, 17 Aug 2022 16:40:01 +0300
>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> Could be, but unless we go through the whole C code looking for checks
>>> of BEGV/ZV and updating the code to also check the "soft bounds" this
>>> won't work reliably enough to replace existing uses of narrowing.
>>
>> The point is to avoid doing that. If we can.
>>
>> For instance, use two overlays in the current buffer with `invisible'
>> property rather than have the display engine refer to the new kind of
>> narrowing bounds.
> 
> That's a time bomb waiting to go off, because invisible text is
> handled at a relatively high level in the display engine, and
> otherwise the invisible property is largely ignored in Emacs.

User-level features should be implementable in terms of primitives 
allowed in Lisp userland.

> Moreover, it will make redisplay slower.  Skipping invisible text is
> faster than iterating through it, but it still takes time, whereas not
> going beyond BEGV..ZV is instantaneous.

Org, as one example, uses invisible text all the time. Other feature too.

> And finally, I don't think I see the benefit of this, even if it'd
> work: you want to get rid of (save-restriction (widen)), but you are
> willing to have to replace that with tests of overlays and invisible
> text all over the place?

No, I don't think the addition of "tests ... all over the place" will be 
needed. The display engine handles the 'invisible' property already.

A number of features/commands will indeed need to know the bounds of the 
user-level narrowing (and we'll have a buffer-local variable for that), 
but that's probably it.

Re: locked narrowing in ELisp

[Eli Zaretskii]

> Date: Wed, 17 Aug 2022 17:03:46 +0300
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 17.08.2022 16:55, Eli Zaretskii wrote:
> >> For instance, use two overlays in the current buffer with `invisible'
> >> property rather than have the display engine refer to the new kind of
> >> narrowing bounds.
> > 
> > That's a time bomb waiting to go off, because invisible text is
> > handled at a relatively high level in the display engine, and
> > otherwise the invisible property is largely ignored in Emacs.
> 
> User-level features should be implementable in terms of primitives 
> allowed in Lisp userland.

I don't see how this is relevant to the concern I raised.  I was
talking about the effects on the display engine.  It doesn't only
display, it also looks at buffer text for various purposes.

> > Moreover, it will make redisplay slower.  Skipping invisible text is
> > faster than iterating through it, but it still takes time, whereas not
> > going beyond BEGV..ZV is instantaneous.
> 
> Org, as one example, uses invisible text all the time. Other feature too.

And Org is indeed relatively slow when you move through a buffer which
has large parts of it hidden by invisible properties.

> > And finally, I don't think I see the benefit of this, even if it'd
> > work: you want to get rid of (save-restriction (widen)), but you are
> > willing to have to replace that with tests of overlays and invisible
> > text all over the place?
> 
> No, I don't think the addition of "tests ... all over the place" will be 
> needed. The display engine handles the 'invisible' property already.
> 
> A number of features/commands will indeed need to know the bounds of the 
> user-level narrowing (and we'll have a buffer-local variable for that), 
> but that's probably it.

I don't think you realize how widespread is use of low-level
primitives and functions in user-level commands.

What you suggest is not a clean design, because it is based on
inaccurate mental model of Emacs internals.  It cannot work reliably,
to the best of my knowledge.

Re: locked narrowing in ELisp

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]

On 17.08.2022 17:20, Eli Zaretskii wrote:
>> Date: Wed, 17 Aug 2022 17:03:46 +0300
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 17.08.2022 16:55, Eli Zaretskii wrote:
>>>> For instance, use two overlays in the current buffer with `invisible'
>>>> property rather than have the display engine refer to the new kind of
>>>> narrowing bounds.
>>>
>>> That's a time bomb waiting to go off, because invisible text is
>>> handled at a relatively high level in the display engine, and
>>> otherwise the invisible property is largely ignored in Emacs.
>>
>> User-level features should be implementable in terms of primitives
>> allowed in Lisp userland.
> 
> I don't see how this is relevant to the concern I raised.  I was
> talking about the effects on the display engine.  It doesn't only
> display, it also looks at buffer text for various purposes.

I guess I didn't understand your concern, sorry. Invisible is handled 
somewhere on the high level, OK. I did not mistake it for 'intangible'.

>>> Moreover, it will make redisplay slower.  Skipping invisible text is
>>> faster than iterating through it, but it still takes time, whereas not
>>> going beyond BEGV..ZV is instantaneous.
>>
>> Org, as one example, uses invisible text all the time. Other feature too.
> 
> And Org is indeed relatively slow when you move through a buffer which
> has large parts of it hidden by invisible properties.

Org uses different properties, a lot. Not just 'invisible'. So I'd 
rather people test the performance of this first before dismissing.

My limited testing didn't show any particular slowdown.

>>> And finally, I don't think I see the benefit of this, even if it'd
>>> work: you want to get rid of (save-restriction (widen)), but you are
>>> willing to have to replace that with tests of overlays and invisible
>>> text all over the place?
>>
>> No, I don't think the addition of "tests ... all over the place" will be
>> needed. The display engine handles the 'invisible' property already.
>>
>> A number of features/commands will indeed need to know the bounds of the
>> user-level narrowing (and we'll have a buffer-local variable for that),
>> but that's probably it.
> 
> I don't think you realize how widespread is use of low-level
> primitives and functions in user-level commands.

Commands that do want to obey narrowing, can take the soft-narrowing 
bounds and apply the narrowing internally.

> What you suggest is not a clean design, because it is based on
> inaccurate mental model of Emacs internals.  It cannot work reliably,
> to the best of my knowledge.

I'm likely missing a lot of things, since I don't usually use this 
feature interactively. All I know is, about once or twice a year, people 
come and ask to make a certain command ignore narrowing. And nobody 
every comes and asks to revert those changes.

Could someone give a few problem scenarios with this patch? Preferably 
ones that should be hard to fix. Adapting isearch took about 2 lines.

[-- Attachment #2: soft-narrow-and-widen.diff --]
[-- Type: text/x-patch, Size: 3613 bytes --]

diff --git a/lisp/bindings.el b/lisp/bindings.el
index 2e32128274..109a1a411f 100644
--- a/lisp/bindings.el
+++ b/lisp/bindings.el
@@ -968,8 +968,8 @@ left-word
 
 (defvar-keymap narrow-map
   :doc "Keymap for narrowing commands."
-  "n" #'narrow-to-region
-  "w" #'widen
+  "n" #'soft-narrow-to-region
+  "w" #'soft-widen
   "g" #'goto-line-relative)
 (define-key ctl-x-map "n" narrow-map)
 
diff --git a/lisp/isearch.el b/lisp/isearch.el
index 31fcf01949..285c92f850 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -1321,9 +1321,10 @@ isearch-mode
   ;; Pushing the initial state used to be before running isearch-mode-hook,
   ;; but a hook might set `isearch-push-state-function' used in
   ;; `isearch-push-state' to save mode-specific initial state.  (Bug#4994)
-  (isearch-push-state)
+  (with-soft-narrow
+   (isearch-push-state)
 
-  (isearch-update)
+   (isearch-update))
 
   (add-hook 'pre-command-hook 'isearch-pre-command-hook)
   (add-hook 'post-command-hook 'isearch-post-command-hook)
@@ -1358,7 +1359,7 @@ isearch-update
 
   (if (and (null unread-command-events)
 	   (null executing-kbd-macro))
-      (progn
+      (with-soft-narrow
         (if (not (input-pending-p))
           (funcall (or isearch-message-function #'isearch-message)))
         (if (and isearch-slow-terminal-mode
diff --git a/lisp/simple.el b/lisp/simple.el
index 1e6e5e11e0..78d0804ee8 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -1647,6 +1647,14 @@ goto-line
       (widen))
     (goto-char pos)))
 
+(defmacro with-soft-narrow (&rest body)
+  (declare (indent 1) (debug t))
+  `(let ((bounds (soft-narrow-bounds)))
+     (save-restriction
+       (when bounds
+         (narrow-to-region (car bounds) (cdr bounds)))
+       ,@body)))
+
 (defun goto-line-relative (line &optional buffer)
   "Go to LINE, counting from line at (point-min).
 The line number is relative to the accessible portion of the narrowed
@@ -1654,7 +1662,8 @@ goto-line-relative
   (declare (interactive-only forward-line))
   (interactive (goto-line-read-args t))
   (with-suppressed-warnings ((interactive-only goto-line))
-    (goto-line line buffer t)))
+    (with-soft-narrow
+      (goto-line line buffer t))))
 
 (defun count-words-region (start end &optional arg)
   "Count the number of words in the region.
@@ -10707,6 +10716,40 @@ lax-plist-put
   "Change value in PLIST of PROP to VAL, comparing with `equal'."
   (declare (obsolete plist-put "29.1"))
   (plist-put plist prop val #'equal))
+
+(defvar soft-narrow--overlays nil)
+
+(defun soft-widen ()
+  (interactive)
+  (when soft-narrow--overlays
+    (with-soft-narrow
+      ;; If cursor is after the cdr ovl or before car ovl,
+      ;; move it inside.
+      (delete-overlay (car soft-narrow--overlays))
+      (delete-overlay (cdr soft-narrow--overlays)))
+    (setq soft-narrow--overlays nil)))
+
+(defun soft-narrow-to-region (beg end)
+  (interactive "r")
+  (soft-widen)
+  (let ((o1 (make-overlay (point-min) beg))
+        (o2 (make-overlay end (point-max) nil t t)))
+    (overlay-put o1 'invisible t)
+    (overlay-put o1 'read-only t)
+    (overlay-put o1 'modification-hooks '(soft-narrow--modif))
+    (overlay-put o2 'invisible t)
+    (overlay-put o2 'read-only t)
+    (overlay-put o2 'modification-hooks '(soft-narrow--modif))
+    (setq soft-narrow--overlays (cons o1 o2))))
+
+(defun soft-narrow--modif (&rest _)
+  (user-error "Not allowed"))
+
+(defun soft-narrow-bounds ()
+  (when soft-narrow--overlays
+    (cons (overlay-end (car soft-narrow--overlays))
+          (overlay-start (cdr soft-narrow--overlays)))))
+
 \f
 
 (provide 'simple)

Re: locked narrowing in ELisp

[Stefan Monnier]

> Could someone give a few problem scenarios with this patch? Preferably ones
> that should be hard to fix. Adapting isearch took about 2 lines.

I don't think there'll be much that's hard to fix, no.  The problem is
rather that there'll be a lot to fix, so it'll basically be half-broken
for a long time.

E.g. `C-x h M-x write-region RET` will end up writing something else
than what I'd expect.


        Stefan

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 18.08.2022 04:58, Stefan Monnier wrote:
>> Could someone give a few problem scenarios with this patch? Preferably ones
>> that should be hard to fix. Adapting isearch took about 2 lines.
> 
> I don't think there'll be much that's hard to fix, no.  The problem is
> rather that there'll be a lot to fix, so it'll basically be half-broken
> for a long time.

That is a problem indeed. So at the very least the corresponding feature 
branch would need attention from somebody (or some people) who use 
narrowing a lot.

> E.g. `C-x h M-x write-region RET` will end up writing something else
> than what I'd expect.

I see what you mean. Some region-related commands might need direct 
support indeed (region-beginning and region-end changes their return 
values once narrowing it applied).

soft-narrow-to-region might also alter the region, if we decide to 
support its programmatic application.

Re: locked narrowing in ELisp

[Eli Zaretskii]

> Date: Thu, 18 Aug 2022 02:13:30 +0300
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 17.08.2022 17:20, Eli Zaretskii wrote:
> >> Date: Wed, 17 Aug 2022 17:03:46 +0300
> >> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >> On 17.08.2022 16:55, Eli Zaretskii wrote:
> >>>> For instance, use two overlays in the current buffer with `invisible'
> >>>> property rather than have the display engine refer to the new kind of
> >>>> narrowing bounds.
> >>>
> >>> That's a time bomb waiting to go off, because invisible text is
> >>> handled at a relatively high level in the display engine, and
> >>> otherwise the invisible property is largely ignored in Emacs.
> >>
> >> User-level features should be implementable in terms of primitives
> >> allowed in Lisp userland.
> > 
> > I don't see how this is relevant to the concern I raised.  I was
> > talking about the effects on the display engine.  It doesn't only
> > display, it also looks at buffer text for various purposes.
> 
> I guess I didn't understand your concern, sorry. Invisible is handled 
> somewhere on the high level, OK. I did not mistake it for 'intangible'.

The concern is that some parts of the display code will ignore the
invisible text and some won't.  The display code doesn't expect
BEGV..ZV restriction to behave that way, it expects these limits to
affect every part of Emacs.

> >>> Moreover, it will make redisplay slower.  Skipping invisible text is
> >>> faster than iterating through it, but it still takes time, whereas not
> >>> going beyond BEGV..ZV is instantaneous.
> >>
> >> Org, as one example, uses invisible text all the time. Other feature too.
> > 
> > And Org is indeed relatively slow when you move through a buffer which
> > has large parts of it hidden by invisible properties.
> 
> Org uses different properties, a lot. Not just 'invisible'. So I'd 
> rather people test the performance of this first before dismissing.

I'm talking from experience: movement through a large Org buffer with
a lot of text hidden is slower.

> >> A number of features/commands will indeed need to know the bounds of the
> >> user-level narrowing (and we'll have a buffer-local variable for that),
> >> but that's probably it.
> > 
> > I don't think you realize how widespread is use of low-level
> > primitives and functions in user-level commands.
> 
> Commands that do want to obey narrowing, can take the soft-narrowing 
> bounds and apply the narrowing internally.

That would mean all of the commands.  A lot of changes (or a lot of
breakage).  It doesn't sound like a good solution, at least not better
than just biting the bullet and introducing 2 different kinds of
narrowing with some ways of telling Emacs internals which of the two
to obey at any particular moment.  (The latter is actually the tricky
part, IMO.)

> I'm likely missing a lot of things, since I don't usually use this 
> feature interactively. All I know is, about once or twice a year, people 
> come and ask to make a certain command ignore narrowing. And nobody 
> every comes and asks to revert those changes.

Making a specific command ignore narrowing is easy.  Your proposal
implicitly assumes that the number of commands that want to ignore
narrowing is much larger than the other kind.  I don't think it's
true, and you haven't provided any evidence to that effect.  Moreover,
I think it might make sense for some commands to honor or ignore the
narrowing depending on the use case, and that is not solved with your
proposal.

> Could someone give a few problem scenarios with this patch? Preferably 
> ones that should be hard to fix. Adapting isearch took about 2 lines.

That's just one command.  Emacs has thousands of them.

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 18.08.2022 09:25, Eli Zaretskii wrote:

>> I guess I didn't understand your concern, sorry. Invisible is handled
>> somewhere on the high level, OK. I did not mistake it for 'intangible'.
> 
> The concern is that some parts of the display code will ignore the
> invisible text and some won't.  The display code doesn't expect
> BEGV..ZV restriction to behave that way, it expects these limits to
> affect every part of Emacs.

That's not a fundamental problem. I believe it's possible to come 99.9% 
close (and even provide better usability, say, by highlighting the 
narrowing bounds to indicate that there is more to the buffer, but it's 
hidden), but if not, the implementation can go a little lower level:

The redisplay could repeat the trick I showed with Isearch and wrap most 
of its logic in (with-soft-narrow ...) -- which translates the soft 
narrowing into an actual one temporarily. Then the 'invisible' property 
won't even be required.

>>>> A number of features/commands will indeed need to know the bounds of the
>>>> user-level narrowing (and we'll have a buffer-local variable for that),
>>>> but that's probably it.
>>>
>>> I don't think you realize how widespread is use of low-level
>>> primitives and functions in user-level commands.
>>
>> Commands that do want to obey narrowing, can take the soft-narrowing
>> bounds and apply the narrowing internally.
> 
> That would mean all of the commands.  A lot of changes (or a lot of
> breakage).  It doesn't sound like a good solution, at least not better
> than just biting the bullet and introducing 2 different kinds of
> narrowing with some ways of telling Emacs internals which of the two
> to obey at any particular moment.

Two kinds of narrowings is inherently a more complex solution. But it 
might be more backward-compatible, yes. Since it just adds "one more 
thing" on top of the existing system.

> (The latter is actually the tricky
> part, IMO.)

If we just make the "locking" feature to be accessible to Lisp, the 
majority of the code shouldn't care or be aware of it. Just some 
particular features/functions/packages will make use of the locking 
argument, limiting the 'widen' calls in the rest of the Emacs from 
widening past the locked region.

>> I'm likely missing a lot of things, since I don't usually use this
>> feature interactively. All I know is, about once or twice a year, people
>> come and ask to make a certain command ignore narrowing. And nobody
>> every comes and asks to revert those changes.
> 
> Making a specific command ignore narrowing is easy.  Your proposal
> implicitly assumes that the number of commands that want to ignore
> narrowing is much larger than the other kind.

That is indeed the key assumption. I'm inclined to believe that not only 
it is larger, but that the set of commands that should ignore user-level 
narrowing also grows faster.

> I don't think it's
> true, and you haven't provided any evidence to that effect.

My observation is that the number of regular users of interactive 
narrowing is not so big. Perhaps limited to a particular subset of 
old-school Emacsers (and some recently-converted sympathizers).

As a result, a lot of problems with commands _not_ ignoring narrowing 
when they should go unreported for years.

E.g. https://debbugs.gnu.org/21262 or 
https://github.com/dgutov/diff-hl/issues/48 (the packages was 3 years 
old at that point, and its main functionality was broken when narrowed) 
https://github.com/dgutov/diff-hl/pull/109 (the corresponding feature 
was 6 year old at that point).

With that in mind, creating any precise statistics doesn't seem 
possible, even if one decides to try.

On the flip side, though, it doesn't seem like diff-hl needs any support 
for user-level narrowing (ignoring it is fine), and company-mode "just 
worked" when I tested it with soft-narrow-to-region.

Same goes for all other packages I maintain.

> Moreover,
> I think it might make sense for some commands to honor or ignore the
> narrowing depending on the use case, and that is not solved with your
> proposal.

Doesn't seem difficult:

(if something
     (with-soft-narrow
      do-the-thing)
   do-the-thing)

or without the macro:

(save-restriction
   (when something
     (let ((bounds (soft-narrow-bounds)))
       (and bounds
            (narrow-to-region (car bounds) (cdr bounds)))))
   do-the-thing)

>> Could someone give a few problem scenarios with this patch? Preferably
>> ones that should be hard to fix. Adapting isearch took about 2 lines.
> 
> That's just one command.  Emacs has thousands of them.

True. In the current state it only solves Isearch (obviously) and the 
basic movement/editing commands by using 'invisible', 'read-only' and 
'modification-hooks' properties.

Stefan's complaint should be fixable by changing either 
mark-whole-buffer or push-mark directly (I'm not sure yet) and 
soft-narrow-to-region to update the region bounds just in case it was 
called from a Lisp program.

I was thinking the vast majority of use cases should be solved by 
altering some key functions. But I can't guarantee that, of course.

Re: locked narrowing in ELisp

[Eli Zaretskii]

> Date: Fri, 19 Aug 2022 02:10:49 +0300
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 18.08.2022 09:25, Eli Zaretskii wrote:
> 
> >> I guess I didn't understand your concern, sorry. Invisible is handled
> >> somewhere on the high level, OK. I did not mistake it for 'intangible'.
> > 
> > The concern is that some parts of the display code will ignore the
> > invisible text and some won't.  The display code doesn't expect
> > BEGV..ZV restriction to behave that way, it expects these limits to
> > affect every part of Emacs.
> 
> That's not a fundamental problem. I believe it's possible to come 99.9% 
> close (and even provide better usability, say, by highlighting the 
> narrowing bounds to indicate that there is more to the buffer, but it's 
> hidden), but if not, the implementation can go a little lower level:
> 
> The redisplay could repeat the trick I showed with Isearch and wrap most 
> of its logic in (with-soft-narrow ...) -- which translates the soft 
> narrowing into an actual one temporarily. Then the 'invisible' property 
> won't even be required.

I think you still underestimate the magnitude of the problem and its
fundamental nature.  The display engine is not a closed subsystem, it
uses a huge number of Emacs infrastructure code from other subsystems:
buffers, text properties, overlays, markers, character composition,
regular expressions, direct access to buffer text, etc.  All of those
places are also used from other commands and functions, outside of
redisplay.  You cannot "wrap [their] logic in (with-soft-narrow ...)",
because their logic is used in many more use cases, and there's no way
such low-level code can distinguish between the cases, nor should it
(because it will make it too slow).

> > That would mean all of the commands.  A lot of changes (or a lot of
> > breakage).  It doesn't sound like a good solution, at least not better
> > than just biting the bullet and introducing 2 different kinds of
> > narrowing with some ways of telling Emacs internals which of the two
> > to obey at any particular moment.
> 
> Two kinds of narrowings is inherently a more complex solution.

Some problems don't have simple solutions.  The narrowing is a very
fundamental feature in Emacs, it permeates all of our code to the
lowest levels.  Such fundamental features cannot be side-stepped by
tricks and hacks.  The change must be in those same basic levels.

> > (The latter is actually the tricky
> > part, IMO.)
> 
> If we just make the "locking" feature to be accessible to Lisp, the 
> majority of the code shouldn't care or be aware of it. Just some 
> particular features/functions/packages will make use of the locking 
> argument, limiting the 'widen' calls in the rest of the Emacs from 
> widening past the locked region.

I'm not talking about locking!  I'm talking about the basic
requirement to distinguish between "user-level" narrowing and the
other kind.  How do you tell the low-level stuff, such as
set_marker_restricted, which of the two narrowing kinds to apply?
That function is used from every place where we set window-point and
window-start positions (as well as many others), so almost any code
that wants to move one of those will eventually call it.  Commands do
that, but redisplay and other places do it as well.

This is the tricky part of introducing two kinds of narrowing; the
rest is very simple and basically mechanical.

> > Making a specific command ignore narrowing is easy.  Your proposal
> > implicitly assumes that the number of commands that want to ignore
> > narrowing is much larger than the other kind.
> 
> That is indeed the key assumption. I'm inclined to believe that not only 
> it is larger, but that the set of commands that should ignore user-level 
> narrowing also grows faster.

What about commands that apply narrowing as part of their internal
operation -- is that "user-level" narrowing or the other kind?  E.g.,
some/many commands that operate on the region when it's active begin
by narrowing to that region -- which kind is that?

> With that in mind, creating any precise statistics doesn't seem 
> possible, even if one decides to try.

If that's so, then any assumptions about which set is significantly
larger cannot be safely made, either.  We should proceed without any
such assumption, i.e. without adopting any solutions which would be
based on such assumptions and will break if the assumptions are proved
false.

> On the flip side, though, it doesn't seem like diff-hl needs any support 
> for user-level narrowing (ignoring it is fine), and company-mode "just 
> worked" when I tested it with soft-narrow-to-region.
> 
> Same goes for all other packages I maintain.

IME, it is not a good idea to base such fundamental design decisions
on a couple of user commands and their needs.

> > Moreover,
> > I think it might make sense for some commands to honor or ignore the
> > narrowing depending on the use case, and that is not solved with your
> > proposal.
> 
> Doesn't seem difficult:
> 
> (if something
>      (with-soft-narrow
>       do-the-thing)
>    do-the-thing)
> 
> or without the macro:
> 
> (save-restriction
>    (when something
>      (let ((bounds (soft-narrow-bounds)))
>        (and bounds
>             (narrow-to-region (car bounds) (cdr bounds)))))
>    do-the-thing)

We are mis-communicating: the problem is not how to use basic
conditionals in Emacs Lisp, the problem is how you design and
implement that "something", such that we won't need to rewrite every
single command out there.

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 19.08.2022 09:31, Eli Zaretskii wrote:

>> The redisplay could repeat the trick I showed with Isearch and wrap most
>> of its logic in (with-soft-narrow ...) -- which translates the soft
>> narrowing into an actual one temporarily. Then the 'invisible' property
>> won't even be required.
> 
> I think you still underestimate the magnitude of the problem and its
> fundamental nature.  The display engine is not a closed subsystem, it
> uses a huge number of Emacs infrastructure code from other subsystems:
> buffers, text properties, overlays, markers, character composition,
> regular expressions, direct access to buffer text, etc.  All of those
> places are also used from other commands and functions, outside of
> redisplay.  You cannot "wrap [their] logic in (with-soft-narrow ...)",
> because their logic is used in many more use cases, and there's no way
> such low-level code can distinguish between the cases, nor should it
> (because it will make it too slow).

Naturally, some uses of 'goto-char' (for example) need to obey the 
narrowing applied by the user, and some -- do not. That's the situation 
today, and that's not going to change.

The question is, would "do not obey" work as a good default.

>>> That would mean all of the commands.  A lot of changes (or a lot of
>>> breakage).  It doesn't sound like a good solution, at least not better
>>> than just biting the bullet and introducing 2 different kinds of
>>> narrowing with some ways of telling Emacs internals which of the two
>>> to obey at any particular moment.
>>
>> Two kinds of narrowings is inherently a more complex solution.
> 
> Some problems don't have simple solutions.  The narrowing is a very
> fundamental feature in Emacs, it permeates all of our code to the
> lowest levels.  Such fundamental features cannot be side-stepped by
> tricks and hacks.  The change must be in those same basic levels.

I think we're talking about two very similar things here, just looking 
at them from different angles.

But if we were going to separate user-level narrowing from "real" 
narrowing, the result should look (at least at the surface) like the 
patch I posted, and it should face the same problems: how to decide 
which commands should obey user-level locking, and how to minimize the 
number of functions we'll have to alter to support it explicitly.

>>> (The latter is actually the tricky
>>> part, IMO.)
>>
>> If we just make the "locking" feature to be accessible to Lisp, the
>> majority of the code shouldn't care or be aware of it. Just some
>> particular features/functions/packages will make use of the locking
>> argument, limiting the 'widen' calls in the rest of the Emacs from
>> widening past the locked region.
> 
> I'm not talking about locking!  I'm talking about the basic
> requirement to distinguish between "user-level" narrowing and the
> other kind.  How do you tell the low-level stuff, such as
> set_marker_restricted, which of the two narrowing kinds to apply?

The solution I picked is this: lower-level code only obeys the "real" 
narrowing. Perhaps the redisplay loop sprinkles in a bit of support for 
"user-level" narrowing by temporarily applying the "real" narrowing 
around the whole main logic when such "user-level" narrowing is present 
(using its bounds).

As a result, with very rare exceptions (Info-mode, maybe), the only kind 
of "persistent" narrowing would be the "user-level" one, but every 
command (or function, or form) will have the freedom to translate it 
into the "real" one for the duration of its execution. The "real" 
narrowing would be thus relegated to lower-level uses, and not as a 
user's tool.

> That function is used from every place where we set window-point and
> window-start positions (as well as many others), so almost any code
> that wants to move one of those will eventually call it.  Commands do
> that, but redisplay and other places do it as well.
> 
> This is the tricky part of introducing two kinds of narrowing; the
> rest is very simple and basically mechanical.
> 
>>> Making a specific command ignore narrowing is easy.  Your proposal
>>> implicitly assumes that the number of commands that want to ignore
>>> narrowing is much larger than the other kind.
>>
>> That is indeed the key assumption. I'm inclined to believe that not only
>> it is larger, but that the set of commands that should ignore user-level
>> narrowing also grows faster.
> 
> What about commands that apply narrowing as part of their internal
> operation -- is that "user-level" narrowing or the other kind?  E.g.,
> some/many commands that operate on the region when it's active begin
> by narrowing to that region -- which kind is that?

It's the other kind, the "real" one.

Sometime later, new functions might crop up (probably in third-party 
code) that decide to alter the user-level narrowing temporarily instead 
because they know which of their callees obey user-level narrowing and 
which do not, and decide to benefit from that distinction. But that 
approach should remain rare.

>> With that in mind, creating any precise statistics doesn't seem
>> possible, even if one decides to try.
> 
> If that's so, then any assumptions about which set is significantly
> larger cannot be safely made, either.  We should proceed without any
> such assumption, i.e. without adopting any solutions which would be
> based on such assumptions and will break if the assumptions are proved
> false.

We could try to make such analysis anyway, just remain aware that the 
difference might be larger (in one particular direction).

If we just have to adopt a solution which goes against the assumption I 
made, well, then, the approach probably won't work.

>> On the flip side, though, it doesn't seem like diff-hl needs any support
>> for user-level narrowing (ignoring it is fine), and company-mode "just
>> worked" when I tested it with soft-narrow-to-region.
>>
>> Same goes for all other packages I maintain.
> 
> IME, it is not a good idea to base such fundamental design decisions
> on a couple of user commands and their needs.

Several fairly high-profile packages are not "a couple of user 
commands". But it's not the whole ecosystem, sure.

>>> Moreover,
>>> I think it might make sense for some commands to honor or ignore the
>>> narrowing depending on the use case, and that is not solved with your
>>> proposal.
>>
>> Doesn't seem difficult:
>>
>> (if something
>>       (with-soft-narrow
>>        do-the-thing)
>>     do-the-thing)
>>
>> or without the macro:
>>
>> (save-restriction
>>     (when something
>>       (let ((bounds (soft-narrow-bounds)))
>>         (and bounds
>>              (narrow-to-region (car bounds) (cdr bounds)))))
>>     do-the-thing)
> 
> We are mis-communicating: the problem is not how to use basic
> conditionals in Emacs Lisp, the problem is how you design and
> implement that "something", such that we won't need to rewrite every
> single command out there.

I considered which commands should be affected the most: they have to be 
those that act on and affect the part of the buffer that the user sees. 
The user shouldn't be able to modify the text they don't see (1), nor 
(probably) have that text affect the commands they are calling (2).

The (1) is implemented by adding the 'read-only' property to the 
overlays which apply the user-level narrowing. It leaves the 
possibilities of the user being able to jump before the first overlay 
(to bob) or after the last, and insert something there. That seems to be 
mostly handled using the FRONT-ADVANCE and REAR-ADVANCE arguments, as 
well as modification-hooks. But there are other options like 
post-command-hook, or making the overlays longer than 0 chars.

As for (2), though, we can't tell that in advance for each command, and 
most have to choose case-by-case. But Stefan gave a good example of when 
we *do* know: when the command does something with the region. The user 
cannot tell the region spans beyond the accessible/visible part of the 
buffer.

So the way to "fix" a lot of them together is to make sure the region 
cannot span beyond the user-level narrowing currently applied. How to do 
that? Either we have to change all of the commands which change the 
region, or alter 'set-mark' (which might be a fairly invasive change, 
stopping certain code from working), or perform the adjustments in 
post-command-hook: after every command, see if the region spans beyond 
the user-level narrowing, and adjust either or both of its bounds if needed.

Re: locked narrowing in ELisp

[Eli Zaretskii]

> Date: Wed, 17 Aug 2022 03:05:05 +0300
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> If the user-level narrow/widen commands didn't use _actual_ narrowing, 
> but display engine tricks or whatever (example: 
> https://github.com/Malabarba/fancy-narrow/), "other parts of the major 
> mode" wouldn't have to do the usual (save-restriction (widen) ...) 
> dance, which a lot of code is littered with.

I don't think this is reliable solution for a general problem such as
this one.  What that package does is prevent _some_ commands and APIs
from escaping the "narrowing".  But in the general case, once you get
low enough into our infrastructure code, there's no way of knowing
what is TRT wrt this "narrowing", because the intent of the caller is
unknown, and quite a few places in our code will happily ignore it.

So I think it is eventually unavoidable to have to define some
property of the narrowing that every piece of Emacs will test in order
to understand whether it should or shouldn't move outside of the
narrowed region.

Re: locked narrowing in ELisp

[Dmitry Gutov]

On 17.08.2022 14:44, Eli Zaretskii wrote:
>> Date: Wed, 17 Aug 2022 03:05:05 +0300
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> If the user-level narrow/widen commands didn't use _actual_ narrowing,
>> but display engine tricks or whatever (example:
>> https://github.com/Malabarba/fancy-narrow/), "other parts of the major
>> mode" wouldn't have to do the usual (save-restriction (widen) ...)
>> dance, which a lot of code is littered with.
> 
> I don't think this is reliable solution for a general problem such as
> this one.  What that package does is prevent _some_ commands and APIs
> from escaping the "narrowing".  But in the general case, once you get
> low enough into our infrastructure code, there's no way of knowing
> what is TRT wrt this "narrowing", because the intent of the caller is
> unknown, and quite a few places in our code will happily ignore it.

Knowing the intent of the narrowing is indeed the problem. A split 
between user-level ("soft") narrowing and "real" narrowing makes sense 
to me.

Calling some narrowing "hard" doesn't answer the question of intent, OTOH.

> So I think it is eventually unavoidable to have to define some
> property of the narrowing that every piece of Emacs will test in order
> to understand whether it should or shouldn't move outside of the
> narrowed region.

In either approach we'd end up with two kinds of narrowings. The 
question for me is, which of the approaches will require more effort 
from Lisp writers to adapt. That is, how many functions and features 
should specifically handle user-level narrowing.

I'd be happy to be able to remove all the (save-restriction (widen) ...) 
calls littering the code.

Re: locked narrowing in ELisp

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The new "fast long-lines" code uses a "locked narrowing" to try and
> avoid being defeated by naive font-lock (and other) code.
>
> This is a good heuristic to try and make sure Emacs stays responsive
> even in the presence of naive packages not designed for such long lines.
>
> Beside the need to override this in those rare packages where we can
> provide better behavior with careful coding, the new code also lacks
> a way to install such a locked narrowing from ELisp, even though it
> seems only natural to use similar heuristics from ELisp code (e.g. when
> jit-lock is run from a timer).
>
> But as pointed out by Lars, this is related to the old discussion about
> "classifying" narrowing, so as to distinguish narrowings installed by
> the user via `C-x n n` from those installed by packages like MMM-mode or
> Info-mode (or nowadays those installed by the redisplay).
> These distinctions are important because various code may need to look
> "outside" of those narrowings in some circumstances, but which "outside"
> to use depends on the particular situation.
>
> We've discussed it in the past and I still don't know what would be
> a good API, but one particular class of `widen` which is known to be
> used within narrowings that should *not* be overridden is those `widen`
> calls used in "parsers" that try to understand the context of
> an operation.
>
> Those `widen` will sometimes be used in circumstances where they may
> actually want to *narrow* rather than widen (e.g. the narrowing might be
> needed to make sure the parser only looks at the part of the buffer that
> it can understand (e.g. in MMM-mode), or it might be needed in order to
> avoid the risk of spending an eternity parsing GBs of text (at the cost
> of misparsing in too-large buffers)).
>
> Last time we discussed these issues, we ended up deciding that
> indentation (like font-lock before it) should "eagerly widen" (i.e. undo
> any user-setup narrowing) before calling the mode-specific code.
> This way, the indentation and font-lock code of a major mode should
> simply never need to call `widen` and things like the MMM-mode (or the
> long-lines code in redisplay) can just setup the narrowing before
> calling that major mode code without any need to "label" the narrowing.
>
> This was an elegant solution, but it seems to be too limited, because
> the parsing code that want(ed) to `widen` is typically also called from
> other parts of the major mode than the indentation or font-lock code.
>
> In the past, we played with having a new widening function (there was
> a `prog-widen` briefly in `prog-mode.el`).  Maybe we should
> (re)introduce such a thing (or maybe an optional arg to `widen` which
> would indicate what "kind" of widening we want to do), coupled with some
> kind of `widen-function` which MMM-mode and the redisplay could
> setup to filter/tweak those widening.
>
> Within an MMM-mode or "redisplay locked narrowing", we *generally* want
> `widen` not to widen past the limits imposed by MMM or by the redisplay,
> but there will be cases where we do want to allow it (e.g. in
> `nlinum-mode` the widening will usually want to be allowed to override
> those narrowings in order to compute the right line number), so we
> clearly not all calls to `widen` are the same.

FWIW, I've already see a workaround in the wild (well, if my job counts
as "wild"), which is to use an indirect buffer to access contents
outside of the "locked narrowing", should widen fail to widen
completely.