bug#69837: 29.2; vtable-update-object only works in visible windows

bug#69837: 29.2; vtable-update-object only works in visible windows

[Adam Porter]

Hi,

I've discovered that `vtable-update-object' only works in buffers that 
are in visible windows.  When trying to update vtables in buffers that 
aren't, the object being updated or replaced fails to be found in the 
cache, apparently because `vtable--cache-key' uses `window-width' in its 
value (so maybe if, when the vtable buffer is not visible, the selected 
window happens to have the same width as the window in which the vtable 
buffer was previously displayed, it will work by chance).

In my case, the vtable is being updated in the background from a timer, 
so its buffer may or may not be visible.  It's preferable to call 
`vtable-update-object' to update two rows in it rather than reverting 
the whole table, because it can be relatively slow to revert the whole 
table when it's large.

I'm not sure of the best way to fix this.  Maybe the vtable's last-used 
window width could be cached and used for the cache key in case its 
buffer is not visible.  I guess that would risk displaying the vtable 
sub-optimally if the next time it's displayed its window width is 
different, but that might be worth it; in that case, the user could 
always revert the table if necessary.

I'm willing to work on a patch for this, but I'd appreciate any 
guidance, since I'm far from an expert on this library, and there are 
many nuances when it comes to buffers, windows, their attributes, etc.

Thanks,
Adam

bug#69837: 29.2; vtable-update-object only works in visible windows

[Eli Zaretskii]

> Date: Sat, 16 Mar 2024 22:41:37 -0500
> From: Adam Porter <adam@alphapapa.net>
> 
> I've discovered that `vtable-update-object' only works in buffers that 
> are in visible windows.  When trying to update vtables in buffers that 
> aren't, the object being updated or replaced fails to be found in the 
> cache, apparently because `vtable--cache-key' uses `window-width' in its 
> value (so maybe if, when the vtable buffer is not visible, the selected 
> window happens to have the same width as the window in which the vtable 
> buffer was previously displayed, it will work by chance).

Does using with-selected-window help to solve the issue?

If not, can you show a recipe, starting from "emacs -Q", that
reproduces the problem, so we could study it in more detail?

bug#69837: 29.2; vtable-update-object only works in visible windows

[Adam Porter]

Hi Eli,

On 3/17/24 01:25, Eli Zaretskii wrote:
>> Date: Sat, 16 Mar 2024 22:41:37 -0500
>> From: Adam Porter <adam@alphapapa.net>
>>
>> I've discovered that `vtable-update-object' only works in buffers that
>> are in visible windows.  When trying to update vtables in buffers that
>> aren't, the object being updated or replaced fails to be found in the
>> cache, apparently because `vtable--cache-key' uses `window-width' in its
>> value (so maybe if, when the vtable buffer is not visible, the selected
>> window happens to have the same width as the window in which the vtable
>> buffer was previously displayed, it will work by chance).
> 
> Does using with-selected-window help to solve the issue?

Sometimes, but not reliably, because it doesn't matter whether the 
window is selected (and the buffer might not have a window, anyway). 
I'll try to give a step-by-step explanation:

1. vtable-update-object looks in the vtable's cache to find the cached 
information about the representation of the object being updated or 
replaced.

2. In the process of doing that, it calls vtable--cache-key, which 
returns a cons cell containing the frame-terminal and window-width.

3. So if the selected frame is on the same terminal, and the selected 
window has the same width, as the ones when the vtable was last 
generated, the cache key will match.  This will allow 
vtable-update-object to access the cached values and look for the object 
in them.

But if the terminal is different, or if the selected window's width is 
different, the cache key will be different, so vtable-update-object will 
fail, even when it could potentially succeed.

In my case, the vtable's buffer is (or can be) in a window that is in a 
non-current tab (using tab-bar-mode) in the same frame, so its window is 
not visible.  And so if the selected window in the current tab has the 
same width as the vtable's window, vtable-update-object may work.  But 
if the widths don't match, it will fail.

(I suppose it may also fail if the vtable's buffer's window is visible 
but has changed width since the vtable was generated, but I haven't 
tested that.)

> If not, can you show a recipe, starting from "emacs -Q", that
> reproduces the problem, so we could study it in more detail?

I had hoped to avoid writing that much code to demonstrate it.  But if 
the explanation above doesn't suffice, let me know and I will.

Thanks,
Adam

bug#69837: 29.2; vtable-update-object only works in visible windows

[Eli Zaretskii]

> Date: Sun, 17 Mar 2024 21:05:56 -0500
> Cc: 69837@debbugs.gnu.org
> From: Adam Porter <adam@alphapapa.net>
> 
> Hi Eli,
> 
> On 3/17/24 01:25, Eli Zaretskii wrote:
> >> Date: Sat, 16 Mar 2024 22:41:37 -0500
> >> From: Adam Porter <adam@alphapapa.net>
> >>
> >> I've discovered that `vtable-update-object' only works in buffers that
> >> are in visible windows.  When trying to update vtables in buffers that
> >> aren't, the object being updated or replaced fails to be found in the
> >> cache, apparently because `vtable--cache-key' uses `window-width' in its
> >> value (so maybe if, when the vtable buffer is not visible, the selected
> >> window happens to have the same width as the window in which the vtable
> >> buffer was previously displayed, it will work by chance).
> > 
> > Does using with-selected-window help to solve the issue?
> 
> Sometimes, but not reliably, because it doesn't matter whether the 
> window is selected (and the buffer might not have a window, anyway). 
> I'll try to give a step-by-step explanation:

I'm sorry, I'm not familiar with the design of vtable, so I will have
to ask you possibly stupid questions.  Apologies in advance.

> 1. vtable-update-object looks in the vtable's cache to find the cached 
> information about the representation of the object being updated or 
> replaced.

Is this cache used just to speed up something (and so if the object is
not in the cache, Emacs will just work harder to obtain the same
information), or is finding the object in the cache absolutely
necessary for working with the object?

> 2. In the process of doing that, it calls vtable--cache-key, which 
> returns a cons cell containing the frame-terminal and window-width.

If working with an object needs its window-width, does it mean objects
are tightly coupled to their windows?  If so, what happens when the
user or Emacs resizes the window? are the cached objects recomputed to
reflect that?

> 3. So if the selected frame is on the same terminal, and the selected 
> window has the same width, as the ones when the vtable was last 
> generated, the cache key will match.  This will allow 
> vtable-update-object to access the cached values and look for the object 
> in them.

This again seems to imply that an object is tightly coupled to its
window, and changes affecting the window must recompute the cached
value.  Right?

Are these keys (window-width, terminal, etc.) only used to look up the
object, or are they needed for processing the objects as well?

> > If not, can you show a recipe, starting from "emacs -Q", that
> > reproduces the problem, so we could study it in more detail?
> 
> I had hoped to avoid writing that much code to demonstrate it.  But if 
> the explanation above doesn't suffice, let me know and I will.

It doesn't have to be code, it could be a (possibly long) list of
instructions what to type.

bug#69837: 29.2; vtable-update-object only works in visible windows

[Adam Porter]

Hi Eli,

On 3/18/24 12:05, Eli Zaretskii wrote:

 > I'm sorry, I'm not familiar with the design of vtable, so I will
 > have to ask you possibly stupid questions.  Apologies in advance.
I'm not very familiar with it, either, so I will have to give you
possibly stupid answers.  :)  But I will try to give accurate ones.

 >> 1. vtable-update-object looks in the vtable's cache to find the
 >> cached information about the representation of the object being
 >> updated or replaced.
 >
 > Is this cache used just to speed up something (and so if the object
 > is not in the cache, Emacs will just work harder to obtain the same
 > information), or is finding the object in the cache absolutely
 > necessary for working with the object?
It appears that the caching is integral to the rendering of vtables.  In
`vtable-insert`, a comment explains:

     ;; We maintain a cache per screen/window width, so that we render
     ;; correctly if Emacs is open on two different screens (or the
     ;; user resizes the frame).

Then `vtable--ensure-cache` is called, which returns either an existing
cache or a newly computed one, and its return value is bound and used
when inserting the table's header line and when inserting each object's
representation.  So it seems that the rendering of objects always uses
the cached values.

 >> 2. In the process of doing that, it calls vtable--cache-key, which
 >> returns a cons cell containing the frame-terminal and
 >> window-width.
 >
 > If working with an object needs its window-width, does it mean
 > objects are tightly coupled to their windows?  If so, what happens
 > when the user or Emacs resizes the window? are the cached objects
 > recomputed to reflect that?
AFAICT, if a vtable's buffer's window is resized, nothing happens
automatically.  But if the vtable is reverted (causing it to be
completely re-rendered), the window's width is taken into account
compared to the width of the table, affecting the vtable's header line,
and possibly the widths of some columns, depending on the table's
specification (see functions `vtable--compute-width` and
`vtable--compute-widths`).  And if an individual object is updated, it
fails if the window width has changed, since the cache key misses.

I suppose the cache serves to sometimes help avoid re-rendering all the
objects in the table, and it uses the window-width as part of the cache
key because the table's buffer could be shown in multiple windows, each
of which could have a different width, requiring a different rendering
of the table (though I can't say I understand exactly how that could
work, since ISTM that rendering it for one window would display the same
rendering in the other window).

 >> 3. So if the selected frame is on the same terminal, and the
 >> selected window has the same width, as the ones when the vtable was
 >> last generated, the cache key will match.  This will allow
 >> vtable-update-object to access the cached values and look for the
 >> object in them.
 >
 > This again seems to imply that an object is tightly coupled to its
 > window, and changes affecting the window must recompute the cached
 > value.  Right?
I think the way it works is roughly like this:

- An object's value(s) affect the width of the columns in its
representation's row.

- A row's columns' widths (or their total width) is compared with the
width of the window to determine how to display the last column (or the
last one visible in the window?).

- So the inverse holds as well: the window's width affects the display
of (at least one of) an object's representation's columns.

So if an object is being re-rendered, and the window's width is the
same, the cache can be used (which also seems to record the line number
at which the object is rendered, which saves from having to iterate
through the table to find the buffer line).  Otherwise, if the window's
width has changed, the cache misses; and since `vtable-update-object`
does not handle this case (it just causes an error), the object is not
updated.  This would require the application to handle the error and
revert the whole table instead.

I would suppose that, in that case, the last-used window width (if it
were known) could be used to update the object, assuming that the width
is either the same or "good enough," to avoid having to re-render the
whole table; if the worst that happened is that a column at the edge of
the table were a bit too wide or narrow compared to the ideal, that
would often be preferable to re-rendering the whole table.  (In my case,
I could have 1,000-2,000 rows in the table, in which case I would prefer
to avoid re-rendering the whole thing; leaving the columns at their
existing width is no problem, especially if some of them are already
past the width of the window--and in my case, the table is always wider
than the window.)

 > Are these keys (window-width, terminal, etc.) only used to look up
 > the object, or are they needed for processing the objects as well?
The window's width appears to be integral to the rendering of the
objects, both individually and when inserting the whole table at once.
The terminal is used as part of the cache key, probably because a
different display could have a different resolution (DPI), which would
make calculations on one invalid on another (I'm sure you know more
about how that works than I do).

 >>> If not, can you show a recipe, starting from "emacs -Q", that
 >>> reproduces the problem, so we could study it in more detail?
 >>
 >> I had hoped to avoid writing that much code to demonstrate it.  But
 >> if the explanation above doesn't suffice, let me know and I will.
 >
 > It doesn't have to be code, it could be a (possibly long) list of
 > instructions what to type.
In my case, I'm using vtable in my `listen` package on ELPA.  If I, e.g.
add all the tracks in my music library to a `listen-queue`, which is
rendered with vtable, it's helpful to be able to update just one track's
representation at a time (so that the currently playing track can have
an arrow next to it, and the previously playing one can have its arrow
removed), because rendering the whole table can take a few seconds.
(Smaller tables, e.g. a hundred tracks or so, render very quickly.)

If a concrete demo is still needed, let me know and I can hack something up.

Thanks,
Adam

bug#69837: 29.2; vtable-update-object only works in visible windows

[Adam Porter]

[-- Attachment #1: Type: text/plain, Size: 1937 bytes --]

Hi Eli,

FWIW, I did some hacking on listen.el attempting to further understand 
and work around this problem, and I've found what seems to be a usable 
workaround (for my purposes, anyway).

The attached code defines a macro within which I call 
`listen-queue--vtable-update-object' (which merely incorporates the fix 
to `vtable-update-object' from bug#69664).  As well, I wrap 
`make-vtable' in a function that also sets two buffer-local variables to 
the values of `frame-terminal' and `window-width' at the time of the 
vtable's creation.  The macro locally overrides the functions 
`frame-terminal' and `window-width' to return those saved values.

This allows the cache key to match unconditionally, which allows the 
vtable's objects to be updated even when its buffer is not visible.

In my limited testing, it seems to work fine.  In my estimation, the 
consequences of doing this in the worst case would be that the rows for 
the updated objects might be drawn with some columns at a slightly 
incorrect width, which is easily rectified by reverting the table 
(usually bound to "g").  As well, that worst case (e.g. imagining a 
vtable whose buffer might be initially displayed on one terminal/monitor 
and later on another with different characteristics) would seem to be 
relatively rare (so for my project, it seems like an obviously good 
thing to do).

For Emacs itself, I'm not sure what the best fix would be.  I suppose a 
workaround like this could be implemented as a fallback in case the 
cache key misses; it would seem better to update the object potentially 
sub-optimally than to error and not update it at all.

Another possibility would be to ignore the frame-terminal and 
window-width in the cache key altogether (i.e. so they would always be 
assumed to be the same), but I'm sure that Lars did it this way for a 
reason, so that would seem unwise.

Let me know how you'd like me to proceed.

Thanks,
Adam

[-- Attachment #2: example.el --]
[-- Type: text/x-emacs-lisp, Size: 1149 bytes --]

(cl-defmacro listen-with-vtable-at (position &rest body)
  "FIXME: Docstring."
  (declare (indent defun))
  (let ((positionᵥ (gensym)))
    `(let ((,positionᵥ ,position))
       (save-excursion
         (goto-char ,positionᵥ)
         (cl-letf* (((symbol-function 'frame-terminal)
                     (lambda (&optional _)
                       listen-vtable-frame-terminal))
                    ((symbol-function 'window-width)
                     (lambda (&optional _ _)
                       listen-vtable-window-width))
                    (table (vtable-current-table))
                    ((symbol-function 'vtable-current-table)
                     (lambda ()
                       table))
                    ((symbol-function 'vtable--recompute-numerical)
                     #'listen-queue--vtable--recompute-numerical))
           ,@body)))))

(defvar-local listen-vtable-frame-terminal nil)
(defvar-local listen-vtable-window-width nil)

(defun listen-make-vtable (&rest args)
  (apply #'make-vtable args)
  (setq-local listen-vtable-frame-terminal (frame-terminal)
              listen-vtable-window-width (window-width)))

bug#69837: 29.2; vtable-update-object only works in visible windows

[Eli Zaretskii]

> Date: Thu, 21 Mar 2024 03:36:02 -0500
> Cc: 69837@debbugs.gnu.org
> From: Adam Porter <adam@alphapapa.net>
> 
> FWIW, I did some hacking on listen.el attempting to further understand 
> and work around this problem, and I've found what seems to be a usable 
> workaround (for my purposes, anyway).
> 
> The attached code defines a macro within which I call 
> `listen-queue--vtable-update-object' (which merely incorporates the fix 
> to `vtable-update-object' from bug#69664).  As well, I wrap 
> `make-vtable' in a function that also sets two buffer-local variables to 
> the values of `frame-terminal' and `window-width' at the time of the 
> vtable's creation.  The macro locally overrides the functions 
> `frame-terminal' and `window-width' to return those saved values.
> 
> This allows the cache key to match unconditionally, which allows the 
> vtable's objects to be updated even when its buffer is not visible.
> 
> In my limited testing, it seems to work fine.  In my estimation, the 
> consequences of doing this in the worst case would be that the rows for 
> the updated objects might be drawn with some columns at a slightly 
> incorrect width, which is easily rectified by reverting the table 
> (usually bound to "g").  As well, that worst case (e.g. imagining a 
> vtable whose buffer might be initially displayed on one terminal/monitor 
> and later on another with different characteristics) would seem to be 
> relatively rare (so for my project, it seems like an obviously good 
> thing to do).
> 
> For Emacs itself, I'm not sure what the best fix would be.  I suppose a 
> workaround like this could be implemented as a fallback in case the 
> cache key misses; it would seem better to update the object potentially 
> sub-optimally than to error and not update it at all.
> 
> Another possibility would be to ignore the frame-terminal and 
> window-width in the cache key altogether (i.e. so they would always be 
> assumed to be the same), but I'm sure that Lars did it this way for a 
> reason, so that would seem unwise.
> 
> Let me know how you'd like me to proceed.

I think you should install your workaround.  I don't see how it could
be worse than what we have now.

P.S. And sorry for a long silence.