bug#66117: 30.0.50; `find-buffer-visiting' is slow when opening large number of buffers

[Dmitry Gutov <dmitry@gutov.dev>]

On 22/09/2023 14:03, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>>> Would it be possible to implement some kind of caching mechanism to be
>>> used by `find-buffer-visiting'?
>>
>> I'm guessing you Cc'd me because of an existing comment inside xref.el?
> 
> Actually, mostly because I though that this discussion might be of
> interest for you.

All right, thank you for that.

>> As you can see I decided not to use this function there, but even
>> get-file-buffer wasn't as fast as I would've wanted, so there's a
>> quick-and-dirty caching solution for sequential lookups (which assumes
>> that the same file would be looked up multiple times in a row).
> 
> For me, the situation is different - a new not-yet-open file is looked
> up multiple times in a row, which is a bit more tricky.

This could be handled with a locally held dictionary as well, but...

>> Putting aside the perspective of maintaining a cache in the core
>> (hopefully this will be discussed later in this feature request), any
>> chance you could initialize that cache yourself first (by iterating
>> through buffers and building file-truename -> buffer map), before you
>> are doing those many lookups? Somewhere in org-agenda-list, perhaps.
>>
>> That should change the complexity from N*M to N+M, more or less.
> 
> Sure. That's my plan to address older Emacs versions. But such iteration
> will involve copy-pasting a big part of `find-buffer-visiting' source
> code, which is not something I want to maintain in a long term (what if
> `find-buffer-visiting' changes in future?).
> 
> Moreover, `find-buffer-visiting' is called by `find-file-noselect', and
> I simply cannot make use of cache there without modifying the code
> upstream; or writing my own version of `find-file-noselect' - bad idea
> maintenance-wise.

...if most of said calls are done through find-file-noselect, I suppose 
that solution is a no-go.

> I think that the best way that will benefit more than Org mode is
> arranging internal cache that will link buffer-file-name,
> buffer-file-truename, and buffer-file-number with buffers; and maintain
> the correctness of the cache if buffer-file-name changes for any reason.

I think that is doable.

It probably won't regress the performance of any particular scenario 
either, but we should benchmark opening a bunch of files this way anyway 
(might actually get faster, due to find-buffer-visiting calls).