xref and leaving "temporary" buffers open

xref and leaving "temporary" buffers open

[Dmitry Gutov]

Hi all,

The original implementation, by Helmut, didn't worry about that, and, 
for instance, used xref-buffer-location for the results of "find 
definitions" operation in the Elisp backend.

I was concerned about having opening lots of buffers, the user being 
surprised at seeing a querying operation make that change in their Emacs 
session. Then switch-to-buffer, as one example, might become more 
troublesome, for having too many options to choose from. Some Emacs 
features might slow down for having to deal with too many buffers (like 
auto-revert?). Anyway, a neat freak like yours truly might get annoyed. 
So I made the current implementations always clean up after themselves.

However, that causes a performance hit: opening a buffer takes time, 
especially when you need to syntax-propertize it (which is a requirement 
for xref--collect-match). So now even if I repeatedly search for the 
same thing, each search incurs the same performance penalty. Or, more 
practically, if a do a search and then press `r' to call 
`xref-query-replace', the latter has to reopen the killed buffers again. 
Which could be implemented, with certain amount of effort, in a deferred 
way so the user doesn't suffer too much, but even that would break if 
some xrefs from the same file appear separately in the list (and that 
might happen, for example, if etags--xref-find-definitions splits the 
results into groups by tag order).

Should we disregard this issue and say that xref doesn't have to close 
the buffers it needed to open during operation?

Another problem with that might be the contents getting out of date: 
what if the user doesn't have global-auto-revert-mode on? Then an xref 
search might get erroneous results from those buffers. Would it be 
allowed to call `revert-buffer' anywhere, as long as the buffer doesn't 
have unsaved changes?

Certain other programs might have a project and all its files loaded, 
but still display only a few of them as the ones the user is editing 
(probably using tabs). Should Emacs have a similar distinction, between 
loaded and open buffers?

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 03:47:11 +0300
> 
> Should we disregard this issue and say that xref doesn't have to close 
> the buffers it needed to open during operation?

How is xref different from Grep or Gid in this regard?  Or from
"M-x gdb", for that matter?  They leave behind buffers that have
names like "*gid foobar*" and "*gdb myprog*", and never look back.

Is xref different?  If so, why?

> Another problem with that might be the contents getting out of date: 
> what if the user doesn't have global-auto-revert-mode on? Then an xref 
> search might get erroneous results from those buffers. Would it be 
> allowed to call `revert-buffer' anywhere, as long as the buffer doesn't 
> have unsaved changes?

I see no reason why not.  Some commands already save the buffer
(perhaps after asking for confirmation) to have the file on disk
consistent with what Emacs has, and this is no different.

> Certain other programs might have a project and all its files loaded, 
> but still display only a few of them as the ones the user is editing 
> (probably using tabs). Should Emacs have a similar distinction, between 
> loaded and open buffers?

I'm not sure I understand how this distinction will come into play, or
why is it relevant to the issue at hand.  Please elaborate.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > Should we disregard this issue and say that xref doesn't have to close
 > the buffers it needed to open during operation?

I'd try to implement some sort of heuristics, like adding a buffer to
some sort of a cache only after its file has been visited by an
application (like xref) at least N times and removing a buffer from that
cache if it has not been visited by an application (like xref) for at
least T minutes.

 > Another problem with that might be the contents getting out of date:
 >  what if the user doesn't have global-auto-revert-mode on? Then an
 >  xref search might get erroneous results from those buffers. Would it
 >  be allowed to call `revert-buffer' anywhere, as long as the buffer
 >  doesn't have unsaved changes?

This should be made customizable.

 > Certain other programs might have a project and all its files loaded,
 > but still display only a few of them as the ones the user is editing
 > (probably using tabs). Should Emacs have a similar distinction,
 > between loaded and open buffers?

I think so.  Each buffer should maintain a list of all the applications
that visited its file and all operations on buffer lists should be aware
of it.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 10:27 AM, Eli Zaretskii wrote:

> How is xref different from Grep or Gid in this regard?  Or from
> "M-x gdb", for that matter?  They leave behind buffers that have
> names like "*gid foobar*" and "*gdb myprog*", and never look back.

No, these are fine.

What I'm talking about, are file buffers that xref commands need to 
open, to read their contents. Such as xref-find-references; see, it can 
use Grep to find textual matches, but it cannot verify that the 
beginning and the end of the match are symbol boundaries without using 
the corresponding major mode, its syntax table, *and* its 
syntax-propertize-function. Basically, the only sane solution is to open 
each buffer and verify.

The question is, should they be killed afterwards.

> I see no reason why not.  Some commands already save the buffer
> (perhaps after asking for confirmation) to have the file on disk
> consistent with what Emacs has, and this is no different.

Hmm yes. I wonder if it'll be appropriate to simply call 
(auto-revert-handler) if each buffer visited for this purpose.

>> Certain other programs might have a project and all its files loaded,
>> but still display only a few of them as the ones the user is editing
>> (probably using tabs). Should Emacs have a similar distinction, between
>> loaded and open buffers?
>
> I'm not sure I understand how this distinction will come into play, or
> why is it relevant to the issue at hand.  Please elaborate.

Hopefully, this is clear now.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 11:28 AM, martin rudalics wrote:

> I'd try to implement some sort of heuristics, like adding a buffer to
> some sort of a cache only after its file has been visited by an
> application (like xref) at least N times and removing a buffer from that
> cache if it has not been visited by an application (like xref) for at
> least T minutes.

N > 1 won't help too much. I'm talking about a situation like 
xref-find-references followed by xref-query-replace. It already gives a 
performance hit (if there are matches in a lot of different files). And 
xref-find-references only opens each buffer once already.

What kind of cache?

> [Automatic reverting] should be made customizable.

I suppose a user option might govern whether xref will prompt before 
reverting. But then, will it prompt for each file? There might be a lot.

> I think so.  Each buffer should maintain a list of all the applications
> that visited its file and all operations on buffer lists should be aware
> of it.

You might want to elaborate. This seems to go further than my suggestion.

Re: xref and leaving "temporary" buffers open

[Stephen Leake]

martin rudalics <rudalics@gmx.at> writes:

>> Should we disregard this issue and say that xref doesn't have to close
>> the buffers it needed to open during operation?
>
> I'd try to implement some sort of heuristics, like adding a buffer to
> some sort of a cache only after its file has been visited by an
> application (like xref) at least N times and removing a buffer from that
> cache if it has not been visited by an application (like xref) for at
> least T minutes.

Better for xref to cache the info it gathered from the buffer (assuming
that's significantly smaller, and faster to access, than the buffer).

>> Another problem with that might be the contents getting out of date:
>>  what if the user doesn't have global-auto-revert-mode on? Then an
>>  xref search might get erroneous results from those buffers. Would it
>>  be allowed to call `revert-buffer' anywhere, as long as the buffer
>>  doesn't have unsaved changes?
>
> This should be made customizable.

Yes. Also provide flush- or refresh-cache functions.

>> Certain other programs might have a project and all its files loaded,
>> but still display only a few of them as the ones the user is editing
>> (probably using tabs). Should Emacs have a similar distinction,
>> between loaded and open buffers?
>
> I think so.  Each buffer should maintain a list of all the applications
> that visited its file and all operations on buffer lists should be aware
> of it.

That makes sense.

One use case would be "goto buffer in project", that completes only on
buffers visiting files that belong to a project (in the xref "project"
sense). The buffers could have a property "projects" that was a list of
some project name/id/root directory.

Another use case (that I've actually implemented differently) is
deleting all buffers that were fetched by a front end to git as part of
reviewing a commit (they display old versions of files). My current
implementation relies on a list of such buffers maintained by the front
end, but it would be better if there were some standard tag on the
buffer. I don't like to close such buffers immediately after the first
ediff session, because I often want to come back to them for some reason.

Functions for deleting some buffers based on such buffer properties
would also be useful. In the projects case, the buffer would not
actually be deleted until there was only one project property left.

-- 
-- Stephe

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 16:26:45 +0300
> 
> What I'm talking about, are file buffers that xref commands need to 
> open, to read their contents. Such as xref-find-references; see, it can 
> use Grep to find textual matches, but it cannot verify that the 
> beginning and the end of the match are symbol boundaries without using 
> the corresponding major mode, its syntax table, *and* its 
> syntax-propertize-function. Basically, the only sane solution is to open 
> each buffer and verify.
> 
> The question is, should they be killed afterwards.

There are arguments either way, AFAIU, so perhaps make this a
defcustom.

> > I see no reason why not.  Some commands already save the buffer
> > (perhaps after asking for confirmation) to have the file on disk
> > consistent with what Emacs has, and this is no different.
> 
> Hmm yes. I wonder if it'll be appropriate to simply call 
> (auto-revert-handler) if each buffer visited for this purpose.

Might be reasonable.  Again, if you are in doubt, let a user option
override that.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > N > 1 won't help too much. I'm talking about a situation like
 > xref-find-references followed by xref-query-replace. It already gives
 > a performance hit (if there are matches in a lot of different
 > files). And xref-find-references only opens each buffer once already.

I'm interested in finding a proper solution of eldoc for C files.  The
current ones are either based on using the preprocessor (which sucks
IMHO) or tags (which I'd prefer).  In the latter case you have to visit
the file referenced by the tag, go to the reference's target and look at
the function's signature.  After that, the corresponding buffer is
usually left open so using that functionality will leave lots of such
buffers around.

N > 1 means that such a buffer has been visited at least twice so
chances are that it will be visited again in the near future and it
might make sense to keep it around for some time.

 > What kind of cache?

The "cache" would consist of the buffers exclusively visited by the
application (xref, eldoc, ...) in the background.

 >> [Automatic reverting] should be made customizable.
 >

 > I suppose a user option might govern whether xref will prompt before
 > reverting. But then, will it prompt for each file? There might be a
 > lot.

That's easy.  Add an option for each reasonable way to handle this and
time will tell.

 >> I think so.  Each buffer should maintain a list of all the applications
 >> that visited its file and all operations on buffer lists should be aware
 >> of it.
 >
 > You might want to elaborate. This seems to go further than my suggestion.

Buffer are eventually killed explicitly or cleaned up in some way.  In
the second case we have to know whether the buffer was visited by the
user (so a flag is needed anyway) but maybe also whether an application
might be interested in keeping the buffer around for some time more.  So
the clean-up routine would query any application that loaded this buffer
in the background whether it agrees to have it killed or not.  How this
is implemented is merely a matter of taste.

martin

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > Better for xref to cache the info it gathered from the buffer (assuming
 > that's significantly smaller, and faster to access, than the buffer).

Obviously.  But I'm afraid that's not always feasible.

 > Functions for deleting some buffers based on such buffer properties
 > would also be useful. In the projects case, the buffer would not
 > actually be deleted until there was only one project property left.

We'd need a smart version of midnight's `clean-buffer-list'.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 05:07 PM, Eli Zaretskii wrote:

>> The question is, should they be killed afterwards.
>
> There are arguments either way, AFAIU, so perhaps make this a
> defcustom.

That's a cop-out answer. Whichever value is the default, will influence 
the implementation (if the buffers are killed right away, they should be 
opened in a special way, otherwise, they must be opened normally, with 
all applicable minor modes).

If the buffers are killed, xref-query-replace will need to account for 
that, and not open too many buffers at the same time. I'm most likely 
missing other issues now.

> Might be reasonable.  Again, if you are in doubt, let a user option
> override that.

Please, let's stop with "implement every behavior a user might want". 
There are better things for a developer to spend time on. Unless a 
particular behavior is requested, a user option is premature.

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 17:24:58 +0300
> 
> On 07/25/2015 05:07 PM, Eli Zaretskii wrote:
> 
> >> The question is, should they be killed afterwards.
> >
> > There are arguments either way, AFAIU, so perhaps make this a
> > defcustom.
> 
> That's a cop-out answer.

Is it ever possible to have a discussion with you without feeling
castigated for my views and opinions?  You asked for opinions, so
please accept them benevolently, even if you don't like them.

> Whichever value is the default, will influence the implementation
> (if the buffers are killed right away, they should be opened in a
> special way, otherwise, they must be opened normally, with all
> applicable minor modes).

My vote is for the default that keeps the buffers.  I see nothing
wrong with having a lot of buffers in an Emacs session.  Personally, I
regularly walk through all my buffers and kill those I no longer need,
because I use desktop.el to restore my sessions, which would otherwise
grow indefinitely.  It's no big deal.  It is much less of a big deal
if sessions are not restored.

> If the buffers are killed, xref-query-replace will need to account for 
> that, and not open too many buffers at the same time.

I don't see why that would be true.  Please elaborate.

> > Might be reasonable.  Again, if you are in doubt, let a user option
> > override that.
> 
> Please, let's stop with "implement every behavior a user might want". 
> There are better things for a developer to spend time on. Unless a 
> particular behavior is requested, a user option is premature.

You asked for an opinion, that's mine.  IME, good engineering
anticipates such requests before they are voiced.  If you can only
resolve a controversy applying your personal preferences, it's a clear
sign that someone, somewhere will be unhappy about it.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 05:12 PM, martin rudalics wrote:

> I'm interested in finding a proper solution of eldoc for C files.  The
> current ones are either based on using the preprocessor (which sucks
> IMHO) or tags (which I'd prefer).  In the latter case you have to visit
> the file referenced by the tag, go to the reference's target and look at
> the function's signature.  After that, the corresponding buffer is
> usually left open so using that functionality will leave lots of such
> buffers around.

A distinction between loaded and open buffers might help with that as well.

> N > 1 means that such a buffer has been visited at least twice so
> chances are that it will be visited again in the near future and it
> might make sense to keep it around for some time.

That may be useful for your eldoc problem, but not so much for xref. In 
the latter case, the issue is not of opening a particular buffer many 
times, but of opening many buffers 2 or 3 times.

> The "cache" would consist of the buffers exclusively visited by the
> application (xref, eldoc, ...) in the background.

A cache is a set of buffers. Okay. How is it implemented? Does 
switch-to-buffer know they're in the cache? Does it take them out of the 
cache?

> That's easy.  Add an option for each reasonable way to handle this and
> time will tell.

If I add an option for *every* reasonable way to handle this (which is 
already a tall order), what could time tell afterwards?

> In
> the second case we have to know whether the buffer was visited by the
> user (so a flag is needed anyway) but maybe also whether an application
> might be interested in keeping the buffer around for some time more.

Flag - yes.

I think a list of applications is going too far. As long as the cleanup 
is only performed when Emacs is idle (and has been idle for a while, 
probably), I think we could could on those applications to reopen 
buffers as needed. For special faces, we could have a second flag, "do 
not clean up".

Whether doing cleanup is at all necessary is a matter of opinion, but I 
guess it might be useful to conserve memory. Maybe that subsystem could 
be smart enough to only trigger cleanup when the buffers exceed a 
certain amount of memory, and tries to kill the oldest opened buffers first.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 05:36 PM, Eli Zaretskii wrote:

> Is it ever possible to have a discussion with you without feeling
> castigated for my views and opinions?  You asked for opinions, so
> please accept them benevolently, even if you don't like them.

I'm sorry, it's just I've seen the suggestion to handle every possible 
behavior too many times, from different people in the Emacs community.

What I'm asking is which behavior people here will actually prefer, 
after weighing the tradeoffs. You've written that below, and thank you 
for that.

> My vote is for the default that keeps the buffers.  I see nothing
> wrong with having a lot of buffers in an Emacs session.  Personally, I
> regularly walk through all my buffers and kill those I no longer need,
> because I use desktop.el to restore my sessions, which would otherwise
> grow indefinitely.  It's no big deal.  It is much less of a big deal
> if sessions are not restored.

That's what I'm also inclining towards now. However, the list of buffers 
automatically opened by xref might be too big for you (or me) to clean 
up manually (if you search for occurrences of some common word, for 
example). After that, I'm not sure what a user might do to undo that. 
(mapc #'kill-buffer (buffer-list)) might be the only option.

>> If the buffers are killed, xref-query-replace will need to account for
>> that, and not open too many buffers at the same time.
>
> I don't see why that would be true.  Please elaborate.

To be clear, I mean it'll need not to open all buffers in advance.

At the moment, it opens all files, and puts markers around each match, 
because if there are several matches on the same line, after the 
replacement is performed on the first of them, the rest will become 
"stale", because the recorded column information will become outdated.

In order not to spend too much time on opening buffers in advance, it'll 
need to open only one or two buffers at the same time, before moving to 
the next match to replace. That will increase the performance, as 
perceived by the user.

> You asked for an opinion, that's mine.  IME, good engineering
> anticipates such requests before they are voiced.  If you can only
> resolve a controversy applying your personal preferences, it's a clear
> sign that someone, somewhere will be unhappy about it.

We can't really support *every* preference. And some of them can even be 
incompatible with architectural decisions providing best tradeoffs.

IMO, it's better to decide on the default workflow that we believe to be 
useful to most, and only after that add variations to that behavior.

Ideally, we might even find a solution that satisfies the needs of both 
kinds of users, which would otherwise ask for different behaviors, 
tailored for their needs. In this instance, a "buffer cache" might be 
that solution, if it'll provide both performance benefits of not having 
to open the buffers several times, as well as the clutter management 
benefits, if the buffers are flagged appropriately.

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 18:09:33 +0300
> Cc: emacs-devel@gnu.org
> 
> > My vote is for the default that keeps the buffers.  I see nothing
> > wrong with having a lot of buffers in an Emacs session.  Personally, I
> > regularly walk through all my buffers and kill those I no longer need,
> > because I use desktop.el to restore my sessions, which would otherwise
> > grow indefinitely.  It's no big deal.  It is much less of a big deal
> > if sessions are not restored.
> 
> That's what I'm also inclining towards now. However, the list of buffers 
> automatically opened by xref might be too big for you (or me) to clean 
> up manually (if you search for occurrences of some common word, for 
> example). After that, I'm not sure what a user might do to undo that. 
> (mapc #'kill-buffer (buffer-list)) might be the only option.

Then perhaps there should be a command to kill all those buffers
created by xref for these purposes.

OTOH, look at the example of info.el's info-apropos: it doesn't keep
around the Info manuals it visits.  So this is also a reasonable
behavior we have elsewhere.

> We can't really support *every* preference. And some of them can even be 
> incompatible with architectural decisions providing best tradeoffs.
> 
> IMO, it's better to decide on the default workflow that we believe to be 
> useful to most, and only after that add variations to that behavior.

If you can identify that now, fine.  If not, there's nothing wrong
with changing the default later, as long as non-default behavior is
available as an option.

> Ideally, we might even find a solution that satisfies the needs of both 
> kinds of users, which would otherwise ask for different behaviors, 
> tailored for their needs.

Again, if that can be identified, fine.  But if not, having several
optional behaviors is the Emacs'y way of not forcing users into
Procrustean situations.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 04:47 PM, Stephen Leake wrote:

> Better for xref to cache the info it gathered from the buffer (assuming
> that's significantly smaller, and faster to access, than the buffer).

Whatever the format of that info, it probably won't work between 
invocations of xref commands with slightly different inputs.

And xref-query-replace is currently implemented via the same mechanism 
as what query-replace-regexp uses, which means it needs to visit every 
buffer in a normal way anyway.

>> This should be made customizable.
>
> Yes. Also provide flush- or refresh-cache functions.

These seem to be separate issues. We might ask the user whether to 
revert an "open" buffer (not one in the cache).

And regarding refresh-cache, I'd rather each application that needs a 
buffer from the cache reverted only the buffers it needs. Maybe the new 
function that opens and cached that kind of buffers (like 
find-file-in-cache), would even do that automatically.

> One use case would be "goto buffer in project", that completes only on
> buffers visiting files that belong to a project (in the xref "project"
> sense). The buffers could have a property "projects" that was a list of
> some project name/id/root directory.

So far, this could be implemented via an arbitrary buffer-local 
variable. Or just simply looping through (buffer-list) and checking 
buffer-file-name against the project-directories; that should be fast 
enough, with reasonable number of open buffers.

> Another use case (that I've actually implemented differently) is
> deleting all buffers that were fetched by a front end to git as part of
> reviewing a commit (they display old versions of files).

Don't you think vc-revision-other-window provides a reasonable solution, 
by adding a suffix to a buffer's name? If that's not enough (for 
instance, if we want to delete them all later, and matching names to 
.*\.~.+~ is not precise enough, we could use a buffer-local var as well).

> Functions for deleting some buffers based on such buffer properties
> would also be useful. In the projects case, the buffer would not
> actually be deleted until there was only one project property left.

What if the user opened it using switch-to-buffer, not using "goto 
buffer in project"? Then it wouldn't have this property set, right?

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > A distinction between loaded and open buffers might help with that as well.

I have thought about this before and am quite sure it's needed.  I don't
want to see such buffers in the buffer list and don't want to
accidentally switch to them.  Actually, it's what keeping me to use the
preprocessor solution at the moment.

 >> N > 1 means that such a buffer has been visited at least twice so
 >> chances are that it will be visited again in the near future and it
 >> might make sense to keep it around for some time.
 >

 > That may be useful for your eldoc problem, but not so much for
 > xref. In the latter case, the issue is not of opening a particular
 > buffer many times, but of opening many buffers 2 or 3 times.

In the eldoc case this might happen as well.  Over the time buffers pile
up.  And I practically never want to switch to them (which is the only
detail I can supply so far).

 > A cache is a set of buffers. Okay. How is it implemented? Does
 > switch-to-buffer know they're in the cache? Does it take them out of
 > the cache?

All file visiting functions must be aware of it and take it out.  As far
as implementation is concerned we have the usual two choices: Pass an
extra argument (clean but costly) or bind some variable around the calls
that visit the file in the background which is probably the way to go.

 > If I add an option for *every* reasonable way to handle this (which is
 > already a tall order), what could time tell afterwards?

The "*every* reasonable way" solution applies only where such
customizations are cheap to implement and the behavior is not easily
predictable.  As fas as querying the user for whether to revert buffers
I see no problem in doing that.  Personally, I'd never opt for a
solution where I'm asked for each file individually.  But the options
"never auto-revert", "always auto-revert without asking" and "if at
least one file is stale ask whether to auto-revert all" seem all pretty
reasonable to me and I would probably try them out.

 > I think a list of applications is going too far. As long as the
 > cleanup is only performed when Emacs is idle (and has been idle for a
 > while, probably), I think we could could on those applications to
 > reopen buffers as needed. For special faces, we could have a second
 > flag, "do not clean up".

OK with me.  My main concern is that such buffers don't show up in my
buffer lists and so one flag is sufficient.

 > Whether doing cleanup is at all necessary is a matter of opinion, but
 > I guess it might be useful to conserve memory. Maybe that subsystem
 > could be smart enough to only trigger cleanup when the buffers exceed
 > a certain amount of memory, and tries to kill the oldest opened
 > buffers first.

The least recently accessed ones, I suppose.  So somewhere we might want
to maintain a timestamp of the least recently background access of these
buffers.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 06:27 PM, Eli Zaretskii wrote:

> Then perhaps there should be a command to kill all those buffers
> created by xref for these purposes.

Suppose we tag them with a buffer-local variable.

What if the use switches to one of these buffers on their own accord, 
with find-file or switch-to-buffer? They might not be aware that that 
buffer was opened by xref, and would be surprised to see M-x 
xref-kill-temporary-buffers kill that buffer as well.

> OTOH, look at the example of info.el's info-apropos: it doesn't keep
> around the Info manuals it visits.  So this is also a reasonable
> behavior we have elsewhere.

It keeps a cache of search results (and initiates full search for any 
input not in cached list). That's nice, but probably solves a different 
problem.

> Again, if that can be identified, fine.  But if not, having several
> optional behaviors is the Emacs'y way of not forcing users into
> Procrustean situations.

Of course. But first we should try.

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 19:07:04 +0300
> 
> On 07/25/2015 06:27 PM, Eli Zaretskii wrote:
> 
> > Then perhaps there should be a command to kill all those buffers
> > created by xref for these purposes.
> 
> Suppose we tag them with a buffer-local variable.
> 
> What if the use switches to one of these buffers on their own accord, 
> with find-file or switch-to-buffer? They might not be aware that that 
> buffer was opened by xref, and would be surprised to see M-x 
> xref-kill-temporary-buffers kill that buffer as well.

You could detect this using buffer-modified-tick, no?  You could also
refrain from killing any of these buffers displayed in some window, or
present in the relevant history lists.

> > OTOH, look at the example of info.el's info-apropos: it doesn't keep
> > around the Info manuals it visits.  So this is also a reasonable
> > behavior we have elsewhere.
> 
> It keeps a cache of search results (and initiates full search for any 
> input not in cached list). That's nice, but probably solves a different 
> problem.

That's not what I meant.  What I meant was that the manuals are not
available once the command finishes, although its quite clear the
command had to visit each and every one of them.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 07:15 PM, Eli Zaretskii wrote:

> You could detect this using buffer-modified-tick, no?  You could also
> refrain from killing any of these buffers displayed in some window, or
> present in the relevant history lists.

The tick doesn't change if the user hasn't modified the contents. And if 
the buffer is not visible just now, that doesn't mean that it's 
unwanted. Both useful heuristics, but still less than ideal, IMO.

> That's not what I meant.  What I meant was that the manuals are not
> available once the command finishes, although its quite clear the
> command had to visit each and every one of them.

Yes, both xref-collect-references and xref-collect-matches delete the 
newly opened buffers in the end as well. Unfortunately, an average 
xref-find-references call takes a longer than info-apropos, IME, in 
large part because it has to (re)open the files.

Re: xref and leaving "temporary" buffers open

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 25 Jul 2015 19:28:10 +0300
> 
> On 07/25/2015 07:15 PM, Eli Zaretskii wrote:
> 
> > You could detect this using buffer-modified-tick, no?  You could also
> > refrain from killing any of these buffers displayed in some window, or
> > present in the relevant history lists.
> 
> The tick doesn't change if the user hasn't modified the contents. And if 
> the buffer is not visible just now, that doesn't mean that it's 
> unwanted. Both useful heuristics, but still less than ideal, IMO.

I meant 3 devices, not 2.

And I think all of them together go a long way towards eliminating
accidental deletion of a buffer the user would like to keep.  Plus,
they can always refrain from invoking the command, if they are afraid
of losing buffers.  And if worse comes to worst, the file can be
visited again.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 07:38 PM, Eli Zaretskii wrote:

> I meant 3 devices, not 2.

I swept the third under the "not visible right now". People delete 
windows, we don't promise any semantics of tying buffers to windows.

> And I think all of them together go a long way towards eliminating
> accidental deletion of a buffer the user would like to keep.

Maybe so. But another concern of mine is that every facility that opens 
buffer in this way, will need to offer its own xxx-delete-opened-buffers 
command. And if we try to consolidate those, the result can't be too far 
from actually having the distinction of open vs. loaded buffers, like 
the cache facility proposed here.

I'll try my hand at the implementation of the latter, and we'll see if 
something unexpected comes up.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > ... the distinction of open vs. loaded buffers, ...

Please let's try to find a better notation.  Both "open" and "loaded"
are very confusing.  "Fore-/background" buffers, "ex-/implicit" buffers
are not good either.  Maybe someone has an idea?

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 09:17 PM, martin rudalics wrote:
>  > ... the distinction of open vs. loaded buffers, ...
>
> Please let's try to find a better notation.  Both "open" and "loaded"
> are very confusing.  "Fore-/background" buffers, "ex-/implicit" buffers
> are not good either.

I'm also worried about that.

Temporary? Might be not too good for the ones we'll want to keep around 
indefinitely (although I've yet to see a solid use case for that).

Hidden? I think this term is already in use for the buffers whose names 
start with a space. At least there's an index entry for "hidden buffers" 
in the manual, pointing to (elisp)Buffer names.

But there don't seem to be a variable or a function in the core that has 
both "hidden" and "buffer" in its name, so it could be a good option.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/25/2015 06:53 PM, martin rudalics wrote:

> All file visiting functions must be aware of it and take it out.  As far
> as implementation is concerned we have the usual two choices: Pass an
> extra argument (clean but costly) or bind some variable around the calls
> that visit the file in the background which is probably the way to go.

How about this:

(defvar-local buffer-hidden nil
   "When non-nil, this buffer is not returned by `buffer-list',
unless passed a non-nil INCLUDE-HIDDEN argument.")

(defvar-local buffer-hidden-last-access nil
"The time when this buffer was last accessed.")

(defun buffer-list (&optional frame include-hidden)

(defvar find-file-hidden nil
   "When non-nil, new file buffers are marked as hidden.
`find-file-noselect', when creating a new buffer, will set
`buffer-hidden' to t, and `buffer-hidden-last-access' to the
current time.
When it's nil, `find-file-noselect' will set `buffer-hidden' and
`buffer-hidden-last-accessed' to nil in the buffer it returns.")

`find-file-noselect' might also become a problem with large amounts of 
hidden buffers, because `find-buffer-visiting' iterates through all 
buffers (and thus is O(N)). Although we should see whether it's an 
actual problem in practice.

> The "*every* reasonable way" solution applies only where such
> customizations are cheap to implement and the behavior is not easily
> predictable.  As fas as querying the user for whether to revert buffers
> I see no problem in doing that.  Personally, I'd never opt for a
> solution where I'm asked for each file individually.  But the options
> "never auto-revert", "always auto-revert without asking" and "if at
> least one file is stale ask whether to auto-revert all" seem all pretty
> reasonable to me and I would probably try them out.

"never auto-revert" sounds like "always ask", but without the option 
"yes", because xref should abort in this case rather than return wrong 
results, and the user would have to be notified of that anyway. So that 
makes for two reasonable values, I think: "always revert" and "ask 
before reverting".

As long as a vague "Some buffers are out of date. Revert?" will suffice, 
that won't be a problem. Naming the first out-of-date buffer should also 
be always possible, but whether we'll be able to name them all, depends 
on the implementation of a given command.

> The least recently accessed ones, I suppose.  So somewhere we might want
> to maintain a timestamp of the least recently background access of these
> buffers.

The time when `find-file-noselect' was last called seems like a good 
approximation of the most recent access time.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > How about this:
 >
 > (defvar-local buffer-hidden nil
 >    "When non-nil, this buffer is not returned by `buffer-list',
 > unless passed a non-nil INCLUDE-HIDDEN argument.")

Good (if you fix the doc-string).  We should then rewrite and change the
name of BUFFER_HIDDEN_P to avoid confusions.

 > (defvar-local buffer-hidden-last-access nil
 > "The time when this buffer was last accessed.")

Good.  I'm not yet sure whether there could be problems with dead
buffers but I don't think so.

 > (defun buffer-list (&optional frame include-hidden)

Good.  Maybe the second argument could be used to not include buffers
starting with a space either.

 > (defvar find-file-hidden nil
 >    "When non-nil, new file buffers are marked as hidden.
 > `find-file-noselect', when creating a new buffer, will set
 > `buffer-hidden' to t, and `buffer-hidden-last-access' to the
 > current time.
 > When it's nil, `find-file-noselect' will set `buffer-hidden' and
 > `buffer-hidden-last-accessed' to nil in the buffer it returns.")

Good.  This should be the only hairy part to implement.

 > `find-file-noselect' might also become a problem with large amounts of
 > hidden buffers, because `find-buffer-visiting' iterates through all
 > buffers (and thus is O(N)). Although we should see whether it's an
 > actual problem in practice.

We can always prune the list when it gets too large.

 > "never auto-revert" sounds like "always ask", but without the option
 > "yes", because xref should abort in this case rather than return wrong
 > results,

Does this remark only apply to `xref-query-replace'?  For a pure `xref'
search we should allow stale contents IMHO.  And does xref always check
the file modification data?  Does it re-run etags?

 > The time when `find-file-noselect' was last called seems like a good
 > approximation of the most recent access time.

Agreed.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/26/2015 02:27 PM, martin rudalics wrote:

> Good (if you fix the doc-string).  We should then rewrite and change the
> name of BUFFER_HIDDEN_P to avoid confusions.

Indeed. But the new macro, seem like, should return true for either 
ephemeral or hidden buffers.

>  > (defvar-local buffer-hidden-last-access nil
>  > "The time when this buffer was last accessed.")
>
> Good.  I'm not yet sure whether there could be problems with dead
> buffers but I don't think so.

Dead as a result of the cleanup routine (to be implemented later)? 
Probably, but we can try to minimize the odds of that.

>  > (defun buffer-list (&optional frame include-hidden)
>
> Good.  Maybe the second argument could be used to not include buffers
> starting with a space either.

I don't know about that. Sometimes you really want to switch to one of 
those buffers while debugging. But there shouldn't be anything 
interesting in the proposed hidden file buffers.

>  > (defvar find-file-hidden nil
>  > ...
>
> Good.  This should be the only hairy part to implement.

It doesn't sound too hard. The buffer-list change looks more complicated 
to me, simply because it's in C.

By the way, I wonder if we're going to discourage, or somehow specially 
handle, calling find-file-noselect with RAWFILE t.

> We can always prune the list when it gets too large.

We'll see. There might be performance benefits to keeping it pretty big. 
Computers have a lot of memory nowadays.

> Does this remark only apply to `xref-query-replace'?  For a pure `xref'
> search we should allow stale contents IMHO.  And does xref always check
> the file modification data?  Does it re-run etags?

That depends on the implementation of a particular command. The ones 
that dispatch to etags, will consult the (possibly out-of-date) TAGS 
file, and etags-goto-tag-location will make adjustments.

Same for Elisp xref backend.

However, the search-based commands (find-references, find-matches) might 
require up-to-date contents.

The only sane alternative, I think, is in addition to running Grep 
iterate through all open buffers and `re-search-forward' through them. 
...or only the stale ones? I'm open to further investigation in this 
direction.

But if we augment Grep with re-search in Elisp, it will complicate the 
handling of stale buffers in xref-query-replace. We won't be able to ask 
to revert all stale buffers in advance, because some matches will be 
from stale buffers. And then, some buffers might have further change.

I guess whether we want to go in this direction, depends on whether we 
think re-search-forward's performance is on the same order of magnitude 
as Grep. IOW, whether having lots of stale buffers might incur visible 
performance penalty.

Re: xref and leaving "temporary" buffers open

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/25/2015 06:53 PM, martin rudalics wrote:
>
>> All file visiting functions must be aware of it and take it out.  As far
>> as implementation is concerned we have the usual two choices: Pass an
>> extra argument (clean but costly) or bind some variable around the calls
>> that visit the file in the background which is probably the way to go.
>
> How about this:
>
> (defvar-local buffer-hidden nil
>   "When non-nil, this buffer is not returned by `buffer-list',
> unless passed a non-nil INCLUDE-HIDDEN argument.")

That means you have to use set-buffer to check for the buffer-local
variable.

Can't we set a property list on the buffer itself? That would be more
efficient to search on.

-- 
-- Stephe

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/26/2015 10:00 PM, Stephen Leake wrote:

> Can't we set a property list on the buffer itself? That would be more
> efficient to search on.

Can we? A buffer is not a symbol.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > Dead as a result of the cleanup routine (to be implemented later)?
 > Probably, but we can try to minimize the odds of that.

A buffer is atomically dead when its name is set to nil.  So
conceptually there should never be any problems.  Yet, there are all
those checks whether a buffer has recursively become dead in
`kill-buffer' which always keep me wondering ...

 > By the way, I wonder if we're going to discourage, or somehow
 > specially handle, calling find-file-noselect with RAWFILE t.

This is one of the problems I see.  Another issue is that of name
management.  Would a hidden buffer's name be uniquified?

martin

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 >> (defvar-local buffer-hidden nil
 >>    "When non-nil, this buffer is not returned by `buffer-list',
 >> unless passed a non-nil INCLUDE-HIDDEN argument.")
 >
 > That means you have to use set-buffer to check for the buffer-local
 > variable.

Do you mean `buffer-local-value' wouldn't work for that variable?

martin

Re: xref and leaving "temporary" buffers open

[Nix]

On 25 Jul 2015, Dmitry Gutov outgrape:

> On 07/25/2015 09:17 PM, martin rudalics wrote:
>>  > ... the distinction of open vs. loaded buffers, ...
>>
>> Please let's try to find a better notation.  Both "open" and "loaded"
>> are very confusing.  "Fore-/background" buffers, "ex-/implicit" buffers
>> are not good either.
>
> I'm also worried about that.
>
> Temporary? Might be not too good for the ones we'll want to keep
> around indefinitely (although I've yet to see a solid use case for
> that).

FWIW, there are existing packages which automatically kill buffers that
weren't modified for a certain period of time and are no longer
displayed in any frame (using the buffer-modified-tick). If you combine
that with a modification such that the comparatively limited set of
commands that explicitly open buffers on user request set a buffer-local
'persistent-buffer' variable in them, you can then arrange to
automatically erase all non-persistent buffers without touching the
packages that open such buffers. (However, this would tend to break
third-party modes that open lots of extra buffers -- but how many of
those are there? It's mostly things, like Gnus, that implement whole
multi-buffer applications inside Emacs, I think. There seem to be
arguments in favour both of the local variable to mark temporariness and
the local variable to mark *non*-temporariness...)

-- 
NULL && (void)

Re: xref and leaving "temporary" buffers open

[Stephen Leake]

martin rudalics <rudalics@gmx.at> writes:

>>> (defvar-local buffer-hidden nil
>>>    "When non-nil, this buffer is not returned by `buffer-list',
>>> unless passed a non-nil INCLUDE-HIDDEN argument.")
>>
>> That means you have to use set-buffer to check for the buffer-local
>> variable.
>
> Do you mean `buffer-local-value' wouldn't work for that variable?

Ah, I did not know about that function. 

-- 
-- Stephe

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/27/2015 07:02 PM, martin rudalics wrote:

> A buffer is atomically dead when its name is set to nil.  So
> conceptually there should never be any problems.  Yet, there are all
> those checks whether a buffer has recursively become dead in
> `kill-buffer' which always keep me wondering ...

We'll see, I suppose.

>  > By the way, I wonder if we're going to discourage, or somehow
>  > specially handle, calling find-file-noselect with RAWFILE t.
>
> This is one of the problems I see.  Another issue is that of name
> management.  Would a hidden buffer's name be uniquified?

Probably. Different buffers with the same name are not allowed, right?

Something else might be worth doing is emulating 
semantic-find-file-noselect in some generic way.

It's hard to draw a line, but maybe we don't need to know the VC status 
of any hidden buffer, or enable any (?) of the minor modes in it. But 
the point is to speed up opening the buffers.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 >> This is one of the problems I see.  Another issue is that of name
 >> management.  Would a hidden buffer's name be uniquified?
 >
 > Probably. Different buffers with the same name are not allowed, right?

What I meant are clients of uniquify.el.  Its users might get a surprise
while working on a buffer visting file ~/foo/bar and xref visits a file
~/bar/bar.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/27/2015 08:34 PM, Nix wrote:

> FWIW, there are existing packages which automatically kill buffers that
> weren't modified for a certain period of time and are no longer
> displayed in any frame (using the buffer-modified-tick). If you combine
> that with a modification such that the comparatively limited set of
> commands that explicitly open buffers on user request set a buffer-local
> 'persistent-buffer' variable in them, you can then arrange to
> automatically erase all non-persistent buffers without touching the
> packages that open such buffers.

I think we'd rather move into the direction of other packages using the 
"hidden buffers" facility, thus indicating that their buffers can 
actually be cleaned up.

Cleaning those up will cause issues already: I'm sure someone, somewhere 
will use find-file-hidden, don't mark the returned buffer as persistent, 
save a reference to it, and then try to use it later.

> (However, this would tend to break
> third-party modes that open lots of extra buffers -- but how many of
> those are there? It's mostly things, like Gnus, that implement whole
> multi-buffer applications inside Emacs, I think. There seem to be
> arguments in favour both of the local variable to mark temporariness and
> the local variable to mark *non*-temporariness...)

A variable to mark temporariness might help a lot, actually. As long as 
it very obvious to the caller, so that we don't end up with heaps of 
hidden buffers, without any of them marked as temporary.

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/28/2015 04:16 PM, martin rudalics wrote:

> What I meant are clients of uniquify.el.  Its users might get a surprise
> while working on a buffer visting file ~/foo/bar and xref visits a file
> ~/bar/bar.

If the file is already open, find-file-hidden should behave the same as 
find-file-noselect.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > If the file is already open, find-file-hidden should behave the same
 > as find-file-noselect.

I suppose you mean that `find-file-hidden' should always behave the same
as `find-file-noselect' but for setting the hidden flag.  Just that that
means that hidden buffers will have some not-so-hidden impact.  BTW it
might probably make sense to use a hash table to find hidden buffers
from their visited file names instead of relying on `get-file-buffer'.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/28/2015 06:10 PM, martin rudalics wrote:

> I suppose you mean that `find-file-hidden' should always behave the same
> as `find-file-noselect' but for setting the hidden flag.

In this case it wouldn't set the flag either. If the file is already 
open, just return its buffer. The caller can then proceed to read its 
contents.

> BTW it
> might probably make sense to use a hash table to find hidden buffers
> from their visited file names instead of relying on `get-file-buffer'.

I suppose. That's the same issue I mentioned WRT find-buffer-visiting, 
right?

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 >> I suppose you mean that `find-file-hidden' should always behave the same
 >> as `find-file-noselect' but for setting the hidden flag.
 >
 > In this case it wouldn't set the flag either. If the file is already open, just return its buffer. The caller can then proceed to read its contents.

Suppose a user is visiting a file ~/foo/bar in a buffer called bar and
has set `uniquify-buffer-name-style' to 'post-forward-angle-brackets.
Now xref visits a file ~/bar/bar which the user has _not_ visited yet.
In this case uniquify will rename the first buffer to bar<foo> and name
the second buffer bar<bar>.  The user will notice that her buffer has
been renamed although she has no idea that a buffer bar<bar> exists.

 > I suppose. That's the same issue I mentioned WRT find-buffer-visiting, right?

Probably.  I forgot.  How many files do we need to make this investment
pay off?

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/28/2015 06:40 PM, martin rudalics wrote:

> In this case uniquify will rename the first buffer to bar<foo> and name
> the second buffer bar<bar>.  The user will notice that her buffer has
> been renamed although she has no idea that a buffer bar<bar> exists.

Yes, it looks like a problem. uniquify is a prime candidate for delayed 
enabling of features. Ideally, we'd avoid uniquifying a hidden buffer, 
but perform uniquification as soon as it's no longer hidden.

>  > I suppose. That's the same issue I mentioned WRT
> find-buffer-visiting, right?
>
> Probably.  I forgot.  How many files do we need to make this investment
> pay off?

Not too many. The exact numbers depend on each file. But e.g.

(with-current-buffer (find-file-noselect "src/xdisp.c")
   (syntax-propertize (point-max)))

takes whooping 400ms the first time.

Re: xref and leaving "temporary" buffers open

[Stefan Monnier]

> It's hard to draw a line, but maybe we don't need to know the VC status of
> any hidden buffer, or enable any (?) of the minor modes in it. But the point
> is to speed up opening the buffers.

I can see something like this working:

- set a buffer-local variable "buffer-file-incomplete" to a function
  which performs the missing setup (e.g. set up VC status and stuff).
- ignore buffers with this var set to a non-nil value in things like
  buffer completion.
- kill those buffers every once in a while.

But I'm not sure the speed up will be ground breaking if we need to call
`syntax-propertize' anyway.


        Stefan

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/29/2015 02:48 AM, Stefan Monnier wrote:

> I can see something like this working:

Right. There were more specific proposals in this thread. You might want 
to comment on those.

> - set a buffer-local variable "buffer-file-incomplete" to a function
>    which performs the missing setup (e.g. set up VC status and stuff).
> - ignore buffers with this var set to a non-nil value in things like
>    buffer completion.

So, you'd limit the changes to the higher level, like 
read-buffer-to-switch and internal-complete-buffer-except?

What about other interfaces we have for buffer switching? 
ido-make-buffer-list-1 calls buffer-list (that one's easy to change, 
since it's in the core), iflipb calls buffer-list, helm-buffers-list 
calls buffer-list.

> - kill those buffers every once in a while.
>
> But I'm not sure the speed up will be ground breaking if we need to call
> `syntax-propertize' anyway.

As long as the buffer-file-incomplete function doesn't change the buffer 
contents, at least we won't have to call syntax-propertize multiple times.

And depending on the file size, the overhead from vc-find-file-hook may 
become noticeable, too. In (find-file-noselect ".gitignore"), it takes 
more than half of the time (15 out of 25 ms), here.

Re: xref and leaving "temporary" buffers open

[Nix]

On 28 Jul 2015, Dmitry Gutov said:

> On 07/27/2015 08:34 PM, Nix wrote:
>> (However, this would tend to break
>> third-party modes that open lots of extra buffers -- but how many of
>> those are there? It's mostly things, like Gnus, that implement whole
>> multi-buffer applications inside Emacs, I think. There seem to be
>> arguments in favour both of the local variable to mark temporariness and
>> the local variable to mark *non*-temporariness...)
>
> A variable to mark temporariness might help a lot, actually. As long as it very obvious to the caller, so that we don't end up with
> heaps of hidden buffers, without any of them marked as temporary.

Yeah. At the moment tempbuf.el works quite well for those of us who like
their buffer lists neat and tidy, but it is necessarily a kludge. I
applaud the search for something better.

(I have actually in the past wondered if it would be nice to have
buffers structured into directories somehow so you could normally ignore
unrelated buffers, but have come to the conclusion that better
navigation is the answer. For me, GNU GLOBAL-driven semantic.el
navigation via CEDET and Icicles-based switch-by-content has sufficed to
control even Emacsen with thousands of buffers open... though this is
mostly Semantic's fault, since it opens files to parse them and often
forgets to close them again. This is something that would definitely
benefit from a temporary-buffer marking, since you certainly can't use
the mode of such buffers as a hint.)

-- 
NULL && (void)

Re: xref and leaving "temporary" buffers open

[Stefan Monnier]

> Right. There were more specific proposals in this thread. You might want to
> comment on those.

I'm afraid I don't have the time to read all those messages.

>> - set a buffer-local variable "buffer-file-incomplete" to a function
>> which performs the missing setup (e.g. set up VC status and stuff).
>> - ignore buffers with this var set to a non-nil value in things like
>> buffer completion.
> So, you'd limit the changes to the higher level, like read-buffer-to-switch
> and internal-complete-buffer-except?

Yes.

> What about other interfaces we have for buffer switching?
> ido-make-buffer-list-1 calls buffer-list (that one's easy to change, since
> it's in the core), iflipb calls buffer-list, helm-buffers-list calls
> buffer-list.

AFAIK they (should) all skip the "hidden buffers" (those starting with
a space), so we'd just have to make those incomplete buffers use a name
with a leading space.

> And depending on the file size, the overhead from vc-find-file-hook may
> become noticeable, too. In (find-file-noselect ".gitignore"), it takes more
> than half of the time (15 out of 25 ms), here.

But for xdisp.c, that's probably negligible compared to the
syntax-propertize call ;-)


        Stefan

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 07/31/2015 12:41 AM, Stefan Monnier wrote:

>> So, you'd limit the changes to the higher level, like read-buffer-to-switch
>> and internal-complete-buffer-except?
>
> Yes.

We'll need to also catch the moment, somehow, when an "incomplete" 
buffer become displayed as a result of user's action. And "complete" its 
initialization at that point.

Would you use window-configuration-change-hook for that?

> AFAIK they (should) all skip the "hidden buffers" (those starting with
> a space), so we'd just have to make those incomplete buffers use a name
> with a leading space.

Indeed. Thinking about it initially, I was under impression that most of 
those tools have a view that shows all buffers including ephemeral 
(which could then be overwhelmed by the full list of "incomplete" 
buffers), but apparently not. The exceptions can adapt later.

By the way, I'm considering naming them "delayed buffers" (like in 
delay-mode-hooks), and the respective function would be find-file-delayed.

>> And depending on the file size, the overhead from vc-find-file-hook may
>> become noticeable, too. In (find-file-noselect ".gitignore"), it takes more
>> than half of the time (15 out of 25 ms), here.
>
> But for xdisp.c, that's probably negligible compared to the
> syntax-propertize call ;-)

Well, at least the slowdown will be proportional to the file size, and 
less impacted by the quantity only (that will improve the case of many 
small files).

And we don't always need to propertize until the end of the file (the 
match could be in the beginning). Opening xdisp.c, even without calling 
syntax-propertize, takes a lot of time now, but that should be optimizable.

Anyway, simply being allowed to open any files and not worry about 
cleaning up their buffers will simplify things for xref considerably.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > We'll need to also catch the moment, somehow, when an "incomplete"
 > buffer become displayed as a result of user's action. And "complete"
 > its initialization at that point.  Would you use
 > window-configuration-change-hook for that?

That's too late.  The operation in question is `find-file-noselect'
which has to find out whether the file is already visited in some
hidden/incomplete/delayed/retarded way.

martin

Re: xref and leaving "temporary" buffers open

[Dmitry Gutov]

On 08/03/2015 09:47 AM, martin rudalics wrote:

> That's too late.  The operation in question is `find-file-noselect'
> which has to find out whether the file is already visited in some
> hidden/incomplete/delayed/retarded way.

switch-to-buffer doesn't call find-file-noselect when switching to an 
existing buffer. Though maybe we just won't perform full initialization 
if someone deliberately switches to a delayed buffer this way.

Re: xref and leaving "temporary" buffers open

[martin rudalics]

 > switch-to-buffer doesn't call find-file-noselect when switching to an
 > existing buffer.

At the time `switch-to-buffer' gets hold of such a buffer it's too late
anyway, IMHO.

 > Though maybe we just won't perform full
 > initialization if someone deliberately switches to a delayed buffer
 > this way.

A user visiting a hidden/delayed buffer is on her own.  I still think
`buffer-list' shouldn't return them by default.

martin

Re: xref and leaving "temporary" buffers open

[Stefan Monnier]

> We'll need to also catch the moment, somehow, when an "incomplete" buffer
> become displayed as a result of user's action. And "complete" its
> initialization at that point.
> Would you use window-configuration-change-hook for that?

The way I see it, those buffers would have a name starting with SPC and
the name wouldn't have to look "normal".  So we don't need to care about
what happens when the user uses C-x b to display the buffer.

We only need to handle the case where find-file-noselect is called on
those files, at which point we should finish the initialization.
I'm not exactly sure what we should do for find-buffer-visiting and
get-file-buffer.


        Stefan