Re: Generation of tags for the current project on the fly

[Dmitry Gutov <dgutov@yandex.ru>]

On 1/15/18 8:37 AM, Eli Zaretskii wrote:

>>> No, I think asking once per project should be enough.
>>
>> Until the end of the current Emacs session? And ask again after restart?
> 
> Yes.
> 
>> What about if the user switches to a different project and then back?
> 
> Ideally, don't ask anymore about that project.

This is doable, ok.

>> There's also another optimization opportunity: performing reindexing in
>> an asynchronous fashion, in the background (maybe after a timeout, too),
>> after any file is changed and saved. This one comes with its own
>> tradeoffs, though.
> 
> Doing that asynchronously could be an automatic action , not in need
> of any user confirmation.  It complicates the implementation a bit,
> but perhaps not too much, so this could be a good design choice.

It would shorten the waits, but do nothing for the CPU and disk usage. 
Those I'm more worried about, actually, as a laptop user with a 
not-so-great battery life on GNU/Linux.

If we just invalidate on save, the rescan doesn't happen until you 
intend to use it again. And with asynchronous approach, they will occur 
again and again, just as you edit and save files. With large projects, 
one CPU core might always be busy this way (or does etags parallelize? 
more cores then).

So maybe someone would prefer this approach, but I'd only go for it only 
as a qualify-of-life improvements when scans are already pretty short.

>> To measure the full time:
>>
>> (benchmark 1 '(progn (etags--project-tags-cleanup)
>> (etags--maybe-use-project-tags)))
> 
> 5.5 sec with warm cache.  This is with an unoptimized Emacs, btw, but
> most of the time is spent by external programs, so perhaps this
> doesn't matter.

Probably doesn't, indeed.

>> To measure the time to generate the list of files only:
>>
>> (benchmark 1 '(all-completions "" (project-file-completion-table
>> (project-current) (list default-directory))))
> 
> 0.95 sec with cold cache, 0.23 with warm cache.

Thanks, so 1 second for file listing for 4.5 seconds for etags. Even if 
we allowed etags to execute in parallel with find, it could only shave 
it down to 4.5 seconds (and probably not even that).

>> How do we figure which files to visit? Do we just visit src/TAGS and
>> expect the rest to be 'include'-d.
> 
> I think just visit TAGS in the directory of the source whose symbol is
> requested, or maybe use locate-dominating-file to look higher in the
> tree if not found in the current directory.

That option is not as easy to code as what I suggested.

Further, if we just visit lisp/TAGS when in lisp/, and xref-etags-mode 
is enabled, we won't be able to find the definition of 'car'.

>> Here's another one: considering the reindexing costs are not always
>> negligible and depend on the size of a project, will there be actual
>> benefit to using the proposed scheme in GNU projects like Emacs, GCC and
>> others (those are the ones that use 'make TAGS')? Or is there a subset
>> of them, at least, which we expect to benefit?
> 
> That's a good question.  But if the tags table is automatically
> produced in the background, the time this takes is much less
> important, and having TAGS always up to date would be a valuable
> feature.  FWIW, I do "make TAGS" in every large project I start
> working on seriously, so at least for me this is important.

You probably do it just once, though, and update very rarely. The old 
way of operation is still going to work.

Can we improve the "warm" reindex times? In the first message of this 
thread I mentioned GNU Global because it reportedly supports incremental 
updates. Can we get such feature in etags, too?

I more or less imagine how I'd implement such a feature using Lisp and 
'etags --append', but that would do nothing to help when the tags are 
generated by make.