bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 8 Oct 2024 01:08:00 +0300
> Cc: pot@gnu.org, spwhitton@spwhitton.name, 73484@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 07/10/2024 22:05, Eli Zaretskii wrote:
> >> Date: Mon, 7 Oct 2024 20:36:47 +0300
> >> Cc: pot@gnu.org, spwhitton@spwhitton.name, 73484@debbugs.gnu.org
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >> On 07/10/2024 19:05, Eli Zaretskii wrote:
> >>
> >>> So what is the conclusion from this?  Are you saying that the long
> >>> scan times in this large tree basically make this new no-fallbacks
> >>> option not very useful, since we still need to carefully include or
> >>> exclude certain files from the scan?  Or should I go ahead and install
> >>> these changes?
> >>
> >> I think that option will be useful, but for better benchmarks and for
> >> end usability as well, I think we need the N^2 thing fixed as well.
> >> Maybe before the rest of the changes.
> > 
> > If this latter part is a precodintion,
> 
> I think we still could use the new flag, just not switch to it (no 
> extension filtering) by default yet.

OK, installed on master.  I leave it up to you whether to close the
bug.

> > then someone else will have to
> > work on this.  I have the new option coded and tested (and
> > documented), but I don't intend to work on redesigning the core etags
> > algorithms to remove the non-linear behavior, that's a much larger
> > project which I currently cannot afford, sorry.
> 
> Do you mind pointing at the places in the code where you already noticed 
> non-linear performance coming from?

The while-loop near line 2020, for example.

Another one is the for-loop near line 1420, which deals with writing
into TAGS the entries of files with no tags.

There may be others, but those are what I saw.  Perhaps it is a good
idea to profile etags while it scans the files during those 15 min, to
see where it spends that much time, because I'm not sure even those
loops can account for that.  It's possible there's something else at
work here which we don't yet understand.

Two aspects that I found trying to understand the long scan times, and
I'd like to mention so they don't become forgotten:

 . If there are compressed files in the directory, etags will
   uncompress them before it attempts to identify their language.
   There are 20 such files in the gecko-dev tree (removing them from
   the list of scanned files had only minor effect on the elapsed
   time, but it could be different in other cases, especially if
   uncompressing them produces very large files).
 . Some files have their language identified by means other than their
   names or extensions: those are the languages that have
   "interpreters" defined in etags.c.  Shell scripts is one such case,
   but not the only one.  So when etags-regen.el passes only files
   with known extensions to etags, it misses those files from TAGS.
   As one example, the file js/src/devtools/rootAnalysis/run_complete
   in the gecko-dev tree is a Perl script, but has no .pl extension.