From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#73484: 31.0.50; Abolishing etags-regen-file-extensions Date: Wed, 09 Oct 2024 22:11:21 +0300 Message-ID: <86v7y12ism.fsf@gnu.org> References: <87tteaznog.fsf@zephyr.silentflame.com> <87jzezzg87.fsf_-_@zephyr.silentflame.com> <37e4b3cd-6363-4f55-9921-92a1182679dc@gutov.dev> <86ttdy50ja.fsf@gnu.org> <75fe4289-da41-454d-ba92-22a92ea7002f@gutov.dev> <86frpe2186.fsf@gnu.org> <8e305b6d-8ca8-4437-990f-183ebc007d18@gutov.dev> <865xqa1ggi.fsf@gnu.org> <86ttdtzoof.fsf@gnu.org> <8d7dc133-9828-4023-821f-e4403f899f81@gutov.dev> <86ttdsxt6x.fsf@gnu.org> <52cb1caa-9e7e-45df-b328-d60948d397f6@gutov.dev> <864j5rxca1.fsf@gnu.org> <87a5fiijy9.fsf@tucano.isti.cnr.it> <86jzelvjh4.fsf@gnu.org> <8b6560a9-e2d6-42ae-ac1d-014700f21804@gutov.dev> <86wmiktzez.fsf@gnu.org> <86ldyzucdd.fsf@gnu.org> <021c625b-adc9-4e19-819c-fe929583e503@gutov.dev> <86ed4ru41x.fsf@gnu.org> <8d86f23e-fdc3-45a5-b3c8-cd4670813e21@gutov.dev> <86ploasq35.fsf@gnu.org> <3e63f532-c6af-4923-880b-01a32cc667ec@gutov.dev> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="2855"; mail-complaints-to="usenet@ciao.gmane.io" Cc: pot@gnu.org, 73484@debbugs.gnu.org, spwhitton@spwhitton.name To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Oct 09 21:12:07 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1syc6Q-0000ac-MV for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 09 Oct 2024 21:12:07 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1syc6E-0007XM-No; Wed, 09 Oct 2024 15:11:54 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1syc6C-0007WG-ME for bug-gnu-emacs@gnu.org; Wed, 09 Oct 2024 15:11:52 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1syc6C-0005eC-6T for bug-gnu-emacs@gnu.org; Wed, 09 Oct 2024 15:11:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=References:In-Reply-To:From:Date:To:Subject; bh=UsSvf/K03s3LHfhD3yE3HzpdxGhS576VWGjkaMUCFXw=; b=jWLqiSA4EIthTby4OkPvRIeeJ701jUgUwLCO8VxNwFtcjbn6YKb+CaOeF796t/gh1hw8+xL899LucArjxvccSAwJYR001jnJx45nttBfFhiqUXmQ+YeqHSaZkrar1pEKktgCrL2hYo/vCFnWQyLpoWJGr4eCYv2gzEJtms3J3AEATJdmSxLLHHJXX0Jfd/wm7h2hVHijkT/5dcPtqCj6gU0pz2AY9d3v0rlFjlFT3YH5Da9eFVoB+Z+aEPclwpngZ09rH7oMrlxnCkW25XrSWbl8l9xz2iJgFnr0kQnlC6odgBywd9rMxTp2kZdw4Opk1FodtgcEOhSN3hCK1KhwgA==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1syc6L-0001M7-KI for bug-gnu-emacs@gnu.org; Wed, 09 Oct 2024 15:12:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 09 Oct 2024 19:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73484 X-GNU-PR-Package: emacs Original-Received: via spool by 73484-submit@debbugs.gnu.org id=B73484.17285011145197 (code B ref 73484); Wed, 09 Oct 2024 19:12:01 +0000 Original-Received: (at 73484) by debbugs.gnu.org; 9 Oct 2024 19:11:54 +0000 Original-Received: from localhost ([127.0.0.1]:57673 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1syc6D-0001Ll-Va for submit@debbugs.gnu.org; Wed, 09 Oct 2024 15:11:54 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:43656) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1syc69-0001LT-Cg for 73484@debbugs.gnu.org; Wed, 09 Oct 2024 15:11:52 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1syc5t-0005cP-70; Wed, 09 Oct 2024 15:11:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=UsSvf/K03s3LHfhD3yE3HzpdxGhS576VWGjkaMUCFXw=; b=FjQ6nLJ/Mj51 WPThyDnXsDtLHYvHh9kI0oyLikBVuGDury/TGS7bhd5WzObiGS3H1MWQyka642wln00B2y+PbX6v2 wtj9CmQad0ciIGFIAZnHOAhox7YzfzermLv2PqzX0CUsZ4606fcOjzWg2t2fYXjU8G0bq1+ZFI5Nl 0p++ibD3ZgznMnTJ5cZSMEPiz4MTRZlROuf+DRYcPIvKSlS9B5xyofBjMf7OJhLakVZoirw3XdJ9y eHjBlaC9zkqV2zPVx/VYG+HyTHBAeeK2xi8Qds+jOatZruyiU2WNbzafkjBP5Wy+iXmJ7svCLRX4i UmXBIfswItx8irScs9FEXg==; In-Reply-To: <3e63f532-c6af-4923-880b-01a32cc667ec@gutov.dev> (message from Dmitry Gutov on Wed, 9 Oct 2024 21:23:37 +0300) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:293225 Archived-At: > Date: Wed, 9 Oct 2024 21:23:37 +0300 > Cc: pot@gnu.org, spwhitton@spwhitton.name, 73484@debbugs.gnu.org > From: Dmitry Gutov > > 'perf' shows me a profile like this: > > 67.31% etags libc.so.6 [.] __strcmp_avx2 > 26.29% etags etags [.] process_file_name > 2.00% etags etags [.] streq > 0.96% etags etags [.] strcmp@plt > 0.32% etags etags [.] readline_internal > 0.11% etags etags [.] HTML_labels > 0.08% etags [kernel.kallsyms] [k] syscall_return_via_sysret > 0.07% etags [kernel.kallsyms] [k] kmem_cache_alloc > 0.06% etags [kernel.kallsyms] [k] entry_SYSRETQ_unsafe_stack > 0.05% etags [kernel.kallsyms] [k] perf_adjust_freq_unthr_context > 0.04% etags etags [.] c_strncasecmp > > So... most of the time is spent in string comparison. > > Here is the nested loop, which if I comment out, makes the parse finish > in ~20 seconds, with all the extra files (except *.js), or in 15s when > using with new flags. > > diff --git a/lib-src/etags.c b/lib-src/etags.c > index a822a823a90..331e3ffe816 100644 > --- a/lib-src/etags.c > +++ b/lib-src/etags.c > @@ -1697,14 +1697,14 @@ process_file_name (char *file, language *lang) > uncompressed_name = file; > } > > - /* If the canonicalized uncompressed name > - has already been dealt with, skip it silently. */ > - for (fdp = fdhead; fdp != NULL; fdp = fdp->next) > - { > - assert (fdp->infname != NULL); > - if (streq (uncompressed_name, fdp->infname)) > - goto cleanup; > - } > + /* /\* If the canonicalized uncompressed name */ > + /* has already been dealt with, skip it silently. *\/ */ > + /* for (fdp = fdhead; fdp != NULL; fdp = fdp->next) */ > + /* { */ > + /* assert (fdp->infname != NULL); */ > + /* if (streq (uncompressed_name, fdp->infname)) */ > + /* goto cleanup; */ > + /* } */ > > inf = fopen (file, "r" FOPEN_BINARY); > if (inf) > > This is basically a "uniqueness" operation using linear search, O(N^2). Yes, this seems to be a protection against the same file name mentioned more than once on the command line.. > Is there a hash table we could use? Something like that should do, yes. > Or perhaps we would skip the search when the canonicalized name is the > same as the original one. That's not the same as the loop above does, I think. > > Two aspects that I found trying to understand the long scan times, and > > I'd like to mention so they don't become forgotten: > > > > . If there are compressed files in the directory, etags will > > uncompress them before it attempts to identify their language. > > There are 20 such files in the gecko-dev tree (removing them from > > the list of scanned files had only minor effect on the elapsed > > time, but it could be different in other cases, especially if > > uncompressing them produces very large files). > > I guess someone might ask for flag "--no-decompress", sometime. Yes, but it's also easy to exclude them via 'find'. > > . Some files have their language identified by means other than their > > names or extensions: those are the languages that have > > "interpreters" defined in etags.c. Shell scripts is one such case, > > but not the only one. So when etags-regen.el passes only files > > with known extensions to etags, it misses those files from TAGS. > > As one example, the file js/src/devtools/rootAnalysis/run_complete > > in the gecko-dev tree is a Perl script, but has no .pl extension. > > This sounds the same as the "hashbang" files that we mentioned > previously. It makes sense for the scan to take longer, of course, > proportional to the number of the detected files. My point was that if someone wants all the Python files, say, submitting only Python extensions to etags might miss some Python scripts.