From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#73484: 31.0.50; Abolishing etags-regen-file-extensions Date: Tue, 08 Oct 2024 16:04:46 +0300 Message-ID: <86ploasq35.fsf@gnu.org> References: <87tteaznog.fsf@zephyr.silentflame.com> <87jzezzg87.fsf_-_@zephyr.silentflame.com> <37e4b3cd-6363-4f55-9921-92a1182679dc@gutov.dev> <86ttdy50ja.fsf@gnu.org> <75fe4289-da41-454d-ba92-22a92ea7002f@gutov.dev> <86frpe2186.fsf@gnu.org> <8e305b6d-8ca8-4437-990f-183ebc007d18@gutov.dev> <865xqa1ggi.fsf@gnu.org> <86ttdtzoof.fsf@gnu.org> <8d7dc133-9828-4023-821f-e4403f899f81@gutov.dev> <86ttdsxt6x.fsf@gnu.org> <52cb1caa-9e7e-45df-b328-d60948d397f6@gutov.dev> <864j5rxca1.fsf@gnu.org> <87a5fiijy9.fsf@tucano.isti.cnr.it> <86jzelvjh4.fsf@gnu.org> <8b6560a9-e2d6-42ae-ac1d-014700f21804@gutov.dev> <86wmiktzez.fsf@gnu.org> <86ldyzucdd.fsf@gnu.org> <021c625b-adc9-4e19-819c-fe929583e503@gutov.dev> <86ed4ru41x.fsf@gnu.org> <8d86f23e-fdc3-45a5-b3c8-cd4670813e21@gutov.dev> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="9130"; mail-complaints-to="usenet@ciao.gmane.io" Cc: pot@gnu.org, 73484@debbugs.gnu.org, spwhitton@spwhitton.name To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Oct 08 15:06:22 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sy9uu-0002Ab-Sg for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 08 Oct 2024 15:06:21 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sy9uX-0005mt-6h; Tue, 08 Oct 2024 09:05:57 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sy9uU-0005ma-Os for bug-gnu-emacs@gnu.org; Tue, 08 Oct 2024 09:05:55 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sy9uU-0004eG-Ex for bug-gnu-emacs@gnu.org; Tue, 08 Oct 2024 09:05:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=References:In-Reply-To:From:Date:To:Subject; bh=IXro6TzKiff35a4Yhfsz5/4haYpWjdevNR+VUHSqQas=; b=T9kzehWJFpxPThgP2Z8skxYVV0wA0i3j1sUUa5PWkif3UxDI++hoVh88b1tB+PQrcqx3reyzEpOIImKbRf3FYO3fgZi/Ee40Qfy9pSyGRd8djMi9V/lo+hJH99jfx7VcQsuuSCj/+3bo5cAQjSMZeaKxWX0J7Wh3IcLuBzPut/9ZYmC7Ox22ItY/BpZ81MHPpk5XYg568o9wdhOzKbmz5bYxGMt/TCZRvBhi1HbfZIwpb7wnS/3TyDi8+MP2PXhFCs98mnLQUz2sIsRZow4TFj6u+QILiQ0KczmBkpj1LD/EQUZtnuP9vtbQOeB42JoJ4sABPpy5ekqNlIeTYxDNxw==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sy9uc-0005ip-OC for bug-gnu-emacs@gnu.org; Tue, 08 Oct 2024 09:06:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 08 Oct 2024 13:06:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73484 X-GNU-PR-Package: emacs Original-Received: via spool by 73484-submit@debbugs.gnu.org id=B73484.172839271421502 (code B ref 73484); Tue, 08 Oct 2024 13:06:02 +0000 Original-Received: (at 73484) by debbugs.gnu.org; 8 Oct 2024 13:05:14 +0000 Original-Received: from localhost ([127.0.0.1]:51627 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sy9tp-0005aH-LY for submit@debbugs.gnu.org; Tue, 08 Oct 2024 09:05:14 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:52420) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sy9tk-0005Yh-3Z for 73484@debbugs.gnu.org; Tue, 08 Oct 2024 09:05:11 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sy9tS-0004IZ-Ir; Tue, 08 Oct 2024 09:04:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=IXro6TzKiff35a4Yhfsz5/4haYpWjdevNR+VUHSqQas=; b=nMrSS9e11PZr 4PVgHwyaL7Sf07CVkLNyt8Uy4Pg5qVN1KVqBBP55hqIyX5fRduMr52tBEQFd3C2ZDMR8rJgONUi/h r3QEFrXlJpOfv4gFQEli3BqbhACsZL7dp/V03Z6dt6MZoSOexdK8LFNggYz6Tr0P5sUkr0638pbge 2ntkoM5otsAUpAGzgqLTaR6eVYfxTMP+JqUynREOQhRwmapWw7p5qzhfj4RqBh/CBcduy80Vb6MTK 4ST31xL3UTGz6kAxCScpCo9Dg0OEP6rGrQHK/gm01BDgJis4o5xFReqFGB1119PmzT/3Aei9omBVI 0S6Ty8rW25Abhzs+HMMz3g==; In-Reply-To: <8d86f23e-fdc3-45a5-b3c8-cd4670813e21@gutov.dev> (message from Dmitry Gutov on Tue, 8 Oct 2024 01:08:00 +0300) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:293155 Archived-At: > Date: Tue, 8 Oct 2024 01:08:00 +0300 > Cc: pot@gnu.org, spwhitton@spwhitton.name, 73484@debbugs.gnu.org > From: Dmitry Gutov > > On 07/10/2024 22:05, Eli Zaretskii wrote: > >> Date: Mon, 7 Oct 2024 20:36:47 +0300 > >> Cc: pot@gnu.org, spwhitton@spwhitton.name, 73484@debbugs.gnu.org > >> From: Dmitry Gutov > >> > >> On 07/10/2024 19:05, Eli Zaretskii wrote: > >> > >>> So what is the conclusion from this? Are you saying that the long > >>> scan times in this large tree basically make this new no-fallbacks > >>> option not very useful, since we still need to carefully include or > >>> exclude certain files from the scan? Or should I go ahead and install > >>> these changes? > >> > >> I think that option will be useful, but for better benchmarks and for > >> end usability as well, I think we need the N^2 thing fixed as well. > >> Maybe before the rest of the changes. > > > > If this latter part is a precodintion, > > I think we still could use the new flag, just not switch to it (no > extension filtering) by default yet. OK, installed on master. I leave it up to you whether to close the bug. > > then someone else will have to > > work on this. I have the new option coded and tested (and > > documented), but I don't intend to work on redesigning the core etags > > algorithms to remove the non-linear behavior, that's a much larger > > project which I currently cannot afford, sorry. > > Do you mind pointing at the places in the code where you already noticed > non-linear performance coming from? The while-loop near line 2020, for example. Another one is the for-loop near line 1420, which deals with writing into TAGS the entries of files with no tags. There may be others, but those are what I saw. Perhaps it is a good idea to profile etags while it scans the files during those 15 min, to see where it spends that much time, because I'm not sure even those loops can account for that. It's possible there's something else at work here which we don't yet understand. Two aspects that I found trying to understand the long scan times, and I'd like to mention so they don't become forgotten: . If there are compressed files in the directory, etags will uncompress them before it attempts to identify their language. There are 20 such files in the gecko-dev tree (removing them from the list of scanned files had only minor effect on the elapsed time, but it could be different in other cases, especially if uncompressing them produces very large files). . Some files have their language identified by means other than their names or extensions: those are the languages that have "interpreters" defined in etags.c. Shell scripts is one such case, but not the only one. So when etags-regen.el passes only files with known extensions to etags, it misses those files from TAGS. As one example, the file js/src/devtools/rootAnalysis/run_complete in the gecko-dev tree is a Perl script, but has no .pl extension.