etags-regen-mode: handling extensionless files

etags-regen-mode: handling extensionless files

[Sean Whitton]

Hello Dmitry,

I'm working on a Perl project which has .pm files (which should be added
to etags-regen-file-extensions; I've written mail about that) but also
programs intended for execution, which have no file extension:
    <https://salsa.debian.org/dgit-team/dgit>.
E.g. 'dgit' and 'git-debrebase' in the project root.

On the one hand, it makes sense not to index these files because they're
programs not libraries, so their internal definitions won't ever be
referenced elsewhere.

On the other hand, it would be nice just to be able to use a simple
M-. to jump to definition, and not have to think about whether the
function is defined in the program or in a library.

Should we have some etags defcustom that allows specifying extra files
to index?  Or, as another idea, maybe etags could somehow include
looking at what imenu finds?  Might be too clever.

Let me know what you think about this use case.

-- 
Sean Whitton

Re: etags-regen-mode: handling extensionless files

[Dmitry Gutov]

Hi Sean,

On 20/09/2024 12:20, Sean Whitton wrote:

> I'm working on a Perl project which has .pm files (which should be added
> to etags-regen-file-extensions; I've written mail about that) but also
> programs intended for execution, which have no file extension:
>      <https://salsa.debian.org/dgit-team/dgit>.
> E.g. 'dgit' and 'git-debrebase' in the project root.
> 
> On the one hand, it makes sense not to index these files because they're
> programs not libraries, so their internal definitions won't ever be
> referenced elsewhere.

They're probably referenced internally though, at least once.

> On the other hand, it would be nice just to be able to use a simple
> M-. to jump to definition, and not have to think about whether the
> function is defined in the program or in a library.
> 
> Should we have some etags defcustom that allows specifying extra files
> to index?  Or, as another idea, maybe etags could somehow include
> looking at what imenu finds?  Might be too clever.

I guess we could add that extra option.

But see my other email regarding etags' hashbang detection.

Re: etags-regen-mode: handling extensionless files

[Sean Whitton]

Hello,

On Fri 20 Sep 2024 at 09:23pm +03, Dmitry Gutov wrote:

> Hi Sean,
>
> On 20/09/2024 12:20, Sean Whitton wrote:
>
>> I'm working on a Perl project which has .pm files (which should be added
>> to etags-regen-file-extensions; I've written mail about that) but also
>> programs intended for execution, which have no file extension:
>>      <https://salsa.debian.org/dgit-team/dgit>.
>> E.g. 'dgit' and 'git-debrebase' in the project root.
>> On the one hand, it makes sense not to index these files because they're
>> programs not libraries, so their internal definitions won't ever be
>> referenced elsewhere.
>
> They're probably referenced internally though, at least once.

Good point.

>> On the other hand, it would be nice just to be able to use a simple
>> M-. to jump to definition, and not have to think about whether the
>> function is defined in the program or in a library.
>> Should we have some etags defcustom that allows specifying extra files
>> to index?  Or, as another idea, maybe etags could somehow include
>> looking at what imenu finds?  Might be too clever.
>
> I guess we could add that extra option.
>
> But see my other email regarding etags' hashbang detection.

Hashbang detection would solve my problem elegantly.

Is my reading of the other thread correct that if we can fix the fortran
fallback then we can enable the hashbang detection?

-- 
Sean Whitton

Re: etags-regen-mode: handling extensionless files

[Dmitry Gutov]

On 22/09/2024 15:02, Sean Whitton wrote:

>> But see my other email regarding etags' hashbang detection.
> 
> Hashbang detection would solve my problem elegantly.
> 
> Is my reading of the other thread correct that if we can fix the fortran
> fallback then we can enable the hashbang detection?

Yep, I think so.

We would probably also discuss etags' auto-detection and its list of 
default extensions, during the next release's development.

Re: etags-regen-mode: handling extensionless files

[Sean Whitton]

Hello,

On Mon 23 Sep 2024 at 08:00pm +03, Dmitry Gutov wrote:

> On 22/09/2024 15:02, Sean Whitton wrote:
>
>>> But see my other email regarding etags' hashbang detection.
>> Hashbang detection would solve my problem elegantly.
>> Is my reading of the other thread correct that if we can fix the fortran
>> fallback then we can enable the hashbang detection?
>
> Yep, I think so.
>
> We would probably also discuss etags' auto-detection and its list of default
> extensions, during the next release's development.

Okay, cool!  Should we have a bug to track this?

-- 
Sean Whitton

Re: etags-regen-mode: handling extensionless files

[Dmitry Gutov]

On 25/09/2024 09:21, Sean Whitton wrote:
>> We would probably also discuss etags' auto-detection and its list of default
>> extensions, during the next release's development.
> Okay, cool!  Should we have a bug to track this?

Sure, please go ahead.

Re: etags-regen-mode: handling extensionless files

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: emacs-devel@gnu.org
> Date: Wed, 25 Sep 2024 07:21:58 +0100
> 
> Hello,
> 
> On Mon 23 Sep 2024 at 08:00pm +03, Dmitry Gutov wrote:
> 
> > On 22/09/2024 15:02, Sean Whitton wrote:
> >
> >>> But see my other email regarding etags' hashbang detection.
> >> Hashbang detection would solve my problem elegantly.
> >> Is my reading of the other thread correct that if we can fix the fortran
> >> fallback then we can enable the hashbang detection?
> >
> > Yep, I think so.
> >
> > We would probably also discuss etags' auto-detection and its list of default
> > extensions, during the next release's development.
> 
> Okay, cool!  Should we have a bug to track this?

We could, but adding an option to disable the Fortran fallback is so
easy that I hope someone will just do it...

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Sean Whitton]

Hello,

On Wed 25 Sep 2024 at 02:41pm +03, Dmitry Gutov wrote:

> On 25/09/2024 09:21, Sean Whitton wrote:
>>> We would probably also discuss etags' auto-detection and its list of default
>>> extensions, during the next release's development.
>> Okay, cool!  Should we have a bug to track this?

We want to replace etags-regen-file-extensions with enabling etags's
hashbang detection support.  That requires disabling its Fortran
fallback.

-- 
Sean Whitton

Re: etags-regen-mode: handling extensionless files

[Francesco Potortì]

Eli Zaretskii <eliz@gnu.org>
>> > We would probably also discuss etags' auto-detection and its list of default
>> > extensions, during the next release's development.

Sean Whitton <spwhitton@spwhitton.name>
>> Okay, cool!  Should we have a bug to track this?
>
>We could, but adding an option to disable the Fortran fallback is so
>easy that I hope someone will just do it...

How about just going with the backward-incompatible change of disabling both fallbacks entirely?  In my opinion the whole fallback idea was already obsolete when I worked on it in 1993.

Today, I can't imagine a situation where it can be useful, that is, where you work on Fortran or C sources without an extension.

On the other hand, if you are working on thirty-years old sources, I argue you should use thirty-years old tools, rather than assuming that today's tool do the right thing on them.

-- 
fp

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Dmitry Gutov]

Hi!

On 25/09/2024 22:27, Sean Whitton wrote:

> On Wed 25 Sep 2024 at 02:41pm +03, Dmitry Gutov wrote:
> 
>> On 25/09/2024 09:21, Sean Whitton wrote:
>>>> We would probably also discuss etags' auto-detection and its list of default
>>>> extensions, during the next release's development.
>>> Okay, cool!  Should we have a bug to track this?
> 
> We want to replace etags-regen-file-extensions with enabling etags's
> hashbang detection support.  That requires disabling its Fortran
> fallback.

Thanks, a fuller plan would look something like this:

- Implement the --no-fortran-fallback flag in etags. Or an environment 
variable, or etc. Use it conditionally in etags-regen-mode.
- Revisit the default lists of extensions that etags recognizes, keeping 
in mind the recent thread we talking this about in - e.g. *.a seems out 
of place for ASM (someone more familiar with assembly dialects please 
feel free to correctme).
- Add new possible value t to etags-regen-file-extensions, and switch 
the default to it.

Re: etags-regen-mode: handling extensionless files

[Eli Zaretskii]

> From: Francesco Potortì <pot@gnu.org>
> Date: Wed, 25 Sep 2024 23:19:10 +0200
> Cc: emacs-devel@gnu.org,
> 	dgutov@yandex.ru,
> 	Sean Whitton <spwhitton@spwhitton.name>
> 
> Eli Zaretskii <eliz@gnu.org>
> >We could, but adding an option to disable the Fortran fallback is so
> >easy that I hope someone will just do it...
> 
> How about just going with the backward-incompatible change of disabling both fallbacks entirely?  In my opinion the whole fallback idea was already obsolete when I worked on it in 1993.

Since no one complained about it, and the only real use case is when
invoking 'etags' from a Lisp program, which can easily use a
non-standard option, I don't see a compelling reason for a
backward-incompatible change in behavior.  In some future version, we
can then flip the default and make the fallback disabled by default.

> Today, I can't imagine a situation where it can be useful, that is, where you work on Fortran or C sources without an extension.

My gray hair tells me that our ability to imagine such situations is
severely limited or biased, and we have enough evidence of this bias
to not trust our imagination in these matters anymore.

> On the other hand, if you are working on thirty-years old sources, I argue you should use thirty-years old tools, rather than assuming that today's tool do the right thing on them.

These arguments are usually not well taken, IME.  People want new
tools because they give them more functionality and performance, but
they do NOT want incompatibilities in the package.  IOW, everyone
likes to have the cake and eat it, too.

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Francesco Potortì]

>- Implement the --no-fortran-fallback flag in etags. Or an environment 
>variable, or etc. Use it conditionally in etags-regen-mode.

If your purpose is to avoid Etags creating false tags on files whose language it cannot detect, you need to disable all fallbacks, rather than just Fortran.

Sorry if I got lost and missed something.

-- 
fp

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Dmitry Gutov]

On 26/09/2024 10:43, Francesco Potortì wrote:
>> - Implement the --no-fortran-fallback flag in etags. Or an environment
>> variable, or etc. Use it conditionally in etags-regen-mode.
> If your purpose is to avoid Etags creating false tags on files whose language it cannot detect, you need to disable all fallbacks, rather than just Fortran.

Yeah, sorry, I guess the next fallback is C?

We'll want to disable both, so the flag would be --no-fallbacks, I guess.

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Eli Zaretskii]

> Date: Thu, 26 Sep 2024 01:30:55 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > We want to replace etags-regen-file-extensions with enabling etags's
> > hashbang detection support.  That requires disabling its Fortran
> > fallback.
> 
> Thanks, a fuller plan would look something like this:
> 
> - Implement the --no-fortran-fallback flag in etags. Or an environment 
> variable, or etc. Use it conditionally in etags-regen-mode.
> - Revisit the default lists of extensions that etags recognizes, keeping 
> in mind the recent thread we talking this about in - e.g. *.a seems out 
> of place for ASM (someone more familiar with assembly dialects please 
> feel free to correctme).
> - Add new possible value t to etags-regen-file-extensions, and switch 
> the default to it.

I understand that we need to disable the Fortran and C fallbacks to
avoid false positives, but what do we want to do if the fallbacks are
disabled and no suitable language parser is found using the file name?
Just skip the file and do nothing? emit a warning? something else?

I also don't understand why enabling the etags' shebang detection
requires to disable the Fortran and C fallbacks: etags looks for
shebang _before_ it falls back to Fortran and C, so what am I missing?

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Eli Zaretskii]

> Cc: 73484@debbugs.gnu.org, spwhitton@spwhitton.name
> Date: Sun, 29 Sep 2024 11:25:45 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> I understand that we need to disable the Fortran and C fallbacks to
> avoid false positives, but what do we want to do if the fallbacks are
> disabled and no suitable language parser is found using the file name?
> Just skip the file and do nothing? emit a warning? something else?

Wait a minute... we already have "--language=none", which means only
do regexp processing, if any.  If no regexps were specified, 'none'
produces a single entry for a file, stating just its name, like this:

  ^L
  foo,0

where ^L is a literal \f character.  Is the intent here to prevent
even that from being written to TAGS?  If not, then we don't need any
new command-line option; instead, etags-regen could simply pass the
"--language=none" option before each file with no extension, and be
done, no?

Or maybe this is "the missing link" between this and the shebang
processing?

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Francesco Potortì]

Eli Zaretskii:
>> I understand that we need to disable the Fortran and C fallbacks to
>> avoid false positives, but what do we want to do if the fallbacks are
>> disabled and no suitable language parser is found using the file name?
>> Just skip the file and do nothing? emit a warning? something else?

Eli Zaretskii:
>Wait a minute... we already have "--language=none", which means only
>do regexp processing, if any.  If no regexps were specified, 'none'
>produces a single entry for a file, stating just its name, like this:
>
>  ^L
>  foo,0
>
>where ^L is a literal \f character.  Is the intent here to prevent
>even that from being written to TAGS?  If not, then we don't need any
>new command-line option; instead, etags-regen could simply pass the
>"--language=none" option before each file with no extension, and be
>done, no?
>
>Or maybe this is "the missing link" between this and the shebang
>processing?

If you set language=none for files whose extension is unknown to Etags, then you give up on shebang processing.  If you do not set language=none and Etags does not recognise any shebang, it defaults to Fortran.  If it does not find any Fortran tags, it defaults to C/C++.  When default processing happens on a file which is neither Fortran nor C/C++, it usually generates no tags, but may occasionally generate fake tags.

AFAIU, the problem is that there are use cases when you have to feed Etags with files that should generate no tags, yet the occasional fake tags are not tolerable.

bug#73484: 31.0.50; Abolishing etags-regen-file-extensions

[Dmitry Gutov]

On 29/09/2024 11:25, Eli Zaretskii wrote:
> I understand that we need to disable the Fortran and C fallbacks to
> avoid false positives, but what do we want to do if the fallbacks are
> disabled and no suitable language parser is found using the file name?
> Just skip the file and do nothing? emit a warning? something else?

Just do nothing. We'd really want to delegate language detection to 
etags rather than doing it inside Elisp - the latter is slower and 
ultimately more limited. But for that etags needs to have a reliable 
detection logic, one without too many false positives (and IME false 
positives here are worse than false negatives, because scanning too much 
can often mean both wrong tags and long scans, and a completion table 
that gets too large because of bogus tags).

For shebangs in particular, however, see Francesco's very good 
explanation. And detecting shebangs in Lisp would not be practical -- 
too slow.