Comparison of tools to search for related files

Comparison of tools to search for related files

[Damien Cassou]

Hi,

I'm implementing jumprel to jump from a file to related files (e.g., its
tests, its CSS, its .h header file…). During discussion for bug#57564,
Eli asked me

      How will this be different from what we already have:

      • the find-file.el package
      • the new command 'find-sibling-file'

I didn't know about these 2 packages so thank you very much for telling
me about them. Because I just learned about them, my description and
comparison below might be incomplete.

In the following I will compare the packages according to my 2 use
cases:

1. In the Emacs core code base, I want to jump from an elisp file (e.g.,
   `lisp/calendar/parse-time.el') to its test file (e.g.,
   `test/lisp/calendar/parse-time-tests.el', note the parallel folder
   hierarchy) and back.

2. In my JavaScript frontend project, I want to jump from a component
   file (e.g., `foo/MyComponent.js') to its Less file (e.g.,
   `foo/MyComponent.less', same folder and different extension) or to
   its UI test file (e.g., `foo/MyComponent.spec.component.js', same
   folder, same extension but different suffix) or to its non-UI test
   file (e.g., `test/foo/myComponent-tests.js', different casing and
   parallel folder hierarchy).

In both use cases, it would be nice to facilitate the creation of
non-existing files: for example, if a buffer visits `MyComponent.js' and
there is no `MyComponent.less', it would be great if Emacs could let me
create it from a list of non-existing related files.

I derive the following required features from these 2 use cases:

Parallel folder hierarchy
      It should be possible to jump from a file in `foo/bar/' to a file
      in `XX/foo/bar/' and back ("XX/" usually equals "test/" or
      "tests/");
Choose candidate
      The user should be presented with a list of related file
      candidates to pick the one to jump to;
Case changing
      Some related files might have a different casing;
Creation
      The user should be presented with the related files that don't
      exist so they can be created automatically (very useful with
      parallel folder hierarchies). I don't consider that a must but a
      nice to have.


1 find-file.el
══════════════

  find-file.el provides `ff-find-other-file' to jump from a file to a
  related file. How a file relates to another is done through
  `ff-other-file-alist' where elements can have 2 different forms:

  ┌────
  │ (REGEXP (EXTENSION...))
  │ (REGEXP FUNCTION)
  └────

  The first form above associates a filename regexp to a list of related
  file extensions. For example, `("\\.c\\'" (".h"))' associates a C file
  to its header file (it seems that C and C++ projects where the main
  target for this package).

  I didn't manage to configure find-file.el for parallel folder
  hierarchies. find-file.el provides `ff-search-directories' but it's
  not completely clear to me if this variable would make it possible to
  implement the use cases using the first form above (aka without
  relying on a function). I have the impression that it wouldn't.

  I managed to configure find-file.el for the cases when files are in
  the same folder as in most of the second use case:

  ┌────
  │ (("\\.spec.component\\.js\\'" (".js" ".less"))
  │  ("\\.less\\'" (".js" ".spec.component.js"))
  │  ("\\.js\\'" (".less" ".spec.component.js")))
  └────

  Unfortunately, only the first file matching from the EXTENSION list is
  used and the user is never presented a choice of candidate. Looking at
  the third line above, this means that if I'm in `MyComponent.js' and
  `MyComponent.less' exists, there is no way to go to
  `MyComponent.spec.component.js'.

  To implement parallel folder hierarchy, it is possible to use the
  second as it allows for more flexibility. The Emacs core use case
  would be implemented with:

  ┌────
  │ (defun my/ff-other-file-for-emacs-core (filename)
  │   (save-match-data
  │     (if (string-match "test/lisp" filename)
  │         (let ((without-test-directory (replace-match "lisp" nil t filename)))
  │           (list (replace-regexp-in-string "-tests\\.el$" ".el" without-test-directory)))
  │       (let ((with-test-directory (string-replace "/lisp/" "/test/lisp/" filename)))
  │         (list (replace-regexp-in-string "\\.el$" "-tests.el" with-test-directory))))))
  └────

  I have the impression that relying on elisp for such use cases will
  limit the usage of the package. For example, I haven't found any place
  in Emacs core code base or documentation where such a function would
  be given to facilitate the life of Emacs core contributors.

  find-file.el has a `ff-file-created-hook' variable to create and
  populate a file if no file exists. While a hook variable allows the
  user to do whatever it wants, it also means the user must write even
  more elisp code to do so.

  find-file.el has a `ff-special-constructs' to match import/include
  lines and open the imported file when cursor is on such a line. This
  seems completely unrelated to the rest of the code in find-file.el
  though and other find-file.el mechanisms are ignored in this case. I'm
  not sure why this code is here.

  Summary of find-file.el:

  Parallel folder hierarchy
        Supported through the creation of functions only. This limits
        the usage of this feature to elisp developers;
  Choose candidate
        find-file.el opens the first existing file and never let the
        user choose;
  Case changing
        Supported through the creation of functions only.
  Creation
        A hook exists which means it is possible but is limited to elisp
        developers.

  Beyond features, I found the code hard to read with very long
  functions, a lot of state mutation and no unit test. The code is
  around 800-line long.


2 find-sibling-file
═══════════════════

  This "package" consists of an interactive function, a customizable
  variable and a helper function.

  The Emacs core use case can easily be implemented by configuring
  `find-sibling-rules':

  ┌────
  │ (("test/lisp/\\(.*\\)-tests\\.el$" "lisp/\\1.el")
  │  ("lisp/\\(.*\\)\\.el$" "test/lisp/\\1-tests.el"))
  └────

  This works great.

  Ignoring parallel folder hierarchy (implemented just like in the
  previous use case) and case changing, the second use case can be

  ┌────
  │ (("\\(.*\\)\\.spec.component\\.js\\'" "\\1.js" "\\1.less")
  │  ("\\(.*\\)\\.less\\'" "\\1.js" "\\1.spec.component.js")
  │  ("\\(.*\\)\\.js\\'" "\\1.spec.component.js" "\\1.less"))
  └────

  This works great but redundancy starts to be annoying. If you need to
  add `*.stories.js' files (as is the case for my JS project), you will
  start suffering.

  If several matching files exist, the user is prompted with a list of
  candidates to choose from.

  I haven't found a way to implement case changing and there is no
  creation mechanism either.

  When regexps are not enough, there is no fallback-to-function
  workaround as was the case with find-file.el. I don't doubt this can
  easily be implemented though.

  The package has a nice feature to let the user switch between the same
  file in two different projects (e.g., `emacs-src-27/lisp/abbrev.el'
  and `emacs-src-28/lisp/abbrev.el'). I don't need the feature but I can
  see how it can be useful.

  Summary of find-sibling-file:

  Parallel folder hierarchy
        Supported with (slightly redundant) regexps;
  Choose candidate
        Supported;
  Case changing
        Unsupported.
  Creation
        Unsupported.

  Beyond features, the code is really simple and only 89-line long. It
  has no unit test though (yet?).


3 jumprel
═════════

  This is the package I'm working on. It provides a command to jump to a
  related file among existing candidates. It features a Domain-Specific
  Language (DSL) to describe the relation between files. For Emacs core,
  it would look like this
  ┌────

  └────

  ┌────
  │ (filename :remove-suffix ".el" :add-suffix "-tests.el" :add-directory "test")
  └────

  This line represents a jumper and must be added to
  `jumprel-jumpers'. This can be done in a `.dir-locals.el' file for
  example. This line is in my opinion much easier to understand and
  modify than the alternatives of the other two packages. Please note
  that this line works to go from a file to its test file and back: this
  limits the redundancy noticed above.

  The JavaScript UI use case would be implemented with these jumpers:

  ┌────
  │ (filename :remove-suffix ".js" :add-suffix "-tests.js" :add-directory "tests" :case-transformer uncapitalize)
  │ (filename :remove-suffix ".js" :add-suffix ".spec.component.js")
  │ (filename :remove-suffix ".js" :add-suffix ".less")
  │ (filename :remove-suffix ".js" :add-suffix ".stories.js")
  └────

  Note that the first line shows an example of using case
  transformation.

  When several files exist, the user is presented with a list of
  candidates just like `find-sibling-file'.

  `find-file.el' natively supports C and C++-based projects (see
  `cc-other-file-alist'). A similar configuration can be achieved with
  jumprel through such a simple jumper:

  ┌────
  │ (filename :remove-suffix ".c" :add-suffix ".h")
  └────

  If the DSL doesn't support your use case, it is possible to fallback
  to implementing a function, just like with find-file.el (I would
  prefer a patch to improve the DSL when it makes sense though). Another
  possibility is to define another DSL. For example, contrary to the
  other two packages, jumprel doesn't have any support for regexp-based
  definitions of file relations. This can easily be implemented by
  defining a new DSL and leveraging `find-sibling-file-search':

  ┌────
  │ (cl-defmethod jumprel-apply ((jumper (head regexp)) place)
  │   "Apply JUMPER to PLACE and return a new place or nil."
  │   (find-sibling-file-search
  │    place
  │    (list (cdr jumper))))
  └────

  With this in place, users can now specify exactly the same patterns as
  they would in `find-sibling-rules'. For example, the jumper below lets
  the user switch between the same file in two different projects (e.g.,
  `emacs-src-27/lisp/abbrev.el' and `emacs-src-28/lisp/abbrev.el'):

  ┌────
  │ (regexp "emacs/[^/]+/\\(.*\\)\\'" "emacs/.*/\\1")
  └────

  Additionally, jumprel provides a simple mechanism to declare how to
  populate files. For Emacs core, the `.dir-locals.el' file could
  contain:

  ┌────
  │ (filename :remove-suffix ".el" :add-suffix "-tests.el" :add-directory "test" :filler auto-insert)
  └────

  The `:filler auto-insert' part indicates that `auto-insert' must be
  called when a test file is created. You can also specify a string
  instead of `auto-insert' to give a default content. The mechanism is
  extensible so I also implemented a way to populate a file based on a
  yasnippet snippet (in my `init.el' file):

  ┌────
  │ (cl-defmethod jumprel-maker-fill ((filler (head yasnippet)) &allow-other-keys &rest)
  │   (when-let* ((snippet (map-elt (cdr filler) :name)))
  │     (yas-expand-snippet (yas-lookup-snippet snippet major-mode))))
  └────

  Which means the user can now specify a yasnippet snippet in their
  `.dir-locals.el' file:

  ┌────
  │ (filename :remove-suffix ".js" :add-suffix ".spec.component.js" :filler (yasnippet :name "componentSpec"))
  └────


  Summary of jumprel:

  Parallel folder hierarchy
        Supported with a simple `:add-directory' directive.
  Choose candidate
        Supported.
  Case changing
        Supported with a simple `:case-transformer' directive.
  Creation
        Supported with a simple `:filler' directive.

  Beyond features, jumprel's code is 403-line long but isn't fully
  documented yet. There are 227 lines of unit tests.

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Comparison of tools to search for related files

[Stefan Monnier]

>   This is the package I'm working on. It provides a command to jump to a
>   related file among existing candidates. It features a Domain-Specific
>   Language (DSL) to describe the relation between files. For Emacs core,
>   it would look like this

I have not looked in detail at your proposal, but it seems to aim to
cover a superset of what's supported by `find-sibling-file` and
`ff-find-other-file`, which is great.

But I keep dreaming of a tool that doesn't require *any* setup at all.
In my ideal world, from a `MyComponent.js` I'd be able to type
`M-x find-file-dwim RET .le RET` and that would jump to
`MyComponent.less`.

Or from `/opt/emacs-27/lisp/subr.el` I'd type `M-x find-file-dwim RET 28
RET` and that'd jump to `/opt/emacs-28/lisp/subr.el`.

Maybe also:

    /foo/lisp/toto.el      +  test     =>  /foo/tests/lisp/toto.el
    /foo/bar/lisp/toto.el  +  baz      =>  /foo/bazaar/toto.el
    /foo/lisp/toto.el      +  te/-te   =>  /foo/test/lisp/toto-tests.el

Admittedly, I'm only thinking of the case where the target file already
exists (we can only find the target file by looking at the filesystem to
see which of the the many possible targets is meant), but it seems like
most of it should be doable.


        Stefan

Re: Comparison of tools to search for related files

[Emanuel Berg]

Stefan Monnier wrote:

>
> But I keep dreaming of a tool that doesn't require *any*
> setup at all. In my ideal world, from a `MyComponent.js` I'd
> be able to type `M-x find-file-dwim RET .le RET` and that
> would jump to `MyComponent.less`.
>
> Or from `/opt/emacs-27/lisp/subr.el` I'd type `M-x
> find-file-dwim RET 28 RET` and that'd jump to
> `/opt/emacs-28/lisp/subr.el`.
>
> Maybe also:
>
>     /foo/lisp/toto.el      +  test     =>  /foo/tests/lisp/toto.el
>     /foo/bar/lisp/toto.el  +  baz      =>  /foo/bazaar/toto.el
>     /foo/lisp/toto.el      +  te/-te   =>  /foo/test/lisp/toto-tests.el

One would then have several algorithms all intended to find
one file from another, but using different methods, and the
DWIM algorithm would try them one by one ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Comparison of tools to search for related files

[Rudolf Schlatte]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>   This is the package I'm working on. It provides a command to jump to a
>>   related file among existing candidates. It features a Domain-Specific
>>   Language (DSL) to describe the relation between files. For Emacs core,
>>   it would look like this
>
> I have not looked in detail at your proposal, but it seems to aim to
> cover a superset of what's supported by `find-sibling-file` and
> `ff-find-other-file`, which is great.
>
> But I keep dreaming of a tool that doesn't require *any* setup at all.
> In my ideal world, from a `MyComponent.js` I'd be able to type
> `M-x find-file-dwim RET .le RET` and that would jump to
> `MyComponent.less`.

Hmm, would `find-file' do the job if the future history (accessible via
`M-n') is filled with all candidate sibling files?

Re: Comparison of tools to search for related files

[Stefan Monnier]

Emanuel Berg [2022-09-06 06:45:30] wrote:

> Stefan Monnier wrote:
>
>>
>> But I keep dreaming of a tool that doesn't require *any*
>> setup at all. In my ideal world, from a `MyComponent.js` I'd
>> be able to type `M-x find-file-dwim RET .le RET` and that
>> would jump to `MyComponent.less`.
>>
>> Or from `/opt/emacs-27/lisp/subr.el` I'd type `M-x
>> find-file-dwim RET 28 RET` and that'd jump to
>> `/opt/emacs-28/lisp/subr.el`.
>>
>> Maybe also:
>>
>>     /foo/lisp/toto.el      +  test     =>  /foo/tests/lisp/toto.el
>>     /foo/bar/lisp/toto.el  +  baz      =>  /foo/bazaar/toto.el
>>     /foo/lisp/toto.el      +  te/-te   =>  /foo/test/lisp/toto-tests.el
>
> One would then have several algorithms all intended to find
> one file from another, but using different methods, and the
> DWIM algorithm would try them one by one ...

No, I'm thinking of the DWIM algorithm as a single one.

It takes a file name FN considers it as a concatenation of PREFIX + STR +
SUFFIX and looks for a file that matches PREFIX + TXT + * + SUFFIX

And if TXT has a slash then the slash decomposes TXT into several such
sub-cases.

The hard part is that we're not told which part is PREFIX and which part
is SUFFIX, so we'd have to use `directory-files` looking for files that
match TXT and filter these matches according to whether they can be
treated as PREFIX + TXT + * + SUFFIX.


        Stefan

Re: Comparison of tools to search for related files

[Eli Zaretskii]

> From: Damien Cassou <damien@cassou.me>
> Cc: Eli Zaretskii <eliz@gnu.org>, Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 05 Sep 2022 22:51:29 +0200
> 
> I'm implementing jumprel to jump from a file to related files (e.g., its
> tests, its CSS, its .h header file…). During discussion for bug#57564,
> Eli asked me
> 
>       How will this be different from what we already have:
> 
>       • the find-file.el package
>       • the new command 'find-sibling-file'
> 
> I didn't know about these 2 packages so thank you very much for telling
> me about them. Because I just learned about them, my description and
> comparison below might be incomplete.
> 
> In the following I will compare the packages according to my 2 use
> cases:

[...]

Thank you for the details and the summaries.

My point in mentioning the existing features is that from where I
stand, it is much better to extend existing features rather than to
introduce a completely separate package with a different
implementation.  I think having several different overlapping features
in core is not good for maintenance.  I was even unhappy when Lars
introduced find-sibling-file, which IMO could have been implemented on
top of find-file.el with some extensions.

So the fact that neither of the 2 existing features fits your use case
doesn't invalidate what I was trying to convey.  I didn't claim that
one or both of the existing features already support your needs, I
wanted to urge you to consider how one of them could be extended to
cover your use cases.  AFAIU, the question whether such extensions are
reasonably practical and will yield convenient solutions -- that
question still stands.

Thanks.

Re: Comparison of tools to search for related files

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org,  Eli Zaretskii <eliz@gnu.org>,  Lars Ingebrigtsen
>  <larsi@gnus.org>
> Date: Tue, 06 Sep 2022 00:26:13 -0400
> 
> Admittedly, I'm only thinking of the case where the target file already
> exists (we can only find the target file by looking at the filesystem to
> see which of the the many possible targets is meant), but it seems like
> most of it should be doable.

I think the case where the target file doesn't yet exist is not much
less important than the other one.  Features like these should support
both.

Re: Comparison of tools to search for related files

[Damien Cassou]

Hi,

in bug#58071 [1] I submitted related-files (formerly jumprel), the tool
I described in the first message of this thread. The question is now if
related-files should be included into Emacs core and, if yes, what to do
about `find-file.el' and `find-sibling-file' whose behavior is, I think,
largely covered by related-files.

What do you think?

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58071

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Comparison of tools to search for related files

[Eli Zaretskii]

> From: Damien Cassou <damien@cassou.me>
> Cc: Eli Zaretskii <eliz@gnu.org>, Lars Ingebrigtsen <larsi@gnus.org>,
>  emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 08:37:05 +0200
> 
> in bug#58071 [1] I submitted related-files (formerly jumprel), the tool
> I described in the first message of this thread. The question is now if
> related-files should be included into Emacs core and, if yes, what to do
> about `find-file.el' and `find-sibling-file' whose behavior is, I think,
> largely covered by related-files.
> 
> What do you think?

Given that we already have 2 such facilities, I personally don't mind
to have a third, which perhaps could replace the other two in the
future.  (Although, to tell the truth, I'd be happier to have just one
package for this.)  But Lars is objected to adding this, AFAIU, so if
we cannot convince him, this should go to ELPA, I think.

I'm also interested in the opinions of others.

Thanks.

Re: Comparison of tools to search for related files

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Given that we already have 2 such facilities, I personally don't mind
> to have a third, which perhaps could replace the other two in the
> future.  (Although, to tell the truth, I'd be happier to have just one
> package for this.)  But Lars is objected to adding this, AFAIU, so if
> we cannot convince him, this should go to ELPA, I think.

My preference would be to have it as en ELPA package, but I won't
protest if you want it included in Emacs core.

> I'm also interested in the opinions of others.

Me too.

Re: Comparison of tools to search for related files

[Felician Nemeth]

>> I'm also interested in the opinions of others.

As a developer of foo-mode, I'd like to implement a related file
functionality for just one backend (find-file or related-file).  As a
user, I'd like to bind just one (global) key to this functionality and
be ignorant how foo-mode or bar-mode implements it.

To paraphrase what Lars wrote earlier, we had
completion-at-point-functions providing a clear completion API between
backends and frontends.  Does a similar API exist for related files?

If I understand correctly find-sibling-file primarily solves a
different problem: to easily switch among different versions of the same
file when for example multiple branches of a project is checked out in
parallel.

But maybe I misunderstand the purpose of related-files.el.  If it's the
user who configures related-files because the user has a special naming
convention, for example, to have files like page.html, page.css,
page.js, then I don't think a common API is necessary.  Then users can
choose to configure whatever package they like.

Re: Comparison of tools to search for related files

[Damien Cassou]

Felician Nemeth <felician.nemeth@gmail.com> writes:
> As a developer of foo-mode, I'd like to implement a related file
> functionality for just one backend (find-file or related-file).


You mean the developer of c-mode would specify that .c and .h are
related? This makes sense.


> As a user, I'd like to bind just one (global) key to this
> functionality and be ignorant how foo-mode or bar-mode implements it.


Indeed.


> To paraphrase what Lars wrote earlier, we had
> completion-at-point-functions providing a clear completion API between
> backends and frontends.  Does a similar API exist for related files?


I'm not sure the comparison stands here because there are not several
frontends and backends with pros and cons. We just need 1 key binding to
jump to a related file and a configuration option to specify how to
compute related files from the current file.


> If I understand correctly find-sibling-file primarily solves a
> different problem: to easily switch among different versions of the same
> file when for example multiple branches of a project is checked out in
> parallel.


find-sibling-file goes beyond that. It allows for specifying a regexp
and an expansion so the user could also specify how to go from a .c file
to a .h file.

For related-files.el, the regexp way of specifying related files is just
one among others: the user can also specify related files through
functions and a recipe DSL or define another way. related-files also
allows creating related files through different mechanisms and users can
add more mechanisms.


> But maybe I misunderstand the purpose of related-files.el.  If it's the
> user who configures related-files because the user has a special naming
> convention, for example, to have files like page.html, page.css,
> page.js


This is indeed the use-case I had in mind and it seems similar to the
one of find-sibling-file.


>, then I don't think a common API is necessary.  Then users can
> choose to configure whatever package they like.


That is true. But I'm not sure we want the same feature implemented 3
times in 3 different ways.

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Comparison of tools to search for related files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You mean the developer of c-mode would specify that .c and .h are
  > related? This makes sense.

In some programs, foo.c and foo.h are generally closely related.  But
that is not always true.  For instance, the C code of GNU Emacs does
not generally pair up header files with source files.

Depending on details, a feature that assumes ,c and .h files a paired
might be convenient with the programs which do that, and harmless with
other programs.  If it doesn't get in the way for the other programs,
then the feature does good and no harm.

But if it tends to be a nuisance when editing the programs that don't
pair the .h and .c files, that starts to be a bad thing.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Comparison of tools to search for related files

[Damien Cassou]

Richard Stallman <rms@gnu.org> writes:
> But if it tends to be a nuisance when editing the programs that don't
> pair the .h and .c files, that starts to be a bad thing.

For this to be a nuisance, it requires:

1. that the user actually tries to use related-files,

2. that the user hasn't configured related-files-jumpers for the
   project,

3. that the default "related" file (e.g., foo.c ⇒ foo.h) would exist but
   that the user would consider it unrelated.

I don't think that is a problem.

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill