bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Eason Huang]

I try to use `M-x project-find-file` in my `.emacs.d` config project. It
takes a long time(about 1 minutes), and finally get a error as bellow:

```
process-file: Variable binding depth exceeds max-specpdl-size
```

The .emacs.d project include 95 submudules of Git, is this too huge for
project.el?
You can visit my config: https://github.com/Eason0210/emacs.d

On other git project with a few submodules(about 14), it works well.
For exmaple this one: https://github.com/emacscollective/emacs.g


And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
Then reproduce the issue, will get the following debug error:

Debugger entered--Lisp error: (excessive-variable-binding)
  call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
  process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
  project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")

  .... there are 1100 lines more here ...

  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("~/.emacs.d/" Git nil)
  #f(compiled-function (dir) #<bytecode -0x1a2f966d7f3875dc>)("~/.emacs.d/")
  mapcan(#f(compiled-function (dir) #<bytecode -0x1a2f966d7f3875dc>) ("~/.emacs.d/"))
  #f(compiled-function (project &optional dirs) #<bytecode -0x13f4e1776db2687e>)((vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  apply(#f(compiled-function (project &optional dirs) #<bytecode -0x13f4e1776db2687e>) (vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  project-files((vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  project-find-file-in(#("init.el" 0 7 (fontified t help-echo "mouse-2: visit this file in other window" mouse-face highlight dired-filename t)) ("~/.emacs.d/") (vc Git "~/.emacs.d/") nil)
  project-find-file(nil)
  funcall-interactively(project-find-file nil)
  command-execute(project-find-file record)
  execute-extended-command(nil "project-find-file" "project-find-fi")
  funcall-interactively(execute-extended-command nil "project-find-file" "project-find-fi")
  command-execute(execute-extended-command)

My platfrom informations:

In GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin21.4.0, NS appkit-2113.40 Version 12.3.1 (Build 21E258))
 of 2022-05-09 built on macbook
Repository revision: 4f1e748df208ced08c7cda8f96e6a5638ad14240
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.3.1

Configured using:
 'configure --with-ns --with-modules
 '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp:/usr/local/share/emacs/site-lisp'
 --with-xwidgets --with-native-compilation
 'CFLAGS=-I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include
 -O2' 'LDFLAGS=-L/usr/local/opt/gcc/lib/gcc/11
 -L/usr/local/opt/gcc/lib/gcc/11/gcc/x86_64-apple-darwin21/11
 -L/usr/local/opt/libgccjit/lib/gcc/11 -I/usr/local/opt/gcc/include
 -I/usr/local/opt/libgccjit/include -Wl,-headerpad_max_install_names''

Configured features:
ACL DBUS GIF GLIB GMP GNUTLS JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XIM XWIDGETS ZLIB

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/d

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media rmc puny
rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config
gnus-util text-property-search time-date mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils help-fns radix-tree
cl-print debug backtrace help-mode find-func thingatpt vc-mtn vc-hg
vc-git diff-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view
easy-mmode pcvs-util vc vc-dispatcher project seq gv subr-x byte-opt
bytecomp byte-compile cconv dired-aux cl-loaddefs cl-lib dired
dired-loaddefs iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice simple cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads
xwidget-internal dbusbind kqueue cocoa ns lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 765142 20103)
 (symbols 48 9014 0)
 (strings 32 215443 3200)
 (string-bytes 1 8854733)
 (vectors 16 21867)
 (vector-slots 8 482129 32483)
 (floats 8 30 306)
 (intervals 56 31551 79)
 (buffers 992 17))


-- 
Eason Huang

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Dmitry Gutov]

On 13.05.2022 15:56, Eason Huang wrote:
> I try to use `M-x project-find-file` in my `.emacs.d` config project. It
> takes a long time(about 1 minutes), and finally get a error as bellow:
> 
> ```
> process-file: Variable binding depth exceeds max-specpdl-size
> ```
> 
> The .emacs.d project include 95 submudules of Git, is this too huge for
> project.el?

Hi! That's a lot, but the above error indicates excess recursion. Do the 
submodules in your repo in turn have checked out submodules inside, and 
so on?

As a workaround, you can set project-vc-merge-submodules to nil (at 
least temporarily, until we get a better fix).

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Eason Huang]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 13.05.2022 15:56, Eason Huang wrote:
>> I try to use `M-x project-find-file` in my `.emacs.d` config project. It
>> takes a long time(about 1 minutes), and finally get a error as bellow:
>> ```
>> process-file: Variable binding depth exceeds max-specpdl-size
>> ```
>> The .emacs.d project include 95 submudules of Git, is this too huge
>> for
>> project.el?
>
> Hi! That's a lot, but the above error indicates excess recursion. Do
> the submodules in your repo in turn have checked out submodules
> inside, and so on?

I try another project with 42 submodules, when I use `Borg` to install
 `Corfu` extensions, I need to set two load-path on `.gitmodules`. And this
 setting will trigger the same issue.

```
[submodule "corfu"]
	load-path = .
	load-path = extensions
	path = lib/corfu
	url = git@github.com:minad/corfu.git
```
if I remove the load-path, just leave it as below, project.el wokrs
well.
```
[submodule "corfu"]
	path = lib/corfu
	url = git@github.com:minad/corfu.git
```

May be the above infomation can help you.

> As a workaround, you can set project-vc-merge-submodules to nil (at
> least temporarily, until we get a better fix).
>

Thanks, I try to set project-vc-merge-submodules to nil, and It works.

By the way, I also tried `projectile` on the huge .emacs.d, It works but
can feel the latency.

-- 
Eason Huang

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Dmitry Gutov]

Hi again, sorry for the delay.

On 13.05.2022 15:56, Eason Huang wrote:
> And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
> Then reproduce the issue, will get the following debug error:
> 
> Debugger entered--Lisp error: (excessive-variable-binding)
>    call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)

Looking at the backtrace again, it seems the problem is not related to 
the number of submodules. It's a plain infloop.

Could you try to help with debugging? Alternatively, you could provide a 
simple repo with this problem that doesn't require Borg to initialize. 
Though it probably doesn't (shouldn't) require Borg anyway, given how 
the problem looks.

What does project--vc-list-files do in your case? It calls 'git 
ls-files' to fetch the list of files in the parent repo, then parses the 
list of submodules in it, and repeats the same call inside each 
submodule (using (concat default-directory module) as target).

Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over and 
over again. So it seems like (project--git-submodules) returns a list 
which has "." as one of its elements.

How does that happen? Do you have a submodule entry which points to "."?

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Eason Huang]

Dmitry Gutov <dgutov@yandex.ru> writes:

Hi Dmitry,

Thanks for reply.

> Hi again, sorry for the delay.
>
> On 13.05.2022 15:56, Eason Huang wrote:
>> And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
>> Then reproduce the issue, will get the following debug error:
>> Debugger entered--Lisp error: (excessive-variable-binding)
>>    call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>
> Looking at the backtrace again, it seems the problem is not related to
> the number of submodules. It's a plain infloop.
>
> Could you try to help with debugging? Alternatively, you could provide
> a simple repo with this problem that doesn't require Borg to
> initialize. Though it probably doesn't (shouldn't) require Borg
> anyway, given how the problem looks.

I create an simple which only include a submodule, I take vertico for
example. you can get it here:

https://github.com/Eason0210/sample-project.git

Steps to reproduce the issue:

1. Create an empty folder call `.emacs.d` (The name is not matter, you
can use any name)

2. cd .emacs.d, and let's say i want to add vertico as an submodule

```
git init
git submodule add --name vertico git@github.com:minad/vertico.git lib/vertico
git add .
git commit -m "add vertico"
```
3. so far so good.
4. Modify the .gitmodules file, add load-path for extensions directory
of vertico repo

```
[submodule "vertico"]
	path = lib/vertico
	url = git@github.com:minad/vertico.git
	load-path = .
	load-path = extensions
   #+end_src
```
5. Now perform ~M-x, project-find-file~ in the .emacs.d directory will
cause the issue.



> What does project--vc-list-files do in your case? It calls 'git
> ls-files' to fetch the list of files in the parent repo, then parses
> the list of submodules in it, and repeats the same call inside each
> submodule (using (concat default-directory module) as target).

I try 'git ls-files .' in vertico submodule in iterm and get this:

```
➜  vertico git:(main) git ls-files .
LICENSE
README.org
extensions/vertico-buffer.el
extensions/vertico-directory.el
extensions/vertico-flat.el
extensions/vertico-grid.el
extensions/vertico-indexed.el
extensions/vertico-mouse.el
extensions/vertico-multiform.el
extensions/vertico-quick.el
extensions/vertico-repeat.el
extensions/vertico-reverse.el
extensions/vertico-unobtrusive.el
vertico.el
```

```
➜  vertico git:(main) git ls-files extensions
extensions/vertico-buffer.el
extensions/vertico-directory.el
extensions/vertico-flat.el
extensions/vertico-grid.el
extensions/vertico-indexed.el
extensions/vertico-mouse.el
extensions/vertico-multiform.el
extensions/vertico-quick.el
extensions/vertico-repeat.el
extensions/vertico-reverse.el
extensions/vertico-unobtrusive.el
```

> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
> and over again. So it seems like (project--git-submodules) returns a
> list which has "." as one of its elements.
>
> How does that happen? Do you have a submodule entry which points to "."?
>

I found that it's this line `load-path = .` cause the issue. Borg need
this line to add the vertico directory to load-path when user add a
subDirectory to load-path. May be it is an issue of Borg?


-- 
Eason Huang

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Dmitry Gutov]

On 13.06.2022 18:00, Eason Huang wrote:
>> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
>> and over again. So it seems like (project--git-submodules) returns a
>> list which has "." as one of its elements.
>>
>> How does that happen? Do you have a submodule entry which points to "."?
>>
> I found that it's this line `load-path = .` cause the issue. Borg need
> this line to add the vertico directory to load-path when user add a
> subDirectory to load-path. May be it is an issue of Borg?

Now that I've tried adding that locally, it indeed what triggered the 
problem. It was a bug in 'project--git-submodules', which I've just 
fixed on master in commit 915b34d280.

I think file listing together with submodules should work fine now in 
your case. Not sure about the performance though: we use one process 
call per submodule, so the overhead might get noticeable with 42 of 
them. But please give it a try.

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Eason Huang]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 13.06.2022 18:00, Eason Huang wrote:
>>> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
>>> and over again. So it seems like (project--git-submodules) returns a
>>> list which has "." as one of its elements.
>>>
>>> How does that happen? Do you have a submodule entry which points to "."?
>>>
>> I found that it's this line `load-path = .` cause the issue. Borg need
>> this line to add the vertico directory to load-path when user add a
>> subDirectory to load-path. May be it is an issue of Borg?
>
> Now that I've tried adding that locally, it indeed what triggered the
> problem. It was a bug in 'project--git-submodules', which I've just
> fixed on master in commit 915b34d280.
>
> I think file listing together with submodules should work fine now in
> your case. Not sure about the performance though: we use one process
> call per submodule, so the overhead might get noticeable with 42 of
> them. But please give it a try.
>

Great, I tried on my huge .emacs.d project, it works well and fast.

Tested on macOS with latest commit.

You can close this bug now. Thanks for your great work on it.

-- 
Eason Huang

bug#55396: 29.0.50; project-find-file don't work on a project with many submodules

[Dmitry Gutov]

On 14.06.2022 14:22, Eason Huang wrote:
> Great, I tried on my huge .emacs.d project, it works well and fast.
> 
> Tested on macOS with latest commit.
> 
> You can close this bug now. Thanks for your great work on it.

Thanks for testing! Closing.