Hi folks, The project.el function project-buffers can return incorrect results when invoked while a let-binding for default-directory is in effect. This is because project-buffers (both the default and vc-based implementations) does its work by inspecting the local value of default-directory in each buffer, and the let-binding temporarily affects this value. To see this in action, start emacs with -Q and evaluate the following forms in order: (require 'project) (find-file-noselect "/tmp/tmpfile") (setq my-project '(transient . "/tmp/")) ;; just the tmpfile (project-buffers my-project) ;; both tmpfile and scratch (let ((default-directory "/tmp/")) (project-buffers my-project)) In the last form, project-buffers includes the current buffer (i.e. the scratch buffer in our example) with the results. (This is true even if the current buffer is visiting a file in some unrelated directory.) This matters because the command project-switch-project let-binds default-directory before calling project-switch-commands. This means that if you set project-switch-commands to some function that calls project-buffers, you will get incorrect results. For example, evaluate the following forms in order: (defun my-list-project-buffers () "List the current project's buffers." (interactive) (let ((buffer-list (project-buffers (project-current t))) (buffer-name (project-prefixed-buffer-name "my-project-buffer-list"))) (with-current-buffer (get-buffer-create buffer-name) (erase-buffer) (save-excursion (dolist (buffer buffer-list) (insert (buffer-name buffer)) (insert ?\n)))) (switch-to-buffer buffer-name))) (setq project-switch-commands #'my-list-project-buffers) ;; list tmpfile but also scratch (project-switch-project "/tmp/") In GNU Emacs 28.2 System Description: macOS Configured using: 'configure --disable-silent-rules --enable-locallisppath=/opt/homebrew/share/emacs/site-lisp --infodir=/opt/homebrew/Cellar/emacs/28.2/share/info/emacs --prefix=/opt/homebrew/Cellar/emacs/28.2 --with-gnutls --without-x --with-xml2 --without-dbus --with-modules --without-ns --without-imagemagick --without-selinux' Configured features: ACL GMP GNUTLS JSON LIBXML2 MODULES NOTIFY KQUEUE PDUMPER THREADS ZLIB Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: ELisp/d Minor modes in effect: tooltip-mode: t global-eldoc-mode: t eldoc-mode: t show-paren-mode: t electric-indent-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t line-number-mode: t indent-tabs-mode: t transient-mark-mode: t Load-path shadows: None found. Features: (shadow regexp-opt sort mail-extr emacsbug message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache json map text-property-search time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-print edebug backtrace help-mode tool-bar find-func vc-mtn vc-hg vc-git diff-mode easy-mmode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc cl-loaddefs cl-lib vc-dispatcher image project seq term/xterm xterm byte-opt gv bytecomp byte-compile cconv iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads kqueue multi-tty make-network-process emacs) Memory information: ((conses 16 72031 7674) (symbols 48 8797 1) (strings 32 26201 1172) (string-bytes 1 817049) (vectors 16 14915) (vector-slots 8 168301 9701) (floats 8 42 454) (intervals 56 324 3) (buffers 992 14))
[-- Attachment #1: Type: text/plain, Size: 2563 bytes --] Hi! On 26.10.2022 03:13, Sean Devlin wrote: > The project.el function project-buffers can return incorrect results > when invoked while a let-binding for default-directory is in > effect. This is because project-buffers (both the default and vc-based > implementations) does its work by inspecting the local value of > default-directory in each buffer, and the let-binding temporarily > affects this value. > > To see this in action, start emacs with -Q and evaluate the following > forms in order: > > (require 'project) > > (find-file-noselect "/tmp/tmpfile") > (setq my-project '(transient . "/tmp/")) > > ;; just the tmpfile > (project-buffers my-project) > > ;; both tmpfile and scratch > (let ((default-directory "/tmp/")) > (project-buffers my-project)) Thanks for the report, I can see the problem. > In the last form, project-buffers includes the current buffer (i.e. the > scratch buffer in our example) with the results. (This is true even if > the current buffer is visiting a file in some unrelated directory.) > > This matters because the command project-switch-project let-binds > default-directory before calling project-switch-commands. This means > that if you set project-switch-commands to some function that calls > project-buffers, you will get incorrect results. > > For example, evaluate the following forms in order: > > (defun my-list-project-buffers () > "List the current project's buffers." > (interactive) > (let ((buffer-list (project-buffers (project-current t))) > (buffer-name (project-prefixed-buffer-name "my-project-buffer-list"))) > (with-current-buffer (get-buffer-create buffer-name) > (erase-buffer) > (save-excursion > (dolist (buffer buffer-list) > (insert (buffer-name buffer)) > (insert ?\n)))) > (switch-to-buffer buffer-name))) > > (setq project-switch-commands #'my-list-project-buffers) Looks like a reimplementation of projectile-ibuffer, seems useful. > ;; list tmpfile but also scratch > (project-switch-project "/tmp/") Not sure how to fix this, though. In bug#53626 we discussed a somewhat similar problem, and a let-binding seems impossible to "escape". What else can we do? One option is to change the signature of every compatible command to take the project object as its first argument. Might have been more realistic when the package was first written, too much breakage now, probably. Another would be to add a new var to help override the project choice without touch default-directory. Something like the attached. Please try it out. [-- Attachment #2: project-current-directory-override.diff --] [-- Type: text/x-patch, Size: 1839 bytes --] diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el index ac278edd40..48ed033fa1 100644 --- a/lisp/progmodes/project.el +++ b/lisp/progmodes/project.el @@ -175,8 +175,14 @@ project-find-functions `cl-defmethod' can dispatch on, like a cons cell, or a list, or a CL struct.") -(defvar project-current-inhibit-prompt nil - "Non-nil to skip prompting the user in `project-current'.") +(define-obsolete-variable-alias + 'project-current-inhibit-prompt + 'project-current-directory-override + "29.1") + +(defvar project-current-directory-override nil + "Value to use instead of `default-directory' when detecting the project. +When it is non-nil, `project-current' will skip always skip prompting.") ;;;###autoload (defun project-current (&optional maybe-prompt directory) @@ -195,11 +201,12 @@ project-current See the doc string of `project-find-functions' for the general form of the project instance object." - (unless directory (setq directory default-directory)) + (unless directory (setq directory (or project-current-directory-override + default-directory))) (let ((pr (project--find-in-directory directory))) (cond (pr) - ((unless project-current-inhibit-prompt + ((unless project-current-directory-override maybe-prompt) (setq directory (project-prompt-project-dir) pr (project--find-in-directory directory)))) @@ -1667,8 +1674,7 @@ project-switch-project (let ((command (if (symbolp project-switch-commands) project-switch-commands (project--switch-project-command)))) - (let ((default-directory dir) - (project-current-inhibit-prompt t)) + (let ((project-current-directory-override dir)) (call-interactively command)))) (provide 'project)
Hi Dmitry, > On Oct 28, 2022, at 8:49 PM, Dmitry Gutov <dgutov@yandex.ru> wrote: > > Hi! > > On 26.10.2022 03:13, Sean Devlin wrote: >> The project.el function project-buffers can return incorrect results >> when invoked while a let-binding for default-directory is in >> effect. This is because project-buffers (both the default and vc-based >> implementations) does its work by inspecting the local value of >> default-directory in each buffer, and the let-binding temporarily >> affects this value. >> To see this in action, start emacs with -Q and evaluate the following >> forms in order: >> (require 'project) >> (find-file-noselect "/tmp/tmpfile") >> (setq my-project '(transient . "/tmp/")) >> ;; just the tmpfile >> (project-buffers my-project) >> ;; both tmpfile and scratch >> (let ((default-directory "/tmp/")) >> (project-buffers my-project)) > > Thanks for the report, I can see the problem. > >> In the last form, project-buffers includes the current buffer (i.e. the >> scratch buffer in our example) with the results. (This is true even if >> the current buffer is visiting a file in some unrelated directory.) >> This matters because the command project-switch-project let-binds >> default-directory before calling project-switch-commands. This means >> that if you set project-switch-commands to some function that calls >> project-buffers, you will get incorrect results. >> For example, evaluate the following forms in order: >> (defun my-list-project-buffers () >> "List the current project's buffers." >> (interactive) >> (let ((buffer-list (project-buffers (project-current t))) >> (buffer-name (project-prefixed-buffer-name "my-project-buffer-list"))) >> (with-current-buffer (get-buffer-create buffer-name) >> (erase-buffer) >> (save-excursion >> (dolist (buffer buffer-list) >> (insert (buffer-name buffer)) >> (insert ?\n)))) >> (switch-to-buffer buffer-name))) >> (setq project-switch-commands #'my-list-project-buffers) > > Looks like a reimplementation of projectile-ibuffer, seems useful. Yeah, in my real configuration, I defined this as an ibuffer filter, but I thought a simpler example would be better for the bug report. > >> ;; list tmpfile but also scratch >> (project-switch-project "/tmp/") > > Not sure how to fix this, though. In bug#53626 we discussed a somewhat similar problem, and a let-binding seems impossible to "escape". > > What else can we do? One option is to change the signature of every compatible command to take the project object as its first argument. Might have been more realistic when the package was first written, too much breakage now, probably. > > Another would be to add a new var to help override the project choice without touch default-directory. > > Something like the attached. Please try it out.<project-current-directory-override.diff> I agree, the fix is not obvious. A workaround I used in my configuration is to add an advice to project-switch-project to wrap it in a with-temp-buffer form. This way, it’s only the default-directory of the temp buffer that gets corrupted. Not a very clean solution, but it seems to work. Unfortunately, you need to remember to do this in many places; for example, I had to do the same thing in my project-ibuffer command. Your solution looks cleaner. I gave it a try (along with disabling my advice), and it seems to work pretty well. Thanks for the fix! There might be some more places where it needs to be applied. For example, project-prefixed-buffer-name still inspects the default-directory. (Maybe project-prefixed-buffer-name should just call project-root or similar?) I think there’s still some fragility in the project-buffers function, since any callers need to be careful not to bind default-directory. It might be useful to call this out in the doc string or in the manual. Thanks for your help!
Hi Sean, On 31.10.2022 19:47, Sean Devlin wrote: >>> In the last form, project-buffers includes the current buffer (i.e. the >>> scratch buffer in our example) with the results. (This is true even if >>> the current buffer is visiting a file in some unrelated directory.) >>> This matters because the command project-switch-project let-binds >>> default-directory before calling project-switch-commands. This means >>> that if you set project-switch-commands to some function that calls >>> project-buffers, you will get incorrect results. >>> For example, evaluate the following forms in order: >>> (defun my-list-project-buffers () >>> "List the current project's buffers." >>> (interactive) >>> (let ((buffer-list (project-buffers (project-current t))) >>> (buffer-name (project-prefixed-buffer-name "my-project-buffer-list"))) >>> (with-current-buffer (get-buffer-create buffer-name) >>> (erase-buffer) >>> (save-excursion >>> (dolist (buffer buffer-list) >>> (insert (buffer-name buffer)) >>> (insert ?\n)))) >>> (switch-to-buffer buffer-name))) >>> (setq project-switch-commands #'my-list-project-buffers) >> >> Looks like a reimplementation of projectile-ibuffer, seems useful. > > Yeah, in my real configuration, I defined this as an ibuffer filter, but I thought a simpler example would be better for the bug report. Right, thanks. >>> ;; list tmpfile but also scratch >>> (project-switch-project "/tmp/") >> >> Not sure how to fix this, though. In bug#53626 we discussed a somewhat similar problem, and a let-binding seems impossible to "escape". >> >> What else can we do? One option is to change the signature of every compatible command to take the project object as its first argument. Might have been more realistic when the package was first written, too much breakage now, probably. >> >> Another would be to add a new var to help override the project choice without touch default-directory. >> >> Something like the attached. Please try it out.<project-current-directory-override.diff> > > I agree, the fix is not obvious. > > A workaround I used in my configuration is to add an advice to project-switch-project to wrap it in a with-temp-buffer form. This way, it’s only the default-directory of the temp buffer that gets corrupted. Not a very clean solution, but it seems to work. Unfortunately, you need to remember to do this in many places; for example, I had to do the same thing in my project-ibuffer command. > > Your solution looks cleaner. I gave it a try (along with disabling my advice), and it seems to work pretty well. Thanks for the fix! Actually I like your solution better. It might be less obvious when reading, but it's shorter and fully backward compatible. So I just pushed that fix to master. > There might be some more places where it needs to be applied. For example, project-prefixed-buffer-name still inspects the default-directory. (Maybe project-prefixed-buffer-name should just call project-root or similar?) In both places where project-prefixed-buffer-name is currently invoked, default-directory is set a specific binding just before that. So I think it would have been fine even with the other fix. Or on a higher level, it acts on the current buffer, so referencing default-directory seems okay. > I think there’s still some fragility in the project-buffers function, since any callers need to be careful not to bind default-directory. It might be useful to call this out in the doc string or in the manual. I suppose it could use improvement, but I'm not sure what phrasing would stop someone from making such a mistake. After all, I knew its implementation and made it anyway. Perhaps the docstring should simply say that the buffers are matched on the basis of their default-directory value. In the default implementation, that is (custom backends could choose their own strategy). Would that help?
Hi Dmitry, > On Nov 1, 2022, at 7:35 PM, Dmitry Gutov <dgutov@yandex.ru> wrote: > > Hi Sean, > > On 31.10.2022 19:47, Sean Devlin wrote: > >>>> In the last form, project-buffers includes the current buffer (i.e. the >>>> scratch buffer in our example) with the results. (This is true even if >>>> the current buffer is visiting a file in some unrelated directory.) >>>> This matters because the command project-switch-project let-binds >>>> default-directory before calling project-switch-commands. This means >>>> that if you set project-switch-commands to some function that calls >>>> project-buffers, you will get incorrect results. >>>> For example, evaluate the following forms in order: >>>> (defun my-list-project-buffers () >>>> "List the current project's buffers." >>>> (interactive) >>>> (let ((buffer-list (project-buffers (project-current t))) >>>> (buffer-name (project-prefixed-buffer-name "my-project-buffer-list"))) >>>> (with-current-buffer (get-buffer-create buffer-name) >>>> (erase-buffer) >>>> (save-excursion >>>> (dolist (buffer buffer-list) >>>> (insert (buffer-name buffer)) >>>> (insert ?\n)))) >>>> (switch-to-buffer buffer-name))) >>>> (setq project-switch-commands #'my-list-project-buffers) >>> >>> Looks like a reimplementation of projectile-ibuffer, seems useful. >> Yeah, in my real configuration, I defined this as an ibuffer filter, but I thought a simpler example would be better for the bug report. > > Right, thanks. > >>>> ;; list tmpfile but also scratch >>>> (project-switch-project "/tmp/") >>> >>> Not sure how to fix this, though. In bug#53626 we discussed a somewhat similar problem, and a let-binding seems impossible to "escape". >>> >>> What else can we do? One option is to change the signature of every compatible command to take the project object as its first argument. Might have been more realistic when the package was first written, too much breakage now, probably. >>> >>> Another would be to add a new var to help override the project choice without touch default-directory. >>> >>> Something like the attached. Please try it out.<project-current-directory-override.diff> >> I agree, the fix is not obvious. >> A workaround I used in my configuration is to add an advice to project-switch-project to wrap it in a with-temp-buffer form. This way, it’s only the default-directory of the temp buffer that gets corrupted. Not a very clean solution, but it seems to work. Unfortunately, you need to remember to do this in many places; for example, I had to do the same thing in my project-ibuffer command. >> Your solution looks cleaner. I gave it a try (along with disabling my advice), and it seems to work pretty well. Thanks for the fix! > > Actually I like your solution better. It might be less obvious when reading, but it's shorter and fully backward compatible. > > So I just pushed that fix to master. Great, thanks! > >> There might be some more places where it needs to be applied. For example, project-prefixed-buffer-name still inspects the default-directory. (Maybe project-prefixed-buffer-name should just call project-root or similar?) > > In both places where project-prefixed-buffer-name is currently invoked, default-directory is set a specific binding just before that. So I think it would have been fine even with the other fix. > > Or on a higher level, it acts on the current buffer, so referencing default-directory seems okay. Sounds good. > >> I think there’s still some fragility in the project-buffers function, since any callers need to be careful not to bind default-directory. It might be useful to call this out in the doc string or in the manual. > > I suppose it could use improvement, but I'm not sure what phrasing would stop someone from making such a mistake. After all, I knew its implementation and made it anyway. > > Perhaps the docstring should simply say that the buffers are matched on the basis of their default-directory value. In the default implementation, that is (custom backends could choose their own strategy). Would that help? Yeah, I think a high-level description of the default strategy would be useful. Thanks again for your help!
On 02.11.2022 17:18, Sean Devlin wrote:
>>> I think there’s still some fragility in the project-buffers function, since any callers need to be careful not to bind default-directory. It might be useful to call this out in the doc string or in the manual.
>>
>> I suppose it could use improvement, but I'm not sure what phrasing would stop someone from making such a mistake. After all, I knew its implementation and made it anyway.
>>
>> Perhaps the docstring should simply say that the buffers are matched on the basis of their default-directory value. In the default implementation, that is (custom backends could choose their own strategy). Would that help?
>
> Yeah, I think a high-level description of the default strategy would be useful.
>
> Thanks again for your help!
I've added some clarification in 7d47651d01.
Hopefully it makes things better, if not -- suggestions welcome.
I'm going to close this in the meantime.
Hi Dmitry,
> On Nov 3, 2022, at 9:18 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 02.11.2022 17:18, Sean Devlin wrote:
>
>>>> I think there’s still some fragility in the project-buffers function, since any callers need to be careful not to bind default-directory. It might be useful to call this out in the doc string or in the manual.
>>>
>>> I suppose it could use improvement, but I'm not sure what phrasing would stop someone from making such a mistake. After all, I knew its implementation and made it anyway.
>>>
>>> Perhaps the docstring should simply say that the buffers are matched on the basis of their default-directory value. In the default implementation, that is (custom backends could choose their own strategy). Would that help?
>> Yeah, I think a high-level description of the default strategy would be useful.
>> Thanks again for your help!
>
> I've added some clarification in 7d47651d01.
>
> Hopefully it makes things better, if not -- suggestions welcome.
>
> I'm going to close this in the meantime.
Thanks, and I saw the corresponding discussion on emacs-devel. Any version of the proposed changes looks good to me. Feel free to close, and thanks again for your help!
> So I just pushed that fix to master.
Unfortunately, this change broke 'C-x p p g':
@@ -1667,9 +1667,10 @@ project-switch-project
- (let ((default-directory dir)
- (project-current-inhibit-prompt t))
- (call-interactively command))))
+ (with-temp-buffer
+ (let ((default-directory dir)
+ (project-current-inhibit-prompt t))
+ (call-interactively command)))))
Because 'C-x p p' switches to a temporary buffer, then
'g' calls 'project-find-regexp' and 'project--read-regexp'.
But (thing-at-point 'symbol t) can't get a symbol at point
in the empty temporary buffer.
On 21/11/22 20:24, Juri Linkov wrote:
>> So I just pushed that fix to master.
> Unfortunately, this change broke 'C-x p p g':
>
> @@ -1667,9 +1667,10 @@ project-switch-project
> - (let ((default-directory dir)
> - (project-current-inhibit-prompt t))
> - (call-interactively command))))
> + (with-temp-buffer
> + (let ((default-directory dir)
> + (project-current-inhibit-prompt t))
> + (call-interactively command)))))
>
> Because 'C-x p p' switches to a temporary buffer, then
> 'g' calls 'project-find-regexp' and 'project--read-regexp'.
> But (thing-at-point 'symbol t) can't get a symbol at point
> in the empty temporary buffer.
Damn it. :-)
Thanks for noticing. I'll need a little time to think about the options,
but we'll probably go with the previous proposed fix.
On 23/11/22 05:48, Dmitry Gutov wrote:
> we'll probably go with the previous proposed fix
And now done, master b37604c263.