Bug? org-id-find not finding some IDs

Bug? org-id-find not finding some IDs

[Jack Kamm]

Hello,

The documentation for org-id-extra-files suggests that org-id-find
should search in Org files visited by Emacs:

> When Org reparses files to remake the list of files and IDs it is
> tracking, it will normally scan the agenda files, the archives
> related to agenda files, ... and any Org file currently visited by
> Emacs.

But, I've found that's not true, or maybe I'm misunderstanding the
doc.

For a minimal example, put the following in test.org:

> * Headline 1
> :PROPERTIES:
> :ID:       b1569c92-3872-4ab9-80eb-274c4950991e
> :END:

Then:

1. emacs -Q test.org
2. C-x b *scratch*
3. M-: (org-id-find "b1569c92-3872-4ab9-80eb-274c4950991e")

Which returns nil. Also, "C-h v org-id-files" doesn't contain
test.org.

Or maybe the docstring for org-id-extra-files should be rewritten to
say "the current buffer" instead of "any Org file currently visited"?

Also, this seems to be the cause of this bug in org-caldav, which is
how I ran into it:
https://github.com/dengste/org-caldav/issues/230

Would there a recommended mitigation for this on org-caldav's side? I'm
thinking it could call

(org-id-update-id-locations org-caldav-files)

early on, to make sure org-caldav-files ends up in org-id-files.

Tested on following versions:
GNU Emacs 28.1 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo version 1.17.6, Xaw3d scroll bars) of 2022-07-10
Org mode version 9.5.2 (release_9.5.2-25-gaf6f12 @ /usr/share/emacs/28.1/lisp/org/)
Org mode version 9.5.4 (release_9.5.4-702-g5a49cc @ /home/jack/dev/org-mode/lisp/)

Re: Bug? org-id-find not finding some IDs

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Jack Kamm <jackkamm@tatersworld.org> writes:

> For a minimal example, put the following in test.org:
>
>> * Headline 1
>> :PROPERTIES:
>> :ID:       b1569c92-3872-4ab9-80eb-274c4950991e
>> :END:
>
> Then:
>
> 1. emacs -Q test.org
> 2. C-x b *scratch*
> 3. M-: (org-id-find "b1569c92-3872-4ab9-80eb-274c4950991e")
>
> Which returns nil. Also, "C-h v org-id-files" doesn't contain
> test.org.

This is indeed inconsistent. The current behaviour is to scan only files
where it is known that IDs are present.

Can you try the attached patch?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-id-update-locations-Scan-all-the-open-Org-buffer.patch --]
[-- Type: text/x-patch, Size: 1644 bytes --]

From 5d166219c9af1f34dccebac55249db926fcfbbe1 Mon Sep 17 00:00:00 2001
Message-Id: <5d166219c9af1f34dccebac55249db926fcfbbe1.1659579786.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 4 Aug 2022 10:21:59 +0800
Subject: [PATCH] org-id-update-locations: Scan all the open Org buffers

* lisp/org-id.el (org-id-update-id-locations): Always scan all the
open Org file buffers for IDs.

Fixes https://orgmode.org/list/87r11zifjy.fsf@gmail.com
---
 lisp/org-id.el | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lisp/org-id.el b/lisp/org-id.el
index f54e7280a..fe2f5f328 100644
--- a/lisp/org-id.el
+++ b/lisp/org-id.el
@@ -491,8 +491,8 @@ (defun org-id-decode (id)
 (defun org-id-update-id-locations (&optional files silent)
   "Scan relevant files for IDs.
 Store the relation between files and corresponding IDs.
-This will scan all agenda files, all associated archives, and all
-files currently mentioned in `org-id-locations'.
+This will scan all agenda files, all associated archives, all open Org
+files, and all files currently mentioned in `org-id-locations'.
 When FILES is given, scan also these files.
 If SILENT is non-nil, messages are suppressed."
   (interactive)
@@ -515,6 +515,8 @@ (defun org-id-update-id-locations (&optional files silent)
 		       org-id-extra-files)
 		     ;; All files known to have IDs.
 		     org-id-files
+                     ;; All Org files open in Emacs.
+                     (mapcar #'buffer-file-name (org-buffer-list 'files t))
 		     ;; Additional files from function call.
 		     files)))))
          (nfiles (length files))
-- 
2.35.1

Re: Bug? org-id-find not finding some IDs

[Jack Kamm]

Ihor Radchenko <yantar92@gmail.com> writes:

> This is indeed inconsistent. The current behaviour is to scan only files
> where it is known that IDs are present.
>
> Can you try the attached patch?

Thanks, Ihor. I tried your patch and it solved my issue -- both the
minimal example I sent, as well as the related issue I was experiencing
with org-caldav.

Jack

Re: Bug? org-id-find not finding some IDs

[Ihor Radchenko]

Jack Kamm <jackkamm@tatersworld.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> This is indeed inconsistent. The current behaviour is to scan only files
>> where it is known that IDs are present.
>>
>> Can you try the attached patch?
>
> Thanks, Ihor. I tried your patch and it solved my issue -- both the
> minimal example I sent, as well as the related issue I was experiencing
> with org-caldav.

Applied onto main via 8f5bf1725.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8f5bf172556564df89fb16ce8ecec68c5b7f0221

Best,
Ihor

> Jack

-- 
Ihor Radchenko,
Org mode maintainer,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Bug? org-id-find not finding some IDs

[Jack Kamm]

Ihor Radchenko <yantar92@gmail.com> writes:

> Applied onto main via 8f5bf1725.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8f5bf172556564df89fb16ce8ecec68c5b7f0221

My sincere apologies, but after a bit of testing, I found that my
requested change had some unforeseen consequences, and I don't think
it's a good idea anymore. So, I'd like to request reverting it.

Doing some research, I found the behavior of org-id-update-id-locations,
to not search all the open Org files, was made in this 2019 commit:

https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9865e6bd8be65229be4eac4f459f62e47fab2be7

The commit message suggests this change in behavior was
intentional. Only, the docstring of org-id-extra-files wasn't updated to
reflect the new behavior, which was the cause of confusion on my end.

In terms of unforeseen consequences of the new commit: I found it caused
some problems for org-caldav. When making changes, org-caldav copies
entries into a backup file at ~/.emacs.d/org-caldav-backup.org. With the
new commit, it is sometimes finding entries in this backup file instead
of the correct file.

To fix the original problem I was experiencing with org-caldav, I think
it may be better to modify org-caldav, instead of upstream Org-mode. To
that end, I submitted this PR to org-caldav:

https://github.com/dengste/org-caldav/pull/250

Re: Bug? org-id-find not finding some IDs

[Ihor Radchenko]

Jack Kamm <jackkamm@tatersworld.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Applied onto main via 8f5bf1725.
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8f5bf172556564df89fb16ce8ecec68c5b7f0221
>
> My sincere apologies, but after a bit of testing, I found that my
> requested change had some unforeseen consequences, and I don't think
> it's a good idea anymore. So, I'd like to request reverting it.
>
> Doing some research, I found the behavior of org-id-update-id-locations,
> to not search all the open Org files, was made in this 2019 commit:
>
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=9865e6bd8be65229be4eac4f459f62e47fab2be7
>
> The commit message suggests this change in behavior was
> intentional. Only, the docstring of org-id-extra-files wasn't updated to
> reflect the new behavior, which was the cause of confusion on my end.

That particular change has not been noticed in the relevant ML thread.
I disagree that skipping the open ID files is the right decision.
We already have org-id-get unconditionally adding arbitrary files
to the ID locations. This org-id-get can be triggered in various
unpredictable places inside Org and third-party packages. Thus, we cannot
really expect consistency even after reverting the commit.

> In terms of unforeseen consequences of the new commit: I found it caused
> some problems for org-caldav. When making changes, org-caldav copies
> entries into a backup file at ~/.emacs.d/org-caldav-backup.org. With the
> new commit, it is sometimes finding entries in this backup file instead
> of the correct file.

I suggest to bind org-id-track-globally to nil file-locally in that
backup file. I believe that org-id-track-globally, when set to t, is
expected to catch all the Org files in Emacs + user-specified files.
Skipping some opened files unpredictably, like it was done before
my commit is chaotic and also caused confusion in this area in my own
Org usage.

If necessary, we may add an extra customization like
org-id-exclude-files to explicitly exclude files from ID tracking.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Bug? org-id-find not finding some IDs

[Jack Kamm]

Ihor Radchenko <yantar92@gmail.com> writes:

> I suggest to bind org-id-track-globally to nil file-locally in that
> backup file.

Thanks for the suggestion. I tried it out, and it initially seemed to
work, but eventually the backup file ended up in org-id-files. It seems
like there is more than one way that files may be added to org-id-files,
and not all will respect a locally-set org-id-track-globally.

However, I found another solution: we can use write-region to append the
entry to org-caldav-backup.org, without opening it. Then this avoids the
problem in normal usage of org-caldav: org-caldav-backup.org won't end
up in org-id-files, since it won't be open in Emacs. I've updated my PR
to org-caldav accordingly.

> If necessary, we may add an extra customization like
> org-id-exclude-files to explicitly exclude files from ID tracking.

I think this would be nice, but it isn't necessary for the time being,
since I found another solution for org-caldav, as described above.