[Bug] org-store-link should not insert a document level ID property

[Bug] org-store-link should not insert a document level ID property

[Matt Lundin]

The introduction of document-level property drawers (commit
1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
inconsistencies in the behavior of org-id and org-store-link.

If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
calling org-store-link above the first headline in an org file will
insert a PROPERTY drawer and an ID at top of the file, so that the file
(call it "~/test.org") looks like this:

--8<---------------cut here---------------start------------->8---
:PROPERTIES:
:ID:       1f43e860-9e7b-4c8f-82b9-6ed3352e589f
:END:

* First headline
--8<---------------cut here---------------end--------------->8---

However, the link that Org actually stores is "[[file:~/test.org]]", so
the ID is irrelevant.

In addition, a link to a document-level ID does not work. Following
"[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:

user-error: Before first headline at position 14 in buffer test.org

So either: 

1. org-id and org-store-link/org-open-link should support document level
   ids in a user-configurable way, or
2. org-id-get-create should detect whether it is above the first heading
   and should not create an id

Option #2 would obviously be the easier fix.

Gustav: were IDs within the scope of your initial thinking about
document level properties? Or are these largely irrelevant?

Best,
Matt

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -----Original Message-----
> From: Matt Lundin <mdl@imapmail.org>
> Sent: den 5 april 2020 00:13
> To: Org Mode List <emacs-orgmode@gnu.org>
> Cc: Gustav Wikström <gustav@whil.se>
> Subject: [Bug] org-store-link should not insert a document level ID property
> 
> The introduction of document-level property drawers (commit
> 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> inconsistencies in the behavior of org-id and org-store-link.
> 
> If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> calling org-store-link above the first headline in an org file will
> insert a PROPERTY drawer and an ID at top of the file, so that the file
> (call it "~/test.org") looks like this:
> 
> --8<---------------cut here---------------start------------->8---
> :PROPERTIES:
> :ID:       1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> :END:
> 
> * First headline
> --8<---------------cut here---------------end--------------->8---
> 
> However, the link that Org actually stores is "[[file:~/test.org]]", so
> the ID is irrelevant.
> 
> In addition, a link to a document-level ID does not work. Following
> "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> 
> user-error: Before first headline at position 14 in buffer test.org
> 
> So either:
> 
> 1. org-id and org-store-link/org-open-link should support document level
>    ids in a user-configurable way, or
> 2. org-id-get-create should detect whether it is above the first heading
>    and should not create an id
> 
> Option #2 would obviously be the easier fix.
> 
> Gustav: were IDs within the scope of your initial thinking about
> document level properties? Or are these largely irrelevant?

Yes, they were in scope and ID should mean something also before first headline. I'll have a look to see what can be done. Thanks for finding this!

> 
> Best,
> Matt

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

[-- Attachment #1: Type: text/plain, Size: 2403 bytes --]

Hi again,

Patch is attached. It's not applied yet as it doesn't include anything about user-configuration yet. @Matt Lundin, care to elaborate what you had in mind in terms of that?

With this patch a link before first headline is stored with the filename (no path) as description. Following the link does what you'd expect. Tests ran fine with the patch applied.

Regards
Gustav

> Hi,
> 
> > -----Original Message-----
> > From: Matt Lundin <mdl@imapmail.org>
> > Sent: den 5 april 2020 00:13
> > To: Org Mode List <emacs-orgmode@gnu.org>
> > Cc: Gustav Wikström <gustav@whil.se>
> > Subject: [Bug] org-store-link should not insert a document level ID property
> >
> > The introduction of document-level property drawers (commit
> > 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> > inconsistencies in the behavior of org-id and org-store-link.
> >
> > If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> > calling org-store-link above the first headline in an org file will
> > insert a PROPERTY drawer and an ID at top of the file, so that the file
> > (call it "~/test.org") looks like this:
> >
> > --8<---------------cut here---------------start------------->8---
> > :PROPERTIES:
> > :ID:       1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> > :END:
> >
> > * First headline
> > --8<---------------cut here---------------end--------------->8---
> >
> > However, the link that Org actually stores is "[[file:~/test.org]]", so
> > the ID is irrelevant.
> >
> > In addition, a link to a document-level ID does not work. Following
> > "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> >
> > user-error: Before first headline at position 14 in buffer test.org
> >
> > So either:
> >
> > 1. org-id and org-store-link/org-open-link should support document level
> >    ids in a user-configurable way, or
> > 2. org-id-get-create should detect whether it is above the first heading
> >    and should not create an id
> >
> > Option #2 would obviously be the easier fix.
> >
> > Gustav: were IDs within the scope of your initial thinking about
> > document level properties? Or are these largely irrelevant?
> 
> Yes, they were in scope and ID should mean something also before first
> headline. I'll have a look to see what can be done. Thanks for finding this!
> 
> >
> > Best,
> > Matt

[-- Attachment #2: 0001-Allow-storing-and-following-ID-links-before-first-he.patch --]
[-- Type: application/octet-stream, Size: 1781 bytes --]

From 88adb690f6de58b116cd59965aba14f3cfa7d490 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gustav=20Wikstr=C3=B6m?= <gustav@whil.se>
Date: Sun, 5 Apr 2020 17:36:01 +0200
Subject: [PATCH] Allow storing and following ID links before first heading

* lisp/org.el (org-find-property): Make org-find-property work
  correctly also before first heading.

* lisp/org-id.el (org-id-store-link): Make it possible to store ID
  links also before first heading.
---
 lisp/org-id.el | 7 +++++--
 lisp/org.el    | 3 ++-
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/lisp/org-id.el b/lisp/org-id.el
index d0fdeeb1b..1ff433bd8 100644
--- a/lisp/org-id.el
+++ b/lisp/org-id.el
@@ -665,8 +665,11 @@ optional argument MARKERP, return the position as a new marker."
     (let* ((link (concat "id:" (org-id-get-create)))
 	   (case-fold-search nil)
 	   (desc (save-excursion
-		   (org-back-to-heading t)
-		   (or (and (looking-at org-complex-heading-regexp)
+		   (org-back-to-heading-or-point-min t)
+		   (or (and (org-before-first-heading-p)
+			    (file-name-nondirectory
+			     (buffer-file-name (buffer-base-buffer))))
+		       (and (looking-at org-complex-heading-regexp)
 			    (if (match-end 4)
 				(match-string 4)
 			      (match-string 0)))
diff --git a/lisp/org.el b/lisp/org.el
index 06891b8bd..bdfeafdfb 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -13133,7 +13133,8 @@ part of the buffer."
 	(while (re-search-forward re nil t)
 	  (when (if value (org-at-property-p)
 		  (org-entry-get (point) property nil t))
-	    (throw 'exit (progn (org-back-to-heading t) (point)))))))))
+	    (throw 'exit (progn (org-back-to-heading-or-point-min t)
+				(point)))))))))
 
 (defun org-delete-property (property)
   "In the current entry, delete PROPERTY."
-- 
2.26.0

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

Just wanted to say that the patch is applied on master. It's solving the problem. Regarding user-configuration that can be added later if needed. It's not wrong right now per se.

Cheers!
Gustav

> -----Original Message-----
> From: Gustav Wikström
> Sent: den 5 april 2020 17:50
> To: Matt Lundin <mdl@imapmail.org>; Org Mode List <emacs-orgmode@gnu.org>
> Subject: RE: [Bug] org-store-link should not insert a document level ID
> property
> 
> Hi again,
> 
> Patch is attached. It's not applied yet as it doesn't include anything about
> user-configuration yet. @Matt Lundin, care to elaborate what you had in mind
> in terms of that?
> 
> With this patch a link before first headline is stored with the filename (no
> path) as description. Following the link does what you'd expect. Tests ran
> fine with the patch applied.
> 
> Regards
> Gustav
> 
> > Hi,
> >
> > > -----Original Message-----
> > > From: Matt Lundin <mdl@imapmail.org>
> > > Sent: den 5 april 2020 00:13
> > > To: Org Mode List <emacs-orgmode@gnu.org>
> > > Cc: Gustav Wikström <gustav@whil.se>
> > > Subject: [Bug] org-store-link should not insert a document level ID
> property
> > >
> > > The introduction of document-level property drawers (commit
> > > 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> > > inconsistencies in the behavior of org-id and org-store-link.
> > >
> > > If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> > > calling org-store-link above the first headline in an org file will
> > > insert a PROPERTY drawer and an ID at top of the file, so that the file
> > > (call it "~/test.org") looks like this:
> > >
> > > --8<---------------cut here---------------start------------->8---
> > > :PROPERTIES:
> > > :ID:       1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> > > :END:
> > >
> > > * First headline
> > > --8<---------------cut here---------------end--------------->8---
> > >
> > > However, the link that Org actually stores is "[[file:~/test.org]]", so
> > > the ID is irrelevant.
> > >
> > > In addition, a link to a document-level ID does not work. Following
> > > "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> > >
> > > user-error: Before first headline at position 14 in buffer test.org
> > >
> > > So either:
> > >
> > > 1. org-id and org-store-link/org-open-link should support document level
> > >    ids in a user-configurable way, or
> > > 2. org-id-get-create should detect whether it is above the first heading
> > >    and should not create an id
> > >
> > > Option #2 would obviously be the easier fix.
> > >
> > > Gustav: were IDs within the scope of your initial thinking about
> > > document level properties? Or are these largely irrelevant?
> >
> > Yes, they were in scope and ID should mean something also before first
> > headline. I'll have a look to see what can be done. Thanks for finding
> this!
> >
> > >
> > > Best,
> > > Matt

RE: [Bug] org-store-link should not insert a document level ID property

[Matthew Lundin]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

Gustav Wikström <gustav@whil.se> writes:

> Hi again,
>
> Patch is attached. It's not applied yet as it doesn't include anything
> about user-configuration yet. @Matt Lundin, care to elaborate what you
> had in mind in terms of that?
>
> With this patch a link before first headline is stored with the
> filename (no path) as description. Following the link does what you'd
> expect. Tests ran fine with the patch applied.

Hi Gustav,

Sorry it's taken so long to get back to you. Thanks for applying this.
It makes the behavior consistent.

What I was thinking of in terms of configuration is being able to
preserve path-based links (instead of IDs) if creating a link above the
first headline. This is the behavior that existed in the past when
org-id-link-to-org-use-id was set to t or
'create-if-interactive-and-no-custom-id.

I've attached a patch that implements this. However, I also know that we
are trying to avoid configuration/feature creep in Org, so I'm
submitting this here for comments. Is this a configuration other people
want?

If not, it would be easy enough for me to hack this through advice-add.

Best,

Matt


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-configuration-of-whether-to-create-IDs-before-.patch --]
[-- Type: text/x-patch, Size: 2275 bytes --]

From 758c7c513c6e5e0457b483dcf2bf7c9299d1015b Mon Sep 17 00:00:00 2001
From: Matt Lundin <mdl@imapmail.org>
Date: Thu, 7 May 2020 16:31:05 -0500
Subject: [PATCH 1/1] Allow configuration of whether to create IDs before first
 heading

This allows users to keep file (path-based) links above the first
heading of a file.

* lisp/ol.el: (org-store-link) New test for whether to use ID links
* lisp/org-id.el: (org-id-create-id-before-first-heading) New variable
---
 lisp/ol.el     |  5 ++++-
 lisp/org-id.el | 11 +++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/lisp/ol.el b/lisp/ol.el
index 0cb1b0a7e..d072ad9f1 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -35,6 +35,7 @@
 (defvar org-comment-string)
 (defvar org-highlight-links)
 (defvar org-id-link-to-org-use-id)
+(defvar org-id-create-id-before-first-heading)
 (defvar org-inhibit-startup)
 (defvar org-outline-regexp-bol)
 (defvar org-src-source-file-name)
@@ -1619,7 +1620,9 @@ non-nil."
 			     (and (eq org-id-link-to-org-use-id
 				      'create-if-interactive-and-no-custom-id)
 				  (not custom-id))))
-		    (and org-id-link-to-org-use-id (org-entry-get nil "ID"))))
+		    (and org-id-link-to-org-use-id (org-entry-get nil "ID")))
+		(or (not (org-before-first-heading-p))
+		    org-id-create-id-before-first-heading))
 	   ;; Store a link using the ID at point
 	   (setq link (condition-case nil
 			  (prog1 (org-id-store-link)
diff --git a/lisp/org-id.el b/lisp/org-id.el
index 34720b7f2..8792aa3cb 100644
--- a/lisp/org-id.el
+++ b/lisp/org-id.el
@@ -123,6 +123,17 @@ nil   Never use an ID to make a link, instead link using a text search for
 	  (const :tag "Only use existing" use-existing)
 	  (const :tag "Do not use ID to create link" nil)))
 
+(defcustom org-id-create-id-before-first-heading t
+  "Non-nil means storing a link before the first heading will use IDs.
+This determines how `org-store-link' generates links before the
+first heading in an Org file when `org-id-link-to-org-use-id' is
+configured to create IDs.
+
+When nil, `org-store-link' will create a normal file link instead
+of an ID if before the first heading."
+  :group 'org-id
+  :type 'boolean)
+
 (defcustom org-id-uuid-program "uuidgen"
   "The uuidgen program."
   :group 'org-id
-- 
2.26.2

Re: [Bug] org-store-link should not insert a document level ID property

[Nicolas Goaziou]

Hello,

Matthew Lundin <mdl@imapmail.org> writes:

> Gustav Wikström <gustav@whil.se> writes:
>
>> Hi again,
>>
>> Patch is attached. It's not applied yet as it doesn't include anything
>> about user-configuration yet. @Matt Lundin, care to elaborate what you
>> had in mind in terms of that?
>>
>> With this patch a link before first headline is stored with the
>> filename (no path) as description. Following the link does what you'd
>> expect. Tests ran fine with the patch applied.
>
> Hi Gustav,
>
> Sorry it's taken so long to get back to you. Thanks for applying this.
> It makes the behavior consistent.
>
> What I was thinking of in terms of configuration is being able to
> preserve path-based links (instead of IDs) if creating a link above the
> first headline. This is the behavior that existed in the past when
> org-id-link-to-org-use-id was set to t or
> 'create-if-interactive-and-no-custom-id.

I don't understand what is the meaning of an ID property for a whole
document. 

AFAIK, ID is associated to a file name, and possibly a location in it.
In this case, the ID is strictly equivalent to the file name, so why
bother?

Regards,

-- 
Nicolas Goaziou

Re: [Bug] org-store-link should not insert a document level ID property

[Matthew Lundin]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Matthew Lundin <mdl@imapmail.org> writes:
>
>> What I was thinking of in terms of configuration is being able to
>> preserve path-based links (instead of IDs) if creating a link above the
>> first headline. This is the behavior that existed in the past when
>> org-id-link-to-org-use-id was set to t or
>> 'create-if-interactive-and-no-custom-id.
>
> I don't understand what is the meaning of an ID property for a whole
> document. 
>
> AFAIK, ID is associated to a file name, and possibly a location in it.
> In this case, the ID is strictly equivalent to the file name, so why
> bother?

I'm not sure I understand the question. Are you asking: Why bother
generating IDs at the top level of a file (which was the change Gustav
introduced)? Or why bother turning off that behavior? I can't address
the former question but I will address the latter. The main reason is
that I find these IDs redundant and visually distracting. I can see how
file IDs would be useful if one is constantly renaming files (or perhaps
writing custom functions that convert files to entries and vice versa).
But in other ways they are more fragile than paths, since a :PROPERTIES:
drawer at the top of a file looks like clutter and is *very* tempting to
delete:

---- beginning of file 
:PROPERTIES:
:ID:       d4ef67e6-ffcd-4df3-b821-b92c0138eb9c
:END:
#+FILETAGS: work inbox
#+CATEGORY: work
---- file continues...

That said, I'm happy to hack together a personal solution by advising
org-id-store-link if we decide not to allow users to customize this
behavior.

Best,
Matt

Re: [Bug] org-store-link should not insert a document level ID property

[Nicolas Goaziou]

Matthew Lundin <mdl@imapmail.org> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>> AFAIK, ID is associated to a file name, and possibly a location in it.
>> In this case, the ID is strictly equivalent to the file name, so why
>> bother?
>
> I'm not sure I understand the question. Are you asking: Why bother
> generating IDs at the top level of a file (which was the change Gustav
> introduced)? Or why bother turning off that behavior? I can't address
> the former question but I will address the latter. 

Sorry for not being clear. This was the first question. I don't
understand why we are generating an ID for the whole file.

> The main reason is that I find these IDs redundant and visually
> distracting. I can see how file IDs would be useful if one is
> constantly renaming files (or perhaps writing custom functions that
> convert files to entries and vice versa).

IIUC, currently, renaming the file breaks the association between the ID
and the file name. IOW, the ID is useless if you rename the file.

Regards,

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi Matt,

> -----Original Message-----
> From: Matthew Lundin <mdl@imapmail.org>
> Sent: den 7 maj 2020 23:41
> To: Gustav Wikström <gustav@whil.se>; Org Mode List <emacs-orgmode@gnu.org>
> Subject: RE: [Bug] org-store-link should not insert a document level ID
> property
> 
> [...]
>
> What I was thinking of in terms of configuration is being able to
> preserve path-based links (instead of IDs) if creating a link above the
> first headline. This is the behavior that existed in the past when
> org-id-link-to-org-use-id was set to t or
> 'create-if-interactive-and-no-custom-id.
> 
> I've attached a patch that implements this. However, I also know that we
> are trying to avoid configuration/feature creep in Org, so I'm
> submitting this here for comments. Is this a configuration other people
> want?

This is a very specific customization. If it was to be implemented I would probably rename it a bit for its purpose to be more clear. My personal opinion of course. Name could be something in the line of "org-id-inhibit-id-before-first-heading" and make it clear in the docstring that the customization really only takes effect if org-id-link-to-org-use-id is not nil. That way one won't wonder as much about the interplay between the functions.

What say you? :)

Kind regards
Gustav

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Sent: den 13 maj 2020 18:45
> To: Matthew Lundin <mdl@imapmail.org>
> Cc: Gustav Wikström <gustav@whil.se>; Org Mode List <emacs-orgmode@gnu.org>
> Subject: Re: [Bug] org-store-link should not insert a document level ID
> property
> 
> Matthew Lundin <mdl@imapmail.org> writes:
> 
> > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> >>
> >> AFAIK, ID is associated to a file name, and possibly a location in it.
> >> In this case, the ID is strictly equivalent to the file name, so why
> >> bother?
> >
> > I'm not sure I understand the question. Are you asking: Why bother
> > generating IDs at the top level of a file (which was the change Gustav
> > introduced)? Or why bother turning off that behavior? I can't address
> > the former question but I will address the latter.
> 
> Sorry for not being clear. This was the first question. I don't
> understand why we are generating an ID for the whole file.

I see some purposes. They maybe aren't for everyone, but they surely are there. 
1) Attachments for the whole file without setting the DIR property.
2) Link targets, making the file name irrelevant for the links.

> > The main reason is that I find these IDs redundant and visually
> > distracting. I can see how file IDs would be useful if one is
> > constantly renaming files (or perhaps writing custom functions that
> > convert files to entries and vice versa).
> 
> IIUC, currently, renaming the file breaks the association between the ID
> and the file name. IOW, the ID is useless if you rename the file.

As long as org-id-track-globally isn't set to nil I don't think what you write is true. And even if renaming the file would break the link it's just momentarily since regenerating the .org-id-locations file should make the link work again.

Regards
Gustav

Re: [Bug] org-store-link should not insert a document level ID property

[Nicolas Goaziou]

Hello,

Gustav Wikström <gustav@whil.se> writes:

> I see some purposes. They maybe aren't for everyone, but they surely are there. 
> 1) Attachments for the whole file without setting the DIR property.

OK. This makes sense, thank you.

> 2) Link targets, making the file name irrelevant for the links.

Again, the file name is relevant, so using id links instead of file
links in this situation doesn't buy us anything.

>> IIUC, currently, renaming the file breaks the association between the ID
>> and the file name. IOW, the ID is useless if you rename the file.
>
> As long as org-id-track-globally isn't set to nil I don't think what
> you write is true. And even if renaming the file would break the link
> it's just momentarily since regenerating the .org-id-locations file
> should make the link work again.

Regenerating ".org-id-locations" is not magical. It looks into
a predefined set of files. If the renaming moves the file out of this
set, there is no way it can associate again the ID to that file.

IOW, I wouldn't trust an ID more than a file name.

Regards,

-- 
Nicolas Goaziou

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -----Original Message-----
> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Sent: den 14 maj 2020 23:21
> To: Gustav Wikström <gustav@whil.se>
> Cc: Matthew Lundin <mdl@imapmail.org>; Org Mode List <emacs-orgmode@gnu.org>
> Subject: Re: [Bug] org-store-link should not insert a document level ID
> property
> 
> [...]
> 
> Regenerating ".org-id-locations" is not magical. It looks into
> a predefined set of files. If the renaming moves the file out of this
> set, there is no way it can associate again the ID to that file.

The same argument could be made for ID on a heading, if that heading is refiled into a location outside of that predefined set of files.

> IOW, I wouldn't trust an ID more than a file name.

But I would. Not entirely unlikely for files to be renamed within my folders. Using ID's helps for that. Be it to a file or a heading. But of course, YMMW.

Kind regards
Gustav