Feature request: IDs for everything

Feature request: IDs for everything

[Tor Erlend Fjelde]

Hi all,

I was wondering if there's a reason why we couldn't have IDs a la org-id.el for everything?
It seem to me that it would be useful to use something like `#+ID` in place of `#+NAME` for tables, blocks, etc. as well as headlines.
Would this go against the intended design of org-id.el, or is this a change that would be welcome but that no one has gotten around to implementing yet?

Also, sorry if this is not correct way to go about this; it's my first time posting to a mailing list. Let me know if there are some guidelines or something somewhere that I should read and make use of.

All the best,
Tor

Re: Feature request: IDs for everything

[Rodrigo Morales]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

Currently, headlines can have an ID (see minimal working example below):

#+BEGIN_SRC org
* My headline 1
* My headline 2
:PROPERTIES:
:ID: e8745be0-906d-4e02-b427-d298f5751f6c
:END:
#+END_SRC

Blocks can't have IDs, but you could use a UUID as the for blocks (see
minimal working example below). You can do the same with tables.

#+BEGIN_SRC org
,#+NAME: 412f567b-26ce-4f21-b07f-f5296c830314
,#+BEGIN_SRC sh
seq 1 3
,#+END_SRC

,#+RESULTS: 412f567b-26ce-4f21-b07f-f5296c830314
,#+begin_example
1
2
3
,#+end_example

,#+HEADER: :noweb yes
,#+BEGIN_SRC sh
echo foo <<412f567b-26ce-4f21-b07f-f5296c830314()>>
,#+END_SRC

,#+RESULTS:
,#+begin_example
foo 1
foo 2
foo 3
foo
,#+end_example
#+END_SRC

What I understand from your message is that you would like to have a new
header argument called #+ID: that could be used for code blocks and tables.
Am I right? If so, I have a question: What could be the added benefits of
having such a header argument? I can think of this benefit: #+NAME would be
used for referencing them through a human-friendly name and the name could
change and the #+ID would be an UUID that is not expected to change.

On Fri, 20 Oct 2023 at 19:13, Tor Erlend Fjelde <tor.github@gmail.com>
wrote:

> Hi all,
>
> I was wondering if there's a reason why we couldn't have IDs a la
> org-id.el for everything?
> It seem to me that it would be useful to use something like `#+ID` in
> place of `#+NAME` for tables, blocks, etc. as well as headlines.
> Would this go against the intended design of org-id.el, or is this a
> change that would be welcome but that no one has gotten around to
> implementing yet?
>
> Also, sorry if this is not correct way to go about this; it's my first
> time posting to a mailing list. Let me know if there are some guidelines or
> something somewhere that I should read and make use of.
>
> All the best,
> Tor
>
>

[-- Attachment #2: Type: text/html, Size: 2467 bytes --]

Re: Feature request: IDs for everything

[Tor Erlend Fjelde]

> What could be the added benefits of having such a header argument? I can think of this benefit: #+NAME would be used for referencing them through a human-friendly name and the name could change and the #+ID would be an UUID that is not expected to change.

This is indeed one use-case I had in mind!


Generally speaking, I would imagine the motivation is the same as for ID vs. CUSTOM_ID for headlines? As in, most arguments made for the introduction of org-id.el in the first place can be made about linking to blocks, etc. too? I haven't seen the original discussions around this, so I'm only speeculating here.

What I specifically had in mind is mainly about what you can expect from an ID vs. NAME. IIUC the ID property is meant to be a "globally" unique identifier, while something like NAME is generally only meant to only be /locally/ unique.
Hence, if you want to interact with such references programmatically, it's useful to have an a "globally" consistent way of referencing not just headlines, but also blocks, etc.

A fairly prominent example is org-roam.el, which, when they moved to v2, decided to solely rely on IDs rather than allowing all standard forms of Org links exactly because "global" consistency is so important for them.

org-id.el also just provides a lot of convenient functionality, e.g. automatic creation of ID if it does not yet exist, which would similarly benefit linking to non-files and -headlines.

Hope that clearifies a bit!

All the best,
Tor


On Fri, 20/10/2023, Rodrigo Morales <moralesrodrigo1100@gmail.com> wrote:

> Currently, headlines can have an ID (see minimal working example below):
>
> #+BEGIN_SRC org
> * My headline 1
> * My headline 2
> :PROPERTIES:
> :ID: e8745be0-906d-4e02-b427-d298f5751f6c
> :END:
> #+END_SRC
>
> Blocks can't have IDs, but you could use a UUID as the for blocks (see
> minimal working example below). You can do the same with tables.
>
> #+BEGIN_SRC org
> ,#+NAME: 412f567b-26ce-4f21-b07f-f5296c830314
> ,#+BEGIN_SRC sh
> seq 1 3
> ,#+END_SRC
>
> ,#+RESULTS: 412f567b-26ce-4f21-b07f-f5296c830314
> ,#+begin_example
> 1
> 2
> 3
> ,#+end_example
>
> ,#+HEADER: :noweb yes
> ,#+BEGIN_SRC sh
> echo foo <<412f567b-26ce-4f21-b07f-f5296c830314()>>
> ,#+END_SRC
>
> ,#+RESULTS:
> ,#+begin_example
> foo 1
> foo 2
> foo 3
> foo
> ,#+end_example
> #+END_SRC
>
> What I understand from your message is that you would like to have a new
> header argument called #+ID: that could be used for code blocks and tables.
> Am I right? If so, I have a question: What could be the added benefits of
> having such a header argument? I can think of this benefit: #+NAME would be
> used for referencing them through a human-friendly name and the name could
> change and the #+ID would be an UUID that is not expected to change.
>
> On Fri, 20 Oct 2023 at 19:13, Tor Erlend Fjelde <tor.github@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I was wondering if there's a reason why we couldn't have IDs a la
>> org-id.el for everything?
>> It seem to me that it would be useful to use something like `#+ID` in
>> place of `#+NAME` for tables, blocks, etc. as well as headlines.
>> Would this go against the intended design of org-id.el, or is this a
>> change that would be welcome but that no one has gotten around to
>> implementing yet?
>>
>> Also, sorry if this is not correct way to go about this; it's my first
>> time posting to a mailing list. Let me know if there are some guidelines or
>> something somewhere that I should read and make use of.
>>
>> All the best,
>> Tor
>>
>>

Re: Feature request: IDs for everything

[Ihor Radchenko]

Tor Erlend Fjelde <tor.github@gmail.com> writes:

> I was wondering if there's a reason why we couldn't have IDs a la
> org-id.el for everything? It seem to me that it would be useful to use
> something like `#+ID` in place of `#+NAME` for tables, blocks, etc. as
> well as headlines.

This has been discussed in the past.

Apart from #+NAME, we also have radio <<<targets>>> that can be used a
link anchors (#+NAME or other affiliated keywords are not allowed, for
example, in items).

We also discussed linking to #+name and <<<target>>> globally, without
specifying the file path.

> Would this go against the intended design of
> org-id.el, or is this a change that would be welcome but that no one
> has gotten around to implementing yet?

Mostly nobody has gotten around.

Although we have at least one caveat that we need to consider - the
current users of the id: links blindly assume that they always link to
headings. This includes many third-party packages, like org-roam.

If we simply allow id: links to point to non-headings, it will be a
major breaking change that may affect third-party packages. So, we
need to design the extended ids carefully to avoid breakage. For
example, `org-id-find' and other API function may need to work in two
modes: (1) legacy, only searching for headings; (2) new, searching for
anything. This can, for example, be done via an extra optional argument.

-----

As an alternative option, we had a proposal that extends id: links to
have a search option:

 [[id:<ID>:search-string]]

Then, we may have top-level drawer like

:PROPERTIES:
:ID: unique-file-id
:END:

And then refer to anything inside the file as

 [[id:unique-file-id:object-id-inside-the-file]]

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Feature request: IDs for everything

[Tor Erlend Fjelde]

> Although we have at least one caveat that we need to consider - the
> current users of the id: links blindly assume that they always link to
> headings. This includes many third-party packages, like org-roam.
>
> If we simply allow id: links to point to non-headings, it will be a
> major breaking change that may affect third-party packages. So, we
> need to design the extended ids carefully to avoid breakage. For
> example, `org-id-find' and other API function may need to work in two
> modes: (1) legacy, only searching for headings; (2) new, searching for
> anything. This can, for example, be done via an extra optional argument.

This is a very good point, and definitively makes things non-trivial.
Nonetheless, I'd be happy to have a go at it if this seems like the way
to go! But I'll probably need quite a bit of help in the process as I
haven't contributed to Org before.

> As an alternative option, we had a proposal that extends id: links to
> have a search option:
>
>  [[id:<ID>:search-string]]
>
> Then, we may have top-level drawer like
>
> :PROPERTIES:
> :ID: unique-file-id
> :END:
>
> And then refer to anything inside the file as
>
>  [[id:unique-file-id:object-id-inside-the-file]]

Indeed; I was actually about to have a go at implementing this because
I thought this would be the quickest way of adding support for referencing
blocks in something like org-roam. But this does seem like a sub-optimal
solution vs. just adding support for more general IDs, and so I thought
it would be better to see if that could be done first (hence I'm here).

All the best,
Tor


On Sat, 21/10/2023, Ihor Radchenko <yantar92@posteo.net> wrote:

> Tor Erlend Fjelde <tor.github@gmail.com> writes:
>
>> I was wondering if there's a reason why we couldn't have IDs a la
>> org-id.el for everything? It seem to me that it would be useful to use
>> something like `#+ID` in place of `#+NAME` for tables, blocks, etc. as
>> well as headlines.
>
> This has been discussed in the past.
>
> Apart from #+NAME, we also have radio <<<targets>>> that can be used a
> link anchors (#+NAME or other affiliated keywords are not allowed, for
> example, in items).
>
> We also discussed linking to #+name and <<<target>>> globally, without
> specifying the file path.
>
>> Would this go against the intended design of
>> org-id.el, or is this a change that would be welcome but that no one
>> has gotten around to implementing yet?
>
> Mostly nobody has gotten around.
>
> Although we have at least one caveat that we need to consider - the
> current users of the id: links blindly assume that they always link to
> headings. This includes many third-party packages, like org-roam.
>
> If we simply allow id: links to point to non-headings, it will be a
> major breaking change that may affect third-party packages. So, we
> need to design the extended ids carefully to avoid breakage. For
> example, `org-id-find' and other API function may need to work in two
> modes: (1) legacy, only searching for headings; (2) new, searching for
> anything. This can, for example, be done via an extra optional argument.
>
> -----
>
> As an alternative option, we had a proposal that extends id: links to
> have a search option:
>
>  [[id:<ID>:search-string]]
>
> Then, we may have top-level drawer like
>
> :PROPERTIES:
> :ID: unique-file-id
> :END:
>
> And then refer to anything inside the file as
>
>  [[id:unique-file-id:object-id-inside-the-file]]
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Feature request: IDs for everything

[Ihor Radchenko]

Tor Erlend Fjelde <tor.github@gmail.com> writes:

>> If we simply allow id: links to point to non-headings, it will be a
>> major breaking change that may affect third-party packages. So, we
>> need to design the extended ids carefully to avoid breakage. For
>> example, `org-id-find' and other API function may need to work in two
>> modes: (1) legacy, only searching for headings; (2) new, searching for
>> anything. This can, for example, be done via an extra optional argument.
>
> This is a very good point, and definitively makes things non-trivial.
> Nonetheless, I'd be happy to have a go at it if this seems like the way
> to go! But I'll probably need quite a bit of help in the process as I
> haven't contributed to Org before.

This is a moderately difficult contribution. As a first step, you will
need to study org-id.el and see what should be changed there to support
the new feature.

If you encounter any difficulties, feel free to ask here.

Also, before you invest too much time into this, please check if you can
do FSF copyright assignment (see
https://orgmode.org/worg/org-contribute.html#copyright). As a GNU
project, we cannot accept large contributions without this paperwork.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Feature request: IDs for everything

[Ihor Radchenko]

Tor Erlend Fjelde <tor.github@gmail.com> writes:

>> As an alternative option, we had a proposal that extends id: links to
>> have a search option:
>>
>>  [[id:<ID>::search-string]]
> ...
> Indeed; I was actually about to have a go at implementing this because
> I thought this would be the quickest way of adding support for referencing
> blocks in something like org-roam. But this does seem like a sub-optimal
> solution vs. just adding support for more general IDs, and so I thought
> it would be better to see if that could be done first (hence I'm here).

FYI, we added [[id:<ID>::search-string]] links to the latest development
version of Org mode. This new feature does not cover global ids for
non-headings though.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Feature request: IDs for everything

[Tor Erlend Fjelde]

Oh, that's great news! Will be very useful; thanks Ihor!

Unfortunately life has come in the way so I haven't had the chance to pursue this further at the moment.

All the best,
Tor

On Wed, 06/03/2024, Ihor Radchenko <yantar92@posteo.net> wrote:

> Tor Erlend Fjelde <tor.github@gmail.com> writes:
>
>>> As an alternative option, we had a proposal that extends id: links to
>>> have a search option:
>>>
>>>  [[id:<ID>::search-string]]
>> ...
>> Indeed; I was actually about to have a go at implementing this because
>> I thought this would be the quickest way of adding support for referencing
>> blocks in something like org-roam. But this does seem like a sub-optimal
>> solution vs. just adding support for more general IDs, and so I thought
>> it would be better to see if that could be done first (hence I'm here).
>
> FYI, we added [[id:<ID>::search-string]] links to the latest development
> version of Org mode. This new feature does not cover global ids for
> non-headings though.
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Feature request: IDs for everything

[Max Nikulin]

On 21/10/2023 20:05, Ihor Radchenko wrote:
> Tor Erlend Fjelde writes:
> 
>> I was wondering if there's a reason why we couldn't have IDs a la
>> org-id.el for everything? It seem to me that it would be useful to use
>> something like `#+ID` in place of `#+NAME` for tables, blocks, etc. as
>> well as headlines.

I think, another complication is referring source code blocks as 
variable values in header arguments and as noweb fragments. It would be 
great to be possible to specify "id:UUID" instead of fuzzy names. 
Perhaps it may be solved by explicit calls to new functions. If 
possible, I would avoid proliferation of keyword in favor of "#+name: 
id:UUID"

And there are references to particular lines inside code blocks...

> Apart from #+NAME, we also have radio <<<targets>>> that can be used a
> link anchors (#+NAME or other affiliated keywords are not allowed, for
> example, in items).

I think, you mean <<anchors>>, not <<<radio>>> targets that activates 
plain text words. It seems, the latter cannot be currently exported as 
explicit links [[radio][Radio]]. I am unsure what conflicts may appear 
during attempt to allow optional explicit types, e.g. <<id:UUID>>.

> We also discussed linking to #+name and <<<target>>> globally, without
> specifying the file path.

 From my point of view, it should be either global id:UUID links or 
file:doc.org::name local links.

> If we simply allow id: links to point to non-headings, it will be a
> major breaking change that may affect third-party packages. So, we
> need to design the extended ids carefully to avoid breakage.

A defcusom user options whether id:UUID links for non-heading elements 
are allowed. It is just an opinion/idea and nothing more.

Are ids for whole files (placed before first headings) are problematic 
for 3rd party packages?

>   [[id:unique-file-id:object-id-inside-the-file]]

Perhaps than it should be id:FILE-UUID::SEARCH with usual search options 
like #CUSTOM_ID, *Heading, etc., with the new variant id:OBJECT-UUID. 
However I am unsure if search should be limited to a subtree if 
HEADING-UUID is specified instead of FILE-UUID.

Re: Feature request: IDs for everything

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> If we simply allow id: links to point to non-headings, it will be a
>> major breaking change that may affect third-party packages. So, we
>> need to design the extended ids carefully to avoid breakage.
>
> A defcusom user options whether id:UUID links for non-heading elements 
> are allowed. It is just an opinion/idea and nothing more.

> Are ids for whole files (placed before first headings) are problematic 
> for 3rd party packages?

Fair point. Even existing id links may not always point to a heading
(when pointing to IDs in top-level property drawer). So, the potential
breakage I was talking about is already there, and third-party packages
already have to adapt.

I still do not feel confident that it will be safe to blindly change
org-id in the proposed way though. Although I cannot think of clear
examples why it would cause problems worse compared to the existing id:
links to files. So, it may just be my unmotivated concern.

>>   [[id:unique-file-id:object-id-inside-the-file]]
>
> Perhaps than it should be id:FILE-UUID::SEARCH with usual search options 
> like #CUSTOM_ID, *Heading, etc., with the new variant id:OBJECT-UUID. 
> However I am unsure if search should be limited to a subtree if 
> HEADING-UUID is specified instead of FILE-UUID.

I indeed meant "::" search option. Sorry for the typo.

In any case, I agree with Tor that search option is less elegant. If we
can go for direct UUIDs, let's do it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>