org-id-locations as a large-scale database store?

org-id-locations as a large-scale database store?

[Laurence von Bottorff]

[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]

I've been using a package called org-brain that is using, I'm all but
certain, the org-id-locations file as its database to store graph-like
relationships between org files and org headings. org-brain is a sort of
graph database which adds a PROPERTIES drawer with just the ID field with a
UUID to an org file, as well as to the org headings in each file. So I
opened this 127k file residing in my .emacs.d directory and apparently
whatever is in my org directory and has a PROPERTIES drawer with ID field
winds up in .org-id-locations represented as plists in one big list, one
for each file that has one or more drawers. But you knowledgeable people
here know all this already. However, it surprised me since such a format as
this

(...("~/Dropbox/org/orgb_fassungen/FassungenDesc.org"
"1ba78dfc-d6c3-46c5-827d-37db94fa9053"
"6809202d-9a4c-4ed0-87fb-798676f448fe"
"f2149157-6ba1-4e24-81c3-4e070fcc5de8"
"5b7c2d9a-2df8-450a-a0c8-9d3b8e703044"
"f3031b9f-3708-493d-9546-f489835c9f17")
("~/Dropbox/org/orgb_fassungen/Fassungen.org"
"e9b683af-90b1-4ece-9afd-18dc28a9fab2")...)

where, e.g., inside the file FassungenDesc.org there are many org headings,
each with a PROPERTIES/ID drawer -- and they're all associated
to FassungenDesc.org with that simple list. So I'm asking if this is a good
design for a really big graph database? I'd like to start using org files
and their containing headings as vertices. But I'm guessing this is not
really all that scalable, not really intended for such a big thing. I saw
another emacs graph database (https://github.com/toshism/netz) that uses
h.el hash to store graph vertices and edges. So again, is this org-brain
approach DOA for anything big? I plan on having potentially hundreds of org
files with very many heading entries each (over a thousand perhaps per
file) representing vertices. What would be a better way of databasing org
files and headings? The PROPERTIES/ID drawer idea is fine, but not throwing
it all in .org-id-locations, right?

Ancillary: If I move or delete .org-id-locations, Emacs will rebuild it,
right? What is the idea behind having such a file structured this way --
other than a simple database? What else uses it?

-- 
⨽
Lawrence Bottorff
Grand Marais, MN, USA
borgauf@gmail.com

[-- Attachment #2: Type: text/html, Size: 2771 bytes --]

Re: org-id-locations as a large-scale database store?

[yeti]

Laurence von Bottorff <borgauf@gmail.com> writes:

> So again, is this org-brain approach DOA for anything big?

I like it, but even my small pool of notes is slow with org-brain.

-------------------------------8<---------------------------------------
~ $ hash-table-size $org-id-locations
1322
------------------------------->8---------------------------------------

-- 
I do not bite, I just want to play.

Re: org-id-locations as a large-scale database store?

[Rick Lupton]

You might be interested in https://www.orgroam.com/ which is a broadly similar idea to org-brain but uses a SQLite database to track id locations and links.

Re: org-id-locations as a large-scale database store?

[Ihor Radchenko]

yeti <yeti@tilde.institute> writes:

> Laurence von Bottorff <borgauf@gmail.com> writes:
>
>> So again, is this org-brain approach DOA for anything big?
>
> I like it, but even my small pool of notes is slow with org-brain.
>
> -------------------------------8<---------------------------------------
> ~ $ hash-table-size $org-id-locations
> 1322
> ------------------------------->8---------------------------------------

Does org-brain have anything to do with org-id-locations?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-id-locations as a large-scale database store?

[yeti]

Ihor Radchenko <yantar92@posteo.net> writes:

> yeti <yeti@tilde.institute> writes:
>
>> Laurence von Bottorff <borgauf@gmail.com> writes:
>>
>>> So again, is this org-brain approach DOA for anything big?
>>
>> I like it, but even my small pool of notes is slow with org-brain.
>>
>> -------------------------------8<---------------------------------------
>> ~ $ hash-table-size $org-id-locations
>> 1322
>> ------------------------------->8---------------------------------------
>
> Does org-brain have anything to do with org-id-locations?

I think so...

E.g. org-brain.el 788-794:

-------------------------------8<---------------------------------------
(defun org-brain-entry-from-id (id)
  "Get entry from ID."
  (unless org-id-locations (org-id-locations-load))
  (when-let ((path (gethash id org-id-locations)))
    (list (org-brain-path-entry-name path)
          (org-brain-headline-at (org-id-find id t))
          id)))
------------------------------->8---------------------------------------

-- 
I do not bite, I just want to play.

Re: org-id-locations as a large-scale database store?

[Ihor Radchenko]

yeti <yeti@tilde.institute> writes:

>> Does org-brain have anything to do with org-id-locations?
>
> I think so...
>
> E.g. org-brain.el 788-794:
>
> -------------------------------8<---------------------------------------
> (defun org-brain-entry-from-id (id)
>   "Get entry from ID."
>   (unless org-id-locations (org-id-locations-load))
>   (when-let ((path (gethash id org-id-locations)))
>     (list (org-brain-path-entry-name path)
>           (org-brain-headline-at (org-id-find id t))
>           id)))

That's redundant. Calling `org-id-find' would suffice (org-id-find
internally arranges org-id-locations to be updated).

In any case, loading and searching id locations scales well for
thousands of entries - I currently have 60+k entries there without
issues.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-id-locations as a large-scale database store?

[Laurence von Bottorff]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

Any docs on what org-id-locations is about, who/what uses it?

On Tue, Mar 12, 2024 at 1:44 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> yeti <yeti@tilde.institute> writes:
>
> >> Does org-brain have anything to do with org-id-locations?
> >
> > I think so...
> >
> > E.g. org-brain.el 788-794:
> >
> > -------------------------------8<---------------------------------------
> > (defun org-brain-entry-from-id (id)
> >   "Get entry from ID."
> >   (unless org-id-locations (org-id-locations-load))
> >   (when-let ((path (gethash id org-id-locations)))
> >     (list (org-brain-path-entry-name path)
> >           (org-brain-headline-at (org-id-find id t))
> >           id)))
>
> That's redundant. Calling `org-id-find' would suffice (org-id-find
> internally arranges org-id-locations to be updated).
>
> In any case, loading and searching id locations scales well for
> thousands of entries - I currently have 60+k entries there without
> issues.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>
>

-- 
⨽
Lawrence Bottorff
Grand Marais, MN, USA
borgauf@gmail.com

[-- Attachment #2: Type: text/html, Size: 2207 bytes --]

Re: org-id-locations as a large-scale database store?

[Ihor Radchenko]

Laurence von Bottorff <borgauf@gmail.com> writes:

> Any docs on what org-id-locations is about, who/what uses it?

It is an internal variable.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>