From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Matt Armstrong Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Fri, 10 Dec 2021 21:13:01 -0800 Message-ID: <87bl1nhggy.fsf@rfc20.org> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> <87a6h8y5j6.fsf@gnus.org> <87fsr0gssu.fsf@rfc20.org> <877dcbncfd.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6011"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Richard Stallman , eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org, Pip Cet , Lars Ingebrigtsen , eliz@gnu.org To: Ihor Radchenko , Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Dec 11 06:14:50 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvuig-0001NE-FI for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 06:14:50 +0100 Original-Received: from localhost ([::1]:33744 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mvuie-0000Dw-LN for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 00:14:48 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:60928) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvuh7-0007xP-J1 for emacs-devel@gnu.org; Sat, 11 Dec 2021 00:13:13 -0500 Original-Received: from relay3-d.mail.gandi.net ([217.70.183.195]:45327) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvuh5-0000O7-9s; Sat, 11 Dec 2021 00:13:13 -0500 Original-Received: (Authenticated sender: matt@rfc20.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id B7ABC60006; Sat, 11 Dec 2021 05:13:04 +0000 (UTC) Original-Received: from matt by naz with local (Exim 4.95) (envelope-from ) id 1mvugv-0055cY-81; Fri, 10 Dec 2021 21:13:01 -0800 In-Reply-To: <877dcbncfd.fsf@localhost> Received-SPF: pass client-ip=217.70.183.195; envelope-from=matt@rfc20.org; helo=relay3-d.mail.gandi.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281648 Archived-At: Ihor Radchenko writes: > Stefan Monnier writes: > >>> Another example is Org Roam (https://www.orgroam.com/), which uses >>> sqlite to manage its index of Org metadata. >> >> Could you give details? >> >> I don't know what is Org-roam (and the above web-page describes it in >> terms that don't speak to me very much). What does it store in the >> database? Why did it choose to use sqlite rather than plain text >> (e.g. simple personal preference on the part of the author, or was >> a text file tried at first but proved too inefficient, if so which >> part)? What kind of database size is common? > > Org-roam stores Org headline metadata like headline title, todo-state, > scheduling info, properties, tags, links to other headings, etc. > > AFAIU, they use sqlite for easier integration with third-party > tools. For example, links between headings can be visualised using > existing web-libraries (see > https://github.com/org-roam/org-roam-ui/blob/main/package.json) > > As for the DB size, I am not sure. However, it should scale with the > number of headings. You may think of 10-100k headline DB records and a > multiple of that number for links between headings. The author gave a presentation on org-roam in EmacsConf 2020: https://emacsconf.org/2020/talks/16/. It is a Zettelkasten note taking system for Emacs built on top of Org. I believe that org-roam uses sqlite primarily for performance, as an index over many the small org files the packages encourages users to create. I suspect that the use case is much like Gnus. I have a memory (that I can't verify as fact now) of the early version using awk to extract the needed info from the org files directly, but this proved to be slow in practice. I suspect that people have found ways to use the database for other things is a secondary benefit.