From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Qiantan Hong Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Sat, 11 Dec 2021 14:13:53 +0000 Message-ID: <492F4AC1-2230-4D4C-9D8E-19659FBFB50B@mit.edu> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> <87bl1olyhr.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27091"; mail-complaints-to="usenet@ciao.gmane.io" Cc: "rms@gnu.org" , Eric Abrahamsen , cesar mena , Ihor Radchenko , "emacs-devel@gnu.org" , Pip Cet , "larsi@gnus.org" , Eli Zaretskii To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Dec 11 15:16:00 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mw3AN-0006ue-W0 for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 15:15:59 +0100 Original-Received: from localhost ([::1]:45846 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mw3AM-0002FZ-S7 for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 09:15:58 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:51852) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mw38f-0000uq-93 for emacs-devel@gnu.org; Sat, 11 Dec 2021 09:14:17 -0500 Original-Received: from outgoing-exchange-7.mit.edu ([18.9.28.58]:33662) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mw38d-0003Ni-H1; Sat, 11 Dec 2021 09:14:12 -0500 Original-Received: from oc11exedge1.exchange.mit.edu (OC11EXEDGE1.EXCHANGE.MIT.EDU [18.9.3.17]) by outgoing-exchange-7.mit.edu (8.14.7/8.12.4) with ESMTP id 1BBEDsVA022319; Sat, 11 Dec 2021 09:13:54 -0500 Original-Received: from oc11expo16.exchange.mit.edu (18.9.4.47) by oc11exedge1.exchange.mit.edu (18.9.3.17) with Microsoft SMTP Server (TLS) id 15.0.1497.26; Sat, 11 Dec 2021 09:13:10 -0500 Original-Received: from oc11expo16.exchange.mit.edu (18.9.4.47) by oc11expo16.exchange.mit.edu (18.9.4.47) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Sat, 11 Dec 2021 09:13:53 -0500 Original-Received: from oc11expo16.exchange.mit.edu ([18.9.4.47]) by oc11expo16.exchange.mit.edu ([18.9.4.47]) with mapi id 15.00.1497.023; Sat, 11 Dec 2021 09:13:53 -0500 Thread-Topic: persistent data feature Thread-Index: AQHX7YIgabWO2VQXG0Wud+XjYmeytqwtq3SA In-Reply-To: Accept-Language: en-US Content-Language: en-US x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [128.12.11.144] Content-ID: <04F39641301BE6419A8E16591770F436@exchange.mit.edu> Received-SPF: pass client-ip=18.9.28.58; envelope-from=qhong@mit.edu; helo=outgoing-exchange-7.mit.edu X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281691 Archived-At: > On Dec 11, 2021, at 5:53 AM, Stefan Monnier wr= ote: >=20 >>>> There will, of course, be large amounts of data, and that will be slow= . >>>> (People already do this with a number things, and the results are >>>> predictably sluggish.) >>>=20 >>> To help me understand this discussion, I think it would help me to have >>> examples of such large databases currently implemented as text files, >>> along with an idea of what "large" means in this context (how many MBs) >>> and where the "slow"ness manifests itself. >>=20 >> I can think of two examples of large text databases: elfeed and recent >> org-mode's caching (org-persist). >=20 > I think I'm beginning to see a pattern: most of those uses (same for > org-roam) are for cache-like data, i.e. data that can be recreated from > other data. Interesting. >=20 > I suspect the same holds for gnus-registry, but doesn't for ecomplete. >=20 > Cache-like data has the property that you don't need to version it, and > users should never need to manipulate it directly: if there's a problem > with it you should be able to just delete it and start over. >=20 > For such cases, there's no great benefit to use a "transparent" text > representation (but it's important to provide some way to rebuild the > cache). I think a good fit to that would be incremental image. Is it possible to have Emacs save/load part of the heap (e.g. reachable from a root object) quickly, potentially by directly mapping the memory in the load case? Is pdumper useful for that purpose?=