From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Tomas Hlavaty Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Sat, 11 Dec 2021 16:38:10 +0100 Message-ID: <87bl1ngnj1.fsf@logand.com> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> <87bl1olyhr.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8821"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Richard Stallman , eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org, Pip Cet , Lars Ingebrigtsen , eliz@gnu.org To: Ihor Radchenko , Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Dec 11 16:39:20 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mw4Sz-00024t-H8 for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 16:39:17 +0100 Original-Received: from localhost ([::1]:43406 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mw4Sx-0007yN-Oy for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 10:39:15 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:40230) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mw4S9-0007KA-4o for emacs-devel@gnu.org; Sat, 11 Dec 2021 10:38:25 -0500 Original-Received: from logand.com ([37.48.87.44]:33458) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mw4S7-0006FB-Bo; Sat, 11 Dec 2021 10:38:24 -0500 Original-Received: by logand.com (Postfix, from userid 1001) id C5D0719EC53; Sat, 11 Dec 2021 16:38:11 +0100 (CET) X-Mailer: emacs 27.2 (via feedmail 11-beta-1 I) In-Reply-To: <87bl1olyhr.fsf@localhost> Received-SPF: pass client-ip=37.48.87.44; envelope-from=tom@logand.com; helo=logand.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281694 Archived-At: On Sat 11 Dec 2021 at 09:30, Ihor Radchenko wrote: > Stefan Monnier writes: > >>> There will, of course, be large amounts of data, and that will be slow. >>> (People already do this with a number things, and the results are >>> predictably sluggish.) >> >> To help me understand this discussion, I think it would help me to have >> examples of such large databases currently implemented as text files, >> along with an idea of what "large" means in this context (how many MBs) >> and where the "slow"ness manifests itself. > > I can think of two examples of large text databases: elfeed and recent > org-mode's caching (org-persist). > > The slowness manifests itself on reading/printing(saving) the database > and from the requirement to load the whole database into memory for > queries. > > We are talking about 300Mb for elfeed and up to 20-30Mb per object > (usually below 1Mb though) for org-persist with 100-200Mb total object > size. On the other hand, my notmuch (emacs email reader) database is 30GB and lives externally out of emacs without issues. Why link sqlite and not xapian? Where does it stop?