From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Tassilo Horn Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Fri, 10 Dec 2021 14:14:22 +0100 Message-ID: <877dccsi69.fsf@gnu.org> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="4253"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.7.5; emacs 29.0.50 Cc: Richard Stallman , eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org, Pip Cet , Lars Ingebrigtsen , eliz@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Dec 10 14:31:40 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvfzw-0000vK-DS for ged-emacs-devel@m.gmane-mx.org; Fri, 10 Dec 2021 14:31:40 +0100 Original-Received: from localhost ([::1]:60030 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mvfzv-0005mo-Fd for ged-emacs-devel@m.gmane-mx.org; Fri, 10 Dec 2021 08:31:39 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:59914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvfxZ-0004NP-0q for emacs-devel@gnu.org; Fri, 10 Dec 2021 08:29:13 -0500 Original-Received: from [2001:470:142:3::e] (port=47248 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvfxL-00059s-3c; Fri, 10 Dec 2021 08:29:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-reply-to:Date:Subject:To:From: References; bh=xCqvoHPiUjeO5ObeGcokNHR+IFUsd5eP9c3PVdyNkSo=; b=nUWTdL6oy1PUq0 EzWS5Xj2UOtq1WjfUaHIlOL4NBLvWvnuoGg4Navh//0vB8QK/njWHyR9zhW7RAht7TSrtEVUNpVNf oqKBXEWF2LMyZeD7q5/iSHKxj1kpGsvVHhRmbWU+9TilfhmB9r15HPPfNy6LSgqiVFWfCYlkvXXaR Y7Fh07CR7rvvC/Z1rlT53/dQ9N0Fa+oecQS3ZGmfeiLG70e/C87WcvHXTOLD7+zKcCJMKtK2RRxCT gsKbpabEk/D+lb0jGvu2US6XcQLLeH3X2V5+ggHjLwO273KPlB3J+xChnvduyzE6TdUK278IaO+GU 9iaVBPqYDvPEL4ca5mGw==; Original-Received: from auth1-smtp.messagingengine.com ([66.111.4.227]:55555) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mvfwy-0000q7-P2; Fri, 10 Dec 2021 08:28:37 -0500 Original-Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailauth.nyi.internal (Postfix) with ESMTP id 8C8B227C0054; Fri, 10 Dec 2021 08:28:34 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Fri, 10 Dec 2021 08:28:34 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrkedvgdehfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpehffgfhvffuffgjkfggtgesthdtredttdertdenucfhrhhomhepvfgrshhsihhl ohcujfhorhhnuceothhsughhsehgnhhurdhorhhgqeenucggtffrrghtthgvrhhnpeevve eikeetkeeviefgfeffiedvteeguddvffeuueduveegtddthedvhfeuveffhfenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhorhhnodhmvg hsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdekieejfeekjeekgedqieefhedvleek qdhtshguhheppehgnhhurdhorhhgsehfrghsthhmrghilhdrfhhm X-ME-Proxy: Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 Dec 2021 08:28:32 -0500 (EST) In-reply-to: X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281596 Archived-At: Stefan Monnier writes: >> There will, of course, be large amounts of data, and that will be >> slow. (People already do this with a number things, and the results >> are predictably sluggish.) > > To help me understand this discussion, I think it would help me to > have examples of such large databases currently implemented as text > files, along with an idea of what "large" means in this context (how > many MBs) and where the "slow"ness manifests itself. I think the Gnus registry (info "(gnus) The Gnus Registry") is a good example. Mine is 3.6 MB but only because I've set `gnus-registry-max-entries' to 10000 and `gnus-registry-track-extra' to nil because I've basically only ever used it to be able to refer to an article's parent. There should be no reason to cap its number of entries or the number of things it tracks. By default it would also track sender, subject, and recipients of messages. I guess that would easily quadruple its size for me. I also think the Gnus registry is an example of where a DB with a specific schema and queries would make sense instead of some persistent variable. There's no value in having that huge pile of hash-maps in memory if I just want to, e.g., move the current mail into the folder where its parent already resides. Bye, Tassilo