From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Tassilo Horn Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Fri, 10 Dec 2021 21:29:44 +0100 Message-ID: <874k7gmadu.fsf@gnu.org> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> <877dccsi69.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35419"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.7.5; emacs 29.0.50 Cc: "rms@gnu.org" , Eric Abrahamsen , cesar mena , emacs-devel@gnu.org, Stefan Monnier , Pip Cet , "larsi@gnus.org" , Eli Zaretskii To: Qiantan Hong Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Dec 10 22:14:26 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvnDl-0008yq-VM for ged-emacs-devel@m.gmane-mx.org; Fri, 10 Dec 2021 22:14:26 +0100 Original-Received: from localhost ([::1]:39698 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mvnDk-0004hl-BA for ged-emacs-devel@m.gmane-mx.org; Fri, 10 Dec 2021 16:14:24 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:42572) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvnCo-0003kW-9R for emacs-devel@gnu.org; Fri, 10 Dec 2021 16:13:26 -0500 Original-Received: from [2001:470:142:3::e] (port=49178 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvnCn-0003wI-FB; Fri, 10 Dec 2021 16:13:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-reply-to:Date:Subject:To:From: References; bh=HUI+QS7OTk9VO0TZIEphjO5zSDeu2vm74KqMS7c1jTA=; b=JZL/wIqjGdhRxp BaW42E4ZowY6WqhXg+gmPD7IUxyHOhL12aFWVj38XgMkTL2kevZyI4UmioZAX5eXqVgZ/mkHQQOfP WXpLPqqu+v3hdatsPHdoRf1mIORgjHX+grK5cEJx2q/RZUXDm0Kbrpq9Rd/5EqpoWCKB5qmv3mdbx TAiETiFRWFgUW2f0d1l8h7OL7r34hsF+M2Yb2SI8FZ7G3mt4/AzGbpMAcREc91E+C9i+734uXPg6r 4aF/gXnx3wsMUMLJLufSkvTp54faxUAs4NSZSSjaTpGSLmwbfZLtzXadBi9pOOE3WnpCmz6QEoguH Zb+dn5sNeuDkAvZx8a5w==; Original-Received: from auth2-smtp.messagingengine.com ([66.111.4.228]:58285) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mvnCn-0001HB-6D; Fri, 10 Dec 2021 16:13:25 -0500 Original-Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailauth.nyi.internal (Postfix) with ESMTP id 9058227C0054; Fri, 10 Dec 2021 16:13:22 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 10 Dec 2021 16:13:22 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrkedvgddugeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfhgfhffvufffjgfkgggtgfesthhqredttderjeenucfhrhhomhepvfgrshhs ihhlohcujfhorhhnuceothhsughhsehgnhhurdhorhhgqeenucggtffrrghtthgvrhhnpe eiudfghffhjeevvedvhfffgeetleeljefffeeggfeugeegleehfeeiueejhfehgeenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhorhhnod hmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdekieejfeekjeekgedqieefhedv leekqdhtshguhheppehgnhhurdhorhhgsehfrghsthhmrghilhdrfhhm X-ME-Proxy: Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 Dec 2021 16:13:19 -0500 (EST) In-reply-to: X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281629 Archived-At: Qiantan Hong writes: >> There should be no reason to cap its number of entries or the number >> of things it tracks. By default it would also track sender, subject, >> and recipients of messages. I guess that would easily quadruple its >> size for me. > I agree the number should not be restricted. I will try to implement > split bucketing in probably the next^2 revision of resist! then. > > There=E2=80=99s still a technical question unanswered. If I=E2=80=99m to = use multiple > files to store one kv store, is it better to let user supply a > directory, or let user supply a filename and we add suffixes? I'd use a directory but don't care too much. >> I also think the Gnus registry is an example of where a DB with a >> specific schema and queries would make sense instead of some >> persistent variable. There's no value in having that huge pile of >> hash-maps in memory if I just want to, e.g., move the current mail >> into the folder where its parent already resides. > I think in your case, even if the db quadruples, it=E2=80=99s still < 20M= B in > textual representation and probably much smaller when resided in the > heap. It=E2=80=99s still a pretty long way to go before memory-residence > become problematic. Sure, but as said I've restricted the registry to 10000 entries. I actually have 100000 mails in my spool and maybe the same number of newsgroup articles. Then we're around 300 MB and that's a size I'd start worrying about. And my main point is that only a tiny fraction of that data will ever be used but which of it will be used cannot be known beforehand. A typical scenario for the Gnus registry is: given a single message, show the complete thread. Unless you have one store per thread, that's a query which will inevitably require loading large parts if not all of the data into memory. Real databases are good in evaluating such queries quickly without excessive memory usage. Bye, Tassilo