From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Robin Tarsiger Newsgroups: gmane.emacs.devel Subject: Re: sqlite3 usage for multisession variable storage Date: Tue, 14 Dec 2021 17:41:12 -0600 Message-ID: <922b09b2-24b1-6f3f-ce5b-7b07675d3637@dasyatidae.com> References: <87tufmjyai.fsf@gnus.org> <87lf0nr2b4.fsf@gnus.org> <87ee6f5wmy.fsf@telefonica.net> <87czlzqy3a.fsf@gnus.org> <874k7bqptr.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36334"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Cc: emacs-devel@gnu.org To: Lars Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Dec 15 00:43:09 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mxHRr-0009HP-Ub for ged-emacs-devel@m.gmane-mx.org; Wed, 15 Dec 2021 00:43:08 +0100 Original-Received: from localhost ([::1]:46738 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxHRq-0002ln-KP for ged-emacs-devel@m.gmane-mx.org; Tue, 14 Dec 2021 18:43:06 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:47766) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxHQ6-0001O8-I6 for emacs-devel@gnu.org; Tue, 14 Dec 2021 18:41:18 -0500 Original-Received: from out1-smtp.messagingengine.com ([66.111.4.25]:47453) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxHQ3-0001zd-At for emacs-devel@gnu.org; Tue, 14 Dec 2021 18:41:18 -0500 Original-Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 7BE185C01BF; Tue, 14 Dec 2021 18:41:14 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Tue, 14 Dec 2021 18:41:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dasyatidae.com; h=message-id:date:mime-version:to:cc:references:from:subject :in-reply-to:content-type:content-transfer-encoding; s=fm3; bh=Z eP43YgQh7/v98zOp8eZo9eHSmw8R4yGV2NnesmEXcg=; b=bFMowKprva79yuGkp 6ODB2pCa1Pl8Sy0Dj+tpHnGoLpBjq0LsYcGn0YD1FjaNu0KjyaeV0KlDZfajmGaw 5gcTqqWZsD2VgaIpv+51YgiknfVfotMa80lO/x8k/DZVeojlSWZgorsSEX5SZ1y4 F8I+DyrcKD52LrO5yckMo+oPkYO1ubdDImVPRJcm0p3J5ztBoRORFYEXeenAsyqi ZVzziCyAUzz3tTOD9f4PlREF5WkftlUN4mehKcPG9q2mkxEhpxVZWkRMiVZ8uCQX QLmslIdHwlTeMi9dOVF+pMVreEMYlxXr3dPO9UOcU6H1SabU6lafUtLn7gmsl060 v5Pmg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=ZeP43YgQh7/v98zOp8eZo9eHSmw8R4yGV2NnesmEX cg=; b=dVOTogxfNGdNc5jNFoYSAj9NsIApQCxfYkQNkT4AzOd2HSBO0FueALDDv xiRHfqxFXPrXtk7ydHnkGP6EYgzmuNEyF7BPbyZVnLsV00L9qSFaSYUz+A3yxyit g61N0yrCcG2AmdPdW5SQBsKqOMmoeXCXwZOIXsgSvOvAiXwjgR7U9NLwEBRprmfz cK6n8sLRL3YbXApeYSdLp1r9Tmg3+FWnDmIgRIVtCUd2JNFLcWG7adekaDx1pm4d sFsvZkbqQvwXEo1ZbNiKVz8+pV4Ql264CjifrdTifIsfn8cDS2ttFNDU2fu1xjyc 7YgcJKMEmQ6EGZTe9y5XxoWqTbt3Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrledugddtiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefkffggfgfvfhfhufgjtgfgsehtjeertddtfeejnecuhfhrohhmpeftohgsihhn ucfvrghrshhighgvrhcuoehrthhtsegurghshigrthhiuggrvgdrtghomheqnecuggftrf grthhtvghrnhepkeetjeffheejgeejgfdvudeuffeiudfhieffgeetuefgheeigeetiefh ueegteffnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh eprhhtthesuggrshihrghtihgurggvrdgtohhm X-ME-Proxy: Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 14 Dec 2021 18:41:13 -0500 (EST) Content-Language: en-US-large In-Reply-To: <874k7bqptr.fsf@gnus.org> Received-SPF: pass client-ip=66.111.4.25; envelope-from=rtt@dasyatidae.com; helo=out1-smtp.messagingengine.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.962, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281952 Archived-At: Lars Ingebrigtsen wrote: > MEMORY seems inappropriate, since several instances will be using the > file at the same time. I'm not sure what the TRUNCATE would bring us. MEMORY doesn't disable the use of exclusive file locking during write transactions. Several concurrent instances successfully committing or rolling back in MEMORY mode will have the same consistency constraints. The difference is in crash recovery; if one instance crashes hard in the middle of a transaction, it's "restore the file from backup" time. Whether that's okay depends on your surrounding strategy, but it feels similar to a subset of the lockfile strategies Emacs already considers okay. TRUNCATE is just slightly less operations than DELETE in the case where you do want to have a full rollback journal and don't care about the directory looking tidy. Hmm. What pattern/frequency/relative-frequency of reads/writes is expected to be common for these variables? And in particular, what about read- modify-write operations? Will applications tend to expect these to be atomic (deliberately or otherwise)? > The `files' backup doesn't lock anything, because it just wants whatever > instance manages to write the file last to do so. But I was pondering > writing to a temp file in the same directory and doing a rename instead. Yeah, overwrites are still non-atomic in the easiest case. :-) >> - Having auto_vacuum be FULL similarly seems like a waste, adding a bunch of >> extra work to each transaction commit. > > Likewise, I didn't see any difference between FULL and INCREMENTAL here, > and FULL means less work, so I left it at FULL. > >> - Given that people have expressed concerns over in-place modification leaving >> unwanted traces of old data, setting secure_delete to ON may be useful here. > > I thought the vacuum was supposed to take care of most of that? Auto-vacuuming only makes sure free pages are deallocated so that the database file shrinks as data is removed, even in FULL mode. It does not necessarily overwrite stale data with zeros, and as mentioned in the documentation, it can actually make fragmentation worse by being too aggressive with moving pages around. Normally I would leave it off. If you want periodic maintenance of the DB file, there may also be application-level maintenance that would be useful, as I extrapolate it? Depending on what's stored in these variables, clearing out stale entries and the like may want to be done at an elisp level. Leaving cleanup+VACUUM to be done explicitly on timer or user request and then having a hook for the non-DB-format-related cleanup might be workable? Not sure. >> - Setting trusted_schema to OFF is recommended for new applications. > > I'm not sure that's relevant for this usag? It cheaply removes a little chunk of potentially surprising behaviors. But you may be right in a sense---it depends on whether elisp assumes it can trust values read from the store. If so, then the entire DB is just in the same "may do anything" trusted zone as the rest of .emacs.d, I suppose? -RTT