From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Arthur Miller Newsgroups: gmane.emacs.devel Subject: Re: sqlite3 Date: Mon, 06 Dec 2021 15:35:07 +0100 Message-ID: References: <87tufmjyai.fsf@gnus.org> <875ys2lyt0.fsf@yahoo.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="13669"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: Po Lu , Lars Ingebrigtsen , emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Dec 06 15:42:29 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1muFCG-0003Nm-Mv for ged-emacs-devel@m.gmane-mx.org; Mon, 06 Dec 2021 15:42:28 +0100 Original-Received: from localhost ([::1]:38506 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1muFCF-0004aB-N3 for ged-emacs-devel@m.gmane-mx.org; Mon, 06 Dec 2021 09:42:27 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:54766) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muF5O-0000d1-Oh for emacs-devel@gnu.org; Mon, 06 Dec 2021 09:35:22 -0500 Original-Received: from mail-oln040092075055.outbound.protection.outlook.com ([40.92.75.55]:18991 helo=EUR04-VI1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muF5M-0004XZ-4p for emacs-devel@gnu.org; Mon, 06 Dec 2021 09:35:22 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kv4tf0xYqQRndx4aTPmZ55C+NdIMmwzC3mz3P9YjqRNgpZ3C1G8JrLHWiU5djseZlxy4GtUYCSDSk3b44/ZEnr136SpDmUk4GGaRiUIeirYo9ff2sK1FAJ/Hosn9qXP1jkd2y1fcod07f+QsE2MPPna1lW2bPCEx8UpFG88XtYdV3kNjdv9NOZ82yMHlFr2H6BoT4Ji12fGiKvqF5t9w51vh1ejj7OioHV+ldX+Rw0iwn5U3+j9UKkGl3evuj15qoU7QLnJ/auY3cIZIYKixoBj7FCGhGTJ23u+x6z5B/wL3tmTmh+FHi6nzZZiKKjcxICYS9hleS+QrAbqRI9ucmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gHt1qwmGv0IVrRFQxRh6e39kNl8laffstoEHVD8Oflc=; b=G7qsqtuGOJfhAMNcYpd8qJnst0vCSEnDW21MEbUV/qMqYTVh41wqjN2lst7QunffT3MhxaxWVH9ShduHH+2mhn6e4nq0MK6kzfr+lMkkw3uDpZBytGCIqycKqI4sDk83GD4Bi8vi8lDub8e4lohCGsKuU088nfRet3PEK5y9aJ3o+eQ1DTm7CquNwSUWoDp8oz9RO7/OkZ8pTI3Ogg7KhwNj8pzGxiWmWyXrfSUj1j1DKWPBlw0uoFCy9ACM4IOg6NsSdht4FyrDANGZkqfa/EQMWXZpNZQmSVvUfQd9IOUvVEDpdMBnnqD0BT6lSiponYkU7EEmJVaV5ugNh0bzQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gHt1qwmGv0IVrRFQxRh6e39kNl8laffstoEHVD8Oflc=; b=jtzJIU9cVtZ0dopyPFgvxInNIMlOX/v6Q62/KyNv6PJsXJzoDuakcFAAybBJKgQpEYcVX0TVC3UHDew+fBYKTxoP6/z10yNqTdSzpw0NGTNdnhZOAUcauHqXVgmt9xNCmkqnsKmvaEbTcdfwRMPcMSCYQ5ytvtpT3fhGAIPrsRPg9stRDCYQ8XvgZznwzwL8WMUjjOXSOgSbnzpY3+grWgtELKSaFQE7wLpfdy8TB0CapIO5nmPDyuZ/I05AO4nf+eG4nIRbpIBSWj6odlb8VheNe/2dthObPQ0Pfnkp/fupYrcaVQI4Jyi01VHCt94K6g/rgP8EC2thd+Cb9ZM2wA== Original-Received: from AM9PR09MB4977.eurprd09.prod.outlook.com (2603:10a6:20b:304::20) by AM0PR09MB2355.eurprd09.prod.outlook.com (2603:10a6:208:d6::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23; Mon, 6 Dec 2021 14:35:15 +0000 Original-Received: from AM9PR09MB4977.eurprd09.prod.outlook.com ([fe80::b012:5fd9:9447:d64a]) by AM9PR09MB4977.eurprd09.prod.outlook.com ([fe80::b012:5fd9:9447:d64a%4]) with mapi id 15.20.4755.021; Mon, 6 Dec 2021 14:35:15 +0000 In-Reply-To: (Arthur Miller's message of "Mon, 06 Dec 2021 15:13:03 +0100") X-TMN: [+6QROkdMaDjczzdx00kegNHyxeMJyVlX] X-ClientProxiedBy: GV3P280CA0103.SWEP280.PROD.OUTLOOK.COM (2603:10a6:150:8::8) To AM9PR09MB4977.eurprd09.prod.outlook.com (2603:10a6:20b:304::20) X-Microsoft-Original-Message-ID: <87ee6pajj8.fsf@live.com> X-MS-Exchange-MessageSentRepresentingType: 1 Original-Received: from pascal.homepc (81.232.177.30) by GV3P280CA0103.SWEP280.PROD.OUTLOOK.COM (2603:10a6:150:8::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4755.16 via Frontend Transport; Mon, 6 Dec 2021 14:35:14 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f703dd2a-e87d-4743-14da-08d9b8c59e1b X-MS-TrafficTypeDiagnostic: AM0PR09MB2355:EE_ X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5cloZUWCE5QVIuFXJf/naNneOthS7VKHd+xEKKX81zrMjtItbIwT89MpggHkV0EPwIXSr75CFSymClaK3ChCncSkQ9+1Hr6VDJNDQSa5hgCia3kfAFWcXVTpwY6HXTesOyvGnASpvpjPp4DTuyAMKm3IwEew5d2HKbBtla3juc9UVqzux+g0yDfZ22Qz5LZXtm/FlQMGufgOkS00c+ejUAyEPWqdORE3weAim/A7qu93wbRfhgRIwDiq4SLzoH8OGqSpLb5IodQ5EZX2cs6STJQ7SzqSb4glEDEob15tGxGwTFtAXZ/zLI+JQG+d5ocjEqiVA6kRzHK4IDldCCoTFXQyQyDHDp1EBi0VfsFu9B/nxF445WGpwgfQg4kppbi5NlGAlZhWeDVw/A/eZvyeTQ4MfZqcyqlK0NKUii8rYirkiLxv+r3xLcJvamYxv27V26j/5HMJvjH0nFNmTxVk4v4NuOi/r2RwlHKskgEr2WUUgg/RFR62G0EbHHnrFD2qy6NwwlvIBBKCphMCJrQ4G6qJ5fhfwXWCRPmPn7oSE2jZcqhJ00s2x0E/NaBESy3/1YKhw8yO3yOdR4tA7KuDiA== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YczftK96ff9/08JjQ8piMiWWzD3eIlDOnl4ctEFUMWhEGp7mfhwBntM6QRJt?= =?us-ascii?Q?MHfV70Drk1Dw3e820/+pc+fgufnUnbZOAhGAJPlUTuwsfOqmXE5LTYRemrpH?= =?us-ascii?Q?FycM6uJ9UBCMRzojpOf6j0jIcTIqMfp4u+eK9c+/azGcQ7CE4fDPTGUfb1Iw?= =?us-ascii?Q?UcGuhQOnJEF/u0BJfbZ7xUrjwN/kA2iG3fhoJr66BwiP8GgmfqtJG1UqfX9J?= =?us-ascii?Q?hAyPzZcan6adw1xzmS3GdOD9jxUFAzDv8xfT7bCZspc8O/V43dDt6NAD7n3U?= =?us-ascii?Q?oVVRZn2IgsZMYHdcEpMecO/mwDwUg3kUm+Haz8IYQj+/GyENAV6CGGegagFz?= =?us-ascii?Q?TFpFGwNBSjAu1zT+rL0s7t6dVnZGcr4B1N5cWx0dG4SZf6L8eUaty5l51vIz?= =?us-ascii?Q?mwsfWSgabijgv6Loky1onkrYM/Q+l2af1OHvJQn5gGhx3DI8QTUOI1AeEIma?= =?us-ascii?Q?GrLTZF56CcSv4+5kByQ8bhV/WIOCSwBs4/QdDP3ku4ZEmKG2Yyc2vnrIU1Nu?= =?us-ascii?Q?Ve7baGUkjw9Vqznm8syNWGiPc6TYBfPo99CLXy2vSpIrVyinNs77aJQPLMbf?= =?us-ascii?Q?9yCPBFl1Xe5dSVv0kOwnq8d76+zXjrQDk1/XIoHn0DD333pTFk8XicBtmruZ?= =?us-ascii?Q?Lr X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-64da6.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: f703dd2a-e87d-4743-14da-08d9b8c59e1b X-MS-Exchange-CrossTenant-AuthSource: AM9PR09MB4977.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2021 14:35:15.3403 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR09MB2355 Received-SPF: pass client-ip=40.92.75.55; envelope-from=arthur.miller@live.com; helo=EUR04-VI1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281122 Archived-At: Arthur Miller writes: > Stefan Monnier writes: > >>> I suggested hashmap serialzied to/from file. I used such approach myself for >>> key-value pairs. It worked fine for me, it was quite simple to read/write >>> it. Currently I am testing a thing, and I use just assoc list to read/write it >>> to a file, and it works fine for me too. >> >> The problem with this approach (which is the standard solution currently >> in ELisp) is that it works OK for simple cases but breaks down when: >> - there's a lot of data. >> - the data is modified by several Emacs sessions at the same time. >> This second case is what can happen with `ecomplete` or `savehist`. > > I wouldn't be surprised, if several processes are involved, if that is what you > mean with several Emacs sessions at the time, than some locking is needed of > course, and sqlite is a winner there. I believe. By the way, after second thought; sqlite locks entire database. If I remember well they advice against using sqlite db when concurrent write access is needed. Does Emacs already has possibility to lock files, or possibly a particular symbol for read/write access; if not, is such mechanism to much work to add? And about lots of data; sqlite mmaps db file, if I am not misstaken about, and Emacs has can mmap files right? How granular and usable is portable dumper? Is it possible to dump a hashmap or some other object to a file, and read in dump at runtime? If those two things would be possible, than there is a fast, multiprocess key-value store with already existing means. And I still think sqlite would be useful in core :). For example, easier importing/exporting with other applications, with some more, 'lispy' interface if we had it in core.