From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id ePDwMiUk2V6xJgAA0tVLHw (envelope-from ) for ; Thu, 04 Jun 2020 16:41:09 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id KPTSLiUk2V7DTAAA1q6Kng (envelope-from ) for ; Thu, 04 Jun 2020 16:41:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 104169401AE for ; Thu, 4 Jun 2020 16:41:08 +0000 (UTC) Received: from localhost ([::1]:55962 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jgsvS-0003fY-AH for larch@yhetil.org; Thu, 04 Jun 2020 12:41:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jgsvO-0003bw-9t for guix-patches@gnu.org; Thu, 04 Jun 2020 12:41:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:35826) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jgsvO-0001lP-0Y for guix-patches@gnu.org; Thu, 04 Jun 2020 12:41:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jgsvN-0002NX-UF for guix-patches@gnu.org; Thu, 04 Jun 2020 12:41:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#41658] [PATCH] fixes / improvements for (guix store database) Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 04 Jun 2020 16:41:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41658 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Caleb Ristvedt Cc: 41658@debbugs.gnu.org Received: via spool by 41658-submit@debbugs.gnu.org id=B41658.15912888479108 (code B ref 41658); Thu, 04 Jun 2020 16:41:01 +0000 Received: (at 41658) by debbugs.gnu.org; 4 Jun 2020 16:40:47 +0000 Received: from localhost ([127.0.0.1]:47372 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jgsv9-0002Mq-3m for submit@debbugs.gnu.org; Thu, 04 Jun 2020 12:40:47 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40948) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jgsv7-0002Md-Ho for 41658@debbugs.gnu.org; Thu, 04 Jun 2020 12:40:46 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36129) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jgsv2-0001kc-1l; Thu, 04 Jun 2020 12:40:40 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=46088 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jgsv1-0006Um-D1; Thu, 04 Jun 2020 12:40:39 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87ftbernat.fsf@cune.org> Date: Thu, 04 Jun 2020 18:40:35 +0200 In-Reply-To: <87ftbernat.fsf@cune.org> (Caleb Ristvedt's message of "Tue, 02 Jun 2020 01:31:38 -0500") Message-ID: <87a71i943g.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: GYwf89EQy2dF Hi, Thanks for the thorough investigation and for the patches! Caleb Ristvedt skribis: > From cce653c590be1506e15044e445aa9805370ac759 Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Mon, 1 Jun 2020 18:50:07 -0500 > Subject: [PATCH 1/4] database: work around guile-sqlite3 bug preventing > statement reset > MIME-Version: 1.0 > Content-Type: text/plain; charset=3DUTF-8 > Content-Transfer-Encoding: 8bit > > guile-sqlite3 provides statement caching, making it unnecessary for sqlit= e to > keep re-preparing statements that are frequently used. Unfortunately it > doesn't quite emulate the semantics of sqlite_finalize properly, because = it > doesn't cause a commit if the statement being finalized is the last "acti= ve" > statement. We work around this by wrapping sqlite-finalize with our own > version that ensures sqlite-reset is called, which does The Right Thing= =E2=84=A2. > > * guix/store/database.scm (sqlite-finalize): new procedure that shadows t= he > sqlite-finalize from (sqlite3). Nice. It would be great if you could report it upstream (Danny and/or myself can then patch it directly in guile-sqlite3 and push out a release) and refer to the issue from here. We can have this patch locally in the meantime, unless it would break once the new guile-sqlite3 is out. WDYT? > From ee24ab21122b1c75a7d67d7062550e15e54ab62f Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Mon, 1 Jun 2020 19:21:43 -0500 > Subject: [PATCH 2/4] database: rewrite query procedures in terms of > with-statement. > > Most of our queries would fail to finalize their statements properly if s= qlite > returned an error during their execution. This resolves that, and also m= akes > them somewhat more concise as a side-effect. > > This also makes some small changes to improve certain queries where behav= ior > was strange or overly verbose. > > * guix/store/database.scm (call-with-statement): new procedure. > (with-statement): new macro. > (last-insert-row-id, path-id, update-or-insert, add-references): rewrit= e to > use with-statement. > (update-or-insert): factor last-insert-row-id out of the end of both > branches. > (add-references): remove pointless last-insert-row-id call. > > * .dir-locals.el (with-statement): add indenting information. LGTM! > From 7d34c27c33aed3e8a49b9796a62a8c19d352e653 Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Mon, 1 Jun 2020 21:43:14 -0500 > Subject: [PATCH 3/4] database: ensure update-or-insert is run within a > transaction > > update-or-insert can break if an insert occurs between when it decides wh= ether > to update or insert and when it actually performs that operation. Puttin= g the > check and the update/insert operation in the same transaction ensures tha= t the > update/insert will only succeed if no other write has occurred in the mid= dle. > > * guix/store/database.scm (call-with-savepoint): new procedure. > (update-or-insert): use call-with-savepoint to ensure the read and the > insert/update occur within the same transaction. That=E2=80=99s a bit beyond my understanding, but I think you can also push= this one. :-) Make sure =E2=80=9Cmake check TESTS=3Dtests/store-database.scm=E2=80=9D is = still happy. Thanks a lot! Ludo=E2=80=99.