From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 2IdoHr2/zF8wRAAA0tVLHw (envelope-from ) for ; Sun, 06 Dec 2020 11:25:49 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id mMsxGr2/zF+wJwAAB5/wlQ (envelope-from ) for ; Sun, 06 Dec 2020 11:25:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 416FE9403AA for ; Sun, 6 Dec 2020 11:25:49 +0000 (UTC) Received: from localhost ([::1]:43686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1klsAm-0004ht-5Z for larch@yhetil.org; Sun, 06 Dec 2020 06:25:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58994) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kls5B-00029o-Qv for guix-devel@gnu.org; Sun, 06 Dec 2020 06:20:01 -0500 Received: from mira.cbaines.net ([212.71.252.8]:38440) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kls59-0005Er-Ky; Sun, 06 Dec 2020 06:20:01 -0500 Received: from localhost (188.28.112.220.threembb.co.uk [188.28.112.220]) by mira.cbaines.net (Postfix) with ESMTPSA id AD5B127BC00; Sun, 6 Dec 2020 11:19:57 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 496e90f8; Sun, 6 Dec 2020 11:19:55 +0000 (UTC) User-agent: mu4e 1.4.13; emacs 27.1 From: Christopher Baines To: guix-devel@gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= , Mathieu Othacehe Subject: Cuirass WAL size issues Date: Sun, 06 Dec 2020 11:19:52 +0000 Message-ID: <87im9fqizr.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -3.90 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 416FE9403AA X-Spam-Score: -3.90 X-Migadu-Scanner: ns3122888.ip-94-23-21.eu X-TUID: YsiBMjUHHnob --=-=-= Content-Type: text/plain Hey, I've been chasing some performance issues in the Guix Build Coordinator and thought that the size of the WAL file was a contributing factor. I believe SQLite checkpoints the WAL file after transactions commit, if the WAL is over 1000 pages in size. At least for the Guix Build Coordinator though, that didn't seem to be working/happening as the WAL file seemed to just grow and grow. I think I'm making some progress on that. It seems that if you don't reset or finalise statements after you're finished with them, bad things happen. I think one of these bad things is that WAL checkpointing stops happening. When I got that to happen in the Guix Build Coordinator, I'm no longer seeing errors when I try and manually run checkpoints now. Switching focus to Cuirass, I think it's got similar issues. I've pushed a few commits to this branch [1], the important one being [2] which rewrites a few procedures to call sqlite-reset when the statement is finished with, rather than calling it before starting to read from the statement. 1: https://git.cbaines.net/guix/cuirass/log/?h=sqlite-statement-reset-fixes 2: https://git.cbaines.net/guix/cuirass/commit/?h=sqlite-statement-reset-fixes&id=e09bc606f25bad11b39c9fae49139ac03b88697c If this makes sense, I can spend the time writing some changelog entries for the commits, and push them. What do you think? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl/MvlhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdCJQ//YK/2c29f5fvoPGbrqEj0JQAEReo7VJAs A7Vn3zlQ+FyHFDZWONshgpc9b88sH3Kos/Exn0BF2QyvpywrWGTFM+6UFXt7f/rx 5E/5/GS0Tof06QBpv1udenWgzjofJV3al3UIA6FPeDtXRJODtAB7SYgqtv7LrKpv nhvaXMEeINN5+L28bZlromwgy5ox2g38KM7tNzFo9C86Bz6Edi8dP393FX1ZR+cm mVMclhljM4TW2lvKYWnJxOkB9vIzEuDX043fivU2XeW/BqRO/fGHupOAv62aN9f9 pp0vLUbhtfHQyYBrWfbItf8GShCVmn355/eM7v51eBnNUBP/7JxzH70XoIpBAwPG TEjivzlbf+LrtOao0ynRp/8gfkIy9eHjwSjdwH2fqVGGuFqfpdo85Ugehd9SIvFG qsxvWj7Nb4QTB93ptkgNEzPoB16Lqnw7unELD6jnpWYlXcZjxAJuH2i4bm9AHl6e EqZCaPV8YOv4E6g3R1gxmHl2U0viFi2DI1U1cd7VbPGS77mZ0EkLSzvOquyDfB8j 9WykjW2JOxjETadj44sHWf/ZZnARYXE7yNRIANqj9E0NbIc14Vd8/h8fPefSMlym BYIqiDdQL4JithppETtfVuhnTx2Elc96YzrYwMgO0oxhUzGDU9sWVMpW9D31CkUl Q+Xo/u1Ed6c= =OwKf -----END PGP SIGNATURE----- --=-=-=--